JaM: PHP App Monitoring Made Easy
A few months back, we, at Kaltura, held an internal hackathon, this post will cover my hackathon project: a PHP monitoring system called JaM.
The Need
I started thinking about JaM following a conversation I had with a colleague of mine. His team had a fatal error in their code because the person who wrote it tested it on PHP 5.4, whilst the production site runs on PHP 5.3. He told me his team looked at the logs and couldn’t find any entries indicating an issue.
The reason for that was someone set the error_log PHP directive to a different log location, so the error did not go to Apache’s error log (which is where the team was expecting to find it).
So, I started thinking about creating a solution that does not require log parsing at all, but instead plugs directly into the Zend Engine and overrides zend_error_cb() to report the errors by email and other methods and only then write to the error log, which is what the original zend_error_cb() function does.
I already knew of one such solution by Zend, called Zend Monitor. But Zend Monitor is not free software, and has dependencies on other Zend proprietary components. I wanted a FOSS solution.
Yet another example of how amazing FOSS is
In my opinion, the first thing you need to do when you get an idea for a new project is check whether or not it already exists, and that’s exactly what I set out to do. It wasn’t very easy to find, but using a combination of “zend_error_cb()” and other keywords, I got to PHP Aware by Mikko Koppanen.
Looking at the project’s README, it felt like it would be a great place to start from, as it already did most of what I wanted to achieve and was written in a very modular way, which is always good.
When I tried to build it, however, it did not compile. That was because it was last worked on in 2011 and the PHP engine has changed quite a bit since.
And so I ported it to support newer PHP versions, added the ElasticSearch backend as per my original plan, and submitted a pull request.
I reached out to Mikko asking if he’d like to merge it. Mikko said that while he is happy someone has taken interest in the project, he was no longer working on it or PHP in general, for that matter, and that he was more than happy for me to take ownership.
And this is how JaM was born.
How does JaM work?
JaM is implemented as a PHP extension which hooks directly into the Zend Engine and overrides its default error handling function: zend_error_cb().
This allows JaM to intercept all PHP events (fatal errors, warnings, notices, basically everything listed here: https://php.net/manual/en/errorfunc.constants.php) so that they can then be sent to multiple, configurable, storage backends.
This means that JaM can be configured to send all your events, or just a subset of them, to your ElasticSearch server, or send SNMP events to your general monitoring system, or email the dev team or all of the above.
Because the ElasticSearch implementation is done by sending the event formatted as JSON over HTTP, it can actually be used by multiple services and not necessarily by an ElasticSearch server.
If your system knows how to process data formatted as JSON, it can easily integrate with JaM.
Note that JaM will operate transparently on ANY PHP ENV where it is loaded. This means you can use to monitor all your PHP apps, Kaltura included, of course, without having to make any code changes to your application or logging.
Supported storage backends
– ElasticSearch
– Email
– SNMP
– Files
The fact that JaM has multiple backends allows you to configure it to send only a subset of events to one backend while logging all events to another.
For instance, think about email alerts. Many applications trigger a lot of notices you’d like to solve but are not critical for the application’s operation.
You therefore might not wish to get them by email, since there will be a lot of them and no immediate action will be taken to fix them.
You can configure JaM to send only what you consider to be critical errors by mail for an immediate response, and log the rest to ElasticSearch or any other backend for later analysis and resolution.
If you do choose to utilize ElasticSearch, you can also enjoy the added benefit of easily visualizing your events using Kibana. This way, you can produce nice graphs out of your PHP errors, without having to write so much as one line of code.
Once JaM is done sending the events to the storage backends, it returns control to the original zend_error_cb() function and the event is logged to the log file, be it the Apache error log or a custom PHP log.
Installing and configuring JaM
JaM if a FOSS project licensed under AGPLv3.
It’s git repo can be found here:
https://github.com/kaltura/jam
JaM is also shipped as both deb and RPM packages, installation and configuration instructions can be found here: https://github.com/kaltura/jam#basic-setup
Call for contributors
As mentioned before, JaM is a FOSS project and as such, we are always happy to accept contributions in the form of pull requests, bug reports and documentation additions.
There is also a PHP7 branch in progress which requires some additional fixes and of course, JaM is built so that it is rather easy to implement additional storage backends.
For those interested in contributing additional backend support, please see: https://github.com/kaltura/jam#creating-additional-storage-backends
Just one more note before we wrap up, JaM was chosen for a session at LinuxFest 2016, which will take place in Bellingham, WA, US on April 23rd & 24th, if you are attending, I’d love to see you there and answer additional questions.
Please see session abstract here.
Thanks for reading!
