Testing of new Zabbix items, triggers, actions, etc is always easier on a separate test instance, which is the reason why we have a few test Zabbix servers. These test servers are usually behind our firewall, but a few weeks ago we found that one test instance wasn’t. To make things even worse, it had the default admin credentials. This wasn’t a big issue, because it was isolated from the rest of our hosts, but it was interesting what happened on that server.
The way we found out that the server was compromised was that it was using 100% CPU. The process which was using all the CPU was a process which we never seen before, nor did any of us ever configure it, and of course it was run by the zabbix user. We killed it instantly, and after some digging around we found out that the executable file was used as an agent for some data mining service on which you can rent computing power to do some tasks.
Let’s talk bugs. The important Zabbix bugs. What are those? The ones that have the most votes in the Zabbix issue tracker.
There are currently 1308 open bugreports. When we looked at this number back in November last year, it was a hundred less. That’s a pretty huge number, is everything bad? Not really, as some might be duplicates and some might be incorrect reports. Not many, though, as there’s constant grooming going on. Most of the remaining are valid bugreports, but not too critical – some are even as minor as an offset of a few pixels in some page. Still a bug, but something we can live with, mostly. We already looked at the top-voted bugreport, now is the time to glance at others same as we did with feature requests.
The bug must be still unfixed to be important. If a new version of Zabbix comes out and the server crashes for all the users, that is the most important bug. Until it is fixed, hopefully, soon.
But there are some long-standing bugs that linger around just below the “fix-it” surface – they’re not terrible enough to be fixed right away, and somewhat complicated usually. Such bugs can be around for many years, sometimes not even being fixed, but going away because a feature gets dropped completely. We’d need a way to measure which of all those known bugs is the most important. And there is a way to find out – same as with features, users can vote on bugreports. The bugreport with the most votes is titled deadlock between server and frontend.