Zabbix – Why You No Audit?

Testing of new Zabbix items, triggers, actions, etc is always easier on a separate test instance, which is the reason why we have a few test Zabbix servers. These test servers are usually behind our firewall, but a few weeks ago we found that one test instance wasn’t. To make things even worse, it had the default admin credentials.  This wasn’t a big issue, because it was isolated from the rest of our hosts, but it was interesting what happened on that server.

The way we found out that the server was compromised was that it was using 100% CPU. The process which was using all the CPU was a process which we never seen before, nor did any of us ever configure it, and of course it was run by the zabbix user. We killed it instantly, and after some digging around we found out that the executable file was used as an agent for some data mining service on which you can rent computing power to do some tasks.

After more investigation we found that the executable was downloaded via cron job of the zabbix user, which basically downloaded the file from Github (account is now banned) and executed it. We knew everything was done through the Zabbix web interface, but we wanted to see how exactly was it done. The first thing we checked was the Zabbix audit log and this is how it looked:

As you can see it had a few failed login attempts and a few successful login/logout ones (which was us). There was not one successful login attempt or anything which wasn’t coming from our own IPs.

Next we checked the Apache access log and the log was filled with this:

45.76.203.110 - - [08/Nov/2016:19:31:12 +0100] "POST /api_jsonrpc.php HTTP/1.1" 200 68 "http://www.baidu.com" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
45.76.203.110 - - [08/Nov/2016:19:31:12 +0100] "POST /api_jsonrpc.php HTTP/1.1" 200 53 "http://www.baidu.com" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
45.76.203.110 - - [08/Nov/2016:19:31:13 +0100] "POST /api_jsonrpc.php HTTP/1.1" 200 178 "http://www.baidu.com" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
45.76.203.110 - - [08/Nov/2016:19:43:15 +0100] "POST /api_jsonrpc.php HTTP/1.1" 200 68 "http://www.baidu.com" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
45.76.203.110 - - [08/Nov/2016:19:43:16 +0100] "POST /api_jsonrpc.php HTTP/1.1" 200 53 "http://www.baidu.com" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
45.76.203.110 - - [08/Nov/2016:19:43:16 +0100] "POST /api_jsonrpc.php HTTP/1.1" 200 178 "http://www.baidu.com" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"

Which means the attacker used the Zabbix API to do the harm, but it was very strange to us that audit log didn’t report any changes. After more digging we found out that the attacker changed the Ping script with some commands which edited crontab of the zabbix user and ran this script against the Zabbix host. The attacker only needed two API calls to achieve this: script.update and script.execute, and those calls won’t be recorded anywhere.

In the mean time we destroyed this test Zabbix instance, but this made us curious. We knew from our previous experience that Zabbix didn’t record all actions in audit log, and it made us very frustrated, but we never checked what was logged and what wasn’t. After searching for bugs on support.zabbix.com we found ZBX-2815 which has a detailed list of actions which are not logged. This bug started having some activity recently, and as far as we can see, the plan is to fix it with Zabbix 3.4 release, which will be a welcome fix by all Zabbix users.

Leave a Reply