Category Archives: Troubleshooting

Debugging Zabbix userparameters

Zabbix agent is easy to extend for data collection with a feature called userparameters. We figured out how they work in the article Create your own items – extend the agent with userparameters. Unfortunately, userparameters sometimes don’t work right away, and not always the cause is obvious. In this article we will explore the most common issues in more detail and learn to use simple methods to debug Zabbix agent userparameters.

Common problems

While some potential problems were mentioned in the previous article, let’s go through them and other issues in more detail.


A fairly common issue is permissions. Do filesystem permissions allow zabbix user to run the script? Do filesystem permissions allow the script to access configuration or data files it needs? What about devices (files in /dev/), are all the permissions correct?

Filesystem permission issues are usually easy to test – try to run the command as the zabbix user. Keep in mind that permissions must allow access all along the path. For example, if the permissions are like this:

drw------- 3 root root 4096 jūn 17 21:40 /data/
drwxr-xr-x 3 root root 4096 jūn 17 21:40 /data/config
drw------- 2 zabbix zabbix 4096 jūn 17 21:40 /data/config/zabbix

User zabbix won’t be able to access directory /data/config/zabbix . Even though it is the owner of that directory, it has no permissions to access /data .

Don’t tee-pee

Closely related to the permission issues, there are a couple of flags for the Zabbix agent that sort of allow testing item keys.

$ zabbix_agentd --help
  -p --print                 Print known items and exit
  -t --test <item key>       Test specified item and exit

These sound perfect for testing userparameters – -p would show whether it is there and -t would test that item alone. Except that my general suggestion about them is:

Never use the -t and -p parameters for the Zabbix agent.

…unless you perfectly understand how they are different from querying a running agent daemon with zabbix_get and how the server queries that agent. Many, many users have been confused and spent a lot of time chasing the wrong things. Until you know your way around those two parameters, avoid them.

Very old versions of Zabbix agent did not support userparameters in the -t and -p flags at all, but there probably are few users with that old version of the agent.

Never tee-pee.

Oh, hello, SELinux

A subset of permission issues – things SELinux might block. If possible, the fastest way to confirm is to set SELinux to permissive and try again. If it works, debug that angle and make it work with SELinux in the enforcing mode.

The environment trap

Another very common problem is related to environment variables. See, Zabbix does not initialise the environment in any way – it just launches the userparameter command and that’s about it. Even if you add variables in the profile or rc files for the zabbix user, those won’t be used unless you explicitly source those files. If your script relies on any environment variables, make sure to set those explicitly.

What variables? Any. Including HOME. For example, if your userparameter uses mysql client utility and you store credentials in ~/.my.cnf like you should, Zabbix agent won’t know where to find the .my.cnf file.

You might suspect it is some environment variable, but not know for sure which one. One way to start about that would be comparing the variables when the script is run manually and works, and when it is run by the Zabbix agent and fails. Common commands to show current variables are set and env. The main functional difference is that set will show local variables, while env won’t – env will only show variables that will be passed to subprocesses. For example, if we have variables like these:

$ variable_a=1
$ export variable_b=2

set will see both, while env will see variable_b only

A subtle difference comes in play when using initscripts. If Zabbix agent has been started upon system boot, it will most likely have different environment compared to agent being started manually – when started manually, variables from the user’s shell will also be inherited. This difference would most likely not be there when using SystemD. SystemD might affect some of the environment variables, including HOME.

What the shell?

When launching userparameter commands, Zabbix simply passes those to /bin/sh . What is that on your system? It could be bash, it could be dash, it could be ash or something else. Make sure you know which one it is. Keep in mind that some shells will act differently, depending on how hey are invoked (for example, bash will work differently when invoked as /bin/sh ).

If it times out, it’s bad

Preferably, the scripts or commands userparameters call should return very quickly. The default timeout on the agent is 3 seconds… and you should not change it. If your script takes longer than a second or two, userparameters might not be the best solution.

Ways to debug

The above identifies the most common problems with userparameters. What to do when it’s not clear what is wrong?  Debugging time. The easiest methods are a bit hackish, but very efficient and not really unique to Zabbix.

Check the logfile

While recent versions of the Zabbix agent often return error message, still check the agent logfile. It might have additional clues.

Run the script manually

Assuming you are using a script for the userparameter, try running that script manually as the zabbix user. It will often provide extra information that will be helpful to figure out what fails. That is also the fastest way to check permission-related issues.

Log important detail

If the script fails in a fairly mysterious way, log interesting bits to a file. Parameters that you intend to pass to some command, payload – anything that you are not 200% sure about.

Bisect the script

If everything else fails, split the script in small bits, simplify it. Similar to logging important bits, start from the final result and go back. At some point the intermediary data won’t match the expectations. If needed, trim the script down to echo 1 like in the first article – maybe the agent daemon configuration file entry is wrong, or located in the wrong file. Instead of spending time on the script detail, trimming the script down will reveal the faulty area sooner.


Zabbix agent userparameters are very flexible and not that hard to debug. If you are familiar with the most common problems, you should have little trouble – and you should be familiar with those problems now.

Seen another issue with userparameters, or have some horror story about them? Please share that with us.

Still wondering about missing notifications in 2017?

If Zabbix keeps on surprising you with its notifications, you might want to try the Action Simulator! The Action Simulator is a community patch that helps you to figure out whether your actions really do as you intend. It first came out for Zabbix 2.0 in 2013 and was downloaded by hundreds of users from all around the world.

The following article gives a brief introduction to the Action Simulator and explains the challenges of developing it for Zabbix 3.2.

Continue reading Still wondering about missing notifications in 2017?

Zabbix – Why You No Audit?

Testing of new Zabbix items, triggers, actions, etc is always easier on a separate test instance, which is the reason why we have a few test Zabbix servers. These test servers are usually behind our firewall, but a few weeks ago we found that one test instance wasn’t. To make things even worse, it had the default admin credentials.  This wasn’t a big issue, because it was isolated from the rest of our hosts, but it was interesting what happened on that server.

The way we found out that the server was compromised was that it was using 100% CPU. The process which was using all the CPU was a process which we never seen before, nor did any of us ever configure it, and of course it was run by the zabbix user. We killed it instantly, and after some digging around we found out that the executable file was used as an agent for some data mining service on which you can rent computing power to do some tasks.

Continue reading Zabbix – Why You No Audit?

Zabbix stopped sending out alerts, HALP!

Zabbix was this reliable friend, always sending you an email, SMS or both when something went down. It sometimes sent you a lot of emails, but you never got angry at Zabbix about that – it was just eager to help you, make sure you did not miss the weekly disaster. But then… last week… Zabbix did not send you an SMS. It did not send you an email. It did not telepathically inform you. But things were DOWN. Server was not RESPONDING.

A fat cat at a table with paws up

Zabbix knew about this. As you review the data, sitting in a dark room, the graphs clearly show the downtime. But there was-no-alert. How is that possible? Wait, what, this is impossible. You can see on the glowing screen that the main action, a crucial piece in getting those alerts, is disabled. That just cannot be, as nobody, NOBODY would ever disable that. How, oh how. Why, oh why.

Zabbix frontend row, showing a disabled action

Continue reading Zabbix stopped sending out alerts, HALP!