Zabbix agent is easy to extend for data collection with a feature called userparameters. We figured out how they work in the article Create your own items – extend the agent with userparameters. Unfortunately, userparameters sometimes don’t work right away, and not always the cause is obvious. In this article we will explore the most common issues in more detail and learn to use simple methods to debug Zabbix agent userparameters.
While some potential problems were mentioned in the previous article, let’s go through them and other issues in more detail.
A fairly common issue is permissions. Do filesystem permissions allow zabbix user to run the script? Do filesystem permissions allow the script to access configuration or data files it needs? What about devices (files in /dev/), are all the permissions correct?
Filesystem permission issues are usually easy to test – try to run the command as the zabbix user. Keep in mind that permissions must allow access all along the path. For example, if the permissions are like this:
drw------- 3 root root 4096 jūn 17 21:40 /data/ drwxr-xr-x 3 root root 4096 jūn 17 21:40 /data/config drw------- 2 zabbix zabbix 4096 jūn 17 21:40 /data/config/zabbix
User zabbix won’t be able to access directory /data/config/zabbix . Even though it is the owner of that directory, it has no permissions to access /data .
Closely related to the permission issues, there are a couple of flags for the Zabbix agent that sort of allow testing item keys.
$ zabbix_agentd --help ... -p --print Print known items and exit -t --test <item key> Test specified item and exit
These sound perfect for testing userparameters – -p would show whether it is there and -t would test that item alone. Except that my general suggestion about them is:
Never use the -t and -p parameters for the Zabbix agent.
…unless you perfectly understand how they are different from querying a running agent daemon with zabbix_get and how the server queries that agent. Many, many users have been confused and spent a lot of time chasing the wrong things. Until you know your way around those two parameters, avoid them.
Very old versions of Zabbix agent did not support userparameters in the -t and -p flags at all, but there probably are few users with that old version of the agent.
Oh, hello, SELinux
A subset of permission issues – things SELinux might block. If possible, the fastest way to confirm is to set SELinux to permissive and try again. If it works, debug that angle and make it work with SELinux in the enforcing mode.
The environment trap
Another very common problem is related to environment variables. See, Zabbix does not initialise the environment in any way – it just launches the userparameter command and that’s about it. Even if you add variables in the profile or rc files for the zabbix user, those won’t be used unless you explicitly source those files. If your script relies on any environment variables, make sure to set those explicitly.
What variables? Any. Including HOME. For example, if your userparameter uses mysql client utility and you store credentials in ~/.my.cnf like you should, Zabbix agent won’t know where to find the .my.cnf file.
You might suspect it is some environment variable, but not know for sure which one. One way to start about that would be comparing the variables when the script is run manually and works, and when it is run by the Zabbix agent and fails. Common commands to show current variables are set and env. The main functional difference is that set will show local variables, while env won’t – env will only show variables that will be passed to subprocesses. For example, if we have variables like these:
$ variable_a=1 $ export variable_b=2
set will see both, while env will see variable_b only
A subtle difference comes in play when using initscripts. If Zabbix agent has been started upon system boot, it will most likely have different environment compared to agent being started manually – when started manually, variables from the user’s shell will also be inherited. This difference would most likely not be there when using SystemD. SystemD might affect some of the environment variables, including HOME.
What the shell?
When launching userparameter commands, Zabbix simply passes those to /bin/sh . What is that on your system? It could be bash, it could be dash, it could be ash or something else. Make sure you know which one it is. Keep in mind that some shells will act differently, depending on how hey are invoked (for example, bash will work differently when invoked as /bin/sh ).
If it times out, it’s bad
Preferably, the scripts or commands userparameters call should return very quickly. The default timeout on the agent is 3 seconds… and you should not change it. If your script takes longer than a second or two, userparameters might not be the best solution.
Ways to debug
The above identifies the most common problems with userparameters. What to do when it’s not clear what is wrong? Debugging time. The easiest methods are a bit hackish, but very efficient and not really unique to Zabbix.
Check the logfile
While recent versions of the Zabbix agent often return error message, still check the agent logfile. It might have additional clues.
Run the script manually
Assuming you are using a script for the userparameter, try running that script manually as the zabbix user. It will often provide extra information that will be helpful to figure out what fails. That is also the fastest way to check permission-related issues.
Log important detail
If the script fails in a fairly mysterious way, log interesting bits to a file. Parameters that you intend to pass to some command, payload – anything that you are not 200% sure about.
Bisect the script
If everything else fails, split the script in small bits, simplify it. Similar to logging important bits, start from the final result and go back. At some point the intermediary data won’t match the expectations. If needed, trim the script down to echo 1 like in the first article – maybe the agent daemon configuration file entry is wrong, or located in the wrong file. Instead of spending time on the script detail, trimming the script down will reveal the faulty area sooner.
Zabbix agent userparameters are very flexible and not that hard to debug. If you are familiar with the most common problems, you should have little trouble – and you should be familiar with those problems now.
Seen another issue with userparameters, or have some horror story about them? Please share that with us.