Debugging Zabbix userparameters

Zabbix agent is easy to extend for data collection with a feature called userparameters. We figured out how they work in the article Create your own items – extend the agent with userparameters. Unfortunately, userparameters sometimes don’t work right away, and not always the cause is obvious. In this article we will explore the most common issues in more detail and learn to use simple methods to debug Zabbix agent userparameters.

Common problems

While some potential problems were mentioned in the previous article, let’s go through them and other issues in more detail.

Permissions

A fairly common issue is permissions. Do filesystem permissions allow zabbix user to run the script? Do filesystem permissions allow the script to access configuration or data files it needs? What about devices (files in /dev/), are all the permissions correct?

Filesystem permission issues are usually easy to test – try to run the command as the zabbix user. Keep in mind that permissions must allow access all along the path. For example, if the permissions are like this:

drw------- 3 root root 4096 jūn 17 21:40 /data/
drwxr-xr-x 3 root root 4096 jūn 17 21:40 /data/config
drw------- 2 zabbix zabbix 4096 jūn 17 21:40 /data/config/zabbix

User zabbix won’t be able to access directory /data/config/zabbix . Even though it is the owner of that directory, it has no permissions to access /data .

Don’t tee-pee

Closely related to the permission issues, there are a couple of flags for the Zabbix agent that sort of allow testing item keys.

$ zabbix_agentd --help
...
  -p --print                 Print known items and exit
  -t --test <item key>       Test specified item and exit

These sound perfect for testing userparameters – -p would show whether it is there and -t would test that item alone. Except that my general suggestion about them is:

Never use the -t and -p parameters for the Zabbix agent.

…unless you perfectly understand how they are different from querying a running agent daemon with zabbix_get and how the server queries that agent. Many, many users have been confused and spent a lot of time chasing the wrong things. Until you know your way around those two parameters, avoid them.

Very old versions of Zabbix agent did not support userparameters in the -t and -p flags at all, but there probably are few users with that old version of the agent.

Never tee-pee.

Oh, hello, SELinux

A subset of permission issues – things SELinux might block. If possible, the fastest way to confirm is to set SELinux to permissive and try again. If it works, debug that angle and make it work with SELinux in the enforcing mode.

The environment trap

Another very common problem is related to environment variables. See, Zabbix does not initialise the environment in any way – it just launches the userparameter command and that’s about it. Even if you add variables in the profile or rc files for the zabbix user, those won’t be used unless you explicitly source those files. If your script relies on any environment variables, make sure to set those explicitly.

What variables? Any. Including HOME. For example, if your userparameter uses mysql client utility and you store credentials in ~/.my.cnf like you should, Zabbix agent won’t know where to find the .my.cnf file.

You might suspect it is some environment variable, but not know for sure which one. One way to start about that would be comparing the variables when the script is run manually and works, and when it is run by the Zabbix agent and fails. Common commands to show current variables are set and env. The main functional difference is that set will show local variables, while env won’t – env will only show variables that will be passed to subprocesses. For example, if we have variables like these:

$ variable_a=1
$ export variable_b=2

set will see both, while env will see variable_b only

A subtle difference comes in play when using initscripts. If Zabbix agent has been started upon system boot, it will most likely have different environment compared to agent being started manually – when started manually, variables from the user’s shell will also be inherited. This difference would most likely not be there when using SystemD. SystemD might affect some of the environment variables, including HOME.

What the shell?

When launching userparameter commands, Zabbix simply passes those to /bin/sh . What is that on your system? It could be bash, it could be dash, it could be ash or something else. Make sure you know which one it is. Keep in mind that some shells will act differently, depending on how hey are invoked (for example, bash will work differently when invoked as /bin/sh ).

If it times out, it’s bad

Preferably, the scripts or commands userparameters call should return very quickly. The default timeout on the agent is 3 seconds… and you should not change it. If your script takes longer than a second or two, userparameters might not be the best solution.

Ways to debug

The above identifies the most common problems with userparameters. What to do when it’s not clear what is wrong?  Debugging time. The easiest methods are a bit hackish, but very efficient and not really unique to Zabbix.

Check the logfile

While recent versions of the Zabbix agent often return error message, still check the agent logfile. It might have additional clues.

Run the script manually

Assuming you are using a script for the userparameter, try running that script manually as the zabbix user. It will often provide extra information that will be helpful to figure out what fails. That is also the fastest way to check permission-related issues.

Log important detail

If the script fails in a fairly mysterious way, log interesting bits to a file. Parameters that you intend to pass to some command, payload – anything that you are not 200% sure about.

Bisect the script

If everything else fails, split the script in small bits, simplify it. Similar to logging important bits, start from the final result and go back. At some point the intermediary data won’t match the expectations. If needed, trim the script down to echo 1 like in the first article – maybe the agent daemon configuration file entry is wrong, or located in the wrong file. Instead of spending time on the script detail, trimming the script down will reveal the faulty area sooner.

Conclusion

Zabbix agent userparameters are very flexible and not that hard to debug. If you are familiar with the most common problems, you should have little trouble – and you should be familiar with those problems now.

Seen another issue with userparameters, or have some horror story about them? Please share that with us.

Auto-registering Linux agents with TLS PSK encryption

Zabbix supports Linux agent auto-registration and it’s a well documented process in the Zabbix manual. However, it’s not really straightforward to mass provision Zabbix agents on hundreds of servers if you want to have Zabbix agent <-> Zabbix server communication encrypted. At least not without some form of additional scripted step in your installation process. For large deployments I usually use Ansible, although this article just briefly covers that part and I’ll mostly focus on how to make sure your agents get registered for encrypted communication.

Continue reading Auto-registering Linux agents with TLS PSK encryption

Create your own items – extend the agent with userparameters

Zabbix supports many different ways of monitoring, including agentless, SNMP and IPMI. Zabbix also provides a monitoring agent, which has a great set of built-in items for monitoring diskspace, processes, memory usage and many other things.

While the list of built-in items is growing with each release, there will always be something else we will want to monitor. Luckily, Zabbix agent is very easy to extend with new items by using a feature called userparameters. Zabbix userparameters are commands that the agent runs and expects an item value to be returned.

Continue reading Create your own items – extend the agent with userparameters

These are not the applications you are looking for

Applications – software applications – mean a specific thing in IT. Usually, that’s a user-oriented piece of software. A web browser, word processor, game. Lately, mobile applications have somewhat lowered the bar, even down to I Am Rich applications.

Screenshot of the "I Am Rich" application, showing the price of USD 999.99
“I Am Rich” sale screen (https://en.wikipedia.org/wiki/File:I_Am_Rich_sale_screen.png)

And then there’s Zabbix with it’s own definition for applications. So what are applications in Zabbix?

Continue reading These are not the applications you are looking for

Still wondering about missing notifications in 2017?

If Zabbix keeps on surprising you with its notifications, you might want to try the Action Simulator! The Action Simulator is a community patch that helps you to figure out whether your actions really do as you intend. It first came out for Zabbix 2.0 in 2013 and was downloaded by hundreds of users from all around the world.

The following article gives a brief introduction to the Action Simulator and explains the challenges of developing it for Zabbix 3.2.

Continue reading Still wondering about missing notifications in 2017?

Zabbix – Why You No Audit?

Testing of new Zabbix items, triggers, actions, etc is always easier on a separate test instance, which is the reason why we have a few test Zabbix servers. These test servers are usually behind our firewall, but a few weeks ago we found that one test instance wasn’t. To make things even worse, it had the default admin credentials.  This wasn’t a big issue, because it was isolated from the rest of our hosts, but it was interesting what happened on that server.

The way we found out that the server was compromised was that it was using 100% CPU. The process which was using all the CPU was a process which we never seen before, nor did any of us ever configure it, and of course it was run by the zabbix user. We killed it instantly, and after some digging around we found out that the executable file was used as an agent for some data mining service on which you can rent computing power to do some tasks.

Continue reading Zabbix – Why You No Audit?

Zabbix bug TOP20, March 2017

Let’s talk bugs. The important Zabbix bugs. What are those? The ones that have the most votes in the Zabbix issue tracker.

TOP3 Zabbix bugreports from the table below in the article

There are currently 1308 open bugreports. When we looked at this number back in November last year, it was a hundred less. That’s a pretty huge number, is everything bad? Not really, as some might be duplicates and some might be incorrect reports. Not many, though, as there’s constant grooming going on. Most of the remaining are valid bugreports, but not too critical – some are even as minor as an offset of a few pixels in some page. Still a bug, but something we can live with, mostly. We already looked at the top-voted bugreport, now is the time to glance at others same as we did with feature requests.

Continue reading Zabbix bug TOP20, March 2017

The most important bug in Zabbix

How do you determine which bugs are important ?

The bug must be still unfixed to be important. If a new version of Zabbix comes out and the server crashes for all the users, that is the most important bug. Until it is fixed, hopefully, soon.

But there are some long-standing bugs that linger around just below the “fix-it” surface – they’re  not terrible enough to be fixed right away, and somewhat complicated usually. Such bugs can be around for many years, sometimes not even being fixed, but going away because a feature gets dropped completely. We’d need a way to measure which of all those known bugs is the most important. And there is a way to find out – same as with features, users can vote on bugreports. The bugreport with the most votes is titled deadlock between server and frontend.

Jira screenshot, showing "deadlock between server and frontend" issue title

Continue reading The most important bug in Zabbix

Created a trigger and it disappeared? Read on for a revelation!

So you had a cluster monitored. As is common with clusters, you wanted to have some cluster-wide parameter adding. Average CPU load, number of nodes online – something not tied to a single cluster node, thus you created a special host to denote the whole cluster. Then you went to that host, clicked “Create trigger”, specified all the items on individual cluster hosts, clicked “Add”… and the trigger was not there. Mysteriously missing.

An empty Zabbix trigger list with a red "NO TRIGGERS" text added

Oh, wait. That trigger actually appeared on all the cluster hosts. Is this a bug?

Continue reading Created a trigger and it disappeared? Read on for a revelation!

Visiting the Open Source Monitoring Conference 2016, Part 4

Returning to the events of the Open Source Monitoring Conference 2016, Avishai Ish-Shalom discussed an engineer’s approach to monitoring. David Hustace from OpenNMS told positive stories about this true-opensource monitoring tool.

OpenNMS mascot with a t-shirt saying "The kiwi bird is a direct descendant of the tyrannosaurus rex. rawr."

Continue reading Visiting the Open Source Monitoring Conference 2016, Part 4