All posts by Richlv

Debugging Zabbix userparameters

Zabbix agent is easy to extend for data collection with a feature called userparameters. We figured out how they work in the article Create your own items – extend the agent with userparameters. Unfortunately, userparameters sometimes don’t work right away, and not always the cause is obvious. In this article we will explore the most common issues in more detail and learn to use simple methods to debug Zabbix agent userparameters.

Common problems

While some potential problems were mentioned in the previous article, let’s go through them and other issues in more detail.


A fairly common issue is permissions. Do filesystem permissions allow zabbix user to run the script? Do filesystem permissions allow the script to access configuration or data files it needs? What about devices (files in /dev/), are all the permissions correct?

Filesystem permission issues are usually easy to test – try to run the command as the zabbix user. Keep in mind that permissions must allow access all along the path. For example, if the permissions are like this:

drw------- 3 root root 4096 jūn 17 21:40 /data/
drwxr-xr-x 3 root root 4096 jūn 17 21:40 /data/config
drw------- 2 zabbix zabbix 4096 jūn 17 21:40 /data/config/zabbix

User zabbix won’t be able to access directory /data/config/zabbix . Even though it is the owner of that directory, it has no permissions to access /data .

Don’t tee-pee

Closely related to the permission issues, there are a couple of flags for the Zabbix agent that sort of allow testing item keys.

$ zabbix_agentd --help
  -p --print                 Print known items and exit
  -t --test <item key>       Test specified item and exit

These sound perfect for testing userparameters – -p would show whether it is there and -t would test that item alone. Except that my general suggestion about them is:

Never use the -t and -p parameters for the Zabbix agent.

…unless you perfectly understand how they are different from querying a running agent daemon with zabbix_get and how the server queries that agent. Many, many users have been confused and spent a lot of time chasing the wrong things. Until you know your way around those two parameters, avoid them.

Very old versions of Zabbix agent did not support userparameters in the -t and -p flags at all, but there probably are few users with that old version of the agent.

Never tee-pee.

Oh, hello, SELinux

A subset of permission issues – things SELinux might block. If possible, the fastest way to confirm is to set SELinux to permissive and try again. If it works, debug that angle and make it work with SELinux in the enforcing mode.

The environment trap

Another very common problem is related to environment variables. See, Zabbix does not initialise the environment in any way – it just launches the userparameter command and that’s about it. Even if you add variables in the profile or rc files for the zabbix user, those won’t be used unless you explicitly source those files. If your script relies on any environment variables, make sure to set those explicitly.

What variables? Any. Including HOME. For example, if your userparameter uses mysql client utility and you store credentials in ~/.my.cnf like you should, Zabbix agent won’t know where to find the .my.cnf file.

You might suspect it is some environment variable, but not know for sure which one. One way to start about that would be comparing the variables when the script is run manually and works, and when it is run by the Zabbix agent and fails. Common commands to show current variables are set and env. The main functional difference is that set will show local variables, while env won’t – env will only show variables that will be passed to subprocesses. For example, if we have variables like these:

$ variable_a=1
$ export variable_b=2

set will see both, while env will see variable_b only

A subtle difference comes in play when using initscripts. If Zabbix agent has been started upon system boot, it will most likely have different environment compared to agent being started manually – when started manually, variables from the user’s shell will also be inherited. This difference would most likely not be there when using SystemD. SystemD might affect some of the environment variables, including HOME.

What the shell?

When launching userparameter commands, Zabbix simply passes those to /bin/sh . What is that on your system? It could be bash, it could be dash, it could be ash or something else. Make sure you know which one it is. Keep in mind that some shells will act differently, depending on how hey are invoked (for example, bash will work differently when invoked as /bin/sh ).

If it times out, it’s bad

Preferably, the scripts or commands userparameters call should return very quickly. The default timeout on the agent is 3 seconds… and you should not change it. If your script takes longer than a second or two, userparameters might not be the best solution.

Ways to debug

The above identifies the most common problems with userparameters. What to do when it’s not clear what is wrong?  Debugging time. The easiest methods are a bit hackish, but very efficient and not really unique to Zabbix.

Check the logfile

While recent versions of the Zabbix agent often return error message, still check the agent logfile. It might have additional clues.

Run the script manually

Assuming you are using a script for the userparameter, try running that script manually as the zabbix user. It will often provide extra information that will be helpful to figure out what fails. That is also the fastest way to check permission-related issues.

Log important detail

If the script fails in a fairly mysterious way, log interesting bits to a file. Parameters that you intend to pass to some command, payload – anything that you are not 200% sure about.

Bisect the script

If everything else fails, split the script in small bits, simplify it. Similar to logging important bits, start from the final result and go back. At some point the intermediary data won’t match the expectations. If needed, trim the script down to echo 1 like in the first article – maybe the agent daemon configuration file entry is wrong, or located in the wrong file. Instead of spending time on the script detail, trimming the script down will reveal the faulty area sooner.


Zabbix agent userparameters are very flexible and not that hard to debug. If you are familiar with the most common problems, you should have little trouble – and you should be familiar with those problems now.

Seen another issue with userparameters, or have some horror story about them? Please share that with us.

Create your own items – extend the agent with userparameters

Zabbix supports many different ways of monitoring, including agentless, SNMP and IPMI. Zabbix also provides a monitoring agent, which has a great set of built-in items for monitoring diskspace, processes, memory usage and many other things.

While the list of built-in items is growing with each release, there will always be something else we will want to monitor. Luckily, Zabbix agent is very easy to extend with new items by using a feature called userparameters. Zabbix userparameters are commands that the agent runs and expects an item value to be returned.

Continue reading Create your own items – extend the agent with userparameters

These are not the applications you are looking for

Applications – software applications – mean a specific thing in IT. Usually, that’s a user-oriented piece of software. A web browser, word processor, game. Lately, mobile applications have somewhat lowered the bar, even down to I Am Rich applications.

Screenshot of the "I Am Rich" application, showing the price of USD 999.99
“I Am Rich” sale screen (

And then there’s Zabbix with it’s own definition for applications. So what are applications in Zabbix?

Continue reading These are not the applications you are looking for

Zabbix bug TOP20, March 2017

Let’s talk bugs. The important Zabbix bugs. What are those? The ones that have the most votes in the Zabbix issue tracker.

TOP3 Zabbix bugreports from the table below in the article

There are currently 1308 open bugreports. When we looked at this number back in November last year, it was a hundred less. That’s a pretty huge number, is everything bad? Not really, as some might be duplicates and some might be incorrect reports. Not many, though, as there’s constant grooming going on. Most of the remaining are valid bugreports, but not too critical – some are even as minor as an offset of a few pixels in some page. Still a bug, but something we can live with, mostly. We already looked at the top-voted bugreport, now is the time to glance at others same as we did with feature requests.

Continue reading Zabbix bug TOP20, March 2017

The most important bug in Zabbix

How do you determine which bugs are important ?

The bug must be still unfixed to be important. If a new version of Zabbix comes out and the server crashes for all the users, that is the most important bug. Until it is fixed, hopefully, soon.

But there are some long-standing bugs that linger around just below the “fix-it” surface – they’re  not terrible enough to be fixed right away, and somewhat complicated usually. Such bugs can be around for many years, sometimes not even being fixed, but going away because a feature gets dropped completely. We’d need a way to measure which of all those known bugs is the most important. And there is a way to find out – same as with features, users can vote on bugreports. The bugreport with the most votes is titled deadlock between server and frontend.

Jira screenshot, showing "deadlock between server and frontend" issue title

Continue reading The most important bug in Zabbix

Created a trigger and it disappeared? Read on for a revelation!

So you had a cluster monitored. As is common with clusters, you wanted to have some cluster-wide parameter adding. Average CPU load, number of nodes online – something not tied to a single cluster node, thus you created a special host to denote the whole cluster. Then you went to that host, clicked “Create trigger”, specified all the items on individual cluster hosts, clicked “Add”… and the trigger was not there. Mysteriously missing.

An empty Zabbix trigger list with a red "NO TRIGGERS" text added

Oh, wait. That trigger actually appeared on all the cluster hosts. Is this a bug?

Continue reading Created a trigger and it disappeared? Read on for a revelation!

Visiting the Open Source Monitoring Conference 2016, Part 4

Returning to the events of the Open Source Monitoring Conference 2016, Avishai Ish-Shalom discussed an engineer’s approach to monitoring. David Hustace from OpenNMS told positive stories about this true-opensource monitoring tool.

OpenNMS mascot with a t-shirt saying "The kiwi bird is a direct descendant of the tyrannosaurus rex. rawr."

Continue reading Visiting the Open Source Monitoring Conference 2016, Part 4

Zabbix stopped sending out alerts, HALP!

Zabbix was this reliable friend, always sending you an email, SMS or both when something went down. It sometimes sent you a lot of emails, but you never got angry at Zabbix about that – it was just eager to help you, make sure you did not miss the weekly disaster. But then… last week… Zabbix did not send you an SMS. It did not send you an email. It did not telepathically inform you. But things were DOWN. Server was not RESPONDING.

A fat cat at a table with paws up

Zabbix knew about this. As you review the data, sitting in a dark room, the graphs clearly show the downtime. But there was-no-alert. How is that possible? Wait, what, this is impossible. You can see on the glowing screen that the main action, a crucial piece in getting those alerts, is disabled. That just cannot be, as nobody, NOBODY would ever disable that. How, oh how. Why, oh why.

Zabbix frontend row, showing a disabled action

Continue reading Zabbix stopped sending out alerts, HALP!

Visiting the Open Source Monitoring Conference 2016, Part 3

Monitoring most often deals with IT infrastructure. Sometimes it diverges a bit and starts caring about temperature and humidity, but in most cases that’s still limited to datacentre monitoring. In a talk at the Open Source Monitoring Conference 2016, Antony Stone covered some real world monitoring that goes a bit further than temperature monitoring. On a more classic-IT note, Shlomi Zadok covered system management with Foreman and security/compliance reporting by integrating with OpenSCAP. Let’s see what these fine gentlemen talked about.

Two cows

Continue reading Visiting the Open Source Monitoring Conference 2016, Part 3

Zabbix API example: creating a graph with LLD-generated items

We talked about the second most-voted feature request for Zabbix, an ability to create a graph containing an item for all LLD-created entities like network interfaces or filesystems. There was a teaser on using the Zabbix API to implement something like that. Now is the time to check out a simple example that shows such a functionality using a Perl module for the Zabbix API, Zabbix::Tiny.

Don’t despair if you have never used Zabbix::Tiny or even Perl before – everybody starts that way, and questions are always welcome. You might also want to review the posts that introduced Zabbix::Tiny:

Our script will do these things:

  • connect to the Zabbix API and obtain the host ID
  • query items by the specified key pattern that belong to the specified host (using the host ID obtained earlier)
  • create a custom graph

Our goal is to automatically create graphs with LLD-generated items like these:
Zabbix graph configuration, showing 4 free diskspace items
Continue reading Zabbix API example: creating a graph with LLD-generated items