Creating JMX LLD Items in Zabbix

With the latest version 3.4 of Zabbix it is now possible to create LLD rules to discover JMX items. However Zabbix documentation was rather confusing so I hope to explain things to you by example. Let us start with what most people would like to monitor – the Java Garbage Collector. It make sense to do a low level discovery (LLD) here as Garbage Collectors can change over time. For example, today it can be PS MarkSweep and tomorrow the developer changes it to G1. Without LLD we have to create an item for every possible Garbage Collector and disable the ones we don’t use. That works for 1 or 10 hosts, but once you have more you want it to be automatic.

Creating the LLD rule

So enough talk, let’s start by creating our LLD rule. In this setup we will try to monitor the Garbage Collectors.

  • The first thing to do is to name it – I went for something easy like “JMX GC Discovery”.
  • Next, we have to choose the rule type. This must be “JMX agent”.
  • The next field is key and here we have to write down our JMX key. For the discovery of Garbage Collectors use jmx.discovery[beans,”java.lang:type=GarbageCollector,name=*”]

Be careful when using “*” in the LLD rule – you could use “*” for the type, but then Zabbix will go through all the types and this will take a while, and have an impact on the performance. It is best to only use it for the name of the beans we would like to discovery.

How do I know that this needs to be java.lang:type=GarbageCollector,name=* ? Best is to use some tool and connect to your Java application. This can be done with tools like JConsole or jmxterm.

JConsole works like an explorer – this way we can browse for the Garbage Collectors, and then we can see that they have the object name java.lang with type GarbageCollector. For more information on JConsole, check out the Java documentation.

Let’s talk about the LLD rule key

The item key jmx.discovery is predefined in Zabbix. In the item key parameters we have choose between beans or attributes to do the discovery. So when to discover by beans and when to discover by attributes? This was something I couldn’t figure out from the Zabbix documentation, but it has been updated since then and should make more sense now. If you know what attributes you want to monitor – and in our case we do because we have been exploring with JConsole and we know we would like to discover all the different GarbageCollectors – then we chose beans. If we have no clue at all, then we have to go for attributes. IMHO you always know what you want, so I personally can’t figure out yet when you would want to use attributes.

Querying the Java gateway directly

We can also query Zabbix Java gateway for debugging purposes. This can be useful if you don’t have a tool like JConsole installed, or if you can only access the JMX port from the Zabbix server because of security restrictions or other reasons.

The easiest might be by using a shell script (reproduced from the official Zabbix blog, in turn adapted from the Zabbix wiki):

#!/usr/bin/env bash

if [ $# != 6 ]
then
   echo "Usage: $0      "
   exit;
fi

# create connection
exec 3<>/dev/tcp/$1/$2

# compose message
MSG="{\"request\": \"java gateway jmx\", \"conn\": \"$3\", \"port\": $4, \"jmx_endpoint\": \"$5\", \"keys\": [\"$6\"]}"

# write message length as zero-padded 16-digit hexadecimal number
printf -v LEN '%016x' "${#MSG}"

# prepare message length in little endian representation
BYTES=""
for i in {0..14..2}
do
    BYTES="\\x${LEN:$i:2}$BYTES"
done

# prepend protocol header and message length
printf "ZBXD\\1$BYTES%s" "$MSG" >&3

# output the result skipping 6 bytes of "ZBXD\\1" header and 8 bytes of message length
tail -c+14 <&3

You can use it this way:

 ./zabbix_get_java.sh 127.0.0.1 10052 <JMX Host IP> <JMX PORT> 'service:jmx:rmi:///jndi/rmi://<JMX Host IP>:<JMX PORT> /jmxrmi' 'jmx.discovery[attributes,\"*:type=MemoryPool,name=Code Cache\"]' | jq '.data[0].value | fromjson | .data' 

When we look at the JMX data for GarbageCollectors, there’s a JSON output like this:

Notice how various values are returned – for example, {#JMXTYPE} can allow for further filtering by the data type.

Zabbix documentation also has a good reference of all the macros being returned by the JMX LLD.

Creating item prototype

Now it’s time to make our item prototype.

Choose JMX agent for the item type and set the key to jmx[{#JMXOBJ},CollectionTime] .

The JMX endpoint can be left as is, unless you have multiple JVMs running on your system with different endpoints.

We can also create another item prototype that monitors the jmx[{#JMXOBJ},CollectionCount] key.

To tie this all together, let’s link our template to hosts and we can see appropriate data flowing in without having all those unsupported items or disabling items manually:

As an example, here’s how the garbage collection time has changed over the course of one month:

Still wondering where the Action Simulator is in 2018?

It hurts to admit it, but this is just another preview and not something that is ready to be released to the general public! But don’t think that I took it easy: I spent a lot of time cleaning up and restructuring the code. I worked on new features and some fundamental issues and then I thought: Instead of going for 3.4, which is nearing its end of life, rather aim for 4.0, which will become the next LTS!

In hindsight, I am not upset with this decision, but unfortunately it added a lot more complexity to the job and that is why it is not done! This is where you could come in: I am looking for help on the frontend part. Please read on if you are interested and let me know!

If you don’t know what Action Simulator is and why is it great, consider reading the introductory post first.

Continue reading Still wondering where the Action Simulator is in 2018?

Forcing a network discovery run

Zabbix network discovery is a way to scan a network range and automatically start monitoring the discovered devices and services. Network discovery can look for basic services like HTTP, IMAP or SMTP. It can also scan the specified network range for SNMP, Zabbix agent or ICMP ping responses.

Zabbix network discovery is not to be confused with the space shuttle

Constant network scanning is not desirable, thus the discovery is usually set to run not too frequently – once every few hours to once every few days. If you have added a new Zabbix action for network discovery that links a fresh template, there is no way to see when will the discovery rule run next time, and no way to force it to run sooner.

Or is there?

Continue reading Forcing a network discovery run

Templating Monitoring of Windows Services

About a year ago, I was inspired by Raymond Kuiper’s Zen and the Art of Zabbix Template Design, and decided to start applying its practices to my own environment. If you’re not familiar with it, I would strongly recommend looking at the linked page, as well his video presentation on the subject. The idea is to create small “Task” templates for very specialized purposes, and then group those into more generalized “Duty” templates, which are grouped into even more generalized “Role” templates that provide a scope for the function of the generalized server, and that finally nests inside of a “Profile” template which gets applied to the monitored server. The process creates an extremely modular system that is very easy to manage. One of the key concepts is to rely on discovery as much as possible to maintain this modularity.

Among the first places I put this into practice was to monitor Windows services. I had envisioned several task templates for monitoring Windows services – one for DFS (distributed file services), one for SQL services, one for Skype For Business services, etc. The only functional differences between them would be that the filters used in each rule would pick just the services for that particular task.

In practice, just doing this created a problem, because while Zabbix does have a very nice Windows service.discovery item, it can only exist once per server. It was easy to create the Tplt::Task::Win::Services::DFS and Tplt::Task::Win::Services::SQL “Task” templates, however trying to link these into the same parent “Duty” template Tplt::Duty::MSSQLServer gave an error like this:

- Discovery rule "service.discovery" already exists on ...

So – how to address this? Continue reading Templating Monitoring of Windows Services

Proxy dependency using the API

One common question we often hear on IRC is: how to make hosts depend on their proxy. The problem being that when a proxy is unable to send data to the server for a long enough time, the nodata() triggers on hosts monitored by the proxy start to fire.
A possible end result is illustrated perfectly by quoting one of the users in #zabbix @ irc.freenode.net

<someuser> lets say one of my large proxies falls behind. I can receive like 2k mails.

Obviously, receiving such a quantity of notifications doesn’t help you correct the issue.

Continue reading Proxy dependency using the API

Debugging Zabbix userparameters

Zabbix agent is easy to extend for data collection with a feature called userparameters. We figured out how they work in the article Create your own items – extend the agent with userparameters. Unfortunately, userparameters sometimes don’t work right away, and not always the cause is obvious. In this article we will explore the most common issues in more detail and learn to use simple methods to debug Zabbix agent userparameters.

Continue reading Debugging Zabbix userparameters

Auto-registering Linux agents with TLS PSK encryption

Zabbix supports Linux agent auto-registration and it’s a well documented process in the Zabbix manual. However, it’s not really straightforward to mass provision Zabbix agents on hundreds of servers if you want to have Zabbix agent <-> Zabbix server communication encrypted. At least not without some form of additional scripted step in your installation process. For large deployments I usually use Ansible, although this article just briefly covers that part and I’ll mostly focus on how to make sure your agents get registered for encrypted communication.

Continue reading Auto-registering Linux agents with TLS PSK encryption

Create your own items – extend the agent with userparameters

Zabbix supports many different ways of monitoring, including agentless, SNMP and IPMI. Zabbix also provides a monitoring agent, which has a great set of built-in items for monitoring diskspace, processes, memory usage and many other things.

While the list of built-in items is growing with each release, there will always be something else we will want to monitor. Luckily, Zabbix agent is very easy to extend with new items by using a feature called userparameters. Zabbix userparameters are commands that the agent runs and expects an item value to be returned.

Continue reading Create your own items – extend the agent with userparameters

These are not the applications you are looking for

Applications – software applications – mean a specific thing in IT. Usually, that’s a user-oriented piece of software. A web browser, word processor, game. Lately, mobile applications have somewhat lowered the bar, even down to I Am Rich applications.

Screenshot of the "I Am Rich" application, showing the price of USD 999.99
“I Am Rich” sale screen (https://en.wikipedia.org/wiki/File:I_Am_Rich_sale_screen.png)

And then there’s Zabbix with it’s own definition for applications. So what are applications in Zabbix?

Continue reading These are not the applications you are looking for

Still wondering about missing notifications in 2017?

If Zabbix keeps on surprising you with its notifications, you might want to try the Action Simulator! The Action Simulator is a community patch that helps you to figure out whether your actions really do as you intend. It first came out for Zabbix 2.0 in 2013 and was downloaded by hundreds of users from all around the world.

The following article gives a brief introduction to the Action Simulator and explains the challenges of developing it for Zabbix 3.2.

Continue reading Still wondering about missing notifications in 2017?