All posts by Patrik Uytterhoeven

Creating JMX LLD Items in Zabbix

With the latest version 3.4 of Zabbix it is now possible to create LLD rules to discover JMX items. However Zabbix documentation was rather confusing so I hope to explain things to you by example. Let us start with what most people would like to monitor – the Java Garbage Collector. It make sense to do a low level discovery (LLD) here as Garbage Collectors can change over time. For example, today it can beĀ PS MarkSweep and tomorrow the developer changes it to G1. Without LLD we have to create an item for every possible Garbage Collector and disable the ones we don’t use. That works for 1 or 10 hosts, but once you have more you want it to be automatic.

Creating the LLD rule

So enough talk, let’s start by creating our LLD rule. In this setup we will try to monitor the Garbage Collectors.

  • The first thing to do is to name it – I went for something easy like “JMX GC Discovery”.
  • Next, we have to choose the rule type. This must be “JMX agent”.
  • The next field is key and here we have to write down our JMX key. For the discovery of Garbage Collectors use jmx.discovery[beans,”java.lang:type=GarbageCollector,name=*”]

Be careful when using “*” in the LLD rule – you could use “*” for the type, but then Zabbix will go through all the types and this will take a while, and have an impact on the performance. It is best to only use it for the name of the beans we would like to discovery.

How do I know that this needs to be java.lang:type=GarbageCollector,name=* ? Best is to use some tool and connect to your Java application. This can be done with tools like JConsole or jmxterm.

JConsole works like an explorer – this way we can browse for the Garbage Collectors, and then we can see that they have the object name java.lang with type GarbageCollector. For more information on JConsole, check out the Java documentation.

Let’s talk about the LLD rule key

The item key jmx.discovery is predefined in Zabbix. In the item key parameters we have choose between beans or attributes to do the discovery. So when to discover by beans and when to discover by attributes? This was something I couldn’t figure out from the Zabbix documentation, but it has been updated since then and should make more sense now. If you know what attributes you want to monitor – and in our case we do because we have been exploring with JConsole and we know we would like to discover all the different GarbageCollectors – then we chose beans. If we have no clue at all, then we have to go for attributes. IMHO you always know what you want, so I personally can’t figure out yet when you would want to use attributes.

Querying the Java gateway directly

We can also query Zabbix Java gateway for debugging purposes. This can be useful if you don’t have a tool like JConsole installed, or if you can only access the JMX port from the Zabbix server because of security restrictions or other reasons.

The easiest might be by using a shell script (reproduced from the official Zabbix blog, in turn adapted from the Zabbix wiki):

#!/usr/bin/env bash

if [ $# != 6 ]
then
   echo "Usage: $0      "
   exit;
fi

# create connection
exec 3<>/dev/tcp/$1/$2

# compose message
MSG="{\"request\": \"java gateway jmx\", \"conn\": \"$3\", \"port\": $4, \"jmx_endpoint\": \"$5\", \"keys\": [\"$6\"]}"

# write message length as zero-padded 16-digit hexadecimal number
printf -v LEN '%016x' "${#MSG}"

# prepare message length in little endian representation
BYTES=""
for i in {0..14..2}
do
    BYTES="\\x${LEN:$i:2}$BYTES"
done

# prepend protocol header and message length
printf "ZBXD\\1$BYTES%s" "$MSG" >&3

# output the result skipping 6 bytes of "ZBXD\\1" header and 8 bytes of message length
tail -c+14 <&3

You can use it this way:

 ./zabbix_get_java.sh 127.0.0.1 10052 <JMX Host IP> <JMX PORT> 'service:jmx:rmi:///jndi/rmi://<JMX Host IP>:<JMX PORT> /jmxrmi' 'jmx.discovery[attributes,\"*:type=MemoryPool,name=Code Cache\"]' | jq '.data[0].value | fromjson | .data' 

When we look at the JMX data for GarbageCollectors, there’s a JSON output like this:

Notice how various values are returned – for example, {#JMXTYPE} can allow for further filtering by the data type.

Zabbix documentation also has a good reference of all the macros being returned by the JMX LLD.

Creating item prototype

Now it’s time to make our item prototype.

Choose JMX agent for the item type and set the key to jmx[{#JMXOBJ},CollectionTime] .

The JMX endpoint can be left as is, unless you have multiple JVMs running on your system with different endpoints.

We can also create another item prototype that monitors the jmx[{#JMXOBJ},CollectionCount] key.

To tie this all together, let’s link our template to hosts and we can see appropriate data flowing in without having all those unsupported items or disabling items manually:

As an example, here’s how the garbage collection time has changed over the course of one month: