Created a trigger and it disappeared? Read on for a revelation!

So you had a cluster monitored. As is common with clusters, you wanted to have some cluster-wide parameter adding. Average CPU load, number of nodes online – something not tied to a single cluster node, thus you created a special host to denote the whole cluster. Then you went to that host, clicked “Create trigger”, specified all the items on individual cluster hosts, clicked “Add”… and the trigger was not there. Mysteriously missing.

An empty Zabbix trigger list with a red "NO TRIGGERS" text added

Oh, wait. That trigger actually appeared on all the cluster hosts. Is this a bug?

Trigger gone

While the Zabbix page probably did not look that intimidating, it was not screaming at you that much, where did that trigger go?

An empty Zabbix trigger list

How did it suddenly end up on other hosts, not the one you clicked “Create trigger” for? That is actually completely normal and there is no bug.

Zabbix triggers are associated with hosts the triggers reference items from. Triggers cannot be associated with other, unrelated hosts.

What does that actually mean? Let’s look at an example.

  • We have 9 hosts called “cluster_host_1” .. “cluster_host_9”
  • We create a host called “cluster” and click “Create trigger” for this new host
  • We add items from all cluster hosts in the cluster

Part of the Zabbix trigger configuration form, showing trigger expression with items from 9 cluster hosts

Notice how we did not include any items from the aggregating host “cluster”. After adding such a trigger, it does not appear for the host “cluster” – instead, it appears for all the individual cluster hosts.

Zabbix host listing, showing one trigger for each of the individual cluster hosts and none for the host "cluster"

This trigger now is included in the trigger count for all the individual cluster hosts, as we referenced an item from each of them. This also means that we cannot sum up trigger counts in this view – in this example we see 9 hosts with a trigger for each, but the total trigger count is still 1.

You can even think of triggers sort of being on the same level as hosts – while permissions “flow” to triggers through hosts, functionally a trigger can check items on any two hosts at the same time, irrespective of the group membership or permissions on those hosts.  This is a very flexible aspect of Zabbix and provides for a lot of advanced configuration opportunities. Most triggers only check a single host, thus from the usability point of view it is simplified to look as if triggers belong to hosts same as items do.

A schematic, showing how items belong to hosts, but a trigger is outside the hosts and references the items
Items belong to hosts, triggers don’t

A workaround

There is a workaround to make this trigger appear for the aggregating host “cluster” – we just have to include some item from that host in the trigger expression. It doesn’t have to be a real item – any item of the trapper type will do. This is the safest type, as you don’t have to make sure it gets any data, and this item should be enabled – disabling it would silence the whole trigger. Once that item is in place, just adding a reference to it at the end of the expression will make this trigger appear for the cluster host – although it will also show up for all the individual cluster nodes still:

... or {cluster:empty_item.last()}=0

Et tu, graph?

By the way, this concept of “referencing” is also true for graphs – a graph includes items and is then associated with all the hosts it references items from. If we would create a custom graph with item from all of our cluster hosts, we would see a graph for each of the 9 hosts while the total graph count would be 1.

There is one difference between triggers and graphs in this regard. If a template trigger references an item from the template and another item from some host, linking this template will produce a trigger on the target host that checks an item on the target host together with the host, directly referenced in the template trigger. Or with a more specific example, in the template we could have:

{template:frontend_app_state.last()}=0 and {backend:db_state.last()}=1

When linking this template to a host, the trigger expression on the host would become:

{frontend_1:frontend_app_state.last()}=0 and {backend:db_state.last()}=1

Now the frontend app going down would not alert if the backend database was down. While a similar functionality can be achieved with trigger dependencies, you might run into situations when the approach above is more suitable.

Now graphs cannot do this. A graph can only include items from any number of hosts or a single template – template and host items cannot be mixed. When creating a graph on a template, this is stressed by disabling the template/host dropdown in the item selection popup, in the upper right corner:

A disabled template/host dropdown

It would be more consistent and flexible if graphs allowed template-and-host-items, and it would allow creating graphs that show statistics on a specific host and then some generic system – like a backend DB host or a core router. It has not been requested so far, though, so apparently there’s not that much interest in such a feature.

Getting back to our misunderstanding regarding how triggers and graphs are associated with hosts – or how triggers and graphs aren’t really created on hosts – don’t worry if you got confused by that. There have been bugreports created about the graph API and trigger configuration, and this topic comes up on the Zabbix IRC channel frequently. It’s just one of those Zabbix oddities that you have to find out about, and then it seems normal – and a powerful feature.

Leave a Reply