Still wondering about missing notifications in 2017?

If Zabbix keeps on surprising you with its notifications, you might want to try the Action Simulator! The Action Simulator is a community patch that helps you to figure out whether your actions really do as you intend. It first came out for Zabbix 2.0 in 2013 and was downloaded by hundreds of users from all around the world.

The following article gives a brief introduction to the Action Simulator and explains the challenges of developing it for Zabbix 3.2.

I’m an expert, I don’t need this!

The Action Simulator was not designed for novice users. It was born to avoid the embarrassment of explaining myself over missing notifications for rare but very important events.  While it certainly is useful for novices, even the most senior users probably have good use for it at some point. It doesn’t really matter how good you are: Mistakes simply happen! Action configuration is fairly complex and Zabbix does not provide easy means of testing any aspect of  it. However, even subtle mis-configuration can stop your important notifications. If you don’t feel like frustrating yourself for hours, trying to find the reason for missing or duplicate notifications, try the Action Simulator — you may like it!

Installation

The Action Simulator introduces changes to the PHP frontend only! This means, you don’t need to recompile anything. It aims to touch as few files as possible. All you need to do is to download a suitable version and patch your frontend. I’m sorry about the somewhat confusing versioning scheme here: It resembles the state of development of the patch and does line up with Zabbix version numbers. You usually want the latest Action Simulator for your version of Zabbix — currently v5.2.0 for Zabbix 3.0. There used to be problems with applying the changes to the CSS files in early 3.0 versions. I hope they are gone now!

If you are afraid of patching your production frontend, or are not allowed by company policy, simply create a copy of it and serve the modified version in parallel. The Action Simulator is not changing anything about your Zabbix configuration — no risk of damage. Neither does it actually execute remote commands or send notifications. It’s really just a simulation!

Using it

The Action Simulator is only exposed through an extra column on the trigger configuration list of hosts (not templates).

The only addition to the frontend is the rightmost column labelled “Actions”.

Clicking the icon in the column opens a pop-up with 4 tables, containing the following:

  1. Details about the simulated event
  2. Resulting notifications (and potentially associated problems)
  3. Resulting remote commands
  4. Action condition evaluation results
Tables 2 to 4
Cutaway of the tables 2, 3 and 4: Number 2 is showing two notifications that can not be performed (red box) and need investigation. Number 3 shows a remote command and number 4 shows condition evaluation for two actions: One matches, the other doesn’t.

If everything acts as conceived, you should see the desired notifications and remote commands in table 2 and 3. Make sure to keep an eye for error symbols in the rightmost column!

If the results don’t live up to your expectations, inspect the “Action” column in these tables for missing or unwanted entries. Table 4 should help you to find to cause then! If you found the problem, change the configuration and simply reload the pop-up. Repeat until everything is correct!

Towards  Zabbix 3.2

3 years after the initial release came the Action Simulator for Zabbix 3.0 — 7 long months after the first 3.0 release and devastating 2 years after the first 2.4 release, which never got an Action Simulator at all! I run LTS versions and thus had no immediate pressure to work on it.  I was also afraid of the changes necessary for 2.4, mostly to accommodate for the introduction of formula-type action conditions, which eventually turned out to be not that difficult. And then there were the rumours of the API leaving the frontend code, which eventually didn’t happen.

Only 4 days after my release for 3.0, Zabbix 3.2 came out. This time I hope to be quicker and not leave 3.2 users in distress of choosing between the many great new Zabbix features or the comfort of having the Action Simulator. Getting it ready for 3.4 should be a walk in the park then!

As a development release, 3.2 is a bit of a moving target and introduces a couple of features that somehow affect how events, actions and operations are handled. Getting the Action Simulator fit for a new Zabbix version requires reading  What’s-New pages, the upgrade notes, the C and PHP source code, the regular and API documentation and some experimenting. Here are the relevant changes I spotted and how they change the Action Simulator:

Event tags

Event tags are a new and powerful concept. Their use in action conditions is rather trivial though, where only the key-value variant is used. This is just one more condition type to handle — very similar to “Application” string matching, actually!

https://www.zabbix.com/documentation/3.2/manual/config/triggers/event_tags

Closing problems manually

This actually requires no change: Is is irrelevant for both action condition evaluation and operations, whether a trigger recovers as a consequence of trigger expression evaluation or by user intervention. I will simply re-label the 3rd calendar widget from “Resolved at” to “Closed/Recovered”, which is also closer to the terms elsewhere in the UI.

https://www.zabbix.com/documentation/3.2/manual/config/events/manual_close

Recovery operations

The Action Simulator never dealt with recovery operations. You have to keep in mind, that it was mostly designed to spot and understand misconfiguration. Before 3.2, recovery notifications were sent out to people who received problem notifications before. It was therefore safe to assume that if they were able to receive those, they will also get the recovery messages. This is no longer the case, since you can now send recovery messages to arbitrary users and you can also execute remote commands. The 3.2 version will therefore have to consider recovery operation and either show them in the existing tables in a distinguishable fashion or use separate tables.

https://www.zabbix.com/documentation/3.2/manual/config/notifications/action/recovery_operations

Pausing operations during maintenance

Maintenance is a bit of a weak spot in the Action Simulator: It does not consider the actual maintenances you set up in frontend. That’s mostly because there  is no existing frontend code to evaluate when a host will enter or leave maintenance. My pragmatic approach for previous releases was to simply assume maintenance to start at a given point in time, specified in the form. This is actually not just a disadvantage, because it also allows you to say “What if”!  So, the principle won’t change here any time soon. What must be adapted though, is the implementation, as “pausing” was not an option, but rather the default previously, if the maintenance state was part of the action conditions.

https://www.zabbix.com/documentation/3.2/manual/config/notifications/action/escalations

And what about Zabbix 3.4?

So far, I couldn’t spot any relevant differences between 3.2 and 3.4 on the draft What’s-New page and my development 3.2 patch even applies to the current state of what is to become 3.4.

There has been recent SCRUM noise around the good old ZBXNEXT-97, a ticket asking for what the Action Simulator provides, among other features. This noise indicated soon-to-come development efforts, but it was apparently cancelled. 120 votes and 7 years just ain’t enough, it seems. This was met with anger and the testimony, that the Action Simulator is important for operating Zabbix successfully, which made me very happy! Since this ticket actually asks for two things (testing media and figuring out what’s gonna happen), it remains unclear what the development focus would have been anyway!

My attention was recently brought to ZBXNEXT-18, which is asking for notifications upon acknowledgements. Looking at the ticket, I can now tell you about the chosen developer team, the number of the sprint and the story points given, but I have no clue what it is that is “in specification”.

That said, I’m still looking forward to the day when design specifications will be public again and not hidden, for no important reason!

2 thoughts on “Still wondering about missing notifications in 2017?”

  1. Dear Volker,
    thanks for the ActionSimulator which I would love to use. I downloaded the latest v 5.2.0 and patched zabbix 3.2.6 on debian, but got 3 failures.
    For “include/views/configuration.triggers.list.php”, Hunk 2 FAILED at 105 and at 241 and for “share/zabbix_actionssim/triggers.php”, Hunk 2 FAILED at 491.

    1. As the article explains, there is no released Action Simulator for 3.2 yet. I will soon post another article on the progress.

Leave a Reply