If Zabbix keeps on surprising you with its notifications, you might want to try the Action Simulator! The Action Simulator is a community patch that helps you to figure out whether your actions really do as you intend. It first came out for Zabbix 2.0 in 2013 and was downloaded by hundreds of users from all around the world.
The following article gives a brief introduction to the Action Simulator and explains the challenges of developing it for Zabbix 3.2.
I’m an expert, I don’t need this!
The Action Simulator was not designed for novice users. It was born to avoid the embarrassment of explaining myself over missing notifications for rare but very important events. While it certainly is useful for novices, even the most senior users probably have good use for it at some point. It doesn’t really matter how good you are: Mistakes simply happen! Action configuration is fairly complex and Zabbix does not provide easy means of testing any aspect of it. However, even subtle mis-configuration can stop your important notifications. If you don’t feel like frustrating yourself for hours, trying to find the reason for missing or duplicate notifications, try the Action Simulator — you may like it!
The Action Simulator introduces changes to the PHP frontend only! This means, you don’t need to recompile anything. It aims to touch as few files as possible. All you need to do is to download a suitable version and patch your frontend. I’m sorry about the somewhat confusing versioning scheme here: It resembles the state of development of the patch and does line up with Zabbix version numbers. You usually want the latest Action Simulator for your version of Zabbix — currently v5.2.0 for Zabbix 3.0. There used to be problems with applying the changes to the CSS files in early 3.0 versions. I hope they are gone now!
If you are afraid of patching your production frontend, or are not allowed by company policy, simply create a copy of it and serve the modified version in parallel. The Action Simulator is not changing anything about your Zabbix configuration — no risk of damage. Neither does it actually execute remote commands or send notifications. It’s really just a simulation!
The Action Simulator is only exposed through an extra column on the trigger configuration list of hosts (not templates).
Clicking the icon in the column opens a pop-up with 4 tables, containing the following:
- Details about the simulated event
- Resulting notifications (and potentially associated problems)
- Resulting remote commands
- Action condition evaluation results
If everything acts as conceived, you should see the desired notifications and remote commands in table 2 and 3. Make sure to keep an eye for error symbols in the rightmost column!
If the results don’t live up to your expectations, inspect the “Action” column in these tables for missing or unwanted entries. Table 4 should help you to find to cause then! If you found the problem, change the configuration and simply reload the pop-up. Repeat until everything is correct!
Towards Zabbix 3.2
3 years after the initial release came the Action Simulator for Zabbix 3.0 — 7 long months after the first 3.0 release and devastating 2 years after the first 2.4 release, which never got an Action Simulator at all! I run LTS versions and thus had no immediate pressure to work on it. I was also afraid of the changes necessary for 2.4, mostly to accommodate for the introduction of formula-type action conditions, which eventually turned out to be not that difficult. And then there were the rumours of the API leaving the frontend code, which eventually didn’t happen.
Only 4 days after my release for 3.0, Zabbix 3.2 came out. This time I hope to be quicker and not leave 3.2 users in distress of choosing between the many great new Zabbix features or the comfort of having the Action Simulator. Getting it ready for 3.4 should be a walk in the park then!
As a development release, 3.2 is a bit of a moving target and introduces a couple of features that somehow affect how events, actions and operations are handled. Getting the Action Simulator fit for a new Zabbix version requires reading What’s-New pages, the upgrade notes, the C and PHP source code, the regular and API documentation and some experimenting. Here are the relevant changes I spotted and how they change the Action Simulator:
Event tags are a new and powerful concept. Their use in action conditions is rather trivial though, where only the key-value variant is used. This is just one more condition type to handle — very similar to “Application” string matching, actually!
Closing problems manually
This actually requires no change: Is is irrelevant for both action condition evaluation and operations, whether a trigger recovers as a consequence of trigger expression evaluation or by user intervention. I will simply re-label the 3rd calendar widget from “Resolved at” to “Closed/Recovered”, which is also closer to the terms elsewhere in the UI.
The Action Simulator never dealt with recovery operations. You have to keep in mind, that it was mostly designed to spot and understand misconfiguration. Before 3.2, recovery notifications were sent out to people who received problem notifications before. It was therefore safe to assume that if they were able to receive those, they will also get the recovery messages. This is no longer the case, since you can now send recovery messages to arbitrary users and you can also execute remote commands. The 3.2 version will therefore have to consider recovery operation and either show them in the existing tables in a distinguishable fashion or use separate tables.
Pausing operations during maintenance
Maintenance is a bit of a weak spot in the Action Simulator: It does not consider the actual maintenances you set up in frontend. That’s mostly because there is no existing frontend code to evaluate when a host will enter or leave maintenance. My pragmatic approach for previous releases was to simply assume maintenance to start at a given point in time, specified in the form. This is actually not just a disadvantage, because it also allows you to say “What if”! So, the principle won’t change here any time soon. What must be adapted though, is the implementation, as “pausing” was not an option, but rather the default previously, if the maintenance state was part of the action conditions.
And what about Zabbix 3.4?
So far, I couldn’t spot any relevant differences between 3.2 and 3.4 on the draft What’s-New page and my development 3.2 patch even applies to the current state of what is to become 3.4.
There has been recent SCRUM noise around the good old ZBXNEXT-97, a ticket asking for what the Action Simulator provides, among other features. This noise indicated soon-to-come development efforts, but it was apparently cancelled. 120 votes and 7 years just ain’t enough, it seems. This was met with anger and the testimony, that the Action Simulator is important for operating Zabbix successfully, which made me very happy! Since this ticket actually asks for two things (testing media and figuring out what’s gonna happen), it remains unclear what the development focus would have been anyway!
My attention was recently brought to ZBXNEXT-18, which is asking for notifications upon acknowledgements. Looking at the ticket, I can now tell you about the chosen developer team, the number of the sprint and the story points given, but I have no clue what it is that is “in specification”.
That said, I’m still looking forward to the day when design specifications will be public again and not hidden, for no important reason!