About a year ago, I was inspired by Raymond Kuiper’s Zen and the Art of Zabbix Template Design, and decided to start applying its practices to my own environment. If you’re not familiar with it, I would strongly recommend looking at the linked page, as well his video presentation on the subject. The idea is to create small “Task” templates for very specialized purposes, and then group those into more generalized “Duty” templates, which are grouped into even more generalized “Role” templates that provide a scope for the function of the generalized server, and that finally nests inside of a “Profile” template which gets applied to the monitored server. The process creates an extremely modular system that is very easy to manage. One of the key concepts is to rely on discovery as much as possible to maintain this modularity.
Among the first places I put this into practice was to monitor Windows services. I had envisioned several task templates for monitoring Windows services – one for DFS (distributed file services), one for SQL services, one for Skype For Business services, etc. The only functional differences between them would be that the filters used in each rule would pick just the services for that particular task.
In practice, just doing this created a problem, because while Zabbix does have a very nice Windows service.discovery item, it can only exist once per server. It was easy to create the Tplt::Task::Win::Services::DFS and Tplt::Task::Win::Services::SQL “Task” templates, however trying to link these into the same parent “Duty” template Tplt::Duty::MSSQLServer gave an error like this:
- Discovery rule "service.discovery" already exists on ...
One common question we often hear on IRC is: how to make hosts depend on their proxy. The problem being that when a proxy is unable to send data to the server for a long enough time, the nodata() triggers on hosts monitored by the proxy start to fire.
A possible end result is illustrated perfectly by quoting one of the users in #zabbix @ irc.freenode.net
<someuser> lets say one of my large proxies falls behind. I can receive like 2k mails.
Obviously, receiving such a quantity of notifications doesn’t help you correct the issue.
Zabbix supports Linux agent auto-registration and it’s a well documented process in the Zabbix manual. However, it’s not really straightforward to mass provision Zabbix agents on hundreds of servers if you want to have Zabbix agent <-> Zabbix server communication encrypted. At least not without some form of additional scripted step in your installation process. For large deployments I usually use Ansible, although this article just briefly covers that part and I’ll mostly focus on how to make sure your agents get registered for encrypted communication.
Zabbix supports many different ways of monitoring, including agentless, SNMP and IPMI. Zabbix also provides a monitoring agent, which has a great set of built-in items for monitoring diskspace, processes, memory usage and many other things.
While the list of built-in items is growing with each release, there will always be something else we will want to monitor. Luckily, Zabbix agent is very easy to extend with new items by using a feature called userparameters. Zabbix userparameters are commands that the agent runs and expects an item value to be returned.
As someone working in IT infrastructure, every now and then you are confronted with a problem that you are not certain how to solve. Often times I have found myself overthinking things and ending up with a complex solution that isn’t very elegant but get’s the job done.