Proxy dependency using the API

One common question we often hear on IRC is: how to make hosts depend on their proxy. The problem being that when a proxy is unable to send data to the server for a long enough time, the nodata() triggers on hosts monitored by the proxy start to fire.
A possible end result is illustrated perfectly by quoting one of the users in #zabbix @ irc.freenode.net

<someuser> lets say one of my large proxies falls behind. I can receive like 2k mails.

Obviously, receiving such a quantity of notifications doesn’t help you correct the issue.

What does Zabbix do already?

Well, unfortunately, Zabbix doesn’t give you an out of the box fix. If you need this, I suggest voting on ZBXNEXT-1891 (currently 33 votes).

Zabbix does already have a concept of trigger dependency though, where one trigger depends on the status of another trigger. So the hard part will be having the nodata() triggers on the hosts depend on the status of the proxy through which it’s being monitored.

So what can I do about it?

Well, the solution I’ve implemented is:

  1. Create a trigger on the proxy host that defines it’s status.
  2. Use the API to:
    • Find all proxies on the system.
    • For each proxy, find the trigger we have defined.
    • Find all hosts monitored through the proxy.
    • List all nodata() triggers on those hosts.
    • Check if they are dependent on the proxy trigger, if not, add the dependency

Implementing this will require enough knowledge on your behalf of Zabbix and its API, because what I post will require modification to fit your specific setup.
The end result would look like this on the dashboard:

Active Proxy Triggers

Two proxies are down, only two triggers are active…
So let’s get started.

One trigger to rule them all

We need a trigger on all of our proxies to determine their status. The best way to get there is modifying the template “Template App Zabbix Proxy” by adding a trigger:

Name: Zabbix proxy not sending data for 4 minutes. (Suppressing triggers)
Problem expression: {Template App Zabbix Proxy:zabbix[queue].nodata(4m)}=1
Recovery expression: {Template App Zabbix Proxy:zabbix[queue].last()}<50

By default, Zabbix gives you a notification after 5 minutes when an agent is unavailable (through the default “Template App Zabbix Agent”). So I’m setting my nodata() trigger to 4 minutes in the problem expression.

The trigger status will only be resolved when the proxy has sent most of their data to the server. Experience taught me that this works very well for my specific installation, but the recovery trigger might need tweaking in your case. I can image in larger installations, the trigger needs a different approach all together, but again, this is what works for me.

Now show me the code

The second step is harder though. We need to do some scripting here. The example I’m posting is in Powershell using the Zabbix.NET library.

[CmdletBinding(PositionalBinding=$false)]
 Param
 (
     [String]$zabbix_newtonsoft_path="C:\..\Newtonsoft.Json.dll",
     [String]$zabbix_zabbixapi_path="C:\..\ZabbixApi.dll",
     [String]$zabbix_user = "myzabbixapiuser",
     [String]$zabbix_pwd = "somepassword",
     [String]$zabbix_api_uri = "http://192.168.123.123/zabbix/api_jsonrpc.php",
     [String]$ProxyTiggerNameStartsWith = "Zabbix proxy not sending data"
 )
 
 # Load the external Zabbix.net and newtonsoft Library into Powershell.
 
 [Reflection.Assembly]::LoadFile($zabbix_newtonsoft_path);
 [Reflection.Assembly]::LoadFile($zabbix_zabbixapi_path);
 
 # Initiate the library and connect.
 
 $Zabbix = New-Object "ZabbixApi.Zabbix" ($zabbix_user, $zabbix_pwd, $zabbix_api_uri);
 $Zabbix.login();
 
 # Let's start with looping through all our proxies.
 
 $proxyList = New-Object System.Collections.ArrayList;
 $params=[System.Dynamic.ExpandoObject]::new();
 $params.output=@("hostid","host");
 $params.proxy_hosts="1";
 $responseObj = $zabbix.objectResponse("host.get", $params);
 $scripterror = 0
 foreach ( $data in $responseObj.result) {
 
     # We have a proxy, now get the host that is monitoring this proxy.
     # In my case, my proxy name ( under administrator - proxies ) has the same name
     # as the host monitoring the proxy. If is this not the case for you,
     # you will need to write other code, or fix the names.
 
     write-output "Getting host for proxy: $($data.host)"
     $params=[System.Dynamic.ExpandoObject]::new();
     $params.output=@("hostid","host");
     $params.filter=@{ host = @($data.host)};
     $responseHostObj = $zabbix.objectResponse("host.get", $params);
     $hostid = $responseHostObj.result.hostid
     write-output "Found agent with id $hostid"
 
     # Now that I have the host that is monitoring the proxy. I need to find the custom
     # trigger I've added through the “Template App Zabbix Proxy”.
     # We can do that through looping all triggers and checking the name of the triggers.
 
     $params=[System.Dynamic.ExpandoObject]::new();
     $params.output=@("triggerid","expression","description");
     $params.hostids= "$hostid";
     $params.functions= "nodata";
     $responseTriggerObj = $zabbix.objectResponse("trigger.get", $params);
     $triggerToDepOn = $null
 
     foreach ( $trigger in $responseTriggerObj.result) {
         if($trigger.description.ToUpper().StartsWith($ProxyTiggerNameStartsWith.ToUpper())){
 
             # We just found the trigger on the proxy on which we need to depend            
 
             $triggerToDepOn = $trigger.triggerid
             break
         }
     }
     if($triggerToDepOn -eq $null){
         write-output "Did not find the master trigger to depend on for proxy $($data.host)/$($data.hostid)"
         continue
     }
 
     # Now that I have the trigger that holds the status of my proxy.
     # We still need to make all hosts monitored by this proxy depend on it.
 
     $params=[System.Dynamic.ExpandoObject]::new();
     $params.output=@("hostid","host");
     $params.proxyids=$data.hostid;
     $params.with_triggers = "1"
     $responseHostOnProxyObj = $zabbix.objectResponse("host.get", $params);
 
     #So for each host monitored by this proxy we are going to check all nodata() triggers.
 
     foreach ( $monitoredHost in $responseHostOnProxyObj.result){
         $params=[System.Dynamic.ExpandoObject]::new();
         $params.output=@("triggerid","expression","description");
         $params.hostids=$monitoredhost.hostid;
         $params.functions = "nodata"
         $params.selectDependencies = ""
         $responseTriggersOnMinitoredHostProxyObj = $zabbix.objectResponse("trigger.get", $params);
 
         foreach ( $nodataTrigger in $responseTriggersOnMinitoredHostProxyObj.result){
 
             # We do not make a self dependency, so check for that.
 
             if($nodataTrigger.triggerid.ToString().CompareTo($triggerToDepOn) -eq 0){ continue }
                 $alreadyDependant = $false
 
             # now check that this trigger doesn't already depend on our trigger
 
             foreach ($dep in $nodataTrigger.dependencies){
                 if($dep.triggerid.CompareTo($triggerToDepOn) -eq 0){
                     $alreadyDependant = $true
                     break
                 }
             }
 
             # and add a dependancy to the trigger of our proxy status if it does not already have one.
 
             if(-Not $alreadyDependant){
                 $params=[System.Dynamic.ExpandoObject]::new();
                 $params.triggerid=$nodataTrigger.triggerid;
                 $params.dependsOnTriggerid=$triggerToDepOn;
                 $responseAddTriggerDepObj = $zabbix.objectResponse("trigger.adddependencies", $params);
                 $responseAddTriggerDepObj.result
             }
         }
     }
 }
 $Zabbix.logout();

 


Some thoughts on this:

  • I run a script like this once a day. Meaning when new nodata() triggers are added to the system (discovered hosts, LLD etc), they do not have the dependency until the script runs.
  • You could run a version of this script after auto-registration, but it will never be perfect since LLD can add nodata() triggers, too.
  • Since this uses the API to loop all hosts and all nodata() triggers, I can imagine that this script causes some load on larger installations. Adding delays between API calls might be appropriate.

Please comment on how you handle “the noise” when one of your proxies goes down.

3 thoughts on “Proxy dependency using the API”

    1. I did … downloaded nuget.exe e run it from PowerShell

      PS C:\Temp> .\nuget.exe install Zabbix.NET
      PS C:\Temp> .\nuget.exe install Newtonsoft.Json

      Then, copy DLL files to your power shell script Path .. Now I am with other erros … 😉

  1. Any Idea ?

    PS C:\Temp> .\zbx-dependencias-nodata.ps1

    GAC Version Location
    — ——- ——–
    False v4.0.30319 C:\Temp\Newtonsoft.Json.dll
    False v4.0.30319 C:\Temp\ZabbixAPI.dll
    Exception calling “login” with “0” argument(s): “Could not load file or assembly ‘Newtonsoft.Json, Version=9.0.0.0,
    Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed’ or one of its dependencies. The system cannot find the file
    specified.”
    At C:\Temp\zbx-dependencias-nodata.ps1:20 char:2
    + $Zabbix.login();
    + ~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : FileNotFoundException

    Exception calling “objectResponse” with “2” argument(s): “Could not load file or assembly ‘Newtonsoft.Json,
    Version=9.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed’ or one of its dependencies. The system cannot find
    the file specified.”
    At C:\Temp\zbx-dependencias-nodata.ps1:28 char:2
    + $responseObj = $zabbix.objectResponse(“host.get”, $params);
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : FileNotFoundException

    Exception calling “logout” with “0” argument(s): “Could not load file or assembly ‘Newtonsoft.Json, Version=9.0.0.0,
    Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed’ or one of its dependencies. The system cannot find the file
    specified.”
    At C:\Temp\zbx-dependencias-nodata.ps1:117 char:2
    + $Zabbix.logout();
    + ~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : FileNotFoundException

Leave a Reply