16. 01. 2024 Patrick Zambelli NetEye, Unified Monitoring

Icinga 2 DSL for Defining the Monitoring Status of Objects with Director

Today I want to present an Icinga 2-based monitoring use case where concepts of the powerful Icinga 2 DSL functional language come into play. The use case is based on mapping the status of a Host/Service Object via passive check results only. For this kind of use case, any accidental active status check could potentially overwrite the current status of the object.

Such use cases typically appear when monitoring incoming SNMP Traps, or mapping the status results provided by a third party Icinga 2 system. In these situations it’s important to avoid an accidental status change by an operator while performing for instance an active check or when clicking on the “Check now” action.

An experienced IcingaWeb2 administrator might think this should be solved via icingaweb2 role settings. Unfortunately as discussed in the open GitHub issue, it’s simply not possible to deny the “actions” permission for just some subset of hosts (i.e., the passive hosts) while simultaneously granting the same permission for other “active” objects. Therefore it’s necessary to prevent the status change on the passive hosts in some other way.

Any approach that implements a Plugin Script to calculate the status for a given Object and then returns that same status would be rather inefficient. In comparison, the Icinga 2 runtime variables provide all the required information, whereas the Icinga2 DSL allows them to be perfectly integrated in the monitoring configuration.

What the Icinga DSL language is

Icinga DSL (Domain Specific Language) is a powerful and flexible language used to define configuration objects and rules in Icinga. It allows users to write custom scripts and expressions to define monitoring behavior, notifications, and other aspects of the Icinga system. With Icinga DSL, users can create complex monitoring rules and configurations that would be difficult or even impossible to achieve using the standard configuration methods. It’s used within Icinga Director, a configuration tool for Icinga, to define advanced monitoring scenarios.

Some of the key advantages for using Icinga DSL include:

  • Flexibility: Icinga DSL lets you create custom scripts and expressions that you can use to define monitoring behavior, notifications, and other Icinga system aspects. This provides a high level of flexibility and allows users to create complex monitoring rules and configurations you otherwise couldn’t with the standard configuration methods like the Director GUI.
  • Powerful features: Icinga DSL includes a range of powerful features and functions, including support for variables, conditional statements, loops, and more. This allows users to create sophisticated monitoring rules and configurations that can adapt to changing conditions in real time. See the language reference guide

The Approach via the Icinga DSL Command

We start by defining a Host Command that can reflect the current status of a monitoring object onto itself in the case of an active check execution. The first step requires defining a command and argument value as a DLS instruction. When generating the DSL instruction with Director web UI, the surrounding “{{ }}” for defining the function is generated.

The function first resolves the host name of the check execution from the runtime variable. This can be accomplished by calling the macro function. In the next step we retrieve the desired host object by calling the “get_host()” method, and finally extract the desired attribute values from the returned object. That integer or string value is returned to the command variable.

To extract the current status code value, we can proceed with the following DSL Expression:

var strhost = macro("$host.name$")
var objhost = get_host(strhost)
var int_exit_status = objhost.last_check_result.state
return int_exit_status

Since the results of a host command should define both:

  • Status code
  • Output description

we slightly change the previous code snippet to the following:

var strhost = macro("$host.name$")
var objhost = get_host(strhost)
var str_output = objhost.last_check_result.output
return str_output

Given that, here’s how it looks in Director:

And here’s what the preview looks like:

If we apply this host command via a host template to the host object(s), we’ll see this result:

And if we perform the Check now action we see:

Inspecting Icinga 2 Object Runtime Values and Evaluating Config Expressions in the Icinga CLI Console

All this seems to be rather magical, but luckily there’s a nice approach to debugging Icinga 2 runtime values via the icinga2 console. Take a look at the documentation for more details about using the CLI console.

Now I’ll give you a short example of how to connect to the console in your NetEye environment and inspect our relevant runtime variable. Connect to the console and then run the following steps:

  • Extract the API credentials from the configuration file. To avoid any restrictions caused by lack of permissions, we connect to the API using the root user with permissions = “*”
# cat /neteye/shared/icinga2/conf/icinga2/conf.d/api-users.conf
/**
 * The ApiUser objects are used for authentication against the API.
 */
object ApiUser "root" {
  password = "your-secret-key"
  permissions = [ "*" ]
}
  • Export the username and password into environment variables
# export ICINGA2_API_USERNAME=root
# export ICINGA2_API_PASSWORD= your-secret-key
  • Start a console session and query for the objects as with the DSL example from above
# icinga2-master console --connect 'https://root@localhost:5665/'
Icinga 2 (version: r2.14.0-1)
<1> => var h = get_host("dummy-selfStatus")
<2> => h.last_check_result
{
        active = false
        check_source = "neteye4.mydomain"
        command = null
        execution_end = 1704906173.819249
        execution_start = 1704906173.819249
        exit_status = 0.000000
        output = "Host UP"
        performance_data = null
        previous_hard_state = 2.000000
        schedule_end = 1704906173.819249
        schedule_start = 1704906173.819249
        scheduling_source = " neteye4.mydomain "
        state = 0.000000
        ttl = 0.000000
        type = "CheckResult"
        vars_after = {
                attempt = 1.000000
                reachable = true
                state = 0.000000
                state_type = 1.000000
        }
        vars_before = {
                attempt = 1.000000
                reachable = true
                state = 2.000000
                state_type = 1.000000
        }
}
  • Now inspect the desired runtime value and voilà, you’ll see the same value that’s shown in the icingaweb2 UI:
<3> => h.last_check_result.state
0.000000
<4> => h.last_check_result.output
OK: Host UP

Enjoy interacting with you Icinga2 objects! 😉

Patrick Zambelli

Patrick Zambelli

Project Manager at Würth Phoenix
After my graduation in Applied Computer Science at the Free University of Bolzano I decided to start my professional career outside the province. With a bit of good timing and good luck I went into the booming IT-Dept. of Geox in the shoe district of Montebelluna, where I realized how a big IT infrastructure has to grow and adapt to quickly changing requirements. During this experience I had also the nice possibility to travel the world, while setting up the various production and retail areas of this company. Arrived at Würth Phoenix I started developing on our monitoring solution NetEye. Today, in my position as Consulting an Project Manager I am continuously heading to implement our solutions to meet the expectation of your enterprise customers.

Author

Patrick Zambelli

After my graduation in Applied Computer Science at the Free University of Bolzano I decided to start my professional career outside the province. With a bit of good timing and good luck I went into the booming IT-Dept. of Geox in the shoe district of Montebelluna, where I realized how a big IT infrastructure has to grow and adapt to quickly changing requirements. During this experience I had also the nice possibility to travel the world, while setting up the various production and retail areas of this company. Arrived at Würth Phoenix I started developing on our monitoring solution NetEye. Today, in my position as Consulting an Project Manager I am continuously heading to implement our solutions to meet the expectation of your enterprise customers.

Leave a Reply

Your email address will not be published. Required fields are marked *

Archive