23. 01. 2020 Benjamin Gröber NetEye

How to Use Tornado for event based monitoring with Icinga 2 (Part 1/2)

Tornado is the spiritual successor of the NetEye EventHandler. Tornado is an open-source, rule based Event Processing engine designed to handle up to millions of events per second. We can leverage this capability to ingest all possibly interesting events of our entire infrastructure, and react to the effectively interesting ones.

In this blog post we will explore the possibility to use tornado to build a good starting point for event based monitoring. In this first part we will see how to teach Tornado to react to incoming events. The second part will explore how to test this configuration from both GUI and CLI, and put tornado under a little stress test.

For the sake of simplicity let’s leave out the first stage “collectors” and assume that your devices send their events directly to tornado. In the real world you will have your tornado collectors taking care of this step, and translate e.g. all Webhooks, Snmptraps, Emails or Syslogs sent to NetEye into Tornado Events.

Approach

We will now create a simple RuleSet that will react to events with the following characteristics:

  • Event Type equals blog
  • IP Address is in the range of our subnet 10.23.24.0/24
  • Whenever the payload starts with the word CRITICAL a passive check result for the service blog on the host with the same Address should be set
  • Whenever the payload starts with the word OK the same should happen with an OK state

The cleanest approach is to add a branch to out Processing tree, for only blog_event events. The content of our rules.d folder will look like this:

 rules.d/
 rules.d/blog
 rules.d/blog/filter.json
 rules.d/blog/blog_subnet
 rules.d/blog/blog_subnet/filter.json
 rules.d/blog/blog_subnet/blog_set_service
 rules.d/blog/blog_subnet/blog_set_service/020_service_blog_ok.json
 rules.d/blog/blog_subnet/blog_set_service/030_service_blog_critical.json

The rules.d/blog/filter.json will contain a simple filter, letting through only those events which have the blog_event type. The same way for rules.d/blog/blog_subnet/filter.json which will let through only events coming from the subnet 10.23.24.0/24 In this way the rest of the branch will not be traversed by all other events of tornado.

rules.d/blog
rules.d/blog/filter.json
rules.d/blog/blog_subnet
rules.d/blog/blog_subnet/filter.json
rules.d/blog/blog_subnet/blog_set_service
rules.d/blog/blog_subnet/blog_set_service/020_service_blog_ok.json
rules.d/blog/blog_subnet/blog_set_service/030_service_blog_critical.json

The rules.d/blog/filter.json will contain a simple filter, letting through only those events which have the blog_event type. The same way for rules.d/blog/blog_subnet/filter.json which will let through only events coming from the subnet 10.23.24.0/24 In this way the rest of the branch will not be traversed by all other events of tornado.

The next stage rules.d/blog/blog_subnet/blog_set_service is a RuleSet containing all Rules to be applied sequentially to events passing through. Using the icinga2 action, it will send a passive-check-result to Icinga2.

If you’re using the Tornado GUI it should look similar to this:

Configuration

For readability reasons I’ll show only a piece of the configuration in the following paragraphs. You can find a complete example, together with a script for replicating the setup on the bottom of this blog post.

Filter

For the first step of filtering, we’ll simply check the type of the event:

"filter": {
  "type": "equal",
  "first": "${event.type}",
  "second": "blog"
}

In the second step we’ll dynamically match the subnet we’re looking for on the address field:

"filter": {
  "type": "regex",
  "regex": "^10.23.24.",
  "target": "${event.payload.address}"
}

Rules

In the RuleSet we have two almost identical rules, differing only in the constraint.WHERE.regex and in the actions.payload.exit_code field. First we match the output, and check for either ^OK or ^CRITICAL

"WHERE": {
  "type": "regex",
  "target": "${event.payload.output}",
  "regex": "^OK\s"
}

When this is matched we want to trigger an icinga2 action, which sets the service state correctly:

"actions": [
  {
    "id": "icinga2",
    "payload": {
      "icinga2_action_name": "process-check-result",
      "icinga2_action_payload": {
        "exit_status": "0",
        "plugin_output": "${event.payload.output}",
        "filter": "host.address==\"${event.payload.address}\" && service.name==\"blog\"",
        "type": "Service"
      }
    }
  }
]

This is a little more verbose, but basically, we are calling the process-check-result endpoint of icinga2, with the given parameters, dynamically specifying the host, on which the blog service shall be affected.

As soon as the events start rolling your event log of the blog services will show activity which should look similar to the following:

If you want to try it out for yourself, you can build Tornado from the Tornado GitHub repository.

Resources

Rules, Setup Script and Stress Test: Download

Benjamin Gröber

Benjamin Gröber

R&D Software Architect at Wuerth Phoenix
Hi, my name is Benjamin and I'm Software Architect in the System Integration Research & Development Team at Wuerth Phoenix. I discovered my passion for Computers and Technology when I got my first PC shortly after my 7th birthday in 1999. Using computers and playing with them soon got boring and so, just a few years later, I taught myself Visual Basic and entered the world of Software Development. Since then I loved trying to keep up with the short lived, fast evolving IT world and exploring new technologies, eventually putting them to good use. Lately I'm investing my free time in the relatively new languages Go and Rust.

Author

Benjamin Gröber

Hi, my name is Benjamin and I'm Software Architect in the System Integration Research & Development Team at Wuerth Phoenix. I discovered my passion for Computers and Technology when I got my first PC shortly after my 7th birthday in 1999. Using computers and playing with them soon got boring and so, just a few years later, I taught myself Visual Basic and entered the world of Software Development. Since then I loved trying to keep up with the short lived, fast evolving IT world and exploring new technologies, eventually putting them to good use. Lately I'm investing my free time in the relatively new languages Go and Rust.

Leave a Reply

Your email address will not be published. Required fields are marked *

Archive