29. 11. 2021 Attilio Broglio NetEye

Dynamically Manage Traps from a PowerMe UPS in NetEye’s Tornado

UPSs are critical devices that need particular attention when you’re monitoring them. Most of them are not very performant with respect to SMNP requests you send them. For this reason, passive monitoring via SNMP traps is the best solution. In this way, monitoring doesn’t burden the hardware with useless requests.

In this session we’ll look at a solution for this problem applied to PowerMe devices using the Tornado engine.

Solution

Thanks to the Tornado collector, SNMP trap notifications are collected and arrive at the Tornado engine in JSON format. Here’s an example of a PowerMe trap message:

{
	"type": "snmptrapd",
	"created_ms": 1624352535574,
	"payload": {
		"protocol": "UDP",
		"PDUInfo": {
			"errorindex": 0,
			"version": 0,
			"messageid": 0,
			"community": "public",
			"transactionid": 462441,
			"requestid": 0,
			"notificationtype": "TRAP",
			"errorstatus": 0,
			"receivedfrom": "UDP: [10.0.0.1]:37721->[192.168.0.1]:162"
		},
		"src_port": "37721",
		"dest_ip": "192.168.0.1",
		"src_ip": "10.0.0.1",
		"oids": {
			"SNMPv2-MIB::snmpTrapOID.0": {
				"datatype": "OID",
				"content": "UPS-MIB::upsTraps.0.3"
			},
			"UPS-MIB::upsAlarmId": {
				"content": "2",
				"datatype": "INTEGER"
			},
			"UPS-MIB::upsAlarmDescr": {
				"content": "UPS-MIB::upsAlarmOnBattery",
				"datatype": "OID"
			},
			"SNMP-COMMUNITY-MIB::snmpTrapCommunity.0": {
				"datatype": "STRING",
				"content": "public"
			},
			"SNMPv2-MIB::snmpTrapEnterprise.0": {
				"content": "UPS-MIB::upsTraps",
				"datatype": "OID"
			},
			"SNMP-COMMUNITY-MIB::snmpTrapAddress.0": {
				"datatype": "IpAddress",
				"content": "10.14.13.131"
			},
			"DISMAN-EVENT-MIB::sysUpTimeInstance": {
				"datatype": "Timeticks",
				"content": "(543280916) 62 days, 21:06:49.16"
			}
		}
	}
}

All the fields in this JSON structure are important for monitoring this device, but in this use case we’ll focus on the following 3 fields, two of which are specific to the MIB OID, while one is a generic field.

OID	VALUE
src_ip	10.0.0.1
oids.SNMPv2-MIB::snmpTrapOID.0	UPS-MIB::upsTraps.0.3
oids.UPS-MIB::upsAlarmDescr	UPS-MIB::upsAlarmOnBattery

The field “src_ip” provides the information about which UPS(IP) is the source of the alarm. This partial information is important in order to link this alarm to the proper host in NetEye. Later on in this article we’ll show how we implement a basic Lookup Table to provide the host name.

The field “oids.SNMPv2-MIB::snmpTrapOID.0” defines the kind of trap message. Looking at the MIB of these devices, traps are grouped into 4 main categories, but alarms are notified using the values 3 and 4.

OID VALUE	OID ALIAS	MEANING
3	upsTrapAlarmEntryAdded	Alarm inserted into the alarm table
4	upsTrapAlarmEntryRemoved	Alarm is removed from the alarm table

So when the field “oids.SNMPv2-MIB::snmpTrapOID.0” contains:

UPS-MIB::upsTraps.0.3 – an alarm has been activated
UPS-MIB::upsTraps.0.4 – an alarm has been removed

The field “oids.UPS-MIB::upsAlarmDescr” provides information about the kind of alarm that activated the trap, in our example it’s “UPS-MIB::upsAlarmOnBattery”.

Once we understand these fields we can build our Tornado Filter/Rules in order to process the trap. The first step is to create the folder/files structure. Here we start with a filter and then add the set of rules that Tornado applies to traps:

powerme_filter
- filter.json
- powerme_rules
  - 0000000000_snmptrap_powerme_upsAlarmOnBattery.json

In filter.json we describe a filter that combines the following 2 constraints:

The field “src_ip” must be present in the set of IP addresses used for PowerMe devices

The field “SNMPv2-MIB::snmpTrapOID.0” must be equal to “UPS-MIB::upsTraps.0.3” or “UPS-MIB::upsTraps.0.4

{
  "description": "Manage PowerMe UPS trap",
  "active": true,
  "filter": {
    "type": "AND",
    "operators": [
      {
        "type": "equals",
        "first": "${event.type}",
        "second": "snmptrapd"
      },
      {
        "type": "OR",
        "operators": [
          {
            "type": "equals",
            "first": "${event.payload.src_ip}",
            "second": "10.0.0.1"
          },
          {
            "type": "equals",
            "first": "${event.payload.src_ip}",
            "second": "10.0.0.2"
          }
         ]
      },
      {
        "type": "OR",
        "operators": [
          {
            "type": "equals",
            "first": "${event.payload.oids.\"SNMPv2-MIB::snmpTrapOID.0\".content}",
            "second": "UPS-MIB::upsTraps.0.3"
          },
          {
            "type": "equals",
            "first": "${event.payload.oids.\"SNMPv2-MIB::snmpTrapOID.0\".content}",
            "second": "UPS-MIB::upsTraps.0.4"
          }
        ]
      }
    ]
  }
}

The “powerme_rules” folder contains all the rules used to describe the actions that Tornado has to perform when a specific trap arrives.

Our one rule is defined with 3 macro blocks: Where, With, and Actions. The first one consists of a description and an additional filter, which only takes into account traps generated for a specific alarm, i.e. upsAlarmOnBattery.

The With section defines some variables and constraints that are used in this section (if one of the variables is not set correctly, the rule will not be applied).

These variables are very important, because by dynamically using alarm_status_value and alarm_status_txt, we don’t need to define rules twice. Indeed it’s sufficient for a single rule to include alarm-fired and alarm-resolved. In the following steps I’ll describe the solution we adopted.

The field “host_name” provides the exact match with the NetEye hostname of the Powerme UPS. Traps arrive with their IP information rather than the host name (the IP is needed in order to create/link a host/service). We use a modifier that maps the IP to the desired host_name:

      "host_name": {
        "from": "${event.payload.src_ip}",
        "regex": {
          "match": "(.+)",
          "group_match_idx": 0,
          "all_matches": false
        },
        "modifiers_post": [
          {
            "type": "ReplaceAll",
            "find": "10.0.0.1",
            "replace": "powerme01.neteyelocal",
            "is_regex": false
          },
          {
            "type": "ReplaceAll",
            "find": "10.0.0.2",
            "replace": " powerme01.neteyelocal ",
            "is_regex": false
          }
        ]
      }

A similar approach is used for “alarm_status_value”. Here we substitute the trap value “UPS-MIB::upsTraps.0.3” with the value 2, which is the NetEye value used for service status.

      "alarm_status_value": {
        "from": "${event.payload.oids.\"SNMPv2-MIB::snmpTrapOID.0\".content}",
        "regex": {
          "match": "(.+)",
          "group_match_idx": 0,
          "all_matches": false
        },
        "modifiers_post": [
          {
            "type": "ReplaceAll",
            "find": "UPS-MIB::upsTraps.0.3",
            "replace": "2",
            "is_regex": false
          },
          {
            "type": "ReplaceAll",
            "find": "UPS-MIB::upsTraps.0.4",
            "replace": "0",
            "is_regex": false
          }
        ]
      }

Also, “alarm_status_txt” maps the plugin output in the correct way with either “OK” or CRITICAL:

      "alarm_status_txt": {
        "from": "${event.payload.oids.\"SNMPv2-MIB::snmpTrapOID.0\".content}",
        "regex": {
          "match": "(.+)",
          "group_match_idx": 0,
          "all_matches": false
        },
        "modifiers_post": [
          {
            "type": "ReplaceAll",
            "find": "UPS-MIB::upsTraps.0.3",
            "replace": "CRITICAL",
            "is_regex": false
          },
          {
            "type": "ReplaceAll",
            "find": "UPS-MIB::upsTraps.0.4",
            "replace": "OK",
            "is_regex": false
          }
        ]
      }

So with these variables properly defined, we create our Action smart_monitoring_action in order to link the trap to the NetEye host/service.

As an additional benefit, we can easily replicate this trap template for any kind of alarm we want to be notified about.

Conclusion

This Tornado SNMP trap example can be easily implemented and can be replicated for all types of PowerMe traps, and can be also used for devices that use the same trap structure.