27. 06. 2022 Attilio Broglio NetEye

Automatically Cleaning up Old Services in NetEye 4

Introduction

In this article I’ll introduce a feature that some customers have frequently asked for. The customers’ request is linked to the fact that automation (whether via Tornado, ad-hoc scripts, etc.) generate a lot of services “on the fly”, for example as shown in my previous blog post. These services are created, updated and eventually manually or automatically closed. Once the problem is resolved, they are no longer needed by the monitoring system and so should be automatically cleaned from NetEye.

Solution

My approach is based on a bash script I named check_service_delete_icingcli_by_hard_state.sh.

This script performs two actions:

Calls NetEye 4’s Icinga API in order to obtain the filtered list of services
Use the icingacli command in order to remove the old unused services

I’ve put some code snippets detailing these at the bottom of this article.

The script is inserted into the /neteye/shared/monitoring/plugins/ directory and is then configured via a command/service with the following parameters:

Once the command and service have been created, the service is added to a target host. It has to be properly configured by setting these 3 fields:

Icinga delete days: Defines the maximum number of days that a service has had since its last Hard State change
Icinga delete state: Defines which Hard State must be used in order to filter services
Icinga delete string: Defines the value of the custom_variable “created_by” in order to filter only services that have this specific property

In the following example the script deletes all services with the custom_variable “created_by” set to apic_fault_check, and that have had their Hard Status set to OK for more than one day. All other services are ignored.

In the NetEye Dashboard, the Service provides basic information like this:

The Code

And here are the code snippets I promised you:

1) Call Director API in order to have all services with predefined filter:

#Call Director API
curl -k -s -u director:@PASSWORD@ -H 'Accept: application/json' -X GET 'https://@FQDN@:5665/v1/objects/services' -d "$(generate_post_data)" | jq -r '.results[] | [.name, .attrs.last_hard_state_change] | @csv' >/$DIR/$var'_delete.tmp'

2) Remove services via the Icinga CLI:

if (($diff > $d2s));then
    icingacli director service delete "$servicename" --host "$hostname" >/dev/null
    if [ $? -eq 0 ]; then
        echo "<strong>DELETED Successfull</strong>: Host $hostname - Service $servicename - Last_Check: $diffg days ago"
        echo ""
    else
        echo "ERROR during DELETE Host $hostname - Service $servicename - Last_Check: $diffg days ago"
        echo ""
    fi
else
    echo "Service not deleted: Host: $hostname - Service: $servicename - Last_Check: $diffg days ago"
    echo ""
fi

These Solutions are Engineered by Humans

Did you find this article interesting? Are you an “under the hood” kind of person? We’re really big on automation and we’re always looking for people in a similar vein to fill roles like this one as well as other roles here at Würth Phoenix.