11. 04. 2024
Business Service Monitoring, NetEye, SLM
Have you ever thought about how to monitor your NetEye system or other critical applications in a network failure scenario?
To manage this scenario, in some customer cases some solutions have been implemented using SMS notifications, thus relying on the support of the mobile network as a notification channel.
But what happens when even the mobile network doesn’t work?
Today we have a more effective solution thanks to Atlassian’s OpsGenie Heartbeats.
What is OpsGenie?
OpsGenie is used by many Wuerth Phoenix customers in addition to the NetEye ecosystem to achieve various objectives including:
- Optimizing NetEye alert management
- Managing the correct escalation of NetEye alerts and other alerts to different teams
- Managing the correct escalation of NetEye alerts and other alerts to different teams during on-call shifts and using the OpsGenie mobile app
But in this post I want to show you the Heartbeat Monitoring feature, which is very useful in case there’s some kind of network failure and NetEye can’t even reach OpsGenie.
What is a Heartbeat for OpsGenie?
In computer science, a heartbeat is a periodic signal generated by hardware or software to indicate normal operation or to synchronize other parts of a computer system. [1]
Atlassian defines OpsGenie Heartbeats in this way: “The Heartbeats feature can be used to ensure that your environment is able to connect to OpsGenie continuously and that:
- There are no network problems between your system and Opsgenie
- Your Heartbeat sender system is working as expected
Heartbeats can also be used to monitor that your periodic tasks are running as expected by using a suitable Heartbeat expiration interval regarding your periodic task run time.“
The idea of a heartbeat is really simple: if the heartbeat stops for a given period of time, you get alerted.
Important: The Heartbeat Monitoring feature is available to Standard and Enterprise plans only.
Configuring a Heartbeat
For details on configuring Heartbeats you can refer to the documentation provided by Atlassian from which I quote this short and important extract to help you better understand this functionality:
When a Heartbeat is added to Opsgenie with an interval of X minutes, your system is expected to send HTTP based Heartbeat requests periodically, at least every X minutes. If a Heartbeat request is not received for more than X minutes, Opsgenie will conclude that there is a problem between your system and Opsgenie, and create an alert according to your settings.
Heartbeat Example: “neteye-master-heartbeat-crontab” to Check if NetEye is Alive
In the following scenario, I created a heartbeat named neteye-master-heartbeat-crontab as follows:
So if within 10 minutes OpsGenie doesn’t receive a ping to this Heartbeat, it will send an alarm signal.
To generate the ping from NetEye you can proceed with a simple curl command in crontab which will ping the heartbeat created using an HTTP POST action.
curl test example
curl -X POST 'https://api.opsgenie.com/v2/heartbeats/neteye-master-heartbeat-crontab/ping' --header 'Authorization: GenieKey XXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX'
{"result":"PONG - Heartbeat received","took":0.007,"requestId":"f7a9fdf0-73b2-4f33-9346-5251d11a9978"}
crontab example
#OpsGenie heartbeat neteye-master-heartbeat-crontab ping
*/5 * * * * curl -s -X POST 'https://api.opsgenie.com/v2/heartbeats/neteye-master-heartbeat-crontab/ping' --header 'Authorization: GenieKey XXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX
Heartbeat Example: “neteye-master-heartbeat-check-scheduler” to Check if NetEye Checks are Working Correctly
In the following scenario, I created a heartbeat named neteye-master-heartbeat-check-scheduler as follows:
This Heartbeat is configured like the previous one, but with a different name. If within 10 minutes OpsGenie doesn’t receive a ping to this Heartbeat it will send an alarm signal, but we will ensure that the ping will be sent from a scheduled NetEye check.
The easiest way to create a scheduled NetEye check is to use this plugin from the OpsGenie GitHub release page, creating a command and a service check following this example:
'/neteye/shared/monitoring/plugins/oghb-linux-amd64' '-action' 'send' '-apiKey' 'XXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX' '-name' 'neteye-master-heartbeat-check-scheduler'
INFO[0000] Successfully sent heartbeat [neteye-master-heartbeat-check-scheduler]
Conclusions
OpsGenie allows for the creation of different Heartbeats that can be used for different purposes and in different systems to ensure that communication between the “monitored” system and OpsGenie is always valid. Otherwise, OpsGenie will send an alert.
In the examples above we’ve seen how to monitor NetEye to ensure that it’s always able to communicate with OpsGenie. Clearly these configurations can be extended to all critical systems for which the same guarantees are desired.
For further information on the integration between OpsGenie and NetEye you can also read this blog post.
[1] Wikipedia source
These Solutions are Engineered by Humans
Did you find this article interesting? Does it match your skill set? Our customers often present us with problems that need customized solutions. In fact, we’re currently hiring for roles just like this and others here at Würth Phoenix.
Giuseppe Di Garbo
Consultant at Würth Phoenix
Hi everybody. I’m Giuseppe and I was born in Milan in 1979. Since the early years of university, I was attracted by the Open Source world and operating system GNU\Linux. After graduation I had the opportunity to participate in a project of a startup for the realization of an Internet Service Provider. Before joining Würth Phoenix as SI consultant, I gained great experience as an IT consultant on projects related to business continuity and implementation of open source software compliant to ITIL processes of incident, change and service catalog management. My free time is completely dedicated to my wife and, as soon as possible, run away from Milan and his caotic time and trekking discover our beautiful mountain near Lecco for relax and lookup the (clean) sky.
Author
Latest posts by Giuseppe Di Garbo
23. 06. 2023
Icinga Web 2, NetEye
Scheduling Downtime on Linux Environments
28. 12. 2022
NetEye
Monitoring Automation in Director
21. 10. 2022
Icinga Web 2, ITOA, NetEye
A Custom Dashboard for Windows and Linux Servers