How to Monitor a Complex Veeam-based Backup System
Veeam is a widely used and well-known backup system.
A customer recently asked me if he could check on the operation of his Veeam-based backup system by verifying the Windows event log, since the standard checks used within the community did not provide him with the current status of his Veeam infrastructure.
This use case requires that the customer be able to check whether over the last X hours there was a specific event with a particular event ID and message. Receiving this event allows the customer to be sure that the backup was successful!
If this event is not present in the last X hours, then the backup is taking too long and the customer would like to be informed of that fact with a critical message, as he has to manually adjust the system in order to prevent any data corruption or overloading.
Design
To implement this use case we decided to use a plugin made available by Icinga called icinga-powershell-plugins, whose documentation can be found on this page.
Among the various plugins there, we found it useful to implement our use case with Invoke-IcingaCheckEventlog whose documentation can be found at this link.
In order to use it we installed the Icinga agent and then the Powershell plugins, using the following commands after starting a Powershell session with administrative rights.
Once that’s done, we were ready to use our Windows client-side script.
I ran the command by hand to check which options to use, then after setting the command I imported the newly available basket so we could use it.
We then create the host, configure the service linked to the command in question called Invoke-IcingaCheckEventlog, and get a result.
Event Monitoring
We looked at that command’s documentation where we saw that event 190 can be monitored. Furthermore, we also verified that the name of the job is inside the message, so you can go and check whether in the last 24 hours there were any instances of event 190 containing the name of the job we scheduled. If there is no such event, it means that the job is running long and therefore we need to manually intervene, and so for this reason we can raise an alarm.
The standard command code doesn’t allow an alarm to be raised when 0 events are found. So we then proceeded to change the PowerShell code to allow for this capability.
We set the various variables as needed before we test the command from the PowerShell command line. Here’s an example command we used:
Invoke-IcingaCheckEventlog -IncludeEventId 190 -Verbosity 3 -IncludeMessage "*'NAME OF THE JOB'*’STATUS OF THE JOB’*" -After 24h -critical 1:
I can also check the actual presence of the backup files using icinga-powershell-check-directory, which allows me to check whether or not the backup file is present.
Again we set the various variables as needed before we test the command in PowerShell. Here’s an example command:
In the end we’ll have two commands along with the service templates that can use them on the Veeam host, and we can therefore satisfy our customer’s use case: monitoring whether in the last X hours a Veeam job has run correctly and has produced a file of at least X megabytes contained in a particular path.
Hi, I’m Franco and I was born in Monza. Over the last 20 years I worked for IBM in various roles. I started as a customer service representative (help desk operator), then I was promoted to Windows expert. In 2004 I changed again and was promoted to consultant, business analyst, then Java developer, and finally technical support and system integrator for Enterprise Content Management (FileNet). Several years ago I became fascinated by the Open Source world, the GNU\Linux operating system, and security in general. And so in the last 4 years during my free time I studied security systems and computer networks in order to extend my knowledge. I came across several open source technologies including the Elastic stack (formerly ELK), and started to explore them and other similar ones like Grafana, Greylog, Snort, Grok, etc. I like to script in Python, too. In addition to studying in my free time I dedicate myself to my family (especially my little daughter) and I like walking, reading, dancing and making pizza for friends and relatives.
Author
Franco Federico
Hi, I’m Franco and I was born in Monza. Over the last 20 years I worked for IBM in various roles. I started as a customer service representative (help desk operator), then I was promoted to Windows expert. In 2004 I changed again and was promoted to consultant, business analyst, then Java developer, and finally technical support and system integrator for Enterprise Content Management (FileNet). Several years ago I became fascinated by the Open Source world, the GNU\Linux operating system, and security in general. And so in the last 4 years during my free time I studied security systems and computer networks in order to extend my knowledge. I came across several open source technologies including the Elastic stack (formerly ELK), and started to explore them and other similar ones like Grafana, Greylog, Snort, Grok, etc. I like to script in Python, too. In addition to studying in my free time I dedicate myself to my family (especially my little daughter) and I like walking, reading, dancing and making pizza for friends and relatives.
In my previous post, we saw how it's possible to build a simple Grafana Data Source Plugin, which we can use to read data from whatever source we'd like to use. In particular, we used it to read data from Read More
NetEye relies on many agents in order to monitor just one server, some examples are: Icinga, Telegraf, Elastic beats, GLPI agent and so on. As a Site Reliability Engineer, I'm responsible for ensuring that all these agents run smoothly. This Read More
Scenario Veritas Backup Exec is widely used for the backup of physical machines through local installed agents both on Windows and Linux hosts. It also supports application backup for Microsoft Exchange, SharePoint and several different Database engines, with the option Read More
A very important, fast-evolving area during the latest NetEye releases has been multi-tenancy. In a system with many tenants, the most complex aspect is probably the proper and orderly management of user permissions. To help administrators in this task, we Read More
If you're familiar with the NetEye SIEM module you probably also know El Proxy, the solution integrated into NetEye to ensure the integrity and inalterability of the logs produced by the SIEM module. Since its introduction in NetEye, the only Read More