03. 05. 2023
Anomaly Detection, ITOA, NetEye
Veeam is a widely used and well-known backup system.
A customer recently asked me if he could check on the operation of his Veeam-based backup system by verifying the Windows event log, since the standard checks used within the community did not provide him with the current status of his Veeam infrastructure.
This use case requires that the customer be able to check whether over the last X hours there was a specific event with a particular event ID and message. Receiving this event allows the customer to be sure that the backup was successful!
If this event is not present in the last X hours, then the backup is taking too long and the customer would like to be informed of that fact with a critical message, as he has to manually adjust the system in order to prevent any data corruption or overloading.
Design
To implement this use case we decided to use a plugin made available by Icinga called icinga-powershell-plugins, whose documentation can be found on this page.
Among the various plugins there, we found it useful to implement our use case with Invoke-IcingaCheckEventlog whose documentation can be found at this link.
In order to use it we installed the Icinga agent and then the Powershell plugins, using the following commands after starting a Powershell session with administrative rights.
Install-Module 'icinga-powershell-plugins';
Use-IcingaOnce that’s done, we were ready to use our Windows client-side script.
I ran the command by hand to check which options to use, then after setting the command I imported the newly available basket so we could use it.
We then create the host, configure the service linked to the command in question called Invoke-IcingaCheckEventlog, and get a result.
Event Monitoring
We looked at that command’s documentation where we saw that event 190 can be monitored. Furthermore, we also verified that the name of the job is inside the message, so you can go and check whether in the last 24 hours there were any instances of event 190 containing the name of the job we scheduled. If there is no such event, it means that the job is running long and therefore we need to manually intervene, and so for this reason we can raise an alarm.
The standard command code doesn’t allow an alarm to be raised when 0 events are found. So we then proceeded to change the PowerShell code to allow for this capability.
We set the various variables as needed before we test the command from the PowerShell command line. Here’s an example command we used:
Invoke-IcingaCheckEventlog -IncludeEventId 190 -Verbosity 3 -IncludeMessage "*'NAME OF THE JOB'*’STATUS OF THE JOB’*" -After 24h -critical 1:I can also check the actual presence of the backup files using icinga-powershell-check-directory, which allows me to check whether or not the backup file is present.
Again we set the various variables as needed before we test the command in PowerShell. Here’s an example command:
Invoke-IcingaCheckDirectory -Path D:\Veeam_Repository\Backup\ -FileNames *.vib -Critical 1: -ChangeYoungerThan 80m -FileSizeGreaterThan 100MBIn the end we’ll have two commands along with the service templates that can use them on the Veeam host, and we can therefore satisfy our customer’s use case: monitoring whether in the last X hours a Veeam job has run correctly and has produced a file of at least X megabytes contained in a particular path.
Franco Federico
Hi, I’m Franco and I was born in Monza. Over the last 20 years I worked for IBM in various roles. I started as a customer service representative (help desk operator), then I was promoted to Windows expert. In 2004 I changed again and was promoted to consultant, business analyst, then Java developer, and finally technical support and system integrator for Enterprise Content Management (FileNet). Several years ago I became fascinated by the Open Source world, the GNU\Linux operating system, and security in general. And so in the last 4 years during my free time I studied security systems and computer networks in order to extend my knowledge. I came across several open source technologies including the Elastic stack (formerly ELK), and started to explore them and other similar ones like Grafana, Greylog, Snort, Grok, etc. I like to script in Python, too. In addition to studying in my free time I dedicate myself to my family (especially my little daughter) and I like walking, reading, dancing and making pizza for friends and relatives.
Author
Latest posts by Franco Federico
10. 12. 2021
APM, NetEye, Real User Experience, Visual Synthetic Monitoring
NEP Alyvix – Ready for Use
07. 05. 2021
Log Management, Log-SIEM, NetEye
Collecting Network and DNS Logs on Your Infrastructure