Disk Space Optimization for the Index Database of NetEye Log Management
As you already know, from version 3.6 we’ve integrated the Elastic Stack (consisting of Elasticsearch, Logstash and Kibana) to the NetEye Log Management.
This integration provides a lot of additional possibilities for log analysis, log correlation, dashboard creations, etc.
Furthermore, it allows to store the collected logs for different periods, which wasn’t possible on prior NetEye versions.
Your NetEye Log Management receives all logs created in your company (Windows Eventlog, Linux Syslogs, Firewall Access logs, VPN logs, etc.). With its filters, Logstash indexes all data and writes them into the Elasticsearch Index Database of the NetEye Log Management.
Now imagine you are collecting logs of 95 systems and together they produce an average of 1000 events per second with peaks of nearly 3000 events per second. These systems produce every day at least 90 GByte of index data on your disk. I think I don’t have to go into further detail to show the importance of optimizing the disk space from time to time =)
In order to optimize your disk space (without adding additional resources), you will have to delete the logs when they aren’t required anymore. Those logs which are required to satisfy the data protection authority are marked with a special tag. In this way, filters can be simply set on these tags. Thanks to the created filters, you will be able to automatically delete the data required for the Italian data protection authority (“garante della privacy”) after 6 months. All other logs can obviously be deleted already after 2 months.
In this way you are able to benefit from two advantages:
- First, you are able to satisfy the rules of the data protection authority.
- Second, you are able to use one single software (NetEye Log Management) to collect and analyze all types of logs.
This is an additional reason why you have no more excuses to not collect all logs using the NetEye Log Management module. =)
Tobias Goller
Author
