29. 04. 2016 TobiasGoller Log Auditing, NetEye

Disk Space Optimization for the Index Database of NetEye Log Management

Disk Space 1

As you already know, from version 3.6 we’ve integrated the Elastic Stack (consisting of Elasticsearch, Logstash and Kibana) to the NetEye Log Management.

This integration provides a lot of additional possibilities for log analysis, log correlation, dashboard creations, etc.

Furthermore, it allows to store the collected logs for different periods, which wasn’t possible on prior NetEye versions.

Your NetEye Log Management receives all logs created in your company (Windows Eventlog, Linux Syslogs, Firewall Access logs, VPN logs, etc.). With its filters, Logstash indexes all data and writes them into the Elasticsearch Index Database of the NetEye Log Management.

Now imagine you are collecting logs of 95 systems and together they produce an average of 1000 events per second with peaks of nearly 3000 events per second. These systems produce every day at least 90 GByte of index data on your disk. I think I don’t have to go into further detail to show the importance of optimizing the disk space from time to time =)

In order to optimize your disk space (without adding additional resources), you will have to delete the logs when they aren’t required anymore. Those logs which are required to satisfy the data protection authority are marked with a special tag. In this way, filters can be simply set on these tags. Thanks to the created filters, you will be able to automatically delete the data required for the Italian data protection authority (“garante della privacy”) after 6 months. All other logs can obviously be deleted already after 2 months.

In this way you are able to benefit from two advantages:

  • First, you are able to satisfy the rules of the data protection authority.
  • Second, you are able to use one single software (NetEye Log Management) to collect and analyze all types of logs.

This is an additional reason why you have no more excuses to not collect all logs using the NetEye Log Management module. =)

 

TobiasGoller

TobiasGoller

Consultant at Würth Phoenix
I’m Tobias and work as SI Consultant on different fields, for commercial products like VMware, Microsoft, Citrix but also for opensource projects like Nagios, OCS Inventory, GLPI NagVis, ntop and the best practice standard ITIL. I have also some certifications for this kind of activities which helps me to improve my activity on the job. I like to introduce the best solutions for the customer necessaries independently if it is a commercial or free product. One of my favorite hobbies is playing in the local music for wind band. As I live in the mountains I love hiking and last but not least I try to spend much of my free time with my family.

Author

TobiasGoller

I’m Tobias and work as SI Consultant on different fields, for commercial products like VMware, Microsoft, Citrix but also for opensource projects like Nagios, OCS Inventory, GLPI NagVis, ntop and the best practice standard ITIL. I have also some certifications for this kind of activities which helps me to improve my activity on the job. I like to introduce the best solutions for the customer necessaries independently if it is a commercial or free product. One of my favorite hobbies is playing in the local music for wind band. As I live in the mountains I love hiking and last but not least I try to spend much of my free time with my family.

Leave a Reply

Your email address will not be published. Required fields are marked *

Archive