NetEye 4 Log Management: Rsyslog and the Elastic Stack
When you need to manage and collect large amounts of data, there can be a lot of hard tasks to do. So we decided to take some of the best Open Source tools to help us do it in the best possible way. Let me introduce you to Rsyslog and the Elastic Stack implementation for NetEye 4 Log Management.
Rsyslog (Version 8)
As the official website says, Rsyslog is the rocket-fast system for log processing. It manages huge amounts of incoming log data, pre-processes it, and delivers it to local destinations at a rate of over one million messages per second.
The version we decided to adopt is version 8, the latest stable release. The main features for this version are:
Multi-threading
TCP, SSL, TLS, and RELP ready
Whole syslog message filtering
A fully configurable output format
We use it to collect data from multiple agents, and we configured it to write the logs into the file system. The configuration includes the rules to allow the agents to send log messages to the Rsyslog service, along with the directives to write the log files into the file system.
The rules for the agents are written in a fully automated way by the Icinga Web 2 Log Manager module, but they are placed in a conf.d/ directory so that when the Rsyslog service is started, manual configuration files can also be loaded by placing them in that same folder (/neteye/shared/rsyslog/conf/rsyslog.d).
The Elastic Stack (Version 6)
Built on an open source foundation, the Elastic Stack lets you reliably and securely take data from any source, in any format, and search, analyze, and visualize it in real time. It is made of three main products: Elasticsearch, Logstash and Kibana.
Logstash
Logstash is a server-side data processing pipeline that ingests data from many different sources and delivers it to a data storage (“stash”).
We use it to read the log files written by Rsyslog in real time and to send the content to Elasticsearch.
Elasticsearch
Elasticsearch is a distributed, RESTful search and analytics engine. It is the heart of the Elastic Stack and the core of the NetEye 4 log storage system.
Kibana
Kibana is a render engine for Elasticsearch data. It allows you to visualize and understand large amounts of data by aggregating them logically and graphically.
We created filters to tell it what data to organize for rendering graphs and charts.
Full Stack Developer at Wuerth Phoenix. I love questioning myself, find new challenges to learn and new adventures to grow up. PHP lover trying to expand my skills studying new languages and tools to improve my professional life.
Author
Gianluca Piccolo
Full Stack Developer at Wuerth Phoenix. I love questioning myself, find new challenges to learn and new adventures to grow up. PHP lover trying to expand my skills studying new languages and tools to improve my professional life.
On February 3rd and 4th, 2024, we attended FOSDEM, a major event where thousands of free and open-source software developers from around the world gather to exchange ideas and collaborate. This year I dedicated much of the second day to Read More
Introduction: Unveiling Elastic APM in Containerized Environments In today's dynamic digital landscape, where every interaction matters, understanding the intricacies of application performance has become paramount. Elastic APM is a powerful toolset within the Elastic Stack included in the NetEye SIEM Read More
In this article, we’ll explore how to configure the “Agent Binary Download” setting and set up your own artifact registry for binary downloads within a NetEye cluster. Prerequisites Before we begin, ensure you have the following prerequisites in place: Your Elastic Agents Read More
We fixed the following issues in the integration between NetEye and Alyvix. Test Case file selection dropdown We fixed an issue in the Test Cases view for which, when switching between the Test Cases of different nodes, the wrong Test Read More
We fixed the following issues in the integration between NetEye and Alyvix. Test Case file selection dropdown We fixed an issue in the Test Cases view for which, when switching between the Test Cases of different nodes, the wrong Test Read More