11. 04. 2024
Business Service Monitoring, NetEye, SLM
28. 10. 2022
Enrico Alberti
Log Management, Log-SIEM, NetEye
Syslog Collection with Elastic under Distributed NetEye Monitoring
Anyone who has joined the beautiful world of logging has collided, sooner or later, with the collection via syslog protocol.
More than 40 years have passed since syslog was invented, and in that time there have been several attempts by the IETF to create a standard around this world (RFC 3164 and RFC 5424). Even today this protocol is a pain whenever we want to collect log data in a STANDARD and Unified Monitoring System.
With NetEye and the Elastic Stack we’ve had quite an easy life, because the most important vendor/devices are parser-ready thanks to the Filebeat Modules and the new Elasticsearch Integrations.
For example, if we have a Cisco Router XYZ (with IP 10.200.36.40) we can enable the module cisco on our Filebeat under NetEye, choose the syslog_port settings (like 5003), load the necessary Ingest Pipeline for related parsing, and in 5 minutes we’ve collected our router logs.
Amazing!!! But…What’s the Catch?
Many devices, due to their age or because they are black-box devices, can unfortunately only send their syslog output to the default 514/UDP port.
This “constraint” is a problem not just for our NetEye environment but also for compliance with modern “Security Best Practice”:
- Ports below
1024are reserved for root/privileged users (based on RFC 1340 by IANA) - Processes and services should be run with the Principle of Least Privilege (POLP)
- Filebeat instances by default run with limited privileges (on NetEye as the
logstashuser)
Obviously, the above points can be resolved if we run the Filebeat service with Super User privileges (strictly not recommended). But then the problem becomes: what if we have MORE THAN ONE log source type that can ONLY send to port 514? How can I distinguish between the traffic flows, and send them to different Filebeat Modules?
The only way to do that is to add a new layer between socket and Filebeat to dispatch the traffic based on specific criteria (usually the source IP address of the device).
First Approach: Use the Native Rsyslog Daemon
As with all standard Linux systems, NetEye also has the Rsyslog program installed and pre-configured to read all logs from port 514.
We only need to create a new config file with the source IP filters and a forwarding action. If we have more than one source type, we can duplicate the file with different port settings for forwarding.
Suppose that we have two Cisco Router XYZs (with IPs 10.200.36.40 and 10.200.36.41). A sample configuration file might be:
if ( $fromhost-ip == '10.200.36.40' or $fromhost-ip == '10.200.36.41' ) then {
action(type="omfwd"
target="filebeat.neteyelocal" port="5003" protocol="udp"
action.resumeRetryCount="-1"
queue.filename="example_fwd"
queue.type="linkedList"
queue.size="10000"
queue.saveOnShutdown="on")
stop
}
Where:
target, port, protocolare the settings of our Filebeat instance into NetEyequeue.type="linkedlist"enables a LinkedList in-memory queuequeue.filenamedefines the disk storage. Backup files are created with the example_fwd prefix, in the working directory specified by the preceding globalworkDirectorydirective- The
action.resumeRetryCount -1setting prevents rsyslog from dropping messages when trying to reconnect to a server that’s not responding - Enabling
queue.saveOnShutdown="on"saves in-memory data if rsyslog shuts down
| Pros | Cons |
|---|---|
| Turnkey solution (no custom installation) | Rule-based config (not easy to manage) |
| Can be balanced on clusters via PCS | Cannot be balanced between multiple-instances (fail-over only) |
| Logs are not exactly as the original, because rsyslog applies by default a template (RSYSLOG_TraditionalForwardFormat) |
Second Approach: Use a Proxy like NGINX
NGINX provide us with another useful solution that can also be used to balance traffic between multiple instances of Filebeat, a feature that’s vital in a distributed environment.
First of all, we have to remember to disable every program that’s listening on the default Syslog port (514). On NetEye this is managed by Rsyslog.
The second step is creating the configuration files under the Nginx config folder.
We create a mapping file (i.e. map.conf) that is used to identify the sources of traffic that have to be forwarded. In this file we can insert the source IP addresses of our Cisco Routers and map them to the variable $backend_srv with the value filebeat_cisco_ios, which is an alias for our remote addresses.
map $remote_addr $backend_srv {
10.200.36.40 "filebeat_cisco_ios";
10.200.36.41 "filebeat_cisco_ios";
}
We also need to configure for server listening and upstream forwarding:
upstream filebeat_cisco_ios {
least_conn;
server filebeat.neteyelocal:5003;
}
server {
listen @@NETEYE_HOSTNAME@@:514 udp;
proxy_pass $backend_srv;
proxy_bind $remote_addr transparent;
}
Where:
least_conn, means that the next request is assigned to the server with the least number of active connectionsserveris the setting of our Filebeat instance into NetEyeNETEYE_HOSTNAMEis the public interface of the system or VIP Cluster IPtransparentallows us to forward a client’s real IP address to the server
| Pros | Cons |
|---|---|
| Easy to manage the config for the customer (mapping file) | Must change the default product config to disable Rsyslog |
| Can be balanced on cluster via PCS | |
| Ready for distributed Filebeats by adding servers to the upstream configuration |
Conclusion
Both solutions are valid, but in a distributed environment with more than one Filebeat instance, only NGINX is a viable option. These approaches can also be extended to Logstash instances if the customer wants.
Enrico Alberti
I’ve always been fascinated by the IT world, especially by the security environment and its architectures.
The common thread in my working experience is the creation of helpful open-source solutions to easily manage the huge amount of security information.
In the past years, my work was especially focused on Cyber Kill Chain, parsing and ELK Stack but in order to start from the beginning...
In 2010 I left my birthplace, the lovely Veneto, looking for a new ´cyber´ adventure in Milan. After graduating in Computer Systems and Networks Security, I worked for 6 years as a Cyber Security Consultant.
During the first 5 years, I explored the deep and manifold world of cybersecurity, becoming passionate about open source solutions. After that, I decided to challenge myself joining a Start-up company focusing on SOC services (I’m a proud member of the Blue Team!).
In Wuerth Phoenix, I would like to personalize the NetEye System for each one of our costumers, in order to develop the perfect product for their needs, by combining all my past experiences and skills.
Author
Latest posts by Enrico Alberti
28. 12. 2023
Log Management, Log-SIEM, NetEye
Monitor Fleet Elastic Agents with NetEye Extension Packs (NEP)
19. 07. 2022
Contribution, NetEye, Unified Monitoring
Integration of Centreon Plugins into NetEye Extension Packs
24. 12. 2021
Log Management, NetEye
Log Management through NetEye Satellites