Grok is a plug-in installed by default in Logstash, which is supplied with the Elastic package (the ELK – Elasticsearch, Logstash and Kibana), one of the integrated modules in our NetEye Unified Monitoring solution.
What is this plug-in for?
First of all, Grok is an English neologism that means “to understand profoundly, intuitively or by empathy, to establish a rapport with”. Grok (the plug-in) is the best way to parse an unstructured log by translating it into structured data.
In fact, it’s thanks to Grok that we are able to interpret the logs received from Logstash, where it extrapolates the fields that will be indexed in Elasticsearch and displayed in Kibana.
By making matches, Grok detects the correspondences in each line in a log and stores the data we need by inserting it into a database field.
When does Grok help us?
Suppose we add a new firewall in the NetEye Log Management that will send its logs to rsyslog. These logs, according to the applied configurations, will be saved in the directory /var/log/rsyslog/ and read by Logstash. However, Logstash on its own is not able to interpret a log. Grok, instead, will read the content line by line to extrapolate the fields that we have defined.
How are a Logstash file and the syntax of Grok constructed?
I admit that the syntax of a Logstash file is not that simple, but with a little work, we can soon understand it.
filter {
if [type] == “syslog” and [message] =~ /.*Safed\[\d+\]\[\d+\].*/ {
grok {
match => [ “message”, “%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:logsource} Safed\[%{POSINT}\]\[%{POSINT}\]:%{SPACE}+ %{GREEDYDATA:message}” ]
add_field => [ “received_at”, “%{@timestamp}” ]
add_tag => “SAFED”
overwrite => [ “message” ]
break_on_match => false
}
In this excerpt from the NetEye Logstash files, we see how the configuration file is structured:
filter – identifies a Logstash component that is able to process the logs it receives
grok – identifies the plug-in used within Logstash to interpret the log and extract the data
This will allow us to build our Grok rules by checking in real time the match with the log line.
One word of advice is to always start with GREEDYDATA (which matches everything) and gradually remove the other parts of the log starting from the beginning.
With a little practice, you’ll realize that it’s simpler than it seems.
Hi, my name is Massimiliano and I'm the youngest SI Consultant in Würth Phoneix (or my colleagues are very old).
I like: my son Edoardo (when he doesn’t cry), my pet-son Charlie, photography, mountains, linux os, open-source technology and everything I don't know.
I don't like: giving up, the blue screen of Windows, the buffering while I’m watching a movie, latecomers and fake news on internet.
I worked for the VEGA project of the European Space Agency and now I'm very happy about being landed in this company.
I'm ready to share all of my knowledge and my passion whit our customers.
Author
Massimiliano De Luca
Hi, my name is Massimiliano and I'm the youngest SI Consultant in Würth Phoneix (or my colleagues are very old).
I like: my son Edoardo (when he doesn’t cry), my pet-son Charlie, photography, mountains, linux os, open-source technology and everything I don't know.
I don't like: giving up, the blue screen of Windows, the buffering while I’m watching a movie, latecomers and fake news on internet.
I worked for the VEGA project of the European Space Agency and now I'm very happy about being landed in this company.
I'm ready to share all of my knowledge and my passion whit our customers.
On February 3rd and 4th, 2024, we attended FOSDEM, a major event where thousands of free and open-source software developers from around the world gather to exchange ideas and collaborate. This year I dedicated much of the second day to Read More
Introduction: Unveiling Elastic APM in Containerized Environments In today's dynamic digital landscape, where every interaction matters, understanding the intricacies of application performance has become paramount. Elastic APM is a powerful toolset within the Elastic Stack included in the NetEye SIEM Read More
In this article, we’ll explore how to configure the “Agent Binary Download” setting and set up your own artifact registry for binary downloads within a NetEye cluster. Prerequisites Before we begin, ensure you have the following prerequisites in place: Your Elastic Agents Read More
We fixed the following issues in the integration between NetEye and Alyvix. Test Case file selection dropdown We fixed an issue in the Test Cases view for which, when switching between the Test Cases of different nodes, the wrong Test Read More
We fixed the following issues in the integration between NetEye and Alyvix. Test Case file selection dropdown We fixed an issue in the Test Cases view for which, when switching between the Test Cases of different nodes, the wrong Test Read More