Grok is a plug-in installed by default in Logstash, which is supplied with the Elastic package (the ELK – Elasticsearch, Logstash and Kibana), one of the integrated modules in our NetEye Unified Monitoring solution.
What is this plug-in for?
First of all, Grok is an English neologism that means “to understand profoundly, intuitively or by empathy, to establish a rapport with”. Grok (the plug-in) is the best way to parse an unstructured log by translating it into structured data.
In fact, it’s thanks to Grok that we are able to interpret the logs received from Logstash, where it extrapolates the fields that will be indexed in Elasticsearch and displayed in Kibana.
By making matches, Grok detects the correspondences in each line in a log and stores the data we need by inserting it into a database field.
When does Grok help us?
Suppose we add a new firewall in the NetEye Log Management that will send its logs to rsyslog. These logs, according to the applied configurations, will be saved in the directory /var/log/rsyslog/ and read by Logstash. However, Logstash on its own is not able to interpret a log. Grok, instead, will read the content line by line to extrapolate the fields that we have defined.
How are a Logstash file and the syntax of Grok constructed?
I admit that the syntax of a Logstash file is not that simple, but with a little work, we can soon understand it.
filter {
if [type] == “syslog” and [message] =~ /.*Safed\[\d+\]\[\d+\].*/ {
grok {
match => [ “message”, “%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:logsource} Safed\[%{POSINT}\]\[%{POSINT}\]:%{SPACE}+ %{GREEDYDATA:message}” ]
add_field => [ “received_at”, “%{@timestamp}” ]
add_tag => “SAFED”
overwrite => [ “message” ]
break_on_match => false
}
In this excerpt from the NetEye Logstash files, we see how the configuration file is structured:
filter – identifies a Logstash component that is able to process the logs it receives
grok – identifies the plug-in used within Logstash to interpret the log and extract the data
This will allow us to build our Grok rules by checking in real time the match with the log line.
One word of advice is to always start with GREEDYDATA (which matches everything) and gradually remove the other parts of the log starting from the beginning.
With a little practice, you’ll realize that it’s simpler than it seems.
Hi, my name is Massimiliano and I'm the youngest SI Consultant in Würth Phoneix (or my colleagues are very old).
I like: my son Edoardo (when he doesn’t cry), my pet-son Charlie, photography, mountains, linux os, open-source technology and everything I don't know.
I don't like: giving up, the blue screen of Windows, the buffering while I’m watching a movie, latecomers and fake news on internet.
I worked for the VEGA project of the European Space Agency and now I'm very happy about being landed in this company.
I'm ready to share all of my knowledge and my passion whit our customers.
Author
Massimiliano De Luca
Hi, my name is Massimiliano and I'm the youngest SI Consultant in Würth Phoneix (or my colleagues are very old).
I like: my son Edoardo (when he doesn’t cry), my pet-son Charlie, photography, mountains, linux os, open-source technology and everything I don't know.
I don't like: giving up, the blue screen of Windows, the buffering while I’m watching a movie, latecomers and fake news on internet.
I worked for the VEGA project of the European Space Agency and now I'm very happy about being landed in this company.
I'm ready to share all of my knowledge and my passion whit our customers.
Have you ever thought about how to monitor your NetEye system or other critical applications in a network failure scenario? To manage this scenario, in some customer cases some solutions have been implemented using SMS notifications, thus relying on the Read More
In my previous blog post, we saw how it's possible to index some documents that we created by crawling our NetEye User Guide, then applying the ELSER model in Elasticsearch to create a bag of words for searching that takes Read More
Once upon a time (in fact it was just a month ago, but it sounds more dramatic this way) I had the opportunity to attend a webinar about Vector Search, Generative AI, and modern NLP by the Elastic Team. One Read More
The event of the year, the NetEye User Group, is back! The User group is not only a chance to inform our customers about new products and releases, but also an occasion to meet and exchange feedback and ideas. This Read More
...also this year, Würth Phoenix & Gravitate organized the annual Usergroup DACH 2023 in Nuremberg. The Usergroup is not only a chance to inform our customers about new products and releases, but also an occasion to meet and exchange feedback Read More