27. 12. 2022 Damiano Chini Development, Log Management, Log-SIEM, NetEye

Adding Settings to Beats Agents’ Templates Based on the Index Name

With the introduction of the Composable Index Templates in Elastic, we at NetEye had to redesign the way index settings and mappings are applied to the indices generated by El Proxy.

In this post I’ll explain:

How the new Composable Index Templates logic has changed with respect to the legacy Index Templates
How these changes impact the way we need to apply settings to the El Proxy indices
How we managed to make sure that our El Proxy indices receive the settings needed by El Proxy and those needed by the specific Beats Agent

The solution explained in the remainder of the post was designed for El Proxy, but it may apply also to other use cases where you need to create new Index Templates based on other existing Index Templates.

Composable Index Templates vs Legacy Index Templates

Elastic 7.8 introduced the new Composable Index Templates, which supersede the legacy Index Templates and have a different logic in the way they can be both defined and applied to the indices.

The main differences are that:

Only one Composable Index Template can be applied to an index, based on its “priority”. On the contrary, many legacy Index Templates can match an index, and are applied to the index with a logic that merges different Legacy Index Templates based on their “order”.
Properties of the Composable Index Templates can be defined in separate Component Templates which can be associated with the Composable Index Template

How Composable Index Templates Change the Logic for El Proxy Indices

With El Proxy, we have the following use case: we want the data coming from Beats Agents (e.g., Filebeat) to receive the settings defined by the Beats Agent’s Index Template, and if the index name matches a particular pattern, we want to define a particular ILM policy for that index.

Achieving this with legacy Index Templates was pretty straight forward. The Beats Agent would simply install its legacy Index Templates in Elasticsearch (e.g., a templates matching the index pattern filebeat-7.8.0-*), and then El Proxy would add an additional legacy Index Template to apply the ILM policy to indices with a predefined pattern (e.g. *-*-elproxysigned-*-six_months-*-*).

With this solution, an index with the name filebeat-7.8.0-elproxysigned-example_tenant-six_months-example_tag-2022.01.01 would match both the legacy Index Template of Filebeat and the legacy Index Template of El Proxy, meaning that the index will receive all the needed settings for Filebeat and for El Proxy.

This approach cannot be taken with Composable Index Templates because only one template can match at a time. So either the settings of the Beats Agent’s Composable Index Template are applied to the index, or those of the El Proxy Composable Index Template. Thus we, the NetEye R&D Team, needed to create a way where we could apply both the settings of the Beats Agent and of El Proxy to a given index pattern.

Adding Settings to Beats Agents’ Composable Index Templates Based on the Index Name

To solve the problem described above we couldn’t just simply modify the Beats Agent’s Composable Index Templates because we wanted to apply the El Proxy settings only in the case where the index name matched a particular pattern.

Our solution instead was to create a separate Composable Index Template matching both the Beats Agent index pattern and the El Proxy index pattern. Following the previous example, the new Composable Index Template would match the index pattern filebeat-7.8.0-elproxysigned-*-six_months-*-*. This Composable Index Template then only needs to import:

A Component Template which contains the settings for the Beats Agent (e.g., filebeat.7.8.0@settings)
A Component Template which contains the mappings for the Beats Agent (e.g., filebeat.7.8.0@mappings)
A Component Template that defines the El Proxy settings (namely, the required ILM policy for the given index pattern)

This solution allows us to address our use case while also keeping things modular and reusable, which is the very reason why Component and Composable Templates were introduced into Elastic. This means that we can reuse the Component Templates multiple times with the settings and mappings of the Beats Agent for each of the different ILM policies, and we can reuse the Component Template with the El Proxy ILM policy for each type and version of Beats Agents.

The solution comes with downside that Beats Agents’ templates by default are not split into Component Templates, so the Component templates <agent>-<version>@settings and <agent>-<version>@mappings need to be created by the user by transforming the Composable Template definition into Component Templates. Luckily this is easily scriptable (and is actually scripted in NetEye) with the Elastic APIs because the syntax for defining Composable Templates is the same as the one for defining Component Templates.

These Solutions are Engineered by Humans

Are you passionate about performance metrics or other modern IT challenges? Do you have the experience to drive solutions like the one above? Our customers often present us with problems that need customized solutions. In fact, we’re currently hiring for roles just like this and other roles here at Würth Phoenix.