With the introduction of the Composable Index Templates in Elastic, we at NetEye had to redesign the way index settings and mappings are applied to the indices generated by El Proxy.
In this post I’ll explain:
The solution explained in the remainder of the post was designed for El Proxy, but it may apply also to other use cases where you need to create new Index Templates based on other existing Index Templates.
Elastic 7.8 introduced the new Composable Index Templates, which supersede the legacy Index Templates and have a different logic in the way they can be both defined and applied to the indices.
The main differences are that:
With El Proxy, we have the following use case: we want the data coming from Beats Agents (e.g., Filebeat) to receive the settings defined by the Beats Agent’s Index Template, and if the index name matches a particular pattern, we want to define a particular ILM policy for that index.
Achieving this with legacy Index Templates was pretty straight forward. The Beats Agent would simply install its legacy Index Templates in Elasticsearch (e.g., a templates matching the index pattern
filebeat-7.8.0-*), and then El Proxy would add an additional legacy Index Template to apply the ILM policy to indices with a predefined pattern (e.g.
With this solution, an index with the name
filebeat-7.8.0-elproxysigned-example_tenant-six_months-example_tag-2022.01.01 would match both the legacy Index Template of Filebeat and the legacy Index Template of El Proxy, meaning that the index will receive all the needed settings for Filebeat and for El Proxy.
This approach cannot be taken with Composable Index Templates because only one template can match at a time. So either the settings of the Beats Agent’s Composable Index Template are applied to the index, or those of the El Proxy Composable Index Template. Thus we, the NetEye R&D Team, needed to create a way where we could apply both the settings of the Beats Agent and of El Proxy to a given index pattern.
To solve the problem described above we couldn’t just simply modify the Beats Agent’s Composable Index Templates because we wanted to apply the El Proxy settings only in the case where the index name matched a particular pattern.
Our solution instead was to create a separate Composable Index Template matching both the Beats Agent index pattern and the El Proxy index pattern. Following the previous example, the new Composable Index Template would match the index pattern
filebeat-7.8.0-elproxysigned-*-six_months-*-*. This Composable Index Template then only needs to import:
This solution allows us to address our use case while also keeping things modular and reusable, which is the very reason why Component and Composable Templates were introduced into Elastic. This means that we can reuse the Component Templates multiple times with the settings and mappings of the Beats Agent for each of the different ILM policies, and we can reuse the Component Template with the El Proxy ILM policy for each type and version of Beats Agents.
The solution comes with downside that Beats Agents’ templates by default are not split into Component Templates, so the Component templates
<agent>-<version>@mappings need to be created by the user by transforming the Composable Template definition into Component Templates. Luckily this is easily scriptable (and is actually scripted in NetEye) with the Elastic APIs because the syntax for defining Composable Templates is the same as the one for defining Component Templates.
Are you passionate about performance metrics or other modern IT challenges? Do you have the experience to drive solutions like the one above? Our customers often present us with problems that need customized solutions. In fact, we’re currently hiring for roles just like this and other roles here at Würth Phoenix.