15. 12. 2022 Rocco Pezzani NetEye

Strategies for Secret Management

In my previous blog post, we had a long discussion about how Icinga 2 manages its configuration in a Distributed Monitoring environment and how this can lead to unwanted disclosure of sensitive data, be it Secrets or Personal Data. The post ended with a set of recommendations/best practices that can be summarized in the statement “Sensitive Data must be enclosed in a Non-Global Zone“. While this single statement is easy to remember, each best practice points to a bad habit that must be discarded. Here’s a recap, just to be sure we all remember what we talked about:

Templates that are not assigned to any Zone (or are assigned to a Global Zone) must contain neither Secret nor Personal data; you can bind the required Data Fields, but do not fill in any Custom Variables
The best place to assign Secret or Personal Data is on a Host Object or Service Object: this way, sensitive data is available only where strictly necessary (the Agent Endpoint and its Parent Endpoints, that is the bare minimum)
If point 2 is not applicable, make sure Templates containing Secrets or Personal Data are explicitly assigned to a Zone that is not Global

Following these recommendations is fairly simple but, as always, coming up with a well-defined design is the best way to avoid future complications. To define a better solution, we should outline some behaviors and define some concepts that can be useful.

Some Basic Concepts

First of all, it’s of utmost importance to understand that an Icinga 2 Agent will receive content only from all the Icinga Global Zones. By no means does it receive data from its Parent Zone, whatever the data and whatever the Zone. This also means that an Agent Endpoint won’t even get the definition of its Host Object.

Let’s look at a quick example:

Server myserver.local is monitored by NetEye
Its Agent Endpoint is named myserver.local (it’s not possible otherwise)
Its Parent Zone is satellite-zone (but also master Zone is fine)
The related Host Object is named myserver.local and its attribute Icinga 2 Agent is set to Yes
The definition of Host Object myserver.local stays in the zone satellite-zone: it is not forwarded to the Agent Endpoint

At first this seems completely illogical, but if you think a little it does make sense: first, the HO is assigned to the Parent Zone, not to the Agent Zone itself; second, the Agent is merely an executor: monitoring is directly managed by one if its Parent Endpoints, making it unnecessary for the Endpoint to know anything other than the names of its Parent Endpoints. It’s the duty of the Parent Endpoints to request the execution of a specific Plugin on the Agent by transmitting all the required data each time. This makes things easier, because we know that an Icinga 2 Agent, our weakest point, can only get shared data across Global Zones.

Then we must start thinking about what Objects can be “shared” as a set of sensitive data: this makes sense because sensitive data is only known by Master or Satellite Endpoints, so grouping Objects together is a logical consequence. We should then define a sort of boundary where all the contained Objects must (or can) share the same data. Let’s call it the Security Domain: everything that shares the same Security Domain can access any sensitive data it contains.

Security Domains, Zones and Tenants

While Icinga 2 doesn’t have the concept of a Security Domain, we can capture the idea perfectly: an Icinga 2 zone IS effectively a security domain. We cannot say the same for a Tenant, because Icinga 2 is not able to represent this concept. But we know what a tenant is, and it’s imperative to find a way to overlap it with an Icinga 2 Zone. Furthermore, it should be remembered that an Icinga 2 Zone must contain a maximum of one tenant; if it contained more, then a tenant’s secrets would be shared with others, a scenario that we absolutely must avoid.

The main constraint is this: Icinga 2 Zones are not a hierarchical structure in terms of data propagation. With the exception of the Master Zone, which knows every Object, each Icinga 2 Zone receives only the data intended for it plus content from the Global Zones. Then, since a Zone is like a sealed box, we thus have only two possible configurations:

A Tenant is covered by exactly one Icinga 2 Zone
A Tenant is covered by two or more Icinga 2 Zones

Therefore, we can apply two possible layouts: Single-Zone Tenant and Multi-Zone Tenant.

Single-Zone Tenant

For a Single Zone Tenant, the Security Domain seamlessly overlaps the Tenant, which means that all monitored objects of a Tenant are assigned to a single Icinga 2 Zone. This is the easiest approach to implement and manage because it only requires that single Icinga 2 Zone. But precisely for this reason it’s the least flexible and scalable:

Only two Icinga 2 Endpoints can manage an Icinga 2 Zone: it’s not possible to scale horizontally (by adding more Endpoints) and must instead scale vertically (by adding resources to the two Endpoints)
No other Icinga 2 Zones are allowed: all monitored objects must be reachable from the 2 Endpoints, even if they are in a DMZ

Multi-Zone Tenant

For a Multi-Zone Tenant, the Security Domain overlaps only part of the Tenant, which means that two or more Icinga 2 Zones are required to fully “cover” a Tenant. This layout requires more management effort: Secrets that overlap one or more Security Domains (literally, if Host Objects assigned to different Icinga 2 Zones require access to one or more common Secrets) must be manually added and updated on all involved Icinga 2 Zones. But this layout is much more flexible:

Can scale both vertically (adding resources to Endpoints) and horizontally (adding additional Icinga 2 Zones with 2 brand new Icinga 2 endpoints)
Devices placed inside a highly secure boundary (such as a DMZ) can be monitored by adding a new dedicated Endpoint

Which layout should I use

Surprisingly, the answer is quite simple and the main reason is that, sadly, there really isn’t a smart way to propagate data between Icinga 2 Zones. The Single-Zone Tenant can easily evolve into a Multi-Zone Tenant: just add a new Icinga Zone 2 when and where it’s needed. The only thing you need to plan ahead for is the naming convention: if the name of a Zone of a Single-Zone Tenant has the same format as that of the Zones of a Multi-Zone Tenant (e.g.: mytenant_myzone_01), then you can switch between layouts simply by adding or removing Zones. This demonstrates that by planning “a little” before creating Zones, the Single-Zone Tenant layout is just like the Multi-Zone Tenant layout. So, as always, it’s just a matter of design, and a very simple one too, which you can easily deal with by thinking about it a little in advance.

Securing Secrets with NEP

The layouts described above can also be applied with NetEye Extension Packs. Like Icinga 2, the NEP framework doesn’t provide the concept of Security Domain, but it is possible to use some of the Templates provided by nep-common to make a zone behave like a Security Domain.

The Host Template Type nx-ht-type-location has been provided by NEP with the objective of setting location data on each Host Object, be it geographical location (like coordinates, address or similar) or logical (Icinga 2 Zone). When a child of nx-ht-type-location is created and assigned to a specific Icinga 2 Zone, it will become (by our definition) a Security Domain. Therefore, it’s possible to bind to it all the necessary Data Fields and fill them with any sort of Secret or Personal Data. To avoid issues, remember:

Bind the Data Field whenever you want on the hierarchy of nx-ht-type-custom, but assign it a value only where a Cluster Zone is explicitly assigned or inherited; never where Cluster Zone is empty
If a Template inherits a Cluster Zone, don’t try to change it: you’ll end up breaking the whole Icinga 2 Configuration
If you’re using a Multi-Zone Tenant layout, don’t try to create a Location Type Template for the Tenant: since it won’t have a value for the Cluster Zone, it will be sent to every Endpoint linked to your monitoring infrastructure, breaking the confidentiality of the whole system

Conclusion

Like almost every scenario, security becomes an issue when the design is poor, or simply does not fit the right requirements. Security must also be a driver in planning and implementing a layout. Therefore, a good NetEye administrator is responsible for planning the most secure design as possible. Also, in order to achieve the maximum available security level, everyone must consider the use of distributed monitoring to forcibly segregate Secrets and Personal Data, and find the right balance between the size of the NetEye infrastructure and the company’s security needs.

These Solutions are Engineered by Humans

Did you like this article? Does it reflect your skills? We often get interesting questions straight from our customers who need customized solutions. In fact, we’re currently hiring for roles just like this and other roles here at Würth Phoenix.