07. 08. 2026
Damiano Chini
NetEye
When valid check results stop arriving: investigating an Overdue incident
How we investigated an unusual Icinga 2 performance issue with no obvious infrastructure cause.
One of the more interesting investigations I have worked on recently started with a puzzling symptom: normal check scheduling was delayed, but there was no apparent reason why.
CPU, memory, and disk I/O looked fine. The incident had started shortly after some monitored objects were removed, but at that point it was only a clue – not yet an explanation.
The symptom: delayed check scheduling
When normal check scheduling falls behind, affected objects can become Overdue. This can happen for familiar reasons: too many checks are scheduled at once, or the system is under pressure from CPU, memory, disk I/O, or another shared resource.
That was our starting point. However, the usual indicators looked healthy, so none of the obvious capacity-related explanations fit the symptom. We had to find out where in the architecture the delay was being introduced: Icinga 2, Icinga DB, Redis, MariaDB, Tornado, or an interaction between them.
The strongest clue came from the timing. Some monitored objects had been removed shortly before the incident, and we had evidence that Tornado was still attempting to submit passive check results for those objects after their deletion. This was not enough to call Tornado the root cause: an event pipeline can legitimately have results in flight while configuration changes propagate. But it gave us a specific condition to test.
What happens when Icinga 2 receives many
process-check-resultactions for objects that no longer exist?
To answer that question, we reproduced the API calls directly instead of reproducing the entire Tornado workflow. This let us check whether those failed requests alone could explain the delayed scheduling.
Reproducing the issue in a test environment
We wrote a small Python load generator that sent many process-check-result requests in parallel to random, non-existing objects.
The behavior was reproducible: requests became slow, normal checks were delayed, and existing objects eventually became Overdue. Running the same test against an existing object did not cause the same issue.
This was the key result. When the target object did not exist, processing stopped inside Icinga 2: there was no check result to write to Icinga DB or Redis. That allowed us to exclude the downstream data path and focus on what happened inside Icinga 2 while it handled the failed request.
The normal resource indicators did not explain the slowdown: CPU, memory, and disk I/O remained healthy. That led us away from capacity concerns and toward blocking or contention in the application code.
Comparing the two code paths
We then started reading and comparing the Icinga 2 source code for an existing and a non-existing target.
The successful path resolves the target and processes the result. The failing path handles an exception and creates DiagnosticInformation. This diagnostic work gathers stack-trace-related details and invokes dladdr() repeatedly.
A single failed request is not the problem. With thousands of concurrent failures, the dladdr() calls contend on the dynamic loader lock, _dl_load_lock. API workers spend time waiting, responses become slow, and normal check scheduling falls behind:
non-existing target → exception → diagnostic information
→ repeated dladdr() calls → loader-lock contention
→ delayed scheduling → checks become Overdue
To verify that contention was the actual root cause, we ran a local eBPF test while reproducing the workload. It showed increased latency in dladdr() – in some cases more than 64 ms – consistent with waiting on _dl_load_lock. This gave us concrete evidence that the slowdown was caused by lock contention in this path, rather than by CPU, memory, disk I/O, or Redis performance.
Why it happened after object removal
The trigger was not deletion alone. It was the combination of deleted objects, a passive result producer still sending results for them, and enough concurrent failed calls to amplify the diagnostic path.
Requests for existing objects complete normally and do not create the same diagnostic information. Requests for deleted objects fail as expected, but at high concurrency their error-handling path delayed unrelated monitoring work.
Reporting the upstream finding
Once we had a reliable reproduction and a plausible explanation, I opened an upstream issue with the Icinga project:
The report includes the reproduction, the comparison with existing objects, and the code-path analysis. We also tested a change that avoids collecting the costly stack-trace information in this case; in our test environment, it removed the problematic behavior.
The Icinga 2 maintainers were very collaborative and responsive during the upstream discussion. They developed a solution, and the fix was released in Icinga 2.16.4 as a fix for high response latency when multiple API requests target non-existing hosts or services.
The main lesson was to keep the investigation simple: start from the symptom, rule out the obvious causes, and test one small hypothesis at a time. Comparing a failing request with a working one pointed us to the right Icinga 2 code path. A local eBPF test then confirmed the cause before we reported it upstream.
Thanks to the Icinga community for the constructive upstream discussion, and to the colleagues involved in testing and analysis.
Damiano Chini
Author
