30. 06. 2026
AI, Kubernetes
30. 09. 2026
Davide Sbetti
Automation, Development, Kubernetes, NetEye
Fewer Builds, More Testing: The New CI Behind the NetEye Operator
At NetEye we’re always working towards more and more integration with Kubernetes. The NetEye Operator is the first piece of that journey we’re working on, with the goal of obtaining a component that ships as a Kubernetes operator rather than as an RPM. The process of building it gave us a rare opportunity, a chance to design its CI from scratch instead of simply extending something that already exists.
We used that opportunity to change where the effort goes. More of the checking happens in fast tests that run at every pull request, fewer expensive builds are needed to get a change validated, and the install, update and upgrade paths are exercised far more often than before. Let’s take a look together at what we’re working on!
Where We Were
Our established model is one distinct build per release. A release of a NetEye component through an RPM is prepared with a pull request, which runs an entire NetEye testing pipeline that is then merged, tagged and built again in staging, with the entire suite of tests also being performed on the tagged version. Installation has long been covered by automated testing, while updates and upgrades were more often verified closer to the release itself.
That model works, and it has an appealing property: Everything we release has gone through that intense testing pipeline. However, it could be more efficient since the same long build is currently needed for each release, yet some intense tests are still generally run more often at the end of the release cycle, in particular to validate upgrade paths.
Push the Cheap Tests Down
The first thing we changed has nothing to do with the nightly builds. It’s simply that far more is now covered by tests that need no build phase at all.
Every pull request runs:
- Go unit tests across the whole module, with the race detector enabled
- Controller integration tests against a real Kubernetes API server using
envtest, also with the race detector - Unit tests for the release tooling itself – the version-consistency checks, the release-source validation, and the Python module that manipulates the OLM catalog
- A drift check that regenerates the deepcopy code, the CRDs and the OLM bundle, and fails if any generated file changed
Pull requests also build both the operator image and the bundle image, complete with SBOM and provenance attestations – but they don’t publish them. The build is proven to work, and nothing is pushed to a registry that would need to be cleaned up later. The entire testing pipeline mentioned above isn’t run for each pull request, since at this level we’d like to push the responsibility more toward those single components that guarantee the compatibility of the API they expose. However, the full integration testing pipeline isn’t something we’re forgetting…
Build Only When There Is Something to Build
In fact, a scheduled candidate build runs against the main branch every night. This is reasonable since the main branch always represents the version that is to be deployed in the NetEye release currently under development, and thus the process doesn’t affect the backporting of bugfixes in already-released versions, which follow a more direct pipeline.
Before it builds anything, it works out whether the branch has actually moved – not by consulting a state file or a marker tag we maintain, but by reading the commit hash embedded in the most recently published candidate tag in the container registry:
neteye-operator:0.1.5-nightly-a1b2c3d4e5f6
└ version ┘ └12-char commit┘
If the current commit already matches, there’s nothing to do and no build happens.
That tag, incidentally, isn’t a version claim. It exists so that the end-to-end suite has something concrete to install. The version number comes later, only after the full pipeline actually validates it.
One Candidate, Three Paths
When the branch has moved, the candidate operator image and bundle image are built in parallel and published, with the same SBOM and provenance attestations a release gets (a candidate built differently from a release wouldn’t tell us much about the release).
The bundle is then published into a rolling candidate channel in our file-based OLM catalog. The workflow resolves the current NetEye release line, updates the catalog in its own repository, validates the result with opm validate, opens a pull request with a signed commit using a narrowly scoped token, waits for that repository’s own checks, and merges.
This step is what makes the rest possible. An OLM-packaged operator is delivered through a catalog, a channel and a cluster extension – so publishing a bundle into a channel is an update event, and changing the declared product version on the NetEye resource is an upgrade request. Both of the operations we most want to test have a single, declarative trigger that a pipeline can “pull”.
From one published candidate, three end-to-end runs then start in parallel:
| install | A clean installation if the candidate succeeds |
| update | An existing installation takes the candidate as an in-channel update |
| upgrade | A NetEye product upgrade succeeds under the candidate operator |
The three runs share a single definition and differ only in which transition they ask for. They also don’t run on a new bespoke harness; they drive the same RHEL-based NetEye build pipeline we already rely on, extended with the parameters it needs in order to know which operator channel and version to use and which transition to perform.
Each run retries a small number of times before giving up to ensure we get a consistent result regardless of any potential flaky tests. Furthermore, being a full-fledged integration pipeline as above, this guarantees a nightly check on the consistency of our work and integrations.
A Release Becomes a Promotion
The release workflow has no human trigger in the normal path. It waits for the check run reported by the end-to-end suite and proceeds only when that suite has passed.
Only then is a version number assigned. The size of the bump is derived from the labels of the pull requests merged since the last release tag. The workflow updates the version, commits it, creates an annotated tag that GitHub signs so it shows as verified, builds and publishes the release images, and adds the bundle to the stable channel for the relevant NetEye release line – referenced by immutable digest, never by tag.
That ordering matters more than it might seem. If you were to put the version number, a validation failure would leave you choosing between moving a tag and discarding a version number. Neither is a good option, and making the number an output of validation removes the choice entirely.
Furthermore, this entire process is automatic, and human intervention is only needed if the nightly build fails in one of the three validation paths, to ensure that potential bugs in the process are promptly fixed.
Where We Are Now
This is work in progress, and the pieces described above are built and running internally, but they’re still being rolled out and tuned. Some stages are still being carried out in a controlled way before they take over autonomously, and parts of the supporting product pipeline are still morphing into their final form. We expect the full flow to become productive over the next few releases.
What we’re confident about is the direction. Push verification earlier, into tests that cost mere seconds. Build only what needs building. Validate one candidate through every path that matters – install, update and upgrade, and let the version number be the last thing that happens.
The NetEye Operator is being developed out in the open at github.com/neteye-platform/neteye-operator, so if you’re the curious type, stop by to give it a look and share your feedback with us!
These Solutions are Engineered by Humans
Did you find this article interesting? Are you an “under the hood” kind of person? We’re really big on automation and we’re always looking for people in a similar vein to fill roles like this one as well as other roles here at Würth IT Italy.
Davide Sbetti
Hi! I'm Davide and I'm a Software Developer with the R&D Team in the "IT System & Service Management Solutions" group here at Würth IT Italy. IT has been a passion for me ever since I was a child, and so the direction of my studies was...never in any doubt! Lately, my interests have focused in particular on data science techniques and the training of machine learning models.
Author
Latest posts by Davide Sbetti
30. 06. 2026
AI, Kubernetes
Load-balancing Requests to LLMs in Kubernetes: A KV-cache Approach with llm-d!