Fewer Builds, More Testing: The New CI Behind the NetEye Operator
NetEye is working always more and more towards an integration with Kubernetes. The NetEye Operator is the first piece of that journey we are working on, with the goal of obtaining a component that ships as a Kubernetes operator rather than as an RPM, and building it gave us a rare opportunity: a chance to design its CI from scratch instead of simply extending something that already existed.
We used that opportunity to change where the effort goes: more of the checking happens in fast tests that run on every pull request, fewer expensive builds are needed to get a change validated, and the install, update and upgrade paths are exercised far more often than before. Let’s give together a look at what we are working on!
Where we were
Our established model is one distinct build per release. A release of a NetEye component through an RPM is prepared with a pull request, which runs an entire NetEye testing pipeline, that is then merged, tagged and built again in staging, with the whole suits of test being performed also on the tagged version. Install has long been covered by automated testing; update and upgrade were verified more often closer to the release itself.
That model works, and it has an appealing property: everything you release went through that intense testing pipeline. However, this could be more efficient since the same long build is currently needed for each release and still some intense tests are generally run more often at the end of the release cycle, in particular to validate upgrade paths.
Push the cheap tests down
The first thing we changed has nothing to do with nightly builds. It is simply that far more is now covered by tests that need no build at all.
Every pull request runs:
- Go unit tests across the whole module, with the race detector enabled;
- controller integration tests against a real Kubernetes API server using
envtest, also with the race detector; - unit tests for the release tooling itself — the version-consistency checks, the release-source validation, and the Python module that manipulates the OLM catalog;
- a drift check that regenerates the deepcopy code, the CRDs and the OLM bundle, and fails if any generated file changed.
Pull requests also build both the operator image and the bundle image, complete with SBOM and provenance attestations — but they do not publish them. The build is proven to work; nothing is pushed to a registry that would later need cleaning up. The entire testing pipeline mentioned above is not run for each pull request, since at this level we would like to push more the responsibility towards the single components which guarantee the compatibility of the API they expose. However, the full integration testing pipeline is not something we are forgetting…
Build only when there is something to build
In fact, a scheduled candidate build runs against the main branch every night. This is reasonable since the main branch always represents the version that is to be deployed in the NetEye release current under development and hence the process does not affect the backport of bugfixes in already released versions, which follow a more direct pipeline.
Before it builds anything, it works out whether the branch has actually moved — not by consulting a state file or a marker tag we maintain, but by reading the commit hash embedded in the most recently published candidate tag in the container registry:
neteye-operator:0.1.5-nightly-a1b2c3d4e5f6
└ version ┘ └ 12-char commit ┘
If the current commit already matches, there is nothing to do and no build happens.
That tag, incidentally, is not a version claim. It exists so that the end-to-end suite has something concrete to install. The version number comes later, only after the full pipeline actually validates it.
One candidate, three paths
When the branch has moved, the candidate operator image and bundle image are built in parallel and published, with the same SBOM and provenance attestations a release gets — a candidate built differently from a release would not tell us much about the release.
The bundle is then published into a rolling candidate channel in our file-based OLM catalog. The workflow resolves the current NetEye release line, updates the catalog in its own repository, validates the result with opm validate, opens a pull request with a signed commit using a narrowly scoped token, waits for that repository’s own checks, and merges.
This step is what makes the rest possible. An OLM-packaged operator is delivered through a catalog, a channel and a cluster extension — so publishing a bundle into a channel is an update event, and changing the declared product version on the NetEye resource is an upgrade request. Both of the operations we most want to test have a single, declarative trigger that a pipeline can pull.
From one published candidate, three end-to-end runs then start in parallel:
| install | a clean installation of the candidate succeeds |
| update | an existing installation takes the candidate as an in-channel update |
| upgrade | a NetEye product upgrade succeeds under the candidate operator |
The three runs share a single definition and differ only in which transition they ask for. They also do not run on a new bespoke harness: they drive the same RHEL-based NetEye build pipeline we already rely on, extended with the parameters it needs to know which operator channel and version to use and which transition to perform.
Each run retries a small number of times before giving up, to ensure we get a consistent result not matter possible flaky tests. Furthermore, being a full-fledged integration pipeline as above, this guarantees a nightly check on the consistency of our work and integrations.
Release becomes a promotion
The release workflow has no human trigger in the normal path. It waits for the check run reported by the end-to-end suite and proceeds only when that suite has passed.
Only then is a version number assigned. The size of the bump is derived from the labels of the pull requests merged since the last release tag. The workflow updates the version, commits it, creates an annotated tag that GitHub signs so it shows as verified, builds and publishes the release images, and adds the bundle to the stable channel for the relevant NetEye release line — referenced by immutable digest, never by tag.
That ordering matters more than it might appear. When the version number comes first, a validation failure leaves you choosing between moving a tag and discarding a version number. Neither is a good option, and making the number an output of validation removes the choice entirely.
Furthermore, this entire process is automatic and an human intervention is only needed if the nightly build fails in one of the three validation paths, to ensure that potential bugs in the process are promptly fixed.
Where we are
This is work in progress and the pieces described above are built and running internally, but they are still being rolled out and tuned: some stages are still being exercised in a controlled way before they take over completely, and parts of the supporting product pipeline are still moving into their final form. We expect the full flow to be productive over the coming releases.
What we are confident about is the direction. Push verification earlier, into tests that cost seconds. Build only what needs building. Validate one candidate through every path that matters — install, update and upgrade and let the version number be the last thing that happens.
The NetEye Operator is developed in the open at github.com/neteye-platform/neteye-operator, so, in case you are curious, stop by to give it a look and share with us your feedbacks!
Davide Sbetti
Author
