30. 12. 2023
Atlassian, Service Management
When you have to publicly release software like we do with NetEye every two months, it’s fundamental that users can verify that software: in our case that the RPMs that we build come from us and not from someone who pretends to be us, that they have not been tampered with, and that they do not harm their installations.
Attackers might, in fact, tamper with NetEye RPMs, uploading a malicious RPM with some evil payload to the repository, even allowing them to perform a supply chain attack against some other company.
RPM Signature
To prevent the above scenario, we can digitally sign our RPMs.
Signing the RPM packages allows NetEye users to verify the integrity and authenticity of the software they download. This mechanism prevents the alteration of the software published on our repositories and, as a consequence, dramatically reduces the supply chain attack surface available for the attackers.
Signing the packages is also another important cornerstone in our path towards delivering more and more secure software.
How to sign RPM packages
RPMs are signed using public-key cryptography. For simplicity, lets imagine that a package maintainer wants to sign his/her RPM my-good-rpm.rpm with his/her private key. Once signed, the RPM is published to a repository that supports signatures.
How can we now verify the signature of this RPM?
We can install the maintainer’s public key in our system and use yum, for example, to carry out the installation. In this case, yum uses the installed public key to verify the integrity of the package.
You can look at this Red Hat article with detailed instructions on how to sign an RPM.
Automating the signing of RPM package
As you can imagine, we produce several gigabytes of RPMs per NetEye release, and so using an error-prone manual process for signing our packages is out of the question. To automate the signing process, however, we not only need an automatic procedure, but also something that easily integrates into our build pipelines, plus a way to securely store our GPG keys.
After some digging around, we decided to give Sigul a try. Sigul is an automated GPG signing system used by Fedora to sign its RPMs and, even if it is not widely adopted outside the Fedora ecosystem, it fills our needs. I must be clear here: it is a powerful yet complex tool that requires care in its configuration and maintenance, and has a relatively steep learning curve.
The Sigul infrastructure
We set up our Sigul infrastructure with the following components:
- The Sigul signing server, an isolated machine not accessible from the outside except for connections coming from the Sigul bridge
- A Sigul bridge, which accepts connections from the Sigul server and from the Sigul client
- A Sigul client, which sends signing requests to the Sigul bridge
The Sigul client is the only machine allowed to send requests to the signing server. These requests are filtered by the bridge, which verifies that requests coming from the client are entitled to be communicated to the server. Once a new request has been completed by the server, the answer is then sent back to the client. The server is completely isolated, and can only connect to the bridge. GPG keys are encrypted and stored on the server.
Automated GPG signing of RPM packages with Jenkins, Ansible, and Sigul
We build our RPM packages in Jenkins pipelines, so we just had to extend a library we already use in our pipelines to add support for the signing of the RPMs, which is, in the end, a new stage in our pipelines that follows the testing phase, and before uploading the packages to our Pulp 2 repos.
The heavy lifting here is done by Ansible, which takes care of :
- Configuring the Docker container used to build the packages
- Contacting the Sigul client to perform the signature
- Getting the signed RPMs back from the Sigul client
- Uploading the signed RPMs to the correct Pulp 2 repository
As I said above, the GPG keys we use to sign the RPMs (RPMs logically belong to different repos and they are signed with different keys) cannot be accessed from the outside, so requests are made by the Sigul client as needed. RPMs without a signature or with the wrong signature cannot be published to the Pulp 2 repos.
A high level overview of our signing process is shown in the diagram below.
Security wrapup
- The Sigul infrastructure is almost completely isolated, and the Sigul server cannot be accessed from outside
- We can completely avoid manipulating GPG keys, as they are safely stored on the Sigul server
- No secret is leaked either in Jenkins or in the Docker containers used for building
- Each repository has its own set of allowed keys, and RPMs must be signed to be published, in order to block any potentially malicious attempt to publish tampered RPMs
- RPMs are checked at installation time, and only those signed with our keys can be installed
Andrea Avancini
DevOps Engineer at Würth Phoenix
Loving understanding of how things work, how things can be automated, and how to apply new technologies when needed. Passionate about technology, open-source software, and security. I found Würth Phoenix the right place for this.
In the past, I co-founded a cybersecurity startup that produces security solutions for mobile apps and blockchain.
Previously, I worked as researcher at Fondazione Bruno Kessler of Trento. My research was mainly focused on web and mobile app security and testing.
I got my PhD in Computer Science at the University of Trento.
Author
Latest posts by Andrea Avancini
18. 10. 2021
NetEye
New Feature for NetEye 4.20
01. 10. 2021
Business Service Monitoring, NetEye
Hosts and NetEye Upgrade