When you have to publicly release software like we do with NetEye every two months, it’s fundamental that users can verify that software: in our case that the RPMs that we build come from us and not from someone who pretends to be us, that they have not been tampered with, and that they do not harm their installations.
Attackers might, in fact, tamper with NetEye RPMs, uploading a malicious RPM with some evil payload to the repository, even allowing them to perform a supply chain attack against some other company.
To prevent the above scenario, we can digitally sign our RPMs.
Signing the RPM packages allows NetEye users to verify the integrity and authenticity of the software they download. This mechanism prevents the alteration of the software published on our repositories and, as a consequence, dramatically reduces the supply chain attack surface available for the attackers.
Signing the packages is also another important cornerstone in our path towards delivering more and more secure software.
RPMs are signed using public-key cryptography. For simplicity, lets imagine that a package maintainer wants to sign his/her RPM
my-good-rpm.rpm with his/her private key. Once signed, the RPM is published to a repository that supports signatures.
How can we now verify the signature of this RPM?
We can install the maintainer’s public key in our system and use
yum, for example, to carry out the installation. In this case,
yum uses the installed public key to verify the integrity of the package.
You can look at this Red Hat article with detailed instructions on how to sign an RPM.
As you can imagine, we produce several gigabytes of RPMs per NetEye release, and so using an error-prone manual process for signing our packages is out of the question. To automate the signing process, however, we not only need an automatic procedure, but also something that easily integrates into our build pipelines, plus a way to securely store our GPG keys.
After some digging around, we decided to give Sigul a try. Sigul is an automated GPG signing system used by Fedora to sign its RPMs and, even if it is not widely adopted outside the Fedora ecosystem, it fills our needs. I must be clear here: it is a powerful yet complex tool that requires care in its configuration and maintenance, and has a relatively steep learning curve.
We set up our Sigul infrastructure with the following components:
The Sigul client is the only machine allowed to send requests to the signing server. These requests are filtered by the bridge, which verifies that requests coming from the client are entitled to be communicated to the server. Once a new request has been completed by the server, the answer is then sent back to the client. The server is completely isolated, and can only connect to the bridge. GPG keys are encrypted and stored on the server.
We build our RPM packages in Jenkins pipelines, so we just had to extend a library we already use in our pipelines to add support for the signing of the RPMs, which is, in the end, a new stage in our pipelines that follows the testing phase, and before uploading the packages to our Pulp 2 repos.
The heavy lifting here is done by Ansible, which takes care of :
As I said above, the GPG keys we use to sign the RPMs (RPMs logically belong to different repos and they are signed with different keys) cannot be accessed from the outside, so requests are made by the Sigul client as needed. RPMs without a signature or with the wrong signature cannot be published to the Pulp 2 repos.
A high level overview of our signing process is shown in the diagram below.