Package, Sign, and Verify Charts

In this blog, we can see how we can Package, Sign and Verify Helm charts.

After we finish building a chart, the next step is to start distributing it to our users or our organization. This is usually a 2 or 3-stage process:

1. Package
2. Sign (Optional but significant security-related improvement)
3. Upload

Let’s explore each stage in depth.

Packaging a Helm Chart

This is the simplest step as the Helm utility already includes a subcommand for packaging charts: helm package /path/to/local/chart. In our case, the command to package our Nginx chart is:

helm package ./nginx

The output shows us that the chart’s files are archived to the /home/user/nginx-0.1.0.tgz file.

[email protected]:~$ helm package ./nginx
Successfully packaged chart and saved it to: /home/user/nginx-0.1.0.tgz

The .tgz extension points out that this is a TAR archive, compressed with Gzip. We can open this up with almost any archive manager like WinRAR, 7-Zip and so on. Inside, we’ll see all the files and directories we added to this chart, in our exercises, neatly grouped together in a single nginx-0.1.0.tgz file that is much easier to move around.

So that’s done, why can’t we just skip to uploading this on some server, make it available to the Internet? Why is it recommended to go through the signing step too?

Signing a Helm Chart

Whenever we download stuff from the Internet, it’s not 100% guaranteed that what we get is actually what the developer/maintainer uploaded. That’s because servers can be hacked and files can be replaced with malicious content. One of the ways to make downloads safer is to cryptographically sign files/packages.

For example, in this case, we built our chart, we know our content is good and safe to use. But how can the users that download our chart be sure that they get exactly what we built for them and not some malicious file uploaded by some hacker?

Helm Provenance and Integrity

Once again, Helm provides us with the necessary tools. To make a long story short, this is a two-step process:

1. Helm uses a private key that only we, the chart developers, have access to. With this key, it produces a digital signature and adds it to what is called a provenance file (we’ll take a deeper look at this file soon).

2. Users download our chart and provenance file and look for this signature. With a public key that they know belongs to us, they can verify if the chart is correctly signed. If the signature is valid, they know that this chart was manually signed by us, the chart’s developers.

Generating Private/Public Key Pairs with gpg

First, we’ll need the private key. We can generate this private key/public key pair with the Gnu Privacy Guard (gpg) utility included by default in most Linux distributions. For the purposes of our exercise, we’ll choose the quickest method to generate these, with this command:

gpg --quick-generate-key "John Smith"

“John Smith” is the “real name” we will use for this key. In a real-world scenario, you should also add other details such as the personal email (in fact an entirely different command is recommended which we will mention here later).

If you get a prompt to password-protect this, you can just use an empty password for the sake of simplicity. In a real-world scenario though, it’s recommended you use a strong password to protect your private key. It will be encrypted with this password and even if someone somehow steals your file, they can’t use it without the correct password.

To speed up the generation of the keys, move the mouse cursor around.

When the keys have been generated, we’ll see output similar to this:

gpg: /home/user/.gnupg/trustdb.gpg: trustdb created
gpg: key 80BA57AAFAAD1CA5 marked as ultimately trusted
gpg: directory '/home/user/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/user/.gnupg/openpgp-revocs.d/8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5.rev'
public and secret key created and signed.

pub   rsa3072 2021-06-17 [SC] [expires: 2023-06-17]
      8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5
uid                      John Smith
sub   rsa3072 2021-06-17 [E]

The pub section shows us the unique identifier of our public key 8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5. In a real scenario, this could be uploaded to an OpenPGP keyserver like https://keyserver.ubuntu.com and users would be able to download it by using this unique identifier and then verify our signatures.

As mentioned before, in a production environment, you should use another command to generate your keys. Something like gpg --full-generate-key would be better since it asks you for more details, and you can fine-tune cryptographic settings, expiration dates, set an email address associated with the key, and so on.

Most Linux distributions nowadays use GnuPG v2, which stores keys in a different format than the previous version. To sign charts, Helm currently prefers the older format. We can convert the new secret keyring format to the old format and store it in a file called secring.gpg with the following command:

gpg --export-secret-keys >~/.gnupg/secring.gpg

Signing and Verifying Helm Charts

Now we can finally repackage our chart and also sign it, with this command:

helm package --sign --key 'John Smith' --keyring ~/.gnupg/secring.gpg ./nginx

The --key parameter expects to receive either the Full Name we associated with our key, or the email address. If we ever forget these details, we can list key details with this command:

[email protected]:~$ gpg --list-keys
/home/user/.gnupg/pubring.kbx
-----------------------------
pub   rsa3072 2021-06-17 [SC] [expires: 2023-06-17]
      8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5
uid           [ultimate] John Smith
sub   rsa3072 2021-06-17 [E]

Provenance File

Previously, when we packaged our chart without signing it, the nginx-0.1.0.tgz file was generated. When also signing, an additional file is generated, called the provenance file. In our case, this will be stored in nginx-0.1.0.tgz.prov. If we look inside it, its content looks similar to this:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

apiVersion: v2
appVersion: 1.16.0
description: Basic Nginx website for our company
maintainers:
- - email: [email protected]
  name: John Smith
name: nginx
type: application
version: 0.1.0

...
files:
  nginx-0.1.0.tgz: sha256:b22a325b03c8e88b6a6a8d1a8e79f5d0498813855174a983426466b6de5a5f71
-----BEGIN PGP SIGNATURE-----

wsDcBAEBCgAQBQJgyp1dCRCAuleq+q0cpQAAGHYMAHsR48WO1zLEiRPEOwdnqxSd
hjtjJ99rWxKtSUwYcdTkka9ChO4Rnddj4xw0mE/Y7I+4nS5LFaUunTu6m+nRASN2
oege4uP+ruBEmiJTw+HHVpK8BqgICMY6tWDB9q4HzoZAlWZM6ldJFny86EsFQARP
JU6zlHAVlV9o7mfwIX7d0XYq1/lFQxYLtjvTzXS5LNYCBwjpT1GDmBGdqIidtPY4
7XXPKWycUtXmhyQFs78+TlM17sOjBfzWyTEPiaO2cOE9uqIrP2ww1aetKM1HdICp
Eyi9hQGzXhGEV0Nh0J3zlPVgJohZ6DfE/mK9vfCwWMVB3dkPYJ8Abtp0HEJr+Eb6
0qZ4UZIjYuznK0Gjl6VaSBQe3+sFilAQflEvmpK/0CMy/dXOHoL43sA1pSdyp2FY
xQNWeYJgNCEGxcINvQ1E7rsUfe760FSxUTHPU3gvsg7Do2yH4rwx2tNR2YI1nhgc
zZJdqO1EYOv1B9wogSCzMkHBLkpgtxByTF/46NUf0Q==
=ZOpx
-----END PGP SIGNATURE-----

If this looks a bit mysterious, here’s the short version of how it works:

This provenance file tells us that the file nginx-0.1.0.tgz should have this exact SHA256 hash: b22a325b03c8e88b6a6a8d1a8e79f5d0498813855174a983426466b6de5a5f71. If we run this command:

sha256sum nginx-0.1.0.tgz

We’ll see that indeed this hash is an exact match.

[email protected]:~$ sha256sum nginx-0.1.0.tgz
b22a325b03c8e88b6a6a8d1a8e79f5d0498813855174a983426466b6de5a5f71  nginx-0.1.0.tgz

If we change only 1 byte in the whole file, the hash would change entirely so this tells us that this file is exactly the one that the provenance file tells us we should have. But the provenance file has been downloaded from the Internet too. Maybe a hacker just generated a corrupted nginx-0.1.0.tgz and then edited the provenance, switching the hash of the genuine file with the hash of his corrupted file. However, the -----BEGIN PGP SIGNATURE----- section is where the real magic happens. This is a signature for the entire provenance file, saying something like this: “I, John Smith, the owner of public key 8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5 have signed the entire contents of this file. You can rest assured that the hash you see here is the correct one if you can verify this signature with my public key and it is valid.” Now the hash in the file cannot be changed as that would invalidate the signature. And the attacker cannot produce a signature that could be verified with your public key.

Verifying Signature of Provenance File

When uploading your chart to some online repository, you should copy both the .tgz chart archive and also the .tgz.prov provenance file. We’ll see how to do that in the next lesson.

When users have both of these files available, they can verify the integrity of the chart (verify signatures) with this command:

[email protected]:~$ helm verify ./nginx-0.1.0.tgz
Error: failed to load keyring: open /home/user/.gnupg/pubring.gpg: no such file or directory

because the newer gpg tool stores public keys in the newer format, in the pubring.kbx file instead of the older pubring.gpg file Helm expects.

As a quick workaround, just so we’re able to see helm verify in action, we can do this:

Export our public key to a file called mypublickey.

gpg --export 'John Smith' > mypublickey

Use the helm verify command again but also point Helm to the location of the public key that can be used to verify the signature.

[email protected]:~$ helm verify --keyring ./mypublickey ./nginx-0.1.0.tgz
Signed by: John Smith
Using Key With Fingerprint: 8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5
Chart Hash Verified: sha256:b22a325b03c8e88b6a6a8d1a8e79f5d0498813855174a983426466b6de5a5f71

If we’d edit the provenance file and change just one character in the sha256 hash string, the verify would correctly fail, indicating the file has been tampered with.

[email protected]:~$ helm verify --keyring ./mypublickey ./nginx-0.1.0.tgz
Error: openpgp: invalid signature: hash tag doesn't match

In a real scenario, our chart users would first download our public key with a command like:

gpg --recv-keys --keyserver keyserver.ubuntu.com 8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5

and only afterward they’d be able to use a helm verify command. This assumes we first uploaded our public key to the keyserver.ubuntu.com keyserver.

The verification phase can also be integrated into the regular helm commands we used so far. For example, to verify when we download a chart from the Internet, or when installing a chart, we just add the –verify parameter to our commands:

helm pull --verify NAME_OF_CHART

helm install --verify NAME_OF_RELEASE NAME_OF_CHART

Of course, if the verification of signatures fails, the install phase is abandoned, so we don’t install charts we can’t trust.

Read the official documentation for more.

Checkout the Helm for the Absolute Beginners course here

Checkout the Complete Kubernetes learning path here