DevOps

Package, Sign, and Verify Helm Charts (Commands & Examples)

After building a helm chart, we need to distribute it to our users or our organization. This process involves packaging, signing, and uploading the chart.

In this blog, we’ll walk you through how to package, sign, and verify a chart. Let us start by creating the chart we'll use to demonstrate the three concepts.

Prerequisites

You'll need access to a running Kubernetes cluster to follow along with the example in this post. If you don’t have access to one, you can use a tool such as minikube to set up a Kubernetes cluster. You also need to have Helm installed on your local machine.

The commands in this blog are executed on KodeKloud's Helm Playground. With this playground, you won't need to go through the hassle of installing any additional software— everything you need is already set up and ready to use.

Creating a Helm Chart

We create a Helm chart using the command helm create. For instance, to create a chart for an app named my-app-chart, we use this command:

helm create nginx

Packaging a Helm Chart

Helm utility includes a subcommand for packaging charts: helm package /path/to/local/chart. In our case, the command to package our Nginx chart is:

helm package ./nginx

The output shows that the chart’s files are archived to the /home/user/nginx-0.1.0.tgz file.

user@debian:~$ helm package ./nginx
Successfully packaged chart and saved it to: /home/user/nginx-0.1.0.tgz

The .tgz extension indicates that this is a TAR archive compressed with Gzip. We can open this up with almost any archive manager like WinRAR, 7-Zip, etc. Inside, we’ll see all the files and directories we added to this chart in our exercises, neatly grouped together in a single nginx-0.1.0.tgz file that is much easier to move around.

So that’s done. Why can’t we upload this on some server and make it available online? Let's see why it is recommended to go through the signing step.

Signing a Helm Chart

Whenever we download code from the Internet, it’s not 100% guaranteed that what we get is actually what the developer/maintainer uploaded. That’s because servers can be hacked, and files can be replaced with malicious content. One of the ways to make downloads safer is to sign files/packages cryptographically.

For example, in this case, we built our chart, and we know our content is good and safe to use. But how can the users that download our chart ensure they get exactly what we built for them and not some malicious file uploaded by hackers? Let's see how.

Helm Provenance and Integrity

With Helm, we can secure and protect the integrity of our charts. This is implemented in two steps.

Helm uses a private key that only we, the chart developers, can access. This key produces a digital signature and adds it to a provenance file (we’ll look deeper at this file soon).
Users download our chart and provenance file and look for this signature. They can verify if the chart is correctly signed with a public key they know belongs to us. If the signature is valid, they know that our developers manually signed this chart.

Generating Private/Public Key Pairs with gpg

First, we’ll need the private key. We can generate this private key/public key pair with the Gnu Privacy Guard (gpg) utility included by default in most Linux distributions (If you are using another OS, you can download and install it using this link). For the purposes of our exercise, we’ll choose the quickest method to generate these with this command:

gpg --quick-generate-key "John Smith"

In the command above, “John Smith” is our “real name” for this key. In a real-world scenario, you should also add other details such as the personal email (in fact, an entirely different command is recommended, which we will mention here later).

If you get a prompt to password-protect this, you can use an empty password for simplicity. In a real-world scenario, though, you should use a strong password to protect your private key. It will be encrypted with this password; even if someone somehow steals your file, they can’t use it without the correct password.

When the keys have been generated, we’ll see an output similar to this:

gpg: /home/user/.gnupg/trustdb.gpg: trustdb created
gpg: key 80BA57AAFAAD1CA5 marked as ultimately trusted
gpg: directory '/home/user/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/user/.gnupg/openpgp-revocs.d/8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5.rev'
public and secret key created and signed.

pub   rsa3072 2021-06-17 [SC] [expires: 2023-06-17]
      8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5
uid                      John Smith
sub   rsa3072 2021-06-17 [E]

The pub section shows us the unique identifier of our public key 8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5. In a real scenario, this could be uploaded to an OpenPGP keyserver like https://keyserver.ubuntu.com. Users could download it using this unique identifier and verify our signatures.

As mentioned before, you should use another command to generate your keys in a production environment. Something like gpg --full-generate-key would be better since it asks you for more details, and you can fine-tune cryptographic settings and expiration dates, set an email address associated with the key, and so on.

Most Linux distributions nowadays use GnuPG v2, which stores keys in a different format than the previous version. Helm currently uses the older format to sign charts. We can convert the new secret keyring format to the old format and store it in a file called secring.gpg with the following command:

gpg --export-secret-keys >~/.gnupg/secring.gpg

Signing and Verifying Helm Charts

Now we can finally repackage our chart and also sign it with this command:

helm package --sign ./nginx --key 'John Smith' --keyring ~/.gnupg/secring.gpg

The --key parameter expects to receive either the full name associated with our key or the email address. If we ever forget these details, we can list key details with this command:

user@debian:~$ gpg --list-keys
/home/user/.gnupg/pubring.kbx
-----------------------------
pub   rsa3072 2021-06-17 [SC] [expires: 2023-06-17]
      8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5
uid           [ultimate] John Smith
sub   rsa3072 2021-06-17 [E]

Provenance File

Previously, when we packaged our chart without signing it, the nginx-0.1.0.tgz file was generated. When also signing, an additional file is generated, called the provenance file. In our case, this will be stored in nginx-0.1.0.tgz.prov. If we look inside it, its content looks similar to this:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

apiVersion: v2
appVersion: 1.16.0
description: Basic Nginx website for our company
maintainers:
- - email: [email protected]
  name: John Smith
name: nginx
type: application
version: 0.1.0

...
files:
  nginx-0.1.0.tgz: sha256:b22a325b03c8e88b6a6a8d1a8e79f5d0498813855174a983426466b6de5a5f71
-----BEGIN PGP SIGNATURE-----

wsDcBAEBCgAQBQJgyp1dCRCAuleq+q0cpQAAGHYMAHsR48WO1zLEiRPEOwdnqxSd
hjtjJ99rWxKtSUwYcdTkka9ChO4Rnddj4xw0mE/Y7I+4nS5LFaUunTu6m+nRASN2
oege4uP+ruBEmiJTw+HHVpK8BqgICMY6tWDB9q4HzoZAlWZM6ldJFny86EsFQARP
JU6zlHAVlV9o7mfwIX7d0XYq1/lFQxYLtjvTzXS5LNYCBwjpT1GDmBGdqIidtPY4
7XXPKWycUtXmhyQFs78+TlM17sOjBfzWyTEPiaO2cOE9uqIrP2ww1aetKM1HdICp
Eyi9hQGzXhGEV0Nh0J3zlPVgJohZ6DfE/mK9vfCwWMVB3dkPYJ8Abtp0HEJr+Eb6
0qZ4UZIjYuznK0Gjl6VaSBQe3+sFilAQflEvmpK/0CMy/dXOHoL43sA1pSdyp2FY
xQNWeYJgNCEGxcINvQ1E7rsUfe760FSxUTHPU3gvsg7Do2yH4rwx2tNR2YI1nhgc
zZJdqO1EYOv1B9wogSCzMkHBLkpgtxByTF/46NUf0Q==
=ZOpx
-----END PGP SIGNATURE-----

If this looks a bit mysterious, here’s the short version of how it works:

This provenance file tells us that the file nginx-0.1.0.tgz should have this exact SHA256 hash: b22a325b03c8e88b6a6a8d1a8e79f5d0498813855174a983426466b6de5a5f71. If we run this command:

sha256sum nginx-0.1.0.tgz

We’ll see that, indeed, this hash is an exact match.

user@debian:~$ sha256sum nginx-0.1.0.tgz
b22a325b03c8e88b6a6a8d1a8e79f5d0498813855174a983426466b6de5a5f71  nginx-0.1.0.tgz

The hash will change entirely if we change only 1 byte in the whole file. The hash tells us that this file is exactly what the provenance file tells us we should have. But the provenance file has been downloaded from the internet, too.

Maybe a hacker just generated a corrupted nginx-0.1.0.tgz and then edited the provenance, switching the hash of the genuine file with the hash of his corrupted file. However, this -----BEGIN PGP SIGNATURE----- section is where the real magic happens. This is a signature for the entire provenance file, saying something like this:

“I, John Smith, the owner of the public key, 8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5have signed the entire contents of this file. You can rest assured that the hash you see here is correct. You can rest assured that the hash you see here is the correct one if you can verify this signature with my public key and it is valid.”

Now, the hash in the file cannot be changed as that would invalidate the signature. The attacker cannot produce a signature that can be verified with your public key.

Verifying Signature of Provenance File

When uploading your chart to some online repository, you should copy both the .tgz chart archive and the .tgz.prov provenance file. We’ll see how to do that in the next lesson.

When users have both of these files available, they can verify the integrity of the chart (verify signatures) with this command:

user@debian:~$ helm verify ./nginx-0.1.0.tgz
Error: failed to load keyring: open /home/user/.gnupg/pubring.gpg: no such file or directory

We get the error because the newer gpg tool stores public keys in the newer format, in the pubring.kbx file instead of the older pubring.gpg file Helm expects. As a quick workaround, just so we can see helm verify in action, we can do the following.

Export our public key to a file called mypublickey.

gpg --export 'John Smith' > mypublickey

Use the helm verify command again, but also point Helm to the location of the public key that can be used to verify the signature.

user@debian:~$ helm verify --keyring ./mypublickey ./nginx-0.1.0.tgz
Signed by: John Smith
Using Key With Fingerprint: 8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5
Chart Hash Verified: sha256:b22a325b03c8e88b6a6a8d1a8e79f5d0498813855174a983426466b6de5a5f71

If we edit the provenance file and change just one character in the sha256 hash string, the verification would correctly fail, indicating the file has been tampered with.

user@debian:~$ helm verify --keyring ./mypublickey ./nginx-0.1.0.tgz
Error: openpgp: invalid signature: hash tag doesn't match

In a real-world scenario, our chart users would first download our public key with a command like:

gpg --recv-keys --keyserver keyserver.ubuntu.com 8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5

And only afterward they’d be able to use a helm verify command. This assumes we first uploaded our public key to the keyserver.ubuntu.com keyserver.

The verification phase can also be integrated into the regular helm commands we have used. For example, to verify when we download a chart from the Internet or when installing a chart, we add the –verify parameter to our commands:

helm pull --verify NAME_OF_CHART

helm install --verify NAME_OF_RELEASE NAME_OF_CHART

Of course, if the verification of signatures fails, the installation phase is abandoned, so we don’t install charts we can’t trust.

ENROLL in our Helm For Beginner's Course to learn other concepts, such as creating Helm Charts, adding dependencies, and testing them.

Conclusion

Helm uses package signing to protect the integrity of charts. This means that each chart is signed by its creator using a private key, and the signature can be verified using the creator's public key. This ensures that the chart has not been tampered with or modified.

Mumshad Mannambeth

Jan 31, 2022 • 8 min read