After building a chart, we need to distribute it to our users or our organization. This process involves packaging, signing, and uploading the chart.
In this blog, we’ll walk you through how to package, sign and verify a chart. Let us start by creating the chart we'll use to demonstrate the three concepts.
You'll need access to a running Kubernetes cluster to follow along with the example in this post. If you don’t have access to one, you can use a tool such as minikube to set up a Kubernetes cluster. You also need to have Helm installed on your local machine.
The commands in this blog are executed on KodeKloud's Helm Playground. With this playground, you won't need to go through the hassle of installing any additional software— everything you need is already set up and ready to use.
Creating a Helm Chart
We create a Helm chart using the command
helm create. For instance, to create a chart for an app named my-app-chart, we use this command:
helm create nginx
Packaging a Helm Chart
Helm utility includes a subcommand for packaging charts:
helm package /path/to/local/chart. In our case, the command to package our Nginx chart is:
helm package ./nginx
The output shows that the chart’s files are archived to the /home/user/nginx-0.1.0.tgz file.
user@debian:~$ helm package ./nginx Successfully packaged chart and saved it to: /home/user/nginx-0.1.0.tgz
The .tgz extension indicates that this is a TAR archive compressed with Gzip. We can open this up with almost any archive manager like WinRAR, 7-Zip, etc. Inside, we’ll see all the files and directories we added to this chart in our exercises, neatly grouped together in a single nginx-0.1.0.tgz file that is much easier to move around.
So that’s done. Why can’t we upload this on some server and make it available online? Let's see why it is recommended to go through the signing step.
Signing a Helm Chart
Whenever we download code from the Internet, it’s not 100% guaranteed that what we get is actually what the developer/maintainer uploaded. That’s because servers can be hacked, and files can be replaced with malicious content. One of the ways to make downloads safer is to sign files/packages cryptographically.
For example, in this case, we built our chart, and we know our content is good and safe to use. But how can the users that download our chart ensure they get exactly what we built for them and not some malicious file uploaded by hackers? Let's see how.
Helm Provenance and Integrity
With Helm, we can secure and protect the integrity of our charts. This is implemented in two steps.
- Helm uses a private key that only we, the chart developers, can access. This key produces a digital signature and adds it to a provenance file (we’ll take a deeper look at this file soon).
- Users download our chart and provenance file and look for this signature. They can verify if the chart is correctly signed with a public key that they know belongs to us. If the signature is valid, they know that our developers manually signed this chart.
Generating Private/Public Key Pairs with gpg
First, we’ll need the private key. We can generate this private key/public key pair with the Gnu Privacy Guard (gpg) utility included by default in most Linux distributions (If you are using another OS, you can download and install it using this link). For the purposes of our exercise, we’ll choose the quickest method to generate these with this command:
gpg --quick-generate-key "John Smith"
In the command above, “John Smith” is our “real name” for this key. In a real-world scenario, you should also add other details such as the personal email (in fact, an entirely different command is recommended, which we will mention here later).
If you get a prompt to password-protect this, you can use an empty password for simplicity. In a real-world scenario, though, you should use a strong password to protect your private key. It will be encrypted with this password; even if someone somehow steals your file, they can’t use it without the correct password.
When the keys have been generated, we’ll see an output similar to this:
gpg: /home/user/.gnupg/trustdb.gpg: trustdb created gpg: key 80BA57AAFAAD1CA5 marked as ultimately trusted gpg: directory '/home/user/.gnupg/openpgp-revocs.d' created gpg: revocation certificate stored as '/home/user/.gnupg/openpgp-revocs.d/8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5.rev' public and secret key created and signed. pub rsa3072 2021-06-17 [SC] [expires: 2023-06-17] 8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5 uid John Smith sub rsa3072 2021-06-17 [E]
The pub section shows us the unique identifier of our public key 8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5. In a real scenario, this could be uploaded to an OpenPGP keyserver like https://keyserver.ubuntu.com. Users could download it using this unique identifier and verify our signatures.
As mentioned before, you should use another command to generate your keys in a production environment. Something like
gpg --full-generate-key would be better since it asks you for more details, and you can fine-tune cryptographic settings, expiration dates, set an email address associated with the key, and so on.
Most Linux distributions nowadays use GnuPG v2, which stores keys in a different format than the previous version. To sign charts, Helm currently uses the older format. We can convert the new secret keyring format to the old format and store it in a file called secring.gpg with the following command:
gpg --export-secret-keys >~/.gnupg/secring.gpg
Signing and Verifying Helm Charts
Now we can finally repackage our chart and also sign it with this command:
helm package --sign ./nginx --key 'John Smith' --keyring ~/.gnupg/secring.gpg
--key parameter expects to receive either the full name associated with our key or the email address. If we ever forget these details, we can list key details with this command:
user@debian:~$ gpg --list-keys /home/user/.gnupg/pubring.kbx ----------------------------- pub rsa3072 2021-06-17 [SC] [expires: 2023-06-17] 8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5 uid [ultimate] John Smith sub rsa3072 2021-06-17 [E]
Previously, when we packaged our chart without signing it, the nginx-0.1.0.tgz file was generated. When also signing, an additional file is generated, called the provenance file. In our case, this will be stored in nginx-0.1.0.tgz.prov. If we look inside it, its content looks similar to this:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 apiVersion: v2 appVersion: 1.16.0 description: Basic Nginx website for our company maintainers: - - email: [email protected] name: John Smith name: nginx type: application version: 0.1.0 ... files: nginx-0.1.0.tgz: sha256:b22a325b03c8e88b6a6a8d1a8e79f5d0498813855174a983426466b6de5a5f71 -----BEGIN PGP SIGNATURE----- wsDcBAEBCgAQBQJgyp1dCRCAuleq+q0cpQAAGHYMAHsR48WO1zLEiRPEOwdnqxSd hjtjJ99rWxKtSUwYcdTkka9ChO4Rnddj4xw0mE/Y7I+4nS5LFaUunTu6m+nRASN2 oege4uP+ruBEmiJTw+HHVpK8BqgICMY6tWDB9q4HzoZAlWZM6ldJFny86EsFQARP JU6zlHAVlV9o7mfwIX7d0XYq1/lFQxYLtjvTzXS5LNYCBwjpT1GDmBGdqIidtPY4 7XXPKWycUtXmhyQFs78+TlM17sOjBfzWyTEPiaO2cOE9uqIrP2ww1aetKM1HdICp Eyi9hQGzXhGEV0Nh0J3zlPVgJohZ6DfE/mK9vfCwWMVB3dkPYJ8Abtp0HEJr+Eb6 0qZ4UZIjYuznK0Gjl6VaSBQe3+sFilAQflEvmpK/0CMy/dXOHoL43sA1pSdyp2FY xQNWeYJgNCEGxcINvQ1E7rsUfe760FSxUTHPU3gvsg7Do2yH4rwx2tNR2YI1nhgc zZJdqO1EYOv1B9wogSCzMkHBLkpgtxByTF/46NUf0Q== =ZOpx -----END PGP SIGNATURE-----
If this looks a bit mysterious, here’s the short version of how it works:
This provenance file tells us that the file nginx-0.1.0.tgz should have this exact SHA256 hash:
b22a325b03c8e88b6a6a8d1a8e79f5d0498813855174a983426466b6de5a5f71. If we run this command:
We’ll see that, indeed, this hash is an exact match.
user@debian:~$ sha256sum nginx-0.1.0.tgz b22a325b03c8e88b6a6a8d1a8e79f5d0498813855174a983426466b6de5a5f71 nginx-0.1.0.tgz
The hash will change entirely if we change only 1 byte in the whole file. The hash tells us that this file is exactly what the provenance file tells us we should have. But the provenance file has been downloaded from the Internet too.
Maybe a hacker just generated a corrupted nginx-0.1.0.tgz and then edited the provenance, switching the hash of the genuine file with the hash of his corrupted file. However, this
-----BEGIN PGP SIGNATURE----- section is where the real magic happens. This is a signature for the entire provenance file, saying something like this:
“I, John Smith, the owner of the public key,
8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5have signed the entire contents of this file. You can rest assured that the hash you see here is correct. You can rest assured that the hash you see here is the correct one if you can verify this signature with my public key, and it is valid.”
Now the hash in the file cannot be changed as that would invalidate the signature. And the attacker cannot produce a signature that could be verified with your public key.
Verifying Signature of Provenance File
When uploading your chart to some online repository, you should copy both the .tgz chart archive and the .tgz.prov provenance file. We’ll see how to do that in the next lesson.
When users have both of these files available, they can verify the integrity of the chart (verify signatures) with this command:
user@debian:~$ helm verify ./nginx-0.1.0.tgz Error: failed to load keyring: open /home/user/.gnupg/pubring.gpg: no such file or directory
We get the error because the newer gpg tool stores public keys in the newer format, in the pubring.kbx file instead of the older pubring.gpg file Helm expects. As a quick workaround, just so we’re able to see helm verify in action, we can do the following.
Export our public key to a file called mypublickey.
gpg --export 'John Smith' > mypublickey
Use the helm verify command again but also point Helm to the location of the public key that can be used to verify the signature.
user@debian:~$ helm verify --keyring ./mypublickey ./nginx-0.1.0.tgz Signed by: John Smith Using Key With Fingerprint: 8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5 Chart Hash Verified: sha256:b22a325b03c8e88b6a6a8d1a8e79f5d0498813855174a983426466b6de5a5f71
If we’d edit the provenance file and change just one character in the sha256 hash string, the verification would correctly fail, indicating the file has been tampered with.
user@debian:~$ helm verify --keyring ./mypublickey ./nginx-0.1.0.tgz Error: openpgp: invalid signature: hash tag doesn't match
In a real-world scenario, our chart users would first download our public key with a command like:
gpg --recv-keys --keyserver keyserver.ubuntu.com 8D40FE0CACC3FED4AD1C217180BA57AAFAAD1CA5
And only afterward they’d be able to use a
helm verify command. This assumes we first uploaded our public key to the
The verification phase can also be integrated into the regular helm commands we used so far. For example, to verify when we download a chart from the Internet or when installing a chart, we add the –verify parameter to our commands:
helm pull --verify NAME_OF_CHART helm install --verify NAME_OF_RELEASE NAME_OF_CHART
Of course, if the verification of signatures fails, the installation phase is abandoned, so we don’t install charts we can’t trust.
ENROLL in our Helm For Beginner'sCourse to learn other concepts, such as creating Helm Charts, adding dependencies, and testing them.
Helm uses package signing to protect the integrity of charts. This means that each chart is signed by its creator using a private key, and the signature can be verified using the creator's public key. This ensures that the chart has not been tampered with or modified in any way.
Why Learn Kubernetes
Kubernetes has witnessed the fastest growth in job searches, with a staggering increase of over 173% in just one year! According to recent surveys conducted by Indeed, the demand for Kubernetes skills is skyrocketing.
To boost your professional credibility, you can earn certifications like CKAD, CKA, and CKS, which will attest to your expertise in Kubernetes. The courses below will help you prepare for these three in-demand certifications.
- Certified Kubernetes Administrator (CKA)
- Certified Kubernetes Application Developer (CKAD)
- Certified Kubernetes Security Specialist (CKS)
Don’t know where to start your certification journey, check out the blog CKAD vs. CKA vs. CKS.
To learn more about the three certifications, check out our CKA, CKAD, CKS - Frequently Asked Questions blog.
More on Helm: