What are Kubernetes Secrets?

A Kubernetes Secret is an object that stores sensitive data, like a password, a certificate, or a token. Generally, an application's sensitive information is packaged with the application binary or passed as environment variables inside a container.

What are Kubernetes Secrets?
Kubernetes Secrets explained

Introduction

In Container Orchestration, Kubernetes is a market leader with great community support. It is an evolved software tool with multiple components working together seamlessly. Kubernetes is well-designed software and illustrates the separation of configuration from the application. Kubernetes provides objects like Secrets to store sensitive data and ConfigMaps to store other configurations.

A Kubernetes Secret is an object that stores sensitive data, like a password, a certificate, or a token. Generally, an application's sensitive information is packaged with the application binary or passed as environment variables inside a container. However, Kubernetes allows users to decouple the storage of such sensitive information in the form of Secrets.

What Are Kubernetes Secrets?

Kubernetes Secrets are objects meant to store sensitive data consumed by the application Pods. The data stored in secrets is base64 encoded, meaning that encoded data is not in an encrypted form, and anyone can easily decode it back to the original plaintext.

Since a user can create Kubernetes Secrets separately from Pods, updating and deleting the Pods does not impact the Secrets. Secrets can be mounted into the Pods or passed as environment variables. The pods can import an external secret and manipulate/change it internally. But the original, external secret will remain unaffected. Secrets are functionally similar to ConfigMaps. The only difference is that the data in the Kubernetes Secrets is base64 encoded.

What Are the Types of Kubernetes Secrets?

There are many types of Kubernetes Secrets for different use cases; some are for storing sensitive data used by applications, Docker registry credentials, some for TLS certificates and service account tokens, and so on.

Application Data or Generic

Application data or Generic secrets are simply the sensitive data used in the applications, like database connection strings, username/passwords, access tokens, etc.

Docker Config

Docker config secrets are used to authenticate against a Docker registry. Docker registries are used to store Docker images used for creating the containers that will run in Kubernetes Pods.

TLS Certificates

TLS Certificates secrets can be created from a public/private key pair. The public key certificate must be .PEM encoded and match the given private key.

Service Account Token

These types of secrets provide a way to programmatically access the Kubernetes APIs and perform operations based on the scope of access provided to a Services Account. Once we create a Service Account, a corresponding secret is generated containing a token. The token can be used to access the Kubernetes APIs.

How and Why to Create Kubernetes Secrets?

Kubernetes provides a way to separate the creation and management of sensitive data used in applications, usually in the form of database credentials, access tokens, and TLS certificates.

This approach saves the user from managing sensitive data in the application; instead, they can offload this responsibility to the Secrets object. Secrets are a source of truth for storing and updating sensitive information. Kubernetes provides multiple ways to pass the secrets to the applications via mounting the secret as a volume or passing the credentials as environment variables object.

Now that we know why we create secrets in Kubernetes let's dive into the how.

We can create secrets using two methods:

  1. Creating a secret using the command line utility kubectl
  2. Creating a secret from a yaml secret definition file.

Creating a Secret Using the Command Line Utility kubectl

Kubernetes Secret object provides us with the capability to make different kinds of secrets, which we have mentioned in the previous sections. Here we will elaborate on creating different types of secrets using the kubectl CLI utility.

Application Data or Generic Secrets

The kubectl command line utility provides a way to create secrets from a literal value, a file or directory containing a credential and an environment file.

kubectl create secret generic user-creds --from-literal username=kodekloud --from-literal password=a//Strong//Password

kubectl create secret generic user-creds --from-literal username=kodekloud --from-file=password=/path/to/file

kubectl create secret generic user-creds --from-env=/path/to/file.env

Docker Config

Docker config secrets are used to authenticate against a Docker registry. Docker registries are used to store Docker images used for creating containers in Kubernetes Pods. There are two ways to make a Dockercfg secret. We can create a secret using a .dockercfg file or by passing in the credentials in the following manner:

kubectl create secret docker-registry kodekloud-reg
--from-file=.dockerconfig=/path/to/somefile.dockerconfig

kubectl create secret docker-registry kodekloud-reg
--docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL

TLS Certificates

TLS Certificates secrets can be created from a public/private key pair. The public key certificate must be .PEM encoded and match the given private key. We can create a TLS secret using the following command:

kubectl create secret tls-secret --cert=/path/to/tls.cert
--key=/path/to/tls.key

Service Account Token

Once a Service Account is created, a corresponding secret containing a token is automatically generated. For example, we can create a Service Account using the following command:

kubectl create serviceaccount kode-kloud-sa

Creating a Secret from a yaml Secret Definition File

Application Data or Generic Secrets

We can create a generic secret from a yaml definition. The yaml definition is processed by the Kubernetes APIs and in turn creates an object similar to the one which can be created from the command line utility. The below definition can be used to create a secret.

apiVersion: v1 
kind: Secret 
metadata: 
  name: nginx 
data: 
  USERNAME: a29kZWtsb3Vk 
  PASSWORD: YS8vU3Ryb25nLy9QYXNzd29yZA== 

Although there is one consideration while defining a secret using a yaml file. We pass the base64 encoded values. We can perform a base64 encode/decode using the following commands on a Linux machine:

# For encoding
echo -n "kodekloud" | base64

# For decoding
echo -n "a29kZWtsb3Vk" | base64 -d

Things to Consider Before Using Kubernetes Secrets

Kubernetes provides a secure means to store sensitive data in the form Secrets. However, there are some drawbacks to it, and before deciding to use secrets, you should consider the following points:

  1. The sensitive data stored in Kubernetes Secrets is encoded in base-64. This is not an encryption technique but simply a text-encoding technique used to store data. For example, humans use base-10 to symbolize numbers like 255. Base-10 has ten digits, from 0 to 9. There is also base-16 which has 16 digits, from 0 to 9, then also A,B,C,D,E,F. The number 255 converted to base-16 is "FF". So base-64 does a similar thing, but with 64 total digits available. Since it's easy to convert from one base to another, it means base-64 data is unprotected.
  2. If a malicious user gets access to a cluster they can read all the secrets and decode the base64 encoded contents.
  3. Kubernetes Secrets rely on a master key for encryption at the ETCD level. ETCD is the key value store (a sort of database) used by the Kubernetes API server to store configuration.

How to Secure Kubernetes Secrets?

Nowadays a common way to add an additional layer to secure secret management on Kubernetes is to introduce a third-party secret management solution, such as Azure Key Vault, Google Secret Manager, AWS Secrets Manager, and Hashicorp Vault. This approach might resolve some issues inherent in Kubernetes, but any third-party solution is not a silver bullet. With a third-party solution, it is still virtually impossible to ensure that:

  1. We are restricting access to secrets. This is an RBAC problem and third-party tools do not currently offer a solution for this problem.
  2. Secrets are not stored on disk or in environment variables.
  3. Applications don't continue to remain the weak point as a compromised application can easily leak secrets.
  4. TLS encrypts data in transit.
  5. Secrets are automatically rotated.

Configure Least-Privilege Access to Secrets

When planning your access control mechanism, such as Kubernetes Role-based Access Control (RBAC), consider the following guidelines for access to Secret objects. You should also follow the other guidelines in RBAC good practices.

  • Services: Restrict watch or list access to only the most privileged, system-level components. Only grant get access for Secrets if the component's normal behavior requires it.
  • People: Restrict get, watch, or list access to Secrets. Only allow cluster administrators to access etcd. This includes read-only access.

Conclusion

Kubernetes Secrets are an effective solution for storing sensitive data. Secret objects do have their pros and cons, but with the proper application of Kubernetes Best Practices and granular RBAC controls, we can have a secure solution to the problem of storing sensitive data.

The best thing about Kubernetes is that it's evolving at an enormous speed with massive community support on Github and Internet forums. As Kubernetes continues to grow, security is at its core and in the coming days we can expect the Kubernetes ecosystem to get more secure. For now secrets are the best ways to store and manage sensitive data.