Container security

10 Kubernetes Security Best Practices to Secure K8 Clusters

Kubernetes (K8s) is an open-source container orchestration tool. It is relatively new and has a complex architecture that needs declarative configurations and management. The State of Kubernetes and Container Security report,” published in 2023 by Stackrox, states that in the period between April 2022 and April 2023, 67% of companies delayed or slowed down deployment due to a security issue.

With high Kubernetes adoption, demand for DevOps/DevSecOps engineers and SREs specializing in Kubernetes security and Kubernetes administrators has grown significantly. The need for security is also reflected in CNCF's introduction of the Certified Kubernetes Security Specialist (CKS) certification.

Enroll in our CKS course to certify your Kubernetes security expertise:

To protect your application from malicious attacks by hackers or accidental data breaches, you need to understand the 'surface' that needs to be protected. This means understanding Kubernetes architectural vulnerabilities and containers lifecycle. In short, you need to look at security in the context of cloud-native security that is modeled across 4 layers, known as 4C’s – Cloud, Clusters, Containers, and Code.

This model is resilient because your application code resides at the innermost layer and is safeguarded by the outer layers. However, it does not guarantee overall security if you only secure your code while the outer layers remain vulnerable. Each layer has its own security requirement that needs to be implemented separately.

Hackers can target any or each of these components. So you will need to harden the Kubernetes cluster and the system from every aspect. There are many dimensions covering all the security aspects of Kubernetes, and it is a vast subject in its entirety that cannot be covered in one single blog. So, we recommend you refer to the official Kubernetes documentation to explore these.

Try the Kubernetes Services Lab for free.

10 Kubernetes Security Best Practices

Below is a checklist covering some of the top Kubernetes security best practices that the Kubernetes administrators need to follow to keep their clusters safe.

1. Keep your Kubernetes version up to date

A new Kubernetes version (e.g., 1.x.y) is released every three months and patch updates that come with bug fixes are also released regularly supporting the latest three versions. Ensure your version is latest or at least within the last three versions and it has all CVE (Common Vulnerabilities and Exposures) fixes.

Managed Kubernetes service providers may provide automatic upgrades. Updating a Kubernetes cluster in a production environment without downtime is quite challenging. Check this blog post from Google.

2. Use authorized container images, scan, and update regularly

Use private container registries and tagged container images, keeping tagged images immutable. Limit the number of users who can create pods and try not to use unknown or unmaintained libraries. Continuously scan Docker container images that run your application for CVE using scanning tools (e.g., Sonatype Nexus, CoreOS Clair, and Dockscan) for all categories like SAST, DAST, IAST, or SCA. Update them using Kubernetes rolling updates when vulnerabilities are found.

3. Network access using Transport Layer Security (TLS)

Network access to the Kubernetes components API server (control plane), etcd (persistent storage database), kubelet agent, kubedns must use an encrypted connection. Ensure that Kubernetes clusters are enabled to use end-to-end TLS. Some components and installation methods may expose ports over HTTP, and admins need to check the settings and fix such vulnerabilities.

4. Kubernetes RBAC authorization and authentication

Kubernetes supports client certificates(x.509), bearer tokens, authenticating proxy, or HTTP basic auth to authenticate API requests. Use an admission controller as an extra layer of validation.

Kubernetes has several forms of access and identity management like LDAP, Active Directory, and SAML, etc. Role-based access control (RBAC), is enabled by default as an authorization mechanism. Make sure that users are assigned proper roles. Assign namespace-specific permissions instead of cluster-wide privileges to minimize exposure. Read more here.

5. Enable network policies.

In Kubernetes, every pod is given a local IP address that can communicate with all other pods on its namespace. Use network policies to define label selectors and control which pods can interact with which pod, over which port, direction, and control ingress/egress traffic.

6. Different namespaces based on the security profile

Namespaces create logical groups/boundaries among a single physical cluster to control access to the service’s resources. Do not include vastly different services in the same cluster. Use different namespaces for different environments like (dev/test/stage/prod).

Using different namespaces mitigates the potential threats affecting all systems. Also, large clusters with unrelated services are harder to monitor and manage.

7. Implement namespace resource quotas

By default, all resources are created with unbounded CPU and memory limits. Do not let containers run without an upper bound on resources to prevent “noisy neighbors” and potential DoS (denial of service) situations. You can assign resource quota policies at the namespace level to limit the overconsumption of the CPU and memory resources a pod is allowed to consume.

8. Restrict access to insecure API server ports

The API server is the main means of communication with Kubernetes components. Besides enabling authentication and RBAC, all unsecured API server ports are locked down. Use a bastion host, configure a VPN, or use an internal network to access the nodes and other infrastructure resources.

9. Secure Dashboard

Limit exposure of Kubernetes Dashboard. Disable public access via the internet. Ensure the Dashboard Service Account is not open and accessible to users. Configure the login page and enable RBAC.

10. Enable Kubernetes Security Logging

Audit logging provides container activity logs to determine the health and security of your Kubernetes. A Fluentd agent running on each node can log container errors and send them to monitoring tools like Stackdriver or Elasticsearch, which can be viewed using Kibana. You can also configure alerts and send notifications using Slack/Email, etc. Kubernetes allows you to create audit policies that let you decide what things you’d like to log and at which level.

To understand in-depth and tackle any Kubernetes-related security issues, take this Certified Kubernetes Security Specialist (CKS) course and become a Kubernetes security expert.

Conclusion

Kubernetes deployments can be highly distributed environments with large clusters. Kubernetes provides the mechanisms to keep your clusters secured but there is a learning curve involved to all configuration options. The best practices mentioned here will help you to get started and let you harden your deployment and reduce the attack surface.

Building a deeper contextual knowledge by reading the official documentation will help you further reduce potential risks, vulnerabilities, and achieve your compliance goals.

People are also reading:

Mumshad Mannambeth

May 1, 2023 • 5 min read