Kubernetes Security Best Practices: A Complete Guide (2023)
Why Do We Need Kubernetes Security Best Practices?
Kubernetes (K8s), the open-source orchestration framework for managing containers (scheduling, load balancing, and distribution) to run especially cloud-native microservice-based application workloads. It is relatively new and has a complex architecture that needs declarative configurations and management. A report on the survey done and published last year (2020) by Stackrox called the “State of Kubernetes and Container Security report” states that in the last 12 months 90 % of respondents had faced major security issues in their Kubernetes environment and in another report found 44% of respondents had delays in application rollouts to production due to security concerns.
Image Source: Stackrox report on Kubernetes security
With high Kubernetes adoption, demand for DevOps/DevSecOps engineers, and SREs specializing in Kubernetes security and Kubernetes administrators have increased manifolds. In fact, only recently, the Cloud Native Computing Foundation (CNCF) has introduced a Certified Kubernetes Security Specialist (CKS) program intended to bridge the knowledge gap.
To protect your application from malicious attacks by hackers or accidental data breaches (sensitive client data like credit card information in a production environment) you will need to understand the key Kubernetes architectural vulnerabilities, platform dependencies, and the container lifecycle (build/ deploy/ run) in Kubernetes deployments. We need to look at Kubernetes security in the context of cloud-native security that is modeled across the layers of 4C’s – Cloud, Clusters, Containers, and Code.
This model is resilient in a way that your application code resides at the innermost layer and is safeguarded by the outer layers. But it does guarantee overall security if you only secure your code while outer layers remain compromised without proper housekeeping. Each layer has its own security requirement that needs to be implemented separately. Now let’s also look at the simple Kubernetes architecture and its components.
Hackers can target any or each of these components. So you will need to harden the Kubernetes cluster and the system from every aspect. There are many dimensions covering all the security aspects in Kubernetes and it is a vast subject in its entirety that cannot be covered in one single blog. So we recommend you refer to the official Kubernetes documentation to explore all of these.
In this blog, we will have a checklist covering some of the top Kubernetes security best practices that the Kubernetes administrators need to follow to keep their clusters safe. Let's discover 10 Kubernetes security best practices!
1. Keep your Kubernetes version up to date.
A new Kubernetes version (e.g. 1.x.y) is released every three months and patch updates that come with bug fixes are also released regularly supporting the latest three versions. Ensure your version is latest or at least within the last three versions and it has all CVE (Common Vulnerabilities and Exposures) fixes. Managed service providers (Cloud operators) may provide automatic upgrades. Updating a Kubernetes cluster in a production environment without downtime is quite challenging. Check this blog post from Google.
2. Use authorized container images, scan, and update regularly.
Use private container registries and tagged container images, keeping tagged images immutable. Limit the number of users who can create pods and try not to use unknown or unmaintained libraries. Continuously scan Docker container images that run your application for CVE using scanning tools (e.g. Sonatype Nexus, CoreOS Clair, and Dockscan) for all categories like SAST, DAST, IAST, or SCA. Update them using Kubernetes rolling updates when vulnerabilities are found.
3. Network access using Transport Layer Security (TLS)
Network access to the Kubernetes components API server (control plane), etcd (persistent storage database), kubelet agent, kubedns must use an encrypted connection. Ensure that Kubernetes clusters have end-to-end TLS enabled. Some components and installation methods may expose local over ports over HTTP and admins need to check the settings for unsecured traffic.
4. Kubernetes RBAC authorization and authentication
Kubernetes supports client certificates(x.509), bearer tokens, authenticating proxy, or HTTP basic auth to authenticate API requests through authentication plugins. Use an admission controller as an extra layer of validation.
Kubernetes has several forms of access and identity management like LDAP, Active Directory, and SAML, etc. Role-based access control (RBAC), is enabled by default as an authorization mechanism. Make sure that users are assigned proper roles. Assign namespace-specific permissions instead of cluster-wide privileges to minimize exposure. Read more here.
5. Enable network policies.
In Kubernetes, every pod is given a local IP address that can communicate with all other pods on its namespace. Use network policies to define label selectors and control which pods can interact with which pod, over which port, direction, and control ingress/egress traffic.
6. Different namespaces based on the security profile
Namespaces create logical groups/boundaries among a single physical cluster to control access to the service’s resources. Do not include vastly different services in the same cluster. Use different namespaces for different environments like (dev/test/stage/prod).
Using different namespaces mitigates the potential threats affecting all systems. Also, large clusters with unrelated services become harder to upgrade, but clusters should be
7. Implement namespace resource quotas
By default, all resources are created with unbounded CPU and memory limits. To prevent “noisy neighbors” and potential DoS (denial of service) situations, do not let containers run without an upper bound on resources. You can assign resource quota policies at the namespace level, in order to limit overconsumption of the CPU and memory resources a pod is allowed to consume.
8. Restrict access to insecure API server ports
The API Server is the main means to communicate with Kubernetes components. Besides enabling authentication and RBAC, lock down all unsecured API server ports. Use a bastion host, configure VPN, or use an internal network to access the nodes and other infrastructure resources.
9. Secure Dashboard
Limit exposure of Kubernetes Dashboard. Disable public access via the internet. Ensure the Dashboard Service Account is not open and accessible to users. Configure the login page and enable RBAC.
10. Enable Kubernetes Security Logging
Audit logging provides container activity logs to determine the health and security of your Kubernetes. A Fluentd agent running on each node can log container errors, and send them to monitoring tools like Stackdriver or Elasticsearch and viewed using Kibana. You can also configure alerts, send notifications using Slack/Email, etc. Kubernetes, allows you to create audit policies that let you decide what things you’d like to log and at which level.
Conclusion
Kubernetes deployments can be highly distributed environments with large clusters. Kubernetes provides the mechanisms to keep your clusters secured but there is a learning curve involved to all configuration options. The best practices mentioned here will help you to get started and let you harden your deployment and reduce the attack surface. Building a deeper contextual knowledge by reading the official documentation will help you further reduce potential risks, vulnerabilities, and achieve your compliance goals.
To understand in-depth and tackle any Kubernetes related security issues, take this Certified Kubernetes Security Specialist (CKS) course and become a Kubernetes security expert.
References: