The Scenario: Securing My Side Project

Recently, while working on a side project to deploy a web application using Kubernetes, I faced a significant issue. One day, I got a call from one of my teammates who sounded worried. He told me that our application was crashing unexpectedly and there were strange activities in the logs. It was clear our Kubernetes setup had security vulnerabilities.

Here’s how I tackled the problem step-by-step:

Step-by-Step Solution

Step 1: Identifying the Threats

First, I checked the audit logs to understand the issue. Audit logs are like a diary of everything happening in the system. I found multiple unauthorized access attempts and unusual API requests. This helped me pinpoint suspicious activities and potential entry points for the attackers.

Realizing the severity, I knew we needed a robust security strategy immediately.

Navigating the Everest of Logs - A Guide to Understanding Kubelet Logs

Use this guide to better understand Kubernetes log, how to locate logs and increasing Kubelet log levels and more.

DevOps BlogNimesha Jinarajadasa

Step 2: Implementing Authentication and Authorization

Our first line of defense was to tighten access controls. I set up Role-Based Access Control (RBAC), giving different keys to different people based on their roles. By defining specific roles and assigning precise permissions, we ensured users only had access to the resources they needed. This minimized the risk of unauthorized activities, significantly reducing the attack surface.

Using RBAC Authorization

Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization. RBAC authorization uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to dynamically configure policies through the Kubernetes API. To enable RBAC, start the API server with the --authorization-mode flag set to a comma-separated list that includes RBAC; for example: kube-apiserver --authorization-mode=Example,RBAC --other-options --more-options API objects The RBAC API declares four kinds of Kubernetes object: Role, ClusterRole, RoleBinding and ClusterRoleBinding.

Kubernetes

Step 3: Securing the API Server

Next, I focused on securing the API server, the heart of Kubernetes, which handles all requests. I enabled TLS encryption, ensuring our communication was protected. Setting up audit logging allowed us to monitor all requests and responses in real-time. Additionally, I restricted access to the API server to trusted IP addresses only, making it harder for potential attackers to exploit vulnerabilities.

Controlling Access to the Kubernetes API

This page provides an overview of controlling access to the Kubernetes API. Users access the Kubernetes API using kubectl, client libraries, or by making REST requests. Both human users and Kubernetes service accounts can be authorized for API access. When a request reaches the API, it goes through several stages, illustrated in the following diagram: Transport security By default, the Kubernetes API server listens on port 6443 on the first non-localhost network interface, protected by TLS.

Kubernetes

Step 4: Protecting the Kubelet

The Kubelet, running on each node, needed securing. I ensured it required proper authentication and enabled TLS to secure its communications. By restricting access to the Kubelet API, we made sure only authorized entities could interact with it, protecting the nodes from unauthorized access.

Step 5: Implementing Network Policies

To control the flow of traffic within our cluster, I set up network policies. These policies specified which parts of the application could communicate with each other, isolating critical services. By implementing these controls, we reduced the attack surface, preventing attackers from moving laterally within the cluster.

I highly recommend the KodeKloud course on Kubernetes Network Policies. It's a fun and hands-on way to learn. You'll get plenty of practical experience and really understand how network policies work in Kubernetes.

Learn-By-Doing Kubernetes Network Policies | KodeKloud

Elevate your Kubernetes Network Policies skills by doing multiple hands-on labs

KodeKloud logo

Step 6: Double-Checking Secrets Management

Sensitive information like API keys and database passwords needed to be secure. Although we had already set up Kubernetes Secrets, I double-checked to ensure that all sensitive information was encrypted and accessible only by authorized parts of the application. This step was crucial in preventing leaks of sensitive information.

Kubectl Create Secret: How to Create, Use, and Access

This article discusses a Kubernetes Secret and how to create and use it. It also discusses some best practices to follow when working with it.

DevOps BlogFaizan Bashir

Step 7: Using Pod Security Admission

Since Pod Security Policies have been deprecated, I used Pod Security Admission to enforce security standards. This approach ensured that pods ran with the least privileges necessary, such as running containers as non-root users and setting resource limits. This minimized the potential damage even if an attacker gained access to a pod.

Step 8: Monitoring and Responding

We already had Prometheus and Grafana set up to monitor the cluster's health and visualize metrics. However, we needed a solution for real-time threat detection. I decided to add Falco to our setup. Falco immediately started monitoring system calls and detecting any suspicious behavior. This addition was crucial, as it alerted us to real-time threats and allowed us to respond quickly and effectively to any potential issues.

CNCF Tool Interviews Series: Falco

Today, we’re diving deep into the world of runtime security with the ever-watchful and slightly witty: Falco.

DevOps BlogNimesha Jinarajadasa

Step 9: Securing the Supply Chain

The supply chain in software development includes everything from writing code, building applications, to deploying them.

Ensuring the integrity of our application images was vital. I made sure all images were signed and scanned for vulnerabilities before deployment, much like checking packages for tampering before using them. This step helped prevent the introduction of vulnerabilities through the supply chain.

Step 10: Conducting Regular Security Audits and Ensuring Compliance

Finally, to maintain a high level of security, I conducted regular security audits. These audits involved:

• Reviewing Configurations: Checking all security configurations to ensure they were up-to-date and correctly applied.
• Penetration Testing: Simulating attacks to identify potential vulnerabilities.
• Compliance Checks: Ensuring that our practices met industry standards and regulations, like GDPR or HIPAA.

These audits helped identify potential weaknesses and allowed us to address them proactively. Regular audits were like health check-ups for our cluster, ensuring everything was in top shape and that our security measures were effective and compliant with the latest standards.

A Long and Challenging Journey

Solving these security issues was a long and challenging process. It took a lot of time and effort because I had no prior knowledge or good resources on Kubernetes security. I had to learn everything on the go, which was frustrating and time-consuming.

But then I discovered that the Linux Foundation published the Kubernetes and Cloud Native Security Associate (KCSA) certification. Having basic knowledge of Kubernetes security is crucial, and a structured learning path like the KCSA certification can make a huge difference. It helps you learn systematically and stay on schedule.

If you check KodeKloud's KCSA course(coming soon), you'll see that all these concepts I tackled during my solution are covered comprehensively. I urge you to take the KCSA certification soon. Being an expert in Kubernetes architecture and administration is never enough until you face a challenge like mine.

If you're interested, I can recommend KodeKloud's KCSA course(coming soon). It covers everything I had to figure out on my own, making it much easier to learn and apply Kubernetes security principles effectively.

Kubernetes and Cloud Native Security Associate (KCSA) | KodeKloud

KodeKloud logo

I had to recommend KodeKloud's KCSA course because, until now, there were no single comprehensive resources for the KCSA certification. KodeKloud has always published excellent courses. The only other option was Michael Levan's LinkedIn course, which just touches on the topics. KodeKloud offers comprehensive learning lessons, and the best part is their labs and quizzes that provide practical, hands-on experience.