When trying to get your containerized application up and running in a Kubernetes cluster, you may run into errors that prevent the application from running. One such error is
In this blog post, we’ll learn how to debug the
CrashLoopBackOff status of a Pod. Our approach? We’ll create a Pod and intentionally leave out certain configuration details to trigger a
CrashLoopBackOff state. Then we’ll explore various troubleshooting mechanisms to fix the error and get the Pod working again. Let’s get started!
What Is CrashLoopBackOff?
In Kubernetes, a Pod enters a state known as
CrashLoopBackOff when an application inside the Pod keeps crashing repeatedly. When this happens, the kubelet – an agent that operates on each worker node – steps in. It attempts to restart the failing application after each crash, creating a cycle of crash, loop (restart), and back off (wait and try again); hence the name,
Below are some factors that can get a Pod into a
- Bugs within the application: If the software inside the Pod contains errors, it might crash, causing the Pod to enter a
- Insufficient resources: Lack of adequate memory or other system resources can prevent the application from running smoothly, causing it to crash.
- Missing or incorrect configuration details: The application may require certain configurations to run correctly. If these details are incorrect or not provided, it can lead to application failure.
- Port conflicts: If multiple applications attempt to bind to the same port, it can cause a conflict, leading to application failure and thus the
To follow along with the example in this post, you’ll need access to a running Kubernetes cluster. If you don’t have access to one, you can use a tool such as minikube to set up a Kubernetes cluster. Also, you’ll need to have kubectl installed on your local machine to interact with the Kubernetes cluster.
Create a Pod
Start by launching your code editor. Next, create a new directory. Within this directory, make a new file with a
.yaml extension. Copy and paste the following content into the new
apiVersion: apps/v1 kind: Deployment metadata: name: mysql-deployment spec: replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql-container image: mysql:8.0
In this file, we have defined a Deployment named
mysql-deployment that is responsible for managing a single Pod. Withing this Pod, there will be one container running the
mysql:8.0 image. The Deployment will ensure that the Pod is always kept running. If the Pod crashes for some reason, the Deployment will automatically recreate the Pod, maintaining the desired count (which in this case is set to 1).
Now run the command below to create the Deployment:
kubectl apply -f <YOUR-FILE-NAME>.yaml
<YOUR-FILE-NAME> with the name of your file.
Executing the command above will produce the following output:
As you can see, a Deployment named
mysql-deployment has been created. Next, let’s check the Pod status by running the following command:
kubectl get pods
This will generate an output similar to the following:
As you can see, the Pod has a
CrashLoopBackOff status as highlighted above.
kubectl get podscommand immediately after creating the Deployment may not display the
CrashLoopBackOffstatus. Instead, you might see a different status like
Error, for example. This variation occurs because the Pod goes through different states during its lifecycle as it starts up, runs, and potentially encounters issues. So, wait for up to five minutes or so before running the
kubectl get podscommand again, to see the
Now, let's focus on the following three columns of the output above and examine their meaning:
READY: This indicates the number of containers in the Pod that are ready.
0/1means there is 1 container in the Pod, and 0 of them are ready. This suggests that the container is not yet running as expected.
STATUS: This shows the current status of the Pod.
CrashLoopBackOffmeans the container in this Pod is failing and is being restarted, over and over again.
RESTARTS: This indicates how many times the container within the Pod has been restarted. The number 5 suggests that the container has crashed and been restarted five times.
CrashLoopbackOff status is a clear indication that something has gone wrong with the Pod. Let’s troubleshoot to find out the cause of this problem.
Kubernetes offers various mechanisms for troubleshooting issues. In this section, we'll explore two steps that will guide you through the debugging process and help you resolve the
Step 1: Using
kubectl describe pod to get Pod details
To address the issue with our Pod, the first step is to get detailed Pod information. We might find some helpful clues that can guide us in the right direction.
Run the following command to fetch Pod details:
kubectl describe pod <POD-NAME>
<POD-NAME> with the name of your Pod.
Running the command above will produce an output similar to the following:
Note that the screenshot above shows only a partial section of the complete output.
Containers section of the output, we see that the container is in a
Waiting state due to
CrashLoopBackOff, with an exit code of 1. Typically, an exit code of 0 means that the process ran successfully without errors, while a non-zero exit code indicates some sort of error. However, in this case, the exit code does not provide us with sufficient information to tell us what we should do next.
To gather more insights, scroll down to the
In the last line of the
Events section, we find the statement
Back-off restarting failed container. This means that the kubelet has attempted to restart the container after a failure but has been unsuccessful. As a result, the kubelet will make another attempt to restart the container. Now, you might wonder, what does "Back-off" mean?
Kubernetes, by default, restarts a Pod when it crashes. If the application doesn’t recover after the restart, Kubernetes doesn’t immediately attempt to restart the Pod again. Instead, it waits for some time.
Why? Because it hopes the waiting time resolves any temporary issue the application may have. If the issue still persists, Kubernetes increases the waiting time between each restart attempt, up to a certain limit. This approach is referred to as the back-off strategy.
Events section does not have any helpful information regarding why the container fails to start. Therefore, our next step is to inspect the Pod logs for any potential clues regarding the issue.
Step 2: Using
kubectl logs to get Pod logs
Pod logs are records of events within an application. They can often provide valuable insights into any issues the application is facing.
To learn more about Pod Logs, check out our blog post: How to Get Pod Logs in Kubernetes.
To fetch the logs, run the following command:
kubectl logs <POD-NAME>
Running the command above will produce the following output:
The logs clearly indicate the reason for the error — we have not set the password in our Deployment configuration. To resolve this issue, one of the options is to define the
MYSQL_ROOT_PASSWORD in the environmental variables section of our Deployment configuration. Let’s do that now.
Add the following details to the
containers block of the
template.spec in your configuration file. Note that you can provide any value you prefer for the password.
env: - name: MYSQL_ROOT_PASSWORD value: secret-password
The final configuration file should look like this:
apiVersion: apps/v1 kind: Deployment metadata: name: mysql-deployment spec: replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql-container image: mysql:8.0 env: - name: MYSQL_ROOT_PASSWORD value: secret-password
After updating the Deployment configuration file, run the following command to apply the changes:
kubectl apply -f <your-file-name>.yaml
After running the command above, you’ll see the following output:
As you can see, Kubernetes has updated the existing Deployment with the new configuration. Next, let’s run the following command to check the Pod status:
kubectl get pods
From the output above, the Pod is up and running as expected. We have resolved the
CrashLoopbackOff state of the Pod.
To LEARN Kubernetes concepts with hands-on labs, ENROLL in our Kubernetes Course:
TEST your Kubernetes expertise with our FREE, fun, and challenging set of hands-on challenges.
In this blog post, you learned how to troubleshoot the
CrashLoopBackOff state of a Pod.
While troubleshooting application errors, you might sometimes need to get a shell to the running container so that you can run commands and interact with the container’s environment. To learn how to open and access a container’s shell, read our blog post: Kubectl Exec: How to Execute Shell Commands Into a Running Container (With Examples)?
More on Kubernetes:
- Vertical Pod Autoscaler (VPA) in Kubernetes Explained through an Example
- What Is Kubernetes DaemonSet and How to Use It?
- What Are Objects Used for in Kubernetes? 11 Types of Objects Explained
- ClusterIP vs. NodePort vs. LoadBalancer: Key Differences and When to Use Them?
- kubectl cp: How to Copy File From Pod to Local (With Examples)?