Solve this question on: `ssh cluster1-controlplane`
The `green-deployment-cka15-trb` deployment is having some issues since the corresponding POD is crashing and restarting multiple times continuously.
Investigate the issue and fix it. Make sure the POD is in a running state and is stable (i.e, `NO RESTARTS!`).
Is POD `stable` now?
My solutions: step by step
Step1: CHeck the deploymentfirst
cluster1-controlplane ~ ➜ k describe deployments.apps green-deployment-cka15-trb
Name: green-deployment-cka15-trb
Namespace: default
CreationTimestamp: Wed, 18 Mar 2026 09:19:48 +0000
Labels: app=green-app-cka15-trb
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=green-app-cka15-trb
Replicas: 1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=green-app-cka15-trb
Containers:
mysql:
Image: mysql:5.6
Port: 3306/TCP
Host Port: 0/TCP
Limits:
cpu: 100m
memory: 256Mi
Requests:
cpu: 50m
memory: 256Mi
Environment:
MYSQL_ROOT_PASSWORD: <set to the key ‘password’ in secret ‘green-root-pass-cka15-trb’> Optional: false
MYSQL_DATABASE: <set to the key ‘database’ in secret ‘green-db-url-cka15-trb’> Optional: false
MYSQL_USER: <set to the key ‘username’ in secret ‘green-user-pass-cka15-trb’> Optional: false
MYSQL_PASSWORD: <set to the key ‘password’ in secret ‘green-user-pass-cka15-trb’> Optional: false
Mounts:
Volumes:
Node-Selectors:
Tolerations:
Conditions:
Type Status Reason
Progressing True NewReplicaSetAvailable
Available False MinimumReplicasUnavailable
OldReplicaSets:
NewReplicaSet: green-deployment-cka15-trb-7ffcd7dd9b (1/1 replicas created)
Events:
Type Reason Age From Message
Normal ScalingReplicaSet 28s deployment-controller Scaled up replica set green-deployment-cka15-trb-7ffcd7dd9b from 0 to 1
Step2: Somthing with secrets:
cluster1-controlplane ~ ➜ k get secrets
NAME TYPE DATA AGE
green-db-url-cka15-trb Opaque 1 44s
green-root-pass-cka15-trb Opaque 1 44s
green-user-pass-cka15-trb Opaque 2 44s
cluster1-controlplane ~ ➜ k get secrets green-db-url-cka15-trb -o yaml
apiVersion: v1
data:
database: Y2thMTVfZGI=
kind: Secret
metadata:
creationTimestamp: “2026-03-18T09:19:48Z”
name: green-db-url-cka15-trb
namespace: default
resourceVersion: “8780”
uid: fd613348-7b5c-4aa4-a60b-1f4e09871436
type: Opaque
cluster1-controlplane ~ ➜ k get secrets green-root-pass-cka15-trb -o yaml
apiVersion: v1
data:
password: WVVJaWRoYjY2Nw==
kind: Secret
metadata:
creationTimestamp: “2026-03-18T09:19:48Z”
name: green-root-pass-cka15-trb
namespace: default
resourceVersion: “8777”
uid: 3846d74d-3e6b-4f31-b361-53e26de32506
type: Opaque
cluster1-controlplane ~ ➜ k get secrets green-user-pass-cka15-trb -o yaml
apiVersion: v1
data:
password: WWNoWkhSY0xrTA==
username: a29kZWtsb3VkX3JheQ==
kind: Secret
metadata:
creationTimestamp: “2026-03-18T09:19:48Z”
name: green-user-pass-cka15-trb
namespace: default
resourceVersion: “8778”
uid: feb46b23-426f-4d9b-b49a-ad25ae96ee55
type: Opaque
Secrets are good
Step3: Let see logs
cluster1-controlplane ~ ➜ k logs pod/green-deployment-cka15-trb-7ffcd7dd9b-z524l
2026-03-18 09:23:29+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.6.51-1debian9 started.
2026-03-18 09:23:30+00:00 [Note] [Entrypoint]: Switching to dedicated user ‘mysql’
2026-03-18 09:23:30+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.6.51-1debian9 started.
2026-03-18 09:23:30+00:00 [Note] [Entrypoint]: Initializing database files
2026-03-18 09:23:31 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2026-03-18 09:23:31 0 [Note] Ignoring --secure-file-priv value as server is running with --bootstrap.
2026-03-18 09:23:31 0 [Note] /usr/sbin/mysqld (mysqld 5.6.51) starting as process 94 …
2026-03-18 09:23:31 94 [Note] InnoDB: Using atomics to ref count buffer pool pages
2026-03-18 09:23:31 94 [Note] InnoDB: The InnoDB memory heap is disabled
2026-03-18 09:23:31 94 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2026-03-18 09:23:31 94 [Note] InnoDB: Memory barrier is not used
2026-03-18 09:23:31 94 [Note] InnoDB: Compressed tables use zlib 1.2.11
2026-03-18 09:23:31 94 [Note] InnoDB: Using Linux native AIO
2026-03-18 09:23:31 94 [Note] InnoDB: Using CPU crc32 instructions
2026-03-18 09:23:31 94 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2026-03-18 09:23:31 94 [Note] InnoDB: Completed initialization of buffer pool
2026-03-18 09:23:31 94 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2026-03-18 09:23:31 94 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
2026-03-18 09:23:31 94 [Note] InnoDB: Database physically writes the file full: wait…
2026-03-18 09:23:31 94 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2026-03-18 09:23:31 94 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
2026-03-18 09:23:31 94 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2026-03-18 09:23:31 94 [Warning] InnoDB: New log files created, LSN=45781
2026-03-18 09:23:31 94 [Note] InnoDB: Doublewrite buffer not found: creating new
2026-03-18 09:23:31 94 [Note] InnoDB: Doublewrite buffer created
2026-03-18 09:23:31 94 [Note] InnoDB: 128 rollback segment(s) are active.
2026-03-18 09:23:31 94 [Warning] InnoDB: Creating foreign key constraint system tables.
2026-03-18 09:23:31 94 [Note] InnoDB: Foreign key constraint system tables created
2026-03-18 09:23:31 94 [Note] InnoDB: Creating tablespace and datafile system tables.
2026-03-18 09:23:31 94 [Note] InnoDB: Tablespace and datafile system tables created.
2026-03-18 09:23:32 94 [Note] InnoDB: Waiting for purge to start
2026-03-18 09:23:32 94 [Note] InnoDB: 5.6.51 started; log sequence number 0
2026-03-18 09:23:32 94 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2026-03-18 09:23:32 94 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
2026-03-18 09:23:32 94 [Note] Binlog end
2026-03-18 09:23:32 94 [Note] InnoDB: FTS optimize thread exiting.
2026-03-18 09:23:32 94 [Note] InnoDB: Starting shutdown…
2026-03-18 09:23:33 94 [Note] InnoDB: Shutdown completed; log sequence number 1625977
2026-03-18 09:23:33 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2026-03-18 09:23:33 0 [Note] Ignoring --secure-file-priv value as server is running with --bootstrap.
2026-03-18 09:23:33 0 [Note] /usr/sbin/mysqld (mysqld 5.6.51) starting as process 117 …
2026-03-18 09:23:33 117 [Note] InnoDB: Using atomics to ref count buffer pool pages
2026-03-18 09:23:33 117 [Note] InnoDB: The InnoDB memory heap is disabled
2026-03-18 09:23:33 117 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2026-03-18 09:23:33 117 [Note] InnoDB: Memory barrier is not used
2026-03-18 09:23:33 117 [Note] InnoDB: Compressed tables use zlib 1.2.11
2026-03-18 09:23:33 117 [Note] InnoDB: Using Linux native AIO
2026-03-18 09:23:33 117 [Note] InnoDB: Using CPU crc32 instructions
2026-03-18 09:23:33 117 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2026-03-18 09:23:34 117 [Note] InnoDB: Completed initialization of buffer pool
2026-03-18 09:23:34 117 [Note] InnoDB: Highest supported file format is Barracuda.
2026-03-18 09:23:34 117 [Note] InnoDB: 128 rollback segment(s) are active.
2026-03-18 09:23:34 117 [Note] InnoDB: Waiting for purge to start
2026-03-18 09:23:34 117 [Note] InnoDB: 5.6.51 started; log sequence number 1625977
2026-03-18 09:23:34 117 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2026-03-18 09:23:34 117 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
2026-03-18 09:23:34 117 [Note] Binlog end
2026-03-18 09:23:34 117 [Note] InnoDB: FTS optimize thread exiting.
2026-03-18 09:23:34 117 [Note] InnoDB: Starting shutdown…
2026-03-18 09:23:36 117 [Note] InnoDB: Shutdown completed; log sequence number 1625987
PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:
/usr/bin/mysqladmin -u root password ‘new-password’
/usr/bin/mysqladmin -u root -h green-deployment-cka15-trb-7ffcd7dd9b-z524l password ‘new-password’
Alternatively you can run:
/usr/bin/mysql_secure_installation
which will also give you the option of removing the test
databases and anonymous user created by default. This is
strongly recommended for production servers.
See the manual for more instructions.
Please report any problems at http://bugs.mysql.com/
The latest information about MySQL is available on the web at
Support MySQL by buying support/licenses at http://shop.mysql.com
Note: new default config file not created.
Please make sure your config file is current
WARNING: Default config file /etc/mysql/my.cnf exists on the system
This file will be read by default by the MySQL server
If you do not want to use this, either remove it, or use the
–defaults-file argument to mysqld_safe when starting the server
2026-03-18 09:23:36+00:00 [Note] [Entrypoint]: Database files initialized
2026-03-18 09:23:36+00:00 [Note] [Entrypoint]: Starting temporary server
2026-03-18 09:23:36+00:00 [Note] [Entrypoint]: Waiting for server startup
cluster1-controlplane ~ ➜
left it
Solution : after exam i stotlly different than my case
ssh cluster1-controlplane
List the pods to check its status
kubectl get pod
its must have crashed already so lets look into the logs.
kubectl logs -f green-deployment-cka15-trb-xxxx
You will see some logs like these ??? not in my case
2022-09-18 17:13:25 98 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-09-18 17:13:25 98 [Note] InnoDB: Memory barrier is not used
2022-09-18 17:13:25 98 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-09-18 17:13:25 98 [Note] InnoDB: Using Linux native AIO
2022-09-18 17:13:25 98 [Note] InnoDB: Using CPU crc32 instructions
2022-09-18 17:13:25 98 [Note] InnoDB: Initializing buffer pool, size = 128.0M
Killed
This might be due to the resources issue, especially the memory, so let’s try to recreate the POD to see if it helps.
kubectl delete pod green-deployment-cka15-trb-xxxx
Now watch closely the POD status
kubectl get pod
Pretty soon you will see the POD status has been changed to OOMKilled.
To confirm if the pod’s memory limits are causing the crash, follow these diagnostic steps:
Check Pod Events and Status:
kubectl describe pod green-deployment-cka15-trb-xxxx
Look under the Events section and in the container’s Last State for entries like:
Reason: OOMKilled
This confirms the pod was terminated because it exceeded its memory limit.
So let’s look into the resources that are assigned to this deployment.
kubectl get deploy
kubectl edit deploy green-deployment-cka15-trb
- Under
resources:→limits:changememoryfrom256Mito512Miand save the changes.
Now watch closely the POD status again
kubectl get pod
It should be stable now.
Note: If OOMKilled still happens, increase the memory limit in increments of 256Mi(256Mi -> 512Mi -> 768Mi -> 1Gi) until stable.