Hi ,
I have a 3 node cluster and have pods (~100+) configured on them.
Some day back implemented memory and CPU limits based on my analysis.
But now frequently many pods are getting killed by OOM, even when the nodes have penalty of memory/CPU.
As the pods are restarting I am unable to find out why or which process consumed process at this time of crash/OOM Killed.
I can see in pod describe , last terminated reason OOM killed and in describe node, killed process ID (java) but the process is already terminated ( not sure if it was part of that POD).
Could anyone please suggest how can I debug on this issue and find out the root cause.
Log snippet:
++++++++++ <<< POD Describe >>> ++++++++++++++++++
State: Running
Started: Wed, 17 Jan 2024 10:53:27
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Mon, 08 Jan 2024 15:11:55
Finished: Wed, 17 Jan 2024 10:53:26
Ready: True
Restart Count: 158
Limits:
cpu: 100m
memory: 256Mi
Requests:
cpu: 100m
memory: 256Mi
Liveness: http-get http://:8080//health delay=2s timeout=1s period=2s #success=1 #failure=3
Readiness: http-get http://:8080//health delay=2s timeout=1s period=2s #success=1 #failure=3
++++++++++++<<<< Node Describe/Event >>>>++++++++++++++++++
Warning OOMKilling 16m kernel-monitor Memory cgroup out of memory: Killed process 3098370 (java) total-vm:5478072kB, anon-rss:150084kB, file-rss:46112kB, shmem-rss:60kB, UID:0 pgtables:720kB oom_score_adj:-997
Warning OOMKilling 10m kernel-monitor Memory cgroup out of memory: Killed process 3106142 (java) total-vm:5478060kB, anon-rss:138520kB, file-rss:46024kB, shmem-rss:60kB, UID:0 pgtables:688kB oom_score_adj:-997
+++++++++++++
Resource Requests Limits
cpu 53604m (41%) 268500m (209%)
memory 37112715770 (6%) 683063790260 (126%)
++++++++++++++++++++++++
Thank you in Advance.
-Subhash