KubeCon 2019 was a great event. Almost 12000 Kubernetes enthusiasts from all over the world were in San Diego last week, making it the largest Kubecon ever.

In one of the keynote sessions, Vickie Cheung (an Engineering Manager at Lyft) who was a co-chair at the event, spoke about the newest features in Kubernetes 1.16.

So what is new with Kubernetes? Plenty! IPv4/IPv6 Dual Stack Support, Overhauled Metrics, Volume Resizing and changes to the Kubernetes API just to name a few.

In this post, I will talk about one of the newest additions to Kubernetes 1.16 — Ephemeral Containers.

If you want to try this feature yourself, check out the KodeKloud guided hands-on lab link at the end of this article.

An ephemeral container is a new, special type of container that can be run temporarily in an existing pod.

How do I benefit from this?

The best use case of this feature would be to carry out a live troubleshooting session on an existing or a crashed pod.

By design, we cannot add a new container to a pod once it has been created. As a result, if you want to inspect it, you would have to “exec” into the pod and troubleshoot using whatever tools that were originally included in the container image.

Most container images, specifically the distro-less images, do not have troubleshooting tools installed. This is done intentionally to keep the image minimalistic, reduce the attack surface and risk of additional security issues.

What if the POD crashes?

In either of the two scenarios above, just an exec into a pod will not help with the troubleshooting.

With this new feature, you can now run an Ephemeral container on an existing Pod. The ephemeral container will share the same network namespace as the base container. You can also enable process namespace sharing. This can be quite handy if you want to see the processes of the base container in the pod.

Before we all get too excited, please note that Ephemeral Containers is in an early alpha release and as such it is not recommended to use it in Production. There can be significant changes in this feature in the future or it may go away entirely.

How to use Ephemeral Containers?

At the time of writing this blog, to make use of this feature, we have to set the EphemeralContainers feature gate to true. Here are the steps to be followed:

a. First, we need to enable this feature. To do this, add — feature-gates=EphemeralContainers=true flag to the command section for the kube-apiserver and scheduler pods.

For example, in a cluster setup by kubeadm with the default configuration, edit the static pod definition files:

  1. /etc/kubernetes/manifests/kube-apiserver.yaml
  2. /etc/kubernetes/manifests/kube-scheduler.yaml

and also add it to the kubelet startup command. (for example add it to /etc/systemd/system/kubelet.service file and reload the service)

add flags to the command
add flag to the kubelet service

Make sure to restart the kubelet service after you make the changes!

b. Create a new pod which we would need to inspect later OR skip to the next step if you have an existing pod that you want to debug.

If you want to check the processes using the Ephemeral Container, set shareProcessNamespace to “true”.

<span class="gx gy ay bq iv b aw iw ix r iy">apiVersion: v1
kind: Pod
  creationTimestamp: null
    run: nginx-check
    name: nginx-check
  <span class="iv gu">shareProcessNamespace: true</span>
    - image: nginx
      name: nginx-check</span>

c. Next, create an ephemeral resource.

In this example, we will call the container “debugger” using the image busybox:1.28

<span class="gx gy ay bq iv b aw iw ix r iy">{
  "apiVersion": "v1",
  "kind": "EphemeralContainers",
  "metadata": {
    "name": "nginx-check"
  "ephemeralContainers": [
      "command": [
      "image": "busybox:1.28",
      "imagePullPolicy": "Always",
      "name": "debugger",
      "stdin": true,
      "tty": true,
      "terminationMessagePolicy": "File"

We will add this container to the pod called “nginx-check”.

d. To add this container to an existing pod, use the replace command as shown below.

<span class="gx gy ay bq iv b aw iw ix r iy">kubectl replace --raw /api/v1/namespaces/default/pods/nginx-check/ephemeralcontainers -f /root/debugger.json</span>

Check the status of the container by running a “kubectl describe pod nginx-check”.

If the previous steps have been completed correctly, the ephemeral container should be in a “Running” state.

e. You can now connect to this container using the “attach” command

<span class="gx gy ay bq iv b aw iw ix r iy">kubectl attach -it nginx-check -c debugger</span>

Since the process namespace sharing is set to true, you can see the processes belonging to the nginx pod.

Ephemeral Containers are designed for a specific purpose. They are not intended to build applications and hence lack many fields that are otherwise available for a regular container. Examples include ports and resource guarantees.

For a complete list, refer: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.16/#ephemeralcontainer-v1-core

Also, it is important to understand that ephemeral containers cannot be added using “kubectl edit”. This is because they make use of ephemeralcontainers handler in the API.

If you would like to try it yourself, KodeKloud provides a free guided hands-on lab where you can deploy an ephemeral container and attach it to a Pod!

Access the Lab here: https://kodekloud.com/p/public-labs/?scenario=kubernetes-ephemeral-containers

Happy Troubleshooting!