Kubernetes

Certified Kubernetes Administrator Exam Series (Part-7): Storage

In the previous blog of this 10-part series, we discussed Security. This blog delves into how Volumes, PVs, PVCs, and Storage Classes help manage storage resource consumption manage the consumption of storage resources in Kubernetes clusters.

Here are the eight other blogs in the series:

The ephemeral nature of containers presents challenges when it comes to managing storage for Kubernetes compute instances. Kubernetes provides Persistent Volume (PV), Persistent Volume Claim (PVC), and Storage Class APIs so administrators can abstract storage consumption from its provision.

A Volume is a storage object that retains and avails data to containers running in a POD. With Volumes, it is easy to manage the sharing of files between containers in a Pod and recover files in case of a container crash. 

Storage in Docker

Before exploring storage in Kubernetes, it is important to understand storage in containers. This section of the tutorial series dives deep into how storage is handled in containers, as this lays the foundation for how Kubernetes provisions and manages volumes to provide storage for containerized applications. Docker storage encompasses two major concepts: storage drivers and volume drivers.

The following chapters are dedicated to giving the candidate a complete understanding of storage concepts in Docker. These concepts have been covered extensively in the DCA exam preparation tutorial series, so any candidate confident with the topic can skim through this part.

Docker Storage

The Docker File System 

When Docker is installed in a host machine, it starts storing container data in a specific directory on the host’s file systems. The default path for this directory is /var/lib/docker and it is run & managed by Docker. This directory contains subfolders such as aufs, containers, images, and volumes, among others. Docker populates these subfolders with information on various components running on the Docker host, including:  

  • Containers
  • Images
  • Storage drivers 
  • Persistent volumes

Layered Architecture

Docker containers are based on application images. A Dockerfile is a configuration file that consists of several instructions which define how the image is built. Each instruction in a Dockerfile represents a layer built by Docker when creating the image. In a Dockerfile, every instruction except for the final one is Read-Only.

To understand the advantage of layered architecture when building images, consider the Dockerfile below:

FROM ubuntu
RUN    apt-get update && apt-get -y install python python3-pip
RUN    pip install flask flask-mysql
COPY ./opt/source code
ENTRYPOINT FLASK_APP=/opt/source-code/app.py flask run

When Docker is building this image, it creates a new layer for each instruction that only includes changes from the previous layer. The result of executing each layer is stored in a cache so they can be reused and shared with other images. This makes building images a lot faster and more efficient.

If Docker is instructed to build a second image, but with the same OS, application, and Python packages but with a different source code and ENTRYPOINT as shown:

FROM ubuntu
RUN    apt-get update && apt-get -y install python python3-pip
RUN    pip install flask flask-mysql
COPY ./app2.py /opt/source code
ENTRYPOINT FLASK_APP=/opt/source-code/app2.py flask run

Docker will reuse the first three layers. It will only need to change the application’s source code and ENTRYPOINT. This layered architecture helps Docker build images faster while also saving disk space.

Docker builds containers by stacking layers on top of each other. Layers created during image builds are Read-Only and cannot be changed once the image build is complete. These layers are known as Image Layers and can only be changed by initiating a new image build. When the container is started using the docker run command, Docker will create a new writable layer on top of the image layer, known as the Container Layer. 

The container layer stores changes made by the container, including files created (logs, temps etc), modifications to existing files, and files deleted. The container layer is ephemeral and only lives as long as the container is running. The layer and any changes written on it are discarded when a container terminates. 

No modifications are made directly by the container to the image layer when the container is running. When modifying a file in the image layer, such as source code when the image is running, Docker creates a copy of the file in the container layer. Every change made by the container will be applied to this copy.

This is known as the Copy-on-Write strategy. This strategy is advantageous since it keeps image layers consistent when being shared between different containers. When the container is destroyed, all the modifications to the writable layer are deleted as well.

Volumes

Volumes are the go-to mechanism for persisting data created by the container layers. A Volume is created using the command:

docker volume create volume1

This command creates a folder named volume1 under volumes in the default docker directory /var/lib/docker.

This volume can then be mounted to a container’s Read/Write layer by specifying the new volume and the location in which it is to be mounted: 

docker run -e MYSQL_ROOT_PASSWORD=root -v volume1:/var/lib/mysql mysql

This command creates a new container mysql inside the folder /var/lib/mysql and mounts the volume created earlier. Any changes made to the database container will be written into the volume. The data written into this volume is active and accessible even when the container is destroyed. If the command is performed before, Docker will automatically create a volume and attach it to the container specified. This process is known as Volume Mounting. 

If the volume is already available elsewhere and data needs to be stored in it, a complete path to the mount folder is specified, for instance: 

docker run -e MYSQL_ROOT_PASSWORD=root -v /data/mysql:/var/lib/mysql mysql

This creates a new volume mysql on the /data folder that reads and persists data from the Read/Write layer of the mysql container. This process is known as Bind Mounting.

The newer --mount flag is preferred to the -v flag since it is explicit and more verbose. The --mount flag allows users to specify more options using comma-separated key-value pairs. Such options include:  type (bind/volume/tmpfs), source (source/src), destination (destination/dst/target), the readonly option and volume-opt options, which can include more than one key-value pair. 

Storage Drivers

Storage drivers enable and manage layered architecture in Docker. Some popular storage drivers include AUFS, BTRS, Overlay1, Overlay2, DeviceMapper and ZFS. Storage Drivers are OS-specific. AUFS, for instance, is the default storage driver for Ubuntu, but it may not work well with CentOS or Fedora. Docker automatically selects the appropriate storage driver based on the Operating System running in a container. 

Volume Driver Plugins

Storage drivers only manage the Docker layered architecture but not the volumes themselves. Volume drivers help connect containers to volumes and manage persistent data. local is Docker’s default volume driver, and it manages the /var/lib/docker directory. Other volume driver plugins exist, including the Local Persistent Volume plugin, which helps  Docker create standalone volumes that can be bound to different directories in multiple containers. 

Other Volume Driver plugins include BeeGFS Volume Plugin, Contiv, DigitalOcean Block Storage Plugin, Flocke, Fluxi, Horcrux, Netshare, OpenStorage, and REX-Ray, among others. Each plugin supports different storage solutions from various cloud vendors. REX-Ray, for instance, provides persistent storage for plenty of platforms such as Amazon EC2, OpenStack, VirtualBox, Google Compute Engine, and EMC, among others.

The command below, for instance, can be used to bind a container to a volume on Amazon’s AWS EBS storage service:

docker run -it \
      --name mysql -e MYSQL_ROOT_PASSWORD=root \
      --volume-driver rexray/ebs \
      --mount src=ebs-vol,target=/var/lib/mysql mysql

This command mounts a container mysql to the volume sourced from the EBS cloud and managed by REX-Ray’ EBS Plugin rexray/ebs.

Container Storage Interface

Earlier, Kubernetes only supported Docker as the container runtime engine, so Docker integrations were baked into Kubernetes. As support for other runtimes increased, Kubernetes developed the Container Runtime Interface (CRI) as an API that allows for separation in the Kubernetes Codebase to allow different runtimes to communicate with the Kubelet service running nodes.

The CRI standardizes interfaces to coordinate the resources used by container runtime engines. Such interfaces include the Container Networking Interface (CNI) and the Container Storage Interface (CSI).  

The Container Storage Interface is a standard interface that connects container workloads with storage platforms by defining how volume drivers are written. This creates a simplified set of specifications that allow various vendors to create storage plugins independent of the Kubernetes development team. The CSI is a global standard, meaning it is supported by most third-party storage vendors and cloud orchestration platforms. This means organizations should not have to worry about vendor or platform lockup.

The CSI defines several Remote Procedure Calls (RPCs) that should be invoked by the orchestrators and then implemented by the Volume Drivers on the storage system. For instance, if the orchestrator invokes a call to provision a new volume, then the Volume driver should create a new volume on the storage. If the orchestrator invokes a call to delete a volume, the driver should decommission the volume. The complete list of RPCs is available on the CSI Github directory here.

Volumes 

Containers are ephemeral; they are only built to run as long as the processes they host are active. A container is initiated to perform a specific task and then discarded as soon as it is done. The data processed by the container is also destroyed as soon as it exits. Docker attaches volumes to containers to persist data generated by their Read/Write layers. Any data processed by the container is stored in these volumes and is available when the container exits.

The ephemeral nature of containers also brings challenges for critical applications running on Kubernetes. This is because when a container crashes, the Kubelet service may restart it, but it comes back up with a clean slate. Multiple containers running in one Pod may also need to share files.

Pods themselves are transient, meaning they terminate as soon as the containers running inside them terminate. Kubernetes volumes create persistent storage for capturing cluster data. Volumes outlive containers and Pods, preserving cluster data even when Pods terminate. Attaching a volume to a Pod ensures that cluster data persists even in the event of Pod failure.

To understand volumes and data persistence, consider a Pod created to generate a random number between 1 and 100 and print it out to /opt/number.out whose specifications are shown: 

apiVersion: v1
kind: Pod
metadata:
  name: random-number-generator
spec: 
  containers:
  - image: alpine
    name: alpine
    command: ["/bin/sh","-c"]
    args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"]

Once the Pod terminates, the random number generated is also deleted. To retain the number in the output, a Volume is created and attached to the Pod:

volumes:
- name: data-volume
  hostpath: 
    path: /data
    Type: directory

The volume is then mounted to specific containers in the Pod by specifying the directory inside the container, for instance: 

volumeMounts:
- mountPath: /opt
  name:data-volume

The complete Pod definition file will be similar to:

apiVersion: v1
kind: Pod
metadata:
  name: random-number-generator
spec: 
  containers:
  - image: alpine
    name: alpine
    command: ["/bin/sh","-c"]
    args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"]
    volumeMounts:
    - mountPath: /opt
      name: data-volume
  volumes:
  - name: data-volume
    hostPath: 
      path: /data
      type: Directory

This means that the volume data-volume will persist data in the container on the R/W directory /opt and store it in the /data directory. The random number output will be available on /data even if the random-number-generator Pod goes down.  

Volume Storage Options

Kubernetes supports multiple storage options implemented as plugins. In the above example, the option used is hostPath which is great for local storage in single-node clusters. In multi-node clusters, however, the hostPath option is not suitable since it will assume a similar storage directory on all nodes while they are, in fact, separate servers.

Kubernetes also offers support for various third-party storage options. These include AzureDisk, AzureFile, AWS ElasticBlockStore (EBS), flocker, glusterfs, gce Persistent Disk, and vSphereVolume, among others. Each option has its own configuration specifications. For instance, to provision an AWS EBS cloud storage option for a Kubernetes Pod, the following configuration is used:  

volumes:
- name: data-volume
  awsElasticBlockStore:
    volumeID: <volume-id>
    fsType: ext4

Persistent Volumes

The volume created in the previous lecture is attached to a specific Pod, and its specifications are written within the Pod. This may be effective for small clusters but not in large production environments. With larger workloads, users create volumes every time a Pod initiates, and this creates a data management challenge. 

A Persistent Volume (PV) is a Kubernetes cluster resource that abstracts storage provision from consumption so storage can be assigned to multiple users. PVs can be provisioned dynamically using Storage Classes or by cluster administrators so that users can use a percentage of the volume when deploying applications. 

The PV is an API resource, a Volume Plugin whose lifecycle is independent of the Pods attached to it, and details how storage is implemented in the cluster. It then creates a shared storage pool out of which users can access a portion using Persistent Volume Claims (PVCs).

A PV can be created by specifying the configurations in a YAML manifest file, as shown:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-vol1
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  hostPath:
     path: /tmp/data

Each PV gets its own set of access modes describing that specific PV’s capabilities.

The access modes are:

  • ReadWriteOnce — the volume can be mounted as read-write by a single node
  • ReadOnlyMany — the volume can be mounted read-only by many nodes
  • ReadWriteMany — the volume can be mounted as read-write by many nodes
  • ReadWriteOncePod — the volume can be mounted as read-write by a single Pod

This creates a PV named pv-vol1 with an access mode ReadWriteOnce and a 1 Gigabyte capacity hosted in the directory  /tmp/data.

Persistent Volume Claims (PVCs)

The PVC is a request made by users to access a portion of PV storage. It is a namespace object, separate from the PV, that allows end users to consume abstracted storage.

Users typically need varying PV specifications for different tasks, and PVCs allow them to specify desired properties such as Volume Mode, Access Mode, and Volume Size, among others. While a PV is a cluster resource, a PVC is a namespace resource. The cluster administrator provisions storage using PV, while users request this storage using PVCs. 

Once a PVC has been created, Kubernetes binds it to a PV with sufficient capacity and matching other properties mentioned in the PVC. These properties include accessMode, Storage Mode, and Volume Mode, among others. 

If more than one PV that matches the requirements of the PV exists, labels and selectors can be used to bind PVCs to specific PVs. If no other match exists, a PVC may be assigned a PV with much larger storage but with other matching properties. Since the PVCs and PVs share a 1:1 relationship, no other PVCs can join to use the extra space.  

A PVC is also created using a YAML definition file, with request specifications similar to the one below:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi

This PVC can then be bound to a Pod so that the Pod can access the PV earlier created. This is performed by specifying the claim in the Pods definition file, as shown:

volumes:
  - name: data-volume
    persistentVolumeClaim: 
      claimName: myclaim

Storage Class

When using PVs, as explored in the previous chapter, a physical storage disk has to be created on Google Cloud every time a volume is created. This is known as static provisioning. Storage Classes offer dynamic storage provisioning. This is where a provisioner such as GCE Storage or AWS EBS is defined to automatically provision storage on the specific cloud service and attach it to Pods when they make a PVC.

A Storage Class object is crucial since administrators can use it to map the quality of storage to certain policies and tags, allowing them to create different ‘profiles’ for users requesting storage. Administrators create different storage classes by stating the name provisioned and other parameters in a YAML definition file.

These parameters vary from one provisioner to another and may include mount options, reclaim policies, and volume expansion capability, among others. A typical manifest file for a Storage Class provisioning volumes using AWS EBS would look similar to:   

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: aws-standard
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Retain
allowVolumeExpansion: true

The Storage Class name is then specified in the PVCs manifest under the  specification, as shown: 

spec:
  accessModes:
ReadOnlyOnce
  storageClassName: aws-standard
  resources:
     requests:
        storage: 500Mi

The Storage Class automatically creates the PV and any other associated storage as soon as it initiates, eliminating the need to create one for each PVC. Each time a user creates a PVC request, the Storage Class orders the provisioner to create a disk on Google Cloud that fulfills the request. There are plenty of Storage Classes from different providers, including  FlexVolume, CephFS, PortworxVolume, GCEPersistentDisk, and Cinder, among others. 

This concludes the storage section of the CKA certification exam.

You can now proceed to the next part of this series: Certified Kubernetes Administrator Exam Series (Part-8): Networking.

Here is the previous part of the series: Certified Kubernetes Administrator Exam Series (Part-6): Security.

Research Questions

Here is a quick quiz with a few questions and sample tasks to help you assess your knowledge. Leave your answers in the comments below and tag us back. 

Quick Tip – Questions below may include a mix of DOMC and MCQ types.

1. Task: Create a Persistent Volume with the given specification.

  • Volume Name: pv-log
  • Storage: 100Mi
  • Access Modes: ReadWriteMany
  • Host Path: /pv/log
  • Reclaim Policy: Retain

solution:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-log
spec:
  persistentVolumeReclaimPolicy: Retain
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 100Mi
  hostPath:
    path: /pv/log

2. Let us claim some of that storage for our application. Create a Persistent Volume Claim with the given specification.

  • Volume Name: claim-log-1
  • Storage Request: 50Mi
  • Access Modes: ReadWriteOnce

solution:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: claim-log-1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Mi

3. What would happen to the PV if the PVC was destroyed?

[A] The PV is scrubbed

[B] The PV is made available again

[C] The PV is not deleted but not available

[D] The PV is deleted as well

4. Create a new Storage Class called delayed-volume-sc that makes use of the below specs:

  •  provisioner: kubernetes.io/no-provisioner
  • volumeBindingMode: WaitForFirstConsumer

Solution:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: delayed-volume-sc
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Conclusion

This section has explored storage concepts and how Kubernetes cluster data persists when using volumes. The ephemeral nature of Docker Containers and Kubernetes Pods necessitates special attention given to storage. The concepts of ephemeral volumes, persistent volumes, and Storage Classes have been explored in-depth, with accompanying labs to help the candidate gain confidence with storage management in Kubernetes both for the CKA exam and for production environments. 

Exam Preparation Course

Our CKA Exam Preparation course explains all the Kubernetes concepts included in the certification’s curriculum. After each topic, you get interactive quizzes to help you internalize the concepts learned. At the end of the course, we have mock exams that will help familiarize you with the exam format, time management, and question types.

Explore our CKA exam preparation course curriculum.

Enroll Now!

Mumshad Mannambeth

• 13 min read