Introduction to Kubernetes Storage
The ephemeral nature of containers presents challenges when it comes to managing storage for Kubernetes compute instances. Kubernetes provides Persistent Volume (PV), Persistent Volume Claim (PVC) and Storage Class APIs so that administrators can abstract the consumption of storage from its provision. A Volume is a storage object that retains and avails data to containers running in a POD. With Volumes, it is easy to manage the sharing of files between containers in a Pod and recover files in case of a container crash.
This section delves into Volumes, PVs, PVCs and Storage Classes that help to manage the consumption of storage resources in Kubernetes clusters. Kubernetes supports many Volume Types and storage options from a wide range of cloud service providers. This section only explores the resources and concepts available within Kubernetes. With a grasp of these concepts, the candidate can provision storage for Kubernetes applications, whether locally, or through other supported volumes such as AWS ElasticBlockStore, azureDisk, cephfs, cinder, etc.
Storage in Docker
Before exploring storage in Kubernetes, it is important to understand storage in containers. This section of the tutorial series dives deep into how storage is handled in containers as this lays the foundation for how Kubernetes provisions and manages volumes to provide storage for containerized applications. Docker storage encompasses two major concepts, storage drivers and volume drivers. The following chapters are dedicated to giving the candidate a complete understanding of storage concepts in Docker. These concepts have been covered extensively in the DCA exam preparation tutorial series, so any candidate confident with the topic can skim through this part.
The Docker File System
When Docker is installed in a host machine, it starts storing container data in a specific directory on the host’s file systems. The default path for this directory is
/var/lib/docker and it is run & managed by Docker. This directory contains subfolders such as aufs, containers, image and volumes among others. Docker populates these subfolders with information on various components running on the Docker host including:
- storage drivers
- persistent volumes
Docker containers are based on application images. A Dockerfile is a configuration file that consists of several instructions which define how the image is built. Each instruction in a Dockerfile represents a layer built by Docker when creating the image. In a Dockerfile, every instruction except for the final one is Read-Only.
To understand the advantage of layered architecture when building images, consider the Dockerfile below:
FROM ubuntu RUN apt-get update && apt-get -y install python python3-pip RUN pip install flask flask-mysql COPY ./opt/source code ENTRYPOINT FLASK_APP=/opt/source-code/app.py flask run
When Docker is building this image, it creates a new layer for each instruction that only includes changes from the previous layer. The result of executing each layer is stored in a cache, so they can be reused and shared with other images. This makes building images a lot faster and efficient. If Docker is instructed to build a second image, but with the same OS, application and python packages but with a different source code and
ENTRYPOINT as shown:
FROM ubuntu RUN apt-get update && apt-get -y install python python3-pip RUN pip install flask flask-mysql COPY ./app2.py /opt/source code ENTRYPOINT FLASK_APP=/opt/source-code/app2.py flask run
Docker will reuse the first three layers. It will only need to change the application’s source code and ENTRYPOINT. This layered architecture helps Docker build images faster while also saving disk space.
Docker builds containers by stacking layers on top of each other. Layers created during image builds are Read-Only and cannot be changed once the image build is complete. These layers are known as Image Layers and can only be changed by initiating a new image build. When the container is started using the
docker run command, Docker will create a new writable layer on top of the image layer known as the Container Layer. The container layer stores changes made by the container, including files created (logs, temps etc), modifications to existing files and files deleted. The container layer is ephemeral and only lives as long as the container is running. The layer and any changes written on it are discarded when a container terminates.
No modifications are made directly by the container to the image layer when the container is running. When modifying a file in the image layer, such as source code when the image is running, Docker creates a copy of the file in the container layer. Every change made by the container will be applied to this copy. This is known as the Copy-on-Write strategy. This strategy is advantageous since it keeps image layers consistent when being shared between different containers. When the container is destroyed, all the modifications to the writable layer are deleted as well.
Volumes are the go-to mechanism for persisting data created by the container layers. A Volume is created using the command:
$ docker volume create volume1
This command creates a folder named
volumes in the default docker directory
This volume can then be mounted to a container’s Read/Write layer by specifying the new volume and the location in which it is to be mounted:
$ docker run -e MYSQL_ROOT_PASSWORD=root -v volume1:/var/lib/mysql mysql
With this command, creates a new container
mysql inside the folder
/var/lib/mysql and mounts the volume created earlier. Any changes made to the database container will be written into the volume. The data written into this volume is active and accessible even when the container is destroyed. If the command is performed before, Docker will automatically create a volume and attach it to the container specified. This process is known as Volume Mounting.
If the volume is already available elsewhere and data needs to be stored in it, a complete path to the mount folder is specified, for instance:
$ docker run -e MYSQL_ROOT_PASSWORD=root -v /data/mysql:/var/lib/mysql mysql
This creates a new volume
mysql on the
/data folder that reads and persists data from the Read/Write layer of the
mysql container. This process is known as Bind Mounting.
--mount flag is preferred to the
-v flag since it is explicit and more verbose. The
--mount flag allows users to specify more options using comma-separated key-value pairs. Such options include:
destination (destination/dst/target), the
readonly option and
volume-opt options, which can include more than one key-value pair.
Storage drivers enable and manage layered architecture in Docker. Some popular storage drivers include AUFS, BTRS, Overlay1, Overlay2, DeviceMapper and ZFS. Storage Drivers are OS-specific. AUFS, for instance, is the default storage driver for Ubuntu but it may not work well with CentOS or Fedora. Docker automatically selects the appropriate storage driver based on the Operating System running in a container.
Volume Driver Plugins
Storage drivers only manage the Docker layered architecture, but not the volumes themselves. Volume drivers help connect containers to volumes and manage persistent data.
local is Docker’s default volume driver, and it manages the
/var/lib/docker directory. Other volume driver plugins exist, including the Local Persistent Volume plugin which helps Docker create standalone volumes that can be bound to different directories in multiple containers.
Other Volume Driver plugins include BeeGFS Volume Plugin, Contiv, DigitalOcean Block Storage Plugin, Flocke, Fluxi, Horcrux, Netshare, OpenStorage and REX-Ray among others. Each plugin supports different storage solutions from various cloud vendors. REX-Ray, for instance, provides persistent storage for plenty of platforms such as Amazon EC2, OpenStack, VirtualBox, Google Compute Engine and EMC among others.
The command below, for instance, can be used to bind a container to a volume on Amazon’s AWS EBS storage service:
$ docker run -it \ --name mysql -e MYSQL_ROOT_PASSWORD=root \ --volume-driver rexray/ebs \ --mount src=ebs-vol,target=/var/lib/mysql mysql
This command mounts a container
mysql to the volume sourced from EBS cloud and managed by REX-Ray’ EBS Plugin
Container Storage Interface
Earlier, Kubernetes only supported Docker as the container runtime engine, so Docker integrations were baked into Kubernetes. As support for other runtimes increased, Kubernetes developed the Container Runtime Interface (CRI) as an API that allows for separation in the Kubernetes Codebase to allow different runtimes to communicate with the Kubelet service running nodes. The CRI standardizes interfaces to coordinate the resources used by container runtime engines. Such interfaces include the Container Networking Interface (CNI) and the Container Storage Interface (CSI).
The Container Storage Interface is a standard interface that connects container workloads with storage platforms by defining how volume drivers are written. This creates a simplified set of specifications that allow various vendors to create storage plugins independent of the Kubernetes development team. The CSI is a global standard, meaning it is supported by most third-party storage vendors and cloud orchestration platforms. This means organizations should not have to worry about vendor or platform lockup.
The CSI defines several Remote Procedure Calls (RPCs) that should be invoked by the orchestrators then implemented by the Volume Drivers on the storage system. For instance, if the orchestrator invokes a call to provision a new volume, then the Volume driver should create a new volume on the storage. If the orchestrator invokes a call to delete a volume, the driver should decommission the volume. The complete list of RPCs is available on the CSI Github directory here.
Containers are ephemeral, that is, they are only built to run as long as the processes they host are active. A container is initiated to perform a specific task then is discarded as soon as it is done. The data processed by the container is also destroyed as soon as it exits. Docker attaches volumes to containers to persist data generated by their Read/Write layers. Any data processed by the container is stored in these volumes, and is available when the container exits.
The ephemeral nature of containers also brings challenges for critical applications running on Kubernetes. This is because when a container crashes, the Kubelet service may restart it but it comes back up with a clean slate. Multiple containers running in one POD may also need to share files. PODs themselves are transient, meaning they terminate as soon as the containers running inside them terminate. Kubernetes volumes create persistent storage for capturing cluster data. Volumes outlive containers and PODs, preserving cluster data even when PODs terminate. Attaching a volume to a POD ensures that cluster data persists even in the event of POD failure.
To understand volumes and data persistence, consider a POD created to generate a random number between 1 and 100 and prints it out to
/opt/number.out whose specifications are shown:
apiVersion: v1 kind: Pod metadata: name: random-number-generator spec: containers: - image: alpine name: alpine command: ["/bin/sh","-c"] args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"]
Once the POD terminates, the random number generated is also deleted. To retain the number in the output, a Volume is created and attached the POD:
volumes: - name: data-volume hostpath: path: /data Type: directory
The volume is then mounted to specific containers in the POD by specifying the directory inside the container, for instance:
volumeMounts: - mountPath: /opt name:data-volume
The complete POD definition file will be similar to:
apiVersion: v1 kind: Pod metadata: name: random-number-generator spec: containers: - image: alpine name: alpine command: ["/bin/sh","-c"] args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"] volumeMounts: - mountPath: /opt name: data-volume volumes: - name: data-volume hostPath: path: /data type: Directory
This means that the volume
data-volume will persist data in the container on the R/W directory
/opt and store it in the
/data directory. The random number output will be available on
/data even if the
random-number-generator POD goes down.
Volume Storage Options
Kubernetes supports multiple storage options implemented as plugins. In the above example, the option used is
hostPath which is great for local storage in single-node clusters. In multi-node clusters, however, the
hostPath option is not suitable since it will assume a similar storage directory on all nodes while they are, in fact, separate servers. Kubernetes also offers support for various third-party storage options. These include AzureDisk, AzureFile, AWS ElasticBlockStore (EBS), flocker, glusterfs, gce Persistent Disk, and vSphereVolume among others. Each option has its own configuration specifications. For instance, to provision an AWS EBS cloud storage option for a Kubernetes POD, the following configuration is used:
volumes: - name: data-volume awsElasticBlockStore: volumeID: <volume-id> fsType: ext4
The volume created in the previous lecture is attached to a specific POD, and its specifications are written within the POD. This may be effective for small clusters, but not in large production environments. With larger workloads, users create volumes every time a POD initiates, and this creates a data management challenge.
A Persistent Volume (PV) is a Kubernetes cluster resource that abstracts storage provision from consumption, so storage can be assigned to multiple users. PVs can be provisioned dynamically using Storage Classes or by cluster administrators so that users can use a percentage of the volume when deploying applications.
The PV is an API resource; a Volume Plugin whose lifecycle is independent of the PODs attached to it, and details how storage is implemented in the cluster. It then creates a shared storage pool out of which users can access a portion using Persistent Volume Claims (PVCs).
A PV can be created by specifying the configurations in a YAML manifest file, as shown:
Each PV gets its own set of access modes describing that specific PV’s capabilities.
The access modes are:
- ReadWriteOnce — the volume can be mounted as read-write by a single node
- ReadOnlyMany — the volume can be mounted read-only by many nodes
- ReadWriteMany — the volume can be mounted as read-write by many nodes
- ReadWriteOncePod — the volume can be mounted as read-write by a single Pod
apiVersion: v1 kind: PersistentVolume metadata: name: pv-vol1 spec: accessModes: - ReadWriteOnce capacity: storage: 1Gi hostPath: path: /tmp/data
This creates a PV named
pv-vol1 with an access mode
ReadWriteOnce and a 1 Gigabyte capacity hosted in the directory
Persistent Volume Claims (PVCs)
The PVC is a request made by users to access a portion of PV storage. It is a namespace object, separate from the PV, that allows end users to consume abstracted storage. Users typically need varying PV specifications for different tasks, and PVCs allow them to specify desired properties such as Volume Mode, Access Mode and Volume Size among others. While a PV is a cluster resource, a PVC is a namespace resource. The cluster administrator provisions storage using PV, while users request for this storage using PVCs.
Once a PVC has been created, Kubernetes binds it to a PV with sufficient capacity and matching other properties mentioned in the PVC. These properties include accessMode, Storage Mode and Volume Mode among others.
If there exists more than one PV that matches the requirements of the PV, Labels and Selectors can be used to bound PVCs to specific PVs. If no other match exists, a PVC may be assigned a PV with much larger storage but with other matching properties. Since the PVCs and PVs share a 1:1 relationship, no other PVCs can join to use the extra space.
A PVC is also created using a YAML definition file, with request specifications similar to the one below:
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: myclaim spec: accessModes: - ReadWriteOnce resources: requests: storage: 500Mi
This PVC can then be bound to a POD so that the POD can access the PV earlier created. This is performed by specifying the claim under in the PODs definition file, as shown:
volumes: - name: data-volume persistentVolumeClaim: claimName: myclaim
When using PVs as explored in the previous chapter, a physical storage disk has to be created on Google Cloud every time a volume is created. This is known as static provisioning. Storage Classes offer dynamic storage provisioning. This is where a provisioner such as GCE Storage or AWS EBS is defined to automatically provision storage on the specific cloud service and attach it to PODs when they make a PVC.
A Storage Class object is crucial since administrators can use them to map quality of storage to certain policies and tags, allowing them to create different ‘profiles’ for users requesting storage. Administrators create different storage classes by stating the name, provisioner and other parameters in a YAML definition file. These parameters vary from one provisioner to another and may include mount options, reclaim policies and volume expansion capability among others. A typical manifest file for a Storage Class provisioning volumes using AWS EBS would look similar to:
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: aws-standard provisioner: kubernetes.io/aws-ebs parameters: type: gp2 reclaimPolicy: Retain allowVolumeExpansion: true
The Storage Class name is then specified in the PVCs manifest under the specification, as shown:
spec: accessModes: ReadOnlyOnce storageClassName: aws-standard resources: requests: storage: 500Mi
The Storage Class automatically creates the PV and any other associated storage as soon as it initiates, eliminating the need to create one for each PVC. Each time a user creates a PVC request, the Storage Class orders the provisioner to create a disk on Google Cloud that fulfils the request. There are plenty of Storage Classes from different providers, including FlexVolume, CephFS, PortworxVolume, GCEPersistentDisk and Cinder among others.
Research Questions & Conclusion
This concludes the storage section of the CKA certification exam. To test your knowledge, it is strongly recommended that you access research questions of all core concepts covered in the coursework and a test to prepare you for the exam. You can also send your feedback to the course developers, whether you have feedback or would like something changed within the course.
Here is a quick quiz with a few questions and sample tasks to help you assess your knowledge. Leave your answers in the comments below and tag us back.
Quick Tip – Questions below may include a mix of DOMC and MCQ types. For sample tasks, we have also provided the right approach/specification along with the question. It is recommended that before you refer to the right answer/approach, you first try on your own and then validate with the right one provided here.
1. Task: Create a Persistent Volume with the given specification.
Volume Name: pv-log
Access Modes: ReadWriteMany
Host Path: /pv/log
Reclaim Policy: Retain
apiVersion: v1 kind: PersistentVolume metadata: name: pv-log spec: persistentVolumeReclaimPolicy: Retain accessModes: - ReadWriteMany capacity: storage: 100Mi hostPath: path: /pv/log
2. Let us claim some of that storage for our application. Create a Persistent Volume Claim with the given specification.
Volume Name: claim-log-1
Storage Request: 50Mi
Access Modes: ReadWriteOnce
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: claim-log-1 spec: accessModes: - ReadWriteOnce resources: requests: storage: 50Mi
3. What would happen to the PV if the PVC was destroyed?
[A] The PV is scrubbed
[B] The PV is made available again
[C] The PV is not deleted but not available
[D] The PV is deleted as well
4. Create a new Storage Class called
delayed-volume-sc that makes use of the below specs:
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: delayed-volume-sc provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer
This section has explored storage concepts and how Kubernetes cluster data is persisted using volumes. The ephemeral nature of Docker Containers and Kubernetes PODs necessitates special attention for storage. The concepts of ephemeral volumes, persistent volumes and Storage Classes have been explored in-depth, with accompanying labs to help the candidate get confident with storage management in Kubernetes both for the CKA exam and for production environments.
More details about KodeKloud’s CKA course with access to the lessons, labs, mock exams and demo can be found here – https://kodekloud.com/courses/certified-kubernetes-administrator-cka/.