Hello all, While trying to take snapshot of etcd cluster,The directory of the p . . .

Kubernetes-Slack Discussions
#1

Annamalai Muthu:
Hello all,
While trying to take snapshot of etcd cluster,The directory of the pod “/etc/kubernetes/manifests/” doesn’t exists… What is the reason?? Do we need to create directory for /etc/kubernetes/manifests???

#2

Tanuja SM:
How Kubernetes is installed? If it is installed using kubeadm way. The etcd is installed as static pod. The default path for the static pods is /etc/kubernetes/manifests. Under this path the manifest files for kube-apiserver, etcd, kube-scheduler and kube-controller-manager will be presesnt.

root@controlplane:/etc/kubernetes/manifests# ls
etcd.yaml kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml

#3

Tanuja SM:
Here is the procedure

ETCD back-up:

• Take the backup:

ETCDCTL_API=3 etcdctl
–endpoints=https://127.0.0.1:2379
–cacert=/etc/kubernetes/pki/etcd/ca.crt
–cert=/etc/kubernetes/pki/etcd/server.crt
–key=/etc/kubernetes/pki/etcd/server.key
snapshot save /opt/snapshot-pre-boot.db

• Check the status of the backup:

ETCDCTL_API=3 etcdctl snapshot status /opt/snapshot-pre-boot.db

• Stop the API server instances

root@controlplane:~# cd /etc/kubernetes/manifests/
root@controlplane:/etc/kubernetes/manifests# ls
etcd.yaml kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml

root@controlplane:/etc/kubernetes/manifests# mv kube-apiserver.yaml …

root@controlplane:/etc/kubernetes/manifests# ls
etcd.yaml kube-controller-manager.yaml kube-scheduler.yaml

• Restore the backup:

root@controlplane:~# ETCDCTL_API=3 etcdctl --data-dir=/var/lib/etcd-from-backup snapshot restore /opt/snapshot-pre-boot.db

• Edit the volume mount in etcd.yaml

root@controlplane:/etc/kubernetes/manifests# vi etcd.yaml

volumes:

  • hostPath:
    path: /etc/kubernetes/pki/etcd
    type: DirectoryOrCreate
    name: etcd-certs
  • hostPath:
    path: /var/lib/etcd-from-backup
    type: DirectoryOrCreate
    name: etcd-data

• Reload system configs

$ systemctl daemon-reload

• Stop and start etcd:

root@controlplane:/etc/kubernetes/manifests# mv etcd.yaml …
root@controlplane:/etc/kubernetes/manifests# kubectl get po
^C
root@controlplane:/etc/kubernetes/manifests# mv …/etcd.yaml .

• Start API server:

root@controlplane:/etc/kubernetes/manifests#mv …/kube-apiserver.yaml .

#4

Aneek Bera:
ETCDCTL_API=3 etcdctl --cacert=“/etc/kubernetes/pki/etcd/ca.crt” --cert=“/etc/kubernetes/pki/etcd/server.crt” --key=“/etc/kubernetes/pki/etcd/server.key” snapshot save /opt/snapshot-pre-boot.db

#5

Aneek Bera:
Restoring from backup:
ETCDCTL_API=3 etcdctl snapshot restore /opt/snapshot-pre-boot.db --data-dir=/var/lib/etcd-from-backup

root@controlplane:~# ls /var/lib/etcd
member
root@controlplane:~# ls /var/lib/etcd-from-backup
member
root@controlplane:~#

Note: ‘etcd-from-backup’ is the place from where the snapshot has been restored to.
But, now, etcd cluster is still configured to connect to /var/lib/etcd. So, we need to change this to new directory in etch yaml file.
Here are the steps:
root@controlplane:~# cd /etc/kubernetes/manifests/
root@controlplane:/etc/kubernetes/manifests# ll
total 28
drwxr-xr-x 1 root root 4096 Sep 20 21:22 ./
drwxr-xr-x 1 root root 4096 Sep 20 20:33 …/
-rw------- 1 root root 2183 Sep 20 20:33 etcd.yaml
-rw------- 1 root root 3807 Sep 20 20:33 kube-apiserver.yaml
-rw------- 1 root root 3314 Sep 20 20:33 kube-controller-manager.yaml
-rw------- 1 root root 1384 Sep 20 20:33 kube-scheduler.yaml
root@controlplane:/etc/kubernetes/manifests# vim etcd.yaml

Come to the bottom of the yaml file and change the host path:
From:

  • hostPath:
    path: /var/lib/etcd
    type: DirectoryOrCreate
    name: etcd-data

To:

  • hostPath:
    path: /var/lib/etcd-from-backup
    type: DirectoryOrCreate
    name: etcd-data
#6

Annamalai Muthu:
Thanks for your response @Aneek Bera @Tanuja SM :pray::pray:

#7

Annamalai Muthu:
@Tanuja SM Why do we need to stop the API servers??? Kindly let me know… :pray: