Question on *etcd snapshot restore*: In lecture it is said to stop kube-apiserve . . .

Kubernetes
#1

Tanumoy Ghosh:
Question on etcd snapshot restore: In lecture it is said to stop kube-apiserver before restoring etcd, but getting below error

root@controlplane:~# kubectl -n kube-system get po | grep api
kube-apiserver-controlplane            1/1     Running   0          4m31s

root@controlplane:~# sudo service kube-apiserver stop
Failed to stop kube-apiserver.service: Unit kube-apiserver.service not loaded.

If I try restore without stopping apiserver, restore command went through but nothing comes up

Same error if I try restarting daemon-reload and etcd after executing etcd restore command

systemctl restart daemon-reload
service etcd restart
#2

Fernando Jimenez:
Hi @Tanumoy Ghosh kube-apiserver-controlplane is a static pod running in the controlplane “host”. Unfortunately, sudo service kube-apiserver stop will not work since that service is not running at the host level.
By the way, in case you would care to know, it is preferable sudo systemctl stop kube-apiserver

#3

Tanumoy Ghosh:
So - I could directly run the etcd snapshot restore option and resources should come up after that, right ?

I am not using --data-dir option in restore option as it has not asked explicitly in the question, or do I need to do that always ?

#4

Hinodeya:
@Tanumoy Ghosh as noted during the exam propably you need to chek if the etcd running as an daemon :wink: systemctl status etcd

#5

Tanumoy Ghosh:
I will trouble both of you a bit more to share your experience - Is it mandatory to always restore from etcd backup in a different --data-dir than the default one /var/lib/etcd

Approach 1: As question didn’t ask to restore in a different --data-dir and <Operating etcd clusters for Kubernetes | Kubernetes documentation> doesn’t have specific mention to use --data-dir while restoring, hence I ran

# ETCDCTL_API=3 etcdctl --endpoints=<https://127.0.0.1:2379> --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot restore /opt/snapshot-pre-boot.db

Nothing got restored actually

Approach 2: Tried restoring using --data-dir=/var/lib/etcd-from-backup and yes the things came up as expected
But for this, I have to change the path of volume etcd-data in static etcd pod yaml to /var/lib/etcd-from-backup , after changing it; the etcd pod remain in pending state without showing any events for the reason being pending

#6

Fernando Jimenez:
Approach 2 can not properly work in the lab/test in kodekloud because the controlplane is not an actual host but rather another pod with kubectl access and some volume mounts. Which it means if you restore to etc-from-backup it is in the jailed controlplane and when you change the etcd manifest, you are telling it a location that it doesn’t exist in the “real” host.

#7

Dear Kodekloud support, I am also facing the same problem during the etcd restore. Even the instructor contradicting his explanation with the solution. In the solution he hasn’t restsrted any kube-apiserver but while explaining he said that you have to stop the kube-apiserver but unfortunately we can’t stop the kube-apiserver because when we issue kube-apiserver stop command we are getting error saying service not found.

Now kindly explain bit more clearly when to stop kube-apiserver and when its not required to restart kube-apiserver. Also what needs to be done during the exam.