Hi All I have tried to to backup etcd using snapshot with below command `k -n ku . . .

Kubernetes-Slack Discussions
#1

Kalesha Gagguturu:
Hi All
I have tried to to backup etcd using snapshot with below command
k -n kube-system exec -it etcd-kubemaster -- etcdctl snapshot save --endpoints=192.168.145.134:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key ./etcd-snapshot-11122022.db
where will be the etcd-snapshot-11122022.db will be located?

#2

Trung Tran:
in this relative path: ./etcd-snapshot-11122022.db
based on where you run this command, just try to execute it and confirm!

#3

Kalesha Gagguturu:

{"level":"info","ts":"2022-12-11T10:46:07.756Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"./etcd-snapshot-11122022.db.part"}
{"level":"info","ts":"2022-12-11T10:46:07.802Z","logger":"client","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2022-12-11T10:46:07.802Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"192.168.145.134:2379"}
{"level":"info","ts":"2022-12-11T10:46:07.891Z","logger":"client","caller":"v3/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2022-12-11T10:46:07.912Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"192.168.145.134:2379","size":"9.6 MB","took":"now"}
{"level":"info","ts":"2022-12-11T10:46:07.912Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"./etcd-snapshot-11122022.db"}
Snapshot saved at ./etcd-snapshot-11122022.db
#4

Kalesha Gagguturu:
Its from my home directory of kubemaster

#5

Kalesha Gagguturu:
but I am not able to find this snapshot file

#6

Kalesha Gagguturu:

gkalesha@kubemaster[Sun Dec 11][04:26 PM]:~$pwd
/home/gkalesha
gkalesha@kubemaster[Sun Dec 11][04:27 PM]:~$ls
gkalesha@kubemaster[Sun Dec 11][04:27 PM]:~$
#7

squeeky mouse:
Hmm… run
k -n kube-system exec -it etcd-kubemaster – pwd
that would give you the folder where the etcdctl cmd was run from

#8

squeeky mouse:
Then look in the folder on kubemaster

#9

Kalesha Gagguturu:
I tried that squeeky but it says executables not found

#10

Kalesha Gagguturu:

gkalesha@kubemaster[Sun Dec 11][06:58 PM]:~$k -n kube-system exec -it etcd-kubemaster -- pwd
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "b1b9085f6b81e0e2e99fc175b51efb8db630e2fdac46e66a4bd18c71aa3f71d2": OCI runtime exec failed: exec failed: unable to start container process: exec: "pwd": executable file not found in $PATH: unknown
#11

squeeky mouse:
Just ssh to the master node and run the etcdctl cmd.

#12

Kalesha Gagguturu:
From master node etcdctl commands not running because etcd is setup with kubeadm as pod

#13

squeeky mouse:
Try
ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
–cacert=/etc/kubernetes/pki/etcd/ca.crt member list

from the controlplane node

#14

Kalesha Gagguturu:
Same output etcdctl command not found on controlplane

#15

Kalesha Gagguturu:
I think I found the issue
there is no etcd client in control plane, thats why we are not able to run etcdctl commands

#16

Kalesha Gagguturu:

gkalesha@kubemaster[Sun Dec 11][11:01 PM]:~$etcdctl --endpoints=127.0.0.1:2379 snapshot save etcd-backup-11122022.db --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
Snapshot saved at etcd-backup-11122022.db
gkalesha@kubemaster[Sun Dec 11][11:01 PM]:~$ls
etcd-backup-11122022.db

So after installing etcd client I was able to take etcd backup

gkalesha@kubemaster[Sun Dec 11][11:05 PM]:~$etcdctl --endpoints=127.0.0.1:2379 snapshot restore --data-dir /home/gkalesha/etcd-restore /home/gkalesha/etcd-backup/etcd-backup-11122022.db
2022-12-11 23:06:10.531450 I | mvcc: restore compact to 142327
2022-12-11 23:06:10.547913 I | etcdserver/membership: added member 8e9e05c52164694d [<http://localhost:2380>] to cluster cdf818194e3a8c32
#17

mjv:
When you are on control plane you can check PID of etcd container (ps faxu | grep etcd).With that info you can fetch etcdctl binary as
find /proc/${PID}/root/ -type f -name 'etcdctl'
Then just copy path from above cmd to /usr/local/bin and you can it.

#18

mjv:
Container root filesytem will be /proc/${PID}/root so you can copy anything from the container to the host filesystem.

[controlplane] $ sudo ps faxu | grep etcd
cloud_u+    5200  0.0  0.0   8168   720 pts/0    S+   08:23   0:00              \_ grep --color=auto etcd
root        1617 14.0  4.7 1112296 382856 ?      Ssl  08:22   0:15  \_ kube-apiserver --advertise-address=172.31.99.188 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=<https://127.0.0.1:2379> --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=<https://kubernetes.default.svc.cluster.local> --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.112.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root        1675  4.3  0.8 11214784 68792 ?      Ssl  08:22   0:04  \_ etcd --advertise-client-urls=<https://172.31.99.188:2379> --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --experimental-initial-corrupt-check=true --experimental-watch-progress-notify-interval=5s --initial-advertise-peer-urls=<https://172.31.99.188:2380> --initial-cluster=controlplane=<https://172.31.99.188:2380> --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=<https://127.0.0.1:2379>,<https://172.31.99.188:2379> --listen-metrics-urls=<http://127.0.0.1:2381> --listen-peer-urls=<https://172.31.99.188:2380> --name=controlplane --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

[controlplane] $ sudo find /proc/1675/root/ -type f -name "etcdctl"
/proc/1675/root/usr/local/bin/etcdctl
#19

Kalesha Gagguturu:
Thank you @mjv
It’s really helpful for me.
I will check this process