ETCD Backup and Restore Issue

Kubernetes
#1

saip:
Seems like with etcdctl 3.4.3 there is no need to specify ETCDCTL_API=3 parameter ? I instantiated a 1.18 cluster with kubeadm and the etcd pod does not accept any of the commands that starts with ETCDCTL_API=3 ?

#2

saip:
k8s-master:~$ kubectl -n kube-system exec etcd-k8s-master – etcdctl version
etcdctl version: 3.4.3
API version: 3.4
k8s-master:~$ kubectl -n kube-system exec -it etcd-k8s-master – ETCDCTL_API=3 etcdctl --help
OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused “exec: \“ETCDCTL_API=3\“: executable file not found in $PATH”: unknown
command terminated with exit code 126
k8s-master:~$

#3

saip:
Any thoughts ?

#4

Kratik Jain:
It’s not like you don’t need to specify ETCDCTL_API=3
Afaik and Whatever the examples I have seen It needs to be declared,
One of basic difference(The command to check version)
in v2
etcdctl --version

in v3
etcdctl version

The issue in your command is not anywhere related to k8s - I don’t know what exactly it is termed(maybe other experts can let us know!) but it is how you input your commands to the underlying container (after --)
I guess as your command includes = and -- (in --help) which are leading your command to not get passed properly.

to pass your command properly try this

kubectl -n kube-system exec -it etcd-k8s-master -- sh -c "ETCDCTL_API=3 etcdctl --help"

this will pass your all commands and args properly

#5

saip:
I always use just “–” and after that the command itself… Dont think we need “-it” here cause I dont need to attach to the terminal of the pod.

#6

saip:
As you can see in the first command in my output above it worked ok.

#7

Kratik Jain:
I am not saying anything about -it ,
Just use sh -c "WHOLE COMMAND" and you’ll be good to go

#8

saip:
Yeah I get you.

#9

saip:
Got rid of the previous messages that I put here for simplicity. I see what you mean about the usage of “=” in the command.

#10

Kratik Jain:
The -c argument is: Read commands from the command_string operand instead of from the standard input.

#11

saip:
Well the bit which really got me puzzled here is the whole backup works when I do this : kubectl -n kube-system exec etcd-k8s-master – etcdctl --cacert=“/etc/kubernetes/pki/etcd/ca.crt” --cert=“/etc/kubernetes/pki/etcd/server.crt” --key=“/etc/kubernetes/pki/etcd/server.key” --endpoints=127.0.0.1:2379 snapshot save /opt/test.db

#12

saip:
That is why I said if etcd has changed with 3.4. I will rephrase my question for acrhiving purposes.

#13

saip:
@Kratik Jain I tested it again and above command works like charm, at least in kubeadm based setup where etcd is a pod. So even without ETCDCTL_API=3 parameter it is always able to save the backup.

#14

saip:
@Rahul Soni are you aware of any changes with etcd 3.4 ?

#15

saip:
Well I found the answer, it has changed. API version is 3 by default with ETCD 3.4.0 : https://chromium.googlesource.com/external/github.com/coreos/etcd/+/refs/heads/master/CHANGELOG-3.4.md#breaking-changes

#16

saip:
@Mumshad Mannambeth @Rahul Soni It would be great if this can be updated in the training for new people. ^^

#17

Kratik Jain:
@saip can you try etcdctl version command as you tried above command,
If it’s ETCDCTL_API=3 , it will print the version otheriwise it will throw error.

#18

saip:
I did that already, it is in my first message ^^^

#19

saip:
Still pasting it here :

#20

saip:
k8s-master:~$ kubectl -n kube-system exec etcd-k8s-master – etcdctl version
etcdctl version: 3.4.3
API version: 3.4