ETCD backup/restore with peer.key and peer.crt

[thump]

I got the question about ssh’ing into node, and perform back up and restore of etcd with peer.key and peer.crt. If I passed those in when backing up, I received “no such file or directory” error on the peer.crt/peer.key paths. Any idea what to do here?

I also tried server.crt and server.key, but all I got was “permission denied”, so I suspect I have to authenticate with the peer flags somehow(?) These peer flags were also not present when checking the etcd pod’s commands. I do see in the docs, under Securing Communication that there are some peer flags for configuring etcd with secure peer comm., but these flags were unknown to the etcdctl tool

Thanks!

[camelel]

The way I always know which cert and key to use is by running:
grep etcd /etc/kubernetes/manifests/kube-apiserver.yaml

Which returns the correct cert, key, ca and endpoints to use when running the etcdctl command:

controlplane ~ ➜ grep etcd /etc/kubernetes/manifests/kube-apiserver.yaml
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- --etcd-servers=https://127.0.0.1:2379

Then using the above information, run the etcdctl command to create the snapshot:

controlplane ~ ➜ etcdctl --cacert=“/etc/kubernetes/pki/etcd/ca.crt” --cert=“/etc/kubernetes/pki/apiserver-etcd-client.crt” --key=“/etc/kubernetes/pki/apiserver-etcd-client.key” --endpoints=127.0.0.1:2379 sn
apshot save /opt/snapshot.db
{“level”:“info”,“ts”:“2024-07-23T15:14:20.996102Z”,“caller”:“snapshot/v3_snapshot.go:65”,“msg”:“created temporary db file”,“path”:“/opt/snapshot.db.part”}
{“level”:“info”,“ts”:“2024-07-23T15:14:21.003448Z”,“logger”:“client”,“caller”:“[email protected]/maintenance.go:212”,“msg”:“opened snapshot stream; downloading”}
{“level”:“info”,“ts”:“2024-07-23T15:14:21.003494Z”,“caller”:“snapshot/v3_snapshot.go:73”,“msg”:“fetching snapshot”,“endpoint”:“127.0.0.1:2379”}
{“level”:“info”,“ts”:“2024-07-23T15:14:21.015910Z”,“logger”:“client”,“caller”:“[email protected]/maintenance.go:220”,“msg”:“completed snapshot read; closing”}
{“level”:“info”,“ts”:“2024-07-23T15:14:21.019437Z”,“caller”:“snapshot/v3_snapshot.go:88”,“msg”:“fetched snapshot”,“endpoint”:“127.0.0.1:2379”,“size”:“1.3 MB”,“took”:“now”}
{“level”:“info”,“ts”:“2024-07-23T15:14:21.019690Z”,“caller”:“snapshot/v3_snapshot.go:97”,“msg”:“saved”,“path”:“/opt/snapshot.db”}
Snapshot saved at /opt/snapshot.db

When you want to just restore, make sure the file you want to restore from resides on the same server where you intend to run the etcdctl snapshot restore command, and then just run:
etcdctl snapshot restore /absolute/path/to/snapshot.file

Attached an image.

Itai

[thump]

This will always do the trick? Even though exam explicitly specifies to use the peer.crt and peer.key? I think I should be able to use those peer ones, all info I can find points to it. I believe I am doing something wrong since I get those “no such file or dir” errors:/

[camelel]

If the question explicitly tells you to use the peer crt/key then use those.
But the “no such file or directory” error is pretty much self explanatory, the file you are looking for doesn’t exist in the path you provided.

I would go to /etc/kubernetes/pki/etcd and list the files there to verify the correct file names.

Try the following command:

etcdctl --cacert=“/etc/kubernetes/pki/etcd/ca.crt” --cert=“/etc/kubernetes/pki/etcd/peer.crt” --key=“/etc/kubernetes/pki/etcd/peer.key” --endpoints=127.0.0.1:2379 snapshot save /opt/snapshot.db

And let me know if it worked.