Ryebridge:
With regard to ETCD, I’m hearing conflicting things about restoring. If for example, you are asked to restore from the original terminal without context change with a file location, does this mean just run the usual restore command like this:
sudo ETCDCTL_API=3 etcdctl --data-dir /opt/newBAK snapshot restore /data/backup/etcd-snapshot-March.db
….or should we run the command, copy the backup data dir to the master node then update the etcd yaml with the new path ? Udemy, Killer.sh all have etcd server on master but could be a possibility that ETCD is on another node.
Vitor Jr.:
Actually I don’t think the way you do it matters. What it matters it is they will ask you to do something and this kind of test is called “desired state”
Vitor Jr.:
So, for this question, as an example, they want the etcd restored. The way you do it doesn’t matters (unless they ask you to record the commands or something like that)
Vitor Jr.:
If you do the restore in the master directly, in the terminal01 and then copied the files it’s your choice
Vitor Jr.:
What matters is the backup restored at the final of the task
Ryebridge:
Thanks @Vitor Jr. Was thinking if ETCD wasn’t installed on the master then we’d have to run the restore on the original terminal, copy the files to master then update etcd with the new path. Maybe ETCD could be located on base node itself, could be a possibility
Vitor Jr.:
So, thinking in a pragmatic way: doesn’t matter where is. If they ask you to do a restore, they will provide where it is located. All you have to do is think about the restore strategy. I try to make everything simple as possible. So, my strategy would be run all commands where the etcd is located, if possible.
Vitor Jr.:
Don’t get me wrong. What I am trying to say is: could be more than one way to achieve this. What it matters, it’s how you analyze, understand, approach and fix the problem.
Vitor Jr.:
This scenario would be about etcd disaster recover but also how do you react, plan and execute this task
Ryebridge:
Yeah, will run all commands where etcd is located then copy files to master backup dir before updating etcd yaml with new file location (safer way would be to move all manifests out then update etcd yaml with new path and move back in again).
Ryebridge:
Actually, I think this could also be run from original terminal so no need to copy or update etcd yaml:
ETCDCTL_API=3 etcdctl --endpoints="<https://127.0.0.1:2379>" --cacert=/opt/certdir/ca.crt --cert=/opt/certdir/etcd-client.crt --key=/opt/certdir/etcd-client.key snapshot restore /var/lib/backup/etcd-snapshot-March.db
Vitor Jr.:
@Ryebridge do you get it that this command you restore those files locally?
Vitor Jr.:
Example: I have an etcd server and then I execute this command from my terminal with etcdctl client. Those files would be restored to my machine and I would have to transfer they to the etcd server host
Vitor Jr.:
Try this in a lab with virtual box or vagrant to check for sure.
Ryebridge:
Theres a few different ways to restore (udemy, killer and docs). I was looking at this option from the docs:
https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#built-in-snapshot
ETCDCTL_API=3 etcdctl --endpoints 10.2.0.9:2379 snapshot restore snapshotdb
…but I just tried it like this on the lab and it doesn’t actually work.
ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt snapshot save /opt/snapshot-pre-boot.db
ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt snapshot restore /opt/snapshot-pre-boot.db
Ryebridge:
I assumed by using the --endpoints option for the restore that it would update the ETCD db directly so no need to update the etcd YAML after running that restore command.
Vitor Jr.:
If I am right the endpoint it’s only to identify where you want to connect
Vitor Jr.:
I think it’s like a database connection
Vitor Jr.:
If I choose to run a dump or restore from my machine, even connecting to a remote database, the physical operation (read/write) files will happen on my machine and not in the server
Ryebridge:
Looks like no option but to restore as per Kodekloud/killer.sh which means having to manually copy the snapshot data to a dir on the master host and update the etcd yaml - I’ve seen a few people use the above way so thought there may have been a faster way to do it without leaving node01.