Hi… again issues with the lab etcd restore/backup, in this case from a k8s vagra . . .

Kubernetes-Slack Discussions
#1

Miguel Gonzalez:
Hi… again issues with the lab etcd restore/backup, in this case from a k8s vagran env.
Backup process:

root@kubemaster:~# ETCDCTL_API=3 etcdctl --endpoints=<https://127.0.0.1:2379> --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key  snapshot save /opt/snapshot-pre-boot.db

Backup status:

root@kubemaster:~# ETCDCTL_API=3 etcdctl -w table snapshot status /root/snapshot-pre-boot.db
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| b4338c5c |    67213 |       1373 |     3.5 MB |
+----------+----------+------------+------------+

Restore backup:

root@kubemaster:~# ETCDCTL_API=3 etcdctl  snapshot restore /opt/snapshot-pre-boot.db --data-dir /var/lib/etcd-from-backup --skip-hash-check
2021-10-31 18:08:53.754463 I | mvcc: restore compact to 66232
2021-10-31 18:08:53.771087 I | etcdserver/membership: added member 8e9e05c52164694d [<http://localhost:2380>] to cluster cdf818194e3a8c32

etcd.yaml file vi /etc/kubernetes/manifests/etcd.yaml:

  volumes:
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs
  - hostPath:
      path: /var/lib/etcd-from-backup ####### NEW PATH
      type: DirectoryOrCreate
    name: etcd-data

I restarted kubelet, docker and daemon-reload

systemctl daemon-reload
systemctl restart docker.service
systemctl restart kubelet.service

Status of docker:

root@kubemaster:~# docker ps -a | grep etcd
d34ac395f4b2   004811815584           "etcd --advertise-cl…"   About a minute ago   Exited (2) About a minute ago             k8s_etcd_etcd-kubemaster_kube-system_2935d46a3487e2c476c3b069b5757be7_8
563c7438b99e   <http://k8s.gcr.io/pause:3.5|k8s.gcr.io/pause:3.5>   "/pause"                 4 minutes ago        Up 4 minutes                              k8s_POD_etcd-kubemaster_kube-system_2935d46a3487e2c476c3b069b5757be7_2

And my kubernetes is died:

vagrant@kubemaster:~$ k get po  -A
The connection to the server 192.168.56.2:6443 was refused - did you specify the right host or port?

Any idea please? I would say my issue is with the paths for volumes

#2

John C:
I think you have to update the dir for etcd to the new path of the export, so you have to edit the /etc/k8s/manifests/etcd.conf and there is a variable called something like dir path and put there the new path of the exported backup ==>/opt/snapshot-pre-boot.db

#3

John C:
so you have to restore with the new path not the one from here

root@kubemaster:~# ETCDCTL_API=3 etcdctl  snapshot restore /opt/snapshot-pre-boot.db --data-dir $NEW_PATH
#4

John C:

root@kubemaster:~# ETCDCTL_API=3 etcdctl  snapshot restore /opt/snapshot-pre-boot.db --data-dir /root/snapshot-pre-boot.db
#5

Miguel Gonzalez:
I don’t think so… the backup was restore with the new path:

ETCDCTL_API=3 etcdctl  snapshot restore /opt/snapshot-pre-boot.db --data-dir /var/lib/etcd-from-backup --skip-hash-check
#6

Miguel Gonzalez:
docker container is rebuilding all the time and with this error:

m-reader-type":"stream MsgApp v2","local-member-id":"8e9e05c52164694d","remote-peer-id":"3ec8d20e7e3a97b3"}
{"level":"info","ts":"2021-10-31T18:36:19.932Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"8e9e05c52164694d","remote-peer-id":"3ec8d20e7e3a97b3"}
{"level":"info","ts":"2021-10-31T18:36:19.936Z","caller":"etcdserver/server.go:728","msg":"started as single-node; fast-forwarding election ticks","local-member-id":"8e9e05c52164694d","forward-ticks":9,"forward-duration":"900ms","election-ticks":10,"election-timeout":"1s"}
{"level":"panic","ts":"2021-10-31T18:36:19.938Z","caller":"membership/cluster.go:453","msg":"failed to update; member unknown","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","unknown-remote-peer-id":"8e9e05c52164694d","stacktrace":"<http://go.etcd.io/etcd/server/v3/etcdserver/api/membership.(*RaftCluster).UpdateAttributes\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/api/membership/cluster.go:453\ngo.etcd.io/etcd/server/v3/etcdserver.(*applierV2store).Put\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/apply_v2.go:92\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyV2Request\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/apply_v2.go:135\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntryNormal\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:2184\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:2116\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:1357\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:1179\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func8\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:1111\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/pkg/schedule/schedule.go:157|go.etcd.io/etcd/server/v3/etcdserver/api/membership.(*RaftCluster).UpdateAttributes\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/api/membership/cluster.go:453\ngo.etcd.io/etcd/server/v3/etcdserver.(*applierV2store).Put\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/apply_v2.go:92\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyV2Request\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/apply_v2.go:135\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntryNormal\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:2184\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:2116\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:1357\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:1179\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func8\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:1111\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/pkg/schedule/schedule.go:157>"}
panic: failed to update; member unknown

goroutine 167 [running]:
<http://go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000184900|go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000184900>, 0xc000185440, 0x3, 0x3)
	/home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global/pkg/mod/go.uber.org/[email protected]/zapcore/entry.go:234 +0x58d
<http://go.uber.org/zap.(*Logger).Panic(0xc000488410|go.uber.org/zap.(*Logger).Panic(0xc000488410>, 0x1221073, 0x20, 0xc000185440, 0x3, 0x3)
	/home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global/pkg/mod/go.uber.org/[email protected]/logger.go:227 +0x85
<http://go.etcd.io/etcd/server/v3/etcdserver/api/membership.(*RaftCluster).UpdateAttributes(0xc0000b81e0|go.etcd.io/etcd/server/v3/etcdserver/api/membership.(*RaftCluster).UpdateAttributes(0xc0000b81e0>, 0x8e9e05c52164694d, 0xc0004c3770, 0xa, 0xc00058f600, 0x1, 0x4, 0x1)
	/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/api/membership/cluster.go:453 +0x951
<http://go.etcd.io/etcd/server/v3/etcdserver.(*applierV2store).Put(0xc0000c8200|go.etcd.io/etcd/server/v3/etcdserver.(*applierV2store).Put(0xc0000c8200>, 0xc000185380, 0x1a7f601, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/apply_v2.go:92 +0x972
<http://go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyV2Request(0xc000394000|go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyV2Request(0xc000394000>, 0xc000185380, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/apply_v2.go:135 +0x277
<http://go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntryNormal(0xc000394000|go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntryNormal(0xc000394000>, 0xc000dbd538)
	/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:2184 +0x511
<http://go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply(0xc000394000|go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply(0xc000394000>, 0xc0000c2030, 0x2, 0x2, 0xc000159100, 0x0, 0x0, 0x0)
	/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:2116 +0x8ed
<http://go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries(0xc000394000|go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries(0xc000394000>, 0xc000159100, 0xc00042cb00)
	/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:1357 +0xe5
<http://go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll(0xc000394000|go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll(0xc000394000>, 0xc000159100, 0xc00042cb00)
	/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:1179 +0x88
<http://go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func8(0x1399c10|go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func8(0x1399c10>, 0xc00058f500)
	/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdserver/server.go:1111 +0x3c
<http://go.etcd.io/etcd/pkg/v3/schedule.(*fifo).run(0xc000412120)|go.etcd.io/etcd/pkg/v3/schedule.(*fifo).run(0xc000412120)>
	/tmp/etcd-release-3.5.0/etcd/release/etcd/pkg/schedule/schedule.go:157 +0xf3
created by <http://go.etcd.io/etcd/pkg/v3/schedule.NewFIFOScheduler|go.etcd.io/etcd/pkg/v3/schedule.NewFIFOScheduler>
	/tmp/etcd-release-3.5.0/etcd/release/etcd/pkg/schedule/schedule.go:70 +0x13b
#7

Miguel Gonzalez:
More information:

root@kubemaster:~# etcdctl -v
etcdctl version: 3.2.17
API version: 2
#8

Miguel Gonzalez:
well… fixed….

#9

Miguel Gonzalez:
the problem was the version, link rerence: https://githubmemory.com/repo/etcd-io/etcd/issues/13119

#10

Miguel Gonzalez:
I updated it https://github.com/etcd-io/etcd/releases

#11

Miguel Gonzalez:
and now it was restored without any problem

#12

Miguel Gonzalez:
@unnivkn qq… in the exam… etcd would be with the right version?

#13

unnivkn:
@Miguel Gonzalez I believe it will be the latest version. Also If you would like to know, I suggest you to submit a ticket with support to get the right version.