Question on etcd and journalctl logs

uzmashafi061 · June 29, 2023, 12:57pm

• kubectl config use-context cluster4

The controlplane node called cluster4-controlplane in the cluster4 cluster is planned for a regular maintenance. In preparation for this maintenance work, we need to take backups of this cluster. However, something is broken at the moment!

Troubleshoot the issues and take a snapshot of the ETCD database using the etcdctl utility at the location /opt/etcd-boot-cka18-trb.db.

Note: Make sure etcd listens at its default port. Also you can SSH to the cluster4-controlplane host using the ssh cluster4-controlplane command from the student-node.
Troubleshooting:
On running the command k get pod -A below error was encountered
The connection totheserver cluster4-controlplane:6443was refused - did you specify therighthost orport?
In soln the command journalctl -u kubelet -f is used, so my question is why are we checking kubelet logs(as we have issue on etcd)? And how to know what logs we need to focus on as there are lots of logs?

mmkmou · June 30, 2023, 12:03am

Hi @uzmashafi061

The issue is not directly linked to the ETCD but to the cluster himself. Please see below a guide how to troubleshooting.

github.com

kodekloudhub/community-faq/blob/main/docs/diagnose-crashed-apiserver.md

# How to Diagnose a Crashed API Server

The API server pod won't come back up - HELP :scream: :scream: :scream:

Perhaps you've made a manifest edit, or perhaps some question has put you into a context where the API server is already broken. You're using `docker ps` or `crictl ps` and see the API server flash up briefly then go away. The container doesn't last long enough for you to grab an ID to pull logs from. Or maybe it never appears in the `ps` output.

Note that these techniques can be used for the other static pods like `etcd` by looking for `etcd` instead of `apiserver` in the commands below.

Steps to take

1.  Restart `kubelet` so you don't have to wait too long in the following steps</br></br>
    ```
    systemctl restart kubelet
    ```

1.  Determine if the kubelet can even start the API server</br></br>
    If there is a syntax error in the YAML manifest, then kubelet will not be able to parse it and will eventually complain. Do the following and watch the output for up to 60 seconds. Note that if you have errors in your pod manifest, kubelet will report them exactly the same way using the [same kind of error messages](./yaml-faq.md#dealing-with-errors) that kubectl does!

    ```
    journalctl -fu kubelet | grep apiserver

This file has been truncated. show original