[kubernetes-the-hard-way] kube-apiserver service cannot start

Kubernetes
#1

Hello,

I followed the steps in kubernetes-the-hard-way to create a k8s lab.
Update 08-bootstrapping-kubernetes-controllers.md
In Bootstrapping the Kubernetes Control Plane, after creating the script for kube-apiserver kube-controller-manager kube-scheduler scripts, then start the 3 services, the kube-apiserver cannot start.
From the error messages, it said encryption key issue, I try to run echo $ENCRYPTION_KEY that created in step 6 have the result, seems not the key get corrupted.
Did anyone hit the same issue? How do I fix it? Thank you.

vagrant@controlplane01:/etc/kubernetes$ sudo systemctl -l status kube-apiserver
â—Ź kube-apiserver.service - Kubernetes API Server
Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Wed 2024-09-04 14:35:51 UTC; 2s ago
Docs: GitHub - kubernetes/kubernetes: Production-Grade Container Scheduling and Management
Process: 10781 ExecStart=/usr/local/bin/kube-apiserver --advertise-address=192.168.56.11 --allow-privileged=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/log/aud>
Main PID: 10781 (code=exited, status=1/FAILURE)
CPU: 701ms

Sep 04 14:35:51 controlplane01 kube-apiserver[10781]: --log-dir string If non-empty, write log files in this directory
Sep 04 14:35:51 controlplane01 kube-apiserver[10781]: --log-file string If non-empty, use this log file
Sep 04 14:35:51 controlplane01 kube-apiserver[10781]: --log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
Sep 04 14:35:51 controlplane01 kube-apiserver[10781]: --logtostderr log to standard error instead of files (default true)
Sep 04 14:35:51 controlplane01 kube-apiserver[10781]: --skip-headers If true, avoid header prefixes in the log messages
Sep 04 14:35:51 controlplane01 kube-apiserver[10781]: --stderrthreshold severity logs at or above this threshold go to stderr (default 2)
Sep 04 14:35:51 controlplane01 kube-apiserver[10781]: -v, --v Level log level for V logs (default 0)
Sep 04 14:35:51 controlplane01 kube-apiserver[10781]: --version version[=true] Print version information and quit
Sep 04 14:35:51 controlplane01 kube-apiserver[10781]: --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
Sep 04 14:35:51 controlplane01 kube-apiserver[10781]: error: error while parsing encryption provider configuration file “/var/lib/kubernetes/encryption-config.yaml”: could not obtain secret for named key key1: illegal base64 data at input byte 0

#2

Hi @kinhk

Did you try to view the contents of the /var/lib/kubernetes/encryption-config.yaml?
The contents should match the secret field with what you get by echoing the ENCRYPTION_KEY on the node where you created the key.

And, you can refer to the much-updated and maintained version of the KTHW here.

Regards.

#3

Hello Santhosh,

Thank you for the response.
Please find the screenshot for your reference.

image
image1765Ă—350 11.9 KB

#4

Do you see the same key on the other controlplane?
controlplane01?

#5

Not the same output on two controlplanes

image

#6

I suggest you delete both the enc configs at /var/lib/kubernetes/encryption-config.yaml and redo from step 6 referring to the link I’ve shared above.

#7

For reference, I ran through this deployment only today as I’m about to do some updates on it (I manage this repo on behalf of Mumshad).

I can say without a doubt that it works, and if something’s gone wrong you either missed a step or did a step incorrectly.

Also I see you’re using tmux. Be very careful that you disable pane sync at the right times, and are logged into the correct nodes when you are syncing, as if you don’t you will royally break it! I’ve done that many times :rofl:

1 Like
#8

@Santosh_KodeKloud Thank you very much, deleted /var/lib/kubernetes/encryption-config.yaml` and redo it from step 6 get work now.
@Alistair_KodeKloud You’re right, I did not disable pane sync at the right time.

image
image926Ă—451 12.8 KB