Hi,
Followed this issue but no luck
opened 02:15AM - 01 Aug 18 UTC
closed 06:55PM - 05 Jan 24 UTC
priority/backlog
area/usability
sig/auth
help wanted
lifecycle/frozen
<!-- This form is for bug reports and feature requests ONLY!
If you're lookin… g for help check [Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes) and the [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/).
If the matter is security related, please disclose it privately via https://kubernetes.io/security/.
-->
**Is this a BUG REPORT or FEATURE REQUEST?**:
> Uncomment only one, leave it on its own line:
>
/kind bug
> /kind feature
**What happened**:
I often get `Invalid padding on input` whenever I try to start Pods or delete secrets. This doesn't happen all of the time, however.
They look something like this:
```
$: kubectl run busybox --image=busybox --command -- sleep 3600
deployment.apps "busybox" created
# Wait a minute or two.
$: pod_name=$(kubectl get pods -l run=busybox -o jsonpath="{.items[0].metadata.name}")
$: kubectl describe pods $pod_name | grep -A 20 "Events"
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 40m default-scheduler Successfully assigned busybox-68654f944b-h92hg to ip-10-0-4-219
Warning FailedMount 10m (x23 over 40m) kubelet, ip-10-0-4-219 MountVolume.SetUp failed for volume "default-token-xqb2r" : Internal error occurred: invalid padding on input
Warning FailedMount 13s (x18 over 38m) kubelet, ip-10-0-4-219 Unable to mount volumes for pod "busybox-68654f944b-h92hg_default(24feb300-952a-11e8-a3b2-0ec4bc8258b0)": timeout expired waiting for volumes to attach or mount for pod "default"/"busybox-68654f944b-h92hg". list of unmounted volumes=[default-token-xqb2r]. list of unattached volumes=[default-token-xqb2r]
```
**What you expected to happen**:
I expected containers for the Pod to initialize successfully.
**How to reproduce it (as minimally and precisely as possible)**:
1. Create an encryption token: `token=$(cat /dev/urandom | head -c 32 | base64)`
2. Put it in this `encryption-config.yaml`:
```
cat >/tmp/encryption-config.yaml <<ENCRYPTION_CONFIG
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: $token
- identity: {}
ENCRYPTION_CONFIG
3. Start the `kube-apiserver` with `--experimental-encryption-provider-config=/tmp/encryption-config.yaml`
4. Attach a few kubelets to it, or restart any bound Kubelets
5. Try to create the `busybox` Pod per above.
```
**Anything else we need to know?**:
1. I'm on AWS. I'm happy to share any configurations required to help debug this problem.
2. My original `encryption-config.yaml` looked like this (I created it per kubernetes-the-hard-way):
```
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: OTEnuvc77R2ehn+Zhlwv1jk+UFVZjl7B3Qw+CWMLzMc=
- identity: {}
```
**Environment**:
- Kubernetes version (use `kubectl version`): 1.10.2
- Cloud provider or hardware configuration: AWS
- OS (e.g. from /etc/os-release): `Ubuntu 18.04 LTS`
- Kernel (e.g. `uname -a`): `Linux ip-10-0-1-42 4.15.0-1010-aws #10-Ubuntu SMP Thu May 24 08:41:34 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux`
- Install tools: none
- Others:
0109 12:04:42.820629 1 reflector.go:569] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key “/registry/secrets/default/my-secret”: no matching prefix found
Hi @prasanna
Can you share your EncryptionConfiguration YAML?
Hi Santosh,
Try setting the identity provider as the first under providers.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- identity: {} # No encryption
- aescbc:
...
Hi Santosh,
