Kubernetes Challenge 3 problems!

er.shrijandra · August 31, 2022, 2:55pm

Hello

I am getting following errors,

root@controlplane ~ ➜ k create deployment worker --image=kodekloud/examplevotingapp_worker -n vote
deployment.apps/worker created

root@controlplane ~ ➜ k get deployments.apps -n vote
NAME READY UP-TO-DATE AVAILABLE AGE
worker 0/1 1 0 8s

root@controlplane ~ ➜ k logs deployment/worker -n vote
Error from server (BadRequest): container “examplevotingapp-worker-2pltp” in pod “worker-74f7f5f8b8-ks7t5” is waiting to start: ContainerCreating

root@controlplane ~ k get pods -n vote
NAME READY STATUS RESTARTS AGE
worker-74f7f5f8b8-ks7t5 0/1 ContainerCreating 0 41s

root@controlplane ~ ➜ k logs worker-74f7f5f8b8-ks7t5 -n vote
System.AggregateException: One or more errors occurred. (No such device or address) —> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No such device or address
at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
at System.Net.Dns.<>c.b__14_1(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
— End of inner exception stack trace —
at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at Npgsql.NpgsqlConnector.Connect(NpgsqlTimeout timeout) at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout) at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout) at Npgsql.ConnectorPool.Allocate(NpgsqlConnection conn, NpgsqlTimeout timeout) at Npgsql.NpgsqlConnection.OpenInternal() at Worker.Program.OpenDbConnection(String connectionString) in /code/src/Worker/Program.cs:line 74 at Worker.Program.Main(String[] args) in /code/src/Worker/Program.cs:line 19 ---> (Inner Exception #0) System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No such device or address at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult) at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult) at System.Net.Dns.<>c.<GetHostAddressesAsync>b__14_1(IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task`1 promise, Boolean requiresSynchronization)<—

Please let me know why I can’t create deployment? Thanks

Alistair_KodeKloud · August 31, 2022, 6:52pm

Hi @er.shrijandra

Without doing the lab again myself, and given the error message, I would suspect that you haven’t done the tasks in the correct order.
The error is saying that it cannot find the database, so please ensure you have deployed the database pod and that it is running without errors before deploying the worker deployment.

er.shrijandra · September 3, 2022, 7:57am

Thank you. i will try again deploying correctly.

amj1641 · August 5, 2023, 7:26am

Same error:

Getting below error in worker pod logs due to which worker pod is in CrashLoopBackOff.

controlplane ➜ k logs worker-78645fb679-h9ns8 -n vote
System.AggregateException: One or more errors occurred. (No such device or address) —> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No such device or address
at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
at System.Net.Dns.<>c.b__14_1(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
— End of inner exception stack trace —
at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at Npgsql.NpgsqlConnector.Connect(NpgsqlTimeout timeout) at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout) at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout) at Npgsql.ConnectorPool.Allocate(NpgsqlConnection conn, NpgsqlTimeout timeout) at Npgsql.NpgsqlConnection.OpenInternal() at Worker.Program.OpenDbConnection(String connectionString) in /code/src/Worker/Program.cs:line 74 at Worker.Program.Main(String[] args) in /code/src/Worker/Program.cs:line 19 ---> (Inner Exception #0) System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No such device or address at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult) at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult) at System.Net.Dns.<>c.<GetHostAddressesAsync>b__14_1(IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task`1 promise, Boolean requiresSynchronization)<—

controlplane ➜ k get pods -n vote
NAME READY STATUS RESTARTS AGE
db-deployment-7bfb789f65-bpl5z 1/1 Running 0 3m1s
redis-deployment-75d4bf6cf6-l7k2m 1/1 Running 0 3m1s
result-deployment-6f46497b48-mjzhd 1/1 Running 0 3m
vote-deployment-74b6568b4d-bs6rs 1/1 Running 0 3m
worker-78645fb679-h9ns8 0/1 CrashLoopBackOff 4 (85s ago) 3m

controlplane ➜

Tried all above steps but worker pod is not coming up. Seems like worker pod’s image is broken?

amj1641 · August 5, 2023, 7:27am

Tried tasks in correct order.

Tej-Singh-Rana · August 5, 2023, 9:40am

Please check the logs of the postgresql pod.

Alistair_KodeKloud · August 5, 2023, 10:10am

And ensure the db pod is using image postgres:9.4

enrico-proietti · October 25, 2023, 8:08am

same error :controlplane ~ ➜ k logs worker
Connected to db
System.AggregateException: One or more errors occurred. (No such device or address) —> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No such device or address
at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
at System.Net.Dns.EndGetHostEntry(IAsyncResult asyncResult)
at System.Net.Dns.<>c.b__16_1(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
— End of inner exception stack trace —
at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at Worker.Program.GetIp(String hostname) in /code/src/Worker/Program.cs:line 123 at Worker.Program.OpenRedisConnection(String hostname) in /code/src/Worker/Program.cs:line 104 at Worker.Program.Main(String[] args) in /code/src/Worker/Program.cs:line 20 ---> (Inner Exception #0) System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No such device or address at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult) at System.Net.Dns.EndGetHostEntry(IAsyncResult asyncResult) at System.Net.Dns.<>c.<GetHostEntryAsync>b__16_1(IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task`1 promise, Boolean requiresSynchronization)<—

Note : “Connected db” in the log above
Db is ready

PostgreSQL init process complete; ready for start up.

LOG: database system was shut down at 2023-10-25 07:28:55 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: autovacuum launcher started
LOG: database system is ready to accept connections

env:-> k exec db-deployment-7bfb789f65-ccnfn -it – bash
root@db-deployment-7bfb789f65-ccnfn:/# env
LANG=en_US.utf8
HOSTNAME=db-deployment-7bfb789f65-ccnfn
PG_MAJOR=9.4
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_PORT=tcp://10.96.0.1:443
PWD=/
HOME=/root
PG_VERSION=9.4.26-1.pgdg90+1
GOSU_VERSION=1.11
KUBERNETES_SERVICE_PORT_HTTPS=443
PGDATA=/var/lib/postgresql/data
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
TERM=xterm
SHLVL=1
KUBERNETES_SERVICE_PORT=443
POSTGRES_HOST_AUTH_METHOD=trust
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/9.4/bin
KUBERNETES_SERVICE_HOST=10.96.0.1
_=/usr/bin/env
and this the the postgres version :9.4
template:
metadata:
creationTimestamp: null
labels:
app: db-deployment
spec:
containers:
- env:
- name: POSTGRES_HOST_AUTH_METHOD
value: trust
image: postgres:9.4
imagePullPolicy: IfNotPresent
name: postgres

Alistair_KodeKloud · October 25, 2023, 5:31pm

It’s not complaining about postgres. It is complaining about redis. Did you start both the redis pod and its service, and name them correctly?

I have just checked the lab and it if fine

alex.hha · December 30, 2023, 1:15pm

Faced with the similar issue. Worker deployment failed to start with the following message

 logs -n vote worker-8db94d9cb-sptv9
System.AggregateException: One or more errors occurred. (No such device or address) ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No such device or address
   at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
   at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
   at System.Net.Dns.<>c.<GetHostAddressesAsync>b__14_1(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Npgsql.NpgsqlConnector.Connect(NpgsqlTimeout timeout)
   at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout)
   at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout)
   at Npgsql.ConnectorPool.Allocate(NpgsqlConnection conn, NpgsqlTimeout timeout)
   at Npgsql.NpgsqlConnection.OpenInternal()
   at Worker.Program.OpenDbConnection(String connectionString) in /code/src/Worker/Program.cs:line 74
   at Worker.Program.Main(String[] args) in /code/src/Worker/Program.cs:line 19
---> (Inner Exception #0) System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No such device or address
   at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
   at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
   at System.Net.Dns.<>c.<GetHostAddressesAsync>b__14_1(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)<---

Redis deployment and service were already created. The worker container was able to start only after I have created db deployment/service.

Alistair_KodeKloud · December 30, 2023, 8:31pm

Correct. The worker pod depends on everything else so it must be started last

The log dump that everyone is so fond of posting indicates a failure to locate the postges database which will happen if postgres and is service are not both running.

alex.hha · December 30, 2023, 8:43pm

and how should I know about that ? At least from the schema it’s not clear, imho

Alistair_KodeKloud · December 31, 2023, 7:46am

It is just something you get to know. If there are databases involved in any kind of deployment, then it’s pretty certain that the applications are going to want to talk to the databases. Therefore you always deploy databases first, then applications.

This is known as dependencies. Applications depend on databases.

An analogy. You are hungry, you go to the refrigerator, there is no food there - you complain!

Satisfying your hunger depends on there being food in the refrigerator. You are the application, the refrigerator containing food is the database with its data.

alex.hha · January 1, 2024, 10:12am

But you didn’t give any insights about the database. Just the following

Create new deployment. name: ‘worker’
image: ‘kodekloud/examplevotingapp_worker’
status: ‘Running’

Maybe you should put env vars into this task. Like - You should add env variables for db connection: DB_HOST=db, DB_USER=db_user, DB_PASSWORD=db_password
In that case it would be straight forward

Alistair_KodeKloud · January 1, 2024, 2:58pm

Adding environment variables would not make the worker pod run if the database is not running first. The connection error is not due to authentication, it is due to the database not being available and ready.

alex.hha · January 1, 2024, 3:23pm

It will give some insights. Isn’t it ?

harishvishwanath · February 8, 2024, 5:13pm

Just faced similar issues as above yesterday, worker app pod goes in to “CrashLoopbackoff” state.

kubectl logs pod/worker-7c5bf6b48f-kt82s -n vote
System.AggregateException: One or more errors occurred. (No such device or address) —> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No such device or address
at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
at System.Net.Dns.<>c.b__14_1(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
— End of inner exception stack trace —
at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at Npgsql.NpgsqlConnector.Connect(NpgsqlTimeout timeout) at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout) at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout) at Npgsql.ConnectorPool.Allocate(NpgsqlConnection conn, NpgsqlTimeout timeout) at Npgsql.NpgsqlConnection.OpenInternal() at Worker.Program.OpenDbConnection(String connectionString) in /code/src/Worker/Program.cs:line 74 at Worker.Program.Main(String[] args) in /code/src/Worker/Program.cs:line 19 ---> (Inner Exception #0) System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No such device or address at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult) at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult) at System.Net.Dns.<>c.<GetHostAddressesAsync>b__14_1(IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task`1 promise, Boolean requiresSynchronization)<—

Attached Screenshot

When i checked the PostgresSQL db Pod logs as below, only warnings about env variables with ‘trust’ , any other suggestions on these

kubectl logs pod/db-deployment-fb698d4f7-mdn27 -n vote

WARNING: POSTGRES_HOST_AUTH_METHOD has been set to “trust”. This will allow
anyone with access to the Postgres port to access your database without
a password, even if POSTGRES_PASSWORD is set. See PostgreSQL
documentation about “trust”:
PostgreSQL: Documentation: 16: 21.4. Trust Authentication
In Docker’s default configuration, this is effectively any other
container on the same system.

     It is not recommended to use POSTGRES_HOST_AUTH_METHOD=trust. Replace
     it with "-e POSTGRES_PASSWORD=password" instead to set a password in
     "docker run".

The files belonging to this database system will be owned by user “postgres”.
This user must also own the server process.

The database cluster will be initialized with locale “en_US.utf8”.
The default database encoding has accordingly been set to “UTF8”.
The default text search configuration will be set to “english”.

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data … ok
creating subdirectories … ok
selecting default max_connections … 100
selecting default shared_buffers … 128MB
selecting default timezone … Etc/UTC
selecting dynamic shared memory implementation … posix
creating configuration files … ok
creating template1 database in /var/lib/postgresql/data/base/1 … ok
initializing pg_authid … ok
setting password … ok
initializing dependencies … ok
creating system views … ok
loading system objects’ descriptions … ok
creating collations … ok
creating conversions … ok
creating dictionaries … ok
setting privileges on built-in objects … ok
creating information schema … ok
loading PL/pgSQL server-side language … ok
vacuuming database template1 … ok
copying template1 to template0 … ok
copying template1 to postgres … ok
syncing data to disk … ok

Success. You can now start the database server using:

WARNING: enabling “trust” authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
–auth-local and --auth-host, the next time you run initdb.
postgres -D /var/lib/postgresql/data
or
pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start…LOG: database system was shut down at 2024-02-08 05:52:31 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
done
server started

/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

LOG: received fast shutdown request
waiting for server to shut down…LOG: aborting any active transactions
.LOG: autovacuum launcher shutting down
LOG: shutting down
LOG: database system is shut down
done
server stopped

PostgreSQL init process complete; ready for start up.

LOG: database system was shut down at 2024-02-08 05:52:33 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: autovacuum launcher started
LOG: database system is ready to accept connections

rob_kodekloud · February 8, 2024, 6:34pm

The worker app’s container is implemented in such a way that it’s not very good at recovering from timing issues with with the postgres pod. There are a number of work-arounds for this, one of which is using the POSTGRES_HOST_AUTH_METHOD hack. Another that I’ve heard works also is to apply the worker app’s YAML only after the postgresql pod is already up. YMMV (“your mileage may vary”). And sometimes, the worker app pod comes up even if you don’t do either of these things.

harishvishwanath · February 10, 2024, 8:14am

Thank you, I redid the challenge and it worked fine this time.