Nginx + Dotnet Docker - Upstream Connection

I am using docker container for my Dotnet WebApi using swagger and PostgreSQL.

I am trying to connect nginx with dotnet on port 5083, when I use example.com:5083/swagger/index.html to GET api content, it works fine. But when I try example.com/swagger/index.html or with https l it only works 2 times before throwing upstream connection closed prematurely error in nginx log and shows 503 bad gateway error on swagger.

There are no errors in dotnet and postgres logs.
The resource consumption is also not exceeding 4% for docker containers.
The response size when successful is not more than 2.5 KB as shown by postman.

How do I fix this?