Follow one request from browser to container. Every networking concept you need appears on that path, in order, and so does every failure you will be paged for.
Highlights
- The networking concepts a DevOps engineer needs are not a syllabus, they are the set that appears on the path a single request travels, which is why tracing one request teaches them in the order you will meet them.
- Symptoms map to layers with surprising reliability, so connection refused, timeout, and reset each point at a different part of the stack before you have run a single command.
- A connection refused arrives instantly and means something answered, while a timeout means nothing did, and that distinction alone eliminates half the possible causes.
- Ephemeral port exhaustion is the classic scaling failure, because a single source address has roughly 28,000 usable outbound ports and a busy proxy consumes them faster than they release.
- Path MTU problems are the cruellest failure mode, since small requests succeed and large ones hang, which looks like an application bug and is not.
- Kubernetes adds three address spaces to reason about rather than a new networking model, and every packet still obeys everything below it.
- Ten commands cover the diagnosis, and knowing which layer each one interrogates matters more than memorising their flags.
A service worked in staging and timed out in production. Same image, same config, same manifest, and the only difference was that production sat behind a VPN gateway which lowered the effective packet size, so the small health check succeeded and the large API response hung forever. Three engineers spent a day on the application before anybody suspected the network, because nothing in the symptom pointed there.
That is the shape of most networking incidents. The failure is somewhere on the path, the symptom rarely names the layer, and the engineer who fixes it quickly is the one who can work down the path in order. This post traces a single request from a browser to a container and back, meets every concept you actually need in the order it appears, and gives you a table mapping symptoms to layers. Three role scenarios follow, one each from DevOps, cloud, and platform engineering, and each breaks at a different layer.
What you actually need to know
The networking concepts a DevOps engineer needs are the ones that appear on the path a request travels: name resolution, addressing and subnets, routing, transport, address translation, encryption, and whatever your platform adds on top, along with the specific way each one fails.
That list is short because it is defined by the path rather than by a syllabus. You do not need to recite the OSI model. You need to know which layer a symptom is pointing at.
The model, used the way practitioners use it
The seven layer model gets taught as trivia and used as a filing system. Its real value is that it tells you where to look when something is wrong, and in practice four layers carry almost everything.
Keeping this in mind pays off as diagnostic sequencing. When something breaks, you are trying to find the lowest layer that is misbehaving, because everything above it will produce confusing symptoms until you do.
Did you know
The OSI model is a reference standard that real networks never implemented. Your traffic actually runs on the TCP/IP model, which collapses those seven layers into four, and the two do not line up cleanly. TLS, for instance, sits awkwardly between transport and application and belongs to neither in the original taxonomy. This is worth knowing because arguments about which layer something belongs to are usually unproductive, while the underlying question of what to check before what is always useful. Keep the sequencing, discard the pedantry.
Following one request
Here is the spine. One request, from a browser to a container running in Kubernetes, with every concept appearing where it actually occurs.
Stop one, the name becomes an address
The browser has api.example.com and needs an address. It checks its own cache, then the operating system's stub resolver, then asks a recursive resolver, which walks the hierarchy and returns an A record with a TTL.
Two things matter here operationally. The TTL governs how long that answer is cached, which is why a DNS change does not take effect immediately and why nobody can flush someone else's cache. And the resolver you use is configured locally, so a pod, a laptop, and a CI runner may all resolve the same name differently.
dig +short api.example.com
dig api.example.com +noall +answer
getent hosts api.example.comThe first returns just the address, which is what you want in a script. The second shows the full answer with its TTL counting down, so running it twice reveals whether you are seeing a cached answer or a fresh one. The third asks the operating system rather than DNS directly, which matters because it also consults /etc/hosts and nsswitch.conf, and a stale hosts file entry beats the entire internet.
Stop two, the address is routed
Armed with an address, the client now needs to send packets to it. It consults its routing table, decides whether the destination is on the local network or needs a gateway, and hands the packet onward.
This is where CIDR notation earns its keep. A subnet mask says which part of an address identifies the network and which identifies the host, and getting this wrong is how two systems that look adjacent turn out to be unable to reach each other.
The two missing addresses in each row are the network address and the broadcast address, which is why a /24 gives you 254 hosts rather than 256. Cloud providers reserve more, typically five per subnet, which catches people out when a /28 turns out to hold eleven usable addresses rather than fourteen.
ip route get 10.0.2.15
ip -brief addr show
traceroute -n api.example.comOf those three, ip route get is the most underused command in this post. Rather than making you read a routing table and work out which rule wins, it asks the kernel directly which route it would use for one specific destination, and answers in one line. The second gives a compact view of every interface and its addresses. The third shows each hop the packet takes, which is how you discover that traffic you assumed stayed inside your VPC is leaving through a NAT gateway.
Stop three, the transport connection
Once the packet arrives, TCP takes over. A connection opens with a three way handshake: the client sends SYN, the server replies SYN ACK, the client sends ACK. That exchange establishes both sequence numbers and, more usefully for you, is what produces the distinct failure modes below.
Connection refused means the handshake reached the host and nothing was listening on that port. The host replied with a reset. This arrives immediately.
Connection timed out means nothing replied at all. Something dropped the packet silently, which is what firewalls and security groups do by design. This takes seconds.
Connection reset means the connection was established and then killed mid conversation, which points at an application crash, a proxy timeout, or an idle connection being reaped.
That distinction is the single most valuable diagnostic in this post. Refused and timeout look similar in a log line and mean completely different things: refused says you reached the right host and the service is not there, timeout says you never got a reply and something is blocking you.
Quick tip
Time your failures. A connection refused comes back in milliseconds because the host actively replied. A timeout takes seconds because your client waited for a reply that never came. Before running any other command, notice which one you got, because that alone tells you whether to look at the service or at the firewall. Engineers who diagnose network problems quickly are almost always doing this without describing it.
Stop four, the encryption
If the request is HTTPS, TLS negotiates on top of the established TCP connection. The client offers its supported versions and ciphers, the server picks and presents its certificate chain, and both derive session keys.
Three failures dominate here. An expired certificate, which is unambiguous. An incomplete chain, where the server sends its own certificate without the intermediate, which works in browsers that cache intermediates and fails in curl and in your services. And a hostname mismatch, where the certificate is valid but not for the name you requested.
openssl s_client -connect api.example.com:443 -servername api.example.com < /dev/null
curl -vI https://api.example.comThe -servername flag is essential and frequently omitted. It sends Server Name Indication, which is how one address serves many certificates, and without it you get whatever default certificate the server offers rather than the one you meant to test. The curl form is quicker for a first look and shows you the negotiated protocol and the certificate summary in a readable form.
Stop five, translation on the way in and out
Between client and container, addresses get rewritten more than most people realise.
SNAT, source translation, rewrites the source address so replies come back to the right place. This is what a NAT gateway does for outbound traffic, and it is why your database sees connections from the gateway rather than from individual pods.
DNAT, destination translation, rewrites the destination, which is what a load balancer or a Kubernetes service does when forwarding to a backend.
The operational consequence is that the address a service sees is frequently not the address the client had, which is why forwarded headers exist and why rate limiting by source address surprises people.
Watch out
Every translated connection is tracked in a table, because the kernel has to remember how to reverse the translation on the way back. That table has a finite size, and when it fills, new connections are dropped silently with no useful error anywhere. A node under heavy outbound load can exhaust conntrack and produce timeouts that look like an application problem, an upstream problem, or anything except what it is. If you see unexplained timeouts that correlate with load rather than with any deployment, check the conntrack table before you check the application.
Stop six, inside the cluster
Kubernetes does not invent new networking, it adds address spaces. Three of them, and knowing which one an address belongs to resolves most cluster networking confusion.
The Service address is the one that confuses people, because nothing is actually bound to it. It exists as a rule in the node's packet filtering, and traffic to it gets rewritten to a pod address on the way through. That is why you cannot ping a Service and why a tcpdump on a Service address returns nothing.
kubectl get endpointslices -l kubernetes.io/service-name=api
kubectl exec -it debug -- nc -vz api.prod.svc.cluster.local 8080Reach for the first of those whenever a Service is not working, because it shows which pods the Service actually selected. An empty result means your selector matches nothing, which is the most common Service problem and is invisible from the Service object itself. The second tests connectivity from inside the cluster, which distinguishes a networking problem from an ingress problem.
Learn the fundamentals underneath all of this
Everything above sits on Linux networking, virtual interfaces, and the addressing concepts that predate every cloud. The DevOps Pre Requisite Course on KodeKloud covers Linux CLI, networking and DNS, IPs and ports, and SSL and TLS basics with hands on labs, which is precisely the ground this post assumes.
Networking concepts mapped to symptoms
This is the artifact. When something breaks, find the symptom, get the layer, and start there.
The second to last row deserves a note, because comparing ip route get from a working and a failing host is the fastest way to find a routing difference, and it is a comparison most people never think to make.
Three scenarios, three roles, three layers
Now the same path, broken in three different places, one per role.
The DevOps engineer, a pipeline that fails only sometimes
Builds fail perhaps one time in five, always at the step that pulls dependencies, with a timeout. Reruns usually succeed. It looks exactly like a flaky registry.
The layer is transport, and specifically ephemeral ports. Every outbound connection from a host needs a local source port, drawn from a range of roughly 28,000 by default, and after closing, a connection sits in TIME_WAIT for a couple of minutes before its port can be reused. A build that opens hundreds of short lived connections in parallel across several concurrent jobs on one runner can exhaust that range, and new connections then fail with no error that mentions ports.
ss -s
ss -tan state time-wait | wc -l
cat /proc/sys/net/ipv4/ip_local_port_rangeRunning ss -s gives a socket summary by state, which shows immediately whether TIME_WAIT is in the tens of thousands. The second counts them precisely. The third shows the available range, and comparing that number against the count from the second command tells you whether you are near the limit. The fix is connection reuse in the client rather than widening the range, though widening buys time.
That failure mode is worth recognising because it presents as intermittency, and intermittency is what engineers most often misattribute to the other party.
The cloud engineer, a VPC peering that half works
Two VPCs are peered. Instances in one can reach instances in the other on port 22 but not on the application port, and the route tables look correct on both sides.
The layer is network filtering, and the specific trap is that traffic must be permitted in four places rather than two. The security group on the source must allow egress, the security group on the destination must allow ingress, and the network ACL on each subnet must permit both directions. Security groups are stateful, so a reply to an allowed connection is automatically allowed. Network ACLs are stateless, so the return traffic needs its own rule, and that rule must cover the ephemeral port range rather than the application port.
ip route get 10.1.0.42
nc -vz 10.1.0.42 8080
traceroute -n -T -p 8080 10.1.0.42Running the route lookup first confirms the packet is heading for the peering connection rather than out through a gateway, which rules out the routing half of the problem in one command. The connectivity test then distinguishes refused from timeout, and a timeout here points squarely at filtering. The third traces the path using TCP on the actual port, which matters because many networks treat ICMP differently from TCP, so a plain traceroute can succeed on a path that your application traffic cannot use.
The stateless network ACL is the piece that catches experienced people, because everything about the configuration looks symmetric until you remember that the return traffic arrives on a high numbered port nobody wrote a rule for.
The platform engineer, a service that works for some pods
A newly deployed Service is reachable from pods on some nodes and not others. Same manifest, same image, and the endpoints list looks populated.
The layer is the link layer, and the cause is MTU. The cluster's overlay network encapsulates each packet, which adds header bytes, so the effective payload size inside the cluster is smaller than the underlying network's. If the pod interface advertises a size the path cannot carry, large packets need fragmenting, and if the do not fragment bit is set and ICMP is being filtered, they are silently dropped instead. Small responses fit and succeed. Large ones vanish.
ip link show eth0
kubectl exec -it debug -- ping -M do -s 1400 10.244.2.7
kubectl exec -it debug -- ping -M do -s 1472 10.244.2.7Checking ip link shows the configured MTU on the interface. The two ping commands are the actual test: -M do forbids fragmentation and -s sets the payload size, so if the smaller packet succeeds and the larger one fails, you have found your path MTU and it is lower than the interface claims. Binary searching between those two sizes gives you the exact number within a few attempts.
In the wild
MTU problems are the cruellest failures in networking because they break the correlation engineers rely on. Every other problem is consistent: the connection works or it does not. An MTU mismatch means small requests succeed, health checks pass, the service reports healthy, and only responses above a certain size disappear. Teams routinely spend days in application code because the evidence says the network is fine, and the network genuinely is fine for every packet small enough to be tested casually. If your symptom includes the words works for small payloads, stop and check the MTU before anything else.
Practise on real infrastructure
Networking is the discipline where reading and doing diverge most, because the concepts are simple and the diagnosis is a skill. The Certified Kubernetes Administrator (CKA) course on KodeKloud covers Services, DNS, network policy, and cluster networking through hands on labs, and the KodeKloud playgrounds give you clusters and Linux machines where you can break connectivity deliberately and diagnose it.
Ten commands worth knowing properly
Grouped by the layer they interrogate, because that is how you will reach for them.
Two of these deserve emphasis. ip route get answers in one line what reading a route table answers in five minutes of squinting. And mtr combines ping and traceroute into a continuous view, which is what makes it the right tool for intermittent loss where a single traceroute shows nothing.
Try this on your own systems
Half an hour, and every step is safe on a system you own.
Pick one service you operate and trace the whole path yourself, in order. Resolve the name and note the TTL. Ask the kernel which route it would use. Test the port with nc and time how long the answer takes. Inspect the certificate with SNI set. Then find the same service's Endpoints if it runs in Kubernetes, and confirm the pod addresses match what you expected.
Write down the layer at which each step operates as you go. You will finish with a map of one real service in your own environment, and the exercise takes longer to describe than to do.
Then break something deliberately in a playground. Point a Service at a selector matching nothing and watch what the failure looks like from a client. Set an MTU lower than your path and watch large requests hang while small ones succeed. Fill an ephemeral port range with a loop and watch new connections start failing. Having seen each symptom once, deliberately, is what makes recognising it under pressure possible.
What you should be able to answer now
Four questions, and each points back at a section if the answer does not come.
When a connection fails, can you tell refused from timeout without checking? The timing gives it away, and that single distinction eliminates roughly half the possible causes before you run anything else.
For one service you operate, do you know which addresses a packet carries at each hop? Between client, load balancer, node, and pod, the source and destination get rewritten more than once, and knowing where is what makes forwarded headers and rate limiting comprehensible.
What is the effective MTU on your cluster network, and how would you check? Most engineers do not know their own number, which is fine right up until the day small requests work and large ones do not.
Which layer would you check first for a failure that only appears under load? Load related failures are almost always transport, meaning ephemeral ports or connection tracking, and looking at the application first is the default mistake.
The VPN gateway from the opening broke nothing that any application code could have fixed. It lowered the effective packet size by a few dozen bytes, and every response above that size disappeared without an error. The day of debugging was not caused by a hard problem. It was caused by nobody asking which layer the symptom pointed at, and the symptom, small things work and big things do not, points at exactly one.
Ready to Get Genuinely Confident With Networking?
Networking rewards practice more than reading, because the concepts are straightforward and the diagnosis is a skill built by repetition. The DevOps Pre Requisite Course on KodeKloud covers Linux, networking, DNS, IPs and ports, and TLS basics from the ground up, the Certified Kubernetes Administrator (CKA) course covers Services, DNS, and network policy inside a cluster, and the KodeKloud playgrounds give you machines where breaking connectivity costs nothing. Start with the layer you are least sure of.
FAQs
Q1: What networking concepts does a DevOps engineer actually need?
The ones that appear on the path a request travels, which is a much shorter list than a networking course covers. Name resolution, including how TTLs cause changes to take effect gradually and how the local resolver configuration can differ between a pod, a laptop, and a CI runner. Addressing and subnets, meaning enough CIDR to know why a /24 gives you 254 usable hosts and why cloud providers reserve five more per subnet. Routing, and specifically how to ask the kernel which route it would use rather than reading a table. Transport, where the distinction between connection refused and connection timed out eliminates half the possible causes of any failure. Address translation, because the address a service sees is often not the address the client had. TLS, mainly the three failures of expiry, incomplete chain, and hostname mismatch. And whatever your platform layers on top, which for Kubernetes means three address spaces rather than a new model. You do not need to recite the OSI layers, you need to know which layer a symptom points at.
Q2: What is the difference between connection refused and connection timed out?
Refused means your packet reached the host and the host actively replied that nothing was listening on that port, which arrives in milliseconds. Timed out means nothing replied at all, so your client waited and eventually gave up, which takes seconds. That difference is the single most useful diagnostic distinction in networking, because refused tells you the routing and filtering are fine and the service is the problem, while timeout tells you something is silently dropping packets, which is precisely what firewalls, security groups, and network policies do by design. A third case is worth knowing: connection reset means the connection was established and then killed mid conversation, which points at an application crash, a proxy idle timeout, or a connection being reaped rather than at anything below. The practical habit is to notice the timing before running any other command, since engineers who diagnose network problems quickly are almost always doing this without articulating it.
Q3: Why do small requests work while large ones hang?
That symptom points at MTU, the maximum size a link will carry, and it is the cruellest failure mode in networking because it breaks the consistency engineers rely on. Health checks pass, small API calls succeed, the service reports healthy, and only responses above a certain size disappear. The cause is usually an encapsulating layer such as a cluster overlay network or a VPN, which adds header bytes and therefore reduces the payload that fits. When a packet is too large and the do not fragment bit is set, the correct behaviour is for a router to send back an ICMP message asking for a smaller size, and many networks filter ICMP, so the packet is dropped in silence instead. You test it with ping -M do -s SIZE, where the flag forbids fragmentation and the size sets the payload, then binary search between a size that works and one that does not until you find the real limit. If your symptom description includes the words works for small payloads, check this before anything else.
Q4: What do I need to know about Kubernetes networking specifically?
Less than people expect, because Kubernetes does not invent a new networking model, it adds address spaces on top of the existing one. There are three. The pod CIDR gives every pod an address that is routable anywhere in the cluster and changes on every restart. The service CIDR gives every Service a virtual address that is only meaningful inside the cluster, and nothing is actually bound to it, which is why you cannot ping a Service and why a packet capture on that address shows nothing. The node network is the real underlying addresses in your VPC. Traffic to a Service address is rewritten to a pod address by rules in the node's packet filtering as it passes through. The single most useful command when a Service is not working is listing its EndpointSlices, because an empty result means the selector matches no pods, which is the most common Service problem and is completely invisible from the Service object itself. Everything below that still applies exactly as it does outside a cluster.
Q5: Why do things work fine and then fail under load?
Load related failures are almost always at the transport layer, and two causes dominate. Ephemeral port exhaustion happens because every outbound connection needs a local source port from a range of roughly 28,000 by default, and a closed connection holds its port in TIME_WAIT for a couple of minutes before it can be reused, so a host opening many short lived connections can run out. Connection tracking exhaustion happens because every translated connection is recorded in a kernel table so the translation can be reversed on the way back, and when that table fills, new connections are dropped silently with no useful error anywhere. Both present as timeouts that correlate with traffic volume rather than with any deployment, which makes them easy to misattribute to an upstream service. Check socket counts by state with ss -s, compare the TIME_WAIT count against the configured port range, and check the conntrack table size on the node. The real fix for the first is connection reuse in the client rather than widening the range.
Q6: Which commands should I learn first?
Five give you the most coverage. Use dig +short to see what a name resolves to, and getent hosts alongside it, because the second consults the hosts file and the search domains while the first does not, and the difference between them has explained many confusing incidents. Use ip route get ADDRESS rather than reading a routing table, because it asks the kernel which route it would actually choose for one destination and answers in a line. Use nc -vz host port to distinguish refused from timeout, which is your fastest branch point. Use ss -tlnp to see what is listening and on which interface, since a service bound to localhost rather than to all interfaces is a common and invisible cause of connection refused. And use ping -M do -s SIZE when the symptom involves size. Beyond those, mtr for intermittent loss and openssl s_client -servername for certificate problems round out the set. Knowing which layer each one interrogates matters more than memorising their flags.
Discussion