Understanding Load Balancing for Beginners

Spreading traffic is the easy half. Noticing that a backend has quietly stopped working is the hard half.

A service had three backends. One of them quietly lost its database connection. The health check asked for the home page, the web server returned 200 because the process was fine, and the load balancer happily sent a third of all traffic to a box that failed every request touching data. The dashboard showed three healthy backends the whole time. The error rate sat at almost exactly thirty three percent for twenty minutes.

Distribution was working perfectly. Detection was not, and that is the part most guides skip. This post covers what a load balancer does, the algorithms and when each fits, the layer 4 and layer 7 split, health checks in the detail they deserve, and what actually happens when something fails. Three failures follow, each with a different cause.

What load balancing is

Load balancing means spreading incoming requests across several backends so none of them gets swamped. It also means checking which backends still work, and routing around the ones that do not.

Both halves matter. A load balancer that spreads perfectly and detects poorly gives you outages that look like random application bugs.

Why you want one, beyond capacity

Capacity is the obvious reason. It is rarely the best one.

Staying up. Any single box can die, get redeployed, or restart. With a load balancer in front, none of that reaches a user.

Deploying. You can pull one box out, update it, and put it back. That is what makes rolling updates and blue green switches possible at all.

Growing. Adding capacity becomes adding a box to a pool, rather than resizing something.

Maintenance without downtime. Patching, certificate renewal, node replacement. All routine instead of scheduled events.

The thread through all four is that load balancing makes individual boxes disposable. Most modern operations practice is built on that.

The algorithms, and when each fits

Five turn up in nearly every load balancer. The choice matters more than the defaults suggest.

Round robin is the default nearly everywhere. It is the right choice less often than that suggests. It assumes every request costs the same. So the moment one endpoint serves a cached reply in five milliseconds while another runs a report for eight seconds, round robin will cheerfully hand one backend three reports in a row while its neighbour takes the easy work.

Least connections adapts on its own. A backend still chewing on slow requests holds more open connections, so it gets fewer new ones. For most real services it is the better default.

That last row is a genuinely neat idea. Checking every backend's connection count gets expensive in a big pool. Picking two at random and sending to the quieter one gets you most of the benefit for almost nothing. You will see it called power of two choices.

Layer 4 and layer 7

This split decides what a load balancer can do. The names help less than the behaviour.

Layer 4 works on TCP or UDP. It sees addresses and ports. It forwards packets without reading them, which makes it very fast and completely blind to what the request says. It works for any protocol, because it does not care.

Layer 7 reads the request. It knows the hostname, the path, the headers, the cookies. So it can route on any of them, terminate TLS, retry a failed request on another backend, and add headers.

That last row is the one that matters here. A layer 4 check opens a TCP connection and calls the backend alive. That is exactly the check that passed while the database was unreachable in the story above. A layer 7 check asks for a real path and reads the answer, which is what makes a useful verdict possible.

Most teams want layer 7 for HTTP services. Use layer 4 for databases, brokers, and anything where the load balancer has no business reading the protocol.

Health checks, in the detail they deserve

Here is the heart of the post, because this is where load balancing works or fails in practice.

A health check is a request the load balancer makes on a schedule to decide whether a backend should get traffic. Three things decide how useful it is. What it asks for. How it judges the answer. And how fast it changes its mind.

What it asks for ranges from opening a TCP connection, to asking for a fixed path, to asking for a path that touches real dependencies. The first proves a process is listening. The second proves the app replies. Only the third proves the box can do its job.

How it judges means which status codes or body content it accepts. Taking any 2xx is normal. Reading the body lets you tell degraded from healthy.

How fast it changes its mind is the thresholds. How many failures in a row before removal. How many successes before it comes back. How often it checks.

upstream api_backend {
    least_conn;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.13:8080 max_fails=3 fail_timeout=30s backup;
    keepalive 32;
}

server {
    listen 443 ssl;
    server_name api.example.com;

    location / {
        proxy_pass http://api_backend;
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 2;
        proxy_connect_timeout 2s;
        proxy_read_timeout 30s;

        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Several settings there are doing specific work.

Using least_conn instead of the default round robin copes with requests of different lengths. The max_fails and fail_timeout pair sets up passive checking, so a backend drops out after three failed real requests in thirty seconds rather than being probed on the side. The backup keyword holds the third server out of rotation until the others are gone, so it is a reserve rather than a normal member.

The proxy_next_upstream line is the one worth copying. It retries a failed request on a different backend. That turns one dead box into a slightly slower response instead of an error. Capping the retries at two stops a genuinely broken request being replayed across your whole pool.

Those two timeouts differ on purpose. A backend should accept a connection almost instantly, while a slow reply is normal on some endpoints. So the two numbers should never match.

Two more terms are worth having.

Liveness asks whether this box should be restarted. Keep it shallow. A liveness check that fails when a shared database slows down will restart every box at once, and turn a slowdown into an outage.

Readiness asks whether this box should get traffic right now. This one can go deep, because it pulls a backend that cannot serve.

Getting those the wrong way round is common and destructive. A deep liveness check turns a dependency problem into a restart storm.

Learn Nginx and the proxy layer properly

All of this is configuration you will actually write, and the ideas carry to whatever load balancer you run. The Nginx for Beginners course on KodeKloud works through upstream blocks, balancing methods, timeouts, and health checking in labs where you edit the config yourself.

What happens when a backend dies

Failure has a timeline. Knowing it explains why outages last as long as they do.

The backend stops working. Requests already on their way to it fail or hang. New requests keep arriving, because the load balancer does not know yet. The health check fails once, then again, then hits its threshold. Only then does the backend get pulled. Traffic shifts to the survivors.

That gap between failing and being pulled is your outage window. It equals your check interval times your failure threshold, plus up to one more interval depending on timing. So a check every ten seconds with a threshold of three gives you roughly thirty to forty seconds of errors. Work that number out for your own setup rather than assuming it is small.

Tightening those numbers is tempting. It has a cost.

Draining, the other half of failover

Everything above is about a backend dying on you. The more common case is you pulling one out on purpose, for a deploy, a patch, or a scale down. Do that badly and you get errors that look exactly like a bug.

Draining means stopping new requests to a box while letting the ones in flight finish, then removing it once they are done or a timeout passes.

Without draining, a deploy kills connections mid request. Users see errors. Your error rate spikes on every deploy. Everyone learns to deploy at night, which is exactly what good deployment practice is meant to avoid.

The sequence that works has four steps. Mark the box as not ready, so the load balancer stops sending new requests. Wait long enough for the load balancer to actually notice, which is at least one check interval, and this is the step people skip. Let the requests in flight finish, up to a timeout you set. Then stop the process.

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 10"]
      readinessProbe:
        httpGet:
          path: /readyz
          port: 8080
        periodSeconds: 5
        failureThreshold: 2
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 10
        failureThreshold: 3

That preStop sleep looks like a hack. It is the correct fix for a real race. When a pod is shutting down, Kubernetes removes it from the Service endpoints and sends the stop signal at the same time. Those two paths travel at different speeds. So without a pause, the process can exit while the load balancer is still sending it work. Sleeping longer than your readiness period covers the gap.

Note the two probes hit different paths on purpose. The readiness path is the deep check with dependencies. The liveness path is shallow and only confirms the process itself is alive. That is the split described above, written as config.

Three failures, three causes

Same load balancer. Three ways it goes wrong.

Errors on every deploy

Every deploy produces a short burst of 502s. Small enough that nobody has prioritised it. Consistent enough that everyone expects it.

The cause. Missing or thin draining. The pod is shutting down while still listed as an endpoint, so requests arrive at a process on its way out.

The fix. A preStop pause longer than your readiness interval. A grace period long enough for real requests to finish. And app code that stops taking new connections while finishing current ones when it gets the stop signal.

Why it lingers. The spike is small and lines up with deploys, so people file it mentally as normal deploy noise. It is a defect. It is fixable. And fixing it is what makes deploying during the working day comfortable.

Uneven load across backends

Six boxes behind a load balancer. Two of them consistently run hotter. Round robin is configured. The boxes are identical. It should be even.

The cause. Almost always long lived connections plus round robin. The algorithm balances connections, not work. So if clients hold connections open, whichever backends got the heavy clients keep them. A new box added during a scale up gets no traffic at all until existing connections close. That is why fresh instances sometimes sit idle beside busy ones.

The fix. Switch to least connections, so new connections favour the quieter boxes. Set a maximum connection lifetime, so long lived clients reconnect and get rebalanced. On the client side, connection pools with a maximum age do the same from the other end.

How to tell. Compare connections per backend against requests per backend. Connections even but requests uneven means your clients differ. Connections uneven means your algorithm or connection lifetimes are the cause.

A health check that caused the outage

During a database slowdown, every application box got pulled from rotation at once. The service went from degraded to completely down.

The cause. A deep health check wired to liveness. Every box depended on the same database. The check failed everywhere at the same moment. The platform removed everything.

The fix. Split the probes properly. Liveness stays shallow and asks only whether this process is alive, so a shared dependency problem never restarts anything. Readiness can go deeper. Even then, ask whether a slow dependency should really pull a box that could still serve cached or reduced answers.

The wider lesson. A health check is a control loop with the power to remove capacity. Control loops that react to shared inputs will act together. So anything checked the same way by every box, and depending on something shared, should raise an alert rather than trigger an automatic action. The automatic action here is your whole fleet walking off the job at once.

Practise on a real cluster

This gets concrete the moment you watch a Service stop routing to a pod that failed its readiness probe. Working through Services, endpoints, probes, and Ingress in labs is what the Certified Kubernetes Administrator (CKA) course on KodeKloud does, and the KodeKloud playgrounds let you kill a pod mid request to see exactly where the errors land.

Sticky sessions

Sticky sessions send a given user to the same backend every time, usually with a cookie the load balancer sets.

It works. Think of it as a bridge rather than a destination. Affinity means the load is no longer balanced, because a backend holding many sessions cannot shed them. It means losing a box logs those users out rather than just moving them. And it makes scaling weaker, because new capacity only helps new sessions.

The alternative is keeping session state somewhere shared, like a cache or a database, so any backend can serve any request. More work up front. It removes the limit for good.

So the honest read is this. Affinity is fine while you move state out of process memory. Needing it forever says something about your architecture, not about your load balancer.

Try this on your own service

Half an hour, and every step is safe on something you own.

Start by finding out what your health check actually asks for. Not what you assume. The real path, and the real pass criteria. If it asks for the home page and takes any 200, you have the setup from the top of this post, and you can improve it this week.

Then work out your detection window. Check interval times failure threshold. That number, in seconds, is how long users get errors when a backend dies. Most people have never worked it out.

Next, kill a backend on purpose somewhere safe and time how long the errors last. Compare against your maths. It usually takes a bit longer than the arithmetic suggests, which is good to know before an incident rather than during one.

Then deploy something and watch your error rate at that moment. A spike means you have a draining problem, and it is fixable.

Finally, check whether your liveness and readiness probes hit the same endpoint. If they do, ask what happens when a shared dependency slows down. The answer may be that everything restarts at once.

What you should be able to answer now

What does your health check actually verify? If the answer is "the process is listening," you could be sending traffic to a box that fails every request touching data. Nothing on your dashboard will say so.

How long does it take to spot a dead backend? Check interval times failure threshold. Worth knowing the number rather than assuming it is small.

What happens to requests in flight during a deploy? With no draining, they fail. That is why your error rate spikes on every deploy, and why people prefer deploying at night.

If your database slowed down, how many boxes would your checks pull? If the answer is all of them, you have a deep check wired to liveness and you are one slow dependency from an outage you caused yourself.

The three backend service at the top had a load balancer doing its spreading job perfectly. What it lacked was a health endpoint that tried a query, which is maybe fifteen lines of code. Spreading traffic comes configured by default. Noticing failure is the part you have to build.

Ready to Build the Traffic Layer Properly?

Load balancing sits between your network and your apps, so being comfortable on both sides makes the middle easy. Take the Nginx for Beginners course on KodeKloud for upstream blocks, balancing methods, and passive health checking you configure yourself. Pair it with the Certified Kubernetes Administrator (CKA) course for probes, endpoints, and how a Service picks eligible pods. Then use the KodeKloud playgrounds to kill a backend mid request and time how long the errors last. Start with your health endpoint.

FAQs