Let's Build a Load Balancer From Scratch

A load balancer, at its absolute simplest, is this:

let next = 0;
function pick(backends) {
  const backend = backends[next % backends.length];
  next += 1;
  return backend;
}

Four lines, and it works. It also has no idea whether any of those backends are alive, gives the same share of traffic to a machine with four cores and one with sixteen, keeps sending requests to a process you are trying to shut down, and falls over the moment two requests need to reach the same server.

Everything after this point is fixing one of those.

What layer are we operating at?

The first real decision, and it is not a detail.

Layer 4 balancers move TCP connections. They read the address and port, choose a backend, and shovel bytes. They do not know what HTTP is, cannot read a path or a header, and cannot retry a failed request because they never understood where one request ended. In exchange they are extremely fast, handle any protocol, and can pass TLS straight through untouched.

Layer 7 balancers parse the protocol. They can route /api to one pool and everything else to another, retry an idempotent request on a different backend, inject tracing headers, terminate TLS, and enforce per-route timeouts. All of that costs CPU and a parse of every request.

Most real systems have both, and use each for what it is good at: layer 4 at the front door to spread connections across a layer 7 tier, layer 7 to make the routing decisions that need to understand the request. Modern hardware makes layer 7 cheap enough that “start at layer 7 unless you have a reason not to” is decent advice.

How do we choose a backend?

Work through the algorithms in that panel and the story tells itself.

Round robin is the default nearly everywhere and assumes every backend is identical. Look at what it does to the p99 when one of the four is degraded: a quarter of your requests are queued behind the slow node, and no amount of averaging hides it.

Weighted round robin fixes the capacity difference by telling the balancer that web-2 is worth two of the others. It does nothing for the degraded node, because weights are static configuration and a machine that got slow this morning does not update its own weight.

Least connections is the first algorithm that uses feedback rather than configuration. A slow backend accumulates in-flight requests and is therefore chosen less. It handles both the capacity difference and the degradation without being told about either, and it is the right default for anything with variable request costs.

IP hash is not really a balancing algorithm at all. It is a stickiness algorithm: the same client reaches the same backend so long as the pool does not change. Useful for in-process session state, and it trades evenness away to get there.

Power of two choices picks two backends at random and takes the less loaded of the pair. It lands far closer to least connections than to round robin while requiring only local information, which is why service meshes like Envoy and Linkerd default to it — with a hundred proxies each holding their own view, “least connections globally” is not something anyone actually knows.

How do we know a backend is alive?

By asking, repeatedly, and by being slow to believe the answer.

The asymmetry in that state machine is the entire point. Three consecutive failures to eject, two consecutive successes to return. A single failed probe means nothing — packet loss exists — and ejecting on it turns a blip into an outage, because the remaining nodes take the load, get slower, fail their own probes, and get ejected too. Cascading failure through health checking is a real and common outage shape.

Three further things separate a health check that helps from one that lies:

Check what actually matters. An endpoint that returns 200 unconditionally proves the process is running and nothing else. One that checks the database connection proves something useful — and one that runs three queries and calls two other services proves that any of five systems being slow will take your entire fleet out of rotation simultaneously. The usual answer is two endpoints: a liveness check that is cheap and local, and a readiness check that is honest about dependencies.

Distinguish “starting” from “broken”. A process that has just booted is not unhealthy, it is not ready. Kubernetes splits these into liveness, readiness and startup probes for exactly this reason, and conflating them produces restart loops.

Consider outlier detection. Passive health checking watches the traffic you are already sending and ejects a backend that is returning 5xx or timing out, without needing a probe at all. It notices things a synthetic check never will — like a node that answers /healthz cheerfully while failing every real request.

How do we take a backend out on purpose?

This is the part people discover during their first zero-downtime deploy, and it is almost entirely about ordering.

The wrong sequence is: stop the process, then remove it from the pool. Every request in flight is dropped, and every request routed in the gap is dropped too.

The right sequence is:

1. The process starts failing its readiness check while continuing to serve.
2. The balancer stops sending it new requests.
3. In-flight requests finish, up to a drain timeout.
4. Only then does the process exit.

That is what “connection draining” means, and the drain timeout needs to be longer than your slowest normal request and shorter than your deployment’s patience. If you are behind a keep-alive-heavy layer 4 balancer, remember that existing connections outlive the pool change — you need Connection: close on the way out, or you will keep receiving traffic on a connection nobody will re-route.

What about when requests must land on the same backend?

Two different problems wear the same clothes.

Session affinity means one user reaching one server, usually because session state lives in that process’s memory. It works, it is fragile, and the correct long-term fix is to move the session out of the process — into Redis, a signed cookie, or a JWT — so any backend can serve any request. Affinity is a workaround, and it should read like one in your architecture documents.

Cache affinity is the interesting one. If your backends each hold a cache, sending the same key to the same node turns four small caches into one large one. Here you do want hashing, and the naive version has a nasty property: hash(key) % n changes for almost every key when n changes, so adding one node invalidates nearly the entire cache at once.

Consistent hashing fixes it. Place both nodes and keys on a ring, and a key belongs to the first node clockwise from it. Adding a node claims only the arc between it and its predecessor; everything else stays exactly where it was. Compare the two numbers in that figure — one node’s share moving, against nearly everything moving.

Virtual nodes are the second half of the idea. One point per server produces a lumpy ring where some servers own far more of the circle than others; placing each server at a hundred-odd points averages it out. Every serious implementation — Cassandra, DynamoDB, Envoy’s ring hash, memcached clients — does this.

What else does a real one do?

The four-line version above is missing all of the following, and every one of them has been the subject of a post-mortem somewhere.

Timeouts, per hop. Connect timeout, request timeout, idle timeout. Without them a slow backend does not fail, it accumulates — and connections you never close are how a balancer runs out of file descriptors.

Retries with a budget. Retrying an idempotent request on a different backend turns a transient failure into a success. Retrying every failed request turns a struggling service into a dead one, because you have just multiplied its load at its worst moment. Cap retries as a percentage of total traffic, not as a per-request count.

Circuit breaking. After enough consecutive failures, stop calling a backend entirely for a while. This is what stops your balancer from being an efficient amplifier of somebody else’s outage.

Backpressure and queue limits. A bounded queue per backend, and a fast 503 when it is full. Unbounded queuing is a memory leak with a latency graph attached: requests sit in a queue long after the client that sent them has given up.

Slow start. A backend that has just joined has cold caches and an empty connection pool. Ramping its share of traffic over thirty seconds stops a fresh node from being knocked over by its own arrival.

Observability. Per-backend request rate, error rate, latency percentiles and active connections. If you cannot see which backend is the slow one, none of the algorithms above will help you find it.

What should you actually run?

Almost certainly not something you wrote. This article is about understanding the machine, not about replacing it.

nginx and HAProxy are the workhorses — decades of production, excellent at layer 4 and layer 7, and configuration you can read. HAProxy has the better health checking and observability out of the box; nginx is more often already installed.

Envoy is the modern answer when routing needs to change dynamically: it takes its configuration from a control plane over an API, which is why every service mesh is built on it.

A cloud load balancer — AWS ALB or NLB, Google Cloud Load Balancing, Azure Load Balancer — when you want somebody else to own the availability of the thing in front of everything else. This is usually correct.

A CDN as your load balancer. Cloudflare and Fastly will do origin selection, health checking and failover at the edge, which puts the decision closer to the user than any of the above.

Write your own only to learn, or when your routing decision depends on something no existing product can see. Both are legitimate reasons — and knowing which one you are in is the whole value of having read this.