Layer 7 Load Balancer From Scratch

A real HTTP reverse proxy on node:http. It spreads requests across backends, removes bad ones through active probes and through observed failures, and forwards the right headers.

const lb = new LoadBalancer({
  backends: [{ id: 'a', port: 9001 }, { id: 'b', port: 9002 }],
  algorithm: 'p2c',                        // 'round-robin' | 'least-connections' | 'p2c'
  healthCheck: { path: '/health', intervalMs: 2000, unhealthyThreshold: 3, healthyThreshold: 2 },
  outlier: { consecutiveFailures: 5, baseEjectionMs: 30_000, maxEjectionPercent: 50 },
});
await lb.listen(8080);

The problem and where it shows up

One server can't take all the traffic and can't be allowed to be a single point of failure. A load balancer gives clients one address and chooses a healthy backend per request. An L7 (HTTP-aware) balancer can also route on paths and headers, retry, and add metadata. You'll find them as AWS ALB, Google Cloud Load Balancing, Envoy (and every Istio sidecar), NGINX, HAProxy, Traefik, and Kubernetes Ingress controllers.

Design

client ─► server.on('request')
            ├─ buffer body (≤ maxBodyBytes, else 413), so a retry can resend it
            ├─ headers: strip hop-by-hop, append X-Forwarded-For, set -Proto/-Host, X-Request-Id
            ├─ pick(): eligible = healthy (active HC) AND not ejected (passive)
            │     round-robin | least-connections | power-of-two-choices
            ├─ http.request(backend, keep-alive agent) ── pipe response back, add X-Upstream
            └─ on error: connect error + idempotent → retry on a different backend
                         timeout → 504, other → 502, nobody eligible → 503

Algorithms (pure classes, unit-tested without sockets)

Picker	Rule	Good at	Weak at
`RoundRobin`	i-th eligible backend, cyclic	Equal backends, equal request cost	Slow or overloaded nodes keep getting their turn
`LeastConnections`	Minimum in-flight requests. Ties rotate	Mixed request cost, slow nodes	O(N) scan. Every LB instance needs accurate counts
`PowerOfTwoChoices`	Sample 2 distinct at random, take the less loaded	Nearly as good as least-conn at O(1)	Randomness can pick two busy nodes

P2C is the "power of two choices" result (Mitzenmacher). With random placement, the maximum load drops from Θ(log n / log log n) to Θ(log log n) just by comparing two samples. It also never herds: many LB instances with stale counts won't all pick the same "least loaded" node, which is a known failure of pure least-conn at scale. The test checks a concrete invariant: the single most-loaded backend is never chosen, and the least-loaded one is chosen with probability exactly 1 - C(n-1,2)/C(n,2) (= 1/2 for n = 4).

Health: two independent signals

Active health checks (checkHealth(), or intervalMs for a timer): GET /health with a timeout. A 2xx is a success. Thresholds add hysteresis: it takes unhealthyThreshold consecutive failures to mark a node down and healthyThreshold consecutive successes to bring it back, so one slow probe doesn't cause flapping.
Passive outlier ejection: real traffic results (connection errors, timeouts, 5xx) feed a per-backend consecutive-failure counter. At consecutiveFailures the backend is ejected for baseEjectionMs × ejectionCount, so repeat offenders stay out longer. maxEjectionPercent refuses to eject past that share of the fleet. If everything is failing, the cause is probably shared (a bad dependency), and ejecting everyone would turn a partial outage into a total one.
A backend is eligible only if it is healthy and not ejected. Passive detection reacts within a few requests. Active checks catch nodes that receive no traffic, and are the only way back from healthy=false.

Header rules

Hop-by-hop headers (RFC 9110 §7.6.1 plus the older RFC 2616 §13.5.1 list: Connection, Keep-Alive, TE, Transfer-Encoding, Upgrade, Proxy-*, Trailer, plus any header named in Connection) are removed in both directions.
X-Forwarded-For is appended: client-supplied, <peer ip>. Only the right-most entries added by proxies you trust are reliable. Anything to the left can be forged by the client. IPv4-mapped IPv6 peers (::ffff:1.2.3.4) are normalised.
X-Forwarded-Proto and X-Forwarded-Host are set if absent. X-Request-Id is preserved or generated (UUID v4) and echoed to the client for log correlation. The response carries X-Upstream (the chosen backend), which is handy for learning but leaks topology in production.

Retries

Only connection refused/reset failures on idempotent methods are retried, and only on a different backend. A refused connection means the backend never saw the request. A timeout doesn't: the request may have run, so retrying a POST could charge a card twice. Timeouts return 504 and are not retried.

Complexity

	Time	Space
pick: RR / P2C	O(N) to build the eligible list (O(1) with an incrementally maintained list) + O(1)	O(N)
pick: least-conn	O(N)
health round	N concurrent probes
per request	O(body) buffering	≤ maxBodyBytes per in-flight request

Failure modes and trade-offs

The LB is a single point of failure itself. Real deployments run several instances behind DNS, anycast, or an L4 tier (ECMP, or AWS NLB in front of ALB), with keepalived/VRRP for failover.
Retry storms: retries multiply load exactly when backends are struggling. Use retry budgets (Envoy retry_budget, e.g. ≤20% of active requests) and jittered backoff. This lab caps it at retries per request.
Health check that lies: /health returning 200 while the real path is broken. Passive ejection covers that. A deep health check that calls the database can mark every node down during a DB blip, which is why the max-ejection cap exists.
In-flight counts are local: with M LB instances, each sees only its own connections. P2C tolerates this better than least-conn.
Buffering bodies costs memory and adds latency (and bounds uploads). Production proxies stream and keep only a bounded retry buffer.
Not implemented: TLS termination, HTTP/2 or gRPC upstreams, WebSocket Upgrade, connection draining on removal, slow start for recovered nodes, weighted backends, sticky sessions.

How production systems do it

Envoy: ROUND_ROBIN, LEAST_REQUEST (P2C by default, with choice_count: 2), RING_HASH, MAGLEV, RANDOM. Active health checking has unhealthy_threshold / healthy_threshold. Outlier detection has consecutive_5xx, base_ejection_time × times ejected, and max_ejection_percent. This lab's passive model copies that design. There's also a panic threshold: if too few hosts are healthy, Envoy balances across all hosts anyway.
NGINX: least_conn, ip_hash, hash … consistent, and (since 1.15) random two least_conn, which is P2C. Open-source NGINX has passive checks only (max_fails, fail_timeout). Active health_check is an NGINX Plus feature.
HAProxy: roundrobin, leastconn, source, uri, and since 2.x random(2) (P2C). Checks use inter plus rise/fall thresholds, and observe layer7 + error-limit + on-error mark-down for passive detection.
AWS ALB: round-robin or "least outstanding requests". Target-group health checks have healthy and unhealthy threshold counts. It appends to X-Forwarded-For and sets X-Forwarded-Proto and X-Forwarded-Port.
Standards: RFC 7239 defines the structured Forwarded: header, the standardised successor to X-Forwarded-*.

Extension exercises

Slow start: after a node recovers, ramp its share linearly over 30 s (Envoy slow_start_config). Show that a cold cache node doesn't get crushed.
Weighted P2C / EWMA: compare an exponentially weighted moving average of latency instead of in-flight counts (Finagle / Linkerd "peak EWMA"). Rerun demo section 6.
Retry budget: allow retries only while retries_in_flight < 0.2 × active_requests, and show that it stops a retry storm.
Connection draining: removeBackend(id) stops new picks but lets in-flight requests finish (with a deadline).
Consistent-hash affinity: add a hash algorithm keyed on a header or cookie using a consistent-hash ring, and measure the remapping when a backend is ejected.

Code & Verify in the Browser Workbench