IP Hash

The load balancer hashes each client's IP address and uses the result to pick a server, so the same client always reaches the same server. That keeps sessions and caches on one machine without the servers sharing anything. Below, each client has its own colour: follow them, then see what an office network or a server leaving does to the spread.

Speed1x

Each server drawn stands for one of your servers, and each dot for 100 real requests; the 8 clients stand for groups of users. Each client always sends from the same address. Switching the algorithm restarts the run with exactly the same requests from the same clients.

Sizing the fleet

Daily active users

Requests per user per day

Peak factorx average

Busiest hour vs the daily average. The book's estimate uses 2x.

One server handlesreq/s

Measured by load testing one server until it cannot go faster.

Target load at peak% busy

Headroom for spikes and for uneven hashing. 50 to 70% is common.

1. Average requests per second
DAU x requests per user per day / 86,400
= 1,000,000 x 50 / 86,400 = 579 req/s
2. Peak requests per second
average x peak factor = 579 x 2 = 1,157 req/s
3. Servers needed n
n = peak / (one server x target load), rounded up = 10
4. Load on each server, if hashing spreads evenly
peak / (n x one server) = 58%
It only spreads evenly when there are many clients of similar size.
5. One address sending 0% of requests
its server takes that share plus its part of the rest: 0% + (1 - 0%) / n of all traffic, a load of 58%
Set the office network's share above to see it. Adding servers barely helps: the office's share still lands on one.
6. A server leaving
taken out of the list (hash over n - 1): about (n - 1) / n of clients move = 90%
marked down: only its own clients move, about 1 / n = 10%
Leave more headroom than with round robin: a hash split is never perfectly even, and a single large network can overload one server however many you add.

Where these formulas come from

Steps 1 and 2 are the estimation method from the back-of-the-envelope chapter of Alex Xu's System Design Interview, Vol. 1, including its 2x peak. Step 6 follows from hashing modulo the number of servers: a key keeps its server when the count changes from n to n - 1 only about 1 time in n. The scene uses only 9 clients, so its counts wander around these averages.

Limits of IP hash

Uneven when clients are uneven

Many users can share one address: a company network, a university, a mobile carrier. All of them land on one server. Raise Office network's share to see it overload.

Changing the servers moves clients

Hashing over a different number of servers sends most clients somewhere new, and whatever their old server held for them is gone. Mark servers down instead of removing them, or use consistent hashing, which moves only about 1 / n of clients (see Data partitioning).

Addresses change

A phone moving from Wi-Fi to mobile data gets a new address and a new server mid-session. NGINX hashes only the first three parts of an IPv4 address, so everyone in the same block of 256 addresses shares a server.

Often a crutch

Stickiness is usually wanted because a server keeps a session in memory. Keeping sessions in a shared store instead lets any server answer any request, so the balancer is free to spread load evenly.

How NGINX and AWS hash clients

NGINX hashes the client's address; AWS's Network Load Balancer hashes each connection instead.

NGINX: ip_hash, and hash with consistent

With ip_hash, "The first three octets of the client IPv4 address, or the entire IPv6 address, are used as a hashing key." It "ensures that requests from the same client will always be passed to the same server except when this server is unavailable." To take a server out temporarily, the documentation says to mark it down "to preserve the current hashing of client IP addresses." Its general hash method warns that adding or removing a server "may result in remapping most of the keys"; with consistent, "only a few keys will be remapped".

AWS Network Load Balancer: a hash per connection

AWS's Network Load Balancer picks a target with a flow hash of the protocol, source and destination address and port, and the TCP sequence number, and keeps each connection on one target. Because the source port differs per connection, the documentation notes that a client's connections "can be routed to different targets." It balances connections, not clients: no stickiness across connections.

Sources and credibility

All are the vendors' own documentation (primary sources). Quoted phrases are short attributed quotations; everything else is described in our words. Check the pages for current behaviour.