Round Robin

A load balancer sits in front of several servers and hands each new request to the next server in a fixed order: 1, 2, 3, then back to 1. It never looks at how busy a server is. Below, change the servers and the requests and see when taking turns spreads the work evenly and when it does not.

Users463 req/sLoad balancerround robin1234next in turnServer 1Server 2Server 3Server 4

Each server drawn stands for one of your servers, and each dot for 100 real requests. Only the speed is slowed, so you can follow single requests: an ordinary request takes 1 second here. The load on each server is the real one from the sizing below. Requests arrive at random, as real traffic does, so the busy figures drift around that load rather than sitting on it.

Sizing the fleet
x average

Busiest hour vs the daily average. The book's estimate uses 2x.

req/s

Measured by load testing one server until it cannot go faster.

% busy

Headroom for spikes, slow requests and a failed server. 50 to 70% is common.

x planned peak

  1. 1. Average requests per second

    DAU x requests per user per day / 86,400
    = 1,000,000 x 50 / 86,400 = 579 req/s

  2. 2. Peak requests per second

    average x peak factor = 579 x 2 = 1,157 req/s

  3. 3. Servers needed n

    n = peak / (one server x target load) = 1,157 / (200 x 60%), rounded up = 10

  4. 4. Load on each server with round robin

    traffic now / (n x one server) = 1,157 / (10 x 200) = 58%
    Round robin splits requests evenly, so this is every server's load, but only while servers and requests are all alike.

  5. 5. When they are not alike

    A half-speed server still gets 1/n of requests: load / 0.5 = 116%
    About one request in 5 taking 4x as long: load x 1.6 = 93% on average, more on an unlucky server
    One server fails: load x n / (n - 1) = 64%
  6. Every server runs at 58%. Anything shown in red above is over 100%: that server cannot keep up and its queue grows until requests fail.
Why changing only DAU barely changes the animation

More users means more servers, and each server's load stays near the target you set. Round robin behaves the same on 4 servers or 4,000 at the same load, so the animation only changes when the load does: through the target, the traffic right now, or rounding when the fleet is small.

Where these formulas come from

Steps 1 and 2 are the estimation method from the back-of-the-envelope chapter of Alex Xu's System Design Interview, Vol. 1, including its 2x peak. Steps 3 to 5 are standard capacity planning: a server's load is the work it is given divided by the work it can do, and round robin gives every server the same number of requests whatever its speed. Losing one of n servers spreads the same traffic over n - 1.

Limits of round robin

Equal turns, not equal work

Round robin counts requests. If some take much longer than others, a server can be stuck on heavy ones while its neighbour sits idle. Turn on Some requests are slow and watch the queues. Least connections sends work to whoever has the least in progress instead.

Blind to server speed

A slower or busier server gets the same share as the rest. If you know the difference in advance, weighted round robin gives bigger servers more turns. If you do not, least connections or least response time adapt as they go.

Finds dead servers late

With passive health checks the balancer only learns a server is down when real requests to it fail. Active checks, which probe each server on a timer, notice sooner. Turn Health checks off and on with a server down to compare.

Plan for a server failing

When one of n servers goes, the rest share its traffic and their load rises by n / (n - 1): from 60% to 80% with four servers. Size the fleet so that still fits, which is why targets well below 100% are normal.

How NGINX and AWS use round robin

Both make round robin the default, because it needs no knowledge of the servers at all.

NGINX: the default, with weights and passive checks

NGINX's load balancing guide says: "When the load balancing method is not specifically configured, it defaults to round-robin." It also says when the spread is even: "when the requests are processed in a uniform manner and completed fast enough", which is the condition the toggles above break. A weight on a server gives it more turns: in the guide's example, weight=3 on one of three servers sends it 3 of every 5 requests. Health checks are passive by default: after a failed request (max_fails, default 1) the server is skipped for fail_timeout (default 10 seconds), then tried again with real requests, as in the scene. Active health checks are part of the paid NGINX Plus.

AWS Application Load Balancer: round robin unless you choose

In AWS's documentation for target groups, the routing algorithm can be round_robin, least_outstanding_requests or weighted_random, and "The default is round_robin." The documentation also says the load balancer "routes requests to the registered targets that are healthy", so unhealthy targets drop out of the rotation. Least outstanding requests is AWS's answer to the slow-request problem above.
Sources and credibility

Both are the vendors' own documentation (primary sources). Quoted sentences are short attributed quotations; everything else is described in our words. Defaults change between versions, so check the pages for current values.