Weighted Round Robin

Round robin where each server has a weight, and a server with weight 3 gets three turns for every one turn a server with weight 1 gets. It lets bigger servers take a bigger share. Below, Server 1 is bigger than the other two: change its weight and see when the load evens out and when it does not.

Users579 req/sLoad balancerweighted round robin12131next in turnServer 13x sizeServer 21x sizeServer 31x size

The three servers drawn are one of your 2 groups of one big and two small servers, and each dot stands for 100 real requests. A bigger server has more workers, so it handles more requests at once. Only the speed is slowed, so you can follow single requests. Requests arrive at random, as real traffic does, so the busy figures drift around the average rather than sitting on it.

Sizing the fleet and its weights
x average

Busiest hour vs the daily average. The book's estimate uses 2x.

req/s

Measured by load testing. A big server handles its size times this.

% busy

Headroom for spikes, slow requests and a failed server. 50 to 70% is common.

x planned peak

  1. 1. Average requests per second

    DAU x requests per user per day / 86,400
    = 1,000,000 x 50 / 86,400 = 579 req/s

  2. 2. Peak requests per second

    average x peak factor = 579 x 2 = 1,157 req/s

  3. 3. Servers needed

    capacity = peak / (small server x target load) = 1,157 / (200 x 60%) = 9.65 small servers' worth
    in groups of one 3x server and two 1x servers: 9.65 / 5, rounded up = 2 big + 4 small

  4. 4. With the weight matching the size

    weight 3: every server at traffic now / total capacity = 58%

  5. 5. With your weight, 3

    big server gets 3 of every 5 requests: 58%
    each small server gets 1 of every 5: 58%
    Plain round robin (weight 1) would give the big server 32% and the small ones 96%.

  6. 6. If a big server slows to half its size

    its weight does not change, so its load doubles: 116%
  7. With the weights matching the sizes, every server runs at 58%. Any value in red above is over 100%: that server cannot keep up, however idle the others are.
Where these formulas come from

Steps 1 and 2 are the estimation method from the back-of-the-envelope chapter of Alex Xu's System Design Interview, Vol. 1, including its 2x peak. Steps 3 to 6 are standard capacity planning: a server with weight w out of a total W receives w / W of the requests, and its load is that share of the traffic divided by what it can handle.

Limits of weighted round robin

Weights are set by hand

A weight is a number in a config file. If a server slows down, from a noisy neighbour, a bad disk or garbage collection, its weight stays the same and it keeps getting the same share. Turn on Server 1 slows to half its size to see it.

Keep weights in step with the fleet

Weights should follow each server's measured capacity. When you change instance types or add bigger machines, the weights have to change too, or the load drifts out of balance without anything failing.

Blocks or a smooth order

Giving a heavy server all its turns at once sends it bursts. Spreading them out, as NGINX does, gives the same totals with a steadier stream. Switch Smooth order off and watch Server 1's queue.

Requests, not work

Like plain round robin, weights divide requests, not the work in them. A few heavy requests can still pile up on one server. Least connections adapts to that; weights cannot.

How NGINX and AWS use weights

NGINX weights single servers; AWS weights whole groups of them.

NGINX: weighted round robin is the default

NGINX's upstream documentation says: "By default, requests are distributed between the servers using a weighted round-robin balancing method." Each server's weight is 1 unless set; in the documentation's example, a server with weight=5 and two without get 5, 1 and 1 of every 7 requests. Since a 2012 change by Maxim Dounin, NGINX spreads those turns out: its commit message gives weights { 5, 1, 1 } producing { a, a, b, a, c, a, a }, "instead of { c, b, a, a, a, a, a } produced previously." The smooth order in the scene is that method.

AWS: weights between target groups

An AWS Application Load Balancer rule can forward to several target groups, each with a weight from 0 to 999. The documentation's example: with weights 10 and 20, "the target group with a weight of 20 receives twice as many requests". That is a common way to move traffic gradually from one version of a service to another. One caution from the same page: if a weighted group has no healthy targets, the load balancer does not fail over to the other group automatically.
Sources and credibility

All three are primary sources: NGINX's documentation, the change itself in NGINX's source history, and AWS's documentation. Quoted phrases are short attributed quotations; everything else is described in our words. Defaults change between versions, so check the pages for current values.