Resource Based

An agent on each server reports how busy the machine is, here its CPU use, every few seconds, and the load balancer sends work according to those reports. It is the only method that sees load the balancer did not cause itself. Below, compare two ways of using the reports with round robin, on the same traffic.

Users463 req/sLoad balancerby spare capacity1234last sent toServer 1Server 2Server 3Server 4

Each server drawn stands for one of your servers, and each dot for 100 real requests. "Busy" on each server is its share of time serving requests; a CPU report also includes any background work. Switching the algorithm restarts the run with exactly the same requests.

Sizing the fleet
x average

Busiest hour vs the daily average. The book's estimate uses 2x.

req/s

Measured by load testing one server until it cannot go faster.

% busy

Headroom for spikes and for background work. 50 to 70% is common.

x planned peak

  1. 1. Average requests per second

    DAU x requests per user per day / 86,400
    = 1,000,000 x 50 / 86,400 = 579 req/s

  2. 2. Peak requests per second

    average x peak factor = 579 x 2 = 1,157 req/s

  3. 3. Servers needed n

    n = peak / (one server x target load), rounded up = 10

  4. 4. Load on each server, on average

    traffic now / (n x one server) = 58%

  5. 5. One server losing 50% of its CPU to other work

    an equal share (round robin) asks it for load / (1 - 0.5) = 116%
    sharing by spare capacity spreads the traffic over what is left: load x n / (n - 0.5) = 61% on every server, at best

  6. 6. Reports every 2 s

    sending everything to the least loaded aims traffic now x T = 1,157 x 2 = 2,315 requests at one server between reports; sharing by spare capacity spreads them by weight instead
  7. Use the reports as weights, not as a single winner, and keep the report interval short compared with how fast load changes.
Where these formulas come from

Steps 1 and 2 are the estimation method from the back-of-the-envelope chapter of Alex Xu's System Design Interview, Vol. 1, including its 2x peak. The rest is standard capacity planning. The second line of step 5 is the best case, the traffic spread over n - 0.5 servers' worth of free CPU; reports that lag behind keep it from getting all the way there.

Limits of resource-based balancing

Reports are always a little old

A report describes the last few seconds. Between reports the balancer is guessing, and if it sends everything to the server that looked quietest, it buries that server. Sharing by spare capacity, or adding a little randomness, avoids that.

Something has to run on every server

Each server needs an agent that measures and reports, and the balancer has to collect and trust those reports. That is more moving parts than counting requests the balancer already sees.

CPU is not the only resource

A server can be short of memory, disk, or connections to its database while its CPU looks idle. Choose the measure that actually runs out first, or combine several.

A full server must still be heard

If a server reporting itself full gets no requests at all, nothing changes its next report. Real systems keep a small minimum share, as the scene does, or probe it now and then.

How HAProxy and gRPC use server reports

Both turn the reports into weights rather than picking a single winner.

HAProxy: agent checks

With an agent check, "the load balancer connects to an agent program running on a backend server", and the agent replies with text that changes how HAProxy treats it. A reply like 50% "Changes the server's weight to a percentage of its current value"; drain stops new connections and down takes the server out. HAProxy's own example agent replies 50% when the server is short of idle CPU time, which is the idea on this page.

gRPC: weighted round robin from backend reports

gRPC's weighted_round_robin policy "requires direct load reporting from backends to clients" and computes each backend's weight as qps / (utilization + ...), using CPU utilization unless the server reports its own measure. It waits a 10 second "blackout period" before trusting a new backend's reports, stops using a weight that has not been reported for 3 minutes, and falls back to plain round robin when fewer than two backends report.
Sources and credibility

Both are primary sources: HAProxy's own documentation, and gRPC's design proposal for the policy (gRFC A58, marked implemented in C-core, Java and Go). Quoted phrases are short attributed quotations; everything else is described in our words. Defaults can change between versions, so check the pages.