Least Response Time

The load balancer times every answer and keeps a moving average for each server. Each new request goes to the server with the lowest average response time, counting the requests it already has in progress. It measures speed directly, where least connections only infers it. Below, compare it with least connections and round robin on the same traffic.

Users463 req/sLoad balancerleast response time1234last sent toServer 1Server 2Server 3Server 4

Each server drawn stands for one of your servers, and each dot for 100 real requests. Only the speed is slowed, so you can follow single requests. Least response time scores each server as its average response time x (its requests in progress + 1) and picks the lowest. Switching the algorithm restarts the run with exactly the same requests.

Sizing the fleet
x average

Busiest hour vs the daily average. The book's estimate uses 2x.

req/s

Measured by load testing one server until it cannot go faster.

% busy

Headroom for spikes, slow requests and a failed server. 50 to 70% is common.

x planned peak

  1. 1. Average requests per second

    DAU x requests per user per day / 86,400
    = 1,000,000 x 50 / 86,400 = 579 req/s

  2. 2. Peak requests per second

    average x peak factor = 579 x 2 = 1,157 req/s

  3. 3. Servers needed n

    n = peak / (one server x target load) = 1,157 / (200 x 60%), rounded up = 10

  4. 4. Load on each server

    traffic now / (n x one server) = 58%

  5. 5. One server half as fast

    an equal share (round robin) asks it for 116%
    moving work off it until all are equally busy puts every server at load x n / (n - 0.5) = 61%

  6. 6. About one request in five taking 4x as long

    every algorithm: load x 1.6 = 93% on average. No algorithm removes work; it can only place it better.
  7. Response-time balancing helps most when servers differ in speed and nobody told the balancer. Any value in red above is over 100%.
Where these formulas come from

Steps 1 and 2 are the estimation method from the back-of-the-envelope chapter of Alex Xu's System Design Interview, Vol. 1, including its 2x peak. The rest is standard capacity planning: the second line of step 5 spreads the same traffic over n - 1 full servers and one half server.

Limits of least response time

Old averages linger

An average only changes when a server answers. A server that had a slow minute and recovered is still seen as slow, gets few requests, and so takes a long time to prove otherwise. Real systems let averages decay over time or send a few probe requests. Turn Server 1 is half as fast on, then off, to watch it.

It measures requests, not only servers

A server that just answered a heavy request looks slow, although it is fine. With very mixed requests the averages are noisy, and least connections can do as well or better.

Fast errors look like fast answers

A server returning errors instantly has a superb response time. Unless errors are left out of the average or caught by health checks, it attracts traffic, as with least connections.

More bookkeeping

The balancer must time every response and keep an average per server, and each balancer only sees its own requests. It is more work than counting connections, which is why it is usually an option rather than the default.

How NGINX uses least response time

NGINX calls it least time, and lets you choose which part of the response to time.

least_time: speed and connections together

With least_time, "a request is passed to the server with the least average response time and least number of active connections, taking into account weights of servers." Ties are "tried in turn using a weighted round-robin balancing method", the same rule used in the scene. NGINX does not document exactly how the two measures are combined or averaged; the score in the scene is one common way to do it.

Which time to measure

least_time header uses the time to receive the response header, and least_time last_byte the time to receive the full response; adding inflight also counts requests still in progress. The documentation notes that before version 1.31.0 the method was available only as part of NGINX's commercial subscription.
Sources and credibility

Both are NGINX's own documentation (primary sources). Quoted phrases are short attributed quotations; everything else is described in our words. Availability changes between versions, so check the pages for your version.