Scalability Fundamentals

Your app works fine for 100 users. One morning someone shares it on a big forum and 100,000 people arrive at once. Pages slow down, then stop loading. Scalability is a system's ability to handle more work by adding resources, without being rebuilt from scratch. There are only two ways to add resources, and each one brings its own problems. This topic walks through both.

A picture to keep in your head

Think of a small restaurant with one cook. More customers arrive than the cook can handle. You have two choices:

• Give the cook a bigger stove and a faster knife. That is vertical scaling.
• Hire more cooks. That is horizontal scaling.

More cooks bring new problems. Someone has to hand out the orders (a load balancer). The recipes must be written on the wall, not kept in one cook's head, so any cook can take any order (stateless servers). And sooner or later every cook is queueing at the same fridge (the database). Keep this picture in mind; the rest of the topic is these four ideas.

The obvious approach: buy a bigger machine

Vertical scaling, or scaling up, means giving the same server more CPU, more memory or faster disks. Your code does not change at all. You get no new failure modes, no network calls between machines, and all your data stays on one box where reading it is fast. For a young product this is very often the right first move.

Why it eventually runs out

• There is a ceiling. Machines only come so big. Once you are on the largest one available, scaling up is over.
• Big machines get expensive. The top end of the hardware range often costs more per unit of capacity than the middle.
• One machine is one point of failure. If it crashes, everything is down. Even a planned upgrade usually means a restart, which is downtime.

Do not underestimate it: Stack Overflow

In 2016 Stack Overflow's own engineer, Nick Craver, published exactly how the site ran. On a single day, 9 February 2016, its load balancers received 209,420,973 HTTP requests, including 66,294,789 page loads. That traffic was served by 9 primary web servers and two SQL Server clusters, with the main Stack Overflow database on machines with 384 GB of RAM each.

The web tier was so efficient that he wrote they were "down to needing only 1 web server", having tested it by accident, and added: "I'm saying it works. I'm not saying it's a good idea." They kept the extra servers for rolling deployments, headroom and redundancy. The lesson: a few big, well-tuned machines go a very long way. Scale out when you have to, not because it sounds impressive.

The fix: add more machines

Horizontal scaling, or scaling out, means running many ordinary servers side by side. There is no ceiling: when traffic grows, you add another server. And a dead server is no longer an outage; it is just one fewer server. The price is that you now have a small distributed system, which brings three problems. Each one has a standard answer.

Step 1: something has to share out the traffic

Users cannot pick a server themselves, so a load balancer sits in front, receives every request and passes it to one of the servers. How it picks matters more than you might expect: taking turns (round robin) is fine when every request costs the same, but when some requests are slow, choosing the least busy server works much better. You can watch each strategy on real traffic in the load balancer visualizations.

Step 2: servers must not remember users

Suppose Alice logs in and server A stores "this is Alice" in its own memory. Her next click is sent by the load balancer to server B, which has never heard of her, so she is logged out. The same goes for shopping carts, upload progress and anything else a server keeps to itself.

The rule is: app servers must be stateless. They can keep nothing between requests that another server would need.

Anything that must be remembered moves out of the server:

• into the request itself, as a signed token the user sends every time (Systema works this way: logging in gives you a signed access token, so any of our servers can check who you are), or
• into a shared store that every server can reach, such as Redis for sessions and the database for everything else.

Some load balancers offer sticky sessions, sending a user back to the same server every time. That hides the problem rather than fixing it: when that server dies, everyone pinned to it loses their session anyway.

Step 3: the database becomes the bottleneck

App servers are easy to multiply precisely because they hold no data. But the data has to live somewhere, and now five or fifty servers are all asking the same database. Teams usually relieve it in this order, cheapest first:

1. Cache hot reads. Keep frequently read results in memory, in a cache such as Redis, so most reads never reach the database. Stack Overflow's Redis handled about 160 billion operations a month with every instance below 2% CPU.
2. Add read replicas. Copies of the database that serve reads, while one primary handles writes.
3. Shard the data. Split the data itself across several databases. This is the most powerful step and the most work, so it comes last. The data partitioning visualizations show how sharding, hashing and consistent hashing decide where each record lives.

How to tell you need to scale

Do not guess; measure. The signals that matter:

• Latency at the slow end. Watch the slowest 1% of requests (the p99), not the average. Averages hide pain.
• Saturation. CPU, memory, disk and database connections creeping towards their limits.
• Errors. Timeouts and failed requests rising with traffic.

Then find the one bottleneck and fix that. Adding app servers when the database is the problem only adds more clients to the queue at the fridge. Rough math helps you see it coming: see Back-of-the-Envelope Estimation.

Vertical or horizontal? A quick guide

• New product, modest traffic: vertical. One good server and a managed database.
• The site must survive a machine failure: horizontal. At least two servers behind a load balancer.
• Traffic swings a lot through the day: horizontal. Add servers at peak, remove them at night.
• The database is struggling: vertical first, then a cache and read replicas, and shard last.

In practice almost every large system uses both: many modest app servers scaled out, and a few powerful database machines scaled up.

Where else this shows up

• Autoscaling in every major cloud adds and removes servers automatically based on load. It only works if the servers are stateless.
• CDNs scale out the delivery of images, scripts and videos by copying them to servers close to users.
• Interviews. Almost every system design question starts with one server and asks what breaks first as traffic grows. This topic is that conversation.

Implementing it in your project

• Make the app stateless: keep sessions in signed tokens (such as JWTs) or in Redis, store uploaded files in object storage rather than on the server's disk, and keep no per-user data in memory.
• Run two instances behind a load balancer (Nginx, Caddy or a cloud load balancer) with a health-check endpoint, so one can restart or fail without downtime.
• Scale up the database first: a larger instance, then a cache in front, then read replicas. Partition last.
• Measure: track p99 latency, CPU and memory of each tier, and database connections, so you scale the actual bottleneck.

---

Sources and credibility

Everything above is written from scratch; the only figures and quotes come from the source below, attributed where used.

Primary (the organisation's own material, highest confidence)

• Stack Overflow: The Architecture - 2016 Edition, Nick Craver, February 2016. Source of the request and page-load counts for 9 February 2016, the web and SQL Server counts, the 384 GB database servers, the Redis figures and the single web server quote.
• Stack Overflow: The Hardware - 2016 Edition, Nick Craver, March 2016. Confirms the server specifications.

These describe Stack Overflow as it ran in 2016, a historical snapshot rather than its setup today.

On copyright: the diagrams on this page were drawn for this article, and the explanations are original. Quoted phrases are short and attributed.