Discord Messaging System

How Discord delivers a chat message to everyone in a server and stores it, built from Discord's own engineering posts. Play the traffic, switch scenarios, and click any component to see what it does and why it exists. The architecture and every figure about Discord are theirs; the traffic in the simulation is an illustrative model.

Speed1x

Guild nodes and the hash ring

Each Discord server (a "guild" internally) has one guild process. "When anything is published in a guild, it is fanned out to every session connected to it." One process per guild gives every message a single, ordered path.

"Discord is a distributed system achieved through consistent hashing": a ring is used "to lookup the node of a particular entity". If a guild node fails, only the guilds on it move to other nodes. Try Node failure.

Learn more: Consistent Hashing

Scenario

g1 members online

Manifold (send once per machine)Request coalescing

Message database

Six servers are chatting. Each message goes to the API, which stores it through the data service and publishes it to the guild node that owns that server on the hash ring. That node's guild process sends it to the session machines holding the server's online members. Node A owns 3 servers, B 1 and C 2, so their message counts grow in that proportion: with only a few servers, a hash ring is rarely even. Click any part of the diagram to see what it does.

Database: ScyllaDB. Discord measured history reads at a p99 of 15 ms and inserts at a steady 5 ms.

Per message in g1, nowdeliveries: 25sends by the guild process: 2guild process busy: 0.1 msat 50 microseconds per send, the midpoint of Discord's measured 30 to 70, and the session machines as drawn

Sizing the fan-out and the reads

Messages a second in one server

Members online

Reads a second of one channel

Database read timems

1. Deliveries
messages x members online = 20 x 30K = 600K/s
2. One guild process, sending to each session
members x 50 microseconds = 1,500 ms per message, so it can publish at most 0.67 messages a second
3. With Manifold
one send per session machine; the per-member work is spread over every machine in parallel
4. Hot channel reads, coalesced
at most one query per read time per data service instance: 1000 / 15 ms = 67/s instead of 5,000/s

Where these formulas come from

The 30 to 70 microseconds per send and the 900 ms to 2.1 s for large guilds are Discord's measurements from 2017. Step 4 is a simplification of coalescing: identical reads arriving while one query is in flight share its result, so one instance sends at most about one query per read time for a single row. Real systems also cache and batch, so treat these as orders of magnitude.

Limits of this design

One process per guild

A single guild process keeps every message in order, but it is also one process doing all of a huge server's publishing. That is why fan-out had to be spread across machines.

Hot partitions

Partitioning by channel keeps a channel's history together, but one very busy channel still lands on one partition. Coalescing and routing by channel reduce, but do not remove, that load.

Fan-out grows with audience

Every message to a server is delivered to every online member. Cost grows with messages times members, the key number in the sizing panel.

Moving data is still work

Consistent hashing limits what moves when a node fails, but those guilds and their members still reconnect, and database migrations, such as Cassandra to ScyllaDB, are large projects of their own.

How it changed over time

2017: Elixir fan-out and Cassandra

Discord described "nearly five million concurrent users and millions of events per second", with guild and session processes in Elixir located by a consistent hash ring. Messages moved from MongoDB to Cassandra, on "a 12 node cluster with a replica factor of 3", at well over 120 million messages a day.

2022: data services and ScyllaDB

With trillions of messages, Discord added Rust data services with request coalescing and moved messages to ScyllaDB in May 2022, going from 177 Cassandra nodes to 72 ScyllaDB nodes. History reads went from a p99 of 40 to 125 ms to 15 ms, and inserts from 5 to 70 ms to a steady 5 ms.

Sources and credibility

All are Discord's own engineering posts and developer documentation. Quoted phrases are short attributed quotations; everything else is described in our words. The simulation is an illustrative model; it does not reproduce Discord's real traffic.