Design a pipeline that reliably ingests high-volume application logs from a large fleet, buffers them against bursts and downstream slowness, and persists them for later query.
Design a centralized logging pipeline. A large fleet of application servers emits log lines continuously; the pipeline must collect them, survive bursts and downstream slowdowns without dropping logs, and persist them durably for later search and analysis. This is a write-dominated, reliability-sensitive system: the value is in not losing logs, even when the persistence/indexing tier is temporarily slow or unavailable.
Direct writes from every app server into the storage tier would couple the whole fleet's logging to storage availability and collapse during storage hiccups or traffic spikes. The pipeline must therefore accept logs into a durable buffer that absorbs bursts and decouples producers from the persistence tier, with consumers draining the buffer into storage at a sustainable rate. Deployment and operational concerns matter here: the collector tier is large, and how it's rolled out and monitored is part of the design.
Design the architecture emphasizing durable buffering and producer/consumer decoupling for reliability. Then document your capacity plan, your deployment approach for the collector fleet, and the trade-offs (durability vs latency, backpressure handling, buffer sizing).