Before you startSystem Design: The Depot#

A very short book for engineers who are asked, in every design review, what happens under load, where the data lives and which copy is right.

The promise

Read this in three evenings and you will be able to take a feature with a stated load and a stated volume of data, estimate it on one sheet of paper, choose how to store, copy, split and cache its data, say what consistency each operation needs, protect it from overload and from the loss of a region, and defend each choice against anyone who says "web scale" with confidence.

It reads as one story. Bramble is an online stationery shop whose system is well organised and has never been busy. Then its founder books a television feature, a back-to-school sale, a launch on a second continent and a limited-edition drop with half a million people arriving at 09:00. Every chapter starts with something breaking. The fix is the chapter's idea. By the end of the chapter, the fix has opened a new crack.

Most ideas arrive four ways: a place in one running metaphor (a network of depots holding stock and serving lorries), a diagram of the mechanism, a one-sentence Remember it as line you can say out loud, and a dated account of who thought of it, when, and what they were reacting to. Every chapter then does what a mentor would do at a whiteboard: it names the traps and their tell-tale signs, gives you a small design with one planted flaw to find, and hands you three questions to ask whoever proposes the mechanism next, plus the one point on which respected practitioners still disagree, argued fairly from both sides.

Who this is for

You build or review systems that other people depend on. You have heard "replica", "shard", "quorum", "CAP" and "p99" in meetings and nodded. You want the part of the field that lets you lead a design review, without reading six hundred pages on each topic first.

The level is "can lead the design review", not "can implement the database". If you have run a sharded datastore in production, been paged for a split brain and can state the CAP theorem as it was proved, this book will be too slow for you; skim the At the whiteboard boxes, chapter 20 and the epilogue.

How to read it

• Straight through the first time. The plot carries the ideas.
• The numbers are fixed. Bramble has one set of figures (orders a day, requests a second, rows, bytes), and every calculation in the book uses them. When a chapter says "not yet", it says at what number the answer changes.
• The Epilogue is the reference: the metaphor table, the review checklist, a numbers card, the design method on one card, the timeline of the field and a reading list by year.

The cast, and where Bramble stands

Bramble sells stationery online: its own range, other sellers' goods, and a monthly subscription box. Its system is split along the lines of the business into a handful of services (Catalogue, which shows products; Ordering, which takes orders; Billing; Fulfilment, which owns stock and ships; Promotions), each with its own database on its own single machine, and they tell one another what happened by sending messages through a broker. Ana is its architect: good at deciding what belongs in which service, and a beginner at everything in this book. Kofi is the staff engineer who has been paged for most of these failures before and carries the history. Priya leads Catalogue and proposes the first fix, which is the natural one and usually wrong in a way worth watching. Marcus is the consultant who proposes what the largest companies run; his arguments are the real ones, and when he is wrong it is because Bramble's numbers do not meet his premise, not because he is foolish. Lena founded the company and keeps selling more than the system can serve.

Code, where it appears, is Java 21 and SQL from Bramble's stock and order tables.

The sibling books

This is the second of three. DDD and Microservices: The Atlas tells how Bramble found its boundaries and owns everything about where they go and how the parts talk: bounded contexts, aggregates, idempotency keys, the outbox, sagas and contract tests. When this book needs one of those it names the Atlas chapter in a sentence and moves on. This book owns data, scale and reliability. A third, on software architecture, owns system-level styles, the analysis of trade-offs and decision records.

Dates and sources

Every "who and when" was checked against the source named in research/milestones.md in the book's repository, which says where only a secondary source could be found, and the theorems of chapters 13 to 15 are stated as their papers state them, with the quotations recorded in the same file. There are no hardware prices and no product limits, because those date within a year.

Contents

Act I: One machine and its limits

1. The Morning the Shop Went Viral (latency, throughput, percentiles, saturation, scaling up)
2. The Back of the Envelope (estimation, peak versus average, headroom, load testing)
3. Two of Everything (scaling out, statelessness, load balancing, health checks)

Act II: The data that would not fit

4. The Query That Read Every Row (indexes, query plans, pagination, connection pools, storage engines)
5. Copies of the Stock Book (replication, lag, read-your-writes, failover)
6. Two Clerks, One Shelf (transactions, isolation levels, locking)
7. Too Big for One Depot (partitioning, consistent hashing, hot keys)
8. A Shelf for Every Kind of Thing (choosing datastores, search, object storage)

Act III: Making it fast

9. The Shelf by the Counter (caching patterns, invalidation, stampedes)
10. Closer to the Customer (the content delivery network, HTTP caching, DNS)
11. The Yard Was Full (queues, load levelling, backpressure)
12. Three Left (the log, batch versus stream, pushing to the browser)

Act IV: When the copies disagree

13. Which Copy Is Right? (CAP, PACELC, consistency models, quorums)
14. Two Head Offices (consensus, leader election, fencing tokens, locks)
15. What Time Is It? (clocks, ordering, identifiers)

Act V: Staying up, and knowing when to stop

16. The Doorman (rate limiting, load shedding, retries with jitter, the waiting room)
17. The Day the Region Went Dark (failure domains, multi-region, recovery objectives, backups)
18. Promises with Numbers (service levels, error budgets, alerting, game days)
19. The Bill (cost, simplicity, scaling back down)
20. The Drop (the design method and the end-to-end design review)

Epilogue

21. The Depot: the metaphor table, the review checklist, the numbers card, the design method card, the timeline, the reading list

Chapter 1 · Act IThe Morning the Shop Went Viral#

The problem

At 07:42 on a Tuesday a breakfast-television presenter holds up Bramble's weekly planner and says she has bought one for everyone she knows. Lena booked the feature three weeks ago and mentioned it to engineering on Monday.

Ana is watching the dashboard when it happens. Product-page traffic, which peaks at 150 views a second on a busy evening, passes 1,500. The tile she trusts, average response time, moves from 90 milliseconds to 190. Amber starts at 500. It looks like a system working a little harder.

It is not. Support's inbox fills with screenshots of a blank page. Lena, at home, cannot load her own shop. Priya has the fix ready: Catalogue's app server is at 100% CPU, so add servers. It would take a day; nothing at Bramble has ever run as more than one copy.

Kofi asks a different question first: where was that average measured? On the server, from the moment a request starts being handled to the moment it finishes. He opens the log of the web proxy that sits in front of it, which stamps each request when it arrives. Half the customers waited more than 2 seconds. One in a hundred waited 14. A great many gave up, and a request that is abandoned in the queue is never timed at all.

The server was timing the work. The customers were timing the wait.

The idea

Think of the shop as a depot. A request is a lorry arriving at the gate and leaving loaded. Two numbers describe how the depot is doing, and they are not the same number.

Latency is a distribution, and an average flattens it. Report percentiles: p50 is the time half the requests beat; p95 and p99 are the times that 95 and 99 in a hundred beat. On an ordinary evening Bramble's product page has a p50 of 90 ms and a p99 of 1.9 seconds, and nobody knew, because the average said 120. That slow end is tail latency: the slowest lorry in a hundred, which is the one that complains. A customer who views twenty pages has an 18% chance of meeting the p99 at least once (1 − 0.99²⁰), and the customers who load most pages are the ones who buy most.

Why did the tail explode? Utilisation is how full the loading bays are: the share of time a resource is busy. Saturation is the queue down the road when they are all full: work that has arrived and cannot start. Lorries do not arrive evenly, so a queue forms well before the bays are full on average, and the wait grows out of all proportion to the load. For one server whose work takes 20 ms:

The rule behind the table is response = service time ÷ (1 − utilisation). Between 50% and 80% nothing seems to happen. Between 90% and 99% the wait grows tenfold for a tenth more traffic. Past 100% there is no steady state at all: the queue grows until something drops lorries.

The diagram also shows the bottleneck: the one narrow gate every lorry must pass. That morning it was the app server's processors, which can render 285 pages a second and no more. The database behind it was 20% busy. Making the database faster would have changed nothing; adding capacity anywhere but the bottleneck never does.

The first money to spend is usually on scaling up: a bigger depot. Kofi resizes the server from 8 cores to 32. It takes eleven minutes and no code. The rest of the day runs at 450 views a second and the bigger machine takes it at under half busy. Scale up first because it is fast, it changes nothing in the software, and one machine has no copies to keep in step. The rule has three limits. A single machine is a single point of failure, however large. The price curve bends: the largest machines cost far more per core than the mid-sized ones. And the gain is less than the cores. Amdahl's law says that if a fraction of the work cannot be shared out, that fraction caps the speed-up: with 5% of the work serial on today's server, four times the cores gives 3.5 times the throughput, and no number of cores gives more than twenty. Bramble's 32 cores saturate near 1,000 views a second, not 1,140. Scaling out, the alternative, has no such ceiling, but it costs a load balancer, shared state and a deployment that must work in more than one place.

Remember it as: time the wait, not the work; watch the slowest lorry in a hundred; a depot nine-tenths full already has a queue down the road.

What Ana learned

• Measure latency as p50, p95 and p99 at the customer's side of the queue, and put those on the dashboard instead of the mean.
• Find the resource that saturates first before spending anything, and keep it under about 70% busy at peak.
• Reach for a bigger machine first, and write down the two things it does not fix: the ceiling and the single point of failure.

…but

The bigger machine holds. Then Lena announces the back-to-school sale: ten times a normal day, in August. Priya says to resize the machine again. Nobody in the room can say whether that is too little, too much, or whether such a machine exists.

Chapter 2 · Act IThe Back of the Envelope#

The problem

Lena wants one answer by Friday: will the shop hold on sale day? She gets three. Priya says to resize the machine again, because resizing worked last time. The hosting account manager says Bramble should take the largest machine on the price list, to be safe. Finance says the budget line for August is whatever engineering can justify, and so far nobody has justified anything.

Ana notices that none of the three answers contains a number of requests. "Ten times a normal day" is the only figure anyone has, and it is a statement about a day. Machines do not fall over across a day. They fall over in the worst minute of it.

Priya has run a test, to be fair. She pointed a script at the product page and measured 3,000 views a second on the original server, which would make the television morning impossible to explain. Kofi reads the script. It requests one product, the weekly planner, as one customer, over and over. After the first request everything it needs is in memory. The real catalogue has 200,000 products and most requests are for something the server has not looked at recently.

Kofi takes a sheet of paper and says the meeting will not end until the sheet has a number for the worst minute, and a second number for what one machine can do about it.

The idea

Back-of-the-envelope estimation is arithmetic with round numbers, done before any design, to find out which of your problems are real. It has four moves.

From a day to a second. A day has 86,400 seconds. Call it 100,000: the sum becomes a matter of dropping five zeros, and the answer is 14% low, which will not matter. Bramble serves 5,000,000 product pages on a normal day, so about 50 requests per second on average; 50,000 orders is one every two seconds.

From average to peak. Traffic is not flat. Bramble's weekday evening runs at three times the daily average: 150 views a second. A sale is burstier, because an email lands in two million inboxes at 09:00, so Kofi uses a factor of four on a day that is itself ten times normal.

From requests to resources. Each product page costs five database queries and weighs 1 MB, of which 50 KB is the page Bramble computes and the rest is images, styles and scripts. Each order stores about 1 KB plus three lines of 0.25 KB.

Two rows are worth the whole exercise. Storage is nothing: a year of orders is about 32 GB, so no decision should be driven by disk. Bandwidth is alarming: 16 Gbit a second is more than Bramble's connection carries. Kofi circles it and writes "later" beside it.

From resources to machines. This needs one measured figure, and a few orders of magnitude to judge designs by before anything is measured:

Memory is a thousand times faster than disk and a million times faster than another continent. Most of this book is a consequence of that table.

The measured figure comes from testing, and there are three kinds. A load test applies the expected peak with a realistic mix and asks whether latency holds. A stress test keeps raising the rate until something breaks, to learn what breaks first and how. A soak test holds a realistic load for hours, to find what leaks. Priya reruns hers with the real catalogue and a production-shaped mix. One 8-core server saturates near 285 views a second.

Nobody should plan to run at saturation; chapter 1's table says why. Headroom is the gap you leave on purpose: plan for the bottleneck to be about 70% busy at the estimated peak, which leaves room for the estimate to be wrong and for a burst inside the peak. At 70%, one server is good for 200 views a second, so the sale needs 2,000 ÷ 200 = 10 servers' worth, and 12 with two spare. The rule bends for work nobody is waiting for: a batch job may run a machine at 95%, because a queue of jobs is not a queue of customers. The cost of headroom is paying for capacity that is idle most of the year, and chapter 19 sends the bill.

The largest machine on the price list has 96 cores, twelve times the original. Amdahl's law with 5% serial work turns twelve times the cores into 7.7 times the throughput: about 2,200 views a second at saturation and 1,540 at 70%. It is short of 2,000, and it is one machine.

Capacity planning is this page kept up to date: the forecast, the measured capacity, and the date on which one crosses the other.

Remember it as: drop five zeros to get from a day to a second, multiply for the peak, and leave a third of every machine empty.

What Ana learned

• Put a peak rate in requests per second on the first page of every design, with the multiplier and where it came from.
• Size the bottleneck for 70% busy at that peak, and state what the spare third is for.
• Believe a load test only after reading what data and what mix of requests it used.

…but

The sheet says the sale needs ten servers' worth of work and that no single machine, at any price, supplies it. On Sunday the one machine Bramble has is restarted for a security patch, and the shop is closed for eleven minutes.

Chapter 3 · Act ITwo of Everything#

The problem

The sheet from chapter 2 says the sale needs ten servers' worth of work, and Sunday's eleven-minute restart says one machine is one failure. So on Monday Priya does the obvious thing: a second Catalogue server, and a load balancer in front of the pair.

By Tuesday support has a new kind of ticket. Baskets empty themselves. Customers are signed out halfway through checkout. A seller uploads a spreadsheet of new prices, and on every second refresh the import page says no file was received. Ana works it out at the whiteboard: the basket and the sign-in live in the memory of whichever server handled the request, the spreadsheet was written to that server's disk, and the load balancer sends the next request to the other one.

Priya's fix takes one line of configuration: sticky sessions, so that each customer always returns to the same server. The tickets stop. On Wednesday a routine deploy restarts server two, and every customer who was stuck to it loses their basket at once. On Thursday the television planner is featured again in a newspaper, one server gets most of the returning customers, and it runs at 95% while its neighbour idles.

Kofi's summary: Bramble now has two machines and has gained neither capacity nor safety, because each customer still depends on exactly one of them.

The idea

There are two ways to get more work done, and they fail differently.

Scaling out only works if the servers are interchangeable. A stateless service is a bay where any loader can serve any lorry, because nothing is kept at the bay. Everything a request needs arrives with it or is fetched from somewhere shared. For Bramble that means two moves. Session state, the basket and the sign-in, goes into a store all servers can reach, keyed by the session cookie. Uploaded files go to shared storage. After that, any server can answer any request, a restart loses nothing, and sticky sessions can be switched off.

The load balancer is the dispatcher at the gate, and there are two kinds. A layer 4 balancer forwards network connections: it sees addresses and ports, is very fast, and knows nothing of what is inside. A layer 7 balancer reads the HTTP request: it can send /checkout to one pool and images to another, retry a failed request elsewhere, and terminate encryption. Bramble needs the second. How the dispatcher chooses a bay matters less than people expect, with one exception:

• Round robin: each server in turn. Fair when requests cost about the same.
• Least connections: the server with the fewest requests in flight. Better when costs vary, but every dispatcher acting on the same slightly stale counts sends a herd to the same server.
• Power of two choices: pick two servers at random and use the less busy. Nearly as even as least connections, with no herd.

The dispatcher must also know which bays are staffed. A health check is its glance: a request every few seconds to an endpoint on each server, and a server that fails is taken out of rotation. What the endpoint tests is a design decision. Too shallow, a static "OK", and a server whose database connections are all dead keeps receiving customers. Too deep, a query against the database, and one database hiccup marks every server unhealthy at once and takes the whole shop down. Test what this server alone can break.

Now count what is left that exists only once. A single point of failure is the only bridge into the estate, and redundancy is building a second one. It counts only if the two fail independently, so the copies go in different zones: separate buildings with separate power and network. Two load balancers, one per zone. Twelve servers, six per zone, so that the sale's ten survive the loss of two.

Every topology in this book is drawn this way. A solid arrow is a request that waits for its answer. A dashed arrow is a transfer nobody waits for: a queued message, a copy, a cache being filled. A box around nodes is a failure domain, labelled with its kind, and everything inside it can fail together. A green cylinder is a store that takes writes; a copy of one is a paler green. Read the diagram for what stands alone: the session store and the database each exist once, in Zone A.

Redundancy is what buys availability, usually quoted in nines:

Parts in a chain multiply: two 99.9% parts that both must work give 99.8%. Parts in parallel multiply their failures: two independent 99% servers, either of which will do, give 99.99%.

Autoscaling adds and removes servers automatically as a signal crosses a threshold. It pays for itself on daily cycles. It does not catch a spike: a new server takes minutes to start, and a sale email lands in one. For a known event, raise the floor beforehand.

Remember it as: keep nothing at the bay, staff two of everything in different buildings, and let the dispatcher send any lorry anywhere.

What Ana learned

• Move every piece of per-customer state out of the app servers before adding the second one.
• For each box on the diagram, ask what happens when it is switched off, and put the copies in different zones.
• Raise capacity ahead of a known event by hand, and leave autoscaling for the daily cycle.

…but

Twelve stateless servers pass the load test at under a quarter busy. The test stops at 800 views a second all the same, because one of the databases behind them is at 100%, and nearly all of that is a single query.

Chapter 4 · Act IIThe Query That Read Every Row#

The problem

The load test stopped at 800 views a second with twelve app servers idling and one database at 100%. The database is Ordering's, and the query log shows where the time goes: the "My orders" page, about one request in a hundred.

On a normal evening that is one or two a second, each taking four seconds, and the database has looked "about 40% busy" for a year without anyone asking why. At 800 views a second it is eight a second. Eight queries arriving each second and each living for four seconds is thirty-two running at once on sixteen cores. They queue. Each holds a database connection while it waits, the app servers run out of connections, and now the cheap queries, the ones behind checkout, cannot get a connection either. One slow page has closed the shop.

Priya's first fix is the one from chapter 3: more app servers. Kofi points out that each new server opens twenty more connections to a database that is already drowning. Her second fix is an index on every column of the orders table, "so it can't happen again".

Ana asks the question she would have asked about a boundary: what, exactly, is the database being asked to do? Kofi types one word in front of the query: EXPLAIN.

The idea

A database answers a query by following a query plan, and it will show you the plan if asked. This one says the database is reading all 40,000,000 orders to find one customer's twenty. That is a full table scan: walking every aisle to find one box.

An index is the aisle signs and the catalogue at the front desk: a second, smaller structure, kept sorted, that says where the rows are. The usual kind is a B-tree: a tree of pages, each holding a few hundred sorted keys and pointers to the pages below. Forty million rows need about four levels, so finding a customer's orders means reading four pages instead of the table.

The index has to match the query. A composite index covers several columns, and their order decides what it is good for: it is sorted by the first column, then by the second within the first. Put the column tested for equality first and the column sorted or ranged on next. An index on (placed_at, customer_id) is sorted by date across all customers and is useless here. A covering index goes one step further and carries every column the query returns, so the table is never visited. In PostgreSQL:

SELECT id, status, total_minor, placed_at
  FROM orders
 WHERE customer_id = 48213
 ORDER BY placed_at DESC, id DESC
 LIMIT 20;

-- Plan before (EXPLAIN ANALYZE, abridged):
--   Sort (top 20)
--     ->  Seq Scan on orders            rows read: 40,000,000
--   Execution time: 4,100 ms

CREATE INDEX orders_by_customer
    ON orders (customer_id, placed_at DESC, id DESC)
    INCLUDE (status, total_minor);

-- Plan after:
--   Index Only Scan using orders_by_customer   rows read: 20
--   Execution time: 0.2 ms

So why not Priya's index on every column? Because every insert and every update must also update every index on the table, and each index competes for memory. Index for the queries you have, found in the slow-query log, and delete the indexes no plan uses. The rule relaxes on a copy that is only read, such as a reporting database, where extra indexes cost disk and little else.

Three more ways one page hurts a database, all on the same screen.

Pagination. "My orders" has a page 2. OFFSET makes the database produce and discard every earlier row, so page 1,000 costs a thousand times page 1. Keyset pagination asks for "the twenty after the last one I showed you", which the index answers directly at any depth. The cost is that you can only move to the next page, not jump to page 400; for an endless list that is no loss.

-- Offset: reads 20,020 rows to return 20
SELECT id, status, total_minor, placed_at
  FROM orders
 WHERE customer_id = 48213
 ORDER BY placed_at DESC, id DESC
 LIMIT 20 OFFSET 20000;

-- Keyset: reads 20 rows, however deep
SELECT id, status, total_minor, placed_at
  FROM orders
 WHERE customer_id = 48213
   AND (placed_at, id) < ('2026-03-14 09:26:53', 39118204)
 ORDER BY placed_at DESC, id DESC
 LIMIT 20;

N+1 queries. The page fetches twenty orders, then loops and fetches each order's lines: one query plus twenty. Each is fast and each is a round trip of half a millisecond or more. Fetch the lines for all twenty in one query.

The connection pool. A database connection is expensive to open and costs the database memory while open, so each app server keeps a fixed set: the fixed number of trolleys; a loader without one waits. Twelve servers with twenty each is 240 connections, and the database has a limit. A pool turns slow queries into a visible queue, which is good, provided waiting for a connection has a timeout and the total across all servers is counted.

Underneath, two things are worth knowing by name. The write-ahead log is the goods-in book, written before anything is put on a shelf: every change is appended to a sequential file and flushed to disk before the database says "committed", and the table pages are updated later. After a crash the log is replayed. Chapter 5 copies that log to other machines. And B-trees are not the only engine. An LSM-tree never updates in place: it collects writes in memory, flushes them as sorted files and merges those in the background.

Remember it as: ask the database for its plan; if it is walking every aisle, put up a sign that matches the question.

What Ana learned

• Read the plan for every query on a page before that page meets a load test.
• Create the index the query needs, with equality columns first, and remove the ones nothing uses.
• Count connections across the whole fleet, give the pool a timeout, and bound every result.

…but

With the index in place the test passes 800 and keeps going. The arithmetic is less kind. Catalogue's database must answer 10,000 queries a second at the sale peak, 99% of them reads, and the test puts one machine's comfortable limit near 6,000. It is also still one machine, in Zone A.

Chapter 5 · Act IICopies of the Stock Book#

The problem

Two numbers and one fright.

The numbers are from chapter 4: Catalogue's database must answer 10,000 queries a second at the sale peak, and one machine is comfortable at about 6,000. Nearly all of those queries are reads. The indexes are right; there is simply more reading to do than one machine can do.

The fright comes on a Wednesday afternoon, when the disk controller in Ordering's database machine starts logging errors. Nothing is lost. But Kofi asks the room how long it would take to rebuild that database on a new machine, and nobody knows, because it has never been done. The only other copy of Bramble's orders is last night's backup, taken at 02:00. Had the disk died at four in the afternoon, fourteen hours of orders, some thirty thousand of them, paid for and not yet shipped, would have existed nowhere.

Priya's proposal covers the first problem: a nightly copy of the catalogue onto a second machine, and send the product pages there. Ana sees what is wrong with it before Kofi says anything. A copy made at night is a day old by evening, and it does nothing for an order placed this afternoon. What Bramble needs is a copy that is at most a moment behind, kept that way continuously, on a machine that can take over.

The idea

Replication is copying the stock book to other depots, continuously. One machine is the leader: the head office book, where every write is made. The others are followers: branch copies. The mechanism is chapter 4's goods-in book. The leader already appends every change to its write-ahead log; it now also ships that log to each follower, and each follower applies the same changes in the same order.

That gives two things at once. A follower can answer reads, so it is a read replica: a leader and two followers offer three machines' worth of reading. And a follower holds everything the leader has, or nearly, so one can be promoted if the leader dies. What replication does not give is more writing: every follower must apply every write, so write capacity stays that of one machine.

"Or nearly" is the whole subject. It depends on whether the leader waits.

An asynchronous follower is always a little behind. Replication lag is the copy that arrives on the evening van: usually milliseconds, and seconds or minutes when the leader is busy, which is exactly when most people are reading. Here the depot picture needs a warning. A box of notebooks is in one depot or another. A record can be in three places at once and slightly out of date in two of them. The book is copied; the stock is not.

Lag produces two anomalies that customers notice.

The customer has just paid and the shop says they have not. Some of them order again. The guarantee they needed is read-your-writes: after your own write, you see it. The cheap way to provide it is to send a customer's reads to the leader for a few seconds after they write; the precise way is to remember the log position of their write and read only from a copy that has reached it. Everyone else's reads may be stale; yours may not.

The second anomaly is time running backwards. A customer refreshes twice, the first read lands on a follower that is up to date and the second on one that is behind, and an order appears and then vanishes. Monotonic reads means never seeing older data than you have already seen, and the simple way to get it is to send each customer to the same follower every time.

Failover is a branch becoming head office. Something must notice the leader has gone, choose the follower with the most complete log, promote it, point the app servers at it, and make sure the old leader does not come back still believing it leads. Each step has a failure of its own. Declare the leader dead too quickly and a slow moment causes a failover; too slowly and the shop is closed for minutes. Promote an asynchronous follower and its missing writes are gone. And if the old leader returns unaware, there are two branches both believing they are head office, each accepting writes the other never sees. That is split brain, and chapter 14 is about preventing it properly. For now Bramble uses the database's own failover manager, with the synchronous follower as the only candidate, and rehearses it.

This chapter's design has one leader. Two others exist and Act IV meets both: multi-leader, where several machines accept writes and conflicts must be resolved, and leaderless, where any copy accepts a write and a reader asks several copies and takes the newest.

Use a single leader with followers by default. The rule gives way when writes must be accepted in two places that cannot always reach each other, and the price is conflict resolution, which is most of chapter 13.

Remember it as: one head office book, copied to the branches; a branch may be a moment behind, so never read your own entry from a branch.

What Ana learned

• Send writes to one leader, reads to followers, and a customer's own reads to the leader for a few seconds after they write.
• Keep one synchronous follower in another zone and say in writing what an asynchronous one can lose.
• Rehearse failover on a schedule, and keep backups as well as replicas.

…but

On the first Friday with replicas, Fulfilment reports an order for a notebook that is not on the shelf: two customers each bought the last one. Priya blames the follower that served the stock check and moves that read to the leader. The following Monday it happens again.

Chapter 6 · Act IITwo Clerks, One Shelf#

The problem

The Juniper A5 is a hand-bound notebook from a seller who makes them one at a time. On Friday at 12:04 there is one in stock. At 12:04:07 a customer in Leeds buys it. At 12:04:07 a customer in Cardiff also buys it. Both are charged, both receive a confirmation, and Fulfilment's picker finds one notebook and two packing slips.

Priya's theory was replication lag: the stock check had read a stale follower. She moved it to the leader, and on Monday it happened again with a fountain pen. So the copies are not the cause. Both customers read the true, current stock level from the same machine, and it was still wrong to act on.

Ana reads the code. It is four lines and each is reasonable. Read the stock level. If it is at least one, carry on. Take payment. Write the stock level back as one less than the value read. She traces two checkouts through those four lines on the whiteboard, interleaved, and the bug is in the gap between the first line and the last. Nothing in the code is wrong for one customer. It is wrong for two.

Kofi has a name for the four lines: check-then-act. He has a second observation, which is that Bramble has been running this code for three years, and the reason it has started failing now is that there are finally enough customers to arrive in the same second.

The idea

Here is the interleaving. Read it the way a picker would replay a mistake.

This is a lost update: two clerks each selling the last box. B's write was computed from a value that A had already changed, and A's sale vanished under it. The tool for preventing it is as old as databases.

A transaction is one clerk's complete stock movement: a group of reads and writes the database treats as a unit. Its promises are abbreviated ACID. Atomicity: all of it happens or none of it does. Consistency: it takes the data from one valid state to another, where "valid" means rules the application declares, such as available >= 0. Isolation: concurrent transactions do not see each other's unfinished work. Durability: once committed, it survives a crash, which is the write-ahead log's job.

Bramble's code ran inside a transaction and oversold anyway, because isolation is not one promise. It is a dial, and isolation in the depot is how much one clerk sees of another's half-finished count. Each setting permits some anomalies:

A dirty read is seeing a change that has not been committed and may be rolled back. A lost update is the notebook. Write skew is two clerks each checking the rota and both going home: each reads the same facts, each makes a decision that is safe alone, and they write different rows, so nothing is overwritten and the rule ("one first-aider on shift") is broken anyway. A phantom is a row that appears in another transaction's search after it has looked: two checkouts each count a customer's purchases of a one-per-person item, find none, and each insert one.

The levels are built on multi-version concurrency control: the database keeps several versions of each row, and a reader sees a snapshot from a moment in the past instead of waiting for writers. Snapshot isolation gives a transaction one snapshot for its whole life (PostgreSQL calls this level "repeatable read"). Serializable guarantees the result equals some one-at-a-time order, and pays for it by aborting transactions that would break that, which the application must retry.

PostgreSQL's default is read committed. So is Oracle's. Bramble's code needed more than its database was set to give. There are three fixes.

-- What the code did: check, then act, with a gap between
SELECT available FROM stock WHERE sku = 'NB-JUNIPER-A5';     -- both read 1
UPDATE stock SET available = 0 WHERE sku = 'NB-JUNIPER-A5';  -- both write 0

-- Pessimistic: the locking read
BEGIN;
SELECT available FROM stock
 WHERE sku = 'NB-JUNIPER-A5'
   FOR UPDATE;                  -- the second checkout waits here
UPDATE stock SET available = available - 1, updated_at = CURRENT_TIMESTAMP
 WHERE sku = 'NB-JUNIPER-A5';   -- run only if the value read was at least 1
COMMIT;

The locking read works and makes every checkout for that item queue. The better fix removes the gap by making the check and the act one statement:

UPDATE stock
   SET available = available - 1, updated_at = CURRENT_TIMESTAMP
 WHERE sku = 'NB-JUNIPER-A5'
   AND available >= 1;
-- 1 row updated: the notebook is yours.
-- 0 rows updated: someone took the last one; say "sold out".

This conditional update is the optimistic lock in its smallest form. The database applies it to one checkout at a time, and the caller learns that it lost from the count of rows changed: zero means sold out, and no payment is taken. The third fix, raising the level to serializable, also works, at the cost of retrying aborted transactions everywhere; Bramble keeps it for the few places where a rule spans several rows.

Locks bring one new failure. An order for a pen and a pad locks the pen row and then the pad row; another order locks the pad and then the pen. Each waits for the other for ever. That is a deadlock. The database detects it and kills one, which must retry. Lock rows in one agreed order, such as by sku, and it cannot form.

Remember it as: never read, decide and then write; make the check and the change one statement, and let the count of rows tell you who lost.

What Ana learned

• Find every read-then-write in the code and turn it into one conditional statement or a locking read.
• Write the isolation level into the design, and name the anomaly each sensitive transaction must not suffer.
• Keep transactions short, with no network call inside, and take locks in one order.

…but

Stock is right. The orders table is now the problem: 40 million orders and 120 million lines, about 150 GB with indexes on a machine with 64 GB of memory. The nightly backup takes six hours, changing a column's type takes the table away for most of a morning, and Marcus, the consultant, has a proposal to split it across four databases.

Chapter 7 · Act IIToo Big for One Depot#

The problem

Ordering's database holds 40 million orders and 120 million order lines: about 150 GB with indexes, on a machine with 64 GB of memory. Tables and indexes together no longer fit in memory, so a lookup for an old order goes to disk. The nightly backup takes six hours. Changing a column's type rewrites the table and takes most of a morning.

Priya's instinct was more memory. Marcus, the consultant Lena calls when the system creaks, proposed something larger, and Priya has built it in the test environment over two weeks: four databases, with each order stored in database number hash(customer_id) % 4. Placing an order is as fast as ever. She demonstrates it on a Thursday and Kofi asks for a week to try to break it.

It takes three days. An office-supplies reseller places 8% of all Bramble's orders, 4,000 a day, and whichever database holds that customer does a third more work than the others. Fulfilment's screen of unshipped orders, which has no customer in its query, now asks all four databases and waits for the slowest. And when Kofi adds a fifth database to see what growing looks like, % 4 becomes % 5, and four orders in every five are suddenly in the wrong place. Moving them takes the weekend.

Nothing in the design is wrong, exactly. Each of those three things is what partitioning is, and Ana wants to know which of them Bramble is obliged to live with.

The idea

Chapter 5 put the same data on several machines. This chapter puts different data on each.

Partitioning, which most engineers call sharding when the shares are on different machines, is splitting stock across depots. The partition key is the rule that says which depot holds which box, and choosing it is the whole design. A query that includes the key goes to one partition. A query that does not must ask them all.

There are two ways to turn a key into a partition. Range partitioning gives each partition a span of keys: orders by month, customers by first letter. Neighbouring keys live together, so a scan across a range is cheap, and the spans must be chosen so that load is even. Hash partitioning runs the key through a hash function and assigns by the result. Load spreads evenly whatever the keys look like, and range scans are lost.

Each of Kofi's three findings has a name.

Hash modulo N and rebalancing. With hash % N, changing N reassigns almost every key: going from four to five keeps a key in place only when both remainders agree, which is one time in five. Consistent hashing fixes this. Picture the hash values as a ring. Each node is placed at many points on it, and a key belongs to the first node point at or after its own hash. Adding a node takes over only the slices just before its points: about one key in five moves when a fifth node joins, and none of them moves between old nodes. It is redrawing depot territories so that only a few streets change hands.

public final class HashRing {
    private static final int POINTS_PER_NODE = 100;   // many points even out the slices
    private final NavigableMap<Long, String> ring = new TreeMap<>();

    public void add(String node) {
        for (int i = 0; i < POINTS_PER_NODE; i++) ring.put(hash(node + "#" + i), node);
    }
    public void remove(String node) {
        ring.values().removeIf(node::equals);
    }
    /** The owner of a key: the first point at or after its hash, wrapping round. */
    public String nodeFor(String partitionKey) {
        var point = ring.ceilingEntry(hash(partitionKey));
        return (point != null ? point : ring.firstEntry()).getValue();
    }
    private static long hash(String s) {
        try {
            byte[] digest = MessageDigest.getInstance("SHA-256")
                                         .digest(s.getBytes(StandardCharsets.UTF_8));
            return ByteBuffer.wrap(digest).getLong();   // the first 8 bytes are enough
        } catch (NoSuchAlgorithmException e) { throw new IllegalStateException(e); }
    }
}

Rebalancing is the moving itself. Systems that do it well fix a large number of small partitions at the start, say 1,024, and move whole partitions between machines, at a time an operator chooses.

Hot keys and skew. Hashing spreads keys evenly. It does not spread load evenly when one key is busier than the rest. The reseller is a hot key: the one box everybody wants, in one depot. The remedies are to split that key further (the reseller's orders by month), to serve it from a cache, or to give it a partition of its own.

Scatter-gather. A query without the partition key is sent to every partition and the answers merged. It is as slow as the slowest partition, and chapter 1's tail arithmetic applies: with four partitions, a one-in-a-hundred slow response becomes one in twenty-five. Secondary indexes face the same choice. A local index lives in each partition and covers only its rows, so writes are cheap and reads by that index scatter. A global index covers all partitions and is itself partitioned, so reads go to one place and every write touches two.

One more cost. A seller's payout that marks forty customers' orders as settled now changes rows on several machines, and doing that atomically is a cross-partition transaction: it needs a commit protocol across machines, or a saga, which the Atlas covers in chapter 16. Choose a key so that things changed together share it.

All of this is the price. So the rule is to try everything cheaper before you shard, in this order: delete or archive what nobody reads; fix the queries and indexes; buy a bigger machine; add read replicas; partition inside one database. The last is what Bramble does. PostgreSQL can split a table by range into child tables on the same machine, and Ordering's tables are now partitioned by month. The current month is 1.5 million orders and its index fits in memory. Old months are read-only, backed up once and never rewritten.

And the number at which Marcus's design becomes right: the leader comfortably takes 5,000 row writes a second, an order is about ten rows, so one machine is good for some 500 orders a second. The sale peaks at 20. Sharding across machines is warranted near 500 orders a second sustained, or when a month of data outgrows one machine.

Remember it as: choose the rule that says which depot holds which box, and put things that change together in the same depot.

What Ana learned

• Before agreeing to partition, write down the write rate and data size one machine cannot take, and compare them with today's.
• Choose the partition key from the queries, so that the frequent ones name it and things that change together share it.
• Ask for the busiest key's share of load before trusting any even-spread diagram.

…but

Orders are in monthly partitions and fit their machine again. Catalogue is next. A search for "dotted notebook A5" takes six seconds, the catalogue backup is a little over 200 GB, nearly all of it photographs, and Marcus has a second slide, proposing that Bramble replace all its databases with one partitioned store.

Chapter 8 · Act IIA Shelf for Every Kind of Thing#

The problem

Lena types "dotted notebook A5" into Bramble's search box during a board meeting and waits six seconds. The query behind it is LIKE '%dotted%' across the names and descriptions of 200,000 products. No B-tree helps: an index is sorted from the first letter, and this asks for a word in the middle, so the database reads every description, every time.

The same week, restoring the catalogue to a test machine takes four hours. The backup is a little over 200 GB. Product text accounts for less than half a gigabyte of it. The rest is the photographs, a million of them, stored in a column of the products table since the first version of the shop and sent to browsers through database connections.

Marcus's new slide has one box on it. Replace the relational databases with a single partitioned key-value store, as Amazon did: no leader to fail over, no schema to migrate, capacity added a node at a time, writes accepted even when machines are down. Priya, who spent a morning last month waiting for a column to be changed, is half persuaded.

Ana notices that the proposal answers neither the search nor the photographs. It answers a question about write volume that nobody at Bramble has asked.

The idea

A depot has shelving, bins, a cold room and a barn, each built for one kind of goods. Datastores are the same, and the way to choose one is to write down the access pattern first: how the data is read (by key, by range, by word, by relationship), how fast it is written, how big it gets, and what must always be true of it.

A search index is the catalogue you can look up by any word on the label. It is built on an inverted index: for every word, a sorted list of the products that contain it. "Dotted notebook A5" becomes three lists intersected, which takes milliseconds, and ranking and spelling tolerance come with it. It is fed from the catalogue database, a moment behind, and can be rebuilt from it at any time. That last property is the rule: the search index answers questions and owns nothing.

Object storage is the bulk barn: big things, fetched whole, by ticket. The photographs move there, the products table keeps only each one's key, and the browser fetches the image directly.

Using several kinds of store, each for what it does well, is polyglot persistence, and it has a price the table does not show. Every store is one more thing to back up, upgrade, monitor, secure and be paged for. And the same fact now lives in two places, the database and the search index, which must be kept in step and will sometimes disagree. The test is this: a new store earns its place when a question cannot be answered by the store you have, and not before.

Document and key-value stores are called schemaless. The data still has a shape; the store has merely stopped checking it. The schema moves into every piece of code that reads the data, which must now cope with every shape ever written.

Now Marcus. His argument is the one Amazon made for its shopping basket: adding an item must succeed even when disks fail and data centres are cut off, because a rejected write is a lost sale, and so the store accepts writes at any node and reconciles the differences afterwards. It is a real design, built for a service taking tens of millions of requests a day at the peak, and everything he says about it is true. Three numbers answer him.

Bramble's sale peak is 20 orders a second, about 200 row writes; one leader comfortably takes 5,000, so Bramble is using 4% of one machine's writing. Its data is 150 GB; the machine holds 1 TB. And a rehearsed failover closes writes for about thirty seconds, a few times a year, which Lena accepts. The premise of Marcus's store is that even that is too much. In exchange it gives up joins, transactions across rows and the conditional update of chapter 6: a store that accepts writes on both sides of a failure cannot promise that the last notebook is sold once.

So the answer is "not yet", with the figures at which it changes: sustained writes near 500 orders a second, or a business that would sooner sell the notebook twice than refuse a sale. And Marcus is right about one thing today. A basket is exactly the data he describes, fetched by key, merged without harm. Bramble's baskets have lived in a key-value store since chapter 3.

Remember it as: write down the questions before choosing the shelf; keep one book of record, and let every other store be a copy you could rebuild.

What Ana learned

• List the access patterns before naming a product, and move a workload to a new store only when the current one cannot answer a question.
• Keep one system of record per fact and make every other store rebuildable from it.
• Answer a proposal built for another company's scale with your own write rate, data size and tolerance for a refused write.

…but

Search is fast and the database is small. In the sale rehearsal, though, every product page still asks the database the same five questions: 10,000 queries a second for content that changes twice a day. Two followers carry 5,000 queries a second each, close to their comfortable 6,000 and with nothing spare if one fails, and Priya has ordered three more to re-read the same rows.

Chapter 9 · Act IIIThe Shelf by the Counter#

The problem

The sale rehearsal passes, narrowly. Every product page asks the database five questions, so 2,000 pages a second is 10,000 queries a second, and the answers change about twice a day. The followers sit close to their limit, re-reading the same rows.

Priya's fix goes live on a Monday: each app server keeps the pages it has built in a map in its own memory and serves them from there. The followers fall quiet. On Tuesday a week of 20% off the weekly planner ends. Some customers see £10, some still see £8, depending on which of the twelve servers answers, and those who see £8 are charged £10 at checkout, because checkout reads the database. Nothing ever removes an entry from the map. Restarting all twelve servers fixes the price and, for four minutes, sends every request back to the database at once.

Version two uses one shared cache and expires every entry after sixty seconds. The prices are right within a minute. But the database graph now has teeth: quiet, then a spike, every sixty seconds exactly. At 09:00:00 in the rehearsal the planner's entry expires, and the three hundred requests that arrive for it in the next second each find nothing, and each run the five queries.

Kofi says both versions are the classic ones, and that the shelf is the easy part. Knowing when to restock it is the design.

The idea

A cache is the shelf by the counter: answers already worked out, kept where they are quick to reach. Chapter 2's table is the reason it pays. Memory is a thousand times faster than disk, and a cached page costs one lookup instead of five queries.

There are four ways to wire a cache to a database, and they differ in who talks to the database and when.

Bramble uses cache-aside, the commonest and the easiest to reason about. Five decisions make it safe.

When does an entry leave? Three ways. Its time to live runs out: the date after which you restock it. It is removed by invalidation: pulling an item off the shelf when the aisle price changes, which here means that when Catalogue or Promotions changes a product they delete its entry. Or it is evicted because memory is full, and the usual policy drops the least recently used. Use the first two together. Invalidation makes changes prompt; the time to live catches the deletes that were missed. Never cache without an expiry. The exception is content that cannot change, keyed by a version, which may be kept for ever at no cost.

What happens when a popular entry expires? A cache stampede: every clerk running to the same aisle when the counter shelf empties.

The fix is single-flight: for each key, one caller loads and the rest wait for its answer. Add a little randomness to each time to live so that entries filled together do not expire together.

public final class ProductPages {
    private final PageCache cache;        // get(sku), put(sku, page, ttl)
    private final PageLoader loader;      // the five queries
    private final ConcurrentHashMap<Sku, CompletableFuture<ProductPage>> loading =
            new ConcurrentHashMap<>();

    public ProductPages(PageCache cache, PageLoader loader) {
        this.cache = cache; this.loader = loader;
    }
    public ProductPage get(Sku sku) {
        Optional<ProductPage> hit = cache.get(sku);            // cache-aside: the shelf first
        if (hit.isPresent()) return hit.get();
        var load = loading.computeIfAbsent(sku, k ->           // single-flight: one loader per sku
                CompletableFuture.supplyAsync(() -> cache.get(k).orElseGet(() -> {
                    ProductPage page = loader.load(k);         // re-checked: a load may just have ended
                    cache.put(k, page, ttlWithJitter());
                    return page;
                })));
        try { return load.join(); }                            // everyone else waits for it
        finally { loading.remove(sku, load); }
    }
    private static Duration ttlWithJitter() {                  // five minutes, give or take 30 s
        return Duration.ofSeconds(270 + ThreadLocalRandom.current().nextInt(61));
    }
}

This protects each server. Across twelve servers the planner page is loaded at most twelve times, not three hundred.

What about things that do not exist? A request for a product that was deleted misses the cache every time and reaches the database every time. Negative caching stores "no such product" for a short while, so that a crawler walking dead links does not become a query load.

Where does the cache live? A local cache is in each server's memory: no network hop, but twelve copies that disagree, each empty after every deploy. A distributed cache is a separate cluster all servers share: one copy, half a millisecond away, still warm after a deploy, and one more thing that can fail. Its keys are spread over its nodes by chapter 7's ring. Bramble uses the distributed one, for the reason Tuesday demonstrated.

How do you know it is working? The hit ratio is the share of reads served from the cache. At 95%, the database sees 500 of the 10,000 queries a second. The figure is touchier than it looks: falling from 99% to 98% doubles what reaches the database.

One thing a cache cannot give is the truth. It is a copy, and its consistency with the database is only ever approximate: a reader can fetch an old row, lose a race with a writer's delete, and put the old row back until the time to live removes it. So nothing is decided from the cache. The price charged and the stock sold are read from the system of record, by chapter 6's conditional update, every time.

Remember it as: keep answers on the shelf by the counter, date every one, pull it when the aisle changes, and never price a sale from it.

What Ana learned

• Give every cache entry both an expiry and an invalidation, and protect popular keys with single-flight.
• Size the database for the load it receives when the cache is cold, or have a rehearsed way to warm it.
• Read anything that decides money or stock from the system of record.

…but

The database is quiet at 2,000 pages a second. The network is not. Each page is still 1 MB, nearly all of it photographs, styles and scripts sent from London: the 16 Gbit a second that chapter 2 circled, on a connection that carries ten. And a trial customer in Toronto reports that the photographs take three seconds to appear.

Chapter 10 · Act IIICloser to the Customer#

The problem

Chapter 2's sheet had a circled line: 2,000 pages a second at 1 MB each is 16 Gbit a second, and Bramble's connection carries ten. The cache of chapter 9 made the pages cheap to build. It did nothing about sending them.

Ana asks why every page is a full megabyte every time, when most visitors have seen the logo and the stylesheet before. Kofi remembers why: a decision from last spring. A stylesheet change had gone out, browsers had kept the old stylesheet for a day as instructed, and customers saw the new page layout with the old styles: buttons over text, a checkout nobody could use. Priya's fix then was to tell browsers never to keep anything. Every visit has downloaded everything since.

Then there is Toronto. Lena has started a trial with a Canadian stationery blogger, and the first customers report three seconds of grey boxes where the photographs should be. A page needs about forty files. Each is requested from London, 80 milliseconds away and back, after an encrypted connection that takes several round trips to set up.

Priya's proposal is a second 10 Gbit line. Kofi's objection is that it puts more lorries on the same road: it does nothing for Toronto, and it pays twice to send the same photograph of the same planner to people who already have it.

The idea

Two questions decide how much traffic a site sends: may this response be reused, and how far away is the nearest copy?

Reuse is declared by the server. Every HTTP response can carry a Cache-Control header saying who may keep it and for how long. max-age gives a lifetime in seconds. public allows shared caches to keep it; private restricts it to the one browser; no-store forbids keeping it at all. A response may also carry an ETag, a fingerprint of its content. When a cached copy's lifetime runs out, the browser makes a conditional request: "I have the version with this fingerprint; has it changed?" If not, the server replies 304 Not Modified with no body. That saves the bytes, though not the round trip.

The stylesheet problem is solved by naming. With versioned asset URLs, a file's name includes a fingerprint of its content: shop.8d1e07.css, planner.4f9a2c.jpg. A given URL can never change its content, so it may be cached for a year. A new stylesheet has a new name, the page refers to the new name, and no browser has an old copy under it. The mismatch of last spring becomes impossible, and nothing ever needs to be purged.

The rule is to give static files a version in the name and a year's lifetime. It cannot apply to URLs you do not control the name of, the page itself or a link in an old email, and those take a short lifetime and an ETag instead.

Distance is solved by someone else's buildings. A content delivery network is corner shops near the customers, stocked from the depot. The provider runs caching servers in hundreds of cities: the edge. Bramble's own servers are the origin. A customer's request goes to the nearest edge, which answers from its cache if it can and otherwise fetches from the origin once, keeps the response as the headers allow, and serves the next thousand customers itself.

The arithmetic changes completely. The 950 KB of files per page never reaches London after the first request for each. What is left for the origin is page HTML at 50 KB, and with product pages reusable for sixty seconds, only the misses. Even with no page caching at all it would be 2,000 × 50 KB: 100 MB a second, under one gigabit. The second line is not needed. Toronto's forty files now come from Toronto.

The edge is also a reverse proxy: a server that accepts the customer's connection on the origin's behalf and makes its own request onward. That lets it do TLS termination, completing the encrypted handshake a few milliseconds from the customer and reusing a few long-lived connections to London, so the several round trips of connection set-up are short ones. Chapter 3's layer 7 load balancer is a reverse proxy too; this is the same idea, placed in every city.

How does a customer reach the nearest edge? Through DNS, the directory that says which depot serves your town. A browser asks for the address of Bramble's name, and with geographic routing the answer depends on where the question came from. DNS answers are themselves cached, for the time to live on the record, by resolvers Bramble does not control. A change to a record is seen only as old answers expire, so a record's time to live is how long a mistake, or a move, takes to reach everyone. Keep it short on records you may need to change in a hurry, at the cost of more lookups; lengthen it where nothing moves.

One last door. If the origin still answers anyone who finds its address, the edge can be bypassed, by a flood or by a scraper. Bramble's load balancers now accept connections only from the network's edge servers.

Remember it as: stock corner shops near the customers, give every file a name that changes when it does, and send only what nobody has seen.

What Ana learned

• Put a fingerprint in the name of every static file and let it be cached for a year.
• Decide for each kind of response whether it is public, private or never stored, and write the header down.
• Shorten a DNS record's lifetime days before the address behind it must change.

…but

The sale opens on a Monday at 09:00 and the pages fly. Orders arrive at twenty a second. Fulfilment's label worker, which turns each paid order into a courier label, manages eight. By ten o'clock the queue between Ordering and Fulfilment holds forty thousand orders and is still growing.

Chapter 11 · Act IIIThe Yard Was Full#

The problem

Monday, sale day, 09:00. Ordering takes orders at twenty a second and publishes each to a queue. On the other side one Fulfilment worker takes each order, asks the courier for a label and sends it to the warehouse printer, at eight a second. Twelve more arrive every second than leave: 43,200 an hour.

Nobody notices. The broker accepts everything it is given, its processor is idle, and every dashboard is green. At eleven the warehouse manager telephones to ask why orders placed at five past nine still have no label.

Priya starts four more workers. Five workers pull forty orders a second, and the courier's label service, which permits twenty-five requests a second, begins to answer slowly and then to refuse. Calls that took 100 milliseconds take a minute. The broker, having heard nothing about a message for thirty seconds, assumes its worker has died and hands the message to another. By noon some six hundred parcels have two labels each, and the courier charges for both.

Ana lists what went wrong. The queue hid a shortfall for two hours. Adding workers moved the bottleneck to somebody else's system. And a message delivered twice did its work twice. The Atlas mentioned backpressure in a paragraph of its chapter 14; this is the mechanism.

The idea

A queue is the waiting yard, and a worker is one of the loaders who clear it. A producer puts a message in and carries on; a consumer takes it out when it is ready. The two no longer need to run at the same speed, or even at the same time.

That makes load levelling possible: lorries arrive in a rush and are loaded at a steady pace. Workers need not match the peak. They need to beat the average, and the yard absorbs the difference at the price of delay. Bramble's peak is 20 orders a second for two hours, falling to about 5 afterwards, and the courier collects at four in the afternoon:

Two workers would meet the courier: the yard grows by four a second through the peak and drains at eleven a second after it. Bramble runs three, because with two an order waits over twenty minutes for its label. That is the first rule of queues. A queue smooths a burst; it does not add capacity. Over any long stretch, more must leave than arrive, or the yard only grows.

Several workers on one queue are competing consumers: each message goes to exactly one of them, so adding workers raises the rate. It raises it until the next bottleneck, which here is the courier at 25 a second, so three workers at 24 is the ceiling and a fourth buys nothing. It also means messages are no longer handled in order.

Watching the yard. Two measurements matter. Queue depth is how many messages are waiting. Consumer lag is how old the oldest one is, and it is the better alarm, because it is stated in the terms the business cares about: a label within fifteen minutes. Monday's failure was a lag of over an hour that no dashboard showed.

Closing the gate. An unbounded queue accepts work for ever and reports nothing, until memory, disk or patience runs out. A bounded queue has a limit, and when it is reached the producer is told: its publish blocks or is refused. That signal is backpressure: closing the gate when the yard is full, and telling the road. It is only useful if it reaches something able to slow down. A batch import can pause. A web request can be answered "try again shortly". A paid order cannot be refused, so for the labels queue the limit is set high, the alarm fires long before it, and Ordering holds unsent messages in its outbox (the Atlas, chapter 15) meanwhile. Always bound the queue. The honest exception is work that must never be turned away, and it costs exactly this: a limit sized as an alarm, and somewhere durable for the producer to wait.

Taking a message is a loan. When a worker takes a message the broker hides it from the others for a visibility timeout. If the worker acknowledges in time, the message is deleted. If it does not, because it crashed or because it was slow, the message becomes visible again and is redelivered to someone else. So every message is delivered at least once and occasionally twice. Making the work safe to repeat is the Atlas's subject: the idempotent consumer and the dead-letter queue in its chapter 15, idempotency keys in its chapter 14. One thing belongs here: set the timeout longer than the slowest honest attempt.

Two more kinds of yard. A delayed message stays invisible until a set time: "release this stock reservation in fifteen minutes unless it has been paid for". The same mechanism runs scheduled jobs. And a priority queue lets urgent work overtake: the simplest form is separate queues with their own workers, so that a million marketing emails cannot stand in front of an order confirmation.

Remember it as: a yard lets lorries arrive in a rush and leave steadily, but only if more leave than arrive and someone is watching.

What Ana learned

• Compare arrival rate with service rate before adding a queue, and work out when it empties after the peak.
• Give every queue a bound, an alarm on the age of its oldest message, and a decision about who is told when it is full.
• Size workers to the slowest thing downstream of them, and make every consumer safe to run twice.

…but

The sale's second day is calm, and Lena has an idea for its last. She wants every product page to say "3 left" and count down as people buy. Priya's sketch has each browser asking for the number once a second. And the only place that hears about each sale is a queue, which hands every message to one reader and then forgets it.

Chapter 12 · Act IIIThree Left#

The problem

Lena wants the last day of the sale to feel like a shop with a crowd in it: every product page showing "3 left", and the number falling while you watch.

Priya's sketch has each open page asking the server for the stock figure once a second. Ana does the arithmetic from chapter 2. At the sale peak 2,000 pages are opened a second, and a visitor keeps a page open for about a minute, so 120,000 pages are open at once. Each asking once a second is 120,000 requests a second: sixty times the load of the product pages themselves, for a number that changes, for most products, a few times an hour.

The second problem is where the number comes from. Every sale is announced as a message on a queue, and chapter 11's queue hands each message to exactly one worker. If the new counter service takes a message, Fulfilment never sees it. Priya adds a second queue and publishes everything twice. Then the search team wants the same messages, to hide sold-out products. Then Lena wants a live sales board. On Wednesday the board's service is deployed and needs to know what was sold on Monday and Tuesday, and those messages were deleted the moment they were handled.

Kofi says the yard was the right tool for work to be done once. This is something else: facts that many readers want, at their own pace, including readers who do not exist yet.

The idea

The log is the goods-in book kept for ever. It is an append-only sequence of records: new ones are added at the end and nothing is changed or removed by being read. Every record has a position, its offset, and each reader keeps its own bookmark saying how far it has got. Readers do not affect each other. One can be an hour behind while another is current, and a reader that did not exist on Monday can start at offset zero on Wednesday.

Retention is set by time or size, not by consumption: Bramble keeps seven days. And replay is ordinary: a reader with a bug is fixed, its bookmark is moved back, and it reads the same records again.

A log is partitioned for the reasons in chapter 7. Records with the same key, here the sku, go to the same partition and stay in order; there is no order between partitions. A consumer group is a team sharing one bookmark: its members divide the partitions among themselves, so a group scales like competing consumers, while separate groups each see every record.

Labels stay on the queue; stock changes move to the log.

Batch and stream. With the facts in a log there are two ways to compute from them. Batch processing is the nightly stocktake: take a bounded set of records, say yesterday's, and process the lot. It is simple and can be rerun, and its answer is hours old. Stream processing is the running tally: handle each record as it arrives and keep the answer current. It gives results in seconds and has to cope with what batch never sees: records arriving late or out of order, and state that must survive a restart. Since a stream never ends, totals are taken over a window: orders in each minute, or units sold in the last fifteen. Lena's daily sales report is a batch. "3 left" is a stream. Use a stream when the answer is worth less every second it is late; otherwise a batch does the same sum with a fraction of the machinery.

Processing twice. A log reader can crash after doing its work and before saving its bookmark. On restart it reads the same records again. Delivery is at least once, as it was for queues, and exactly-once delivery cannot be built. What can be built is effectively-once processing: a result that is the same however many times a record is handled. A stock event does not say "one fewer"; it says "the planner now has 3 left, change number 5,117". Applying it twice changes nothing.

Telling the browser. Four ways, in rising order of machinery:

The counter only ever flows one way, so Bramble uses server-sent events: the push servers hold 120,000 idle connections, which an event-driven server manages easily, and send a few bytes when a number changes. It is the counter display that changes without anyone asking.

Fan-out. One sale of the planner must reach everyone watching the planner. Fan-out on write does the work when the fact arrives: push to every open connection for that sku. Fan-out on read stores the latest figure once and lets each viewer fetch it when they look. Writing is cheap when few are watching and ruinous when thirty thousand are: every sale becomes thirty thousand messages. So the push is coalesced to one update per product per second, however many sales occurred.

Remember it as: a yard gives each job to one loader and forgets it; the goods-in book is kept, and every reader has a bookmark.

What Ana learned

• Put facts that several systems need on a log, keep tasks on a queue, and say which is which.
• Make every record safe to process twice, by carrying the new value or an identifier, before trusting any total.
• Choose polling through the edge, server-sent events or a WebSocket by which side speaks and how late an update may be.

…but

The sale ends as the best week Bramble has had. Lena's next slide says North America, from January, with a second set of servers in Virginia so that Toronto is not eighty milliseconds from everything. In the first test the link between London and Virginia is cut for four minutes. Both sides keep selling, from what each believes is the stock.

Chapter 13 · Act IVWhich Copy Is Right?#

The problem

Bramble opens in North America in January, with app servers and a copy of every database in Virginia. The copies are followers: every write still crosses the Atlantic to the leader in London, 80 milliseconds there and back. A checkout makes several writes, and from Toronto it feels slow.

Priya's fix is the natural one. Make Virginia a leader too. Each region accepts writes from its own customers and sends them to the other in the background, so everybody's writes are local.

Kofi tests it by cutting the link between the regions for four minutes. Both sides keep trading. When the link returns there are forty boxes of a fine black pen in the warehouse, and London has sold thirty-one of them while Virginia has sold twenty-two. Fifty-three sold, forty to ship. A customer who changed her delivery address through one region and her telephone number through the other has lost one of the two, and nothing recorded which.

Nothing malfunctioned. Ana's question, when two copies of a record disagree, which copy is right, turns out to have no general answer. It has to be decided in advance, separately for each kind of data, and the deciding has a theory.

The idea

Start with where the depot picture breaks, because the break is the subject. Physical stock cannot be in two places: forty pens are forty pens. A record saying "forty pens" can be in London and in Virginia at once, and each copy can be reduced separately. Nothing in a copied number prevents both sides subtracting from the same forty. Only a rule about who may write prevents it.

A network partition is the road between two depots being closed, so that each must decide whether to keep trading. Partitions are rare and they are not optional. What a system does during one is a choice, and the CAP theorem states the limit of that choice. As proved, it says this: in an asynchronous network, one with no shared clock and where messages may be lost, it is impossible to implement a read/write data object that guarantees both availability, meaning "every request received by a non-failing node in the system must result in a response", and atomic consistency, meaning the object behaves as if it were on a single node, answering one operation at a time.

During a partition, each operation on shared data must either be refused on one side, which keeps it consistent, or be answered on both sides from what each has, which keeps it available and lets the copies diverge. The theorem says nothing about a healthy network, nothing about speed, and nothing that makes "consistent, available, partition-tolerant: pick two" a menu. The choice exists only while the road is closed, and it can be made differently per operation.

Daniel Abadi's PACELC adds the half that matters every day: if there is a partition, a system trades availability against consistency; else, when all is well, it trades latency against consistency. Toronto's slow checkout was that second trade. A write that must be confirmed by London costs 80 ms whether or not anything is broken.

"Consistency" here names a scale, and three points on it are worth knowing.

• Linearizability: everyone reads the one book at head office. Each operation appears to take effect at a single instant between its start and its finish, so once a write has completed, every later read sees it. This is the C in CAP.
• Causal consistency: operations that depend on one another are seen by everyone in the same order, so a reply never appears before its question. Unrelated operations may be seen in different orders.
• Eventual consistency: the copies agree once the vans have done their rounds. If writes stop, all copies converge. It promises nothing about what a read returns meanwhile.

Two of these words are each used for two things.

A database can offer either of the last two without the other.

Quorums are how a system without a leader approaches consistency. Keep N copies. A write succeeds when W copies confirm it; a read asks R copies and takes the newest. If R + W > N, every read set overlaps every write set: enough signatures from enough depots that any two decisions must share a signatory. With N = 3, W = 2 and R = 2, one copy may be unreachable and work continues. Choosing R and W per request is tunable consistency: W = 1 is fast and may be lost; R = 1 is fast and may be stale. The overlap is weaker than it sounds. The paper that popularised it says only that R + W > N "yields a quorum-like system", and with concurrent writes and nodes standing in for failed ones it is not linearizable.

When copies have diverged, something must reconcile them. Last write wins keeps the version with the later timestamp and throws the other away, silently; chapter 15 shows how arbitrary "later" is. Siblings keeps both versions and hands them to the application, which merges them: a basket becomes the union of both baskets.

Bramble decides per operation.

Browsing and baskets choose availability and low latency: they are served locally, may be seconds stale, and merge without harm. Stock and payment choose consistency: one leader, in London, and during a partition Virginia can fill baskets and cannot check out. North America is a tenth of orders, 5,000 a day, so a four-minute partition delays about fourteen orders. That is the price, and it is cheaper than thirteen pens that do not exist. The answer changes when North America has a warehouse of its own, or passes half of all orders: stock held there would have its leader there, and no record would have two writers.

Remember it as: when the road between depots closes, decide for each kind of entry whether to stop writing or reconcile later, and decide before it closes.

What Ana learned

• Sort operations into those that may be stale and those that may be refused, and write the list down before adding a second site.
• Give every piece of data one place where it is written, unless it merges harmlessly.
• Ask of any eventually consistent store what makes the copies agree again, and what is thrown away when they do.

…but

So stock has one leader, in London, and everything depends on there being exactly one. In February the machine running that leader freezes for forty seconds. The failover manager promotes the follower. Then the old leader wakes, unaware that anything has happened, and carries on accepting orders.

Chapter 14 · Act IVTwo Head Offices#

The problem

At 14:11 on a Tuesday in February the virtual machine running the stock leader in London stops. It has not crashed. The host underneath it is being moved, and for forty seconds the machine executes nothing at all.

The failover manager does its job. Thirty seconds without a heartbeat means the leader is dead, so it promotes the follower in Zone B and begins repointing the app servers.

At second forty the old leader resumes. From its point of view no time has passed. It still believes it is the leader, and it is configured to carry on alone if its follower stops answering. Three app servers still hold connections to it, and it accepts their writes. For ninety seconds, until someone notices, Bramble has two leaders. Eleven orders are recorded on a machine whose changes go nowhere, and two of them are for stock the new leader has already sold.

This is the split brain that chapter 5 named and postponed. Priya's first fix is to raise the timeout to two minutes, so that a forty-second pause is survived. Kofi points out the two costs: every real failure now closes the shop for two minutes, and nothing limits a pause to forty seconds. Her second idea is a second manager to watch the first. Ana asks who decides when the two managers disagree, and nobody answers.

The idea

Every node proves it is alive by sending a heartbeat every second or so, and failure detection is noticing silence. The trouble is that silence has three causes. The node is dead, or it is paused, or the network between you and it is broken. From outside they look the same. A timeout is therefore a guess, and a design that is only correct when the guess is right is not correct.

Two mechanisms make the guess safe to get wrong. One ensures that only one new leader can be chosen. The other ensures that an old leader, whatever it believes, cannot do damage.

Choosing one leader: majority. Let the decision be made by a fixed set of voters, and require more than half to agree. Any two majorities of the same set share at least one voter, and a voter votes once per round, so two different leaders cannot both be chosen. It is chapter 13's overlapping signatures, applied to an election. Three voters survive the loss of one; five survive two. On the losing side of a partition no majority exists, so no leader is elected there and that side stops: consistent, not available.

Use an odd number of voters, in at least three failure domains. Four voters tolerate one loss, as three do, at the cost of another machine. Two is the worst number: a majority of two is two, so when either is lost, or the link between them, the survivor cannot tell whether it is the majority, and must either stop or risk a split brain.

Consensus is the general problem: the depot managers agreeing, by majority, on one head office. More exactly, it is getting several machines to agree on a value, such as who leads or what the next entry in a log is, so that a decision once made is never unmade, though machines crash and messages are lost. Paxos is the classic algorithm. Raft is the one written to be understood, and it has three parts. Leader election: time is divided into numbered terms, and at most one leader can be elected in a given term. Log replication: the leader appends each change to its log and counts it committed when a majority hold it. Safety: an entry once committed appears in the log of every later leader. Such systems keep working while a majority of the servers are running and can reach one another.

Nobody at Bramble should write this. A coordination service (etcd, ZooKeeper and Consul are the usual ones) is a small store that runs consensus among three or five voters and offers the results as simple operations: elect a leader, hold a lease, keep a little configuration. Bramble's database failover is rebuilt on one, with voters in Zone A, Zone B and a third zone that holds a vote and no data.

Stopping the old leader: leases and fencing. Leadership is granted as a lease: an appointment that expires unless renewed. A leader that cannot renew must stop. But the paused machine of 14:11 did not know its lease had expired; it woke and wrote. A lease limits how long two leaders can believe they lead. It does not stop the stale one acting.

That takes a fencing token: the numbered key issued with each appointment; the lock refuses an older number. Each time the coordination service grants leadership it issues a number larger than any before. Every write carries the writer's token, and the store that receives writes remembers the highest it has seen and refuses anything lower.

public final class FencedStockRepository implements StockRepository {
    private final Map<Sku, StockLevel> stock = new HashMap<>();
    private long highestTokenSeen = 0;        // stored with the data in a real system

    /** Every write names the fencing token issued with the writer's lease. */
    @Override
    public synchronized void save(StockLevel level, long fencingToken) {
        if (fencingToken < highestTokenSeen) {
            throw new StaleLeaderException(
                    "token " + fencingToken + " is older than " + highestTokenSeen);
        }
        highestTokenSeen = fencingToken;
        stock.put(level.sku(), level);
    }
}

Notice where the check is. The coordination service cannot protect the store; only the store can refuse a write. That is the limit of every distributed lock. Inside one process a lock is absolute. Across machines, the holder can be paused beyond the lock's expiry and resume in the middle of its work, so the lock is a strong hint and the token is the guarantee. A lock without one is adequate for avoiding duplicated effort, and inadequate where a second writer would corrupt something.

With both mechanisms the split brain is resolved: the majority rule means only one side can appoint a leader, and the token means the other side's late writes are refused.

Remember it as: a majority appoints one head office and numbers the appointment, and the lock refuses any key carrying an older number.

What Ana learned

• Let a consensus-backed coordination service with an odd number of voters in three places choose the leader, and write none of it yourself.
• Treat every timeout as a guess, and make the design safe for the case where the node was only slow.
• Make the store that receives writes check a fencing token, because only it can refuse a stale leader.

…but

There is one leader now, provably. A week later support escalates an order whose cancellation is timestamped a second before the order was placed, and a customer whose change of address was accepted and then vanished. Both involved two servers writing in the same moment. In both, the later clock won, and the later clock was wrong.

Chapter 15 · Act IVWhat Time Is It?#

The problem

Support has two cases that make no sense.

The first is order 49206518. The customer placed it and cancelled it a second later, having ordered the wrong size. The record says it was cancelled at 10:15:01 and placed at 10:15:02. The nightly reconciliation job, which sorts events by timestamp, found a cancellation for an order that did not yet exist, ignored it, and the wrong size was shipped.

The second is a change of address. The customer typed "12 Mill Lane", saw the mistake, and corrected it to "14" a second later. The profile says 12. Profiles are merged field by field, as chapter 13's exercise advised, and within a field the later timestamp wins. Here the later write lost.

Ana finds the cause in an afternoon. The two requests in each case were handled by different servers. One server's clock was two seconds fast; a synchronisation job had been failing quietly for a month. So the earlier event carried the later time.

Priya's fix is to repair the job and add an alarm, and that is worth doing. Kofi's point is that it changes the size of the problem and not its nature. Synchronised clocks agree to within some milliseconds on a good day and drift apart when anything is wrong. Two events closer together than the error cannot be put in order by reading clocks, and the two events that matter are always close together.

The idea

A computer has two kinds of clock, and most bugs with time come from using the wrong one.

The wall clock is the clock on each depot's wall, all slightly different. The difference between two machines' wall clocks is clock skew, and it is never zero. The monotonic clock is a stopwatch: useless for saying when, exact for saying how long.

So wall-clock timestamps from different machines cannot settle which of two close events came first. This is what goes wrong when they are asked to:

Last write wins is therefore a way of losing data. With skewed clocks the wrong write wins. With perfect clocks, of two writes made at the same moment, one is still thrown away and nobody is told. It is acceptable only for values where a lost concurrent update does no harm, such as "last seen at".

The alternative is to stop asking what time it was and ask what could have caused what. A logical clock is numbering the entries instead of timing them. In the simplest, the Lamport timestamp, every node keeps a counter. It adds one for each event, sends the counter with every message, and on receiving a message sets its own counter to one more than the larger of the two. The guarantee is one-directional: if event a could have influenced event b, then a's number is lower than b's. The converse does not hold. A lower number does not mean "happened first"; the two events may be concurrent, with no connection between them. Breaking ties by node name gives a total order that everyone agrees on, and Lamport's own paper calls that order somewhat arbitrary. It is enough for the first case: carried on each message, the counter puts a cancellation after the order it refers to.

A vector clock keeps one counter per node and carries the whole set. Comparing two of them gives three possible answers: this one came before, that one came before, or neither, which means they were concurrent. That third answer is what last write wins hides. It is how a store knows it holds siblings, in chapter 13's sense, and must merge them instead of choosing.

A CRDT, a conflict-free replicated data type, is a data structure designed so that concurrent changes always merge to the same result on every copy, with no coordination: a counter, a set, a basket that takes the union. And TrueTime is the opposite bet, made by Google's Spanner: put atomic clocks and satellite receivers in every data centre, have the clock report an interval that is certain to contain the true time, and make each transaction wait out the uncertainty, a few milliseconds, before it commits. Then a transaction that commits before another starts always has the earlier timestamp. It works, with special hardware and a wait on every write.

Time has one more job, which is naming things. A unique identifier is a docket number no two depots can issue twice.

The Atlas wrote OrderId.next() and never said what was inside. This is what Bramble puts there:

public final class OrderIdGenerator {
    private static final long EPOCH = 1_704_067_200_000L;    // 2024-01-01T00:00:00Z in milliseconds
    private final long machineId;                             // 0 to 1,023, unique per running process
    private long lastMillis = -1;
    private long sequence = 0;                                // 0 to 4,095 within one millisecond

    public OrderIdGenerator(long machineId) { this.machineId = machineId; }

    public synchronized OrderId next(long wallClockMillis) {
        if (wallClockMillis < lastMillis) {                   // the wall clock has stepped back
            throw new IllegalStateException("clock moved back; refusing to issue identifiers");
        }
        if (wallClockMillis == lastMillis) {
            if (sequence == 0xFFF) throw new IllegalStateException("4,096 issued this millisecond");
            sequence++;
        } else {
            sequence = 0;
            lastMillis = wallClockMillis;
        }
        return new OrderId(((wallClockMillis - EPOCH) << 22) | (machineId << 12) | sequence);
    }
}

Forty-one bits of time, ten of machine, twelve of sequence. The machine numbers come from chapter 14's coordination service. The identifiers sort roughly by creation, which keeps the index compact, but "roughly" is the word: they order rows for storage, and they do not say which of two orders was really first.

Remember it as: every depot's wall clock is slightly wrong, so number the entries instead of timing them, and never let a clock decide whose write survives.

What Ana learned

• Never order events from different machines by wall-clock time; carry a counter or a version instead.
• Use the monotonic clock for every timeout and duration, and the wall clock only to show people a time.
• Generate identifiers on any machine from time, a machine number and a sequence, and treat their order as approximate.

…but

The clocks are distrusted and the identifiers sort. Then Lena has news: the illustrator has said yes. Kofi books a rehearsal for the drop at a tenth of its size, and it is over in four minutes: one script and a wave of retries, arriving together, flatten checkout.

Chapter 16 · Act VThe Doorman#

The problem

The drop is 10,000 numbered notebooks by an illustrator with four million followers, on sale at 09:00 on a Friday in May. Marketing expects half a million people in the first minute.

Kofi's rehearsal runs at one-tenth scale on a Tuesday night, with a test product and rented load generators. It fails in three ways in four minutes.

First, one generator, playing a scalper's script, sends 400 add-to-basket requests a second from a single account. Bramble has a limit, which Priya added in the spring: 50 requests a second per account, counted on each app server. There are twelve servers, so its real allowance is 600, and all 400 pass.

Second, checkout slows, and the shop's own page code does what it was written to do: retry after one second, then two, then four. Every browser that failed in the same second retries in the same second. The servers begin to recover, are hit by the returning wave, fall over, and are hit by the next, which is larger.

Third, the servers treat everything alike. Requests for recommendations, for the live stock counter, and from a search engine's crawler queue beside payments, and all of them time out together.

The Atlas gave rate limiting and load shedding a paragraph in its chapter 14. The drop needs the mechanisms.

The idea

Three defences, from the outside in: limit each client, protect the service whoever is asking, and make clients behave.

Rate limiting is the doorman's allowance per haulier: a cap on what one client may ask, so that a script spends its own allowance and nobody else's. Four algorithms are in use.

The token bucket is the usual choice, because real clients are bursty and it says plainly how bursty they may be.

public final class TokenBucket {
    private final long capacity;            // the largest burst allowed
    private final double refillPerNano;     // the steady rate
    private double tokens;
    private long lastRefill;

    public TokenBucket(long capacity, double tokensPerSecond, long nowNanos) {
        this.capacity = capacity;
        this.refillPerNano = tokensPerSecond / 1_000_000_000.0;
        this.tokens = capacity;
        this.lastRefill = nowNanos;
    }
    /** True if the request may pass; false means answer 429 with a Retry-After. */
    public synchronized boolean tryAcquire(long nowNanos) {
        tokens = Math.min(capacity, tokens + (nowNanos - lastRefill) * refillPerNano);
        lastRefill = nowNanos;
        if (tokens < 1.0) return false;
        tokens -= 1.0;
        return true;
    }
}

The algorithm is the small part. Where the limit is enforced decides whether it works. The count must be one count: held at the edge, or in a store every server shares, never in each server's memory. And it must come first. A limiter that runs after the request has been authenticated, parsed and looked up in the database has already spent what it was meant to save. Enforce per-client limits at the edge, before the origin sees the request. A limit that needs to know something only the service knows, such as a seller's plan, has to sit in the service, and costs a lookup per request.

Load shedding protects the service from the total, whoever is sending it. It is turning lorries away at the gate before the yard jams. Chapter 1's table says a saturated server serves everyone badly; a server that refuses the excess in a millisecond serves the rest well. Admission control is the decision at the door: count the requests in flight, and when the count passes what the service can finish in time, refuse new ones at once. Refusal must have an order. Bramble's is payments and checkout first, then browsing, then recommendations and the live counter, then crawlers. Shedding the last two classes is graceful degradation: serving the essentials and closing the gift-wrap counter. The page still sells notebooks; it stops suggesting pens.

Clients must behave, including Bramble's own. A failed call should be retried only if it is safe to repeat, which the Atlas's chapter 14 covers under idempotency keys, and then with three rules. Exponential backoff: wait longer after each failure, so a struggling service gets room. Jitter: randomise each wait, because backoff alone keeps clients that failed together in step; choosing a random wait between zero and the backoff ceiling spreads the returning wave flat. And a retry budget: retries may be at most a set share of traffic, say a tenth, because three layers each making three attempts turn one failure into twenty-seven requests.

One client-side technique adds load on purpose. A hedged request attacks tail latency: if a read has not answered by the time 95 in a hundred usually have, send a second copy to another server and take whichever returns first. It costs about 5% more requests and cuts the p99 sharply. It is only for reads that are safe to run twice, and it must stop when the service is busy, or it is a retry storm with a better name.

For the drop none of this is enough alone. Half a million arrivals in a minute is 8,300 a second, four times the largest load Bramble has served, and only 10,000 of those people can succeed. A virtual waiting room is numbered tickets and a car park outside the estate. Visitors are held on a static page served from the edge, each is given a place in line, and they are admitted at the rate checkout has been tested to serve: 100 a second, five times the sale's peak. Everyone else sees an honest position and then an honest "sold out". The waiting room is not for ordinary days. It is one more component in front of everything, and it earns that position only when arrivals are known to exceed capacity.

Remember it as: give every haulier an allowance at the gate, turn lorries away before the yard jams, and tell them not to all come back at once.

What Ana learned

• Keep one count per client, at the edge, and check it before doing any other work.
• Decide the order in which requests are refused before the day they must be.
• Give every client backoff, jitter and a retry budget, starting with the shop's own pages.

…but

The second rehearsal passes: the crowd waits outside, the script gets its allowance and no more, and checkout takes a steady hundred a second. Kofi asks what nobody has asked. All of it runs in London. There is a document, written in March, for moving the shop to Virginia if London is lost, and nobody has ever followed it.

Chapter 17 · Act VThe Day the Region Went Dark#

The problem

Wednesday, 23:00, ten weeks before the drop. Kofi blocks every network path into and out of London, as if the region had vanished, and starts a stopwatch. The document from March says Virginia will be serving customers within thirty minutes.

The first step is to open the document. It is on the wiki, and the wiki runs in London. Kofi has a printout.

Step two works: the copy of the orders database in Virginia is promoted to leader by hand, with London cut off at the edge so that nothing can reach the old one. It was replicating asynchronously and was six seconds behind. At the evening rate of an order and a half a second, nine orders that customers saw confirmed do not exist in Virginia.

Step three is to grow Virginia from the two app servers that serve North America to twelve. New servers fetch the application from the image registry. The registry is in London. So is the pipeline that could publish the image somewhere else. Priya finds a copy on a laptop.

Step four, changing the DNS records, works in five minutes, because chapter 10 shortened their lifetime.

Virginia is serving at 02:40: three hours and forty minutes. On Thursday Kofi tries the last line of the document, restoring the orders database from backup. It takes nine hours, and the backup from the second of June will not open at all.

Nothing was wrong with any single component. The plan needed the place it was a plan for losing.

The idea

A failure domain is everything that floods together. A machine is one. So is an availability zone, a building with its own power, cooling and network, and a region, a town with several such buildings some kilometres apart. So, less obviously, is anything shared: one configuration service, one registry, one deploy that goes everywhere at once. Blast radius is how much stops when a domain fails. The design work is to put copies in different domains and then hunt for whatever still joins them.

Zones and regions buy different things. Zones are a millisecond or two apart, close enough for the synchronous replication of chapter 5, so losing a zone need lose no data. Regions are far apart, 80 ms for Bramble, so replication between them is asynchronous and the standby is always a little behind. That gap has a name, and so does the stopwatch.

Both are business decisions with a price. Lena accepts ten seconds of orders at risk, since the payment provider's records allow them to be rebuilt, and will not accept an afternoon closed.

Read it as chapter 3 taught: look for what stands alone. The registry box exists once, and Virginia has one zone and a sixth of the servers.

There are two shapes for a second region.

Bramble is between the two: Virginia is active for reading and passive for writing. That is a legitimate place to stay. Full active-active is the right answer only for data that has a conflict design: each record with a single home region, or values that merge. Chapter 19 puts a figure on when it pays.

Wednesday's lessons become rules.

Nothing needed for recovery may live only in the region being recovered from. Registry, pipeline, secrets, documents and backups are copied to Virginia, and the printout stays in the drawer.

The standby must be able to take the load without asking anyone. Scaling up during a disaster depends on the things that are failing. Virginia's two servers already cover a normal peak of 150 pages a second; it stays that size and is raised by hand before an event. The same holds between zones: twelve servers across two zones leaves six if one is lost, good for 1,200 pages a second and short of a sale's 2,000. Priya doubles the floor to twelve per zone. The price is idle machines, which chapter 19 will question.

A backup is the copy in the fireproof safe, and a restore test is opening the safe to check you can read it. Replicas protect against losing a machine. Only a backup protects against a mistake, because a mistake is replicated. A backup that has never been restored is a hope. Bramble now restores one every month, to a different region, and times it.

Failover has a second half. Failback is returning home: London comes back holding nine orders Virginia never saw, must be reconciled, rebuilt as a follower and promoted at a quiet hour. It is rehearsed as well.

Two wider ideas. Data residency: the law in some places requires certain personal data to stay within a jurisdiction, which constrains where copies and backups may be put, so the question goes to a lawyer before the second region is chosen. And cells: instead of one large system per region, run several complete, independent copies, each a self-contained depot serving its own customers, so a fire stays in one building. A bad deploy or a runaway customer then harms one cell's share of customers. A cell-based architecture costs a routing layer and many of everything, and it suits platforms with hundreds of servers and customers who must be shielded from each other. Bramble has two dozen servers; its two regions are cells enough.

The second rehearsal, a week later, takes fourteen minutes.

Remember it as: know what floods together, keep the safe in another town, and open it before the fire.

What Ana learned

• State the recovery point and recovery time as numbers the founder has agreed to, and measure both in a rehearsal.
• Keep every tool the recovery needs, including the plan, outside the region it recovers.
• Restore a backup on a schedule and practise failing back as well as failing over.

…but

Lena hears "fourteen minutes" and asks whether that is good. Nobody can answer. Bramble has never written down how much failure it is prepared to accept, so it has no way to say whether fourteen minutes is a triumph or a breach.

Chapter 18 · Act VPromises with Numbers#

The problem

"Is fourteen minutes good?" Lena's question about the failover has no answer, and the week supplies three more like it.

Priya was on call and was paged thirty-one times: a cache node's processor over 80%, a disk at 70%, a follower two seconds behind. None of the thirty-one touched a customer. On Thursday evening the payment provider slowed down and three checkouts in every hundred failed for forty minutes. Nobody was paged, because no alert watches whether checkouts succeed.

Then sales forwards a contract for signature. The office-supplies reseller, 8% of Bramble's orders, has been promised "99.99% availability" with refunds for any breach. Nobody in engineering was asked what Bramble achieves, because nobody in engineering knows.

Ana sees one gap behind all three. Bramble measures its machines in detail and has never stated, as a number, what its customers are owed. So it cannot say whether an outage was acceptable, whether a page was worth waking someone for, or whether a contract can be kept. The Atlas introduced service level objectives in a paragraph of its chapter 20. This chapter is how they are set and what they are for.

The idea

Three terms, routinely confused, and each built on the one before.

A service level indicator must be measured where the customer is, for chapter 1's reason, and it must count what the customer cares about: a checkout that returns an error page quickly is fast and bad. A service level objective turns it into a promise. A service level agreement is always looser than the objective behind it, so that the alarm rings before the lawyer does. The contract on Lena's desk has it backwards.

The objective is 99.9% and not 100% for a reason. A hundred percent is unreachable: the customer's own connection is less reliable than that. By a common rule of thumb, each added nine costs ten times the last. And a system that may never fail may never be changed. The gap is useful. It is called the error budget: the late deliveries you may still afford this quarter.

Bramble takes 1,500,000 orders in thirty days. At 99.9%, the budget is 1,500 failed checkouts, or about 43 minutes of complete outage. Now Lena's question has an answer. Fourteen minutes is a third of a month's budget: affordable twice in a month, not four times. Thursday's incident, 108 failed checkouts, spent about 7%.

A budget only matters if spending it has consequences, agreed beforehand. Bramble's error-budget policy is two sentences. While budget remains, teams release as they please and may spend it on purpose. When it is gone, releases stop, except fixes, until the thirty-day figure recovers. Both halves matter: the first is permission to take risks, and the second is the only thing that makes the objective more than a wish.

What to watch. For any service, four measurements cover most of what goes wrong, the golden signals: latency, traffic, errors and saturation. Chapter 1 met three of them. They belong on the dashboard.

What to be woken for. Alert on symptoms, not causes. A symptom is the customer being hurt: checkouts failing, pages slow. A cause is a full disk or a busy processor, and there are hundreds, most of which hurt nobody on any given night. Causes belong on dashboards and in tickets. The rule gives way for a cause that is certain to become a symptom soon, such as a disk that will be full in four hours, and the price of the exception is a page that might have waited.

The symptom worth paging on is the budget being spent too fast. Burn rate is the speed of spending relative to the speed that would use exactly the whole budget in the window. At a burn rate of 1, the budget lasts thirty days. Thursday's 3% failure rate against a budgeted 0.1% was a burn rate of 30: at that pace the month's budget is gone in a day. Bramble pages when the rate has been above 14 for an hour, which is 2% of the month's budget gone, and opens a ticket when it has been above 1 for three days. Priya's thirty-one pages become none, and Thursday becomes one.

Testing the promise. An objective says what should happen when things fail. Chaos engineering is finding out, by causing failures deliberately and watching: stop a server, cut a link, slow a dependency, with a stated expectation, a limited blast radius and a way to stop. A game day is the same thing as a scheduled exercise for people as well as machines; chapter 17's rehearsal was one. It is the fire drill, and the error budget is what pays for it.

Learning from the breach. After an incident, a blameless postmortem records what happened, what it cost in budget, what made it hard to notice and what will change. Blameless is a method, not a courtesy. It assumes everyone acted sensibly on what they knew and asks what the system told them. People who expect blame stop reporting what they saw, and the next incident takes longer.

Remember it as: measure at the gate, publish a promise short of perfect, and spend the difference on purpose.

What Ana learned

• State what customers are owed as a ratio with a target and a window, and measure it where they are.
• Agree in advance what happens when the error budget runs out, and keep every contract looser than the objective.
• Page people only when the budget is burning fast, and send everything else to a dashboard.

…but

A month on, every objective is met, and by a wide margin: checkout has run at 99.99% since the target was set. The same week finance sends the infrastructure bill. It was £60,000 a month before the sale and is £105,000 now. And Marcus has a proposal for the drop: three regions, all active.

Chapter 19 · Act VThe Bill#

The problem

Finance sends one line: infrastructure was £60,000 a month before the sale and is £105,000 now. Bramble takes 1,500,000 orders a month, as it did then. Each order used to cost four pence to serve. It now costs seven.

Nobody decided to spend the extra £45,000. Ana and Kofi spend a Friday afternoon finding it, line by line, and each line has a history. The app-server floor was doubled from 12 to 24 after chapter 17's rehearsal, so that either zone could carry a sale alone, and there is no sale; the fleet peaks at 2% busy. Each busy database gained three more followers in the fortnight before chapter 9's cache went in; with nineteen reads in twenty now answered by the cache, they idle. Six machines run a stream-processing cluster for the live sales board, which Lena opens twice a week. Three years of backups sit on the fastest storage there is. And £8,000 a month is for data leaving the provider: debug logs, uncompressed, shipped to an outside logging service.

Marcus's proposal arrives the same afternoon, and it is the opposite of a saving. For the drop, and for good, Bramble should run in three regions, all active, every write accepted wherever it lands. His estimate is £30,000 a month and two engineers.

The idea

Cost is the rent, paid on every depot whether or not lorries come. The first tool is a unit. A monthly total says nothing; cost per order, or per request, can be compared across months and attached to decisions. Four pence became seven with no change in what customers received, and a design review can now ask what a proposal adds to the figure.

The second tool is a rule for telling headroom from waste. Chapter 2's spare third and chapter 17's standby in Virginia are capacity that a number justifies: a peak, a recovery time. Waste is capacity whose number has gone. So Kofi's list has three columns: what load justified this, what that load is today, and who would notice if it went.

That is £39,000 removed: £66,000 a month, 4.4 pence an order. Priya removes the followers herself; she added them.

Over-provisioning versus autoscaling. A fleet fixed at the size of its worst day is idle on every other day. Autoscaling follows the daily cycle, and chapter 3's rule covers the rest: raise the floor by hand before a known event. The half that was forgotten is the other direction. Every temporary increase gets an end date when it is made.

Storage tiers and egress. Providers sell storage in tiers: fast and dear, slower and cheaper, and archival, which costs almost nothing to keep and a fee to read. Data should move down as it ages; chapter 7's old order partitions are candidates. Egress is the charge for bytes leaving: to the internet, to another region, to a third party. It appears on no architecture diagram, and any design that moves data across a boundary pays it for ever.

Managed versus self-run. A managed database or queue costs more per unit than the same software on your own machines, and comes with someone else's on-call. The usual rule is to rent the managed service until its bill exceeds the cost of people who could run it well. The rule bends for large, steady workloads, where owning can be much cheaper, and what it costs is those people's attention, which was otherwise going into the product.

Operational complexity as a cost. The stream cluster's invoice was £7,000. Its real cost included the two engineers who understood it, its upgrades and its pages. Every component carries that second bill, and it is why chapter 8 counted stores and chapter 17 counted regions. Simplicity is capacity too: the capacity to change things.

Scaling back down is therefore a practice with a date in the diary: each quarter, every component is asked for its number again.

Now Marcus. His case is the one Netflix made: a standby is never truly tested, so make every region live and failover disappears. Three numbers answer it. Bramble's objective is 99.9%, a budget of 43 minutes a month, and a rehearsed failover takes 14: the present design meets the promise. Fourteen dark minutes at half an order a second is 420 orders, about £10,000 at an average order of £24; twice a year, £20,000, against £360,000 a year for his design. And stock cannot be merged, as chapter 13 showed, so "every write accepted anywhere" would mean either consensus across continents on every checkout or the chance of selling the same notebook twice. For the drop itself, which lasts under two minutes, there is a cheaper protection: if London is unhealthy at 08:45, the drop moves to Saturday.

So: not yet. His design is right when a regional outage costs more than the design does, which at today's prices is about eighteen times Bramble's volume, or when the objective must be 99.99%, whose monthly budget of four minutes no failover fits inside.

Remember it as: rent is paid on every depot whether or not lorries come, so ask each depot, every quarter, which lorries it is for.

What Ana learned

• Track the cost of one order, and ask of every proposal what it adds to that figure.
• Give every component a number that justifies it, and every temporary increase an end date.
• Answer a proposal for more scale with what an outage costs, what the design costs, and the volume at which the two meet.

…but

The bill is back to 4.4 pence an order. The drop is nine days away: 10,000 notebooks, half a million people at 09:00, and so far a list of mechanisms that have never been drawn as one design. Lena wants to see it on Thursday. Marcus will be presenting his.

Chapter 20 · Act VThe Drop#

The problem

Thursday, eight days before the drop. Marcus presents first, and his slide is handsome: fourteen boxes. The 10,000 notebooks are split into sixteen partitions across two regions, so that no single record is hot. Purchase requests are appended to a log, and a stream processor allocates units from it. Reservations live in a new key-value store that accepts writes in either region. Both regions are active.

Lena asks one question: will it sell exactly 10,000? Marcus says the partitions reconcile within seconds, so eventually, yes.

Ana has no diagram. She has one page, and it starts with what must be true: each notebook is numbered, so 10,001 is not an apology, it is a second customer holding number 4,117. She asks for an hour at the whiteboard and says she will draw nothing for the first twenty minutes.

What Bramble has never done is use the previous nineteen chapters in order, starting from the requirement and ending with what it chose not to build. Kofi says that order is the method, and that the method matters more than any box.

The idea

A design is produced in eight steps, and the diagram is the fifth.

1. Requirements. What must be true, and what may give. Sell 10,000 numbered notebooks, one per customer, never 10,001. Tell each buyer within seconds whether they have one. Hold it for five minutes while they pay; release it if they do not. Keep the rest of the shop working. Serve people roughly in order of arrival. May give: a buyer in Toronto waits a little longer; nobody can reserve during a failover.

2. Estimates. Half a million arrivals in a minute is 8,300 a second, four times the sale peak. Only 10,000 can succeed, so 98% of arrivals need a cheap, honest "no". Checkout has been tested at 100 a second, five times the sale's 20, so the notebooks are reserved in about 100 seconds. One stock row takes 100 updates a second against a leader good for 5,000: 2%.

3. Interface. POST /drop/reservations returns a reservation and its expiry, or "sold out", or "slow down" with a time to retry. POST /orders pays for a reservation. GET /drop/status returns the number left. Both writes are idempotent; the Atlas's chapter 14 covers idempotency keys.

4. Data model. One stock row for the notebook. A reservations table: customer, expiry, state, unique per customer. Taking a reservation is chapter 6's conditional update and one insert, in one short transaction. This is a reservation with expiry: stock is decremented at once, and a delayed message from chapter 11 fires five minutes later and, if the reservation is still unpaid, marks it expired and adds one back.

5. High-level design. Now the boxes, each one from a chapter.

6. Deep dives. The stock row is a hot key on purpose: 100 short updates a second, with no network call inside the transaction, is well within one row's reach; it stops being so somewhere past a thousand. Once the count reaches zero, "sold out" is served from the edge, so the other 490,000 arrivals never touch London. The number left is polled through the edge, a second stale, because pushing to half a million browsers would be chapter 12's fan-out storm.

7. Failure modes. For each box, what customers see when it fails. The leader dies: chapter 14's election promotes the synchronous follower in about thirty seconds, no reservation is lost, and the waiting room keeps the crowd outside meanwhile. The payment provider slows: reservations simply hold, and Lena may extend the five minutes. The waiting room fails: the edge says "back shortly" and nothing behind it floods. London is unhealthy at 08:45: the drop moves to Saturday.

8. Trade-offs. Say what was given up. This design refuses reservations for thirty seconds if the leader fails, charges Toronto 80 ms a write, and depends on one region for one morning. In exchange it cannot oversell.

Then the review. Ana puts both designs through questions the team has asked before.

The decision is Ana's design, and Marcus agrees on the numbers. One thing of his is kept: every reservation is appended to the log, so the allocation can be audited afterwards. The decision is written up as a record, a practice the software architecture book covers.

Deliberately not built: partitioned stock, a second writing region, a new datastore, a stream allocator, cells.

What would change the decision: a need to reserve more than 1,000 units a second; drops of a million units; stock held in two regions for local delivery; an objective of 99.99%; North America passing half of orders.

Remember it as: requirements, numbers, interface, data, boxes, risks, failures, trade-offs, in that order, and write down what you chose not to build.

What Ana learned

• Start every design from what must be true and a peak rate, and draw nothing until both are written.
• Put every proposal, including your own, through the same questions, and keep one alternative on the page.
• Record what was not built and the measurement that would change your mind.

…but

The drop opens at 09:00 on Friday. In the first minute 512,000 people arrive, and the notebooks are reserved in 104 seconds. Three hundred and twelve reservations expire unpaid and are offered again; by 09:09 all 10,000 are sold, none twice. There is no next crack, only the next drop. The method is on a card by Ana's desk, and the epilogue is that card, at length.

EpilogueThe Depot#

This is the card by Ana's desk, at length. Nothing here is new. It is arranged for the second reading, for the design review, and for the moment someone says a word in a meeting and you want to know what it is for and how it goes wrong.

The metaphor table

Every concept in the book has a place in one network of depots. If you can place a word here, you can usually explain it.

One warning travels with the table: a box of stock is in one depot or another, and a record can be in three at once, slightly out of date in two. Chapters 5 and 13 are about that difference.

The diagram legend

Colour means the same thing in every diagram. A solid arrow is a request that waits for its answer; a dashed arrow is a transfer nobody waits for; a box around nodes is a failure domain, labelled as a zone or a region; a store takes writes unless it is drawn as a copy.

The review checklist

Every trap from every chapter, with the sign a reviewer would notice. Read down the section that matches the design in front of you.

Measuring and scaling (chapters 1–3)

• Averages that hide the tail (ch. 1, The Morning the Shop Went Viral): a dashboard whose only latency tile is a mean, measured on the server.
• Scaling out before measuring (ch. 1, The Morning the Shop Went Viral): a proposal to add servers that does not name the saturated resource.
• Running hot (ch. 1, The Morning the Shop Went Viral): a resource at 85% or more on a normal peak, described as efficient.
• Coordinated omission (ch. 1, The Morning the Shop Went Viral): a load test whose generator waits for each reply before sending the next request.
• Designing for the average day (ch. 2, The Back of the Envelope): a capacity figure obtained by dividing a daily total by the seconds in a day and stopping there.
• False precision (ch. 2, The Back of the Envelope): an estimate of 57.87 requests per second built on a peak factor somebody guessed.
• Load testing with unrealistic data (ch. 2, The Back of the Envelope): a test that requests one product, as one user, against an empty database.
• No headroom (ch. 2, The Back of the Envelope): a plan in which estimated peak equals measured capacity.
• Sticky sessions as state (ch. 3, Two of Everything): a balancer configured to pin each customer to one server so that their session survives.
• The health check that lies (ch. 3, Two of Everything): a health endpoint that returns a constant, or one that tests a dependency every server shares.
• The single load balancer (ch. 3, Two of Everything): a diagram with two of every server and one box in front of them.
• Autoscaling on the wrong signal (ch. 3, Two of Everything): a scaling rule on a metric the bottleneck does not move, such as memory on a service limited by CPU.
• State on local disk (ch. 3, Two of Everything): an upload, a report or a session written to the server's own filesystem.

Data (chapters 4–8)

• An index for every column (ch. 4, The Query That Read Every Row): a table with more indexes than queries, added "to be safe".
• Offset pagination at depth (ch. 4, The Query That Read Every Row): OFFSET with a number supplied by the client.
• N+1 queries (ch. 4, The Query That Read Every Row): a query inside a loop over the results of another query, often hidden by an object mapper.
• The unbounded query (ch. 4, The Query That Read Every Row): a SELECT with no LIMIT on a table that grows.
• Pool exhaustion (ch. 4, The Query That Read Every Row): requests failing with "timed out waiting for a connection" while the database's processors are not full.
• Reading your own write from a replica (ch. 5, Copies of the Stock Book): a page shown immediately after a write whose query goes to a follower.
• The failover never tested (ch. 5, Copies of the Stock Book): a failover described in a document and never carried out on purpose.
• Asynchronous replication assumed durable (ch. 5, Copies of the Stock Book): a claim that "no committed data is lost" over a design with only asynchronous followers.
• The replica as backup (ch. 5, Copies of the Stock Book): "we have replicas" given as the answer to "what if someone deletes the table?".
• Check-then-act (ch. 6, Two Clerks, One Shelf): a read, an if in application code, then a write that depends on the read.
• Assuming the default isolation level is serializable (ch. 6, Two Clerks, One Shelf): a design that says "it's in a transaction, so it's safe" and names no level.
• The long transaction (ch. 6, Two Clerks, One Shelf): a network call, a file upload or user thinking time between BEGIN and COMMIT.
• Inconsistent lock order (ch. 6, Two Clerks, One Shelf): two code paths that lock the same rows in different orders.
• Sharding too early (ch. 7, Too Big for One Depot): a partitioning design with no figure for the write rate or data size one machine could not take.
• The monotonic partition key (ch. 7, Too Big for One Depot): range partitioning on a timestamp or a counter in order to spread writes.
• The hot partition (ch. 7, Too Big for One Depot): one partition at full utilisation while the others are quiet.
• Hash modulo N (ch. 7, Too Big for One Depot): hash(key) % numberOfNodes in routing code.
• Routine cross-partition transactions (ch. 7, Too Big for One Depot): an everyday operation that must change rows in two partitions atomically.
• A database chosen by fashion (ch. 8, A Shelf for Every Kind of Thing): a store selected before anyone has listed the queries or the write rate.
• Polyglot sprawl (ch. 8, A Shelf for Every Kind of Thing): a new kind of store with each new feature and no owner for its backups.
• The search index as source of truth (ch. 8, A Shelf for Every Kind of Thing): a write that goes to the search index first, or data that exists nowhere else.
• Blobs in the database (ch. 8, A Shelf for Every Kind of Thing): a column of images or documents in a transactional table.
• Schemaless meaning no schema (ch. 8, A Shelf for Every Kind of Thing): "we don't need migrations" beside reading code full of checks for missing fields.

Speed (chapters 9–12)

• The cache as source of truth (ch. 9, The Shelf by the Counter): data written to the cache and nowhere else, or a decision such as a charge made from a cached value.
• No expiry (ch. 9, The Shelf by the Counter): a put without a time to live.
• Stampede on expiry (ch. 9, The Shelf by the Counter): a database load graph with regular spikes at the cache's time to live.
• Caching to hide a slow query (ch. 9, The Shelf by the Counter): a cache added because one query takes seconds.
• A personalised response under a shared key (ch. 9, The Shelf by the Counter): a cache key that names the page but not the customer, for a value that includes something about the customer.
• The cold cache at deploy (ch. 9, The Shelf by the Counter): a cache held in each server's memory, or flushed on release.
• Uncacheable by default (ch. 10, Closer to the Customer): no-store, or no caching header at all, on files that are identical for every visitor.
• Purging instead of versioning (ch. 10, Closer to the Customer): a release step that calls the network's purge function for files whose names did not change.
• Private pages cached at the edge (ch. 10, Closer to the Customer): public, or a missing private, on a response that contains one customer's data.
• The long DNS time to live before a move (ch. 10, Closer to the Customer): a record with a lifetime of a day on a name that is about to change address.
• The unprotected origin (ch. 10, Closer to the Customer): an origin address that answers requests from anywhere.
• The unbounded queue (ch. 11, The Yard Was Full): a queue with no maximum length and a producer that has never had a publish refused.
• The queue as database (ch. 11, The Yard Was Full): code that inspects the queue to answer a question, or messages kept for days as the only record of something.
• The invisible slow consumer (ch. 11, The Yard Was Full): monitoring of the broker's processor and disk with no alarm on depth or on the age of the oldest message.
• Work that cannot be repeated (ch. 11, The Yard Was Full): a consumer that charges, prints or sends with no way to recognise a message it has seen before.
• Polling everything (ch. 12, Three Left): a timer in the page that requests a value every second from the origin.
• Exactly-once assumed (ch. 12, Three Left): a consumer that adds, increments or sends for each record, with its bookmark saved in a separate step.
• The fan-out storm (ch. 12, Three Left): one event producing a message per follower or viewer, with no limit on either.
• A stream where a batch would do (ch. 12, Three Left): a streaming pipeline feeding a report read once a day.

Agreement (chapters 13–15)

• "We chose CA" (ch. 13, Which Copy Is Right?): a design for several machines that claims consistency and availability and says partitions will not happen.
• One consistency level for everything (ch. 13, Which Copy Is Right?): a single setting, strong or eventual, applied to every table.
• Eventual consistency with no convergence story (ch. 13, Which Copy Is Right?): "it will be consistent eventually" with no statement of what reconciles diverged copies.
• Silent last write wins (ch. 13, Which Copy Is Right?): conflict resolution by timestamp on whole records, with nothing logged.
• Home-made consensus (ch. 14, Two Head Offices): leader election written in application code, with a table of heartbeats and a "who is alive" query.
• A lock without a fencing token (ch. 14, Two Head Offices): a distributed lock with an expiry, and writes that carry nothing the store can check.
• A timeout taken as proof of death (ch. 14, Two Head Offices): a failover that acts on silence and does nothing to stop the old leader.
• The two-node cluster (ch. 14, Two Head Offices): two voters, or two sites with no third.
• Ordering by wall clock (ch. 15, What Time Is It?): a comparison of timestamps that were taken on different machines, in a sort, a merge or a conflict rule.
• The auto-increment identifier across partitions (ch. 15, What Time Is It?): a database sequence used as the key of data that is, or will be, written in more than one place.
• A timestamp as a unique identifier (ch. 15, What Time Is It?): a key built from the time to the millisecond.
• Trusting the client's clock (ch. 15, What Time Is It?): a time sent by a browser or a phone and used to order, expire or validate.

Staying up (chapters 16–20)

• The per-server rate limit (ch. 16, The Doorman): a limit held in each server's memory behind a load balancer.
• Retries without jitter (ch. 16, The Doorman): a retry delay that is a constant or a pure power of two.
• Limiting after the expensive work (ch. 16, The Doorman): a limiter that runs after authentication, parsing or a database read.
• Shedding without priorities (ch. 16, The Doorman): an overload policy that drops requests at random, or oldest first, whatever they are.
• Hedging a request that is not idempotent (ch. 16, The Doorman): a duplicate request sent for speed on a call that creates or charges.
• Backups never restored (ch. 17, The Day the Region Went Dark): a backup job with a green tick and no record of a restore.
• Failover never practised (ch. 17, The Day the Region Went Dark): a recovery document older than the last change to the system.
• Active-active without a conflict design (ch. 17, The Day the Region Went Dark): two regions accepting writes to the same records, and "the database replicates both ways" as the whole answer.
• The shared dependency across zones (ch. 17, The Day the Region Went Dark): one service, in one zone, that every zone calls: configuration, sessions, a registry.
• A recovery plan that needs the failed region (ch. 17, The Day the Region Went Dark): a recovery step that fetches something, an image, a secret or the plan itself, from the place being recovered from.
• An objective of one hundred percent (ch. 18, Promises with Numbers): "zero downtime" as a target, or four nines chosen because it sounds right.
• An alert for every cause (ch. 18, Promises with Numbers): a pager that fires on processor, memory and disk thresholds.
• An agreement without an objective (ch. 18, Promises with Numbers): a contractual availability figure that engineering does not measure, or one stricter than the internal target.
• Blame in the postmortem (ch. 18, Promises with Numbers): a root cause that is a person's name, or "human error".
• No error-budget policy (ch. 18, Promises with Numbers): an objective on a dashboard and no agreed consequence for missing it.
• Résumé-driven scaling (ch. 19, The Bill): a technology proposed with its famous users named and its required load unstated.
• Scaling for load that never came (ch. 19, The Bill): a component sized from a forecast, running at a few percent busy a year later.
• Ignoring egress (ch. 19, The Bill): a design that copies data between regions, or out to another company, with no figure for bytes a month.
• Never scaling back (ch. 19, The Bill): a capacity increase with a start date and no end date.
• Boxes before requirements (ch. 20, The Drop): a design presented as a diagram, with no sentence saying what must be true.
• A design with no numbers (ch. 20, The Drop): "a large spike" or "high traffic" where a rate should be.
• A single design with no alternative (ch. 20, The Drop): a proposal with no section on what else was considered.
• Ignoring failure modes (ch. 20, The Drop): a diagram in which every arrow works.
• Building for permanent peak (ch. 20, The Drop): capacity sized for the one extreme hour and left in place.

The numbers card

Orders of magnitude (chapter 2).

Waiting against utilisation (chapter 1): response = service time ÷ (1 − utilisation). At 50% busy a request takes twice its service time; at 80%, five times; at 90%, ten; at 99%, a hundred.

Availability in nines (chapters 3 and 18).

Bramble's numbers. A day is 86,400 seconds; call it 100,000.

The design method card

1. Requirements. What must always be true, and what may give.
2. Estimates. A peak rate per second, with its multiplier; storage; bandwidth.
3. Interface. The few operations, and what each returns when refused.
4. Data model. The records, the one place each is written, and what is conditional.
5. High-level design. Boxes, each justified by a line in step 2.
6. Deep dives. The hot key, the slowest dependency, the largest fan-out.
7. Failure modes. For each box, what customers see when it fails.
8. Trade-offs. What was given up, what was not built, and the measurement that would change the decision.

The timeline

Three eras. In the first, the theory was written, mostly for machines that shared a room. In the second, a few very large websites outgrew the single database and published what they built in its place. In the third, the rest of the industry adopted those designs and then learned to ask whether it needed them.

To 1999: the foundations. Queueing, the relational model, transactions, the B-tree and the logical clock are all older than the web. So are the first answers to its growth: consistent hashing, the content delivery network and consensus.

2000 to 2010: the large websites. Brewer named the trade that Google and Amazon were about to make in public. Memcached, MapReduce, Bigtable, Dynamo and Chubby each replaced something a single machine used to do, and by 2010 the costs of the trade had names too.

2011 onward: everyone else, and second thoughts. The log, understandable consensus and trustworthy clocks made the large designs available to ordinary teams. Then came the practices for running them, and the essays asking whether a given team should.

The reading list, by year

Fifteen to read first; every other source is in research/milestones.md. Titles in quotation marks are free online.

• 1978: Leslie Lamport, "Time, Clocks, and the Ordering of Events in a Distributed System"
• 1981: Jim Gray, "The Transaction Concept: Virtues and Limitations"
• 1995: Hal Berenson and others, "A Critique of ANSI SQL Isolation Levels"
• 2007: Giuseppe DeCandia and others, "Dynamo: Amazon's Highly Available Key-value Store"
• 2007: Michael Nygard, Release It!
• 2012: Eric Brewer, "CAP Twelve Years Later: How the 'Rules' Have Changed"
• 2013: Jeffrey Dean and Luiz André Barroso, "The Tail at Scale"
• 2013: Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction"
• 2014: Diego Ongaro and John Ousterhout, "In Search of an Understandable Consensus Algorithm"
• 2015: Marc Brooker, "Exponential Backoff and Jitter"
• 2015: Frank McSherry, Michael Isard and Derek Murray, "Scalability! But at what COST?"
• 2016: Betsy Beyer and others (editors), Site Reliability Engineering
• 2016: Martin Kleppmann, "How to do distributed locking"
• 2017: Martin Kleppmann, Designing Data-Intensive Applications, the one book to read after this one
• 2017: Oz Nova, "You Are Not Google"

For where the boundaries go and how the parts talk, read DDD and Microservices: The Atlas. For system-level styles, trade-off analysis and decision records, read the sibling book on software architecture.