About this bookBefore you start#

Computer Networks: The Sorting Office

A short story about one Friday outage and eighty years of networks: what every piece is, why it exists, what it replaced, and where it breaks.

The promise

You type 10.0.0.0/16 into a form and hope. You paste a DNS record someone gave you. You open port 443 because the tutorial said so. You have been in meetings where people said "it's a BGP thing" or "the pods can't reach the endpoint" and you nodded. This book is for you. By the end you will be able to name every hop between a phone and a pod in a cloud cluster, say what each one does, why it was invented and what it replaced, and say which attack lives at that hop and what stops it.

It is not a reference manual and it has no labs. It is a story, and it is short: thirty-one chapters of about fourteen hundred words each, roughly three and a half hours end to end, or one chapter over a coffee.

Who it is for

Software engineers, DevOps engineers and cloud practitioners who build and run things on networks without ever having been taught networks. No prior knowledge is assumed beyond having used a browser and a terminal. Network engineers will find it too gentle, and that is by design.

How to read it

Two stories run side by side. In the present, Nadia, a junior DevOps engineer on call for her first weekend, chases an outage one layer at a time from Friday afternoon to Monday morning. In the past, Ruth, a network engineer two weeks from retirement, answers each of Nadia's "why is it like this?" with how it came to be. Every chapter has the same five parts:

Read it in order: the mystery is fair, and every clue is given before the reveal. If you only want the concepts, read The idea in each chapter and the epilogue, which holds the whole journey, a glossary and a checklist. For depth on cryptography, identity and defence beyond the network, this book points to its sibling, Cybersecurity: The Keep.

The cast

• Nadia Haddad, junior DevOps engineer at Kettle, eight months in. She writes everything down.
• Ruth Okafor, network engineer, sixty-three. Leased lines in the eighties, an ISP in the nineties, a data centre in the noughties, the cloud after that. Two weeks left.
• Sam Price, engineering lead. Believes it is always DNS.
• Jonas Lind, platform engineer. Made a change on Friday afternoon for good reasons.
• Kettle, a coffee-subscription business: forty staff, thirty thousand subscribers, a storefront on a CDN, checkout running in pods on a managed Kubernetes cluster inside one cloud VPC, and payments handled by an outside provider called Paystream.

The diagrams

Every diagram uses the same colours, so a shape means the same thing in every chapter.

Above each chapter a strip shows where that chapter sits on the journey a checkout request makes: the wire, the phone, Wi-Fi, the ISP, the internet, DNS, the CDN edge, the load balancer, the VPC, the node, the pod, and the payment API.

Dates and sources

Every year and name in the Back then sections was checked against a primary or authoritative source before writing; the list is in the book's repository. Real people are described only by what is on the public record, and nothing they did not say is put in their mouths. Kettle, Paystream and the four characters are invented.

Contents

Prologue. The Page That Wouldn't Load

Act I. Before the network

1. The Machine Everyone Queued For (sharing, time-sharing, terminals, client and server)
2. The Busy Signal (circuit switching versus packet switching)
3. Lo (ARPANET, hosts and the subnet, email)

Act II. Agreeing how to talk

4. A Network of Networks (internetworking, gateways, the end-to-end principle)
5. Envelopes Inside Envelopes (protocols, layers, OSI and TCP/IP)
6. Everyone Talks at Once (Ethernet, MAC addresses, switches, frames)
7. Registered Post and Postcards (TCP and UDP, ports, the handshake)
8. Flag Day (standards, RFCs, the 1983 cutover, change windows)

Act III. Finding each other

9. Every House Needs an Address (IPv4, subnets, CIDR)
10. Who Has This Address? (ARP, DHCP, the default gateway)
11. It's Always DNS (names, resolvers, records, TTL)
12. The Sorting Office (routers and routing tables)
13. The Map Nobody Owns (autonomous systems, BGP)
14. Running Out of Houses (private ranges, NAT, IPv6)

Act IV. The web on top

15. The Worm (firewalls, the DMZ, the perimeter)
16. A Page Made of Links (HTTP)
17. Talking in Private (TLS and certificates)
18. The Tunnel Home (VPNs)
19. One Name, Many Servers (load balancers and proxies)
20. Closer to You (caches and CDNs)
21. The Flood (DDoS and retry storms)
22. Faster Than a Handshake (HTTP/2, HTTP/3, QUIC)

Act V. The data centre

23. Floors Full of Switches (VLANs, spine-leaf)
24. Machines Inside Machines (virtual switches, overlays, SDN)

Act VI. Someone else's network

25. Your Own Private Slice (VPC, subnets, route tables, gateways)
26. Two Kinds of Guard (security groups and network ACLs)
27. Connecting the Islands (peering, transit, private endpoints, hybrid)
28. Never Trust the Inside (segmentation and zero trust)

Act VII. Networks for containers

29. Every Pod Gets an Address (container networking, CNI)
30. Services That Move (Services, cluster DNS, Ingress, NetworkPolicy)
31. The Mesh (service mesh, mTLS, eBPF)

Epilogue. The Whole Journey (the incident review, one request end to end, the glossary, the checklist)

PrologueThe Page That Wouldn't Load#

At 16:02 on a Friday in late September, the checkout page at Kettle stopped loading, and nobody noticed for eleven minutes.

The storefront was fine. You could browse the single-origin Ethiopian, read about the roaster in Leeds, put a bag of beans in your basket and press the big terracotta button that said Subscribe. Then you waited. A small spinner turned. Some people, perhaps a third of them, got through to a confirmation and an email. The rest watched the spinner until the page gave up and showed them a grey box that said 504 Gateway Time-out, which is not a sentence anyone at Kettle had written.

Nadia Haddad found out the way on-call engineers usually find out, which is from someone who is not on call. Sam Price, the engineering lead, leaned over the partition with his phone held out as if it were slightly too hot.

"Marketing says checkout is broken."

"Which bit?"

"The bit where people give us money."

She had been at Kettle for eight months and on the on-call rota for three. This was her first weekend holding the phone alone, and it was not yet the weekend. She opened the dashboards. The dashboards were green. Forty-one pods in the checkout deployment, all healthy. The cluster's nodes were fine. CPU was low. Memory was low. The load balancer was serving requests and reporting, in its calm way, that about a third of them were succeeding.

"It's always DNS," said Sam, and went to talk to the CEO.

Nadia tried the checkout herself. It worked. She tried again. It hung. She tried on her phone over mobile data and it worked, and then she tried once more and it hung. She wrote the times in the paper notebook she kept because she had once lost a whole incident timeline when a laptop died, and wrote next to them: not all, not none.

By 16:30 the incident channel had thirty people in it, which is a lot for a company of forty. Someone had posted a screenshot of the grey box. Someone else had posted "restart it?" with a question mark that was doing a lot of work.

Ruth Okafor arrived at the edge of Nadia's desk the way she arrived everywhere, which was slowly and with a mug. Ruth was sixty-three and had two weeks left. She had been kept on past the move to the cloud to hand over the old office network, the thing of beige boxes in the cupboard that still connected the printers and the door entry system to whatever the rest of the company now was. She had been a network engineer since before Nadia was born. She had, people said, run an internet service provider in the nineties, out of a building in Reading with a diesel generator and a fax machine.

"What's broken?" said Ruth.

"Checkout."

"What's checkout?"

Nadia started to answer and found she could not. Checkout was a deployment. It was forty-one pods. It was a Service in front of them, and an Ingress in front of that, and a load balancer in front of that, and a CDN in front of that, and DNS in front of that. It was a call to Paystream, the payment provider, which went out of the cluster somehow and came back somehow, and she had never once thought about the somehow. It was a URL. It was the big terracotta button.

"Right," said Ruth, and pulled a chair over. "Then that's the problem. Not the outage. You can't debug what you can't name." She put the mug down. "Start at the beginning. What is a server?"

"I know what a server is."

"Good. Which one of yours is down?"

Nadia looked at the forty-one green pods and did not have an answer.

Chapter 1 · Act IThe Machine Everyone Queued For#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

It was 16:30. Nadia listed the pods, because it gave her hands something to do.

$ kubectl get pods -n checkout | head -4
checkout-7d4f9-2kq8x   1/1   Running   0   6d
checkout-7d4f9-5mzpl   1/1   Running   0   6d
checkout-7d4f9-9rbtw   1/1   Running   0   6d

Forty-one lines like that. Running, ready, no restarts, six days old. The nine nodes under them were Ready too, three in each zone. Nothing had crashed. Nothing was even warm.

"Someone in the channel says the server is down," she said.

"Which server?"

"Checkout."

"That's a name. I asked which machine." Ruth turned the mug a quarter turn, which Nadia would learn over the weekend meant she was settling in. "When your phone hung at twenty past four, something on the other end didn't answer. Which of those forty-one was it?"

Nadia did not know. The request had gone to the load balancer, and the load balancer had handed it to a pod, and she had no idea which pod, or whether it had been the same one twice.

"Pick one, then. Ask it directly."

She opened a tunnel to the first pod on the list and sent it a health request. It answered ok in four milliseconds. The second: ok. The third: ok. She did ten and stopped.

"They're all healthy."

"Then either the server isn't down, or the server isn't one of those." Ruth sat back. "You said it like it was a box. When I started, it was. One machine, one room, and a queue out of the door. Let me tell you about the queue."

The idea

A computer in 1961 cost more than the building it sat in. Sharing it was not a nicety; it was the only way most people would ever touch one. The first way to share was batch processing: collect everyone's jobs, run them one after another, and hand the results back. It kept the machine busy and the people waiting.

Time-sharing turned that round. The machine switches between users many times a second, giving each a short turn, so that from the chair it feels like a private computer. The chair itself is a terminal: a keyboard and a screen (in 1961, a typewriter) with no computer inside it. It sends keystrokes down a wire and prints whatever comes back. The wire is a serial line, which carries one bit after another. When the terminal sat across town rather than down the corridor, a modem at each end turned the bits into tones a telephone line could carry, and the far modem turned them back.

Look at the shape of that. One side asks and waits. The other side holds the expensive thing and answers. The side that asks is the client; the side that answers is the server. The words came later, but the arrangement is 1961. A server is a role, not a box. It is whatever is asked.

Three things follow from that picture, and all three matter at Kettle.

First, the client and the server need not share a room. From the terminal's chair you cannot tell whether the machine is next door or in another city, and the modem was invented so that you would not need to.

Second, the wire is part of the arrangement. A terminal with a cut line and a terminal with a crashed machine look identical from the chair: you type, and nothing comes back. Everything in this book sits between those two explanations.

Third, the thing you call to reach a server, a telephone number in 1961, does not have to name one machine.

When someone in the channel writes "the server is down", they mean the top half of that diagram: one machine, one room, a light that is either on or off. Kettle's checkout is the bottom half. One name, forty-one machines, and the thing that owns the name is a reception desk sending each visitor to any free clerk. "Which server is down?" is a real question, and the dashboards cannot answer it, because they report on the clerks and not on the visit.

Remember it as: A server is whatever answers; a client is whatever asks. The name you call is not the machine that picks up.

Nadia's health checks had asked ten pods "are you alive?", and ten pods had said yes. The forty-one green lights were forty-one honest answers to a question that nobody pressing the terracotta button was asking.

…but

The load balancer's graph had not moved while they talked. Roughly one request in three was failing and two in three were going through, and it had been like that since 16:02. If a server was down, it was managing to be down only some of the time.

Chapter 2 · Act IThe Busy Signal#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

By 16:45 Nadia had drawn a line down a notebook page: works on the left, hangs on the right. She pressed the terracotta button twelve times with a test card. Eight confirmations. Four spinners that ended in the grey box. From her phone, over mobile data, it was seven and five. The load balancer's own log told the same story in its own way.

16:41:07  POST /checkout  200    412 ms
16:41:19  POST /checkout  504  30004 ms
16:41:53  POST /checkout  200    388 ms
16:42:10  POST /checkout  504  30002 ms

A third failing, flat since 16:02. No ramp, no spike, no pattern she could see in which attempts failed. Sam's theory in the channel was that Paystream, the payment provider, had gone down. Nadia's objection, which she typed and then deleted, was that if a payment provider went down, nobody would get through.

Ruth had read the log over her shoulder. "When I was at the ISP, customers used to ring and say the line was engaged. That meant every circuit into the exchange was taken. Nobody new got in, and the ones already on stayed on. This looks like a busy signal." She tapped the page. "One in three gets the engaged tone."

"So it's capacity?"

"But it isn't one. A busy signal comes in waves: you get it, and you keep getting it until someone hangs up. This is a coin toss on every press. Something is letting two through and losing the third, every few seconds, all afternoon. Telephones don't fail like that." She picked the mug up again. "Letters do."

The idea

A telephone call works by circuit switching: before you say a word, the network reserves a path from your handset to theirs, through every exchange in between, and holds it for the whole call. The path is yours whether you speak or breathe. If no path is free, you get the busy signal. If any link on the path fails, the call drops, all of it, at once. In the postal picture, it is booking a private road for the whole trip.

Packet switching reserves nothing. The message is cut into packets, small pieces that each carry the destination's address, and each packet travels on its own. The lines are shared by everyone's packets, mixed together, like letters sharing every road.

The junctions work by store-and-forward: a node receives a whole packet, holds it in memory, checks it, and sends it on down the next line as soon as that line is free. If the usual line is down, the node picks another. If the next line is busy, the packet waits in memory for a few thousandths of a second and then goes.

This is why packets survive damage. A circuit network has two states: there is a path, or there is not. A packet network has as many states as it has packets. Lose a node and the packets go round it. Lose one packet and the sender sends that one again, while the other ninety-nine arrive. The network can be nine-tenths working, and that is not an emergency; it is Tuesday.

It also means a packet network fails in a way a telephone never does. Every packet succeeds or fails on its own. One conversation can lose one packet in a hundred and barely notice. Another can lose every packet that happens to pass through one bad node, while the conversation beside it, routed one line over, sails through. Partial failure, some requests working and some not, with no pattern at the surface, is not a special case of packet switching. It is the normal condition, with the volume turned up.

Two numbers describe any line, and they are not the same number. Bandwidth is how many bits a line carries per second: the width of the road. Latency is how long one bit takes to get from one end to the other: the length of the road. Store-and-forward adds latency at every node, because a packet must arrive whole before it can leave. A line can have bandwidth to spare and still be slow, and a line can be fast and still be narrow.

Remember it as: Circuits book the road; packets share it. A booked road fails all at once; shared roads lose letters one at a time.

Kettle's checkout request did not book a road. It was a stream of packets, each making its own way, and "a third fail" is what a packet network looks like from above when something on one path is eating letters and nothing else is wrong.

…but

A checkout request was not one conversation. The phone talked to checkout, and checkout talked to Paystream, and Nadia saw she had been counting only the first half. The checkout logs would say which half was losing letters.

Chapter 3 · Act ILo#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

At 17:00 Nadia opened the checkout logs and searched for the request id from one of her own failed attempts. Checkout had seen it. That was the first thing. The phone had reached checkout, the basket had been checked, the price had been worked out, all in a few milliseconds. Then one step had started and not finished.

17:01:12  req=8f3a  step=basket.validate     ok     3 ms
17:01:12  req=8f3a  step=payment.authorise   start  upstream=api.paystream.example
17:02:12  req=8f3a  step=payment.authorise   failed 60004 ms

She found a request that had succeeded. Same steps, same upstream, and payment.authorise had come back in 410 ms. She searched for every failure since 16:02. Every one of them had died on the same line.

"So the phone reaches us every time," said Ruth. "What fails is us reaching somebody else."

"Paystream. We send them the card, they send back yes or no."

"Then that's your hop. Not phone to checkout. Checkout to Paystream." Ruth pulled the notebook towards her and drew two boxes with a long gap between them. "Your log says the far end didn't answer in time. It doesn't say why. It can't. From where that log is standing, 'their machine fell over' and 'the letters never arrived' look exactly the same." She wrote checkout in one box and Paystream in the other and left the gap empty. "In 1969 a man typed two letters and the far end fell over. Harold was in the room. It took them a phone call to find out which of those two it was."

The idea

ARPANET was paid for by the American defence department's research agency to solve the problem from chapter 1: expensive machines, scattered across the country, that researchers wanted to share. Every site had a different computer, with different ideas about what a character was. Teaching each one to speak to every other meant new work at every site each time a site joined.

The answer, suggested by Wes Clark while the plan was still on paper, was to take the networking out of the hosts altogether. Put a small, identical computer at every site and let those computers talk to each other. That was the IMP: a dedicated machine whose only job was packets. The IMPs and the leased lines between them formed the subnet, the part of the network that moves packets and knows nothing about what they mean. The design called for each IMP to join at least two others, so that one dead line never cut a site off.

The hosts were the sites' own computers, each attached to its IMP by a short cable. A host's job was to run programs. An IMP's job was to deliver. The IMP cut the message into packets, chose a line, stored and forwarded, sent again if the next IMP did not confirm, and reassembled the packets at the far end before handing them to the far host.

That division of labour is the oldest line in networking, and it is still the one engineers draw. Hosts on one side, subnet on the other. "My app" and "the network". The IMPs are long gone, but the question the line poses is unchanged: did the far host fail, or did the letters not get there?

The subnet could carry packets, but packets are not a conversation. For two programs on two hosts to open a connection, agree it was open, send bytes in order and close it again, both sides needed a protocol: a written agreement about what to send, in what order, and what each message means. NCP was that agreement, and it ran only on the hosts. The IMPs never read it. They carried it.

Follow the letters. Both IMPs did their job for all three. The subnet was fine. The failure was in a host, and from Kline's chair it looked like silence, which is also what a failure in the subnet would have looked like. The only way to tell the difference was Duvall on the telephone saying what his end had seen.

Email was the surprise. The network was built so that hosts could share machines, and within a few years its main cargo was people writing to people. The lesson has not aged: the use you build a network for is rarely the use that fills it.

Remember it as: Hosts run programs; the subnet moves packets. "My app" and "the network" are those two, and a log only sees one.

Kettle's checkout log had reported a host-to-host failure. Checkout is a host. Paystream is a host. Everything in the gap Ruth had left on the page is subnet, and the log could say only that the far host had not answered in time. It could not say whether Paystream had fallen over like the SDS 940, or whether the letters had never arrived.

…but

The log said the far end had not answered. It did not say why. The quickest way to find out whether Paystream was up was to ask Paystream, so Nadia opened a new tab on her laptop.

Chapter 4 · Act IIA Network of Networks#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

At 17:10 Nadia had one fact she trusted. The checkout logs said the step that failed was checkout calling Paystream, not a customer calling checkout. So the next question wrote itself: was Paystream down?

She asked it from her laptop.

$ curl -s https://api.paystream.example/v1/health
{"status":"ok","region":"eu-west","latency_ms":41}

She ran it five times. Five answers, none slower than 180 ms. She opened Paystream's status page. Every row was green. All systems operational. Last incident: 23 days ago. She pasted the screenshot into the channel and somebody replied with a thumbs-up, as if that settled it.

"It's not them," she said.

"It's not them from here," said Ruth.

"Where else is there?"

Ruth put the mug down. "How does your laptop get to Paystream?"

Nadia thought about it. The office Wi-Fi. The box in the cupboard. Whoever the office bought its internet from. Then, somehow, Paystream. "Three or four networks," she said. "I don't know whose."

"And the pod?"

The pod was in the cluster. The cluster was in the VPC. The VPC went out to the world through something, and she realised she had never once asked what. "Different networks," she said slowly. "A different road."

"Count the roads," said Ruth. "Then tell me what your green status page proves."

Nadia wrote in the notebook: Works from laptop. Status green. Proves nothing about the pod's road. She underlined road and did not yet know why.

The idea

A single network, like ARPANET, is a set of machines that all agree on one set of rules. Internetworking is the trick of joining networks that do not agree, so that a host on one can talk to a host on another without either network changing its rules. The internet is that trick, done at scale, and the small letter i in internetwork is where the name comes from.

The join is made by a gateway: a machine with a foot in two networks, speaking each one's local rules on its own side and passing a common packet between them. Today we call gateways routers, and a later chapter is all about how they choose a direction. For now the shape is enough.

The common packet needs a common idea of an address, one that means the same thing on every network it crosses. That is the job of the internet protocol, IP, and the addresses it carries are the ones you type into forms. A packet is a letter in an envelope. Each network is a postal service with its own vans and its own habits; the gateway is the counter where one service hands the letter to the next. The letter does not care.

What the paper refused to do is as important as what it did. It did not ask the middle to be reliable. Every network along the way may lose, duplicate or reorder packets. Only the two hosts at the ends keep track, and only they recover.

This is the end-to-end principle: put a function at the ends of the network, where it can be done completely, and keep the middle simple. The slogan engineers use is "the network is dumb and the edges are smart". A dumb middle is cheap, it scales, and you can add a new kind of network to it without asking permission. The cost is that the middle will never explain itself. When a packet vanishes, no gateway sends a letter of apology. The ends have to notice, by waiting, and then decide what to do.

Remember it as: The middle only carries. Only the two ends can know whether the letter arrived, so the ends do the checking.

Both halves of that idea were sitting on Nadia's desk. Her laptop and the checkout pod reached Paystream across different sets of networks, through different gateways, so "Paystream answers my laptop" was a fact about one road and said nothing about the other. And nothing on the pod's road was going to tell her where the packets went. Checkout had noticed the way ends notice, by waiting, and the wait was what the customers were watching.

…but

If Paystream had answered the pod and said no, the answer would carry a number, the way a web error does. Nadia went back to the checkout logs to find the number. There was not one.

Chapter 5 · Act IIEnvelopes Inside Envelopes#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

At 17:20 Nadia filtered the checkout logs down to the failed payment calls and read one properly, instead of scrolling past it. She had expected a 500 from Paystream, or a 502, or any of the three-digit numbers she knew how to search for.

17:19:41 ERROR checkout payment: POST /v1/authorise failed
         connect timeout after 60000 ms (api.paystream.example:443)

There was no number. There was no reply at all. The word that stopped her was connect.

"What's the difference," said Ruth, reading over her shoulder, "between that and a 500?"

"A 500 means they got my request and fell over." Nadia said it slowly, hearing it. "This means I never got to send the request. It waited a minute for a connection and gave up."

"So the failure is below HTTP."

"Below?"

Ruth took the notebook, drew seven boxes in a stack, and wrote a word in each. Then she crossed out three of them. "Nobody uses this stack," she said. "Everyone talks about it. That's going to annoy you for the rest of your career."

Nadia looked at the four boxes that were left. HTTP was in the top one. The word connect belonged, she suspected, one or two boxes down. And the checkout logs, which had told her everything about the top box, had almost nothing to say about the ones below it.

The idea

A protocol is an agreement about messages: what they look like, what order they come in, and what each side does when it gets one. Two programs that share a protocol can talk without sharing anything else. A layer is a protocol's place in a stack: it does one job, offers that job as a service to the layer above, and relies on the layer below for its own. The point of the stack is that you can replace one layer without rewriting the others. Your browser does not care whether the wire is copper, glass or air.

Here is the stack OSI drew, next to the stack the internet uses.

The four-layer stack is the one that runs. TCP, the transport protocol in the fourth row, is the one that numbers and signs for every piece of data; the next chapters explain it properly. IP in the third row is the common packet from the last chapter. Ethernet and Wi-Fi in the second row are the rules of one local network. Nobody runs the seven-layer version. Yet everyone says layer 4 when they mean transport, ports and connections, and layer 7 when they mean the application's own conversation, which for Kettle is HTTP. The numbers came from OSI, TCP/IP never gave its layers any, and shorthand wins. You will hear it on load balancers, firewalls and in incident channels. Nobody says layer 5 or layer 6, because nothing lives there.

What the layers do is encapsulation. Each layer takes the message handed down from above, puts it inside its own envelope with its own header on the front, and hands it down again. Envelopes inside envelopes, and each office reads only its own.

At the far end the envelopes come off in the opposite order. The link layer opens the frame and passes the IP packet up. IP checks the address and passes the TCP segment up. TCP checks the port and the numbering and passes the bytes up. Only then does an HTTP server see a request. Every layer sees only its own header and treats everything inside as a sealed parcel.

The order matters for errors as much as for data. Transport has to open a connection before the application can say a word.

If the second message gets no answer, the sequence stops on the second line. The application layer on the far side is never told anything, because nothing reached it. The application on the near side is told only that the layer beneath it waited, and for how long.

Remember it as: Each layer is an envelope with its own address on the front, and each office along the way opens only its own.

That is what connect timeout means. Checkout's layer 4 asked for a connection and layer 4 at Paystream never replied, so HTTP at Paystream never heard from the pod, and no server there wrote a log line about it. Nadia's laptop test had been a layer 7 question, and the pod's problem was a layer below it.

…but

Below HTTP, below TCP, below IP, Ruth's table ended at a wire. Nadia had seen Ruth's wires, in the cupboard with the beige boxes. She could not think where the wire was in the cloud.

Chapter 6 · Act IIEveryone Talks at Once#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

The cupboard was at the end of the corridor, and at 17:25 Nadia went and looked in it, because she had never properly looked. Two beige boxes with a row of green lights each, a tangle of blue cables, a label in Ruth's handwriting that said DO NOT UNPLUG, THIS MEANS YOU. Every cable came from a desk or a printer and ended in one of those boxes. That was the wire, then. That was where the frames from the bottom of Ruth's table went.

She went back to her desk and looked for the same thing in the cloud console. There were nodes. Each node had a page, and on the page was a section called Network interfaces, and in it was a single row.

eni-0b7c2e   10.0.18.79   0a:58:0a:00:12:4f   subnet-a   attached to node-18

An address she recognised. A second address she did not, six pairs of letters and digits. A subnet. No switch. No cupboard. Nothing with lights.

"There's no switch," she said. "The table ends at a wire, and there isn't one. So where does a frame go?"

Ruth looked at the row for a while. "That second address is the name on the van," she said. "The frames are still there. You just don't own the wire."

"Then who does?"

"The people you rent from. Which is the first interesting thing you've said about the cloud all afternoon." Ruth sat down. "Before you can ask where their switch is, you need to know what a switch is for, and that means knowing what it replaced."

The idea

Ethernet is a set of rules for one local network: how machines share one medium, what a message on it looks like, and how each machine is named. It is the second row of the table, the link layer, and it is still the commonest thing under an IP packet in any building.

The message is a frame: the unit the link layer carries for one hop, holding an IP packet inside it. A frame starts with the address of the machine it is for and the machine it is from, then a type field saying what is inside, then the payload, then a checksum so a damaged frame can be thrown away. In the postal picture the frame is the mailbag the letter rides in for one leg. The bag is unpacked at every stop; the letter is not.

The addresses in a frame are MAC addresses: 48-bit numbers written as six pairs of hex, such as 0a:58:0a:00:12:4f, stamped into every network interface when it is made, with the first three pairs saying who made it. A MAC address means something only on the local network, the one wire or one room. It is the name painted on the van for one street, and the van never leaves the street.

On Metcalfe's cable everyone heard everything, and two machines sending at once caused a collision: both frames ruined, both senders stopping and waiting a random time before trying again.

The first office networks replaced the long cable with a hub, a box that repeats every bit it receives on one port out of every other port. Electrically it was the same single room: everyone still heard everyone, and collisions grew with every desk added. The switch changed the room into corridors. A switch watches the source address on each frame that arrives and learns which MAC address lives on which port. Then it sends each frame out of one port only, the one that leads to its destination. Each cable becomes a private two-way conversation between the machine and the switch, and collisions stop happening.

A frame has a largest size. The MTU, the maximum transmission unit, is the biggest payload one frame on a link will carry, and for Ethernet it is 1,500 bytes. An IP packet larger than the MTU of the next link must be cut into pieces or dropped, and every envelope between the application and the frame takes bytes out of that 1,500.

Wi-Fi is Ethernet's radio cousin. Its frames have the same shape and the same MAC addresses, so IP cannot tell the difference. The air is a shared room like Metcalfe's cable, with two problems the cable did not have: a radio cannot hear a collision while it is transmitting, so Wi-Fi tries to avoid collisions by waiting and acknowledging every frame; and anyone with an antenna is in the room, so the frames are encrypted.

Remember it as: Ethernet is a room where everyone hears everyone. A switch turns the room into corridors, one per machine.

In Kettle's cloud the switch has not gone away. Every node and every pod has a network interface with a MAC address, and the frames it sends go to a switch written in software on the physical machine the node runs on, owned and operated by the provider. The frames are there; the cupboard is someone else's.

…but

At 17:29 somebody in the channel typed doing a rolling restart of checkout to clear it. Nadia started typing wait, and on the dashboard the first of the forty-one pods was already gone.

Chapter 7 · Act IIRegistered Post and Postcards#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

The restart finished at 17:33. By 17:35 the load balancer's calm little graph said that four checkouts in five were failing, up from one in three, and the channel had gone quiet in the way channels do when everyone is re-reading what they typed.

"It got worse," said Sam, from the doorway.

"It got worse," Nadia agreed. She did not yet know why, and she wrote the time down so that someone could ask her later.

Ruth had a suggestion that was not a suggestion. "Stop reading logs. Watch the packets." So Nadia started a debug pod on the same node as a failing checkout pod and ran a capture, filtering on port 443, and let one failed payment call go through it. She trimmed the paste to the columns that mattered, because the channel's screenshots were already unreadable.

17:36:02.114  Flags [S], seq 2811449031, win 62727, length 0
17:36:03.140  Flags [S], seq 2811449031, win 62727, length 0
17:36:05.171  Flags [S], seq 2811449031, win 62727, length 0

Three lines. The same flag, the same number, one second apart, then two seconds apart. Nothing came back. Not an error, not a refusal, not a single frame in the other direction.

"S is for SYN," said Ruth. "That's the pod knocking on the door. Three knocks, each one waiting longer. Nobody answers."

"So the door's locked?"

"A locked door says no. You'd see a reply with an R in it, straight away, and the log would say refused. This is nobody answering." Ruth tapped the empty space under the third line. "Dropped, not refused. Something is taking those packets and throwing them in the bin, and it isn't telling anyone."

The idea

The transport layer offers two services, and the postal metaphor is exact for once. TCP is registered post: every byte is numbered, every delivery is signed for, and anything lost is sent again, so the two programs at the ends see one unbroken, ordered stream. UDP is a postcard: cheap, fast, no receipt, and you find out it was lost only by noticing that no answer came. Web traffic, email and database connections use TCP. Phone-book lookups, video calls and games use UDP, because a late answer is worse than a missing one.

Both use ports: 16-bit numbers that say which program in the building a message is for. The IP address is the street address; the port is the flat number. Servers listen on well-known ports, 443 for HTTPS, 80 for plain HTTP, 53 for the phone book. A client picks a temporary high port for its side, such as 51734, and the pair of addresses and pair of ports together name one conversation.

TCP begins with the three-way handshake. The client sends a SYN carrying a random starting sequence number, the number of the first byte it will send. The server answers with a SYN of its own plus an ACK, an acknowledgement, saying which byte it expects next. The client acknowledges that, and the connection is open. From then on each side numbers its bytes and the other side signs for them.

The timer is the heart of retransmission. A sender keeps a copy of everything not yet acknowledged. If the acknowledgement does not arrive in time, it sends the copy again, and doubles the wait before the next try: one second, two, four, eight. That doubling is the same random-ish patience that ALOHA used, and it is why the capture showed gaps of one second and then two. A SYN that is never answered is retried on the same schedule until the client gives up, and Kettle's payment client gives up after 60 s.

Doubling is also half of congestion control. The other half is Jacobson's slow start: a new connection sends a little, doubles what it sends every round trip while acknowledgements keep coming, and cuts back the moment one goes missing. Loss is the only signal the dumb middle ever gives, so TCP treats every loss as the network asking it to slow down.

A connection is a thing with a life. Each end keeps it in a state, and the diagram below is a simplified form of the one in RFC 793 that Harold pinned up.

The arrow that matters tonight is the one from SYN_SENT back to CLOSED. There are two ways out of SYN_SENT that are not success. If the SYN reaches a host where nothing is listening on that port, the host replies at once with a reset, RST, and the client reports connection refused within a millisecond. If the SYN reaches nothing, or reaches something that discards it in silence, there is no reply, and the client retries, and retries, and finally times out.

Remember it as: TCP is registered post, UDP is a postcard. A refusal is a reply. A timeout is silence, and silence means something is in the way.

That is the whole of Ruth's "dropped, not refused". Paystream's door was not locked, because a locked door answers. The pod's SYNs went somewhere that read them and said nothing, three times, and checkout's sixty seconds were spent listening to that nothing.

…but

Something between the pod and Paystream was throwing the knocks away without a word, and it had not been there at lunchtime, because at lunchtime people had bought coffee. Nadia turned to a new page and wrote what changed? at the top of it.

Chapter 8 · Act IIFlag Day#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

What changed? is a question with a tool, and at 18:00 Nadia opened it: the change log, where every merge to the platform repository landed with a time and a name. She scrolled back to Friday afternoon. Two small things in the morning. Nothing at lunch. Then one line at 15:58.

#412  platform: private connectivity to payments      merged 15:58  jonas.lind
      Payment traffic to Paystream now stays inside the VPC. Checked from a node, works.

Four minutes before the first spinner. She read it twice, and then she read the time of the first failed checkout in her notebook, 16:02, and sat very still.

"Jonas," said Sam, who had come to look.

"Don't," said Ruth.

"I wasn't going to." He was already typing. Jonas answered from a train, in the slightly too cheerful way of a man who has been asked a question he suspects is bad news. He was on his way to his cousin's wedding. He would have no signal from Saturday morning. He had tested it. From a node. It had worked, and the dashboards had been green when he left at four.

"He's not wrong about the dashboards," said Nadia.

"Nobody's wrong yet," said Ruth. "Write down what you know. A change at 15:58, failures at 16:02, a restart at 17:30 that made it worse. That's not a cause. That's a timeline. But I've seen a lot of timelines, and I've never seen one where four minutes was a coincidence."

Nadia wrote it down. Then she wrote the title of the change again, underneath, and underlined one word. Private.

The idea

A protocol is an agreement, and an agreement needs somewhere to be written down and some way to be changed. The internet's answer is odd by the standards of any other industry, and it has outlasted every rival.

An RFC is a numbered document in a series that began with Crocker's bathroom note. The name still stands for Request for Comments, though many are now the standard itself: TCP is RFC 793, and every protocol in this book has a number. An RFC is never edited after publication. When something changes, a new RFC updates or replaces the old one, so the history is always on the record.

The body that looks after them is the IETF, the Internet Engineering Task Force. It has no membership and no votes. Anyone can join a working group's mailing list, and a document advances when the group reaches rough agreement and the idea has been shown to work. David Clark, one of the authors of the end-to-end paper, put the creed on a slide at a meeting in the early nineties: the IETF rejects kings, presidents and voting, and believes in rough consensus and running code. A standard that nobody has implemented twice, with the two implementations talking to each other, is not finished.

That is also why the seven-layer model lost. OSI was consensus without running code. TCP/IP was running code that found its consensus afterwards. The lesson is not that standards are bad. It is that a standard is only as real as the things that run it.

The flag day teaches the other half. The 1983 cutover worked, and it hurt, for the same reason: everything changed at once. Postel's plan gave the change what it needed to survive. A date announced far ahead. A long period when both protocols ran side by side, so a host could switch early and still talk to one that had not. Rehearsals, where the old protocol was switched off and switched back on, so the people who had not noticed the memo found out on a day that did not matter. And on the day, people in the machine rooms, watching.

Every team has since borrowed the pieces, and the borrowed version is the change window: an agreed time when a change may happen, chosen so that the people who made it are present, the people who use the system are not, and the way back is known before the way forward is tried. A good change announces itself, runs the old and new side by side while it can, moves a little traffic first, and watches. A big-bang change does none of these, and its tell is a single vertical line on a graph with everything after it a different colour.

Remember it as: Rough consensus and running code. Agree in the open, prove it works twice, and never change everything on the same morning.

Jonas's merge was a flag day with none of Postel's protections. At 15:58 every payment call at Kettle moved to a new road at once, with no rehearsal, no side-by-side period, no window, and the one person who understood it on a train. The 17:30 restart was a second one. Nothing about that made Jonas wrong, and Ruth said so, twice.

…but

Private connectivity. Nadia read the two words again. Private from what, and seen from where? Jonas had checked from a node. The pods were not nodes. She opened a terminal, and this time, instead of her laptop, she picked a pod.

Chapter 9 · Act IIIEvery House Needs an Address#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

By 18:30 the change log had been pasted into the channel four times, and each time someone added a different emoji. platform: private connectivity to payments, merged at 15:58 by Jonas Lind. Sam had rung Jonas once, got voicemail, and said "wedding, leave him" in a voice that meant it. Nobody wrote Jonas's name again. Nadia read the pull request title three times and then wrote in her notebook: private connectivity. Private from what?

"If it's private," she said, "it has a different address. That's what private means, isn't it? A different door."

"Then look at the door," said Ruth.

Nadia still had the debug pod from the capture earlier. She opened a shell into it and asked it the one question she knew how to ask.

$ dig +short api.paystream.example
10.20.4.31

Then she ran the same command on her laptop, on the office Wi-Fi.

$ dig +short api.paystream.example
203.0.113.10

She wrote both in the notebook and drew a line between them. Same name. Two answers. One of them started with ten, which she had always been told meant ours.

"Read me the first one," said Ruth.

"Ten, twenty, four, thirty-one."

"And what does ten twenty mean to you?"

"It's... internal? Our nodes are ten dot zero dot something. It looks like our stuff."

"Looks like," said Ruth. "Is it yours? Which bit of it is the street and which bit is the house?"

Nadia opened her mouth and closed it. She had typed 10.0.0.0/16 into a form on her second week at Kettle, because the form had asked for it and the example had shown it. She had never once asked what the slash did.

The idea

An IPv4 address is thirty-two bits, written as four numbers from 0 to 255 with dots between them. It is the street address of a building: it says where a machine is, not what it is called. Every packet carries two, one for the sender and one for the receiver, in the envelope that chapter 5 called the IP header.

The address has two parts. The network part says which network the machine is on, and the host part says which machine on that network. The split is not fixed; it is told to the machine when it gets its address, as a subnet mask, a pattern of ones for the network bits and zeros for the host bits, or, in the modern shorthand, as a prefix length after a slash. The prefix is the postcode district and the host part is the house number.

10.0.0.0/16 reads: "the network is the first sixteen bits, 10.0, and everything after that is a house number." That is the whole of the slash. There are sixteen host bits, so 65,536 addresses, from 10.0.0.0 to 10.0.255.255. A CIDR block is any address written with a prefix length like this, and CIDR is the name for the scheme in which the length can be anything from /0 to /32.

A subnet is a CIDR block carved from a larger one. Kettle's VPC, the private slice of the cloud it rents, is 10.0.0.0/16, and inside it each availability zone has a subnet for nodes: 10.0.1.0/24, 10.0.2.0/24, 10.0.3.0/24. Carving matters because the address is how a machine decides what to do with a packet. It compares the destination to its own address, using its mask, and asks one question.

If the destination is on the same network, the machine delivers it directly, on its own street. If not, it hands the packet to a gateway, a machine on its street whose job is to carry things further. The next chapter is about how it finds either of them.

Some blocks are set aside and mean the same thing everywhere. 127.0.0.1 is the machine itself. 0.0.0.0/0 is every address there is, which is why it turns up in rules that mean "anywhere". Three blocks are reserved for private use, addresses that are never handed out on the public internet and that any organisation may use inside its own walls: 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16. Chapter 14 tells the story of why they exist. For now the rule of thumb is the one Nadia already half knew: an address starting with 10. is somebody's inside, and the only question is whose.

Remember it as: an address is a postcode and a house number in one, and the slash says where the postcode ends.

Read 10.20.4.31 with a /16 and the postcode is 10.20. Read Kettle's VPC and the postcode is 10.0. They are both private, both plausible, and not the same district. Whatever was answering to Paystream's name inside the cluster lived in a block Nadia had never been told existed.

…but

10.20.4.31 was valid, private and perfectly ordinary, in a district nobody had mentioned. Something inside the VPC was answering to it. It occurred to Nadia that two things might be.

Chapter 10 · Act IIIWho Has This Address?#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

At 19:00 Nadia had a theory, and she liked it more than she should have. Two things with the same address. She had seen it once in Ruth's cupboard, two printers that had both been typed 192.168.1.20 by two different people, each answering to the name whenever it felt like it. Half the print jobs vanished. It had looked, she remembered, exactly like a third of everything failing.

"An address clash," she said. "Something old at 10.20.4.31, something new at 10.20.4.31, and the pods get whichever answers first."

Sam, passing with a sandwich, said that sounded very plausible, which was what Sam said about theories he could not check.

"Then find both of them," said Ruth.

Nadia searched the cloud console for the address. Not a machine, not a pod. A network interface, the cloud's name for one card in one machine with one address painted on it. And not one: three, one in each availability zone, all born at the same minute.

eni-0a9f3c   10.20.4.31   endpoints-a   created 15:58   VPC Endpoint Interface
eni-07d21e   10.20.5.31   endpoints-b   created 15:58   VPC Endpoint Interface
eni-0b44a7   10.20.6.31   endpoints-c   created 15:58   VPC Endpoint Interface

Every button on the page was greyed out. The interfaces belonged to something that lived in the platform account, where Jonas worked and Nadia did not have a login. But she could read them, and there was no fourth line. Nothing else in the VPC claimed 10.20.4.31. The address had exactly one owner, and the owner was eleven days younger than her notebook.

"Not a clash," she said, and crossed it out, which felt worse than it should have.

"Good," said Ruth. "Now. When your laptop wants to send something to 10.0.1.1, how does it know which van to put it in?"

The idea

A machine that has only now been plugged in knows nothing. It has to answer three questions before it can send a single packet anywhere useful: what is my address, how do I reach the machine next to me, and where do I send everything else.

DHCP, the Dynamic Host Configuration Protocol, answers the first. It is the council assigning a house number when you move in. The new machine has no address, so it broadcasts to everyone on its street, and a DHCP server replies with an offer. The exchange has four steps, and engineers remember them by their initials.

The answer is a lease: the address is lent, not given, for a fixed time, and the machine must renew it or lose it. With the address comes the subnet mask from the last chapter, the address of a gateway, and the address of a name server. In a cloud, the DHCP server is not a machine you can find; the estate itself answers, and the lease is for whatever the interface has been assigned.

ARP, the Address Resolution Protocol, answers the second question. The machine now knows that 10.0.1.1 is on its own street, but the street, the Ethernet from chapter 6, only delivers to MAC addresses, the name painted on the van. So the machine shouts down the corridor: who lives at number 1? The owner answers, and the asker keeps the answer in its ARP cache for a few minutes.

The third question is answered by one line of configuration. The default gateway is the address on your own street to which you send everything that is not on your own street. It is the sign in the post room that says "everything else goes to the main sorting office". The machine ARPs for the gateway's MAC address once, and from then on every packet for another network goes into a frame addressed to the gateway, with the real destination still written inside the envelope. The gateway is a router, and chapter 12 is about what it does next.

Two details matter in a cloud. First, nobody shouts on a real wire. The host that runs your virtual machine answers the ARP question itself, from a table of every interface in the estate, and it will not relay an answer from anyone else. Second, an address does not belong to a machine. It belongs to a network interface, the cloud's record of one card with one MAC address and one or more IP addresses, in one subnet. An interface can be attached to a virtual machine, or to a managed service that has no machine you will ever see. The estate knows exactly where each interface is, and it delivers to the interface, not to whatever is behind it.

Remember it as: DHCP gives you a house number, ARP finds the van for your own street, and the default gateway takes everything else.

That was why there could be no clash. 10.20.4.31 was one interface, in one subnet, created at 15:58, and the estate delivered to it without ever asking who lived there. Nadia's pods were sending their packets to the right door; the door was new.

…but

So nothing was fighting over 10.20.4.31. It had one owner, born at 15:58, and the only wrong thing about it was that a pod asking for api.paystream.example was sent there at all. Something was doing the sending.

Chapter 11 · Act IIIIt's Always DNS#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

At 19:30 Sam typed "it's always DNS" for the third time, and this time Nadia did not roll her eyes, because she was starting to think he was half right.

Something was telling the pods that api.paystream.example lived at 10.20.4.31. Pods do not invent addresses. They ask. So she asked the same question again from a node, and this time she read the whole answer instead of the short one.

;; ANSWER SECTION:
api.paystream.example.   60   IN   A   10.20.4.31
;; SERVER: 10.0.0.2

Two things she had never looked at before. The answer had come from 10.0.0.2, an address at the very bottom of Kettle's own block, which meant the directory being consulted was inside the VPC. And the 60 meant the answer was only good for a minute. She had read somewhere that the payment client library kept addresses for five minutes regardless. She wrote 60 s vs 5 min and underlined it.

Then she asked a different directory. From the same node she sent the question to a public resolver on the internet instead of the one the node had been given.

$ dig +short @1.1.1.1 api.paystream.example
203.0.113.10

"Same name," she said. "Two phone books. Ours says ten twenty. Everyone else's says the real one."

"Which one is lying?" said Ruth.

"Neither. That's the problem. Ours was told to say it."

Ruth nodded as if that were the first properly interesting thing anyone had said all evening. "Then it isn't DNS that's broken. DNS is doing exactly what someone asked it to do at two minutes to four."

The idea

DNS is the phone book of the network, except that nobody prints it. It turns a name people can remember, api.paystream.example, into an address packets can use. When a machine needs that address, it does not go looking itself. It asks a recursive resolver, a server run by its network or its cloud whose job is to do the walking, and the resolver asks a chain of other servers in turn.

The chain follows the tree. At the top are the root servers, which know only who is responsible for each top-level domain, the last label in a name: com, uk, example. The top-level domain's servers know who is responsible for each name beneath them. At the bottom is the authoritative server for paystream.example, run by Paystream or its DNS provider, which holds the actual records and gives the final answer.

Each answer is a record, and records come in types. The ones an engineer meets most often fit in one table.

Every record comes with a TTL, time to live, a number of seconds the resolver may remember the answer for before it must ask again. Caching is why DNS is fast and why it is confusing. The resolver remembers; the operating system remembers; often the application remembers too, for a time of its own choosing that has nothing to do with the TTL. A record changed at 15:58 reaches a machine that asked at 15:57 one minute later, a machine whose client library caches for five minutes at 16:03, and a machine holding an answer it will not re-check until it restarts never.

A resolver inside a cloud network can be told to answer a public name with a private address. That is called split-horizon DNS: the same name gives one answer from inside the estate and another from outside. Clouds sell it as a private DNS zone, a private phone book that overrides the public one for anyone who asks from within the VPC. It is not a fault. It is a feature, and it is what Friday's change switched on.

Two words to recognise for later. DNSSEC adds signatures to records so a resolver can tell a forged answer from a real one. Encrypted DNS carries the question itself inside an encrypted connection so that nobody on the path can read or change it. Neither was in play at Kettle, and neither would have helped: the private answer was signed by nobody and intended by somebody.

Remember it as: DNS is directory enquiries with a memory: the answer you get depends on who you ask and how long ago they last asked.

So Sam was half right. It was DNS in the sense that DNS was handing out a new address, and not DNS in the sense that DNS was obeying. The caches explain the shape of the failure too: pods drifted onto the new answer one at a time as their caches expired, a third at first, more as the evening went on, until the 17:30 restart emptied every cache at once.

…but

DNS had done what it was told: inside the estate, send Paystream's name to 10.20.4.31. The pods were obeying, and their packets were leaving with the right address on the front. Whether they could get there was a question for the sorting office.

Chapter 12 · Act IIIThe Sorting Office#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

At 20:00 the office had emptied to the three of them and the hum of the fridge. Sam had gone to sit with the CEO. Nadia had the packets leaving the pods with 10.20.4.31 on the front, and she had the interface at the far end. What she did not have was the road between.

"If the address is in a different district," she said, "then something has to carry it there. A gateway. Which means a route."

"Then read the routes," said Ruth.

The cloud console had a page for it. Every subnet in the VPC was attached to a route table, a short list of where packets for each destination should go. Nadia opened the one for the node subnets.

Destination       Target
10.0.0.0/16       local
10.20.0.0/16      local
0.0.0.0/0         nat-0c71e2

There was one more line, between the second and the last. She did not read it, because it was not what she was looking for. What she was looking for was there: 10.20.0.0/16, target local. The estate would carry it itself. No gateway, no box in the middle, nothing to blame.

To be sure, she went back to the debug pod and traced the road.

$ traceroute -n 10.20.4.31
 1  * * *
 2  * * *

"Stars," she said. "No hops at all. Not even one."

"Local means the estate delivers it by hand," said Ruth. "One street to the next, no sorting office between. The stars are the other end not answering, which we already knew. Routing takes the letter to the door, Nadia. It doesn't make anybody open it."

Nadia wrote routing: fine and, after a moment, annoyingly.

The idea

A router is a machine with more than one street, whose only job is to take a packet from one and hand it to the next on the way to its destination. It is the sorting office. It does not read the letter and it does not care who sent it. It reads the postcode on the front and puts the letter in a pigeonhole.

The pigeonholes are the routing table, a list of rows, each with a destination prefix and a next hop. The next hop is either an interface, meaning "this street, deliver it directly", or the address of another router on one of its streets, meaning "hand it to them". Rows are written as CIDR blocks from chapter 9, and a packet's destination may match more than one.

That is where the one rule of routing comes in. A router chooses the row with the longest prefix match: among all the rows whose block contains the destination, the one with the most network bits wins, because it is the most specific. A /24 beats a /16, which beats a /8, which beats the last row of all.

The last row is the default route, 0.0.0.0/0, which matches every address and wins only when nothing else does. It is "everything else goes to the main sorting office". On a laptop it points at the home router. In Kettle's VPC it points at the NAT gateway, the machine that carries traffic out to the internet, which chapter 14 explains. The word local in a cloud route table means the estate itself is the next hop: every subnet in the VPC can reach every other, and the cloud's fabric delivers the packet without a router you can see.

Routing is hop by hop. No router knows the whole road. Each one chooses only the next office and trusts that office to choose the one after, which is why two routers that each think the other is the way forward can pass a packet between them for ever. To stop the for ever, every packet carries a TTL, a counter that starts at some number like 64 and goes down by one at each router. When it reaches zero the router drops the packet and sends a small message back to the sender saying time exceeded, and from where.

That small message is the whole trick of traceroute. It sends a packet with a TTL of 1 and notes who complained. Then a TTL of 2, and a TTL of 3, until the destination itself answers. Each complaint names one router on the road.

A row of stars means nobody complained at that distance. The router may be configured not to answer, or there may be no router there at all, because the estate delivers locally and the far end is silent.

Routes get into tables in two ways. A static route is typed by a person and never changes. A dynamic route is learned from a routing protocol, in which routers tell each other what they can reach. Inside one organisation the protocols are called interior: RIP, which counts hops and is rarely used now, and OSPF, in which every router shares a map of its links and computes the shortest paths itself. A cloud route table is static, written by whoever owns the VPC, and the local rows are written by the cloud and cannot be removed.

Remember it as: a router reads only the postcode, picks the most specific pigeonhole, and hands the letter one office on.

Kettle's table had a row for 10.20.0.0/16, it was the longest match, and it said local. The road to the new door was short, straight and correctly signed, which meant the thing that was dropping the packets was not on the road.

…but

The estate carried the letter to the door. Before Friday, it had carried the same letter a great deal further: out of the building, across the city and back in. Nadia wanted to see that old road.

Chapter 13 · Act IIIThe Map Nobody Owns#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

At 20:30 Nadia did something she had not done all evening: she sent a packet to the address that used to work. Not the name, which the VPC's phone book would turn into 10.20.4.31, but Paystream's public address itself, straight from the debug pod.

$ traceroute -n 203.0.113.10
 1  * * *
 4  198.51.100.7    1.1 ms
 9  203.0.113.10    7.4 ms

Nine hops. One of them, at hop 4, was the NAT gateway's public address, the door the estate used to send things into the world. After that the road went somewhere she had never looked: a handful of routers with addresses that belonged to the cloud provider, then one that did not, then Paystream. And Paystream answered. The old road still existed. Nobody had closed it. Jonas had built a shorter one beside it and moved the signpost.

She wanted to know whose the middle routers were. Ruth showed her where to ask: a public register of who announces which blocks of addresses to the rest of the internet.

$ whois -h whois.radb.net 203.0.113.10
route:    203.0.113.0/24
origin:   AS64500
descr:    Paystream Ltd

"AS," said Nadia. "Autonomous system?"

"A country, roughly," said Ruth. "With its own post office. The cloud is one. Paystream's hosting is another. Before Friday your checkout letters left one country, crossed at least one more, and came back. Now they never leave the estate. The internet never saw tonight's packets, Nadia. Whatever is wrong with them, it's wrong inside your fence."

Nadia wrote old road: out and back. New road: never leaves. It was, she thought, the first thing all evening that made the change sound like an improvement.

The idea

Inside one organisation, routers share a map and agree on the shortest path. The internet is not one organisation. It is tens of thousands of networks, each run by a company, a university, a cloud or a government, each with its own routers and its own rules. Each of those is an autonomous system, an AS, identified by a number, and the question of how a packet crosses from one to the next is the question of how the internet exists at all.

The answer is BGP, the Border Gateway Protocol. A router at the edge of each AS keeps a session with a router at the edge of each neighbouring AS. Over that session it announces the address blocks it can deliver to, and with each block the AS path, the list of AS numbers the announcement has passed through on its way. A neighbour that hears the same block from two directions prefers, all else being equal, the shorter path. Then it adds its own number and passes the announcement on. The picture a router ends up with is not a map of the internet. It is a list of blocks and, for each, the best next-door neighbour to hand it to.

Neighbours come in two kinds, and the difference is money. Transit is a service: a smaller network pays a larger one to carry its traffic to the whole internet and announce its blocks to everyone. Peering is an agreement between two networks to exchange traffic between their own customers directly, usually for free, because both save on transit. Much of that peering happens at an internet exchange, a building where many networks plug into one shared switch and can peer with each other over it. It is the postal treaty: countries agree which of each other's mail they will carry, and on what terms.

Two rules from chapter 12 still hold out here, and both have teeth. Routing is hop by hop, so no AS controls the road beyond its own border. And the longest prefix wins, so an AS that announces a more specific block than the real owner takes the traffic, whatever its path length. BGP has no way of checking that the network announcing a block is allowed to. It was built on trust, and it still mostly runs on it.

The repair, slowly spreading, is RPKI, a signed statement from the registry that owns a block saying which AS may announce it. A router that checks these statements can drop announcements that fail. Many large networks now check. Many do not. The map of the internet is still a thing nobody owns, assembled every minute from what each network tells the next.

A cloud customer is not an AS. The cloud provider is, and it speaks to the world on behalf of every VPC inside it. A packet from a pod to the public internet leaves the estate through the NAT gateway, is announced to the world under the provider's AS number, and comes back the same way.

Remember it as: BGP is countries' post offices telling each other which addresses they can reach, and believing each other.

That was the old road for Kettle's checkout: out through the cloud's AS, across an exchange or a transit link to Paystream's AS, and back. The new road did not cross a single border, so nothing on the internet could be at fault, and nothing on it could help.

…but

If the new road was so short and so private, why had Jonas built it at all? The old one still worked. Nadia went back to the pull request and, this time, read the description instead of the title.

Chapter 14 · Act IIIRunning Out of Houses#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

At 21:00 Nadia read the pull request properly, and found that Jonas had written four careful lines that nobody in the channel had quoted.

Why: during the September sale the NAT gateway hit ErrorPortAllocation
(peak 1,900/min) and checkouts failed for 20 min. NAT data processing is
also our largest network bill. Move Paystream to a private endpoint, no NAT.
Tested from node ip-10-0-1-42: curl https://api.paystream.example -> 200

She remembered the September sale: a sharp tooth in the error rate at the busiest ten minutes of the year, and a post-mortem with the words port allocation in it that she had not understood and had not asked about. Jonas had understood it. He had fixed it, on a Friday, and tested the fix.

"He tested it," she said. "From a node. It worked."

"From a node," said Ruth, and let it sit there. "Tell me what the NAT gateway does."

"It... lets the pods reach the internet. They've got private addresses, so it swaps them for a public one on the way out."

"One public one. For how many pods?"

"Forty-one. More in a sale. Each one opening connections."

"And how many desks does a post room have?"

Nadia looked at the word port in the pull request and felt it change meaning. Then Ruth said the other thing, the one Nadia wrote down without yet knowing why it mattered.

"Something else. On the old road, whatever address a pod had, Paystream never saw it. It saw the post room's address. The gateway's. That's gone now. Whoever's standing at the new door is seeing the pods as they really are, for the first time."

The idea

IPv4 has about four billion addresses, and the world has far more than four billion things that want one. The early allocations were generous, whole class A blocks to single organisations, and by the mid-nineties it was clear the pool would exhaust. Two answers were designed at once: a bigger pool, and a way to share the pool you had. The second is the one almost every engineer lives behind today.

RFC 1918 defines three private address ranges. Addresses in them are never announced on the public internet, so any organisation may use them inside its own network, and millions do, all using the same numbers.

A machine with a private address can send a packet to the public internet, but nobody out there can send one back, because no router outside knows a road to 10.0.1.42. NAT, Network Address Translation, is the box at the edge that solves this. It is the company post room: one public address on the building, and inside it a table of which desk each letter belongs to.

Sharing one public address among many machines needs more than swapping the address. Two pods may both open a connection to Paystream at the same moment, and the replies must find their way back to the right one. So the translator rewrites the source port too, the flat number from chapter 7, and keeps a row for each connection. This is port translation, and it is what nearly every NAT does.

The table is the limit. A port number is sixteen bits, so one public address has at most 65,536 ports, and in practice a translator keeps perhaps 55,000 of them for use. Each open connection from inside to the same outside address and port needs a row of its own. During a sale, hundreds of pods each opening dozens of connections to one payment API can use every row, and the next connection has nowhere to go. The gateway reports an error that engineers see as port allocation failed and customers see as a spinner. That is port exhaustion, and it was Jonas's September.

Two side effects of NAT matter as much as the thing itself. The first is that it hides the inside. Paystream's logs for the old road showed one address, the gateway's, for every pod at Kettle, because that is the only address that ever left the building. The second is that it blocks unsolicited inbound traffic, since a packet arriving from outside with no row in the table has no desk to go to and is dropped. This is NAT as accidental security: real protection, by accident, which people mistake for a firewall.

When the carrier runs out of public addresses too, it puts its customers' NAT boxes behind another one. This is carrier-grade NAT: a home router translates to a middle address, and the carrier's translator turns that into a public one. The middle layer needs addresses that are neither private, because customers already use those inside their homes, nor public, so a separate range was set aside for it, 100.64.0.0/10. Ruth said it turned up in the oddest places, which Nadia wrote down without asking why.

The bigger pool exists. IPv6 addresses are 128 bits long, written in hexadecimal with colons, like 2001:db8::1, and there are enough that nobody needs to share one. Adoption is slow for a plain reason: NAT works well enough, and IPv6 only helps once both ends and every road between speak it, which is the flag-day problem from chapter 8 spread across the whole planet. Most networks now run both, and most traffic still arrives over the old one.

Remember it as: NAT is the company post room: one public address outside, desk numbers inside, and a finite number of pigeonholes.

Friday's change took NAT out of the road to Paystream, which was the right cure for port exhaustion and for the bill. It also meant that, for the first time, the thing at the far end of the road saw each pod's own address instead of the post room's.

…but

Jonas had taken the post room out of the road for a good reason, and the new road was short, private and silent. A letter that arrives and is neither answered nor refused smells like a guard with a list. But the private letterbox had been installed from the platform account, and Nadia's badge did not open that door.

Chapter 15 · Act IVThe Worm#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

By 22:00 the office was down to two people and the hum of the fridge. Nadia's notebook had a page headed What we know, and the list had stopped growing an hour ago. Inside a pod the name went to 10.20.4.31. The route to it was local. Jonas had built the endpoint to stop paying the NAT gateway. A SYN left the pod and nothing came back.

Ruth had the capture from the debug pod open again. "Nothing," she said. "Not a reset. Not an error note from a router. A machine that is switched off gives you this. So does a machine that is on and has decided not to speak to you."

"A firewall."

"Something with a list. What do you have in your account that has a list?"

"Nothing called a firewall. I'd have seen it."

"You'd have seen a box with firewall written on it. That is not the same thing."

Nadia went back to the interface she had found on Friday evening. She could still see it. What she could not see was anything attached to it.

$ cloud network-interface list --filter address=10.20.4.31
10.20.4.31  eni-0a9f3c  endpoints-a  owner=platform-shared  rules: (no access)

"Platform-shared," Nadia read. "That's Jonas's account. The one with the endpoints and the shared subnets. I've got no role in it."

"Who grants one?"

"Jonas. Or Sam, for read-only, and Sam's been on the phone to the CEO since seven."

"Then ask Sam in the morning." Ruth stood and picked up her mug. "The thing you need to look at is behind a door you can't open tonight. Write down the question you would ask it, and go home."

Nadia wrote: Is there a guard on 10.20.4.31, and what is on its list? She underlined list, closed the notebook and went to find her coat.

The idea

A firewall is a guard at the door with a list. Packets arrive, the guard compares each one with the list, and the packet goes through or it does not. That is the whole job. Everything else is detail about what is on the list and how much the guard remembers.

The simplest kind is a packet filter. It looks at one packet at a time and only at the envelope: where it came from, where it is going, which protocol, which port. A rule might say allow TCP to port 25 on the mail server from anywhere or refuse everything to port 23. Packet filters are fast and dumb. Their weakness is replies. If a machine inside is allowed to open a connection out, the answers have to be allowed back in, and a filter that cannot tell an answer from a fresh attempt has to leave a gap for anything that looks like one. Attackers learned to make their packets look like answers.

A stateful firewall closes that gap by remembering. When it sees the first packet of a connection go out, the SYN from chapter 7, it writes down the pair of addresses and ports. When the reply comes back it checks the note and lets it through. A packet that matches no note and no rule is treated as a stranger.

The list can be written two ways round. A list of things to refuse lets everything else in; a list of things to allow refuses everything else. The second is called default deny, and after 1988 it became the rule for anything facing the outside. You do not list what is dangerous. You list what is needed, and the guard turns away the rest.

Some machines have to be reachable from outside, or there would be no point having a connection. The pattern that emerged was the DMZ, a demilitarised zone: a small network between two guards, holding only the things the outside is meant to see, such as the web server and the mail relay. The outside can reach the DMZ. The DMZ can reach a few named things inside. Nothing outside reaches inside directly.

When the guard turns a packet away there are two ways to do it. Reject sends something back: a TCP reset, or a small error note, so the sender learns at once that the door is shut and gives up. Drop sends nothing. The packet vanishes, the sender waits for its own timer to run out, tries again, waits again, and then reports a timeout. Drop is the default almost everywhere, because it tells an attacker nothing and costs the guard nothing. The price is paid by honest senders, who cannot tell a shut door from an empty building.

Remember it as: A firewall is a guard with a list; stateful means it remembers who went out so it can let the reply back in.

Put all of this together and you have the perimeter model: a hard shell around a soft centre. One boundary, well guarded. Inside it, machines trust each other, because the guard already checked. The model is older than the cloud, and the cloud borrowed its words and its habits. Every VPC has lists at its edges, and the guard still drops rather than rejects.

That is why Nadia's capture looked the way it did. A refusal would have come back in a millisecond as connection refused. Silence for sixty seconds is a packet dropped on purpose, by something with a list, and the list was not in her account.

…but

On the walk to the station Nadia thought about the grey box the customers were seeing. 504 Gateway Time-out. Checkout's own log said connect timeout, and nobody at Kettle had written either sentence. Something between the phone and the pod was giving up first, and speaking for both of them.

Chapter 16 · Act IVA Page Made of Links#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

At 08:00 on Saturday Nadia was at her kitchen table with the laptop, the notebook and the Kettle app open on her phone. The overnight numbers were worse: since the restart at 17:30, four checkouts in five were failing. Sam had posted a screenshot from a customer. A spinner, then the grey box.

She tried it herself and counted under her breath. The spinner turned for thirty seconds, almost exactly, and the box appeared. She tried from the laptop, against the same address the app used, and asked for the time.

$ curl -o /dev/null -w '%{http_code} after %{time_total}s\n' -X POST https://kettle.example/api/checkout
504 after 30.04s

Thirty seconds. Then she opened the checkout logs for the same minute.

08:03:12 checkout-7d9f  POST /checkout  error="connect timeout"  peer=10.20.4.31:443  waited=60.0s

Sixty seconds. She rang Ruth, who answered on the second ring and sounded as if she had been up for hours.

"The customer gets a 504 at thirty seconds," Nadia said. "Checkout gives up at sixty. So when the customer sees the error, checkout is still waiting. It's still trying. For another half a minute."

"Who sends the 504?"

"Not checkout. It never got that far." She looked at the path in her notebook, the one Ruth had made her draw on Friday. Phone, CDN, load balancer, pod. "The load balancer. It's the thing that hands requests to the pods. It waits thirty seconds for a pod to answer, and then it answers for the pod."

"And what does the phone do when it gets a 504?"

Nadia looked at the app. The grey box had a button on it. The button said Try again.

The idea

Everything in this book so far has been envelopes. HTTP is the letter inside them. It is the thing the phone wrote and the thing checkout read, and it is short enough to read by eye.

A URL names what you want: a scheme such as https, a host such as kettle.example, an optional port (443 if the scheme is https and you say nothing), a path such as /api/checkout, and sometimes a query after a ?. The host part goes to DNS. The rest goes in the request.

An HTTP request is a few lines of text. The first line carries a method, the verb, and the path. GET asks for something and promises not to change anything. POST sends something that will change things, such as an order. PUT and DELETE do what they say. After the first line come headers, one per line, each a name and a value: Host says which site you mean, Content-Type says what the body is, Cookie carries the scrap of memory. Then a blank line, then the body, if there is one.

The response has the same shape the other way round. Its first line carries a status code, a three-digit number whose first digit tells you the kind of answer. 2xx is success, 200 the usual one. 3xx means look elsewhere. 4xx means the request was wrong: 404 for a path that is not there, 403 for not allowed. 5xx means the server side failed: 500 is a crash, 503 is overloaded, and 502 and 504 are special. They are not sent by the server you asked for. They are sent by something in between, on its behalf, to say that it could not get an answer: 502 for a bad one, 504 for none at all in the time allowed.

HTTP is stateless: each request stands alone, and the server keeps nothing between one and the next. That is what lets one address serve millions, and what lets any pod of forty-one answer any request. A cookie is how a site remembers you anyway. The server sends a Set-Cookie header with a value it will recognise, the browser stores it, and sends it back as a Cookie header on every later request to that site. The memory lives in the client. The server only has to recognise the token when it comes back.

Every hop a request passes through has a timeout: how long it will wait for the next hop before it stops waiting and tells the one behind it. The phone has one. The load balancer has one. Checkout has one for Paystream. Each was set by a different person on a different day, and nothing makes them agree. Whichever hop has the shortest clock speaks first, and it speaks for everything behind it.

Remember it as: HTTP is a letter and a reply, each standing alone; cookies are the memory; every hop on the way has its own clock.

The question mark in that table was in Nadia's notebook, and she did not like it. At Kettle the load balancer's thirty seconds ran out first, so the customer saw a 504 while checkout was still waiting for a SYN-ACK that would never come. For the next thirty seconds that pod was busy doing nothing, and the customer had a button that said Try again.

…but

Sam rang at twenty-five past eight. He had been reading the cloud provider's documentation in bed, which was never a good sign, and he had an idea. The endpoint had a hostname of its own, a long one. What if checkout called that, and left api.paystream.example out of it?

Chapter 17 · Act IVTalking in Private#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

Sam's idea, at 08:30, went like this. The endpoint in the platform account had a hostname of its own, generated by the cloud: vpce-0f3a9c1d7b2e.pay.endpoints.cloud.example. It pointed at the same three private addresses. Checkout's configuration held the Paystream address as a URL. Change the URL to the endpoint's own name, and whatever DNS was doing to api.paystream.example would stop mattering. "It's a config change in our account," he said. "We can ship that. If it's DNS, that's the fix."

Nadia did not think it was DNS any more, but she had learned to test the idea rather than argue with it. The endpoint name would not resolve from her kitchen, so she looked instead at what Paystream would present when a connection arrived. From the laptop, against the public address, she asked for the certificate and read the names on it.

$ openssl s_client -connect api.paystream.example:443 </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
X509v3 Subject Alternative Name:
    DNS:api.paystream.example, DNS:*.paystream.example

Two names, both Paystream's. Nothing under cloud.example. A client told to call vpce-0f3a9c1d7b2e.pay.endpoints.cloud.example would connect, be handed this certificate, look for its own hostname on the list, and hang up.

"Then skip the name," said Sam. "Hard-code 10.20.4.31. It's not going to move."

She tried the equivalent from the kitchen, against Paystream's public address.

$ curl https://203.0.113.10/v1/ping
curl: (60) SSL: no alternative certificate subject name matches target host name '203.0.113.10'

"Same thing," she said. "The client checks the name it was asked for against the certificate. An address isn't on it either."

"There's a flag to turn that check off."

Ruth, who had been on the call the whole time and silent, spoke for the first time. "There is. And then anyone standing between you and Paystream can be Paystream, and you'll pay them."

The idea

TLS is a sealed, tamper-evident envelope with a verified return address. It wraps a connection, usually a TCP one, and gives the two ends three things. Privacy: nobody in between can read what is inside. Integrity: nobody in between can change it without the change being noticed. Identity: the client knows which server it is talking to. HTTPS is HTTP inside that envelope, on port 443 instead of 80. Every other protocol this book meets later can go inside the same envelope.

The hard part is the third thing. Scrambling bytes is well understood. Knowing that the machine you are scrambling them for is Paystream, and not something between you and Paystream, needs a piece of paper.

A certificate is that paper: a statement, signed by a certificate authority that your client already trusts, that the holder of a particular key is entitled to a list of names. The list is what matters here. It is a list of DNS names, such as api.paystream.example, and it can carry a wildcard such as *.paystream.example. It very rarely carries an IP address. When a client connects, the server presents the certificate and proves it holds the key. The client checks three things: that the signature leads back to an authority it trusts, that the dates are current, and that the name it was asked to call is on the list. Fail any of the three and the client hangs up before a byte of HTTP is sent.

The first message carries the name the client wants, in the clear, before anything is sealed. That is SNI, Server Name Indication, and it exists because one address often serves many names. Without it the server would have to guess which certificate to show before the client had said which site it wanted. With it, the client says the name, the server picks the matching paper, and the check can succeed.

That is why the name is the thing. The client decides what name it is calling, from the URL it was given. The certificate lists names. If the two do not meet, there is no connection, however correct the address underneath. Tell a client to call 10.20.4.31 and it will look for 10.20.4.31 on the list.

The check is not fussiness. Anyone who can sit between a client and a server, at a café router, at an ISP, or at a private address inside a VPC, can accept the client's connection and open their own to the real server, reading and changing everything that passes. That is a man-in-the-middle, and the certificate check is the only thing that stops it. The attacker can present a certificate, but not one with the right name signed by an authority the client trusts. Turn the check off and the envelope is still sealed. It is sealed to whoever answered.

The handshake costs a round trip in TLS 1.3, two in older versions, on top of the one TCP needs. For a request a second that is nothing. For a new connection on every request it adds up, which is why clients keep connections open and why a later chapter cares about that.

Remember it as: TLS seals the envelope and checks the return address: the name you asked for must be on the certificate, or you hang up.

How the keys are agreed, what the signatures are made of and how an authority earns trust are the business of Cybersecurity: The Keep. For this book the point is the name. Sam's shortcut failed because the check was doing its job: Paystream's certificate names Paystream, and the endpoint's hostname belongs to the cloud. The connection would be sealed and would go nowhere.

…but

At 09:10 Sam's approval came through: a read-only role in the platform account, at last. Nadia clicked the console link, signed in, and got one line in return. Sign-in from this network is not permitted.

Chapter 18 · Act IVThe Tunnel Home#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

Sign-in from this network is not permitted. Nadia read it three times. The role was real. Sam had sent her the confirmation. The platform account's console had looked at where her request came from, which at 09:30 on a Saturday was a kitchen in a flat with a broadband router, and shut the door.

"It's allowed from the office," Sam said. "Jonas locked it down after the audit. The console only takes sign-ins from the office address."

"I can be in the office in forty minutes."

"Or," said Ruth, "you can bring the office to you. There's a VPN. It runs on one of my beige boxes. It's on the handover list, page four."

Nadia had used the VPN once, in her first week, to reach a printer. She opened the client, signed in with her usual account and a code from her phone, and watched the small icon turn from grey to green. Then she asked a public website what address it saw her coming from, before and after.

$ curl -s https://myaddr.example/      # home broadband
81.2.69.144
$ curl -s https://myaddr.example/      # with the VPN up
198.51.100.20

The second address was Kettle's. Her packets were leaving the flat sealed, arriving at the office, being unwrapped there and going out again through the office's front door, so that the console saw them coming from the building. The video call with Sam stuttered and froze, because the video was now going to the office and back as well.

She reloaded the console. It let her in. For the first time since Friday afternoon she could see the platform account: the shared subnets in 10.20.0.0/16, the endpoints, and in the list of endpoints, one created at 15:58 the day before, named paystream-private.

The idea

A VPN, a virtual private network, is a diplomatic pouch. Your packets travel through the public post, sealed, addressed to an embassy, and when the pouch is opened on the other side the letters inside are delivered as if they had been posted there. The public network carries them. It cannot read them, and it does not know where they are really going.

The mechanism is the envelopes-inside-envelopes trick from chapter 5, with a seal. The laptop builds a packet for a private address, say the office file server. Instead of sending it, a VPN client encrypts the whole packet and puts it inside a new one addressed to the VPN gateway, a machine with a public address at the edge of the private network. The outer packet crosses the home router, the ISP and the internet like any other. The gateway strips the outer packet, checks the seal, and forwards the inner one onto the private network. Replies come back the same way in reverse.

IPsec does the sealing at the IP layer, packet by packet, which suits gateways. The other common kind, a TLS VPN, carries the same inner packets inside a TLS connection on port 443, which suits laptops behind hotel firewalls. The shape is the same. What differs is which layer does the wrapping.

There are two ways to use a tunnel. Site-to-site joins two networks: a gateway at each end, always on, and nobody in either office knows it is there. This is what replaced the leased line, and it is what joins an office to a cloud network in a later chapter. Remote access joins one machine to a network on demand: the laptop at the kitchen table becomes, for as long as the icon is green, a machine in the office.

How much becomes the office is a choice. With split tunnelling only traffic for the private ranges goes through the tunnel, and everything else, the video call and the music, goes straight out of the home router as before. With a full tunnel everything goes through the office, which is why Nadia's call froze. Full tunnels are safer in one way, since the office sees and can filter everything, and worse in another, since the office link now carries every employee's video.

Remember it as: A VPN is a diplomatic pouch: your packets travel sealed through the public post and come out inside the office.

Now put this next to chapter 15. The firewall drew a line around the building and trusted what was inside. The VPN moves that line. A laptop in a kitchen, on a home network the company has never seen, is inside the perimeter, with the perimeter's trust, and the guard at the door sees a colleague. That is the VPN's whole value and its whole risk. Everything the laptop has picked up comes in with it, and anyone holding its credentials comes in from anywhere. The ciphers and the seals belong to Cybersecurity: The Keep. The network lesson is shorter: a rule that trusts an address lets in whoever can arrange to have that address.

That is what let Nadia in: the console trusted the office's address, the VPN gave her the office's address, and the lock had checked where she came from, not who she was.

…but

The endpoint named paystream-private had a status column. Nadia clicked through. Three network interfaces, one per zone, 10.20.4.31 among them, and next to each a green tick and the word Healthy. Something had checked those interfaces in the last minute and found them well. Checkout had been trying for seventeen hours and had never once got an answer.

Chapter 19 · Act IVOne Name, Many Servers#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

The VPN had taken three attempts and a code from her phone, and now the platform account's console was open on the kitchen table. Nadia had never been inside it. It looked like Kettle's own account, the same grey menus in the same places, but the names in it were Jonas's names, and there were more of them than she expected.

She searched for "paystream". One object, created on Friday at 15:58, state available. Three network interfaces, one in each of the endpoint subnets, and there, in a neat row, were the addresses she had been chasing since Friday evening.

vpce-0f3a9c1d7b2e   paystream-private   available   created Fri 15:58
  10.20.4.31   10.20.5.31   10.20.6.31
  provider health: healthy (3 of 3)

"Healthy," she said into the phone. Ruth had rung at ten from her kitchen in Reading and said she would stay on the line while she did the crossword. "Three of three. Everything on this page says it's fine."

"Who sends the health checks?"

Nadia read the line again. It belonged to the provider's side: Paystream's own load balancer, checking Paystream's own servers behind the endpoint. "Paystream does. To itself."

"From where?"

"From inside Paystream."

"And your balancer. The one in front of checkout. What does it check?"

She opened Kettle's account, the one she knew. Forty-one targets, forty-one healthy. The check was a GET to /healthz every ten seconds, and /healthz returned 200 whenever the process was running. It did not call Paystream. It did not call anything.

"So," said Ruth. "Two sets of green lights, and not one of them has ever sent a packet from a pod to that endpoint. You've been reading a thermometer that isn't in the patient."

Nadia wrote it down and underlined it: health checks test the checker's path, not the customer's.

The idea

A load balancer is a machine, or a managed service, that owns one address and stands in front of many servers. Clients connect to the one address. The balancer passes each connection, or each request, to one of the servers behind it and sends the answer back. It is the reception desk: one address on the door, visitors sent to any free clerk, and nobody outside needing to know how many clerks there are today.

The cheapest way to have one name and many servers is round-robin DNS: give the name several A records and let the name server hand them out in rotation. It costs nothing and it half works. DNS does not know which servers are alive, so a dead one stays in the rotation until a person removes it, and resolvers cache whichever answer they got for the record's TTL, so clients are pinned to one server for minutes at a time. It is still used to spread traffic across balancers in different regions; behind the name, a real balancer does the work.

There are two kinds, named after the layers from chapter 5. An L4 load balancer works at the transport layer: it sees addresses and ports, picks a server for each new TCP connection and forwards the packets, never reading what is inside. It is fast and it understands nothing. An L7 load balancer works at the application layer: it reads the HTTP request and can send /api/pay to one set of servers and /images to another, add a header, or reject a bad request before any server sees it. To do that it has to be a reverse proxy: a server that accepts the client's connection, makes its own separate connection to a backend, and copies the request across. Two connections, two handshakes, and the backend sees the proxy's address, not the client's, unless the proxy writes the client's address into a header. Kettle's public balancer is this kind.

A balancer only knows what it asks. A health check is a small request it sends to every server every few seconds; a server that fails a few in a row is taken out of rotation until it passes again. The check is a question, and the question is chosen by whoever set the balancer up. "Is the process listening?" is the usual one. It is not "can this server do the thing customers need?", and the gap between those two questions is where Saturday morning's green lights came from.

Three more things ride on the balancer. A sticky session sends the same client back to the same server each time, by a cookie or by source address, for applications that keep state in memory. TLS termination is the balancer holding the certificate and ending the encrypted connection itself, so the certificate must carry the public name, as chapter 17 found, and the hop inside can be plain HTTP or a second TLS connection. And keep-alive is the agreement to keep a TCP connection open after a request so the next one can skip the handshake; a connection pool is a client holding several such open connections ready for reuse. Pools sit between the balancer and the pods, and between each pod and Paystream. They make everything faster, and they remember the world as it was when they were opened.

Remember it as: One address on the door, any free clerk behind it, and a reception desk that only knows what it asks.

At Kettle the balancer asks /healthz, and /healthz says yes while every payment behind it times out. Paystream's balancer asks its own servers, from its own side of the endpoint. Nobody is asking the question that matters, which is whether a pod can reach the address it has been given.

…but

At 10:25 Sam forwarded a message from Marketing. They had checked from four phones and a laptop, and the site was up, and could engineering please stop saying it was down.

Chapter 20 · Act IVCloser to You#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

Nadia loaded the storefront on her phone with the Wi-Fi off, the way Marketing would have. It was fast. The home page, the Ethiopian, the roaster in Leeds, the basket: all of it came up before she had finished lowering her thumb. She could see why they were cross.

Then she asked the site to tell her where the page had come from.

$ curl -sI https://www.kettle.example/ | grep -iE 'cache|age'
cache-control: public, max-age=3600
age: 2210
x-cache: HIT

Age 2210. The page in her hand had been made thirty-seven minutes ago and was being handed out again from a copy. The server that built it, Kettle's storefront pods behind Kettle's load balancer, had not been asked. She tried the checkout page.

$ curl -sI https://www.kettle.example/checkout | grep -iE 'cache|age'
cache-control: no-store
x-cache: MISS

No copy, by instruction. Every request for anything to do with paying went through to the origin, and from the origin to a pod, and from the pod to an address that did not answer.

"Marketing checked the forms," said Ruth, when Nadia explained it. "The branch office has plenty of forms. It doesn't have a cashier."

"So the site is up."

"The shop window is up. Tell Sam that, in those words. It'll save him a conversation with the CEO." A pause, and the scratch of a pencil. "And write down what 'up' meant to the person who said it. That's a fact about the incident, not a complaint about Marketing."

Nadia typed the shop window is up, the till is not into the channel, and three people reacted with a coffee cup.

The idea

A cache is a copy kept close so the next request does not have to go all the way to the source. Your browser has one. So does the resolver from chapter 11; a cached DNS answer is the same idea with a TTL on it. The web's largest caches belong to a CDN, a content delivery network: many servers in many cities that hold copies of a site's pages, images and scripts and answer for the site in its own name. Each of those places is an edge location, a building with servers near a large population and good connections to the local networks, and the server you own, the one behind the load balancer from chapter 19, is the origin. The branch office holds copies of the popular forms; the head office prints them.

A request reaches the nearest edge first. If the edge has a fresh copy, it answers at once, and the origin never hears about it. If not, the edge fetches from the origin, keeps a copy if it is allowed to, and answers. What "allowed" means comes from the origin's own headers: cache-control: public, max-age=3600 says anyone may keep this for an hour; no-store says nobody may keep it at all. A checkout page, a basket, anything that differs per person or changes the world when you ask for it, is no-store, and for it the edge is only a corridor.

How does a phone in Leeds find the Manchester edge? Two ways. The CDN's DNS can look at where the question came from and answer with the address of a nearby edge, which is round-robin DNS with a map. Or the CDN uses anycast: the same address is announced from every edge location at once, through the BGP of chapter 13, and the ordinary routing of the internet carries each packet to whichever announcement is nearest. One address, two hundred buildings, and your packet goes to the closest one without anyone choosing. It also means a flood aimed at that address is split across all two hundred, which is a point the next chapter needs.

What the edge keeps is filed under a cache key: usually the full URL, sometimes with a header or two, such as the language. Two requests with the same key get the same copy, which is why a page that differs by user must say so or be no-store. A copy lives until its age passes its max-age, or until the origin tells the edge to forget it early, which is cache invalidation, also called a purge. Invalidation is the part everyone gets wrong, because it is the only part that needs the origin to remember every place it has left a copy. The edge also stands between the internet and the origin as a security layer, where a WAF, a web application firewall, inspects each request for attacks on the application itself; that work belongs to Cybersecurity: The Keep.

Remember it as: The branch office has copies of the popular forms, so "the site is up" only means the forms are in stock.

Kettle's storefront is a shelf of forms with an hour on them, and the edge has been handing them out since Friday without needing anyone in the building. Checkout is no-store. It walks past the shelf, through the origin, into a pod, and stops.

…but

At 10:50 the load balancer's own graph moved. Requests to checkout were climbing, double what a Saturday morning should carry and still rising, and Sam asked the question the whole channel had been thinking. "Are we being attacked as well?"

Chapter 21 · Act IVThe Flood#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

Nadia had seen the shape before, on a slide at a conference: a flat line and then a cliff going the wrong way. Requests to /api/pay were running at six times a normal Saturday and the balancer's queue was growing. If she had been told nothing else she would have said the same as Sam.

She looked at where it was coming from. Not a thousand strangers. One user agent, Kettle's own mobile app, and a short list of customers. She picked one order number and searched the checkout logs for it.

10:58:31 pay order=K-88213 req=a71 try=1 paystream: connect timeout 60 s
10:59:31 pay order=K-88213 req=a71 try=2 paystream: connect timeout 60 s
11:00:31 pay order=K-88213 req=a71 try=3 paystream: connect timeout 60 s
11:01:33 pay order=K-88213 req=c09 try=1 paystream: connect timeout 60 s

One order. Checkout had tried Paystream three times, a minute apart, with nothing between the tries except the wait for the timeout. Then a new request number: the app, having heard nothing, had sent the whole order again. She scrolled. Three app requests, three tries each. Nine attempts for one tap, each one holding a checkout worker for a minute, and the customer, who had seen only a spinner and a grey box, pressing the button again.

"What's between the tries?" asked Ruth.

"Nothing. Try, wait sixty seconds, try."

"And the app?"

"The same. Three times, no wait."

"Then nobody is attacking you," said Ruth. "You're doing it to yourselves. Multiply it out."

Nadia did. A third of customers failing on Friday evening, each becoming nine, had been a nuisance. By Saturday morning, with four in five failing and every failure taking a full minute, the nine had become the load.

She wrote retry storm in the notebook with a question mark, and then crossed the question mark out.

The idea

A denial of service, DoS, is an attack that makes something unavailable not by breaking in but by using it up: the bandwidth of a line, the table of half-open connections, the memory of a server, the patience of a queue. A DDoS is the distributed kind, from many machines at once, so that no single address can be blocked and the total is larger than any one line could carry. In the postal picture it is a million junk letters to one address. The letterbox is full, the postman cannot get the real post in, and the person inside has done nothing wrong.

There are three places to hit: the pipe, with more bytes than the line can carry; the protocol, with the first packet of chapter 7's handshake sent over and over and never the third, until the server's table of half-open connections is full; and the application, with requests that are cheap to make and expensive to answer.

The pipe is the easiest to fill, because of two tricks. Reflection is sending a request to an innocent third party with the source address forged to be the victim's; the reply goes to the victim. UDP makes this possible, because a postcard needs no handshake and so carries no proof of who sent it. Amplification is choosing a third party whose answers are much bigger than its questions, so that a small stream of forged questions becomes a large stream of unwanted answers. A DNS query of sixty bytes can return an answer of three thousand; other services have done worse.

The many machines usually belong to someone else. A botnet is a collection of computers an attacker has taken over and can command together, and since Mirai the computers need not be computers; anything with a network port and a default password will do.

The defences are layered. Rate limiting is the application refusing to do more than so many requests per second for one client, one token or one address, and answering the rest with a quick "later" that costs almost nothing. Scrubbing is diverting all traffic for a site through a centre that drops the rubbish and forwards the rest, usually by announcing the site's prefix from the scrubbing centre for the length of the attack. And anycast absorption is the CDN trick from the last chapter: an address announced from two hundred edges spreads any flood across two hundred buildings, and no single line fills.

None of that would have helped Kettle on Saturday morning, because the flood was coming from inside. A retry is a client trying again after a failure. A retry storm is what happens when every layer does it, each on its own, and none of them waits. The app retries three times. Checkout retries three times. Three times three is nine attempts for one tap, and the service they are all aimed at is the one that is already failing, which now fails more slowly, which makes every attempt take longer, which fills more workers, which fails more customers, who press the button again.

The cure is as old as chapter 7, where Van Jacobson's fix for a congested network was for every sender to wait longer after each failure. Backoff means waiting, and waiting longer each time, before a retry; jitter means adding a random amount so that a thousand clients that failed together do not retry together. Beyond that: retry at one layer, not every layer; give the whole operation a budget of attempts; and stop sending for a while when almost everything is failing. Those are not network settings. They are lines in the app and in checkout.

Remember it as: A flood doesn't care who sent it: a million junk letters or your own retries, the letterbox fills the same way.

Saturday's spike was never an attack. It was Friday's failure, multiplied by nine, arriving at the one address that could not answer it.

…but

The nine explained the flood. It did not explain the one number nobody had touched since Friday: at 16:02 a third of checkouts had failed and two thirds had gone through, on pods that were all running the same code and all asking the same name. Nadia went to find out which pods the successful ones had used.

Chapter 22 · Act IVFaster Than a Handshake#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

By noon Nadia had a table she had not known she could make. The logs recorded which pod had served each checkout, and when each pod had started, and between 16:02 and 17:30 on Friday the pods fell into two groups so cleanly that she checked it twice.

pod            started       16:02 to 17:30     after 17:30
checkout-7f2   Thu 09:14     412 ok of 418      gone
checkout-9c1   Fri 16:11     3 ok of 301        gone
checkout-b40   Fri 17:31     -                  70 ok of 360

Pods running since Thursday kept paying people. Pods that started after four o'clock on Friday did not. At 17:30 somebody had restarted the lot, and every pod had become a pod that started after four o'clock. The seventy that still went through after that were paid with gift credit and had never called Paystream.

"What does the payment client do with its connections?" asked Ruth.

Nadia found the library's documentation. It kept a pool. On first use it looked the name up, opened a handful of connections to the answer, and reused them for every call afterwards, for as long as they stayed busy and the far end kept them open. There was a setting for how long to keep them, and Kettle had never changed it.

"So a pod from Thursday," she said slowly, "looked up api.paystream.example on Thursday. Got the public address. Opened its connections and kept them. When the answer changed on Friday, nobody told the connections, because a connection isn't to a name. It's to an address. The old pods never asked again."

"Until?"

"Until a connection in the pool died and it opened a new one. Then it asked, got 10.20.4.31, and that connection never came back. That's why it got worse through the evening." She looked at the third row. "And at 17:30 we closed every connection that was still working. To clear it."

"The restart didn't break anything," said Ruth. "It stopped the thing that was hiding the break."

Nadia drew a box round the table and wrote across the top: a third failed because a third of the connections were new.

The idea

Chapter 19 introduced keep-alive as an agreement to leave a TCP connection open after a request. HTTP/1.1 made that the default, and it is still the HTTP most servers speak to each other. It has one rule that costs: on a single connection, responses come back in the order the requests were sent. If the first response is slow, everything queued behind it waits, which is head-of-line blocking: the one at the front of the line holds up the line. Clients work around it by opening several connections and spreading requests across them, which spends handshakes and connection slots to buy parallelism.

HTTP/2 keeps the one connection and removes the rule. It splits every request and response into small frames, each labelled with a stream number, and sends them interleaved, so that many requests and responses are in flight on the same connection at once and each finishes when it is ready. That is multiplexing: many conversations sharing one connection without waiting for one another. The connection is opened once, with one TCP handshake and one TLS handshake, and then kept for as long as both ends like, which is often hours.

There was one line left to block on, and it was TCP's. TCP is registered post: every byte is numbered and delivered in order, so if one packet is lost, the receiver holds everything that arrived after it until the resend comes, even frames of streams that had nothing to do with the loss. QUIC is a transport built on top of UDP that numbers each stream separately, so a lost packet delays only the stream it belonged to. It carries TLS inside itself, so a new connection needs one round trip and a resumed one needs none, and it names connections by an identifier rather than by addresses and ports, so a phone that walks from Wi-Fi to mobile keeps its connection. HTTP/3 is HTTP on QUIC, and it is what your browser speaks to most large sites today.

Two names ride on these. gRPC is a way for services to call each other, built on HTTP/2 streams, and is what many clusters use between pods. A WebSocket is an HTTP connection upgraded into a plain two-way channel that stays open for as long as a chat window needs it. Both are long-lived by design.

All of this shares one consequence. A connection is made to an address, not to a name. The name is looked up once, when the connection is opened, and never again while it stays open; chapter 11's TTL says how long an answer may be remembered, but an open connection is not remembering the answer, it is living in it. Change the name's answer and nothing happens to any connection already open. The change arrives at the pace of connections closing, which, for a pooled HTTP/2 client, can be a day.

Remember it as: An open connection is a conversation already in progress: it never asks directory enquiries again.

That is Friday evening's shape. Pods with old pooled connections kept talking to Paystream's public address along the old path and paid people; every new connection asked the name, got 10.20.4.31, and waited for a reply that never came. Failures grew as pools turned over, and at 17:30 the restart closed every old connection at once and turned a third into four fifths.

…but

The old connections had gone out through the NAT gateway wearing the node's address. The new ones went to an address inside the VPC, so the NAT gateway never touched them, and the endpoint saw each pod as itself. Nadia realised she had never asked what a pod's own address was, or where it came from, and opened the VPC's page to read it from the top.

Chapter 23 · Act VFloors Full of Switches#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

Lunch was sandwiches eaten over keyboards. By 14:00 the HTTP/2 explanation had been written up in the channel, the 17:30 restart had a line in the timeline that no longer needed a question mark, and Sam had gone quiet, which Nadia had learned meant he was with the CEO.

She was still on the VPN and still in the platform account's console. On Friday night she had chased a single address through it. Now she wanted the whole shape. She found the VPC and listed its subnets.

kettle-nodes-a   10.0.1.0/24   zone a     kettle-endpoints-a   10.20.4.0/24   zone a
kettle-nodes-b   10.0.2.0/24   zone b     kettle-endpoints-b   10.20.5.0/24   zone b
kettle-nodes-c   10.0.3.0/24   zone c     kettle-endpoints-c   10.20.6.0/24   zone c

Three zones, and in each zone a subnet for the nodes, a smaller one for shared endpoints, and, further down the list, a public one for the load balancer. 10.20.4.31 sat in the first endpoint subnet, which agreed with Friday's route table. It was the first tidy thing she had seen all weekend.

Ruth, on the video call with her mug, looked at the shared screen. "One per building, one per job. We called those VLANs."

"Buildings?"

"Zones are buildings. Separate power, separate floors full of switches." Ruth paused. "What's above the subnets?"

Nadia clicked up a level to the VPC itself, and found the list of address blocks it owned.

kettle-prod   10.0.0.0/16   10.20.0.0/16   100.64.0.0/16

She had known about the first block since her first week. She had found the second on Friday night. Nobody had ever mentioned the third. She had filtered the subnet list by the name "kettle-", and when she took the filter off, three more subnets appeared inside 100.64, one per zone, created a year ago, with names that told her nothing.

"That's shared address space," said Ruth. "The range the ISPs were given for the insides of their big NATs. Odd thing to find in a VPC."

Nadia wrote 100.64.0.0/16: why? in the notebook and boxed it.

"Later," said Ruth. "Understand the subnets first. The cloud copied them from something, and I built the something."

The idea

A data centre is a building that exists to hold computers, and the network inside it is designed for the opposite of the internet: short distances, enormous volumes, and one owner.

The switches from chapter 6 are the bricks. A switch learns which MAC address lives on which port and forwards each frame to that port alone. When it does not know, it floods the frame to every port, and when the frame is a broadcast it floods on purpose. Two switches with two cables between them form a loop, and a flooded frame goes round a loop forever. Spanning tree is the protocol that stops this: the switches elect one of themselves as the root and switch off any link that would make a second path, leaving a tree. The spare links come back only when a live one fails. It keeps the network up and wastes half the cables.

A VLAN, a virtual LAN, lets one set of switches behave as several unconnected ones. Each port belongs to a numbered VLAN, and frames from one VLAN are never forwarded to a port in another. Between switches, a trunk carries every VLAN on one cable by adding the 802.1Q tag to each frame on the way in and stripping it on the way out. To get from one VLAN to another a frame must go through a router, which is where you put the rules about who may talk to whom. In the postal picture, VLANs are separate post rounds sharing one building: the same corridors, different sacks, and a sorting office between them.

For twenty years the standard shape was the three-tier network: an access switch at the top of each rack, a distribution switch for each row that joins the racks and holds the VLAN routing, and a core pair that joins the rows and faces the internet. Traffic that enters from outside and goes to a server is called north-south, because that is how it runs on the diagram. Traffic between two servers inside the building is east-west.

The tree suits north-south traffic and punishes east-west. A web server talking to a database two racks away sends every frame up through the distribution and core and back down, over links shared with everyone, half of which spanning tree has switched off.

Spine-leaf is Clos's exchange rebuilt in Ethernet. Every rack has a leaf switch. Every leaf connects to every spine switch, and nothing else connects to the spines. Any server reaches any other in exactly three hops, leaf to spine to leaf, and because the links between tiers are routed rather than bridged, all of them carry traffic at once and nothing is switched off. To add capacity you add a spine. VLANs stop at the leaf, which is why the data centre later needed a way to stretch a network across a routed fabric, and that is the next chapter.

Remember it as: a data centre is a tree that grew sideways: VLANs split one building into rounds, spine-leaf gives every rack an equal road.

Kettle's subnets are VLANs with the wiring removed. One per zone, one per job, and a router between them that Nadia cannot see. The question that mattered on Saturday was the same one Ruth's VLANs had always raised: who is allowed to cross from one round to another, and who decides.

…but

Ruth's VLANs had ended at a switch she could touch. Kettle's subnets ended at nothing Nadia could point to, and the router between them, the one that was supposed to hold the list, did not seem to exist anywhere either. If there was no box in the path, what had dropped the pod's packet?

Chapter 24 · Act VMachines Inside Machines#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

At 15:00 Nadia went looking for the box.

If the subnets were VLANs, something sat between them and held the list. On Ruth's floor it had been a router with a name and a rack number. Nadia opened a shell in Friday's debug pod and traced the route to the endpoint, expecting to find whatever was dropping her packets somewhere along the way.

$ traceroute -n -q 1 10.20.4.31
 1  10.0.3.44   0.21 ms
 2  *
 3  *

One hop, the node the pod was running on, and then nothing. No router. The trail ended as if the endpoint were on the same wire as the node and had chosen not to reply.

"Where's the router?" she said.

"Same place as the switches," said Ruth. "You don't own it, so you don't see it."

"Then what's enforcing the rule? Something is dropping a SYN."

Ruth told her where to look. Nadia found the endpoint's network interface, the one with the address 10.20.4.31, and opened its details. Attached to the interface itself, not to the subnet, not to any router, was a security group: a named list of allowed traffic, which she had seen on every node without ever asking what it was. Before she could open it Sam's face appeared on the call, his phone held against his chest.

"Paystream want to know whether we are going to keep hammering them."

"Tell them it stops when we stop retrying," said Ruth, without looking up. Then, to Nadia: "Don't read the rule yet. Read where it lives. It's on the interface. There is no box. The host that interface lives on checks every packet in and out of it, and you will never log in to that host."

Nadia wrote: firewall = per interface, done by the host. Then, underneath: which host?

"Whichever one the provider put it on," said Ruth. "I learned that in the 2010s, and I'm still not sure I like it."

The idea

Virtualisation is running several computers on one. A hypervisor is the program that does it: it owns the real processor, memory and network card, and shares them out among virtual machines, each of which sees a processor, some memory and a network card of its own. The card it sees is a virtual interface, and the hypervisor decides what happens to every frame that leaves it.

Inside the host sits a virtual switch. It behaves like the switches of chapter 6, learning which virtual interface holds which MAC address, except that it is software with a table, and the table can hold rules as well as addresses. The physical card is one of its ports; so is every virtual machine.

An overlay network is a network that exists only in those tables. The physical switches and routers form the underlay: they carry packets between hosts and know nothing else. When virtual machine A sends a frame to virtual machine B, the virtual switch on A's host looks up which host currently holds B, wraps the whole frame in a new packet addressed to that host, and sends it. B's host unwraps it and hands the original frame to B. VXLAN is the common way of wrapping: the frame goes inside a UDP packet to port 4789, with a small header carrying a twenty-four-bit VNI, the number of the virtual network it belongs to. The wrapping point on each host is a VTEP.

This is where the postal picture stops helping. The best it can offer is letters inside parcels carried by another courier, which is true as far as it goes, but it leads you to look for the courier's van, and there is none. Here is the same thing without the metaphor. A is given an address in a network that has no cables. That network is a table, copied to every host, saying which host holds which address. The cables carry only host-to-host packets. If B moves to a third host, the table changes and A's next frame goes to the new host, while A's address, B's address and the network they share have not changed at all, because none of them were ever on a wire.

Who writes the table? Software-defined networking, SDN, is the name for separating two jobs: the data plane, the hosts and switches that forward packets by consulting a table, and the control plane, a controller that decides what the tables should say and pushes it down. OpenFlow was one protocol for the pushing. Every large cloud runs a controller of its own design that knows where every virtual interface is and what it is allowed to do.

That last item is the one that matters on Saturday. In a building, the rules live in a box in the path. In an overlay there is no need for the box, because every packet already passes through a virtual switch on its way out of one interface and another on its way into the next. So the rules are attached to the interface, and the host that owns the interface applies them, on the way out and again on the way in. The firewall is everywhere and nowhere. It keeps state, like the stateful firewalls of chapter 15, so a reply is allowed if the request was. When a packet matches no rule the host drops it before the destination sees it, and nothing in the path sends a refusal.

Remember it as: the cloud's network is a table on every host, and the firewall is a rule on every interface, enforced by the host that owns it.

This is why Nadia's traceroute found one hop and then nothing, and why on Friday the SYN went out and nothing came back: the pod's packet reached the host holding 10.20.4.31, and that host compared it with the interface's list and threw it away.

…but

If every interface carried its own guard, the question was no longer "where is the firewall" but "how many interfaces, and which rules on each". Nadia could not answer, because in eight months she had never once drawn Kettle's VPC. She turned to a clean page.

Chapter 25 · Act VIYour Own Private Slice#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

If the guard stood at every network interface, Ruth had said, then the question was not "where is the firewall" but "which interface, in which subnet, with which rules". At 15:40 on Saturday Nadia realised she could not answer the middle part. She had been typing 10.0.0.0/16 into forms for eight months and had never once drawn the thing it described.

She was still on the company VPN, looking at the platform account's console, which Jonas had built and nobody else had ever needed to open. She turned to a clean page in the notebook and wrote Kettle VPC at the top.

Three address blocks were attached to it. 10.0.0.0/16, the one everybody used. 10.20.0.0/16, which the console called shared-endpoints. And 100.64.0.0/16, which she had found an hour earlier and still could not place.

Under them, subnets, in threes. Three zones, named a, b and c. Each zone had a public subnet with a leg of the load balancer in it, a private subnet with the cluster's nodes in it, an endpoint subnet carved from 10.20.0.0/16, and a subnet from 100.64.0.0/16 called only cluster-secondary, which told her nothing. Twelve subnets. She drew twelve boxes and ran out of page by the third column.

Then the route tables. The one attached to the private subnets read:

10.0.0.0/16      local
10.20.0.0/16     local
100.64.0.0/16    local
0.0.0.0/0        nat-gateway-a

The table on the public subnets was the same except for the last line, which pointed at something called the internet gateway. The table on the endpoint subnets had no last line at all. She drew arrows. At 16:00 she held the page up to the camera.

"That," said Ruth, "is the first time anyone at Kettle has drawn it. Keep it. You'll be looking at it all evening."

The idea

A VPC (AWS: VPC; Azure: virtual network, VNet) is a private network that you define inside a cloud provider's building: a gated estate you rent inside a city you don't own. Nothing is inside it until you put something there, and nothing can reach it until you build a way in.

The first decision is the address block, written as a CIDR prefix from chapter 9. 10.0.0.0/16 gives 65,536 addresses. Choose it from the private ranges of chapter 14, choose it large, and choose a block that nothing else you own uses, because one day you will want to connect this network to another, and two networks with the same addresses cannot be told apart. A VPC can carry more than one block. Kettle's 10.0.0.0/16 was chosen in a hurry; the second and third blocks were added later, when the first ran short, and the route tables learned about them with no help from anyone.

Inside the block you cut subnets, smaller prefixes such as 10.0.1.0/24. Each subnet lives in exactly one availability zone (AWS: Availability Zone; Azure: availability zone): a separate building with its own power and cooling, close enough to the others for fast links and far enough that a fire in one does not take out the rest. That is why Kettle's subnets come in threes. One of each kind in each zone, so that losing a zone loses a third of the capacity and none of the services.

A subnet is not public or private by nature. It is public or private by its route table (AWS: route table; Azure: route table, or user-defined routes): the sorting office's pigeonholes from chapter 12, now a short list you write yourself and attach to each subnet. Every table begins with a local route, meaning that every block attached to the VPC is reachable directly, without leaving. That one line is why 10.20.4.31 was reachable from the nodes the moment it existed.

A public subnet is one whose table sends 0.0.0.0/0, everything else, to an internet gateway (AWS: internet gateway; Azure: no separate object, outbound access is built in or set by a NAT gateway): the door between the estate and the city. Machines in a public subnet can be given a public address and be reached from the internet. Kettle's load balancer lives there. A private subnet has no such route. Its machines have private addresses only, and nothing on the internet can start a conversation with them. Kettle's nodes live there. When a private machine needs to reach out, for a software update or a payment, its table sends 0.0.0.0/0 to a NAT gateway (AWS: NAT gateway; Azure: NAT Gateway): the company post room from chapter 14, sitting in a public subnet, swapping the private address for one public address and remembering who asked. Replies come back through it. Fresh conversations from outside do not.

The three kinds of subnet differ in one line of one table each.

A packet leaving a node to a public address follows the private table to the end: not local, so out through the NAT gateway, then the internet gateway, then the city.

Remember it as: A VPC is an estate you rent inside someone's city: you choose the postcodes, write the pigeonholes, and build every gate yourself.

That sequence is the path every call to Paystream took until Friday. Since 15:58 the destination has been 10.20.4.31, which matches the second line of every table, local, so the packet never reaches the NAT gateway or the internet gateway at all. It goes straight to the endpoint interface, and straight into whatever guard stands at it.

…but

The endpoint interface sat in a subnet whose only route was local. Nothing in the drawing could lose a packet on the way there. Only the guard at the interface could, and a guard who keeps no record is half a guard. Ruth asked whether the platform account had flow logs turned on.

Chapter 26 · Act VITwo Kinds of Guard#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

Flow logs were not on. Jonas had never needed them. Nadia ate a sandwich at the kitchen table, copied the drawing out again more neatly, and at 17:35 turned on flow logs for the endpoint's three network interfaces. The console told her, in the unhurried way of consoles, that records would start to appear in about ten minutes.

While she waited she opened the security group on the endpoint. Jonas had written it on Friday. It had one inbound rule.

Inbound   TCP 443   from 10.0.0.0/16    "kettle nodes"
Outbound  all       to 0.0.0.0/0        (default)

Nodes lived in 10.0.0.0/16. Jonas had tested from a node. The rule was correct for the test. She ran the same connection check from a node herself, so that at least one line in the log would be hers.

The first records arrived at 17:58. Her test was there, 10.0.2.14 to 10.20.4.31, port 443, ACCEPT. Everything else was not hers.

100.64.3.17    10.20.4.31  443  TCP  REJECT
100.64.5.102   10.20.4.31  443  TCP  REJECT
100.64.3.17    10.20.4.31  443  TCP  REJECT

Pages of it. Every rejected line came from an address beginning 100.64. The rule allowed 10.0.0.0/16 and mentioned nothing else. She laid the rule next to the log and, at 18:00, wrote under both of them: What is 100.64? Who lives there?

"Now you've seen the drop," said Ruth. "Chapter seven's missing reply, written down by the thing that dropped it. Before you fix the rule, you need to know which kind of guard wrote it."

The idea

A security group (AWS: security group; Azure: network security group) is a guard for each flat. It is attached to a network interface, not to a subnet, and it travels with the interface. Its rules say only what is allowed: port, protocol, and where from. There is no deny line. Anything no rule allows is dropped, silently, with no refusal sent back, which is why checkout saw a timeout and not an error. A security group is stateful: it remembers each conversation it lets in, and lets the replies out without a rule for them, and the same the other way round.

A network ACL (AWS: network ACL; Azure: NSG rules on a subnet; there is no separate stateless list) is the checkpoint at the district boundary. It is attached to a subnet and checks every packet crossing in or out. Its rules are numbered and read in order, and they can say deny as well as allow. It is stateless: it keeps no notebook, so each packet is judged alone, and a reply needs its own rule covering the ephemeral ports it will be addressed to. By default it allows everything, which is why most people never notice it.

Both apply to the same packet. A packet from one subnet to an interface in another crosses the first subnet's checkpoint on the way out, the second subnet's checkpoint on the way in, and then meets the security group at the door.

Flow logs (AWS: VPC Flow Logs; Azure: VNet flow logs) are the record that neither guard keeps on its own. Turned on for an interface, a subnet or a whole VPC, they write one line per conversation: source, destination, port, protocol, how many packets, and whether the host let it through or not. They hold no payload, they arrive a few minutes late, and they do not say which of the two guards made the decision. They are still the only way to see a silent drop without a capture at both ends.

Two habits decide whether these guards help or hurt. The first is the source. A rule can allow a range, from 10.0.0.0/16, or it can allow a group: anything whose own security group is the one named. A rule that says allow 443 from the cluster's group follows the machines wherever their addresses go. A rule that says from 10.0.0.0/16 follows the addresses, and trusts anything that has one. The second habit is 0.0.0.0/0. It means from anywhere, it is what the tutorial said to type, and it turns the guard into a doorstop.

Remember it as: Security group: a guard per flat, remembers who it let in. Network ACL: a checkpoint at the district boundary, no memory, both ways.

The endpoint's guard had one line, and the packets from 100.64 matched none of it. Allow-only means no match is a silent drop, and a silent drop at the far end is a SYN that gets nothing back: the capture from chapter 7, finally explained from the other side.

…but

"I could widen it," Nadia said. "Allow 100.64.0.0/16 as well." "You could," said Ruth. "Do you know what you'd be letting in?" She did not. And there was a harder question under that one, which Sam rejoined the call at 18:40 to ask on the CEO's behalf: why had Jonas needed a private letterbox at all?

Chapter 27 · Act VIConnecting the Islands#

1. the wire
2. phone
3. Wi-Fi
4. ISP
5. internet
6. DNS
7. CDN edge
8. load balancer
9. VPC
10. node
11. pod
12. payment API

The problem

Sam's question, which was the CEO's question, was whether they could roll Jonas's change back. Nadia had thought about it. Deleting the endpoint would put api.paystream.example back to its public addresses, and the old path through the NAT gateway was still there in every route table. It would probably work. Probably was doing the same work the question mark had done on Friday.

"Before you do that," said Ruth, "tell me why he built it."

Nadia had the ticket from a month ago. During the September sale the NAT gateway in zone a had run out of ports, and the graph Jonas had attached to the ticket still looked like a cliff.

nat-gateway-a   ErrorPortAllocation     peak 1,412 per minute   14 Sept 19:02
nat-gateway-a   BytesOutToDestination   2.1 TB                  September

Two terabytes of payment traffic out through the post room, every byte of it charged, and on the busiest evening of the year the post room had run out of envelopes.

Ruth held up her tablet to the camera and drew an island. "Kettle's VPC. Draw me every way off it."

Nadia drew the internet gateway and the NAT gateway, because those were on her page. Ruth drew the rest: a link to another estate, a hub with many spokes, a sealed pouch through the public post, a private road, and a letterbox. By 19:00 there were six ways off the island, and Nadia had written Jonas's name under the letterbox.

"He chose well," said Ruth. "That's what makes this annoying."

The idea

A VPC is an island. Sooner or later something on it needs something off it, and there are only a few kinds of crossing.

Peering (AWS: VPC peering; Azure: VNet peering) joins two VPCs directly. Each side adds a route for the other's block, and traffic passes between them on the provider's own network, never on the internet. Two rules follow from how it is built. The blocks must not overlap, because a router cannot tell two 10.0.1.14s apart. And it is not transitive: a link from A to B and a link from B to C does not give A a path to C.

A transit hub (AWS: Transit Gateway; Azure: Virtual WAN hub) is a router you attach islands to. Each VPC gets one link to the hub, the hub holds one route table, and anything attached can reach anything else the table allows. Ten VPCs peered in pairs need forty-five links; ten VPCs on a hub need ten.

Hybrid connectivity is the crossing to premises you own. The cheap way is a VPN (AWS: Site-to-Site VPN; Azure: VPN Gateway), the sealed pouch of chapter 18, carried over the public internet between your office router and a gateway in the cloud. The expensive way is a dedicated line (AWS: Direct Connect; Azure: ExpressRoute): a physical circuit from your building to the provider's edge, with predictable latency and a predictable bill, ordered in months. Both usually land on the transit hub.