Before you startLeading a Java Team: The Yard#

A very short book for a lead who knows how software is built and has just inherited a team that builds it in Java. Fifteen arguments the team will have in the first ninety days, the mechanism and the history behind each one, and the decision a lead makes and defends.

The promise

Read this in two evenings and you will walk into your first design review already knowing the shape of the arguments: why the build takes nine minutes and whether Gradle would fix it, why the veteran wants Lombok and the junior wants records, why every test is green and nothing is tested, why the database in the test is not the database in production, why a library upgrade broke an unrelated module, why the pod died at two in the morning with no stack trace, and why a few payouts go missing on every deploy. You will know what each thing is called, what it actually does, who argued about it first and in what year, and what the lead in this book decided and why. You will also know where the lead was wrong, because a lead who is never wrong is a lead nobody tells anything.

It reads as one story. Sorrel is a mid-sized insurer with about forty Java services on a shared Kubernetes cluster. Sam arrives from twelve years of .NET to lead the Payouts team, six people who pay approved claims by reading the policy system, writing to the ledger and instructing a bank gateway. Every chapter opens on two people who want different things. The chapter's idea is the mechanism underneath the argument. The chapter ends with Sam's decision, in Sam's words to the team, and the crack that becomes the next chapter.

Who this is for

You lead, or are about to lead, a Java team, and your own hands learned the trade somewhere else: .NET most likely, but Go, Python or TypeScript will do. You can read Java. What you cannot yet do is tell which of the team's arguments matter, who is right, and what it costs to decide either way. That is what this book is for.

It is not a Java tutorial. Its sibling, Java for C# Developers, holds the mechanics at 170,000 words, and every chapter here ends by naming the handbook chapters that carry the detail. Read the story here, look up the syntax there.

If you have led Java teams for years, this book will be slow for you. Read the At the whiteboard boxes and the epilogue's decision log, and argue with them.

The stack, and why it is this one

The book favours what Sorrel runs, because the author's team runs it: Maven, through the wrapper, never the system install; Spring Boot 3.5 moving to 4; Amazon DocumentDB through the MongoDB API, with Spring Data MongoDB; JUnit 5 and Mockito; services on Kubernetes that call other systems inside the company. Gradle, Kotlin, WebFlux and JPA each get a fair hearing in the chapter where someone proposes them, and each is declined for Sorrel with the reason stated and the condition under which Sorrel would revisit. If your team runs Gradle, the arguments still hold; the kit commands will not.

How to read it

• Straight through the first time. The plot carries the ideas, and each chapter's crack is the reason the next idea exists.
• The boxes are where it becomes a tool. The kit is what to type. The traps is a checklist. At the whiteboard is what to ask in the meeting. How we got here is the story to tell when someone asks why it is like this.
• The epilogue is the reference: fifteen architecture decision records in Sam's hand, the map from this book to the handbook, every command collected by task, a timeline, and a glossary.

The cast

Sam is the new lead, from .NET, and the reader's eyes. Rafael is the veteran who wrote the parent POM eight years ago and has been right about most things since. Aisha is two years in and asks the questions a lead cannot. Oren runs the platform team: the cluster, the registry, the artifact repository, and a number for everything. Bea owns the ledger on another team and answers the phone when Payouts breaks it.

There is no villain. The arguments are the ordinary ones, and both sides are usually right about something.

Dates, sources and versions

Every "who and when" in this book was checked against the primary source named in research/milestones.md. Every claim about a version, a property or a DocumentDB limitation is recorded in research/claims.md with the page it came from and the date it was fetched, because those move. Where this book and the handbook disagree, the handbook's later verification date wins and this book is wrong.

Following along

Every command in this book runs against the companion project in examples/payouts on an ordinary laptop with a JDK 25 or later and Docker. Maven arrives through the wrapper. Chapters that touch the cluster use a local kind cluster that one script creates, and every command against it is read-only or confined to the payouts namespace. Nothing is pointed at a machine you do not own.

Tested on: macOS 15 with JDK 26.0.1, Docker 27.5.1, kind 0.33.0, kubectl 1.31. The companion project targets Java 25 and the Spring Boot version named in its parent POM.

Contents

Act I: The first week — what is actually here

1. Three Services, Three Javas (the parent POM, the release flag, the Boot BOM)
2. The Build That Downloads the Internet (the local repository, settings.xml, the stores, reproducible builds)
3. Which Boot Are We On (the upgrade to Boot 4 as a project, the release matrix)

Act II: The first month — the build and the tests

4. Nine Minutes (Gradle proposed, measured, declined for now)
5. The Lombok Ban (annotation processors, records, reversing a decision in public)
6. All Green, Nothing Tested (JUnit, Mockito, what a unit is, slices)
7. The Database That Isn't (Testcontainers, the DocumentDB gap, a feature contract)
8. One Document, Two Writers (the document model, optimistic locking, idempotency)
9. The Jar in the Audit (the dependency tree, nearest-wins, Equifax)
10. The Upgrade That Broke Something Else (no lockfile, SNAPSHOT drift, the enforcer)

Act III: The first quarter — running it

11. The Ledger Is Down, So Are We (timeouts, retries, the breaker, calling other teams)
12. Let's Go Reactive (virtual threads, pinning, when WebFlux wins)
13. OOMKilled (the JVM in a container, the first incident)
14. The Deploy That Dropped Requests (readiness, graceful shutdown, rolling updates)
15. The Magic Box (how Spring actually works)

Epilogue: The Decision Log

Chapter 1 · Act IThree Services, Three Javas#

The problem

On Sam's second morning, Oren sent a spreadsheet. One row per container image on the cluster, one column for the base image. The Payouts team's rows read eclipse-temurin:21, eclipse-temurin:21 and eclipse-temurin:11.

"We're on 21," Rafael said, before Sam had finished the question. "It's in the parent POM. Has been since the Christmas before last."

"And the third one?"

"Reports. It builds the monthly file for finance. Nobody has touched it in three years and nobody wants to. It works."

Sam had led this conversation before, in a different language. In .NET it ended with a Directory.Build.props and a target framework that every project inherited, and anyone who overrode it had to say why in a pull request. Here the parent POM said 21, Rafael said 21, and one of the three services was running a runtime two major versions behind, with security updates that had stopped arriving.

Aisha found it in ten minutes. The reports module had its own maven.compiler.release property set to 11, three lines below the line that inherited the parent, and its own dependencyManagement block pinning an older Spring Boot. Both had been added in a hurry during an incident, with no comment, by someone who had since left.

"So which Java are we on?" Sam asked.

"All of them," Aisha said.

The idea

A Maven build has two trees that look like one. The first is aggregation: a root pom.xml lists <modules>, and ./mvnw package at the root builds them in dependency order. The second is inheritance: each module names a <parent>, and takes from it every property, plugin configuration and managed dependency version the parent declares. The two trees usually coincide. They do not have to, and when they diverge a module can sit in the build and still ignore the site office.

The site office is the parent POM. It should say three things once. The Java release, through maven.compiler.release, which makes the compiler reject any API newer than the release named, rather than merely emitting old bytecode the way source and target did. The versions of every library the company has decided on, in dependencyManagement, which sets a version without adding the dependency, so a module declares spring-data-mongodb with no version and gets the managed one. And the plugins, with their versions, so that every bay in the yard runs the same tools.

Spring Boot's own versions arrive in one of two ways, and the choice is the first thing a lead from .NET gets wrong. Inheriting spring-boot-starter-parent gives you Boot's managed versions, its plugin configuration and sensible defaults for free, at the price of your only <parent> slot. Importing the spring-boot-dependencies BOM into your own dependencyManagement gives you the versions only, and leaves the parent slot for the company's parent, which is what Sorrel has. Rafael chose the import eight years ago, correctly, and nobody had written down why.

Inheritance is a promise a child can break. Any property, any managed version, any plugin setting can be redeclared in a module, and the nearer declaration wins. That is what Reports had done. The only document that tells the truth about a module is its effective POM: the parent, the imports and the overrides merged into one file, which Maven will print on request and which nobody reads until a morning like this one.

Two flags matter for the rest of the book. -pl payouts-api builds one module. -am adds the modules it depends on, which is the flag people forget, and then wonder why the build used last week's payouts-domain from the local repository.

Remember it as: the parent POM is a promise every module can break; the effective POM is what is actually true.

Sam's decision, to the team that afternoon: one parent, one release, one Boot. Any override of a parent property needs a comment saying why and a date, and the pull request has to show the effective POM diff. Reports moves to 21 this week, since it compiles on 21 already and Oren has the image. It moves to 25 with everyone else, and not before, because Sam had been there four days and the first thing you change should not be the thing nobody understands. Rafael asked whether the comment rule was a .NET habit. Sam said yes, and that it was a good one.

What Sam learned

• Ask for the effective POM before believing anyone, including the parent.
• A rule in the parent is only a rule if overriding it costs a comment and a review.
• Move the thing nobody understands last, and move it to where everyone else is first.

…but

Reports moved to 21 on Thursday. On Friday a new laptop arrived for Aisha, and the first clean build took twenty-two minutes, most of it a progress bar downloading jars from somewhere Sam could not name, and Oren said the CI runners did the same thing every night. Sam asked where it all came from, and nobody had the same answer.

Chapter 2 · Act IThe Build That Downloads the Internet#

The problem

Aisha's new laptop finished its first build at twenty-two minutes. Oren's CI runners, which started from nothing every night by design, took nine, and four of those were a progress bar.

"That's Maven," Rafael said. "First build fills the cache. After that it's fast. Everyone's laptop has a gigabyte of jars in it and nobody notices."

Oren noticed. The platform team ran an artifact repository, a Nexus, and had for two years. Roughly half of Sorrel's builds went through it. The other half, Payouts included, went straight to Maven Central over the office connection, which was why the nightly builds of twenty teams all slowed down at the same time. He wanted every download routed through the Nexus, and he wanted it in writing.

Rafael did not. "Central has been up for twenty years. The Nexus went down twice last quarter. If it's the only road, we can't build when it's down."

Sam had seen this one from the other side. In .NET it was a nuget.config with a private feed and a lock file, and a build that failed loudly when a package was not where it said it would be. The thing that worried Sam was not the nine minutes. It was that nobody in the room could say, for any one jar in the build, where it had come from, whether it was the same bytes as last month, or what would happen if it stopped being there.

"Where does spring-core come from?" Sam asked.

Three people gave three answers, and all of them were partly right.

The idea

Every dependency in a Maven build resolves along one path. The build reads the POM, asks the local repository at ~/.m2/repository whether it already has that artifact at that version, and only on a miss asks a remote. The remote is Maven Central unless settings.xml says otherwise, and the usual way it says otherwise is a mirror: one entry that says every request for any repository goes to this address instead. A company artifact repository, Nexus or Artifactory, sits at that address, fetches from Central on the company's behalf, keeps a copy, and from then on answers from its own disk.

That is the whole mechanism, and three things follow from it. The local repository is a cache that is never cleared, which is why Rafael's laptop was fast and Aisha's was not. The mirror is the only point at which a company can see, count or block what it depends on, which is why Oren wanted it. And a build that cannot reach the mirror cannot build, which is why Rafael did not.

Two smaller pieces close the gaps. The Maven wrapper, mvnw, is a script committed with the code that downloads the Maven version named in .mvn/wrapper/maven-wrapper.properties and runs it, so that every laptop and every runner builds with the same Maven, and nobody installs one. And project.build.outputTimestamp in the parent makes a build reproducible: with it set, the same sources produce byte-identical jars on any machine, which turns "is this the same build?" from a question into a checksum. Maven 3.10, released in September 2026, turns that mode on by default, which is one more reason the wrapper should name a current Maven.

The remaining question is Rafael's, and it is a fair one. A mirror that is the only road is a single point of failure. The answer is not to avoid the mirror but to make the road's condition visible: ./mvnw --offline builds from the local cache alone and fails fast if anything is missing, which tells you in seconds whether you depend on the stores today, and dependency:go-offline fills the cache deliberately before a flight or an outage. The platform team, for its part, owes the mirror an uptime it can state.

Remember it as: every jar came from somewhere; settings.xml says where, the local repository says it was once, and the mirror is the only place anyone can count.

What Sam told the team, and Oren: everything goes through the Nexus, in every settings.xml and on every runner, and the entry carries a comment with Oren's name and the date. The wrapper is committed in every repository, the parent sets the output timestamp, and CI caches the local repository keyed on the POM hashes so the nine minutes become two. In return the platform team publishes an uptime figure for the Nexus and a documented fallback, because a road with no stated condition is not one Sam will make the only road. Oren asked what the fallback should be. Sam said Central, probably, and that it was fine as long as it was written down and nobody was surprised.

What Sam learned

• A build's dependencies have an address, and the address should be one that someone at the company owns.
• The local repository is why builds are fast and why nobody can say what is in them; count through the mirror instead.
• Never make something the only road without a stated condition and a stated fallback.

…but

With the cache warm, the nightly build settled at nine minutes, and Rafael said nine minutes was the Maven tax and the only way to stop paying it was to stop using Maven. He had a Gradle branch, a benchmark, and a slot in Thursday's design review.

Chapter 3 · Act IWhich Boot Are We On#

The problem

Spring Boot 4 had been out since November, and Rafael wanted it. "One line in the parent. Everything else is managed. We've done it for every minor since 2.1. And 3.5 went out of open-source support in June, so we're already running on borrowed time."

Oren wanted it too, for a different reason: the platform team's dashboards used a metrics format that Boot 4 emitted by default and Boot 3.5 needed configuring for. Two teams had moved already. Payouts was on the list of ones that had not.

Sam had done a framework major before, from .NET Framework to .NET Core, and the memory was of a line in a project file that took an afternoon and a migration that took a quarter. The difference between those two things was the question nobody in the room had asked.

"What does 4 remove?" Sam asked.

Rafael listed what he knew: the web starter had a new name, some properties had moved, a couple of deprecated things from 3.x were gone. "The migration guide has a list. It's mostly renames."

Aisha, who had opened the guide while he spoke, read out the length of the list. Then she read out the Spring Framework version underneath Boot 4, and the Jakarta version underneath that, and the three libraries in the parent POM whose Boot 4 compatibility nobody had checked, one of which was the resilience library every service used to talk to the ledger.

"So it's one line," Sam said, "and we find out what the line costs after we've typed it."

The idea

A Boot major is not a version. It is a change of baseline: the Spring Framework generation underneath, the Jakarta EE level that defines the servlet and persistence APIs, the minimum Java, and the set of third-party libraries whose versions Boot manages. Boot 4 sits on Spring Framework 7 and raises the Jakarta baseline, which means every library that touches a servlet, a bean validator or a persistence provider has to have been rebuilt for that baseline too. The one-line change in the parent moves all of that at once, and anything in the build that Boot does not manage stays where it was and may no longer fit.

The place to start is the release matrix: which Java, which Boot, which Framework, and which versions of the libraries your parent POM manages outside Boot's BOM. At Sorrel that list is short, and three entries on it mattered: the resilience library, the DocumentDB driver pin, and a test library that had been added for one module. Each has its own compatibility statement, and each is a thing a lead can check in an hour before anyone types the line.

Then the renames. The web starter became spring-boot-starter-webmvc, which is the change every tutorial written before the release will get wrong, and the migration guide lists the properties that moved. Renames are the cheap part; a build that fails to compile is a build you can fix. The expensive part is behaviour that changes without failing: a default that flipped, a property that is now ignored, a health group that reports differently. Those are found by tests, which is why chapter 6 comes before anyone upgrades anything, and by a canary, which is why chapter 14 does too.

The upgrade is therefore a project, with the shape every migration has: inventory what is pinned outside the BOM, read the guide for the removals, move one service that has good tests, run it beside the old one, then move the rest. The one-line version of the story is true only of the last step.

Remember it as: a Boot major moves the baseline under every library you did not pin; the line in the parent is the last step, not the first.

Sam's decision, in the Monday stand-up: Boot 4 is on the roadmap for the quarter, and the first thing on it is a list, not a version. Aisha owns the matrix: every library the parent pins outside the BOM, its Boot 4 compatible version, and a yes or no. Rafael picks the service with the best tests to go first, which will tell everyone something about the tests. Oren gets his metrics format from the one service that moves first, which is enough to prove his dashboards. Nobody changes the parent until the matrix is complete. Rafael pointed out that 3.5's support had already ended and asked how long the list would take. Sam said a week, and that a week of list was cheaper than a quarter of surprises, and that the minors had never needed this because the word major meant something.

What Sam learned

• A major is a baseline change, and the inventory of what sits outside the baseline is the first deliverable.
• Move the service with the best tests first, because the tests are the instrument.
• Renames are the cheap part; the expensive part does not fail the build.

…but

Aisha's matrix took three days, and two of them were spent waiting for the build. Every change to the parent meant a full build to see what broke, and a full build was nine minutes on CI and longer on a laptop. By Wednesday Rafael had a proposal on the board, with a benchmark attached, and the proposal was Gradle.

Chapter 4 · Act IINine Minutes#

The problem

Rafael's slide had two numbers on it. The Maven build of the Payouts repository, on a CI runner with a warm cache: nine minutes and ten seconds. The same code on a Gradle branch he had built over the weekend: three minutes and forty seconds. "Same tests, same modules, same machine. Gradle does incremental compilation and caches what it's already done. Maven rebuilds the world every time. We're paying five and a half minutes a build, forty builds a day, for XML."

The room was with him. Aisha had just spent two days waiting on that build. Oren liked any number that went down.

Sam liked the number too, and distrusted the slide, because it answered a question nobody had asked. Nobody had asked what the nine minutes were. In .NET Sam had once replaced a build system to save four minutes and found afterwards that three of them had been a test that slept. "Before we change the tool," Sam said, "can we see where the time goes?"

"It's Maven," Rafael said. "That's where the time goes."

Oren had the runner logs. He put the timings up beside Rafael's slide. Compilation of both modules: one minute fifty. Packaging: twenty seconds. Tests: six minutes fifty, run one class at a time, in a single process, with a database container started for each class that needed one.

"So the build is two minutes," Sam said. "And the tests are seven."

The idea

A build is a pipeline of stages, and the stages have different shapes. Compilation scales with source size and is bounded by the compiler. Tests scale with the number of tests, the time each one takes, and how many run at once. A build tool can change the first by doing less of it, and the second only by running it differently. Rafael's benchmark had compared the two tools on both at once, and the five and a half minutes were almost entirely the second.

Gradle's genuine advantages are real and worth stating. It does incremental compilation: it tracks which classes changed and recompiles only them and their dependants, where Maven's compiler plugin, by default, recompiles a module when any source in it changes. It has a build cache that stores the outputs of tasks keyed on their inputs, so a test task whose inputs are unchanged is not run at all, locally or on a shared cache across the team. On a large multi-module build with many unchanged modules those two things are decisive. On a two-module build they are worth most of the one minute fifty, and nothing of the six minutes fifty.

Maven, for its part, has two flags that address the seven minutes, and both were off. -T 1C builds independent modules in parallel, one thread per core, which helps when modules do not depend on each other. Surefire's forkCount runs test classes in several JVMs at once, and reuseForks keeps those JVMs alive between classes, so the container started for one class serves the next. Together they turn a serial test run into a parallel one. With them on, Oren's runner did the same build in four minutes five: the compile unchanged, the tests down to two minutes ten.

That leaves a fair comparison: Gradle at three forty against Maven at four oh five, with the remaining difference being incremental compilation that a two-module build barely uses. Against it sit the costs of moving: a build file that is a program rather than a document, which the chapter on Lombok will show matters for who can read it; every other Sorrel team on Maven, with the parent POM and the Nexus configuration written for it; and a rewrite of the thing the team understands least, three weeks into a new lead's tenure.

The honest condition under which Sorrel would revisit is also clear, and worth writing down so that it is a decision and not a mood: when the build is compile-bound, meaning compilation is most of the wall time, across many modules that mostly do not change. That is the shape Gradle was built for. It is not the shape of Payouts.

Remember it as: measure the stages before you change the tool; a slow build is usually slow tests, and the tool cannot fix a test.

Sam wrote the decision on the whiteboard under Rafael's slide: not now, with the numbers. Parallel modules and forked, reused test JVMs go into the parent this week, which gets everyone to four minutes. The two minutes of tests that remain are chapter 6's problem, not a build tool's. Gradle is revisited when compilation is more than half of a build that is more than ten minutes, and that sentence goes in the decision log so nobody has to have this meeting from memory. Rafael said he would hold Sam to the sentence. Sam said that was the point of writing it down.

What Sam learned

• Time the stages before comparing the tools; the comparison is otherwise about the wrong stage.
• Maven's defaults run tests one at a time in one process, and the fix is two settings in the parent.
• Write the condition for revisiting a decision, so that revisiting it is a measurement and not an argument.

…but

Four minutes bought goodwill, and Sam spent it on the ninth morning, in a pull request review, on a class with eleven annotations above it and no code inside. Sam asked what generated the code. Rafael said Lombok did, and that every class in the repository used it, and Sam said that would stop.

Chapter 5 · Act IIThe Lombok Ban#

The problem

The class was called Payout. It had eleven annotations above its name, @Data, @Builder, @NoArgsConstructor, @AllArgsConstructor and seven more, and below the name it had nine fields and nothing else. No constructor, no accessors, no equals. The pull request changed one field's type and the diff was two lines, and Sam, reading it on the ninth morning, could not tell from the file what the class could do.

"Lombok," Rafael said. "It generates all of that at compile time. Every class in the repository uses it. It's been there since before me."

Sam knew the argument from the .NET side, where the same impulse produced source generators and the same objection: code you cannot read in the file is code you cannot review in the file. Twelve years of analyser rules and a team that had learned to live with them had left Sam with a reflex, and the reflex fired. By lunchtime there was a message in the team channel: Lombok is removed from the parent POM, new code does not use it, existing classes are migrated as they are touched, and Aisha would open the first pull request that afternoon.

Aisha's pull request, that afternoon, was four thousand lines. Three hundred and six classes used Lombok. Removing it from the parent broke every one of them at once, so the migration was all or nothing, and the generated accessors that nobody had read turned out to be called from every test, every mapper and two other teams' client libraries. Rafael said nothing in the channel. In the review he wrote one comment: "We could have just stopped adding it."

The idea

Lombok is an annotation processor, a program the compiler runs during compilation, and it is an unusual one. Ordinary processors read annotations and write new source files beside yours. Lombok reaches into the compiler's own syntax tree and adds methods to the class being compiled, which is why Payout has accessors with no accessors in the file and why the IDE needs a plugin to know they exist. It works, it has worked since 2009, and it depends on compiler internals that the JDK team never promised to keep.

The thing Sam had reacted to was real: a class whose behaviour is generated is a class you review by trusting the generator. What Sam had got wrong was the shape of the remedy. There are three ways out of a dependency on generated code, and they have very different costs. Removing it is a migration of every class at once, because the processor is all-or-nothing at the build level. Stopping adding it costs nothing today and shrinks the problem with every class that is rewritten for another reason. Replacing it where it fits means records, which since Java 16 give a value class its constructor, accessors, equals and toString in one line of language rather than one line of annotation, with no processor, no plugin and no reach into the compiler. A Payout document is a value, and a record is the better Payout. A mutable builder-heavy service class is not, and a record would not fit it.

The second thing Sam had got wrong was the order of operations. A lead's first week is for finding out what is true. A rule issued on day nine, by analogy with another platform, before counting the classes it touches, is a rule issued to the wrong team. The cost was a week of noise, a four-thousand-line pull request closed unmerged, and a message in the channel that Rafael had been too polite to answer.

Aisha asked, the same week, whether Kotlin would have avoided all this, since data classes do what records do and more. The honest answer is that Kotlin on Maven works, through its own compiler plugin, and that almost nobody does it, because the Kotlin ecosystem's tooling, documentation and examples assume Gradle. A Kotlin module on a Maven team is a second language, a second compiler in the build, and a dependency on the one build tool the team had just decided not to adopt. Not now, for the same reason as chapter 4, and the reason is gravity rather than merit.

Remember it as: stop adding before you start removing, and never issue a rule before you have counted what it touches.

Sam's decision, posted in the channel under the original message and left there: the ban is withdrawn. Lombok stays in the parent. New classes do not use it, and documents and values that are touched for any other reason become records, with no migration ticket and no deadline. Aisha's pull request is closed with thanks, and the three classes in it that were already records are merged. The message ends with the sentence Sam had wanted to avoid writing: I should have counted first. Rafael replied with a thumbs up, which on this team meant a great deal.

What Sam learned

• Count what a rule touches before issuing it, and issue it as "stop adding" before "start removing".
• A record is the language's answer to the value case; a processor that edits the compiler is a tool's answer to all cases, and it is renewed every JDK release.
• Reverse a wrong decision in the place it was made, in the same words, and say the part you would rather not.

…but

Aisha's closed pull request had one lasting effect: while converting Payout to a record she had broken a test, fixed the test, and then noticed that the test could not have failed in the first place, because it mocked the repository, mocked the service, and asserted that the mock returned what the mock had been told to return. The suite was at eighty-four percent coverage, and she asked Sam what the number measured.

Chapter 6 · Act IIAll Green, Nothing Tested#

The problem

The number on the dashboard was eighty-four percent, in green, and Rafael was proud of it. "We've held that line for three years. Every service over eighty. The platform team's gate won't merge below seventy-five."

Aisha put the test on the screen. It was called PayoutServiceTest, it was forty lines long, and it did this: it mocked the repository, it mocked the service that used the repository, it told the mocked service that createPayout returned a payout, it called createPayout on the mocked service, and it asserted that the result was the payout it had just been told to return. It had passed every day for three years. It could not fail. It was not testing the service; it was testing Mockito.

"That one's old," Rafael said. "There are better ones."

There were. There were also sixty-one like it, and Aisha had counted them. The pattern had a name on the team, "the template", and new tests were written by copying it. Coverage counted the lines the mocks walked through. The bug that had taken a day to find the previous month, a payout marked paid without its version bumped, had been covered by three green tests, all of which had mocked the thing that was wrong.

Sam had seen this in .NET, where it was called the Moq tax, and had no idea it could reach eighty-four percent. "What does the number measure?" Aisha asked again, and this time she was asking Rafael.

The idea

A test is a claim about behaviour that can be false. That is the whole definition, and it is the one the template failed. A test that mocks the thing it is testing is asserting that the mock framework works, and the mock framework does. The coverage tool, which counts lines executed, cannot tell the difference, because the lines were executed. Coverage measures what was run, not what was checked.

What should be mocked is the boundary: the thing the unit cannot run without, and cannot run truthfully in a test. For PayoutService the boundary is the ledger, which is another team's system across the network, and the bank gateway. The repository is not a boundary; it is a Spring Data interface over a database that chapter 7 will show can run in a container in two seconds. The service is never a boundary, because it is the thing under test.

Mockito, used well, polices some of this itself. Its strict stubs mode, the default under its JUnit 5 extension, fails a test that stubs a method the test never calls, which catches the template's first symptom. What it cannot catch is a test with no real object in it, and that is the reviewer's job: every test should name the one real thing it exercises, and if there is none, it is not a test.

Spring Boot adds a middle layer that the team had skipped entirely. A slice test starts only one part of the application: @WebMvcTest starts the web layer with the controller and nothing below it, so the request mapping, validation and error shape are tested against a real HTTP stack with the service replaced; @DataMongoTest starts only the Mongo repositories against a real database. Slices run in a second or two. The full @SpringBootTest, which starts everything, is the right test for the whole service and the wrong test for a controller's status code, because it costs twenty seconds of context start and the team had been paying that twenty seconds in sixty places. Where a bean must be replaced inside a slice or a full context, @MockitoBean is the current annotation; @MockBean, which the template used, was deprecated in Boot 3.4 and the migration in chapter 3 will remove it.

The pyramid that follows is ordinary and the team had inverted it: many unit tests with real domain objects, a handful of slices per service, one or two full-context tests per service, and the mocks only at the ledger and the bank. The two minutes of test time from chapter 4 was sixty full contexts starting one after another.

Remember it as: coverage counts what ran, not what was checked; a test that cannot fail is a line on a dashboard, not a test.

When Aisha had finished, Sam gave the team the decision: the template is deleted, with its sixty-one copies, and the coverage number is allowed to fall and will be reported honestly to the platform team with the reason. Rafael writes the replacement guidance, because the next template will be copied too and it should be his. Every new test names its one real object. Slices for controllers and repositories, one full-context test per service, mocks at the ledger and the bank only. Aisha pairs with whoever writes the first ten. Would the gate block them at seventy-five, Rafael asked. Probably, Sam said; the gate measured the wrong thing, and that was a conversation with Oren, not a reason to keep sixty-one tests that could not fail.

What Sam learned

• A test is a claim that can be false; if nothing in it is real, nothing in it can be false.
• Mock the boundary: the other team's system, the bank. Never the repository, never the subject.
• Coverage is a floor on what ran, not a measure of what was checked, and a gate on it rewards the template.

…but

The first replacement test Aisha wrote ran the repository against a real database in a container, and it was green, and the pull request it guarded went to production on Friday with Sam's approval, and at 17:40 the service refused every write with an error none of them had seen before. The container had been running MongoDB. Production was not.

Chapter 7 · Act IIThe Database That Isn't#

The problem

At 17:40 on Friday payouts-api stopped writing. Every save returned the same error, which none of them had seen before and which Oren read out from the log: Retryable writes are not supported. The pods were healthy, the database was up, the reads worked, and nothing could be paid.

The change had been Aisha's, and Sam had approved it two hours earlier on the strength of the new test from chapter 6: a repository test against a real database in a container, green, with the query actually run. The change moved the database connection from a hand-written configuration class to Spring Data's own property, which was the right thing to do and which the test had exercised end to end. Against MongoDB.

"Production isn't MongoDB," Rafael said, once the rollback had gone through and the writes had resumed. "It's DocumentDB. Amazon's thing. It speaks the Mongo wire protocol and the driver thinks it's Mongo. Mostly it is."

"Mostly," Sam said.

The old configuration class had a connection string with retryWrites=false on the end and a comment from 2021 that said do not remove, DocumentDB. Aisha's property had the same host and credentials and not the parameter, because MongoDB did not need it, and the container in the test was MongoDB. The test had been true about the wrong database.

Sam had signed it off. That was the part that would not go away.

The idea

Amazon DocumentDB is not MongoDB. It is a database of Amazon's own, with its own storage engine, that emulates the MongoDB API: it accepts the wire protocol, answers the way a MongoDB server would for the operations it supports, and declines the ones it does not. The driver cannot tell the difference, and neither can Spring Data MongoDB above it. The only party that knows the difference is the person who reads Amazon's list of what is and is not supported, and at Sorrel that person had left in 2021 with the comment as their monument.

The list is long and changes with each compatibility version Amazon ships, and it has a shape worth knowing. Some things are absent on the version you run and present on a later one: retryable writes, which the modern driver turns on by default, are supported from the 8.0.2 engine and rejected at the first write by everything before it, and Sorrel's cluster was created by a script that took the default, which is 5.0. Some operators and stages in the aggregation pipeline are unsupported or limited, and the unsupported list is published per version. Some behaviours differ quietly, in index types, in collation, in how a change stream has to be enabled. None of it is hidden; the failure is that nobody was reading the document against the code.

Testcontainers, meanwhile, is exactly what chapter 6 asked for: a real database, started in Docker for the duration of a test, torn down after. The practice plot has real ground. The trouble is that there is no DocumentDB container, because DocumentDB is a service and not a binary, so the real database in the test is MongoDB and the real database in production is an emulation of it. The test proves the query runs on MongoDB. Whether it runs on DocumentDB is a second question the test cannot ask.

The answer is not to stop using Testcontainers, the best tool the team has, nor a DocumentDB-shaped environment for every pull request, which Oren can provide only nightly. It is a feature contract: a short, owned document listing every MongoDB feature the code uses, each one checked against Amazon's supported list for the compatibility version Sorrel runs, with a date. Every query operator, every aggregation stage, every index type, every driver option. The pull request that adds a feature adds a line. The test suite gets one more test, which reads the connection options and refuses to start if retryWrites is not false, so that the next person who tidies the configuration meets the comment as a failing test rather than as a Friday.

Remember it as: the test ran against a real database; production runs an emulation of it, and the gap is a list someone must own.

Sam's decision, in the incident review on Monday, began with the sentence Sam owed: I approved a change on the strength of a test that could not have caught this, and I should have asked what database the test ran against. Then: the feature contract exists by Wednesday, Rafael owns it because he knows which features the old code leans on, and it lives in the repository beside the connection configuration. The driver options are asserted by a test. Oren provides a shared DocumentDB instance for a nightly integration run, which is the only place the contract is checked against the real thing, and the nightly run's result is in the channel every morning. Aisha asked whether the change had been wrong. Sam said the change had been right, the test had been right, and the approval had been wrong, and that those are three different things.

What Sam learned

• "Green" is not an approval criterion; "green against the database production runs" is, and when that is impossible, the gap must have an owner.
• A driver option with a warning comment is a test waiting to be written.
• Say which of the change, the test and the approval was wrong; they are three different things, and the lead's is usually the third.

…but

The nightly run against the real DocumentDB found nothing for a week, and then on the second Thursday it found something that was not a compatibility gap at all: two pods had paid the same claim, four seconds apart, and the bank had sent the money twice.

Chapter 8 · Act IIOne Document, Two Writers#

The problem

Claim 4471 was paid at 09:12:07 by one pod and at 09:12:11 by the other. The bank gateway, which does what it is told, sent two payments. The customer, who was honest, rang up.

Rafael had the sequence on the whiteboard by ten. The claim arrived from the policy system as a message. Both pods received it, because the message had been redelivered after a timeout on the first. Each pod checked whether a payout existed for the claim, found none, created one, saved it, and called the bank. The check and the save were four seconds apart on one pod and four seconds apart on the other, and the two four-second windows overlapped.

"We need a transaction," Rafael said. "Check and insert, atomically. DocumentDB does transactions now. We should have used them from the start."

Aisha had a different diagram. On hers there was a field on the payout called idempotencyKey, which held the claim number, and the question she had written next to it was: why isn't this unique?

Sam had met the same bug in .NET on a message queue. "Before we add a transaction," Sam said, "what does the database already know how to refuse?"

"A duplicate key," Aisha said. "If we let it."

Rafael said a unique index was not a design. Aisha said a transaction around a race was a lock around a bug. Both were partly right, and Sam asked them to draw the second write.

The idea

A document database holds each payout as one document: the claim, the amount, the status, the bank reference, together, written and read as a unit. That is its strength, and it reshapes the concurrency problem: most of what must stay consistent is already inside one document, and a single-document write is atomic without anyone asking.

What is not atomic is a check followed by a write, which is what both pods did. The fix for that is not to make the check-and-write atomic but to make the write refuse: a unique index on idempotencyKey means the second pod's insert fails with a duplicate-key error, which the service catches and treats as "already paid, return the existing payout". The idempotency key turns a redelivered message from a second payment into a second look at the first one. DocumentDB supports unique indexes, and the code had even annotated the field with one. What nobody knew was that Spring Data MongoDB stopped creating annotated indexes automatically in its version 3.0, behind a property that defaults to off, so the annotation had been a comment for four years and the index had never been built.

The second race is the one that would have come next. Two updates to the same existing payout, one marking it paid and one cancelling it, each reading the document, changing it, and saving the whole thing back. The last save wins and the first is lost. The fix is optimistic locking: a @Version field that Spring Data increments on every save and checks on every write, so that the second writer, holding a stale version, gets a failure instead of silently overwriting. The service retries or reports; it does not lose the update. Chapter 6's bug, the payout marked paid without its version bumped, was this mechanism half-installed.

Then Rafael's transaction, which deserves a fair answer. DocumentDB does support multi-document transactions, from its 4.0 compatibility version on, with limits Amazon publishes. They are the right tool when two documents must change together or not at all, and the wrong tool for a race that a unique index refuses more cheaply, because a transaction costs a round trip, holds resources, and must be retried on conflict, which is exactly the retry the idempotency key makes unnecessary. The honest rule: one document, one write, no transaction; two documents that must agree, a transaction, used rarely and named in the feature contract from chapter 7.

What remains is the bank. The payout document and the instruction to the bank gateway are two writes to two systems, and no transaction spans them. That is the problem the Atlas's transactional outbox solves, by writing the instruction into the same document store in the same write and letting a separate process deliver it, and chapter 11 is where Sorrel meets it.

Remember it as: let the database refuse the duplicate and the stale write; a transaction is for two documents that must agree, not for a race.

Sam drew the decision between the two diagrams: unique index on the idempotency key, created explicitly at startup rather than by an annotation nobody checks, and verified by the nightly run against DocumentDB; @Version on every document that is updated after creation, with the service treating a version failure as a retry, not an error; the duplicate-key path tested against the real index, not a mock. Transactions where two documents must move together, which today is nowhere, and the feature contract says so. Aisha asked whether the bank call was now safe. No, Sam said, it was now safe to retry, which was different, and the rest was chapter 11's problem and Bea's phone number.

What Sam learned

• A unique index is a decision the database enforces; a check in code is a hope two pods can share.
• Every document that is updated after creation carries a version, and a version failure is a retry.
• A transaction is for two documents that must agree; a race is for the index, and two systems are for the outbox.

…but

The index went in on Thursday and the nightly run was green on Friday, which was the morning security's spreadsheet arrived with forty-one findings, three of them critical, and one of the criticals in a library nobody on the team had ever heard of.

Chapter 9 · Act IIThe Jar in the Audit#

The problem

The email came from security, with a spreadsheet attached and a deadline in the subject line. The platform team's new scanner had walked every container image on the cluster and listed every library in each one against the public vulnerability database. The Payouts rows had forty-one findings, three of them critical, and one of the criticals was a library Rafael had never heard of.

"We don't use that," he said. "It's not in any POM. Look." He searched the repository. The name was not in it.

"It's in the image," Oren said. "The scanner doesn't read POMs. It reads jars."

Sam had done this in .NET with a package graph and a tool that drew it, and the answer had always been the same shape: the library you have never heard of is three steps below one you have. Aisha ran the dependency tree for the reports module, the one that had been on Java 11 in chapter 1, and there it was, four levels down, pulled in by a configuration library that pulled in a text-handling library at a version from 2020, with a critical finding published in 2022 against it.

"So it's transitive," Rafael said. "It's not our code."

"It's in our image," Sam said. "With our name on it. And the question security is going to ask is the one I'm about to ask, which is how long it has been there and why nobody knew." Rafael did not have the answer. Nobody did, because nobody had ever asked the build to list what it shipped.

The idea

A Maven build ships a dependency tree, not a dependency list. Each library you name brings the libraries it names, and so on down, and the tree for a modest Spring Boot service has two or three hundred nodes, of which the POM names perhaps thirty. The other two hundred and seventy are in the image with your name on them, and the scanner is right to count them.

When two branches of the tree ask for the same library at different versions, Maven has to choose one, and the rule it uses is nearest wins: the declaration closest to the root of the tree takes the version, and every other request for that library gets it too, whether it asked for a newer one or not. The rule is simple and deterministic, and it has one consequence a lead must know: adding a library can silently downgrade another library's dependency, because your new direct dependency is nearer than the transitive one that was there before. Gradle chose the opposite rule, highest wins, and both rules surprise people in different ways.

The tree above shows the common case: the fixed version is in the build already, on a different branch, and loses on distance. The fix is to make the version a decision rather than an accident, by managing it in the parent's dependencyManagement, which overrides distance for every branch at once; chapter 10 is about the tools that keep that decision true.

Finding the jar is one command. Triage is the lead's job, and it has a shape. For each finding: is the vulnerable code reachable from ours, or is it a class we never load; is there a fixed version, and does it fit under the managed versions we already have; and what does the vendor of the library between us and it say. Three tools answer three of those questions. dependency:tree shows the path. versions:display-dependency-updates lists what newer versions exist. OWASP's dependency-check plugin does what the platform scanner did, at build time, before the image exists, so the finding arrives in a pull request rather than an email. None of them answers reachability, which remains a human reading a CVE and a stack of code, and the honest triage says so.

Remember it as: the build ships a tree, not a list; nearest wins, and the jar you have never heard of has your name on it.

Sam's decision, in the reply to security and to the team: the three criticals are fixed this week by managing the versions in the parent, with the tree diff in each pull request. The dependency-check plugin goes into the parent so that the next finding fails a build rather than arriving in a spreadsheet, with a suppression file that requires a reason and a review date for every entry. Aisha owns the triage document, one line per finding, including the thirty-eight that are not critical, because the ones that are not reachable today will be reachable after someone adds a feature. Rafael owns reading the trees, because he wrote the parent and knows which branches are his. Oren asked whether Sam expected every team to do this. This team would, Sam said; the rest was Oren's call.

What Sam learned

• The image is the list; the POM is the part of the list someone wrote down.
• Nearest wins, so a version that matters is managed in the parent, where distance does not apply.
• Triage is reachability, fix availability and fit, recorded per finding with a date; the tool does the first command and the lead does the rest.

…but

Managing the three versions in the parent fixed the three criticals, and on Tuesday the nightly build of payouts-api failed with a method-not-found error in a class nobody had changed. Rafael traced it to a library that had quietly moved from one version to another over the weekend, because its version ended in the word SNAPSHOT, and the parent had not managed that one.

Chapter 10 · Act IIThe Upgrade That Broke Something Else#

The problem

The nightly build of payouts-api failed on Tuesday with a method-not-found error in a class nobody had changed, in a module nobody had touched, after a weekend in which nobody had merged anything. Rafael, who takes a build failure on an unchanged tree personally, had the cause before stand-up: a library called sorrel-commons, owned by Oren's platform team and used by every service, at version 1.4-SNAPSHOT. The platform team had pushed a new build of it on Saturday. The method Payouts called had been renamed.

"That's what SNAPSHOT is for," Oren said. "Teams get fixes without waiting for a release. It's been that way for years."

"It's been breaking people for years," Rafael said. "You changed a public method on a Saturday and every service that resolves the snapshot got it on Sunday."

Oren said the rename had been in the changelog. Rafael said nobody reads a changelog for a dependency that changes under them without a version bump, because the whole point of a version is that it does not. Both looked at Sam.

Sam had a .NET reflex here too, and this time checked it before speaking. The reflex was a lock file, which NuGet has and Maven does not. The question underneath was simpler. "Which of our dependencies can change without a change to our build?"

Aisha ran the tree with versions showing and read out three: the snapshot, a version range someone had written in 2022, and a managed version that chapter 9's fix had pinned in the parent but which one module had been overriding since before the pin existed.

The idea

Maven has no lockfile. A build resolves its tree afresh from the POMs, and the result is the same every time only if every version in the tree is fixed. Three things make a version not fixed. A SNAPSHOT is a version that names a moving target by design: 1.4-SNAPSHOT means "the latest build of what will become 1.4", and Maven checks the repository for a newer one daily. A version range such as [1.2,2.0) lets the resolver choose the highest available, which changes when someone publishes. And a transitive dependency with no managed version is chosen by nearest-wins, which changes when any nearer declaration changes. A build that contains any of the three can fail on an unchanged tree, and Sorrel's contained all three.

The tool that makes the tree a decision is the one chapter 9 reached for: dependencyManagement in the parent, which fixes a version for every branch regardless of distance. The tool that keeps the decision true is the enforcer plugin, which fails the build when a rule is broken. Two of its rules matter here. requireReleaseDeps forbids SNAPSHOT dependencies on the main branch, so a moving target can be used on a feature branch and must be replaced with a release before merge. requireUpperBoundDeps fails the build when nearest-wins has chosen a version lower than one some branch of the tree asked for, which is the silent downgrade from chapter 9 made loud. A third, banDuplicatePomDependencyVersions, catches the module that declared the same dependency twice.

Two more pieces close the loop. A BOM import is how a family of libraries publishes its own agreed versions, and importing it into the parent's dependencyManagement fixes the whole family at once; Boot's is the one every service already imports, and the resilience library and the test library in chapter 3's matrix publish their own. And dependency:analyze reports two things a tree never shows: dependencies the module declares and does not use, which are weight and attack surface, and classes the module uses from dependencies it does not declare, which arrive transitively today and vanish when a neighbour's tree changes.

Oren's point stands and deserves its answer. Teams do need fixes without a release ceremony, and a SNAPSHOT on a branch, consumed by the one team testing the fix, is the right tool for that. What is wrong is a SNAPSHOT on main, consumed by forty services that did not ask. The rule is therefore about branches, not about SNAPSHOTs, and a platform team that publishes a library owes its users a release cadence short enough that nobody needs the snapshot on main.

Remember it as: Maven has no lockfile, so every version is a decision or an accident; the enforcer is how a decision stays one.

Sam put the decision in the channel with Oren copied: the enforcer goes into the parent with release-only dependencies on main and the upper-bound rule on, which will fail the build today in two places, and those two are fixed today. The range from 2022 becomes a version. The module override from chapter 9 is deleted. dependency:analyze runs in CI as a warning for a month and a failure after. The platform team publishes sorrel-commons as a release every fortnight, and Payouts consumes releases on main and snapshots on branches only. Oren said a fortnight was a lot of releases. Sam said a Saturday rename was a lot of broken builds, and that releases were cheaper.

What Sam learned

• Every version that can move without a commit is a build that can fail without a change; find all three kinds.
• SNAPSHOTs belong on branches, and a platform team that needs them on main needs a release cadence instead.
• The enforcer turns the parent's decision into a rule the build keeps; a warning is not a rule.

…but

The enforcer failed the build twice on the first day and never again, and the first month ended with a tree that could not move without a commit. The first quarter opened on a Tuesday at 11:20 when Bea's ledger stopped answering, and Payouts, which had nothing wrong with it, stopped with it.

Chapter 11 · Act IIIThe Ledger Is Down, So Are We#

The problem

At 11:20 on Tuesday the ledger slowed down. It did not fail; Bea's service answered every request, in thirty to forty seconds instead of two hundred milliseconds, because a reporting job on her side was holding a lock it should not have held. By 11:26 Payouts was not answering at all. Every request thread in both pods was waiting on a ledger call that had not come back, the readiness probe could not get a thread either, and Kubernetes took the pods out of service for being unready. Nothing was wrong with Payouts. It was simply waiting, with everything it had, for someone else.

"We need retries," Rafael said at 11:40, on the call with Bea. "Three attempts, then fail."

"You have retries," Bea said. "That's why I'm on this call. Every request you send me is arriving three times. My job is slow because you tripled my load the moment I got slow."

Rafael had not known about the retries. Aisha found them: a client library from 2022 that retried every failure by default, including a timeout, including a timeout on a write. Sam, who had sat on the other end of a call like this in .NET, asked the question that the whole chapter answers.

"How long do we wait for the ledger before we give up?"

Nobody knew. Aisha found that too. The HTTP client had no timeout configured, which meant it waited forever, which was why both pods had been waiting since 11:21 with every thread they owned.

The idea

A service that calls another service inherits that service's failures, and the only question is how. Without limits, it inherits them completely: a slow dependency becomes a slow caller, a stalled dependency becomes a stalled caller, and a caller with a thread pool becomes a stalled caller with no threads. Resilience is the set of mechanisms that let a caller fail partially, quickly and on purpose rather than totally, slowly and by accident.

The first is the one Sam asked for. A connect timeout bounds how long to wait for the connection; a read timeout bounds how long to wait for the response. Spring's RestClient sets neither by default. With a two-second read timeout, the ledger's thirty-second responses become two-second failures, each thread is returned to the pool, and Payouts answers its probe and its other callers. That alone would have kept Payouts up on Tuesday; it would have failed every payout, correctly, instead of failing everything.

The second is the one Rafael asked for, used the way Bea needs. A retry repeats a failed call, and it is safe only when the call is idempotent, which chapter 8 made the ledger write, and kind only when it backs off: a short wait, then a longer one, with jitter so that two pods do not retry in step, and a budget so that a dependency in trouble sees a few extra requests and not a multiple of its load. The 2022 library retried instantly, three times, on everything, which is the mechanism for turning a slow dependency into a dead one.

The third is the circuit breaker, which is a memory. It counts recent outcomes in a sliding window; when the failure rate crosses a threshold it opens, and for a set time every call fails immediately without touching the dependency, which gives Bea's lock time to clear. Then it lets a few trial calls through and closes again if they succeed. A bulkhead is the fourth: a cap on how many calls to one dependency may be in flight at once, so that the ledger can consume at most, say, twenty of the pool's threads and the other callers keep the rest. Spring Framework 7 ships a retry and a concurrency limit in its core; it does not ship a breaker, and resilience4j does, which is why the companion project carries it.

The fifth is not a library. The ledger is Bea's, and the agreement between the two teams, what the ledger promises, how fast, and what it returns when it cannot, is a contract that both sides test. Sorrel's internal services authenticate to each other with client credentials from the company's identity provider, which is right, and which also means a token request is one more call with one more timeout to set.

Remember it as: a timeout bounds the wait, a retry with backoff forgives a blip, a breaker remembers, a bulkhead contains; without the first, the others never run.

Sam's decision, on the call, with Bea listening: timeouts on every outbound call by Thursday, connect one second, read two, starting with the ledger and the token endpoint. The 2022 client library is removed and its retries replaced with resilience4j's, exponential with jitter, on idempotent calls only, three attempts across a budget. A breaker and a bulkhead on the ledger, with the thresholds in configuration and the state in a metric Oren can graph. A contract test with Bea's team, owned jointly, run in both pipelines. Rafael asked whether the breaker would make payouts fail when the ledger was merely slow. Sam said yes, fast and visibly, which was the point, and that a payout that failed in two seconds could be retried by the thing that owned the retry, and a pod that waited forever could not.

What Sam learned

• The first question for every outbound call is how long we wait; until that has an answer, nothing else protects the pod.
• A retry is a gift to the caller and a load on the callee; make it idempotent, back it off, and budget it.
• The breaker's state is a metric or it is a secret; the other team's phone call is not a monitoring system.

…but

With timeouts in place the pods stayed up through the next ledger stall, and the graph of blocked request threads during it was still a wall. Rafael looked at the wall and said the real fix was to stop blocking threads at all, and that the way to do that was WebFlux.

Chapter 12 · Act IIILet's Go Reactive#

The problem

Rafael's second proposal had a graph instead of a benchmark. During the Thursday ledger stall, with timeouts in place, every request thread in payouts-api had spent two seconds blocked, waiting, and the pool of two hundred had been full for most of a minute. "Blocking is the disease," he said. "The timeout treats the symptom. WebFlux stops threads waiting at all. Non-blocking end to end: the request, the ledger call, the database. One event loop, a handful of threads, no pool to exhaust."

Aisha asked how much of the code would change. Rafael said most of it: the controller, the service, the client, and the repository, which would need the reactive Mongo driver. He said it was the right architecture and that the team would learn it.

Sam had led a team through the async-everything migration in .NET, where async and await had crept from the edges to the core over eighteen months, and remembered what the proposal left out: every library in the path had to speak the dialect, and a stack trace through it was read with grief.

"What Java are we on?" Sam asked. "Twenty-one," Aisha said, "twenty-five at the end of the quarter." "Then before we rewrite the service," Sam said, "I'd like to know what a thread costs."

Oren had the number: about a megabyte of stack per platform thread, and a ceiling of two hundred because somebody had set it in 2019.

The idea

A blocked thread is only expensive if threads are expensive, and until Java 21 they were. Each platform thread is a kernel thread with a fixed stack, so a service that blocks one per request can afford a few hundred, which is why pools exist and why a stalled dependency fills one. The reactive answer, WebFlux and Reactor, is to never block: a request becomes a chain of callbacks on a small event loop, each step returning before its result exists. The throughput is real. The price is that every library in the path must be written that way, that the code reads as a description of a pipeline rather than as a sequence of steps, and that a stack trace shows the pipeline's machinery rather than the program.

Virtual threads, final in Java 21 in 2023, are the other answer. A virtual thread is a thread scheduled by the JVM rather than the kernel, with a stack that lives on the heap and grows as needed, costing a few hundred bytes rather than a megabyte. When it blocks on a socket or a lock, the JVM unmounts it from the carrier thread it was running on and runs another; when the socket answers, it is mounted again. The code is the same blocking code. The ledger call still waits two seconds; the waiting now costs nothing anyone can measure, and there is no pool of two hundred to fill, because there are as many threads as there are requests.

Spring Boot has supported this since 3.2 behind one property, spring.threads.virtual.enabled, which runs the web server's request handling and most of Spring's own executors on virtual threads. The Mongo driver blocks, and that is fine, because a blocked virtual thread is the point. Two cautions remain. Pinning is the case where a virtual thread cannot be unmounted and holds its carrier while it waits: it happened inside synchronized blocks until JDK 24 removed that limitation in 2025, and still happens inside native code, so a library that blocks in a native frame can quietly turn virtual threads back into a pool. And a virtual thread is not a free lunch for CPU-bound work, which gains nothing from being unmounted.

When does WebFlux still win? When the work is genuinely a stream: a feed of events to thousands of open connections, backpressure between a fast producer and a slow consumer, or a stack that is already reactive end to end and would lose by converting back. Payouts receives a request, calls two systems and writes a document. It is the shape virtual threads were made for.

Remember it as: blocking is only a disease when threads are expensive; on Java 21 and later, turn on virtual threads before you rewrite anything.

Sam decided in the design review, with Rafael's graph still on the screen: the property goes on in payouts-api this week, and Oren repeats the ledger stall in the test environment on Friday. If the wall on the graph is gone, the WebFlux proposal is closed with thanks and the reason written down; if it is not, Rafael's proposal comes back with Sam's support. The pool ceiling from 2019 is deleted either way. Rafael said virtual threads were a bet on the JDK and WebFlux was proven. Sam said the JDK was a bet Sorrel had already made, forty services deep, and that the right time to learn the bet was wrong was before rewriting a service for the alternative.

On Friday the wall was gone. Rafael closed the proposal himself.

What Sam learned

• Ask what a thread costs before deciding that blocking is the problem.
• One property on Java 21 or later gives most services what a rewrite would, and the test is a repeat of the incident with the graph open.
• When the veteran closes his own proposal, say so in the decision log; it is the reason the next proposal will be honest too.

…but

Virtual threads went to production on the following Tuesday, and at 02:14 on Wednesday one of the two pods died with no stack trace, no log line and an exit code of 137, which Oren's dashboard explained as the container exceeding its memory limit. Rafael said the limit was too low, and Oren said the service had used what it was given.

Chapter 13 · Act IIIOOMKilled#

The problem

The pod died at 02:14 and left nothing behind. No exception, no heap dump, no final log line; the last entry was a routine payout at 02:13:58 and the next was the replacement pod starting. The dashboard had one word, OOMKilled, and one number, exit code 137, and Oren explained both before stand-up: the container had been using more memory than its limit of five hundred and twelve megabytes, and the kernel had killed it.

"Then the limit's too low," Rafael said. "It's a Java service. Give it a gigabyte."

"It had three hundred and eighty-four for the heap and used all of it, plus a hundred and sixty for everything else," Oren said. "I don't give a service more because it asked. I give it more when someone tells me what it needs and why."

Sam had a .NET memory of this too: a runtime that had grown into its container because nobody had told it where the fence was, and a week of tuning that ended in understanding rather than a bigger box.

"What does this service need," Sam asked, "and how would we know?"

Aisha pulled the image's startup flags. The heap had been set to seventy-five percent of the container's memory. The other twenty-five percent was expected to cover everything that was not heap. Nobody could say what that everything was, or how big, and the kernel had just said it was more than a hundred and twenty-eight megabytes.

The idea

A JVM in a container has to answer one question at startup: how much memory is there? Until 2018 it answered by asking the operating system, which told it about the whole machine, so a JVM with a one-gigabyte limit on a sixty-four-gigabyte host sized its heap for sixty-four, ran happily past its limit, and was killed without warning. Container awareness, default since JDK 10, reads the container's limit instead. The JVM now knows the fence line. What it does with it is the next decision.

By default the JVM takes a quarter of the limit for the heap, and MaxRAMPercentage changes the fraction. The companion project's image, like Sorrel's, had set it to seventy-five, which is common and which is a claim: that everything the JVM needs that is not heap fits in the remaining quarter. That everything has a name per part. Metaspace holds class metadata and grows with the number of classes loaded, which for a Spring Boot service is many. The code cache holds compiled code. Each thread has a stack; virtual threads keep theirs on the heap, carrier threads do not. Direct buffers are what the network and the driver use for I/O. None of these appear in a heap graph, and all of them count against the limit.

The JVM can account for all of it, on request. Native memory tracking, turned on with a flag and read with jcmd, reports each category, and reading it once from a pod under production load is the hour of work that replaces the week of guessing. Sorrel's number came to about two hundred megabytes outside the heap, not a hundred and twenty-eight, and the difference was metaspace and the code cache, both of which grow with the size of the application and neither of which the seventy-five percent had budgeted for.

Two more things turn the kernel's silence into a message. ExitOnOutOfMemoryError makes the JVM exit the moment the heap runs out, with a log line and the option of a heap dump, instead of spending its last minutes collecting garbage in a loop; the kernel kill is for the case where the whole process exceeds the limit, and a JVM whose heap is sized honestly rarely reaches it. And requests and limits should be equal for memory: a request below the limit tells the scheduler the pod needs less than it will use, which is how two services that each fit are placed on a node that fits neither.

Remember it as: the heap is the part you can see; the limit is the whole; size from a native memory report, not from a percentage someone chose.

Sam's decision, in the incident review: the limit is not raised until the service's footprint has been measured, and it is measured today, from a pod under load, with native memory tracking on. The heap is then set from the measurement, with the non-heap total and a margin written beside it in the manifest as a comment, and the percentage flag goes. Exit on out-of-memory goes into the image so that the next death has a stack trace. Memory request equals limit. Oren adds an alert on container memory at ninety percent of the limit, because the heap graph that everyone watches is the one graph that could not have predicted this. Aisha asked whether the answer would turn out to be a gigabyte after all. It might, Sam said, and it would then be a gigabyte with a reason, which was the only kind Oren was offering.

The measurement said six hundred and forty megabytes with headroom. Oren gave it seven hundred and sixty-eight.

What Sam learned

• The JVM knows its limit; it does not know what the rest of the process needs, and only a native memory report does.
• A pod that dies silently needs two changes: a flag so the heap case speaks, and an alert on the whole container so the rest does.
• "Give it more" is the right answer only after "how much does it need" has a measured one.

…but

With the limits set from measurement, the service stopped dying at night and started failing at lunchtime instead, in a way that was harder to see: Bea's team reported that about a dozen ledger callbacks a week got a connection error, and the dozen always arrived in the three minutes around a Payouts deploy.

Chapter 14 · Act IIIThe Deploy That Dropped Requests#

The problem

Bea's number was a dozen a week. Twelve ledger callbacks, out of some forty thousand, got a connection reset or a gateway error from Payouts, and when she plotted them against time every one sat inside the three minutes around a Payouts deploy. Payouts deployed twice a day. "Your rollouts drop requests," she said. "I've been retrying them from my side for a month. I'd rather you stopped dropping them."

"Boot does graceful shutdown," Rafael said. "By default, since 3.4. The server stops taking new connections and finishes the ones in flight. We don't drop anything."

Oren had the pod events, and the pod events said something slightly different. On a deploy, Kubernetes sent each old pod a termination signal and, at the same moment, began removing it from the service's endpoints. The two things happened in parallel and the second took longer. For about a second after the signal, the old pod's address was still being handed out, and a request that arrived in that second met a server that had just stopped listening.

"So it's a race," Sam said. "Between the pod stopping and the cluster noticing."

"It's always a race," Oren said. "The question is who's been told to wait."

Aisha read the manifest. No preStop hook. A termination grace period of thirty seconds, which was also Spring's default shutdown timeout. A readiness probe that checked the same endpoint as liveness. And a rolling update with the defaults, on a deployment of two pods.

The idea

A deploy is a sequence of handoffs, and every dropped request is a handoff where one party moved before the other was told. Kubernetes replaces pods by starting a new one, waiting for it to be ready, then stopping an old one, and each of those three words hides a decision the manifest has to make.

Ready is the readiness probe, and it should mean "can take traffic now", which is a different question from "is alive". Boot exposes both as separate health groups, liveness and readiness, and the readiness group goes false during shutdown by itself; a manifest that points both probes at the same general endpoint makes a pod that is alive look ready, and a pod that is draining look dead. A startup probe covers the first thirty seconds of a Spring context coming up, so that the liveness probe does not kill a pod for being slow to start.

Stopping is where Bea's dozen came from. The termination signal and the endpoint removal are issued together, and the removal propagates to every node's proxy on its own schedule. A preStop hook runs before the signal, inside the container, and the usual hook is a sleep of a few seconds: long enough for the endpoint removal to reach every proxy, so that by the time Spring begins its graceful shutdown no new request can be routed to the pod. The sleep is not a hack; it is the only way the pod has of waiting for a system it cannot see.

Then the arithmetic. Spring's graceful shutdown finishes in-flight requests within spring.lifecycle.timeout-per-shutdown-phase, thirty seconds by default. Kubernetes gives the pod terminationGracePeriodSeconds, also thirty by default, and that period includes the preStop hook. With both at thirty and a five-second hook, Spring can still be finishing a request when the kernel kills the pod. The fix is to make the grace period longer than the hook plus the shutdown timeout, which at Sorrel became twenty seconds for Spring, five for the hook, and forty for the pod.

Finally the rolling update itself. The defaults allow a quarter of pods to be unavailable and a quarter extra to be created, rounded for a two-pod deployment into "one down at a time", which halves capacity during every deploy. maxUnavailable: 0 with maxSurge: 1 starts the new pod first and takes the old one only when the new one is ready, which costs one pod's worth of memory for a minute and no capacity at all.

Remember it as: a deploy is three handoffs, start, ready, stop; the pod must wait for the cluster to notice, and the grace period must outlast the wait.

With Bea and Oren in the review, Sam decided: the manifest gets a preStop sleep of five seconds, a grace period of forty, separate readiness and liveness probes on Boot's health groups, a startup probe, and zero unavailable with one surge. Spring's shutdown timeout comes down to twenty. Aisha writes a deploy test for the kind cluster that sends requests continuously through a rollout and counts failures, and the count has to be zero before the change merges. Bea keeps her retries for a month and then takes them out, because a dozen a week that Payouts cannot see is a dozen Payouts should be seeing. Bea asked, politely, whether Boot's graceful shutdown had been doing anything at all. Everything, Sam said, except waiting for the one party nobody had told it about.

The deploy test counted zero on the third try. The first two found a load balancer timeout that was shorter than the hook.

What Sam learned

• Readiness, liveness and startup are three questions; a manifest that asks one has answered the other two wrong.
• The app's graceful shutdown cannot wait for the cluster to stop routing; a preStop sleep is the pod's only way to wait, and the grace period must cover it.
• Count the failures during a rollout, in a test, before and after; a partner team's retries are not a measurement.

…but

The deploy test went green and Bea took out her retries, and the quarter had three weeks left. Sam spent one of them reading the service's conditions report, four hundred lines of it, and found that no transaction manager existed, and that one of Sam's own @Transactional methods had, it turned out, never been transactional at all.

Chapter 15 · Act IIIThe Magic Box#

The problem

Sam ran the service with its debug flag on a quiet Tuesday because nobody ever had, and read the conditions report: every piece of configuration Boot had considered, with the reason it had applied or not. One line said the transaction support had not been configured, because no transaction manager existed. Sam, who had written a @Transactional on markPaid in PayoutService a month earlier by .NET reflex, asked how that method had been working.

"Spring Data Mongo doesn't make a transaction manager," Rafael said. "You declare one if you want one. If that annotation ever ran, it would fail at the call. It can't have run."

Aisha followed the call. markPaid was invoked from another method in the same class, and so the annotation had never run: Spring applies it through a proxy that intercepts calls from outside, and a call from inside the same object never meets the proxy. Two silences had hidden each other: the annotation did nothing, so the missing manager was never missed, and chapter 8's test passed because the version check was doing the work.

"So the framework silently ignored an annotation I wrote," Sam said, "and would have silently failed it if it hadn't."

"It reported both," Aisha said, pointing at the line about the manager.

Sam, after twelve years with a framework that did less and said more, said the thing a lead is not supposed to say in the third month: I do not trust what I cannot see.

The idea

Spring is not magic. It is three mechanisms, each of which keeps a record, and a lead who can read the records can trust the box because it is no longer closed.

The first is the container: Spring creates the objects your application is made of, called beans, wires each one's dependencies, and manages its lifecycle from construction to destruction. The beans are listable from Actuator, each with a name, a type and the configuration that caused it.

The second is auto-configuration. Boot ships a long list of configuration classes, each guarded by @Conditional conditions: on a class being present, on no bean of a type existing yet, on a property. At startup Boot evaluates every condition and keeps a report. Run the application with --debug and the report is printed in full: every auto-configuration that matched, with the condition that matched it, and every one that did not, with the reason. Transaction support was in the second list, under a condition that said no transaction manager bean existed, which for a Mongo service is the default: Boot wires the client and the repositories and leaves the manager to you. A line in a report the team could have read any day in four years.

The third is the proxy, which is how most annotations do their work. @Transactional, @Cacheable and the resilience annotations from chapter 11 all work by Spring handing other beans a wrapper around yours, so that a call from outside passes through code that opens the transaction, checks the cache or counts the failure first. The rule that follows is the one Sam fell over: a call that does not pass through the proxy is not intercepted, and a method calling another method on this does not pass through the proxy. The self-invocation trap is old, documented, and caught by nobody's compiler. The fix is structural: the annotated method lives on a bean called from another bean, so that the boundary the annotation marks is a boundary between objects.

What makes all three trustworthy is that each one tells you what it did. The beans endpoint lists the container. The conditions report explains auto-configuration. The proxy is visible in a stack trace, where the interceptor's frames sit above your method, and absent when it did not run. A framework that records all it does is a box with a log; read the log once per service when you inherit it, and again when anything surprises you.

Remember it as: Spring is a container, auto-configuration and proxies, and each keeps a record; the magic is only the part you have not read.

Sam's decision, on the last Friday of the quarter: every engineer on the team reads the conditions report and the beans list of one service they own, this month, and writes down the three things in it they did not expect. The self-invocation trap becomes a review rule with the diagram above pinned beside it, and markPaid moves to where the proxy can see it, which the chapter 8 test now checks by asserting the transaction actually opened. No transaction manager is declared until chapter 8's first real transaction needs one, and when it is, the report will say so. Rafael asked whether Sam trusted the box now. Sam said no, and that trust was not the point; the point was that the box could be asked, and that the team now knew how.

What Sam learned

• Spring is a container, a list of guarded auto-configurations and a set of proxies, and each of the three keeps a record you can read.
• An annotation on a method called from inside its own bean is a comment; the proxy only sees calls from outside.
• Distrusting a framework is fine on the first day; by the ninetieth, the job is to know how to ask it what it did.

…but

Ninety days ended on a Thursday, with Sam holding a document of fifteen decisions, each with a date, the argument it had settled, who had been right, and the condition under which it would be reopened. Rafael asked what it was for, and Sam said it was for the next lead, and for the next time any of them had the same argument from memory.

EpilogueEpilogue: The Decision Log#

Sam's handover at ninety days. Fifteen decisions in the form the next lead can argue with, then the map from this book to the handbook, every command collected by task, the timeline, and a glossary.

The fifteen decisions

Each record follows the shape Michael Nygard set out in 2011: context, decision, consequences, and the status it holds today. Every one names the argument it settled, who was right, and the condition under which it is reopened.

ADR 1: One parent, one release, one Boot

Context. Three services, three Java versions; a module overriding the parent's release and Boot version with no comment. Decision. Every module inherits sorrel-parent; any override of a parent property carries a comment with a reason and a date, and the pull request shows the effective POM diff. Reports moved to 21 immediately and to 25 with everyone else. Consequences. Overrides are visible and rare. Moving the least-understood module last cost a week and no incidents. Who was right. Rafael, that the parent said 21; Aisha, that it did not matter. Reopen when a module has a documented reason to stay behind for more than one quarter.

ADR 2: Every dependency through the artifact repository

Context. Half of Sorrel's builds resolved from Central directly; nobody could say where a jar had come from. Decision. All resolution through the Nexus mirror in every settings.xml and on every runner; the wrapper committed; the output timestamp set; CI caches the local repository keyed on POM hashes. The platform team publishes an uptime figure and a written fallback. Consequences. Nine-minute builds became four before any test changed. One road, with a stated condition. Who was right. Oren on the mirror; Rafael on the single point of failure. Reopen when the Nexus misses its stated uptime for two consecutive months.

ADR 3: Boot 4 is a project, not a version

Context. 3.5 out of open-source support since June; a one-line upgrade proposed. Decision. Inventory first: every dependency pinned outside the BOM, its Boot 4 compatible version, yes or no. The best-tested service moves first, runs beside the old, then the rest. Consequences. A week of list; one service moved with two behaviour changes caught by its tests; the parent changed last. Who was right. Rafael on urgency; Aisha on the length of the list. Reopen when the next major ships, with this record as the template.

ADR 4: Maven stays; Gradle is revisited on a measurement

Context. A Gradle branch built in three forty against nine ten. Decision. Measure the stages. Parallel modules and forked, reused test JVMs in the parent. Gradle is reconsidered when compilation is more than half of a build longer than ten minutes. Consequences. Four minutes five on Maven. The condition is written here so the meeting is never held from memory. Who was right. Rafael that the build was slow; Oren that it was the tests. Reopen when the written condition is met.

ADR 5: No new Lombok, records for values, no forced removal

Context. A ban issued on day nine by analogy with .NET; three hundred and six classes; a four-thousand-line pull request. Decision. The ban withdrawn in the place it was issued. Lombok stays in the parent; new classes do not use it; values and documents touched for any reason become records; no migration ticket. Kotlin declined for Gradle gravity. Consequences. A week of noise, then a shrinking count. The lead's first public reversal. Who was right. Rafael: "we could have just stopped adding it." Reopen when the Lombok count falls below fifty, at which point removal is one pull request.

ADR 6: A test names its one real object

Context. Eighty-four percent coverage; sixty-one tests that mocked the subject and could not fail. Decision. The template deleted. Every test names the real object it exercises. Slices for controllers and repositories, one full-context test per service, mocks only at the ledger and the bank. Coverage is reported honestly and the gate conversation goes to the platform team. Consequences. Coverage fell to seventy-one and the number means something. Two minutes of tests became forty seconds. Who was right. Aisha, first. Reopen when a slice cannot express a test the team needs, which has not happened.

ADR 7: A feature contract for DocumentDB, owned and dated

Context. A green Testcontainers suite against MongoDB; production on DocumentDB 5.0 refusing retryable writes; the lead approved the change. Decision. A contract listing every operator, stage, index type and driver option the code uses, checked against Amazon's supported list for the engine version Sorrel runs, with a date per line. A test asserts the driver options. A nightly run against a real DocumentDB instance. Consequences. One compatibility gap found in the first month, in a pull request rather than on a Friday. Who was right. Rafael on "mostly"; the lead was wrong on the approval. Reopen when Sorrel moves to engine 8.0, at which point every line is rechecked.

ADR 8: Let the database refuse

Context. Two pods paid one claim; the unique-index annotation had never created an index. Decision. The unique index on the idempotency key created explicitly at startup; @Version on every document updated after creation, with a version failure treated as a retry; transactions only where two documents must agree, which today is nowhere; the ledger write goes through an outbox. Consequences. The duplicate path is tested with two threads. Who was right. Aisha on the index; Rafael that a transaction will one day be needed. Reopen when a second document has to move with a payout.

ADR 9: The image is the list

Context. Forty-one scanner findings, one critical in a transitive jar nobody had heard of, four levels down. Decision. Critical versions managed in the parent; dependency-check in the build at the critical threshold, with a suppression file that requires a reason and a review date; a triage document, one line per finding with reachability and a date. Consequences. The next critical failed a build on a Tuesday instead of arriving by email on a Friday. Who was right. Oren: the scanner reads jars. Reopen when the suppression file exceeds twenty entries, which is the sign that the gate is being worked around.

ADR 10: No version moves without a commit

Context. A platform library changed under the build on a Saturday; a range from 2022; a module override older than the pin. Decision. The enforcer in the parent: release dependencies only on main, upper-bound rule on. SNAPSHOTs on branches only. dependency:analyze as a warning for a month, then a failure. The platform team releases sorrel-commons fortnightly. Consequences. Two failures on day one, none since. Who was right. Rafael on the breakage; Oren that teams need fixes fast, which is what the fortnight is for. Reopen when the fortnightly cadence slips and teams start asking for snapshots on main again.

ADR 11: Every outbound call has a timeout before it has anything else

Context. The ledger slowed to thirty seconds and Payouts, with no timeouts and a retrying client, stopped entirely and tripled the ledger's load. Decision. Connect one second, read two, on every call including the token endpoint. Retries only on idempotent calls, exponential with jitter, budgeted. A breaker and a bulkhead on the ledger with state in a metric. A contract test with Bea's team in both pipelines. Consequences. The next ledger stall failed payouts in two seconds and kept the pods up. Who was right. Bea. Reopen when a dependency's measured latency makes the fixed numbers wrong twice in a quarter, at which point adaptive limits are considered.

ADR 12: Virtual threads before any rewrite

Context. A WebFlux proposal with a graph of blocked threads. Decision. spring.threads.virtual.enabled on; the ledger stall repeated in test; the pool ceiling from 2019 deleted. WebFlux reconsidered only for a genuine stream. Consequences. The wall on the graph was gone on Friday. Rafael closed his own proposal. Who was right. Rafael that blocking was the symptom; the JDK on the cure. Reopen when a service is a stream with backpressure, or pinning in native code is found under load.

ADR 13: Size memory from a native memory report

Context. A pod killed at 02:14 with exit code 137 and nothing else; seventy-five percent heap on a half-gigabyte limit. Decision. Footprint measured on a pod under load with native memory tracking; heap set from the measurement with the arithmetic in a manifest comment; the percentage flag removed; exit on out-of-memory in the image; memory request equal to limit; an alert on container memory at ninety percent. Consequences. Seven hundred and sixty-eight megabytes, with a reason. No night deaths since. Who was right. Oren: "I give it more when someone tells me what it needs." Reopen when the measured non-heap footprint changes by more than a fifth after an upgrade.

ADR 14: A deploy waits for the cluster to notice

Context. A dozen ledger callbacks a week failing in the three minutes around a deploy; Boot's graceful shutdown on and doing everything except waiting for routing. Decision. A five-second preStop sleep; a forty-second grace period over a twenty-second Spring shutdown; readiness, liveness and startup as three probes on Boot's health groups; zero unavailable, one surge. A deploy test that counts failures during a rollout and must count zero. Consequences. Zero on the third try; Bea removed her retries. Who was right. Oren: "it's always a race; the question is who's been told to wait." Reopen when the measured endpoint propagation on Sorrel's cluster exceeds the sleep.

ADR 15: Read the report, assert the effect

Context. No transaction manager in a service with a @Transactional on a self-invoked method that had never been transactional; two silences hiding each other. Decision. Every engineer reads the conditions report and beans list of one service they own and records three surprises. Self-invocation of annotated methods is a review rule with the diagram pinned up. Tests assert the effect of an annotation, never its presence. Consequences. Four surprises found in the first month, one of them a bean nobody wanted and one a bean nobody had declared. Who was right. Aisha: "it reported both." Reopen when the framework stops logging what it did, which it has not in twenty-three years.

The map to the handbook

This book tells the argument; Java for C# Developers holds the mechanics. The pointers from every chapter, collected.

The kit, by task

Every command in the book, grouped by what you are trying to find out. All of them run against examples/payouts; the ones marked cluster need the kind cluster from make kind-up.

What is actually in the build

./mvnw -q -f examples/payouts/pom.xml -pl payouts-api help:effective-pom -Doutput=/tmp/eff.xml   # the merged truth for one module
./mvnw -q -f examples/payouts/pom.xml -N help:evaluate -Dexpression=project.parent.version -DforceStdout
./mvnw -q -f examples/payouts/pom.xml -pl payouts-api -am compile dependency:tree -DoutputFile=/tmp/tree.txt  # every jar that ships
./mvnw -f examples/payouts/legacy/pom.xml -pl payouts-reports dependency:tree -Dverbose -Dincludes='*:commons-text'   # who brought one jar
./mvnw -q -f examples/payouts/pom.xml -pl payouts-api -am compile dependency:list -DexcludeGroupIds=org.springframework,org.springframework.boot
./mvnw -f examples/payouts/pom.xml -pl payouts-api -am compile dependency:analyze -DignoreNonCompile      # unused declared, used undeclared
./mvnw -q -f examples/payouts/pom.xml help:effective-settings                                        # which mirror this machine uses
cat examples/payouts/.mvn/wrapper/maven-wrapper.properties                                           # which Maven the wrapper pins

Whether the build is safe to ship

./mvnw -f examples/payouts/pom.xml versions:display-dependency-updates
./mvnw -f examples/payouts/legacy/pom.xml -pl payouts-reports dependency-check:check -DfailBuildOnCVSS=7
./mvnw -f examples/payouts/legacy/pom.xml enforcer:enforce -Denforcer.rules=requireReleaseDeps,requireUpperBoundDeps
./mvnw -q -f examples/payouts/pom.xml dependency:tree | grep -E 'SNAPSHOT|[\[\(][0-9]'               # versions that can move
./mvnw -q -f examples/payouts/pom.xml --offline compile                                              # can we build with the road closed

Where the time goes

time ./mvnw -q -f examples/payouts/pom.xml -DskipTests compile
time ./mvnw -q -f examples/payouts/pom.xml test -DforkCount=1C -DreuseForks=true
time ./mvnw -q -f examples/payouts/pom.xml -T 1C -DskipTests package
grep -h 'Tests run' examples/payouts/*/target/surefire-reports/*.txt | sort -t, -k5 -r | head -5

What the tests actually check

./mvnw -q -f examples/payouts/pom.xml -pl payouts-domain test -Dtest=MockEverythingTest -Dsurefire.failIfNoSpecifiedTests=false
./mvnw -q -f examples/payouts/pom.xml -pl payouts-domain test -Dtest=PayoutRepositoryMongoTest -Dsurefire.failIfNoSpecifiedTests=false
grep -rl '@Mock' examples/payouts --include='*Test.java' | xargs grep -l 'Service service' | wc -l   # tests that mock their subject
grep -rhoE '\$[a-zA-Z]+' examples/payouts/*/src/main/java | sort | uniq -c | sort -rn              # the DocumentDB feature contract

What the service does under load

docker compose -f examples/payouts/docker-compose.yml up -d mongo
./mvnw -q -f examples/payouts/pom.xml -DskipTests install
./mvnw -q -f examples/payouts/pom.xml -pl payouts-api spring-boot:run -Dspring-boot.run.profiles=fakeledger &
curl -s -X PUT 'http://localhost:8080/ledger/mode?delay=10s'                                        # stall the fake ledger
curl -s http://localhost:8080/actuator/circuitbreakers | jq '.circuitBreakers.ledger | {state, failureRate}'
jcmd $(jps -l | grep -m1 payouts | cut -d' ' -f1) Thread.dump_to_file -overwrite -format=json /tmp/threads.json   # virtual threads, which Actuator cannot see
curl -s http://localhost:8080/actuator/beans | jq '.contexts[].beans | keys | length'
./mvnw -q -f examples/payouts/pom.xml -pl payouts-api spring-boot:run -Dspring-boot.run.profiles=fakeledger -Dspring-boot.run.arguments=--debug 2>&1 | sed -n '/CONDITIONS EVALUATION REPORT/,/Exclusions/p'

What the cluster sees (cluster)

kubectl -n payouts exec deploy/payouts-api -- jcmd 1 VM.native_memory summary
kubectl -n payouts get pod -l app=payouts-api -o jsonpath='{range .items[*]}{.metadata.name} {.status.containerStatuses[0].lastState.terminated.reason}{"\n"}{end}'
kubectl -n payouts exec deploy/payouts-api -- curl -s localhost:8080/actuator/health/readiness
kubectl -n payouts rollout restart deploy/payouts-api && kubectl -n payouts rollout status deploy/payouts-api

Timeline

Glossary

• Artifact repository. The company's own store of jars, proxying Maven Central; the only place a dependency can be counted. Chapter 2.
• Auto-configuration. Boot's list of guarded configuration classes, each applied only when its conditions hold; the conditions report says which did. Chapter 15.
• BOM. A bill of materials: a POM whose only job is to agree a family of versions, imported into dependencyManagement. Chapters 1, 10.
• Bulkhead. A cap on in-flight calls to one dependency, so that it cannot consume every thread. Chapter 11.
• Circuit breaker. A memory of recent failures that, once a threshold is crossed, fails calls immediately for a while and then tries again. Chapter 11.
• Container awareness. The JVM reading the container's memory and CPU limits rather than the host's; default since JDK 10. Chapter 13.
• Dependency management. The parent POM section that fixes a version for every module without adding the dependency. Chapters 1, 9, 10.
• DocumentDB. Amazon's database that emulates the MongoDB API on its own engine; compatible with a published list, per engine version. Chapter 7.
• Effective POM. The parent, imports and overrides merged into one document; the only POM that tells the truth. Chapter 1.
• Enforcer. The Maven plugin that fails a build when a rule is broken: no snapshots, no silent downgrades. Chapter 10.
• Feature contract. The owned, dated list of every database feature the code uses, checked against what production supports. Chapter 7.
• Idempotency key. A value on a request that makes a repeat of it harmless, enforced by a unique index. Chapter 8.
• Metaspace. The JVM's memory for class metadata, outside the heap and inside the container limit. Chapter 13.
• Nearest wins. Maven's rule for choosing between versions of the same artifact: the declaration closest to the root. Chapters 9, 10.
• Optimistic locking. A version field checked on every write so that a stale writer fails instead of overwriting. Chapter 8.
• Pinning. A virtual thread unable to unmount from its carrier while blocked; in synchronized until JDK 24, in native code still. Chapter 12.
• preStop. A hook Kubernetes runs before the termination signal; a short sleep lets routing catch up. Chapter 14.
• Proxy. The wrapper Spring hands other beans so that annotations like @Transactional can act on calls from outside. Chapter 15.
• Readiness probe. The question "can this pod take traffic now", distinct from liveness and startup. Chapter 14.
• Release flag. maven.compiler.release: compile for a Java version and reject newer APIs. Chapter 1.
• Retry budget. The total extra load a retrying caller may put on a dependency; the difference between a retry and a storm. Chapter 11.
• Self-invocation. A call to an annotated method from inside its own bean, which bypasses the proxy. Chapter 15.
• Slice test. A Spring test that starts one layer: the web layer, or the repositories. Chapter 6.
• SNAPSHOT. A version that names a moving target; for branches, never for main. Chapter 10.
• Testcontainers. A real database in a container for the duration of a test; real MongoDB, which is not DocumentDB. Chapters 6, 7.
• Virtual thread. A JVM-scheduled thread with a heap-allocated stack, cheap enough to block. Chapter 12.
• Wrapper. mvnw: the committed script that downloads and runs the Maven version the repository names. Chapter 2.