Before you start · 5 min readFrom Docker to Kubernetes: A Short Story and Three Manuals#

A short story for developers and DevOps engineers who have a vague idea of Docker and Kubernetes and would like a clear one, followed by three manuals: one for the shell that runs underneath all of it, and two for when they want to write the files themselves.

The promise

Read this in an evening or two and you will know what every core piece of Docker and Kubernetes is for, why it exists, and the twenty percent of commands that do eighty percent of the work. You will also know what Helm, Istio, Prometheus, Grafana and Argo CD add on top, and when you would reach for each. And when you inherit a repository written in 2018, you will know what every old word in it became.

Then, if you want it, Part IV makes you the person who writes and reviews the files: Dockerfiles that are small, cached, non-root and shut down cleanly; Compose projects with health-gated startup and per-environment overrides; and Kubernetes manifests that pass the restricted security standard, roll out without dropping a request, and survive a node being drained. And it teaches the language underneath every RUN line and every entrypoint, from first principles: what a command, a process and a quote really are, then grep, sed, awk and jq, up to writing a production bash script and reading anyone else's.

The story reads as one plot. Maya, a developer at a tiny startup, ships an app called Pantry. Every chapter starts with something breaking or somebody asking for something the current setup cannot do. The fix is the chapter's concept. By the end of the chapter, a new crack has appeared. Concepts you learn because you needed them are the ones you remember.

Every concept comes with three memory hooks: a place in one running metaphor (a kitchen, then a city), a diagram of how it actually works, and a one-sentence Remember it as line you can repeat out loud. Every chapter ends with the commands, a hands-on section you can paste into a terminal, the traps that bite silently, five recall questions (answers in the epilogue), and exactly three takeaways.

Who this is for

You have run docker run at least once and have heard of Pods. You may have copied a YAML file without knowing what half of it did. You want to be the person on the team who knows what is going on, without reading two thousand pages of documentation. If you are already an SRE with years of cluster time, this book will be too slow for you.

How to read it

• Straight through the first time. The plot carries the concepts.
• Skim on a re-read: each chapter has the same shape, so jump to The idea for the concept or The commands for the recipe.
• The Epilogue is the reference: the full metaphor table, the diagram legend, and one cheat sheet of every command in the book.
• Part IV, The Manuals comes after the epilogue. The story stands on its own; the manuals are reference you read after it or come back to when you sit down to write a Dockerfile, a Compose file or a manifest. Each chapter climbs from basic to advanced, ends in one annotated production file, a review checklist and a broken file to fix.
• The Shell manual needs no prior chapter, not even the story. Read it first if set -e and exec "$@" look like noise to you, or come to it whenever a script confuses you.

Following along

The companion app lives in examples/pantry/. It is a fifty-line Python web service that greets you and counts visits in Redis. You need:

• Docker Desktop (or Docker Engine on Linux) with at least 8 GB of memory allocated if you plan to install Act III's tools.
• kubectl and kind for Act II. The kind cluster config is examples/kind-config.yaml.
• helm, istioctl and argocd CLIs for Act III, each introduced in its chapter.

Commands are written for a POSIX shell (macOS, Linux, or WSL2 on Windows). If you get lost, examples/reset.sh <chapter> puts your machine back at that chapter's starting line.

Tested on: Docker 27.5 · Docker Compose 2.32 · kind 0.33 (Kubernetes 1.37) · kubectl 1.31 · Helm 4.3 · Istio 1.31 · kube-prometheus-stack chart 80.x · Argo CD 3.5 · Shell manual: bash 5.2 with GNU tools and BusyBox 1.37 in the shell lab (examples/manual/shell-lab/), and macOS /bin/bash 3.2 with BSD tools. Newer versions will almost certainly work; the commands here are the stable ones.

Contents

Act I: One kitchen (Docker)

1. It Works On My Machine
2. Write the Recipe Down
3. Memory and Neighbours
4. Ship It
5. The Whole Band
6. The Night It Fell Over

Act II: The city (Kubernetes)

7. Meet the City
8. The Smallest Thing That Runs
9. Someone Who Never Sleeps
10. A Number That Never Changes
11. Sticky Notes and the Locked Drawer
12. Are You Alive?
13. Memory That Survives, Again
14. Districts, Key Cards and Rush Hour
15. The Night It Fell Over, Again

Interludes

• Old Maps: the history of containers, orchestrators and moving YAML, told through an inherited repository
• The Landlord's Week: emptying a node safely and putting a ceiling on a district

Act III: The city grows up (the ecosystem)

16. Flat-Pack Furniture (Helm)
17. A Concierge at Every Door (Istio)
18. The Control Room (Prometheus and Grafana)
19. Nobody Touches the City by Hand (Argo CD)

Epilogue

20. The Map: metaphors, timeline, old words, cheat card, cheat sheet, recall answers, where to go next

Part IV: The Manuals

• The Manuals: how to use them, and a map of every chapter

The Shell manual

1. The Ground Floor
2. Every Command Is a Process
3. How the Shell Reads a Line
4. Variables and Quoting
5. Exit Status and Tests
6. Redirection and Pipes
7. Loops, Globs and Lines
8. Functions and Arrays
9. Processes and Signals
10. Finding Things: grep, find and xargs
11. sed
12. awk
13. The Pipeline Toolkit
14. Watching the System
15. Writing a Script
16. Portability
17. Reading Any Script

The Docker manual

1. Every Instruction
2. ENTRYPOINT, CMD and PID 1
3. Context and Cache
4. Multi-Stage Builds
5. Users and Permissions
6. Supply Chain and Secrets
7. Networks
8. Volumes
9. Compose: The File
10. Compose: The Project

The Kubernetes manual

1. Reading Any Manifest
2. The Pod Spec
3. Security Context
4. Lifecycle and Rollouts
5. Resources and Placement
6. Config and Secrets
7. Storage
8. Networking
9. Identity and Access
10. Packaging: Helm and Kustomize
11. Debugging

The Bridge

• The Bridge: every Docker concept mapped to its Kubernetes field

Chapter 1 · Act I · 4 min readIt Works On My Machine#

The problem

Pantry is three weeks old. It is a small web service that says hello and counts how many people have said hello back, and it stores that count in Redis. Maya wrote it, Maya runs it, and on Maya's laptop it has never once failed.

Today is demo day. Sam from sales has borrowed the app to show a customer, and Sam's laptop is a different story. Python is the wrong version. Redis is not installed, and the installer wants an admin password Sam does not have. Forty minutes before the meeting, Sam sends Maya a screenshot of a stack trace and the words "it doesn't work".

Maya knows exactly what is wrong. Pantry does not just need its own code; it needs a particular Python, a particular set of libraries, a running Redis, and an environment variable telling it where Redis lives. On her laptop all of that happens to be there. On Sam's laptop none of it is. The app is fine. The surroundings are the problem.

She has a vague memory of a colleague saying "just put it in a container". She has forty minutes to find out what that means.

The idea

A container is a process running on your machine with its own private view of the filesystem, network and process list. It is not a virtual machine. There is no second operating system booting inside it. It is your kernel, running your program, with walls around it. Those walls mean the program sees exactly the files and libraries it was packaged with, and nothing from the host that could surprise it.

Where do the files come from? From an image. An image is a read-only, versioned snapshot of a filesystem plus a little metadata: which program to start, which port it listens on, which environment variables to set. You download an image once and can start as many containers from it as you like. Each container gets its own thin writable layer on top of the image, so changes inside one container never leak into another.

Think of the image as a recipe card. It is written down, it has a version, and it does not change while you cook. A container is a dish cooked from that card. You can cook the same dish three times for three tables, throw one away when the table leaves, and the card is untouched. This is the single most important distinction in Docker, and almost every early confusion comes from mixing the two up. docker images lists recipe cards. docker ps lists dishes.

Remember it as: an image is the recipe card, a container is a dish cooked from it, and you can cook as many as you like.

A container has a short life cycle, and it helps to know the states:

Because a stopped container keeps its writable layer, you can docker start it again and find your files still there. Because that layer is deleted with docker rm, anything you want to keep must live somewhere else, which is the subject of chapter 3.

Maya's forty-minute fix is embarrassingly small. Instead of installing Redis on Sam's laptop, Sam runs one command that downloads the Redis image and starts a container from it. Redis is now listening on port 6379 of Sam's laptop, no admin password required, and it can be thrown away after the meeting with one more command.

The commands

# Start a container in the background (-d), give it a name, publish a port host:container
docker run -d --name redis -p 6379:6379 redis:7-alpine

# Run something interactive and throw the container away when it exits
docker run -it --rm python:3.12-slim python -c 'print("hello from inside")'

# What is running? (-a also shows stopped containers)
docker ps
docker ps -a

# Read a container's stdout/stderr; -f follows like tail -f
docker logs -f redis

# Run a command inside a running container (get a shell with: docker exec -it redis sh)
docker exec -it redis redis-cli ping

# Stop (SIGTERM, then SIGKILL after 10s) and remove
docker stop redis
docker rm redis

# See what images (recipe cards) you have locally
docker images

Try it

1. Start Redis in a container:

   docker run -d --name redis -p 6379:6379 redis:7-alpine
   

Docker prints a long container ID. The first run also pulls the image; the second time it starts in under a second.

2. Prove it is running and talk to it:

   docker ps
   docker exec -it redis redis-cli ping
   

   CONTAINER ID   IMAGE            COMMAND                  STATUS         PORTS                    NAMES
   3f9c2a1b8e44   redis:7-alpine   "docker-entrypoint.s…"   Up 5 seconds   0.0.0.0:6379->6379/tcp   redis
   PONG
   

3. Run a throwaway container to feel how cheap they are:

   docker run -it --rm python:3.12-slim python -c 'import sys; print(sys.version)'
   

You get a Python 3.12 prompt's worth of output on a machine that may not have Python installed at all. When the process exits, --rm deletes the container.

4. Stop and remove Redis, then check it is gone:

   docker stop redis && docker rm redis
   docker ps -a
   

The image is still there (docker images). Only the dish was thrown away, not the recipe.

Recall

1. What is the difference between an image and a container, in one sentence?
2. Which command lists images, and which lists running containers?
3. What does --rm do, and when would you not want it?
4. After docker stop and docker rm, is the image still on your machine?
5. In the kitchen metaphor, what is a container?

What Maya learned

• An image is a read-only recipe; a container is a running dish cooked from it, and the two are listed by different commands.
• A container is just a process with walls, so starting one costs about as much as starting the program itself.
• run, ps, logs, exec, stop and rm cover almost everything you do with a single container.

…but

Sam's demo works. Redis came from a container, but Pantry itself still runs from a folder of source files with python app.py, and it still needs the right Python and the right libraries. The customer asks a reasonable question: "Can you give us this to try on our own machines?" Maya cannot hand them her laptop. She needs a recipe card for Pantry.

Chapter 2 · Act I · 4 min readWrite the Recipe Down#

The problem

The customer wants to try Pantry on their own machines. Maya's first instinct is a README: install Python 3.12, create a virtualenv, pip install these three libraries, run Redis somehow, export REDIS_HOST, then python app.py. She writes it, reads it back, and realises it is the same list of things that broke on Sam's laptop, just written down politely.

Yesterday Redis came out of a container and it was wonderful, because somebody at Redis had already written the recipe. Nobody has written the recipe for Pantry. Maya has to write it herself.

She has one more constraint. She changes app.py twenty times a day. If every change means a full ten-minute rebuild of "download Python, install libraries, copy code", she will never use this. The recipe has to be cheap to re-cook when only the last step changed.

The idea

A Dockerfile is the recipe. It is a plain text file with one instruction per line, read top to bottom, and docker build turns it into an image. Here is Pantry's entire recipe:

FROM python:3.12-slim                       # start from someone else's recipe
WORKDIR /app                                # cd, creating the directory if needed
COPY requirements.txt .                     # copy the dependency list in first...
RUN pip install --no-cache-dir -r requirements.txt   # ...and install it
COPY app.py .                               # now the code, which changes most often
ENV PANTRY_VERSION=v1                       # default environment
EXPOSE 8080                                 # documentation: the app listens here
CMD ["python", "app.py"]                    # what to run when a container starts

Every instruction produces a layer: a snapshot of what changed on disk after that step. An image is a stack of layers, and here is the trick that makes daily use bearable. Docker caches each layer, keyed on the instruction and the files it touched. When you rebuild, Docker walks down the recipe, reusing cached layers until it hits the first step whose inputs changed. Everything after that point is rebuilt; everything before is free.

That is why the recipe copies requirements.txt and installs libraries before copying app.py. Editing the code invalidates only the last two layers, which take a second. Editing the dependency list invalidates the install step, which is rare. Order the recipe so the things that change most often come last.

Remember it as: a Dockerfile is a recipe, each line is a step that becomes a cached layer, so put the steps that change most often at the bottom.

A few details matter in practice. FROM picks a base image; python:3.12-slim is a trimmed Debian with Python installed, and slim is usually the right choice. RUN executes at build time, CMD at container start time; confusing the two is the second most common Dockerfile mistake. A .dockerignore file next to the Dockerfile lists paths that should never be copied in, such as .git and __pycache__, which keeps builds fast and images clean.

Finally, tags. An image name like pantry:v1 is repository:tag. The tag is free text; v1, 2024-06-01 and latest are all just labels. latest is not magic and does not mean newest, it is simply the tag used when you do not give one. Tag every build with something meaningful and treat latest with suspicion.

Two instructions look alike and are not. CMD is the default command, replaced entirely if you pass one on docker run; ENTRYPOINT is the command that always runs, with CMD or your arguments appended to it. A tool image sets ENTRYPOINT ["redis-cli"] so that docker run redis-cli ping reads naturally; a service like Pantry only needs CMD. Both come in two forms. The exec form, CMD ["python", "app.py"], runs your program as process 1. The shell form, CMD python app.py, runs /bin/sh -c "python app.py", so process 1 is a shell and your program is its child. That matters at stop time: Docker sends SIGTERM to process 1, the shell does not pass it on, and after ten seconds Docker kills everything. Use the exec form, and make sure whatever runs as process 1 actually handles the signal.

The commands

# Build an image from the Dockerfile in the current directory and tag it
docker build -t pantry:v1 .

# Build with a different value for an ARG in the Dockerfile
docker build -t pantry:v2 --build-arg PANTRY_VERSION=v2 --build-arg PANTRY_ERROR_RATE=0.3 .

# Which recipe cards do I have?
docker images

# Give an existing image a second name (same layers, no copy)
docker tag pantry:v1 pantry:latest

# Show the layers an image is made of, biggest first tells you where the weight is
docker history pantry:v1

# Delete an image you no longer need (fails if a container still uses it)
docker rmi pantry:latest

Try it

1. From examples/pantry, build the image and time it:

   cd examples/pantry
   docker build -t pantry:v1 .
   

The first build pulls the base image and installs the libraries. It ends with naming to docker.io/library/pantry:v1.

2. Build again without changing anything:

   docker build -t pantry:v1 .
   

Every step says CACHED. Now open app.py, change the greeting, save, and build a third time. Only the COPY app.py step and the ones after it run.

3. Look at the layers:

   docker history pantry:v1
   

   IMAGE          CREATED BY                                      SIZE
   a33fb03b2eb8   CMD ["python" "app.py"]                         0B
   <missing>      ENV PANTRY_VERSION=v1 PANTRY_ERROR_RATE=0       0B
   <missing>      COPY app.py . # buildkit                        12.3kB
   <missing>      RUN /bin/sh -c pip install --no-cache-dir -r…   18.7MB
   <missing>      COPY requirements.txt . # buildkit              12.3kB
   <missing>      WORKDIR /app                                    8.19kB
   ...
   

The libraries sit in one 19 MB cached layer. Your code is a few kilobytes on top, and the metadata lines cost nothing.

4. Run it. Pantry needs Redis, so start that first on a shared network (chapter 3 explains this line):

   docker network create pantry-net
   docker run -d --name redis --network pantry-net redis:7-alpine
   docker run -d --name pantry --network pantry-net -p 8000:8080 pantry:v1
   curl localhost:8000/
   

   Hello from Pantry (v1). You are visitor #1. Served by 84ba985b15bd.
   

The recipe works. Note -p 8000:8080: port 8000 on your laptop is the serving hatch, port 8080 inside the container is the kitchen door. They do not have to match.

Gotchas

• Process 1 in a container only receives SIGTERM if it installs a handler; shells do not, and neither does Flask's dev server, so docker stop pantry waits the full ten seconds. docker run --init puts a tiny init at PID 1 and the stop takes a quarter of a second; a real server such as gunicorn handles it itself.
• COPY . . before pip install means any code edit re-runs the install. Copy the dependency list first, install, then copy the code.
• COPY . . also ships everything in the directory, .git and data files included; a .dockerignore keeps stray files out of the context and the image.
• An ARG declared before FROM is not visible after it unless you declare it again.

Then and now

You will see MAINTAINER lines and ENV key value without an equals sign in older Dockerfiles; both were replaced by LABEL and ENV key=value years ago and still build, with warnings. Old Maps tells the story.

Recall

1. Why does the Dockerfile copy requirements.txt before app.py?
2. What is the difference between RUN and CMD?
3. Which form of CMD lets your program receive SIGTERM, and why?
4. What does the latest tag actually mean?
5. In the kitchen metaphor, what is a layer?

What Maya learned

• A Dockerfile is a recipe read top to bottom; docker build cooks it into a tagged image.
• Each instruction is a cached layer, so put slow, rarely-changing steps first and your code last.
• Tags are just labels; latest is the default label, not a promise of freshness.

…but

The customer's trial goes well, and one of their engineers asks where the visit counter is kept. Maya says "in Redis, in the container". The engineer asks what happens to the count when the container is removed. Maya opens a terminal to check and does not like the answer.

Go deeper

Docker manual 1 · Every Instruction · Docker manual 2 · ENTRYPOINT, CMD and PID 1 · Docker manual 3 · Context and Cache

Chapter 3 · Act I · 3 min readMemory and Neighbours#

The problem

Maya runs the experiment. Redis in a container, ten visits to Pantry, counter reads ten. She removes the Redis container, starts a fresh one from the same image, and the counter reads one. The data lived in the container's thin writable layer, and docker rm took the layer with it.

That is only half of the day's trouble. Pantry finds Redis by the hostname redis. On her laptop that works because both containers are on the network she created yesterday, but a colleague who copied the commands without the network line gets redis unreachable. And the customer's engineer wants to know why Pantry is on port 8000 on her machine but 8080 in the logs.

Two separate things have gone wrong: containers forget, and containers cannot see each other by default. Both are deliberate, and both have a proper fix.

The idea

Containers forget on purpose. The writable layer is meant to be disposable, so that a container can be replaced without ceremony. Anything that must outlive the container goes on a volume: a directory managed by Docker that lives outside any container and is mounted into one at a path. Remove the container, the volume stays. Start a new container with the same volume, the data is back.

There are two flavours. A named volume (-v pantry-data:/data) is created and stored by Docker; you rarely need to know where on disk. A bind mount (-v $(pwd):/app) mounts a directory from your laptop straight into the container. Bind mounts are for development: edit code on the host, see it change inside the container immediately. Named volumes are for data.

Remember it as: a volume is the pantry shelf; dishes get thrown away, the shelf stays.

Containers cannot see each other by default, and that is also on purpose. Each container gets its own network stack. On the default network, containers can only reach each other by IP address, which changes. On a user-defined network (docker network create pantry-net), Docker runs a small DNS server and every container can reach every other by its --name. That is why REDIS_HOST=redis works: redis is a hostname on pantry-net.

Ports are a separate idea. Inside its network, a container's port is just open; Pantry listens on 8080 and Redis on 6379, and other containers on the same network can connect to those directly. Nothing outside the network can, until you publish a port with -p host:container. Publishing punches a hole from your laptop's port 8000 through to the container's 8080.

Remember it as: a network is the corridor between kitchens, and a published port is a serving hatch to the street.

Redis is unpublished in this picture, and that is right: only Pantry needs it. Publish as little as you can.

The commands

# Volumes: create, list, inspect where it lives, remove
docker volume create pantry-data
docker volume ls
docker volume rm pantry-data

# Mount a named volume (data) or a host directory (dev) into a container
docker run -d --name redis -v pantry-data:/data redis:7-alpine redis-server --appendonly yes
docker run --rm -it -v "$(pwd)":/app -w /app python:3.12-slim python app.py

# Networks: create one so containers can find each other by name
docker network create pantry-net
docker network ls
docker run -d --name redis --network pantry-net redis:7-alpine

# Publish a container port on the host: -p host:container
docker run -d --name pantry --network pantry-net -p 8000:8080 pantry:v1

# Everything Docker knows about a container (IP, mounts, env, ports) as JSON
docker inspect pantry

# Live CPU and memory per container, and copying files in or out
docker stats --no-stream
docker cp pantry:/app/app.py ./app-from-container.py

Try it

1. Clean up yesterday's containers and start Redis on a named volume, with persistence turned on:

   docker rm -f pantry redis
   docker run -d --name redis --network pantry-net -v pantry-data:/data redis:7-alpine redis-server --appendonly yes
   docker run -d --name pantry --network pantry-net -p 8000:8080 pantry:v1
   curl localhost:8000/; curl localhost:8000/
   

You are visitor #1, then #2.

2. Destroy Redis and bring it back:

   docker rm -f redis
   docker run -d --name redis --network pantry-net -v pantry-data:/data redis:7-alpine redis-server --appendonly yes
   curl localhost:8000/
   

   Hello from Pantry (v1). You are visitor #3. Served by 84ba985b15bd.
   

The shelf survived the dish.

3. See the network doing its job:

   docker exec pantry python -c "import socket; print(socket.gethostbyname('redis'))"
   docker network inspect pantry-net --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}'
   

Pantry resolved redis to the IP the network assigned to the Redis container.

4. Try to reach Redis from your laptop:

   nc -zv localhost 6379
   

It refuses. Redis has no published port, and that is the correct state for a database.

Gotchas

• A bind mount path must be absolute: -v app.py:/x creates a named volume called app.py. Use "$(pwd)/app.py".
• On the default bridge network containers cannot resolve each other by name; only a network you created has DNS.
• localhost inside a container is the container. To reach your laptop, Docker Desktop offers host.docker.internal.
• A container running as a non-root USER may not be able to write to a bind-mounted directory owned by you; check ownership before blaming the app.

Then and now

You will see --link redis:redis and "data-only containers" in older guides; user-defined networks (Docker 1.9, 2015) and named volumes replaced both, and --link still works while everyone tells you not to use it. Old Maps tells the story.

Recall

1. Where does a named volume live, and what happens to it on docker rm?
2. Why does REDIS_HOST=redis resolve inside pantry-net but not on the default network?
3. What does -p 8000:8080 open, and which side is the container?
4. Which command shows a container's IP, mounts and environment as JSON?
5. In the kitchen metaphor, what is a published port?

What Maya learned

• Containers forget by design; put data on a named volume, and it survives the container being removed.
• Containers on a user-defined network can reach each other by container name, on their real ports.
• -p host:container opens a port to the outside world; publish only what the outside world needs.

…but

Pantry now survives restarts and finds its database by name. The customer signs. Their first request is not a feature: "Can you give us the image? We do not want to build it ourselves." Maya has an image on her laptop and no way to hand it over except, apparently, email.

Go deeper

Docker manual 7 · Networks · Docker manual 8 · Volumes

Chapter 4 · Act I · 3 min readShip It#

The problem

The customer wants the image, not the source. docker save can write an image to a tar file, and Maya's is 213 MB, which is on the wrong side of the email attachment limit and the wrong side of dignity.

Then she looks at the size again. Two hundred megabytes for fifty lines of Python? She opens docker history and finds most of it is the base image, plus the libraries. The customer's engineer, who has clearly done this before, asks two questions: "Where do you publish images?" and "Do you really need a compiler in production?"

Maya does not publish images anywhere, and she had not thought about compilers at all. Both questions turn out to be the same lesson: an image is a thing you ship, and shipping has rules.

The idea

Images live in a registry. Docker Hub is the public one, and the one you have been pulling from all along; redis:7-alpine is shorthand for docker.io/library/redis:7-alpine. Companies run private registries (GitHub Container Registry, AWS ECR, Google Artifact Registry, a self-hosted Harbor) that work exactly the same way. The verbs are push to upload and pull to download, and the image name tells Docker where to go.

A full image name has up to four parts: registry/namespace/repository:tag. Leave out the registry and Docker assumes Docker Hub. Leave out the tag and it assumes latest. To push to your own account on Docker Hub, the image must be named yourname/pantry:v1; that is what docker tag is for. Every push and pull is layer by layer, and layers already present on the other side are skipped, which is another reason to keep the slow, stable layers at the top of the Dockerfile.

Remember it as: a registry is the cookbook library; you push a recipe card in, anyone with access pulls it out.

The compiler question is about what ends up inside the image. Some Python packages need a C compiler to install. The lazy fix is to build from the full python:3.12 base image, which includes compilers, and then ship that: 1.6 GB, most of it tools nobody will run in production. The right fix is a multi-stage build: use a fat image to do the installing, then copy only the results into a slim final image. The Dockerfile has two FROM lines, and only the last stage is shipped.

FROM python:3.12 AS builder                     # stage 1: the workshop, 1.6 GB, has compilers
WORKDIR /app
COPY requirements.txt .
RUN python -m venv /venv && /venv/bin/pip install --no-cache-dir -r requirements.txt

FROM python:3.12-slim                           # stage 2: the shop floor, small
WORKDIR /app
COPY --from=builder /venv /venv                 # take only the finished goods
COPY app.py .
ENV PATH=/venv/bin:$PATH
USER nobody                                     # do not run as root
CMD ["python", "app.py"]

Pantry's multi-stage image is 217 MB. The builder stage alone is 1.63 GB. Nothing from the builder except the /venv directory made it into the image the customer gets. Smaller images pull faster, start faster, and contain fewer things an attacker could use. USER nobody is the other habit worth forming today: nothing in the image needs root, so do not give it root.

The commands

# Log in to a registry (Docker Hub by default; pass a hostname for others)
docker login
docker login ghcr.io

# Name the image for the registry, then push
docker tag pantry:v1 yourname/pantry:v1
docker push yourname/pantry:v1

# On another machine: pull and run
docker pull yourname/pantry:v1
docker run -d -p 8000:8080 yourname/pantry:v1

# Build a multi-stage Dockerfile (-f picks a file); --target stops at a stage
docker build -t pantry:slim -f Dockerfile.multistage .
docker build -t pantry:builder --target builder -f Dockerfile.multistage .

# Reclaim disk: remove stopped containers, unused networks, dangling images and build cache
docker system prune

Try it

1. Build both stages and compare:

   docker build -t pantry:slim -f Dockerfile.multistage .
   docker build -t pantry:builder --target builder -f Dockerfile.multistage .
   docker images pantry
   

   REPOSITORY   TAG       SIZE
   pantry       builder   1.63GB
   pantry       slim      217MB
   pantry       v1        213MB
   

The workshop is seven times bigger than what ships.

2. Run the slim image in place of v1 and confirm nothing changed for the user:

   docker rm -f pantry
   docker run -d --name pantry --network pantry-net -p 8000:8080 pantry:slim
   curl localhost:8000/
   docker exec pantry whoami
   

The greeting is the same, and the process now runs as nobody.

3. (Optional: needs a free Docker Hub account.) Publish it:

   docker login
   docker tag pantry:slim yourname/pantry:v1
   docker push yourname/pantry:v1
   

Then on any other machine, docker run -d -p 8000:8080 yourname/pantry:v1 is the whole installation guide. Everything in this book works without a registry account, because chapter 7 loads images straight into the local cluster.

4. See how much disk Docker is using and tidy up:

   docker system df
   docker system prune
   

Gotchas

• docker build produces an image for the machine it runs on: arm64 on an Apple Silicon Mac. Docker Desktop runs it anyway through emulation, so you notice only when an amd64 server says exec format error. Check with docker image inspect --format '{{.Architecture}}', and build with --platform linux/amd64 or multi-arch with buildx.
• docker push pantry:v1 is refused: without a registry and namespace it means docker.io/library/pantry, which is not yours. Tag it yourname/pantry:v1 first.
• Anonymous pulls from Docker Hub are rate-limited (since 2020); a CI server that pulls a lot without docker login starts failing at random.

Then and now

You will see older build output with Step 3/8 lines; that was the legacy builder, replaced by BuildKit as the default in Docker 23 (2023), whose output looks quite different. Old Maps tells the story.

Recall

1. What are the four parts of a full image name, and which two have defaults?
2. Why is the multi-stage image 217 MB when its builder stage is 1.6 GB?
3. Which command pushes an image, and what must its name include?
4. Why run as USER nobody?
5. In the kitchen metaphor, what is a registry?

What Maya learned

• An image is shipped by pushing it to a registry and pulling it somewhere else; the image name says which registry.
• A multi-stage build uses a fat image to build and a slim image to ship, so compilers never reach production.
• Run as a non-root user and prune regularly; small, clean images are faster and safer.

…but

The customer pulls the image and runs it. Twenty minutes later they call: "It says Redis is unreachable." Of course it does. Pantry is one container, but the app is two containers, a network and a volume, started in the right order with the right flags. Maya has those flags in her shell history. The customer does not.

Go deeper

Docker manual 4 · Multi-Stage Builds · Docker manual 5 · Users and Permissions · Docker manual 6 · Supply Chain and Secrets

Chapter 5 · Act I · 3 min readThe Whole Band#

The problem

Maya writes the customer a shell script: create the network, create the volume, start Redis with persistence, wait a moment, start Pantry with the network and the port and the environment variable. It works. Then she needs to change one flag, and realises she now maintains a shell script with six docker run invocations that nobody else understands, that cannot tell whether Redis is actually ready before Pantry starts, and that has no idea how to stop everything cleanly.

She counts the moving parts. Two containers. One network. One volume. One port. One environment variable. A startup order. This is not a container problem any more; it is a "several containers that belong together" problem, and she is solving it with the wrong tool.

The idea

Docker Compose describes a set of containers that run together as one application, in one YAML file, and gives you one command to start, stop and inspect the lot. Each container is a service. Compose creates a network for the project automatically, so services reach each other by service name, and it manages named volumes declared in the file.

Here is all of Pantry:

services:
  web:
    build: .                      # build from the Dockerfile here, or use image: yourname/pantry:v1
    ports:
      - "8000:8080"
    environment:
      REDIS_HOST: redis           # the service name is the hostname
    depends_on:
      redis:
        condition: service_healthy   # wait until redis's healthcheck passes
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/healthz')"]
      interval: 10s
  redis:
    image: redis:7-alpine
    command: ["redis-server", "--appendonly", "yes"]
    volumes:
      - pantry-data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s

volumes:
  pantry-data:

Two of those lines deserve attention. A healthcheck is a command Docker runs inside the container on a timer; if it fails repeatedly, the container is marked unhealthy. On its own that only changes what docker ps shows. Combined with depends_on and condition: service_healthy, it makes Compose wait for Redis to actually answer PONG before starting Pantry. "Started" and "ready" are different things, and this is the first time in the book that difference matters. It will not be the last.

Remember it as: Compose is the dinner menu; one file lists every dish, and one command cooks them all in the right order.

The file is committed next to the code. Anyone with the repository runs docker compose up and gets the same two containers, network and volume that Maya has, with none of the shell history. It is also the honest answer to "how do I run this locally?" for most services that need a database, a queue or a cache.

Compose is a single-machine tool. It knows nothing about a second host, and if a container dies it will restart it only if you asked with restart: unless-stopped. Keep that in mind.

The commands

# Start everything in the background, building images if needed
docker compose up -d
docker compose up -d --build

# What is running in this project, and is it healthy?
docker compose ps

# Logs from all services, or one; -f follows
docker compose logs -f
docker compose logs -f web

# Run a command inside a service's container
docker compose exec redis redis-cli GET visits

# Rebuild images without starting anything
docker compose build

# Print the effective file after defaults and variables are applied; old keys warn here
docker compose config

# Stop and remove containers and the network (volumes survive); -v removes volumes too
docker compose down
docker compose down -v

Try it

1. Tear down the hand-made containers and start the composed version:

   docker rm -f pantry redis
   docker compose up -d --build
   docker compose ps
   

   NAME             STATUS
   pantry-redis-1   Up 9 seconds (healthy)
   pantry-web-1     Up 3 seconds (health: starting)
   

Redis was healthy before web was even created. A few seconds later web is healthy too.

2. Use it, then stop and start it, and watch the counter:

   curl localhost:8000/; curl localhost:8000/
   docker compose down
   docker compose up -d
   curl localhost:8000/
   

   Hello from Pantry (v1). You are visitor #3. Served by 30ef7b1ee3b1.
   

down removed the containers and the network but kept the volume, so the counter survived. The hostname changed because it is a new container.

3. Peek at the data directly:

   docker compose exec redis redis-cli GET visits
   

4. Start again from nothing:

   docker compose down -v
   

The -v deletes the volume. The next up will greet visitor #1.

Gotchas

• A plain depends_on: [redis] orders start, not readiness: web comes up while Redis is still booting and the first requests fail. Only condition: service_healthy waits.
• docker compose down keeps named volumes on purpose; only down -v deletes them, and there is no confirmation.
• The project name, and so the container and volume names, comes from the directory name. Two checkouts in differently named folders are two separate projects.
• ports: - 8000:8080 unquoted is parsed as a number by YAML in some cases; always quote port mappings.

Then and now

You will see docker-compose with a hyphen, a version: "3" line at the top, and a file called docker-compose.yml; the Go plugin docker compose (2020) ignores version and prefers compose.yaml. Old Maps tells the story.

Recall

1. What does a service name become on the project network?
2. What is the difference between depends_on with and without condition: service_healthy?
3. Which command removes containers but keeps the data, and which removes both?
4. Where does Compose get the project name from?
5. In the kitchen metaphor, what is the Compose file?

What Maya learned

• Compose describes a multi-container app in one file and runs it with one command; service names are hostnames.
• Healthchecks plus depends_on: condition: service_healthy make startup order mean "ready", not just "started".
• down keeps volumes and down -v deletes them; know which one you are typing.

…but

The customer is happy. Pantry has real users now. Maya moves it from her laptop to a rented server, runs docker compose up -d there, and goes home. At 3 a.m. her phone rings.

Go deeper

Docker manual 9 · Compose: The File · Docker manual 10 · Compose: The Project

Chapter 6 · Act I · 4 min readThe Night It Fell Over#

The problem

Pantry is down. Maya logs in to the server at 3:10 a.m. and finds the web container exited two hours ago with a Python traceback from a dependency bug that shows up once in a few thousand requests. Nothing restarted it, because nothing was told to. She adds restart: unless-stopped to the Compose file, brings it up, and goes back to bed feeling that she has fixed it.

At 9 a.m. the customer's launch email goes out and traffic triples. One Pantry container on one server cannot keep up; requests queue, then time out. Maya can run three copies of web on the same server, but they would fight over port 8000, and the server has only so much CPU anyway. She rents a second server. Now she has two Compose files on two machines, no way to send traffic to both, and no way to know when one of them dies.

At 11 a.m. she ships a fix for the traceback. docker compose up -d --build stops the old container and starts the new one. For eight seconds, Pantry is down again, on purpose, in the middle of the morning rush.

Nothing here is a bug in Pantry. Every one of these problems is a thing a single Docker host, however well configured, simply does not do.

The idea

Maya sits down with a coffee and writes the list of what she has and what she needs.

The pattern in the right-hand column is that every item is about many copies on many machines, and about something watching them. A restart policy is a bandage on one container on one host. What Maya needs is a system whose whole job is to be told "I want three copies of this image, reachable at one address, always" and to make that true, keep it true when servers die, and change it gracefully when she ships a new version.

That system is an orchestrator, and the one the industry settled on is Kubernetes. It takes a description of what you want (the desired state) and runs a loop, forever, comparing it to what exists (the actual state) and fixing the difference. Container died? The loop notices and starts another. Server died? The loop moves its containers elsewhere. New image? The loop replaces the copies one at a time and stops if the new ones fail to come up.

Remember it as: Docker cooks one meal in one kitchen; Kubernetes runs the whole city and keeps every kitchen open.

It is worth saying plainly that Kubernetes does not replace Docker. Everything from the first five chapters carries over untouched. The image Maya built is the image Kubernetes will run. The Dockerfile, the registry, the idea that containers forget and need volumes, the idea that "started" is not "ready": all of it is the foundation. Kubernetes adds a control plane, a vocabulary for describing what you want, and the loop.

The cost is real. Kubernetes has more concepts than Docker, and its YAML can be intimidating the first time. The next nine chapters introduce those concepts one at a time, each because Maya needs it, and each with a picture. By chapter 15 she will be able to diagnose a 3 a.m. outage in five minutes with four commands.

The commands

The one thing a single host can offer, so you know what it does and does not solve:

# Restart the container if it exits, unless you stopped it yourself (Compose: restart: unless-stopped)
docker run -d --restart unless-stopped --name pantry -p 8000:8080 pantry:v1

# Make the app crash from the inside (docker kill counts as "you stopped it" and is NOT restarted)
curl localhost:8000/crash

# Update means stop, then start: count the seconds of downtime
docker compose up -d --build

# What Docker saw: the container died and was started again
docker events --since 30s --until 1s --filter container=pantry --format '{{.Action}}'

Try it

1. Start Pantry with a restart policy, then kill it and watch Docker bring it back:

   docker compose down -v
   docker network create pantry-net 2>/dev/null
   docker run -d --name redis --network pantry-net redis:7-alpine
   docker run -d --restart unless-stopped --name pantry --network pantry-net -p 8000:8080 pantry:v1
   sleep 2; curl localhost:8000/crash; sleep 3
   docker ps --filter name=pantry --format '{{.Names}} {{.Status}}'
   docker events --since 30s --until 1s --filter container=pantry --format '{{.Action}}'
   

   pantry Up 3 seconds
   create
   start
   die
   start
   

Pantry has a /crash endpoint that makes the process exit, which is the honest way to simulate a bug. (docker kill would not trigger a restart: Docker treats that as you stopping it on purpose.)

That handles the 3 a.m. crash. Now try the 9 a.m. problem.

2. Try to run a second copy on the same port:

   docker run -d --name pantry2 --network pantry-net -p 8000:8080 pantry:v1
   

   docker: Error response from daemon: ... Bind for 0.0.0.0:8000 failed: port is already allocated.
   

One port, one container. There is no built-in way to say "spread across these" without adding a load balancer by hand.

3. Measure the 11 a.m. problem:

   docker rm -f pantry pantry2
   docker run -d --restart unless-stopped --name pantry --network pantry-net -p 8000:8080 pantry:v1
   ( for i in $(seq 1 12); do curl -s -o /dev/null -m 1 -w '%{http_code}\n' localhost:8000/ || echo down; sleep 0.5; done ) &
   sleep 1; docker rm -f pantry && docker run -d --name pantry --network pantry-net -p 8000:8080 pantry:v2
   wait
   

   200
   200
   000
   down
   000
   down
   ...
   200
   500
   

Every down is half a second with no Pantry at all, about four seconds in total. That is downtime you chose. (The 500 at the end is v2 being deliberately flaky, which comes in useful later.)

4. Clean up before the city:

   docker rm -f pantry redis
   docker network rm pantry-net
   

Gotchas

• docker stop and docker kill both count as "you stopped it"; a container with --restart unless-stopped stays down afterwards. Only a crash from inside triggers the restart.
• A restart policy restarts a container that exits; it does nothing for one that hangs while its process stays alive. That needs a health check, which Docker records but does not act on.
• The eight seconds of downtime during compose up -d --build is not a bug to fix in Compose; it is the absence of a rolling update.

Then and now

You will see docker stack deploy and Compose files with a deploy: section in repositories from 2017 to 2019; that was Docker Swarm, the orchestrator that lost to Kubernetes, and Compose ignores deploy: today. Old Maps tells the story.

Recall

1. Name the three things a single Docker host cannot do that the 3 a.m. story exposed.
2. Why did docker kill not trigger the restart policy while /crash did?
3. What is "desired state", and what does the orchestrator do with it?
4. Does Kubernetes replace the images and Dockerfiles from chapters 1 to 5?
5. In the metaphor, what does Docker cook and what does Kubernetes run?

What Maya learned

• A restart policy heals one container on one host; it cannot scale, spread traffic, or roll out without downtime.
• Every 3 a.m. problem is a "many copies on many machines, watched continuously" problem.
• Kubernetes is that watcher: you declare the desired state and a loop keeps reality matching it, using the same images.

…but

Raj, who joined last month from a company that ran hundreds of services, listens to Maya's list and nods. "You need a cluster," he says. "Let me show you the city." He opens a terminal and types kind create cluster.

Go deeper

Docker manual 2 · ENTRYPOINT, CMD and PID 1 · Docker manual 5 · Users and Permissions

Chapter 7 · Act II · 4 min readMeet the City#

The problem

Raj's terminal says kind create cluster, and ninety seconds later Maya is looking at two Docker containers pretending to be two servers. "That's a cluster," Raj says. "Same thing the cloud gives you, just smaller and on your laptop."

Maya's questions come fast. If it is all containers, what is running Pantry? Nothing yet. Where do the requests go? Nowhere yet. What is the kubectl he keeps typing, and why does it talk to the cluster and not a server? And when he says "the control plane decides where things run", who exactly is deciding?

Raj does what good ops people do. Before running anything, he draws the map. Kubernetes has a lot of parts, but they fall into two groups, and once Maya can name them she will be able to read every kubectl output for the rest of the book.

The idea

A cluster is a set of machines, called nodes, managed as one. Some nodes run your containers; these are the worker nodes. One or more nodes run the management software; together they are the control plane. In a cloud, the provider runs the control plane for you and you only see workers. In kind, both are Docker containers on your laptop.

Think of the cluster as a city. Each node is a building. The control plane is city hall, and city hall has four departments worth knowing:

• The API server is the front desk. Every request, from you, from kubectl, from every other component, goes through it. Nobody talks to anything else directly.
• etcd is the records room. It is a small key-value database and it is the only place cluster state is stored. Everything else can be restarted and rebuilt from the records.
• The scheduler is the housing office. When a new Pod needs a home, the scheduler picks a node with enough room and writes that decision back to the records.
• The controller manager runs the inspectors. Each controller watches one kind of record ("there should be three copies of this") and takes action when reality drifts.

Every building has a kubelet, the caretaker. It watches the records for Pods assigned to its node, asks the container runtime to start them, reports their health, and restarts them if they die. The kubelet is the only part of Kubernetes that actually runs containers.

Remember it as: the cluster is a city, nodes are buildings, city hall keeps the records and everyone else watches the records and does their part.

The important word in the diagram is watches. Nothing in Kubernetes commands anything else. You write what you want into the records via the front desk, and each department notices and acts. That is what makes it self-healing: the inspectors never stop looking.

kubectl is the front-desk phone. It reads a config file (~/.kube/config) that can hold several clusters, and the currently selected one is called the context. Most of the "why is this not working" moments in a new team come from being pointed at the wrong context. The three commands for that are in the list below; learn them first.

Two kubectl verbs will carry you through the whole book. get lists things, and -o wide shows more columns while -o yaml shows everything. describe tells the story of one thing, including the events the departments logged about it, which is where the answers are when something is wrong. A third verb, explain, is the built-in manual for any field of any resource, and it means you never have to guess what a YAML key does.

The commands

# Create a local cluster from a config file (two nodes, ports mapped); delete it when done
kind create cluster --config examples/kind-config.yaml
kind delete cluster --name pantry

# Which cluster am I talking to? Switch if it is the wrong one
kubectl config current-context
kubectl config get-contexts
kubectl config use-context kind-pantry

# Where is the front desk, and is it answering?
kubectl cluster-info

# The buildings, with more columns
kubectl get nodes -o wide

# Everything running in every namespace (the control plane is just Pods too)
kubectl get pods -A

# The story of one thing, events included
kubectl describe node pantry-worker

# The built-in manual for any resource or field
kubectl explain pod.spec.containers

# Copy an image from your laptop into the cluster's nodes (no registry needed)
kind load docker-image pantry:v1 pantry:v2 --name pantry

Try it

1. Create the cluster (this is the config the whole book uses):

   cd examples
   kind create cluster --config kind-config.yaml
   kubectl config current-context
   

   kind-pantry
   

2. Look at the buildings:

   kubectl get nodes -o wide
   

   NAME                   STATUS   ROLES           AGE   VERSION   INTERNAL-IP   OS-IMAGE
   pantry-control-plane   Ready    control-plane   60s   v1.37.0   172.20.0.3    Debian GNU/Linux 13
   pantry-worker          Ready    <none>          45s   v1.37.0   172.20.0.2    Debian GNU/Linux 13
   

Run docker ps in another terminal: the two nodes are two containers.

3. Look at city hall. It runs as Pods, in a namespace called kube-system:

   kubectl get pods -A
   

You will see kube-apiserver-..., etcd-..., kube-scheduler-..., kube-controller-manager-... and a kube-proxy and kindnet per node. Every box in the diagram is on that list.

4. Read the manual for something you will write tomorrow:

   kubectl explain pod.spec.containers.image
   

5. Load Pantry's images into the cluster so no registry is needed:

   cd pantry
   docker build -t pantry:v1 .
   docker build -t pantry:v2 --build-arg PANTRY_VERSION=v2 --build-arg PANTRY_ERROR_RATE=0.3 .
   kind load docker-image pantry:v1 pantry:v2 --name pantry
   

The images are now on both nodes' disks. On a real cluster this step is docker push to a registry the nodes can pull from.

Gotchas

• An image tagged :latest, or with no tag, gets imagePullPolicy: Always, so the kubelet ignores the copy you loaded into kind and tries Docker Hub: ErrImagePull. Use a real tag like v1, or set imagePullPolicy: IfNotPresent.
• kubectl talks to whichever context is current, not to the cluster you are thinking about. After creating or switching clusters, run kubectl config current-context before anything else.
• kind load docker-image copies the image onto the nodes that exist now; a node added later, or a recreated cluster, needs it loaded again.

Then and now

You will see the control-plane node called master in older manifests and tutorials; the label went in Kubernetes 1.24 and the taint in 1.25. Old Maps tells the story, and why "Kubernetes dropped Docker" changed nothing.

Recall

1. Name the four control-plane departments and one sentence on what each does.
2. Which component actually starts containers on a node?
3. Which command tells you which cluster kubectl is pointed at?
4. What does "everything watches the API server" buy you?
5. In the city metaphor, what is a node and what is etcd?

What Maya learned

• A cluster is nodes plus a control plane; the control plane stores desired state in etcd and everything else watches it and acts.
• The kubelet on each node is the only thing that runs containers; the scheduler only decides where.
• kubectl config current-context, get, describe and explain are the first four commands to reach for in any cluster.

…but

The city is built and the images are inside it, but nothing is running. Raj hands Maya a YAML file with eleven lines in it and says, "That is the smallest thing that runs."

Go deeper

Kubernetes manual 1 · Reading Any Manifest

Chapter 8 · Act II · 4 min readThe Smallest Thing That Runs#

The problem

Maya wants to run one copy of Pantry in the cluster, the way she ran one container in chapter 1, and see it answer a request. That is all. Raj's YAML file is eleven lines, and every one of them looks like it means something.

She tries to skip the file. Surely there is a kubectl run pantry:v1 that just works? There is, sort of, and Raj lets her try it. It works. Then he asks her to run it again on Tuesday with a different environment variable, and on Wednesday to explain to a colleague exactly what she ran on Tuesday. Shell history is not a system. The file is the system.

The second surprise is the word Kubernetes uses. She asked for a container. Kubernetes gave her a Pod.

The idea

A Pod is the smallest thing Kubernetes runs. It is one or more containers that share a network address and can share storage, scheduled together onto the same node, living and dying together. Nearly every Pod you write will have exactly one container in it. The reason the concept exists is the "nearly": sometimes a program needs a helper right beside it, sharing its localhost, and Act III has a famous example.

Think of a Pod as an apartment. The containers inside share the address and the kitchen. Pantry lives alone in its apartment. Later a concierge will move in.

Remember it as: a Pod is an apartment: one address, usually one tenant, sometimes a helper in the spare room.

Here is the file, with the four parts every Kubernetes resource has:

apiVersion: v1          # which version of the API this kind of thing lives in
kind: Pod               # what it is
metadata:               # name, labels, namespace: how it is identified
  name: pantry
  labels:
    app: pantry
spec:                   # what you want: the desired state
  containers:
    - name: web
      image: pantry:v1
      ports:
        - containerPort: 8080
      env:
        - name: REDIS_HOST
          value: redis

apiVersion, kind, metadata, spec. Every Deployment, Service, ConfigMap and Ingress in this book has those same four top-level keys, and the only one that changes shape is spec. When Kubernetes accepts the file it adds a fifth section, status, which is what it observed. Desired state is yours; observed state is the cluster's; the gap between them is what the controllers work on.

The file goes in with kubectl apply -f. apply is idempotent: run it again with no changes and nothing happens, change one line and only that changes. It is the verb you will use most, and it is why the file is the system.

Nobody writes these files from a blank page. The honest way is to ask kubectl for a skeleton and edit it: kubectl create deployment pantry --image=pantry:v1 --dry-run=client -o yaml > deployment.yaml. --dry-run=client means "build the object but do not send it", and -o yaml prints it. The same trick works for create service, create configmap and most other kinds, and it is how experienced people produce YAML that is correct on the first try.

One more resident of the apartment is worth knowing now. An init container runs to completion before the main container starts, and the Pod stays Init:0/1 until it exits successfully. It is the helper who arrives before the tenant moves in: waiting for Redis to answer, fetching a config file, running a migration. k8s/pod-init.yaml waits for Redis with a two-line shell loop, which is the most common init container in the world.

A Pod has a life cycle, and its phase is the first thing to look at when something is off:

Pending means "not yet placed or not yet started" and almost always means either no node has room or the image cannot be pulled. Running with a climbing RESTARTS count means the program keeps dying. Chapter 15 turns this diagram into a checklist.

One more thing you will do constantly: reach a Pod from your laptop without exposing it to anyone. kubectl port-forward opens a tunnel from a local port to a port in the Pod, through the API server. It is the Kubernetes equivalent of -p 8000:8080 and it is for you, not for users.

The commands

# Send a file to the front desk; apply is idempotent, run it as often as you like
kubectl apply -f k8s/pod.yaml

# List Pods; -o wide adds IP and node, -w watches for changes
kubectl get pods
kubectl get pods -o wide
kubectl get pods -w

# The story of one Pod, events at the bottom
kubectl describe pod pantry

# Everything the cluster knows about it, including status
kubectl get pod pantry -o yaml

# Logs (-f follows, -c picks a container in a multi-container Pod)
kubectl logs -f pantry

# Run a command inside; get a shell with -- sh
kubectl exec -it pantry -- sh

# Tunnel local port 8000 to the Pod's 8080 (Ctrl-C to stop)
kubectl port-forward pod/pantry 8000:8080

# A quick throwaway Pod for poking around the network, deleted on exit
kubectl run curl --rm -it --image=curlimages/curl:8.11.1 -- sh

# Generate a correct skeleton instead of typing YAML from memory
kubectl create deployment pantry --image=pantry:v1 --dry-run=client -o yaml

# Delete by file or by name
kubectl delete -f k8s/pod.yaml
kubectl delete pod pantry

Try it

1. Pantry needs Redis. Apply Raj's Redis file first (it is a Deployment and a Service, both explained in the next two chapters; for now it is "a Redis called redis"), then the Pod:

   cd examples/pantry
   kubectl apply -f k8s/redis.yaml
   kubectl apply -f k8s/pod.yaml
   kubectl get pods -o wide
   

   NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE
   pantry                   1/1     Running   0          3s    10.244.1.3   pantry-worker
   redis-5d9f8c7b6d-x2k9p   1/1     Running   0          8s    10.244.1.2   pantry-worker
   

The scheduler put both on the worker; the control-plane node is reserved for city hall.

2. Talk to it through a tunnel:

   kubectl port-forward pod/pantry 8000:8080 &
   curl localhost:8000/
   kill %1
   

   Hello from Pantry (v1). You are visitor #1. Served by pantry.
   

The hostname is the Pod name. Pantry is running in the city.

3. Read its story:

   kubectl describe pod pantry | tail -8
   

The Events section shows Scheduled, Pulled, Created, Started: one line per department that touched it.

4. Delete it and watch what happens:

   kubectl delete pod pantry
   kubectl get pods
   

It is gone, and nothing brings it back. A bare Pod has no inspector watching it. That is the problem for the next chapter.

Then and now

You will see tutorials where kubectl run nginx --image=nginx creates a Deployment; it did until Kubernetes 1.18 (2020) removed the generators, and since then it creates only a Pod. Old Maps tells the story.

Recall

1. What are the four top-level keys of every Kubernetes resource, and which one the cluster adds?
2. Why does kubectl apply -f twice with no changes do nothing?
3. How do you generate a Deployment manifest without writing it by hand?
4. What does an init container guarantee about the main container?
5. In the city metaphor, what is a Pod?

What Maya learned

• A Pod is the unit Kubernetes runs: one or more containers sharing an address, almost always one.
• Every resource is apiVersion, kind, metadata, spec; apply -f makes the file the source of truth.
• get, describe, logs, exec and port-forward are the Pod toolkit; describe shows the events that explain problems.

…but

Maya deletes the Pod and it stays deleted. She kills the process inside it and the kubelet restarts the container, which is nice, but if the node dies the Pod dies with it and nobody notices. This is exactly the 3 a.m. problem, and a Pod on its own does not solve it. "Correct," says Raj. "Pods are not supposed to. You never run a Pod by itself. You ask someone to run it for you."

Go deeper

Kubernetes manual 1 · Reading Any Manifest · Kubernetes manual 2 · The Pod Spec · The Bridge

Chapter 9 · Act II · 4 min readSomeone Who Never Sleeps#

The problem

Maya's list from chapter 6 is on the whiteboard: self-healing, scaling, safe rollout. A single Pod delivers none of them. Delete it and it is gone. Ask for three and you write three files. Ship a new version and you delete the old Pod and create a new one, which is the eight seconds of downtime she was trying to escape.

"Who watches the Pods?" she asks. Raj writes one word on the whiteboard: Deployment. "You don't run Pods. You tell a Deployment how many Pods you want and what they should look like. It never sleeps."

The idea

A Deployment is a record that says: keep N copies of this Pod template running, and when the template changes, move from the old copies to the new ones gradually. A controller in city hall watches every Deployment and makes it true.

It does the job through a middle manager. The Deployment creates a ReplicaSet, whose only responsibility is "exactly N Pods matching this label selector must exist". If a Pod dies or a node vanishes, the ReplicaSet creates a replacement. When you change the Deployment's Pod template (a new image, say), the Deployment creates a second ReplicaSet for the new template and shifts the count from old to new, one Pod at a time. The old ReplicaSet is kept at zero so you can roll back.

The dotted lines are the important part. The ReplicaSet does not "own" Pods by holding a list of them. It selects them by labels, and labels are just key-value tags in a resource's metadata. Every Pod the Deployment creates gets app: pantry plus a hash of the template; the ReplicaSet counts Pods with those labels. Labels and selectors are how everything in Kubernetes finds everything else, and the next chapter uses the same trick for traffic.

Think of the Deployment as the building manager who guarantees N identical apartments are always occupied, and labels as the name tags on the doors the manager checks against a list.

Remember it as: a Deployment is the building manager who never sleeps: N tenants, always, and new tenants move in one at a time.

Here is the Deployment's spec, which wraps the Pod from the last chapter:

spec:
  replicas: 3
  selector:
    matchLabels:
      app: pantry            # must match the template's labels
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1            # may run one extra Pod during a rollout
      maxUnavailable: 0      # may never drop below 3 ready Pods
  template:                  # the Pod, minus apiVersion/kind/name
    metadata:
      labels:
        app: pantry
    spec:
      containers:
        - name: web
          image: pantry:v1

A rollout with those two numbers looks like this over time:

At no point are fewer than three Pods ready, so users never notice. If the new Pods fail to become ready, the rollout stops with the old ones still serving, and rollout undo shifts the counts back the other way. That is the 11 a.m. problem solved by arithmetic.

The commands

# Create or update the Deployment from the file (the normal way to change anything)
kubectl apply -f k8s/deployment.yaml

# The three layers, in one listing (-l filters by label)
kubectl get deploy,rs,pods -l app=pantry

# Change the number of copies now (the file still says 3; apply would put it back)
kubectl scale deploy/pantry --replicas=5

# Change the image without editing the file (handy in scripts and demos)
kubectl set image deploy/pantry web=pantry:v2

# Watch a rollout finish, see the history, go back one revision
kubectl rollout status deploy/pantry
kubectl rollout history deploy/pantry
kubectl rollout undo deploy/pantry

# Restart every Pod gracefully (new Pods first): picks up new ConfigMaps, clears bad state
kubectl rollout restart deploy/pantry

# Show labels, or add one
kubectl get pods --show-labels
kubectl label pod <name> tier=web

Try it

1. Apply the Deployment and look at all three layers:

   kubectl apply -f k8s/deployment.yaml
   kubectl rollout status deploy/pantry
   kubectl get deploy,rs,pods -l app=pantry
   

   NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/pantry   3/3     3            3           2s

   NAME                                DESIRED   CURRENT   READY   AGE
   replicaset.apps/pantry-647759c9cb   3         3         3       2s

   NAME                          READY   STATUS    RESTARTS   AGE
   pod/pantry-647759c9cb-64rhd   1/1     Running   0          2s
   pod/pantry-647759c9cb-7cfk6   1/1     Running   0          2s
   pod/pantry-647759c9cb-wtwht   1/1     Running   0          2s
   

Pod names are <deployment>-<template hash>-<random>. The hash is the label the ReplicaSet selects on.

2. Kill one and watch the manager replace it:

   kubectl delete pod $(kubectl get pod -l app=pantry -o jsonpath='{.items[0].metadata.name}')
   kubectl get pods -l app=pantry
   

There are still three, and one of them is a few seconds old. The 3 a.m. problem is solved.

3. Roll out v2 and watch it happen:

   kubectl set image deploy/pantry web=pantry:v2
   kubectl rollout status deploy/pantry
   

   Waiting for deployment "pantry" rollout to finish: 1 out of 3 new replicas have been updated...
   Waiting for deployment "pantry" rollout to finish: 2 out of 3 new replicas have been updated...
   Waiting for deployment "pantry" rollout to finish: 1 old replicas are pending termination...
   deployment "pantry" successfully rolled out
   

4. v2 is flaky (it returns errors on purpose). Go back:

   kubectl rollout history deploy/pantry
   kubectl rollout undo deploy/pantry
   kubectl get pods -l app=pantry -o jsonpath='{.items[*].spec.containers[0].image}'
   

   pantry:v1 pantry:v1 pantry:v1
   

Both ReplicaSets still exist: kubectl get rs shows one at 3 and one at 0. The undo just moved the numbers.

Gotchas

• A Deployment's selector cannot be changed after creation; kubectl apply fails with "field is immutable". Delete and recreate, or leave the selector alone and change labels elsewhere.
• Without a readiness probe, a Pod counts as available the instant its container starts, so a rolling update can finish while the new version is still booting. Chapter 12 fixes this.
• kubectl scale and set image change the cluster, not the file; the next apply -f puts the file's values back. Edit the file, or accept that you are experimenting.
• Template labels must match the selector or the Deployment is rejected; a typo in one of them is the usual cause of "selector does not match template labels".

Then and now

You will see kind: ReplicationController and Deployments under apiVersion: extensions/v1beta1 in older manifests; the address stopped working in Kubernetes 1.16 (2019) and apps/v1 made the selector mandatory. Old Maps tells the story.

Recall

1. What does a ReplicaSet do that a Deployment does not, and the other way round?
2. How does a ReplicaSet know which Pods are "its" Pods?
3. Which two numbers make a rollout zero-downtime, and what did we set them to?
4. Which command goes back one revision?
5. In the city metaphor, what is a Deployment?

What Maya learned

• Never run bare Pods; a Deployment keeps N copies alive and replaces any that die, on any node.
• A Deployment manages ReplicaSets, which select Pods by label; changing the template creates a new ReplicaSet and shifts Pods across gradually.
• scale, set image, rollout status, rollout undo and rollout restart are the day-to-day Deployment verbs.

…but

Three copies of Pantry are running, and each has its own Pod IP. Which one should a user talk to? Maya port-forwards to one of them, deletes that Pod to prove the manager works, and her tunnel dies with it. The new Pod has a new IP. The manager keeps the apartments full; it does not give anyone a phone number.

Go deeper

Kubernetes manual 4 · Lifecycle and Rollouts

Chapter 10 · Act II · 4 min readA Number That Never Changes#

The problem

Three Pantry Pods, three IP addresses, and every one of them temporary. Maya's port-forward died when its Pod did. Redis has the same problem from the other side: Pantry finds it by the name redis, and in chapter 8 that worked, but she never asked why. If the Redis Pod is replaced, its IP changes, and something must be keeping the name pointed at the right place.

Then the customer asks for a URL. Not a tunnel, not a port on a node, a proper https://pantry.example.com that their users can type. Nothing Maya has so far answers to a hostname.

Two questions, then: how do Pods find each other when everything moves, and how does the outside world find the Pods at all?

The idea

A Service is a stable name and IP for a group of Pods, chosen by label selector. Traffic sent to the Service is spread across whichever Pods currently match. Pods come and go; the Service does not. Every Service also gets a DNS name inside the cluster, <name>.<namespace>.svc.cluster.local, and from the same namespace just <name> works. That is why REDIS_HOST=redis has been working: redis is a Service.

Think of it as a phone number for a group of apartments. Tenants move in and out; the number stays on the sign. The selector is the list of doors the number rings.

Remember it as: a Service is a phone number for a group of Pods that never changes when the Pods do.

Services come in three types, and the difference is only who can dial the number:

NodePort and LoadBalancer build on ClusterIP; each type is the one below plus a door. On a laptop, kind maps a NodePort to localhost for you. On a cloud, LoadBalancer costs money per Service, which is why nobody exposes twenty Services that way.

They do the job for "find each other". They do not answer to a hostname. For that there is Ingress: a set of rules saying "requests for host X and path Y go to Service Z". An Ingress is only a record; something has to read it and act, and that something is an Ingress controller, a reverse proxy (usually NGINX) running in the cluster, exposed once, routing to every Service by hostname. One front gate, many signposts.

Remember it as: Ingress is the city's front gate with a signpost: one entrance, routed by hostname and path.

The metaphor stretches in one place, so here it is plainly: an Ingress can also terminate TLS (the certificate lives in a Secret named in the Ingress) and rewrite paths. Those are proxy features, not gates, and the controller's documentation covers them.

The commands

# Create the Service; get shows type, cluster IP and ports
kubectl apply -f k8s/service.yaml
kubectl get svc

# Which Pod IPs is the Service currently pointing at? (empty = your selector is wrong)
kubectl get endpoints pantry

# Tunnel to the Service instead of a Pod: survives Pod restarts
kubectl port-forward svc/pantry 8000:80

# Test DNS and routing from inside the cluster with a throwaway Pod
kubectl run curl --rm -it --image=curlimages/curl:8.11.1 -- sh -c 'curl -s http://pantry/; nslookup redis'

# Install an Ingress controller (kind flavour: listens on the node's port 80)
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml

# Create the Ingress and see which host it answers to
kubectl apply -f k8s/ingress.yaml
kubectl get ingress
kubectl describe ingress pantry

Try it

1. Create the Service and look at its endpoints:

   kubectl apply -f k8s/service.yaml
   kubectl get svc pantry
   kubectl get endpoints pantry
   

   NAME     TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
   pantry   ClusterIP   10.96.80.31   <none>        80/TCP    2s
   NAME     ENDPOINTS                                            AGE
   pantry   10.244.1.4:8080,10.244.1.5:8080,10.244.1.6:8080      2s
   

Three Pod IPs behind one cluster IP. Delete a Pod and run the endpoints command again: one IP has changed, the Service has not.

2. Call it from inside the city, by name:

   kubectl run curl --rm -it --restart=Never --image=curlimages/curl:8.11.1 -- -s http://pantry/
   

   Hello from Pantry (v1). You are visitor #1. Served by pantry-647759c9cb-xthw5.
   

Run it a few times: the Pod name at the end changes as the Service spreads the load.

3. Open a door on the node. The book's kind config maps node port 30080 to your laptop:

   kubectl apply -f k8s/service-nodeport.yaml
   curl localhost:30080/
   kubectl delete -f k8s/service-nodeport.yaml
   

4. Install the Ingress controller, wait for it, and add the front gate:

   kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
   kubectl wait -n ingress-nginx --for=condition=Ready pod -l app.kubernetes.io/component=controller --timeout=180s
   kubectl apply -f k8s/ingress.yaml
   curl -H 'Host: pantry.localtest.me' localhost/
   curl -o /dev/null -w '%{http_code}\n' localhost/
   

   Hello from Pantry (v1). You are visitor #4. Served by pantry-647759c9cb-g6hp9.
   404
   

With the right hostname you reach Pantry; without it the gate has no signpost and says 404. (*.localtest.me resolves to 127.0.0.1 on most networks, in which case plain curl pantry.localtest.me/ works too; some home routers block that, so the book uses the Host header.)

Gotchas

• Three ports, three meanings: port is the Service's own, targetPort is the container's, nodePort is the door on the node. Mixing the first two is the most common Service bug.
• A Service whose selector matches nothing is created happily and routes nowhere; kubectl get endpoints showing <none> is the only tell.
• From another namespace, redis does not resolve; redis.default or redis.default.svc does.
• An Ingress without an ingressClassName, on a cluster with no default class, is accepted and ignored by every controller: no ADDRESS in kubectl get ingress, and 503 from the gate.

Then and now

You will see the kubernetes.io/ingress.class annotation and networking.k8s.io/v1beta1 Ingresses; v1 arrived in 1.19 and the old address went in 1.22 (2021), and since 1.33 get endpoints warns about EndpointSlices. Old Maps tells the story.

Recall

1. What is a Service, in one sentence, and what makes its address stable?
2. Which command shows the Pod IPs behind a Service, and what does an empty result mean?
3. Who can reach a ClusterIP, a NodePort and a LoadBalancer Service?
4. What reads an Ingress resource, and what happens without it?
5. In the city metaphor, what are the Service and the Ingress?

What Maya learned

• A Service is a stable name, IP and DNS entry for whichever Pods match its selector; check endpoints when it seems broken.
• ClusterIP is for inside, NodePort opens a fixed port on every node, LoadBalancer asks the cloud for a public IP.
• Ingress routes by hostname and path through one controller, so a whole cluster needs one public entrance, not one per Service.

…but

Pantry has a URL. The customer's engineer opens the deployment YAML to review it and finds REDIS_HOST: redis hard-coded in the Pod template, and the greeting hard-coded in the image. "How do we change these for staging?" she asks. "Rebuild the image?" Maya opens her mouth and closes it again.

Go deeper

Kubernetes manual 8 · Networking

Chapter 11 · Act II · 4 min readSticky Notes and the Locked Drawer#

The problem

Staging needs a different greeting, a different Redis, and soon a different API token for the payment provider. Maya's options so far are to bake values into the image (one image per environment, which defeats the point of images) or to write them into the Deployment's Pod template (one Deployment file per environment, all nearly identical, drifting apart). Neither is right.

The token is worse. It cannot go in the image, because images get pushed to registries and pulled by whoever has access. It cannot go in the Deployment file, because the file is committed to Git. It has to be somewhere, and that somewhere has to be readable by the Pod and by nobody else.

Raj's answer is two resources that look almost identical and are treated very differently.

The idea

A ConfigMap is a named bag of key-value pairs that lives in the cluster and is injected into Pods at start-up, either as environment variables or as files. The Deployment stays the same in every environment; only the ConfigMap changes. Configuration has moved out of the image and out of the Pod template into a thing you can apply on its own.

A Secret is the same bag with a different label on it. The values are base64-encoded (that is encoding, not encryption; anyone with get secret rights can read them), and the difference is in how the cluster handles them: Secrets can be encrypted at rest in etcd, RBAC lets you grant access to ConfigMaps without granting access to Secrets, and tooling knows not to print them. Put anything you would not paste into a public chat in a Secret.

Remember it as: a ConfigMap is sticky notes on the fridge, a Secret is the locked drawer; both are read by the Pod, only one is meant to be seen.

There are two ways in, and they behave differently. Environment variables are simple and are what most apps read, but they are fixed when the container starts; change the ConfigMap and the Pod keeps its old values until it is restarted (kubectl rollout restart is for exactly this). Mounted files update in place a minute or so after the ConfigMap changes, which suits config files a program re-reads. Pantry uses both: settings as env vars, a message-of-the-day as a file.

Here is the piece of the Deployment that does the wiring; the ConfigMap and Secret themselves are three-line files:

containers:
  - name: web
    image: pantry:v1
    envFrom:
      - configMapRef: { name: pantry-config }    # every key becomes an env var
      - secretRef:    { name: pantry-secret }
    volumeMounts:
      - name: notes
        mountPath: /etc/pantry
        readOnly: true
volumes:
  - name: notes
    configMap:
      name: pantry-config
      items:
        - key: motd.txt
          path: motd.txt                          # one key becomes one file

The Secret file in the repository uses stringData so it is readable, and it contains a fake token. In real life the file is not committed; the Secret is created by kubectl create secret from a CI variable, or by an operator that syncs from a vault.

The commands

# Create from files (fine for config; the Secret file here holds a fake token)
kubectl apply -f k8s/configmap.yaml -f k8s/secret.yaml

# Create from the command line: the usual way for real Secrets
kubectl create configmap pantry-config --from-literal=GREETING='Hi' --from-file=motd.txt
kubectl create secret generic pantry-secret --from-literal=API_TOKEN="$TOKEN"

# See them; Secrets show sizes, not values
kubectl get configmap,secret
kubectl describe configmap pantry-config

# Read a Secret value back (base64, then decode)
kubectl get secret pantry-secret -o jsonpath='{.data.API_TOKEN}' | base64 -d

# Apply the Deployment that consumes them, then confirm from inside a Pod
kubectl apply -f k8s/deployment-config.yaml
kubectl exec deploy/pantry -- sh -c 'echo $GREETING; cat /etc/pantry/motd.txt'

# Env vars are read at start: restart Pods to pick up a changed ConfigMap
kubectl rollout restart deploy/pantry

Try it

1. Create both and look at how they are shown:

   kubectl apply -f k8s/configmap.yaml -f k8s/secret.yaml
   kubectl get configmap pantry-config -o yaml | head -12
   kubectl get secret pantry-secret -o yaml | grep -A2 '^data'
   

The ConfigMap shows its values. The Secret shows API_TOKEN: czNjcjN0..., which is base64 and decodes with one command.

2. Wire them into Pantry and check from inside:

   kubectl apply -f k8s/deployment-config.yaml
   kubectl rollout status deploy/pantry
   kubectl exec deploy/pantry -- sh -c 'echo GREETING=$GREETING; echo API_TOKEN=$API_TOKEN; cat /etc/pantry/motd.txt'
   

   GREETING=Welcome to the Pantry
   API_TOKEN=s3cr3t-token-do-not-commit-real-ones
   Today's special: containers, freshly orchestrated.
   

kubectl exec deploy/pantry picks a current Pod of the Deployment for you.

3. Change a value and see which path notices:

   kubectl patch configmap pantry-config -p '{"data":{"GREETING":"Hello, staging","motd.txt":"Menu changed.\n"}}'
   sleep 60; kubectl exec deploy/pantry -- sh -c 'echo $GREETING; cat /etc/pantry/motd.txt'
   

The file says Menu changed.; the env var still says Welcome to the Pantry. Then:

   kubectl rollout restart deploy/pantry && kubectl rollout status deploy/pantry
   curl -H 'Host: pantry.localtest.me' localhost/
   

   Hello, staging (v1). You are visitor #6. Served by pantry-7c9d4b8f6-q2xkm.
   

4. Put it back for the next chapter:

   kubectl apply -f k8s/configmap.yaml
   kubectl rollout restart deploy/pantry
   

Gotchas

• Environment variables are read once at container start; a changed ConfigMap does nothing to running Pods until rollout restart. Mounted files do update, after up to a minute.
• A Secret is base64, and kubectl get secret -o yaml shows it to anyone with read access. Protection is RBAC and never committing real values, not the encoding.
• A ConfigMap is capped at about 1 MiB; a 2 MB file is refused with too large, and belongs in the image or a volume instead.
• Keys used with envFrom must be valid environment variable names; a key like motd.txt is silently skipped as an env var and only works as a mounted file.

Recall

1. What is the actual difference between a ConfigMap and a Secret?
2. Which injection path picks up a changed value without a restart?
3. How do you read a Secret's value back in plain text?
4. Why does one image now serve staging and production?
5. In the city metaphor, what is a Secret?

What Maya learned

• Configuration lives in ConfigMaps and secrets in Secrets, injected as env vars or files, so one image and one Deployment serve every environment.
• A Secret is base64, not encrypted; its protection is in access control and handling, so never commit real ones.
• Env vars need a Pod restart to change; mounted files update in place.

…but

Staging is configured. During the first load test, one Pantry Pod starts answering slowly, then stops answering at all, while still showing Running. The Service keeps sending it a third of the traffic. Kubernetes cannot tell a Pod that is up from a Pod that is working, because nobody has told it how to check.

Go deeper

Kubernetes manual 6 · Config and Secrets

Chapter 12 · Act II · 4 min readAre You Alive?#

The problem

During the load test one Pantry Pod wedged. The process was alive, so the kubelet saw a healthy container. The Pod showed Running and 1/1, so the Service kept sending it a third of all requests, and a third of all requests timed out. Kubernetes did exactly what it had been told, which was nothing.

Two other things surfaced in the same afternoon. When Maya scaled to ten Pods, several were sent traffic in the half-second between the process starting and Redis being reachable, and returned errors. And one Pod, sharing a node with a memory-hungry neighbour, was slow for no reason of its own.

Raj sums it up: "You never told the city how to check on a tenant, and you never told it how much space a tenant needs."

The idea

A probe is a check the kubelet runs against a container on a timer. There are three, and they answer three different questions:

Each can be an HTTP GET (a 2xx or 3xx passes), a TCP connect, or a command run inside the container. Pantry's /healthz returns 200 only when Redis answers, so it is a good readiness check and an acceptable liveness check.

The mistake everyone makes once is using the same aggressive check for liveness and readiness. A readiness failure is gentle: stop sending traffic, keep the Pod. A liveness failure is violent: kill and restart. If Redis goes down for a minute and liveness checks Redis, every Pantry Pod is restarted in a loop for something that was never their fault. Make readiness strict and liveness lenient (longer period, more failures allowed), and give slow starters a startup probe so liveness does not kill them mid-boot.

Remember it as: liveness asks "are you awake?" and restarts you, readiness asks "are you taking visitors?" and just stops the traffic.

Resources are the other half. Every container should declare a request, the CPU and memory it needs to run properly, and a limit, the most it may ever use. The scheduler uses requests to pick a node with room; that is the only information it has. The kubelet enforces limits: a container that exceeds its memory limit is killed (OOMKilled), and one that exceeds its CPU limit is throttled.

resources:
  requests: { cpu: 50m,  memory: 64Mi }    # what the lease promises: 5% of a core, 64 MiB
  limits:   { cpu: 500m, memory: 128Mi }   # the ceiling: half a core, 128 MiB

Think of it as the lease: requests are what you are promised, limits are the ceiling you cannot exceed. Without requests, the scheduler packs Pods onto nodes blind, which is how the memory-hungry neighbour happened. m is millicores (1000m = one CPU), Mi is mebibytes. Set requests from what you measure with kubectl top, and set memory limits close to requests, because memory cannot be throttled, only killed.

The commands

# Apply the Deployment with probes and resources
kubectl apply -f k8s/deployment-probes.yaml

# READY shows containers passing readiness; RESTARTS climbs on liveness failures
kubectl get pods -l app=pantry

# See probe results and OOM kills in the events
kubectl describe pod <pod> | grep -A12 Events

# Install metrics-server on kind (cloud clusters have it already), then measure
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl patch -n kube-system deploy metrics-server --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl top pod
kubectl top node

# What has been promised on each node, and to whom
kubectl describe node pantry-worker | grep -A6 'Allocated resources'

Try it

1. Roll out the probed version:

   kubectl apply -f k8s/deployment-probes.yaml
   kubectl rollout status deploy/pantry
   

The rollout takes a little longer now: a new Pod only counts as available once its readiness probe passes, so maxUnavailable: 0 really means zero.

2. Make Redis disappear and watch readiness do its job:

   kubectl scale deploy/redis --replicas=0
   sleep 15; kubectl get pods -l app=pantry
   kubectl get endpoints pantry
   

   NAME                      READY   STATUS    RESTARTS   AGE
   pantry-5c8f7d9b4-2jk8x    0/1     Running   0          2m
   ...
   NAME     ENDPOINTS   AGE
   pantry   <none>      40m
   

0/1: running, not ready, no traffic. Nothing was restarted. Bring Redis back and the endpoints return within seconds:

   kubectl scale deploy/redis --replicas=1
   

3. Break one Pod and watch liveness restart it:

   kubectl port-forward deploy/pantry 8000:8080 &
   curl localhost:8000/crash; kill %1
   sleep 5; kubectl get pods -l app=pantry
   

One Pod shows RESTARTS 1. Its events say Container web failed liveness probe, will be restarted or simply that the process exited and was restarted; either way the kubelet handled it.

4. Measure what Pantry actually uses:

   kubectl top pod -l app=pantry
   

   NAME                     CPU(cores)   MEMORY(bytes)
   pantry-6b6ff9b46-64857   3m           28Mi
   pantry-6b6ff9b46-8wfhf   2m           27Mi
   

Two millicores and 28 MiB. The request of 50m and 64Mi is generous, and the limit of 128Mi has room. Chapter 15 shows what happens when it does not.

Gotchas

• A liveness probe that checks a dependency (Redis) restarts every Pantry Pod whenever Redis blinks, for a fault that was never theirs. Liveness checks the process; readiness checks the dependencies.
• CPU over the limit is throttled, memory over the limit is killed. A memory limit set from a guess is how OOMKilled happens to healthy apps.
• Without a startup probe, a slow-booting container can fail its liveness checks before it is up and be restarted forever, looking exactly like a crash loop.
• kubectl top shows "Metrics API not available" until metrics-server is installed and has had a minute to collect.

Then and now

You will see Heapster in older monitoring guides and dashboards; it was retired in Kubernetes 1.13 (2018) in favour of metrics-server, which is why kubectl top has a prerequisite today. Old Maps tells the story.

Recall

1. What happens on a readiness failure, and what happens on a liveness failure?
2. Which endpoint does Pantry use for liveness and which for readiness, and why are they different?
3. What does the scheduler use requests for, and what does the kubelet use limits for?
4. Which command shows real CPU and memory use per Pod?
5. In the city metaphor, what are requests and limits?

What Maya learned

• Readiness gates traffic and liveness triggers restarts; make readiness strict, liveness lenient, and add a startup probe for slow boots.
• Requests are what the scheduler reserves, limits are what the kubelet enforces; memory over the limit means the container is killed.
• Measure with kubectl top and set requests from reality, not guesses.

…but

Pantry's Pods are honest about their health now. Then Raj scales the Redis Deployment to zero and back up, and the visit counter starts again at one. Maya has seen this before, in chapter 3, and she knows the word for what is missing. She just does not know what it is called in the city.

Go deeper

Kubernetes manual 4 · Lifecycle and Rollouts · Kubernetes manual 5 · Resources and Placement

Chapter 13 · Act II · 4 min readMemory That Survives, Again#

The problem

The Redis Pod was replaced, and the visit counter went back to one. Maya recognises the shape immediately: a Pod's filesystem is the container's writable layer, and it dies with the container, exactly as in chapter 3. Then she needed a volume. Now she needs a volume that works when the Pod might come back on a different node, where a directory on the old node's disk is no use at all.

There is a second wrinkle. Redis is a database. Databases care about identity: they want the same name and the same disk every time they start, not a random Pod name and whichever disk is free. A Deployment, which treats all its Pods as interchangeable, is the wrong manager for that.

The idea

Kubernetes separates the disk from the request for a disk. A PersistentVolume (PV) is a real piece of storage: a cloud disk, an NFS export, a directory on a node. A PersistentVolumeClaim (PVC) is a request: "I need 100 MiB, read-write by one node at a time." The cluster binds a claim to a volume that satisfies it, and the Pod mounts the claim, never the volume directly. That indirection is what lets the same Pod spec work on a laptop and in three clouds.

Nobody wants to create PVs by hand, so a StorageClass names a provisioner that creates them on demand. kind ships one called standard; clouds ship their own. A PVC that names no class gets the default.

Think of storage units: the PV is the unit, the PVC is the rental request, and the StorageClass is the storage company that builds a unit when someone asks. Delete the Pod and the claim, and the unit, stay. Delete the claim and the class's reclaim policy decides whether the unit is wiped.

Remember it as: a PVC is the rental request, the PV is the storage unit, and the Pod mounts the request so it does not care which unit it got.

One thing surprises people on kind and on most clouds: a fresh PVC shows Pending until a Pod uses it. The class's binding mode is WaitForFirstConsumer, so the unit is built on the node where the Pod is scheduled, not somewhere the Pod cannot reach. It is not an error.

For Redis's identity problem there is a second manager: the StatefulSet. It is a Deployment for things with names. Its Pods are numbered (redis-0, redis-1), start and stop in order, each get a stable DNS name through a headless Service, and each get their own PVC from a template, so redis-0 always mounts data-redis-0 even after being rescheduled. Databases, queues and anything that says "node ID" in its docs go in a StatefulSet. Everything else stays in a Deployment.

The whole StatefulSet spec adds three things to what you already know: a serviceName pointing at a headless Service, a volumeClaimTemplates list, and that is all.

kind: StatefulSet
spec:
  serviceName: redis                 # headless Service (clusterIP: None): gives redis-0.redis a DNS name
  replicas: 1
  template:
    spec:
      containers:
        - name: redis
          args: ["redis-server", "--appendonly", "yes"]
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:              # one PVC per Pod, named data-<pod>
    - metadata: { name: data }
      spec:
        accessModes: ["ReadWriteOnce"]
        resources: { requests: { storage: 100Mi } }

The commands

# What storage companies exist, and which is the default?
kubectl get storageclass

# Make a claim, and watch it wait for a consumer
kubectl apply -f k8s/pvc.yaml
kubectl get pvc

# Replace the throwaway Redis with the StatefulSet version
kubectl delete -f k8s/redis.yaml
kubectl apply -f k8s/redis-statefulset.yaml
kubectl rollout status sts/redis

# The claim it made for itself, and the volume that was built
kubectl get pvc,pv

# StatefulSet Pods have stable names; delete one and the same name comes back
kubectl delete pod redis-0
kubectl get pods -l app=redis -w

# Copy a file out of (or into) a Pod
kubectl cp redis-0:/data/appendonlydir ./redis-backup

Try it

1. See the storage company and make a standalone claim:

   kubectl get storageclass
   kubectl apply -f k8s/pvc.yaml
   kubectl get pvc scratch
   

   NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      AGE
   standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   5m
   NAME      STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   scratch   Pending                                      standard       8s
   

Pending, and it will stay that way until a Pod mounts it. That is the binding mode, not a fault.

2. Swap Redis for the version that remembers:

   kubectl delete -f k8s/redis.yaml
   kubectl apply -f k8s/redis-statefulset.yaml
   kubectl rollout status sts/redis
   kubectl get pvc,pv
   

   NAME                                 STATUS   VOLUME                                     CAPACITY   STORAGECLASS
   persistentvolumeclaim/data-redis-0   Bound    pvc-92ac9f2c-5034-485d-96f8-875737557612   100Mi      standard
   persistentvolumeclaim/scratch        Pending                                                        standard
   

data-redis-0 was claimed by the StatefulSet's template, bound, and a PV was built for it.

3. Count some visits, destroy the Pod, and count again:

   curl -H 'Host: pantry.localtest.me' localhost/; curl -H 'Host: pantry.localtest.me' localhost/
   kubectl delete pod redis-0
   kubectl wait --for=condition=Ready pod/redis-0 --timeout=90s
   curl -H 'Host: pantry.localtest.me' localhost/
   

   Hello from Pantry (v1). You are visitor #1. ...
   Hello from Pantry (v1). You are visitor #2. ...
   pod "redis-0" deleted
   Hello from Pantry (v1). You are visitor #3. ...
   

Same name, same claim, same data. The pantry shelf has moved to the city.

4. Clean up the unused claim:

   kubectl delete pvc scratch
   

Gotchas

• A claim can never shrink (can not be less than status.capacity), and it can grow only if its StorageClass allows expansion; kind's standard does not, so size claims generously.
• Deleting a StatefulSet does not delete its PVCs, on purpose; the data waits for a StatefulSet of the same name. Delete the claims yourself if you really mean it.
• A default StorageClass with reclaim policy Delete wipes the volume when the claim is deleted; check kubectl get storageclass before deleting a claim on a real cluster.

Recall

1. What is the relationship between a PV, a PVC and a StorageClass?
2. Why is a fresh PVC on kind Pending, and is that a fault?
3. What does a StatefulSet give Redis that a Deployment does not?
4. Which command lists claims and volumes together?
5. In the city metaphor, what is a PVC?

What Maya learned

• Pods mount a PersistentVolumeClaim; a StorageClass provisions a PersistentVolume to satisfy it, so the Pod never names a disk.
• A claim showing Pending on kind or a cloud is usually waiting for its first Pod, not failing.
• StatefulSets give Pods stable names and their own claims; use them for databases and Deployments for everything else.

…but

Pantry is healthy, configured and durable. The company hires a second team, who want a cluster for their own service and would rather not share a default namespace with Maya's. Then the launch traffic returns, and Maya finds herself running kubectl scale by hand at lunchtime every day.

Go deeper

Kubernetes manual 7 · Storage

Chapter 14 · Act II · 5 min readDistricts, Key Cards and Rush Hour#

The problem

The new team's service is called Shop, and on day one one of its engineers runs kubectl delete deploy pantry in the wrong terminal. Nothing was lost, thanks to chapter 9, but everyone agrees this should not have been possible. Two teams, one cluster, one flat list of everything, and every developer holding the admin credentials Raj handed out on day one.

Meanwhile, the traffic. Pantry needs six Pods at lunchtime and two overnight. Maya has a calendar reminder to run kubectl scale at 11:45 and again at 14:30, which is a restart policy for humans.

And there is a growing list of small jobs: reset a counter once, print a report every minute, run a log shipper on every node. None of them is a long-running web service, and Deployments keep trying to restart the ones that finish.

Four problems, and it turns out to be one short chapter, because all four answers are small.

The idea

Namespaces are districts. Every namespaced resource (Pods, Services, ConfigMaps, most things) lives in exactly one, names only need to be unique within it, and kubectl commands act on one at a time (-n shop, or -A for all). Pantry gets pantry, Shop gets shop, and kubectl delete deploy pantry from the shop district finds nothing to delete. Namespaces also carry quotas and, most usefully, are the unit that permissions attach to. DNS respects them too: pantry.pantry.svc from anywhere, pantry only from inside its own district.

RBAC (role-based access control) is the key cards. A Role lists what may be done to which resources in one namespace (get, list, watch on pods). A RoleBinding hands that Role to a subject: a user, a group, or a ServiceAccount, which is the identity a Pod runs as. Cluster-wide versions (ClusterRole, ClusterRoleBinding) exist for things like nodes and for admins. Shop's engineers get a binding in shop and nothing in pantry.

Key cards say who may ask city hall for what. They say nothing about which apartments may talk to each other; by default every Pod can reach every other Pod in the cluster. A NetworkPolicy closes corridors: it selects Pods and lists who may connect to them, on which ports. k8s/networkpolicy.yaml says that Redis accepts connections only from Pods labelled app: pantry, on 6379, and nothing else. A policy needs a network plugin that enforces it; kind's does, and so do all the cloud ones.

Remember it as: a Namespace is a district and RBAC is the key card that opens some doors in some districts and none in others.

The HorizontalPodAutoscaler (HPA) is hiring temps in rush hour. It watches a metric (CPU utilisation against the request from chapter 12, by default) and adjusts a Deployment's replica count between a minimum and a maximum to hold the metric near a target. It is a control loop like everything else, and it only works if the Pods declare CPU requests and metrics-server is installed.

Remember it as: the HPA hires temps when the queue grows and lets them go when it shrinks, within the numbers you set.

The last three are workload types for things that are not web servers, and a table is all they need:

The commands

# Districts: create, list, act inside one (-n), or everywhere (-A)
kubectl create namespace pantry-staging
kubectl get namespaces
kubectl get pods -n pantry-staging
kubectl get pods -A

# Key cards: apply a ServiceAccount + Role + RoleBinding, then ask "can X do Y?"
kubectl apply -f k8s/rbac.yaml
kubectl auth can-i list pods -n pantry-staging --as=system:serviceaccount:pantry-staging:pod-reader
kubectl auth can-i delete pods -n pantry-staging --as=system:serviceaccount:pantry-staging:pod-reader

# Corridors: who may talk to whom
kubectl apply -f k8s/networkpolicy.yaml
kubectl get networkpolicy

# Rush hour: create the autoscaler and watch it think
kubectl apply -f k8s/hpa.yaml
kubectl get hpa -w

# Contractors and cleaners
kubectl apply -f k8s/job.yaml && kubectl logs job/reset-counter
kubectl apply -f k8s/cronjob.yaml && kubectl get cronjob
kubectl apply -f k8s/daemonset.yaml && kubectl get ds

Try it

1. Make a district and a key card, and test the card:

   kubectl apply -f k8s/namespace.yaml -f k8s/rbac.yaml
   kubectl auth can-i list pods -n pantry-staging --as=system:serviceaccount:pantry-staging:pod-reader
   kubectl auth can-i delete pods -n pantry-staging --as=system:serviceaccount:pantry-staging:pod-reader
   kubectl auth can-i list pods -n default --as=system:serviceaccount:pantry-staging:pod-reader
   

   yes
   no
   no
   

Read Pods in its district: yes. Delete them: no. Read Pods next door: no. auth can-i with --as is the fastest way to debug any permission question.

2. Hire temps. Pantry already has CPU requests from chapter 12; add the autoscaler and some load:

   kubectl apply -f k8s/hpa.yaml
   kubectl run load --image=busybox:1.36 --restart=Never -- sh -c 'while true; do wget -qO- http://pantry/ >/dev/null 2>&1; done'
   sleep 75; kubectl get hpa pantry
   

   NAME     REFERENCE           TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
   pantry   Deployment/pantry   cpu: 254%/50%   2         6         6          76s
   

One busy loop drove tiny Pods to 254% of their request, and the HPA went to the maximum. Delete the load Pod and, after a five-minute cool-down, it scales back to two.

   kubectl delete pod load
   

3. Run a contractor and a cleaner:

   kubectl apply -f k8s/job.yaml
   kubectl wait --for=condition=Complete job/reset-counter --timeout=60s
   kubectl logs job/reset-counter
   kubectl apply -f k8s/cronjob.yaml
   sleep 70; kubectl get jobs
   

The Job printed OK and finished. The CronJob has created its first child Job, named report-visits-<timestamp>, and kubectl logs job/<that name> shows the count.

4. Put a caretaker in every building:

   kubectl apply -f k8s/daemonset.yaml
   kubectl get ds caretaker
   

   NAME        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   AGE
   caretaker   2         2         2       2            2           10s
   

Two nodes, two Pods. The manifest carries a toleration so it may also run on the control-plane node, which normally refuses ordinary workloads.

Gotchas

• The HPA does nothing, and says nothing, if the Pods have no CPU requests; kubectl describe hpa shows <unknown> for the target. Requests first, autoscaler second.
• CronJob schedules run in UTC unless you set spec.timeZone; a "7 a.m." job that fires at 8 is this.
• kubectl delete namespace can hang in Terminating for ever when a resource inside has a finalizer nobody will honour; look for it before you start removing finalizers by hand.
• A NetworkPolicy with an empty podSelector: {} selects every Pod in the namespace; with policyTypes: [Ingress] and no rules, that is "deny all inbound".

Then and now

You will see PodSecurityPolicy (removed in 1.25, 2022), autoscaling/v2beta2 HPAs, batch/v1beta1 CronJobs, and code that expects every ServiceAccount to have a token Secret (not since 1.24). Old Maps tells the story.

Recall

1. What three things does a Namespace separate?
2. Which command answers "can this ServiceAccount delete Pods here?"
3. What two prerequisites does the HPA need before it will scale anything?
4. Which resource controls which Pods may connect to Redis?
5. In the city metaphor, what are a Job, a CronJob and a DaemonSet?

What Maya learned

• Namespaces split a cluster into districts for naming, quotas and permissions; -n picks one and -A sees all.
• RBAC grants verbs on resources in a namespace to users or ServiceAccounts; auth can-i --as answers any permission question.
• The HPA scales a Deployment on CPU against its requests; Jobs run to completion, CronJobs on a schedule, DaemonSets once per node.

…but

The city is well run. At 3 a.m., for the first time since Act I, Maya's phone rings. Four different things are wrong at once, and the dashboard says only Pending, ImagePullBackOff, CrashLoopBackOff and OOMKilled. This time she has the tools. She just needs the order.

Go deeper

Kubernetes manual 3 · Security Context · Kubernetes manual 5 · Resources and Placement · Kubernetes manual 9 · Identity and Access

Chapter 15 · Act II · 3 min readThe Night It Fell Over, Again#

The problem

3:04 a.m. Shop's team pushed four changes before going home, and the cluster has four unhappy Pods. Maya looks at kubectl get pods and sees the four words every Kubernetes engineer learns to read at a glance:

NAME                              READY   STATUS             RESTARTS      AGE
broken-crash-864749c4c6-d447b     0/1     CrashLoopBackOff   3 (43s ago)   60s
broken-image-85b97ddc-9wmxd       0/1     ImagePullBackOff   0             60s
broken-oom-7cd56c996d-xk8lb       0/1     OOMKilled          3 (40s ago)   60s
broken-pending-7866dd6949-7j2t8   0/1     Pending            0             60s

In Act I, one dead container took her two hours to find. Tonight there are four, and she has Raj's rule taped to her monitor: the status tells you which question to ask, and there are only four questions.

The idea

Every broken Pod is stuck at one of four points in its life: it could not be placed, its image could not be fetched, its process keeps dying, or it is being killed by the kubelet for breaking its lease. The STATUS column tells you which, and each has one command that reveals the cause.

Remember it as: Pending asks the scheduler, ImagePull asks the registry, CrashLoop asks the app's previous log, OOMKilled asks the lease.

The tools are ones you already own, with two additions. describe shows the Pod's events, which is where the scheduler and kubelet explain themselves. logs --previous shows the output of the last crashed container rather than the one currently restarting, which is where the stack trace lives. get events --sort-by shows the whole namespace's recent history in time order, which is how you notice that four things broke at 2:58 a.m. and not four separate times. exec lets you test from inside (is Redis reachable from here? does /healthz answer on localhost?). And debug attaches a throwaway container with real tools to a Pod whose image has no shell at all.

Two habits make this fast. First, always read the bottom of describe; the events are last and the newest is at the bottom. Second, when a Deployment's Pods are wrong, fix the Deployment (kubectl apply a corrected file or kubectl set image), never the Pod; the manager will just replace whatever you edit by hand.

The reverse case, Running and 1/1 but users see errors, is a chapter 12 problem: the probes are not checking the right thing. Running and 0/1 is a readiness failure, and the events say why.

These four have a STATUS word. The failures that do not, the Pod that is Running and 1/1 while users see errors, or the Service that routes nowhere, are the silent kind, and each chapter's Gotchas section lists the ones its concept produces.

The commands

# Triage: statuses, then the recent history of the whole district in order
kubectl get pods
kubectl get events --sort-by=.lastTimestamp | tail -20

# The story of one Pod; events are at the bottom
kubectl describe pod <pod>

# Logs of the crashed attempt, not the current retry
kubectl logs <pod> --previous

# Test from inside: is the dependency reachable? does the probe path work?
kubectl exec -it <pod> -- sh
kubectl exec <pod> -- python -c "import socket; print(socket.gethostbyname('redis'))"

# A tools container attached to a Pod that has no shell of its own
kubectl debug -it <pod> --image=busybox:1.36 --target=web

# The Pod's last exit reason in one line
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

# Fix the manager, not the Pod
kubectl set image deploy/<name> web=pantry:v1
kubectl apply -f <corrected>.yaml

Try it

Reproduce the night. The four broken Deployments are in k8s/broken/:

kubectl apply -f k8s/broken/
sleep 60; kubectl get pods -l 'app in (broken-pending,broken-image,broken-crash,broken-oom)'

1. Pending. Ask the scheduler:

   kubectl describe pod -l app=broken-pending | tail -5
   

   Warning  FailedScheduling  ...  0/2 nodes are available: 1 Insufficient memory, 1 node(s) had untolerated taint ...
   

The manifest requests 64 GiB of memory. Nobody has a flat that big. Fix: k8s/broken/pending.yaml, lower the request, apply.

2. ImagePullBackOff. Ask the registry:

   kubectl describe pod -l app=broken-image | grep -E 'Failed|BackOff' | tail -2
   

   Warning  Failed   ...  Failed to pull image "pantry:v1-typo": ... not found
   Normal   BackOff  ...  Back-off pulling image "pantry:v1-typo"
   

A tag that does not exist. Fix: kubectl set image deploy/broken-image web=pantry:v1.

3. CrashLoopBackOff. Ask the previous log:

   kubectl logs -l app=broken-crash --previous
   

   cannot find REDIS_HOST=rediss
   

A typo in an environment variable, printed by the app as it died. Fix the manifest, apply.

4. OOMKilled. Ask the lease:

   kubectl get pod -l app=broken-oom -o jsonpath='{.items[0].status.containerStatuses[0].lastState.terminated.reason}'
   

   OOMKilled
   

The Pod allocates 200 MiB and its limit is 64 MiB. Fix: raise the limit or stop allocating. Then look at the whole night in order:

   kubectl get events --sort-by=.lastTimestamp | grep -i broken | tail
   

5. Clean up:

   kubectl delete -f k8s/broken/
   

Four failures, four questions, four commands. Maya is back in bed by 3:20.

Recall

1. For each of Pending, ImagePullBackOff, CrashLoopBackOff and OOMKilled, which command answers "why"?
2. Where in kubectl describe output are the events, and in what order?
3. Why fix the Deployment rather than the Pod?
4. Which command shows the whole namespace's recent history in time order?
5. In the metaphor, what do the four status words ask?

What Maya learned

• The STATUS column names the failure and picks the command: describe for Pending, ImagePull and OOM, logs --previous for crashes.
• Events, read bottom-up, are where the scheduler and kubelet explain themselves; get events --sort-by shows the whole timeline.
• Fix the Deployment, not the Pod; the manager will replace anything you patch by hand.

…but

Pantry is a solved problem. Then the company grows. Shop's team arrives with a service, a deadline, and a repository left behind by a contractor who has since moved on. Maya opens it on a quiet Friday afternoon and finds docker-compose.yml files with a version: line, a script that calls docker stack deploy, a Helm chart whose README says to run helm init, and a warning in bold that Kubernetes is about to drop Docker. Every word is one she almost knows.

Go deeper

Kubernetes manual 11 · Debugging

Interlude · 7 min readOld Maps#

The problem

Shop's previous contractor has left, and left a repository. Maya opens it on a quiet Friday afternoon, expecting to read it in an hour, and finds a language she almost speaks.

There is a docker-compose.yml that starts with version: "2" and wires the services together with links:. There is a deploy.sh containing one line, docker stack deploy -c stack.yml pantry, and a stack.yml full of a deploy: key she has never seen. There is a Helm chart whose README says "run helm init first", with its dependencies in a separate file called requirements.yaml. There is a Deployment with apiVersion: extensions/v1beta1 and no selector. There is an Ingress whose class is set by an annotation. There is something called a PodSecurityPolicy. And there is a note at the top of the README, in bold, warning that "Kubernetes is dropping Docker support, we need to migrate off Docker before it breaks."

Some of these will apply to the cluster. Some will fail. Some will apply and silently do nothing. Maya cannot tell which is which, and she cannot tell whether the bold warning is true.

Raj reads it over her shoulder and laughs, not unkindly. "None of this is wrong," he says. "It's just old. Let me tell you when each of these was true."

The idea

Everything in that repository was correct when it was written. Understanding it means knowing three stories, each of which ends with a sentence about what survived.

Arc one: how containers escaped Docker. Docker appeared in 2013 and for two years "container" and "Docker" meant the same thing. In 2015 Docker and others founded the Open Container Initiative (OCI), which wrote down two standards: what an image is, and how a runtime starts a container from one. Once those were standards, anyone could build a runtime, and the runtime that mattered was containerd, the engine underneath Docker itself. Kubernetes talks to runtimes through its Container Runtime Interface (CRI). Docker did not speak CRI, so Kubernetes carried an adapter called the dockershim. In Kubernetes 1.20 (December 2020) the project announced the shim would go, and the internet read it as "Kubernetes drops Docker". In 1.24 (May 2022) it went. Nothing about images changed: a docker build still produces an OCI image, and every cluster still runs it, through containerd directly instead of through Docker. The README's bold warning was true, and it did not matter.

Remember it as: the recipe card became a standard, so the caretaker stopped borrowing Docker's oven and bought his own.

Arc two: how the orchestrator war ended. Kubernetes was announced in 2014 and reached 1.0 in July 2015. Docker answered with its own orchestrator: Swarm mode, built into Docker 1.12 (June 2016), and a version 3 of the Compose file format with a deploy: key (Compose 1.10 and Docker 1.13, January 2017) so that one file could describe both a laptop and a cluster. docker stack deploy read that file and ran it on a swarm. For about two years teams had to choose. Then Docker itself shipped Kubernetes inside Docker Desktop (version 18.06, July 2018), which was the end of the argument. Compose survived, as the best way to run several containers on one machine, which is exactly how this book used it. A stack.yml is a Compose file with a deploy: section that Compose ignores; Swarm still exists and some small shops still run it happily.

Remember it as: two cities were built; one kept its kitchens and gave up on being a city.

Arc three: how the YAML kept moving. Kubernetes promises that apiVersion, kind, metadata, spec never change shape, and it has kept that promise. What changes is the addresses. Deployments started life in extensions/v1beta1; that address stopped working in 1.16 (September 2019), and apps/v1 requires the selector the old one let you omit. kubectl run created a Deployment until 1.18 (2020), when its generators were removed and it began creating only Pods, which is why old tutorials' kubectl run nginx --image=nginx did something different. The node that runs city hall was labelled master until the label was removed in 1.24 and the matching taint in 1.25; today it is control-plane. Ingress moved from networking.k8s.io/v1beta1 to v1 in 1.19, the old address was removed in 1.22 (2021), and with it the kubernetes.io/ingress.class annotation gave way to ingressClassName. PodSecurityPolicy was removed in 1.25 (2022) in favour of Pod Security Admission, which is a namespace label rather than a resource. Under the hood, kube-dns became CoreDNS by default in 1.13 (2018), Heapster gave way to metrics-server the same year, ServiceAccounts stopped getting a token Secret created automatically in 1.24, and in 1.33 (2025) the Endpoints API you met in chapter 10 began warning that EndpointSlices are its replacement.

The tools moved too. Helm 2 ran a server inside the cluster called Tiller, installed with helm init, with chart dependencies in requirements.yaml; Helm 3 (November 2019) removed Tiller and moved dependencies into Chart.yaml, and Helm 4 (November 2025) is the version in this book. Istio's telemetry ran through a component called Mixer, deprecated in 1.5 (2020) and removed in 1.8; its sidecars became optional when ambient mode reached GA in 1.24 (2024). Docker's own builder was replaced by BuildKit by default in Docker 23 (2023), which is why old build output looks different, and Docker Hub began rate-limiting anonymous pulls in November 2020.

Remember it as: the four-key shape never changed; only the street addresses did.

Alive, just not here. Some names you will meet are not old, only absent from this book. Podman runs OCI containers without a daemon and accepts most docker commands unchanged. Colima and Rancher Desktop are alternatives to Docker Desktop on a Mac, which became a paid product for larger companies in 2021. minikube is kind's older sibling, a one-node cluster in a VM. Kompose translates a Compose file into Kubernetes manifests, which is a reasonable first draft and a poor final one. Swarm, as said, still runs in small shops. All of them use the same images.

The commands

# Which API addresses does this cluster actually serve? (the old ones will be missing)
kubectl api-resources --api-group=extensions
kubectl api-resources | grep -i ingress

# Validate a manifest against the live cluster without creating anything
kubectl apply --dry-run=server -f old-maps/deployment-v1beta1.yaml

# The manual for the current address of any kind
kubectl explain deployment
kubectl explain deployment --api-version=apps/v1 --recursive | head -40

# Compose: parse and print the effective file; old keys produce warnings here
docker compose -f old-maps/docker-compose.yml config

# Helm: lint an inherited chart; Helm 2 charts and their READMEs show their age
helm lint old-maps/chart-helm2
helm version --short

Try it

The inherited files are in examples/pantry/old-maps/, each with a *.modern.* neighbour that is what you would write today.

1. Start with the Compose file. Nothing is broken, but the tool tells you it is old:

   cd examples/pantry
   docker compose -f old-maps/docker-compose.yml config | head -3
   

   level=warning msg="old-maps/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion"
   name: old-maps
   services:
   

version: is ignored, links: still works, and compose.modern.yaml is the same file without either.

2. The Swarm script fails on the first line, because your Docker is not a swarm:

   old-maps/deploy.sh
   

   Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" ...
   

Read stack.yml anyway. It is chapter 5's file with a deploy: block: replicas, update policy, restart policy. Every one of those ideas is a Deployment now.

3. Ask the cluster about the old addresses:

   kubectl apply --dry-run=server -f old-maps/deployment-v1beta1.yaml
   kubectl apply --dry-run=server -f old-maps/ingress-v1beta1.yaml
   kubectl apply --dry-run=server -f old-maps/psp.yaml