One Server, Three Languages#
Learn C, Go and Rust from scratch by building the same Redis-compatible server three times, and understand the power and the weakness of each.
October 2026 edition
16 chapters and 43 exercises. Reading time about 8 hours 35 minutes, measured by the build. Building time is yours. This copy was built on 12 October 2026.
Preface#
You learn by building. So this book gives you one thing to build, and asks you to build it three times.
The thing is kv, a small server that speaks the Redis protocol. From chapter 4 on, the real redis-cli can talk to it. By the end it keeps keys and values in a hash table you wrote and serves many clients at once. It removes a key when its time limit runs out, survives a crash by replaying a log, and answers redis-benchmark at a respectable speed.
You build kv in C first, then in Go, then in Rust. The order is the order of history. C was made in 1972 to write an operating system. Go and Rust were both made, about forty years later, to fix the problems that forty years of C had shown. Go fixed them by making the language smaller and adding a garbage collector. Rust fixed them by making the compiler stricter. Building the same server in all three is the clearest way to feel the difference.
Who this book is for#
You wrote some C at university and have forgotten most of it. You used Go for a while and have forgotten that too. You have heard that Rust is good for systems programming and have never written any. You want to understand what each language is good at, what each one is bad at, and what the tools around each one are like. You do not want a long book.
No chapter assumes that you remember the chapter before. Each one restates the few facts it needs. Many readers read English as a second language, so every sentence holds one fact and no idiom.
How to read it#
Every teaching chapter has the same shape. A short lead says what the chapter is about. A table compares the concept across the three languages. Then you build the feature in C, in Go, and in Rust. Each build has an exercise. The exercise shows a design and partial code. You write the rest. If you are stuck, open the folded solution under the exercise. It holds the complete listing. The chapter closes with a verdict, a small scorecard of which language was strong, which was weak, and why.
Four kinds of box appear in the text. A history box says why a language is the way it is. A rough edge box warns about a mistake that many people make. A practice box names the convention professionals follow. A platforms box explains where macOS, Linux and Windows differ.
Code appears side by side on a wide screen and in tabs on a narrow one. Press 1, 2 or 3 to follow one language through the whole book, or use the Follow buttons in the sidebar. Press โK to search.
The code#
Every listing in this book is taken from a file in the code/ folder of the
book's repository. There is one complete copy of kv for each chapter and each language, so
code/rust/ch09 is exactly what chapter 9 ends with. The same test script drives
all three versions through redis-cli and compares the replies. The build of the
book reads the listings from those files, so the text and the code cannot disagree.
The book targets C17 with clang 17 (notes on C23 and gcc), Go 1.27 and Rust 1.99, edition 2024, tokio 1, with redis-cli 8.10 as the client. Version-sensitive facts were verified from 12 October 2026 onwards. The edition is October 2026.
This copy was built on 12 October 2026. Reading time is about 8 hours 35 minutes, measured by the build. The time you spend building is yours, and it is the point.
Fifty years, three answers#
C was made in 1972 to write an operating system. Go and Rust were made about forty years later to fix what forty years of C had shown. Go made the language smaller and added a garbage collector. Rust made the compiler stricter. This chapter tells the three stories, because every design choice in this book has a reason in them.
One problem, three answers#
Every program in this book does the same job. It takes bytes from a network socket, keeps some of them in memory, and sends bytes back. That is systems programming. The hardware has not changed its nature since 1972. Memory is still a flat run of bytes. A socket is still a stream. What changed is how much a mistake costs.
The timeline below is the shape of the whole chapter. Read it once now and again at the end.
1969 to 1973: why C#
Bell Labs, 1969. Ken Thompson has a DEC PDP-7 that nobody else wants, a machine with little memory and almost no software. On it he has written the first Unix. He needs a language to write programs for it, and the languages he knows are too big for the machine. So he makes a small one and calls it B. B was BCPL, a language that Martin Richards had designed in the mid-1960s, "squeezed into 8K bytes of memory", in Dennis Ritchie's words. Ritchie also wrote that "B can be thought of as C without types".
B had one type. Every value was a machine word. On the PDP-7 that was fine, because the machine addressed memory in words. In 1970 the group got a PDP-11. The PDP-11 addressed memory in bytes. It also had no floating-point hardware at first. A language with one word-sized type did not fit a machine that worked in bytes. B was also slow, because its compiler produced code that an interpreter ran, not machine code.
So in 1971 Ritchie added a character type to B and made it produce PDP-11 machine code. During 1971 to 1973 he added more types and then structures. The result was C. In the summer of 1973 the Unix kernel was rewritten in C. An attempt in 1972 had failed. The attempt in 1973 succeeded because C now had structures, which the kernel needed to describe its own tables.
Ritchie called the rule that turns an array into a pointer "the crucial jump in the evolutionary chain between typeless BCPL and typed C". An array name in C is the address of its first element. That rule is why
buf[3] and *(buf + 3) mean the same thing. It is also why C cannot
check an array index at run time. The length is not part of the value.
Unix spread, and C spread with it. Brian Kernighan and Ritchie published The C Programming Language in 1978. In the same years the language was made portable. A compiler that could produce code for more than one machine was described in 1978, and Unix was moved to the Interdata 8/32. From then on, C was the language that operating systems were written in. In 1983 an ANSI committee began to write a standard. It finished at the end of 1989, and ISO adopted it in 1990. Newer standards followed in 1999, 2011, 2018 and 2024. The 2018 standard is called C17 and is the one this book uses. The 2024 standard is called C23.
In 1995, when it began the next revision, the committee wrote down what it wanted to keep. It called this the spirit of C, and listed it as five short rules. The first was "Trust the programmer". The second was "Don't prevent the programmer from doing what needs to be done". The fifth was "Make it fast, even if it is not guaranteed to be portable". Those rules explain almost everything about C that surprises a reader coming from another language. C does not check an array index. C does not know when to free memory. C lets you add one to a pointer and read whatever is there. The programmer is trusted to be right.
Ritchie's own summary of the language was "quirky, flawed, and an enormous success". Every word of that is true.
What forty years of C cost#
Trusting the programmer worked when one careful person wrote a kernel. It worked less well when thousands of people wrote browsers, and strangers on the internet sent those browsers input. The first stranger arrived in 1988.
On the evening of 2 November 1988 a program began to copy itself from one Unix machine to the next, across the whole internet. Robert Tappan Morris, a graduate student at Cornell, released it from a machine at MIT. One of the ways it got in was a buffer overflow in the finger service. That service was a C program that read a request into a fixed array without checking the length. The worm sent a request longer than the array, and the bytes past the end became code that ran. About 6,000 machines were infected, by the figure most often reported. The case became the first felony conviction under the Computer Fraud and Abuse Act. The bug was a C program trusting its input. Thirty years later it was still the most common kind.
In 2019 Microsoft reported that about 70 percent of the vulnerabilities it reported each year were memory safety bugs. The Chromium project reported the same figure for its own code. Around 70 percent of its serious security bugs, in a sample of 912 high and critical bugs since 2015, were memory safety problems. A memory safety bug is a read or write that the program should not have made. An index past the end of an array is one. Using memory after it was freed is another. C allows both, because of the rules above.
Memory was one cost. Build time was another. By 2007, Rob Pike wrote, a large C++ binary at Google took 45 minutes to build. The C++ language inherited the C way of sharing declarations, which is to copy a header file into every file that includes it. In one measurement, 4.2 megabytes of source expanded to more than 8 gigabytes of compiler input. That is about 2,000 bytes of work for every byte a programmer wrote. The problem was old. In 1984 a Unix build was found to include the same header file 37 times for one program.
The third cost was concurrency. By 2007 a computer had several processors, and a server had to use all of them. C gives you threads and locks and nothing else. A mistake with a lock is a data race, and a data race in C is undefined behaviour. The program may do anything.
The C committee has said this itself. The charter for the 2024 standard states that "Trust the programmer, as a goal, is outdated in respect to the security and safety programming communities". The committee kept the goal. A sixth rule, added for the 2011 standard, says "Make support for safety and security demonstrable". Two other groups of people did not wait for the committee. They made new languages.
2007 to 2012: why Go#
Google, 21 September 2007. A C++ build is running, and it will take 45 minutes. While it runs, Robert Griesemer, Rob Pike and Ken Thompson start to sketch a new language on a whiteboard. Thompson had made B in 1969. Now he was in the room again, 38 years later, to replace what B had become. Pike's own account says the idea came during one of those builds. The Go FAQ says the language "was born out of frustration with existing languages and environments". It names the frustrations. Large programs were too complex to build. Multiprocessor computers were "becoming universal but most languages offered little help to program them". And a programmer had to "choose either efficient compilation, efficient execution, or ease of programming".
The answers in Go are all answers to those frustrations. Each one is a thing C does not do.
- A garbage collector. The FAQ says the lifetime of objects is "one of the biggest sources of bookkeeping in systems programs". Go frees memory for you. The collector pauses the program, but by now the pauses are "often to the sub-millisecond range".
- Goroutines and channels. The concurrency comes from the idea of communicating sequential processes, which Tony Hoare described in 1978. Go took it through two earlier languages, Newsqueak and Limbo. A goroutine is cheap, and a channel sends a value from one to another.
- Fast builds. A Go file names only the packages it imports, and never copies in a header. An import that is not used is a compile error. So the compiler reads each dependency once, and the build of a large program takes seconds.
- No exceptions. The FAQ says the try-catch-finally idiom "results in
convoluted code". A Go function returns its error as a second value, and the caller checks it.
Go keeps
panicfor the cases that are truly exceptional. - One format. The tool
gofmtdecides how code looks. Pike wrote that this removed "an entire class of argument between programmers".
The syntax came from C. The packages and declarations came from the Pascal, Modula and Oberon family, which Griesemer knew from Niklaus Wirth. Go became an open source project on 10 November 2009. Go 1 was released on 28 March 2012 with a promise. A program written for Go 1 "will continue to compile and run correctly, unchanged, over the lifetime of that specification". The promise still holds. The version in this book is Go 1.27, from August 2026.
Go was designed for a company with a very large code base and a very large number of programmers. A language that is small is a language that everyone on the team reads the same way. That is why Go has no classes, no generics until 2022, and one loop keyword. The plainness is the point. When Go feels limited in this book, remember who it was built for.
2006 to 2015: why Rust#
Mozilla, 2006. The company's product is a web browser written in C++, and a browser is the program that the 70 percent figure above describes. Every memory bug in it is a hole that a web page can reach. Graydon Hoare, an engineer there, starts a language of his own in his spare time and calls it Rust. He works on it alone for three years. From late 2009 Mozilla pays him to work on it, and in 2010 he presents it to the company. He hopes it will be "slightly less annoying" than the languages Mozilla uses. The two aims he gives are better memory safety and more concurrency.
Go answered the memory problem with a garbage collector. Mozilla did not want its engine to pause for a collector, or to pay for one in memory. So Rust needed a different answer. It found one in the compiler. Every value in Rust has exactly one owner. When the owner goes out of scope, the value is freed. A value can be lent to another part of the program, but the compiler checks every loan. A loan may not outlive its owner. While one part of the program can change a value, no other part can read it. These rules are checked before the program runs, so they cost nothing at run time. The checker is called the borrow checker, and it is the part of Rust that new programmers argue with.
The same rules solve the concurrency problem. A data race needs two threads with access to
one value, and at least one of them writing. The ownership rules do not allow that. A Rust program that compiles, and uses no unsafe, has no data races. The compiler proves it before the program runs. Go has a race detector, but
the detector finds a race only when a test meets one.
Rust 1.0 was released on 15 May 2015. The announcement called it "the official beginning of our commitment to stability". It described the aim as "low-level control over performance with high-level convenience and safety guarantees", "without requiring a garbage collector or runtime". A new version has come out about every six weeks since then. The versions this book uses are 1.97 from July 2026, 1.98 from August and 1.99 from October. Larger changes to the language arrive as an edition, and a program chooses its edition. This book uses the 2024 edition, which shipped with Rust 1.85 in February 2025.
In February 2021 the Rust Foundation was created to own the language, with AWS, Google, Huawei, Microsoft and Mozilla as founding members. Mozilla, "the original home of the Rust project", gave it the trademarks and the package registry. The language no longer belongs to one company.
C trusts you. Go checks a little, and a garbage collector handles the rest. Rust refuses to compile a program until it can prove that the memory rules hold. The proof is yours to supply, in the types you write. That is why a Rust program takes longer to write the first time and why the compiler's messages are so long. The compiler is doing the work that in C you do in your head, and in Go you leave to the collector.
Two answers to one question#
The question was what to do about the costs of trusting the programmer. Go and Rust gave different answers, and the table shows where they differ. The rest of the book is the long form of this table.
| The cost | C | Go | Rust |
|---|---|---|---|
| Memory that is freed too early | a crash or a security hole | impossible, the collector frees it | does not compile, the owner frees it |
| Memory that is never freed | a leak | unreachable objects are always freed, but a kept reference is a leak | freed at the end of scope, but a reference cycle or mem::forget can leak |
| Two threads writing one value | undefined behaviour | a race the detector may find | does not compile |
| A forgotten error check | the error is lost | the error is lost, but a linter can warn | the compiler warns on an unused Result |
| A slow build | the headers are copied into every file | packages are read once, builds take seconds | crates are compiled once, but the checks take time |
| The cost you pay | your attention, all the time | memory and pauses for the collector | the time to satisfy the compiler |
No answer is free. Go pays with a collector and with some things it refuses to let you express. Rust pays with the time it takes to learn, and with code that spells out more than C or Go would ask for. C pays nothing in advance, and everything later. In each chapter of this book you will feel one of those costs directly, because you will build the same thing three times.
What this means for kv#
The server you will build, kv, is a small version of Redis. Redis itself is written in C, and is a well written C program. It was started by Salvatore Sanfilippo in 2009, the year Go was announced. It keeps every key in memory, serves many clients from one thread, and writes an append-only log to survive a crash. Each of those three ideas is a chapter in this book. In each one you see the C answer first, then what Go and Rust change.
Toolchains and hello, three times#
Three programs, one line of output each. Build all three and look at the files you got. On the author's Mac:
$ ./hello-c; ./hello-go; ./hello-rust
hello from C
hello from Go
hello from Rust
$ ls -l hello-c hello-go hello-rust | awk '{print $5, $9}'
8440 hello-c
2516816 hello-go
414728 hello-rustThe same line, three times. One file is 8 kilobytes, one is 2.5 megabytes, and one sits between them. What is in the big one that is not in the small one? Where did the small one leave it?
Install a C compiler, Go and Rust. Compile one small program in each. Then look at what the
compiler made, because a binary is the thing all three languages exist to produce. The chapter
also installs redis-cli, which the server will talk to from chapter 4.
kv does not exist yet. By the end of this chapter you have three programs that print a line. You also know what a compiler, a linker and a build tool each do. Every later chapter assumes those three tools are on your path.
| The toolchain | C | Go | Rust |
|---|---|---|---|
| Compiler | cc (clang or gcc) | go | rustc, driven by cargo |
| Build tool | Make, or CMake for large projects | go build, with no build file | cargo build, with Cargo.toml |
| Project file | none, a Makefile by convention | go.mod | Cargo.toml |
| Formatter | clang-format, optional | gofmt, universal | rustfmt, universal |
| Hello binary on macOS | 8 KB, links the C library | 2.5 MB, the runtime is inside | 415 KB, the standard library is inside |
| Where the standard library is | a shared file the OS provides | compiled into every binary | compiled into every binary |
Install the three toolchains#
Each language has one official way to install it. For Go and Rust use that way, and not a package from a distribution, so that the version matches the baseline of this book. The commands are below for each platform.
On macOS, run xcode-select --install. It installs Apple's clang, which answers
to the name cc. On Debian and Ubuntu, run
sudo apt install build-essential, which installs gcc, make and the C library
headers. On Windows there are two choices. The Microsoft way is to download the Build Tools for
Visual Studio and select the "Desktop development with C++" workload. The compiler is then
cl.exe, and you run it from a Developer Command Prompt, because the tools need
many environment variables set. The other way is to install WSL and use the Ubuntu line above.
The C code in this book is written for POSIX, the Portable Operating System Interface that
macOS and Linux share. So WSL is the simpler choice on Windows.
Go is installed from a package. On macOS the package installs into
/usr/local/go and adds it to your path. On Linux, download the archive from
go.dev/dl and extract it:
$ rm -rf /usr/local/go && tar -C /usr/local -xzf go1.27.0.linux-amd64.tar.gz
$ export PATH=$PATH:/usr/local/go/bin
$ go version
go version go1.27.0 linux/amd64On Windows, run the MSI installer and open a new command prompt. The check is the same,
go version.
Rust is installed by rustup, which also manages versions later. On macOS,
Linux and WSL the command is one line:
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ rustc --version
rustc 1.99.0 (b940084d7 2026-09-28)On Windows, download and run rustup-init.exe. It may ask you to install the
Visual Studio C++ Build Tools, because Rust uses the Microsoft linker there. Finally, install
the Redis client. On macOS run brew install redis. On Debian and Ubuntu run
sudo apt install redis-tools. Check with redis-cli --version.
The Go you run may not be the Go you installed. A go.mod file
names a Go version. Since Go 1.21, when that version is newer than the installed one, the
go command downloads the newer toolchain and uses it. The author's machine had Go
1.23 installed, and go version printed 1.27 inside this book's code. That is
convenient, and it is also surprising. The environment variable GOTOOLCHAIN=local
turns it off.
C: from source to binary in four steps#
The C program is six lines:
#include <stdio.h>
int main(void) {
printf("hello from C\n");
return 0;
}Compile and run it. The flags after -std=c17 turn on every useful warning and
make each warning an error. Use them on every C file you ever compile:
$ cc -std=c17 -Wall -Wextra -Werror -o hello hello.c
$ ./hello
hello from CThe one command ran four tools, and it helps to see each one. The
preprocessor copies stdio.h into the file, and expands macros.
Those six lines become 571. The compiler turns C into assembly. The
assembler turns assembly into an object file, which is machine code with
holes where the addresses of other functions go. The linker fills the holes
and joins the object file with the C library. You can stop after any step:
$ cc -std=c17 -E hello.c | wc -l # preprocess only
571
$ cc -std=c17 -S -O2 -o - hello.c # compile to assembly, print it
_main:
pushq %rbp
movq %rsp, %rbp
leaq L_str(%rip), %rdi
callq _puts
xorl %eax, %eax
popq %rbp
retq
$ cc -std=c17 -c -o hello.o hello.c # assemble to an object file
$ nm hello.o # the names it defines (T) and needs (U)
0000000000000000 T _main
U _printfTwo details in that output matter for the rest of the book. The assembly calls
_puts, not _printf. The compiler saw a format string with no
arguments and chose the cheaper function. The compiler is allowed to do that, and much more,
whenever the result is the same. The object file was built without -O2, so it lists _printf as undefined, marked U. The linker resolves it to the C library, which on macOS is the shared
file libSystem.B.dylib. Your binary does not contain printf. It
contains a note that says where to find it.
The -O2 is the optimisation level, and it is the reason for puts.
At -O0, the default, the compiler translates each line as written, which is what
you want under a debugger. At -O2 it may reorder, inline and replace, as long as
the program behaves the same. At -O0 the hello program is 34 lines of assembly
that call printf. At -O2 it is 29 that call puts.
Every release build in this book uses -O2, and chapter 13 measures what it
is worth.
A C compiler reads one file at a time and knows nothing about the others. For a call to
printf to compile, the file must have seen a declaration of printf
first. Before header files, a declaration that several files needed was written again in each of them. A header file is
that declaration written once and copied in by the preprocessor. The scheme was cheap on a
PDP-11 and expensive. By 1984 one Unix program was found to include the same header 37 times. At Google in 2007 the same scheme turned 4.2 megabytes of source into 8 gigabytes of compiler input.
Go and Rust both replaced headers with modules, which chapter 7 covers.
Add two lines to main. The first prints the size of an int, a
long and a pointer. The second prints the compiler version. The operator
sizeof gives a size in bytes, and the format %zu prints one. The macro __VERSION__ is a string that gcc and clang define. It is not part of the C standard, and cl.exe on Windows defines _MSC_VER instead. On the author's machine the
output is int 4, long 8, pointer 8 bytes.
int main(void) {
printf("hello from C\n");
/* your two lines here */
return 0;
}Stuck?Show the full listing
#include <stdio.h>
int main(void) {
printf("hello from C\n");
/* sizeof gives a size in bytes. The %zu format prints a size_t. */
printf("int %zu, long %zu, pointer %zu bytes\n",
sizeof(int), sizeof(long), sizeof(void *));
printf("compiler " __VERSION__ "\n");
return 0;
}The C compiler accepts a great many programs that are wrong. The warnings catch a large share
of them, and -Werror stops you from ignoring one. Every C file in this book is
compiled with those three flags, and with -std=c17 so that the compiler applies one
known standard.
C: two files, one binary#
kv will not fit in one file for long. Split the hello program in two and the four steps above explain how C programs of any size are built. The second file holds the greeting. Its header declares the one function other files may call:
#ifndef GREET_H
#define GREET_H
/* Declared here, defined in greet.c. Every file that includes this header can call it. */
void greet(const char *who);
#endif#include <stdio.h>
#include "greet.h"
/* static: only this file can see the name. The linker never hears of it. */
static const char *prefix(void) {
return "hello from";
}
/* No static: this name has external linkage. units.c calls it through greet.h. */
void greet(const char *who) {
printf("%s %s\n", prefix(), who);
}The first file includes the header and calls the function:
#include "greet.h"
int main(void) {
greet("two files");
return 0;
}Each .c file, with the headers pasted in, is a translation
unit. The compiler reads one unit at a time and knows nothing about the others. So
units.c compiles because greet.h promised that greet
exists somewhere. The Makefile compiles each unit to an object file and links the two
object files:
# Two translation units. Each .c is compiled alone into a .o, and the linker joins them.
units: units.o greet.o
$(CC) $(CFLAGS) -o $@ units.o greet.o
units.o: units.c greet.h
$(CC) $(CFLAGS) -c units.c
greet.o: greet.c greet.h
$(CC) $(CFLAGS) -c greet.cnm shows what each object file defines and what it still needs. The letters
are the linkage of each name. T is a function this file
defines and other files may use. U is a name this file uses and someone else
must define:
$ make units
$ nm units.o
U _greet
0000000000000000 T _main
$ nm greet.o
0000000000000000 T _greet
U _printfprefix is missing from greet.o. It was declared
static, so its name never leaves the file, and at -O2 the compiler
pasted its body into greet. A function without static has external
linkage, which means every file in the program shares its name. That is why every public
name in kv starts with kv_, and why every private one is
static. Leave an object file out and the linker says what is missing:
$ cc -o units units.o
Undefined symbols for architecture x86_64:
"_greet", referenced from:
_main in units.o
ld: symbol(s) not found for architecture x86_64Hold on to that error. In chapter 7 the same mechanism holds the modules of kv together. In chapter 14 it is what linking a library means. In chapter 15 it is how Go and Rust call C at all.
Go: one command, one binary#
A Go program is a package. The file below is the whole of package main, and
the function main is where the program starts:
package main
import (
"fmt"
"runtime"
"unsafe"
)
func main() {
fmt.Println("hello from Go")
}Go needs one more file, go.mod, which names the module and the Go version it
is written for:
module hello
go 1.27Then one command compiles and links. There is no Makefile, because the go tool
finds every file in the folder itself:
$ go build -o hello .
$ ./hello
hello from Go
$ go run . # compile and run in one step, for development
hello from GoThe binary is 2.5 megabytes, three hundred times the C one. It contains the Go runtime,
which is the garbage collector, the scheduler for goroutines, and the code that talks to the
operating system. On Linux it has no dependency on the C library at all. On macOS it calls libSystem for system calls and nothing else. That is why a
Go binary can be copied to another machine of the same kind and run there with nothing
installed. Chapter 14 uses that fact.
Every variable starts at zero#
Look at the two declarations in the solution below, var i int and
var p *int. The declarations give no value, and both variables are usable. In
Go every variable begins as its zero value. A number is 0, a string is
empty, a bool is false. A pointer, a slice, a map or a function is nil. A
struct is the zero value of each field. There is no uninitialised memory to read
by mistake, which in C is one of the ways a program is wrong without a warning. The rule
shapes how Go code is written. A type is designed so that its zero value is ready to use, and
most types need no constructor.
The documentation is in the source#
The comment above an exported name is its documentation, and go doc prints
it. Ask about the function the hello program calls:
$ go doc fmt.Println
package fmt // import "fmt"
func Println(a ...any) (n int, err error)
Println formats using the default formats for its operands and writes toThe same text is what pkg.go.dev shows for every public module. When you
write a package in chapter 7, a sentence above each exported name is all the documentation
needs.
Add two lines to main. The first prints the size of an int and of a
pointer, with unsafe.Sizeof, which works like sizeof in C. The second
prints runtime.Version(), runtime.GOOS and
runtime.GOARCH. Import the packages runtime and unsafe.
An import you do not use is an error in Go, so add each import together with the line that
uses it.
func main() {
fmt.Println("hello from Go")
// your two lines here
}Stuck?Show the full listing
package main
import (
"fmt"
"runtime"
"unsafe"
)
func main() {
fmt.Println("hello from Go")
// unsafe.Sizeof is a compile-time constant, like sizeof in C.
var i int
var p *int
fmt.Printf("int %d, pointer %d bytes\n", unsafe.Sizeof(i), unsafe.Sizeof(p))
fmt.Println("compiler", runtime.Version(), runtime.GOOS, runtime.GOARCH)
}gofmt -l . lists the files whose layout is not the standard one, and
gofmt -w . fixes them. go vet ./... finds mistakes the compiler
accepts, such as a wrong format verb. Both tools come with Go, and every Go project runs them.
There is no setting for the style, which is the point.
Rust: cargo owns the project#
In Rust you rarely run the compiler, rustc, yourself. The tool
cargo creates the project, builds it, runs it, tests it and fetches its
dependencies. cargo new hello makes a folder with two files. The first is the
project file:
[package]
name = "hello"
version = "0.1.0"
edition = "2024"
[dependencies]The edition line names the version of the language rules. There are four,
2015, 2018, 2021 and 2024, and a new one comes every three years. An edition may change what
code means, such as which names are keywords, so a crate says which edition it is written
for. Crates of different editions link together, so a dependency written in 2018 compiles
unchanged in a 2024 project. That is how Rust changes the language without breaking the
world. The 2024 edition is the current one and this book uses it. The second file is
src/main.rs:
fn main() {
println!("hello from Rust");
}The exclamation mark means that println! is a macro, not a function. A macro
can check the format string at compile time, and this one does. A build has two modes. The
default is a debug build, which compiles quickly. The release build is slower to compile and
runs faster:
$ cargo build --release
Compiling hello v0.1.0
Finished `release` profile [optimized] target(s)
$ ./target/release/hello
hello from Rust
$ cargo run --release # build if needed, then runEverything cargo makes goes under target/. cargo new already lists it in the project's .gitignore. The binary is 415 kilobytes. Like the Go one it contains its standard
library. Like the C one it calls the C library of the operating system for a call such as
write. Rust sits between C and Go.
Add two lines to main. The first prints the size of four types, with size_of::<T>(), which the
compiler evaluates at compile time. The four types are i32, i64,
usize and a reference. The second prints std::env::consts::OS
and std::env::consts::ARCH. Then run cargo clippy and
cargo fmt, and see whether either tool reports something.
fn main() {
println!("hello from Rust");
// your two lines here
}Stuck?Show the full listing
fn main() {
println!("hello from Rust");
// size_of is evaluated at compile time, like sizeof in C.
println!(
"i32 {}, i64 {}, usize {}, pointer {} bytes",
size_of::<i32>(),
size_of::<i64>(),
size_of::<usize>(),
size_of::<&i32>()
);
println!("target {} {}", std::env::consts::OS, std::env::consts::ARCH);
}cargo fmt formats the project. cargo clippy is a linter with
more than 800 checks, from a needless clone to a loop that should be an iterator. Every Rust
snapshot in this book passes cargo clippy -- -D warnings, which makes each
warning an error, and cargo fmt --check.
The documentation is in the source, and it is tested#
A comment that starts with three slashes, ///, is a doc
comment. cargo doc turns every doc comment in your crate and in every
dependency into a website under target/doc, offline:
$ cargo doc
Documenting hello v0.1.0
Finished `dev` profile [unoptimized + debuginfo] target(s)
$ ls target/doc
crates.js hello help.htmlA code block inside a doc comment is a test. cargo test compiles it and runs
it, so an example in the documentation cannot go stale. Chapter 12 writes one.
What a binary is#
All three programs are files the operating system can load and run. On macOS the format is
called Mach-O. On Linux it is called ELF, the Executable and Linkable Format. On Windows it
is called PE, the Portable Executable. The file command names the format:
$ file hello
hello: Mach-O 64-bit executable x86_64A binary holds machine code, the constant data such as the string to print, and a list of
the shared libraries it needs. On macOS the list is printed by otool -L, on Linux
by ldd, and on Windows by dumpbin /dependents. The three hello
programs name the same library:
$ otool -L hello
hello:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1356.0.0)So the three binaries differ in what they carry, not in what they are. The C binary carries
almost nothing and depends on the system for everything. The Go binary carries its own
runtime. The Rust binary carries its standard library and depends on the system for the rest.
The sizes in the table at the top of the chapter follow from that. The programs print the
same word sizes too. An int is 4 bytes in C and 8 in Go. A pointer is 8 bytes in
all three. A Rust usize is the size of a pointer. The pointer size comes from the machine. The size of int is a choice each language made, and C and Go chose differently.
On Linux the C binary is an ELF file and links libc.so.6, the GNU C library.
The Go binary links nothing at all, because Go makes its system calls directly on Linux. On
Windows the C program links the Windows C runtime, and Rust links the system libraries it
needs. Go loads kernel32.dll and a few other system libraries at start, because Windows has no stable system call interface. The chapter on shipping explains what that means for copying
a binary from one machine to another.
Build tools, and the lack of one#
C has no build tool of its own. The convention is make, which Stuart Feldman wrote at Bell Labs in 1976. It
reads a file called Makefile. The file lists each output, the inputs it depends
on, and the command that makes it. The Makefile for the hello program is short:
CC ?= cc
CFLAGS ?= -std=c17 -Wall -Wextra -Werror -O2
hello: hello.c
$(CC) $(CFLAGS) -o $@ hello.c
# Two translation units. Each .c is compiled alone into a .o, and the linker joins them.
units: units.o greet.o
$(CC) $(CFLAGS) -o $@ units.o greet.o
units.o: units.c greet.h
$(CC) $(CFLAGS) -c units.c
greet.o: greet.c greet.h
$(CC) $(CFLAGS) -c greet.c
clean:
rm -rf hello units *.o hello.dSYM units.dSYM
.PHONY: cleanThe line hello: hello.c says that hello is made from
hello.c, and the indented line below says how. make runs the command
only when the input is newer than the output. The ?= sets a variable only when
the environment has not set it, which is how the book's test script adds sanitizer flags
later. Large C projects use CMake, which writes Makefiles for you from a description that
works on every platform. The Redis source, by comparison, is plain Make.
Eight spaces are not a tab. The command line under a rule must start with a
tab character, and an editor that converts tabs to spaces produces the error
missing separator. Feldman has said the tab was a workaround in the first version, kept so as not to break the first users. Every C programmer since has met it.
Go and Rust each come with a build tool, and each tool is the only one anyone uses. For Go
the description of the project is go.mod, and the build is go build.
For Rust it is Cargo.toml and cargo build. Both tools also run
tests, fetch dependencies, and format code. The difference from C is not that these tools are
cleverer than Make. It is that there is one of them, so every project looks the same.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| Installing | OK | Strong | Strong | A C compiler is one system command away on macOS and Linux. Go and Rust are one download each, on every platform. |
| Building | Weak | Strong | Strong | C has no build tool, only a convention. Go and Rust each have one tool that every project uses. |
| Seeing what you made | Strong | OK | OK | C shows every step from preprocessor to linker. Go and Rust hide the steps, which is usually what you want. |
| Binary size | Strong | Weak | OK | 8 KB for C, 2.5 MB for Go because the runtime is inside, 415 KB for Rust. |
| Running elsewhere | Weak | Strong | OK | The C binary needs the right C library on the target, the Go binary needs nothing, and Rust is in between. |
- Translation unit. A C file with its headers pasted in, compiled alone, and joined to the others by the linker.
- Zero value. A Go variable starts as zero, nil or empty, so a type is designed to be usable without a constructor.
- Edition. A Rust crate names the version of the language rules it is written for, and crates of different editions link together.
Echo server#
You have a Redis client and no Redis. Ask it anyway:
$ redis-cli -p 6380 PING
Could not connect to Redis at 127.0.0.1:6380: Connection refusedNothing is listening on the port. Every chapter from here on changes what the program on that port does. This one makes something listen.
A server that sends back what it receives. It is the smallest program that opens a port, accepts a client and moves bytes in both directions. Every later chapter adds to this one. The chapter teaches the socket calls that all three languages share, and how each language wraps them.
kv has to be reachable over the network. A client such as redis-cli connects to
a port, sends bytes, and reads bytes back. Before kv can understand a command it must be able to
hold that conversation at all. So the first version of kv is an echo server. Whatever the
client sends, the server sends back, one client at a time.
| The socket calls | C | Go | Rust |
|---|---|---|---|
| Open a port | socket, setsockopt, bind, listen | net.Listen("tcp", ":6380") | TcpListener::bind("0.0.0.0:6380") |
| Wait for a client | accept, returns an int | ln.Accept(), returns a net.Conn | listener.incoming(), yields a TcpStream |
| Read bytes | read(fd, buf, n), returns a count or -1 | conn.Read(buf), returns a count and an error | stream.read(&mut buf), returns a Result |
| Copy in to out | a loop you write, with a short-write loop inside | io.Copy(conn, conn) | io::copy(&mut reader, &mut writer) |
| Close | close(fd), which you must remember | defer conn.Close() | dropped at the end of scope |
| An error | -1 and a number in errno | a second return value | an Err you must handle or pass on |
The four calls in the first row are all there is. A socket is a file descriptor that the operating system connects to the network instead of to a disk. To bind is to give the socket an address and a port. To listen is to tell the operating system that this socket accepts connections, and how many may wait. To accept is to block until a client connects, and to get a second socket that talks to that one client. The first socket keeps listening. The second one is read and written like a file, and closed like one.
The calls come from 4.2BSD, the Berkeley version of Unix released in 1983, and are called
the Berkeley sockets API. The operating system standard POSIX, the Portable Operating System
Interface, adopted them. Windows copied them in 1992 under the name Winsock, with a few
changes. Go and Rust do not replace these calls. They call them. Underneath, net.Listen
and TcpListener::bind are the same four calls. So the C version of this chapter is also an explanation of the
other two.
C: four calls, and a way to clean up#
The port comes from an environment variable, so the tests can pick one that is free. The
C library function getenv returns a pointer to the value, or NULL
when the variable is not set. The pointer belongs to the environment, not to you, so you do
not free it:
/* The port comes from the environment, so the tests can pick a free one. */
static int kv_port(void) {
const char *s = getenv("KV_PORT");
if (s == NULL || *s == '\0') {
return 6380;
}
return atoi(s);
}Opening the port takes the four calls, and each of them can fail. The pattern below is the
most common shape in C that does more than one thing that can fail. Each failure prints its
reason and jumps to cleanup, which releases what was acquired so far:
/* Open a listening socket on every interface. Returns the socket, or -1 after
* printing the reason. The caller owns the socket and must close() it. */
static int kv_listen(int port) {
int fd = socket(AF_INET, SOCK_STREAM, 0);
if (fd < 0) {
perror("socket");
return -1;
}
/* Without this, a restart within a minute fails with "address in use". */
int yes = 1;
if (setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes) < 0) {
perror("setsockopt");
goto cleanup;
}
struct sockaddr_in addr;
memset(&addr, 0, sizeof addr);
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_ANY);
addr.sin_port = htons((uint16_t)port);
if (bind(fd, (struct sockaddr *)&addr, sizeof addr) < 0) {
perror("bind");
goto cleanup;
}
if (listen(fd, 16) < 0) {
perror("listen");
goto cleanup;
}
return fd;
cleanup:
close(fd);
return -1;
}Read the function from the bottom. There is one place where the socket is closed, and every
failure after the socket exists reaches it. The goto is not a sign of bad code
here. It is how C expresses "undo what I did" without exceptions and without destructors.
perror prints the name you give it, a colon, and a message. The message is the
text for the current value of errno. That is the global number which every
failed system call sets.
A function that acquires more than one resource has one label, cleanup, at the
end. Each acquisition that fails jumps there. The cleanup code releases resources in the
reverse order of acquisition. Each release must be safe to run when the resource was never
acquired. For example, set a descriptor to -1 first, and skip close when it is
still -1. The Linux kernel is
written this way, and so is Redis.
A server that you restart soon after it stopped fails to bind. When a connection over TCP, the Transmission
Control Protocol, closes, the operating system keeps its address in a waiting state for a
while. The wait stops late packets from reaching a new program. The option SO_REUSEADDR says that
your server may bind anyway. Every server sets it. Forget it and the second start of kv prints
bind: Address already in use.
errno: the number a failed call leaves behind#
Every system call in kv_listen reports failure the same way. It returns -1
and puts a number in errno, a variable that errno.h declares. The
number says what went wrong. EADDRINUSE is the port being taken, and
EINTR is a signal arriving mid-call. strerror turns the number
into words, and perror prints your label, a colon and those words, which is
what perror("bind") did above. Three rules keep errno honest.
Read it right after the call that failed, because the next call may overwrite it. Do not
read it after a call that succeeded, because a library function may set it even then and
never resets it to zero. And since C11 it is thread-local, so a thread in chapter 11 cannot
see another thread's number. Go and Rust carry the same number inside their error values,
and chapter 4 shows the three designs side by side.
Serving a client is a loop. read returns the number of bytes it put in the
buffer, zero when the client closed the connection, or -1 on an error. Writing those bytes
back needs a loop of its own, because write may send fewer bytes than you asked
when the network is busy. Both calls can also be interrupted by a signal, in which case
errno is EINTR and the right response is to try again.
Write kv_serve. Read into a 4 KB buffer until read returns zero or
an error. After each read, write everything back, and keep writing until every byte is out.
Treat EINTR as "try again" in both calls. The caller closes the client socket.
static void kv_serve(int client) {
char buf[4096];
for (;;) {
ssize_t n = read(client, buf, sizeof buf);
/* your code: handle n < 0 and n == 0, then write all n bytes back */
}
}Stuck?Show the full listing
/* Copy everything the client sends back to it, until the client closes. */
static void kv_serve(int client) {
char buf[4096];
for (;;) {
ssize_t n = read(client, buf, sizeof buf);
if (n < 0) {
if (errno == EINTR) {
continue;
}
perror("read");
return;
}
if (n == 0) {
return; /* the client closed the connection */
}
/* write() may send fewer bytes than asked, so loop until all are out */
ssize_t off = 0;
while (off < n) {
ssize_t w = write(client, buf + off, (size_t)(n - off));
if (w < 0) {
if (errno == EINTR) {
continue;
}
perror("write");
return;
}
off += w;
}
}
}The main function ties the pieces together. It accepts one client, serves it to the end, closes it, and accepts the next. A second client waits in the operating system's queue until the first one is done. Chapter 9 removes that limit:
int main(void) {
int port = kv_port();
int fd = kv_listen(port);
if (fd < 0) {
return 1;
}
fprintf(stderr, "kv listening on port %d\n", port);
for (;;) {
int client = accept(fd, NULL, NULL);
if (client < 0) {
if (errno == EINTR) {
continue;
}
perror("accept");
break;
}
kv_serve(client);
close(client);
}
close(fd);
return 1;
}Build and run it with the Makefile from chapter 2, which has grown a SANITIZE
variable for the tests:
CC ?= cc
CFLAGS ?= -std=c17 -Wall -Wextra -Werror -O2 -g
SANITIZE ?=
kv: kv.c
$(CC) $(CFLAGS) $(SANITIZE) -o $@ kv.c
clean:
rm -rf kv kv.dSYM
.PHONY: cleanThen talk to it with nc, the netcat tool that comes with
macOS and most Linux systems. Type a line and the server sends it back:
$ make
$ ./kv &
kv listening on port 6380
$ nc localhost 6380
hello
helloPress Ctrl-C to stop nc, and kill %1 to stop the
server.
Go: the loop you did not have to write#
The Go version has the same three parts. The port function reads the environment with
os.Getenv, which returns the empty string when the variable is unset. There is no
NULL to check, and nothing to free:
// The port comes from the environment, so the tests can pick a free one.
func port() string {
if p := os.Getenv("KV_PORT"); p != "" {
return p
}
return "6380"
}Serving a client is one call. io.Copy reads from its second argument and
writes to its first until the reader returns end of file. Its signature is
Copy(dst io.Writer, src io.Reader). An io.Reader is any type with
a Read method. An io.Writer is any type with a Write
method. A net.Conn has both, so copying a connection to itself is an echo. A
file, a buffer and a compressed stream have them too, which is why one copy function serves
them all. Chapter 7 names this mechanism. The short-write loop and the retry on interruption
are inside io.Copy and the runtime:
Write serve. It takes a net.Conn, closes it when the function
returns, copies the connection to itself, and returns the error from the copy. Use
defer for the close.
func serve(conn net.Conn) error {
// your code here
}Stuck?Show the full listing
// serve copies everything the client sends back to it, until the client closes.
func serve(conn net.Conn) error {
defer conn.Close()
_, err := io.Copy(conn, conn)
return err
}defer conn.Close() runs when the function returns, by whichever path. Write it
on the line after the connection is created, so that no later return can forget it. This is
the Go form of the C cleanup label, and it needs no label.
Two rules of defer#
defer has two rules that surprise everyone once. Deferred calls run in the
reverse order of their defer lines, last in first out, which is the order a
cleanup label releases things. And the arguments of a deferred call are evaluated at the
defer line, not when the call runs. A closure that reads a variable sees the
value at exit, while an argument carries the value at the defer. The test
below is in the chapter's snapshot, and go test runs it:
// Two rules of defer, as a test you can run with go test: the deferred calls run last in
// first out, and each one's arguments are evaluated at the defer line, not at the exit.
func deferOrder(log *[]string) {
n := 1
defer func() { *log = append(*log, "third, closure sees n =", itoa(n)) }()
defer func(v int) { *log = append(*log, "second, argument was", itoa(v)) }(n)
n = 2
*log = append(*log, "first, the body")
}
func TestDeferOrder(t *testing.T) {
var log []string
deferOrder(&log)
want := []string{"first, the body", "second, argument was", "1", "third, closure sees n =", "2"}
for i := range want {
if i >= len(log) || log[i] != want[i] {
t.Fatalf("log = %q, want %q", log, want)
}
}
}The main function listens and loops. net.Listen makes the four calls of the C version for you. It also sets
SO_REUSEADDR, which you can see in the file net/sockopt_bsd.go of
the Go source. An error from Accept is printed and the
loop continues, because one bad connection must not stop the server:
func main() {
ln, err := net.Listen("tcp", ":"+port())
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Fprintln(os.Stderr, "kv listening on port", port())
for {
conn, err := ln.Accept()
if err != nil {
fmt.Fprintln(os.Stderr, "accept:", err)
continue
}
if err := serve(conn); err != nil {
fmt.Fprintln(os.Stderr, "client:", err)
}
}
}$ go run .
kv listening on port 6380Go lets you drop an error on the floor. serve(conn) on a line
of its own compiles, and the error is lost. The compiler checks that a value you assign is
used, but not that a value you never assign is looked at. go vet does not catch
it either. A separate linter called errcheck does. This book does not add it, and writes every check out instead.
In this chapter the check is written out, as if err := serve(conn); err != nil.
Rust: a Result at every step#
In Rust the environment variable is a Result. std::env::var returns
Ok(value) or an error when the variable is unset or not valid text.
unwrap_or_else turns the error case into the default:
/// The port comes from the environment, so the tests can pick a free one.
fn port() -> String {
std::env::var("KV_PORT").unwrap_or_else(|_| "6380".to_string())
}Serving a client uses io::copy, which does the same as io.Copy in Go, with the arguments in the other order. There is one new idea. A TcpStream is one value,
and the ownership rules allow one mutable use of it at a time. The simplest way to get two handles on it is to clone the stream. try_clone asks the
operating system for a second descriptor to the same connection, and that can fail, so it
returns a Result too:
Write serve. Clone the stream with try_clone, copy from the clone
to the original with io::copy, and return Ok(()). Use ?
after each call that returns a Result, so that an error returns from the function
at once.
fn serve(stream: TcpStream) -> io::Result<()> {
// your code here
}Stuck?Show the full listing
/// Copy everything the client sends back to it, until the client closes.
fn serve(stream: TcpStream) -> io::Result<()> {
let mut reader = stream.try_clone()?;
let mut writer = stream;
io::copy(&mut reader, &mut writer)?;
Ok(())
}A main that returns a Result#
The main function returns io::Result<()>, so ? works inside it
and a failure to bind prints the error and exits with a non-zero status. The loop uses
incoming(), which yields one Result per connection attempt. A match
handles both arms, and the compiler would refuse the program if one arm were missing:
fn main() -> io::Result<()> {
let port = port();
let listener = TcpListener::bind(format!("0.0.0.0:{port}"))?;
eprintln!("kv listening on port {port}");
for stream in listener.incoming() {
match stream {
Ok(stream) => {
if let Err(e) = serve(stream) {
eprintln!("client: {e}");
}
}
Err(e) => eprintln!("accept: {e}"),
}
}
Ok(())
}$ cargo run --release
kv listening on port 6380Nothing closes the stream, and nothing needs to. When serve returns, the
TcpStream it owns goes out of scope, and Rust runs its destructor, which calls
close. The clone is closed the same way. This is the third way to do what
cleanup did in C and defer did in Go. It is the only one of the
three that cannot be forgotten.
The compiler warns about an unused Result and then compiles the program.
serve(stream); on a line of its own produces
warning: unused Result that must be used. The book's build turns warnings into
errors with cargo clippy -- -D warnings, so in this book the line does not
compile. Without that flag it does, and the error is lost as in Go. The difference from Go is
that Rust told you.
?, expect and unwrap#
Three ways to take a value out of a Result appear in every Rust program,
and each one is a decision. ? passes the error to the caller, and is the
default. expect("the reason this cannot fail") stops the program with that
message when it does fail. It is for a failure that would mean a bug, and the message says
why you believed that. unwrap() is expect with no reason, and
belongs in examples and tests. kv uses ? almost everywhere. In
port above, unwrap_or_else is a fourth way, which gives a default
instead of failing.
Errors, three ways#
The three versions of serve differ most in how they say that something went
wrong, and the difference runs through the whole book. In C a function returns -1, and the
reason is a number in the global errno. The caller must know to check, and must
check before any other call overwrites the number. In Go a function returns the error as a
second value, and nil means success. The caller must still choose to look. In
Rust a function returns Result, which is either Ok(value) or
Err(reason), and the value is inside. To use the value at all, the caller must
take it out. The ways to take it out are match, ?, and
unwrap. The last one crashes on an error and is for examples only. Chapter 4 builds a parser in all three styles,
where the difference is larger.
Winsock has the same calls with the same names, with three differences that matter. A program must call WSAStartup before any of them. A socket is a handle of its own type, not an int, and it is closed with closesocket. And the option SO_REUSEADDR means something stronger on
Windows, where it allows two programs to bind the same port. The C version of kv is written
for POSIX and does not build on Windows without changes. The Go and Rust versions build there unchanged, because their standard libraries contain the differences.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| Opening a port | Weak | Strong | Strong | C needs four calls, a struct and an option. Go and Rust need one call that sets the option for you. |
| Closing it again | Weak | OK | Strong | C needs a cleanup label and Go needs a defer you must write, but Rust closes at the end of scope by itself. |
| The echo loop | Weak | Strong | Strong | C needs the short-write loop and the EINTR retry. Both standard libraries contain them. |
| A lost error | Weak | Weak | OK | C and Go let a return value drop. Rust warns, and with the book's flags refuses. |
| Seeing what happens | Strong | OK | OK | The C version shows every system call. The other two hide them, which is what you want on most days. |
| Windows | Weak | Strong | Strong | The C code needs Winsock changes. Go and Rust build unchanged. |
| What you must learn first | Strong | OK | Weak | C is the calls themselves. Rust asks for Result, ?, ownership and try_clone before an echo works. |
- errno. A failed C call returns -1 and leaves a number you must read at once.
- defer. Runs at function exit, last in first out, with its arguments taken at the defer line.
- Result. The value is inside, and
?,expectandmatchare the ways out.
Speaking RESP#
Start the chapter 3 server and point the real client at it:
$ redis-cli -p 6380 PING
1) "PING"
$ redis-cli -p 6380 SET a 1
1) "SET"
2) "a"
3) "1"The client asked for PONG and got its own question back, as a list. The
server echoed the bytes the client sent, and the client read them as a reply. Whatever those
bytes are, kv has to understand them before it can answer.
Redis clients talk in a small wire format called RESP, the Redis serialization protocol.
This chapter teaches kv to read it and write it. After this chapter the real
redis-cli can say PING to your server and get PONG
back. The chapter also compares how the three languages say "something went wrong", because a
parser is where that question is asked most often.
The echo server sends bytes back. redis-cli does not want its bytes back. It
wants an answer in the format it expects. When you type ECHO hi into
redis-cli, it sends these 22 bytes to the server:
*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\nWritten with one element per line, the request reads as follows. An array of 2 bulk
strings. The first has 4 bytes, ECHO. The second has 2 bytes, hi. The server
replies with a bulk string of its own, $2\r\nhi\r\n. Every message in RESP is
built from five types, each marked by its first byte:
+OK\r\n a simple string, for short replies such as OK and PONG
-ERR unknown command\r\n an error, which the client shows as an error
:42\r\n an integer
$5\r\nhello\r\n a bulk string: the length, then exactly that many bytes
*2\r\n... an array: the count, then that many messages
$-1\r\n the null bulk string, which GET returns for a missing keyThe parts are separated by \r\n, a carriage return and a line feed, which this
chapter calls a line ending. A bulk string carries its length in front, so the bytes inside
can be anything, including line endings. That one rule is what makes the format both readable
and fast to parse. There is no quoting and no escaping.
The Redis protocol specification names its three aims as "Simple to implement. Fast to
parse. Human readable." The length in front of a bulk string means the parser never scans the
payload for special characters, as a JSON parser must. The second version of the protocol,
RESP2, became the standard in Redis 2.0. A third version, RESP3, adds types such as maps and
a null of its own. A client chooses it with the HELLO command. kv speaks
RESP2, which every client still speaks. The specification also allows an inline command. That
is a plain line of words, so that a person with only telnet can talk to a server. Those work in kv too, which is why nc still works after this chapter.
| Reading a request | C | Go | Rust |
|---|---|---|---|
| Where the bytes wait | a buffer you grow with realloc | a bufio.Reader | a BufReader |
| Reading a line | memchr for the line feed | r.ReadBytes('\n') | r.read_until(b'\n', &mut line) |
| Reading n bytes | check the length, then take a pointer | io.ReadFull(r, buf) | r.read_exact(&mut buf) |
| The arguments | pointers into the buffer, no copy | [][]byte, one copy | Vec<Vec<u8>>, one copy |
| Not enough bytes yet | the status KV_PARSE_INCOMPLETE | the reader blocks until they arrive | the reader blocks until they arrive |
| Bytes that are not RESP | the status KV_PARSE_BAD | the error value errProtocol | an io::Error of kind InvalidData |
| The client went away | read returns 0 | io.EOF | Ok(None) |
The table shows the first large difference between C and the other two. Go and Rust read from a buffered reader that blocks until the bytes arrive, so the parser can ask for a line and wait. C has stdio, a buffered reader built for text files, and kv does not use it. The event loop of chapter 9 needs to see the bytes that have arrived so far. The C parser works on whatever bytes have arrived so far, and says "incomplete" when the request is not all there. That design is more work now. In chapter 9 it is the design the C event loop needs.
C: a buffer, a parser and three answers#
C needs a place to keep bytes between reads, because one read may return half
a request, or two and a half. The structure below is a growable byte buffer. The comment on
data says who owns it. The book writes such a comment on every pointer in a
structure, because nothing else in C will tell you:
/* A growable byte buffer. It owns data, and kv_buf_free releases it. */
typedef struct {
char *data; /* owned; NULL until the first append */
size_t len;
size_t cap;
} kv_buf;
static bool kv_buf_reserve(kv_buf *b, size_t extra) {
if (b->len + extra <= b->cap) {
return true;
}
size_t cap = b->cap ? b->cap * 2 : 1024;
while (cap < b->len + extra) {
cap *= 2;
}
char *p = realloc(b->data, cap);
if (p == NULL) {
return false; /* the old block is still valid and still ours */
}
b->data = p;
b->cap = cap;
return true;
}
static bool kv_buf_append(kv_buf *b, const void *src, size_t n) {
if (!kv_buf_reserve(b, n)) {
return false;
}
memcpy(b->data + b->len, src, n);
b->len += n;
return true;
}
/* Drop the first n bytes. memmove allows the two ranges to overlap. */
static void kv_buf_consume(kv_buf *b, size_t n) {
memmove(b->data, b->data + n, b->len - n);
b->len -= n;
}
static void kv_buf_free(kv_buf *b) {
free(b->data);
b->data = NULL;
b->len = b->cap = 0;
}Look at kv_buf_reserve. When realloc fails it returns
NULL and leaves the old block alone, so the code must not write the result into
b->data before checking it. Writing b->data = realloc(b->data, cap)
is a common mistake that loses the buffer on failure. kv_buf_consume uses
memmove, not memcpy, because the source and the destination overlap.
memcpy on overlapping memory is undefined behaviour.
A parsed command is a small structure of pointers into the buffer. No bytes are copied. The cost of that choice is a rule. The pointers are valid only until the buffer changes, and the comment says so. The parser returns one of three statuses, and the names of the statuses are the whole design:
/* One parsed request. The arguments point into the request buffer, so they are valid
* only until that buffer is consumed. A kv_cmd owns nothing. */
typedef struct {
size_t argc;
const char *argv[KV_MAX_ARGS];
size_t arglen[KV_MAX_ARGS];
} kv_cmd;
typedef enum {
KV_PARSE_OK, /* cmd is filled in, and *used bytes were taken */
KV_PARSE_INCOMPLETE, /* read more bytes and try again */
KV_PARSE_BAD /* the bytes are not RESP: close the connection */
} kv_parse_status;Two helpers find a line ending and read a number. The number reader refuses more than 18
digits, so that the multiplication cannot overflow a 64-bit long. Signed overflow is
undefined behaviour in C, and a length field is exactly where an attacker would put a large
number. A third helper splits an inline line into words:
/* The '\n' that ends the line starting at p, or NULL when it has not arrived yet. */
static const char *kv_find_eol(const char *p, const char *end) {
return memchr(p, '\n', (size_t)(end - p));
}
/* A decimal number between p and the line end. False when the text is not a number. */
static bool kv_parse_number(const char *p, const char *eol, long *out) {
const char *stop = eol;
if (stop > p && stop[-1] == '\r') {
stop--;
}
if (p == stop || stop - p > 18) {
return false;
}
long v = 0;
for (; p < stop; p++) {
if (*p < '0' || *p > '9') {
return false;
}
v = v * 10 + (*p - '0');
}
*out = v;
return true;
}
/* An inline command: words separated by spaces, as typed into nc or telnet. */
static kv_parse_status kv_parse_inline(const char *data, const char *eol, kv_cmd *cmd,
size_t *used) {
const char *p = data;
const char *stop = eol;
if (stop > p && stop[-1] == '\r') {
stop--;
}
while (p < stop) {
while (p < stop && *p == ' ') {
p++;
}
if (p == stop) {
break;
}
const char *word = p;
while (p < stop && *p != ' ') {
p++;
}
if (cmd->argc == KV_MAX_ARGS) {
return KV_PARSE_BAD;
}
cmd->argv[cmd->argc] = word;
cmd->arglen[cmd->argc] = (size_t)(p - word);
cmd->argc++;
}
*used = (size_t)(eol + 1 - data);
return KV_PARSE_OK;
}kv.c sets three limits. KV_MAX_ARGS is 64 arguments and KV_MAX_BULK is 1 MB for one bulk string. KV_MAX_REQUEST is twice that. The Go and Rust files set the same two limits under the names maxBulk and MAX_BULK.
A header that says $999999999 asks the server to allocate a
gigabyte before one byte of payload has arrived. A parser that trusts the number
does it, and a few such connections empty the machine. All three parsers in this chapter
refuse a length over the limit before they read anything. The protocol test
04-limits.test sends that header to each of them. Redis has the same limit,
proto-max-bulk-len, set to 512 MB.
Write the loop in kv_parse that reads the n bulk strings after the
array header. For each one, check that the next byte is $ and find its line ending.
Read the length. Then check that the whole string and its line ending have arrived. Return
KV_PARSE_INCOMPLETE when bytes are missing and KV_PARSE_BAD when a
byte is wrong. Store a pointer and a length in cmd, and set *used
to the number of bytes the request took. Treat a length above KV_MAX_BULK as bad.
static kv_parse_status kv_parse(const char *data, size_t len, kv_cmd *cmd, size_t *used) {
const char *end = data + len;
const char *p = data;
cmd->argc = 0;
if (len == 0) return KV_PARSE_INCOMPLETE;
const char *eol = kv_find_eol(p, end);
if (eol == NULL) return KV_PARSE_INCOMPLETE;
if (*p != '*') return kv_parse_inline(data, eol, cmd, used);
long n;
if (!kv_parse_number(p + 1, eol, &n) || n > KV_MAX_ARGS) return KV_PARSE_BAD;
p = eol + 1;
for (long i = 0; i < n; i++) {
/* your code here */
}
cmd->argc = (size_t)n;
*used = (size_t)(p - data);
return KV_PARSE_OK;
}Stuck?Show the full listing
/* Parse one request from data[0..len). The request is an array of bulk strings, such as
* "*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n", or one inline line. */
static kv_parse_status kv_parse(const char *data, size_t len, kv_cmd *cmd, size_t *used) {
const char *end = data + len;
const char *p = data;
cmd->argc = 0;
if (len == 0) {
return KV_PARSE_INCOMPLETE;
}
const char *eol = kv_find_eol(p, end);
if (eol == NULL) {
return KV_PARSE_INCOMPLETE;
}
/* The first byte says what the client sent. A client sends arrays. The other four
* type bytes are replies, which only a server sends, so they are not a request. */
switch (*p) {
case '*':
break;
case '+':
case '-':
case ':':
case '$':
return KV_PARSE_BAD;
default:
return kv_parse_inline(data, eol, cmd, used);
}
long n;
if (!kv_parse_number(p + 1, eol, &n) || n > KV_MAX_ARGS) {
return KV_PARSE_BAD;
}
p = eol + 1;
for (long i = 0; i < n; i++) {
if (p >= end) {
return KV_PARSE_INCOMPLETE;
}
if (*p != '$') {
return KV_PARSE_BAD;
}
eol = kv_find_eol(p, end);
if (eol == NULL) {
return KV_PARSE_INCOMPLETE;
}
long blen;
if (!kv_parse_number(p + 1, eol, &blen) || blen > KV_MAX_BULK) {
return KV_PARSE_BAD;
}
p = eol + 1;
if (end - p < blen + 2) {
return KV_PARSE_INCOMPLETE;
}
if (p[blen] != '\r' || p[blen + 1] != '\n') {
return KV_PARSE_BAD;
}
cmd->argv[i] = p;
cmd->arglen[i] = (size_t)blen;
p += blen + 2;
}
cmd->argc = (size_t)n;
*used = (size_t)(p - data);
return KV_PARSE_OK;
}switch, and the cases that run into each other#
Look at the top of the solution. The first byte of a request decides what it is, and
the decision is a switch on that byte. A switch compares one
integer against a list of constant cases and jumps to the one that matches. The four reply
markers are stacked with no break between them. Any one of the four reaches
the same return. That is fallthrough, and it is the feature of switch that
bites. Forget a break after a case that has a body, and the next case runs
too. The compiler says nothing, which is why every case in kv ends in a
break, a return or a continue.
The serving loop has a second switch, on the parse status, and that one
has no default on purpose. The status is an enum, a set of named
integers. With -Wall, the compiler reports a switch on an enum
that leaves a name out. Add a fourth status next year and every
switch that forgot it fails to build. A default would hide that.
The same trick does not work for a chain of if statements, which is why the
loop changed shape in this chapter.
A pointer into a buffer dies when the buffer grows. realloc
may move the block, and every pointer into the old block then points at freed memory. The
kv_cmd holds such pointers. kv is safe because it runs the command and consumes
the buffer before the next read appends to it. Move one line and it is not safe,
and the compiler will not say a word. Chapter 12 shows AddressSanitizer catching exactly this.
Replies are appended to a second buffer. Each reply function returns false
when memory runs out, and the callers pass that on:
static bool kv_reply_line(kv_buf *out, char first, const char *s) {
return kv_buf_append(out, &first, 1) && kv_buf_append(out, s, strlen(s)) &&
kv_buf_append(out, "\r\n", 2);
}
static bool kv_reply_simple(kv_buf *out, const char *s) { return kv_reply_line(out, '+', s); }
static bool kv_reply_error(kv_buf *out, const char *s) { return kv_reply_line(out, '-', s); }
static bool kv_reply_bulk(kv_buf *out, const char *s, size_t n) {
char head[32];
int h = snprintf(head, sizeof head, "$%zu\r\n", n);
return kv_buf_append(out, head, (size_t)h) && kv_buf_append(out, s, n) &&
kv_buf_append(out, "\r\n", 2);
}
static bool kv_reply_array(kv_buf *out, size_t n) {
char head[32];
int h = snprintf(head, sizeof head, "*%zu\r\n", n);
return kv_buf_append(out, head, (size_t)h);
}A function that takes any number of arguments#
Two error messages need a value inside them, the command name in one and the unknown
word in the other. printf solves this with a format and any number of
arguments, and C lets you write such a function yourself. The ... in the
parameter list is the variadic part. va_list from stdarg.h walks
the arguments. va_start points it after the last named parameter. And
vsnprintf is the snprintf that takes a va_list
instead of the arguments themselves:
/* An error reply with a format, as printf would write it. The ... takes any arguments,
* va_list walks them, and vsnprintf is the printf that accepts a va_list. */
static bool kv_reply_errorf(kv_buf *out, const char *fmt, ...) {
char msg[128];
va_list ap;
va_start(ap, fmt);
vsnprintf(msg, sizeof msg, fmt, ap);
va_end(ap);
return kv_reply_error(out, msg);
}The compiler cannot check the arguments of a variadic function against the format,
because the function's type says nothing about them. Both gcc and clang check
printf itself by name. An attribute, __attribute__((format)),
asks for the same check on yours. It is a compiler extension, not part of the standard,
and kv does without it. Every format in kv is a literal in the call, so the mistake the
check would catch cannot happen.
Running a command is a chain of comparisons on the first argument. Command names in Redis
are case-insensitive, so the comparison uses strncasecmp from
strings.h, with the length, because the argument is not a C string. It has no
terminating zero byte. That is a theme of the next chapter:
/* True when argument i of the command is name, ignoring case. */
static bool kv_arg_is(const kv_cmd *c, size_t i, const char *name) {
size_t n = strlen(name);
return c->arglen[i] == n && strncasecmp(c->argv[i], name, n) == 0;
}
static bool kv_reply_arity(kv_buf *out, const char *name) {
return kv_reply_errorf(out, "ERR wrong number of arguments for '%s' command", name);
}
/* Run one command and append its reply to out. */
static bool kv_exec(const kv_cmd *c, kv_buf *out) {
if (c->argc == 0) {
return kv_reply_error(out, "ERR empty command");
}
if (kv_arg_is(c, 0, "PING")) {
if (c->argc == 1) {
return kv_reply_simple(out, "PONG");
}
if (c->argc == 2) {
return kv_reply_bulk(out, c->argv[1], c->arglen[1]);
}
return kv_reply_arity(out, "ping");
}
if (kv_arg_is(c, 0, "ECHO")) {
if (c->argc != 2) {
return kv_reply_arity(out, "echo");
}
return kv_reply_bulk(out, c->argv[1], c->arglen[1]);
}
if (kv_arg_is(c, 0, "COMMAND")) {
return kv_reply_array(out, 0); /* redis-cli asks on connect; an empty list is fine */
}
int n = c->arglen[0] < 40 ? (int)c->arglen[0] : 40;
return kv_reply_errorf(out, "ERR unknown command '%.*s'", n, c->argv[0]);
}The serving loop ties it together. After each read it runs every complete
request in the buffer, because a client may send several at once, which Redis calls a
pipeline. Then it writes all the replies in one call. A request that grows past
KV_MAX_REQUEST without ending is treated as bad, so that a client cannot make
the server grow its buffer without limit. The write loop from chapter 3 becomes a function first:
/* write() may send fewer bytes than asked, so loop until all are out. */
static bool kv_write_all(int fd, const char *p, size_t n) {
size_t off = 0;
while (off < n) {
ssize_t w = write(fd, p + off, n - off);
if (w < 0) {
if (errno == EINTR) {
continue;
}
perror("write");
return false;
}
off += (size_t)w;
}
return true;
}/* Read requests, run them, send the replies, until the client closes. */
static void kv_serve(int client) {
kv_buf in = {0};
kv_buf out = {0};
char tmp[4096];
for (;;) {
ssize_t n = read(client, tmp, sizeof tmp);
if (n < 0) {
if (errno == EINTR) {
continue;
}
perror("read");
break;
}
if (n == 0) {
break; /* the client closed the connection */
}
if (!kv_buf_append(&in, tmp, (size_t)n)) {
break;
}
/* A pipeline holds several requests in one read, so loop until one is incomplete. */
bool bad = false;
for (;;) {
kv_cmd cmd;
size_t used = 0;
kv_parse_status st = kv_parse(in.data, in.len, &cmd, &used);
/* No default: with -Wall the compiler reports a status this switch forgot. */
switch (st) {
case KV_PARSE_INCOMPLETE:
bad = in.len > KV_MAX_REQUEST; /* a request that never ends is not a request */
break;
case KV_PARSE_BAD:
bad = true;
break;
case KV_PARSE_OK:
if (!kv_exec(&cmd, &out)) {
bad = true;
}
kv_buf_consume(&in, used);
continue;
}
break;
}
if (bad) {
kv_reply_error(&out, "ERR Protocol error");
}
if (out.len > 0 && !kv_write_all(client, out.data, out.len)) {
break;
}
out.len = 0;
if (bad) {
break;
}
}
kv_buf_free(&in);
kv_buf_free(&out);
}Build it, run it, and talk to it with the real client:
$ make && ./kv &
$ redis-cli -p 6380 PING
PONG
$ redis-cli -p 6380 ECHO "hello world"
"hello world"
$ redis-cli -p 6380 FOO
(error) ERR unknown command 'FOO'A pointer in C carries no information about who frees it or how long it stays valid. So
the comment must. kv_buf says "owned". kv_cmd says "valid only until
the buffer is consumed". A function that returns a pointer says whether the caller must free
it. Redis does this, the Linux kernel does this, and so does every C codebase that has
survived long enough to be read by strangers.
Go: a reader that waits, and an error that names itself#
Go has bufio.Reader, which reads ahead into its own buffer and gives you
lines or exact counts of bytes. The parser becomes a set of calls that block until the bytes
are there. The "incomplete" status disappears, because the reader waits instead. The first
helper reads a line and removes its ending:
// readLine returns one line without its line ending. A bare LF is accepted as well as
// CRLF, so that a command typed into nc works. io.EOF means the client closed the
// connection cleanly between commands.
func readLine(r *bufio.Reader) ([]byte, error) {
line, err := r.ReadBytes('\n')
if err != nil {
if err == io.EOF && len(line) > 0 {
return nil, io.ErrUnexpectedEOF
}
return nil, err
}
line = bytes.TrimSuffix(line, []byte("\n"))
return bytes.TrimSuffix(line, []byte("\r")), nil
}Two errors from the standard library carry meaning here. io.EOF means the
connection closed, and readLine returns it unchanged when it happens between
requests, which is the normal way for a client to leave. io.ErrUnexpectedEOF
means the connection closed in the middle of something, which is not normal. A third error,
errProtocol, is this file's own. It is a package-level variable, and a caller
tests for it with errors.Is. Such a value is called a sentinel error.
An error is any value with an Error method#
There is nothing special about an error in Go. error is an interface with
one method, Error() string, and any type with that method is an error. A
sentinel is the smallest one, a value with a fixed message. Sometimes a caller needs more
than the fact of a failure, such as which byte was wrong. Then the error becomes a type of
your own with fields:
// errProtocol means the bytes were not RESP. The connection is closed after it. A caller
// that only needs to know that much tests errors.Is(err, errProtocol).
var errProtocol = errors.New("protocol error")
// A ProtocolError says which byte was wrong. Unwrap returns the sentinel, so errors.Is still
// finds errProtocol in it, and errors.As gives a caller the byte.
type ProtocolError struct {
Byte byte
}
func (e *ProtocolError) Error() string { return fmt.Sprintf("protocol error at byte %q", e.Byte) }
func (e *ProtocolError) Unwrap() error { return errProtocol }
func protoErr(b byte) error { return &ProtocolError{Byte: b} }Three things in that block are the whole of Go error handling. Error makes
*ProtocolError an error. Unwrap says that a
ProtocolError contains the sentinel, so errors.Is(err, errProtocol)
is still true for one. And errors.Is and errors.As walk that
chain of Unwrap calls. Is asks whether a given value is in the
chain. As asks whether a value of a given type is, and gives it to you. In the
reader, every error from the standard library is wrapped with what kv was doing. The
wrapping is fmt.Errorf with the %w verb, which adds a message and
keeps the cause in the chain.
Write the loop in readCommand that reads n bulk strings. For each
one, read a line and check that it starts with $. Convert the rest to a size with
strconv.Atoi. Then read exactly size+2 bytes with
io.ReadFull. Check that the last two bytes are the line ending. Return
errProtocol for a wrong byte, and turn an io.EOF inside a request
into io.ErrUnexpectedEOF. Treat a size above maxBulk as bad. for range n runs the body n times, and is new in Go 1.22.
func readCommand(r *bufio.Reader) ([][]byte, error) {
line, err := readLine(r)
if err != nil {
return nil, err
}
if len(line) == 0 || line[0] != '*' {
return bytes.Fields(line), nil
}
n, err := strconv.Atoi(string(line[1:]))
if err != nil || n < 0 || n > maxArgs {
return nil, errProtocol
}
args := make([][]byte, 0, n)
for range n {
// your code here
}
return args, nil
}Stuck?Show the full listing
// readCommand reads one request: an array of bulk strings such as
// "*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n", or one inline line of words. Every error from the
// reader is wrapped with what kv was doing, and the cause stays reachable with errors.Is.
func readCommand(r *bufio.Reader) ([][]byte, error) {
line, err := readLine(r)
if err != nil {
return nil, fmt.Errorf("read command: %w", err)
}
if len(line) == 0 || line[0] != '*' {
return bytes.Fields(line), nil
}
n, err := strconv.Atoi(string(line[1:]))
if err != nil || n < 0 || n > maxArgs {
return nil, protoErr('*')
}
args := make([][]byte, 0, n)
for range n {
line, err := readLine(r)
if err != nil {
return nil, fmt.Errorf("read argument: %w", unexpected(err))
}
if len(line) == 0 || line[0] != '$' {
return nil, protoErr(first(line))
}
size, err := strconv.Atoi(string(line[1:]))
if err != nil || size < 0 || size > maxBulk {
return nil, protoErr('$')
}
buf := make([]byte, size+2)
if _, err := io.ReadFull(r, buf); err != nil {
return nil, fmt.Errorf("read argument: %w", unexpected(err))
}
if buf[size] != '\r' || buf[size+1] != '\n' {
return nil, protoErr(buf[size])
}
args = append(args, buf[:size])
}
return args, nil
}
// unexpected turns a clean EOF in the middle of a request into an error that says so.
func unexpected(err error) error {
if err == io.EOF {
return io.ErrUnexpectedEOF
}
return err
}
func first(line []byte) byte {
if len(line) == 0 {
return '\n'
}
return line[0]
}Writing replies uses a bufio.Writer, which collects small writes and sends
them in one system call when it is flushed:
func writeSimple(w *bufio.Writer, s string) error {
_, err := w.WriteString("+" + s + "\r\n")
return err
}
func writeError(w *bufio.Writer, s string) error {
_, err := w.WriteString("-" + s + "\r\n")
return err
}
func writeBulk(w *bufio.Writer, b []byte) error {
if _, err := w.WriteString("$" + strconv.Itoa(len(b)) + "\r\n"); err != nil {
return err
}
if _, err := w.Write(b); err != nil {
return err
}
_, err := w.WriteString("\r\n")
return err
}
func writeArrayLen(w *bufio.Writer, n int) error {
_, err := w.WriteString("*" + strconv.Itoa(n) + "\r\n")
return err
}The command switch is the C chain of comparisons, written as a switch on an
upper-cased string. Go strings are immutable and can hold any bytes, so converting an
argument to a string costs one copy and nothing else:
// execute runs one command and writes its reply.
func execute(args [][]byte, w *bufio.Writer) error {
if len(args) == 0 {
return writeError(w, "ERR empty command")
}
name := strings.ToUpper(string(args[0]))
switch name {
case "PING":
switch len(args) {
case 1:
return writeSimple(w, "PONG")
case 2:
return writeBulk(w, args[1])
}
return writeArity(w, "ping")
case "ECHO":
if len(args) != 2 {
return writeArity(w, "echo")
}
return writeBulk(w, args[1])
case "COMMAND":
return writeArrayLen(w, 0) // redis-cli asks on connect; an empty list is fine
}
shown := string(args[0])
if len(shown) > 40 {
shown = shown[:40]
}
return writeError(w, "ERR unknown command '"+shown+"'")
}
func writeArity(w *bufio.Writer, name string) error {
return writeError(w, "ERR wrong number of arguments for '"+name+"' command")
}The serving loop reads a request, runs it, and flushes the writer when the reader has no bytes waiting. That last check is the Go form of the C pipeline loop. While a pipeline is still being read, replies collect in the writer, and go out together:
// serve reads requests, runs them and sends the replies, until the client closes.
func serve(conn net.Conn) error {
defer conn.Close()
r := bufio.NewReader(conn)
w := bufio.NewWriter(conn)
for {
args, err := readCommand(r)
// The error is wrapped, so == would miss io.EOF. errors.Is walks the chain.
if errors.Is(err, io.EOF) {
return nil
}
// errors.AsType finds the typed error in the chain and gives us its byte. Before
// Go 1.26 the same check was errors.As(err, &pe) with a declared *ProtocolError.
if pe, ok := errors.AsType[*ProtocolError](err); ok {
writeError(w, "ERR Protocol error")
w.Flush()
return fmt.Errorf("closing client: %w", pe)
}
if err != nil {
return err
}
if err := execute(args, w); err != nil {
return err
}
// A pipeline holds several requests; flush once the reader has nothing waiting.
if r.Buffered() == 0 {
if err := w.Flush(); err != nil {
return err
}
}
}
}The function returns nil on a clean close, and the error otherwise. On a
protocol error it first tells the client why. The three tests on err are the
Go way of handling three different failures, and the order matters. The specific checks come
before the general err != nil. Two of them changed shape in this chapter.
err == io.EOF became errors.Is(err, io.EOF), because the reader
now wraps its errors, and a wrapped io.EOF is not equal to the bare one. And
errors.AsType[*ProtocolError](err) pulls the typed error out of the chain so
the log line can name the byte. AsType is new in Go 1.26. Before it the same
check was errors.As(err, &pe) with a declared pe, and you
will read that form in most code.
The moment any function in the chain wraps with %w, every
err == io.EOF above it becomes false. The comparison sees the outer
value. errors.Is sees the chain. In this book every comparison with a sentinel
is errors.Is, including the ones that would work today with ==.
The function below may start wrapping tomorrow. The replay in chapter 11 is where that
caught the author.
A caller that only needs to know that something went wrong tests err != nil.
A caller that must tell one failure from another compares with errors.Is, so
define a sentinel for that failure. A caller that needs the details, a byte, a path, a
count, gets a type with fields and finds it with errors.As. Give the type an
Unwrap that returns the sentinel, and both callers are served by one value.
errors.Join combines several errors into one that Is and
As search through, for the case where a loop collects failures.
Rust: an Option for the end, a Result for the rest#
The Rust reader is the BufRead trait, and read_until does the same as ReadBytes in Go. The line function returns io::Result<Option<Vec<u8>>>,
which is a mouthful and exactly right. The outer Result is "did the read fail".
The inner Option is "was there a line at all". A clean close is
Ok(None), and nobody has to remember that a particular error value means
"not really an error":
/// One line without its line ending. A bare LF is accepted as well as CRLF, so that a
/// command typed into nc works. `Ok(None)` means the client closed the connection
/// cleanly between commands.
fn read_line(r: &mut impl BufRead) -> Result<Option<Vec<u8>>> {
let mut line = Vec::new();
if r.read_until(b'\n', &mut line)? == 0 {
return Ok(None);
}
if line.pop() != Some(b'\n') {
return Err(io::Error::from(io::ErrorKind::UnexpectedEof).into());
}
if line.last() == Some(&b'\r') {
line.pop();
}
Ok(Some(line))
}
fn parse_len(digits: &[u8], type_byte: u8) -> Result<usize> {
std::str::from_utf8(digits)
.ok()
.and_then(|s| s.parse().ok())
.ok_or(KvError::Protocol(type_byte))
}The ? after read_until returns any read error at once, and
it does one more thing, which the next section explains. parse_len turns the
digits into a number in two steps. Each step returns an Option, and
and_then chains them. The final ok_or turns a missing number
into the protocol error, with the type byte that was being read.
One enum for everything that can go wrong#
Two kinds of failure can come out of the reader, a failed read and bytes that are not
RESP, and they need different answers. In Rust the two become one enum, and
each variant carries what its caller needs. Io holds the
io::Error itself. Protocol holds the byte that was wrong. A
match on the enum must handle both, and the compiler refuses the program if a
third variant is added and one match forgets it:
/// Everything that can go wrong while reading a request. Each variant carries what the
/// caller needs to know: the io error itself, or the byte that was not RESP.
#[derive(Debug)]
pub enum KvError {
Io(io::Error),
Protocol(u8),
}
/// How the error prints, with `{}`. `to_string` and `eprintln!` use this.
impl fmt::Display for KvError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
KvError::Io(e) => write!(f, "io error: {e}"),
KvError::Protocol(b) => write!(f, "protocol error at byte {:?}", *b as char),
}
}
}
/// Joining the standard error trait lets a `KvError` sit in a `Box<dyn Error>`.
impl std::error::Error for KvError {}
/// `?` calls this when an `io::Error` must become a `KvError`. Without it, every read in
/// this file would need `.map_err(KvError::Io)`.
impl From<io::Error> for KvError {
fn from(e: io::Error) -> Self {
KvError::Io(e)
}
}
/// The other direction, so a function that returns `io::Result` can use `?` on ours.
impl From<KvError> for io::Error {
fn from(e: KvError) -> Self {
match e {
KvError::Io(e) => e,
KvError::Protocol(_) => io::Error::new(io::ErrorKind::InvalidData, e.to_string()),
}
}
}
pub type Result<T> = std::result::Result<T, KvError>;The four impl blocks are the standard shape of an error type, and every
Rust error you meet has them. Display says how the error prints with
{}. The empty Error block declares it an error, so it can live in
a Box<dyn Error> with every other error. And From is the
one that matters most. When ? meets an io::Error inside a function
that returns our enum, it calls From::from to convert it. That is
why no line in the reader says map_err. The second From goes the
other way, so a function that still returns io::Result can use ?
on ours. The last line gives the file its own Result, with the error type
filled in, so the signatures below stay short.
Write the loop in read_command that reads n bulk strings. Read a
line with read_line, and treat None as an unexpected end of file.
Check the first byte, parse the size, read exactly size + 2 bytes with
read_exact, check the last two, then truncate and push. The
let ... else at the top shows the shape for a missing line. Treat a size above MAX_BULK as bad.
pub fn read_command(r: &mut impl BufRead) -> io::Result<Option<Command>> {
let Some(line) = read_line(r)? else {
return Ok(None);
};
if line.first() != Some(&b'*') {
let words = line.split(|b| *b == b' ').filter(|w| !w.is_empty());
return Ok(Some(words.map(<[u8]>::to_vec).collect()));
}
let n = parse_len(&line[1..])?;
if n > MAX_ARGS {
return Err(protocol_error());
}
let mut args = Vec::with_capacity(n);
for _ in 0..n {
// your code here
}
Ok(Some(args))
}Stuck?Show the full listing
/// One request: an array of bulk strings such as `*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n`,
/// or one inline line of words. `Ok(None)` means the client has gone.
pub fn read_command(r: &mut impl BufRead) -> Result<Option<Command>> {
let Some(line) = read_line(r)? else {
return Ok(None);
};
if line.first() != Some(&b'*') {
let words = line.split(|b| *b == b' ').filter(|w| !w.is_empty());
return Ok(Some(words.map(<[u8]>::to_vec).collect()));
}
let n = parse_len(&line[1..], b'*')?;
if n > MAX_ARGS {
return Err(KvError::Protocol(b'*'));
}
let mut args = Vec::with_capacity(n);
for _ in 0..n {
let line = read_line(r)?.ok_or(io::Error::from(io::ErrorKind::UnexpectedEof))?;
if line.first() != Some(&b'$') {
return Err(KvError::Protocol(line.first().copied().unwrap_or(b'\n')));
}
let size = parse_len(&line[1..], b'$')?;
if size > MAX_BULK {
return Err(KvError::Protocol(b'$'));
}
let mut buf = vec![0u8; size + 2];
r.read_exact(&mut buf)?;
if &buf[size..] != b"\r\n" {
return Err(KvError::Protocol(buf[size]));
}
buf.truncate(size);
args.push(buf);
}
Ok(Some(args))
}The reply functions take impl Write, which means any writer, and the
write! macro formats straight into it:
pub fn write_simple(w: &mut impl Write, s: &str) -> io::Result<()> {
write!(w, "+{s}\r\n")
}
pub fn write_error(w: &mut impl Write, s: &str) -> io::Result<()> {
write!(w, "-{s}\r\n")
}
pub fn write_bulk(w: &mut impl Write, b: &[u8]) -> io::Result<()> {
write!(w, "${}\r\n", b.len())?;
w.write_all(b)?;
w.write_all(b"\r\n")
}
pub fn write_array_len(w: &mut impl Write, n: usize) -> io::Result<()> {
write!(w, "*{n}\r\n")
}Running a command is a match on the upper-cased name as bytes. The
let ... else handles an empty request. A bulk string is Vec<u8>
and not String, because a Rust String must be valid UTF-8 and a
Redis value need not be. When the name must be shown in an error message,
from_utf8_lossy replaces any invalid bytes with a marker:
/// Run one command and write its reply.
pub fn execute(args: &[Vec<u8>], w: &mut impl Write) -> io::Result<()> {
let Some(name) = args.first() else {
return resp::write_error(w, "ERR empty command");
};
match name.to_ascii_uppercase().as_slice() {
b"PING" => match args.len() {
1 => resp::write_simple(w, "PONG"),
2 => resp::write_bulk(w, &args[1]),
_ => write_arity(w, "ping"),
},
b"ECHO" => match args.len() {
2 => resp::write_bulk(w, &args[1]),
_ => write_arity(w, "echo"),
},
// redis-cli asks on connect; an empty list is fine
b"COMMAND" => resp::write_array_len(w, 0),
_ => {
let shown = String::from_utf8_lossy(&name[..name.len().min(40)]);
resp::write_error(w, &format!("ERR unknown command '{shown}'"))
}
}
}
fn write_arity(w: &mut impl Write, name: &str) -> io::Result<()> {
resp::write_error(
w,
&format!("ERR wrong number of arguments for '{name}' command"),
)
}The serving loop matches on the three outcomes of read_command. The arm
Err(e @ KvError::Protocol(_)) matches one variant and, with @,
also binds the whole error to e so it can be returned. That is the Rust form
of errors.As, checked by the compiler rather than searched at run time. The
compiler also checks that the four arms cover every case:
/// Read requests, run them and send the replies, until the client closes. The function
/// returns a `KvError`, so `?` converts every `io::Error` on the way with `From`.
fn serve(stream: TcpStream) -> resp::Result<()> {
let mut reader = BufReader::new(stream.try_clone()?);
let mut writer = BufWriter::new(stream);
loop {
match resp::read_command(&mut reader) {
Ok(Some(args)) => cmd::execute(&args, &mut writer)?,
Ok(None) => return Ok(()),
Err(e @ KvError::Protocol(_)) => {
resp::write_error(&mut writer, "ERR Protocol error")?;
writer.flush()?;
return Err(e);
}
Err(e) => return Err(e),
}
// A pipeline holds several requests; flush once the reader has nothing waiting.
if reader.buffer().is_empty() {
writer.flush()?;
}
}
}String::from_utf8(bytes) fails on a value that is not text.
Redis keys and values are bytes, and a client may store an image under a key. A parser that
converts every argument to String rejects such a client with an error that
looks like a bug in the client. Keep arguments as Vec<u8> and convert only
for display, with from_utf8_lossy, or for commands that need text, such as a
number.
Three ways to say what went wrong#
Each version of the parser had to express four different outcomes. The request is complete. The bytes are not RESP. The client closed the connection cleanly. The client closed it in the middle of a request. The table shows the three vocabularies.
| The outcome | C | Go | Rust |
|---|---|---|---|
| A value, or a failure | a return code and an out-parameter | two return values, (v, err) | one value, Result<T, E> |
| The reason for a failure | the global errno, or an enum of your own | an error value, found with errors.Is or errors.As | the Err payload, an enum matched on its variant |
| Passing a failure up | if (x < 0) return -1; on every call | if err != nil { return err } on every call | one character, ? |
| Ignoring a failure | write nothing, and nothing warns | write nothing, and nothing warns | the compiler warns about an unused Result |
| "Not an error, but nothing" | a special return value, such as 0 from read | a special error value, io.EOF | Option, which is not an error at all |
| Adding a reason | a format string into a fixed array | fmt.Errorf("...: %w", err) | a variant of your enum, and From so that ? converts |
C has one mechanism, the return value, and uses it for everything. The design works, and
the whole of Unix is built on it, but it has two costs that the book will keep meeting. The
first is that the check is optional and silent when missing. The second is that the value and the error share one channel. A function that returns a
pointer uses NULL for failure. A function that returns a count uses -1. The
caller must know which.
Go separates the value from the error, which fixes the second cost, and leaves the first
in place by choice. The Go FAQ says the designers found try-catch-finally
produced "convoluted code". The price is the three lines of if err != nil after
every call, and a Go file of any size has many such checks. Rust puts the value inside the Result, so that the caller cannot reach the value without facing the failure. The
? operator makes the common case one character. The price is a type system that
the programmer must learn before any of it makes sense. There is also a long error message
from the compiler when a ? is used where the error types do not match.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| Reading bytes | Weak | Strong | Strong | C has stdio, which blocks, so kv grows its own buffer. Both standard libraries have one. |
| Holding the arguments | Strong | OK | OK | The C parser copies nothing. Go and Rust copy each argument once, which is simpler and almost never matters. |
| Saying "not RESP" | OK | OK | Strong | C and Go use a value by convention. Rust uses a kind the compiler knows about and a match the compiler checks. |
| Saying "the client left" | OK | Weak | Strong | Go reuses an error value for a non-error. Rust has Option for exactly this. |
| Passing an error up | Weak | Weak | Strong | C and Go repeat the check at every call. Rust writes one character. |
| Binary values | Strong | Strong | OK | C and Go treat all bytes alike. Rust makes you choose between String and Vec<u8>, and the wrong choice rejects real data. |
| An error with details | Weak | OK | Strong | C has an enum and a fixed buffer, and Go has a type found by searching a chain. Rust has an enum variant the compiler matches. |
- switch. One integer against constant cases. A case with no break continues into the next, and an enum switch with no default is checked by the compiler.
- Error chain. A Go error wraps its cause with
%w. The functionserrors.Isanderrors.Assearch the chain, and==cannot. - From and ?. A Rust error enum with a
Fromimpl lets?convert every lower error into yours.
GET and SET#
Start the chapter 4 server and ask it to remember something:
$ redis-cli -p 6380 SET name kv
(error) ERR unknown command 'SET'
$ redis-cli -p 6380 GET name
(error) ERR unknown command 'GET'It speaks the protocol and has nothing to say. Every command it will ever learn is a name, a number of arguments and a function. This chapter gives it a place to keep that list, and the first five entries.
kv starts to remember. SET stores a value under a key and GET returns
it. Two commands take several keys, DEL and EXISTS. INCR
treats a value as a number. The chapter is about strings and bytes, because a key and a value are
bytes, and the three languages disagree about what a string is.
A Redis value is a sequence of bytes. It may be the text hello, the digits
42, or the first kilobyte of a JPEG. A Redis key is the same. The server must keep a copy of both and find a key again later. It must compare two keys for
equality. For INCR it must read a value as an integer and write one back. Each of those steps asks the
question "what is a string?", and each language answers it differently.
| A string | C | Go | Rust |
|---|---|---|---|
| What it is | a pointer to bytes that end with a zero byte | an immutable sequence of bytes | a sequence of bytes that must be valid UTF-8 |
| Bytes with a length | a pointer and a size_t, by convention | []byte | Vec<u8>, or a borrowed &[u8] |
| Length | strlen walks to the zero, or you carry the length | len(s), stored with the value | v.len(), stored with the value |
| Equality | memcmp with the lengths | == on strings, bytes.Equal on slices | == on both |
| As a map key | a hash you write, in chapter 6 | string, because a slice cannot be a key | Vec<u8>, because it is hashable |
| To an integer | you parse the digits, watching for overflow | strconv.ParseInt | str::parse::<i64>() after a UTF-8 check |
| Who owns the bytes | whoever the comment says | the garbage collector | exactly one binding, checked by the compiler |
| A command, as data | a function pointer in a table, built by a macro | a map from the name to a func | a fn pointer in a const table, built by a macro |
In BCPL, the language before B, a string began with its length in one byte. B dropped the count and ended a string with a marker character instead, and C kept that rule with the zero byte as the marker. Ritchie gave two reasons. The one-byte count limited a string to 255 characters, and "maintaining the count seemed, in our experience, less convenient than using a terminator". Fifty years later, that marker is why a C string cannot hold a zero byte. It is also why every C string function walks the whole string to find its end. Go and Rust store the length with the bytes, as BCPL did.
C: bytes, a length and an owner#
The arguments that the parser produces are not C strings. They are pointers into the request buffer with a length beside them, and no zero byte ends them. Every function in kv that touches a key or a value takes a pointer and a length. The C string library is never called on them. The first helpers copy, compare and parse such bytes:
/* A copy of n bytes that the caller owns and must free. Never NULL on success, even
* for n == 0, because malloc(0) may return NULL and that would look like failure. */
static char *kv_bytes_dup(const char *src, size_t n) {
char *p = malloc(n ? n : 1);
if (p == NULL) {
return NULL;
}
memcpy(p, src, n);
return p;
}
static bool kv_bytes_eq(const char *a, size_t alen, const char *b, size_t blen) {
return alen == blen && memcmp(a, b, alen) == 0;
}
/* Parse a signed decimal integer from bytes that have no terminating zero. Strict, as
* Redis is: no spaces, no plus sign, no leading zeros, and no overflow. */
static bool kv_parse_int(const char *s, size_t n, long long *out) {
size_t i = 0;
bool neg = false;
if (n == 0) {
return false;
}
if (s[0] == '-') {
neg = true;
i = 1;
if (n == 1) {
return false;
}
}
if (s[i] == '0' && n - i > 1) {
return false;
}
unsigned long long v = 0;
for (; i < n; i++) {
if (s[i] < '0' || s[i] > '9') {
return false;
}
unsigned d = (unsigned)(s[i] - '0');
if (v > (ULLONG_MAX - d) / 10) {
return false;
}
v = v * 10 + d;
}
if (!neg && v > (unsigned long long)LLONG_MAX) {
return false;
}
if (neg && v > (unsigned long long)LLONG_MAX + 1) {
return false;
}
*out = neg ? -(long long)(v - 1) - 1 : (long long)v;
return true;
}Three details deserve a look. kv_bytes_dup asks for at least one byte. The reason is that malloc(0)
is allowed to return NULL, which would look like an out-of-memory failure. The integer parser works in
unsigned arithmetic and checks before every multiplication, because overflow of a signed
integer is undefined behaviour in C. The last line turns the unsigned result into a negative number. It avoids computing
-v when v is one more than the largest positive value. That value is exactly the smallest negative one, and the obvious
expression would overflow on the way there.
Calling a string function on a key reads past the key. The key
hello in a request is followed by \r\n and the rest of the buffer,
not by a zero byte. strlen walks on until it finds a zero somewhere, and
strcmp compares bytes that belong to another argument. Most of the time there is no crash. Both are undefined behaviour, and AddressSanitizer, which chapter 12 introduces, reports such a read.
The rule in kv is simple. A key or a value is always a pointer and a length, and
string.h is for messages the program writes itself.
In April 2014 anyone on the internet could read 64 kilobytes of a web server's memory by sending it one message. The TLS heartbeat extension of OpenSSL, a C library, let a client send a payload and its length, and the server sent the payload back. The code allocated the reply "based on the length field in the requesting message, without regard to the actual size of that message's payload". A length of 65,535 and a payload of one byte returned that byte and whatever lay after it. Passwords, keys, other people's sessions. The bug is CVE-2014-0160 in the Common Vulnerabilities and Exposures list, and it had been in the library since the end of 2011. About 17 percent of the secure web servers on the internet were believed to be open to it on the day it was disclosed. It is the trusted length of chapter 4 and the pointer without a length of this chapter, in one line of C.
The store in this chapter is a list of entries. A lookup walks the list and compares each key, which is slow once there are thousands of keys. Chapter 6 replaces the list with a hash table. The entry owns copies of its key and value, and the comments say so:
/* One key and its value. Both are owned copies, freed with the entry. */
typedef struct {
char *key;
size_t klen;
char *val;
size_t vlen;
} kv_entry;
/* Every entry, in a list. A lookup walks the whole list, which chapter 6 fixes. */
typedef struct {
kv_entry *items; /* owned */
size_t len;
size_t cap;
} kv_store;
static kv_entry *kv_store_find(kv_store *s, const char *key, size_t klen) {
for (size_t i = 0; i < s->len; i++) {
if (kv_bytes_eq(s->items[i].key, s->items[i].klen, key, klen)) {
return &s->items[i];
}
}
return NULL;
}Write kv_store_set. Copy the value first, because that can fail. If the key
exists, free its old value and keep the new one. If not, grow the list when it is full, copy
the key, and append an entry. On any failure, free what you copied and return
false, leaving the store as it was.
static bool kv_store_set(kv_store *s, const char *key, size_t klen, const char *val,
size_t vlen) {
char *v = kv_bytes_dup(val, vlen);
if (v == NULL) return false;
/* your code here */
}Stuck?Show the full listing
/* Store a copy of val under a copy of key, replacing any old value. */
static bool kv_store_set(kv_store *s, const char *key, size_t klen, const char *val,
size_t vlen) {
char *v = kv_bytes_dup(val, vlen);
if (v == NULL) {
return false;
}
kv_entry *e = kv_store_find(s, key, klen);
if (e != NULL) {
free(e->val); /* the old value, which nothing else points to */
e->val = v;
e->vlen = vlen;
return true;
}
if (s->len == s->cap) {
size_t cap = s->cap ? s->cap * 2 : 16;
kv_entry *items = realloc(s->items, cap * sizeof *items);
if (items == NULL) {
free(v);
return false;
}
s->items = items;
s->cap = cap;
}
char *k = kv_bytes_dup(key, klen);
if (k == NULL) {
free(v);
return false;
}
s->items[s->len++] = (kv_entry){k, klen, v, vlen};
return true;
}
/* Remove a key. The last entry moves into the hole, so the order is not kept. */
static bool kv_store_del(kv_store *s, const char *key, size_t klen) {
kv_entry *e = kv_store_find(s, key, klen);
if (e == NULL) {
return false;
}
free(e->key);
free(e->val);
*e = s->items[--s->len];
return true;
}
static void kv_store_free(kv_store *s) {
for (size_t i = 0; i < s->len; i++) {
free(s->items[i].key);
free(s->items[i].val);
}
free(s->items);
s->items = NULL;
s->len = s->cap = 0;
}kv_store_del moves the last entry into the hole that the deleted one leaves.
That keeps the list packed without moving everything after it, at the cost of order, which a
store does not promise. A value is freed in three places. kv_store_set frees the old one on replacement, kv_store_del frees it, and at exit kv_store_free walks the list once. Every malloc in this file has one
free that matches it, and a reader can find the pair.
INCR is where bytes and numbers meet. The stored value is bytes. The command
parses them, adds one, and stores bytes again. The error messages are the ones Redis uses,
so that a client written for Redis understands kv. The check against
LLONG_MAX comes before the add, and the section on integers at the end of the
chapter says why it must:
/* INCR: the value is bytes that must parse as an integer, and the new value is bytes too.
* The check against LLONG_MAX comes before the add, because overflowing a signed integer
* is undefined behaviour, and the compiler may assume it never happens. */
static bool kv_cmd_incr(const kv_cmd *c, kv_store *s, kv_buf *out) {
long long n = 0;
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
if (e != NULL && !kv_parse_int(e->val, e->vlen, &n)) {
return kv_reply_error(out, "ERR value is not an integer or out of range");
}
if (n == LLONG_MAX) {
return kv_reply_error(out, "ERR increment or decrement would overflow");
}
n++;
char text[32];
int len = snprintf(text, sizeof text, "%lld", n);
if (!kv_store_set(s, c->argv[1], c->arglen[1], text, (size_t)len)) {
return false;
}
return kv_reply_int(out, n);
}A table of functions#
Chapter 4 ran a command through a chain of if statements, one per name.
Five more commands make the chain long, and the arity check is repeated in every link. The
shape underneath is the same each time. A name, how many arguments it takes, and what to
do. That is a row of a table, and C can hold the "what to do" in a function
pointer. A function pointer is the address of a function, with the signature it
expects spelled out, so the compiler knows how to call through it. Every command becomes
a function of one shape:
/* Each command is a function of this one shape, so a table can hold them all. */
typedef bool (*kv_cmd_fn)(const kv_cmd *c, kv_store *s, kv_buf *out);
static bool kv_cmd_ping(const kv_cmd *c, kv_store *s, kv_buf *out) {
(void)s;
if (c->argc == 1) {
return kv_reply_simple(out, "PONG");
}
if (c->argc == 2) {
return kv_reply_bulk(out, c->argv[1], c->arglen[1]);
}
return kv_reply_arity(out, "ping");
}
static bool kv_cmd_echo(const kv_cmd *c, kv_store *s, kv_buf *out) {
(void)s;
return kv_reply_bulk(out, c->argv[1], c->arglen[1]);
}
static bool kv_cmd_set(const kv_cmd *c, kv_store *s, kv_buf *out) {
if (!kv_store_set(s, c->argv[1], c->arglen[1], c->argv[2], c->arglen[2])) {
return false;
}
return kv_reply_simple(out, "OK");
}
static bool kv_cmd_get(const kv_cmd *c, kv_store *s, kv_buf *out) {
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
return e ? kv_reply_bulk(out, e->val, e->vlen) : kv_reply_nil(out);
}
static bool kv_cmd_del(const kv_cmd *c, kv_store *s, kv_buf *out) {
long long n = 0;
for (size_t i = 1; i < c->argc; i++) {
n += kv_store_del(s, c->argv[i], c->arglen[i]);
}
return kv_reply_int(out, n);
}
static bool kv_cmd_exists(const kv_cmd *c, kv_store *s, kv_buf *out) {
long long n = 0;
for (size_t i = 1; i < c->argc; i++) {
n += kv_store_find(s, c->argv[i], c->arglen[i]) != NULL;
}
return kv_reply_int(out, n);
}
static bool kv_cmd_command(const kv_cmd *c, kv_store *s, kv_buf *out) {
(void)c;
(void)s;
return kv_reply_array(out, 0); /* redis-cli asks on connect; an empty list is fine */
}The table is the list of rows. It is written once, as a macro that calls another macro,
X, on each row. Each place that needs the list defines X to say
what to make of a row, uses the list, and undefines X again. The first use
builds the array of rows. The second counts them, by turning each row into
+1. This is called an X-macro, and it is how C keeps one list in one place
when several things have to agree with it:
/* The command list, written once. Each use of KV_COMMANDS defines X to say what to make
* of a row. The first use below builds the table, the second counts it. A name is spelled
* as the arity error must show it, and arity is counted as Redis counts it: a positive
* number is the exact argc, name included, and a negative one is a minimum. */
#define KV_COMMANDS \
X(ping, kv_cmd_ping, -1) \
X(echo, kv_cmd_echo, 2) \
X(set, kv_cmd_set, 3) \
X(get, kv_cmd_get, 2) \
X(del, kv_cmd_del, -2) \
X(exists, kv_cmd_exists, -2) \
X(incr, kv_cmd_incr, 2) \
X(append, kv_cmd_append, 3) \
X(strlen, kv_cmd_strlen, 2) \
X(getrange, kv_cmd_getrange, 4) \
X(command, kv_cmd_command, -1)
typedef struct {
const char *name;
kv_cmd_fn fn;
int arity;
} kv_command;
static const kv_command kv_commands[] = {
#define X(name, fn, arity) {#name, fn, arity},
KV_COMMANDS
#undef X
};
enum {
#define X(name, fn, arity) +1
KV_COMMAND_COUNT = 0 KV_COMMANDS
#undef X
};Running a command is now a walk down the table. The arity check happens once, in the loop, and the function is called through its pointer with the same arguments the chain used to pass:
/* Run one command against the store and append its reply to out: find the row, check
* the arity, call the function through its pointer. */
static bool kv_exec(const kv_cmd *c, kv_store *s, kv_buf *out) {
if (c->argc == 0) {
return kv_reply_error(out, "ERR empty command");
}
for (size_t i = 0; i < KV_COMMAND_COUNT; i++) {
const kv_command *cmd = &kv_commands[i];
if (!kv_arg_is(c, 0, cmd->name)) {
continue;
}
int argc = (int)c->argc;
bool ok = cmd->arity < 0 ? argc >= -cmd->arity : argc == cmd->arity;
if (!ok) {
return kv_reply_arity(out, cmd->name);
}
return cmd->fn(c, s, out);
}
int n = c->arglen[0] < 40 ? (int)c->arglen[0] : 40;
return kv_reply_errorf(out, "ERR unknown command '%.*s'", n, c->argv[0]);
}When several things of one shape are chosen by name, put them in a table of structs.
Each struct holds a function pointer, and the code walks the table. Redis does this with its command table, and
every interpreter, driver model and plugin system in C does the same. A chain of
if grows with every feature, and a table grows by one row.
Growing, measuring and slicing bytes#
Three more commands work on the bytes of a value. APPEND makes a value
longer, STRLEN measures it, and GETRANGE cuts a piece out by
offset. Each one is a few lines, and each one meets a rule of C bytes:
/* APPEND key value: grow the stored bytes in place. A missing key starts empty. */
static bool kv_cmd_append(const kv_cmd *c, kv_store *s, kv_buf *out) {
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
if (e == NULL) {
if (!kv_store_set(s, c->argv[1], c->arglen[1], c->argv[2], c->arglen[2])) {
return false;
}
return kv_reply_int(out, (long long)c->arglen[2]);
}
char *v = realloc(e->val, e->vlen + c->arglen[2] + 1);
if (v == NULL) {
return false; /* e->val is still valid: realloc leaves it alone on failure */
}
memcpy(v + e->vlen, c->argv[2], c->arglen[2]);
e->val = v;
e->vlen += c->arglen[2];
return kv_reply_int(out, (long long)e->vlen);
}
/* STRLEN key: the length in bytes, which is the only length the store knows. */
static bool kv_cmd_strlen(const kv_cmd *c, kv_store *s, kv_buf *out) {
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
return kv_reply_int(out, e ? (long long)e->vlen : 0);
}
/* GETRANGE key start end: a slice by byte offsets, both inclusive. A negative offset
* counts from the end, and an offset past either end is clamped, as Redis does. */
static bool kv_cmd_getrange(const kv_cmd *c, kv_store *s, kv_buf *out) {
long long start, end;
if (!kv_parse_int(c->argv[2], c->arglen[2], &start) ||
!kv_parse_int(c->argv[3], c->arglen[3], &end)) {
return kv_reply_error(out, "ERR value is not an integer or out of range");
}
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
long long len = e ? (long long)e->vlen : 0;
if (start < 0) {
start += len;
}
if (end < 0) {
end += len;
}
if (start < 0) {
start = 0;
}
if (end >= len) {
end = len - 1;
}
if (len == 0 || start > end) {
return kv_reply_bulk(out, "", 0);
}
return kv_reply_bulk(out, e->val + start, (size_t)(end - start + 1));
}APPEND asks realloc to grow the block the value lives in.
When the block cannot grow in place, realloc moves it, copies the bytes and
frees the old one. So the pointer in the entry must be replaced. On failure it returns
NULL and the old block is untouched, which is why the result goes into a new
variable first. STRLEN answers in bytes, because the store knows no other
length. GETRANGE is arithmetic on offsets, with four clamps, and then a pointer
plus a length into the value. No byte is copied until the reply is written. The one
function from the C string library in this chapter is snprintf, which turns
the number in INCR into digits. It takes the size of its buffer, writes at
most that many bytes with a terminator, and returns how many it wanted to write. Its
older cousin sprintf takes no size and is how buffers overflow.
Run it, and the first real session with the real client works:
$ make && ./kv &
$ redis-cli -p 6380
127.0.0.1:6380> SET name kv
OK
127.0.0.1:6380> GET name
"kv"
127.0.0.1:6380> INCR visits
(integer) 1
127.0.0.1:6380> INCR name
(error) ERR value is not an integer or out of range
127.0.0.1:6380> DEL name visits
(integer) 2Any function that takes bytes takes two parameters, the pointer and the length, in that
order. A structure that holds bytes holds both fields side by side. The habit is older than
kv. The read and write calls of POSIX, the Portable Operating System Interface, take a pointer and a length. Redis keeps a length in front of every string it stores. A C string is only for text the program itself
writes, such as an error message, where the zero byte is guaranteed.
Go: two kinds of string, and a copy between them#
Go has both answers to the question. A string is immutable bytes with a
length. A []byte is mutable bytes with a length. Converting one to the other
copies the bytes, because the string must not change when the slice does. The store uses a
string for the key, because a map key must support ==, and a slice
does not. The value stays a slice:
// store keeps every key and its value. The parser allocated each slice, so the store can
// keep them without copying: nothing else holds them.
type store struct {
m map[string][]byte
}
func newStore() *store {
return &store{m: make(map[string][]byte)}
}
func (s *store) get(key []byte) ([]byte, bool) {
v, ok := s.m[string(key)]
return v, ok
}
func (s *store) set(key, val []byte) {
s.m[string(key)] = val
}
func (s *store) del(key []byte) bool {
_, ok := s.m[string(key)]
delete(s.m, string(key))
return ok
}
func (s *store) has(key []byte) bool {
_, ok := s.m[string(key)]
return ok
}The value is not copied on set. The key is converted to a string, which copies it once, because a map key must be comparable. The parser allocated a fresh slice for every argument, and nothing else holds it. The
garbage collector keeps it alive for as long as the map refers to it. The question that took a paragraph of comments in C does not arise. The
cost is in the collector, which must find out, from time to time, which slices are still
referred to.
// The two ways INCR fails. Their text is what the client sees.
var (
errNotInteger = errors.New("ERR value is not an integer or out of range")
errOverflow = errors.New("ERR increment or decrement would overflow")
)Write the incr method. Read the current value if there is one, parse it with
strconv.ParseInt, and return errNotInteger when parsing fails. Return
errOverflow when the value is already math.MaxInt64. Otherwise store
the new value, formatted with strconv.FormatInt, and return it.
func (s *store) incr(key []byte) (int64, error) {
var n int64
// your code here
return n, nil
}Stuck?Show the full listing
// incr adds one to the integer stored under key, which counts as 0 when absent.
func (s *store) incr(key []byte) (int64, error) {
var n int64
if cur, ok := s.get(key); ok {
var err error
n, err = strconv.ParseInt(string(cur), 10, 64)
if err != nil {
return 0, errNotInteger
}
}
if n == math.MaxInt64 {
return 0, errOverflow
}
n++
s.set(key, []byte(strconv.FormatInt(n, 10)))
return n, nil
}The two errors are package-level values with the exact text the client should see. The
command code does not look inside them. It writes err.Error() as the reply,
which keeps the knowledge of what can go wrong next to the code that knows it.
A map of functions#
In Go a function is a value like any other. It can sit in a variable, in a struct field
or in a map, and a func type names its shape. The command table is a map from
the upper-cased name to a struct that holds the function and its arity:
// A handler is any function of this shape. The table maps the upper-cased name to one,
// with the arity as Redis counts it: a positive number is the exact argc, name included,
// and a negative one is a minimum.
type handler func(args [][]byte, s *store, w *bufio.Writer) error
type command struct {
run handler
arity int
}
var commands = map[string]command{
"PING": {ping, -1},
"ECHO": {echo, 2},
"SET": {set, 3},
"GET": {get, 2},
"DEL": {del, -2},
"EXISTS": {exists, -2},
"INCR": {incr, 2},
"APPEND": {appendCmd, 3},
"STRLEN": {strlenCmd, 2},
"GETRANGE": {getrange, 4},
"COMMAND": {commandCmd, -1},
}Running a command is a map lookup, an arity check, and a call. The chain of cases is gone, and adding a command is one line in the map and one function:
// execute runs one command against the store and writes its reply: look the name up,
// check the arity, call the function the table holds.
func execute(args [][]byte, s *store, w *bufio.Writer) error {
if len(args) == 0 {
return writeError(w, "ERR empty command")
}
name := strings.ToUpper(string(args[0]))
cmd, ok := commands[name]
if !ok {
shown := string(args[0])
if len(shown) > 40 {
shown = shown[:40]
}
return writeError(w, "ERR unknown command '"+shown+"'")
}
if n := len(args); (cmd.arity < 0 && n < -cmd.arity) || (cmd.arity > 0 && n != cmd.arity) {
return writeArity(w, strings.ToLower(name))
}
return cmd.run(args, s, w)
}
func writeArity(w *bufio.Writer, name string) error {
return writeError(w, "ERR wrong number of arguments for '"+name+"' command")
}Two of the handlers show what a function value can be. countKeys takes a
function from a key to a bool and counts the keys for which it is true. DEL
passes s.del, which is a method value. It is the method with
its receiver already bound, so it can be called later with the key alone.
EXISTS passes a closure, a function literal that captures
s from the surrounding function and calls get on it. A closure
captures by reference. It sees the variable as it is when the closure runs, not as it was
when the closure was made. The defer test of chapter 3 showed the same
rule:
func ping(args [][]byte, _ *store, w *bufio.Writer) error {
switch len(args) {
case 1:
return writeSimple(w, "PONG")
case 2:
return writeBulk(w, args[1])
}
return writeArity(w, "ping")
}
func echo(args [][]byte, _ *store, w *bufio.Writer) error {
return writeBulk(w, args[1])
}
func set(args [][]byte, s *store, w *bufio.Writer) error {
s.set(args[1], args[2])
return writeSimple(w, "OK")
}
func get(args [][]byte, s *store, w *bufio.Writer) error {
if v, ok := s.get(args[1]); ok {
return writeBulk(w, v)
}
return writeNil(w)
}
// countKeys counts the keys for which hit returns true. DEL and EXISTS differ only in
// which method they pass: s.del removes as it counts, s.has only looks.
func countKeys(keys [][]byte, hit func([]byte) bool) int64 {
var n int64
for _, key := range keys {
if hit(key) {
n++
}
}
return n
}
func del(args [][]byte, s *store, w *bufio.Writer) error {
return writeInt(w, countKeys(args[1:], s.del))
}
func exists(args [][]byte, s *store, w *bufio.Writer) error {
return writeInt(w, countKeys(args[1:], s.has))
}
func incr(args [][]byte, s *store, w *bufio.Writer) error {
n, err := s.incr(args[1])
if err != nil {
return writeError(w, err.Error())
}
return writeInt(w, n)
}
func commandCmd(_ [][]byte, _ *store, w *bufio.Writer) error {
return writeArrayLen(w, 0) // redis-cli asks on connect; an empty list is fine
}Bytes, runes and a builder#
Three more commands work on the bytes of a value. APPEND uses the built-in
append. It may write into the spare capacity of the old slice or allocate a
new array, and the map keeps whichever slice comes back. STRLEN
replies with len, which counts bytes. The character count of a string is a
different question in Go. A string is bytes in UTF-8, and a character outside ASCII takes
more than one byte. A rune is one character as an int32,
range over a string walks runes, and utf8.RuneCount counts them.
GETRANGE slices by byte offset, which is what Redis does and what may cut a
rune in half. The test below is in the chapter's snapshot. It shows the three facts and
strings.Builder, which grows a string without copying it at every step:
// A string is bytes. len counts them, and a character outside ASCII takes more than one.
func TestBytesAndRunes(t *testing.T) {
s := "hรฉllo"
if len(s) != 6 || utf8.RuneCountInString(s) != 5 {
t.Fatalf("len %d, runes %d", len(s), utf8.RuneCountInString(s))
}
// range walks runes, not bytes: the index jumps from 1 to 3 over the two bytes of รฉ
var idx []int
for i := range s {
idx = append(idx, i)
}
if idx[2] != 3 {
t.Fatalf("indexes %v", idx)
}
// s[1:2] is one byte, half of รฉ, and not a valid string on its own
if utf8.ValidString(s[1:2]) {
t.Fatal("half a rune should not be valid")
}
// strings.Builder grows a string without copying it at every step
var b strings.Builder
for i := 0; i < 3; i++ {
b.WriteString("ab")
}
if b.String() != "ababab" {
t.Fatal(b.String())
}
}// appendCmd grows the stored bytes. append may reuse the old slice's spare capacity or
// allocate a new array; either way the map is updated with the slice it returns.
func appendCmd(args [][]byte, s *store, w *bufio.Writer) error {
old, _ := s.get(args[1])
v := append(old, args[2]...)
s.set(args[1], v)
return writeInt(w, int64(len(v)))
}
// strlenCmd replies with the length in bytes. len counts bytes, and so does Redis; the
// number of characters is a different question, which utf8.RuneCount answers.
func strlenCmd(args [][]byte, s *store, w *bufio.Writer) error {
v, _ := s.get(args[1])
_ = utf8.RuneCount(v) // the count a user would expect for text, which STRLEN is not
return writeInt(w, int64(len(v)))
}
// getrange slices by byte offsets, both inclusive. A negative offset counts from the end,
// and an offset past either end is clamped, as Redis does.
func getrange(args [][]byte, s *store, w *bufio.Writer) error {
start, err1 := strconv.ParseInt(string(args[2]), 10, 64)
end, err2 := strconv.ParseInt(string(args[3]), 10, 64)
if err1 != nil || err2 != nil {
return writeError(w, "ERR value is not an integer or out of range")
}
v, _ := s.get(args[1])
n := int64(len(v))
if start < 0 {
start += n
}
if end < 0 {
end += n
}
start = max(start, 0)
end = min(end, n-1)
if n == 0 || start > end {
return writeBulk(w, nil)
}
return writeBulk(w, v[start:end+1])
}strconv.ParseInt accepts +5 and 007.
Redis stores both and then refuses to INCR them, and the C version of kv does the same. The Go version accepts them, and so
does the Rust version, because both standard libraries parse numbers the same generous way.
The book leaves the difference in and tells you about it, because that is what happens in
real ports. A strict parser in Go is a short function, and the C version shows what it must check.
Rust: a value has one owner#
Rust has the same two kinds as Go, String for text and Vec<u8>
for bytes, with one more rule. A String must be valid UTF-8, and the compiler
will not let you build one from bytes without a check. A Redis key is bytes, so the store is
a map from Vec<u8> to Vec<u8>. A Vec can
be a key because it supports hashing and equality:
/// Every key and its value. The store owns both: a key or value that is put in is moved,
/// not copied, and is freed when it is replaced, deleted or the store is dropped.
#[derive(Default)]
pub struct Store {
map: HashMap<Vec<u8>, Vec<u8>>,
}
impl Store {
pub fn new() -> Self {
Self::default()
}
/// A borrowed view of the value, which lives as long as the borrow of the store.
pub fn get(&self, key: &[u8]) -> Option<&[u8]> {
self.map.get(key).map(Vec::as_slice)
}
pub fn set(&mut self, key: Vec<u8>, val: Vec<u8>) {
self.map.insert(key, val);
}
pub fn del(&mut self, key: &[u8]) -> bool {
self.map.remove(key).is_some()
}
/// Append to the value under key, which starts empty, and return the new length. The
/// entry API finds or inserts in one lookup, and `extend_from_slice` grows the `Vec`
/// in place when it has room.
pub fn append(&mut self, key: Vec<u8>, val: Vec<u8>) -> usize {
let v = self.map.entry(key).or_default();
v.extend_from_slice(&val);
v.len()
}
}Two signatures carry the whole ownership story. get takes &self
and returns Option<&[u8]>, a borrowed view that lives only as long as the
borrow of the store. set takes Vec<u8> by value, which means
the caller gives the bytes away. Nothing is copied, and nothing can be used twice.
The command function takes its arguments by value for the same reason. For SET, mem::take moves the key and the value out of the argument
list and leaves empty vectors behind. The store receives them without a copy.
A table of fn pointers, built by a macro#
Rust has function pointers too, with the type fn(...) -> ..., and a
const array of structs can hold them. The signature has one new thing in it.
The writer is &mut dyn Write, which means any writer, chosen at run
time. The helpers in resp take impl Write, which means any
writer chosen at compile time. The compiler makes a copy of each helper per writer
type. A table of pointers cannot hold a copy per type, so the handlers take the run-time
form. Chapter 7 returns to the difference. The table itself is written through a
macro_rules! macro, which turns each name => function, arity
line into an Entry:
/// Every command is a function of this one shape, so a table can hold them. The writer is
/// `dyn Write`, any writer decided at run time, because a table cannot hold one function
/// per writer type. Chapter 7 explains `dyn`.
type Handler = fn(&mut Command, &mut Store, &mut dyn Write) -> io::Result<()>;
/// A row of the table: the name as the client spells it, the function, and the arity as
/// Redis counts it. Positive is the exact argc, name included; negative is a minimum.
struct Entry {
name: &'static [u8],
run: Handler,
arity: i32,
}
/// The table, written once. The macro turns each `name => function, arity` line into an
/// `Entry`, so the list reads like a list and not like a struct literal eleven times.
macro_rules! commands {
($($name:literal => $run:ident, $arity:expr;)*) => {
const COMMANDS: &[Entry] = &[$(Entry { name: $name, run: $run, arity: $arity }),*];
};
}
commands! {
b"PING" => ping, -1;
b"ECHO" => echo, 2;
b"SET" => set, 3;
b"GET" => get, 2;
b"DEL" => del, -2;
b"EXISTS" => exists, -2;
b"INCR" => incr, 2;
b"APPEND" => append, 3;
b"STRLEN" => strlen, 2;
b"GETRANGE" => getrange, 4;
b"COMMAND" => command, -1;
}A macro_rules! macro matches the tokens between its brackets against a
pattern and writes code from a template. $name:literal captures a literal and
$run:ident a name. $arity:expr captures an expression, and
$( ... ;)* repeats for every line. The macro runs before type checking, so
what it writes is checked as if you had typed it. The lookup is then a search of the
array, and the call goes through the pointer:
/// Run one command against the store and write its reply: find the row, check the arity,
/// call the function. The command is taken by value, so that SET can move the key and
/// the value into the store instead of copying them.
pub fn execute(mut args: Command, store: &mut Store, w: &mut impl Write) -> io::Result<()> {
let Some(name) = args.first() else {
return resp::write_error(w, "ERR empty command");
};
let name = name.to_ascii_uppercase();
let Some(entry) = COMMANDS.iter().find(|e| e.name == name.as_slice()) else {
let shown = String::from_utf8_lossy(&args[0][..args[0].len().min(40)]);
return resp::write_error(w, &format!("ERR unknown command '{shown}'"));
};
let argc = args.len() as i32;
let ok = if entry.arity < 0 {
argc >= -entry.arity
} else {
argc == entry.arity
};
if !ok {
return write_arity(w, entry.name);
}
(entry.run)(&mut args, store, w)
}
fn write_arity(w: &mut dyn Write, name: &[u8]) -> io::Result<()> {
let name = String::from_utf8_lossy(name).to_ascii_lowercase();
resp::write_error(
w,
&format!("ERR wrong number of arguments for '{name}' command"),
)
}fn ping(args: &mut Command, _: &mut Store, w: &mut dyn Write) -> io::Result<()> {
match args.as_slice() {
[_] => resp::write_simple(w, "PONG"),
[_, msg] => resp::write_bulk(w, msg),
_ => write_arity(w, b"ping"),
}
}
fn echo(args: &mut Command, _: &mut Store, w: &mut dyn Write) -> io::Result<()> {
resp::write_bulk(w, &args[1])
}
fn set(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
let [_, key, val] = args.as_mut_slice() else {
unreachable!("arity checked")
};
store.set(mem::take(key), mem::take(val));
resp::write_simple(w, "OK")
}
fn get(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
match store.get(&args[1]) {
Some(val) => resp::write_bulk(w, val),
None => resp::write_nil(w),
}
}
fn del(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
let n = args[1..].iter().filter(|k| store.del(k)).count();
resp::write_int(w, n as i64)
}
fn exists(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
let n = args[1..].iter().filter(|k| store.get(k).is_some()).count();
resp::write_int(w, n as i64)
}
fn incr(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
match store.incr(&args[1]) {
Ok(n) => resp::write_int(w, n),
Err(e) => resp::write_error(w, e.message()),
}
}
fn command(_: &mut Command, _: &mut Store, w: &mut dyn Write) -> io::Result<()> {
resp::write_array_len(w, 0) // redis-cli asks on connect; an empty list is fine
}Each handler is a match on the argument slice. One arm can say
"PING with nothing", the next "PING with one message". A
let ... else with a slice pattern takes the key and the value out of
SET. The unreachable! says that the arity check above makes
the other case impossible.
str, String, char and a slice of bytes#
Three more commands work on the bytes of a value. APPEND uses the map's
entry API to find or insert in one lookup and extends the Vec in place.
STRLEN is len, in bytes. GETRANGE slices the
Vec by byte offset. None of them can be written on a String. A
String is a Vec<u8> with a promise, that its bytes are
valid UTF-8. A slice at a byte offset can break the promise. The test below
is in the chapter's snapshot. A str has a length in bytes and a count of
char, which are different. A slice of a str must land on a
character boundary, and one that does not is a panic. The same bytes as a
&[u8] can be cut anywhere, which is why kv stores values as bytes:
/// A `str` is bytes that are valid UTF-8. `len` counts the bytes, `chars` the
/// characters, and a slice must land on a character boundary.
#[test]
fn bytes_and_chars() {
let s = "hรฉllo";
assert_eq!(s.len(), 6);
assert_eq!(s.chars().count(), 5);
assert_eq!(&s[0..1], "h");
assert_eq!(&s[1..3], "รฉ");
assert!(s.is_char_boundary(3));
assert!(!s.is_char_boundary(2));
// the same bytes as a slice can be cut anywhere, which is what GETRANGE needs
assert_eq!(&s.as_bytes()[1..2], &[0xC3]);
}
/// Slicing a `str` in the middle of a character is a panic, not a corrupt string.
#[test]
#[should_panic(expected = "byte index 2 is not a char boundary")]
fn half_a_char_panics() {
let s = "hรฉllo";
let _ = &s[0..2];
}/// APPEND key value: grow the stored bytes. The value is moved in when the key is new.
fn append(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
let [_, key, val] = args.as_mut_slice() else {
unreachable!("arity checked")
};
let n = store.append(mem::take(key), mem::take(val));
resp::write_int(w, n as i64)
}
/// STRLEN key: the length in bytes. `len` on a `Vec<u8>` counts bytes, and so does Redis.
/// The number of characters is a different question, which `chars().count()` answers on
/// a `str`, and which the store cannot ask, because a value need not be text.
fn strlen(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
let n = store.get(&args[1]).map_or(0, <[u8]>::len);
resp::write_int(w, n as i64)
}
/// GETRANGE key start end: a slice by byte offsets, both inclusive. A negative offset
/// counts from the end, and an offset past either end is clamped, as Redis does.
fn getrange(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
let (Some(start), Some(end)) = (parse_int(&args[2]), parse_int(&args[3])) else {
return resp::write_error(w, "ERR value is not an integer or out of range");
};
let val = store.get(&args[1]).unwrap_or(b"");
let len = val.len() as i64;
let start = if start < 0 { start + len } else { start }.max(0);
let end = if end < 0 { end + len } else { end }.min(len - 1);
if len == 0 || start > end {
return resp::write_bulk(w, b"");
}
resp::write_bulk(w, &val[start as usize..=end as usize])
}/// Why INCR could not add one.
pub enum IncrError {
NotInteger,
Overflow,
}
impl IncrError {
pub fn message(&self) -> &'static str {
match self {
Self::NotInteger => "ERR value is not an integer or out of range",
Self::Overflow => "ERR increment or decrement would overflow",
}
}
}
/// The bytes as a signed 64-bit integer, if that is what they are.
pub fn parse_int(bytes: &[u8]) -> Option<i64> {
std::str::from_utf8(bytes).ok()?.parse().ok()
}Write the incr method. Use parse_int for the current value. Turn its None into
IncrError::NotInteger with ok_or.
Use checked_add, which returns None on overflow, and turn that into
IncrError::Overflow. Store the new value with to_string and
into_bytes, and return it.
pub fn incr(&mut self, key: &[u8]) -> Result<i64, IncrError> {
let n = match self.get(key) {
None => 0,
Some(bytes) => parse_int(bytes).ok_or(IncrError::NotInteger)?,
};
// your code here
}Stuck?Show the full listing
impl Store {
/// Add one to the integer stored under key, which counts as 0 when absent.
pub fn incr(&mut self, key: &[u8]) -> Result<i64, IncrError> {
let n = match self.get(key) {
None => 0,
Some(bytes) => parse_int(bytes).ok_or(IncrError::NotInteger)?,
};
let n = n.checked_add(1).ok_or(IncrError::Overflow)?;
self.set(key.to_vec(), n.to_string().into_bytes());
Ok(n)
}
}IncrError is an enum with two cases, and the command code matches on it to
write the reply. This is the Rust form of the two Go sentinel values. The difference is that a match on an enum must handle every case. So a third
kind of failure, added later, is a compile error at every place that forgot it.
A borrowed value blocks every write to the store until it is dropped. The following three lines do not compile, and the compiler says why:
let old = store.get(b"a");
store.set(b"a".to_vec(), b"2".to_vec());
println!("{:?}", old);error[E0502]: cannot borrow `store` as mutable because it is also borrowed as immutable
| let old = store.get(b"a");
| ----- immutable borrow occurs here
| store.set(b"a".to_vec(), b"2".to_vec());
| ^^^^^ mutable borrow occurs here
| println!("{:?}", old);
| --- immutable borrow later used hereIn C the same three lines compile, and old points at freed memory after the
set. The Rust error is the C bug, caught before the program runs. When you need
the old value after a write, copy it first with to_vec(). That is what the C
code did with kv_bytes_dup, and it is what the compiler is asking for.
Three answers to the largest integer plus one#
Start any of the three servers and run SET big 9223372036854775807, then
INCR big. All three reply with the Redis error, because each one checks
before it adds. Remove the check and the three languages give three different answers,
and that is the lesson of this section.
In C, adding one to LLONG_MAX is undefined behaviour. The standard does
not say what happens, so the compiler is allowed to assume it never happens, and it does.
A check written after the add, such as if (n + 1 < n), is removed by the
optimiser as impossible. The check must come before, as kv_cmd_incr writes
it. The parser of chapter 5 had the same rule and refused more than 18 digits for the same
reason. Unsigned arithmetic is the exception. It wraps around by definition, which is why
a hash function in chapter 6 is written with unsigned types.
In Go, integers wrap. The largest int64 plus one is the smallest, with no
error, no panic and no flag. The test in the chapter's snapshot shows it:
// Go integers wrap. The largest int64 plus one is the smallest, with no error and no panic,
// which is why incr checks against math.MaxInt64 before it adds.
func TestOverflowWraps(t *testing.T) {
n := int64(math.MaxInt64)
n++
if n != math.MinInt64 {
t.Fatalf("got %d", n)
}
}In Rust, the answer depends on the build. A debug build panics on the overflow, with the
message attempt to add with overflow. A release build wraps, as Go does.
Three methods say what you want in their names, checked_add,
wrapping_add and saturating_add. They behave the same in both
builds. kv uses checked_add, which returns None on overflow, and
turns that into the Redis error:
/// Three ways to add one to the largest integer: a debug build panics on plain `+`,
/// a release build wraps, and `checked_add` returns `None` in both. kv uses the last.
#[test]
fn overflow_three_ways() {
let n = i64::MAX;
assert_eq!(n.checked_add(1), None);
assert_eq!(n.wrapping_add(1), i64::MIN);
assert_eq!(n.saturating_add(1), i64::MAX);
}
#[test]
#[cfg(debug_assertions)]
#[should_panic(expected = "attempt to add with overflow")]
fn plain_add_panics_in_debug() {
let n = std::hint::black_box(i64::MAX);
let _ = n + 1;
}A test for signed overflow that runs after the arithmetic may be deleted by the
compiler. Because the overflow cannot happen in a correct program, a branch that
only runs when it did happen is dead code to the optimiser. The same reasoning removes a
NULL check after a pointer has already been used. UndefinedBehaviorSanitizer,
in chapter 12, reports the overflow at run time, and -ftrapv turns it into a
crash. Both are off in a release build.
Who owns the bytes#
Every version of SET had to answer one question. The bytes arrived in a
request buffer. Who keeps them, and who frees them? C answered with a copy and a comment. The
store makes its own copy with kv_bytes_dup and frees it on replacement. The
comment on the field is the only record of the rule. Go answered with the collector. The
store keeps the parser's slice. The collector frees it when nothing refers to it any
more, at a time the program does not choose. Rust answered with a move. The store takes the
bytes from the argument list, which is left empty. The compiler refuses any later use of
what was moved.
All three answers work. The C answer costs a copy and a rule the compiler does not know.
The Go answer costs memory until the next collection and a pause when it runs. The Rust
answer costs nothing at run time. It costs a type signature that says Vec<u8>
in place of the char * of C. Chapter 6 makes the costs larger, because the store becomes
a structure that must free and move things itself.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| A key is bytes | OK | Strong | OK | C carries a length beside every pointer, Go has []byte and string, and Rust makes you pick Vec<u8> because String rejects binary data. |
| Copying and owning | Weak | Strong | Strong | C copies and comments, Go lets the collector own everything, and Rust moves and checks. |
| Parsing a number | Weak | OK | OK | C needs a careful parser to avoid undefined behaviour. The other two accept forms Redis rejects. |
| Overflow | Weak | OK | Strong | C must check before it adds and Go checks against a constant, while Rust has checked_add built in. |
| A missing case | Weak | Weak | Strong | An enum match in Rust must be complete. In C and Go, a new error kind can be forgotten at a call site. |
| Reading the code | OK | Strong | OK | The Go store is the shortest. The Rust signatures say more and take longer to read. |
| A table of commands | OK | Strong | OK | C needs an X-macro and a function pointer type, and Go has a map of funcs. Rust needs dyn Write and a macro for the rows. |
| Text against bytes | Weak | OK | Strong | C has no text type at all, and Go has rune helpers. Rust makes a str keep its promise and a Vec<u8> make none. |
- Function pointer. A function is a value with an address, and a table of them replaces a chain of ifs in all three languages.
- Bytes against text. A length in bytes is not a count of characters, and a slice at a byte offset can cut a character in half.
- Signed overflow. Undefined in C, a silent wrap in Go, a panic in a Rust debug build, and a checked add in kv.
Your own hash table#
Start the chapter 5 server and fill it with twenty thousand keys. Then let the Redis benchmark tool ask for them, one client, one request at a time:
$ redis-cli -p 6380 --pipe < twenty-thousand-sets.txt
errors: 0, replies: 20000
$ redis-benchmark -p 6380 -c 1 -n 5000 -r 20000 -t get -q
GET: 6257.82 requests per second, p50=0.151 msecSix thousand a second, from a program that does nothing but compare bytes. Real Redis
answers the same question from one client about twenty times faster. Every
GET is walking the list from chapter 5, and the list has twenty thousand
entries.
The list from chapter 5 gets slower with every key. This chapter replaces it with a hash table that you write yourself, in all three languages. In C the table becomes the store. In Go and Rust the standard map stays, and your table stands beside it. The chapter is about memory, because a hash table must allocate, move and free its own parts.
With 20,000 keys in the store, the C server from chapter 5 answers about 6,300
GET requests a second from one client. Each lookup walks half the list on
average. The same server with the hash table from this chapter answers about 17,700, and
the median request takes 39 microseconds instead of 151. That is the measurement on the
author's machine, and chapter 13 shows how to take it yourself. The difference grows with
the number of keys, because the list takes time in proportion to the keys and the table
does not.
A hash table turns a key into a number, called its hash. The low bits of that number are an
index into an array of buckets. Two keys may land in the same bucket, so each
bucket holds a chain of entries, and a lookup walks that short chain. When the table holds
more entries than buckets, the array doubles and every entry moves to its new bucket. The
chapter also adds two commands that ask about the whole table, DBSIZE and KEYS.
| The table | C | Go | Rust |
|---|---|---|---|
| A bucket | a pointer to the first entry of a chain | a slice of entries | a Vec of entries |
| A chain link | a next pointer the entry owns | the next element of the slice | the next element of the Vec |
| Growing | calloc a bigger array, relink every entry | make a bigger slice, append every entry | drain every chain into a bigger Vec |
| Deleting | unlink, then free the key, the value and the entry | move the last entry into the hole | swap_remove, which drops the entry |
| What frees the old array | you, with free | the garbage collector, later | the old Vec is dropped at the end of the function |
| The hash | FNV-1a, written here | FNV-1a, written here | FNV-1a here, SipHash in the standard map |
| Who would use it | everyone, because C has no map | nobody, the map is built in | nobody, HashMap is built in |
Doubling the array is the simplest way to grow a table, and it is what this chapter does.
It has a cost. The move of every entry takes a time proportional to the table, and a Redis
server with ten million keys cannot stop for that long. So the Redis dictionary keeps two
arrays during a resize and moves a few buckets on every operation. The function is
dictRehash(dict *d, int n) in dict.c, and a field named
rehashidx records how far the move has gone. The built-in map of Go also grows incrementally. It splits a large map into
several small tables that grow one at a time. The server kv needs no such trick, and the simple version is
easier to see.
C: an array of chains, and every pointer accounted for#
The hash is FNV-1a, a few lines that multiply and exclusive-or. It is fast and spreads keys well, and it is not safe against an attacker who chooses keys that collide. The Rust section says why that matters:
/* FNV-1a: a small, fast hash of bytes. Not safe against an attacker who chooses keys
* to collide, which Redis guards against with a secret seed. Chapter 12 returns to it. */
static uint64_t kv_hash(const char *s, size_t n) {
uint64_t h = 14695981039346656037ULL;
for (size_t i = 0; i < n; i++) {
h ^= (unsigned char)s[i];
h *= 1099511628211ULL;
}
return h;
}Two details of the hash are rules of C integers. The multiply overflows on almost every
step, and that is fine, because uint64_t is unsigned and unsigned arithmetic
wraps by definition. A signed type here would be undefined behaviour. And each byte is
cast to unsigned char before it is mixed in. A plain char may be
signed or unsigned, by platform. On this machine it is signed, so the byte 0xC3 of an
รฉ is the value -61. Mixing -61 into a 64-bit hash sets the top 56 bits.
The cast makes it 195 on every platform, and the hash the same everywhere. The same cast
belongs in front of every ctype.h call, for the same reason.
An entry now has a next pointer, and the comment says that the entry owns it.
That one word decides who frees a chain. The table is an array of chain heads. The count of buckets is always a power of two, so
that the index is the hash masked with cap - 1 instead of a division:
/* One key and its value, in a chain of entries that share a bucket. The key and the
* value are owned copies. The next pointer is owned too: freeing an entry frees its
* chain. */
typedef struct kv_entry {
char *key;
size_t klen;
char *val;
size_t vlen;
struct kv_entry *next;
} kv_entry;
/* A hash table with separate chaining. buckets[i] is the head of a chain, or NULL. The
* table grows when it holds more entries than buckets, so a chain stays short. */
typedef struct {
kv_entry **buckets; /* owned array of cap pointers */
size_t cap; /* a power of two, or 0 before the first insert */
size_t len; /* how many entries, over all chains */
} kv_store;
static kv_entry *kv_store_find(kv_store *s, const char *key, size_t klen) {
if (s->cap == 0) {
return NULL;
}
size_t i = kv_hash(key, klen) & (s->cap - 1);
for (kv_entry *e = s->buckets[i]; e != NULL; e = e->next) {
if (kv_bytes_eq(e->key, e->klen, key, klen)) {
return e;
}
}
return NULL;
}
/* The command list, written once. Each use of KV_COMMANDS defines X to say what to make
* of a row. The first use builds the table, the second counts it. A positive arity is
* the exact argc, name included, and a negative one is a minimum, as Redis counts. */
#define KV_COMMANDS \
X(ping, kv_cmd_ping, -1) \
X(echo, kv_cmd_echo, 2) \
X(set, kv_cmd_set, 3) \
X(get, kv_cmd_get, 2) \
X(del, kv_cmd_del, -2) \
X(exists, kv_cmd_exists, -2) \
X(incr, kv_cmd_incr, 2) \
X(append, kv_cmd_append, 3) \
X(strlen, kv_cmd_strlen, 2) \
X(getrange, kv_cmd_getrange, 4) \
X(dbsize, kv_cmd_dbsize, 1) \
X(keys, kv_cmd_keys, 2) \
X(command, kv_cmd_command, -1)
typedef struct {
const char *name;
kv_cmd_fn fn;
int arity;
} kv_command;
static const kv_command kv_commands[] = {
#define X(name, fn, arity) {#name, fn, arity},
KV_COMMANDS
#undef X
};
enum {
#define X(name, fn, arity) +1
KV_COMMAND_COUNT = 0 KV_COMMANDS
#undef X
};Growing allocates a new array with calloc, which returns memory set to zero,
so every new bucket starts as an empty chain. Then it walks every old chain and pushes each
entry onto the front of its new one. No key or value is copied. Only the next
pointers change, which is the point of keeping entries as separate allocations:
/* Double the bucket array and move every entry to its new chain. No entry is copied or
* freed: only the next pointers change. */
static bool kv_store_grow(kv_store *s) {
size_t cap = s->cap ? s->cap * 2 : 16;
kv_entry **buckets = calloc(cap, sizeof *buckets);
if (buckets == NULL) {
return false;
}
for (size_t i = 0; i < s->cap; i++) {
kv_entry *e = s->buckets[i];
while (e != NULL) {
kv_entry *next = e->next; /* read it before the chain is changed */
size_t j = kv_hash(e->key, e->klen) & (cap - 1);
e->next = buckets[j];
buckets[j] = e;
e = next;
}
}
free(s->buckets);
s->buckets = buckets;
s->cap = cap;
return true;
}The line kv_entry *next = e->next must come first. The loop
sets e->next to the head of the new chain, and after that the old chain is gone.
Swap the two lines and the loop follows the new chain instead of the old one. It loses
every entry after the first, and ends in a cycle or a crash. The compiler accepts both orders.
Nothing but your reading of the code tells them apart.
Inserting has three allocations that can each fail, and the cleanup must free exactly the
ones that succeeded. The code calls free on every pointer, because free(NULL) does
nothing. That rule makes the failure path three lines instead of a tree of conditions:
/* Store a copy of val under a copy of key, replacing any old value. */
static bool kv_store_set(kv_store *s, const char *key, size_t klen, const char *val,
size_t vlen) {
char *v = kv_bytes_dup(val, vlen);
if (v == NULL) {
return false;
}
kv_entry *e = kv_store_find(s, key, klen);
if (e != NULL) {
free(e->val);
e->val = v;
e->vlen = vlen;
return true;
}
if (s->len >= s->cap && !kv_store_grow(s)) {
free(v);
return false;
}
e = malloc(sizeof *e);
char *k = kv_bytes_dup(key, klen);
if (e == NULL || k == NULL) {
free(e);
free(k);
free(v);
return false;
}
size_t i = kv_hash(key, klen) & (s->cap - 1);
*e = (kv_entry){k, klen, v, vlen, s->buckets[i]};
s->buckets[i] = e;
s->len++;
return true;
}Write kv_store_del. Walk the chain with a pointer to a pointer. Then removing the first entry of a chain and
removing a middle one are the same operation. When
the key matches, make the previous link skip the entry, free its key, its value and the entry
itself, and reduce the count.
static bool kv_store_del(kv_store *s, const char *key, size_t klen) {
if (s->cap == 0) return false;
size_t i = kv_hash(key, klen) & (s->cap - 1);
for (kv_entry **link = &s->buckets[i]; *link != NULL; link = &(*link)->next) {
/* your code here */
}
return false;
}Stuck?Show the full listing
/* Remove a key. The chain is walked with a pointer to the pointer that leads to each
* entry, so that the head of the chain and the middle of it are the same case. */
static bool kv_store_del(kv_store *s, const char *key, size_t klen) {
if (s->cap == 0) {
return false;
}
size_t i = kv_hash(key, klen) & (s->cap - 1);
for (kv_entry **link = &s->buckets[i]; *link != NULL; link = &(*link)->next) {
kv_entry *e = *link;
if (kv_bytes_eq(e->key, e->klen, key, klen)) {
*link = e->next;
free(e->key);
free(e->val);
free(e);
s->len--;
return true;
}
}
return false;
}
static void kv_store_free(kv_store *s) {
for (size_t i = 0; i < s->cap; i++) {
kv_entry *e = s->buckets[i];
while (e != NULL) {
kv_entry *next = e->next;
free(e->key);
free(e->val);
free(e);
e = next;
}
}
free(s->buckets);
s->buckets = NULL;
s->cap = s->len = 0;
}The pointer-to-pointer walk is an old C idiom, and it is worth a moment. link
points at whichever pointer currently leads to e. At the head of the chain that
is the bucket itself. In the middle it is the next field of the entry before.
Writing *link = e->next removes e in both cases without a special
case for the head. Linus Torvalds has said that code which special-cases the head tells him "this person
doesn't understand pointers". The version without the special case is also harder to read
the first time.
KEYS walks every bucket and every chain. It needs a pattern matcher, and the one here supports * and ?. The Go and Rust files hold the same function in their glob regions. The reply is an array whose length is
known only after the walk, so the keys are collected in a second buffer first:
/* KEYS pattern: every matching key, as an array. The count is known only after the
* walk, so the keys are collected first and the header is written in front of them. */
static bool kv_cmd_keys(const kv_cmd *c, kv_store *s, kv_buf *out) {
kv_buf keys = {0};
size_t n = 0;
for (size_t i = 0; i < s->cap; i++) {
for (kv_entry *e = s->buckets[i]; e != NULL; e = e->next) {
if (kv_glob(c->argv[1], c->arglen[1], e->key, e->klen)) {
if (!kv_reply_bulk(&keys, e->key, e->klen)) {
kv_buf_free(&keys);
return false;
}
n++;
}
}
}
bool ok = kv_reply_array(out, n) && kv_buf_append(out, keys.data, keys.len);
kv_buf_free(&keys);
return ok;
}Open kv.c and find each malloc and calloc. For each
one, find the free that releases it. The value of an entry is freed in kv_store_set on replacement. The key and the value are freed in kv_store_del and in kv_store_free. The bucket array is freed in kv_store_grow and in
kv_store_free. If you cannot find the pair for an allocation, that is a leak,
and chapter 12 shows how Valgrind finds it for you.
Where a value lives, and for how long#
Every object in C has a storage duration, and the table uses all
three. A local variable such as i in kv_store_grow has
automatic duration. It lives on the stack from its declaration to the end of its block,
and its address must not be kept after that. A static variable, and anything
declared outside a function, lives for the whole run, in memory the loader set up before
main. And everything malloc returns has allocated duration. It
lives on the heap until free, whatever block or function ends in between. The
entries must outlive the function that made them, so they are allocated. The bucket index
is needed for a few lines, so it is a local. Returning the address of a local is the
mistake the three durations make possible, and the compiler warns about the plain form
of it.
What C does instead of generics#
The table stores bytes under bytes, and nothing else. A table of integers under
strings would be a second copy of the file with the types changed. Go and Rust write the
table once with a type parameter, as the next two sections show. C has three ways to
avoid the copy, each with a cost, and the file generic.c in the chapter's
snapshot runs all three. make test builds it, and every claim in it is an
assert. The first way is a void * and an element size. The
function moves bytes without knowing what they are, and the caller casts them back:
/* 1. A void pointer and an element size. The caller says how big one element is, and the
* function moves bytes without knowing what they are. qsort and bsearch work this way.
* The compiler checks nothing: pass the wrong size and the sort is silently wrong. */
static int cmp_u32(const void *a, const void *b) {
uint32_t x = *(const uint32_t *)a;
uint32_t y = *(const uint32_t *)b;
return (x > y) - (x < y);
}
static void sort_demo(void) {
uint32_t xs[] = {30, 10, 20};
qsort(xs, 3, sizeof xs[0], cmp_u32);
assert(xs[0] == 10 && xs[1] == 20 && xs[2] == 30);
}The cast (const uint32_t *)a is a claim the compiler cannot check. Pass
qsort a size of 2 and it sorts garbage in silence. The second way is a macro
that writes the typed code for each type. The macro is the template, and each use is a
copy with T filled in. ## pastes two tokens into one name, so
name##_push becomes vec_u32_push:
/* 2. A macro that writes a typed container for each element type. The macro is the
* template, and each use is a copy with T filled in. A Rust generic compiles the same
* way, one copy per type, with the difference that Rust checks the template once. */
#define DEFINE_VEC(name, T) \
typedef struct { \
T *data; /* owned */ \
size_t len, cap; \
} name; \
static bool name##_push(name *v, T x) { \
if (v->len == v->cap) { \
size_t cap = v->cap ? v->cap * 2 : 8; \
T *p = realloc(v->data, cap * sizeof *p); \
if (p == NULL) { \
return false; \
} \
v->data = p; \
v->cap = cap; \
} \
v->data[v->len++] = x; \
return true; \
}
DEFINE_VEC(vec_u32, uint32_t)
DEFINE_VEC(vec_str, const char *)
static void vec_demo(void) {
vec_u32 nums = {0};
vec_str words = {0};
for (uint32_t i = 0; i < 100; i++) {
assert(vec_u32_push(&nums, i * i));
}
assert(vec_str_push(&words, "hello") && vec_str_push(&words, "world"));
assert(nums.len == 100 && nums.data[9] == 81);
assert(strcmp(words.data[1], "world") == 0);
free(nums.data);
free(words.data);
}That is also how a Rust generic is compiled, one copy per type. The difference is that
Rust checks the template once. A C macro is checked only when it is used, in error
messages that point at the use. The third way is _Generic, a compile-time
switch on the type of an expression, which lets one name reach several functions:
/* 3. _Generic: a compile-time switch on the type of an expression. It is how C11 gives
* one name to several functions, which is as close as C gets to overloading. */
static const char *name_u32(uint32_t x) {
(void)x;
return "uint32_t";
}
static const char *name_str(const char *s) {
(void)s;
return "const char *";
}
#define type_name(x) _Generic((x), uint32_t: name_u32, const char *: name_str, char *: name_str)(x)
static void generic_demo(void) {
uint32_t n = 7;
const char *s = "seven";
assert(strcmp(type_name(n), "uint32_t") == 0);
assert(strcmp(type_name(s), "const char *") == 0);
}Redis uses the first way and the second. Its dictionary stores void * keys
and values, with a table of function pointers that say how to hash, compare and free them.
Chapter 7 returns to that table, because it is also how C does an interface.
What a struct is made of, and an arena#
A kv_entry is 40 bytes on this machine, five fields of 8. Change the
order of fields in a struct, or mix sizes, and the compiler inserts padding. Every type
has an alignment, and a member must start at a multiple of its own.
offsetof reads where a member starts, and alignof reads the
alignment. _Static_assert checks a claim about either when the file is
compiled, so a wrong one never runs:
/* What a struct is made of. Every type has an alignment, and the compiler inserts padding
* so that each member starts at a multiple of its own alignment. One byte of tag before
* an eight-byte number costs seven bytes of padding, and reordering the members saves it.
* offsetof reads the layout, and _Static_assert checks it when the program is compiled. */
typedef struct {
uint8_t tag;
uint64_t n;
uint8_t flag;
} loose;
typedef struct {
uint64_t n;
uint8_t tag;
uint8_t flag;
} tight;
_Static_assert(sizeof(loose) == 24, "two bytes of tags cost 14 bytes of padding");
_Static_assert(sizeof(tight) == 16, "the same members, reordered, need 6");
_Static_assert(offsetof(loose, n) == 8, "n starts at the next multiple of 8");
_Static_assert(alignof(uint64_t) == 8, "a 64-bit integer is aligned to 8 bytes");
static void layout_demo(void) {
printf("loose %zu bytes, tight %zu bytes, char is %s\n", sizeof(loose), sizeof(tight),
CHAR_MIN < 0 ? "signed" : "unsigned");
}Chapter 15 needs these numbers, because a struct that crosses into Go or Rust must have the same layout on both sides. The last demonstration is an arena, the other way to own memory in C. One block is given out from the front, each piece rounded up to the largest alignment, and the whole block is freed at once. A parser that builds a tree for one request frees a thousand things with one call. So does a game that throws away a frame, and no piece leaks:
/* An arena: one block, handed out from the front, freed all at once. Each piece starts
* at a multiple of the largest alignment, so any type may live in it. A parser that
* builds a tree per request, or a game that frees everything per frame, uses one. */
typedef struct {
char *base; /* owned */
size_t used;
size_t cap;
} arena;
static void *arena_alloc(arena *a, size_t n) {
size_t align = alignof(max_align_t);
size_t start = (a->used + align - 1) / align * align; /* round up */
if (start + n > a->cap) {
return NULL;
}
a->used = start + n;
return a->base + start;
}
static void arena_demo(void) {
arena a = {malloc(4096), 0, 4096};
assert(a.base != NULL);
char *name = arena_alloc(&a, 5);
uint64_t *count = arena_alloc(&a, sizeof *count);
memcpy(name, "kv!!", 5);
*count = 42;
assert((uintptr_t)count % alignof(uint64_t) == 0); /* padded past the 5 bytes */
assert(a.used == 16 + sizeof *count);
free(a.base); /* one free for every piece */
}$ make test
loose 24 bytes, tight 16 bytes, char is signed
generic: all checks passedGo: the same table, with nothing to free#
The Go version keeps the shape of the C one and drops the bookkeeping. A chain is a
slice, so there is no next pointer. The entry has two fields, and the types
of both are type parameters. K comparable says the key may
be any type that supports ==, which is what a lookup needs. V
any says the value may be anything at all. The table is written once, and the
compiler produces the code for each pair of types a program uses. The hash comes from
the caller as a function, because comparable promises equality and nothing
about bytes:
// fnv1a hashes bytes the way kv does in C, so the two tables spread keys the same way.
func fnv1a(b []byte) uint64 {
h := uint64(14695981039346656037)
for _, c := range b {
h ^= uint64(c)
h *= 1099511628211
}
return h
}// entry is one key and its value in a chain. There is no next pointer: a chain is a
// slice, and the garbage collector owns everything. K and V are type parameters: the
// table is written once, for any key that supports == and any value at all.
type entry[K comparable, V any] struct {
key K
val V
}
// Table is a hash table with separate chaining. It grows when it holds more entries
// than buckets, so a chain stays short. The hash function comes from the caller, because
// the constraint comparable promises == and nothing else.
type Table[K comparable, V any] struct {
hash func(K) uint64
buckets [][]entry[K, V]
n int
}
func NewTable[K comparable, V any](hash func(K) uint64) *Table[K, V] {
return &Table[K, V]{hash: hash}
}
func (t *Table[K, V]) Get(key K) (V, bool) {
if len(t.buckets) == 0 {
var zero V
return zero, false
}
chain := t.buckets[t.hash(key)&uint64(len(t.buckets)-1)]
for i := range chain {
if chain[i].key == key {
return chain[i].val, true
}
}
var zero V
return zero, false
}A constraint is an interface used as a bound. comparable
and any are built in. cmp.Ordered covers the types with a less-than,
and a union such as ~int | ~string names a set of types.
Go had no generics until version 1.18 in 2022, thirteen years into its life, and the
design kept them small on purpose. There are no generic methods, and a type parameter
cannot be used as a receiver. The question of when to use them has a short answer. A
container or an algorithm over several types, such as this table, is a generic. A
function that takes behaviour is an interface, which chapter 7 covers.
Write the grow method. Make a slice of buckets twice as long. Walk every old chain, and append each entry to its
new chain. Use fnv1a and the new length to find the index. Then replace t.buckets. Nothing is freed, because nothing in Go
is freed by you.
func (t *table) grow() {
buckets := make([][]entry, 2*len(t.buckets))
// your code here
t.buckets = buckets
}Stuck?Show the full listing
func (t *Table[K, V]) Set(key K, val V) {
if len(t.buckets) == 0 {
t.buckets = make([][]entry[K, V], 16)
}
i := t.hash(key) & uint64(len(t.buckets)-1)
chain := t.buckets[i]
for j := range chain {
if chain[j].key == key {
chain[j].val = val
return
}
}
if t.n >= len(t.buckets) {
t.grow()
i = t.hash(key) & uint64(len(t.buckets)-1)
}
t.buckets[i] = append(t.buckets[i], entry[K, V]{key, val})
t.n++
}
// grow doubles the bucket slice and moves every entry to its new chain. The old slices
// become garbage, and the collector frees them.
func (t *Table[K, V]) grow() {
buckets := make([][]entry[K, V], 2*len(t.buckets))
for _, chain := range t.buckets {
for _, e := range chain {
i := t.hash(e.key) & uint64(len(buckets)-1)
buckets[i] = append(buckets[i], e)
}
}
t.buckets = buckets
}The old slices become garbage when t.buckets is replaced, and the collector
frees them at some later moment. Deleting moves the last entry of a chain into the hole, as the C list did in chapter 5. The
removed entry is garbage too:
func (t *Table[K, V]) Del(key K) bool {
if len(t.buckets) == 0 {
return false
}
i := t.hash(key) & uint64(len(t.buckets)-1)
chain := t.buckets[i]
for j := range chain {
if chain[j].key == key {
// move the last entry into the hole, then shorten the chain by one
chain[j] = chain[len(chain)-1]
t.buckets[i] = chain[:len(chain)-1]
t.n--
return true
}
}
return false
}
// Len is the number of entries, and Keys is every key in no particular order.
func (t *Table[K, V]) Len() int { return t.n }
func (t *Table[K, V]) Keys() []K {
keys := make([]K, 0, t.n)
for _, chain := range t.buckets {
for _, e := range chain {
keys = append(keys, e.key)
}
}
return keys
}The table is never used by the server. It is checked by a test, which runs five thousand
random operations against a built-in map and stops at the first disagreement. The test is
in table_test.go, and go test finds it by the file name and the
function name:
// TestTableAgainstMap runs random operations on the table and on a map, and checks
// after each one that the table agrees with the map. The table is instantiated with
// string keys and string values; the compiler writes that version of the code.
func TestTableAgainstMap(t *testing.T) {
tb := NewTable[string, string](hashString)
ref := map[string]string{}
r := rand.New(rand.NewPCG(1, 2))
for step := range 5000 {
key := fmt.Sprintf("k%d", r.IntN(200))
switch r.IntN(3) {
case 0:
val := fmt.Sprintf("v%d", step)
tb.Set(key, val)
ref[key] = val
case 1:
_, want := ref[key]
if got := tb.Del(key); got != want {
t.Fatalf("step %d: del %q = %v, want %v", step, key, got, want)
}
delete(ref, key)
case 2:
want, ok := ref[key]
got, found := tb.Get(key)
if found != ok || got != want {
t.Fatalf("step %d: get %q = %q,%v, want %q,%v", step, key, got, found, want, ok)
}
}
if tb.Len() != len(ref) {
t.Fatalf("step %d: table has %d entries, map has %d", step, tb.Len(), len(ref))
}
}
// the generic slices and maps packages: collect the map's keys, sort both, compare
want := slices.Sorted(maps.Keys(ref))
got := tb.Keys()
slices.Sort(got)
if !slices.Equal(got, want) {
t.Fatalf("keys differ: %v vs %v", got, want)
}
}
// TestTableWithIntKeys is the same table with another key type. Nothing in table.go
// changed; the compiler made a second copy for int keys.
func TestTableWithIntKeys(t *testing.T) {
tb := NewTable[int, []byte](func(k int) uint64 { return uint64(k) * 11400714819323198485 })
for i := range 100 {
tb.Set(i, []byte{byte(i)})
}
if v, ok := tb.Get(42); !ok || v[0] != 42 {
t.Fatalf("get 42 = %v, %v", v, ok)
}
if tb.Del(42) != true || tb.Len() != 99 {
t.Fatal("del")
}
}$ go test
PASS
ok kv 0.425s
$ go test -bench Get -benchmem
BenchmarkTableGet-12 70199774 17.49 ns/op 0 B/op 0 allocs/op
BenchmarkMapGet-12 62301384 20.14 ns/op 0 B/op 0 allocs/opThe test ends with the generic standard packages. maps.Keys walks the
reference map, and slices.Sorted orders it. slices.Sort orders
the table's list and slices.Equal compares both lists. Before 1.18 each of those
was a loop you wrote. The second command runs the two benchmarks at the bottom of the test
file. On the author's machine the hand-written table answers a lookup in 17 nanoseconds
and the built-in map in 20. That is not a reason to use your own. The built-in map grows
without a pause, and is what every Go programmer reads without thinking. The store keeps
the map, and adds the two new commands:
// store keeps every key and its value in a map. The hand-written table in table.go
// would work too, and the tests show it does, but the map is faster and nobody would
// use anything else in Go.
type store struct {
m map[string][]byte
}
func newStore() *store {
return &store{m: make(map[string][]byte)}
}
func (s *store) get(key []byte) ([]byte, bool) {
v, ok := s.m[string(key)]
return v, ok
}
func (s *store) set(key, val []byte) {
s.m[string(key)] = val
}
func (s *store) del(key []byte) bool {
_, ok := s.m[string(key)]
delete(s.m, string(key))
return ok
}
func (s *store) size() int {
return len(s.m)
}
// keys returns every key that matches the pattern, in no particular order.
func (s *store) keys(pattern []byte) [][]byte {
var out [][]byte
for k := range s.m {
if glob(pattern, []byte(k)) {
out = append(out, []byte(k))
}
}
return out
}Where the compiler puts a value#
Nothing in the Go table says stack or heap, and the compiler decides. A value that no pointer outlives stays on the stack of its function, which is free. A value that a pointer could reach after the function returns moves to the heap, where the collector must find it later. The decision is called escape analysis, and the compiler will show it:
$ go build -gcflags=-m . 2>&1 | grep -E 'table.go|store.go' | head -4
./table.go:8:6: can inline fnv1a
./table.go:8:12: b does not escape
./store.go:30:21: key does not escape
./store.go:31:22: string(key) does not escapeb does not escape means the slice passed to fnv1a never
leaves it, so no copy is made. string(key) does not escape in
del is the compiler noticing something. The string made from the key is only
used to look in the map, so it is built on the stack and never allocated. In
set the same conversion escapes, because the map keeps it. This is the tool
to open when a Go program allocates more than you expect, and chapter 13 does.
append may or may not allocate. When a chain has room, the new
entry goes into the existing array, and any other slice that pointed at that array sees it.
In del, the shortened slice chain[:len(chain)-1] still points at
the same array as before. The slot past the new end still holds a copy of the entry that was moved. Its key and value stay reachable until the next append writes over them. None of this is a bug here. It is the reason that a Go programmer must know what a slice is. A slice is a pointer, a
length and a capacity, and not an array.
Rust: the table that drops itself#
In Rust the chain is a Vec<Entry>, and the entry owns its key and value.
Dropping the table drops the outer Vec, which drops every chain, which drops
every entry, which frees every byte. No function in table.rs frees anything,
and nothing leaks. The table is generic too, in two type parameters. The
where clause on the impl block is the bound.
It says what the table needs from a key, a view of it as bytes to hash and equality to
find it. AsRef<[u8]> and PartialEq are traits, which
chapter 7 explains. A key type that has both compiles. One that lacks either is refused
at the impl, not deep inside a method:
/// FNV-1a, the same hash the C version of kv uses.
fn fnv1a(bytes: &[u8]) -> u64 {
bytes.iter().fold(0xcbf2_9ce4_8422_2325, |h, &b| {
(h ^ u64::from(b)).wrapping_mul(0x0100_0000_01b3)
})
}/// One key and its value in a chain. The chain is a Vec, which owns its entries, and
/// the entries own their bytes. Dropping the table drops all of it. `K` and `V` are type
/// parameters: the table is written once for any key that can be seen as bytes and
/// compared, and any value at all.
struct Entry<K, V> {
key: K,
val: V,
}
/// A hash table with separate chaining. It grows when it holds more entries than
/// buckets, so a chain stays short.
pub struct Table<K, V> {
buckets: Vec<Vec<Entry<K, V>>>,
len: usize,
}
/// An empty table, for every `K` and `V`. `derive(Default)` would also demand that `K`
/// and `V` have defaults, which an empty table does not need, so this is written out.
impl<K, V> Default for Table<K, V> {
fn default() -> Self {
Self {
buckets: Vec::new(),
len: 0,
}
}
}
/// The bounds say what the table needs from a key: a view of it as bytes, to hash, and
/// equality, to find it. A key type with more is fine; one with less does not compile.
impl<K, V> Table<K, V>
where
K: AsRef<[u8]> + PartialEq,
{
fn index(&self, key: &K) -> usize {
(fnv1a(key.as_ref()) & (self.buckets.len() as u64 - 1)) as usize
}
pub fn get(&self, key: &K) -> Option<&V> {
if self.buckets.is_empty() {
return None;
}
let chain = &self.buckets[self.index(key)];
chain.iter().find(|e| e.key == *key).map(|e| &e.val)
}Two shapes in this file have appeared since chapter 4 without a name. The hash is a
fold, an iterator method that walks the bytes and carries
a value along. get is iter().find(...).map(...), a chain where
each call wraps the one before and nothing runs until the end asks for a value. And the
|e| e.key == *key inside find is a closure,
a function written in place that captures key from the method around it.
The C loop with an index, the Go for range and this chain do the same work.
Chapter 8 gives iterators and closures a section of their own, because SCAN
needs both.
The compiler produces one copy of every method for each pair of types a program uses,
which is called monomorphisation. The copy for byte keys and the copy for &str keys share no code at run
time. So a generic costs nothing to call and something to compile. The
Default block is written out rather than derived. The derive would also
demand a default for the key and the value types, which an empty table does not need.
Write set. If the buckets are empty, create sixteen. Find the chain by
index. If an entry with the key exists, replace its value. Otherwise grow when
the table is full, compute the index again, because the length changed, and push a new
entry. The compiler will tell you if you hold a borrow of a chain across the call to
grow.
pub fn set(&mut self, key: Vec<u8>, val: Vec<u8>) {
if self.buckets.is_empty() {
self.buckets.resize_with(16, Vec::new);
}
// your code here
}Stuck?Show the full listing
pub fn set(&mut self, key: K, val: V) {
if self.buckets.is_empty() {
self.buckets.resize_with(16, Vec::new);
}
let i = self.index(&key);
if let Some(e) = self.buckets[i].iter_mut().find(|e| e.key == key) {
e.val = val;
return;
}
if self.len >= self.buckets.len() {
self.grow();
}
let i = self.index(&key);
self.buckets[i].push(Entry { key, val });
self.len += 1;
}
/// Double the buckets and move every entry to its new chain. The entries are moved,
/// not copied: `drain` takes them out of the old chains.
fn grow(&mut self) {
let mut buckets: Vec<Vec<Entry<K, V>>> = Vec::new();
buckets.resize_with(self.buckets.len() * 2, Vec::new);
let mask = buckets.len() as u64 - 1;
for chain in &mut self.buckets {
for e in chain.drain(..) {
let i = (fnv1a(e.key.as_ref()) & mask) as usize;
buckets[i].push(e);
}
}
self.buckets = buckets;
}Look at grow. The call chain.drain(..) moves every entry out of
the old chain and into the new one. There is no copy of a key or a value, as in C, and the
old chains are empty when the loop ends. The last line replaces self.buckets,
and the old outer Vec is dropped at that moment, not later. Deleting uses
swap_remove, which does what the Go version did and drops the removed entry on
the same line:
pub fn del(&mut self, key: &K) -> bool {
if self.buckets.is_empty() {
return false;
}
let i = self.index(key);
let chain = &mut self.buckets[i];
match chain.iter().position(|e| e.key == *key) {
Some(j) => {
// move the last entry into the hole; the removed entry is dropped here
chain.swap_remove(j);
self.len -= 1;
true
}
None => false,
}
}
pub fn len(&self) -> usize {
self.len
}
pub fn is_empty(&self) -> bool {
self.len == 0
}
}The test lives in the same file, in a module marked #[cfg(test)], which the
compiler leaves out of the real binary:
#[cfg(test)]
mod tests {
use super::Table;
use std::collections::HashMap;
/// A small deterministic random number generator, so the test is the same every run.
fn next(state: &mut u64) -> u64 {
*state ^= *state << 13;
*state ^= *state >> 7;
*state ^= *state << 17;
*state
}
#[test]
fn agrees_with_hashmap() {
let mut table: Table<Vec<u8>, Vec<u8>> = Table::default();
let mut reference: HashMap<Vec<u8>, Vec<u8>> = HashMap::new();
let mut rng = 0x2545_f491_4f6c_dd1d_u64;
for step in 0..5000 {
let key = format!("k{}", next(&mut rng) % 200).into_bytes();
match next(&mut rng) % 3 {
0 => {
let val = format!("v{step}").into_bytes();
table.set(key.clone(), val.clone());
reference.insert(key, val);
}
1 => {
assert_eq!(
table.del(&key),
reference.remove(&key).is_some(),
"step {step}"
);
}
_ => {
assert_eq!(table.get(&key), reference.get(&key));
}
}
assert_eq!(table.len(), reference.len());
}
}
/// The same table with another key type. `&str` is bytes with `as_ref`, and a
/// `u32` value needs nothing. The compiler made a second copy of every method.
#[test]
fn works_with_str_keys_and_integer_values() {
let mut table: Table<&str, u32> = Table::default();
for (i, word) in ["one", "two", "three"].iter().enumerate() {
table.set(word, i as u32);
}
assert_eq!(table.get(&"two"), Some(&1));
assert!(table.del(&"two"));
assert_eq!(table.len(), 2);
}$ cargo test
running 3 tests
test table::tests::agrees_with_hashmap ... ok
test table::tests::works_with_str_keys_and_integer_values ... ok
test table::tests::two_owners_with_rc_and_refcell ... okTwo owners of one table#
Every value in Rust has one owner, and the table so far has had one. A program
sometimes needs two parts to hold one thing, and no part is the owner. The standard
library has two types for that in one thread. Rc is a reference
count. Cloning an Rc makes a second owner and adds one to the count, and the
value is dropped when the count reaches zero. RefCell moves the borrow rules
from compile time to run time. borrow and borrow_mut give a
shared or a mutable reference, and a second mutable borrow while one is live is a panic.
The rule did not go away. It moved to where the compiler could not see it. This is called
interior mutability. Cell is the same idea for a small
copyable value, with a getter and a setter instead of references. The third test shows all three:
/// Two owners of one table, in one thread. `Rc` counts the owners and frees the table
/// when the last one goes; `RefCell` moves the borrow rules to run time, so each
/// owner can borrow it mutably in turn. A second mutable borrow at the same moment
/// is a panic, not a compile error: the check moved, it did not go away.
#[test]
fn two_owners_with_rc_and_refcell() {
use std::cell::{Cell, RefCell};
use std::rc::Rc;
let shared: Rc<RefCell<Table<&str, u32>>> = Rc::new(RefCell::new(Table::default()));
let writer = Rc::clone(&shared);
let reader = Rc::clone(&shared);
assert_eq!(Rc::strong_count(&shared), 3);
writer.borrow_mut().set("hits", 1);
assert_eq!(reader.borrow().get(&"hits"), Some(&1));
drop(writer);
assert_eq!(Rc::strong_count(&shared), 2);
// Cell is the same idea for a Copy value: change it through a shared reference
let counter = Cell::new(0);
let bump = |c: &Cell<u32>| c.set(c.get() + 1);
bump(&counter);
bump(&counter);
assert_eq!(counter.get(), 2);
// the run-time check: a borrow_mut while a borrow is live
let guard = shared.borrow();
assert!(shared.try_borrow_mut().is_err());
drop(guard);
assert!(shared.try_borrow_mut().is_ok());
}Both types refuse to cross a thread. Chapter 9 replaces them with Arc and
Mutex, which are the same two ideas with a lock. A chain written as a linked
list, the way the C table does it, would need one more type, Box. A struct
cannot contain itself without a pointer, and chapter 8 writes one.
As in Go, the server keeps the standard HashMap. The standard map uses a
hash called SipHash 1-3, which the documentation describes as "selected to provide resistance
against HashDoS attacks". An attacker who knows the hash function can send thousands of keys that land in one
bucket. Every lookup of such a key then walks that one chain, which holds every key the attacker sent. FNV-1a has no
defence against that. Redis uses SipHash too, and seeds it with random bytes at startup, for the same reason.
On 28 December 2011 one HTTP request could hold a web server at full CPU for hours. Alexander Klink and Julian Wรคlde showed where the hole was. Nearly every web platform kept its form parameters in a hash table with a known hash and no secret. A POST with tens of thousands of parameter names chosen to collide turned every insert into a walk of one long chain. The advisory listed Java, PHP, Python, Ruby, the V8 JavaScript engine and the servers built on them. The fix across the industry was a hash with a secret seed, and SipHash was designed in 2012 for that purpose. Your FNV-1a table has exactly this hole. The standard maps of Go and Rust, and Redis, do not.
A module nobody calls produces a warning for every function in it. The
binary never calls Table, so the compiler reports each method as dead code, and
the book's clippy -D warnings turns that into an error. The file opens with
#![allow(dead_code)] and a comment that says why. In C, an unused function that is not static compiles in silence, because the compiler cannot know whether another file calls it. A static one gets a warning under -Wall, which the book's -Werror turns into an error.
Memory, three ways#
The chapter touched memory in every function. The table shows what each language asked of you.
| The question | C | Go | Rust |
|---|---|---|---|
| Who frees an entry | you, in three places that must agree | the collector, when nothing refers to it | the owner, at the end of its scope |
| Forgetting to free | a leak that grows until the process dies | impossible for unreachable memory | impossible, the compiler inserts the drop |
| Freeing twice | undefined behaviour, often a security hole | impossible | does not compile |
| Using after free | undefined behaviour, often a security hole | impossible, the reference keeps it alive | does not compile |
| Keeping a pointer across a move | your responsibility, and grow relies on it | safe, the entry is reachable from the new slice | checked, and set recomputes the index after grow |
| The cost | your attention | memory held until the next collection, and a pause | the time to satisfy the borrow checker |
The C table is about 140 lines, and about a third of them exist to allocate, check and
free. The Go table is 75 lines with none of that. The price is a collector that runs in the
background and holds freed memory until it does. The Rust table is also 75 lines, and it
frees memory at the moment the C one does. The price is a compiler that refused several
early drafts of set. Each table is the right answer for its language, and the
differences between them are the differences between the languages.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| Writing the table | Weak | Strong | OK | C needs an owner comment on every pointer and three cleanup paths, Go is the shortest, and Rust argues about borrows in set. |
| Freeing memory | Weak | Strong | Strong | C must free in three places that agree. Go and Rust free for you, in different ways. |
| Freeing on time | Strong | Weak | Strong | C and Rust free on the line that removes an entry. Go frees at the next collection. |
| Testing it | Weak | Strong | Strong | C has no test runner in the box. go test and cargo test are built in. |
| Shipping unused code | Weak | OK | Strong | C warns only for static functions and Go ignores unused functions, while Rust reports each one. |
| Needing to write one | Strong | Weak | Weak | In C the table is the store. In Go and Rust it is an exercise, because the map is built in. |
| Writing it once for any type | Weak | OK | Strong | C has void pointers, macros and _Generic, each with a hole. Go has type parameters since 2022, and Rust has bounds the compiler checks at the impl. |
| Seeing where a value lives | Strong | OK | Strong | C says it in every line, and Rust says it in the types. Go decides for you and shows its decision with a flag. |
- Storage duration. A C value lives on the stack to the end of its block, in static memory for the run, or on the heap until freed.
- Type parameter. A table written once for any key and value, with a bound that says what the type must support.
- Rc and RefCell. Two owners of one value in one thread, with the borrow rules checked at run time instead of compile time.
Modules, packages and dependencies#
The chapter 6 file is 868 lines. Cut it the obvious way, by moving the hash table into
store.c as it stands, and compile the two files together:
$ wc -l kv.c
868 kv.c
$ cc -std=c17 -Wall -Wextra -Werror -o kv kv.c store.c
kv.c:357:44: error: unknown type name 'kv_store'
kv.c:359:42: error: unknown type name 'kv_store'
kv.c:370:42: error: unknown type name 'kv_store'
kv.c:375:41: error: unknown type name 'kv_store'
...Twenty-five errors. Every function that uses the store has lost the word
kv_store. The file that defines it is now a different file, and C compiles
one file at a time. The cut needs a third file, which says what the second one
offers.
kv is nearly nine hundred lines in one file, and the next chapters double it. This chapter splits
the server into parts, in each language's own way. C gets header files and a Makefile that
knows about them. Go gets packages under internal. Rust gets a library crate
with a thin binary. It also covers dependencies, and the tools that keep code in one style.
The server does not change in this chapter. Every test from chapter 6 still passes, and no command is added. What changes is where the code lives, how one part sees another, and how a reader finds anything. That sounds like housekeeping. It is also where the three languages differ most in their daily feel. A module system is what you touch every time you open a file.
| The unit | C | Go | Rust |
|---|---|---|---|
| A module | a .c file with a .h beside it | a folder, one package | a file, declared with mod |
| A private name | static | a lower-case first letter | the default, without pub |
| A public name | declared in the header, prefixed kv_ | an upper-case first letter | pub |
| Using another module | #include "store.h", copied in by the preprocessor | import "kv/internal/store" | use kv::store::Store |
| Building many files | a Makefile that lists them, or finds them | go build ./cmd/kv | cargo build |
| A dependency | copy the source in, or a system package | go get, recorded in go.mod | cargo add, recorded in Cargo.toml |
| The exact versions | wherever you wrote them down | go.sum | Cargo.lock |
| The registry | none | any Git host, through a proxy | crates.io |
| The formatter | clang-format, with a config file | gofmt, with no options | rustfmt, with few |
| Behaviour a caller may swap | a struct of function pointers, and an opaque type | an interface, satisfied by having the methods | a trait, implemented with impl Store for |
A module system is a promise about what stays the same. Go 1 was released in 2012 with a written promise. Programs written to the Go 1 specification "will continue to compile and run correctly, unchanged, over the lifetime of that specification". Fourteen years later the promise holds, and a Go package from 2012 is meant to still build. Rust made a different promise. The language can change in ways that would break old code, but only in a new edition. A
crate names its edition in Cargo.toml. The 2024 edition shipped with Rust 1.85 in February 2025. A crate
on the 2015 edition still builds with today's compiler, next to one on the 2024 edition, in
the same program. C made no promise at all, and relies on the fact that no compiler vendor
dares to break the code that already exists.
C: headers, static, and a Makefile that knows about both#
C has no modules. It has files, and a convention. A module is a .c file that
holds the code and a .h file that holds the declarations other files may use.
The buffer from chapter 4 becomes two files. The header is the public part:
#ifndef KV_BUF_H
#define KV_BUF_H
#include <stdbool.h>
#include <stddef.h>
/* A growable byte buffer. It owns data, and kv_buf_free releases it. Start one with
* kv_buf b = {0}. */
typedef struct {
char *data; /* owned; NULL until the first append */
size_t len;
size_t cap;
} kv_buf;
bool kv_buf_append(kv_buf *b, const void *src, size_t n);
void kv_buf_consume(kv_buf *b, size_t n);
void kv_buf_free(kv_buf *b);
#endifThree conventions are at work. The first lines and the last line are a header
guard. The preprocessor copies a header into every file that includes it. A file that includes
buf.h twice, once directly and once through resp.h, would see the
typedef twice. The guard makes the second
copy empty. The second convention is the prefix. C has one namespace for the whole program. So every public name in kv begins with
kv_, and a name from another library cannot collide with it. The third is that the header says what the caller must do, in
comments, because nothing else will. The .c file holds the rest, and marks the
one function that only it uses as static:
/* Private: only this file calls it, so it is static and not in the header. */
static bool kv_buf_reserve(kv_buf *b, size_t extra) {
if (b->len + extra <= b->cap) {
return true;
}
size_t cap = b->cap ? b->cap * 2 : 1024;
while (cap < b->len + extra) {
cap *= 2;
}
char *p = realloc(b->data, cap);
if (p == NULL) {
return false; /* the old block is still valid and still ours */
}
b->data = p;
b->cap = cap;
return true;
}A static function is invisible outside its file. That is the whole of
information hiding in C. There is no private keyword and no module boundary, only a function
the linker never hears about. Every file in src/ follows the same shape. The header of the store is the most useful one to read. It says who owns every pointer,
and how long a pointer from kv_store_find stays valid:
Write store.h for the hash table from chapter 6. It needs a guard, the two
structures, and a declaration for each of the four public functions. Put a comment on the
entry that says what it owns, and a comment on kv_store_find that says how long
the returned pointer is valid. Everything that only store.c uses stays out of the header and becomes
static. That includes the hash function and the grow function.
#ifndef KV_STORE_H
#define KV_STORE_H
#include <stdbool.h>
#include <stddef.h>
/* your declarations here */
#endifStuck?Show the full listing
#ifndef KV_STORE_H
#define KV_STORE_H
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
/* The keys and values, in a hash table with separate chaining. */
/* One key and its value, in a chain of entries that share a bucket. The key, the value
* and the next entry are owned. */
typedef struct kv_entry {
char *key;
size_t klen;
char *val;
size_t vlen;
struct kv_entry *next;
} kv_entry;
/* How the table treats a key: a struct of function pointers, which is how C describes
* behaviour a caller may choose. A store is made with one of the types below, and every
* hash and comparison inside it goes through that type. Redis calls the same thing a
* dictType. */
typedef struct {
const char *name;
uint64_t (*hash)(const char *key, size_t klen, uint64_t seed);
bool (*eq)(const char *a, size_t alen, const char *b, size_t blen);
bool seeded; /* kv_store_new draws a random seed, so keys cannot be chosen to collide */
} kv_store_type;
extern const kv_store_type kv_fnv_type; /* FNV-1a, the same hash on every run */
extern const kv_store_type kv_seeded_type; /* FNV-1a mixed with a seed drawn at start */
/* The store itself is opaque: this header says it exists, store.c says what is in it.
* Other files hold a pointer and call the functions below, and cannot reach inside. */
typedef struct kv_store kv_store;
/* A new, empty store of the given type, or NULL when out of memory. Free it with
* kv_store_free, which also frees every entry. */
kv_store *kv_store_new(const kv_store_type *type);
void kv_store_free(kv_store *s);
size_t kv_store_len(const kv_store *s);
/* The entry for key, or NULL. The pointer is valid until the next set or del. */
kv_entry *kv_store_find(kv_store *s, const char *key, size_t klen);
/* Store a copy of val under a copy of key, replacing any old value. */
bool kv_store_set(kv_store *s, const char *key, size_t klen, const char *val, size_t vlen);
bool kv_store_del(kv_store *s, const char *key, size_t klen);
/* Call fn once for every entry, with ctx passed through, until fn returns false. The
* callback must not set or del: the chain it is walking would move under it. */
void kv_store_each(kv_store *s, bool (*fn)(kv_entry *e, void *ctx), void *ctx);
#endifAn opaque type, and a struct of functions#
The header above still shows the fields of kv_store. So any file that
includes it can reach inside and walk the buckets, as KEYS did in chapter 6.
The second half of the header takes that away. It declares the struct without defining
it, and store.c holds the definition. Every other file has a pointer to
something it cannot look inside. That is called an opaque type, and it
is the strongest form of hiding C has. FILE from stdio.h is the
one you have used all along:
/* The store itself is opaque: this header says it exists, store.c says what is in it.
* Other files hold a pointer and call the functions below, and cannot reach inside. */
typedef struct kv_store kv_store;
/* A new, empty store of the given type, or NULL when out of memory. Free it with
* kv_store_free, which also frees every entry. */
kv_store *kv_store_new(const kv_store_type *type);
void kv_store_free(kv_store *s);
size_t kv_store_len(const kv_store *s);
/* The entry for key, or NULL. The pointer is valid until the next set or del. */
kv_entry *kv_store_find(kv_store *s, const char *key, size_t klen);
/* Store a copy of val under a copy of key, replacing any old value. */
bool kv_store_set(kv_store *s, const char *key, size_t klen, const char *val, size_t vlen);
bool kv_store_del(kv_store *s, const char *key, size_t klen);
/* Call fn once for every entry, with ctx passed through, until fn returns false. The
* callback must not set or del: the chain it is walking would move under it. */
void kv_store_each(kv_store *s, bool (*fn)(kv_entry *e, void *ctx), void *ctx);Because nobody outside can walk the table, the table walks itself.
kv_store_each takes a function pointer and a void *, and calls
the function once per entry with that pointer passed back. The void * is the
context, whatever the caller needs the callback to see. The callback casts it back to
the real type on its first line. This pair, a function pointer and a context pointer, is
how C passes a closure, and KEYS now uses it:
void kv_store_each(kv_store *s, bool (*fn)(kv_entry *e, void *ctx), void *ctx) {
for (size_t i = 0; i < s->cap; i++) {
for (kv_entry *e = s->buckets[i]; e != NULL; e = e->next) {
if (!fn(e, ctx)) {
return;
}
}
}
}/* What the KEYS walk needs to know, passed to the callback through a void pointer. */
typedef struct {
const char *pat;
size_t plen;
kv_buf keys; /* the matching keys, as replies, collected before the count is known */
size_t n;
bool ok;
} kv_keys_ctx;
/* Called once per entry by kv_store_each. The context comes back as a void pointer, and
* the first line turns it into the real type again. */
static bool kv_keys_visit(kv_entry *e, void *ctx) {
kv_keys_ctx *k = ctx;
if (!kv_glob(k->pat, k->plen, e->key, e->klen)) {
return true;
}
if (!kv_reply_bulk(&k->keys, e->key, e->klen)) {
k->ok = false;
return false; /* stop the walk */
}
k->n++;
return true;
}
/* KEYS pattern: every matching key, as an array. The walk is the store's; this file only
* supplies what to do with each entry. */
static bool kv_cmd_keys(const kv_cmd *c, kv_store *s, kv_buf *out) {
kv_keys_ctx k = {c->argv[1], c->arglen[1], {0}, 0, true};
kv_store_each(s, kv_keys_visit, &k);
bool ok = k.ok && kv_reply_array(out, k.n) && kv_buf_append(out, k.keys.data, k.keys.len);
kv_buf_free(&k.keys);
return ok;
}The header also lets a caller choose how the table treats a key. A
kv_store_type is a struct whose fields are function pointers, one for the
hash and one for equality, and a store is made with one. Inside store.c every
hash is s->type->hash(...), a call through a pointer, so the table never
names a hash function. This is the C form of an interface, and Redis builds its
dictionary on exactly this struct, which it calls dictType:
/* How the table treats a key: a struct of function pointers, which is how C describes
* behaviour a caller may choose. A store is made with one of the types below, and every
* hash and comparison inside it goes through that type. Redis calls the same thing a
* dictType. */
typedef struct {
const char *name;
uint64_t (*hash)(const char *key, size_t klen, uint64_t seed);
bool (*eq)(const char *a, size_t alen, const char *b, size_t blen);
bool seeded; /* kv_store_new draws a random seed, so keys cannot be chosen to collide */
} kv_store_type;
extern const kv_store_type kv_fnv_type; /* FNV-1a, the same hash on every run */
extern const kv_store_type kv_seeded_type; /* FNV-1a mixed with a seed drawn at start *//* FNV-1a: a small, fast hash of bytes. The seed is mixed into the starting value, so a
* seeded store hashes the same key differently on every run. */
static uint64_t kv_fnv1a(const char *s, size_t n, uint64_t seed) {
uint64_t h = 14695981039346656037ULL ^ seed;
for (size_t i = 0; i < n; i++) {
h ^= (unsigned char)s[i];
h *= 1099511628211ULL;
}
return h;
}
const kv_store_type kv_fnv_type = {"fnv", kv_fnv1a, kv_bytes_eq, false};
const kv_store_type kv_seeded_type = {"seeded", kv_fnv1a, kv_bytes_eq, true};
/* Eight random bytes from the operating system, or the clock when that fails. */
static uint64_t kv_random_seed(void) {
uint64_t seed = 0;
FILE *f = fopen("/dev/urandom", "rb");
if (f != NULL) {
size_t got = fread(&seed, sizeof seed, 1, f);
fclose(f);
if (got == 1) {
return seed;
}
}
return (uint64_t)(uintptr_t)&seed * 2654435761ULL;
}The two types here differ in one field. The seeded one mixes eight random bytes into
the hash when the store is made. So the chain an attacker's keys land in cannot be
predicted. That is the fix for the hash flooding of chapter 6, and the server uses it by
default. KV_HASH=fnv picks the plain one, for a run that must be repeatable.
kv_store_new is where the choice is applied:
kv_store *kv_store_new(const kv_store_type *type) {
kv_store *s = calloc(1, sizeof *s);
if (s == NULL) {
return NULL;
}
s->type = type;
s->seed = type->seeded ? kv_random_seed() : 0;
return s;
}
size_t kv_store_len(const kv_store *s) {
return s->len;
}The main file now includes four headers and nothing else of its own. Those four lines are the C form of an import list:
#include "buf.h"
#include "cmd.h"
#include "resp.h"
#include "store.h"/* KV_HASH=fnv asks for the unseeded hash, which makes a run repeatable. The default
* seeds the hash, so that nobody can choose keys that all land in one chain. */
const char *hash = getenv("KV_HASH");
const kv_store_type *type = hash && strcmp(hash, "fnv") == 0 ? &kv_fnv_type : &kv_seeded_type;
kv_store *store = kv_store_new(type);
if (store == NULL) {
perror("kv_store_new");
close(fd);
return 1;
}With six source files, the Makefile from chapter 2 is not enough. The one below finds
every .c file, compiles each into an object file under build/, and
links them. The -MMD flag asks the compiler to write a small file beside each
object that lists the headers it included. make reads those files, so a change
to store.h rebuilds exactly the files that include it:
CC ?= cc
CFLAGS ?= -std=c17 -Wall -Wextra -Werror -O2 -g
SANITIZE ?=
# every .c under src/ becomes a .o under build/, and kv is linked from all of them
SRC := $(wildcard src/*.c)
OBJ := $(SRC:src/%.c=build/%.o)
kv: $(OBJ)
$(CC) $(CFLAGS) $(SANITIZE) -o $@ $(OBJ)
# -MMD writes a .d file beside each .o that lists the headers it included, so that a
# change to a header rebuilds exactly the files that use it
build/%.o: src/%.c | build
$(CC) $(CFLAGS) $(SANITIZE) -MMD -c -o $@ $<
build:
mkdir -p build
-include $(OBJ:.o=.d)
clean:
rm -rf kv kv.dSYM build
.PHONY: cleanInclude a header twice and the compiler reports a redefinition.
main.c includes buf.h directly and again through
cmd.h. Without the guard the second copy defines kv_buf a second
time, and the error names the header, not the file that included it twice. Every header
gets a guard on the day it is created. Many compilers accept #pragma once as a
shorter form, and it is not in the standard.
A .c file includes its own header before any other, as buf.c
does. Suppose the header forgot an include it needs, such as stddef.h for
size_t. The mistake then shows in the file that owns the header. It does not hide
in another file that included things in a luckier order.
Dependencies in C: the package manager that is not there#
C has no package manager, and that sentence has to be said carefully, because it has
several. The operating system has one. A C program that needs a library usually says
apt install libssl-dev or brew install openssl. Then it asks a tool
called pkg-config where the headers went. The second way is to copy the source
into your own tree, which is called vendoring. Redis does this. Its deps/ folder
holds its own copies of the libraries it needs, and they build with the rest. The third way is a package manager that is not part of the language, such as vcpkg.
Microsoft describes vcpkg as "a free and open-source C/C++ package manager maintained by
Microsoft and the C++ community". Conan is another. It works with CMake, Make and MSBuild.
Both are used. There is no default.
Each way has a cost. A system package is whatever version the operating system has. A vendored copy must be updated yourself. A third-party manager is one more thing to install before the build works. The server kv has no dependency. That is not unusual for a C program, and it is one reason C programs so often write things themselves.
Go: a folder is a package, and a capital letter is an export#
A Go package is a folder. Every file in the folder begins with the same
package line and sees everything in the other files. Another package sees only
the names that begin with a capital letter. The spec puts it in one sentence. An identifier is exported if "the first character of the identifier's name is a Unicode uppercase letter".
There is no keyword. The protocol code moves to internal/resp, and
readCommand becomes ReadCommand:
// ReadCommand reads one request: an array of bulk strings, or one inline line of words.
// It returns io.EOF when the client closed the connection between commands.
func ReadCommand(r *bufio.Reader) ([][]byte, error) {
line, err := readLine(r)
if err != nil {
return nil, fmt.Errorf("read command: %w", err)
}
if len(line) == 0 || line[0] != '*' {
return bytes.Fields(line), nil
}
n, err := strconv.Atoi(string(line[1:]))
if err != nil || n < 0 || n > maxArgs {
return nil, protoErr('*')
}
args := make([][]byte, 0, n)
for range n {
line, err := readLine(r)
if err != nil {
return nil, fmt.Errorf("read argument: %w", unexpected(err))
}
if len(line) == 0 || line[0] != '$' {
return nil, protoErr(first(line))
}
size, err := strconv.Atoi(string(line[1:]))
if err != nil || size < 0 || size > maxBulk {
return nil, protoErr('$')
}
buf := make([]byte, size+2)
if _, err := io.ReadFull(r, buf); err != nil {
return nil, fmt.Errorf("read argument: %w", unexpected(err))
}
if buf[size] != '\r' || buf[size+1] != '\n' {
return nil, protoErr(buf[size])
}
args = append(args, buf[:size])
}
return args, nil
}The folder is named internal on purpose. The Go 1.4 release notes state the rule. A package with internal in its path
"can be imported only by code in the directory tree rooted at" the parent of that folder. So kv/internal/store can
be used by kv/cmd/kv and by nobody outside the module. The rule is in the
go tool, not in the language, and it is how a Go program says "this is not an
API". The import path begins with the module path from go.mod:
// Execute runs one command against the store and writes its reply. The two imports
// above are this module's own packages: kv is the module path from go.mod.
func Execute(args [][]byte, s Storer, w *bufio.Writer) error {The store exports a type and its methods, and keeps the map itself private, so that no other package can reach into it:
// Store keeps every key and its value in a map. Exported, so that package cmd can use
// it; its field is not, so that only this package can touch the map.
type Store struct {
m map[string][]byte
}
func New() *Store {
return &Store{m: make(map[string][]byte)}
}
func (s *Store) Get(key []byte) ([]byte, bool) {
v, ok := s.m[string(key)]
return v, ok
}
func (s *Store) Set(key, val []byte) {
s.m[string(key)] = val
}
func (s *Store) Del(key []byte) bool {
_, ok := s.m[string(key)]
delete(s.m, string(key))
return ok
}
func (s *Store) Len() int {
return len(s.m)
}
// Keys returns every key that matches the pattern, in no particular order.
func (s *Store) Keys(pattern []byte) [][]byte {
var out [][]byte
for k := range s.m {
if glob(pattern, []byte(k)) {
out = append(out, []byte(k))
}
}
return out
}An interface is a set of methods#
The commands do not need the map. They need six methods, and an
interface is a type that lists methods and nothing else. Any type that
has those methods satisfies the interface, with no declaration that says so. The
interface lives in the cmd package, next to the code that calls it. Go puts
an interface on the side that uses it. The package that implements it never hears of it:
// A Storer is whatever the commands need from a store: these six methods, and nothing
// about how they are done. Any type with these methods is a Storer, without saying so.
// The interface lives here, with the code that uses it, which is where Go puts one.
type Storer interface {
Get(key []byte) ([]byte, bool)
Set(key, val []byte)
Del(key []byte) bool
Len() int
Keys(pattern []byte) [][]byte
Incr(key []byte) (int64, error)
}Two types satisfy it. *store.Store is the map. *store.ByteTable
is the table from chapter 6, wrapped so that its methods have the right shapes. Nothing
in the file names cmd.Storer, and it could not, because store
is imported by cmd and a Go import cannot go both ways:
// ByteTable is the hand-written table from chapter 6, used as a store. It has the same
// methods as Store, so it satisfies the cmd.Storer interface without naming it. Nothing
// here says "implements": the methods are the whole of the contract.
type ByteTable struct {
t *Table[string, []byte]
}
func NewByteTable() *ByteTable {
return &ByteTable{t: NewTable[string, []byte](func(s string) uint64 { return fnv1a([]byte(s)) })}
}
func (b *ByteTable) Get(key []byte) ([]byte, bool) { return b.t.Get(string(key)) }
func (b *ByteTable) Set(key, val []byte) { b.t.Set(string(key), val) }
func (b *ByteTable) Del(key []byte) bool { return b.t.Del(string(key)) }
func (b *ByteTable) Len() int { return b.t.Len() }
func (b *ByteTable) Keys(pattern []byte) [][]byte {
var out [][]byte
for _, k := range b.t.Keys() {
if glob(pattern, []byte(k)) {
out = append(out, []byte(k))
}
}
return out
}
func (b *ByteTable) Incr(key []byte) (int64, error) {
var n int64
if cur, ok := b.Get(key); ok {
var err error
n, err = strconv.ParseInt(string(cur), 10, 64)
if err != nil {
return 0, ErrNotInteger
}
}
if n == math.MaxInt64 {
return 0, ErrOverflow
}
n++
b.Set(key, []byte(strconv.FormatInt(n, 10)))
return n, nil
}The main function chooses. A variable of the interface type holds one or the other, and every call through it goes to whichever is inside:
// Either store satisfies cmd.Storer. The variable's type is the interface, and the
// value inside it is whichever the environment asked for.
var s cmd.Storer = store.New()
if os.Getenv("KV_STORE") == "table" {
s = store.NewByteTable()
}This is why io.Copy in chapter 3 could copy a socket, a file or a buffer
with one function. It is why error in chapter 4 was a type with one method.
The small interfaces of the standard library are the ones most code accepts, such as
io.Reader, io.Writer and error. A type that has a
Read method joins them without asking.
Embedding, and the nil that is not nil#
A struct may hold a field with no name, only a type, which is called
embedding. The methods of the embedded type are promoted to the outer
struct, so a wrapper can replace one method and inherit the rest. The test below wraps any
Storer and counts the writes:
// counting wraps any Storer and counts the writes. The embedded field lends the wrapped
// store's methods to counting, so only Set is written here: the rest are promoted.
type counting struct {
Storer
sets int
}
func (c *counting) Set(key, val []byte) {
c.sets++
c.Storer.Set(key, val)
}
func TestEmbeddedStorerPromotesMethods(t *testing.T) {
c := &counting{Storer: store.New()}
var buf bytes.Buffer
w := bufio.NewWriter(&buf)
for _, line := range [][]string{{"SET", "a", "1"}, {"SET", "b", "2"}, {"GET", "a"}} {
var args [][]byte
for _, s := range line {
args = append(args, []byte(s))
}
if err := Execute(args, c, w); err != nil {
t.Fatal(err)
}
}
w.Flush()
if c.sets != 2 || c.Len() != 2 || !bytes.Contains(buf.Bytes(), []byte("$1\r\n1\r\n")) {
t.Fatalf("sets %d, len %d, out %q", c.sets, c.Len(), buf.String())
}
}An interface value is two words, a type and a pointer. That shape has a trap, and the second test walks into it on purpose:
// A nil pointer inside an interface is not a nil interface. The interface holds a type
// and a pointer; the type is set, so the interface is not nil, and a method call reaches
// the pointer receiver with nil. This is the trap every Go programmer meets once.
func TestNilPointerInInterfaceIsNotNil(t *testing.T) {
var p *store.Store // nil
var s Storer = p // a Storer holding (*store.Store, nil)
if s == nil {
t.Fatal("an interface holding a typed nil compares unequal to nil")
}
var none Storer // no type, no value
if none != nil {
t.Fatal("an interface with no type is nil")
}
}A function that returns an interface, and returns a nil pointer of a concrete
type, returns something that is not nil. The caller's if err != nil
is true, and the method call inside it runs on a nil receiver. The rule is to return the
literal nil, never a nil pointer, from a function whose result type is an
interface. The test above is the whole explanation, and most Go programmers learn it from
a crash.
A function that takes an argument asks for the smallest interface that has what it
needs. That is why io.Copy takes a Reader and not a file. A
function that returns a value returns the concrete type, so the caller has every method.
And an interface is defined when the second implementation exists, not before. One
implementation is a struct.
The function glob is now in its own file in the store package.
Write TestGlob in glob_test.go. Make a table of cases with a
pattern, a key and the expected answer, and a loop that calls t.Errorf for each
mistake.
Include a case where * must match an empty string, and one where
? must not. Run it with go test ./..., which tests every package
in the module.
func TestGlob(t *testing.T) {
cases := []struct {
pat, key string
want bool
}{
// your cases here
}
for _, c := range cases {
// your check here
}
}Stuck?Show the full listing
func TestGlob(t *testing.T) {
cases := []struct {
pat, key string
want bool
}{
{"*", "", true},
{"*", "anything", true},
{"user:*", "user:1", true},
{"user:*", "order:1", false},
{"us?r:1", "user:1", true},
{"us?r:1", "usr:1", false},
{"*:2", "user:2", true},
{"a*b*c", "aXXbYYc", true},
{"a*b*c", "aXXbYY", false},
}
for _, c := range cases {
if got := glob([]byte(c.pat), []byte(c.key)); got != c.want {
t.Errorf("glob(%q, %q) = %v, want %v", c.pat, c.key, got, c.want)
}
}
}$ go build -o kv ./cmd/kv
$ go test ./...
? kv/cmd/kv [no test files]
? kv/internal/cmd [no test files]
? kv/internal/resp [no test files]
ok kv/internal/store 0.020skv still has no dependency, and Go makes that a normal state. The standard library has the
network, the buffered reader, the map and the test runner, and most Go programs of this
size use nothing else. When a dependency is needed, go get adds a line to
go.mod and a hash to go.sum. The hash is checked on every later download, against the file and against a public log
that Google runs. So a changed upload cannot replace what you tested. There is no central registry. A module path is a URL,
and the go tool fetches through a proxy that caches every version it has seen.
Delete the last use of a package and the build fails. The error is
"strings" imported and not used. Go does this on purpose, so that a dependency
graph never holds a package that nothing needs, and the build stays fast. Editors remove the
line for you, and new Go programmers curse it for a week.
Rust: a module is a file you must announce#
kv already had modules in Rust. Chapter 4 wrote mod resp; and
mod cmd;, and each line told the compiler to read a file of that name. The new step is to put the modules in a library. Then a test or a benchmark can use the
parser and the store without starting a server. A crate with both src/lib.rs and
src/main.rs builds a library and a binary of the same name:
pub mod cmd;
pub mod resp;
pub mod store;
pub mod table;The pub on each mod line makes the module visible outside the
library, and the pub on each function inside makes the function visible. Without
either, the item is private to the file. The binary uses the library through its name:
use kv::store::{MapStore, Store};
use kv::table::Table;
use kv::{cmd, resp};Create src/lib.rs with four public mod lines. Remove the mod lines from main.rs. Replace them with
use lines that name the library, such as kv::store::Store. Build, and run
cargo clippy. It will tell you that Store has a public len and no
is_empty. That is a convention it enforces for public types, and it never
mentioned it while the type was private. Add a one-line is_empty that returns self.map.is_empty().
// src/lib.rs
// your four mod lines here
// src/main.rs
// your use lines hereStuck?Show the full listing
//! kv as a library: the protocol, the store and the commands. The binary in main.rs is a
//! thin wrapper, and a test or a benchmark can use the crate without starting a server.
pub mod cmd;
pub mod resp;
pub mod store;
pub mod table;use kv::store::{MapStore, Store};
use kv::table::Table;
use kv::{cmd, resp};A trait is a promise of methods#
The Rust store is now a trait, a set of methods a type promises, and
two types keep the promise. Unlike a Go interface, the promise is written down. A type
says impl Store for MapStore and lists the methods, and the compiler checks
that none is missing and none has the wrong shape:
/// What the commands need from a store. A trait is a set of methods a type promises, and
/// any type that writes an `impl Store for` block with these methods is a store. The
/// server is written against the trait, so the type behind it can change.
pub trait Store {
/// A borrowed view of the value, which lives as long as the borrow of the store.
fn get(&self, key: &[u8]) -> Option<&[u8]>;
fn set(&mut self, key: Vec<u8>, val: Vec<u8>);
fn del(&mut self, key: &[u8]) -> bool;
fn len(&self) -> usize;
/// Every key that matches the pattern, in no particular order.
fn keys(&self, pattern: &[u8]) -> Vec<&[u8]>;
fn is_empty(&self) -> bool {
self.len() == 0
}
/// Add one to the integer stored under key, which counts as 0 when absent. A default
/// method: written once here, in terms of `get` and `set`, and shared by every
/// implementation unless one replaces it.
fn incr(&mut self, key: &[u8]) -> Result<i64, IncrError> {
let n = match self.get(key) {
None => 0,
Some(bytes) => parse_int(bytes).ok_or(IncrError::NotInteger)?,
};
let n = n.checked_add(1).ok_or(IncrError::Overflow)?;
self.set(key.to_vec(), n.to_string().into_bytes());
Ok(n)
}
}Two of the methods have bodies in the trait. is_empty and
incr are default methods, written once in terms of the
others and shared by every implementation unless one replaces them. The map store
implements the five that remain:
/// The store the server uses: the standard HashMap. The hand-written table in table.rs
/// implements the same trait, and `KV_STORE=table` runs the server on it.
#[derive(Default)]
pub struct MapStore {
map: HashMap<Vec<u8>, Vec<u8>>,
}
impl MapStore {
pub fn new() -> Self {
Self::default()
}
}
impl Store for MapStore {
fn get(&self, key: &[u8]) -> Option<&[u8]> {
self.map.get(key).map(Vec::as_slice)
}
fn set(&mut self, key: Vec<u8>, val: Vec<u8>) {
self.map.insert(key, val);
}
fn del(&mut self, key: &[u8]) -> bool {
self.map.remove(key).is_some()
}
fn len(&self) -> usize {
self.map.len()
}
fn keys(&self, pattern: &[u8]) -> Vec<&[u8]> {
self.map
.keys()
.map(Vec::as_slice)
.filter(|k| glob(pattern, k))
.collect()
}
}The table from chapter 6 implements the same five. The block lives in
table.rs, next to the type. It could as well live next to the trait, because
Rust allows an impl wherever the crate owns the trait or the type. That is
the orphan rule. A crate may implement its own trait for any type, or
any trait for its own type. It may never implement a foreign trait for a foreign type, so
two crates cannot disagree about what Display means for Vec:
/// The hand-written table as a store: the same trait as MapStore, so the server can run
/// on either. The methods are those of the trait; the body of each is one line.
impl crate::store::Store for Table<Vec<u8>, Vec<u8>> {
fn get(&self, key: &[u8]) -> Option<&[u8]> {
Table::get(self, &key.to_vec()).map(Vec::as_slice)
}
fn set(&mut self, key: Vec<u8>, val: Vec<u8>) {
Table::set(self, key, val);
}
fn del(&mut self, key: &[u8]) -> bool {
Table::del(self, &key.to_vec())
}
fn len(&self) -> usize {
Table::len(self)
}
fn keys(&self, pattern: &[u8]) -> Vec<&[u8]> {
self.buckets
.iter()
.flatten()
.map(|e| e.key.as_slice())
.filter(|k| crate::store::glob(pattern, k))
.collect()
}
}Static and dynamic dispatch#
There are two ways to call a method of a trait, and kv uses both. The main function
builds a Box<dyn Store>, a box holding whichever type the environment
asked for. dyn Store is a trait object. A pointer to one
carries a second pointer to a table of the type's methods. Every call looks the method
up in that table at run time:
// Either type implements Store. A Box<dyn Store> holds whichever one the environment
// asked for, and every call through it is decided at run time.
let mut store: Box<dyn Store> = match std::env::var("KV_STORE").as_deref() {
Ok("table") => Box::new(Table::default()),
_ => Box::new(MapStore::new()),
};The test in store.rs takes the other road. fn
behaves_like_a_store<S: Store> is generic with a bound, and the compiler
writes one copy of it per type it is called with. Each copy calls the methods directly,
which is static dispatch, the same mechanism as the generic table of
chapter 6. impl Store in an argument position is the short spelling of the
same bound:
#[cfg(test)]
mod tests {
use super::{MapStore, Store};
use crate::table::Table;
/// One test for every store. `S: Store` is a bound, and the compiler writes a copy of
/// this function for each type it is called with: static dispatch, no trait object.
fn behaves_like_a_store<S: Store>(mut store: S) {
assert!(store.is_empty());
store.set(b"a".to_vec(), b"1".to_vec());
assert_eq!(store.get(b"a"), Some(&b"1"[..]));
assert_eq!(store.incr(b"a"), Ok(2));
assert_eq!(store.keys(b"*").len(), 1);
assert!(store.del(b"a"));
assert!(store.get(b"a").is_none());
}
#[test]
fn the_map_store_is_a_store() {
behaves_like_a_store(MapStore::new());
}
#[test]
fn the_table_is_a_store() {
behaves_like_a_store(Table::<Vec<u8>, Vec<u8>>::default());
}
}The handlers of chapter 5 took &mut dyn Write for the reason that
appears again here. A table of function pointers holds one function, so the function
must work for any writer at run time. The helpers in resp take
impl Write + ?Sized, which is the generic form. The ?Sized is a
note that the type may be a trait object, whose size the compiler does not know. One
rule limits which traits can be objects. A trait with a generic method cannot be a
dyn. The same holds for a method that returns Self. The method
table could not hold either. Store has no such method, so it
can.
One more word of visibility appeared in this chapter. parse_int is
pub(crate), which every module in the crate may use and no user of the
library can see. Between private and pub, it is the one that most helpers
want.
A dependency in Rust is a line in Cargo.toml, which cargo add
writes for you, and a crate on crates.io, which is the one registry. The Cargo book draws the
line between the two files in one sentence each. "Cargo.toml is about describing
your dependencies in a broad sense, and is written by you." "Cargo.lock contains exact information about your dependencies. It is
maintained by Cargo and should not be manually edited." The first dependency of kv arrives
in chapter 9. That is when the lock file appears. Two more Cargo words will come up
before then. A feature is a named switch in Cargo.toml that
turns part of a crate on. The code behind it is marked with a cfg attribute
that names the feature. tokio in chapter 9 is built from features. Few programs need all
of it. A workspace is a folder of crates that share one
Cargo.lock and one target directory, for a project with a
library and several binaries. Go has the same two ideas under other names. A build tag
is a feature, and a go.work file is a workspace. Chapter 14 uses a workspace
when a module must build with a local copy of another.
Add src/glob.rs and nothing happens. The compiler reads only
the files that a mod line names, starting from lib.rs or
main.rs. A forgotten mod line produces an error about an unresolved
name somewhere else, and the file that holds the name is never mentioned. Go reads every file
in the folder. C reads only what the Makefile lists. Rust reads what the tree of
mod declarations reaches.
Three answers to "where does code come from"#
Set the three languages side by side and the pattern of the book repeats. C has a convention where the other two have a tool, and the convention has several competing forms. Go has one tool with no registry, which fits a company that hosts its own code. Rust has one tool with one registry, which fits a community that shares small crates. The number of dependencies a program of this size has in practice follows from that. A C program has few, because each one is work. A Go program has few, because the standard library is wide. A Rust program often has dozens, because adding one is a single command, and that is the weakness of the strongest answer. Chapter 14 shows what every dependency costs at the border, and chapter 15 what a C one costs.
One style, enforced by a tool#
All three languages now have a formatter, and the difference is who decides the style.
clang-format takes a configuration file, and the one for kv is four settings:
# The layout every C file in kv keeps. clang-format -i src/*.c src/*.h applies it.
BasedOnStyle: LLVM
IndentWidth: 4
ColumnLimit: 100
AllowShortFunctionsOnASingleLine: InlineIt comes with clang, so it is installed on macOS with Xcode and on Linux as a package. A
project that uses it must agree on the configuration first. gofmt
has no configuration. The Go team decided the style when the language was released. The result is that every Go
file anyone has ever read looks the same. rustfmt is between clang-format and gofmt. It has a configuration file,
rustfmt.toml, and almost nobody writes one. The book runs all three formatters
in make verify. A file that is not formatted fails the build.
Beside each formatter is a linter that finds mistakes the compiler allows.
clang-tidy for C must be installed and configured. go vet comes with
Go and runs in make verify, and staticcheck adds more.
cargo clippy comes with Rust, has more than 800 checks, and is the tool that
asked for is_empty above. The verdict for this chapter is mostly about these
tools, because the day-to-day experience of a language is the experience of its tools.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| Splitting code | Weak | Strong | OK | C needs guards, prefixes and a Makefile that tracks headers, while Go needs a folder and Rust a mod line that is easy to forget. |
| Hiding a name | OK | Strong | Strong | C has static, for functions only. Go and Rust hide by default or by a letter. |
| Adding a dependency | Weak | OK | Strong | C has three competing ways and no default, while Go and Rust each have one command, and Rust also has one registry. |
| Having too many | Strong | Strong | Weak | C and Go programs stay small by cost and by a wide standard library. Rust programs grow dozens of crates. |
| Keeping one style | OK | Strong | Strong | clang-format needs a config everyone agrees on. gofmt and rustfmt need nothing. |
| Finding mistakes | Weak | OK | Strong | clang-tidy must be installed. go vet is built in. clippy is built in and has 800 checks. |
| Swapping an implementation | OK | Strong | Strong | C has a struct of function pointers that nobody checks. Go has implicit interfaces, and Rust has checked traits with two kinds of dispatch. |
| Hiding a struct | Strong | Strong | Strong | C has the opaque type, and Go and Rust have unexported fields. All three keep the map out of reach. |
- Opaque type. A C header declares a struct it does not define, so other files hold a pointer and nothing more.
- Interface. A Go type satisfies an interface by having its methods, and a nil pointer inside one is not nil.
- Trait object. A
dyn Storeis dispatched at run time through a method table, and a generic bound is dispatched at compile time.
Data types#
Fill the chapter 7 server with a million keys. Then, from one terminal, ask for all of them, and fifty milliseconds later, from another, ask for one:
$ redis-cli -p 6380 --pipe < million-sets.txt
errors: 0, replies: 1000000
$ time redis-cli -p 6380 KEYS '*' > /dev/null # terminal one
real 0m0.75s
$ time redis-cli -p 6380 GET user:1 # terminal two, 50 ms later
"v"
real 0m0.68sA GET takes no measurable time on its own. This one took most of a
second, because it waited for another client's KEYS to finish walking a
million entries. One question, and everyone waits. Redis answers it with a command that
walks the table a page at a time, and with values that are more than strings.
A key can hold more than a string. This chapter gives kv lists, hashes and sets, and a
TYPE command that says which. It adds SCAN, which walks the keys a
page at a time. The chapter is about sum types, which each language builds in its own way,
and about iteration.
Redis is not a map from strings to strings. Its documentation calls it a data structure
server. The value under a key may be a string, a list, a hash of fields or a set of
members. There are several more, and kv takes the first four. Each needs the same three things from
the store. A value must say which kind it is. A command must take the value apart as that
kind, and refuse any other kind with the error Redis uses, WRONGTYPE. And a
command such as HGETALL or SMEMBERS must walk whatever is
inside. The first need is a sum type, a value that is one of several
shapes. The third is iteration. The three languages answer both, and the
answers are the chapter.
| One value, four kinds | C | Go | Rust |
|---|---|---|---|
| The value | a struct with a tag and a union | an any, holding one of four concrete types | an enum with data in each variant |
| Asking which | switch (v->kind) | a type switch, switch v := x.(type) | match, which must cover every variant |
| Taking one kind | read the union member the tag names, and nothing checks | a type assertion, v.(*List), with an ok | a pattern, Value::List(l), in a match or let else |
| Adding a fifth kind | the compiler warns at each switch with no default | nothing warns; a missed case falls to default | the compiler refuses every match that forgot it |
| A list | an array of byte strings, shifted along for LPUSH | a slice of byte slices | a VecDeque, with both ends cheap |
| A hash and a set | a nested store, the set with empty values | a map, and a generic Set[T] | a HashMap and a HashSet |
| Walking a collection | a callback with a void * context | an iterator function that range calls | an Iterator, with map, filter and collect |
Salvatore Sanfilippo began Redis in 2009 while trying to improve the scalability of his startup, a real-time web log analyser. He had met "significant problems in scaling some types of workloads using traditional database systems". A log analyser keeps lists of recent events and sets of visitors. A list in a string store is a string the client must rewrite on every push. A list in the server is one command. The documentation calls Redis "a data structure server", and the name Remote Dictionary Server undersells it. The value is where the work is.
C: a tag and a union#
A C value that may be one of four things is a struct of two parts. The
first is a tag, an enum that says which shape is live. The
second is a union, a type whose members share the same bytes. So a
value holds one string, or one list, or one table, and never more than one at a time.
The pair is called a tagged union, and C checks nothing about it. Read the list member of
a value whose tag says string and you read the string's pointer as an array length:
/* Which of the four shapes a value has. The number is the tag of the union below. */
typedef enum { KV_STRING, KV_LIST, KV_HASH, KV_SET } kv_kind;
/* A tagged union: the tag says which member of the union is live, and reading another
* member is a bug the compiler will not catch. A hash and a set are each a nested store,
* whose entries hold strings; a set keeps its values empty and uses only the keys. */
typedef struct {
kv_kind kind;
union {
kv_bytes str;
kv_list list;
kv_store *table; /* owned, for KV_HASH and KV_SET */
} u;
} kv_value;
const char *kv_kind_name(kv_kind kind); /* "string", "list", "hash" or "set" */
void kv_value_free(kv_value *v);A hash and a set are each a nested store. The store of chapter 7 is opaque, holds
values under byte keys, and can be made with the same type as its parent. So a hash is a
store whose values are strings. A set is a store whose values are empty. Nothing new
was written for either. The list is new. It is an array of byte strings, and
LPUSH shifts the whole array along by one, which costs the length of the
list. Redis uses a linked structure there. kv keeps the array, because the chapter is
about the union, and says what it costs:
/* A list: an array of owned byte strings. RPUSH appends; LPUSH shifts the array along by
* one, which costs the length of the list. Redis keeps a linked structure instead, and
* pays for that in other ways. */
typedef struct {
kv_bytes *items; /* owned array of owned strings */
size_t len;
size_t cap;
} kv_list;Freeing a value is a switch on the tag with a case per kind and no
default. The compiler reports a kind this switch forgot. That is the only
check C gives a tagged union, and the reason the switch is written that way:
const char *kv_kind_name(kv_kind kind) {
switch (kind) {
case KV_STRING:
return "string";
case KV_LIST:
return "list";
case KV_HASH:
return "hash";
case KV_SET:
return "set";
}
return "none";
}
bool kv_bytes_set(kv_bytes *b, const char *src, size_t n) {
char *copy = kv_bytes_dup(src, n);
if (copy == NULL) {
return false;
}
free(b->p);
b->p = copy;
b->len = n;
return true;
}
static void kv_list_free(kv_list *l) {
for (size_t i = 0; i < l->len; i++) {
free(l->items[i].p);
}
free(l->items);
}
bool kv_list_push(kv_list *l, const char *src, size_t n, bool front) {
char *copy = kv_bytes_dup(src, n);
if (copy == NULL) {
return false;
}
if (l->len == l->cap) {
size_t cap = l->cap ? l->cap * 2 : 4;
kv_bytes *items = realloc(l->items, cap * sizeof *items);
if (items == NULL) {
free(copy);
return false;
}
l->items = items;
l->cap = cap;
}
if (front) {
memmove(l->items + 1, l->items, l->len * sizeof *l->items);
l->items[0] = (kv_bytes){copy, n};
} else {
l->items[l->len] = (kv_bytes){copy, n};
}
l->len++;
return true;
}
/* Free whatever the tag says is live. The switch has no default, so a fifth kind would
* be reported by the compiler here, with -Wall, until it is handled. */
void kv_value_free(kv_value *v) {
switch (v->kind) {
case KV_STRING:
free(v->u.str.p);
break;
case KV_LIST:
kv_list_free(&v->u.list);
break;
case KV_HASH:
case KV_SET:
kv_store_free(v->u.table);
break;
}
*v = (kv_value){KV_STRING, {{NULL, 0}}};
}Every command that takes a value apart starts in one of two helpers. The first returns
the string of an entry, or writes the WRONGTYPE error and returns
NULL. The second returns an entry as a given kind, making an empty one when
the key is new. Each is the tag check written once:
#define KV_WRONGTYPE "WRONGTYPE Operation against a key holding the wrong kind of value"
/* The string of an entry, or NULL after writing the WRONGTYPE error. Every string
* command starts here, so the check against the tag is in one place. */
static kv_bytes *kv_string_of(kv_entry *e, kv_buf *out) {
if (e->val.kind != KV_STRING) {
kv_reply_error(out, KV_WRONGTYPE);
return NULL;
}
return &e->val.u.str;
}
/* The entry for key as the given kind, made empty when the key is new, or NULL after
* writing the WRONGTYPE error when it holds another kind. */
static kv_entry *kv_entry_as(kv_store *s, const kv_cmd *c, kv_kind kind, kv_buf *out, bool *oom) {
bool created;
*oom = false;
kv_entry *e = kv_store_insert(s, c->argv[1], c->arglen[1], &created);
if (e == NULL) {
*oom = true;
return NULL;
}
if (created) {
e->val.kind = kind;
if (kind == KV_HASH || kind == KV_SET) {
e->val.u.table = kv_store_new(kv_store_type_of(s));
if (e->val.u.table == NULL) {
kv_store_del(s, c->argv[1], c->arglen[1]);
*oom = true;
return NULL;
}
}
} else if (e->val.kind != kind) {
kv_reply_error(out, KV_WRONGTYPE);
return NULL;
}
return e;
}The string commands of chapter 5 change by one line each, the call to
kv_string_of. GETRANGE and LRANGE share a clamp,
because a list is sliced by position exactly as a string is sliced by byte:
/* LPUSH and RPUSH: the same function, told which end by the table. */
static bool kv_push(const kv_cmd *c, kv_store *s, kv_buf *out, bool front) {
bool oom;
kv_entry *e = kv_entry_as(s, c, KV_LIST, out, &oom);
if (e == NULL) {
return !oom;
}
for (size_t i = 2; i < c->argc; i++) {
if (!kv_list_push(&e->val.u.list, c->argv[i], c->arglen[i], front)) {
return false;
}
}
return kv_reply_int(out, (long long)e->val.u.list.len);
}
static bool kv_cmd_lpush(const kv_cmd *c, kv_store *s, kv_buf *out) {
return kv_push(c, s, out, true);
}
static bool kv_cmd_rpush(const kv_cmd *c, kv_store *s, kv_buf *out) {
return kv_push(c, s, out, false);
}
/* The list of an entry, or NULL after the WRONGTYPE error. */
static kv_list *kv_list_of(kv_entry *e, kv_buf *out) {
if (e->val.kind != KV_LIST) {
kv_reply_error(out, KV_WRONGTYPE);
return NULL;
}
return &e->val.u.list;
}
static bool kv_cmd_llen(const kv_cmd *c, kv_store *s, kv_buf *out) {
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
if (e == NULL) {
return kv_reply_int(out, 0);
}
kv_list *l = kv_list_of(e, out);
return l == NULL || kv_reply_int(out, (long long)l->len);
}
static bool kv_cmd_lrange(const kv_cmd *c, kv_store *s, kv_buf *out) {
long long start, end;
if (!kv_parse_int(c->argv[2], c->arglen[2], &start) ||
!kv_parse_int(c->argv[3], c->arglen[3], &end)) {
return kv_reply_error(out, "ERR value is not an integer or out of range");
}
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
if (e == NULL) {
return kv_reply_array(out, 0);
}
kv_list *l = kv_list_of(e, out);
if (l == NULL) {
return true;
}
size_t from, to;
if (!kv_clamp(start, end, (long long)l->len, &from, &to)) {
return kv_reply_array(out, 0);
}
if (!kv_reply_array(out, to - from + 1)) {
return false;
}
for (size_t i = from; i <= to; i++) {
if (!kv_reply_bulk(out, l->items[i].p, l->items[i].len)) {
return false;
}
}
return true;
}Write kv_cmd_sadd. Get the entry as KV_SET with
kv_entry_as, which writes the error for a key of another kind. Then insert
each member into the nested store with kv_store_insert, and count the ones
that were created. Reply with the count. Return false only when memory runs
out, which is what oom reports.
static bool kv_cmd_sadd(const kv_cmd *c, kv_store *s, kv_buf *out) {
bool oom;
kv_entry *e = kv_entry_as(s, c, KV_SET, out, &oom);
if (e == NULL) {
return !oom;
}
/* your code here */
}Stuck?Show the full listing
static bool kv_cmd_sadd(const kv_cmd *c, kv_store *s, kv_buf *out) {
bool oom;
kv_entry *e = kv_entry_as(s, c, KV_SET, out, &oom);
if (e == NULL) {
return !oom;
}
long long added = 0;
for (size_t i = 2; i < c->argc; i++) {
bool created;
if (kv_store_insert(e->val.u.table, c->argv[i], c->arglen[i], &created) == NULL) {
return false;
}
added += created;
}
return kv_reply_int(out, added);
}
static bool kv_cmd_sismember(const kv_cmd *c, kv_store *s, kv_buf *out) {
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
if (e == NULL) {
return kv_reply_int(out, 0);
}
kv_store *set = kv_table_of(e, KV_SET, out);
return set == NULL || kv_reply_int(out, kv_store_find(set, c->argv[2], c->arglen[2]) != NULL);
}
static bool kv_cmd_smembers(const kv_cmd *c, kv_store *s, kv_buf *out) {
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
if (e == NULL) {
return kv_reply_array(out, 0);
}
kv_store *set = kv_table_of(e, KV_SET, out);
if (set == NULL) {
return true;
}
if (!kv_reply_array(out, kv_store_len(set))) {
return false;
}
kv_members_ctx m = {out, false, true};
kv_store_each(set, kv_members_visit, &m);
return m.ok;
}HGETALL and SMEMBERS walk the nested store with the callback
of chapter 7. One callback serves both, told by its context whether to write the values
as well as the keys:
/* HSET key field value [field value ...]: the fields go into the nested store. */
static bool kv_cmd_hset(const kv_cmd *c, kv_store *s, kv_buf *out) {
if (c->argc % 2 != 0) {
return kv_reply_arity(out, "hset");
}
bool oom;
kv_entry *e = kv_entry_as(s, c, KV_HASH, out, &oom);
if (e == NULL) {
return !oom;
}
long long added = 0;
for (size_t i = 2; i + 1 < c->argc; i += 2) {
added += kv_store_find(e->val.u.table, c->argv[i], c->arglen[i]) == NULL;
if (!kv_store_set(e->val.u.table, c->argv[i], c->arglen[i], c->argv[i + 1], c->arglen[i + 1])) {
return false;
}
}
return kv_reply_int(out, added);
}
/* The nested table of an entry, or NULL after the WRONGTYPE error. */
static kv_store *kv_table_of(kv_entry *e, kv_kind kind, kv_buf *out) {
if (e->val.kind != kind) {
kv_reply_error(out, KV_WRONGTYPE);
return NULL;
}
return e->val.u.table;
}
static bool kv_cmd_hget(const kv_cmd *c, kv_store *s, kv_buf *out) {
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
if (e == NULL) {
return kv_reply_nil(out);
}
kv_store *h = kv_table_of(e, KV_HASH, out);
if (h == NULL) {
return true;
}
kv_entry *f = kv_store_find(h, c->argv[2], c->arglen[2]);
return f ? kv_reply_bulk(out, f->val.u.str.p, f->val.u.str.len) : kv_reply_nil(out);
}
/* The context of the HGETALL and SMEMBERS walks: the reply so far, and whether a
* reply failed to append. */
typedef struct {
kv_buf *out;
bool with_values;
bool ok;
} kv_members_ctx;
static bool kv_members_visit(kv_entry *e, void *ctx) {
kv_members_ctx *m = ctx;
m->ok = kv_reply_bulk(m->out, e->key, e->klen) &&
(!m->with_values || kv_reply_bulk(m->out, e->val.u.str.p, e->val.u.str.len));
return m->ok;
}
static bool kv_cmd_hgetall(const kv_cmd *c, kv_store *s, kv_buf *out) {
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
if (e == NULL) {
return kv_reply_array(out, 0);
}
kv_store *h = kv_table_of(e, KV_HASH, out);
if (h == NULL) {
return true;
}
if (!kv_reply_array(out, kv_store_len(h) * 2)) {
return false;
}
kv_members_ctx m = {out, true, true};
kv_store_each(h, kv_members_visit, &m);
return m.ok;
}SCAN costs C the least. The store's buckets are an array, and a
cursor is an index into it. The walk starts at the cursor's bucket and visits whole
buckets until it has seen enough entries. Then it returns the next bucket as the new
cursor, or 0 at the end. A client that calls SCAN with each returned cursor sees every key
that was there throughout, without the server ever holding the whole list:
size_t kv_store_scan(kv_store *s, size_t cursor, size_t count,
bool (*fn)(kv_entry *e, void *ctx), void *ctx) {
size_t seen = 0;
while (cursor < s->cap) {
for (kv_entry *e = s->buckets[cursor]; e != NULL; e = e->next) {
seen++;
if (!fn(e, ctx)) {
return 0;
}
}
cursor++;
if (seen >= count) {
break;
}
}
return cursor < s->cap ? cursor : 0;
}static bool kv_cmd_type(const kv_cmd *c, kv_store *s, kv_buf *out) {
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
return kv_reply_simple(out, e ? kv_kind_name(e->val.kind) : "none");
}
/* The context of a SCAN walk: the pattern, and the matching keys collected so far. */
typedef struct {
const char *pat;
size_t plen;
kv_buf keys;
size_t n;
bool ok;
} kv_scan_ctx;
static bool kv_scan_visit(kv_entry *e, void *ctx) {
kv_scan_ctx *k = ctx;
if (k->pat != NULL && !kv_glob(k->pat, k->plen, e->key, e->klen)) {
return true;
}
if (!kv_reply_bulk(&k->keys, e->key, e->klen)) {
k->ok = false;
return false;
}
k->n++;
return true;
}
/* SCAN cursor [MATCH pattern] [COUNT n]: a page of keys and the cursor for the next page.
* The reply is an array of two things, the cursor as a string and the array of keys. */
static bool kv_cmd_scan(const kv_cmd *c, kv_store *s, kv_buf *out) {
long long cursor;
if (!kv_parse_int(c->argv[1], c->arglen[1], &cursor) || cursor < 0) {
return kv_reply_error(out, "ERR invalid cursor");
}
kv_scan_ctx k = {NULL, 0, {0}, 0, true};
long long count = 10;
for (size_t i = 2; i + 1 < c->argc; i += 2) {
if (kv_arg_is(c, i, "MATCH")) {
k.pat = c->argv[i + 1];
k.plen = c->arglen[i + 1];
} else if (kv_arg_is(c, i, "COUNT")) {
if (!kv_parse_int(c->argv[i + 1], c->arglen[i + 1], &count) || count < 1) {
return kv_reply_error(out, "ERR value is not an integer or out of range");
}
} else {
return kv_reply_error(out, "ERR syntax error");
}
}
if (c->argc % 2 != 0) {
return kv_reply_error(out, "ERR syntax error");
}
size_t next = kv_store_scan(s, (size_t)cursor, (size_t)count, kv_scan_visit, &k);
char text[32];
int len = snprintf(text, sizeof text, "%zu", next);
bool ok = k.ok && kv_reply_array(out, 2) && kv_reply_bulk(out, text, (size_t)len) &&
kv_reply_array(out, k.n) && kv_buf_append(out, k.keys.data, k.keys.len);
kv_buf_free(&k.keys);
return ok;
}Reading a union member that is not the live one is undefined behaviour in
the general case, and the compiler will not say a word. The only defence is the
habit the helpers above enforce. Check the tag, then read the member, and never pass a
union member around without the tag beside it. A switch on the tag with no
default keeps the check honest when a kind is added.
Go: any, and a switch on the type#
Go has no union. It has any, the interface with no methods, which every
value satisfies. The store holds an any under each key, and the four kinds
are four concrete types inside it. A type switch asks which. It is a
switch whose cases are types, and Kind is one. The 'a in its name says that the reference it holds borrows the store, which chapter 10 explains:
// A value is one of four things, and the store holds it as `any`: an interface with no
// methods, which any value satisfies. The four concrete types are the cases, and a type
// switch takes the value apart. ErrWrongType is what a command replies when the case is
// not the one it needs.
var ErrWrongType = errors.New("WRONGTYPE Operation against a key holding the wrong kind of value")
// Kind names the case, as TYPE reports it. The switch has no default on purpose: a
// value of a fifth type would reach "none", and the tests would say so.
func Kind(v any) string {
switch v.(type) {
case []byte:
return "string"
case *List:
return "list"
case Hash:
return "hash"
case *Set[string]:
return "set"
}
return "none"
}A type assertion asks for one type and gets a second result that says
whether the value had it. With that ok an assertion never panics. Without
it, one that fails stops the program, which is the form to avoid on data from a client.
The string commands start in stringOf, and the three collection kinds start
in as, which is generic over the kind it returns:
// stringOf is the value as bytes, or ErrWrongType. The assertion v.([]byte) asks whether
// the value inside the interface has that type; the second result says. Every string
// command starts here, so the check is in one place.
func stringOf(v any) ([]byte, error) {
b, ok := v.([]byte)
if !ok {
return nil, store.ErrWrongType
}
return b, nil
}
// as returns the value under key as T, making one with mk when the key is new. A key
// that holds another kind is ErrWrongType. T is a type parameter, so one function
// serves *store.List, store.Hash and *store.Set[string].
func as[T any](s Storer, key []byte, mk func() T) (T, error) {
if v, ok := s.Get(key); ok {
t, isT := v.(T)
if !isT {
var zero T
return zero, store.ErrWrongType
}
return t, nil
}
t := mk()
s.Set(key, t)
return t, nil
}Nothing warns when a type switch forgets a case. A fifth kind added to the store falls
to the default of every switch that did not know it, and TYPE
would answer none. That is the cost of a sum type built from an interface,
and the tests are what catch it. The list, the hash and the set are three types of their
own. The set is generic, because nothing in it depends on what the members are:
// A Hash is a map from field to value, with nothing to add: a map already does it all.
type Hash map[string][]byte
// A Set is a map with no values. It is generic, because nothing about it depends on
// what the members are, only that they can be compared.
type Set[T comparable] struct {
m map[T]struct{}
}
func NewSet[T comparable]() *Set[T] { return &Set[T]{m: make(map[T]struct{})} }
// Add reports whether the member was new.
func (s *Set[T]) Add(v T) bool {
_, had := s.m[v]
s.m[v] = struct{}{}
return !had
}
func (s *Set[T]) Has(v T) bool { _, ok := s.m[v]; return ok }
func (s *Set[T]) Len() int { return len(s.m) }
// All yields every member, in no particular order.
func (s *Set[T]) All() iter.Seq[T] {
return func(yield func(T) bool) {
for v := range s.m {
if !yield(v) {
return
}
}
}
}A function that range can call#
Since Go 1.23, range can loop over a function. The function takes the
loop body, called yield, and calls it once per item until it returns
false. The type iter.Seq[T] names that shape. The list's
Range is one, and LRANGE loops over it without ever seeing the
slice inside:
// A List is a slice of byte strings. PushBack appends; PushFront makes room at the
// start by copying the slice along, which costs the length of the list.
type List struct {
items [][]byte
}
func (l *List) PushBack(v []byte) { l.items = append(l.items, v) }
func (l *List) PushFront(v []byte) { l.items = append([][]byte{v}, l.items...) }
func (l *List) Len() int { return len(l.items) }
// Range yields the items from start to end, both inclusive, with negative offsets
// counted from the end and out-of-range offsets clamped. It is an iterator: a function
// that takes the loop body, and range calls it. The caller never sees the slice.
func (l *List) Range(start, end int64) iter.Seq[[]byte] {
return func(yield func([]byte) bool) {
n := int64(len(l.items))
if start < 0 {
start += n
}
if end < 0 {
end += n
}
start = max(start, 0)
end = min(end, n-1)
for i := start; i <= end; i++ {
if !yield(l.items[i]) {
return
}
}
}
}Write lrange. Parse the two offsets with strconv.ParseInt, and
reply with the Redis error when either fails. Get the list with listOf, and
reply with its error when the key holds another kind. Then loop over
l.Range(start, end) with range, append each item to a slice,
and write the slice with writeBulks. The count must be known before the
items, which is why the iterator is drained first.
func lrange(args [][]byte, s Storer, w *bufio.Writer) error {
// your code here
}Stuck?Show the full listing
func push(args [][]byte, s Storer, w *bufio.Writer, front bool) error {
l, err := as(s, args[1], func() *store.List { return &store.List{} })
if err != nil {
return resp.WriteError(w, err.Error())
}
for _, v := range args[2:] {
if front {
l.PushFront(v)
} else {
l.PushBack(v)
}
}
return resp.WriteInt(w, int64(l.Len()))
}
func lpush(args [][]byte, s Storer, w *bufio.Writer) error { return push(args, s, w, true) }
func rpush(args [][]byte, s Storer, w *bufio.Writer) error { return push(args, s, w, false) }
// listOf is the list under key, an empty one when the key is missing, or an error.
func listOf(s Storer, key []byte) (*store.List, error) {
v, ok := s.Get(key)
if !ok {
return &store.List{}, nil
}
l, isList := v.(*store.List)
if !isList {
return nil, store.ErrWrongType
}
return l, nil
}
func llen(args [][]byte, s Storer, w *bufio.Writer) error {
l, err := listOf(s, args[1])
if err != nil {
return resp.WriteError(w, err.Error())
}
return resp.WriteInt(w, int64(l.Len()))
}
// lrange collects the page with range over the list's iterator. The count is needed
// before the items, so the iterator is drained into a slice first.
func lrange(args [][]byte, s Storer, w *bufio.Writer) error {
start, err1 := strconv.ParseInt(string(args[2]), 10, 64)
end, err2 := strconv.ParseInt(string(args[3]), 10, 64)
if err1 != nil || err2 != nil {
return resp.WriteError(w, "ERR value is not an integer or out of range")
}
l, err := listOf(s, args[1])
if err != nil {
return resp.WriteError(w, err.Error())
}
var page [][]byte
for item := range l.Range(start, end) {
page = append(page, item)
}
return writeBulks(w, page)
}A Go map cannot be scanned from a position. Each range over a map starts
in a new random order, by design, so that no program comes to depend on the order. The
map store sorts its keys and uses the cursor as an index into the sorted list, which costs
a sort per page. The table from chapter 6 owns its buckets and scans them as the C store
does, in one walk. That difference is the reason Redis owns its table, and the one place
in the book where the hand-written table is the better store:
// Scan returns a page of keys from the cursor on, and the cursor for the next page, or
// 0 when the walk is done. A Go map cannot be walked from a position: each range starts
// in a new random order. So the keys are sorted and the cursor is an index into that
// order, which costs a sort per page. The hand-written table scans by bucket instead.
func (s *Store) Scan(cursor uint64, count int) (uint64, [][]byte) {
sorted := slices.Sorted(maps.Keys(s.m))
if cursor >= uint64(len(sorted)) {
return 0, nil
}
end := min(int(cursor)+count, len(sorted))
var page [][]byte
for _, k := range sorted[cursor:end] {
page = append(page, []byte(k))
}
if end == len(sorted) {
return 0, page
}
return uint64(end), page
}// Scan walks the buckets from cursor, collecting keys until at least count have been
// seen or the buckets end. The returned cursor is the next bucket, or 0 at the end: a
// cursor is a bucket index, which is why the walk can resume.
func (t *Table[K, V]) Scan(cursor uint64, count int) (uint64, []K) {
var page []K
for cursor < uint64(len(t.buckets)) {
for _, e := range t.buckets[cursor] {
page = append(page, e.key)
}
cursor++
if len(page) >= count {
break
}
}
if cursor >= uint64(len(t.buckets)) {
return 0, page
}
return cursor, page
}v.(*List) on a value that is not a *List stops the
program with interface conversion. On a value that came from a
client, that is a crash any client can cause. The two-result form, l, ok :=
v.(*List), never panics, and every assertion in kv has the ok. A
type switch has the same safety with no extra code, because an unmatched type takes
the default branch.
Rust: an enum with data in each variant#
A Rust enum is the tagged union with the check the C version lacks. Each
variant carries its own data, and the compiler knows which variant is live. The only way
to read the data is a pattern that names the variant. A match must cover
every variant, so a fifth kind added next year is a compile error at every
match that forgot it:
/// The four kinds of value, as one enum. Each variant carries its own data, and a match
/// must handle every variant, so a fifth kind added later is a compile error at every
/// place that forgot it. This is the sum type that C builds by hand with a tag and a
/// union, and Go builds with an interface and a type switch.
pub enum Value {
Str(Vec<u8>),
List(VecDeque<Vec<u8>>),
Hash(HashMap<Vec<u8>, Vec<u8>>),
Set(HashSet<Vec<u8>>),
}
impl Value {
/// The name TYPE reports.
pub fn kind(&self) -> &'static str {
match self {
Value::Str(_) => "string",
Value::List(_) => "list",
Value::Hash(_) => "hash",
Value::Set(_) => "set",
}
}
}
/// The error a command replies when the value is not the kind it needs.
#[derive(Debug, PartialEq)]
pub struct WrongType;
impl WrongType {
pub const MESSAGE: &'static str =
"WRONGTYPE Operation against a key holding the wrong kind of value";
}The trait gains five methods that work on a value of any kind, and the string commands
become default methods on top of them. get returns a
Result of an Option, because a string command has three
outcomes, the string, no key, and a key of another kind. incr uses
? on that result. A From turns the wrong-type error into the
IncrError it returns:
/// What the commands need from a store. Five methods work on values of any kind, and the
/// string commands are default methods written on top of them.
pub trait Store {
fn value(&self, key: &[u8]) -> Option<&Value>;
fn value_mut(&mut self, key: &[u8]) -> Option<&mut Value>;
/// The value under key, made with `make` when the key is new.
fn value_or_insert(&mut self, key: Vec<u8>, make: fn() -> Value) -> &mut Value;
/// Store a value of any kind, replacing whatever kind was there.
fn insert(&mut self, key: Vec<u8>, val: Value);
fn del(&mut self, key: &[u8]) -> bool;
fn len(&self) -> usize;
/// Every key that matches the pattern, in no particular order.
fn keys(&self, pattern: &[u8]) -> Vec<&[u8]>;
/// A page of up to `count` keys from `cursor` on, and the cursor of the next page,
/// or 0 when the walk is complete.
fn scan(&self, cursor: usize, count: usize) -> (usize, Vec<&[u8]>);
fn is_empty(&self) -> bool {
self.len() == 0
}
/// The string under key: `None` for a missing key, an error for another kind.
fn get(&self, key: &[u8]) -> Result<Option<&[u8]>, WrongType> {
match self.value(key) {
None => Ok(None),
Some(Value::Str(s)) => Ok(Some(s)),
Some(_) => Err(WrongType),
}
}
fn set(&mut self, key: Vec<u8>, val: Vec<u8>) {
self.insert(key, Value::Str(val));
}
/// Add one to the integer stored under key, which counts as 0 when absent.
fn incr(&mut self, key: &[u8]) -> Result<i64, IncrError> {
let n = match self.get(key)? {
None => 0,
Some(bytes) => parse_int(bytes).ok_or(IncrError::NotInteger)?,
};
let n = n.checked_add(1).ok_or(IncrError::Overflow)?;
self.set(key.to_vec(), n.to_string().into_bytes());
Ok(n)
}
}The commands take a value apart with let ... else, which matches one
variant and leaves the function when the value is another. The list is a
VecDeque, which pushes at either end in constant time. The arguments are
moved into it with drain, so no byte of a pushed item is copied:
/// LPUSH and RPUSH: the same function, told which end by a flag. A `VecDeque` pushes at
/// either end in constant time, which is why the list is one and not a `Vec`.
fn push(
args: &mut Command,
store: &mut dyn Store,
w: &mut dyn Write,
front: bool,
) -> io::Result<()> {
let key = mem::take(&mut args[1]);
let Value::List(list) = store.value_or_insert(key, || Value::List(VecDeque::new())) else {
return write_wrongtype(w);
};
for item in args.drain(2..) {
if front {
list.push_front(item);
} else {
list.push_back(item);
}
}
resp::write_int(w, list.len() as i64)
}
fn lpush(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
push(args, store, w, true)
}
fn rpush(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
push(args, store, w, false)
}
/// The value under key as the kind the command needs: `None` for a missing key, an
/// error for another kind. `Kind<'a, T>` is a name for the shape all three helpers return.
type Kind<'a, T> = Result<Option<&'a T>, WrongType>;
/// The list under key, an empty one when the key is missing, or the wrong kind.
fn list_of<'a>(store: &'a dyn Store, key: &[u8]) -> Kind<'a, VecDeque<Vec<u8>>> {
match store.value(key) {
None => Ok(None),
Some(Value::List(l)) => Ok(Some(l)),
Some(_) => Err(WrongType),
}
}
fn llen(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
match list_of(store, &args[1]) {
Ok(l) => resp::write_int(w, l.map_or(0, VecDeque::len) as i64),
Err(WrongType) => write_wrongtype(w),
}
}
/// LRANGE key start end: a slice of the list by position. `range` on a `VecDeque` is
/// an iterator over the items in that range, and `write_bulks` drains it.
fn lrange(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
let (Some(start), Some(end)) = (parse_int(&args[2]), parse_int(&args[3])) else {
return resp::write_error(w, "ERR value is not an integer or out of range");
};
let list = match list_of(store, &args[1]) {
Ok(Some(l)) => l,
Ok(None) => return resp::write_array_len(w, 0),
Err(WrongType) => return write_wrongtype(w),
};
match clamp(start, end, list.len()) {
Some(range) => write_bulks(w, list.range(range).map(Vec::as_slice)),
None => resp::write_array_len(w, 0),
}
}Closures and iterators, named at last#
Rust has written iterators since chapter 4 without naming them. An
iterator is any value with a next method that gives the
next item or None. Every adaptor comes with that one method. map
changes each item and filter keeps some. skip and
take cut a range. flat_map turns each item into several, and
collect builds a collection from what is left. Each adaptor wraps the one before and does nothing
until something at the end asks for an item. That is laziness, and the
scan below depends on it. skip(cursor) and take(count) describe
a page of a million keys without touching the other keys:
/// A HashMap walks in one fixed order while it is not changed, so the cursor is a
/// count of keys already seen. `skip` and `take` are lazy: nothing runs until
/// `collect` asks, and the map is never copied. One more `next` tells whether a
/// page follows.
fn scan(&self, cursor: usize, count: usize) -> (usize, Vec<&[u8]>) {
let mut rest = self.map.keys().skip(cursor);
let page: Vec<&[u8]> = rest.by_ref().take(count).map(Vec::as_slice).collect();
let next = if rest.next().is_some() {
cursor + page.len()
} else {
0
};
(next, page)
}The |k| glob(p, k) inside a filter is a
closure, a function written where it is used, which can capture the
variables around it. A closure has one of three traits, by what it does with what it
captured. Fn only reads, FnMut changes something, and
FnOnce consumes it and can run once. filter asks for an
FnMut, so a closure that counts as it filters is allowed. sadd
uses that to count the members that were new:
fn sadd(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
let key = mem::take(&mut args[1]);
let Value::Set(set) = store.value_or_insert(key, || Value::Set(HashSet::new())) else {
return write_wrongtype(w);
};
let added = args.drain(2..).filter(|m| set.insert(m.clone())).count();
resp::write_int(w, added as i64)
}
fn set_of<'a>(store: &'a dyn Store, key: &[u8]) -> Kind<'a, HashSet<Vec<u8>>> {
match store.value(key) {
None => Ok(None),
Some(Value::Set(s)) => Ok(Some(s)),
Some(_) => Err(WrongType),
}
}
fn sismember(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
match set_of(store, &args[1]) {
Ok(s) => resp::write_int(w, i64::from(s.is_some_and(|s| s.contains(&args[2])))),
Err(WrongType) => write_wrongtype(w),
}
}
fn smembers(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
match set_of(store, &args[1]) {
Ok(Some(s)) => write_bulks(w, s.iter().map(Vec::as_slice)),
Ok(None) => resp::write_array_len(w, 0),
Err(WrongType) => write_wrongtype(w),
}
}Write hgetall. Get the hash with hash_of. Reply with an empty
array when the key is missing and with the error when it holds another kind. Then build a
Vec of borrowed slices, two per field, with iter and
flat_map. Collect it, and write it with write_bulks.
The turbofish on collect is optional here, because the type of the variable
says what to collect into.
fn hgetall(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
let hash = match hash_of(store, &args[1]) {
Ok(Some(h)) => h,
Ok(None) => return resp::write_array_len(w, 0),
Err(WrongType) => return write_wrongtype(w),
};
// your code here
}Stuck?Show the full listing
fn hset(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
if !args.len().is_multiple_of(2) {
return write_arity(w, b"hset");
}
let key = mem::take(&mut args[1]);
let Value::Hash(hash) = store.value_or_insert(key, || Value::Hash(HashMap::new())) else {
return write_wrongtype(w);
};
let mut added = 0;
let mut pairs = args.drain(2..);
while let (Some(field), Some(val)) = (pairs.next(), pairs.next()) {
added += usize::from(hash.insert(field, val).is_none());
}
resp::write_int(w, added as i64)
}
fn hash_of<'a>(store: &'a dyn Store, key: &[u8]) -> Kind<'a, HashMap<Vec<u8>, Vec<u8>>> {
match store.value(key) {
None => Ok(None),
Some(Value::Hash(h)) => Ok(Some(h)),
Some(_) => Err(WrongType),
}
}
fn hget(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
match hash_of(store, &args[1]) {
Ok(h) => match h.and_then(|h| h.get(&args[2])) {
Some(v) => resp::write_bulk(w, v),
None => resp::write_nil(w),
},
Err(WrongType) => write_wrongtype(w),
}
}
/// HGETALL: every field and value, as one flat array. `flat_map` turns each pair into
/// two items, and the count is twice the number of fields.
fn hgetall(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
let hash = match hash_of(store, &args[1]) {
Ok(Some(h)) => h,
Ok(None) => return resp::write_array_len(w, 0),
Err(WrongType) => return write_wrongtype(w),
};
let flat: Vec<&[u8]> = hash
.iter()
.flat_map(|(f, v)| [f.as_slice(), v.as_slice()])
.collect();
write_bulks(w, flat.into_iter())
}Writing an iterator of your own is one method. The file chain.rs in the
snapshot is the C chain of chapter 6 in Rust, kept for two lessons the server did not
need. A type that contains itself must do so through a Box, a pointer to the
heap that owns what it points to. The compiler must know the size of a value, and a
Chain inside a Chain has none. And a type becomes an iterator
by implementing next. Every adaptor then works on it, and so does
for:
/// A recursive enum. Without the `Box`, a `Chain` would contain a `Chain` directly, and
/// the compiler could not say how large one is. `Box` puts the next link on the heap,
/// which makes the size one pointer, and it owns that link, so dropping the head drops
/// the whole chain.
pub enum Chain {
Link(Vec<u8>, Box<Chain>),
End,
}
impl Chain {
pub fn push(self, item: Vec<u8>) -> Chain {
Chain::Link(item, Box::new(self))
}
}
/// An iterator over a chain: a value with one method, `next`, which gives the next item
/// or `None`. Everything else, `map`, `filter`, `count` and the `for` loop, comes with
/// the trait. The iterator takes the chain apart as it goes, so it owns it.
pub struct IntoIter(Chain);
impl Iterator for IntoIter {
type Item = Vec<u8>;
fn next(&mut self) -> Option<Vec<u8>> {
match std::mem::replace(&mut self.0, Chain::End) {
Chain::Link(item, rest) => {
self.0 = *rest;
Some(item)
}
Chain::End => None,
}
}
}
impl IntoIterator for Chain {
type Item = Vec<u8>;
type IntoIter = IntoIter;
fn into_iter(self) -> IntoIter {
IntoIter(self)
}
}Three methods give three kinds of iterator over a collection. iter borrows
each item, iter_mut borrows each mutably, and into_iter takes
the collection apart and gives the items away. The chain only has the last, because its
next consumes a link.
keys.iter().map(|k| store.del(k)); deletes nothing. The
adaptor builds a value that would delete on each next, and no one calls
next. The compiler warns that the iterator is unused, and the book's build
makes the warning an error. A side effect wants a for loop, or a call such as
count() at the end, as del in this chapter writes it.
Iteration, three ways#
Every collection in this chapter is walked three times, once per language, and the three walks are the three answers to one question. Who holds the loop?
| Walking a collection | C | Go | Rust |
|---|---|---|---|
| Who holds the loop | the collection, which calls back | the collection, which calls yield | the caller, which calls next |
| What the caller supplies | a function pointer and a void * context | the loop body, through range | nothing, and adaptors between |
| Stopping early | the callback returns false | the body returns, and yield returns false | the caller stops calling next |
| Composing two walks | by hand, with a second callback | by hand, or with the iter package | filter on map on skip, for free |
| Where the state lives | in the context struct | in the closure the collection made | in the iterator value |
| The cost of a step | an indirect call | an indirect call, which the compiler often inlines | nothing: the chain compiles to a loop |
The C and Go answers are the same shape, a collection that runs the loop and hands each
item to a function. The Rust answer inverts it. The caller runs the loop, and the
collection only knows how to give the next item. That inversion is why Rust iterators
compose. A filter does not need to know what it filters, only how to ask for
the next item and whether to pass it on. The slices and maps
packages of Go now take and return an iterator function, a decade after Rust. C never
will.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| A value of several kinds | Weak | OK | Strong | C has a tag nothing checks, and Go has a type switch nothing checks. Rust has a match the compiler checks. |
| Adding a kind later | OK | Weak | Strong | C warns at each switch with no default, and Go falls to default in silence. Rust refuses to build until every match knows. |
| A list, a hash, a set | Weak | Strong | Strong | C writes each, or nests the table. Go and Rust have them in the standard library. |
| Walking a collection | Weak | OK | Strong | C needs a callback and a context struct, and Go has range over a function. Rust has adaptors that compose. |
| Scanning by cursor | Strong | OK | OK | C and the hand-written table scan buckets. The Go map sorts, and the Rust map counts from the start. |
| Reading the code | Weak | Strong | OK | Thirteen C commands need a helper each, and Go reads like prose. Rust reads well once match and let else are familiar. |
- Sum type. One value that is one of several shapes, with a tag that says which. C checks nothing, Go checks at a type switch, and Rust checks at compile time.
- Cursor. A number the client sends back so the server can continue a walk where it stopped, without holding the whole list.
- Iterator. A value with a
nextmethod, which every adaptor wraps and nothing runs until something asks.
Many clients#
Start the chapter 8 server and point redis-benchmark at it with its default
fifty connections. Then, from a second terminal, ask it one question:
$ redis-benchmark -p 6380 -c 50 -n 1000 -t ping -q
PING_INLINE: rps=0.0 (overall: 262.2) avg_msec=0.000 (overall: 0.043)
$ redis-cli -p 6380 PING # terminal two
Could not connect to Redis at 127.0.0.1:6380: Operation timed outThe benchmark answered a few hundred requests in its first moment and then nothing. It never finishes. The second terminal waited fifteen seconds and could not even connect. One client is being served. Forty-nine are waiting, and the server does not know they exist.
kv serves one client at a time. This chapter makes it serve thousands. C gets an event loop on one thread, with the kernel's own list of ready sockets. Go gets a goroutine for each client and a load generator built from channels. Rust gets a thread for each client, the same tool built from scoped threads, and then the server on tokio.
The scene has a precise shape. The chapter 8
server accepts one connection and serves it until it closes. The benchmark keeps every
connection open until the whole run ends, so the first client is never done. Behind it, the kernel keeps a queue of connections that are complete but not yet accepted. The server chose its length with listen(fd, 16) in chapter 3, so sixteen clients sit in that queue. The other thirty-three sent their first packet and got no answer at all. The kernel drops a connection it cannot queue, and the client tries again a second later, and again. That is the fifteen seconds.
There are two ways out, and the three languages take both. The first is to keep one thread and never block it. The thread asks the operating system which sockets have bytes waiting, reads from those, and asks again. That is an event loop, and it is what Redis does. The second is to give each client its own thread of control and let it block. The operating system or a runtime then switches between them. Go does that with goroutines. Rust does it with threads first, and then with tasks on an asynchronous runtime. That runtime is an event loop written as if each client had its own thread.
| Concurrency | C | Go | Rust |
|---|---|---|---|
| One client is | a heap struct the poller hands back | a goroutine | a thread, then a task |
| Who waits | one call, for all clients at once | each goroutine, on its own read | each thread on its read, then the tokio runtime |
| Which sockets are ready | poll, kqueue or epoll, chosen by #ifdef | the runtime asks, you never see it | the runtime asks, you never see it |
| Platform code lives in | a file per system, under #ifdef | a file per system, with a //go:build line | one function per system, with #[cfg] |
| Memory for a client | two buffers you size | a stack of a few kilobytes that grows | a 2 MiB stack, then a task of a few kilobytes, mostly its buffers |
| The store is shared by | nobody, there is one thread | a sync.RWMutex | an Arc<Mutex<Store>> |
| Forgetting the lock | not possible | a data race, which go test -race may find | does not compile |
| Fifty workers talk through | nothing: C has no load generator here | channels and a WaitGroup | an mpsc channel and a scope |
| Using all the cores | no | yes, the runtime spreads goroutines | yes, with threads or with tokio |
The question "which socket can I use now?" got different answers on different systems,
and every language has to hide the difference. Unix from Berkeley, the Berkeley Software Distribution (BSD), answered first with select. poll came later and takes an array instead of a fixed-size set of bits.
FreeBSD 4.1 added kqueue, which is what macOS uses. Linux added
epoll in October 2002. All of those report readiness. They say "this
socket has bytes", and the program then reads. Windows NT 3.5 took the other shape with I/O
completion ports. A program hands the operating system a buffer and a read request, and the
port reports completion. It says "this read has finished, and the bytes are in your
buffer". The Go runtime and tokio both contain a layer that makes the two shapes look the same. That
is why the Go and Rust versions of kv run on Windows unchanged, and the C one does not.
C: one thread, and a poller to say who is ready#
The loop keeps a structure for each client, holding the socket, the bytes read so far, and the replies not yet written. The buffers are the ones from chapter 4. The clients live on the heap, one allocation each, and the loop keeps an array of pointers to them. The poller in the next section stores a pointer to each client, so a client must never move:
/* One client: its socket, the bytes read so far, and the replies not yet written. */
typedef struct {
int fd;
kv_buf in;
kv_buf out;
bool writing; /* the poller is watching for room to write */
bool closing; /* after a protocol error: close once out is written */
bool dead; /* gone; removed once the batch of events it was part of is done */
} kv_conn;
/* Every client, beside the poller that watches their sockets. The array holds
* pointers, so a client stays where it is in memory when another one is removed, and
* the tag the poller hands back stays valid. */
typedef struct {
kv_conn **conns;
size_t len;
size_t cap;
kv_poller *poller;
} kv_clients;Every socket in the loop is non-blocking. A blocking read on a socket with
no bytes would stop the one thread, and every other client with it. A non-blocking one
returns -1 with errno set to EAGAIN, which means "nothing right
now", and the loop moves on:
/* A non-blocking socket returns EAGAIN instead of waiting, which the loop relies on. */
static bool kv_set_nonblocking(int fd) {
int flags = fcntl(fd, F_GETFL, 0);
return flags >= 0 && fcntl(fd, F_SETFL, flags | O_NONBLOCK) >= 0;
}Accepting clients follows the same rule. When the poller says the listening socket
is ready, there may be several connections waiting, so the loop accepts until
accept itself says EAGAIN:
/* Accept every waiting client. The listening socket is non-blocking, so accept()
* returns EAGAIN when the queue is empty. */
static void kv_accept_all(int listen_fd, kv_clients *cs) {
for (;;) {
int fd = accept(listen_fd, NULL, NULL);
if (fd < 0) {
if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) {
return;
}
perror("accept");
return;
}
if (!kv_set_nonblocking(fd) || !kv_clients_add(cs, fd)) {
close(fd);
}
}
}Reading from a ready client follows the pattern once more. Read until the socket is
empty, then run every complete request in the buffer and queue the replies in the output
buffer. The parser from chapter 4 was written for exactly this moment. Its KV_PARSE_INCOMPLETE status looked like extra work in a one-client server.
Here it lets the loop stop in the middle of a request and come back later:
/* Read what has arrived, run every complete request, and queue the replies. Returns
* false when the client has gone or must be dropped. */
static bool kv_conn_input(kv_conn *c, kv_store *store) {
char tmp[4096];
for (;;) {
ssize_t n = read(c->fd, tmp, sizeof tmp);
if (n < 0) {
if (errno == EAGAIN || errno == EWOULDBLOCK) {
break; /* nothing more for now */
}
if (errno == EINTR) {
continue;
}
return false;
}
if (n == 0) {
return false; /* the client closed */
}
if (!kv_buf_append(&c->in, tmp, (size_t)n)) {
return false;
}
}
for (;;) {
kv_cmd cmd;
size_t used = 0;
kv_parse_status st = kv_parse(c->in.data, c->in.len, &cmd, &used);
if (st == KV_PARSE_INCOMPLETE && c->in.len > KV_MAX_REQUEST) {
st = KV_PARSE_BAD; /* a request that never ends is not a request */
}
/* No default: with -Wall the compiler reports a status this switch forgot. */
switch (st) {
case KV_PARSE_INCOMPLETE:
return true; /* the poller says when more bytes arrive */
case KV_PARSE_BAD:
kv_reply_error(&c->out, "ERR Protocol error");
c->closing = true;
return true;
case KV_PARSE_OK:
if (!kv_exec(&cmd, store, &c->out)) {
return false;
}
kv_buf_consume(&c->in, used);
break;
}
}
}Write kv_conn_output. Loop while the output buffer holds bytes. Call
write with everything that is queued, and consume as many bytes as it took. When
it returns EAGAIN, return true and stop, because the socket is
full and the poller will say when it has room. Retry on EINTR. Return
false on any other error.
static bool kv_conn_output(kv_conn *c) {
while (c->out.len > 0) {
/* your code here */
}
return true;
}Stuck?Show the full listing
/* Write as much of the queued reply as the socket will take right now. */
static bool kv_conn_output(kv_conn *c) {
while (c->out.len > 0) {
ssize_t w = write(c->fd, c->out.data, c->out.len);
if (w < 0) {
if (errno == EAGAIN || errno == EWOULDBLOCK) {
return true; /* the socket is full: the poller will say when to continue */
}
if (errno == EINTR) {
continue;
}
return false;
}
kv_buf_consume(&c->out, (size_t)w);
}
return true;
}One function serves a client that the poller reported. It reads if there is input and writes if a reply is queued. Then it does two things that every event loop has to get right. It asks to be told about room to write only while a reply is waiting, because a socket is almost always writable. And it does not free a finished client. It marks it dead, because the batch of events may hold a second event for the same client:
/* One client had an event. Read if there is input, write if there is room or a reply is
* queued, then ask to be told about room only while a reply is waiting. A socket is
* almost always writable, so asking every time would make every wait return at once. */
static void kv_conn_ready(kv_clients *cs, kv_conn *c, const kv_event *ev, kv_store *store) {
bool alive = true;
if (ev->readable || ev->hangup) {
alive = kv_conn_input(c, store);
}
if (alive && (ev->writable || c->out.len > 0)) {
alive = kv_conn_output(c);
}
if (!alive || (c->closing && c->out.len == 0)) {
c->dead = true; /* not freed yet: this batch may hold another event for it */
return;
}
bool want = c->out.len > 0;
if (want != c->writing && kv_poller_want_write(cs->poller, c->fd, c, want)) {
c->writing = want;
}
}The loop itself is short. Wait, walk the batch, accept when the listening socket is ready and serve the client otherwise, then remove the dead. The removal walks backwards, so that moving the last client into a hole does not skip anyone:
int kv_server_run(int listen_fd, kv_store *store) {
kv_clients cs = {0};
cs.poller = kv_poller_new();
/* the listening socket is watched too, with a null tag to tell it from a client */
if (cs.poller == NULL || !kv_set_nonblocking(listen_fd) ||
!kv_poller_add(cs.poller, listen_fd, NULL)) {
kv_poller_free(cs.poller);
return 1;
}
fprintf(stderr, "kv waits with %s\n", kv_poller_name());
kv_event events[64];
for (;;) {
int n = kv_poller_wait(cs.poller, events, 64, -1);
if (n < 0) {
if (errno == EINTR) {
continue;
}
perror("wait");
break;
}
for (int i = 0; i < n; i++) {
kv_conn *c = events[i].tag;
if (c == NULL) {
kv_accept_all(listen_fd, &cs);
} else if (!c->dead) {
kv_conn_ready(&cs, c, &events[i], store);
}
}
/* walk backwards, because removing a client moves the last one into its slot */
for (size_t i = cs.len; i-- > 0;) {
if (cs.conns[i]->dead) {
kv_clients_remove(&cs, i);
}
}
}
for (size_t i = cs.len; i-- > 0;) {
kv_clients_remove(&cs, i);
}
free(cs.conns);
kv_poller_free(cs.poller);
return 1;
}About two hundred lines of new code, and the server now serves fifty clients from one thread. It also uses one core, which is the limit of this design, and the one Redis still lives with for running commands. The main function shrinks to a call, with one addition. Every client is an open file, and a process starts with a soft limit on those, often 256 on macOS and 1,024 on Linux. The server raises its limit to the hard one, as Redis does, and says how many clients that leaves room for:
/* Every client is an open file, and a process starts with a soft limit on those, often
* 256 on macOS and 1024 on Linux. Raise it to the hard limit, as Redis does, and say
* how many clients that leaves room for. */
static void kv_raise_fd_limit(void) {
struct rlimit lim;
if (getrlimit(RLIMIT_NOFILE, &lim) < 0) {
return;
}
if (lim.rlim_cur < lim.rlim_max && setrlimit(RLIMIT_NOFILE, &(struct rlimit){lim.rlim_max, lim.rlim_max}) == 0) {
lim.rlim_cur = lim.rlim_max;
}
if (lim.rlim_cur == RLIM_INFINITY) {
fprintf(stderr, "kv has no limit on open files\n");
} else {
/* the listener, the poller, stdio and some spare take the rest */
fprintf(stderr, "kv can hold %llu clients\n", (unsigned long long)lim.rlim_cur - 32);
}
}Forget O_NONBLOCK on one socket and a slow client stops the
server. The poller says the socket is readable, and the loop calls read. A client
that sent half a request and went away keeps the one thread waiting for the other half. The other forty-nine clients see a server that has stopped
answering. Nothing crashes and nothing is reported, which makes it one of the harder bugs to find in
a C server. That is one reason the kv_set_nonblocking call is the first thing
done with every new socket.
C again: poll, kqueue and epoll behind one header#
The loop above never names poll. It talks to a poller, which is six functions in a header, and the header decides which of three files provides them. The preprocessor decides, from macros the compiler defines for the target. There is no runtime check and no configuration file. The files that do not match compile to nothing, because everything in them sits under an #ifdef:
/* One of poller_poll.c, poller_kqueue.c and poller_epoll.c holds the code; the other two
* compile to nothing. The compiler predefines a macro for the target: __linux__ on
* Linux, __APPLE__ on macOS, __FreeBSD__ on FreeBSD. KV_POLL on the command line, as in
* `make kv KV_POLL=1`, forces the portable poll() version on any system. */
#if defined(KV_POLL)
#define KV_POLLER_POLL 1
#elif defined(__linux__)
#define KV_POLLER_EPOLL 1
#elif defined(__APPLE__) || defined(__FreeBSD__)
#define KV_POLLER_KQUEUE 1
#else
#define KV_POLLER_POLL 1
#endifThe six functions are the smallest shape the three system calls have in common. A socket is added with a tag, which comes back in every event for that socket, and the tag is the client pointer. A wait fills an array of events and says how many:
typedef struct kv_poller kv_poller;
/* What a wait reports for one socket: the tag given when it was added, and which of
* the three things happened to it. */
typedef struct {
void *tag;
bool readable;
bool writable;
bool hangup;
} kv_event;
kv_poller *kv_poller_new(void);
void kv_poller_free(kv_poller *p);
/* Watch fd for input. The tag comes back in every event for fd. */
bool kv_poller_add(kv_poller *p, int fd, void *tag);
/* Also watch fd for room to write, or stop watching for it. */
bool kv_poller_want_write(kv_poller *p, int fd, void *tag, bool want);
/* Stop watching fd. Call it before close(). */
void kv_poller_remove(kv_poller *p, int fd);
/* Block until a socket is ready or timeout_ms passes, where -1 means wait for ever.
* Fills up to max events and returns how many, 0 on timeout, or -1 with errno set. */
int kv_poller_wait(kv_poller *p, kv_event *events, int max, int timeout_ms);
/* "poll", "kqueue" or "epoll", for the startup line. */
const char *kv_poller_name(void);poll is in POSIX, the Portable Operating System Interface, so its file builds anywhere, and make kv KV_POLL=1 asks for it on any system. Its cost is that every call gives the kernel the whole array to walk. So one wait costs more with every client. kqueue on macOS and epoll on Linux keep the list in the kernel. Adding a socket is one call. A wait sends nothing and gets back only the sockets that changed. The Linux file is the shorter one, and its state is one descriptor:
/* The whole state is one descriptor: the kernel holds the list. */
struct kv_poller {
int ep;
};
kv_poller *kv_poller_new(void) {
kv_poller *p = malloc(sizeof *p);
if (p == NULL) {
return NULL;
}
p->ep = epoll_create1(0);
if (p->ep < 0) {
free(p);
return NULL;
}
return p;
}
void kv_poller_free(kv_poller *p) {
if (p != NULL) {
close(p->ep);
free(p);
}
}
/* One socket has one entry with one mask of events, so a change replaces the mask. */
static bool kv_ep_ctl(kv_poller *p, int op, int fd, uint32_t mask, void *tag) {
struct epoll_event change = {.events = mask, .data.ptr = tag};
return epoll_ctl(p->ep, op, fd, &change) == 0;
}
bool kv_poller_add(kv_poller *p, int fd, void *tag) {
return kv_ep_ctl(p, EPOLL_CTL_ADD, fd, EPOLLIN, tag);
}
bool kv_poller_want_write(kv_poller *p, int fd, void *tag, bool want) {
return kv_ep_ctl(p, EPOLL_CTL_MOD, fd, EPOLLIN | (want ? EPOLLOUT : 0), tag);
}
/* close() drops a socket from the list by itself, unless another descriptor refers to
* the same socket, which is why the explicit call is the safe habit. */
void kv_poller_remove(kv_poller *p, int fd) {
epoll_ctl(p->ep, EPOLL_CTL_DEL, fd, NULL);
}
int kv_poller_wait(kv_poller *p, kv_event *events, int max, int timeout_ms) {
struct epoll_event ready[64];
if (max > 64) {
max = 64;
}
int n = epoll_wait(p->ep, ready, max, timeout_ms);
if (n <= 0) {
return n;
}
for (int i = 0; i < n; i++) {
uint32_t mask = ready[i].events;
events[i] = (kv_event){
.tag = ready[i].data.ptr,
.readable = (mask & EPOLLIN) != 0,
.writable = (mask & EPOLLOUT) != 0,
.hangup = (mask & (EPOLLHUP | EPOLLERR)) != 0,
};
}
return n;
}
const char *kv_poller_name(void) {
return "epoll";
}The designated initialisers, .events = mask and .data.ptr = tag, name the fields they set and leave the rest zero. They are the C way to build a struct without remembering the order of its fields. Redis has the same three files and a fourth for select, and picks one the same way.
Write poller_kqueue.c for macOS and FreeBSD. The whole state is one descriptor from kqueue(). One call, kevent, both changes the list and waits on it. So write a helper that makes one change with EV_SET and asks for no results. A filter is one kind of event on one socket. Watching a socket for reading and for writing is two entries, and a wait can report the socket twice. The loop in the previous section already copes with that. Treat EV_DELETE of a filter that was never added as success.
struct kv_poller {
int kq;
};
static bool kv_kq_change(kv_poller *p, int fd, int16_t filter, uint16_t flags, void *tag) {
/* your code here: EV_SET, then kevent with one change and no results */
}Stuck?Show the full listing
/* The whole state is one descriptor: the kernel holds the list. */
struct kv_poller {
int kq;
};
kv_poller *kv_poller_new(void) {
kv_poller *p = malloc(sizeof *p);
if (p == NULL) {
return NULL;
}
p->kq = kqueue();
if (p->kq < 0) {
free(p);
return NULL;
}
return p;
}
void kv_poller_free(kv_poller *p) {
if (p != NULL) {
close(p->kq);
free(p);
}
}
/* kevent() both changes the list and waits on it. This call only changes it: one
* entry, no room for results, no timeout. A filter is one kind of event on one socket,
* so a socket watched for reading and for writing is two entries. */
static bool kv_kq_change(kv_poller *p, int fd, int16_t filter, uint16_t flags, void *tag) {
struct kevent change;
EV_SET(&change, (uintptr_t)fd, filter, flags, 0, 0, tag);
if (kevent(p->kq, &change, 1, NULL, 0, NULL) < 0) {
/* deleting a filter that was never added is not an error worth stopping for */
return (flags & EV_DELETE) && errno == ENOENT;
}
return true;
}
bool kv_poller_add(kv_poller *p, int fd, void *tag) {
return kv_kq_change(p, fd, EVFILT_READ, EV_ADD, tag);
}
bool kv_poller_want_write(kv_poller *p, int fd, void *tag, bool want) {
return kv_kq_change(p, fd, EVFILT_WRITE, want ? EV_ADD : EV_DELETE, tag);
}
/* close() drops a socket from every kqueue by itself; this only makes it explicit. */
void kv_poller_remove(kv_poller *p, int fd) {
kv_kq_change(p, fd, EVFILT_READ, EV_DELETE, NULL);
kv_kq_change(p, fd, EVFILT_WRITE, EV_DELETE, NULL);
}
int kv_poller_wait(kv_poller *p, kv_event *events, int max, int timeout_ms) {
struct kevent ready[64];
if (max > 64) {
max = 64;
}
struct timespec ts = {timeout_ms / 1000, (timeout_ms % 1000) * 1000000L};
int n = kevent(p->kq, NULL, 0, ready, max, timeout_ms < 0 ? NULL : &ts);
if (n <= 0) {
return n;
}
/* A socket ready for both reading and writing is two entries, with the same tag.
* The loop that gets them copes with that, so they are passed on as they are. */
for (int i = 0; i < n; i++) {
events[i] = (kv_event){
.tag = ready[i].udata,
.readable = ready[i].filter == EVFILT_READ,
.writable = ready[i].filter == EVFILT_WRITE,
.hangup = (ready[i].flags & (EV_EOF | EV_ERROR)) != 0,
};
}
return n;
}
const char *kv_poller_name(void) {
return "kqueue";
}Build the server both ways, and the startup line names the poller. At fifty clients they run at the same speed, because fifty sockets are not enough for the cost of poll to show:
$ make kv && ./kv
kv waits with kqueue
$ make clean && make kv KV_POLL=1 && ./kv
kv waits with pollThe three pollers all report readiness, so one loop fits all of them. Windows reports completion, so a fourth file would not be enough. The loop itself would change, and chapter 14 comes back to what that means for porting the C version.
Go: one word, and one lock#
The Go change in main is the keyword go. The call
serve(conn, s) runs on a new goroutine, and the loop goes straight back to
Accept. The serving function is unchanged. It still blocks on its reads, which
is fine, because it blocks only its own goroutine, and the runtime runs another:
func main() {
ln, err := net.Listen("tcp", ":"+port())
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Fprintln(os.Stderr, "kv listening on port", port())
if n, ok := sys.OpenFiles(); ok {
fmt.Fprintln(os.Stderr, "kv can hold", n-32, "clients")
}
s := store.New()
for {
conn, err := ln.Accept()
if err != nil {
fmt.Fprintln(os.Stderr, "accept:", err)
continue
}
// the one word that changes: the client is served on its own goroutine, and
// the loop goes straight back to accept
go func() {
if err := serve(conn, s); err != nil {
fmt.Fprintln(os.Stderr, "client:", err)
}
}()
}
}A goroutine is not a thread. The Go FAQ says one costs "little overhead beyond the memory for the stack", which is a few kilobytes. It also says it "is practical to create hundreds of thousands of goroutines in the same address space". The event loop that C needed is inside the runtime, and it talks to
kqueue or epoll or a completion port on your behalf.
The runtime in one paragraph#
As many threads run Go code as GOMAXPROCS allows, which by default is the number of CPUs the process may use. Each thread takes a goroutine from a queue and runs it until it blocks, or for ten milliseconds. A goroutine that blocks on a socket is parked, and the runtime's poller queues it again when the socket is ready. One that blocks in a system call keeps its thread, and the runtime starts another. None of this appears in the code, and all of it appears in the numbers at the end of the chapter.
The price appears in the store. Fifty goroutines now call Set and
Get at the same time, and a Go map is not safe for that. The language FAQ says that a mutex on every map operation "would slow down most programs and add safety to few". So the store adds its own, a
read-write lock. Readers take RLock and may hold it together. Writers take
Lock and hold it alone:
// Store keeps every key and its value in a map, behind a lock. Many goroutines read and
// write the store at once, and a Go map is not safe for that without one. The value is
// any of the four kinds in value.go.
type Store struct {
mu sync.RWMutex
m map[string]any
}
func New() *Store {
return &Store{m: make(map[string]any)}
}
// Get takes the read lock, which any number of readers may hold together.
func (s *Store) Get(key []byte) (any, bool) {
s.mu.RLock()
defer s.mu.RUnlock()
v, ok := s.m[string(key)]
return v, ok
}
// Set takes the write lock, which excludes every other reader and writer.
func (s *Store) Set(key []byte, val any) {
s.mu.Lock()
defer s.mu.Unlock()
s.m[string(key)] = val
}
func (s *Store) Del(key []byte) bool {
s.mu.Lock()
defer s.mu.Unlock()
_, ok := s.m[string(key)]
delete(s.m, string(key))
return ok
}
func (s *Store) Len() int {
s.mu.RLock()
defer s.mu.RUnlock()
return len(s.m)
}
// Keys returns every key that matches the pattern, in no particular order.
func (s *Store) Keys(pattern []byte) [][]byte {
s.mu.RLock()
defer s.mu.RUnlock()
var out [][]byte
for k := range s.m {
if glob(pattern, []byte(k)) {
out = append(out, []byte(k))
}
}
return out
}
// Scan returns a page of keys from the cursor on, in sorted order, and the cursor for
// the next page, or 0 when the walk is done.
func (s *Store) Scan(cursor uint64, count int) (uint64, [][]byte) {
s.mu.RLock()
defer s.mu.RUnlock()
sorted := slices.Sorted(maps.Keys(s.m))
if cursor >= uint64(len(sorted)) {
return 0, nil
}
end := min(int(cursor)+count, len(sorted))
var page [][]byte
for _, k := range sorted[cursor:end] {
page = append(page, []byte(k))
}
if end == len(sorted) {
return 0, page
}
return uint64(end), page
}Rewrite Incr so that it holds the write lock from the moment it reads the old
value to the moment it writes the new one. Do not call Get and Set inside it. Each of those takes the lock
again, and a Go lock cannot be taken twice by the same goroutine. Read and write the map directly.
func (s *Store) Incr(key []byte) (int64, error) {
// your code here: lock, read s.m, parse, add, write s.m, unlock
}Stuck?Show the full listing
// Incr holds the write lock from the read to the write, so that two clients that
// increment the same key at the same moment cannot both read the same old value.
func (s *Store) Incr(key []byte) (int64, error) {
s.mu.Lock()
defer s.mu.Unlock()
var n int64
if cur, ok := s.m[string(key)]; ok {
b, isStr := cur.([]byte)
if !isStr {
return 0, ErrWrongType
}
var err error
n, err = strconv.ParseInt(string(b), 10, 64)
if err != nil {
return 0, ErrNotInteger
}
}
if n == math.MaxInt64 {
return 0, ErrOverflow
}
n++
s.m[string(key)] = []byte(strconv.FormatInt(n, 10))
return n, nil
}Without the single lock, two clients can both read 41, both write 42, and one increment
is lost. The test below adds one to the same key from fifty goroutines, two hundred times
each, and expects exactly ten thousand. The WaitGroup counts the goroutines, and its Go method, new in Go 1.25, starts one and counts it in one call. Wait blocks until every one of them has returned:
// TestIncrFromManyGoroutines adds one to the same key from 50 goroutines, 200 times
// each. With the lock the result is 10000. Without it, go test -race reports the data
// race, and the count comes out short.
func TestIncrFromManyGoroutines(t *testing.T) {
s := New()
var wg sync.WaitGroup
for range 50 {
wg.Go(func() {
for range 200 {
if _, err := s.Incr([]byte("n")); err != nil {
t.Error(err)
}
}
})
}
wg.Wait()
got, _ := s.Get([]byte("n"))
if b, _ := got.([]byte); string(b) != "10000" {
t.Fatalf("n = %s, want 10000", got)
}
}$ go test -race ./...
ok kv/internal/store 1.514sThe -race flag turns on the race detector. It watches every memory access
during the test, and reports two goroutines that touch the same byte without a lock between
them. Remove the lock from Incr and the test fails twice. The count comes out
short, and the detector prints both stack traces. Keep the flag on in every test run, which
the book's make verify does.
A race the test never provokes is a race the detector never reports. The detector watches real accesses, so a missing lock on a path the test does not run stays hidden until production runs it. It is a tool for finding races, not a proof that there are none. The Rust compiler makes the other kind of claim later in this chapter.
Go again: fifty workers and three channels#
The scene said that one client was served and forty-nine waited. A tool that counts who got answered can prove it, and it is the best place to meet the rest of Go's concurrency. kvload is a worker pool. One goroutine for each connection, a channel of jobs they all take from, and channels of answers they all send to:
// answer is one round trip: which connection made it and how long it took.
type answer struct {
worker int
took time.Duration
}
// worker owns one connection. It takes job after job from the channel until the channel
// is closed and empty, and reports each round trip on answers, or its first error on
// errs, after which it gives up.
func worker(id int, addr string, jobs <-chan int, answers chan<- answer, errs chan<- error) {
conn, err := net.Dial("tcp", addr)
if err != nil {
errs <- fmt.Errorf("worker %d: %w", id, err)
return
}
defer conn.Close()
r := bufio.NewReader(conn)
for range jobs {
start := time.Now()
if _, err := conn.Write([]byte("PING\r\n")); err != nil {
errs <- fmt.Errorf("worker %d: %w", id, err)
return
}
if _, err := r.ReadString('\n'); err != nil {
errs <- fmt.Errorf("worker %d: %w", id, err)
return
}
answers <- answer{id, time.Since(start)}
}
}The for range jobs loop receives from the channel until the channel is closed and empty, and then ends. The channel types in the signature say which way each one flows, and the compiler holds the worker to that. The main function fills the jobs channel, starts the workers, and then waits on three things at once:
func main() {
addr := flag.String("addr", "127.0.0.1:6380", "the server")
clients := flag.Int("c", 50, "connections")
n := flag.Int("n", 10000, "requests in total")
timeout := flag.Duration("timeout", 5*time.Second, "give up after this long")
flag.Parse()
// A buffered channel holds every job up front. Closing it tells the workers that
// nothing more is coming: their range loops end once the buffer is empty.
jobs := make(chan int, *n)
for i := range *n {
jobs <- i
}
close(jobs)
answers := make(chan answer, *n)
errs := make(chan error, *clients)
var wg sync.WaitGroup
for id := range *clients {
wg.Go(func() { worker(id, *addr, jobs, answers, errs) })
}
// Only the sender may close a channel, and here there are many senders, so a
// goroutine of its own waits for all of them and closes both channels after.
go func() {
wg.Wait()
close(answers)
close(errs)
}()
start := time.Now()
var took []time.Duration
byWorker := map[int]int{}
failed := 0
deadline := time.After(*timeout)
// select waits on three channels at once. A closed channel is set to nil, and a nil
// channel never fires, which takes its case out of the select until the loop ends.
for answers != nil || errs != nil {
select {
case a, ok := <-answers:
if !ok {
answers = nil
continue
}
took = append(took, a.took)
byWorker[a.worker]++
case err, ok := <-errs:
if !ok {
errs = nil
continue
}
failed++
fmt.Fprintln(os.Stderr, err)
case <-deadline:
// the workers still blocked on a read are left behind: the process is
// about to exit, so their goroutines go with it
fmt.Printf("gave up after %v\n", *timeout)
answers, errs = nil, nil
}
}
report(took, byWorker, *clients, failed, time.Since(start))
}Four rules of channels are in that function. A channel with a buffer holds that many values without a receiver, so jobs can be filled before any worker exists. Closing a channel is how a sender says "no more". The language specification says that "sending to or closing a closed channel causes a run-time panic". So only the side that sends may close, and only once. That is why a goroutine of its own waits for every worker and then closes the two result channels. A select waits on several channels and takes the first that is ready, and with a default case it does not wait at all. And a nil channel "is never ready for communication", which sounds useless until a case must be switched off. Setting a closed channel to nil is the idiom, and the loop ends when both are nil.
The deadline case is the one to study. time.After returns a channel that delivers once, after the duration. When it fires, the workers still blocked on a read are abandoned, and the program exits with them still there. That is a goroutine leak, the Go bug that replaces the C memory leak. A goroutine that waits for a reply that never comes lives until the process ends, with its stack. Here the process ends at once. In a server that runs for a year it matters, and the cure is a timeout or a context, which chapter 10 adds. Run the tool against the chapter 8 server and then against this one:
$ go run ./cmd/kvload # the chapter 8 server, one client at a time
10000 answered by 17 of 50 connections, 9984 of them by one, 0 failed, 2.07 s in all
p50 0.045 ms, max 539.894 ms
$ go run ./cmd/kvload # the chapter 9 server
10000 answered by 50 of 50 connections, 232 of them by one, 0 failed, 0.11 s in all
p50 0.525 ms, max 1.861 msSeventeen is one plus sixteen. One connection was served and took nearly every job. Sixteen sat in the kernel's queue with their first PING already sent, and were answered once each when the first one closed. The other thirty-three connected after the jobs were gone. The second run is the chapter. Every connection was served, in a tenth of a second.
The rest of the sync package#
The rest of the sync package is short, and kv needs none of it yet. Once "will perform exactly one action", however many goroutines call it. Pool is a cache of temporary objects, such as buffers, that relieves "pressure on the garbage collector". sync/atomic gives a counter or a flag that many goroutines may change without a lock. Its own documentation says most code should use channels or a mutex instead. The Go proverb behind all of it says to share memory by communicating, and not to communicate by sharing memory. The channels in kvload are that proverb, and the lock in the store is its exception.
Go's platform files: a line at the top#
C chose its poller with #ifdef. Go has no preprocessor, so it chooses files. A comment on the first line, before the package clause, says which systems a file belongs to, and the compiler ignores the file everywhere else. The one place kv needs it is the open-file limit. Since Go 1.19, programs that import the os package "automatically increase the open file limit (RLIMIT_NOFILE) to the maximum allowed value". Reading it back is a system call that Windows does not have, so there are two files. One for every Unix:
// OpenFiles reports how many files, and so how many clients, the process may hold open.
// The Go runtime has already raised the soft limit to the hard one at startup.
func OpenFiles() (uint64, bool) {
var lim syscall.Rlimit
if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &lim); err != nil {
return 0, false
}
return lim.Cur, true
}And one for everything else, with the opposite constraint, so that exactly one of them is ever in the build:
// OpenFiles reports false where the system has no such limit to ask about.
func OpenFiles() (uint64, bool) {
return 0, false
}A file name ending in _linux.go is the same constraint without the comment. GOOS=windows go vet ./... checks the other file without a Windows machine.
Rust: a thread for each client, and a compiler that checks the sharing#
The thread version is the Go version with the sharing made explicit. The store is wrapped in two types. An Arc counts how many owners the store
has. A Mutex lets one thread at a time use it. Each new thread gets a clone of the
Arc, which is a new owner of the same store:
fn main() -> io::Result<()> {
let port = port();
let listener = TcpListener::bind(format!("0.0.0.0:{port}"))?;
eprintln!("kv listening on port {port}");
if let Some(n) = kv::sys::open_files() {
eprintln!("kv can hold {} clients", n - 32);
}
let store = Arc::new(Mutex::new(MapStore::new()));
for stream in listener.incoming() {
match stream {
Ok(stream) => {
// a clone of the Arc is a new owner of the same store, moved into the thread
let store = Arc::clone(&store);
thread::spawn(move || {
if let Err(e) = serve(stream, store) {
eprintln!("client: {e}");
}
});
}
Err(e) => eprintln!("accept: {e}"),
}
}
Ok(())
}Inside the serving function, the lock is taken for one command. It is released at the end
of the line, when the guard goes out of scope. The unwrap_or_else handles a
poisoned lock, which is what a mutex becomes when a thread panics while holding it. The store
is still usable, so the server takes it anyway:
/// The store is shared: an Arc counts the owners, and a Mutex lets one thread at a
/// time use it. The lock is taken for one command and released at the end of the line.
fn serve(stream: TcpStream, store: Arc<Mutex<MapStore>>) -> resp::Result<()> {
let mut reader = BufReader::new(stream.try_clone()?);
let mut writer = BufWriter::new(stream);
loop {
match resp::read_command(&mut reader) {
Ok(Some(args)) => {
let mut store = store.lock().unwrap_or_else(|e| e.into_inner());
cmd::execute(args, &mut *store, &mut writer)?;
}
Ok(None) => return Ok(()),
Err(e @ resp::KvError::Protocol(_)) => {
resp::write_error(&mut writer, "ERR Protocol error")?;
writer.flush()?;
return Err(e);
}
Err(e) => return Err(e),
}
if reader.buffer().is_empty() {
writer.flush()?;
}
}
}What happens without the Arc and the Mutex? The compiler
refuses. Try to move one Store into two threads and it says
error[E0382]: use of moved value: store. Try to share a reference instead and the compiler refuses that too, because the thread may outlive main.
The only way through is a type that is safe to share, and Arc<Mutex<T>>
is the simplest one. The rule is the one the borrow checker applied in chapter 5. A value can have many readers or one writer, never both, and a thread is only another
reader or writer. A data race in a Rust program that compiles without unsafe is not possible. The race
detector that Go runs at test time is work the Rust compiler does at compile time.
Send and Sync#
The check has a name, and you met the single-threaded half of it in chapter 6. Replace the Arc with an Rc. Replace the Mutex with a RefCell, which shares and mutates a value as well on one thread. The compiler says:
error[E0277]: `Rc<RefCell<MapStore>>` cannot be sent between threads safely
--> src/main.rs:57:31
|
57 | thread::spawn(move || {
| ------------- ^------
= help: within `{closure@src/main.rs:57:31: 57:38}`, the trait `Send` is not implemented
for `Rc<RefCell<MapStore>>`
note: required by a bound in `spawn`
|
128 | F: Send + 'static,Send and Sync are two marker traits with no methods. Send means a value may move to another thread. Sync means a reference to it may be shared between threads. The compiler derives both for every type from its parts. A type that holds an Rc loses Send. The reason is that Rc counts its owners without atomic operations, and two threads updating that count at once "is undefined behavior". The spawn function demands Send of its closure, as the note says. So thread safety in Rust is two traits, derived by the compiler and demanded where a value crosses between threads. There is nothing to remember, and no way to forget.
Rust again: the load generator with scoped threads#
The Rust kvload has the shape of the Go one, with each Go idea replaced by the Rust one. A worker owns one connection. The jobs are a counter, an AtomicUsize, rather than a channel. Its fetch_add returns the old value, so every job number is given out once without a lock. The answers go back on an mpsc channel, which has many senders and one receiver, as its name says. The timeout is on the socket, for a reason that comes in a moment:
/// One connection. It takes job numbers from the shared counter until they run out and
/// sends each round trip on the channel. An error ends the worker, and the caller
/// reports it. The read timeout is what lets a worker give up on a server that never
/// answers, because a scoped thread must finish before the scope ends.
fn worker(
id: usize,
addr: SocketAddr,
jobs: &AtomicUsize,
total: usize,
timeout: Duration,
answers: &mpsc::Sender<Answer>,
) -> io::Result<()> {
let mut stream = TcpStream::connect_timeout(&addr, timeout)?;
stream.set_read_timeout(Some(timeout))?;
let mut reader = BufReader::new(stream.try_clone()?);
let mut line = String::new();
// fetch_add returns the old value, so every job number is handed out exactly once
while jobs.fetch_add(1, Ordering::Relaxed) < total {
let start = Instant::now();
stream.write_all(b"PING\r\n")?;
line.clear();
if reader.read_line(&mut line)? == 0 {
return Err(io::ErrorKind::UnexpectedEof.into());
}
// send fails only when main has stopped listening, and then there is nothing to do
let _ = answers.send(Answer {
worker: id,
took: start.elapsed(),
});
}
Ok(())
}The main function is the interesting half. The workers borrow the counter and the sender from main's own stack frame, with no Arc and no cloning of anything but the sender. That is what thread::scope is for. The standard library says that "the scope guarantees all threads will be joined at the end of the scope". No thread outlives the data it borrows, so the borrow checker allows it:
fn main() -> io::Result<()> {
let (addr, clients, total, timeout) = args();
let jobs = AtomicUsize::new(0);
let (tx, rx) = mpsc::channel();
let failed = AtomicUsize::new(0);
let start = Instant::now();
// The scope borrows jobs, failed and tx from this stack frame, and the threads it
// spawns can use them without an Arc, because the scope joins every thread before
// it returns. No thread outlives the data.
let mut took: Vec<Duration> = thread::scope(|s| {
for id in 0..clients {
let tx = tx.clone();
let (jobs, failed) = (&jobs, &failed);
s.spawn(move || {
if let Err(e) = worker(id, addr, jobs, total, timeout, &tx) {
failed.fetch_add(1, Ordering::Relaxed);
eprintln!("worker {id}: {e}");
}
});
}
// Each worker holds a clone of the sender. Once main drops its own, the receiver
// ends when the last worker drops the last clone, which ends the for loop.
drop(tx);
let mut took = Vec::new();
let mut by_worker = vec![0usize; clients];
for answer in rx {
took.push(answer.took);
by_worker[answer.worker] += 1;
}
let answered = by_worker.iter().filter(|n| **n > 0).count();
let most = by_worker.iter().max().copied().unwrap_or(0);
println!(
"{} answered by {answered} of {clients} connections, {most} of them by one, {} failed, {:.2} s in all",
took.len(),
failed.load(Ordering::Relaxed),
start.elapsed().as_secs_f64()
);
took
});
if !took.is_empty() {
took.sort();
let ms = |d: Duration| d.as_secs_f64() * 1e3;
println!(
"p50 {:.3} ms, max {:.3} ms",
ms(took[took.len() / 2]),
ms(took[took.len() - 1])
);
}
Ok(())
}The channel ends the loop. Each worker holds a clone of the sender. When the last worker drops the last clone, the receiver reports that the channel is disconnected, and the for loop ends. The main thread has to drop its own sender first, or the loop would wait for ever. That replaces the WaitGroup, the closing goroutine and the nil channels of the Go version. The scope replaces the leak. A scoped thread must finish before the scope returns, so a stuck worker cannot be abandoned the way the Go version abandons it. The read timeout on the socket is what lets it give up and report an error. The result against the chapter 8 server is the same seventeen:
$ cargo run --release --bin kvload # the chapter 8 server
10000 answered by 17 of 50 connections, 9984 of them by one, 0 failed, 2.06 s in all
p50 0.033 ms, max 410.443 ms
$ cargo run --release --bin kvload # the chapter 9 server
10000 answered by 50 of 50 connections, 221 of them by one, 0 failed, 0.12 s in all
p50 0.540 ms, max 2.066 msAtomics#
An atomic is an integer that many threads may change without a lock, because the processor does each operation in one step. Every operation takes an Ordering, which says what else it promises about memory. Relaxed promises nothing beyond the counter itself, which is enough here and not enough for publishing data to another thread. A mutex or a channel promises that. Both are built from atomics, and so is Arc, which is why it may cross threads and Rc may not.
RwLock#
Go's store took a read-write lock, and Rust has the same type. A RwLock "allows a number of readers or at most one writer at any point in time". The server does not use it yet, because every command takes the store mutably. Chapter 13 makes the commands say which of them only read, and measures what the read lock gives.
A few days after it landed on Mars in July 1997, the Pathfinder lander began resetting itself. Its software ran on VxWorks, with tasks at fixed priorities and a shared area of memory behind a mutex. A low-priority task that published weather data took the mutex. A medium-priority communications task then preempted it and ran for a long time. The high-priority task that managed the shared memory waited for the mutex, behind a task that could not run. A watchdog noticed that the high-priority task had not run and reset the whole system. The engineers at the Jet Propulsion Laboratory reproduced it on the replica in their lab, with tracing on, and saw a classic priority inversion. The mutex had been created with priority inheritance turned off. They uploaded a short C program that changed the flag from FALSE to TRUE, and the resets stopped. A lock is a promise about memory, which Go checks at test time and Rust at compile time. It is also a promise about time, who holds it and for how long, and no language checks that one. Taking the lock for one command and dropping it at the end of the line is the answer kv has.
Rust's platform code: an attribute on the item#
Rust answers the platform question with an attribute, not a file. #[cfg(unix)] on a function, a module or a statement keeps it out of the build on every other system. #[cfg(target_os = "linux")] names one system. The open-file limit is the one place kv needs it, and the standard library has no call for it. So the function asks the C library, through the libc crate, which tokio already depends on. That call is the first unsafe block in the book. The word marks a promise the compiler cannot check, and chapter 15 is about it:
/// How many files, and so how many clients, the process may hold open. The standard
/// library has no call for it, so this asks the C library, which chapter 15 explains.
#[cfg(unix)]
pub fn open_files() -> Option<u64> {
let mut lim = libc::rlimit {
rlim_cur: 0,
rlim_max: 0,
};
// SAFETY: getrlimit writes one struct through a valid pointer and reads nothing.
let rc = unsafe { libc::getrlimit(libc::RLIMIT_NOFILE, &mut lim) };
(rc == 0 && lim.rlim_cur != libc::RLIM_INFINITY).then_some(lim.rlim_cur)
}The other version carries the opposite condition, so that exactly one of them is compiled:
/// Where the system has no such limit to ask about, the answer is None.
#[cfg(not(unix))]
pub fn open_files() -> Option<u64> {
None
}Rust once more: tasks on tokio#
A thread for each client works for fifty clients and for five thousand. Each thread has a 2 MiB stack, on every main platform. So fifty thousand threads would reserve a hundred gigabytes of stack. The operating system limits how many threads a process may create long before that. The answer is the C answer, an event loop, with the Go answer on top of it. That is a cheap
unit of concurrency that looks like a thread. In Rust that is an async
task, and the runtime that schedules tasks is a library, not part of the language. The
one almost everyone uses is tokio, and it becomes the first line in the
[dependencies] section of Cargo.toml:
$ cargo add tokio@1 --features rt-multi-thread,net,io-util,macros
Adding tokio v1.53.2 to dependenciesThat command also writes Cargo.lock, with the exact version of tokio and of
the fourteen crates it depends on. Every later build uses those versions until you ask for
new ones, which is what the lock file is for. The server is a second binary in the same crate, src/bin/kv_async.rs. Its
serving function looks like the thread version with one word added after every read and
write:
/// The replies are written into a Vec first, with the ordinary synchronous command
/// code from the library, and then sent with one awaited write. The lock is never held
/// across an await, so a plain std Mutex is the right one.
async fn serve(stream: TcpStream, store: Arc<Mutex<MapStore>>) -> resp::Result<()> {
let mut reader = BufReader::new(stream);
let mut out = Vec::new();
loop {
match read_command(&mut reader).await {
Ok(Some(args)) => {
let mut store = store.lock().unwrap_or_else(|e| e.into_inner());
cmd::execute(args, &mut *store, &mut out)?;
}
Ok(None) => return Ok(()),
Err(e @ KvError::Protocol(_)) => {
resp::write_error(&mut out, "ERR Protocol error")?;
reader.get_mut().write_all(&out).await?;
return Err(e);
}
Err(e) => return Err(e),
}
if reader.buffer().is_empty() {
reader.get_mut().write_all(&out).await?;
out.clear();
}
}
}What a future is#
An async fn does not run when it is called. It returns a future, which the standard library defines as "a value that might not have finished computing yet". A future has one method, poll. It "attempts to resolve the future into a final value" and returns either Ready with the value or Pending. "Futures alone are inert", the documentation says. Nothing happens until a runtime polls them. A waker, a handle the future stores and the socket wakes, tells the runtime when to poll again. Each .await in the function above is a point
where the task may return Pending and let another task use the thread. That is the C event loop with the bookkeeping done by the compiler. It turns the function into a state machine with one state per await.
Pin in one paragraph#
That state machine is why Pin appears in every async signature. Between two awaits, the function's local variables live inside the future, and one may be a reference to another. A value that points into itself must never move, or the pointer would point at the old address. Pin is a wrapper that promises a value will "remain located at the same place in memory" until it is dropped. You write the word only when you implement a future yourself, which this book never does.
The parser from chapter 4 has to be written again with .await after each read.
A function that blocks must not be called from a task, because it stops the thread and every task on it:
/// The parser from chapter 4 again, with `.await` after every read. A blocking read
/// would stop the whole thread; an awaited one lets the thread run another task.
async fn read_line(r: &mut BufReader<TcpStream>) -> resp::Result<Option<Vec<u8>>> {
let mut line = Vec::new();
if r.read_until(b'\n', &mut line).await? == 0 {
return Ok(None);
}
if line.pop() != Some(b'\n') {
return Err(io::Error::from(io::ErrorKind::UnexpectedEof).into());
}
if line.last() == Some(&b'\r') {
line.pop();
}
Ok(Some(line))
}
fn parse_len(digits: &[u8], type_byte: u8) -> resp::Result<usize> {
std::str::from_utf8(digits)
.ok()
.and_then(|s| s.parse().ok())
.ok_or(KvError::Protocol(type_byte))
}
async fn read_command(r: &mut BufReader<TcpStream>) -> resp::Result<Option<Command>> {
let Some(line) = read_line(r).await? else {
return Ok(None);
};
if line.first() != Some(&b'*') {
let words = line.split(|b| *b == b' ').filter(|w| !w.is_empty());
return Ok(Some(words.map(<[u8]>::to_vec).collect()));
}
let n = parse_len(&line[1..], b'*')?;
if n > MAX_ARGS {
return Err(KvError::Protocol(b'*'));
}
let mut args = Vec::with_capacity(n);
for _ in 0..n {
let line = read_line(r)
.await?
.ok_or(io::Error::from(io::ErrorKind::UnexpectedEof))?;
if line.first() != Some(&b'$') {
return Err(KvError::Protocol(line.first().copied().unwrap_or(b'\n')));
}
let size = parse_len(&line[1..], b'$')?;
if size > MAX_BULK {
return Err(KvError::Protocol(b'$'));
}
let mut buf = vec![0u8; size + 2];
r.read_exact(&mut buf).await?;
if &buf[size..] != b"\r\n" {
return Err(KvError::Protocol(buf[size]));
}
buf.truncate(size);
args.push(buf);
}
Ok(Some(args))
}Write the main of the tokio version. Mark it #[tokio::main] and
make it async. Bind a tokio::net::TcpListener. In a loop, await accept and clone
the Arc. Then tokio::spawn an async block that calls
serve. The shape is the thread version with thread::spawn
replaced and .await added.
#[tokio::main]
async fn main() -> io::Result<()> {
// your code here
}Stuck?Show the full listing
#[tokio::main]
async fn main() -> io::Result<()> {
let port = port();
let listener = TcpListener::bind(format!("0.0.0.0:{port}")).await?;
eprintln!("kv (tokio) listening on port {port}");
let store = Arc::new(Mutex::new(MapStore::new()));
loop {
let (stream, _) = listener.accept().await?;
let store = Arc::clone(&store);
// a task, not a thread: it costs a few hundred bytes and no kernel resources
tokio::spawn(async move {
if let Err(e) = serve(stream, store).await {
eprintln!("client: {e}");
}
});
}
}select!, cancellation and blocking#
Two more pieces of tokio appear in the next chapters. tokio::select! is the select of Go, for futures. It "waits on multiple concurrent branches, returning when the first branch completes, cancelling the remaining branches". Cancelling means dropping, and that is the whole of cancellation in Rust. A dropped future stops, because a future that is not polled does nothing. Chapter 10 uses it to wait for a client and for a shutdown signal at once. The other piece is the rule about blocking. A blocking call inside a task stops the thread from advancing every other task on it. The cure is spawn_blocking, which "runs the provided closure on a thread dedicated to blocking operations", and chapter 11 uses it for fsync.
Hold a std::sync::Mutex guard across .await and the task
will not compile. The error is future cannot be sent between threads safely. The note says that
MutexGuard is not Send, and that "await occurs here, with
mut guard maybe used later". It is the same Send check as the thread version, on the future this time. A task may
resume on another thread after an await, and a mutex must be unlocked by the thread that
locked it. The serving function above takes the lock, runs the command into a buffer, and
drops the guard before the write is awaited. The tokio crate has its own mutex for the cases where a lock must be held across an await.
It is slower, which is why the code avoids needing it.
The command code in the library did not change for tokio. It writes replies into any
Write, and a Vec<u8> is one. The task fills a vector with the
synchronous code and sends the vector with one awaited write. That keeps the async surface
of a program small, which makes the compiler's messages shorter and the code easier to test
without a runtime.
What the four servers measure#
Each server was given fifty connections and two hundred thousand requests by
redis-benchmark, driven from four threads, on a machine with twelve cores. The
numbers are requests a second, and chapter 13 explains how to take and read such numbers
properly.
C, kqueue, one thread SET 72,700 GET 79,900
Go, goroutines SET 88,700 GET 79,800
Rust, threads SET 99,900 GET 88,800
Rust, tokio SET 99,800 GET 88,800Three things in the table are worth more than the numbers. The C server uses one core and is within a third of the others, which is why Redis stayed single-threaded for so long. The Go and Rust servers use every core. A lock on one store stops them from using the cores well. Chapter 13 returns to that. And the tokio version is no faster than the thread version at fifty clients, because fifty threads are cheap. Its advantage starts where threads run out, at tens of thousands of connections.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| Serving many clients | Weak | Strong | OK | C needs 200 lines of event loop and three pollers, Go one keyword, and Rust an Arc, a Mutex and a clone per thread. |
| Using every core | Weak | Strong | Strong | The C loop is one thread by design. Go and Rust spread clients over every core. |
| A forgotten lock | n/a | Weak | Strong | Go compiles it and the detector may catch it. Rust refuses to compile it. |
| Platform code | OK | Strong | Strong | C needs a file per poller under ifdef. Go and Rust need one guarded function, and their runtimes hide the pollers. |
| Memory for a client | Strong | Strong | OK | C sizes two buffers and a goroutine is a few kilobytes, while a Rust thread is 2 MiB until tokio fixes that at the cost of a dependency. |
| Reading the code | Weak | Strong | OK | The C loop is subtle and the Go version is the chapter 7 code, while the tokio version is a second copy of the parser. |
| Windows | Weak | Strong | Strong | The C loop reports readiness and Windows reports completion. Both runtimes hide that. |
- Event loop. One thread asks the kernel which sockets are ready and serves those. poll walks an array every time. The other two keep a list in the kernel.
- Unit of concurrency. A C client is a struct. A Go client is a goroutine of a few kilobytes. A Rust client is a 2 MiB thread or a small tokio task.
- Who checks the lock. Nobody in C, the race detector at test time in Go, and the Send and Sync traits at compile time in Rust. None of them checks how long you hold it.
Expiry#
Give the chapter 9 C server a value of a million bytes. Then be a bad client. Ask for it eight times, read sixteen bytes of the eight megabytes, and hang up. Do it in a loop:
$ redis-cli -p 6380 -x SET big < million-x.txt
OK
$ for i in 1 2 3; do
> (exec 3<>/dev/tcp/127.0.0.1/6380; for _ in 1 2 3 4 5 6 7 8; do printf 'GET big\r\n'; done >&3
> head -c 16 <&3 >/dev/null)
> kill -0 $pid && echo "hang-up $i: the server is still here" || { wait $pid; echo "hang-up $i: gone, status $?"; break; }
> done
hang-up 1: the server is still here
hang-up 2: gone, status 141Status 141 is 128 plus 13, and signal 13 is SIGPIPE. One client that did not want its reply has killed the server for every other client, and nothing was logged.
A cache forgets. This chapter gives kv keys that die on time, with two options on SET and
the commands EXPIRE and TTL. It also teaches the server about signals, so that a hang-up no longer kills it and a kill stops it cleanly. And the Rust parser stops copying, which is where lifetimes come in.
Redis is used as a cache more than as anything else, and a cache entry is only useful with a lifetime. A session that lasts an hour is a key with a deadline. So is a rate limit that resets each minute, and a lock that must not outlive a crashed holder. After the deadline the key must behave as if it had been deleted, whether or not anybody asks for it.
That last clause is where the work is. A key that is read after its deadline is the simple case. The read notices and deletes it. A key that nobody reads again must still go, or the server fills with dead entries. So there are two mechanisms. Lazy expiry checks the deadline on every access. Active expiry runs on a timer and removes dead keys that nobody has touched. Redis does both, and so does kv.
| Time, signals and background work | C | Go | Rust |
|---|---|---|---|
| A deadline | an int64_t of milliseconds, 0 for none | a time.Time, zero for none | an Option<Instant> |
| Reading the clock | clock_gettime(CLOCK_MONOTONIC) | time.Now(), which carries both clocks | Instant::now() |
| Lazy expiry | kv_store_find unlinks a dead entry | Get reports it absent, the sweep removes it | get filters it out, the sweep removes it |
| The sweep runs | in the event loop, between waits | on a goroutine with a ticker | on a thread with a sleep, then a tokio task with an interval |
| A hang-up mid-reply | SIGPIPE, ignored since this chapter | an EPIPE error, by default | an EPIPE error, by default |
| Ctrl-C or kill | a handler writes to a pipe the loop waits on | a context that the signal cancels | a thread that waits for the signal, then a scope that waits for the threads |
| Stopping the clients | the loop returns | AfterFunc closes each connection | the socket is shut, or the task is dropped |
| The arguments of a request | pointers into the buffer, by convention | copies, with the capacity capped | slices of the buffer, by lifetime |
The whole chapter rests on one question, "is this entry dead yet?", and each language
answers it in its own shape. The C entry stores a deadline as a number and treats zero as
never. Go stores a time.Time and treats the zero time as never. Rust stores an
Option, so "never" is a case the compiler makes every reader handle:
/* 0 means the key never expires */
typedef struct kv_entry {
/* ... key, value ... */
int64_t expires_at;
struct kv_entry *next;
} kv_entry;
static bool kv_expired(const kv_entry *e, int64_t now) {
return e->expires_at != 0 && e->expires_at <= now;
}// the zero time means the key never expires
type entry struct {
val []byte
expires time.Time
}
func (e entry) expired(now time.Time) bool {
return !e.expires.IsZero() && !now.Before(e.expires)
}// None means the key never expires
struct Entry {
val: Vec<u8>,
expires: Option<Instant>,
}
impl Entry {
fn expired(&self, now: Instant) -> bool {
self.expires.is_some_and(|at| at <= now)
}
}Every operating system offers two clocks. The wall clock says what time it is, and it can jump, when an administrator sets it or when the network corrects it. The monotonic clock says how much time has passed since some moment, and it only moves forward. A deadline measured on the wall clock expires at once when the clock jumps ahead an hour, and never when it jumps back. Redis stores expiry times as Unix timestamps, on the wall clock, because it must write them to disk and read them back on another machine. Its documentation warns that setting the computer time forward expires keys at once. The server kv keeps everything in memory, so it uses the monotonic clock. A deadline is then a reading of a clock that only moves forward, and the number means nothing to another process.
C: a deadline on the entry, and a sweep with a cursor#
The clock is one function, in a file of its own because chapter 11 will need it too.
CLOCK_MONOTONIC is POSIX, the Portable Operating System Interface, and the call
is clock_gettime:
/* Milliseconds on the monotonic clock, which only moves forward. It is not the time of
* day: that clock jumps when the system time is set, and a deadline must not. */
int64_t kv_now_ms(void);An entry gains a field. The comment says what zero means, which matters because every check in the file depends on it:
/* One key and its value. expires_at is a deadline in milliseconds on the monotonic
* clock, or 0 for a key that never expires. The key, the value and the next entry are
* owned. */
typedef struct kv_entry {
char *key;
size_t klen;
kv_value val;
int64_t expires_at;
struct kv_entry *next;
} kv_entry;Lazy expiry goes into kv_store_find, which now takes the current time. An
entry past its deadline is unlinked on the spot, which is why the walk uses the
pointer-to-pointer form from chapter 6 again. The special value now = 0 turns
the check off, for the callers that have already decided the key is alive:
static bool kv_expired(const kv_entry *e, int64_t now) {
return e->expires_at != 0 && e->expires_at <= now;
}
/* Unlink and free the entry that *link points to. */
static void kv_unlink(kv_store *s, kv_entry **link) {
kv_entry *e = *link;
*link = e->next;
kv_entry_free(e);
s->len--;
}
kv_entry *kv_store_find(kv_store *s, const char *key, size_t klen, int64_t now) {
if (s->cap == 0) {
return NULL;
}
size_t i = kv_index(s, key, klen, s->cap);
for (kv_entry **link = &s->buckets[i]; *link != NULL; link = &(*link)->next) {
kv_entry *e = *link;
if (s->type->eq(e->key, e->klen, key, klen)) {
if (kv_expired(e, now)) {
kv_unlink(s, link); /* lazy expiry: the key dies when it is touched */
return NULL;
}
return e;
}
}
return NULL;
}Write kv_store_sweep. Start at the bucket the store's cursor
names, walk its chain, unlink every expired entry, and move to the next bucket. Stop after
looking at budget entries or after one full round of the buckets, and leave the
cursor where the next call should start. Count what you removed. A bucket index wraps with
the mask cap - 1, because cap is a power of two.
size_t kv_store_sweep(kv_store *s, int64_t now, size_t budget) {
size_t removed = 0;
size_t seen = 0;
/* your code here */
return removed;
}Stuck?Show the full listing
size_t kv_store_sweep(kv_store *s, int64_t now, size_t budget) {
size_t removed = 0;
size_t seen = 0;
for (size_t n = 0; n < s->cap && seen < budget; n++) {
kv_entry **link = &s->buckets[s->cursor];
while (*link != NULL && seen < budget) {
seen++;
if (kv_expired(*link, now)) {
kv_unlink(s, link); /* *link now points at the next entry */
removed++;
} else {
link = &(*link)->next;
}
}
s->cursor = (s->cursor + 1) & (s->cap - 1);
}
return removed;
}The budget matters because the sweep runs on the one thread that serves every client. A sweep that walks ten million keys stops the server for the time that takes. The cursor makes a long walk into many short ones. Redis uses a different bound, a random sample, but the purpose is the same. The sweep is called from the event loop. The loop now waits at most a tenth of a second, so that keys expire on time even when no client is talking:
/* Wait at most 100 ms, so that keys expire even when no client is talking. */
int n = kv_poller_wait(cs.poller, events, 64, 100);
if (n < 0) {
if (errno == EINTR) {
continue;
}
perror("wait");
break;
}
kv_store_sweep(store, kv_now_ms(), 200);The SET command grows two options, and a plain SET removes any
deadline, because that is what Redis does and what clients expect. A number that is not a
number, and a lifetime of zero, get the two error texts Redis uses:
/* SET key value [EX seconds | PX milliseconds]. A plain SET removes any deadline, as
* Redis does; a SET with EX or PX sets a new one. */
static bool kv_cmd_set(const kv_cmd *c, kv_store *s, kv_buf *out, int64_t now) {
if (c->argc != 3 && c->argc != 5) {
return kv_reply_arity(out, "set");
}
int64_t ttl_ms = 0;
if (c->argc == 5) {
long long n;
bool ex = kv_arg_is(c, 3, "EX");
if (!ex && !kv_arg_is(c, 3, "PX")) {
return kv_reply_error(out, "ERR syntax error");
}
if (!kv_parse_int(c->argv[4], c->arglen[4], &n)) {
return kv_reply_error(out, "ERR value is not an integer or out of range");
}
if (n <= 0 || n > INT64_MAX / 1000) {
return kv_reply_error(out, "ERR invalid expire time in 'set' command");
}
ttl_ms = ex ? n * 1000 : n;
}
if (!kv_store_set(s, c->argv[1], c->arglen[1], c->argv[2], c->arglen[2])) {
return false;
}
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1], 0);
e->expires_at = ttl_ms ? now + ttl_ms : 0;
return kv_reply_simple(out, "OK");
}EXPIRE and TTL are short. The one surprise is that
EXPIRE with zero or a negative number deletes the key at once. The Redis
documentation says a non-positive timeout results in the key "being deleted rather than
expired", and the test file checks for it:
/* EXPIRE key seconds: 1 if the key exists and got a deadline, 0 if it does not exist. */
static bool kv_cmd_expire(const kv_cmd *c, kv_store *s, kv_buf *out, int64_t now) {
long long secs;
if (!kv_parse_int(c->argv[2], c->arglen[2], &secs)) {
return kv_reply_error(out, "ERR value is not an integer or out of range");
}
if (secs > INT64_MAX / 1000) { /* now + secs * 1000 must not overflow */
return kv_reply_error(out, "ERR invalid expire time in 'expire' command");
}
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1], now);
if (e == NULL) {
return kv_reply_int(out, 0);
}
if (secs <= 0) {
kv_store_del(s, c->argv[1], c->arglen[1]); /* a deadline in the past is a DEL */
} else {
e->expires_at = now + secs * 1000;
}
return kv_reply_int(out, 1);
}
/* TTL key: the seconds left, -1 for a key with no deadline, -2 for a missing key. */
static bool kv_cmd_ttl(const kv_cmd *c, kv_store *s, kv_buf *out, int64_t now) {
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1], now);
if (e == NULL) {
return kv_reply_int(out, -2);
}
if (e->expires_at == 0) {
return kv_reply_int(out, -1);
}
return kv_reply_int(out, (e->expires_at - now + 500) / 1000); /* nearest, as Redis */
}kv_store_find can now free the entry it is asked about. Any
pointer to that entry held across the call is a pointer to freed memory. Inside kv_exec_incr the code finds the entry and reads its deadline. Then it
calls kv_store_set and finds the entry again. The first pointer must not be
trusted after a call that can change the table. This rule was true in chapter 6. It became easier to break in this chapter, because a
function named find no longer sounds like one that writes.
Go: a ticker, and a clock the test can hold still#
A Go time.Time carries both clocks. The documentation says a value from time.Now "contains both a wall clock
reading and a monotonic clock reading". Sub and Before use the
monotonic one. So a deadline is a plain
time.Time, and a zero value means none:
// entry is a value of any kind and its deadline. A zero time means the key never
// expires. The deadline is a time.Time, which carries a monotonic reading, so a change
// to the system clock does not move it.
type entry struct {
val any
expires time.Time
}
func (e entry) expired(now time.Time) bool {
return !e.expires.IsZero() && !now.Before(e.expires)
}Every method takes now as a parameter instead of calling
time.Now itself. That is a small decision with a large effect. The command code
reads the clock once per command. So every check inside one command, such as the expiry test and the deadline it sets, sees the same moment. And a test can pass any time it likes,
which is how the expiry test runs in a millisecond without sleeping:
// TestExpiry uses a clock the test controls, so it never sleeps.
func TestExpiry(t *testing.T) {
s := New()
t0 := time.Now()
s.Set([]byte("k"), []byte("v"))
s.SetExpiry([]byte("k"), t0.Add(time.Second), t0)
if _, ok := s.Get([]byte("k"), t0.Add(999*time.Millisecond)); !ok {
t.Fatal("key died early")
}
if _, ok := s.Get([]byte("k"), t0.Add(time.Second)); ok {
t.Fatal("key outlived its deadline")
}
if n := s.Sweep(t0.Add(time.Second), 100); n != 1 {
t.Fatalf("sweep removed %d keys, want 1", n)
}
if s.Len() != 0 {
t.Fatalf("store holds %d keys after the sweep, want 0", s.Len())
}
}Get holds the read lock, and a dead key cannot be removed under a read lock.
So Get reports the key absent and leaves the removal to the sweep. The sweep
takes the write lock, walks the map, and stops after a budget. Go promises no order for a map walk, and the runtime starts each walk at a different place. That gives the sweep a fresh sample without any cursor:
// SetExpiry gives a live key a deadline. It reports whether the key was there.
func (s *Store) SetExpiry(key []byte, at time.Time, now time.Time) bool {
s.mu.Lock()
defer s.mu.Unlock()
e, ok := s.m[string(key)]
if !ok || e.expired(now) {
return false
}
e.expires = at
s.m[string(key)] = e
return true
}
// TTL returns the time left, and a status: -2 for a missing key, -1 for a key with
// no deadline, 0 for a key with one.
func (s *Store) TTL(key []byte, now time.Time) (time.Duration, int) {
s.mu.RLock()
defer s.mu.RUnlock()
e, ok := s.m[string(key)]
if !ok || e.expired(now) {
return 0, -2
}
if e.expires.IsZero() {
return 0, -1
}
return e.expires.Sub(now), 0
}
// Sweep removes keys past their deadline, looking at no more than budget of them. The
// map iterates in a different order each time, so each call sees a fresh sample.
func (s *Store) Sweep(now time.Time, budget int) int {
s.mu.Lock()
defer s.mu.Unlock()
removed := 0
for k, e := range s.m {
if budget == 0 {
break
}
budget--
if e.expired(now) {
delete(s.m, k)
removed++
}
}
return removed
}Write sweep, which main starts with go sweep(ctx, s). Make a time.NewTicker for a tenth of a second and stop it with defer. Loop over a select with two cases. When the context is done, return. When the ticker's channel delivers a time, call s.Sweep with it and a budget of 200. The goroutine takes the store's lock like any other, and the ticker makes it sleep in between.
func sweep(ctx context.Context, s *store.Store) {
// your code here
}Stuck?Show the full listing
// sweep removes keys past their deadline ten times a second, so that a key nobody reads
// still dies on time. It takes the store's lock like any other goroutine. The select
// waits on the ticker and on the context at once, and returns when the server stops.
func sweep(ctx context.Context, s *store.Store) {
tick := time.NewTicker(100 * time.Millisecond)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case now := <-tick.C:
s.Sweep(now, 200)
}
}
}A function that calls time.Now can only be tested with real time, which
means tests that sleep and tests that fail on a slow machine. A function that takes
now can be tested with any time at all. The store does the second. The command
layer reads the clock once and passes it down. The same shape works in Rust and in C, and
the C version of kv does it too, in kv_exec, though without the test.
Rust: an Option for the deadline, a thread and then a task for the sweep#
Rust separates the clocks into two types. SystemTime is the wall clock.
Instant is the monotonic one, and its documentation says it "will never go
backwards". A deadline is Option<Instant>, so the "none" case is a case
the compiler makes you handle rather than a zero you must remember:
/// A value and its deadline. `None` means the key never expires. An `Instant` is a
/// reading of the monotonic clock, so a change to the system time cannot move it.
struct Entry {
val: Value,
expires: Option<Instant>,
}
impl Entry {
fn expired(&self, now: Instant) -> bool {
self.expires.is_some_and(|at| at <= now)
}
}get borrows the store immutably, so it filters a dead key out and leaves it
for the sweep, as the Go version does. TTL has three answers, and the enum Ttl gives each a name. The
command code then matches on three names instead of on -2, -1 and a number:
fn set_expiry(&mut self, key: &[u8], at: Instant, now: Instant) -> bool {
match self.map.get_mut(key) {
Some(e) if !e.expired(now) => {
e.expires = Some(at);
true
}
_ => false,
}
}
fn ttl(&self, key: &[u8], now: Instant) -> Ttl {
match self.map.get(key) {
None => Ttl::Missing,
Some(e) if e.expired(now) => Ttl::Missing,
Some(Entry { expires: None, .. }) => Ttl::Never,
Some(Entry {
expires: Some(at), ..
}) => Ttl::Left(*at - now),
}
}
/// `retain` walks the whole map, which is fine for a store of this size; Redis
/// samples instead, to bound the time.
fn sweep(&mut self, now: Instant) -> usize {
let before = self.map.len();
self.map.retain(|_, e| !e.expired(now));
before - self.map.len()
}The SET options are a second match arm with five elements. The inner match on the unit and the parsed number has one arm for each error
text. The compiler checks that the arms are complete:
/// SET key value [EX seconds | PX milliseconds]. A plain SET removes any deadline, as
/// Redis does; a SET with EX or PX sets a new one.
fn set(args: &Command, store: &mut dyn Store, now: Instant, w: &mut dyn Write) -> io::Result<()> {
match args.as_slice() {
[_, key, val] => {
// the one copy: the store must own its key and value, the buffer does not
store.set(key.to_vec(), val.to_vec());
resp::write_simple(w, "OK")
}
[_, key, val, unit, n] => {
let ttl = match (unit.to_ascii_uppercase().as_slice(), parse_int(n)) {
(b"EX" | b"PX", None) => {
return resp::write_error(w, "ERR value is not an integer or out of range");
}
(b"EX", Some(n)) if n > 0 && n <= i64::MAX / 1000 => Duration::from_secs(n as u64),
(b"PX", Some(n)) if n > 0 => Duration::from_millis(n as u64),
(b"EX" | b"PX", Some(_)) => {
return resp::write_error(w, "ERR invalid expire time in 'set' command");
}
_ => return resp::write_error(w, "ERR syntax error"),
};
let Some(at) = now.checked_add(ttl) else {
return resp::write_error(w, "ERR invalid expire time in 'set' command");
};
store.set(key.to_vec(), val.to_vec());
store.set_expiry(key, at, now);
resp::write_simple(w, "OK")
}
_ => write_arity(w, b"set"),
}
}In the thread version, spawn a thread inside the scope. It sleeps 100 milliseconds, locks the store, and
calls sweep with Instant::now(), until the stopping flag is raised. It borrows the store from the scope, with no clone. In the tokio version, spawn a task instead, with
tokio::time::interval, and a select! between the tick and the stop channel. The tokio feature
time must be added to Cargo.toml for the interval to exist.
s.spawn(|| {
// your code here
});Stuck?Show the full listing
// Ten times a second, a thread of its own removes keys past their deadline. It
// takes the lock like any other thread, and sleeps in between.
s.spawn(|| {
while !stopping.load(Ordering::Relaxed) {
thread::sleep(Duration::from_millis(100));
let mut store = store.lock().unwrap_or_else(|e| e.into_inner());
store.sweep(Instant::now());
}
});// The sweep is a task like the clients. The await on the interval costs no thread,
// and select! ends the task when the stop arrives.
let sweeper = Arc::clone(&store);
let mut stop_rx = stop.subscribe();
tokio::spawn(async move {
let mut tick = tokio::time::interval(Duration::from_millis(100));
loop {
tokio::select! {
_ = tick.tick() => {
let mut store = sweeper.lock().unwrap_or_else(|e| e.into_inner());
store.sweep(Instant::now());
}
_ = stop_rx.recv() => return,
}
}
});The two sweepers differ in one way that matters. The thread version occupies a thread that spends almost all its time asleep, which is cheap at one sweeper and wasteful at a thousand timers. The task version occupies nothing while it waits. The await on the interval hands the thread back to the runtime, and the runtime wakes the task when the tick arrives. That is the same trade as chapter 9, seen from the other side.
The Rust sweep has no budget. retain visits every entry
under the lock, which is fine at ten thousand keys and a pause at ten million. The C version
has a cursor, and the Go version has a budget on an iteration that starts at a random
place. A Rust HashMap iterates in a fixed order for a given state, so taking the
first 200 entries would sweep the same 200 every time. A bounded sweep in Rust needs its own
cursor, or a random sample, as Redis takes. The book leaves it unbounded and says so, which
is the honest version of a difference between the three standard libraries.
C: a signal, a flag and a pipe#
Back to the scene. A write to a socket whose other end has gone raises SIGPIPE, and the default action for SIGPIPE is to end the process. The kernel does not ask. That default is from the days when a pipeline such as cat file | head was the only pipe. A writer with nobody left to read should end, without a message. A server is the opposite case. One line in main changes the default to ignoring the signal. The write then fails with EPIPE instead, and the loop closes that one client:
/* A write to a client that has gone raises SIGPIPE, which kills the process by
* default. Ignore it, and write() reports EPIPE instead, which closes that client.
* Linux can ask for the same with MSG_NOSIGNAL on send(), and macOS with the
* SO_NOSIGPIPE socket option. Ignoring the signal covers both. */
signal(SIGPIPE, SIG_IGN);
/* Ctrl-C and kill send SIGINT and SIGTERM, which also end the process by default.
* From here on they end the event loop instead, through a pipe it waits on. */
int stop_fd = kv_stop_fd();
if (stop_fd < 0) {
perror("kv_stop_fd");
return 1;
}The same line adds the second signal story. Ctrl-C sends SIGINT and kill sends SIGTERM, and both end the process at once by default, in the middle of whatever it was doing. A server would rather finish the command it is running, and in chapter 11 write the last bytes of its log. So the two signals get a handler. A handler is a function the kernel calls between any two instructions of the program, even in the middle of malloc. It may only do two things safely. It may set a flag of the one type made for that. And it may call the functions on the async-signal-safe list, which holds write and not printf:
/* Which signal asked the server to stop, or 0. sig_atomic_t is the one integer type
* that a signal handler may write while the program reads it, without a torn value, and
* volatile tells the compiler the value can change behind its back. */
extern volatile sig_atomic_t kv_stop_signal;/* A signal handler runs between any two instructions of the program, even in the middle
* of malloc or printf. So it may only do what is safe at any moment: set the flag, and
* call functions on the async-signal-safe list. write() is on the list. The byte wakes
* the loop, which is otherwise asleep in its wait and would not look at the flag. */
static void kv_on_stop(int sig) {
int saved = errno; /* write() may change errno, and the interrupted code needs it */
kv_stop_signal = sig;
if (write(kv_stop_write, "s", 1) < 0) {
/* the pipe is full or closed: the loop is already awake, or already gone */
}
errno = saved;
}The flag alone is not enough, and the reason is a race. The loop is asleep in its wait when the signal arrives, and it checks the flag only when it wakes. A signal that arrives a moment before the wait is missed until the next event, which may be an hour away. The answer is the self-pipe trick, which Daniel Bernstein described in 1991. The handler writes one byte into a pipe, and the loop watches the pipe's other end like a socket. A signal is then an event, delivered through the same wait as everything else:
int kv_stop_fd(void) {
int fds[2];
if (pipe(fds) < 0) {
return -1;
}
/* non-blocking, so that a handler can never wait on a full pipe */
for (int i = 0; i < 2; i++) {
int flags = fcntl(fds[i], F_GETFL, 0);
if (flags < 0 || fcntl(fds[i], F_SETFL, flags | O_NONBLOCK) < 0) {
return -1;
}
}
kv_stop_write = fds[1];
/* sigaction, not signal(): it says exactly what happens, on every Unix. SA_RESTART
* lets a read or accept that the signal interrupted carry on, because the pipe,
* not the interruption, is what tells the loop to stop. */
struct sigaction sa = {0};
sa.sa_handler = kv_on_stop;
sigemptyset(&sa.sa_mask);
sa.sa_flags = SA_RESTART;
if (sigaction(SIGINT, &sa, NULL) < 0 || sigaction(SIGTERM, &sa, NULL) < 0) {
return -1;
}
return fds[0];
}sigaction is the call to install a handler. The older signal is for SIG_IGN only, because what it does to an interrupted call differs between systems. The flag SA_RESTART says that a read or an accept the signal interrupted should carry on as if nothing happened. The pipe, not the interruption, is what tells the loop to stop. In the loop, the byte is an event with a tag of its own:
/* A byte on the pipe is the signal, delivered to the loop as an event like
* any other. Nothing was interrupted in the middle, so there is nothing to
* repair: the loop ends, and main closes everything in order. */
if (c == (kv_conn *)&stop_tag) {
rc = 0;
break;
}Nothing was interrupted in the middle, so nothing needs repair. The loop returns zero, and main frees the store and closes the socket in order, as it would have after any other return. The protocol tests gain a line, terminate, which sends SIGTERM and expects exit status 0 within two seconds. From this chapter on, every server in the book passes it.
A signal handler that calls printf, malloc or free can deadlock or corrupt the heap. Those functions take locks and keep state, and the handler may run while the main program is inside them. The symptom is a server that hangs once a month, on a signal that arrived at the wrong instruction. The rule is short. A handler sets a sig_atomic_t, writes a byte, or calls _exit, and everything else happens in the loop. The volatile on the flag is the other half. Without it the compiler may read the flag once and keep the value in a register for ever.
Go: a context that a signal cancels#
Go's runtime handles SIGPIPE on a socket by itself. The os/signal package says a write to a broken pipe on any descriptor but 1 and 2 "will fail with a syscall.EPIPE error". So the Go server survived the scene without a line of code. For SIGINT and SIGTERM, the same package offers a channel, and a context. One call turns the signal into a cancelled context, and every part of the server that holds the context can see it:
// Ctrl-C and kill send SIGINT and SIGTERM, which end a Go program at once by default.
// NotifyContext changes that: the signal cancels a context instead, and every part of
// the server that holds the context sees it. A second signal, after stop() has run,
// ends the program the default way, for a server that will not finish.
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()context in one paragraph#
A Context "carries deadlines, cancellation signals, and other request-scoped values" down a chain of calls, as the first parameter of every function in the chain. WithCancel gives a child that a function call cancels. WithTimeout and WithDeadline give one that time cancels. Done is a channel that closes when the context is cancelled, which is what a select waits on, and Err says why. Values ride along with WithValue, for request-scoped data such as a trace identifier. The documentation warns against using them to pass ordinary parameters. kv needs only the cancel. The sweeper's select waits on the ticker and on ctx.Done() at once, and returns when the server stops.
A clean stop has three parts, and each is a few lines. Closing the listener makes a blocked Accept "unblocked and return errors", which ends the loop. context.AfterFunc runs a function when the context is cancelled, and the function closes a connection, which ends that client's serve. The WaitGroup from chapter 9 waits for every serve to return:
// Three parts to a clean stop. Closing the listener makes Accept return an error,
// which ends the loop. AfterFunc closes each client when the context is cancelled,
// which ends its serve. The WaitGroup waits for every serve to return.
var clients sync.WaitGroup
context.AfterFunc(ctx, func() { ln.Close() })
for {
conn, err := ln.Accept()
if err != nil {
if ctx.Err() != nil {
break
}
fmt.Fprintln(os.Stderr, "accept:", err)
continue
}
clients.Go(func() {
defer context.AfterFunc(ctx, func() { conn.Close() })()
if err := serve(conn, s); err != nil && ctx.Err() == nil {
fmt.Fprintln(os.Stderr, "client:", err)
}
})
}
fmt.Fprintln(os.Stderr, "kv stopping:", context.Cause(ctx))
clients.Wait()panic and recover#
One more defence goes into serve, and it is against the book's own bugs. A panic "stops the ordinary flow of control", unwinds the goroutine's stack running every deferred call, and then ends the whole process. An index out of range in one command handler would take down every client. recover "regains control of a panicking goroutine", and the specification says it works only when called directly by a deferred function. So the deferred function below catches the panic, turns it into an error for this one connection, and the connection closes. The bug is logged and is still a bug. The result is named, err, because a deferred function can set a named result and nothing else:
// A panic in one command would end the whole process, and every other client with
// it. recover, which only works inside a deferred function, turns it into an error
// for this one connection, which closes. The bug is still logged, and still a bug.
defer func() {
if r := recover(); r != nil {
err = fmt.Errorf("panic: %v", r)
}
}()That is the Go rule for the two kinds of failure. An error is a value for the caller, returned and checked, as every err in this book has been. A panic is a bug in the program. recover is for the one place that must survive bugs in code below it, which is a server's per-connection loop. A library that panics at a caller's mistake is thought badly of. A library that returns an error for its own bugs hides them.
Rust: a thread that waits for the signal, then a scope that waits for the threads#
The Rust standard library ignores SIGPIPE before main runs, so the thread server of chapter 9 logged client: Broken pipe (os error 32) in the scene and went on. It has no signal handling of its own, so the stop goes through the C library again, with the pattern that fits threads best. The signals are blocked in main before any thread exists, so every thread inherits the block. One thread then asks for them with sigwait, which "suspends execution of the calling thread until one of the signals" arrives. No handler ever runs between two instructions. An ordinary thread gets the signal as a return value and may do anything:
/// Block SIGINT and SIGTERM in the calling thread. Every thread spawned afterwards
/// inherits the mask, so the one thread that calls `wait_for_stop` is the only one that
/// ever sees them, and no handler runs in the middle of anything.
#[cfg(unix)]
pub fn block_stop_signals() {
// SAFETY: the set is initialised by sigemptyset before any use, and the mask call
// reads it and writes nothing else.
unsafe {
let mut set = std::mem::zeroed::<libc::sigset_t>();
libc::sigemptyset(&mut set);
libc::sigaddset(&mut set, libc::SIGINT);
libc::sigaddset(&mut set, libc::SIGTERM);
libc::pthread_sigmask(libc::SIG_BLOCK, &set, std::ptr::null_mut());
}
}
/// Wait until SIGINT or SIGTERM arrives, and say which. sigwait takes the signal off
/// the pending list and returns it to an ordinary thread, which may then do anything.
#[cfg(unix)]
pub fn wait_for_stop() -> i32 {
// SAFETY: as above, and sigwait writes one int through a valid pointer.
unsafe {
let mut set = std::mem::zeroed::<libc::sigset_t>();
libc::sigemptyset(&mut set);
libc::sigaddset(&mut set, libc::SIGINT);
libc::sigaddset(&mut set, libc::SIGTERM);
let mut sig = 0;
libc::sigwait(&set, &mut sig);
sig
}
}
/// Elsewhere there is nothing to block, and nothing to wait for: Ctrl-C ends the
/// process the default way.
#[cfg(not(unix))]
pub fn block_stop_signals() {}
#[cfg(not(unix))]
pub fn wait_for_stop() -> i32 {
loop {
std::thread::park();
}
}With that, the whole server fits in one thread::scope. The store, the stop flag and the list of clients are locals of main, lent to every thread. The Arc from chapter 9 is gone. The sweeper and the signal thread run inside the scope. Each client runs inside it. And the scope ends only when all of them have returned, which is the waiting part of a clean stop, done by the type system:
// One thread waits for the signal. It raises the flag and then wakes the accept
// loop the only way a plain listener can be woken: by connecting to it.
s.spawn(|| {
let sig = sys::wait_for_stop();
eprintln!("kv stopping on signal {sig}");
stopping.store(true, Ordering::Relaxed);
let _ = TcpStream::connect(("127.0.0.1", port.parse().unwrap_or(6380)));
});// Shutting a socket makes its read return 0, and that client's serve returns.
// The scope then joins every thread, so nothing is still running when main ends.
for c in clients.lock().unwrap_or_else(|e| e.into_inner()).iter() {
let _ = c.shutdown(Shutdown::Both);
}Two details are unusual, and both are honest. A plain listener can only be woken by a connection. So the signal thread connects to its own server, and the accept loop sees the flag and breaks. And a client blocked in read is woken by shutting its socket from outside, through a second handle. After that, "all currently blocked and future reads will return Ok(0)". The tokio version has no such problem, because tokio owns the event loop:
/// Completes when Ctrl-C or kill asks the process to stop. Asking tokio for the signal
/// is what turns off the default, which would end the process at once.
async fn stop_signal() {
#[cfg(unix)]
{
use tokio::signal::unix::{SignalKind, signal};
let mut term = signal(SignalKind::terminate()).expect("SIGTERM handler");
tokio::select! {
_ = tokio::signal::ctrl_c() => {}
_ = term.recv() => {}
}
}
#[cfg(not(unix))]
{
let _ = tokio::signal::ctrl_c().await;
}
}Asking tokio for a signal is what turns off the default. From then on the accept loop waits on two futures with select!, the next connection and the stop. The stop reaches the tasks through the two channels from chapter 9's last section. A broadcast channel, where "each sent value is seen by all consumers", tells every task. An mpsc channel, where every task holds a sender it never uses, lets main wait until the last sender is dropped:
// Two channels for a clean stop. The broadcast channel tells: every task holds a
// receiver, and one send reaches all of them. The mpsc channel waits: every task
// holds a sender it never uses, and recv() returns None once the last one is dropped.
let (stop, _) = broadcast::channel::<()>(1);
let (done, mut all_done) = mpsc::channel::<()>(1);loop {
let stream = tokio::select! {
accepted = listener.accept() => accepted?.0,
_ = stop_signal() => break,
};
let store = Arc::clone(&store);
let mut stop_rx = stop.subscribe();
let done = done.clone();
tokio::spawn(async move {
// A task waits on its client and on the stop at once. When the stop wins,
// the serve future is dropped where it stands, between two awaits, and that
// is the whole of cancellation: nothing to flag, nothing to catch.
tokio::select! {
res = serve(stream, store) => {
if let Err(e) = res {
eprintln!("client: {e}");
}
}
_ = stop_rx.recv() => {}
}
drop(done);
});
}
eprintln!("kv (tokio) stopping");
let _ = stop.send(());
drop(done);
all_done.recv().await; // None, once every task has dropped its senderThe line to look at is the select! inside the task. When the stop wins, the serve future is dropped where it stands, at whichever await it was parked on. That is cancellation in Rust, all of it. The lock is never held across an await, so a command is never cut in half. The socket closes when the future's locals are dropped. Go needed an AfterFunc and a closed connection to get the same effect, because a goroutine cannot be dropped, only persuaded.
The parser stops copying#
Every parser so far has copied each argument out of the buffer the bytes arrived in. Redis does not, and from this chapter the Rust version of kv does not either. A SET copies its key and value when it stores them, and only then. A GET copies nothing at all. The change is one type, and the type is a lifetime:
/// A parsed request: the command name and its arguments, as slices of the buffer the
/// request arrived in. Nothing is copied. The lifetime `'a` is the buffer's, and it says
/// that a `Command` cannot outlive the bytes it points into. A command that needs to
/// keep an argument, as SET keeps its value, copies it then, and only then.
pub type Command<'a> = Vec<&'a [u8]>;
/// What `parse` found in the bytes so far.
#[derive(Debug)]
pub enum Parsed<'a> {
/// A whole request, and how many bytes of the buffer it used.
Done(Command<'a>, usize),
/// Not a whole request yet: read more bytes and call again.
Incomplete,
}Lifetimes, in the one place kv needs them#
A struct or a type alias that holds a reference must say what the reference borrows from, and 'a is that name. The Rust book puts it in one sentence. An instance "can't outlive the reference it holds". So a Command<'a> cannot be kept after the buffer it points into is changed or freed, and the compiler refuses a program that tries. The annotation changes nothing about how long anything lives. It only gives the compiler a name to check against, where it cannot infer one. In a function it usually can. The elision rules say that one reference in means the reference out borrows from it. So parse below writes '_, and the compiler fills in the rest:
/// One request from the front of `buf`: an array of bulk strings such as
/// `*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n`, or one inline line of words. The returned
/// `Command` borrows `buf`, which the signature says with the elided lifetime: one
/// reference in, so the one reference out lives as long as it.
pub fn parse(buf: &[u8]) -> Result<Parsed<'_>> {
let Some((line, mut used)) = take_line(buf, 0) else {
return Ok(Parsed::Incomplete);
};
if line.first() != Some(&b'*') {
let words = line.split(|b| *b == b' ').filter(|w| !w.is_empty());
return Ok(Parsed::Done(words.collect(), used));
}
let n = parse_len(&line[1..], b'*')?;
if n > MAX_ARGS {
return Err(KvError::Protocol(b'*'));
}
let mut args = Vec::with_capacity(n);
for _ in 0..n {
let Some((line, start)) = take_line(buf, used) else {
return Ok(Parsed::Incomplete);
};
if line.first() != Some(&b'$') {
return Err(KvError::Protocol(line.first().copied().unwrap_or(b'\n')));
}
let size = parse_len(&line[1..], b'$')?;
if size > MAX_BULK {
return Err(KvError::Protocol(b'$'));
}
let end = start + size;
if buf.len() < end + 2 {
return Ok(Parsed::Incomplete);
}
if &buf[end..end + 2] != b"\r\n" {
return Err(KvError::Protocol(buf[end]));
}
args.push(&buf[start..end]);
used = end + 2;
}
Ok(Parsed::Done(args, used))
}
/// The line that starts at `from`, without its ending, and where the next line starts.
/// A bare LF is accepted as well as CRLF, so that a command typed into nc works.
fn take_line(buf: &[u8], from: usize) -> Option<(&[u8], usize)> {
let rest = &buf[from..];
let nl = rest.iter().position(|b| *b == b'\n')?;
let line = rest[..nl].strip_suffix(b"\r").unwrap_or(&rest[..nl]);
Some((line, from + nl + 1))
}
fn parse_len(digits: &[u8], type_byte: u8) -> Result<usize> {
std::str::from_utf8(digits)
.ok()
.and_then(|s| s.parse().ok())
.ok_or(KvError::Protocol(type_byte))
}The serving loop is where the borrow is felt. It reads into a Vec, parses a command out of it, runs it, and then removes the used bytes with drain. The drain changes the vector, and a Command that still pointed into it would be a bug. The compiler allows the call because the borrow ended at the last use of args, not at the end of the block. That is the non-lexical rule from Rust 2018, and without it this loop would need an extra scope:
/// Bytes are read into a buffer and whole requests are parsed out of it without a
/// copy. The store is shared: the Mutex lets one thread at a time use it, and the scope
/// in main lends it to every client, so no Arc is needed.
fn serve(mut stream: TcpStream, store: &Mutex<MapStore>) -> resp::Result<()> {
let mut writer = BufWriter::new(stream.try_clone()?);
let mut buf: Vec<u8> = Vec::with_capacity(4096);
let mut chunk = [0u8; 4096];
loop {
let n = stream.read(&mut chunk)?;
if n == 0 {
return Ok(()); // the client closed, or the server shut the socket
}
buf.extend_from_slice(&chunk[..n]);
let mut used = 0;
loop {
match resp::parse(&buf[used..]) {
Ok(Parsed::Done(args, n)) => {
let mut store = store.lock().unwrap_or_else(|e| e.into_inner());
cmd::execute(&args, &mut *store, &mut writer)?;
used += n;
}
Ok(Parsed::Incomplete) => break,
Err(e @ resp::KvError::Protocol(_)) => {
resp::write_error(&mut writer, "ERR Protocol error")?;
writer.flush()?;
return Err(e);
}
Err(e) => return Err(e),
}
}
// the arguments borrowed the buffer, and they are gone, so the buffer may change
buf.drain(..used);
if buf.len() > resp::MAX_REQUEST {
return Err(resp::KvError::Protocol(b'*')); // a request that never ends
}
writer.flush()?;
}
}Three smaller things fall out of the same change. A &[u8] is Copy, so a handler can use an argument twice without a clone, where a Vec would have moved. The unknown-command reply uses String::from_utf8_lossy, which returns a Cow, "a clone-on-write smart pointer". It borrows when the bytes are valid text and allocates only when it must replace a byte. And thread::spawn demands 'static of its closure, as chapter 9's error message showed, which means "owns everything it holds". A Command can never be sent to a spawned thread, and a scoped thread may borrow it. The tokio server gained the most. Its second copy of the parser, with an await after every read, is gone. A parser that works on bytes already in memory has nothing to wait for.
Go cannot say any of this in a type, and so it chose to copy. What it can say is how much an argument may grow in place. Each argument is a slice of a buffer with two bytes of line ending after it. append "re-uses the underlying array" when there is room. So an append to such an argument would write over the line ending, or over whatever a later parser keeps there. The next command would see it. The full slice expression buf[:size:size] sets the capacity to the length, so an append must copy:
// buf[:size] would share its spare two bytes with the line ending. The full slice
// expression caps the capacity at the length, so an append to the argument copies.
args = append(args, buf[:size:size])// Every argument is a slice into a buffer the parser allocated, with two bytes of line
// ending after it. The full slice expression buf[:size:size] sets the capacity to the
// length, so that an append to an argument must copy instead of writing over the line
// ending, or over whatever a later version of the parser keeps there.
func TestArgumentsDoNotShareSpareCapacity(t *testing.T) {
r := bufio.NewReader(strings.NewReader("*3\r\n$3\r\nSET\r\n$1\r\nk\r\n$2\r\nhi\r\n"))
args, err := ReadCommand(r)
if err != nil {
t.Fatal(err)
}
val := args[2]
if cap(val) != len(val) {
t.Fatalf("cap %d, len %d: an append would write into the parser's buffer", cap(val), len(val))
}
grown := append(val, '!')
if &grown[0] == &val[0] {
t.Fatal("append reused the argument's memory")
}
if string(val) != "hi" || string(grown) != "hi!" {
t.Fatalf("val %q, grown %q", val, grown)
}
}C has had the borrowed parser since chapter 4, and the rule since then has been one comment. The const on each argument is the part of it the compiler can check. It is a promise not to write through the pointer, carried through every call that takes one:
/* One parsed request. The arguments point into the request buffer, so they are valid
* only until that buffer is consumed. A kv_cmd owns nothing: the comment is the whole
* of the rule, and const is the part of it the compiler can check. */
typedef struct {
size_t argc;
const char *argv[KV_MAX_ARGS];
size_t arglen[KV_MAX_ARGS];
} kv_cmd;On 18 February 2017 a security researcher found other people's cookies and passwords in pages served by Cloudflare. Cloudflare's edge servers rewrote HTML with a parser generated by Ragel, and the generated C checked for the end of the buffer with if (++p == pe). A pointer that had already stepped past the end never equalled it. The parser read on, into memory that held fragments of other customers' requests. The company's report says that "had the check been done using >= instead of == jumping over the buffer end would have been caught". About one request in three million leaked, over five months, and search engines had cached some of the leaked pages. A parser that returns pointers into a buffer is the fastest kind and the most dangerous kind. The C rule is a comment and a habit. The Rust rule is the lifetime on Command<'a>, and the compiler would have refused the bug.
Background work, three ways#
The sweep is the first work in kv that no client asked for. Each language had a natural place to put it. In C it went into the loop that already existed, with a timeout on the wait, and the loop does the sweep between clients. There is no second thread and no lock. In Go it went into a goroutine with a ticker, and the lock on the store made that safe without a thought. In Rust it went into a thread, and then into a task, and the compiler checked that the store was shared correctly both times.
Each placement reflects the chapter before. The C loop is one thread, so the sweep must share it and must be short, which is why it has a budget. The Go and Rust servers have many threads of control, so the sweep can be one more. The cost is a lock that every client also pays. Chapter 11 adds the second kind of background work, writing to disk, where the cost of sharing is higher and the three answers differ more.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| A deadline | OK | Strong | Strong | C uses zero to mean none, which the comments must say. Go's zero Time and Rust's Option say it in the type. |
| The clock | OK | Strong | Strong | C picks the clock by name, while Go carries both in one value and Rust gives each a type. |
| Lazy expiry | Strong | OK | OK | C deletes on find. Go and Rust must leave it to the sweep because of the read lock or the borrow. |
| The sweep | Strong | Strong | Weak | C has a cursor and Go a random sample. Rust's retain walks the whole map. |
| A client that hangs up | Weak | Strong | Strong | C dies unless one line ignores SIGPIPE. Both runtimes turn it into an error. |
| Stopping cleanly | OK | Strong | OK | C needs a handler, a pipe and a rule about what a handler may do. Go needs a context, and Rust the C library or tokio. |
| A bug in one command | Weak | Strong | OK | C corrupts or crashes, and Go recovers per connection. Rust panics per thread, and the scope reports it. |
| The parser's arguments | OK | OK | Strong | C borrows by comment, Go copies and caps the capacity, Rust borrows by lifetime and the compiler checks it. |
| Testing time | Weak | Strong | Strong | Go and Rust tests pass a clock in. The C store has no test runner yet. |
- Two expiries. Lazy expiry checks the deadline on every access. Active expiry sweeps on a timer, with a budget, so that a key nobody reads still dies.
- A signal is an event. The C handler writes a byte into a pipe the loop waits on. Go turns it into a cancelled context. Rust gives it to one thread, or to a select! in tokio.
- A lifetime is a name. Command<'a> says the arguments borrow the buffer and cannot outlive it. The annotation changes nothing about how long anything lives.
Persistence#
Start this chapter's C server with a log file, pour a million writes into it, and pull the plug a second and a half in. Then start it again:
$ KV_AOF=kv.aof ./kv &
$ redis-benchmark -p 6380 -t set -n 1000000 -r 100000 -q &
$ kill -9 $pid # no signal handler runs for this one
$ ls -l kv.aof
-rw-r--r-- 1 you staff 2686230 Oct 12 kv.aof
$ KV_AOF=kv.aof ./kv
kv replayed 59694 commands from kv.aof
$ redis-cli -p 6380 DBSIZE
45076Fifty-nine thousand commands came back from a file the server never had time to force
to disk. Now set KV_FSYNC=always and run the benchmark again. It manages 43
commands a second. The chapter is about the space between those two numbers.
Stop kv and every key is gone. This chapter gives it an append-only file, a log of every write in the bytes the client
sent. A restart replays the log through the parser, and the store comes back. The chapter is about files and fsync, about the thread that waits for the disk, and about what a crash takes.
A cache may forget, but a store that forgets on every restart is not much use. The simplest way to remember is to write down every command that changed something, in order. At startup, run them all again. Redis calls this the append-only file, and its documentation says the commands "are logged using the same format as the Redis protocol itself". That one decision means the parser you already have is also the loader, and a text editor can read the file.
Two details make it work. The first is expiry. A command such as SET k v EX 60
means "sixty seconds from now", and "now" is gone by the time the file is replayed. So the
file stores a deadline on the wall clock instead, as PEXPIREAT k 1760000000000,
which is a Redis command too. The second is fsync. A write call
hands bytes to the kernel, which keeps them in memory and writes them to the disk when it
likes. fsync waits until they are on the disk. The first survives a crash of
the process, which is what the scene showed. Only the second survives a crash of the machine, and it is slow.
| The file, and the thread that waits for the disk | C | Go | Rust |
|---|---|---|---|
| Open for appending | open(path, O_WRONLY | O_CREAT | O_APPEND) | os.OpenFile with the same flags | OpenOptions::new().create(true).append(true) |
| One record | encode into a kv_buf, then write | encode into a bufio.Writer, then Flush | encode into a BufWriter, then flush |
| Forcing it to disk | fsync, or F_FULLFSYNC on macOS | f.Sync(), which does the same | file.sync_all(), which does the same |
| Who waits for the disk | a pthread, asleep on a condition variable | a goroutine, ranging over a channel | a scoped thread, ranging over an mpsc receiver, or spawn_blocking |
| How the loop asks | lock, set a flag, signal, unlock | a send on a channel with room for one | a send on the channel |
| What checks the sharing | ThreadSanitizer, at test time | the race detector, at test time | Send, at compile time |
| Replay | kv_parse in a loop over the file | resp.ReadCommand on a bufio.Reader | resp::parse on a buffer read from the file |
| Log and store together | one thread, so no question | the log has its own lock | one lock around a Db with both |
The append-only file arrived in Redis 1.1, and it has kept the protocol format since. The persistence documentation gives the reason in practical terms. The file is "an easy to understand and parse format". If you flush the database by mistake, "you can still save your data set just by stopping the server, removing the latest command, and restarting". The cost is size. A counter incremented a hundred times is one key in memory and a hundred records on disk. Redis solves that by rewriting the file in the background, as "the minimal set of operations needed to create the current data set". The server kv does not rewrite, and the chapter says where that leaves it.
C: a file descriptor, a write, and a thread for the fsync#
The file is opened once, with O_APPEND, which makes every write
go to the end of the file whatever else has happened to it. The policy is one of three,
and they are the three that Redis offers under the names always,
everysec and no:
/* How often the file is forced to disk. Redis has the same three settings. */
typedef enum {
KV_FSYNC_ALWAYS, /* after every command: slow and safe */
KV_FSYNC_EVERYSEC, /* once a second: lose at most a second of writes */
KV_FSYNC_NO /* never: the operating system decides */
} kv_fsync_policy;
typedef struct {
int fd; /* -1 when persistence is off */
kv_fsync_policy policy;
bool dirty; /* bytes written since the last fsync was asked for */
int64_t last_request; /* monotonic ms of the last request to the syncer */
kv_syncer syncer; /* the thread that waits for the disk */
} kv_aof;bool kv_aof_open(kv_aof *a, const char *path, kv_fsync_policy policy) {
/* O_APPEND: every write goes to the end, even if another process appends too */
int fd = open(path, O_WRONLY | O_CREAT | O_APPEND, 0644);
if (fd < 0) {
perror(path);
return false;
}
*a = (kv_aof){.fd = fd, .policy = policy, .last_request = kv_now_ms()};
if (!kv_syncer_start(&a->syncer, fd)) {
perror("kv_syncer_start");
close(fd);
a->fd = -1;
return false;
}
return true;
}Recording a command means encoding it as an array of bulk strings, which is the format
the client sent and the format kv_parse reads. The one rewrite turns a relative
expiry into an absolute one, with the wall clock from a second function in
clock.c. The record goes to the kernel with write at once. That is the system call, not fwrite, which would put a buffer of the C library's own in front of the kernel's. The Go gotcha below is what such a buffer does to a log. With the always policy it goes to the disk before the reply, on the one thread, and that is the 43 a second:
static bool kv_write_all(int fd, const char *p, size_t n) {
size_t off = 0;
while (off < n) {
ssize_t w = write(fd, p + off, n - off);
if (w < 0) {
if (errno == EINTR) {
continue;
}
perror("aof write");
return false;
}
off += (size_t)w;
}
return true;
}
/* PEXPIREAT key <ms since 1970>, as the absolute form of EX, PX and EXPIRE. */
static bool kv_encode_pexpireat(kv_buf *out, const char *key, size_t klen, int64_t at) {
char digits[32];
int n = snprintf(digits, sizeof digits, "%lld", (long long)at);
const char *argv[3] = {"PEXPIREAT", key, digits};
size_t arglen[3] = {9, klen, (size_t)n};
return kv_aof_encode(out, 3, argv, arglen);
}
bool kv_aof_record(kv_aof *a, const kv_cmd *c, int64_t wall_now) {
if (a->fd < 0) {
return true;
}
kv_buf out = {0};
bool ok;
long long n;
if (kv_arg_is(c, 0, "SET") && c->argc == 5 && kv_parse_int(c->argv[4], c->arglen[4], &n) &&
n > 0) {
/* SET key value EX|PX n becomes SET key value, then PEXPIREAT key <absolute> */
int64_t at = wall_now + (kv_arg_is(c, 3, "EX") ? n * 1000 : n);
ok = kv_aof_encode(&out, 3, c->argv, c->arglen) &&
kv_encode_pexpireat(&out, c->argv[1], c->arglen[1], at);
} else if (kv_arg_is(c, 0, "EXPIRE") && c->argc == 3 &&
kv_parse_int(c->argv[2], c->arglen[2], &n)) {
if (n <= 0) {
const char *argv[2] = {"DEL", c->argv[1]};
size_t arglen[2] = {3, c->arglen[1]};
ok = kv_aof_encode(&out, 2, argv, arglen);
} else {
ok = kv_encode_pexpireat(&out, c->argv[1], c->arglen[1], wall_now + n * 1000);
}
} else {
ok = kv_aof_encode(&out, c->argc, c->argv, c->arglen);
}
ok = ok && kv_write_all(a->fd, out.data, out.len);
kv_buf_free(&out);
if (!ok) {
return false;
}
a->dirty = true;
if (a->policy == KV_FSYNC_ALWAYS) {
/* always: wait for the disk here, before the reply, on the one thread */
if (!kv_fsync(a->fd)) {
perror("fsync");
}
a->dirty = false;
}
return true;
}In the input function, a write command is recorded after it runs and before its reply is
sent. So a client never sees OK for a write the file does not have:
/* The command ran; now it goes to the file, before the reply goes to the client. */
if (kv_cmd_is_write(&cmd) && !kv_aof_record(aof, &cmd, kv_wall_ms())) {
return false;
}C again: the second thread#
With everysec, the fsync happens once a second, and the question is who waits for it. On this machine one takes 23 milliseconds, and the event loop cannot stop for 23 milliseconds without every client noticing. So the fsync gets a thread of its own, the first second thread in the C version of kv. The thread tools come from POSIX, the Portable Operating System Interface. A thread is started with pthread_create from a function and one void *, and waited for with pthread_join. The two threads share a struct, and the struct says which rule guards each field:
typedef struct {
pthread_t thread;
pthread_mutex_t mu; /* guards wanted and stop */
pthread_cond_t cv; /* signalled when either changes */
bool wanted; /* an fsync has been asked for */
bool stop; /* the thread should return */
int fd;
/* when the last fsync finished, in monotonic ms; written by the thread and read by
* the loop without the lock, which is what an atomic is for */
_Atomic int64_t last_sync;
} kv_syncer;Three fields are guarded by a mutex, which one thread at a time may hold. A thread that reads or writes them without holding it has a data race. C says a data race is undefined behaviour, as chapter 6 said of an overrun. Nothing stops you forgetting the lock. The thread itself spends its life asleep on a condition variable, which is a place to wait until another thread says something has changed. The wait "atomically release[s] mutex and cause[s] the calling thread to block", and the mutex is held again when it returns:
/* The thread. It sleeps on the condition variable until the loop asks for an fsync or
* for it to stop, and it never holds the mutex while it waits for the disk. */
static void *kv_syncer_run(void *arg) {
kv_syncer *s = arg;
pthread_mutex_lock(&s->mu);
for (;;) {
/* a while, not an if: a wait may return early, and the loop may change its mind */
while (!s->wanted && !s->stop) {
pthread_cond_wait(&s->cv, &s->mu); /* releases mu while asleep, takes it back after */
}
if (s->stop) {
break;
}
s->wanted = false;
pthread_mutex_unlock(&s->mu);
if (!kv_fsync(s->fd)) {
perror("fsync");
}
/* release: every write the fsync made visible is published with the time */
atomic_store_explicit(&s->last_sync, kv_now_ms(), memory_order_release);
pthread_mutex_lock(&s->mu);
}
pthread_mutex_unlock(&s->mu);
return NULL;
}The wait is inside a while, never an if, and the manual says why. "Spurious wakeups from the pthread_cond_timedwait() or pthread_cond_wait() functions may occur", so the condition must be checked again on every return. The fourth field is an atomic. The thread writes the time the fsync finished. The loop reads it without taking the lock, to know whether the last request is still running. An atomic operation is one no other thread can see half done, and the memory order on it says what else it promises. The release on the store and the acquire on the load pair up. Everything the thread wrote before the store is visible to the loop after the load. The loop's side is three short functions:
bool kv_syncer_start(kv_syncer *s, int fd) {
*s = (kv_syncer){.fd = fd};
atomic_init(&s->last_sync, kv_now_ms());
if (pthread_mutex_init(&s->mu, NULL) != 0 || pthread_cond_init(&s->cv, NULL) != 0) {
return false;
}
/* A signal is delivered to any thread that does not block it. The stop handler must
* run on the event loop's thread, so this thread is created with every signal
* blocked, which it inherits, and the loop's mask is put back afterwards. */
sigset_t all, before;
sigfillset(&all);
pthread_sigmask(SIG_BLOCK, &all, &before);
int rc = pthread_create(&s->thread, NULL, kv_syncer_run, s);
pthread_sigmask(SIG_SETMASK, &before, NULL);
return rc == 0;
}
void kv_syncer_request(kv_syncer *s) {
pthread_mutex_lock(&s->mu);
s->wanted = true;
pthread_cond_signal(&s->cv);
pthread_mutex_unlock(&s->mu);
}
int64_t kv_syncer_last(const kv_syncer *s) {
/* acquire: pairs with the release in the thread, so the reader sees the fsync */
return atomic_load_explicit(&s->last_sync, memory_order_acquire);
}
void kv_syncer_stop(kv_syncer *s) {
pthread_mutex_lock(&s->mu);
s->stop = true;
pthread_cond_signal(&s->cv);
pthread_mutex_unlock(&s->mu);
pthread_join(s->thread, NULL);
pthread_cond_destroy(&s->cv);
pthread_mutex_destroy(&s->mu);
}The event loop has woken ten times a second since chapter 10, for the sweep, and the request joins it. Once a second, when there is something to force and the last fsync has finished, the loop asks and goes straight back to its clients:
void kv_aof_tick(kv_aof *a, int64_t now) {
if (a->fd < 0 || !a->dirty || a->policy != KV_FSYNC_EVERYSEC) {
return;
}
if (now - a->last_request < 1000) {
return; /* not yet a second since the last request */
}
if (kv_syncer_last(&a->syncer) < a->last_request) {
return; /* the last fsync is still running: a slow disk gets one at a time */
}
a->dirty = false;
a->last_request = now;
kv_syncer_request(&a->syncer); /* returns at once; the thread waits for the disk */
}One line in the start function is about signals, and it answers a question chapter 10 left open. A signal is delivered to any thread that does not block it. The stop handler must run on the loop's thread, whose wait it is meant to interrupt. So the fsync thread is created with every signal blocked, a mask it inherits from the thread that created it. The loop's own mask is put back at once. ThreadSanitizer found the need for that line, in a way. Under it, a signal that landed on the sleeping thread was never acted on, and the server did not stop.
The thread is where the macOS surprise lives. On macOS, fsync only passes the bytes to the drive. The manual says that "the drive may not physically write the data to the platters for quite some time". The call that reaches the disk is fcntl(fd, F_FULLFSYNC). The Go and Rust standard libraries both make it when you ask for a sync on macOS. So the C version makes it too, behind an #ifdef:
bool kv_fsync(int fd) {
#ifdef __APPLE__
/* On macOS fsync() only hands the bytes to the drive, which may hold them in its own
* cache. F_FULLFSYNC asks the drive to write them, and is what Go and Rust call. */
return fcntl(fd, F_FULLFSYNC) >= 0;
#else
return fsync(fd) >= 0;
#endif
}Two threads sharing a struct is where the book's C tooling gains a sanitizer. ThreadSanitizer watches every access from every thread and reports two that touch the same memory without a lock or an atomic between them. It cannot run together with AddressSanitizer, so the build has a target of its own. make verify-c-tsan rebuilds every snapshot that calls pthread_create and runs the tests again:
$ make verify-c-tsan
## C (TSan) code/c/ch11
== c ch11: 207/207 passedThread safety of the C library#
POSIX requires every function in the standard to be thread-safe "except for the following functions". The list is short but has old friends on it. strtok keeps its place in a static. localtime and strerror return pointers to one shared buffer. getenv and rand share state. Each has a safe form with _r on the end, and the syncer uses perror, which is on the safe side. C11 standardised threads in <threads.h>, with thrd_create and mtx_t. But the header is optional and macOS does not provide it, so kv uses the POSIX names that every Unix has.
Write kv_aof_load. Open the file read-only, and return 0 when it does not
exist, because a missing file is an empty store. Read it in chunks into a kv_buf. After each chunk, run kv_parse
in a loop. Execute each complete command with kv_exec into a reply buffer that
nobody reads. Count the
commands. Bytes left over at the end are a truncated last command. Report them and drop
them.
long kv_aof_load(const char *path, kv_store *store) {
int fd = open(path, O_RDONLY);
if (fd < 0) {
return errno == ENOENT ? 0 : -1;
}
/* your code here */
}Stuck?Show the full listing
long kv_aof_load(const char *path, kv_store *store) {
int fd = open(path, O_RDONLY);
if (fd < 0) {
return errno == ENOENT ? 0 : -1; /* no file yet is an empty store, not an error */
}
kv_buf in = {0};
kv_buf out = {0}; /* the replies, which nobody reads */
char tmp[65536];
long applied = 0;
for (;;) {
ssize_t n = read(fd, tmp, sizeof tmp);
if (n < 0 && errno == EINTR) {
continue;
}
if (n <= 0) {
break;
}
if (!kv_buf_append(&in, tmp, (size_t)n)) {
applied = -1;
break;
}
/* the same parser and the same executor as a live client */
for (;;) {
kv_cmd cmd;
size_t used = 0;
kv_parse_status st = kv_parse(in.data, in.len, &cmd, &used);
if (st != KV_PARSE_OK) {
break;
}
kv_exec(&cmd, store, &out);
out.len = 0;
kv_buf_consume(&in, used);
applied++;
}
}
if (in.len > 0) {
fprintf(stderr, "aof: dropped %zu trailing bytes of a truncated command\n", in.len);
}
kv_buf_free(&in);
kv_buf_free(&out);
close(fd);
return applied;
}The absolute deadline needs a command of its own, PEXPIREAT, which turns a
wall-clock time back into a monotonic one for the store. A deadline already in the past
deletes the key, which is how an expired key stays expired across a restart:
/* PEXPIREAT key <ms since 1970>: the absolute form that the append-only file uses. The
* wall-clock deadline is turned into a monotonic one for the store. */
static bool kv_cmd_pexpireat(const kv_cmd *c, kv_store *s, kv_buf *out, int64_t now) {
long long at;
if (!kv_parse_int(c->argv[2], c->arglen[2], &at)) {
return kv_reply_error(out, "ERR value is not an integer or out of range");
}
kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1], now);
if (e == NULL) {
return kv_reply_int(out, 0);
}
int64_t left = at - kv_wall_ms();
if (left <= 0) {
kv_store_del(s, c->argv[1], c->arglen[1]);
} else {
e->expires_at = now + left;
}
return kv_reply_int(out, 1);
}Startup replays the file before it serves the first client, so that no client sees an empty store, and then opens the file for appending. At the other end, the clean stop from chapter 10 joins the thread, forces the last bytes to disk and closes the file:
/* KV_AOF names the file. Replay it first, then open it for appending. */
kv_aof aof = {.fd = -1, .policy = KV_FSYNC_EVERYSEC};
const char *path = getenv("KV_AOF");
if (path != NULL && *path != '\0') {
kv_fsync_policy policy;
if (!kv_fsync_policy_parse(getenv("KV_FSYNC"), &policy)) {
fprintf(stderr, "KV_FSYNC must be always, everysec or no\n");
return 1;
}
long applied = kv_aof_load(path, store);
if (applied < 0) {
perror(path);
return 1;
}
fprintf(stderr, "kv replayed %ld commands from %s\n", applied, path);
if (!kv_aof_open(&aof, path, policy)) {
return 1;
}
}After write returns, the bytes are in the kernel, not on the
disk. If the process is killed, the kernel still has them and writes them out, which is why the scene lost nothing.
If the machine loses power, they are gone. fsync makes the kernel write them now and wait. The Linux manual adds that
even then the directory entry for a new file may not have reached the disk. On macOS, fsync itself is not enough, and the section above says what is. Most databases have a paragraph like this one in their source.
Redis takes its snapshots with fork, which "creates a new process by duplicating the calling process". The child writes the whole dataset to a file while the parent goes on serving. The kernel copies a page only when one of them writes to it. The child "is created with a single thread", the one that called fork. That is why a program with threads must be careful what the child does. Windows has no fork. After completion ports, that is the second reason why Redis on Windows was always a port and never the same program. The server kv uses a thread instead of a process, which is the choice Go and Rust make everywhere.
Go: the same reader, backwards#
The Go log is a struct with a lock, because many goroutines record into it. The policy is parsed from the environment into one of three constants:
// Policy says how often the file is forced to disk. Redis has the same three settings.
type Policy int
const (
Always Policy = iota // after every command: slow and safe
EverySec // once a second: lose at most a second of writes
No // never: the operating system decides
)
func ParsePolicy(s string) (Policy, error) {
switch strings.ToLower(s) {
case "", "everysec":
return EverySec, nil
case "always":
return Always, nil
case "no":
return No, nil
}
return 0, fmt.Errorf("KV_FSYNC must be always, everysec or no, not %q", s)
}
// Log is the open file. Many goroutines record into it, so it has a lock of its own.
type Log struct {
mu sync.Mutex
f *os.File
w *bufio.Writer
policy Policy
dirty bool
syncs chan struct{} // a request to the syncer goroutine; room for one, so a second waits for nobody
done sync.WaitGroup
}Recording uses the WriteArrayLen and WriteBulk functions from
the protocol package, which were written to answer clients and serve for log records too. The buffer is flushed after every record. The book's own test found the bug
that this line fixes, and the gotcha below tells the story:
// Record appends one command. A relative expiry is rewritten as an absolute PEXPIREAT,
// because the file may be replayed much later. The bytes reach the kernel before this
// returns: a policy delays only the fsync, never the write, so a crash of the process
// loses nothing and only a crash of the machine can lose the last second.
func (l *Log) Record(args [][]byte, now time.Time) error {
l.mu.Lock()
defer l.mu.Unlock()
for _, cmd := range rewrite(args, now) {
if err := resp.WriteArrayLen(l.w, len(cmd)); err != nil {
return err
}
for _, a := range cmd {
if err := resp.WriteBulk(l.w, a); err != nil {
return err
}
}
}
if err := l.w.Flush(); err != nil {
return err
}
l.dirty = true
if l.policy == Always {
return l.syncLocked()
}
return nil
}
// rewrite turns one command into the commands the file stores.
func rewrite(args [][]byte, now time.Time) [][][]byte {
name := strings.ToUpper(string(args[0]))
switch {
case name == "SET" && len(args) == 5:
n, err := strconv.ParseInt(string(args[4]), 10, 64)
if err != nil || n <= 0 {
break
}
d := time.Duration(n) * time.Millisecond
if strings.ToUpper(string(args[3])) == "EX" {
d = time.Duration(n) * time.Second
}
return [][][]byte{args[:3], pexpireat(args[1], now.Add(d))}
case name == "EXPIRE" && len(args) == 3:
n, err := strconv.ParseInt(string(args[2]), 10, 64)
if err != nil {
break
}
if n <= 0 {
return [][][]byte{{[]byte("DEL"), args[1]}}
}
return [][][]byte{pexpireat(args[1], now.Add(time.Duration(n)*time.Second))}
}
return [][][]byte{args}
}
func pexpireat(key []byte, at time.Time) [][]byte {
return [][]byte{[]byte("PEXPIREAT"), key, []byte(strconv.FormatInt(at.UnixMilli(), 10))}
}Go again: a goroutine that waits for the disk#
The Go version of the fsync thread is a goroutine and a channel. It is shorter than the C version by the whole of the mutex and the condition variable. The channel has room for one request. A sender that finds it full is a request nobody needs, because one is already waiting. So the select with a default drops it without blocking. The goroutine ranges over the channel and ends when Close closes it, and the WaitGroup is how Close waits for that:
// syncer runs on a goroutine of its own and calls fsync whenever it is asked, so that no
// client ever waits for the disk while holding the log's lock. The range ends when Close
// closes the channel, and the WaitGroup lets Close wait for that.
func (l *Log) syncer() {
for range l.syncs {
if err := l.f.Sync(); err != nil {
fmt.Fprintln(os.Stderr, "aof:", err)
}
}
}
// Tick is called once a second. With everysec it flushes the buffer to the kernel and
// asks the syncer for an fsync, without waiting for it. The select with a default never
// blocks: if a request is already waiting in the channel, this one is not needed.
func (l *Log) Tick() {
l.mu.Lock()
defer l.mu.Unlock()
if !l.dirty || l.policy != EverySec {
return
}
if err := l.w.Flush(); err != nil {
fmt.Fprintln(os.Stderr, "aof:", err)
return
}
l.dirty = false
select {
case l.syncs <- struct{}{}:
default:
}
}The file is the one shared thing. The os package says that "the methods of File are safe for concurrent use". So a Sync on the goroutine and a Write under the lock may overlap. Go's Sync on macOS makes the same F_FULLFSYNC call as the C version, which is why the always policy measures the same 44 a second. The ticker goroutine from chapter 10 calls Tick once a second and returns when the context is cancelled.
Write Replay. Open the file, and return 0 and no error when it does not
exist. Wrap it in a bufio.Reader and call resp.ReadCommand in a
loop, handing each command to apply. Stop at io.EOF. Treat
io.ErrUnexpectedEOF as a truncated last command, print a line about it, and
return what you have.
func Replay(path string, apply func(args [][]byte)) (int, error) {
f, err := os.Open(path)
// your code here
}Stuck?Show the full listing
// Replay reads the file with the same parser a client goes through and hands each
// command to apply. A truncated last command is dropped, as Redis drops it. A missing
// file is an empty store, not an error.
func Replay(path string, apply func(args [][]byte)) (int, error) {
f, err := os.Open(path)
if errors.Is(err, os.ErrNotExist) {
return 0, nil
}
if err != nil {
return 0, err
}
defer f.Close()
r := bufio.NewReaderSize(f, 1<<16)
n := 0
for {
args, err := resp.ReadCommand(r)
if errors.Is(err, io.EOF) {
return n, nil
}
if errors.Is(err, io.ErrUnexpectedEOF) {
fmt.Fprintln(os.Stderr, "aof: dropped a truncated command at the end")
return n, nil
}
if err != nil {
return n, err
}
apply(args)
n++
}
}The command function takes the log as a parameter, and records a write before running
it. During replay the log is nil, so nothing is recorded twice. The replies
during replay go to a writer over io.Discard. Because the log and the store have different locks, two goroutines can record in one order and run in the other. A replay can then end with a different value for a key that both wrote. The Rust version below closes that gap with one lock:
// Execute runs one command against the store and writes its reply. A write command is
// also recorded in the log, when there is one. During replay the log is nil.
func Execute(args [][]byte, s Storer, log *aof.Log, w *bufio.Writer) error {
now := time.Now()
if len(args) == 0 {
return resp.WriteError(w, "ERR empty command")
}
name := strings.ToUpper(string(args[0]))
cmd, ok := commands[name]
if !ok {
shown := string(args[0])
if len(shown) > 40 {
shown = shown[:40]
}
return resp.WriteError(w, "ERR unknown command '"+shown+"'")
}
if n := len(args); (cmd.arity < 0 && n < -cmd.arity) || (cmd.arity > 0 && n != cmd.arity) {
return arity(w, strings.ToLower(name))
}
if log != nil && aof.IsWrite(name) {
if err := log.Record(args, now); err != nil {
return err
}
}
return cmd.run(args, s, w, now)
}// KV_AOF names the file. Replay it first, with no log, then open it for appending.
var log *aof.Log
if path := os.Getenv("KV_AOF"); path != "" {
policy, err := aof.ParsePolicy(os.Getenv("KV_FSYNC"))
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
sink := bufio.NewWriter(io.Discard)
n, err := aof.Replay(path, func(args [][]byte) { cmd.Execute(args, s, nil, sink) })
if err != nil {
fmt.Fprintln(os.Stderr, "aof:", err)
os.Exit(1)
}
fmt.Fprintf(os.Stderr, "kv replayed %d commands from %s\n", n, path)
if log, err = aof.Open(path, policy); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
go func() {
tick := time.NewTicker(time.Second)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
log.Tick()
}
}
}()
}The first Go version lost every write on restart, and the test caught it.
The log used a bufio.Writer, as every Go file writer does, and flushed it once
a second from the ticker. The test stops the server with a signal, which runs no Go code. So the bytes still in the
buffer never reached the kernel. The C version had no
such bug, because it has no buffer between the record and write. The fix is the
Flush after every record. A policy decides when to fsync. It must
never decide when to write.
Rust: the store and the log under one lock#
The Rust log is a struct around a BufWriter<File>, and it has no lock
of its own. Instead the store and the log live in one struct, Db, and the one
lock from chapter 9 now guards both. A command and its record happen together,
and no other command can come between them:
/// Everything a command may touch. One Mutex around this struct means a command and its
/// record in the log happen together, with no other command in between.
pub struct Db {
pub store: MapStore,
pub aof: Option<Aof>,
}Write Db::execute. If there is a log and the command is a write, record it
with the current wall-clock time, then run it with cmd::execute. Record first, so that a command the file holds is one the server accepted. Use a let chain, which the 2024 edition allows, one if with two conditions joined by &&.
pub fn execute(&mut self, args: &Command, w: &mut impl Write) -> io::Result<()> {
// your code here
}Stuck?Show the full listing
impl Db {
/// Record a write command, then run it. Recording comes first, so that a command
/// the file holds is one that was accepted, and a crash between the two loses a
/// reply and never a write.
pub fn execute(&mut self, args: &Command, w: &mut impl Write) -> io::Result<()> {
if let Some(log) = self.aof.as_mut()
&& args.first().is_some_and(|name| aof::is_write(name))
{
log.record(args, aof::wall_ms())?;
}
crate::cmd::execute(args, &mut self.store, w)
}
}Recording mirrors the C version. The flush after each record is the same fix as in Go. The first Rust version had the same bug, and the same test found it:
/// Append one command. A relative expiry is rewritten as an absolute PEXPIREAT,
/// because the file may be replayed much later. The bytes reach the kernel before this
/// returns: a policy delays only the fsync, never the write, so a crash of the process
/// loses nothing and only a crash of the machine can lose the last second.
pub fn record(&mut self, args: &Command<'_>, now_ms: i64) -> io::Result<()> {
for cmd in rewrite(args, now_ms) {
resp::write_array_len(&mut self.file, cmd.len())?;
for a in &cmd {
resp::write_bulk(&mut self.file, a)?;
}
}
self.file.flush()?;
self.dirty = true;
if self.policy == Policy::Always {
self.sync()?;
}
Ok(())
}
/// Flush the buffer and force the file to disk, on the calling thread. The always
/// policy does this after every record, and main does it once, last of all.
pub fn sync(&mut self) -> io::Result<()> {
self.file.flush()?;
if mem::replace(&mut self.dirty, false) && self.policy != Policy::No {
self.file.get_ref().sync_all()?;
}
Ok(())
}Rust again: a scoped thread on a channel#
The Rust fsync thread needs the file and nothing else, so the log gives it a second handle. The standard library says a cloned File "shares the same underlying file handle", and an fsync through either forces what was written through both. The other method is the once-a-second question, and it uses mem::replace, which puts a new value in a place and returns the old one. Its cousin mem::take puts the default there instead, and mem::swap exchanges two places. All three exist because Rust will not let you move a value out of a &mut and leave a hole:
impl Aof {
/// A second handle to the same file, for the thread that waits for the disk. An
/// fsync through either handle forces everything written through both.
pub fn handle(&self) -> io::Result<File> {
self.file.get_ref().try_clone()
}
/// Once a second, with everysec: flush the buffer to the kernel and say whether an
/// fsync is owed. `mem::replace` reads the flag and clears it in one expression.
pub fn owes_sync(&mut self) -> io::Result<bool> {
self.file.flush()?;
Ok(self.policy == Policy::EverySec && mem::replace(&mut self.dirty, false))
}
}The thread lives in the scope from chapter 10, beside the sweeper and the clients. It owns the handle and the receiving end of an mpsc channel, and it does one thing for each message. The sweeper owns the sending end. When the sweeper returns at shutdown, the sender is dropped and the channel reports that it is disconnected. The for loop ends, and the scope joins the thread. No flag and no stop message:
// The thread that waits for the disk. It receives a request on the channel,
// calls fsync through its own handle, and never touches the lock. The loop ends
// when every sender is dropped, which happens when the sweeper returns.
let (sync_tx, sync_rx) = mpsc::channel::<()>();
if let Some(file) = sync_file {
s.spawn(move || {
for () in sync_rx {
if let Err(e) = file.sync_all() {
eprintln!("aof: {e}");
}
}
});
}// every tenth tick, once a second, the everysec policy flushes the log and
// asks the syncer for an fsync, without waiting for it
if tick.is_multiple_of(10) {
match db.aof.as_mut().map(Aof::owes_sync) {
Some(Ok(true)) => {
let _ = sync_tx.send(());
}
Some(Err(e)) => eprintln!("aof: {e}"),
_ => {}
}
}The tokio version cannot spawn a thread that blocks, and fsync blocks. This is the case chapter 9 promised spawn_blocking for. The call runs on a thread tokio keeps for exactly that, and the sweeper task goes on at once. The handle is in an Arc, because each blocking call needs its own owner:
// fsync blocks, and a task must not. spawn_blocking runs the call on
// a thread kept for exactly that, and the task goes on at once.
if owed && let Some(file) = sync_file.clone() {
tokio::task::spawn_blocking(move || {
if let Err(e) = file.sync_all() {
eprintln!("aof: {e}");
}
});
}Replay reads the file in chunks into the same kind of buffer the server reads a socket into. The same parse takes whole commands out of it. The parser does not know whether it is
reading a socket or a file, and the replies go into io::sink():
/// Replay the file into the store, through the same parser a client goes through, from
/// the same kind of buffer. A truncated last command is dropped, as Redis drops it. A
/// missing file is an empty store.
pub fn replay(path: &Path, store: &mut MapStore) -> io::Result<usize> {
let mut file = match File::open(path) {
Ok(f) => f,
Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(0),
Err(e) => return Err(e),
};
let mut sink = io::sink();
let mut buf: Vec<u8> = Vec::new();
let mut chunk = vec![0u8; 1 << 16];
let mut n = 0;
loop {
let got = file.read(&mut chunk)?;
if got == 0 {
if !buf.is_empty() {
eprintln!("aof: dropped a truncated command at the end");
}
return Ok(n);
}
buf.extend_from_slice(&chunk[..got]);
let mut used = 0;
while let Parsed::Done(args, len) = resp::parse(&buf[used..])? {
crate::cmd::execute(&args, store, &mut sink)?;
n += 1;
used += len;
}
buf.drain(..used);
}
}The log has a Drop that syncs the file, and a signal never calls
it. Drop runs when a value goes out of scope in a program that is
still running. A process ended by kill -9, by the test runner, or by a power cut
runs no more Rust code. So the Drop is a courtesy for a clean exit. The clean stop from chapter 10 does the same sync on purpose, last of all. The flush after every record is
what the durability of the log rests on, exactly as in Go.
Rust's ownership rules say nothing about this. The operating system does.
One thread, three ways#
The same thread was written three times in this chapter, and the three versions are a fair picture of the three languages with threads. The C version is a struct, a mutex, a condition variable and an atomic. A comment on each field says which rule guards it, and a sanitizer checks the comments. The Go version is a channel, and the channel is both the message and the lock. The Rust version is a channel too. The compiler checked that the handle could cross to the thread, and that the thread could not outlive the scope.
| The fsync thread | C | Go | Rust |
|---|---|---|---|
| Starting it | pthread_create with a function and a void * | wg.Go | s.spawn inside the scope |
| Waiting for work | pthread_cond_wait in a while | for range ch | for () in rx |
| Asking for work | lock, flag, signal, unlock | a send, dropped when one is waiting | a send |
| Telling it to stop | a second flag under the same lock | close(ch) | drop the sender |
| Waiting for it to end | pthread_join | wg.Wait | the end of the scope |
| A forgotten lock | undefined behaviour, found by TSan if the test provokes it | a data race, found by the detector if the test provokes it | does not compile |
What a crash takes#
Three kinds of failure, and three policies, make a small table that is worth learning. Every version of kv behaves the same way, because all three call the same kernel.
| After a crash of | C | Go | Rust |
|---|---|---|---|
| The process, with any policy | nothing lost, the kernel has every record | the same | the same |
The machine, policy always | nothing lost | the same | the same |
The machine, policy everysec | up to one second of writes lost | the same | the same |
The machine, policy no | whatever the kernel had not written, usually seconds | the same | the same |
| A truncated last record | dropped at replay | the same | the same |
The price of always is the scene's number, and it is why Redis defaults to
everysec. On the author's machine, with the disk really flushed, the C server answered 43 SET commands a second with one client. With fifty clients it answered 44, because the disk is the whole cost. With everysec it answered 14,700 from one client and 47,700 from fifty. The no policy was no faster, because one fsync a second on a thread of its own costs the clients nothing. The scene's file held 59,694 commands in 2,686,230 bytes, 45 bytes a command, which is the cost of
a format a person can read. And a kill -9 never cut a record in half, because each record is one write call and the kernel either has it or does not. Only the machine's death can leave a truncated last record, and all three versions drop it at replay, as Redis does.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| Writing the file | OK | Strong | Strong | C encodes into a buffer and calls write. Go and Rust reuse the protocol writers and flush. |
| Replay | Strong | Strong | Strong | All three reuse the parser from chapter 4 unchanged, which is the point of a log in RESP. |
| A crash of the process | Strong | OK | OK | C writes straight through. Go and Rust needed a flush the book's test had to find. |
| The clock on disk | OK | Strong | OK | C converts between two clocks itself, while Go's time.Time holds both and Rust converts with a helper. |
| Log and store together | Strong | OK | Strong | One C thread needs no lock, Go takes two locks, and Rust takes one around both. |
| The fsync thread | Weak | Strong | Strong | C needs a mutex, a condition variable, an atomic and a sanitizer. Go needs a channel, and Rust a channel the compiler checks. |
| Reaching the disk on macOS | OK | Strong | Strong | C must know about F_FULLFSYNC itself. Both standard libraries know. |
- write and fsync. write passes bytes to the kernel, which survives a dead process. fsync waits for the disk, which survives a dead machine. On this disk that is 43 a second.
- Condition variable. A place a thread sleeps until another signals it, always inside a while, because a wait may return for no reason.
- A thread's channel. In Go and Rust the channel is the message and the lock. Closing it, or dropping the last sender, is how the thread learns to stop.
Testing and fuzzing#
Take the chapter 4 parser and remove the one check that compares a length field with the room it has. Give the result to a fuzzer, with a single valid request as its only hint:
$ make fuzz-trusting
INFO: seed corpus: files: 1 min: 22b max: 22b total: 22b rss: 35Mb
==15==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5030005a29d0
WRITE of size 41 at 0x5030005a29d0 thread T0
#1 0x556e366fc450 in trusting_parse /w/bugs/trusting_parser.c:77:9
0x5030005a29d0 is located 0 bytes after 32-byte region
elapsed 2 s
$ od -c crash-8c6f20f4728eeea01d638af72038b7d1c045481f | head -1
0000000 * 2 \r \n $ 4 1 \r \n E \0 \0 \0 \0 ( \0Two seconds. The fuzzer turned $4 into $41, padded the
argument to match, and wrote nine bytes past a thirty-two byte buffer. No unit test in this
book thought of that input, and no person would have.
Every chapter so far was checked by one script that talks to kv through
redis-cli. This chapter adds the tests that live beside the code, in each
language's own runner, and the tools that find what tests cannot. A fuzzer throws millions
of inputs at the parser. A sanitizer or a race detector watches the program run. The chapter
is about what each language can prove, and when.
The protocol tests in tests/protocol/ are the same for all three servers,
and that is their value. They do not say which function is wrong, and they only send what a
person thought to write. The parser from chapter 4 has five ways to fail on bad input, and the script tries one. A unit test can try all five, and a fuzzer can try five million
things nobody thought of.
| The tools | C | Go | Rust |
|---|---|---|---|
| A unit test | a main you write, with a CHECK macro | a TestXxx function in a _test.go file | a #[test] function in a mod tests |
| Running them | make test, a target you write | go test ./... | cargo test |
| A test that lives in the docs | no | an Example function, whose output is checked | a code block in a doc comment, compiled and run |
| A test from outside | no | a _test package | the tests/ directory, a crate of its own |
| A fuzzer | libFuzzer, with a clang that has it | go test -fuzz, built in | cargo-fuzz, on the nightly toolchain |
| Memory errors | AddressSanitizer, or Valgrind on Linux | not possible in safe code | not possible in safe code |
| Data races | ThreadSanitizer | go test -race | the compiler, at compile time |
| Undefined behaviour | UndefinedBehaviorSanitizer | defined, except under a data race | Miri, for unsafe code |
| What the book runs | the tests under ASan and UBSan, and TSan for the threads | go vet, go test -race | cargo test, clippy -D warnings |
C has no test runner in its standard library, and no way to find a bad pointer at run time. Both gaps were filled from outside. Valgrind, released by Julian Seward in 2002, runs a program on a simulated processor and watches every memory access. The program runs twenty to thirty times slower. AddressSanitizer, from Google in 2012, puts the checks into the compiled code instead, at about twice the cost. clang and gcc both carry it. Go and Rust were designed after both tools existed and built the answers in. Go's runtime can watch for races and its test tool can fuzz. Rust's compiler refuses the programs that the sanitizer would have caught. The C tools remain the reason that C is still written at all.
C: a runner you write, and a sanitizer that watches#
A C test is a program. The runner for kv is thirty lines. A counter, and a macro that prints the file and the
line of a failed check and carries on. One assertion is checked by the compiler instead of the test. _Static_assert takes a constant expression, and if it is false "a compile-time error occurs and the compiler must display message". A test suite that assumes a limit should fail to build when the limit changes, not fail at run time:
static int failed = 0;
static int passed = 0;
/* Report a failure with its location, and carry on to the next check. */
#define CHECK(cond) \
do { \
if (cond) { \
passed++; \
} else { \
failed++; \
fprintf(stderr, "%s:%d: check failed: %s\n", __FILE__, __LINE__, #cond); \
} \
} while (0)
/* The tests assume a request may carry at least four arguments. The compiler checks it,
* so a smaller limit fails to build instead of failing at run time. */
_Static_assert(KV_MAX_ARGS >= 4, "the tests use requests of up to four arguments");The tests for the parser cover the three answers from chapter 4, and the inline form. The one that matters most is the second. It cuts a valid request at every byte and checks that the parser says "incomplete" and never "bad". The event loop depends on exactly that:
Write test_parse_incomplete_at_every_cut. Take a valid request, and for
every length from 0 to one less than the whole, call kv_parse on that prefix.
Each call must return KV_PARSE_INCOMPLETE. A single KV_PARSE_BAD
would make the server drop a client in the middle of a slow request.
static void test_parse_incomplete_at_every_cut(void) {
const char *req = "*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n";
/* your loop here */
}Stuck?Show the full listing
static void test_parse_complete(void) {
const char *req = "*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n";
kv_cmd cmd;
size_t used = 0;
CHECK(kv_parse(req, strlen(req), &cmd, &used) == KV_PARSE_OK);
CHECK(used == strlen(req));
CHECK(cmd.argc == 2);
CHECK(cmd.arglen[0] == 4 && memcmp(cmd.argv[0], "ECHO", 4) == 0);
CHECK(cmd.arglen[1] == 2 && memcmp(cmd.argv[1], "hi", 2) == 0);
}
static void test_parse_incomplete_at_every_cut(void) {
/* every prefix of a valid request is incomplete, never bad */
const char *req = "*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n";
for (size_t n = 0; n < strlen(req); n++) {
kv_cmd cmd;
size_t used = 0;
CHECK(kv_parse(req, n, &cmd, &used) == KV_PARSE_INCOMPLETE);
}
}
static void test_parse_bad(void) {
const char *bad[] = {"*x\r\n", "*1\r\n#4\r\nECHO\r\n", "*1\r\n$4\r\nECHOXX",
"*1\r\n$99999999\r\n", "*100\r\n"};
for (size_t i = 0; i < sizeof bad / sizeof bad[0]; i++) {
kv_cmd cmd;
size_t used = 0;
CHECK(kv_parse(bad[i], strlen(bad[i]), &cmd, &used) == KV_PARSE_BAD);
}
}
static void test_parse_inline(void) {
const char *req = " SET a b\r\n";
kv_cmd cmd;
size_t used = 0;
CHECK(kv_parse(req, strlen(req), &cmd, &used) == KV_PARSE_OK);
CHECK(cmd.argc == 3);
CHECK(cmd.arglen[2] == 1 && cmd.argv[2][0] == 'b');
CHECK(used == strlen(req));
}The list of tests is written once, as an X-macro, the trick from chapter 5's command table. Each use of the list gives X a new meaning. Here it makes a table of names and function pointers, and main walks the table. A name on the command line runs one test, and a new test is one line:
/* The list of tests, written once. Each use of the list defines X to mean something
* different: a declaration, an entry in a table, or a call. Adding a test is one line
* here, and nothing else to keep in step. */
#define KV_TESTS \
X(parse_complete) \
X(parse_incomplete_at_every_cut) \
X(parse_bad) \
X(parse_inline) \
X(parse_int) \
X(glob)
static const struct {
const char *name;
void (*run)(void);
} tests[] = {
#define X(name) {#name, test_##name},
KV_TESTS
#undef X
};
/* Run every test, or only those whose name contains the argument: ./test_resp parse */
int main(int argc, char **argv) {
const char *only = argc > 1 ? argv[1] : "";
int ran = 0;
for (size_t i = 0; i < sizeof tests / sizeof tests[0]; i++) {
if (strstr(tests[i].name, only) == NULL) {
continue;
}
int before = failed;
tests[i].run();
printf("%s %s\n", failed == before ? "ok " : "FAIL", tests[i].name);
ran++;
}
printf("%d tests, %d checks passed, %d failed\n", ran, passed, failed);
return failed == 0 ? 0 : 1;
}The Makefile gains a test target. It compiles the test file with every source file except main.c, all under AddressSanitizer and UndefinedBehaviorSanitizer, and runs the result. The sanitizers are the point. A test that
passes under them has also proved that it read no freed memory and overflowed no integer
on the way:
# The unit tests: every source under src/ except main.c, plus the test file, compiled
# together under the sanitizers. make test fails when a check fails.
test: build/test_resp
./build/test_resp
build/test_resp: tests/test_resp.c $(SRC) | build
$(CC) $(CFLAGS) -fsanitize=address,undefined -o $@ tests/test_resp.c $(filter-out src/main.c,$(SRC))$ make test
ok parse_complete
ok parse_incomplete_at_every_cut
ok parse_bad
ok parse_inline
ok parse_int
ok glob
6 tests, 53 checks passed, 0 failed
$ ./build/test_resp glob
ok glob
1 tests, 7 checks passed, 0 failedTo see what the sanitizer does when there is something to find, the snapshot includes the bug from the chapter 4 gotcha, on purpose. It keeps a pointer into a buffer, grows the buffer, and reads through the pointer:
int main(void) {
kv_buf b = {0};
kv_buf_append(&b, "GET key\r\n", 9);
const char *key = b.data + 4; /* points at "key", inside the buffer */
char filler[2048];
memset(filler, 'x', sizeof filler);
kv_buf_append(&b, filler, sizeof filler); /* the buffer grows, and may move */
printf("%.3s\n", key); /* reads through the old pointer */
kv_buf_free(&b);
return 0;
}$ make bugs
==32154==ERROR: AddressSanitizer: heap-use-after-free on address 0x619000000584
READ of size 1 at 0x619000000584 thread T0
#2 0x00010cec1930 in main use_after_free.c:16
0x619000000584 is located 4 bytes inside of 1024-byte region
freed by thread T0 here:
#0 0x00010d3d9a47 in realloc+0x87
#1 0x00010cec1aa1 in kv_buf_append buf.c:27
#2 0x00010cec191e in main use_after_free.c:15The report names the line that read the freed memory, the line that freed it, and the
realloc inside kv_buf_append that did the freeing. Without the sanitizer the program prints key most of the time, because the
old bytes are usually still there. That is the whole problem with undefined behaviour. On Linux,
Valgrind gives the same report without recompiling, and also lists every block that was
never freed. The book ran the test binary under it in a container:
$ valgrind --leak-check=full ./test_resp
53 checks passed, 0 failed
==215== All heap blocks were freed -- no leaks are possible
==215== ERROR SUMMARY: 0 errors from 0 contextsA debugger session#
The third way to see the same bug is to watch it happen. A debugger stops the program at a line and shows what the variables hold. Build the bug without optimisation and with debug information, and give gdb a script of commands. The session below stops before the buffer grows, prints the buffer's address and the pointer into it, steps one line, and prints them again:
$ cc -O0 -g -o use_after_free bugs/use_after_free.c src/buf.c
$ gdb -q -batch -x session.gdb ./use_after_free
Breakpoint 1, main () at bugs/use_after_free.c:15
15 kv_buf_append(&b, filler, sizeof filler); /* the buffer grows, and may move */
$1 = 0x7ffff7f55020 "GET key\r\n"
$2 = 0x7ffff7f55024 "key\r\n"
16 printf("%.3s\n", key); /* reads through the old pointer */
$3 = 0x7ffff7f51030 "GET key\r\n", 'x' <repeats 191 times>...
$4 = 0x7ffff7f55024 <error: Cannot access memory at address 0x7ffff7f55024>
$5 = 4096One next, and the buffer's data moved from one address to another while key kept the old one. The debugger cannot even read it now. On macOS the debugger is lldb, with the same commands in a different dialect. The book recorded this session in a Linux container, because gdb is what the container had. A session like this takes a minute, and it is the right tool when a sanitizer says what happened and you want to see why.
The same snapshot holds a second bug on purpose, for the sanitizer chapter 11 introduced. Two threads add to one counter with no lock, which is the simplest data race there is:
static long counter = 0;
static void *add(void *arg) {
(void)arg;
for (int i = 0; i < 100000; i++) {
counter++; /* a read, an add and a write, and another thread between them */
}
return NULL;
}
int main(void) {
pthread_t a, b;
pthread_create(&a, NULL, add, NULL);
pthread_create(&b, NULL, add, NULL);
pthread_join(a, NULL);
pthread_join(b, NULL);
printf("%ld\n", counter); /* 200000 if nothing went wrong, which something did */
return 0;
}$ make race
WARNING: ThreadSanitizer: data race (pid=28355)
Write of size 8 at 0x00010c76e000 by thread T2:
#0 add race.c:12
Previous write of size 8 at 0x00010c76e000 by thread T1:
#0 add race.c:12
Location is global 'counter' at 0x00010c76e000
SUMMARY: ThreadSanitizer: data race race.c:12 in add
200000The count came out right, at 200,000, and the race was reported anyway. That is the difference between a detector and a test. The test checks the answer, which was correct by luck. ThreadSanitizer "is a tool that detects data races", and it saw two threads write the same eight bytes with nothing between them, whatever the answer. The clang documentation puts its cost at "about 5x-15x" in time. That is why the book runs it on its own target, make verify-c-tsan, rather than on every build.
A fuzzer is the last tool, and the scene showed what it does. The one that comes with clang is libFuzzer, "an in-process, coverage-guided, evolutionary fuzzing engine". It calls one function of yours millions of times with inputs it mutates. It watches which branches each input reaches, so that it can mutate the interesting ones further. The target for kv feeds the input through the parser and the executor, as the server would:
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
kv_store store = {0};
kv_buf out = {0};
const char *p = (const char *)data;
size_t len = size;
/* run every complete command in the input, as the server would */
for (;;) {
kv_cmd cmd;
size_t used = 0;
if (kv_parse(p, len, &cmd, &used) != KV_PARSE_OK) {
break;
}
kv_exec(&cmd, &store, &out);
p += used;
len -= used;
}
kv_buf_free(&out);
kv_store_free(&store);
return 0;
}The scene's target is the same shape, pointed at the parser that trusts its length field. The parser is the chapter 4 one with a single line missing, and the line is the one the chapter 4 gotcha was about:
/* One request, with the arguments copied out. The bulk length is trusted: nothing
* checks it against the room in the argument. A length of 40 writes 40 bytes into 32,
* and the attacker chooses the 8 that land on whatever the heap keeps next. */
int trusting_parse(const char *data, size_t len, trusting_cmd *cmd) {
const char *end = data + len;
const char *p = data;
const char *eol = find_eol(p, end);
if (eol == NULL || *p != '*') {
return -1;
}
long n;
if (!parse_number(p + 1, eol, &n) || n > TRUSTING_MAX_ARGS) {
return -1;
}
p = eol + 1;
for (long i = 0; i < n; i++) {
if (p >= end || *p != '$' || (eol = find_eol(p, end)) == NULL) {
return -1;
}
long blen;
if (!parse_number(p + 1, eol, &blen)) {
return -1;
}
p = eol + 1;
if (end - p < blen + 2) {
return -1;
}
memcpy(cmd->argv[i], p, (size_t)blen); /* the missing check was here */
cmd->arglen[i] = (size_t)blen;
p += blen + 2;
}
cmd->argc = (size_t)n;
return 0;
}Two things made the two seconds possible. Each argument's buffer is a heap allocation of its own, so AddressSanitizer sees the first byte past it. And the fuzzer started from one valid request, so it did not have to discover what a request looks like. Without the seed, and with the arguments inside one struct, the same fuzzer ran twenty-two million inputs in a minute and found nothing. A fuzzer is only as good as the signal it gets, and a seed corpus and a sanitizer are most of that signal.
The clang that comes with Xcode cannot link a fuzzer. The error is
library 'libclang_rt.fuzzer_osx.a' not found. The fuzz targets of the Makefile run in a Linux container instead, with the
one-line docker run in their comment. The clang 19 in the container links them
without complaint. Against the real kv parser the fuzzer found nothing. That is the usual result after
the unit tests have done their work, and no reason to skip it.
A C test suite without sanitizers checks the answers and nothing else. With
-fsanitize=address,undefined it also checks every memory access and every
arithmetic operation the tests perform. The cost is about twice the run time, which for a
test suite is nothing. kv's make test always builds with them, and so should
yours. The one sanitizer that cannot join them is ThreadSanitizer, which needs a build of its own.
Go: the runner is in the box, and so is the fuzzer#
Go tests live in files that end in _test.go, in the same package as the code,
and go test finds them by name. The parser test is a table, which is the Go shape for "the same check on many inputs".
Each row is a subtest, so that a failure names the case. Three calls in it are the rest of the testing package. t.Run "runs f as a subtest of t called name". t.Parallel "signals that this test is to be run in parallel with (and only with) other parallel tests". The rows share nothing, so they may. And the comparison moved into a helper that calls t.Helper, so that a failure reports the row's line and not the helper's:
func TestReadCommand(t *testing.T) {
cases := []struct {
name string
input string
want []string
err error
}{
{"array", "*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n", []string{"ECHO", "hi"}, nil},
{"inline", " SET a b\r\n", []string{"SET", "a", "b"}, nil},
{"bare LF", "PING\n", []string{"PING"}, nil},
{"empty value", "*3\r\n$3\r\nSET\r\n$1\r\nk\r\n$0\r\n\r\n", []string{"SET", "k", ""}, nil},
{"closed", "", nil, io.EOF},
{"cut short", "*2\r\n$4\r\nEC", nil, io.ErrUnexpectedEOF},
{"not a number", "*x\r\n", nil, ErrProtocol},
{"wrong marker", "*1\r\n#4\r\nECHO\r\n", nil, ErrProtocol},
{"too many", "*100\r\n", nil, ErrProtocol},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
t.Parallel() // the cases share nothing, so they may run at once
got, err := ReadCommand(bufio.NewReader(strings.NewReader(c.input)))
if !errors.Is(err, c.err) {
t.Fatalf("err = %v, want %v", err, c.err)
}
expectArgs(t, got, c.want)
})
}
}
// expectArgs compares the parsed arguments with the expected strings. t.Helper makes a
// failure report the caller's line, not this one.
func expectArgs(t *testing.T, got [][]byte, want []string) {
t.Helper()
if len(got) != len(want) {
t.Fatalf("got %q, want %q", got, want)
}
for i := range got {
if string(got[i]) != want[i] {
t.Fatalf("arg %d = %q, want %q", i, got[i], want[i])
}
}
}Write FuzzReadCommand. Add three seed inputs with f.Add. Then
call f.Fuzz with a function that parses the input and returns on an error. When
the parse succeeds, write the arguments back out with the protocol writers, parse them
again, and fail the test if anything changed. That round trip is a property the parser must
keep for every input, and it is the kind of check a fuzzer is good at.
func FuzzReadCommand(f *testing.F) {
f.Add([]byte("*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n"))
f.Fuzz(func(t *testing.T, data []byte) {
// your code here
})
}Stuck?Show the full listing
// FuzzReadCommand lets the fuzzer mutate the input. Whatever it sends, the parser must
// not panic, and a request it accepts must survive a round trip through the writers.
func FuzzReadCommand(f *testing.F) {
f.Add([]byte("*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n"))
f.Add([]byte("PING\r\n"))
f.Add([]byte("*1\r\n$3\r\nGET\r\n"))
f.Fuzz(func(t *testing.T, data []byte) {
args, err := ReadCommand(bufio.NewReader(bytes.NewReader(data)))
if err != nil {
return
}
var buf bytes.Buffer
w := bufio.NewWriter(&buf)
WriteArrayLen(w, len(args))
for _, a := range args {
WriteBulk(w, a)
}
w.Flush()
again, err := ReadCommand(bufio.NewReader(&buf))
if err != nil {
t.Fatalf("round trip failed: %v", err)
}
if len(again) != len(args) {
t.Fatalf("round trip changed the argument count")
}
for i := range args {
if !bytes.Equal(again[i], args[i]) {
t.Fatalf("round trip changed argument %d", i)
}
}
})
}$ go test ./... # the seeds run as ordinary tests
ok kv/internal/resp 0.021s
$ go test -fuzz=FuzzReadCommand -fuzztime=10s ./internal/resp
fuzz: elapsed: 11s, execs: 581307 (50220/sec), new interesting: 34 (total: 37)
PASSThe fuzzer ran half a million inputs in ten seconds. Had one failed, Go would have written it to testdata/fuzz/FuzzReadCommand/.
From then on go test would run it with the seeds, as a regression test, until
the bug was fixed. The race detector from chapter 9 is the other built-in tool, and it is worth seeing
what it prints. A scratch program that runs the chapter 5 INCR from two goroutines with no lock prints this under the detector:
$ go run -race .
WARNING: DATA RACE
Write at 0x00c000186000 by goroutine 7:
runtime.mapassign_faststr()
main.main.func1()
Previous read at 0x00c000186000 by goroutine 6:
runtime.mapaccess2_faststr()
Goroutine 7 (running) created at:
sync.(*WaitGroup).Go()errors.Join#
One Go change in this chapter is not a test but a thing tests find. The log's Close from chapter 11 must sync the file and then close it, and either may fail. Returning the first error would skip the close. errors.Join "returns an error that wraps the given errors. Any nil error values are discarded". It "returns nil if every value in errs is nil". So both calls run, and the caller hears about both. errors.Is and errors.As from chapter 4 look inside a joined error as they look inside a wrapped one:
// The file is closed even when the sync failed, and the caller hears about both.
// errors.Join drops the nils, so with no error at all it returns nil.
return errors.Join(sync, l.f.Close())The race detector and the fuzzer report what they saw, not what exists.
A race on a path the test never ran is not reported. An input the fuzzer never generated is
not tried. Both tools are excellent and both are statistical. That is why kv runs the race detector in make verify on every change and keeps the fuzz corpus. The number of things seen then keeps growing.
The next section shows the other kind of tool.
Rust: the compiler already ran, and the tools check the rest#
Rust has three places for a test, and kv now uses all three. The first is the file under test, in a module marked #[cfg(test)]. The attribute "tells Rust to compile and run the test code only when you run cargo test, not when you run cargo build". A helper in that module is compiled only for tests too, which is where a function that turns a typed line into a reply belongs:
#[cfg(test)]
mod tests {
use super::*;
use crate::store::MapStore;
/// A command typed as one inline line, run against the store, and its reply as
/// text. The helper is compiled only for tests, like the module around it.
fn run(store: &mut MapStore, line: &str) -> String {
let args: Command = line.split(' ').map(str::as_bytes).collect();
let mut out = Vec::new();
execute(&args, store, &mut out).unwrap();
String::from_utf8(out).unwrap()
}
#[test]
fn incr_counts_from_nothing() {
let mut store = MapStore::new();
assert_eq!(run(&mut store, "INCR n"), ":1\r\n");
assert_eq!(run(&mut store, "INCR n"), ":2\r\n");
assert_eq!(run(&mut store, "GET n"), "$1\r\n2\r\n");
}
#[test]
fn an_expiry_in_the_past_deletes_the_key() {
let mut store = MapStore::new();
run(&mut store, "SET k v");
assert_eq!(run(&mut store, "EXPIRE k 0"), ":1\r\n");
assert_eq!(run(&mut store, "EXISTS k"), ":0\r\n");
}
}The second place is the documentation. A code block in a doc comment is a test, because "rustdoc supports executing your documentation examples as tests", and cargo test runs them. The example in the comment on parse is the one a reader of the docs sees. It cannot go stale, because a stale example stops the build:
/// One request from the front of `buf`: an array of bulk strings such as
/// `*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n`, or one inline line of words. The returned
/// `Command` borrows `buf`, which the signature says with the elided lifetime: one
/// reference in, so the one reference out lives as long as it.
///
/// A code block in a doc comment is a test. `cargo test` compiles and runs it, and
/// `cargo doc` shows it, so an example that stops being true stops the build:
///
/// ```
/// use kv::resp::{Parsed, parse};
/// let Ok(Parsed::Done(args, used)) = parse(b"*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n") else {
/// unreachable!("a whole request")
/// };
/// assert_eq!(args, [&b"ECHO"[..], b"hi"]);
/// assert_eq!(used, 22);
/// assert!(matches!(parse(b"*2\r\n$4\r\nEC"), Ok(Parsed::Incomplete)));
/// ```
pub fn parse(buf: &[u8]) -> Result<Parsed<'_>> {
let Some((line, mut used)) = take_line(buf, 0) else {
return Ok(Parsed::Incomplete);
};
if line.first() != Some(&b'*') {
let words = line.split(|b| *b == b' ').filter(|w| !w.is_empty());
return Ok(Parsed::Done(words.collect(), used));
}
let n = parse_len(&line[1..], b'*')?;
if n > MAX_ARGS {
return Err(KvError::Protocol(b'*'));
}
let mut args = Vec::with_capacity(n);
for _ in 0..n {
let Some((line, start)) = take_line(buf, used) else {
return Ok(Parsed::Incomplete);
};
if line.first() != Some(&b'$') {
return Err(KvError::Protocol(line.first().copied().unwrap_or(b'\n')));
}
let size = parse_len(&line[1..], b'$')?;
if size > MAX_BULK {
return Err(KvError::Protocol(b'$'));
}
let end = start + size;
if buf.len() < end + 2 {
return Ok(Parsed::Incomplete);
}
if &buf[end..end + 2] != b"\r\n" {
return Err(KvError::Protocol(buf[end]));
}
args.push(&buf[start..end]);
used = end + 2;
}
Ok(Parsed::Done(args, used))
}
/// The line that starts at `from`, without its ending, and where the next line starts.
/// A bare LF is accepted as well as CRLF, so that a command typed into nc works.
fn take_line(buf: &[u8], from: usize) -> Option<(&[u8], usize)> {
let rest = &buf[from..];
let nl = rest.iter().position(|b| *b == b'\n')?;
let line = rest[..nl].strip_suffix(b"\r").unwrap_or(&rest[..nl]);
Some((line, from + nl + 1))
}
fn parse_len(digits: &[u8], type_byte: u8) -> Result<usize> {
std::str::from_utf8(digits)
.ok()
.and_then(|s| s.parse().ok())
.ok_or(KvError::Protocol(type_byte))
}The third place is the tests/ directory next to src/. Each file there is an integration test. "Cargo will compile each of the files as an individual crate", and a crate of its own sees only what the library makes pub. That is the point. An integration test cannot reach into private functions, so it tests the crate as a user would. The one in kv runs a request through the parser and the executor, the way the server does:
/// One request through the parser and the executor, as the server runs it, and the
/// reply bytes. A helper that every test in the file shares is an ordinary function.
fn round_trip(store: &mut MapStore, request: &[u8]) -> Vec<u8> {
let mut out = Vec::new();
match resp::parse(request) {
Ok(Parsed::Done(args, used)) => {
assert_eq!(used, request.len(), "the whole request is used");
cmd::execute(&args, store, &mut out).expect("a Vec never fails to write");
}
other => panic!("not a whole request: {other:?}"),
}
out
}#[test]
fn set_then_get() {
let mut store = MapStore::new();
assert_eq!(
round_trip(&mut store, b"*3\r\n$3\r\nSET\r\n$1\r\nk\r\n$2\r\nhi\r\n"),
b"+OK\r\n"
);
assert_eq!(
round_trip(&mut store, b"*2\r\n$3\r\nGET\r\n$1\r\nk\r\n"),
b"$2\r\nhi\r\n"
);
}
#[test]
fn the_wrong_kind_is_an_error_and_not_a_panic() {
let mut store = MapStore::new();
round_trip(&mut store, b"*3\r\n$5\r\nLPUSH\r\n$1\r\nl\r\n$1\r\nx\r\n");
let reply = round_trip(&mut store, b"*2\r\n$3\r\nGET\r\n$1\r\nl\r\n");
assert!(
reply.starts_with(b"-WRONGTYPE"),
"{}",
String::from_utf8_lossy(&reply)
);
}
#[test]
fn an_unknown_command_is_named_in_the_error() {
let mut store = MapStore::new();
assert_eq!(
round_trip(&mut store, b"*1\r\n$4\r\nNOPE\r\n"),
b"-ERR unknown command 'NOPE'\r\n"
);
}Write the unit test a_request_survives_a_round_trip in the parser's test module. Build a command with three arguments, one of which contains a line ending. Write it
with write_array_len and write_bulk into a Vec<u8>.
Parse the vector, and assert that the result equals the command. The line ending inside a value is the case that a
length-prefixed format exists for.
#[test]
fn a_request_survives_a_round_trip() {
let args = vec![b"SET".to_vec(), b"k".to_vec(), b"a value\r\nwith a line".to_vec()];
// your code here
}Stuck?Show the full listing
#[cfg(test)]
mod tests {
use super::*;
/// The parser works on bytes already in memory, so a test needs no socket.
fn parse_one(input: &[u8]) -> Result<Parsed<'_>> {
parse(input)
}
fn args_of(input: &[u8]) -> Vec<Vec<u8>> {
match parse_one(input).unwrap() {
Parsed::Done(args, used) => {
assert_eq!(used, input.len());
args.iter().map(|a| a.to_vec()).collect()
}
Parsed::Incomplete => panic!("incomplete"),
}
}
#[test]
fn an_array_of_bulk_strings() {
assert_eq!(
args_of(b"*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n"),
vec![b"ECHO".to_vec(), b"hi".to_vec()]
);
}
#[test]
fn an_inline_line_and_a_bare_lf() {
let got = args_of(b" SET a b\n");
assert_eq!(got.len(), 3);
assert_eq!(got[2], b"b");
}
#[test]
fn nothing_yet_is_incomplete() {
for cut in [&b""[..], b"*2\r\n$4\r\nEC", b"*2\r\n$4\r\nECHO\r\n"] {
assert!(matches!(parse_one(cut).unwrap(), Parsed::Incomplete));
}
}
#[test]
fn bad_bytes_are_a_protocol_error() {
for bad in [&b"*x\r\n"[..], b"*1\r\n#4\r\nECHO\r\n", b"*100\r\n"] {
assert!(matches!(parse_one(bad).unwrap_err(), KvError::Protocol(_)));
}
}
#[test]
fn a_request_survives_a_round_trip() {
let args = vec![
b"SET".to_vec(),
b"k".to_vec(),
b"a value\r\nwith a line".to_vec(),
];
let mut bytes = Vec::new();
write_array_len(&mut bytes, args.len()).unwrap();
for a in &args {
write_bulk(&mut bytes, a).unwrap();
}
assert_eq!(args_of(&bytes), args);
}
}cargo test, and what comes after the dashes#
cargo test runs every unit test, every integration test and every doc test, in that order, each set in a binary of its own. --lib runs only the unit tests, --test roundtrip only that integration file, and --doc only the documentation. A name after the command is a filter, so cargo test incr runs the tests with incr in their name. Everything after a lone -- goes to the test binary rather than to cargo. Two are worth knowing. -- --no-capture shows what a test prints, which the harness hides unless the test fails. -- --test-threads 1 runs the tests one at a time, because Rust runs them in parallel by default and Go only when asked.
Fuzzing in Rust is cargo-fuzz, which the fuzzing book calls "the recommended
tool for fuzz testing Rust code". It drives the same libFuzzer that C used, through a crate
called libfuzzer-sys, and it needs the nightly toolchain. The targets live in a
small crate of their own under fuzz/, and the target for kv is the C one in
Rust:
fuzz_target!(|data: &[u8]| {
let mut input = data;
let mut store = kv::store::MapStore::new();
let mut out = Vec::new();
// run every complete command in the input, as the server would
while let Ok(kv::resp::Parsed::Done(args, used)) = kv::resp::parse(input) {
let _ = kv::cmd::execute(&args, &mut store, &mut out);
input = &input[used..];
}
});$ cargo install cargo-fuzz
$ cargo +nightly fuzz run parse -- -max_total_time=20
#343809 DONE cov: 335 ft: 1611 corp: 356/36Kb exec/s: 16371
Done 343809 runs in 21 second(s)The last tool is Miri, an interpreter that runs a Rust program slowly and checks every
memory access against the rules. Its README calls it "an Undefined Behavior detection tool for Rust", and it finds the
same things AddressSanitizer finds in C. The server kv has no unsafe code of its own, so
Miri has nothing to find until chapter 15. It still runs:
$ rustup +nightly component add miri
$ cargo +nightly miri test --lib
test result: ok. 12 passed; 0 failed; finished in 92.81sNinety seconds for tests that take a hundredth of a second natively. That is the cost of checking rules at run time that the compiler could not check at compile time. Safe Rust avoids that cost by having the compiler check them first.
cargo-fuzz and Miri both need nightly, and the fuzz crate must stay out of the
main one. The nightly toolchain is a second compiler installed beside the stable
one, chosen per command with +nightly. The fuzz/ directory is a separate package with an empty [workspace]
table. The main Cargo.toml has one too. So a cargo command in one directory
never reads the files of the other. Forget the one in fuzz/ and the first error names a workspace you did not know you had.
Fuzzing, three ways, one engine#
All three fuzzers in this chapter are coverage-guided. They watch which branches an input reaches and mutate the inputs that reach new ones.
That is why a fuzzer finds the corner cases of a parser within seconds. Two of the three
fuzzers are the same engine. The engine called libFuzzer is
what clang links into the C target and what libfuzzer-sys links into the Rust
one. Go wrote its own, with the same design, and built it into go test. The rates on the author's machine were 17,000 inputs a second for Rust and 50,000 for
Go. The small C target in a container ran at 350,000. None found a bug
in kv's parser. The unit tests had already covered the cases a fuzzer finds first, which is
the order the book recommends. The fuzzer's own case is the one in the scene, the input that nobody writes down.
What finds what#
| The bug | C | Go | Rust |
|---|---|---|---|
| Reading freed memory | ASan at run time, if a test reaches it | cannot be written | cannot be written |
| A data race | ThreadSanitizer, in its own build | -race at run time, if a test reaches it | the compiler, always |
| Integer overflow | UBSan at run time | wraps, by the language definition | panics in a debug build, wraps in release |
| A panic or crash on bad input | a sanitizer, or a core dump | the fuzzer, in go test | the fuzzer, on nightly |
| A wrong answer | your tests | your tests | your tests |
The table is the chapter in four lines. C finds its bugs with tools that watch the program run, which means a test must reach the bug. Go has two such tools built in, and a language definition that removes the undefined cases. Rust removes two whole rows at compile time, and uses the same run-time tools as C for what remains. The last row is the same for all three, and it is where most bugs live.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| A test runner | Weak | Strong | Strong | C has none, so you write one. Go and Rust have one in the box, with no setup. |
| Where a test may live | Weak | OK | Strong | C has the file you write, and Go the package and an Example. Rust has the module, the docs and a directory for outsiders. |
| Fuzzing | OK | Strong | OK | libFuzzer is one flag on Linux and a container on a Mac, Go has it built in, and Rust needs nightly and a second crate. |
| Memory errors | OK | Strong | Strong | C finds them at run time with ASan or Valgrind. Go and Rust cannot write them in safe code. |
| Data races | OK | OK | Strong | C and Go find the races a test provokes, with a sanitizer or the detector. Rust refuses them at compile time. |
| Undefined behaviour | Weak | Strong | OK | C has it everywhere and UBSan finds some. Go has almost none, a data race being the exception, and Rust confines it to unsafe, where Miri checks it. |
| Speed of the tools | Strong | Strong | Weak | ASan costs two times and the race detector two to twenty, while Miri costs nine thousand. |
- A detector sees what ran. A sanitizer, the race detector and a fuzzer report the bugs a test reached. The compiler's refusal is the only kind of proof.
- Seed and signal. A fuzzer needs a valid input to start from and a sanitizer to tell it when it has won. With both, two seconds. Without, twenty-two million runs for nothing.
- Three homes for a Rust test. The cfg(test) module sees everything, the doc comment is read by users, and the tests directory sees only what is pub.
Benchmarking and profiling#
Time the same request through the three parsers, ten million times each. Each language has its own way to say "run this loop and divide", and the chapter shows all three. The answers, on the author's machine:
C, kv_parse 56 ns 0 allocations
Rust, parse 179 ns 1 allocation
Go, ReadCommand 532 ns 9 allocations, 184 bytesGo is three times slower than Rust and nine times slower than C, on the one piece of kv that is pure computation. The chapter finds out why, finds out whether it matters, and halves it. Then it turns to the servers, where a Rust number is the one that is wrong.
The book has quoted numbers since chapter 6. This chapter shows how to take one that means something. It also shows how to find where the time goes when the number is worse than you expected. It benchmarks and profiles all four servers, and follows two bad results from a profile to a fix. Measurement is the only argument that ends a discussion about speed.
Chapter 9 reported that the four servers answer about 70,000 to 100,000 requests a second from fifty clients. That number is true and nearly useless, because it measures a round trip on the loopback interface more than it measures kv. The Redis documentation says so in its own words. When the client and the server share a machine, "the CPU is the limiting factor with redis-benchmark". By default the tool "does not use pipelining or any parallelism at all". A benchmark is a question, and a question has to be chosen.
| Measuring | C | Go | Rust |
|---|---|---|---|
| The load generator | redis-benchmark | the same | the same |
| A micro-benchmark | a loop and the clock | go test -bench | an example binary, or criterion |
| Counting allocations | none to count | -benchmem | a global allocator that counts |
| What the compiler decided | -O0 to -O3, measured | -gcflags=-m | the release profile, LTO |
| A CPU profile | sample on macOS, perf on Linux | runtime/pprof, then go tool pprof | sample or perf, with symbols kept in the release build |
| Memory knobs | none: you free it | GOGC and GOMEMLIMIT | none: you drop it |
| The surprise | none | nine allocations a request | a wall at 200,000 requests a second with pipelining |
| The fix | none needed | five allocations, from a free list the reader already had | a read lock, after four experiments that failed |
Taking a number#
Every number in this chapter comes from one command shape. Fifty connections, four client threads so that the client is not the bottleneck, two hundred thousand requests, and two settings of the pipeline. A pipeline of 16 means each connection sends sixteen commands before it waits for a reply. The documentation calls pipelining "a feature often exploited by real world applications". It changes what is measured. Without it, the number is mostly the cost of a round trip. With it, the number is mostly the cost of the server:
$ redis-benchmark -p 6380 -c 50 -n 200000 --threads 4 -t set,get -q # one at a time
$ redis-benchmark -p 6380 -c 50 -n 200000 --threads 4 -t set,get -q -P 16 # sixteen at a time no pipeline pipeline of 16
SET GET SET GET
C, poll, one thread 88,700 79,900 796,800 800,000
Go, goroutines 88,700 88,700 800,000 800,000
Rust, threads 99,950 88,700 159,700 199,400
Rust, tokio 99,950 99,850 160,000 199,800The left half says the four servers are the same, within noise, because the left half
measures the loopback. The right half says something is wrong with the two Rust servers.
Both stop at a fifth or a quarter of what C and Go reach. The second half of the chapter is about that column,
and the tools that explain it. First, the shape of a single measurement. The full output
of the benchmark, without -q, gives a distribution and not one number:
Summary:
throughput summary: 79617.83 requests per second
latency summary (msec):
avg min p50 p95 p99 max
0.540 0.064 0.487 0.919 1.319 5.103The median request took half a millisecond. One in a hundred took more than 1.3 milliseconds, and the slowest took five. A server is judged by the p99 column more often than by the average, because the average hides the requests that a user notices.
A profiler interrupts a program many times a second and writes down what it was doing. The result is a list of stacks with a count against each, and the count says where the time went. In 2011 Brendan Gregg drew those stacks as a picture he called a flame graph. He describes it as "a visualization of hierarchical data that I created to visualize stack traces of profiled software". Each box is a function, and its width is the share of samples that passed through it. The boxes above it are the functions it called. The x axis "is not the passage of time". Every profiler in this chapter produces the same kind of data, and every one of them can draw that picture. The chapter reads the lists instead, because a list fits in a book.
C: a clock, a loop and five flags#
A server profile answers "where does the time go". A micro-benchmark answers "how long does this function take", which is a different question with a different tool. Go has one built in, and Rust has one on the nightly toolchain. C has a clock, which is enough:
Write a benchmark for kv_parse. Build one SET request as a string and read the monotonic clock, with kv_now_ms from chapter 10. Parse the
request ten million times in a loop. Read the clock again, and print the nanoseconds for
one parse. Count the successful parses and print
the count too, so that the compiler cannot remove the loop as unused work. The Makefile
target bench builds it with the release flags and no sanitizer.
int main(void) {
const char *req = "*3\r\n$3\r\nSET\r\n$16\r\nkey:000000001234\r\n$3\r\nxxx\r\n";
/* your code here */
}Stuck?Show the full listing
int main(void) {
const char *req = "*3\r\n$3\r\nSET\r\n$16\r\nkey:000000001234\r\n$3\r\nxxx\r\n";
size_t len = strlen(req);
long runs = 10 * 1000 * 1000;
kv_cmd cmd;
size_t used = 0;
long ok = 0;
int64_t start = kv_now_ms();
for (long i = 0; i < runs; i++) {
ok += kv_parse(req, len, &cmd, &used) == KV_PARSE_OK;
}
int64_t elapsed = kv_now_ms() - start;
/* ok is printed so that the compiler cannot remove the loop as unused work */
printf("%ld parses in %lld ms: %.1f ns each (%ld ok)\n", runs, (long long)elapsed,
elapsed * 1e6 / (double)runs, ok);
return 0;
}The optimisation levels, measured#
The one number that changes a C benchmark most is a compiler flag. The book has built with -O2 since chapter 2, and the Makefile has a target that builds the same benchmark at every level. The gcc manual describes -O0 as "Reduce compilation time and make debugging produce the expected results. This is the default." At -O2 the compiler "performs nearly all supported optimizations that do not involve a space-speed tradeoff". -O3 adds the ones that do, such as unrolling loops. -Os optimises for size:
$ make bench-levels
-O0 10000000 parses in 998 ms: 99.8 ns each
-O1 10000000 parses in 560 ms: 56.0 ns each
-O2 10000000 parses in 586 ms: 58.6 ns each
-O3 10000000 parses in 551 ms: 55.1 ns each
-Os 10000000 parses in 622 ms: 62.2 ns eachThe whole of the gain is between -O0 and -O1, and the rest is noise. That is the usual shape. A parser is small functions and short loops, and the first level already inlines the small functions, such as kv_find_eol, into their one caller. The keyword inline would ask for the same thing by name. But its intent "is to serve as a hint", and "the compilers can (and usually do) ignore presence or absence of the inline specifier". At -O1 and above the compiler has already made that decision from the size of the function, and kv has no inline anywhere. A second keyword makes a promise rather than a request. restrict on a pointer parameter says that nothing else reaches the memory it points to, so the compiler may reorder loads and stores freely. The benchmark below adds two arrays with and without it:
/* Without restrict, the compiler must allow for dst and src overlapping, so each store
* may change the next load, and the loop is kept in order or guarded by a check. With
* the promise, the loop may add eight or sixteen numbers in one instruction. */
static void add_plain(int32_t *dst, const int32_t *src, size_t n) {
for (size_t i = 0; i < n; i++) {
dst[i] += src[i];
}
}
static void add_restrict(int32_t *restrict dst, const int32_t *restrict src, size_t n) {
for (size_t i = 0; i < n; i++) {
dst[i] += src[i];
}
}$ make bench-copy
plain 62 ms for 100000 passes over 4096 ints
restrict 62 ms for 100000 passes over 4096 ints (100% of the time, sum 1677312000000)No difference, and the reason is worth more than a difference would have been. clang vectorises both loops. For the plain one it adds a run-time check that the arrays do not overlap, and takes the fast path when they do not. They never do. The promise bought nothing that the compiler could not check itself. A broken promise is undefined behaviour, as the C reference says in its own words. The lesson is the chapter's. A keyword that is supposed to make code faster is a hypothesis, and the benchmark is the experiment.
A sampler and a short answer#
macOS ships a sampling profiler called sample. Given a process and a number of seconds, it writes every stack it saw. It adds a summary
sorted by the function at the top of the stack. The C server was sampled for five seconds while the benchmark ran:
$ sample $(pgrep kv) 5 -file kv.sample
$ grep -A12 "Sort by top of stack" kv.sample
read (in libsystem_kernel.dylib) 1741
write (in libsystem_kernel.dylib) 1561
poll (in libsystem_kernel.dylib) 427
_platform_memmove 67
kv_server_run (in kv) 31
kv_parse (in kv) 17
strncasecmp_l 16
kv_exec (in kv) 13Three system calls hold ninety percent of the samples. The parser and the command
code together hold one percent. The C server is not spending its time in C. It is
spending it in the kernel, moving bytes in and out of sockets, and no change to
kv_parse would be visible in the benchmark. That is the most common result
of profiling a server, and the most useful, because it stops you from optimising the wrong
thing. On Linux the same question is asked with perf record, which the manual
describes as "Run a command and record its profile into perf.data". It is answered with
perf report.
A profile is cheap and a guess is expensive. The sampler above took five seconds to run and ten seconds to read, and it closed a question that could have cost a day. The second half of the rule matters as much. Every change made for speed is measured again with the same command. A change that does not move the number is reverted, however clever it was.
Go: nine allocations, and the flag that names them#
The Go benchmark is a function in a test file, and the testing package runs it. Functions "of the form func BenchmarkXxx(*testing.B) are considered benchmarks". b.Loop runs the body for as long as the package needs to get a stable number. The flag -benchmem adds two columns that the scene's number came from:
// BenchmarkReadCommand times the parser on one SET request. go test -bench runs the loop
// for as long as it needs to get a stable number. The reader is reused, so that the
// number is the parser and not the cost of a 4 KB buffer per request.
func BenchmarkReadCommand(b *testing.B) {
req := []byte("*3\r\n$3\r\nSET\r\n$16\r\nkey:000000001234\r\n$3\r\nxxx\r\n")
r := bufio.NewReader(nil)
for b.Loop() {
r.Reset(bytes.NewReader(req))
if _, err := ReadCommand(r); err != nil {
b.Fatal(err)
}
}
}$ go test -run xxx -bench ReadCommand -benchmem ./internal/resp/
BenchmarkReadCommand-12 2267204 532.2 ns/op 184 B/op 9 allocs/opEscape analysis#
Nine allocations for a request with three arguments. Go decides for each value whether it may live on the goroutine's stack, which is free, or must go on the heap. There the garbage collector (GC) will find it later. The guide to the collector says a value escapes to the heap when "the Go compiler cannot determine its lifetime". The compiler will say which, for every line, with one flag. The manual describes -m as "Print optimization decisions. Higher values or repetition produce more detail":
$ go build -gcflags=-m ./internal/resp/ 2>&1 | grep "escapes to heap"
internal/resp/resp.go:54:14: make([][]byte, 0, n) escapes to heap
internal/resp/resp.go:67:14: make([]byte, size + 2) escapes to heap
internal/resp/resp.go:77:16: append escapes to heapThose three are the arguments and the slice that holds them, four allocations the parser must make, because its caller keeps them. The other five were a surprise. bufio.Reader.ReadBytes returns a fresh slice for each line, four lines in this request, and strconv.Atoi wants a string, which the conversion from bytes allocates. Both values are dead by the next line of the parser. The reader has a second method, ReadSlice, which returns a view into the reader's own buffer, valid until the next read. That is exactly as long as the parser needs it. The number is parsed from the bytes without a string:
// readLine returns one line without its line ending. ReadSlice hands back a view into
// the reader's own buffer, valid until the next read, so no allocation is made for a
// line. Chapter 13 measured the difference. A line longer than the buffer is an error,
// which for a length field it should be.
func readLine(r *bufio.Reader) ([]byte, error) {
line, err := r.ReadSlice('\n')
if err != nil {
if err == io.EOF && len(line) > 0 {
return nil, io.ErrUnexpectedEOF
}
if err == bufio.ErrBufferFull {
return nil, protoErr(line[0])
}
return nil, err
}
line = bytes.TrimSuffix(line, []byte("\n"))
return bytes.TrimSuffix(line, []byte("\r")), nil
}
// atoi parses a non-negative decimal from bytes without making a string of them, which
// strconv.Atoi would need and the compiler would allocate.
func atoi(b []byte) (int, bool) {
if len(b) == 0 || len(b) > 18 {
return 0, false
}
n := 0
for _, c := range b {
if c < '0' || c > '9' {
return 0, false
}
n = n*10 + int(c-'0')
}
return n, true
}$ go test -run xxx -bench ReadCommand -benchmem ./internal/resp/
BenchmarkReadCommand-12 3196206 369.9 ns/op 162 B/op 5 allocs/opFrom 532 to 370 nanoseconds, and from nine allocations to five, for a change of fifteen lines that the flag pointed at. The five that remain are the ones the caller needs, plus the error values that are never made on the happy path. The Go parser is still twice as slow as the Rust one, which allocates once. It is six times slower than the C one, which allocates nothing. Each of those is a design, not a flaw. Go copies so that the arguments are safe to keep. The rest of the gap is the collector's accounting on every make, which is the price of not writing free.
GOGC, GOMEMLIMIT and the Pool#
The collector has two knobs, both environment variables, and both are worth one experiment. "The GOGC variable sets the initial garbage collection target percentage", and "the default is GOGC=100". A collection runs when the heap has grown by that percentage since the last one. The guide puts the trade in one sentence. "Doubling GOGC will double heap memory overheads and roughly halve GC CPU cost." The second knob is a ceiling. "The GOMEMLIMIT variable sets a soft memory limit for the runtime", and with GOGC=off the collector runs only when the limit is near. The server was run under each setting with the pipelined SET benchmark, and the collections counted with GODEBUG=gctrace=1:
setting SET a second collections resident memory
GOGC=100 (the default) 796,800 11 10 MB
GOGC=25 800,000 41 8 MB
GOGC=400 800,000 2 21 MB
GOGC=off GOMEMLIMIT=32MiB 800,000 1 26 MBThe throughput did not move, because this server allocates little and the collector is not where its time goes. The memory did, in the direction the guide says. A server that must fit in a container with a fixed memory should set the limit. The guide recommends it when "the Go program is the only program with access to some set of resources". sync.Pool, from chapter 9, is the third tool. It is "a set of temporary objects that may be individually saved and retrieved", so that a buffer is reused instead of collected. The five remaining allocations are not worth one. A pool pays when the object is large and the rate is high. The reader's 4 KB buffer is the one object in kv that fits, and it lives for the whole connection already.
The profiler is a package#
Go's profiler is in the standard library. The package runtime/pprof "writes
runtime profiling data in the format expected by the pprof visualization tool", and
starting it is one call. The chapter 13 server starts a CPU profile when an environment
variable names a file, and the clean stop from chapter 10 ends it:
At the start of main, if KV_CPUPROFILE names a file, create
the file and call pprof.StartCPUProfile. Then arrange for the profile to be stopped and the file closed when main returns. That is after the clean stop has waited for every client. Two defer calls do it, in the order that runs last first.
Without the stop, the profile file stays empty.
if path := os.Getenv("KV_CPUPROFILE"); path != "" {
// your code here
}Stuck?Show the full listing
// KV_CPUPROFILE names a file. The profile runs until the server is told to stop, with
// Ctrl-C or kill, so that a benchmark can be profiled from start to end.
if path := os.Getenv("KV_CPUPROFILE"); path != "" {
f, err := os.Create(path)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
if err := pprof.StartCPUProfile(f); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
defer f.Close()
defer pprof.StopCPUProfile()
}$ KV_CPUPROFILE=cpu.prof ./kv &
$ redis-benchmark -p 6380 -c 50 -n 300000 --threads 4 -t get -q
$ kill -INT %1
$ go tool pprof -top kv cpu.prof
Duration: 4.40s, Total samples = 11.61s (263.58%)
flat flat% sum% cum cum%
6.51s 56.07% 56.07% 6.51s 56.07% syscall.rawsyscalln
1.89s 16.28% 72.35% 1.89s 16.28% runtime.pthread_cond_wait
1.20s 10.34% 82.69% 1.20s 10.34% runtime.kevent
0.82s 7.06% 89.75% 0.82s 7.06% runtime.pthread_cond_signal
0.01s 0.086% 97.59% 3.66s 31.52% internal/poll.(*FD).Read
0.01s 0.086% 97.67% 2.88s 24.81% internal/poll.(*FD).WriteThe total is 263 percent, because the profile adds up the time of every thread, and the
server used almost three cores. More than half of that time was in system calls. A tenth was in kevent, which
is the Go runtime asking the kernel which sockets are ready. Most of the rest was in the
scheduler, parking and waking threads. The cum column
says a third of all time was spent somewhere under Read. Again, the code that
kv wrote is not in the list, and the nine allocations that the micro-benchmark found are invisible at the server's scale. Both measurements are true. The server one says which to act on.
Rust: one allocation, a profile, four experiments, and a lock#
Stable Rust has no benchmark runner in the box. The one most projects use is criterion, "a statistics-driven micro-benchmarking tool". It runs under cargo bench and compares each run with the last. The server kv keeps to the standard library, and its benchmark is an example binary with the C loop in it. It adds one thing the C version could not. A program may name its own allocator, with "a static item whose type implements the GlobalAlloc trait", and this one counts:
/// Every allocation in the program goes through the global allocator, and a program may
/// name its own. This one counts the calls and passes them on to the system allocator.
struct Counting;
static ALLOCS: AtomicU64 = AtomicU64::new(0);
// SAFETY: every call is passed through to System unchanged, which is itself a correct
// allocator, and the counter has no effect on the memory handed back.
unsafe impl GlobalAlloc for Counting {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
ALLOCS.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc(layout) }
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
unsafe { System.dealloc(ptr, layout) }
}
}
#[global_allocator]
static GLOBAL: Counting = Counting;fn main() {
let req: &[u8] = b"*3\r\n$3\r\nSET\r\n$16\r\nkey:000000001234\r\n$3\r\nxxx\r\n";
let runs = 10_000_000u32;
let mut ok = 0u32;
let before = ALLOCS.load(Ordering::Relaxed);
let start = Instant::now();
for _ in 0..runs {
// the parser takes a slice and copies nothing, so no allocation is made per argument
ok += u32::from(matches!(
kv::resp::parse(req),
Ok(kv::resp::Parsed::Done(..))
));
}
let elapsed = start.elapsed();
let allocs = ALLOCS.load(Ordering::Relaxed) - before;
println!(
"{runs} parses in {} ms: {:.1} ns each ({ok} ok), {} allocations a parse",
elapsed.as_millis(),
elapsed.as_nanos() as f64 / f64::from(runs),
allocs / u64::from(runs)
);
}The release profile#
The counter says one allocation a parse, the Vec that holds the borrowed arguments. The time depends on the profile. The dev profile of Cargo has no optimisation and full debug information. The release profile is opt-level = 3. The same example, both ways, and then with two settings the release profile leaves off by default. lto = "fat", link-time optimisation (LTO), optimises across every crate at once. codegen-units = 1 gives the compiler the whole crate in one piece, at the cost of a slower build:
cargo run --example bench_parse 1460.9 ns (dev: opt-level 0)
cargo run --release --example bench_parse 179.0 ns (release: opt-level 3)
with lto = "fat" and codegen-units = 1 149.2 ns
with #[inline(never)] on the two helpers 185.2 nsEight times between the two profiles, which is why a Rust benchmark run without --release measures nothing. The last line is the Rust form of the C inline experiment. The attribute #[inline] "suggests whether a copy of the attributed function's code should be placed in the caller". The reference says that "in every form the attribute is a hint". Forbidding it on the two helpers cost three percent, which is what the compiler's own decision was worth, and kv does not write the attribute. The release build keeps line information when Cargo.toml says so, which costs nothing at run time and lets a profiler name inlined functions:
# Keep function names in the release build, so a profiler can show them. The code is
# as fast as without this; only the binary is larger.
[profile.release]
debug = 1The Rust servers stop at 200,000 requests a second with pipelining, and the first question is where the time goes. The sampler names Rust functions, and its summary for the thread server under load is this:
__recvfrom (in libsystem_kernel.dylib) 67711
__accept (in libsystem_kernel.dylib) 1411
__semwait_signal (in libsystem_kernel.dylib) 1409
__sendto (in libsystem_kernel.dylib) 1252
__psynch_mutexwait (in libsystem_kernel.dylib) 689
core::slice::memchr::memchr_aligned (in kv) 256
kv::cmd::execute (in kv) 25
kv::resp::read_line (in kv) 24Sixty-seven thousand samples in recvfrom are not CPU time.
The sampler records every thread on every tick, and the thread server has fifty threads
that spend most of their lives blocked in a read. The C server has one thread, so its
list was honest. The Rust list is dominated by waiting. The lines that describe work are lower down. The one that matters is
__psynch_mutexwait, which is a thread waiting for a mutex that another thread
holds. On Linux, perf samples CPU time
by default and does not have this problem.
One line of a profile is a hypothesis, not an answer. The author tested it with experiments, each a few minutes, and the table is the record:
experiment SET GET conclusion
one client, pipeline 16 201,000 the client limits one connection
50 clients, GET only / SET only 200,000 253,000 the count of clients changes nothing
parking_lot's user-space mutex 200,000 199,800 not the kind of mutex
TCP_NODELAY on the socket 159,700 199,200 not Nagle's algorithm
one socket handle instead of try_clone 199,600 199,400 not the clone
each thread with a private store, no lock 800,000 796,800 the lockThe last line is not a server anyone would ship, because the threads no longer share data, but it answers the question. Without the shared lock the Rust server is as fast as the C one. With it, every command from every thread passes through one mutex. A contended mutex on macOS is a pthread mutex, which passes the lock from one thread to the next through the kernel. The Rust standard library uses a futex on Linux and the pthread mutex on other Unix systems. The same server in a Linux container reached 398,000. Go has the same lock around its store and reaches 800,000. The comment above the Go mutex explains why. A waiting goroutine is parked on a semaphore of the runtime. In what the comment calls "starvation mode", ownership passes "from the unlocking goroutine to the waiter at the front of the queue". The kernel is not involved.
The first fix was to shrink what the lock covers. The command code now returns the reply as a value, and the server writes it after the lock is released:
/// A reply as a value, so that a command can be run under a lock and written after
/// the lock is released. Formatting and the write to the socket cost more than the
/// store operation itself, and neither needs the store.
pub enum Reply {
Simple(&'static str),
Error(String),
Bulk(Vec<u8>),
Int(i64),
Nil,
Array(Vec<Reply>),
}
pub fn write_reply(w: &mut (impl Write + ?Sized), reply: &Reply) -> io::Result<()> {
match reply {
Reply::Simple(s) => write_simple(w, s),
Reply::Error(s) => write_error(w, s),
Reply::Bulk(b) => write_bulk(w, b),
Reply::Int(n) => write_int(w, *n),
Reply::Nil => write_nil(w),
Reply::Array(items) => {
write_array_len(w, items.len())?;
items.iter().try_for_each(|item| write_reply(w, item))
}
}
}It made no difference. Both servers stayed at 200,000, because the cost was the handoff and not the hold. The change stays, because formatting under a lock is a habit worth losing, and because the second fix needed it. The second fix is the reader-writer lock chapter 9 promised. Reads of the store take a shared lock that any number of threads may hold together. Writes take the exclusive one:
Write execute_read. It takes the arguments by reference and the store by shared reference. It returns Some(reply) for the commands that only read the store, and
None for anything else. The table of commands says which is which, with Run::Read and Run::Write, so the function is a lookup and a call.
pub fn execute_read(args: &Command, store: &dyn Store) -> Option<Reply> {
// your code here
}Stuck?Show the full listing
/// The reply of a command that only reads the store, or `None` when the command writes
/// and the caller must take the exclusive lock and call `execute`.
pub fn execute_read(args: &Command, store: &dyn Store) -> Option<Reply> {
match lookup(args) {
Ok(Entry {
run: Run::Read(f), ..
}) => Some(f(args, store, Instant::now())),
Ok(_) => None,
Err(reply) => Some(reply),
}
}The serving loop tries the shared lock first and takes the exclusive one only for a write:
// A read takes the shared lock, which many threads hold at once. A
// write takes the exclusive one. Either way the lock is released
// before the reply is formatted or written.
let reply = {
let db = db.read().unwrap_or_else(|e| e.into_inner());
db.execute_read(&args)
};
let reply = match reply {
Some(reply) => reply,
None => {
let mut db = db.write().unwrap_or_else(|e| e.into_inner());
db.execute(&args)?
}
};
resp::write_reply(&mut writer, &reply)?; no pipeline pipeline of 16
SET GET SET GET
Rust, threads, before 99,950 88,700 159,700 199,400
Rust, threads, after 99,800 99,850 264,900 796,800
Rust, tokio, after 99,850 99,255 199,600 800,000Reads are fixed. GET with pipelining went from 200,000 to 800,000, the C
and Go figure, because fifty readers no longer queue. Writes are not fixed, because fifty
writers still do, and 265,000 is where the pthread handoff leaves them. The next step would be to split the store into shards with a lock each, so that writers
to different keys never meet. That is what every fast Rust store does, and what kv does
not need. The
chapter stops here, with a number that is honest about what was fixed and what was not.
The method#
The two halves of the chapter are the same method. A benchmark gave a number that looked wrong. A tool gave one line to suspect, the escape flag in Go and a profile in Rust. Each experiment changed one thing and measured again with the same command. The Go fix was kept because it moved the number. The first Rust fix was kept and measured, found to do nothing, and said so. The second was measured, found to fix half the problem, and said so. The C flags were measured and found to be noise, and said so. Nothing in that sequence needed a particular language. The same steps find the same kind of answer in all three, with their own tools. The "golden rule" of the benchmark documentation applies to all of it. "Only compare apples and apples."
On macOS, sample needs nothing installed, and Instruments, started from the
command line with xctrace record, gives a time profile with a graphical
viewer. Both are macOS tools and do not run on Linux. There, perf is the tool for C and Rust, and
its output feeds the flame graph scripts directly. Go's pprof works everywhere
Go does, because the profiler is inside the runtime and not in the operating system. On Windows, Visual Studio has a CPU profiler for native code, and pprof again
covers Go.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| A micro-benchmark | OK | Strong | OK | C has a clock and a loop, and Go has go test -bench with allocation counts. Rust needs an example binary or a crate. |
| Seeing what the compiler did | OK | Strong | OK | Go prints its decisions for every line. C and Rust measure each flag, profile and attribute yourself. |
| A profiler in the box | OK | Strong | OK | C and Rust use the operating system's profiler. Go carries its own, and it works the same everywhere. |
| Reading the profile | Strong | OK | Weak | One C thread gives an honest list and Go needs the cum column, while the fifty Rust threads drown the list in waiting. |
| The cost of memory | Strong | OK | Strong | C and Rust allocate what the code says. Go allocates what escapes, and a flag has to tell you which. |
| A fast server | Strong | Strong | OK | C and Go reach 800,000 with pipelining. Rust needs a read lock to match on reads and stays behind on writes. |
| Why it was slow | OK | Strong | Weak | The Go runtime hides lock handoff from the kernel. Rust's standard mutex is the platform's, and macOS makes it expensive. |
| Honest numbers | Strong | Strong | Strong | redis-benchmark and the same command shape treat all three alike. |
- Two questions, two tools. A micro-benchmark says how long a function takes. A profile says where a server's time goes. The parser was nine times slower in Go and one percent of the server in all three.
- Escape analysis. Go puts a value on the heap when it cannot prove its lifetime, and -gcflags=-m names each one. Four of nine allocations were the caller's, and five were the parser's habits.
- A flag is a hypothesis. -O3, inline, restrict, LTO and #[inline] each moved the number by a few percent or not at all. The benchmark decides, not the keyword.
Shipping#
Build the C server on a current Linux, copy the binary to a Linux that is six years older, and run it:
$ docker run --rm -v "$PWD":/w -w /w gcc:14 sh -c 'ldd --version | head -1; gcc -O2 -pthread -o kv src/*.c'
ldd (Debian GLIBC 2.41-12+deb13u4) 2.41
$ docker run --rm -v "$PWD":/w -w /w debian:buster sh -c 'ldd --version | head -1; ./kv'
ldd (Debian GLIBC 2.28-10+deb10u3) 2.28
./kv: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./kv)Nothing in kv changed between the two machines. The C library did, and the binary carries a note that names the version it was built against. This chapter is about that note, and about everything else a program has to settle before it leaves the machine it was written on.
kv works on the machine it was written on. This chapter builds it for other machines, Linux on two kinds of processor and Windows, from a Mac. It gives each server the options, the configuration and the version a shipped program needs. It covers linking, cross-compilation, the three operating systems, and what the newest C standard changes.
To ship a program, three questions need answers. What does the binary carry with it, and
what does it expect to find on the target machine? Which machines is it built for, and from
where? And how does a user get a particular version, know which one they have, and tell it
where to listen? Each
language has settled answers. They differ enough that the same questions produce three
different make dist targets.
| Shipping | C | Go | Rust |
|---|---|---|---|
| A plain build links | the system C library, dynamically | nothing, with cgo off | the system C library, dynamically |
| A static binary | -static with musl, through zig | the default on Linux | the musl target |
| Cross-compiling | zig cc -target | GOOS= and GOARCH= | rustup target add, then cargo zigbuild |
| Options | getopt, and a die that never returns | the flag package | std::env::args, or the clap crate |
| A config file | none here, by hand elsewhere | encoding/json into a struct | none here, the serde crate elsewhere |
| Logging | fprintf(stderr, ...) | log/slog, with fields | eprintln! |
| A version in the binary | a macro you define | go:embed of a generated file | CARGO_PKG_VERSION at build time |
| Platform code | #ifdef and a file per system | a build constraint per file | #[cfg] per item |
| A Linux server, stripped | 45 KB | 2.5 MB | 521 KB |
| Windows | no, the C code is Unix-only | GOOS=windows, unchanged | x86_64-pc-windows-gnu, unchanged |
What a binary carries#
Chapter 2 showed what the three hello programs linked, and the picture has not changed.
A C program calls the C library for everything and links it dynamically by default. The
binary then holds a note that says "find libc.so.6 on this machine", and the scene showed the rest of the note. Each function it uses is tagged with the library version that introduced it. A library too old for the tag refuses to load the program. On Linux, a Go program with CGO_ENABLED=0 makes its system calls itself and links nothing. On macOS it still links libSystem, as chapter 2 showed. A Rust program carries its own standard library and still calls the C library for the system. So it links libc.so.6 as the C one does, with the same note.
The old answer was to build on the oldest system you support. The newer answer is to link statically against musl, a small C library written for this. The binary then carries its own copy and asks for nothing. Go made that the default. The other two can choose it.
Shared libraries exist to save memory and disk. One copy of the C library serves every program on the machine, which mattered when memory was small and expensive. The cost was that a program depended on the exact library that was there. A program moved to another machine often met a different one. Containers reversed the trade. A container image holds one program, so there is nothing to share, and a binary that needs nothing can run in an empty image. Go chose to link nothing from the start, and the choice fit the container world that arrived a few years later.
C: zig as the cross-compiler#
Cross-compiling C has always been possible and has rarely been pleasant. The compiler must be built for the target, and so must the C library and its headers. Those pieces must be found and told about each other. The Zig project changed that. The project overview says that Zig "is also a C compiler" and carries a C library for 97 targets. As a result, "Zig builds for all supported targets independently of the host", with "no separate cross toolchain". One download, and zig cc -target names the machine:
Write the dist target. Loop over two targets,
x86_64-linux-musl and aarch64-linux-musl. For each, run zig cc over every source file with the usual flags and three more. -target names the machine, -static puts the C library inside the binary, and -s strips the symbols. Name each output after its target under dist/.
ZIG ?= zig
TARGETS := x86_64-linux-musl aarch64-linux-musl
dist: $(SRC)
mkdir -p dist
# your loop hereStuck?Show the full listing
# Cross-compile with zig cc, which carries a C compiler, a linker and the C libraries of
# every target in one download. -static with musl gives a binary that needs no libc file,
# and -s strips the symbols, which is most of the size.
ZIG ?= zig
TARGETS := x86_64-linux-musl aarch64-linux-musl
dist: $(SRC)
mkdir -p dist
for t in $(TARGETS); do \
$(ZIG) cc -target $$t -static -s -std=c17 -Wall -Wextra -Werror -O2 -o dist/kv-$$t $(SRC) || exit 1; \
done
ls -l dist$ make dist
-rwxr-xr-x 53952 kv-aarch64-linux-musl
-rwxr-xr-x 44824 kv-x86_64-linux-musl
$ file dist/kv-x86_64-linux-musl
ELF 64-bit LSB executable, x86-64, statically linkedForty-five kilobytes, for a server with a hash table, an event loop and a log. The static
musl library costs almost nothing, because only the functions kv calls are linked in. Before
the -s flag the same file was 1.5 megabytes, and the difference was the symbol table and the debugging information. The binary was then run in a Linux container that had never seen a
compiler:
$ docker run --rm -v "$PWD/dist":/dist redis:8 bash -c \
'KV_PORT=6380 /dist/kv-x86_64-linux-musl & sleep 1; redis-cli -p 6380 PING'
PONG
$ docker run --rm --platform linux/arm64 -v "$PWD/dist":/dist redis:8 bash -c \
'KV_PORT=6380 /dist/kv-aarch64-linux-musl & sleep 1; redis-cli -p 6380 PING'
PONGThe second command ran the arm64 binary under emulation on an Intel machine, which is slow and is enough to prove the build. Both answered, and both ran without a file from the system. The scene's binary would have answered on the old Debian too, had it been built this way.
C again: options, and a function that does not come back#
A shipped server is started by a script or a service manager, which passes options. The environment variables kv has used since chapter 3 are not enough. The same standard gives C one function for the job. getopt "parses the command-line arguments", from a string that lists the option letters, and a letter followed by a colon "requires an argument". It returns the letters one at a time, leaves an argument in optarg, and "returns -1" when the options are done:
/* The settings, from the command line first and the environment second. */
typedef struct {
int port;
const char *aof; /* NULL for no persistence */
const char *fsync; /* NULL for the default */
const char *hash; /* "fnv" for the unseeded hash */
} kv_options;
static const char USAGE[] = "usage: kv [-p port] [-a file] [-f always|everysec|no]\n"
" -p port the port to listen on (KV_PORT), 6380 by default\n"
" -a file append every write to this file (KV_AOF)\n"
" -f policy how often to fsync the file (KV_FSYNC)\n";
/* getopt walks argv for the options in its string. A letter followed by a colon takes an
* argument, which getopt leaves in optarg. It returns -1 when the options are done, and
* optind then says where the rest of the arguments start. */
static kv_options kv_options_parse(int argc, char **argv) {
const char *port = getenv("KV_PORT");
kv_options o = {
.port = port && *port ? atoi(port) : 6380,
.aof = getenv("KV_AOF"),
.fsync = getenv("KV_FSYNC"),
.hash = getenv("KV_HASH"),
};
int c;
while ((c = getopt(argc, argv, "p:a:f:h")) != -1) {
switch (c) {
case 'p':
o.port = atoi(optarg);
break;
case 'a':
o.aof = optarg;
break;
case 'f':
o.fsync = optarg;
break;
case 'h':
fputs(USAGE, stdout);
exit(0);
default: /* getopt has already printed which option it did not know */
die("%s", USAGE);
}
}
if (optind < argc) {
die("kv: unexpected argument '%s'", argv[optind]);
}
if (o.port <= 0 || o.port > 65535) {
die("kv: not a port: %d", o.port);
}
if (o.aof != NULL && *o.aof == '\0') {
o.aof = NULL;
}
return o;
}The function that reports a bad option is the one to look at. It never returns, and it says so with _Noreturn. That is the C11 specifier that "specifies that the function does not return to its point of invocation". The compiler then knows that the code after a call to die is unreachable, and the reader knows it too. C23 deprecates the keyword for the attribute [[noreturn]], and the section below says why. The other word on the line is older than both. __attribute__ is a GNU extension, outside the standard but shared by gcc and clang. The format attribute says a function takes printf-style "arguments that should be type-checked against a format string". So a wrong %d in a call to die is an error at compile time, as it is for printf itself:
/* Print a message the way printf would, and exit. _Noreturn tells the compiler, and
* the reader, that nothing follows a call. The attribute is a GNU extension that gcc
* and clang share, outside the standard: it checks the arguments against the format
* string at compile time, as the compiler does for printf itself. */
__attribute__((format(printf, 1, 2))) _Noreturn static void die(const char *fmt, ...) {
va_list ap;
va_start(ap, fmt);
vfprintf(stderr, fmt, ap);
va_end(ap);
fputc('\n', stderr);
exit(2);
}The C version of kv does not build for Windows. It uses poll, kqueue or epoll and the socket calls of POSIX, the Portable Operating System Interface. Chapter 9 explained that Windows wants a different loop. A C program that must run on both keeps two versions of the networking code behind one header. It chooses between them with #ifdef _WIN32, as chapter 9 chose a poller. kv does not, and says so. The other two languages carry that
#ifdef inside their standard libraries, which is the whole difference in the
next two sections.
C17 against C23#
Every C file in this book is built with -std=c17, because every compiler the book names accepts it. The newer standard, ISO/IEC 9899:2024, is in clang 17 and gcc 14, and the snapshot has a tour of it in one file. Most of the list is C catching up with the other two languages, feature by feature. nullptr is a null with a type of its own, so that it cannot be passed where an integer is wanted. constexpr is a constant the compiler evaluates, for the places a #define was used. typeof and auto take a type from an expression, as the other two languages do. Attributes in double brackets replace __attribute__ for the common cases. The booleans are keywords without a header. #embed puts the bytes of a file into an array at compile time, which Go does with go:embed and Rust with include_bytes!:
/* nullptr has a type of its own, so it cannot be mistaken for an int, as 0 or NULL can */
static const char *first_or(const char *s, const char *fallback) {
return s != nullptr ? s : fallback;
}
/* [[nodiscard]] makes the compiler warn when a caller ignores the result */
[[nodiscard]] static int parse_digit(char c) {
return c >= '0' && c <= '9' ? c - '0' : -1;
}
/* #embed pastes the bytes of a file into an array at compile time */
static const unsigned char banner[] = {
#embed "banner.txt"
};
int main(void) {
/* constexpr: a true compile-time constant, usable where a #define was needed before */
constexpr int limit = 64;
/* typeof and auto: the type comes from the expression, as in the other two languages */
typeof(limit) half = limit / 2;
auto digit = parse_digit('7');
/* binary literals and digit separators */
unsigned mask = 0b1111'0000;
/* true and false are keywords, and bool needs no header */
bool ok = digit == 7;
static_assert(limit > 0); /* no message needed any more */
printf("%s %d %d %u %d %.*s", first_or(nullptr, "default"), half, digit, mask, ok,
(int)sizeof banner, banner);
return 0;
}$ make c23
default 32 7 240 1 kv, three languagesThe standard also removes something. A function defined the way Kernighan and Ritchie wrote them in 1978, with the parameter types on separate lines, is no longer C. That is the first removal of a feature from the language in its history. The server kv stays on C17 for now. The one line that would change is the _Noreturn above.
Go: two variables#
Go cross-compiles by design. The compiler is one program that contains the code generators for every target. The standard library is compiled for the target as part of the build.
go help environment describes the two variables. GOOS is "The
operating system for which to compile code", and GOARCH is "The architecture,
or processor, for which to compile code". A third, CGO_ENABLED, says "Whether
the cgo command is supported", and setting it to 0 keeps the C library out:
Write a dist target. Loop over the four targets below. Split each into the two variables, add .exe for Windows, and set CGO_ENABLED=0. Build with -ldflags="-s -w", which drops the symbol table and the debugging information.
TARGETS := linux/amd64 linux/arm64 windows/amd64 darwin/arm64
dist:
mkdir -p dist
# your loop hereStuck?Show the full listing
# Cross-compile by setting two variables. CGO_ENABLED=0 keeps the binary free of the C
# library, so that it runs on any Linux of the right architecture, and in a scratch
# container. -s -w drop the symbol table and debug information, which halves the size.
TARGETS := linux/amd64 linux/arm64 windows/amd64 darwin/arm64
dist:
mkdir -p dist
for t in $(TARGETS); do \
os=$${t%/*}; arch=$${t#*/}; ext=""; [ $$os = windows ] && ext=.exe; \
CGO_ENABLED=0 GOOS=$$os GOARCH=$$arch go build -ldflags="-s -w" -o dist/kv-$$os-$$arch$$ext ./cmd/kv || exit 1; \
done
ls -l dist$ make dist
-rwxr-xr-x 2541858 kv-darwin-arm64
-rwxr-xr-x 2556064 kv-linux-amd64
-rwxr-xr-x 2490528 kv-linux-arm64
-rwxr-xr-x 2677760 kv-windows-amd64.exeFour binaries for three operating systems, from one command, with nothing installed
beyond Go. Each is about 2.5 megabytes, because each carries the runtime, and the Linux ones link nothing. They answered PING in the same containers as the C ones.
The Windows one was not run, because this book has no Windows machine. The two Windows binaries in this chapter are the one case that was built and not tested. The file command at least
agrees that it is "PE32+ executable (console) x86-64, for MS Windows".
A Go program that imports a package which uses cgo links the C library and loses the
freedom to run anywhere. The usual sources are the net package's name
resolver and the os/user package, both of which have pure Go paths that
CGO_ENABLED=0 selects. Build with it off, and turn it on only for a package
that has no other way, which chapter 15 shows.
Go again: flags, a file, a log and a version#
The Go server reads its settings in three layers, each over the last. The environment comes first, for the test runner. A JSON file comes second, if a flag names one. The flags on the command line come last. The flag package "implements command-line flag parsing". Each call such as flag.StringVar registers a flag with its default and its help text. flag.Parse reads the arguments, and -h prints the help that the package wrote from those calls. The file is read with encoding/json, which "parses the JSON-encoded data and stores the result in the value pointed to by v". The tag on each field of the struct "gives the name of the field" in the file. A field must be exported to be filled:
// config is every setting the server takes. The json tags name the keys in a config
// file, and the same struct is filled from the environment and the flags.
type config struct {
Port string `json:"port"`
AOF string `json:"aof"`
Fsync string `json:"fsync"`
}
func env(name, fallback string) string {
if v := os.Getenv(name); v != "" {
return v
}
return fallback
}
// loadConfig reads the settings in order: the environment, then a JSON file if -config
// names one, then the flags given on the command line, each layer over the last.
func loadConfig() config {
cfg := config{Port: env("KV_PORT", "6380"), AOF: os.Getenv("KV_AOF"), Fsync: os.Getenv("KV_FSYNC")}
file := flag.String("config", "", "a JSON file with port, aof and fsync")
flag.StringVar(&cfg.Port, "port", cfg.Port, "the port to listen on")
flag.StringVar(&cfg.AOF, "aof", cfg.AOF, "append every write to this file")
flag.StringVar(&cfg.Fsync, "fsync", cfg.Fsync, "how often to fsync the file: always, everysec or no")
showVersion := flag.Bool("version", false, "print the version and exit")
flag.Parse()
if *showVersion {
fmt.Print(version)
os.Exit(0)
}
if *file != "" {
data, err := os.ReadFile(*file)
if err == nil {
err = json.Unmarshal(data, &cfg)
}
if err != nil {
slog.Error("config", "file", *file, "err", err)
os.Exit(2)
}
// a flag given on the command line still wins over the file
flag.Visit(func(f *flag.Flag) {
switch f.Name {
case "port":
cfg.Port = f.Value.String()
case "aof":
cfg.AOF = f.Value.String()
case "fsync":
cfg.Fsync = f.Value.String()
}
})
}
return cfg
}$ cat kv.json
{"port": "6395", "fsync": "no"}
$ ./kv -config kv.json -aof kv.aof
2026/10/12 20:33:04 INFO listening port=6395 version=8f75db5-dirty
2026/10/12 20:33:04 INFO open files clients=122848
2026/10/12 20:33:04 INFO replayed commands=0 file=kv.aof
2026/10/12 20:33:05 INFO stopping cause="interrupt signal received"The lines it printed come from the fourth piece, log/slog. It "provides structured logging, in which log records include a message, a severity level" and key-value pairs. The text handler writes key=value. The JSON handler, "whose output is line-delimited JSON", is one line to switch to, for a collector that wants fields and not sentences. The version in the first line is the fifth piece, and it took two directives:
// go generate runs the command in the directive and writes version.txt, which is checked
// in. go:embed then compiles the file's contents into the binary as a string, so that a
// shipped kv can say which commit it is.
//
//go:generate sh -c "git describe --always --dirty > version.txt"
//go:embed version.txt
var version stringgo generate "runs commands described by directives within existing files". The manual is firm that it "is never run automatically by go build, go test, and so on. It must be run explicitly". So the generated file is checked in. go:embed then does what the C23 #embed does. The embed package "provides access to files embedded in the running Go program". The files are read "from the package directory or subdirectories at compile time", into a string, a byte slice or a file system.
For thirteen years, a goroutine started inside a Go loop saw the loop's last value, not its own. The variable was "created once and updated by each iteration", so every closure captured the same one. The bug was so common that go vet had a check for it, and it reached production often enough that the Go team changed the language. In Go 1.22, "each iteration of the loop creates new variables, to avoid accidental sharing bugs". A change to what existing code means is the thing a language with a compatibility promise cannot do, and Go found a way. The new meaning applies "only in packages contained in modules that declare go 1.22 or later in their go.mod files". The line go 1.27 in kv's module is a shipping decision, then, and not only a minimum. It says which language the code was written in. The preview ran for four months before the release and produced "zero reports of any problems in production code".
Rust: a target, a linker, and zig again#
Rust is in between. The compiler can produce code for any target, and
rustup target add fetches the standard library for one. The rustup book is
precise about what that gives you. "rustup target add only installs the Rust
standard library for a given target", and "There are typically other tools necessary to
cross-compile, particularly a linker". The linker, and the C library it links against, are
the C problem again. The common answer is the same tool as in C. cargo-zigbuild describes
itself as "Compile Cargo project with zig as linker for easier cross compiling":
Add a dist profile to Cargo.toml that inherits from
release, turns debugging information off, and strips the symbols. Then write a dist target in a Makefile. It runs cargo zigbuild --profile dist for each of the three targets below and copies each binary under dist/. Each target must first be added with rustup target add.
x86_64-unknown-linux-musl
aarch64-unknown-linux-gnu
x86_64-pc-windows-gnuThe profile comes first.
[profile.dist]
# your settings hereStuck?Show the full listing
# Keep function names in the release build, so a profiler can show them. The code is
# as fast as without this; only the binary is larger.
[profile.release]
debug = 1
# The profile for shipping: the release settings, without the symbols.
[profile.dist]
inherits = "release"
debug = 0
strip = true# Cross-compile with cargo-zigbuild, which lets zig link for the target. Each target is
# a rustup component: rustup target add <triple>. The musl target is static. The dist
# profile in Cargo.toml strips the symbols that the release profile keeps for profiling.
TARGETS := x86_64-unknown-linux-musl aarch64-unknown-linux-gnu x86_64-pc-windows-gnu
dist:
mkdir -p dist
for t in $(TARGETS); do \
cargo zigbuild --profile dist -q --target $$t || exit 1; \
cp target/$$t/dist/kv dist/kv-$$t 2>/dev/null || cp target/$$t/dist/kv.exe dist/kv-$$t.exe; \
done
ls -l dist$ cargo install --locked cargo-zigbuild
$ rustup target add x86_64-unknown-linux-musl aarch64-unknown-linux-gnu x86_64-pc-windows-gnu
$ make dist
-rwxr-xr-x 444416 kv-aarch64-unknown-linux-gnu
-rwxr-xr-x 1020416 kv-x86_64-pc-windows-gnu.exe
-rwxr-xr-x 521024 kv-x86_64-unknown-linux-muslThe musl binary is static and the gnu one is not. In the arm64 container, ldd lists three shared libraries for it, starting with libc.so.6. That is the note from the first section, and it ran because the container had them.
The musl one ran because it needed nothing. The chapter 13 profile setting,
debug = 1, is why a separate profile exists. It keeps symbols for the profiler,
and a shipped binary does not want them. Stripping took the musl build from 7.3 megabytes
to 521 kilobytes.
The gnu target prints a warning on every build. It reads
warning: linker stderr: ignoring deprecated linker optimization setting "1",
under the lint linker_messages. It comes from zig's linker, which is being
driven with a flag meant for another, and the binary is fine. Cross-compiling in Rust means rustup, cargo and the linker cooperating. A message from one of them that the others do not understand is the normal cost of that.
Options from the standard library, and the crate everyone uses#
The Rust standard library gives a program its arguments and nothing more. std::env::args "returns the arguments that this program was started with", as an iterator, and "the first element is traditionally the path of the executable". Everything after that is a loop with a match. The server kv writes it itself, because the book has one dependency and will not add a second for this. The function that reports a bad option returns !, the never type, which is what _Noreturn is in C with the compiler's full attention. A call to it can stand where a value is expected, because no value will ever be needed:
/// Every setting the server takes, from the environment first and the command line
/// second. The standard library gives an iterator over the arguments and nothing else.
/// Most Rust programs reach the same result with the clap crate, which writes the
/// parser and the help text from a struct like this one.
struct Options {
port: u16,
aof: Option<String>,
fsync: String,
}
const USAGE: &str = "usage: kv [-p port] [-a file] [-f always|everysec|no]
-p port the port to listen on (KV_PORT), 6380 by default
-a file append every write to this file (KV_AOF)
-f policy how often to fsync the file (KV_FSYNC)";
fn env(name: &str) -> Option<String> {
std::env::var(name).ok().filter(|v| !v.is_empty())
}
/// A usage error is reported and the process ends with status 2, as getopt programs do.
/// The return type `!` is Rust's _Noreturn: this function never comes back.
fn usage_error(msg: &str) -> ! {
eprintln!("kv: {msg}\n{USAGE}");
std::process::exit(2)
}
fn options() -> Options {
let mut o = Options {
port: 6380,
aof: env("KV_AOF"),
fsync: env("KV_FSYNC").unwrap_or_default(),
};
let mut port = env("KV_PORT").unwrap_or_else(|| "6380".to_string());
let mut args = std::env::args().skip(1);
while let Some(flag) = args.next() {
let mut value = || {
args.next()
.unwrap_or_else(|| usage_error(&format!("{flag} needs a value")))
};
match flag.as_str() {
"-p" => port = value(),
"-a" => o.aof = Some(value()).filter(|v| !v.is_empty()),
"-f" => o.fsync = value(),
"-h" => {
println!("{USAGE}");
std::process::exit(0)
}
other => usage_error(&format!("unknown option '{other}'")),
}
}
o.port = match port.parse() {
Ok(port) => port,
Err(_) => usage_error(&format!("not a port: {port}")),
};
o
}Most Rust programs do not write this loop. They add clap, "Command Line Argument Parser for Rust", and put a #[derive(Parser)] on a struct like Options. The parser, the help text and --version are written for them. A config file goes the same way, through the serde crate. It derives the reading of JSON or TOML into a struct, as the Go package does with tags. Both crates are in nearly every shipped Rust program, and the standard library has no such thing. Go put the same two things in its standard library, which is the same decision made in opposite directions. The version needs no crate at all. Cargo sets CARGO_PKG_VERSION from Cargo.toml at build time, and env!("CARGO_PKG_VERSION") puts it in the binary.
Three sizes, three reasons#
Linux x86_64, stripped what is inside
C, musl, static 44,824 bytes kv, and the parts of musl it calls
Rust, musl, static 521,024 bytes kv, the Rust standard library, and musl
Go, CGO_ENABLED=0 2,556,064 bytes kv, and the Go runtime with its collectorThe C binary is small because C has no runtime and the linker keeps only what is referenced. The Rust binary is ten times larger, because its standard library is inside and is itself written in Rust, with formatting, collections and the allocator. The Go binary is fifty times larger, because the garbage collector, the scheduler and the network poller are inside. They are there whether a program uses them or not. None of the three sizes is a problem for a server. All three are a fact a reader should know before choosing a language for a device with a megabyte of flash.
The three operating systems, in one place#
The platform boxes in earlier chapters each covered one difference. Set together, they are the list of what a program must know when it leaves its first machine. The last three rows are how each language says "this code is for that system". The book used all three in chapters 9 and 10.
| The difference | C | Go | Rust |
|---|---|---|---|
| Readiness and completion | poll and kqueue on macOS, epoll on Linux, completion ports on Windows | the runtime hides it | the runtime hides it |
| Sockets | one API from BSD, with Winsock variants on Windows | hidden | hidden |
| Threads and locks | pthreads, or Windows threads | hidden | hidden, but the mutex differs underneath |
| The C library | libc.so.6, libSystem, or the Windows C runtime | not used | used through a thin layer |
| Line endings and paths | \n and /, or \r\n and \ | path/filepath and runtime.GOOS | std::path and cfg!(windows) |
| Choosing code per system | #ifdef __linux__, from macros the compiler predefines | //go:build unix on the first line, or _linux.go in the name | #[cfg(unix)] or #[cfg(target_os = "linux")] on an item |
| The unit it chooses | a region of a file, or a whole file that compiles to nothing | a whole file, always | one item: a function, a module, a statement |
| Where it shows in kv | the pollers, the fsync call, every #include | internal/sys | sys.rs, and the mutex in chapter 13 |
The last row is the honest one. The Go version of kv has one file that knows which operating system it runs on, and so does the Rust version. The Rust one still ran twice as fast on Linux as on macOS under contention, because the standard mutex is a different object on each. Hiding a platform is not the same as making it disappear.
Versions and releases#
A version is a promise about change, and the three ecosystems make the promise the same
way, with three numbers. Semantic versioning, which Go modules and cargo both read, gives each number a meaning. Increase the "MAJOR version when you make incompatible API changes". Increase the "MINOR version when you add functionality in a backward compatible manner". Increase the "PATCH version when you make backward compatible bug fixes". kv's Rust crate has carried a version in
Cargo.toml since chapter 2. Its Go module carries one as a Git tag, which
go install reads, and the binary now carries the commit. Its C version carries one wherever you choose to write it.
How a user gets the program differs more. A Go program with a public module path is installed with go install path@version, which fetches and builds it. Or it is shipped as the one binary this chapter made. A Rust program is installed with
cargo install from crates.io or a Git URL, or shipped as a binary. A C program
is shipped as source, and becomes a package when a distribution packages it, or as a
binary built as above. Redis ships as source and as distribution packages, and that is the
C norm. The one binary with nothing to install is the Go norm, and increasingly the Rust
one.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| Cross-compiling | OK | Strong | OK | C needs zig and gets a small static binary. Go needs two variables, and Rust needs a target, a linker and zig. |
| Static linking | OK | Strong | OK | C and Rust choose musl. Go links nothing by default. |
| Binary size | Strong | Weak | OK | 45 KB, 2.5 MB and 521 KB for the same server. |
| Options and config | OK | Strong | OK | C has getopt and nothing for a file, and Go has flag, json and slog in the box. Rust has the arguments and a crate for the rest. |
| Windows | Weak | Strong | Strong | The C code is Unix-only. Go and Rust build for Windows unchanged. |
| Knowing the platform | Weak | Strong | OK | C sees every difference and Go hides all of them. Rust hides most and leaks one through the mutex. |
| A release | Weak | Strong | Strong | C has no mechanism. Go and Rust read a tag or a version field and install from a path. |
- The note in the binary. A dynamic binary names the C library version it needs, and an older machine refuses it. A static musl binary, or a Go binary, carries everything and asks for nothing.
- Three ways to choose code per system. C uses the preprocessor and macros the compiler predefines. Go uses a constraint on a whole file. Rust uses an attribute on one item.
- A function that never returns. C says _Noreturn or [[noreturn]] and the compiler trusts it. Rust says ! and the compiler knows it. Go has os.Exit and no way to say so.
Talking to C#
Take the chapter 6 pattern matcher, which Go and C both have. Time the Go version against the C version called through cgo, on the same input, from the same benchmark:
$ go test -run xxx -bench Glob -benchmem ./internal/cmatch/
BenchmarkGlobGo-12 49871018 24.75 ns/op 0 B/op 0 allocs/op
BenchmarkGlobCgo-12 15145387 76.56 ns/op 0 B/op 0 allocs/opThe C function does the same work in the same number of instructions, and the call takes three times as long. The difference is the border, and the chapter is about what stands on it. The Rust call to the same function costs nothing, and the chapter says why.
Every operating system, database driver and codec exposes a C interface, so Go and Rust
both have a door to C. This chapter builds a small C library from kv and calls it from Go through cgo and from Rust through an extern block. It passes a struct and a union across, where the layout of memory is the interface. Then it calls Rust from C.
The hash and the pattern matcher from chapter 6 are the library. They are small, they take bytes and a length, and they allocate nothing, which makes them the easiest kind of C to call from anywhere. The same two functions are already written in Go and Rust, so each call can be checked against a native version, and timed against it.
| Talking to C | C | Go | Rust |
|---|---|---|---|
| Declaring a C function | a header | a C comment before import "C" | an unsafe extern "C" block |
| Passing bytes | a pointer and a size_t | unsafe.Pointer and C.size_t | as_ptr() and len() |
| A struct across the border | its layout, by the rules of the ABI | C.struct_name, laid out as C did | #[repr(C)], laid out as C did |
| A union across the border | a tag and the same bytes | a byte array, read with unsafe | a union, read with unsafe |
| Who compiles the C file | you, with make | the go tool, from the package directory | build.rs, with the cc crate |
| A trivial call costs | nothing | about three times a Go call | the same as a C call |
| Being called from C | the normal case | //export | #[unsafe(no_mangle)] pub unsafe extern "C" |
| What it does to the build | nothing | needs a C compiler and cgo on | needs a C compiler for the C part |
The border is a C function#
A compiled function has a name, a place in memory, and a convention for where its arguments and its result go. That convention is the application binary interface (ABI), and the C one is the one every operating system defines and documents. A Go or Rust function has its own. A caller in either language cannot reach a C function without agreeing to the C convention for that one call. The Rust book puts it in one sentence. "The "C" part defines which application binary interface (ABI) the external function uses".
The border has a second property. Nothing on it is checked. The C compiler trusts the
header, Go trusts the comment above import "C", and Rust trusts the
extern block. A wrong type in any of them compiles and misbehaves at run
time. That is why the library below is written to make the declarations hard to get
wrong.
Unix was written in C, and its system calls were given C wrappers. So the first interface every other language had to reach was a C one. Every operating system since has
kept the habit, and so has every library that wants to be used from more than one
language. Go got its door early. The Go blog described cgo in March 2011, a year before Go 1, with C as a "pseudo-package" for "C's name space". Rust had
extern blocks before 1.0, and the 2024 edition made them say
unsafe, because that is what they always were. In October 2022 the border was crossed in the other direction at the largest scale there is. Linus Torvalds accepted the Rust for Linux work into the kernel, released as Linux 6.1, and the first drivers written in Rust followed in 6.8. A kernel is C that calls everything, and it now calls Rust.
C: a library that is easy to call#
A C library is a header and an archive. The header is the only thing the other side
reads, so its shape decides how pleasant the border is. The rules are short. Take a pointer
and a length, never a C string, because Go and Rust bytes have no terminating zero. Return
an int, not a bool, because every language agrees on what a C
int is. Allocate nothing, so that no one has to ask which side frees it.
Declare both functions. kvmatch_hash takes a pointer and a byte count and
returns a uint64_t. kvmatch_glob takes a pattern and a key, each
as a pointer and a length, and returns 1 for a match and 0 otherwise. Include the headers
the types need.
#ifndef KVMATCH_H
#define KVMATCH_H
// your declarations here
#endifStuck?Show the full listing
/* FNV-1a over n bytes. */
uint64_t kvmatch_hash(const char *s, size_t n);
/* 1 when the key matches the Redis-style pattern (* and ? only), else 0. An int, not a
* bool, because every language agrees on what a C int is. */
int kvmatch_glob(const char *pat, size_t plen, const char *key, size_t klen);The bodies are the chapter 6 code with the lengths made explicit. The archive is made
with ar, the tool every C library is built with:
# The hash and the matcher as a static library, for chapter 15. ar bundles object files
# into an archive that a linker can search; this is what every C library is.
lib: build/libkvmatch.a
build/libkvmatch.a: lib/kvmatch.c lib/kvmatch.h | build
$(CC) $(CFLAGS) -c -o build/kvmatch.o lib/kvmatch.c
ar rcs $@ build/kvmatch.o$ make lib
cc -std=c17 -Wall -Wextra -Werror -O2 -g -c -o build/kvmatch.o lib/kvmatch.c
ar rcs build/libkvmatch.a build/kvmatch.o
$ ls -l build/libkvmatch.a
-rw-r--r-- 4264 build/libkvmatch.aA struct and a union on the border#
Two more functions pass something larger than a number. One fills a struct with both answers at once. The other fills a tagged union, the shape chapter 8 used for a value of several kinds. On the border, the shape of a struct is not a detail of the compiler. It is the interface. Every C compiler lays out a struct by the same rule, the one the ABI writes down. Each field sits at the first offset after the previous one that its alignment allows. The struct's size is rounded up to its own alignment. A uint8_t followed by a uint64_t is therefore sixteen bytes, with seven of them padding:
/* Both answers in one call. The struct is part of the interface, and so is its layout:
* the one-byte flag is followed by seven bytes of padding, so that the hash sits at an
* offset of 8, and every caller on every side of the border must agree. */
typedef struct kvmatch_stats {
uint8_t matched; /* 0 or 1 */
uint64_t hash; /* of the key */
} kvmatch_stats;
void kvmatch_glob_stats(const char *pat, size_t plen, const char *key, size_t klen,
kvmatch_stats *out);
/* A number parsed from bytes. A union holds the integer or the double in the same eight
* bytes, and the tag is the only record of which one is there. */
enum { KVMATCH_NONE = 0, KVMATCH_INT = 1, KVMATCH_FLOAT = 2 };
typedef struct kvmatch_number {
int kind; /* KVMATCH_INT or KVMATCH_FLOAT, or KVMATCH_NONE for not a number */
union {
int64_t i;
double f;
} as;
} kvmatch_number;
/* Parse up to 63 bytes as a decimal integer or, with a point, a double. Returns the kind. */
int kvmatch_parse_number(const char *s, size_t n, kvmatch_number *out);The C reference defines the alignment requirement as "the number of bytes between successive addresses at which objects of this type can be allocated". offsetof gives the offset of a field "from the beginning of an object of specified type", padding included. A small program asks the compiler for all of the numbers. The Go and Rust tests below assert the same ones:
int main(void) {
printf("kvmatch_stats: size %zu, alignment %zu, matched at %zu, hash at %zu\n",
sizeof(kvmatch_stats), alignof(kvmatch_stats), offsetof(kvmatch_stats, matched),
offsetof(kvmatch_stats, hash));
printf("kvmatch_number: size %zu, alignment %zu, kind at %zu, as at %zu, as is %zu bytes\n",
sizeof(kvmatch_number), alignof(kvmatch_number), offsetof(kvmatch_number, kind),
offsetof(kvmatch_number, as), sizeof(((kvmatch_number *)0)->as));
return 0;
}$ make layout
kvmatch_stats: size 16, alignment 8, matched at 0, hash at 8
kvmatch_number: size 16, alignment 8, kind at 0, as at 8, as is 8 bytesFour kilobytes, and the C side is done. C has no door to build, because C is the door. The rest of the chapter is the other two languages arriving at it.
Go: cgo#
cgo is part of the go tool. A Go file that has import "C" is a cgo file. The comment directly above that import is the preamble. The documentation says it "is used as a header when compiling the C parts of the package". Every C name in the
preamble appears in Go under the C prefix, so C.size_t is the C
type and C.kvmatch_glob is the function. Any .c file in the
package directory is compiled with the system C compiler and linked in, which is why
kvmatch.c sits beside cmatch.go.
Write Glob(pat, key []byte) bool and Hash(b []byte) uint64
over the C functions. Get the pointer to a slice with
unsafe.SliceData and convert it to *C.char through
unsafe.Pointer. Decide what to pass for an empty slice, whose pointer may be
nil.
/*
#include "kvmatch.h"
*/
import "C"
import "unsafe"
func Glob(pat, key []byte) bool {
// your code here
}Stuck?Show the full listing
/*
#include "kvmatch.h"
*/
import "C"
import "unsafe"
// Glob reports whether key matches the pattern, by calling the C function. The slices
// are passed as a pointer and a length, and C must not keep the pointer: after the call
// returns, the garbage collector may free the bytes, or move them if they live on a stack.
func Glob(pat, key []byte) bool {
return C.kvmatch_glob(cstr(pat), C.size_t(len(pat)), cstr(key), C.size_t(len(key))) != 0
}
// Hash returns the FNV-1a hash of b, from the C code.
func Hash(b []byte) uint64 {
return uint64(C.kvmatch_hash(cstr(b), C.size_t(len(b))))
}
// cstr gives C a pointer to the first byte of a slice, or a pointer to a static byte for
// an empty slice, because unsafe.SliceData of an empty slice may be nil and C must not
// receive nil for a length of zero.
func cstr(b []byte) *C.char {
if len(b) == 0 {
return (*C.char)(unsafe.Pointer(&empty))
}
return (*C.char)(unsafe.Pointer(unsafe.SliceData(b)))
}
var empty byteThe pointer is the delicate part, because Go has a garbage collector and C does not
know about it. The cgo documentation gives the rule. "All Go pointers passed to C must point to pinned Go memory". A pointer passed as a function argument has "the memory they point to implicitly pinned for the duration of the call". After that, "C code may
not keep a copy of a Go pointer after the call returns". kvmatch_glob reads
and returns, so it obeys without trying. A C library that keeps a buffer after the call needs runtime.Pinner, or a copy made with C.CBytes. The copy is freed with C.free, in the C way.
The struct and the union come across under their C names, with the prefix the documentation describes. "To access a struct, union, or enum type directly, prefix it with struct_, union_, or enum_". Go lays the struct out as the C compiler did, padding and all. The test asks the unsafe package for the numbers. The union is the odd one. Go has no unions, so "C's union types are represented as a Go byte array with the same length". The value is read out of those bytes with a conversion that the tag chooses:
// cStats is the C struct, under its cgo name. Go lays it out exactly as the C compiler
// did, padding included, because the bytes cross the border as they are. The type stays
// private: only a cgo file can see the fields of a C struct.
type cStats = C.struct_kvmatch_stats
// Layout asks the Go compiler what it decided about the C struct, with the questions
// sizeof, offsetof and alignof answer in C. The offset is pointer arithmetic, because
// unsafe.Offsetof does not accept a field of a type that cgo generated.
func Layout() (size, hashOffset, align uintptr) {
var s cStats
at := uintptr(unsafe.Pointer(&s.hash)) - uintptr(unsafe.Pointer(&s))
return unsafe.Sizeof(s), at, unsafe.Alignof(s)
}
// GlobStats fills the C struct through a pointer to a Go value, which the cgo rules
// allow because the struct holds no Go pointers, and takes the two answers out of it.
func GlobStats(pat, key []byte) (matched bool, hash uint64) {
var out cStats
C.kvmatch_glob_stats(cstr(pat), C.size_t(len(pat)), cstr(key), C.size_t(len(key)), &out)
return out.matched != 0, uint64(out.hash)
}
// Number is the parsed value, as Go would rather have it: an interface holding an int64
// or a float64, instead of the C union and its tag.
type Number any
// ParseNumber calls the C parser. cgo represents the union as a byte array of its size,
// so the value is read out of those bytes with an unsafe conversion, chosen by the tag.
func ParseNumber(b []byte) Number {
var n C.kvmatch_number
switch C.kvmatch_parse_number(cstr(b), C.size_t(len(b)), &n) {
case C.KVMATCH_INT:
return int64(*(*C.int64_t)(unsafe.Pointer(&n.as)))
case C.KVMATCH_FLOAT:
return float64(*(*C.double)(unsafe.Pointer(&n.as)))
}
return nil
}// The layout Go sees must be the layout C wrote: a one-byte flag, seven bytes of
// padding, and the hash at offset 8. Layout asks the compiler through unsafe.
func TestLayoutAgreesWithC(t *testing.T) {
if size, at, align := Layout(); size != 16 || at != 8 || align != 8 {
t.Fatalf("size %d, hash at %d, align %d", size, at, align)
}
matched, hash := GlobStats([]byte("user:*"), []byte("user:1"))
if !matched || hash != store.Fnv1a([]byte("user:1")) {
t.Fatalf("GlobStats = %v, %d", matched, hash)
}
if n, ok := ParseNumber([]byte("42")).(int64); !ok || n != 42 {
t.Fatalf("42 parsed as %v", ParseNumber([]byte("42")))
}
if f, ok := ParseNumber([]byte("4.5")).(float64); !ok || f != 4.5 {
t.Fatalf("4.5 parsed as %v", ParseNumber([]byte("4.5")))
}
if ParseNumber([]byte("x")) != nil {
t.Fatal("x is not a number")
}
}The first test calls both matchers on the same inputs, and the benchmark from the scene times them:
// The price of the border: the same match, in Go and through cgo.
var sink bool
func BenchmarkGlobGo(b *testing.B) {
pat, key := []byte("user:*"), []byte("user:1")
for b.Loop() {
sink = store.Glob(pat, key)
}
}
func BenchmarkGlobCgo(b *testing.B) {
pat, key := []byte("user:*"), []byte("user:1")
for b.Loop() {
sink = Glob(pat, key)
}
}The C function does the same work as the Go one and takes three times as long. The difference, about fifty nanoseconds, is the crossing. A goroutine runs on a small stack that can grow, and C expects a large fixed one. So the runtime switches stacks on the way in and back on the way out. It also tells the scheduler that this goroutine is away. Fifty nanoseconds is nothing for a call that then decodes a video frame. It is everything for a call made once per byte.
A program that imports a cgo package cannot be built with
CGO_ENABLED=0. The chapter 14 dist target still works
here, because the kv server does not import cmatch. The moment it did, the build would need a C compiler for every target, and a cross-compiler for the foreign ones. The binary would then link the C library. The one-file freedom of
chapter 14 is paid for with the rule that the program contains no C.
Put every cgo call in one small package with a Go interface. Pass a pointer and a length, nothing that holds a Go pointer, and keep the calls coarse. One call that processes a
buffer costs the same crossing as one call that processes a byte. Check the standard library first. The net resolver and os/user both have a pure Go
path, and most of what a program needs from C already has one.
Rust: extern and unsafe#
Rust has no separate tool. An extern "C" block declares functions that live somewhere else, in Rust syntax with C types. The 2024 edition requires the word unsafe in front of it. The edition guide explains why. "It is the responsibility of the author of the extern block to ensure that the signatures are correct". A wrong one "may result in undefined behavior". The compiler checks that the Rust side of each call matches the declaration. It cannot
check that the declaration matches the C.
Declare both functions in an unsafe extern "C" block. Use *const u8 for the pointers and usize for the lengths. The C int is std::ffi::c_int. Then write safe
glob(pat: &[u8], key: &[u8]) -> bool and
hash(bytes: &[u8]) -> u64 around them, each with a
SAFETY comment that says why the call is sound.
unsafe extern "C" {
// your declarations here
}
pub fn glob(pat: &[u8], key: &[u8]) -> bool {
// your code here
}Stuck?Show the full listing
// What the C header declares, in Rust spelling. The compiler takes these on trust: a
// wrong type here is undefined behaviour, which is why the block is unsafe to call.
unsafe extern "C" {
fn kvmatch_hash(s: *const u8, n: usize) -> u64;
fn kvmatch_glob(pat: *const u8, plen: usize, key: *const u8, klen: usize) -> c_int;
fn kvmatch_glob_stats(
pat: *const u8,
plen: usize,
key: *const u8,
klen: usize,
out: *mut Stats,
);
fn kvmatch_parse_number(s: *const u8, n: usize, out: *mut CNumber) -> c_int;
}/// Does the key match the pattern? The C function reads `plen` bytes from `pat` and
/// `klen` from `key` and keeps neither pointer, so a slice is safe to pass for the
/// duration of the call.
pub fn glob(pat: &[u8], key: &[u8]) -> bool {
// SAFETY: both pointers come from live slices and the lengths are their lengths;
// the C code does not write through them and does not keep them.
unsafe { kvmatch_glob(pat.as_ptr(), pat.len(), key.as_ptr(), key.len()) != 0 }
}
/// FNV-1a, from the C code.
pub fn hash(bytes: &[u8]) -> u64 {
// SAFETY: as above, a live slice and its length.
unsafe { kvmatch_hash(bytes.as_ptr(), bytes.len()) }
}The five powers, and the pointer they are for#
The wrapper is the whole idea of unsafe in Rust. The Rust book lists five
things that only an unsafe block may do. Dereference a raw pointer. Call an
unsafe function or method. Access or modify a mutable static variable. Implement an
unsafe trait. Access fields of a union. A call into C is the second, and the block around
it is a boundary, not a switch. Inside it, the borrow checker still runs and every other
rule still holds. Only the five powers are added, and the SAFETY comment
records the argument that they were used correctly. Everything that calls
glob is ordinary safe Rust and cannot break the rule the comment states.
The first power is about the type that chapter 10's lifetimes never touched. A raw pointer, *const T or *mut T, is a reference with the checks removed. The book says raw pointers "are allowed to ignore the borrowing rules" and "aren't guaranteed to point to valid memory". They "are allowed to be null" and "don't implement any automatic cleanup". Making one is safe, which is what as_ptr does. Reading through one is the first of the five powers. The only place kv does it is on the way back from C, in the exported function below.
The struct is declared again in Rust, with one attribute. A plain Rust struct has no promised layout. The reference says the order of the fields "does not have to be the same as the order in which the fields are specified". #[repr(C)] gives it C's rule instead, field by field in declaration order, so that the bytes mean the same on both sides. The union is a union, a type Rust has for exactly this case. Reading a field of it is the fifth power, "because Rust can't guarantee the type of the data currently being stored in the union instance". The tag the C code set is the argument that the read is sound:
/// The C struct, declared again in Rust. `repr(C)` makes the compiler lay it out as C
/// would, fields in order with C's padding, instead of reordering them as it may for a
/// plain Rust struct. The test below checks the numbers against the C ones.
#[repr(C)]
#[derive(Clone, Copy, Debug, Default)]
pub struct Stats {
pub matched: u8,
pub hash: u64,
}
/// The C union: eight bytes that are an integer or a double. Reading a field of a union
/// is unsafe, because the compiler cannot know which one was last written.
#[repr(C)]
#[derive(Clone, Copy)]
union NumberBits {
i: i64,
f: f64,
}
#[repr(C)]
#[derive(Clone, Copy)]
struct CNumber {
kind: c_int,
as_: NumberBits,
}
/// What the parser found, as a Rust enum: the tag and the bytes become one value that
/// a match must take apart, and no wrong field can be read.
#[derive(Debug, PartialEq)]
pub enum Number {
Int(i64),
Float(f64),
}
pub fn glob_stats(pat: &[u8], key: &[u8]) -> Stats {
let mut out = Stats::default();
// SAFETY: live slices and their lengths, and `out` is a valid Stats the C code fills.
unsafe { kvmatch_glob_stats(pat.as_ptr(), pat.len(), key.as_ptr(), key.len(), &mut out) };
out
}
pub fn parse_number(bytes: &[u8]) -> Option<Number> {
let mut n = CNumber {
kind: 0,
as_: NumberBits { i: 0 },
};
// SAFETY: a live slice and its length, and `n` is a valid CNumber the C code fills.
// The union field read below is chosen by the tag the C code set, which is the
// whole contract of a tagged union.
unsafe {
match kvmatch_parse_number(bytes.as_ptr(), bytes.len(), &mut n) {
1 => Some(Number::Int(n.as_.i)),
2 => Some(Number::Float(n.as_.f)),
_ => None,
}
}
}The test asks for the same numbers that the C program printed, with size_of, align_of and offset_of!. The three agree, as they must. A program that declares a C struct itself does this for every struct. That is why most Rust projects with a large C library do not declare them at all. They run bindgen, which "automatically generates Rust FFI bindings to C and C++ libraries". FFI is the foreign function interface. It reads the header, writes the repr(C) structs and the extern blocks, and runs from a build script like the one below.
Someone must compile the C file. A build script does, before the crate compiles, using
the cc crate, which finds the system compiler and links the result into the
crate:
// A build script runs before the crate compiles. This one compiles the C library with
// the cc crate, which finds the system C compiler and links the result into the crate.
fn main() {
cc::Build::new()
.file("../../c/ch15/lib/kvmatch.c")
.include("../../c/ch15/lib")
.warnings(true)
.compile("kvmatch");
println!("cargo:rerun-if-changed=../../c/ch15/lib/kvmatch.c");
println!("cargo:rerun-if-changed=../../c/ch15/lib/kvmatch.h");
}# The build script needs a C compiler driver; cc finds the system one.
[build-dependencies]
cc = "1"
# Besides the Rust library, produce a static library a C program can link: libkv.a.
[lib]
crate-type = ["lib", "staticlib"]$ cargo test -q cmatch
running 4 tests
....
test result: ok. 4 passed; 0 failedA crossing from Rust to C costs what a C to C call costs. Rust threads use the system stack, and the Rust ABI differs from the C one only in what the compiler is free to choose. So the compiler emits a plain call. The benchmark that the Go section needed does not have a result to show here, because there is no difference to measure.
cargo miri test cannot run a test that calls into C. The tool interprets Rust and knows nothing about the compiled C code, so it reports an unsupported
foreign function and stops. It can still check the Rust side of the border. The test for
the exported function below passes under miri, because the raw pointers it builds slices
from come from Rust. Test the C side with the C tools from chapter 12 and the Rust side with miri. No tool sees both at once.
In January 2020 the most used Rust web framework was taken down by its author for a weekend, over the word unsafe. Sergey Davidoff had published an analysis of the framework's internals. He found code that "violates memory safety" by giving out "multiple mutable references to the same data". He wrote that the maintainers "have refused to investigate it". The discussion that followed on Reddit was large and unkind. The maintainer, Nikolay Kim, wrote "I am done with open source" and threatened to "make [Actix] repos private and then delete them". He restored them within days, writing that "a lot of people depend on actix". He passed the project to a new leader, under whom it continues. The Register noted at the time that "there is extensive use of unsafe code in Actix, leading to debate about what should be fixed". The lesson is not about one maintainer. A Rust project is safe exactly to the extent of its unsafe blocks, and the word draws reviewers the way an open port draws scanners. Every such block in kv has a SAFETY comment that states the argument, which is the habit the dispute was about.
The other way: C calls Rust#
Rust can also be the library. A function marked extern "C" uses the C
convention, and #[unsafe(no_mangle)] keeps its name as written, so that a
C linker can find it. The function below is the chapter 6 matcher behind a C signature:
/// The other direction: a Rust function with a C name and a C signature, for a C program
/// to call. It is the Rust matcher from chapter 6. `#[unsafe(no_mangle)]` keeps the name
/// as written, and `extern "C"` uses the C calling convention.
///
/// # Safety
/// `pat` must point to `plen` readable bytes and `key` to `klen`, as a C caller would
/// promise in a comment. Rust cannot check it, and a wrong length here is the same bug
/// it would be in C.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn kv_rust_glob(
pat: *const u8,
plen: usize,
key: *const u8,
klen: usize,
) -> c_int {
if pat.is_null() || key.is_null() {
return 0;
}
// SAFETY: the contract above, which the caller accepted by calling an unsafe function.
let (pat, key) = unsafe {
(
std::slice::from_raw_parts(pat, plen),
std::slice::from_raw_parts(key, klen),
)
};
c_int::from(crate::store::glob(pat, key))
}It is marked unsafe because calling it is unsafe. The caller promises the lengths,
and nothing checks the promise. That is the contract every C function has always had,
now written in the signature instead of in a comment. With
crate-type = ["lib", "staticlib"] in Cargo.toml, cargo also
produces libkv.a, and a C program links it like any archive:
/* The declaration a header would hold. Rust exported this name with #[unsafe(no_mangle)]. */
int kv_rust_glob(const char *pat, size_t plen, const char *key, size_t klen);
int main(void) {
const char *pat = "user:*";
const char *key = "user:1";
int ok = kv_rust_glob(pat, strlen(pat), key, strlen(key));
printf("kv_rust_glob(\"%s\", \"%s\") = %d\n", pat, key, ok);
return ok ? 0 : 1;
}# A C program calling Rust: build the Rust crate as a static library, then link it like
# any other. The flags after -lkv are what the Rust standard library needs from the
# system on each platform.
RUST := ../../rust/ch15
from-rust: build/from_rust
./build/from_rust
build/from_rust: from_rust/main.c | build
cargo build --release -q --manifest-path $(RUST)/Cargo.toml
$(CC) $(CFLAGS) -o $@ from_rust/main.c -L$(RUST)/target/release -lkv $(RUST_LIBS)
# What the Rust standard library needs from the system. On Linux, set RUST_LIBS to the
# list that `cargo rustc --release -- --print native-static-libs` prints.
ifeq ($(shell uname),Darwin)
RUST_LIBS ?= -framework CoreFoundation
endif$ make from-rust
cargo build --release -q --manifest-path ../../rust/ch15/Cargo.toml
cc ... -o build/from_rust from_rust/main.c -L../../rust/ch15/target/release -lkv -framework CoreFoundation
./build/from_rust
kv_rust_glob("user:*", "user:1") = 1
$ ls -l ../../rust/ch15/target/release/libkv.a build/from_rust
-rw-r--r-- 19629264 libkv.a
-rwxr-xr-x 1498400 build/from_rustThe archive is nineteen megabytes because it holds the whole crate, tokio included. The C program is 1.5 megabytes, because the linker kept only what one matcher needs. That is still the Rust runtime and its allocator. The -framework
CoreFoundation at the end is what the Rust standard library asks of macOS. On Linux the list is different and longer, and cargo rustc --release -- --print native-static-libs prints it. The Rust side knows nothing of this.
The C side, as always, must.
Go can be called from C as well, with a //export comment and -buildmode=c-archive. But the archive carries the Go runtime, which starts its scheduler and collector inside the host program. It is done, and it is rarely a
comfortable fit. The Rust case has no runtime to start. That is why Rust libraries are the ones you find inside C programs, Python packages and browsers, and now inside the kernel.
What a crossing costs#
| The cost of the border | C | Go | Rust |
|---|---|---|---|
| Per call | nothing | about 50 nanoseconds and a stack switch | nothing |
| Memory rules | yours, as ever | pin for the call, copy to keep | the borrow checker stops at the border |
| A struct | laid out by the ABI | laid out by the ABI, fields under their C names | laid out by the ABI, with repr(C) |
| A union | a tag you must honour | bytes you must convert | a field read that needs unsafe |
| What the compiler checks | the header, against the body | the preamble, against the Go side | the extern block, against the Rust side |
| What nothing checks | the header, against the caller | the preamble, against the C | the extern block, against the C |
| Tools that cross | the sanitizers | the race detector does not | miri does not |
| The build | make | a C compiler, per target | a C compiler, per target, for the C part |
The row that matters most is the one that nothing checks. In all three languages the declaration is a promise. The chapter 6 tests, run on both sides of the border, are the only thing that keeps it. That is the C way of working, and at the border, every language works the C way.
Verdict##
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| Calling C | Strong | OK | Strong | Go pays a stack switch per call and loses static builds. C and Rust pay nothing per call. |
| Being called from C | Strong | Weak | Strong | A Go archive carries a runtime that must start. A Rust archive is a plain library. |
| Safety at the border | Weak | OK | OK | C checks nothing anywhere. Go and Rust check their own side and trust the declaration. |
| A struct on the border | Strong | OK | Strong | C owns the rule. Go follows it under C names, and Rust follows it with repr(C) and can assert the numbers. |
| A union on the border | OK | Weak | OK | C reads the field the tag names. Go converts the bytes itself, and Rust names the field and marks the read unsafe. |
| Making the rules visible | Weak | OK | Strong | C has comments and Go has a documented pointer rule. Rust puts unsafe in the signature. |
| Build cost of C code | Strong | Weak | OK | Go needs cgo on for every target. Rust needs the cc crate and a compiler for the C part only. |
- The border is a C function. A name, a pointer, a length and an int, laid out by the ABI. The declaration on each side is a promise that no compiler checks.
- Layout is the interface. A struct's offsets, padding and size are fixed by the ABI. C, Go and Rust compute the same sixteen bytes, and repr(C) is how Rust promises to.
- Five powers. An unsafe block adds raw pointer reads, unsafe calls, mutable statics, unsafe traits and union fields, and nothing else. The SAFETY comment is the argument that they were used correctly.
The verdict#
Thirteen chapters each ended with a small scorecard. This chapter gathers every row into one table, reads what the pattern says about each language, and turns it into advice. When to choose C, when Go, when Rust, and when the right answer is a language this book did not cover.
One warning before the table. The rows score one project, a key-value server, built by one person who had to learn each language's way as they went. A different project moves rows. A device with sixty-four kilobytes of memory has no row for a garbage collector, because it cannot afford one. A web service has no row for binary size, because it does not care. Read the table as evidence about this kind of program, not as a ranking of languages.
The scorecard#
| Topic | C | Go | Rust | Why |
|---|---|---|---|---|
| 02 Toolchains and hello, three times | ||||
| Installing | OK | Strong | Strong | A C compiler is one system command away on macOS and Linux. Go and Rust are one download each, on every platform. |
| Building | Weak | Strong | Strong | C has no build tool, only a convention. Go and Rust each have one tool that every project uses. |
| Seeing what you made | Strong | OK | OK | C shows every step from preprocessor to linker. Go and Rust hide the steps, which is usually what you want. |
| Binary size | Strong | Weak | OK | 8 KB for C, 2.5 MB for Go because the runtime is inside, 415 KB for Rust. |
| Running elsewhere | Weak | Strong | OK | The C binary needs the right C library on the target, the Go binary needs nothing, and Rust is in between. |
| 03 Echo server | ||||
| Opening a port | Weak | Strong | Strong | C needs four calls, a struct and an option. Go and Rust need one call that sets the option for you. |
| Closing it again | Weak | OK | Strong | C needs a cleanup label and Go needs a defer you must write, but Rust closes at the end of scope by itself. |
| The echo loop | Weak | Strong | Strong | C needs the short-write loop and the EINTR retry. Both standard libraries contain them. |
| A lost error | Weak | Weak | OK | C and Go let a return value drop. Rust warns, and with the book's flags refuses. |
| Seeing what happens | Strong | OK | OK | The C version shows every system call. The other two hide them, which is what you want on most days. |
| Windows | Weak | Strong | Strong | The C code needs Winsock changes. Go and Rust build unchanged. |
| What you must learn first | Strong | OK | Weak | C is the calls themselves. Rust asks for Result, ?, ownership and try_clone before an echo works. |
| 04 Speaking RESP | ||||
| Reading bytes | Weak | Strong | Strong | C has stdio, which blocks, so kv grows its own buffer. Both standard libraries have one. |
| Holding the arguments | Strong | OK | OK | The C parser copies nothing. Go and Rust copy each argument once, which is simpler and almost never matters. |
| Saying "not RESP" | OK | OK | Strong | C and Go use a value by convention. Rust uses a kind the compiler knows about and a match the compiler checks. |
| Saying "the client left" | OK | Weak | Strong | Go reuses an error value for a non-error. Rust has Option for exactly this. |
| Passing an error up | Weak | Weak | Strong | C and Go repeat the check at every call. Rust writes one character. |
| Binary values | Strong | Strong | OK | C and Go treat all bytes alike. Rust makes you choose between String and Vec<u8>, and the wrong choice rejects real data. |
| An error with details | Weak | OK | Strong | C has an enum and a fixed buffer, and Go has a type found by searching a chain. Rust has an enum variant the compiler matches. |
| 05 GET and SET | ||||
| A key is bytes | OK | Strong | OK | C carries a length beside every pointer, Go has []byte and string, and Rust makes you pick Vec<u8> because String rejects binary data. |
| Copying and owning | Weak | Strong | Strong | C copies and comments, Go lets the collector own everything, and Rust moves and checks. |
| Parsing a number | Weak | OK | OK | C needs a careful parser to avoid undefined behaviour. The other two accept forms Redis rejects. |
| Overflow | Weak | OK | Strong | C must check before it adds and Go checks against a constant, while Rust has checked_add built in. |
| A missing case | Weak | Weak | Strong | An enum match in Rust must be complete. In C and Go, a new error kind can be forgotten at a call site. |
| Reading the code | OK | Strong | OK | The Go store is the shortest. The Rust signatures say more and take longer to read. |
| A table of commands | OK | Strong | OK | C needs an X-macro and a function pointer type, and Go has a map of funcs. Rust needs dyn Write and a macro for the rows. |
| Text against bytes | Weak | OK | Strong | C has no text type at all, and Go has rune helpers. Rust makes a str keep its promise and a Vec<u8> make none. |
| 06 Your own hash table | ||||
| Writing the table | Weak | Strong | OK | C needs an owner comment on every pointer and three cleanup paths, Go is the shortest, and Rust argues about borrows in set. |
| Freeing memory | Weak | Strong | Strong | C must free in three places that agree. Go and Rust free for you, in different ways. |
| Freeing on time | Strong | Weak | Strong | C and Rust free on the line that removes an entry. Go frees at the next collection. |
| Testing it | Weak | Strong | Strong | C has no test runner in the box. go test and cargo test are built in. |
| Shipping unused code | Weak | OK | Strong | C warns only for static functions and Go ignores unused functions, while Rust reports each one. |
| Needing to write one | Strong | Weak | Weak | In C the table is the store. In Go and Rust it is an exercise, because the map is built in. |
| Writing it once for any type | Weak | OK | Strong | C has void pointers, macros and _Generic, each with a hole. Go has type parameters since 2022, and Rust has bounds the compiler checks at the impl. |
| Seeing where a value lives | Strong | OK | Strong | C says it in every line, and Rust says it in the types. Go decides for you and shows its decision with a flag. |
| 07 Modules, packages and dependencies | ||||
| Splitting code | Weak | Strong | OK | C needs guards, prefixes and a Makefile that tracks headers, while Go needs a folder and Rust a mod line that is easy to forget. |
| Hiding a name | OK | Strong | Strong | C has static, for functions only. Go and Rust hide by default or by a letter. |
| Adding a dependency | Weak | OK | Strong | C has three competing ways and no default, while Go and Rust each have one command, and Rust also has one registry. |
| Having too many | Strong | Strong | Weak | C and Go programs stay small by cost and by a wide standard library. Rust programs grow dozens of crates. |
| Keeping one style | OK | Strong | Strong | clang-format needs a config everyone agrees on. gofmt and rustfmt need nothing. |
| Finding mistakes | Weak | OK | Strong | clang-tidy must be installed. go vet is built in. clippy is built in and has 800 checks. |
| Swapping an implementation | OK | Strong | Strong | C has a struct of function pointers that nobody checks. Go has implicit interfaces, and Rust has checked traits with two kinds of dispatch. |
| Hiding a struct | Strong | Strong | Strong | C has the opaque type, and Go and Rust have unexported fields. All three keep the map out of reach. |
| 08 Data types | ||||
| A value of several kinds | Weak | OK | Strong | C has a tag nothing checks, and Go has a type switch nothing checks. Rust has a match the compiler checks. |
| Adding a kind later | OK | Weak | Strong | C warns at each switch with no default, and Go falls to default in silence. Rust refuses to build until every match knows. |
| A list, a hash, a set | Weak | Strong | Strong | C writes each, or nests the table. Go and Rust have them in the standard library. |
| Walking a collection | Weak | OK | Strong | C needs a callback and a context struct, and Go has range over a function. Rust has adaptors that compose. |
| Scanning by cursor | Strong | OK | OK | C and the hand-written table scan buckets. The Go map sorts, and the Rust map counts from the start. |
| Reading the code | Weak | Strong | OK | Thirteen C commands need a helper each, and Go reads like prose. Rust reads well once match and let else are familiar. |
| 09 Many clients | ||||
| Serving many clients | Weak | Strong | OK | C needs 200 lines of event loop and three pollers, Go one keyword, and Rust an Arc, a Mutex and a clone per thread. |
| Using every core | Weak | Strong | Strong | The C loop is one thread by design. Go and Rust spread clients over every core. |
| A forgotten lock | n/a | Weak | Strong | Go compiles it and the detector may catch it. Rust refuses to compile it. |
| Platform code | OK | Strong | Strong | C needs a file per poller under ifdef. Go and Rust need one guarded function, and their runtimes hide the pollers. |
| Memory for a client | Strong | Strong | OK | C sizes two buffers and a goroutine is a few kilobytes, while a Rust thread is 2 MiB until tokio fixes that at the cost of a dependency. |
| Reading the code | Weak | Strong | OK | The C loop is subtle and the Go version is the chapter 7 code, while the tokio version is a second copy of the parser. |
| Windows | Weak | Strong | Strong | The C loop reports readiness and Windows reports completion. Both runtimes hide that. |
| 10 Expiry | ||||
| A deadline | OK | Strong | Strong | C uses zero to mean none, which the comments must say. Go's zero Time and Rust's Option say it in the type. |
| The clock | OK | Strong | Strong | C picks the clock by name, while Go carries both in one value and Rust gives each a type. |
| Lazy expiry | Strong | OK | OK | C deletes on find. Go and Rust must leave it to the sweep because of the read lock or the borrow. |
| The sweep | Strong | Strong | Weak | C has a cursor and Go a random sample. Rust's retain walks the whole map. |
| A client that hangs up | Weak | Strong | Strong | C dies unless one line ignores SIGPIPE. Both runtimes turn it into an error. |
| Stopping cleanly | OK | Strong | OK | C needs a handler, a pipe and a rule about what a handler may do. Go needs a context, and Rust the C library or tokio. |
| A bug in one command | Weak | Strong | OK | C corrupts or crashes, and Go recovers per connection. Rust panics per thread, and the scope reports it. |
| The parser's arguments | OK | OK | Strong | C borrows by comment, Go copies and caps the capacity, Rust borrows by lifetime and the compiler checks it. |
| Testing time | Weak | Strong | Strong | Go and Rust tests pass a clock in. The C store has no test runner yet. |
| 11 Persistence | ||||
| Writing the file | OK | Strong | Strong | C encodes into a buffer and calls write. Go and Rust reuse the protocol writers and flush. |
| Replay | Strong | Strong | Strong | All three reuse the parser from chapter 4 unchanged, which is the point of a log in RESP. |
| A crash of the process | Strong | OK | OK | C writes straight through. Go and Rust needed a flush the book's test had to find. |
| The clock on disk | OK | Strong | OK | C converts between two clocks itself, while Go's time.Time holds both and Rust converts with a helper. |
| Log and store together | Strong | OK | Strong | One C thread needs no lock, Go takes two locks, and Rust takes one around both. |
| The fsync thread | Weak | Strong | Strong | C needs a mutex, a condition variable, an atomic and a sanitizer. Go needs a channel, and Rust a channel the compiler checks. |
| Reaching the disk on macOS | OK | Strong | Strong | C must know about F_FULLFSYNC itself. Both standard libraries know. |
| 12 Testing and fuzzing | ||||
| A test runner | Weak | Strong | Strong | C has none, so you write one. Go and Rust have one in the box, with no setup. |
| Where a test may live | Weak | OK | Strong | C has the file you write, and Go the package and an Example. Rust has the module, the docs and a directory for outsiders. |
| Fuzzing | OK | Strong | OK | libFuzzer is one flag on Linux and a container on a Mac, Go has it built in, and Rust needs nightly and a second crate. |
| Memory errors | OK | Strong | Strong | C finds them at run time with ASan or Valgrind. Go and Rust cannot write them in safe code. |
| Data races | OK | OK | Strong | C and Go find the races a test provokes, with a sanitizer or the detector. Rust refuses them at compile time. |
| Undefined behaviour | Weak | Strong | OK | C has it everywhere and UBSan finds some. Go has almost none, a data race being the exception, and Rust confines it to unsafe, where Miri checks it. |
| Speed of the tools | Strong | Strong | Weak | ASan costs two times and the race detector two to twenty, while Miri costs nine thousand. |
| 13 Benchmarking and profiling | ||||
| A micro-benchmark | OK | Strong | OK | C has a clock and a loop, and Go has go test -bench with allocation counts. Rust needs an example binary or a crate. |
| Seeing what the compiler did | OK | Strong | OK | Go prints its decisions for every line. C and Rust measure each flag, profile and attribute yourself. |
| A profiler in the box | OK | Strong | OK | C and Rust use the operating system's profiler. Go carries its own, and it works the same everywhere. |
| Reading the profile | Strong | OK | Weak | One C thread gives an honest list and Go needs the cum column, while the fifty Rust threads drown the list in waiting. |
| The cost of memory | Strong | OK | Strong | C and Rust allocate what the code says. Go allocates what escapes, and a flag has to tell you which. |
| A fast server | Strong | Strong | OK | C and Go reach 800,000 with pipelining. Rust needs a read lock to match on reads and stays behind on writes. |
| Why it was slow | OK | Strong | Weak | The Go runtime hides lock handoff from the kernel. Rust's standard mutex is the platform's, and macOS makes it expensive. |
| Honest numbers | Strong | Strong | Strong | redis-benchmark and the same command shape treat all three alike. |
| 14 Shipping | ||||
| Cross-compiling | OK | Strong | OK | C needs zig and gets a small static binary. Go needs two variables, and Rust needs a target, a linker and zig. |
| Static linking | OK | Strong | OK | C and Rust choose musl. Go links nothing by default. |
| Binary size | Strong | Weak | OK | 45 KB, 2.5 MB and 521 KB for the same server. |
| Options and config | OK | Strong | OK | C has getopt and nothing for a file, and Go has flag, json and slog in the box. Rust has the arguments and a crate for the rest. |
| Windows | Weak | Strong | Strong | The C code is Unix-only. Go and Rust build for Windows unchanged. |
| Knowing the platform | Weak | Strong | OK | C sees every difference and Go hides all of them. Rust hides most and leaks one through the mutex. |
| A release | Weak | Strong | Strong | C has no mechanism. Go and Rust read a tag or a version field and install from a path. |
| 15 Talking to C | ||||
| Calling C | Strong | OK | Strong | Go pays a stack switch per call and loses static builds. C and Rust pay nothing per call. |
| Being called from C | Strong | Weak | Strong | A Go archive carries a runtime that must start. A Rust archive is a plain library. |
| Safety at the border | Weak | OK | OK | C checks nothing anywhere. Go and Rust check their own side and trust the declaration. |
| A struct on the border | Strong | OK | Strong | C owns the rule. Go follows it under C names, and Rust follows it with repr(C) and can assert the numbers. |
| A union on the border | OK | Weak | OK | C reads the field the tag names. Go converts the bytes itself, and Rust names the field and marks the read unsafe. |
| Making the rules visible | Weak | OK | Strong | C has comments and Go has a documented pointer rule. Rust puts unsafe in the signature. |
| Build cost of C code | Strong | Weak | OK | Go needs cgo on for every target. Rust needs the cc crate and a compiler for the C part only. |
One hundred and one rows. C was strong in 28, ok in 29 and weak in 43. Go was strong in 58, ok in 30 and weak in 13. Rust was strong in 57, ok in 37 and weak in 7. One row for C had no mark, because a forgotten lock cannot happen on one thread. The thorough edition added rows for abstraction, generics, sum types and iteration in chapters 7 and 8. It added rows for signals, background work, platform code and the structs on the border in the chapters after. Those sums are less interesting than where the marks fall, because each language's weak rows belong to one or two families.
C: the price of seeing everything#
Almost every weak row for C is the same row. Memory that must be freed by the right code at the right time. An error that must be checked by the next line or is lost. A platform difference that the program must know about, because nothing stands between it and the operating system. A value of several kinds that is a tag and a union the compiler never checks against each other. A lock, a condition variable and an atomic that a comment, not a type, says how to use. Those are not five weaknesses. They are one property, that C hides nothing, seen from five sides.
The same property fills C's strong rows. The binary was forty-five kilobytes. The profile read like the source, because the source is what ran. The sanitizers found the use-after-free in chapter 12 in a second, because they watch the real memory operations. And at the border in chapter 15, C was not a language calling another. It was the border. Every other language's door is shaped like a C function, and that will not change in your career.
Choose C when the program must be small, or must run where no runtime fits. Choose it when the program is itself the thing other languages will call, or when the platform is the subject. An operating system, a C library, a device driver, a codec, an interpreter that other languages embed. Choose it knowing that the compiler checks types and nothing else. The chapter 12 tools are not optional in C. They are the part of the compiler that was left out.
Go: the server language#
Go's strong rows cluster around the task this book set. A server that talks to many
clients, in one word, go, with a runtime that schedules them across every
core. A test runner, a fuzzer, a race detector and a profiler in the box, with no
decision to make about any of them. A channel that is both the message and the lock, which made the fsync thread of chapter 11 the shortest of its three versions. One command to build for any machine, and a binary
that needs nothing. The chapter 9 server was the shortest of the four. It was also the one that used every core without a line of code written for the purpose.
Go's weak rows are the cost of the runtime and the cost of simplicity. The binary is 2.5 megabytes before the program starts. Calling C costs a stack switch and the one-file freedom. The language has no way to say that a function can fail except a second return value. Chapter 3 showed what happens when the caller ignores it. The forgotten lock in chapter 9 compiled and ran, and only the race detector, run on purpose, said anything. A value of several kinds is an interface and a type switch, and the compiler does not say when a case is missing. The parser allocated nine times a request until a flag said where. What lives on the heap is the compiler's decision and not the programmer's. Go trusts the programmer to check every error and every lock. It gives better tools than C for finding the ones they missed, and no more protection at compile time.
Choose Go when the program is a network service, a command-line tool, or anything a team must read, build and deploy with the least effort. Choose it when the people who will maintain it are many, or new, or both. Go has one way to do most things, and the formatter enforces it. Do not choose it for a library that C or Python will embed, or for a device with a megabyte of flash. Do not choose it when correctness depends on the compiler catching a race.
Rust: the compiler as the reviewer#
Rust is strong in the rows that were weak for C. The hash table freed itself. The lost
error could not be lost, because a Result that is dropped is a warning and a
? is a line. The forgotten lock did not compile. The data race did not
compile. The value of several kinds was an enum, and a match that forgot a kind did not compile either. The parser that borrowed its arguments from the buffer could not outlive it, because a lifetime said so. At the border in chapter 15, the unsafe block was a labelled
boundary with a written argument, not a mood. Rust is what C would be if the compiler
checked the comments.
Rust's weak rows are the cost of that checking, and one surprise. The compiler is slower. A clean release build of chapter 15 took 19 seconds in Rust, 8 in Go and 1 in C. The chapter 12 tools took the longest to run. The chapter 13 profile was the hardest to read, because the sampler counted fifty sleeping threads and buried the one line that mattered. The mutex in chapter 13 was a different object on macOS and on Linux, and the standard library did not say so. And in chapter 6, writing a hash table was harder than in Go, and harder than the rest of Rust had been. A structure that owns its entries and lends references to them is exactly what the borrow checker was built to question. That one is a real cost. Rust is at its most awkward when writing the data structures it is best at using.
Choose Rust when the program must be fast and must be correct. Choose it when the people who will run it are not the people who wrote it. A database engine, a proxy, a parser for untrusted input, a command-line tool that must never crash. A library that will be embedded in C, in Python or in a browser. Choose it knowing that the first month is spent
arguing with the compiler, and that the argument is the point. Do not choose it for a program that will be rewritten next quarter, or for a team that has no time to learn it. A half-learned Rust is slower to write than Go, and a program full of unwrap() crashes when Go would have returned an error.
When the answer is none of them#
This book chose a server so that all three languages would be in their element. Many programs are not servers. A script that runs once, a report, a prototype that exists to be thrown away. A program whose whole cost is the time of the person writing it. For those, a language with a garbage collector, a large library and no build step is the right choice. That it runs many times slower than C does not appear in any row that matters. Python, JavaScript, Ruby and their relatives exist for this, and a systems programmer who refuses them is choosing the wrong tool out of pride.
The honest version of the question is not "which language is best". It is "what does this program need to be true, and which compiler will check it". C checks the types. Go checks the types and watches the running program. Rust checks the types, the lifetimes and the sharing. Each check has a price in time and in freedom, and the right language is the one whose price buys a guarantee the program needs.
What this book did not do#
A list, so that the scorecard is read with it. No chapter ran a Windows binary, so every
Windows row rests on documentation and a file command. The Rust
chapters used tokio and did not compare it with another runtime. The C server ran kqueue and epoll but never completion ports, because the book has no Windows machine. ThreadSanitizer ran the C tests on Linux only, because on macOS it defers a signal handler for ever. The garbage collector was measured by its count of collections and its memory, not by its pauses, because kv never allocated enough to show one. No chapter sharded the Rust store, so its write number in chapter 13 stayed where the mutex left it. And the three appendices teach the long tail of each language in two sentences each, which is a reminder and not a lesson. Each of those would change a row or
add one.
Where to go from here#
kv is small enough to understand and large enough to extend. The next features, in rough order of difficulty, are more data types, replication to a second server, and a cluster that shares keys by hash. Each is a chapter this book did not write. Each is a chance to learn whether the scorecard above survives contact with a harder problem. Pick the language whose weak rows worried you least and build the feature. Then build it again in the language whose weak rows worried you most. The second build is where the learning is.
Appendix A: Rosetta cheat card#
The constructs kv used most, side by side. Each block shows the same small thing in C, Go and Rust, in the form the chapters settled on. Use the Follow buttons at the top of the page to put your language first.
A value and a type#
int n = 42;
const char *name = "kv";
size_t len = strlen(name);
uint64_t big = 1ULL << 40;
bool ok = n > 0; /* stdbool.h */n := 42
name := "kv"
length := len(name)
var big uint64 = 1 << 40
ok := n > 0let n = 42;
let name = "kv";
let len = name.len();
let big: u64 = 1 << 40;
let ok = n > 0;
let mut count = 0; // mutableA function that can fail#
/* 0 on success, -1 and errno on failure */
int kv_open(const char *path, int *fd_out) {
int fd = open(path, O_RDONLY);
if (fd < 0) return -1;
*fd_out = fd;
return 0;
}
int fd;
if (kv_open("kv.aof", &fd) < 0) {
perror("open");
}func open(path string) (*os.File, error) {
f, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("open: %w", err)
}
return f, nil
}
f, err := open("kv.aof")
if err != nil {
log.Fatal(err)
}fn open(path: &str) -> io::Result<File> {
let f = File::open(path)?;
Ok(f)
}
let f = match open("kv.aof") {
Ok(f) => f,
Err(e) => {
eprintln!("open: {e}");
std::process::exit(1);
}
};A struct with a function on it#
typedef struct {
char *data;
size_t len;
} kv_buf;
void kv_buf_clear(kv_buf *b) {
b->len = 0;
}
kv_buf b = {0};
kv_buf_clear(&b);type Buf struct {
data []byte
}
func (b *Buf) Clear() {
b.data = b.data[:0]
}
var b Buf
b.Clear()struct Buf {
data: Vec<u8>,
}
impl Buf {
fn clear(&mut self) {
self.data.clear();
}
}
let mut b = Buf { data: Vec::new() };
b.clear();Bytes and strings#
/* a string is bytes with a zero at the end */
char s[] = "PING";
size_t n = strlen(s); /* 4 */
if (memcmp(s, "PING", 4) == 0) { }
char copy[16];
snprintf(copy, sizeof copy, "%s", s);s := "PING" // immutable
b := []byte(s) // a copy, mutable
if string(b) == "PING" { }
if bytes.Equal(b, []byte("PING")) { }
n := len(b) // 4, in byteslet s = "PING"; // &str, UTF-8
let b: &[u8] = s.as_bytes(); // no copy
let owned: Vec<u8> = b.to_vec(); // a copy
if b == b"PING" { }
let n = b.len(); // 4A growable array#
size_t cap = 4, len = 0;
int *xs = malloc(cap * sizeof *xs);
if (len == cap) {
int *p = realloc(xs, 2 * cap * sizeof *xs);
if (p == NULL) { free(xs); return -1; }
xs = p;
cap *= 2;
}
xs[len++] = 7;
free(xs);xs := make([]int, 0, 4)
xs = append(xs, 7)
for i, x := range xs {
fmt.Println(i, x)
}
// nothing to freelet mut xs = Vec::with_capacity(4);
xs.push(7);
for (i, x) in xs.iter().enumerate() {
println!("{i} {x}");
}
// freed when xs goes out of scopeA map#
/* no map in the standard library:
write one (chapter 6) or use a library */
kv_store_set(&s, key, klen, val, vlen);
kv_entry *e = kv_store_find(&s, key, klen, now);
if (e == NULL) { /* missing */ }
kv_store_del(&s, key, klen);m := map[string][]byte{}
m["k"] = []byte("v")
if v, ok := m["k"]; ok { use(v) }
delete(m, "k")let mut m: HashMap<Vec<u8>, Vec<u8>> = HashMap::new();
m.insert(b"k".to_vec(), b"v".to_vec());
match m.get(b"k".as_slice()) {
Some(v) => { /* found */ }
None => { /* missing */ }
}
m.remove(b"k".as_slice());Heap memory, and who frees it#
kv_entry *e = malloc(sizeof *e);
if (e == NULL) return -1;
e->key = malloc(klen);
/* ... */
free(e->key); /* in the same order */
free(e); /* you decided, you free */e := &entry{key: make([]byte, klen)}
// the collector frees e when nothing
// points to it any morelet e = Box::new(Entry { key: vec![0; klen] });
// freed at the end of the scope that owns it
let moved = e; // ownership moves, e is gone
drop(moved); // or free it earlyA thread and a shared counter#
static int count;
static pthread_mutex_t mu = PTHREAD_MUTEX_INITIALIZER;
void *work(void *arg) {
pthread_mutex_lock(&mu);
count++;
pthread_mutex_unlock(&mu);
return NULL;
}
pthread_t t;
pthread_create(&t, NULL, work, NULL);
pthread_join(t, NULL);var mu sync.Mutex
var count int
var wg sync.WaitGroup
wg.Go(func() {
mu.Lock()
count++
mu.Unlock()
})
wg.Wait()let count = Arc::new(Mutex::new(0));
let c = Arc::clone(&count);
let t = thread::spawn(move || {
*c.lock().unwrap() += 1;
// unlocked here, when the guard drops
});
t.join().unwrap();A value that is one of several kinds#
typedef struct {
enum { KV_STR, KV_LIST } kind;
union { kv_bytes str; kv_list list; } as;
} kv_value;
switch (v->kind) { /* the compiler does not check kind against as */
case KV_STR: use(v->as.str); break;
case KV_LIST: use(v->as.list); break;
}var v any = []byte("hi") // any value, with its type inside
switch val := v.(type) {
case []byte:
use(val)
case *List:
use(val)
default: // no error if a case is missing
}enum Value { Str(Vec<u8>), List(VecDeque<Vec<u8>>) }
match v { // every variant, or it does not compile
Value::Str(s) => use_str(s),
Value::List(l) => use_list(l),
}Calling a function chosen at run time#
typedef bool (*kv_handler)(const kv_cmd *, kv_store *, kv_buf *);
static const struct { const char *name; kv_handler fn; } commands[] = {
{"GET", kv_cmd_get}, {"SET", kv_cmd_set},
};
for (size_t i = 0; i < 2; i++)
if (strcmp(commands[i].name, name) == 0)
return commands[i].fn(&cmd, store, &out);type Storer interface { // satisfied by any type with these methods
Get(key []byte) ([]byte, bool)
Set(key, val []byte)
}
var handlers = map[string]func(args [][]byte, s Storer) error{
"GET": get, "SET": set,
}
return handlers[name](args, store)trait Store { // implemented by name, not by shape
fn get(&self, key: &[u8]) -> Option<&[u8]>;
fn set(&mut self, key: Vec<u8>, val: Vec<u8>);
}
type Handler = fn(&Command, &mut dyn Store) -> Reply; // dyn: a vtable
const COMMANDS: &[(&[u8], Handler)] = &[(b"GET", get), (b"SET", set)];Walking every key#
static void print_key(const char *k, size_t n, void *ctx) {
fprintf(ctx, "%.*s\n", (int)n, k);
}
kv_store_each(store, print_key, stdout); /* a callback and a void * */for k := range s.Keys() { // Keys returns an iter.Seq[[]byte]
fmt.Println(string(k))
}for k in store.keys() { // keys returns impl Iterator<Item = &[u8]>
println!("{}", String::from_utf8_lossy(k));
}
let n = store.keys().filter(|k| k.starts_with(b"user:")).count(); // lazyHanding work to another thread#
pthread_mutex_lock(&s->mu); /* a flag under a mutex, and a signal */
s->wanted = true;
pthread_cond_signal(&s->cv);
pthread_mutex_unlock(&s->mu);
/* the thread: while (!s->wanted) pthread_cond_wait(&s->cv, &s->mu); */jobs := make(chan int, 10)
go func() {
for j := range jobs { // ends when jobs is closed and empty
work(j)
}
}()
jobs <- 1
close(jobs)let (tx, rx) = mpsc::channel();
thread::scope(|s| {
s.spawn(move || for j in rx { work(j) }); // ends when tx is dropped
tx.send(1).unwrap();
drop(tx);
});Stopping on Ctrl-C#
static volatile sig_atomic_t stopped;
static void on_signal(int sig) { stopped = sig; write(pipe_w, "s", 1); }
struct sigaction sa = {.sa_handler = on_signal, .sa_flags = SA_RESTART};
sigaction(SIGINT, &sa, NULL);
/* the loop: poll on pipe_r as well, and stop when it is readable */ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()
select {
case <-ctx.Done(): // the signal arrived
case <-work:
}tokio::select! {
_ = tokio::signal::ctrl_c() => break, // the signal arrived
accepted = listener.accept() => serve(accepted?),
}
// with std: block the signals, and sigwait on one threadReading a whole file#
FILE *f = fopen("kv.aof", "rb");
if (f == NULL) return -1;
char buf[4096];
size_t n;
while ((n = fread(buf, 1, sizeof buf, f)) > 0) {
/* use buf[0..n] */
}
fclose(f);data, err := os.ReadFile("kv.aof")
if err != nil {
return err
}
// data is the whole filelet data = std::fs::read("kv.aof")?;
// data: Vec<u8>, the whole filePrinting#
printf("%s=%zu (%d)\n", key, len, n);
fprintf(stderr, "kv: %s\n", strerror(errno));fmt.Printf("%s=%d (%d)\n", key, len(key), n)
fmt.Fprintf(os.Stderr, "kv: %v\n", err)
log.Printf("listening on %s", addr)println!("{key}={} ({n})", key.len());
eprintln!("kv: {err}");
println!("{:?}", bytes); // debug formBuild, test, check#
make # build
make test # your runner
make SANITIZE=-fsanitize=address,undefined
clang-format -i src/*.c
make dist # zig cc, per targetgo build ./cmd/kv
go test ./...
go test -race ./...
go vet ./... && gofmt -l .
GOOS=linux GOARCH=arm64 go build ./cmd/kvcargo build --release
cargo test
cargo clippy -- -D warnings
cargo fmt
cargo zigbuild --target x86_64-unknown-linux-muslAppendix B: Glossary#
The terms the book uses, each with the section that explains it. The first use of a term in each chapter links here.
| Term | Meaning | See |
|---|---|---|
| ABI | Application binary interface. The convention for how a compiled function receives its arguments and returns its result. | The border is a C function |
| AddressSanitizer | A compiler mode that watches every memory access and reports a use after free or an access outside a block. | C: a runner you write, and a sanitizer that watches |
| AOF | Append-only file. kv's log of every write, replayed at start to rebuild the store. | C: a file descriptor, a write, and a thread for the fsync |
| async | Rust code that pauses at an await point instead of blocking a thread, run by a runtime such as tokio. | Rust once more: tasks on tokio |
| atomic | An operation on a shared variable that no other thread can see half done, with a memory order that says what else it promises. | C again: the second thread |
| borrow checker | The part of the Rust compiler that tracks who may read or change a value and for how long. | Rust: a value has one owner |
| build constraint | The comment on the first line of a Go file, //go:build, that says which systems or settings the file is for. | Go's platform files: a line at the top |
| bulk string | The RESP form of a byte string. A dollar sign, a length, a line break, the bytes, a line break. | C: a buffer, a parser and three answers |
| cfg | The Rust attribute, #[cfg(...)], that keeps an item out of the build on every system or setting it does not name. | Rust's platform code: an attribute on the item |
| cgo | The part of the go tool that lets Go code call C and be called from it. | Go: cgo |
| chaining | A hash table design where each bucket holds a linked list of the entries that hash to it. | C: an array of chains, and every pointer accounted for |
| channel | A queue that goroutines or threads send values through, which in Go and Rust is also the lock. | Go again: fifty workers and three channels |
| clippy | The Rust linter. It reports patterns that compile but are slower, longer or riskier than they need to be. | One style, enforced by a tool |
| closure | A function literal that captures variables from the scope around it. | Go: two kinds of string, and a copy between them |
| condition variable | A place a thread sleeps until another thread signals it, always inside a loop that checks the condition again. | C again: the second thread |
| context | The Go value that carries a cancellation, a deadline and request values down a chain of calls. | context in one paragraph |
| crate | A Rust package. A library crate, a binary crate, or both, described by Cargo.toml. | Rust: a module is a file you must announce |
| cross-compiling | Building on one machine a binary that runs on another kind of machine. | C: zig as the cross-compiler |
| cursor | A number the client sends back so that the server can continue a walk where it stopped. | C: a tag and a union |
| data race | Two threads touching the same memory at the same time, one of them writing, with no lock between them. | Go: one word, and one lock |
| deadline | The moment a key expires, stored with the key as a reading of the monotonic clock. | C: a deadline on the entry, and a sweep with a cursor |
| dynamic linking | Linking where the binary holds only the name of a library and finds it on the machine at start. | What a binary carries |
| epoll | The Linux call that waits for many sockets at once. kqueue on macOS and completion ports on Windows do the same job. | C: one thread, and a poller to say who is ready |
| escape analysis | The Go compiler's decision, for each value, whether it may live on the stack or must go on the heap. | Escape analysis |
| event loop | One thread that asks which sockets are ready and serves each in turn. | C: one thread, and a poller to say who is ready |
| extern block | The Rust declaration of functions that live in another language, marked unsafe since the 2024 edition. | Rust: extern and unsafe |
| file descriptor | The small integer a Unix system gives a program for an open file or socket. | C: four calls, and a way to clean up |
| flame graph | A picture of a profile in which each box is a function and its width is the share of samples through it. | Taking a number |
| fsync | The system call that makes the operating system write a file's bytes to the disk before it returns. | What a crash takes |
| fuzzing | Testing with generated inputs that a tool mutates toward the paths it has not yet run. | Fuzzing, three ways, one engine |
| garbage collector | The part of the Go runtime that finds memory nothing points to and frees it. | Go: the same table, with nothing to free |
| generic | A function or type written once for many types, with type parameters that the compiler fills in. | Go: the same table, with nothing to free |
| goroutine | A function the Go runtime runs on its own small stack, a few kilobytes, scheduled over the real threads. | Go: one word, and one lock |
| goroutine leak | A goroutine that waits for a message or a reply that never comes, and lives until the process ends. | Go again: fifty workers and three channels |
| hash table | An array of buckets, with a hash of the key choosing the bucket. The store under GET and SET. | C: an array of chains, and every pointer accounted for |
| header | A C file of declarations, included by every file that uses them. | C: headers, static, and a Makefile that knows about both |
| header guard | The #ifndef at the top of a header that stops it being read twice. | C: headers, static, and a Makefile that knows about both |
| interface | A Go type that names methods, satisfied by any type that has them, without being told. | Go: a folder is a package, and a capital letter is an export |
| iterator | A value with a next method that yields one item at a time, which adaptors wrap and nothing runs until asked. | Rust: an enum with data in each variant |
| lazy expiry | Checking a key's deadline when the key is read, and deleting it then. | C: a deadline on the entry, and a sweep with a cursor |
| libc | The C standard library, which also wraps the system calls. glibc and musl are two of them. | What a binary carries |
| libFuzzer | The in-process, coverage-guided fuzzing engine in clang, which cargo-fuzz drives for Rust. | C: a runner you write, and a sanitizer that watches |
| lifetime | The name, written 'a, that says how long a reference in a type is valid, which the compiler checks and never extends. | Lifetimes, in the one place kv needs them |
| linker | The tool that joins compiled files and libraries into one binary. | What a binary is |
| miri | An interpreter for Rust that checks unsafe code for undefined behaviour while it runs. | Rust: the compiler already ran, and the tools check the rest |
| module | In Go, a set of packages with a version. In Rust, a file or block of code with its own names. | Go: a folder is a package, and a capital letter is an export |
| monotonic clock | A clock that only goes forward, unaffected by the user changing the time. | C: a deadline on the entry, and a sweep with a cursor |
| musl | A small C library for Linux that is designed to be linked statically. | What a binary carries |
| mutex | A lock. One thread holds it at a time, and the others wait. | Go: one word, and one lock |
| ownership | The Rust rule that every value has one owner, and is freed when the owner goes away. | Rust: a value has one owner |
| package | In Go, a folder of files that share a name. In Rust, a crate with a Cargo.toml. | Go: a folder is a package, and a capital letter is an export |
| pipelining | Sending many commands before reading any reply, so that the network wait is paid once. | Taking a number |
| poll | The portable call that waits until one of a set of sockets is ready. | C: one thread, and a poller to say who is ready |
| poller | The loop's question to the kernel, which socket is ready, asked with poll, kqueue or epoll. | C again: poll, kqueue and epoll behind one header |
| profile | A record of where a running program spent its time, taken by sampling the stack. | C: a clock, a loop and five flags |
| race detector | The Go tool, go test -race, that watches memory access and reports a data race it sees. | Go: the runner is in the box, and so is the fuzzer |
| raw pointer | A Rust pointer, *const T or *mut T, with the borrow checker's guarantees removed, which only an unsafe block may read through. | The five powers, and the pointer they are for |
| region | A marked part of a listing that the book pulls in by name, so the text and the code cannot drift. | The code |
| repr(C) | The Rust attribute that lays a struct or union out as a C compiler would, so that the bytes mean the same on both sides. | The five powers, and the pointer they are for |
| RESP | The Redis serialization protocol. Arrays of bulk strings in, and simple strings, errors, integers, bulk strings or nil out. | C: a buffer, a parser and three answers |
| Result | The Rust type that is either a value or an error, and must be looked at. | Rust: a Result at every step |
| sanitizer | A compiler mode that adds checks for memory errors, undefined behaviour or data races. | C: a runner you write, and a sanitizer that watches |
| scoped thread | A Rust thread spawned inside thread::scope, which may borrow local data because the scope joins it before returning. | Rust again: the load generator with scoped threads |
| self-pipe trick | A signal handler that writes one byte into a pipe the event loop waits on, so that a signal becomes an event. | C: a signal, a flag and a pipe |
| semantic versioning | Three numbers, major, minor and patch, that say what kind of change a release holds. | Versions and releases |
| Send and Sync | The two Rust marker traits that say a value may move to another thread, or be shared between threads, derived by the compiler. | Send and Sync |
| static linking | Linking where the binary carries its libraries inside and needs nothing from the machine. | What a binary carries |
| sum type | A value that is one of several kinds, with a tag that says which. A tagged union in C, an interface in Go, an enum in Rust. | C: a tag and a union |
| sweep | The background pass that deletes expired keys nobody has asked for. | C: a deadline on the entry, and a sweep with a cursor |
| tokio | The Rust async runtime that the book's second Rust server runs on. | Rust once more: tasks on tokio |
| trait object | A value of unknown concrete type behind dyn Trait, called through a table of function pointers at run time. | Rust: a module is a file you must announce |
| type switch | The Go switch on the dynamic type inside an interface value, one case per type. | Go: any, and a switch on the type |
| undefined behaviour | A C or unsafe Rust operation the standard does not define, after which the compiler may do anything. | C: a runner you write, and a sanitizer that watches |
| union | A C or Rust type whose fields share the same bytes, so that only one is meaningful at a time. | A struct and a union on the border |
| unsafe | The Rust keyword that marks a block where the five extra powers are allowed, and the author takes responsibility. | Rust: extern and unsafe |
| X-macro | A list written once as a macro, reused with a different meaning of X for a table, a declaration or a call. | C: a runner you write, and a sanitizer that watches |
| zig | A toolchain that includes a C compiler and the C libraries of many targets, used here to cross-compile. | C: zig as the cross-compiler |
Appendix C: C in one chapter#
Every C concept the book teaches, in the groups of the concept index, with one to three sentences each. A concept that a chapter teaches links to that section. A concept kv never needed is taught here, and links to its official reference. Open this appendix six months after the book, when a construct looks familiar and its name has gone.
Types and values#
| Concept | What to remember | Where |
|---|---|---|
| integer types and stdint.h | int, long and friends have platform sizes; int32_t and uint64_t say exactly what they are. | C: an array of chains, and every pointer accounted for c/ch06/kv.c |
| char signedness | Plain char may be signed or unsigned by platform, so a byte over 127 can become negative. | C: an array of chains, and every pointer accounted for c/ch06/kv.c |
| integer promotion and the usual conversions | Small types widen to int before arithmetic, so two chars add as ints. When a signed and an unsigned operand meet, the signed one becomes unsigned, and -1 is suddenly the largest value. Compare sizes as size_t and counts as int, and never mix them in one expression. | reference |
| signed overflow is undefined | Overflowing a signed integer is undefined behaviour, so the compiler may assume it never happens. | Three answers to the largest integer plus one c/ch05/kv.c |
| unsigned wrap | Unsigned arithmetic wraps around modulo 2^n, by definition, which is how a hash function is allowed to overflow. | Three answers to the largest integer plus one c/ch05/kv.c |
| bool | stdbool.h since C99, a keyword in C23. Any non-zero value is true. | C17 against C23 c/ch14/extras/c23.c |
| enum | Named integer constants, with no check that a value is one of the names. | switch, and the cases that run into each other c/ch04/kv.c |
| sizeof | The size of a type or an object in bytes, computed at compile time, with no parentheses needed for an object. | What a struct is made of, and an arena c/ch06/generic.c |
| size_t, ssize_t and ptrdiff_t | size_t is the unsigned type of a size and an index, and sizeof yields it. ssize_t is the signed one that read and write return, so that -1 can mean failure. ptrdiff_t is the type of one pointer minus another. | reference |
| casts | An explicit conversion the compiler will not question, so each one is a claim you must be able to defend. | What C does instead of generics c/ch06/generic.c |
| const and const-correctness | A promise not to write through a pointer, checked by the compiler and carried through every call. | The parser stops copying c/ch10/src/resp.h |
| volatile | Every read and write must happen, because something outside the program can change the value. | C: a signal, a flag and a pipe c/ch10/src/stop.h |
| static, extern and linkage | static hides a name inside one file; extern says a name is defined elsewhere. | C: two files, one binary hello/c/greet.c |
| translation units | Each .c file compiles alone, with the headers pasted in, and the linker joins the results. | C: two files, one binary hello/c/units.c |
| storage duration | Automatic variables live until the block ends, static ones for the whole run, allocated ones until free. | Where a value lives, and for how long c/ch06/kv.c |
| stack and heap | Locals live on the stack and vanish with the function; malloc gives memory that outlives it. | Where a value lives, and for how long c/ch06/kv.c |
| zero initialisation | = {0} sets every member to zero, and a static variable is zero without it. | C: an array of chains, and every pointer accounted for c/ch06/kv.c |
| designated initialisers | {.fd = -1, .len = 0} names the members, so the order of the struct does not matter. | C again: poll, kqueue and epoll behind one header c/ch09/src/poller_epoll.c |
| compound literals | (struct pollfd){fd, POLLIN, 0} builds a value in place without a named variable. | C: an array of chains, and every pointer accounted for c/ch06/kv.c |
Pointers and memory#
| Concept | What to remember | Where |
|---|---|---|
| pointers | An address plus a type, and the type says how many bytes one step is. | C: an array of chains, and every pointer accounted for c/ch06/kv.c |
| pointer arithmetic | p + 1 moves by one element, not one byte, so the type of p decides the step. The result is only defined inside the same object, or one past its end, and the chapter 4 parser stays inside its buffer by comparing against end. | reference |
| arrays decay to pointers | An array name used as a value becomes a pointer to its first element, and the length is gone. That is why a function takes a pointer and a length, and why sizeof an array parameter is the size of a pointer. | reference |
| pointer to pointer | How a function gives its caller a new pointer, as in a grow function that may move a buffer. | C: an array of chains, and every pointer accounted for c/ch06/kv.c |
| void pointer | A pointer to anything, which malloc returns and a callback context uses. | What C does instead of generics c/ch06/generic.c |
| NULL and nullptr | The null pointer. C23 adds nullptr with its own type. | C17 against C23 c/ch14/extras/c23.c |
| malloc, calloc, realloc and free | Ask for bytes, get them zeroed, grow them in place or elsewhere, give them back. | Growing, measuring and slicing bytes c/ch05/kv.c |
| ownership conventions | A comment on every pointer says who frees it, because the compiler will not. | The parser stops copying c/ch10/src/resp.h |
| alignment and offsetof | Each type has an alignment, structs get padding to keep it, and offsetof finds a member's position. | C: an array of chains, and every pointer accounted for c/ch06/generic.c |
| flexible array members | A struct whose last member is an array of unknown size, declared as data[], is allocated in one block with room after its header. Redis stores its strings this way, with the length and the bytes in one allocation. | reference |
| restrict | A promise that two pointers do not overlap, which lets the compiler optimise. | The optimisation levels, measured c/ch13/bench/bench_copy.c |
| strict aliasing | Reading an object through a pointer of a different type is undefined, except through char or unsigned char. The compiler may assume an int and a float never overlap, which is why a memcpy is the way to reinterpret bytes. | reference |
| use after free, double free and leaks | The three memory bugs C lets you write, and what each looks like under AddressSanitizer. | C: an array of chains, and every pointer accounted for c/ch06/kv.c |
| arenas | Allocate many small things from one block and free them all at once. | What a struct is made of, and an arena c/ch06/generic.c |
Structs and unions#
| Concept | What to remember | Where |
|---|---|---|
| struct | Members laid out in order, with padding, passed by value unless you pass a pointer. | C: an array of chains, and every pointer accounted for c/ch06/kv.c |
| typedef | A new name for a type, which is how kv_store stands for struct kv_store. | C: an array of chains, and every pointer accounted for c/ch06/kv.c |
| opaque pointers | A header declares struct kv_store without its members, so callers cannot touch them. | An opaque type, and a struct of functions c/ch07/src/store.h |
| union | Members that share the same bytes, so only one is meaningful at a time. | C: a tag and a union c/ch08/src/store.h |
| tagged union | A union plus an enum that says which member is live: C's hand-built sum type. | C: a tag and a union c/ch08/src/store.h |
| bitfields | Members with a width in bits, packed by the compiler in an order it chooses. | C: a tag and a union |
Functions and control#
| Concept | What to remember | Where |
|---|---|---|
| switch and fallthrough | A jump table on an integer, where a case without break runs into the next one. | switch, and the cases that run into each other c/ch04/kv.c |
| do while | A loop that runs its body once before it tests the condition. The test runner's CHECK macro is built on one with a condition of 0, so that it runs exactly once. | reference |
| goto cleanup | One exit path at the end of a function that frees everything, reached by goto from each failure. | C: four calls, and a way to clean up c/ch03/kv.c |
| function pointers | A pointer to code, declared with the signature it points to, which is how C passes behaviour. | A table of functions c/ch05/kv.c |
| callbacks with a void context | A function pointer plus a void * that the callee passes back, which is C's closure. | An opaque type, and a struct of functions c/ch07/src/cmd.c |
| variadic functions | Functions that take any number of arguments, read with va_list, as printf does. | A function that takes any number of arguments c/ch04/kv.c |
| inline | A hint to paste the function body at each call, with linkage rules that surprise. | The optimisation levels, measured c/ch13/Makefile |
| static functions | A function only this file can call, which the compiler can warn about when unused. | C: headers, static, and a Makefile that knows about both c/ch07/src/buf.c |
| _Noreturn | Marks a function that never returns, such as one that prints an error and exits. | C again: options, and a function that does not come back c/ch14/src/main.c |
Preprocessor#
| Concept | What to remember | Where |
|---|---|---|
| #include | Pastes a file in place. Angle brackets search the system, quotes search here first. | C: headers, static, and a Makefile that knows about both c/ch07/src/main.c |
| header guards and #pragma once | The #ifndef at the top of a header that stops it being pasted twice. | C: headers, static, and a Makefile that knows about both c/ch07/src/buf.h |
| object and function macros | Text replacement before compilation, with or without arguments, with no types. | A table of functions c/ch05/kv.c |
| do { } while (0) | The wrapper that makes a macro of several statements behave like one statement, so that it can follow an if without braces and still take a semicolon. The CHECK macro in chapter 12 is one. | reference |
| X-macros | One list of entries that expands into a table, an enum and a set of names at once. | C: bytes, a length and an owner c/ch05/kv.c |
| #ifdef platform guards | Choose code at compile time by platform, as in __linux__ against __APPLE__. | C again: poll, kqueue and epoll behind one header c/ch09/src/poller.h |
| _Static_assert | A check the compiler runs, such as that a struct has the size you think. | C: a runner you write, and a sanitizer that watches c/ch12/tests/test_resp.c |
| _Generic | A compile-time switch on the type of an expression, the nearest C gets to overloading. | What C does instead of generics c/ch06/generic.c |
| predefined macros | __FILE__, __LINE__, __func__ and the platform and compiler names. | C again: poll, kqueue and epoll behind one header c/ch09/src/poller.h |
| __attribute__ and compiler extensions | Non-standard annotations such as format checking and cleanup, understood by gcc and clang. | C again: options, and a function that does not come back c/ch14/src/main.c |
Strings and I/O#
| Concept | What to remember | Where |
|---|---|---|
| NUL-terminated strings | A string is bytes up to the first zero, so a string cannot hold a zero and its length is a walk. | C: bytes, a length and an owner c/ch05/kv.c |
| strlen, memcpy and memcmp | The three calls behind most string code, and what each assumes about its arguments. | C: bytes, a length and an owner c/ch05/kv.c |
| the strncpy trap | strncpy fills the whole destination and may leave no terminating zero when the source is as long as the limit. A bounded copy of a C string wants snprintf, and a copy of bytes with a length wants memcpy, which kv uses throughout. | reference |
| snprintf | Formats into a buffer of a known size and never overruns it, returning what it wanted to write. | Growing, measuring and slicing bytes c/ch05/kv.c |
| pointer and length discipline | Every buffer travels with its length, which is how kv handles bytes that may contain zero. | C: a buffer, a parser and three answers c/ch04/kv.c |
| stdio against read and write | FILE * buffers in user space; read and write are the system calls underneath. | C: a file descriptor, a write, and a thread for the fsync c/ch11/src/aof.c |
| errno, strerror and perror | A failed call returns -1 and sets a thread-local number, which strerror turns into words. | errno: the number a failed call leaves behind c/ch03/kv.c |
Concurrency and signals#
| Concept | What to remember | Where |
|---|---|---|
| pthreads and C11 threads | Create a thread with a function and a void *, join it to wait. | C again: the second thread c/ch11/src/syncer.c |
| mutex | pthread_mutex_lock and unlock around every access to shared data, with nothing to stop you forgetting. | C again: the second thread c/ch11/src/syncer.c |
| condition variable | Sleep until another thread signals, always inside a loop that rechecks the condition. | C again: the second thread c/ch11/src/syncer.c |
| atomics and memory order | stdatomic.h gives operations no other thread can see half done, with an ordering you choose. | C again: the second thread c/ch11/src/syncer.c |
| thread safety of libc | Most of libc is safe to call from threads; strtok, localtime and a few others are not. | Thread safety of the C library |
| signals and sigaction | The kernel interrupts the program to run a handler, which may only do a few safe things. | C: a signal, a flag and a pipe c/ch10/src/stop.c |
| sig_atomic_t and the self-pipe trick | A handler sets a flag or writes one byte to a pipe that the poll loop watches. | C: a signal, a flag and a pipe c/ch10/src/stop.c |
| SIGPIPE | A write to a peer that has gone raises a signal that kills the process unless ignored. | C: a signal, a flag and a pipe c/ch10/src/main.c |
| fork | Clone the process. The Unix way to start a program, which Windows does not have. | C again: the second thread |
Build and tooling#
| Concept | What to remember | Where |
|---|---|---|
| compiler flags and -O levels | -O0 for debugging, -O2 for release, and what the optimiser is allowed to assume. | The optimisation levels, measured c/ch13/Makefile |
| warnings as errors | -Wall -Wextra -Werror turns the compiler's advice into a gate. | C: from source to binary in four steps hello/c/Makefile |
| static libraries | An archive of object files that the linker copies from, made with ar. | C: a library that is easy to call c/ch15/Makefile |
| Make | Targets, prerequisites and recipes, with tabs, and a rule for how a .o comes from a .c. | Build tools, and the lack of one hello/c/Makefile |
| CMake | A generator of build files for larger projects, which most C libraries now use. | Build tools, and the lack of one |
| pkg-config | Asks an installed library for the flags to compile and link against it. | Dependencies in C: the package manager that is not there |
| AddressSanitizer and UndefinedBehaviorSanitizer | Compile-time instrumentation that reports memory errors and undefined behaviour as they happen. | C: a runner you write, and a sanitizer that watches c/ch12/Makefile |
| ThreadSanitizer | The sanitizer that reports data races, which cannot run together with AddressSanitizer. | C again: the second thread c/ch11/src/syncer.c |
| valgrind | Runs the unmodified binary under an emulator and reports leaks and bad accesses, on Linux. | C: a runner you write, and a sanitizer that watches |
| a debugger session | lldb or gdb: break, step, print, backtrace, and the -g flag that makes them useful. | A debugger session |
| clang-format | The formatter, driven by a .clang-format file, since C has no single style. | Build tools, and the lack of one |
| C17 against C23 | nullptr, constexpr, typeof, auto, #embed, attributes and true keywords, and how far clang and gcc support them. | C17 against C23 c/ch14/extras/c23.c |
Conventions#
| Concept | What to remember | Where |
|---|---|---|
| name prefixes | kv_ on every public name, because C has one namespace for the whole program. | C: headers, static, and a Makefile that knows about both c/ch07/src/buf.h |
| header hygiene | A header declares, includes only what it needs, and never defines a variable. | C: headers, static, and a Makefile that knows about both c/ch07/src/buf.h |
| error returns | Return a code or a bool and put the result in an out parameter, consistently. | C: bytes, a length and an owner c/ch05/kv.c |
| ownership comments | Who frees it, written next to the pointer, every time. | C: bytes, a length and an owner c/ch05/kv.c |
| const parameters | A const char * parameter promises the caller the bytes come back unchanged. | C: bytes, a length and an owner c/ch05/kv.c |
Appendix D: Go in one chapter#
Every Go concept the book teaches, in the groups of the concept index, with one to three sentences each. A concept that a chapter teaches links to that section. A concept kv never needed is taught here, and links to its official reference. Open this appendix six months after the book, when a construct looks familiar and its name has gone.
Basics#
| Concept | What to remember | Where |
|---|---|---|
| packages and imports | A folder is a package, an import is a path, and an unused import is an error. | Go: one command, one binary hello/go/main.go |
| zero values | Every variable starts as zero, nil or empty, so a struct is usable without a constructor. | Every variable starts at zero hello/go/main.go |
| := and var | Inside a function, := declares and assigns in one step and infers the type. var works anywhere and takes an explicit type or a zero value. A := in an inner scope can declare a new variable that shadows one outside, which is the usual way an err goes missing. | reference |
| const and iota | A const is untyped until it is used, so 1 << 40 is fine where an int would overflow. Inside one const block, iota counts from 0 on each line, which is how the aof package names its three policies. | reference |
| basic types and conversions | int is the machine word, every conversion is written out, and there is no implicit widening. | Three answers to the largest integer plus one go/ch05/strings_test.go |
| runes, strings, bytes and UTF-8 | A string is immutable bytes, range gives runes, and len counts bytes. | Bytes, runes and a builder go/ch05/strings_test.go |
| arrays against slices | An array has a fixed length in its type; a slice is a view of one with a length and a capacity. | Go: the same table, with nothing to free go/ch06/table.go |
| len, cap, append and aliasing | append writes into spare capacity, so two slices can share and overwrite the same array. | Bytes, runes and a builder go/ch05/exec.go |
| copy and the full slice expression | copy duplicates bytes, and s[a:b:c] caps the capacity to stop append from aliasing. | The parser stops copying go/ch10/internal/resp/resp.go |
| maps | Reading a nil map is fine and writing one panics; iteration order is random on purpose. | Go: two kinds of string, and a copy between them go/ch05/store.go |
| comma-ok | v, ok := m[k] separates a missing key from a zero value, and the same form works for assertions and channels. | Go: two kinds of string, and a copy between them go/ch05/store.go |
| structs and composite literals | Fields in order or by name, and a pointer to a literal with &. | Go: the same table, with nothing to free go/ch06/table.go |
| pointers against values | Everything is passed by value, so a function that must change the caller's struct takes a pointer. A method on a pointer receiver can change the value and a method on a value receiver gets a copy. The store's methods all take *Store. | reference |
| the forms of range | Over a slice, a map, a string, a channel, an integer since 1.22, and a function since 1.23. | A function that range can call go/ch08/internal/store/value.go |
| switch and type switch | No fallthrough by default, cases can be expressions, and switch v := x.(type) branches on a type. | Go: any, and a switch on the type go/ch08/internal/store/value.go |
| labeled break and continue | The way out of a nested loop, with a label on the outer one. | A function that range can call |
Functions and methods#
| Concept | What to remember | Where |
|---|---|---|
| multiple returns and named results | A function returns a value and an error together, and named results document them. | Go: any, and a switch on the type go/ch08/internal/cmd/cmd.go |
| variadic functions | f(args ...string) takes any number, and f(slice...) spreads one. | Bytes, runes and a builder go/ch05/exec.go |
| closures and capture | A function literal captures variables by reference, and since 1.22 each loop iteration has its own. | Go: two kinds of string, and a copy between them go/ch05/exec.go |
| defer | Runs at function exit, last in first out, with its arguments evaluated at the defer line. | Two rules of defer go/ch03/defer_test.go |
| panic and recover | A panic unwinds the stack; a deferred recover stops it, which is how one bad client does not kill a server. | panic and recover go/ch10/cmd/kv/main.go |
| methods and receivers | A function with a receiver, on a pointer when it changes the value, on a value when it does not. | Go: a folder is a package, and a capital letter is an export go/ch07/internal/store/store.go |
| method sets | A value of type T has the methods with value receivers, and a *T has those and the pointer-receiver ones too. That decides which of the two satisfies an interface: a *Store satisfies Storer, and a Store does not. | reference |
| method values | s.Get without the call is a function bound to s, which a dispatch table can hold. | A map of functions go/ch05/exec.go |
| embedding and promotion | An unnamed field lends its methods to the outer struct, as an embedded sync.Mutex does. | Embedding, and the nil that is not nil go/ch07/internal/cmd/cmd_test.go |
Interfaces#
| Concept | What to remember | Where |
|---|---|---|
| interfaces and implicit satisfaction | A set of method signatures, and any type with those methods satisfies it without saying so. | An interface is a set of methods go/ch07/internal/cmd/cmd.go |
| small interfaces | io.Reader, io.Writer, fmt.Stringer and error have one method each, and most code accepts them. | An interface is a set of methods go/ch07/internal/cmd/cmd.go |
| any | The empty interface, which every value satisfies, and which a type switch takes apart. | Go: any, and a switch on the type go/ch08/internal/store/value.go |
| type assertion | v, ok := x.(T) asks whether the dynamic value is a T, and without ok it panics. | Go: any, and a switch on the type go/ch08/internal/cmd/cmd.go |
| the nil interface trap | An interface holding a nil pointer is not nil, which makes a returned *MyErr look like an error. | Embedding, and the nil that is not nil go/ch07/internal/cmd/cmd_test.go |
| interface embedding | An interface may list other interfaces, and gets their methods. io.ReadWriter is io.Reader and io.Writer in one declaration, and a type that has both methods satisfies all three. | reference |
| when not to use an interface | Accept interfaces, return structs, and define one only when a second implementation exists. | Embedding, and the nil that is not nil go/ch07/internal/cmd/cmd.go |
Generics#
| Concept | What to remember | Where |
|---|---|---|
| type parameters | func Max[T cmp.Ordered](a, b T) T, since Go 1.18, with the type inferred at the call. | Go: the same table, with nothing to free go/ch06/table.go |
| constraints, comparable and any | An interface used as a bound, comparable for map keys, and a union of types with |. | Go: the same table, with nothing to free go/ch06/table.go |
| the slices, maps and cmp packages | Sort, search, clone and compare, written once with generics instead of per type. | Go: the same table, with nothing to free go/ch06/table_test.go |
| when to use generics | A container or an algorithm over many types, not a method that takes an interface. | Go: the same table, with nothing to free go/ch06/table.go |
Errors#
| Concept | What to remember | Where |
|---|---|---|
| the error interface | Anything with an Error() string method, and nil means no error. | An error is any value with an Error method go/ch04/resp.go |
| custom error types | A struct with fields the caller can read, returned as an error. | An error is any value with an Error method go/ch04/resp.go |
| wrapping with %w | fmt.Errorf("read config: %w", err) adds context and keeps the cause reachable. | An error is any value with an Error method go/ch04/resp.go |
| errors.Is, errors.As and errors.Join | Walk the wrapped chain for a sentinel, for a type, or combine several errors. | errors.Join go/ch12/internal/aof/aof.go |
| sentinel against typed errors | var ErrNotFound = errors.New(...) for a condition, a type when the caller needs fields. | An error is any value with an Error method go/ch04/resp.go |
| panics against errors | An error is a value for the caller; a panic is a bug in the program. | panic and recover |
Concurrency#
| Concept | What to remember | Where |
|---|---|---|
| goroutines | go f() runs f on a stack of a few kilobytes, multiplexed over the real threads. | Go: one word, and one lock go/ch09/cmd/kv/main.go |
| channels | A typed queue between goroutines; unbuffered ones hand off, buffered ones hold a few. | Go again: fifty workers and three channels go/ch09/cmd/kvload/main.go |
| close and range over a channel | The sender closes, the receiver's range ends, and a receive from a closed channel gives zero. | Go again: fifty workers and three channels go/ch09/cmd/kvload/main.go |
| nil channels | A nil channel blocks forever, which is a way to switch a select case off. | Go again: fifty workers and three channels go/ch09/cmd/kvload/main.go |
| select | Wait on several channels at once, with default for a non-blocking try and a timer for a timeout. | Go again: fifty workers and three channels go/ch09/cmd/kvload/main.go |
| sync.Mutex and RWMutex | Lock around shared data, with RLock for readers, and nothing stops you forgetting. | Go: one word, and one lock go/ch09/internal/store/store.go |
| sync.WaitGroup | Add before starting, Done when finished, Wait for all, and wg.Go since 1.25. | Go: one word, and one lock go/ch09/internal/store/store_test.go |
| sync.Once and sync.Pool | Run something exactly once, and reuse buffers instead of allocating. | The rest of the sync package |
| sync/atomic | Counters and flags that need no lock, as atomic.Int64. | The rest of the sync package |
| context | Cancellation, a deadline and request values that flow down every call. | context in one paragraph go/ch10/cmd/kv/main.go |
| worker pools | A fixed number of goroutines reading jobs from one channel. | Go again: fifty workers and three channels go/ch09/cmd/kvload/main.go |
| data races and the race detector | Two goroutines, one writing, no sync, and go test -race finds it. | Go: one word, and one lock go/ch09/internal/store/store_test.go |
| GOMAXPROCS and the scheduler | Goroutines run on P logical processors over M threads, and a blocking system call hands its P over. | The runtime in one paragraph |
| goroutine leaks | A goroutine blocked forever on a channel nobody will touch again. | Go again: fifty workers and three channels go/ch09/cmd/kvload/main.go |
Runtime and memory#
| Concept | What to remember | Where |
|---|---|---|
| the garbage collector | Concurrent mark and sweep, with short pauses and memory held until the next cycle. | Go: the same table, with nothing to free go/ch06/table.go |
| escape analysis | go build -gcflags=-m says which values move to the heap, and why. | Escape analysis go/ch13/internal/resp/resp.go |
| GOGC and GOMEMLIMIT | How often the collector runs, and a ceiling on the heap. | GOGC, GOMEMLIMIT and the Pool |
| pprof | CPU and memory profiles from a running program, read as a graph or a flamegraph. | The profiler is a package go/ch13/cmd/kv/main.go |
Tooling and packaging#
| Concept | What to remember | Where |
|---|---|---|
| go.mod and go.sum | The module path and its dependencies, and the hashes that pin them. | Go: one command, one binary hello/go/go.mod |
| semantic import versioning | A v2 module has /v2 in its import path, so two majors can coexist. | Go: a folder is a package, and a capital letter is an export |
| go get and replace | Add or upgrade a dependency, or point a module at a local folder. | Go: a folder is a package, and a capital letter is an export |
| workspaces | go.work lets several modules build together while you change them. | Rust: a module is a file you must announce |
| internal packages | A package under internal/ can only be imported from inside its parent. | Go: a folder is a package, and a capital letter is an export go/ch07/internal/store/store.go |
| go vet, gofmt and staticcheck | The checker, the one formatter, and the linter most teams add. | Go: one command, one binary |
| build tags and platform files | //go:build linux, or a _linux.go suffix, picks a file by platform. | Go's platform files: a line at the top go/ch09/internal/sys/limit_unix.go |
| go generate and go:embed | Run a tool from a comment, and compile a file into the binary. | Go again: flags, a file, a log and a version go/ch14/cmd/kv/main.go |
| table-driven tests | A slice of cases and one loop, the idiom every Go test file uses. | Go: a folder is a package, and a capital letter is an export go/ch07/internal/store/glob_test.go |
| subtests, t.Parallel and helpers | t.Run names a case, t.Parallel runs them together, t.Helper hides a helper from the report. | Go: the runner is in the box, and so is the fuzzer go/ch12/internal/resp/resp_test.go |
| benchmarks | func BenchmarkX(b *testing.B) with b.Loop, and go test -bench. | Go: nine allocations, and the flag that names them go/ch13/internal/resp/bench_test.go |
| fuzzing | func FuzzX(f *testing.F) with go test -fuzz, built into the toolchain. | Go: the runner is in the box, and so is the fuzzer go/ch12/internal/resp/resp_test.go |
| cross-compiling | GOOS and GOARCH, with CGO_ENABLED=0 for a static binary. | Go: two variables go/ch14/Makefile |
| cgo | import "C" and a preamble, with a cost on every call. | Go: cgo go/ch15/internal/cmatch/cmatch.go |
| the module proxy and vendoring | proxy.golang.org serves modules, and go mod vendor copies them into the repository. | Go: a folder is a package, and a capital letter is an export |
| go doc | Doc comments on exported names, read with go doc and on pkg.go.dev. | The documentation is in the source |
Standard library tour#
| Concept | What to remember | Where |
|---|---|---|
| fmt | Printf and its verbs: %v for anything, %q for a quoted string, %d and %s as in C, and %w to wrap an error in Errorf. Println adds spaces and a newline, and a type with a String method prints as it chooses. | reference |
| strconv | Atoi, ParseInt and Itoa, with errors, not panics. | Go: a reader that waits, and an error that names itself go/ch04/resp.go |
| strings and bytes | The same functions twice, for string and for []byte, plus strings.Builder. | Bytes, runes and a builder go/ch05/strings_test.go |
| bufio | A buffered reader and writer over any io.Reader or io.Writer. | Go: a reader that waits, and an error that names itself go/ch04/resp.go |
| io and os | The interfaces, and files, environment and arguments. | Go: the loop you did not have to write go/ch03/main.go |
| net | net.Listen, Accept and net.Conn, which is an io.ReadWriteCloser. | Go: the loop you did not have to write go/ch03/main.go |
| time | time.Time and Duration, the monotonic reading hidden inside Now, Timer and Ticker. | Go: a ticker, and a clock the test can hold still go/ch10/internal/store/store.go |
| encoding/json | Struct tags and Marshal, the most used package after fmt. | Go again: flags, a file, a log and a version go/ch14/cmd/kv/main.go |
| flag | Command-line flags with defaults and a generated usage. | Go again: flags, a file, a log and a version go/ch14/cmd/kv/main.go |
| log/slog | Structured logging with levels and key-value pairs. | Go again: flags, a file, a log and a version go/ch14/cmd/kv/main.go |
| os/signal | signal.Notify delivers a signal on a channel, and NotifyContext cancels a context. | Go: a context that a signal cancels go/ch10/cmd/kv/main.go |
Appendix E: Rust in one chapter#
Every Rust concept the book teaches, in the groups of the concept index, with one to three sentences each. A concept that a chapter teaches links to that section. A concept kv never needed is taught here, and links to its official reference. Open this appendix six months after the book, when a construct looks familiar and its name has gone.
Basics#
| Concept | What to remember | Where |
|---|---|---|
| let, mut and shadowing | A binding is immutable unless it says mut, which the compiler enforces. A second let with the same name shadows the first, and may change its type, which is how a parsed value replaces the string it came from. | reference |
| integer overflow rules | Debug builds panic on overflow, release builds wrap, and checked_add and wrapping_add say what you mean. | Three answers to the largest integer plus one rust/ch05/src/store.rs |
| as and From casts | as converts between numbers and may truncate or wrap, so n as u8 of 300 is 44. From and Into convert where the conversion cannot fail, and the ? operator uses From to turn one error type into another. | reference |
| tuples, arrays and slices | A tuple groups a fixed set of values of mixed types, and a pattern takes it apart. An array [T; N] has a length in its type. A slice &[T] is a borrowed view of a run of elements, with its length beside the pointer, and it is what every function in kv takes. | reference |
| Vec | The growable array that owns its elements and frees them when it drops. | Rust: a value has one owner rust/ch05/src/store.rs |
| String, str, char and bytes | String owns valid UTF-8, &str borrows it, char is a code point, and as_bytes gives the bytes. | str, String, char and a slice of bytes rust/ch05/src/store.rs |
| structs and impl blocks | Data in a struct, behaviour in impl, and a value is built with a literal or a new function. | Rust: the table that drops itself rust/ch06/src/table.rs |
| methods and the forms of self | &self reads, &mut self changes, self consumes. | Rust: the table that drops itself rust/ch06/src/table.rs |
| enums with data | Each variant can carry its own fields, which is a sum type the compiler checks. | One enum for everything that can go wrong rust/ch04/src/resp.rs |
| Option and Result | Some or None for a value that may be absent, Ok or Err for one that may fail. | Rust: an Option for the end, a Result for the rest rust/ch04/src/resp.rs |
| match and patterns | Every case must be handled, with guards, bindings, slice and struct patterns, and @. | Rust: an Option for the end, a Result for the rest rust/ch04/src/main.rs |
| if let, let else and while let | Match one pattern without the full match, or leave early when it does not match. | Rust: an Option for the end, a Result for the rest rust/ch04/src/resp.rs |
| loops and labels | loop runs until break, and break may carry a value out of it. while and for are the usual ones, and for takes anything that is an iterator. A label such as 'outer: lets a break or continue name the loop it means. | reference |
| expressions against statements | A block is an expression, and its value is the last line without a semicolon. So if, match and a plain block all produce values, and a function returns the value of its final expression. A semicolon turns an expression into a statement that yields nothing. | reference |
| const and static | A const is inlined, a static has one address, and a mutable static needs unsafe. | A table of fn pointers, built by a macro rust/ch05/src/cmd.rs |
| modules and visibility | mod, pub, pub(crate), use and paths, with files as modules. | Rust: a module is a file you must announce rust/ch07/src/store.rs |
| Cargo | Crates, features, workspaces, profiles, build.rs and the lockfile, with semantic versioning. | Rust: cargo owns the project hello/rust/Cargo.toml |
| doc comments and doc tests | /// becomes documentation, and a code block in it is compiled and run by cargo test. | Rust: the compiler already ran, and the tools check the rest rust/ch12/src/resp.rs |
| rustfmt and clippy | The one formatter and the linter with hundreds of checks, both standard. | Rust: cargo owns the project |
| editions | A crate picks 2015, 2018, 2021 or 2024, and crates of different editions link together. | Rust: cargo owns the project hello/rust/Cargo.toml |
Ownership#
| Concept | What to remember | Where |
|---|---|---|
| move | Assigning or passing a value gives it away, and the old name cannot be used again. | Rust: a value has one owner rust/ch05/src/cmd.rs |
| Copy against Clone | Small plain values copy on assignment; everything else moves unless you call clone. | Lifetimes, in the one place kv needs them |
| borrows | & lends a value for reading, &mut for changing, and the owner waits until the borrow ends. | Rust: a value has one owner rust/ch05/src/store.rs |
| mutable or shared, never both | Any number of readers or one writer at a time, which is the rule that prevents data races. | Rust: a value has one owner |
| non-lexical lifetimes | A borrow ends at its last use, not at the end of the block. | Lifetimes, in the one place kv needs them rust/ch10/src/main.rs |
| lifetime annotations and elision | 'a names how long a reference lives, and the compiler fills in the common cases. | Lifetimes, in the one place kv needs them rust/ch10/src/resp.rs |
| 'static | Lives for the whole program, as a string literal does, or owns everything it holds. | Lifetimes, in the one place kv needs them |
| structs that hold references | Command<'a> borrows from a buffer and cannot outlive it. | The parser stops copying rust/ch10/src/resp.rs |
| Drop and RAII | A value is cleaned up when its owner goes out of scope, in reverse order of declaration. | Rust: the table that drops itself rust/ch06/src/table.rs |
| mem::take, replace and swap | Move a value out of a place you only have &mut to, leaving a default behind. | Rust again: a scoped thread on a channel rust/ch11/src/aof.rs |
| interior mutability | Cell, RefCell and Mutex let you change a value through a shared reference, checked at runtime. | Two owners of one table rust/ch06/src/table.rs |
| Rc and Arc | Shared ownership by reference count, Arc when threads are involved. | Two owners of one table rust/ch06/src/table.rs |
| Box | One owner of a value on the heap, which recursive types and trait objects need. | Closures and iterators, named at last rust/ch08/src/chain.rs |
| Deref | A type that implements Deref is treated as the type it dereferences to where a reference is wanted. That is why a &String is accepted as a &str and a &Vec<T> as a &[T], and why a Box<T> can be used as a T. | reference |
| borrowing from a Vec | A slice of a Vec blocks every push until the slice is dropped. | Lifetimes, in the one place kv needs them rust/ch10/src/main.rs |
| Cow | Borrowed or owned, decided at runtime, for a value that usually needs no copy. | Lifetimes, in the one place kv needs them rust/ch10/src/cmd.rs |
Traits and generics#
| Concept | What to remember | Where |
|---|---|---|
| defining and implementing a trait | A set of methods a type promises, implemented in an impl Trait for Type block. | A trait is a promise of methods rust/ch07/src/store.rs |
| default methods | A trait method with a body that implementations may keep or replace. | A trait is a promise of methods rust/ch07/src/store.rs |
| derive | #[derive(Debug, Clone, PartialEq)] asks the compiler to write the usual implementations of those traits from the fields. The book derives Debug on every error type, so that a test can print one, and Default on the stores. | reference |
| the standard traits | Debug and Display for printing, Clone and Copy for copying, Default for an empty value, PartialEq, Eq, Hash and Ord for comparing and hashing, From and Into for converting, AsRef for borrowing in any form, Iterator for walking, Read and Write for bytes, and Error for errors. Most are derived, and the rest are an impl block each. | reference |
| generics and bounds | fn largest<T: Ord>(xs: &[T]) works for any T that can be ordered, and where clauses spell long bounds out. | Rust: the table that drops itself rust/ch06/src/table.rs |
| impl Trait in argument and return position | fn f(w: impl Write) is a generic in disguise, and -> impl Iterator hides a type. | Static and dynamic dispatch rust/ch07/src/store.rs |
| trait objects and dyn | Box<dyn Store> picks the implementation at runtime through a vtable. | Static and dynamic dispatch rust/ch07/src/main.rs |
| object safety | A trait with generic methods or Self returns cannot be a dyn. | Static and dynamic dispatch rust/ch07/src/store.rs |
| static against dynamic dispatch | Generics copy the code per type and inline; dyn calls through a pointer and keeps one copy. | Static and dynamic dispatch rust/ch07/src/store.rs |
| monomorphisation | Each use of a generic with a new type compiles to its own code, which is fast and slow to build. | Rust: the table that drops itself rust/ch06/src/table.rs |
| the orphan rule | You can implement a trait for a type only if you own one of them. | A trait is a promise of methods rust/ch07/src/table.rs |
| operator overloading | An operator is a trait: + is Add, [] is Index, == is PartialEq. A type that implements the trait gets the operator, and nothing else is allowed to redefine one. | reference |
| associated types and constants | Iterator's Item is a type chosen by the implementation, not by the caller. | Closures and iterators, named at last rust/ch08/src/chain.rs |
Closures and iterators#
| Concept | What to remember | Where |
|---|---|---|
| closures | |x| x + 1 captures its surroundings by reference, by mutable reference, or by move. | Closures and iterators, named at last rust/ch08/src/cmd.rs |
| Fn, FnMut and FnOnce | What a closure may do with what it captured, which decides where it can be passed. | Closures and iterators, named at last rust/ch08/src/cmd.rs |
| iterators and adaptors | map, filter, enumerate, zip, take and the rest, each a lazy wrapper around the last. | Closures and iterators, named at last rust/ch08/src/store.rs |
| collect and the turbofish | collect::<Vec<_>>() builds any collection, and the type says which. | Closures and iterators, named at last rust/ch08/src/cmd.rs |
| implementing Iterator | One method, next, and every adaptor works on your type. | Closures and iterators, named at last rust/ch08/src/chain.rs |
| iter, iter_mut and into_iter | Borrow the elements, borrow them mutably, or take them. | Closures and iterators, named at last rust/ch08/src/chain.rs |
| laziness | Nothing runs until something consumes the chain, so an unused map is a warning. | Closures and iterators, named at last rust/ch08/src/store.rs |
Errors#
| Concept | What to remember | Where |
|---|---|---|
| custom error enums | One enum with a variant per failure, matched exhaustively by the caller. | One enum for everything that can go wrong rust/ch04/src/resp.rs |
| Display and Error | Implement Display for the message and Error to join the ecosystem. | One enum for everything that can go wrong rust/ch04/src/resp.rs |
| From and ? | ? converts the error with From, so an io::Error becomes your KvError without a line of code. | One enum for everything that can go wrong rust/ch04/src/resp.rs |
| Box<dyn Error> | A boxed trait object that can hold any error type at all. A main or a tool that only reports errors returns Result<(), Box<dyn Error>> and uses ? on every error it meets, at the cost of no longer being able to match on the kind. | reference |
| thiserror and anyhow | The two crates most projects use for errors. thiserror derives Display and From for an error enum like the chapter 4 KvError, so that the impl blocks are one attribute each. anyhow is the Box<dyn Error> idea with context, for programs that report errors and never match on them. | reference |
| panics and the unwrap policy | unwrap and expect are for bugs that cannot happen, with expect saying why. | ?, expect and unwrap rust/ch03/src/main.rs |
| Result from main | fn main() -> Result<(), Box<dyn Error>> prints the error and exits non-zero. | A main that returns a Result rust/ch03/src/main.rs |
Concurrency#
| Concept | What to remember | Where |
|---|---|---|
| threads | thread::spawn takes a closure that must be 'static, and join waits. | Rust: a thread for each client, and a compiler that checks the sharing rust/ch09/src/main.rs |
| scoped threads | thread::scope lets threads borrow from the stack, because all are joined before it returns. | Rust again: the load generator with scoped threads rust/ch09/src/bin/kvload.rs |
| Send and Sync | Marker traits the compiler checks: safe to move to a thread, safe to share between threads. | Send and Sync |
| Arc<Mutex<T>> | Shared ownership plus one writer at a time, the shape most shared state takes. | Rust: a thread for each client, and a compiler that checks the sharing rust/ch09/src/main.rs |
| RwLock | Many readers or one writer, when reads dominate. | Rust: one allocation, a profile, four experiments, and a lock rust/ch13/src/main.rs |
| mpsc channels | Many senders, one receiver, and the receiver's loop ends when every sender drops. | Rust again: the load generator with scoped threads rust/ch09/src/bin/kvload.rs |
| atomics | AtomicBool and AtomicUsize with an Ordering, for flags and counters. | Atomics rust/ch09/src/bin/kvload.rs |
| async and await | An async fn returns a Future, and await pauses it instead of blocking the thread. | Rust once more: tasks on tokio rust/ch09/src/bin/kv_async.rs |
| what a Future is | A state machine with a poll method that a runtime drives. | What a future is |
| Pin | A future that borrows itself cannot move, and Pin is the promise that it will not. | Pin in one paragraph |
| tokio | spawn, tasks, the multi-threaded runtime, and its own Mutex and channels. | Rust once more: tasks on tokio rust/ch09/src/bin/kv_async.rs |
| select! | Wait on several futures and take the first, with cancellation of the rest. | select!, cancellation and blocking |
| cancellation | Dropping a future stops it, which is both convenient and a trap for half-done work. | select!, cancellation and blocking |
| blocking inside async | A std Mutex held across await, or a long computation, stalls the whole runtime. | select!, cancellation and blocking |
Unsafe and FFI#
| Concept | What to remember | Where |
|---|---|---|
| raw pointers | *const T and *mut T, with no borrow checking, dereferenced only in unsafe. | The five powers, and the pointer they are for rust/ch15/src/cmatch.rs |
| unsafe blocks and functions | The keyword that marks where the compiler's rules stop and yours start. | Rust: extern and unsafe rust/ch15/src/cmatch.rs |
| the five powers | Dereference a raw pointer, call unsafe code, touch a mutable static, implement an unsafe trait, read a union. | The five powers, and the pointer they are for |
| repr(C) | Lay a struct out as C would, so both sides agree on the bytes. | The five powers, and the pointer they are for rust/ch15/src/cmatch.rs |
| extern blocks | Declare C functions, unsafe since the 2024 edition, and export your own with no_mangle. | Rust: extern and unsafe rust/ch15/src/cmatch.rs |
| miri | The interpreter that runs tests and reports undefined behaviour in unsafe code. | Rust: the compiler already ran, and the tools check the rest |
| bindgen | Generates the extern declarations from a C header. | The five powers, and the pointer they are for |
Macros#
| Concept | What to remember | Where |
|---|---|---|
| macro_rules! | Pattern-based macros that write code from a template, for tables and repetition. | A table of fn pointers, built by a macro rust/ch05/src/cmd.rs |
| derive and attribute macros | #[derive(...)] and #[tokio::main] are procedural macros, which run Rust code at compile time to generate code. They come from crates, and the book uses them without writing one. A macro_rules! macro, like the command table in chapter 5, is the simpler kind. | reference |
| println!, format!, vec! and assert! | The standard macros. println! and format! take a format string checked at compile time with a variable number of arguments, which a function cannot. vec! builds a Vec from a list, and assert! and assert_eq! are the test checks that print both sides when they fail. | reference |
Testing and tooling#
| Concept | What to remember | Where |
|---|---|---|
| unit, integration and doc tests | #[cfg(test)] modules in the file, the tests/ directory, and examples in doc comments. | Rust: the compiler already ran, and the tools check the rest rust/ch12/tests/roundtrip.rs |
| cargo test flags | Filter by name, show output, run one thread, and ignore the slow ones. | cargo test, and what comes after the dashes |
| benchmarks with criterion | Stable Rust has no built-in bench, and criterion is the crate that fills the gap. | Rust: one allocation, a profile, four experiments, and a lock rust/ch13/examples/bench_parse.rs |
| cargo-fuzz | libFuzzer targets in a fuzz/ directory, run on nightly. | Rust: the compiler already ran, and the tools check the rest rust/ch12/fuzz/fuzz_targets/parse.rs |
| cargo doc | Builds the documentation of your crate and every dependency, offline. | Rust: the compiler already ran, and the tools check the rest rust/ch12/src/resp.rs |
Appendix F: Concept index#
Every concept the book claims to teach, per language and per group, with the section
that teaches it and the kv file that uses it. The build generates this index from
the three files under concepts/ and fails when a concept has no section.
C#
Types and values#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| integer types and stdint.h | int, long and friends have platform sizes; int32_t and uint64_t say exactly what they are. | C: an array of chains, and every pointer accounted for | c/ch06/kv.c |
| char signedness | Plain char may be signed or unsigned by platform, so a byte over 127 can become negative. | C: an array of chains, and every pointer accounted for | c/ch06/kv.c |
| integer promotion and the usual conversions | Small types widen to int before arithmetic, so two chars add as ints. When a signed and an unsigned operand meet, the signed one becomes unsigned, and -1 is suddenly the largest value. Compare sizes as size_t and counts as int, and never mix them in one expression. | Appendix C | |
| signed overflow is undefined | Overflowing a signed integer is undefined behaviour, so the compiler may assume it never happens. | Three answers to the largest integer plus one | c/ch05/kv.c |
| unsigned wrap | Unsigned arithmetic wraps around modulo 2^n, by definition, which is how a hash function is allowed to overflow. | Three answers to the largest integer plus one | c/ch05/kv.c |
| bool | stdbool.h since C99, a keyword in C23. Any non-zero value is true. | C17 against C23 | c/ch14/extras/c23.c |
| enum | Named integer constants, with no check that a value is one of the names. | switch, and the cases that run into each other | c/ch04/kv.c |
| sizeof | The size of a type or an object in bytes, computed at compile time, with no parentheses needed for an object. | What a struct is made of, and an arena | c/ch06/generic.c |
| size_t, ssize_t and ptrdiff_t | size_t is the unsigned type of a size and an index, and sizeof yields it. ssize_t is the signed one that read and write return, so that -1 can mean failure. ptrdiff_t is the type of one pointer minus another. | Appendix C | |
| casts | An explicit conversion the compiler will not question, so each one is a claim you must be able to defend. | What C does instead of generics | c/ch06/generic.c |
| const and const-correctness | A promise not to write through a pointer, checked by the compiler and carried through every call. | The parser stops copying | c/ch10/src/resp.h |
| volatile | Every read and write must happen, because something outside the program can change the value. | C: a signal, a flag and a pipe | c/ch10/src/stop.h |
| static, extern and linkage | static hides a name inside one file; extern says a name is defined elsewhere. | C: two files, one binary | hello/c/greet.c |
| translation units | Each .c file compiles alone, with the headers pasted in, and the linker joins the results. | C: two files, one binary | hello/c/units.c |
| storage duration | Automatic variables live until the block ends, static ones for the whole run, allocated ones until free. | Where a value lives, and for how long | c/ch06/kv.c |
| stack and heap | Locals live on the stack and vanish with the function; malloc gives memory that outlives it. | Where a value lives, and for how long | c/ch06/kv.c |
| zero initialisation | = {0} sets every member to zero, and a static variable is zero without it. | C: an array of chains, and every pointer accounted for | c/ch06/kv.c |
| designated initialisers | {.fd = -1, .len = 0} names the members, so the order of the struct does not matter. | C again: poll, kqueue and epoll behind one header | c/ch09/src/poller_epoll.c |
| compound literals | (struct pollfd){fd, POLLIN, 0} builds a value in place without a named variable. | C: an array of chains, and every pointer accounted for | c/ch06/kv.c |
Pointers and memory#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| pointers | An address plus a type, and the type says how many bytes one step is. | C: an array of chains, and every pointer accounted for | c/ch06/kv.c |
| pointer arithmetic | p + 1 moves by one element, not one byte, so the type of p decides the step. The result is only defined inside the same object, or one past its end, and the chapter 4 parser stays inside its buffer by comparing against end. | Appendix C | |
| arrays decay to pointers | An array name used as a value becomes a pointer to its first element, and the length is gone. That is why a function takes a pointer and a length, and why sizeof an array parameter is the size of a pointer. | Appendix C | |
| pointer to pointer | How a function gives its caller a new pointer, as in a grow function that may move a buffer. | C: an array of chains, and every pointer accounted for | c/ch06/kv.c |
| void pointer | A pointer to anything, which malloc returns and a callback context uses. | What C does instead of generics | c/ch06/generic.c |
| NULL and nullptr | The null pointer. C23 adds nullptr with its own type. | C17 against C23 | c/ch14/extras/c23.c |
| malloc, calloc, realloc and free | Ask for bytes, get them zeroed, grow them in place or elsewhere, give them back. | Growing, measuring and slicing bytes | c/ch05/kv.c |
| ownership conventions | A comment on every pointer says who frees it, because the compiler will not. | The parser stops copying | c/ch10/src/resp.h |
| alignment and offsetof | Each type has an alignment, structs get padding to keep it, and offsetof finds a member's position. | C: an array of chains, and every pointer accounted for | c/ch06/generic.c |
| flexible array members | A struct whose last member is an array of unknown size, declared as data[], is allocated in one block with room after its header. Redis stores its strings this way, with the length and the bytes in one allocation. | Appendix C | |
| restrict | A promise that two pointers do not overlap, which lets the compiler optimise. | The optimisation levels, measured | c/ch13/bench/bench_copy.c |
| strict aliasing | Reading an object through a pointer of a different type is undefined, except through char or unsigned char. The compiler may assume an int and a float never overlap, which is why a memcpy is the way to reinterpret bytes. | Appendix C | |
| use after free, double free and leaks | The three memory bugs C lets you write, and what each looks like under AddressSanitizer. | C: an array of chains, and every pointer accounted for | c/ch06/kv.c |
| arenas | Allocate many small things from one block and free them all at once. | What a struct is made of, and an arena | c/ch06/generic.c |
Structs and unions#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| struct | Members laid out in order, with padding, passed by value unless you pass a pointer. | C: an array of chains, and every pointer accounted for | c/ch06/kv.c |
| typedef | A new name for a type, which is how kv_store stands for struct kv_store. | C: an array of chains, and every pointer accounted for | c/ch06/kv.c |
| opaque pointers | A header declares struct kv_store without its members, so callers cannot touch them. | An opaque type, and a struct of functions | c/ch07/src/store.h |
| union | Members that share the same bytes, so only one is meaningful at a time. | C: a tag and a union | c/ch08/src/store.h |
| tagged union | A union plus an enum that says which member is live: C's hand-built sum type. | C: a tag and a union | c/ch08/src/store.h |
| bitfields | Members with a width in bits, packed by the compiler in an order it chooses. | C: a tag and a union |
Functions and control#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| switch and fallthrough | A jump table on an integer, where a case without break runs into the next one. | switch, and the cases that run into each other | c/ch04/kv.c |
| do while | A loop that runs its body once before it tests the condition. The test runner's CHECK macro is built on one with a condition of 0, so that it runs exactly once. | Appendix C | |
| goto cleanup | One exit path at the end of a function that frees everything, reached by goto from each failure. | C: four calls, and a way to clean up | c/ch03/kv.c |
| function pointers | A pointer to code, declared with the signature it points to, which is how C passes behaviour. | A table of functions | c/ch05/kv.c |
| callbacks with a void context | A function pointer plus a void * that the callee passes back, which is C's closure. | An opaque type, and a struct of functions | c/ch07/src/cmd.c |
| variadic functions | Functions that take any number of arguments, read with va_list, as printf does. | A function that takes any number of arguments | c/ch04/kv.c |
| inline | A hint to paste the function body at each call, with linkage rules that surprise. | The optimisation levels, measured | c/ch13/Makefile |
| static functions | A function only this file can call, which the compiler can warn about when unused. | C: headers, static, and a Makefile that knows about both | c/ch07/src/buf.c |
| _Noreturn | Marks a function that never returns, such as one that prints an error and exits. | C again: options, and a function that does not come back | c/ch14/src/main.c |
Preprocessor#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| #include | Pastes a file in place. Angle brackets search the system, quotes search here first. | C: headers, static, and a Makefile that knows about both | c/ch07/src/main.c |
| header guards and #pragma once | The #ifndef at the top of a header that stops it being pasted twice. | C: headers, static, and a Makefile that knows about both | c/ch07/src/buf.h |
| object and function macros | Text replacement before compilation, with or without arguments, with no types. | A table of functions | c/ch05/kv.c |
| do { } while (0) | The wrapper that makes a macro of several statements behave like one statement, so that it can follow an if without braces and still take a semicolon. The CHECK macro in chapter 12 is one. | Appendix C | |
| X-macros | One list of entries that expands into a table, an enum and a set of names at once. | C: bytes, a length and an owner | c/ch05/kv.c |
| #ifdef platform guards | Choose code at compile time by platform, as in __linux__ against __APPLE__. | C again: poll, kqueue and epoll behind one header | c/ch09/src/poller.h |
| _Static_assert | A check the compiler runs, such as that a struct has the size you think. | C: a runner you write, and a sanitizer that watches | c/ch12/tests/test_resp.c |
| _Generic | A compile-time switch on the type of an expression, the nearest C gets to overloading. | What C does instead of generics | c/ch06/generic.c |
| predefined macros | __FILE__, __LINE__, __func__ and the platform and compiler names. | C again: poll, kqueue and epoll behind one header | c/ch09/src/poller.h |
| __attribute__ and compiler extensions | Non-standard annotations such as format checking and cleanup, understood by gcc and clang. | C again: options, and a function that does not come back | c/ch14/src/main.c |
Strings and I/O#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| NUL-terminated strings | A string is bytes up to the first zero, so a string cannot hold a zero and its length is a walk. | C: bytes, a length and an owner | c/ch05/kv.c |
| strlen, memcpy and memcmp | The three calls behind most string code, and what each assumes about its arguments. | C: bytes, a length and an owner | c/ch05/kv.c |
| the strncpy trap | strncpy fills the whole destination and may leave no terminating zero when the source is as long as the limit. A bounded copy of a C string wants snprintf, and a copy of bytes with a length wants memcpy, which kv uses throughout. | Appendix C | |
| snprintf | Formats into a buffer of a known size and never overruns it, returning what it wanted to write. | Growing, measuring and slicing bytes | c/ch05/kv.c |
| pointer and length discipline | Every buffer travels with its length, which is how kv handles bytes that may contain zero. | C: a buffer, a parser and three answers | c/ch04/kv.c |
| stdio against read and write | FILE * buffers in user space; read and write are the system calls underneath. | C: a file descriptor, a write, and a thread for the fsync | c/ch11/src/aof.c |
| errno, strerror and perror | A failed call returns -1 and sets a thread-local number, which strerror turns into words. | errno: the number a failed call leaves behind | c/ch03/kv.c |
Concurrency and signals#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| pthreads and C11 threads | Create a thread with a function and a void *, join it to wait. | C again: the second thread | c/ch11/src/syncer.c |
| mutex | pthread_mutex_lock and unlock around every access to shared data, with nothing to stop you forgetting. | C again: the second thread | c/ch11/src/syncer.c |
| condition variable | Sleep until another thread signals, always inside a loop that rechecks the condition. | C again: the second thread | c/ch11/src/syncer.c |
| atomics and memory order | stdatomic.h gives operations no other thread can see half done, with an ordering you choose. | C again: the second thread | c/ch11/src/syncer.c |
| thread safety of libc | Most of libc is safe to call from threads; strtok, localtime and a few others are not. | Thread safety of the C library | |
| signals and sigaction | The kernel interrupts the program to run a handler, which may only do a few safe things. | C: a signal, a flag and a pipe | c/ch10/src/stop.c |
| sig_atomic_t and the self-pipe trick | A handler sets a flag or writes one byte to a pipe that the poll loop watches. | C: a signal, a flag and a pipe | c/ch10/src/stop.c |
| SIGPIPE | A write to a peer that has gone raises a signal that kills the process unless ignored. | C: a signal, a flag and a pipe | c/ch10/src/main.c |
| fork | Clone the process. The Unix way to start a program, which Windows does not have. | C again: the second thread |
Build and tooling#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| compiler flags and -O levels | -O0 for debugging, -O2 for release, and what the optimiser is allowed to assume. | The optimisation levels, measured | c/ch13/Makefile |
| warnings as errors | -Wall -Wextra -Werror turns the compiler's advice into a gate. | C: from source to binary in four steps | hello/c/Makefile |
| static libraries | An archive of object files that the linker copies from, made with ar. | C: a library that is easy to call | c/ch15/Makefile |
| Make | Targets, prerequisites and recipes, with tabs, and a rule for how a .o comes from a .c. | Build tools, and the lack of one | hello/c/Makefile |
| CMake | A generator of build files for larger projects, which most C libraries now use. | Build tools, and the lack of one | |
| pkg-config | Asks an installed library for the flags to compile and link against it. | Dependencies in C: the package manager that is not there | |
| AddressSanitizer and UndefinedBehaviorSanitizer | Compile-time instrumentation that reports memory errors and undefined behaviour as they happen. | C: a runner you write, and a sanitizer that watches | c/ch12/Makefile |
| ThreadSanitizer | The sanitizer that reports data races, which cannot run together with AddressSanitizer. | C again: the second thread | c/ch11/src/syncer.c |
| valgrind | Runs the unmodified binary under an emulator and reports leaks and bad accesses, on Linux. | C: a runner you write, and a sanitizer that watches | |
| a debugger session | lldb or gdb: break, step, print, backtrace, and the -g flag that makes them useful. | A debugger session | |
| clang-format | The formatter, driven by a .clang-format file, since C has no single style. | Build tools, and the lack of one | |
| C17 against C23 | nullptr, constexpr, typeof, auto, #embed, attributes and true keywords, and how far clang and gcc support them. | C17 against C23 | c/ch14/extras/c23.c |
Conventions#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| name prefixes | kv_ on every public name, because C has one namespace for the whole program. | C: headers, static, and a Makefile that knows about both | c/ch07/src/buf.h |
| header hygiene | A header declares, includes only what it needs, and never defines a variable. | C: headers, static, and a Makefile that knows about both | c/ch07/src/buf.h |
| error returns | Return a code or a bool and put the result in an out parameter, consistently. | C: bytes, a length and an owner | c/ch05/kv.c |
| ownership comments | Who frees it, written next to the pointer, every time. | C: bytes, a length and an owner | c/ch05/kv.c |
| const parameters | A const char * parameter promises the caller the bytes come back unchanged. | C: bytes, a length and an owner | c/ch05/kv.c |
Go#
Basics#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| packages and imports | A folder is a package, an import is a path, and an unused import is an error. | Go: one command, one binary | hello/go/main.go |
| zero values | Every variable starts as zero, nil or empty, so a struct is usable without a constructor. | Every variable starts at zero | hello/go/main.go |
| := and var | Inside a function, := declares and assigns in one step and infers the type. var works anywhere and takes an explicit type or a zero value. A := in an inner scope can declare a new variable that shadows one outside, which is the usual way an err goes missing. | Appendix D | |
| const and iota | A const is untyped until it is used, so 1 << 40 is fine where an int would overflow. Inside one const block, iota counts from 0 on each line, which is how the aof package names its three policies. | Appendix D | |
| basic types and conversions | int is the machine word, every conversion is written out, and there is no implicit widening. | Three answers to the largest integer plus one | go/ch05/strings_test.go |
| runes, strings, bytes and UTF-8 | A string is immutable bytes, range gives runes, and len counts bytes. | Bytes, runes and a builder | go/ch05/strings_test.go |
| arrays against slices | An array has a fixed length in its type; a slice is a view of one with a length and a capacity. | Go: the same table, with nothing to free | go/ch06/table.go |
| len, cap, append and aliasing | append writes into spare capacity, so two slices can share and overwrite the same array. | Bytes, runes and a builder | go/ch05/exec.go |
| copy and the full slice expression | copy duplicates bytes, and s[a:b:c] caps the capacity to stop append from aliasing. | The parser stops copying | go/ch10/internal/resp/resp.go |
| maps | Reading a nil map is fine and writing one panics; iteration order is random on purpose. | Go: two kinds of string, and a copy between them | go/ch05/store.go |
| comma-ok | v, ok := m[k] separates a missing key from a zero value, and the same form works for assertions and channels. | Go: two kinds of string, and a copy between them | go/ch05/store.go |
| structs and composite literals | Fields in order or by name, and a pointer to a literal with &. | Go: the same table, with nothing to free | go/ch06/table.go |
| pointers against values | Everything is passed by value, so a function that must change the caller's struct takes a pointer. A method on a pointer receiver can change the value and a method on a value receiver gets a copy. The store's methods all take *Store. | Appendix D | |
| the forms of range | Over a slice, a map, a string, a channel, an integer since 1.22, and a function since 1.23. | A function that range can call | go/ch08/internal/store/value.go |
| switch and type switch | No fallthrough by default, cases can be expressions, and switch v := x.(type) branches on a type. | Go: any, and a switch on the type | go/ch08/internal/store/value.go |
| labeled break and continue | The way out of a nested loop, with a label on the outer one. | A function that range can call |
Functions and methods#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| multiple returns and named results | A function returns a value and an error together, and named results document them. | Go: any, and a switch on the type | go/ch08/internal/cmd/cmd.go |
| variadic functions | f(args ...string) takes any number, and f(slice...) spreads one. | Bytes, runes and a builder | go/ch05/exec.go |
| closures and capture | A function literal captures variables by reference, and since 1.22 each loop iteration has its own. | Go: two kinds of string, and a copy between them | go/ch05/exec.go |
| defer | Runs at function exit, last in first out, with its arguments evaluated at the defer line. | Two rules of defer | go/ch03/defer_test.go |
| panic and recover | A panic unwinds the stack; a deferred recover stops it, which is how one bad client does not kill a server. | panic and recover | go/ch10/cmd/kv/main.go |
| methods and receivers | A function with a receiver, on a pointer when it changes the value, on a value when it does not. | Go: a folder is a package, and a capital letter is an export | go/ch07/internal/store/store.go |
| method sets | A value of type T has the methods with value receivers, and a *T has those and the pointer-receiver ones too. That decides which of the two satisfies an interface: a *Store satisfies Storer, and a Store does not. | Appendix D | |
| method values | s.Get without the call is a function bound to s, which a dispatch table can hold. | A map of functions | go/ch05/exec.go |
| embedding and promotion | An unnamed field lends its methods to the outer struct, as an embedded sync.Mutex does. | Embedding, and the nil that is not nil | go/ch07/internal/cmd/cmd_test.go |
Interfaces#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| interfaces and implicit satisfaction | A set of method signatures, and any type with those methods satisfies it without saying so. | An interface is a set of methods | go/ch07/internal/cmd/cmd.go |
| small interfaces | io.Reader, io.Writer, fmt.Stringer and error have one method each, and most code accepts them. | An interface is a set of methods | go/ch07/internal/cmd/cmd.go |
| any | The empty interface, which every value satisfies, and which a type switch takes apart. | Go: any, and a switch on the type | go/ch08/internal/store/value.go |
| type assertion | v, ok := x.(T) asks whether the dynamic value is a T, and without ok it panics. | Go: any, and a switch on the type | go/ch08/internal/cmd/cmd.go |
| the nil interface trap | An interface holding a nil pointer is not nil, which makes a returned *MyErr look like an error. | Embedding, and the nil that is not nil | go/ch07/internal/cmd/cmd_test.go |
| interface embedding | An interface may list other interfaces, and gets their methods. io.ReadWriter is io.Reader and io.Writer in one declaration, and a type that has both methods satisfies all three. | Appendix D | |
| when not to use an interface | Accept interfaces, return structs, and define one only when a second implementation exists. | Embedding, and the nil that is not nil | go/ch07/internal/cmd/cmd.go |
Generics#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| type parameters | func Max[T cmp.Ordered](a, b T) T, since Go 1.18, with the type inferred at the call. | Go: the same table, with nothing to free | go/ch06/table.go |
| constraints, comparable and any | An interface used as a bound, comparable for map keys, and a union of types with |. | Go: the same table, with nothing to free | go/ch06/table.go |
| the slices, maps and cmp packages | Sort, search, clone and compare, written once with generics instead of per type. | Go: the same table, with nothing to free | go/ch06/table_test.go |
| when to use generics | A container or an algorithm over many types, not a method that takes an interface. | Go: the same table, with nothing to free | go/ch06/table.go |
Errors#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| the error interface | Anything with an Error() string method, and nil means no error. | An error is any value with an Error method | go/ch04/resp.go |
| custom error types | A struct with fields the caller can read, returned as an error. | An error is any value with an Error method | go/ch04/resp.go |
| wrapping with %w | fmt.Errorf("read config: %w", err) adds context and keeps the cause reachable. | An error is any value with an Error method | go/ch04/resp.go |
| errors.Is, errors.As and errors.Join | Walk the wrapped chain for a sentinel, for a type, or combine several errors. | errors.Join | go/ch12/internal/aof/aof.go |
| sentinel against typed errors | var ErrNotFound = errors.New(...) for a condition, a type when the caller needs fields. | An error is any value with an Error method | go/ch04/resp.go |
| panics against errors | An error is a value for the caller; a panic is a bug in the program. | panic and recover |
Concurrency#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| goroutines | go f() runs f on a stack of a few kilobytes, multiplexed over the real threads. | Go: one word, and one lock | go/ch09/cmd/kv/main.go |
| channels | A typed queue between goroutines; unbuffered ones hand off, buffered ones hold a few. | Go again: fifty workers and three channels | go/ch09/cmd/kvload/main.go |
| close and range over a channel | The sender closes, the receiver's range ends, and a receive from a closed channel gives zero. | Go again: fifty workers and three channels | go/ch09/cmd/kvload/main.go |
| nil channels | A nil channel blocks forever, which is a way to switch a select case off. | Go again: fifty workers and three channels | go/ch09/cmd/kvload/main.go |
| select | Wait on several channels at once, with default for a non-blocking try and a timer for a timeout. | Go again: fifty workers and three channels | go/ch09/cmd/kvload/main.go |
| sync.Mutex and RWMutex | Lock around shared data, with RLock for readers, and nothing stops you forgetting. | Go: one word, and one lock | go/ch09/internal/store/store.go |
| sync.WaitGroup | Add before starting, Done when finished, Wait for all, and wg.Go since 1.25. | Go: one word, and one lock | go/ch09/internal/store/store_test.go |
| sync.Once and sync.Pool | Run something exactly once, and reuse buffers instead of allocating. | The rest of the sync package | |
| sync/atomic | Counters and flags that need no lock, as atomic.Int64. | The rest of the sync package | |
| context | Cancellation, a deadline and request values that flow down every call. | context in one paragraph | go/ch10/cmd/kv/main.go |
| worker pools | A fixed number of goroutines reading jobs from one channel. | Go again: fifty workers and three channels | go/ch09/cmd/kvload/main.go |
| data races and the race detector | Two goroutines, one writing, no sync, and go test -race finds it. | Go: one word, and one lock | go/ch09/internal/store/store_test.go |
| GOMAXPROCS and the scheduler | Goroutines run on P logical processors over M threads, and a blocking system call hands its P over. | The runtime in one paragraph | |
| goroutine leaks | A goroutine blocked forever on a channel nobody will touch again. | Go again: fifty workers and three channels | go/ch09/cmd/kvload/main.go |
Runtime and memory#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| the garbage collector | Concurrent mark and sweep, with short pauses and memory held until the next cycle. | Go: the same table, with nothing to free | go/ch06/table.go |
| escape analysis | go build -gcflags=-m says which values move to the heap, and why. | Escape analysis | go/ch13/internal/resp/resp.go |
| GOGC and GOMEMLIMIT | How often the collector runs, and a ceiling on the heap. | GOGC, GOMEMLIMIT and the Pool | |
| pprof | CPU and memory profiles from a running program, read as a graph or a flamegraph. | The profiler is a package | go/ch13/cmd/kv/main.go |
Tooling and packaging#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| go.mod and go.sum | The module path and its dependencies, and the hashes that pin them. | Go: one command, one binary | hello/go/go.mod |
| semantic import versioning | A v2 module has /v2 in its import path, so two majors can coexist. | Go: a folder is a package, and a capital letter is an export | |
| go get and replace | Add or upgrade a dependency, or point a module at a local folder. | Go: a folder is a package, and a capital letter is an export | |
| workspaces | go.work lets several modules build together while you change them. | Rust: a module is a file you must announce | |
| internal packages | A package under internal/ can only be imported from inside its parent. | Go: a folder is a package, and a capital letter is an export | go/ch07/internal/store/store.go |
| go vet, gofmt and staticcheck | The checker, the one formatter, and the linter most teams add. | Go: one command, one binary | |
| build tags and platform files | //go:build linux, or a _linux.go suffix, picks a file by platform. | Go's platform files: a line at the top | go/ch09/internal/sys/limit_unix.go |
| go generate and go:embed | Run a tool from a comment, and compile a file into the binary. | Go again: flags, a file, a log and a version | go/ch14/cmd/kv/main.go |
| table-driven tests | A slice of cases and one loop, the idiom every Go test file uses. | Go: a folder is a package, and a capital letter is an export | go/ch07/internal/store/glob_test.go |
| subtests, t.Parallel and helpers | t.Run names a case, t.Parallel runs them together, t.Helper hides a helper from the report. | Go: the runner is in the box, and so is the fuzzer | go/ch12/internal/resp/resp_test.go |
| benchmarks | func BenchmarkX(b *testing.B) with b.Loop, and go test -bench. | Go: nine allocations, and the flag that names them | go/ch13/internal/resp/bench_test.go |
| fuzzing | func FuzzX(f *testing.F) with go test -fuzz, built into the toolchain. | Go: the runner is in the box, and so is the fuzzer | go/ch12/internal/resp/resp_test.go |
| cross-compiling | GOOS and GOARCH, with CGO_ENABLED=0 for a static binary. | Go: two variables | go/ch14/Makefile |
| cgo | import "C" and a preamble, with a cost on every call. | Go: cgo | go/ch15/internal/cmatch/cmatch.go |
| the module proxy and vendoring | proxy.golang.org serves modules, and go mod vendor copies them into the repository. | Go: a folder is a package, and a capital letter is an export | |
| go doc | Doc comments on exported names, read with go doc and on pkg.go.dev. | The documentation is in the source |
Standard library tour#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| fmt | Printf and its verbs: %v for anything, %q for a quoted string, %d and %s as in C, and %w to wrap an error in Errorf. Println adds spaces and a newline, and a type with a String method prints as it chooses. | Appendix D | |
| strconv | Atoi, ParseInt and Itoa, with errors, not panics. | Go: a reader that waits, and an error that names itself | go/ch04/resp.go |
| strings and bytes | The same functions twice, for string and for []byte, plus strings.Builder. | Bytes, runes and a builder | go/ch05/strings_test.go |
| bufio | A buffered reader and writer over any io.Reader or io.Writer. | Go: a reader that waits, and an error that names itself | go/ch04/resp.go |
| io and os | The interfaces, and files, environment and arguments. | Go: the loop you did not have to write | go/ch03/main.go |
| net | net.Listen, Accept and net.Conn, which is an io.ReadWriteCloser. | Go: the loop you did not have to write | go/ch03/main.go |
| time | time.Time and Duration, the monotonic reading hidden inside Now, Timer and Ticker. | Go: a ticker, and a clock the test can hold still | go/ch10/internal/store/store.go |
| encoding/json | Struct tags and Marshal, the most used package after fmt. | Go again: flags, a file, a log and a version | go/ch14/cmd/kv/main.go |
| flag | Command-line flags with defaults and a generated usage. | Go again: flags, a file, a log and a version | go/ch14/cmd/kv/main.go |
| log/slog | Structured logging with levels and key-value pairs. | Go again: flags, a file, a log and a version | go/ch14/cmd/kv/main.go |
| os/signal | signal.Notify delivers a signal on a channel, and NotifyContext cancels a context. | Go: a context that a signal cancels | go/ch10/cmd/kv/main.go |
Rust#
Basics#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| let, mut and shadowing | A binding is immutable unless it says mut, which the compiler enforces. A second let with the same name shadows the first, and may change its type, which is how a parsed value replaces the string it came from. | Appendix E | |
| integer overflow rules | Debug builds panic on overflow, release builds wrap, and checked_add and wrapping_add say what you mean. | Three answers to the largest integer plus one | rust/ch05/src/store.rs |
| as and From casts | as converts between numbers and may truncate or wrap, so n as u8 of 300 is 44. From and Into convert where the conversion cannot fail, and the ? operator uses From to turn one error type into another. | Appendix E | |
| tuples, arrays and slices | A tuple groups a fixed set of values of mixed types, and a pattern takes it apart. An array [T; N] has a length in its type. A slice &[T] is a borrowed view of a run of elements, with its length beside the pointer, and it is what every function in kv takes. | Appendix E | |
| Vec | The growable array that owns its elements and frees them when it drops. | Rust: a value has one owner | rust/ch05/src/store.rs |
| String, str, char and bytes | String owns valid UTF-8, &str borrows it, char is a code point, and as_bytes gives the bytes. | str, String, char and a slice of bytes | rust/ch05/src/store.rs |
| structs and impl blocks | Data in a struct, behaviour in impl, and a value is built with a literal or a new function. | Rust: the table that drops itself | rust/ch06/src/table.rs |
| methods and the forms of self | &self reads, &mut self changes, self consumes. | Rust: the table that drops itself | rust/ch06/src/table.rs |
| enums with data | Each variant can carry its own fields, which is a sum type the compiler checks. | One enum for everything that can go wrong | rust/ch04/src/resp.rs |
| Option and Result | Some or None for a value that may be absent, Ok or Err for one that may fail. | Rust: an Option for the end, a Result for the rest | rust/ch04/src/resp.rs |
| match and patterns | Every case must be handled, with guards, bindings, slice and struct patterns, and @. | Rust: an Option for the end, a Result for the rest | rust/ch04/src/main.rs |
| if let, let else and while let | Match one pattern without the full match, or leave early when it does not match. | Rust: an Option for the end, a Result for the rest | rust/ch04/src/resp.rs |
| loops and labels | loop runs until break, and break may carry a value out of it. while and for are the usual ones, and for takes anything that is an iterator. A label such as 'outer: lets a break or continue name the loop it means. | Appendix E | |
| expressions against statements | A block is an expression, and its value is the last line without a semicolon. So if, match and a plain block all produce values, and a function returns the value of its final expression. A semicolon turns an expression into a statement that yields nothing. | Appendix E | |
| const and static | A const is inlined, a static has one address, and a mutable static needs unsafe. | A table of fn pointers, built by a macro | rust/ch05/src/cmd.rs |
| modules and visibility | mod, pub, pub(crate), use and paths, with files as modules. | Rust: a module is a file you must announce | rust/ch07/src/store.rs |
| Cargo | Crates, features, workspaces, profiles, build.rs and the lockfile, with semantic versioning. | Rust: cargo owns the project | hello/rust/Cargo.toml |
| doc comments and doc tests | /// becomes documentation, and a code block in it is compiled and run by cargo test. | Rust: the compiler already ran, and the tools check the rest | rust/ch12/src/resp.rs |
| rustfmt and clippy | The one formatter and the linter with hundreds of checks, both standard. | Rust: cargo owns the project | |
| editions | A crate picks 2015, 2018, 2021 or 2024, and crates of different editions link together. | Rust: cargo owns the project | hello/rust/Cargo.toml |
Ownership#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| move | Assigning or passing a value gives it away, and the old name cannot be used again. | Rust: a value has one owner | rust/ch05/src/cmd.rs |
| Copy against Clone | Small plain values copy on assignment; everything else moves unless you call clone. | Lifetimes, in the one place kv needs them | |
| borrows | & lends a value for reading, &mut for changing, and the owner waits until the borrow ends. | Rust: a value has one owner | rust/ch05/src/store.rs |
| mutable or shared, never both | Any number of readers or one writer at a time, which is the rule that prevents data races. | Rust: a value has one owner | |
| non-lexical lifetimes | A borrow ends at its last use, not at the end of the block. | Lifetimes, in the one place kv needs them | rust/ch10/src/main.rs |
| lifetime annotations and elision | 'a names how long a reference lives, and the compiler fills in the common cases. | Lifetimes, in the one place kv needs them | rust/ch10/src/resp.rs |
| 'static | Lives for the whole program, as a string literal does, or owns everything it holds. | Lifetimes, in the one place kv needs them | |
| structs that hold references | Command<'a> borrows from a buffer and cannot outlive it. | The parser stops copying | rust/ch10/src/resp.rs |
| Drop and RAII | A value is cleaned up when its owner goes out of scope, in reverse order of declaration. | Rust: the table that drops itself | rust/ch06/src/table.rs |
| mem::take, replace and swap | Move a value out of a place you only have &mut to, leaving a default behind. | Rust again: a scoped thread on a channel | rust/ch11/src/aof.rs |
| interior mutability | Cell, RefCell and Mutex let you change a value through a shared reference, checked at runtime. | Two owners of one table | rust/ch06/src/table.rs |
| Rc and Arc | Shared ownership by reference count, Arc when threads are involved. | Two owners of one table | rust/ch06/src/table.rs |
| Box | One owner of a value on the heap, which recursive types and trait objects need. | Closures and iterators, named at last | rust/ch08/src/chain.rs |
| Deref | A type that implements Deref is treated as the type it dereferences to where a reference is wanted. That is why a &String is accepted as a &str and a &Vec<T> as a &[T], and why a Box<T> can be used as a T. | Appendix E | |
| borrowing from a Vec | A slice of a Vec blocks every push until the slice is dropped. | Lifetimes, in the one place kv needs them | rust/ch10/src/main.rs |
| Cow | Borrowed or owned, decided at runtime, for a value that usually needs no copy. | Lifetimes, in the one place kv needs them | rust/ch10/src/cmd.rs |
Traits and generics#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| defining and implementing a trait | A set of methods a type promises, implemented in an impl Trait for Type block. | A trait is a promise of methods | rust/ch07/src/store.rs |
| default methods | A trait method with a body that implementations may keep or replace. | A trait is a promise of methods | rust/ch07/src/store.rs |
| derive | #[derive(Debug, Clone, PartialEq)] asks the compiler to write the usual implementations of those traits from the fields. The book derives Debug on every error type, so that a test can print one, and Default on the stores. | Appendix E | |
| the standard traits | Debug and Display for printing, Clone and Copy for copying, Default for an empty value, PartialEq, Eq, Hash and Ord for comparing and hashing, From and Into for converting, AsRef for borrowing in any form, Iterator for walking, Read and Write for bytes, and Error for errors. Most are derived, and the rest are an impl block each. | Appendix E | |
| generics and bounds | fn largest<T: Ord>(xs: &[T]) works for any T that can be ordered, and where clauses spell long bounds out. | Rust: the table that drops itself | rust/ch06/src/table.rs |
| impl Trait in argument and return position | fn f(w: impl Write) is a generic in disguise, and -> impl Iterator hides a type. | Static and dynamic dispatch | rust/ch07/src/store.rs |
| trait objects and dyn | Box<dyn Store> picks the implementation at runtime through a vtable. | Static and dynamic dispatch | rust/ch07/src/main.rs |
| object safety | A trait with generic methods or Self returns cannot be a dyn. | Static and dynamic dispatch | rust/ch07/src/store.rs |
| static against dynamic dispatch | Generics copy the code per type and inline; dyn calls through a pointer and keeps one copy. | Static and dynamic dispatch | rust/ch07/src/store.rs |
| monomorphisation | Each use of a generic with a new type compiles to its own code, which is fast and slow to build. | Rust: the table that drops itself | rust/ch06/src/table.rs |
| the orphan rule | You can implement a trait for a type only if you own one of them. | A trait is a promise of methods | rust/ch07/src/table.rs |
| operator overloading | An operator is a trait: + is Add, [] is Index, == is PartialEq. A type that implements the trait gets the operator, and nothing else is allowed to redefine one. | Appendix E | |
| associated types and constants | Iterator's Item is a type chosen by the implementation, not by the caller. | Closures and iterators, named at last | rust/ch08/src/chain.rs |
Closures and iterators#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| closures | |x| x + 1 captures its surroundings by reference, by mutable reference, or by move. | Closures and iterators, named at last | rust/ch08/src/cmd.rs |
| Fn, FnMut and FnOnce | What a closure may do with what it captured, which decides where it can be passed. | Closures and iterators, named at last | rust/ch08/src/cmd.rs |
| iterators and adaptors | map, filter, enumerate, zip, take and the rest, each a lazy wrapper around the last. | Closures and iterators, named at last | rust/ch08/src/store.rs |
| collect and the turbofish | collect::<Vec<_>>() builds any collection, and the type says which. | Closures and iterators, named at last | rust/ch08/src/cmd.rs |
| implementing Iterator | One method, next, and every adaptor works on your type. | Closures and iterators, named at last | rust/ch08/src/chain.rs |
| iter, iter_mut and into_iter | Borrow the elements, borrow them mutably, or take them. | Closures and iterators, named at last | rust/ch08/src/chain.rs |
| laziness | Nothing runs until something consumes the chain, so an unused map is a warning. | Closures and iterators, named at last | rust/ch08/src/store.rs |
Errors#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| custom error enums | One enum with a variant per failure, matched exhaustively by the caller. | One enum for everything that can go wrong | rust/ch04/src/resp.rs |
| Display and Error | Implement Display for the message and Error to join the ecosystem. | One enum for everything that can go wrong | rust/ch04/src/resp.rs |
| From and ? | ? converts the error with From, so an io::Error becomes your KvError without a line of code. | One enum for everything that can go wrong | rust/ch04/src/resp.rs |
| Box<dyn Error> | A boxed trait object that can hold any error type at all. A main or a tool that only reports errors returns Result<(), Box<dyn Error>> and uses ? on every error it meets, at the cost of no longer being able to match on the kind. | Appendix E | |
| thiserror and anyhow | The two crates most projects use for errors. thiserror derives Display and From for an error enum like the chapter 4 KvError, so that the impl blocks are one attribute each. anyhow is the Box<dyn Error> idea with context, for programs that report errors and never match on them. | Appendix E | |
| panics and the unwrap policy | unwrap and expect are for bugs that cannot happen, with expect saying why. | ?, expect and unwrap | rust/ch03/src/main.rs |
| Result from main | fn main() -> Result<(), Box<dyn Error>> prints the error and exits non-zero. | A main that returns a Result | rust/ch03/src/main.rs |
Concurrency#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| threads | thread::spawn takes a closure that must be 'static, and join waits. | Rust: a thread for each client, and a compiler that checks the sharing | rust/ch09/src/main.rs |
| scoped threads | thread::scope lets threads borrow from the stack, because all are joined before it returns. | Rust again: the load generator with scoped threads | rust/ch09/src/bin/kvload.rs |
| Send and Sync | Marker traits the compiler checks: safe to move to a thread, safe to share between threads. | Send and Sync | |
| Arc<Mutex<T>> | Shared ownership plus one writer at a time, the shape most shared state takes. | Rust: a thread for each client, and a compiler that checks the sharing | rust/ch09/src/main.rs |
| RwLock | Many readers or one writer, when reads dominate. | Rust: one allocation, a profile, four experiments, and a lock | rust/ch13/src/main.rs |
| mpsc channels | Many senders, one receiver, and the receiver's loop ends when every sender drops. | Rust again: the load generator with scoped threads | rust/ch09/src/bin/kvload.rs |
| atomics | AtomicBool and AtomicUsize with an Ordering, for flags and counters. | Atomics | rust/ch09/src/bin/kvload.rs |
| async and await | An async fn returns a Future, and await pauses it instead of blocking the thread. | Rust once more: tasks on tokio | rust/ch09/src/bin/kv_async.rs |
| what a Future is | A state machine with a poll method that a runtime drives. | What a future is | |
| Pin | A future that borrows itself cannot move, and Pin is the promise that it will not. | Pin in one paragraph | |
| tokio | spawn, tasks, the multi-threaded runtime, and its own Mutex and channels. | Rust once more: tasks on tokio | rust/ch09/src/bin/kv_async.rs |
| select! | Wait on several futures and take the first, with cancellation of the rest. | select!, cancellation and blocking | |
| cancellation | Dropping a future stops it, which is both convenient and a trap for half-done work. | select!, cancellation and blocking | |
| blocking inside async | A std Mutex held across await, or a long computation, stalls the whole runtime. | select!, cancellation and blocking |
Unsafe and FFI#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| raw pointers | *const T and *mut T, with no borrow checking, dereferenced only in unsafe. | The five powers, and the pointer they are for | rust/ch15/src/cmatch.rs |
| unsafe blocks and functions | The keyword that marks where the compiler's rules stop and yours start. | Rust: extern and unsafe | rust/ch15/src/cmatch.rs |
| the five powers | Dereference a raw pointer, call unsafe code, touch a mutable static, implement an unsafe trait, read a union. | The five powers, and the pointer they are for | |
| repr(C) | Lay a struct out as C would, so both sides agree on the bytes. | The five powers, and the pointer they are for | rust/ch15/src/cmatch.rs |
| extern blocks | Declare C functions, unsafe since the 2024 edition, and export your own with no_mangle. | Rust: extern and unsafe | rust/ch15/src/cmatch.rs |
| miri | The interpreter that runs tests and reports undefined behaviour in unsafe code. | Rust: the compiler already ran, and the tools check the rest | |
| bindgen | Generates the extern declarations from a C header. | The five powers, and the pointer they are for |
Macros#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| macro_rules! | Pattern-based macros that write code from a template, for tables and repetition. | A table of fn pointers, built by a macro | rust/ch05/src/cmd.rs |
| derive and attribute macros | #[derive(...)] and #[tokio::main] are procedural macros, which run Rust code at compile time to generate code. They come from crates, and the book uses them without writing one. A macro_rules! macro, like the command table in chapter 5, is the simpler kind. | Appendix E | |
| println!, format!, vec! and assert! | The standard macros. println! and format! take a format string checked at compile time with a variable number of arguments, which a function cannot. vec! builds a Vec from a list, and assert! and assert_eq! are the test checks that print both sides when they fail. | Appendix E |
Testing and tooling#
| Concept | In one line | Taught in | In kv |
|---|---|---|---|
| unit, integration and doc tests | #[cfg(test)] modules in the file, the tests/ directory, and examples in doc comments. | Rust: the compiler already ran, and the tools check the rest | rust/ch12/tests/roundtrip.rs |
| cargo test flags | Filter by name, show output, run one thread, and ignore the slow ones. | cargo test, and what comes after the dashes | |
| benchmarks with criterion | Stable Rust has no built-in bench, and criterion is the crate that fills the gap. | Rust: one allocation, a profile, four experiments, and a lock | rust/ch13/examples/bench_parse.rs |
| cargo-fuzz | libFuzzer targets in a fuzz/ directory, run on nightly. | Rust: the compiler already ran, and the tools check the rest | rust/ch12/fuzz/fuzz_targets/parse.rs |
| cargo doc | Builds the documentation of your crate and every dependency, offline. | Rust: the compiler already ran, and the tools check the rest | rust/ch12/src/resp.rs |
End of the book