One Server, Three Languages#

Three terminals, one server: C, Go and Rust each answering a redis-cli PING C $ ./kv $ redis-cli PING PONG 1972 trust the programmer GO $ go run . $ redis-cli PING PONG 2009 keep it simple, add a GC RUST $ cargo run $ redis-cli PING PONG 2015 safety without a GC the same protocol, the same tests, the same client

Learn C, Go and Rust from scratch by building the same Redis-compatible server three times, and understand the power and the weakness of each.

Compiled bykodebot

October 2026 edition

C17 with clang 17 (notes on C23 and gcc) Go 1.27 Rust 1.99, edition 2024, tokio 1 redis-cli 8.10

16 chapters and 43 exercises. Reading time about 8 hours 35 minutes, measured by the build. Building time is yours. This copy was built on 12 October 2026.

Preface#

You learn by building. So this book gives you one thing to build, and asks you to build it three times.

The thing is kv, a small server that speaks the Redis protocol. From chapter 4 on, the real redis-cli can talk to it. By the end it keeps keys and values in a hash table you wrote and serves many clients at once. It removes a key when its time limit runs out, survives a crash by replaying a log, and answers redis-benchmark at a respectable speed.

You build kv in C first, then in Go, then in Rust. The order is the order of history. C was made in 1972 to write an operating system. Go and Rust were both made, about forty years later, to fix the problems that forty years of C had shown. Go fixed them by making the language smaller and adding a garbage collector. Rust fixed them by making the compiler stricter. Building the same server in all three is the clearest way to feel the difference.

Who this book is for#

You wrote some C at university and have forgotten most of it. You used Go for a while and have forgotten that too. You have heard that Rust is good for systems programming and have never written any. You want to understand what each language is good at, what each one is bad at, and what the tools around each one are like. You do not want a long book.

No chapter assumes that you remember the chapter before. Each one restates the few facts it needs. Many readers read English as a second language, so every sentence holds one fact and no idiom.

How to read it#

Every teaching chapter has the same shape. A short lead says what the chapter is about. A table compares the concept across the three languages. Then you build the feature in C, in Go, and in Rust. Each build has an exercise. The exercise shows a design and partial code. You write the rest. If you are stuck, open the folded solution under the exercise. It holds the complete listing. The chapter closes with a verdict, a small scorecard of which language was strong, which was weak, and why.

Four kinds of box appear in the text. A history box says why a language is the way it is. A rough edge box warns about a mistake that many people make. A practice box names the convention professionals follow. A platforms box explains where macOS, Linux and Windows differ.

Code appears side by side on a wide screen and in tabs on a narrow one. Press 1, 2 or 3 to follow one language through the whole book, or use the Follow buttons in the sidebar. Press โŒ˜K to search.

The code#

Every listing in this book is taken from a file in the code/ folder of the book's repository. There is one complete copy of kv for each chapter and each language, so code/rust/ch09 is exactly what chapter 9 ends with. The same test script drives all three versions through redis-cli and compares the replies. The build of the book reads the listings from those files, so the text and the code cannot disagree.

The book targets C17 with clang 17 (notes on C23 and gcc), Go 1.27 and Rust 1.99, edition 2024, tokio 1, with redis-cli 8.10 as the client. Version-sensitive facts were verified from 12 October 2026 onwards. The edition is October 2026.

This copy was built on 12 October 2026. Reading time is about 8 hours 35 minutes, measured by the build. The time you spend building is yours, and it is the point.

Fifty years, three answers#

Origins · Chapter 01· 13 min read

C was made in 1972 to write an operating system. Go and Rust were made about forty years later to fix what forty years of C had shown. Go made the language smaller and added a garbage collector. Rust made the compiler stricter. This chapter tells the three stories, because every design choice in this book has a reason in them.

One problem, three answers#

Every program in this book does the same job. It takes bytes from a network socket, keeps some of them in memory, and sends bytes back. That is systems programming. The hardware has not changed its nature since 1972. Memory is still a flat run of bytes. A socket is still a stream. What changed is how much a mistake costs.

The timeline below is the shape of the whole chapter. Read it once now and again at the end.

1965 1975 1985 1995 2005 2015 2025 C 1972 Unix in C, 1973 ANSI C, 1989 Rust 2006 Go 2007 Go 1, 2012 Rust 1.0, 2015 forty years of C: memory bugs, slow builds, hard concurrency
The timeline of the chapter. C in 1972, Unix in C in 1973, ANSI C in 1989. Rust began in 2006 and Go in 2007. Go 1 in 2012, Rust 1.0 in 2015. Both new languages answer the same forty years of C.

1969 to 1973: why C#

Bell Labs, 1969. Ken Thompson has a DEC PDP-7 that nobody else wants, a machine with little memory and almost no software. On it he has written the first Unix. He needs a language to write programs for it, and the languages he knows are too big for the machine. So he makes a small one and calls it B. B was BCPL, a language that Martin Richards had designed in the mid-1960s, "squeezed into 8K bytes of memory", in Dennis Ritchie's words. Ritchie also wrote that "B can be thought of as C without types".

B had one type. Every value was a machine word. On the PDP-7 that was fine, because the machine addressed memory in words. In 1970 the group got a PDP-11. The PDP-11 addressed memory in bytes. It also had no floating-point hardware at first. A language with one word-sized type did not fit a machine that worked in bytes. B was also slow, because its compiler produced code that an interpreter ran, not machine code.

So in 1971 Ritchie added a character type to B and made it produce PDP-11 machine code. During 1971 to 1973 he added more types and then structures. The result was C. In the summer of 1973 the Unix kernel was rewritten in C. An attempt in 1972 had failed. The attempt in 1973 succeeded because C now had structures, which the kernel needed to describe its own tables.

๐Ÿ“œWhy C has pointers and arrays the way it does

Ritchie called the rule that turns an array into a pointer "the crucial jump in the evolutionary chain between typeless BCPL and typed C". An array name in C is the address of its first element. That rule is why buf[3] and *(buf + 3) mean the same thing. It is also why C cannot check an array index at run time. The length is not part of the value.

Unix spread, and C spread with it. Brian Kernighan and Ritchie published The C Programming Language in 1978. In the same years the language was made portable. A compiler that could produce code for more than one machine was described in 1978, and Unix was moved to the Interdata 8/32. From then on, C was the language that operating systems were written in. In 1983 an ANSI committee began to write a standard. It finished at the end of 1989, and ISO adopted it in 1990. Newer standards followed in 1999, 2011, 2018 and 2024. The 2018 standard is called C17 and is the one this book uses. The 2024 standard is called C23.

In 1995, when it began the next revision, the committee wrote down what it wanted to keep. It called this the spirit of C, and listed it as five short rules. The first was "Trust the programmer". The second was "Don't prevent the programmer from doing what needs to be done". The fifth was "Make it fast, even if it is not guaranteed to be portable". Those rules explain almost everything about C that surprises a reader coming from another language. C does not check an array index. C does not know when to free memory. C lets you add one to a pointer and read whatever is there. The programmer is trusted to be right.

Ritchie's own summary of the language was "quirky, flawed, and an enormous success". Every word of that is true.

What forty years of C cost#

Trusting the programmer worked when one careful person wrote a kernel. It worked less well when thousands of people wrote browsers, and strangers on the internet sent those browsers input. The first stranger arrived in 1988.

๐Ÿ“ฐThe Morris worm, 1988

On the evening of 2 November 1988 a program began to copy itself from one Unix machine to the next, across the whole internet. Robert Tappan Morris, a graduate student at Cornell, released it from a machine at MIT. One of the ways it got in was a buffer overflow in the finger service. That service was a C program that read a request into a fixed array without checking the length. The worm sent a request longer than the array, and the bytes past the end became code that ran. About 6,000 machines were infected, by the figure most often reported. The case became the first felony conviction under the Computer Fraud and Abuse Act. The bug was a C program trusting its input. Thirty years later it was still the most common kind.

In 2019 Microsoft reported that about 70 percent of the vulnerabilities it reported each year were memory safety bugs. The Chromium project reported the same figure for its own code. Around 70 percent of its serious security bugs, in a sample of 912 high and critical bugs since 2015, were memory safety problems. A memory safety bug is a read or write that the program should not have made. An index past the end of an array is one. Using memory after it was freed is another. C allows both, because of the rules above.

Memory was one cost. Build time was another. By 2007, Rob Pike wrote, a large C++ binary at Google took 45 minutes to build. The C++ language inherited the C way of sharing declarations, which is to copy a header file into every file that includes it. In one measurement, 4.2 megabytes of source expanded to more than 8 gigabytes of compiler input. That is about 2,000 bytes of work for every byte a programmer wrote. The problem was old. In 1984 a Unix build was found to include the same header file 37 times for one program.

The third cost was concurrency. By 2007 a computer had several processors, and a server had to use all of them. C gives you threads and locks and nothing else. A mistake with a lock is a data race, and a data race in C is undefined behaviour. The program may do anything.

The C committee has said this itself. The charter for the 2024 standard states that "Trust the programmer, as a goal, is outdated in respect to the security and safety programming communities". The committee kept the goal. A sixth rule, added for the 2011 standard, says "Make support for safety and security demonstrable". Two other groups of people did not wait for the committee. They made new languages.

2007 to 2012: why Go#

Google, 21 September 2007. A C++ build is running, and it will take 45 minutes. While it runs, Robert Griesemer, Rob Pike and Ken Thompson start to sketch a new language on a whiteboard. Thompson had made B in 1969. Now he was in the room again, 38 years later, to replace what B had become. Pike's own account says the idea came during one of those builds. The Go FAQ says the language "was born out of frustration with existing languages and environments". It names the frustrations. Large programs were too complex to build. Multiprocessor computers were "becoming universal but most languages offered little help to program them". And a programmer had to "choose either efficient compilation, efficient execution, or ease of programming".

The answers in Go are all answers to those frustrations. Each one is a thing C does not do.

  • A garbage collector. The FAQ says the lifetime of objects is "one of the biggest sources of bookkeeping in systems programs". Go frees memory for you. The collector pauses the program, but by now the pauses are "often to the sub-millisecond range".
  • Goroutines and channels. The concurrency comes from the idea of communicating sequential processes, which Tony Hoare described in 1978. Go took it through two earlier languages, Newsqueak and Limbo. A goroutine is cheap, and a channel sends a value from one to another.
  • Fast builds. A Go file names only the packages it imports, and never copies in a header. An import that is not used is a compile error. So the compiler reads each dependency once, and the build of a large program takes seconds.
  • No exceptions. The FAQ says the try-catch-finally idiom "results in convoluted code". A Go function returns its error as a second value, and the caller checks it. Go keeps panic for the cases that are truly exceptional.
  • One format. The tool gofmt decides how code looks. Pike wrote that this removed "an entire class of argument between programmers".

The syntax came from C. The packages and declarations came from the Pascal, Modula and Oberon family, which Griesemer knew from Niklaus Wirth. Go became an open source project on 10 November 2009. Go 1 was released on 28 March 2012 with a promise. A program written for Go 1 "will continue to compile and run correctly, unchanged, over the lifetime of that specification". The promise still holds. The version in this book is Go 1.27, from August 2026.

๐Ÿ“œWhy Go looks plain

Go was designed for a company with a very large code base and a very large number of programmers. A language that is small is a language that everyone on the team reads the same way. That is why Go has no classes, no generics until 2022, and one loop keyword. The plainness is the point. When Go feels limited in this book, remember who it was built for.

2006 to 2015: why Rust#

Mozilla, 2006. The company's product is a web browser written in C++, and a browser is the program that the 70 percent figure above describes. Every memory bug in it is a hole that a web page can reach. Graydon Hoare, an engineer there, starts a language of his own in his spare time and calls it Rust. He works on it alone for three years. From late 2009 Mozilla pays him to work on it, and in 2010 he presents it to the company. He hopes it will be "slightly less annoying" than the languages Mozilla uses. The two aims he gives are better memory safety and more concurrency.

Go answered the memory problem with a garbage collector. Mozilla did not want its engine to pause for a collector, or to pay for one in memory. So Rust needed a different answer. It found one in the compiler. Every value in Rust has exactly one owner. When the owner goes out of scope, the value is freed. A value can be lent to another part of the program, but the compiler checks every loan. A loan may not outlive its owner. While one part of the program can change a value, no other part can read it. These rules are checked before the program runs, so they cost nothing at run time. The checker is called the borrow checker, and it is the part of Rust that new programmers argue with.

The same rules solve the concurrency problem. A data race needs two threads with access to one value, and at least one of them writing. The ownership rules do not allow that. A Rust program that compiles, and uses no unsafe, has no data races. The compiler proves it before the program runs. Go has a race detector, but the detector finds a race only when a test meets one.

Rust 1.0 was released on 15 May 2015. The announcement called it "the official beginning of our commitment to stability". It described the aim as "low-level control over performance with high-level convenience and safety guarantees", "without requiring a garbage collector or runtime". A new version has come out about every six weeks since then. The versions this book uses are 1.97 from July 2026, 1.98 from August and 1.99 from October. Larger changes to the language arrive as an edition, and a program chooses its edition. This book uses the 2024 edition, which shipped with Rust 1.85 in February 2025.

In February 2021 the Rust Foundation was created to own the language, with AWS, Google, Huawei, Microsoft and Mozilla as founding members. Mozilla, "the original home of the Rust project", gave it the trademarks and the package registry. The language no longer belongs to one company.

๐Ÿ“œWhy Rust is hard at first

C trusts you. Go checks a little, and a garbage collector handles the rest. Rust refuses to compile a program until it can prove that the memory rules hold. The proof is yours to supply, in the types you write. That is why a Rust program takes longer to write the first time and why the compiler's messages are so long. The compiler is doing the work that in C you do in your head, and in Go you leave to the collector.

Two answers to one question#

The question was what to do about the costs of trusting the programmer. Go and Rust gave different answers, and the table shows where they differ. The rest of the book is the long form of this table.

The costCGoRust
Memory that is freed too earlya crash or a security holeimpossible, the collector frees itdoes not compile, the owner frees it
Memory that is never freeda leakunreachable objects are always freed, but a kept reference is a leakfreed at the end of scope, but a reference cycle or mem::forget can leak
Two threads writing one valueundefined behavioura race the detector may finddoes not compile
A forgotten error checkthe error is lostthe error is lost, but a linter can warnthe compiler warns on an unused Result
A slow buildthe headers are copied into every filepackages are read once, builds take secondscrates are compiled once, but the checks take time
The cost you payyour attention, all the timememory and pauses for the collectorthe time to satisfy the compiler

No answer is free. Go pays with a collector and with some things it refuses to let you express. Rust pays with the time it takes to learn, and with code that spells out more than C or Go would ask for. C pays nothing in advance, and everything later. In each chapter of this book you will feel one of those costs directly, because you will build the same thing three times.

What this means for kv#

The server you will build, kv, is a small version of Redis. Redis itself is written in C, and is a well written C program. It was started by Salvatore Sanfilippo in 2009, the year Go was announced. It keeps every key in memory, serves many clients from one thread, and writes an append-only log to survive a crash. Each of those three ideas is a chapter in this book. In each one you see the C answer first, then what Go and Rust change.

Next

Three languages, three toolchains, and one line of output each. Chapter 2 builds that line three times and ends with three files of very different sizes. Two of them are carrying something the third one left behind.

Toolchains and hello, three times#

Origins · Chapter 02· 20 min read· 148 lines of code
▶Scene

Three programs, one line of output each. Build all three and look at the files you got. On the author's Mac:

$ ./hello-c; ./hello-go; ./hello-rust
hello from C
hello from Go
hello from Rust
$ ls -l hello-c hello-go hello-rust | awk '{print $5, $9}'
8440 hello-c
2516816 hello-go
414728 hello-rust

The same line, three times. One file is 8 kilobytes, one is 2.5 megabytes, and one sits between them. What is in the big one that is not in the small one? Where did the small one leave it?

Install a C compiler, Go and Rust. Compile one small program in each. Then look at what the compiler made, because a binary is the thing all three languages exist to produce. The chapter also installs redis-cli, which the server will talk to from chapter 4.

kv does not exist yet. By the end of this chapter you have three programs that print a line. You also know what a compiler, a linker and a build tool each do. Every later chapter assumes those three tools are on your path.

The toolchainCGoRust
Compilercc (clang or gcc)gorustc, driven by cargo
Build toolMake, or CMake for large projectsgo build, with no build filecargo build, with Cargo.toml
Project filenone, a Makefile by conventiongo.modCargo.toml
Formatterclang-format, optionalgofmt, universalrustfmt, universal
Hello binary on macOS8 KB, links the C library2.5 MB, the runtime is inside415 KB, the standard library is inside
Where the standard library isa shared file the OS providescompiled into every binarycompiled into every binary

Install the three toolchains#

Each language has one official way to install it. For Go and Rust use that way, and not a package from a distribution, so that the version matches the baseline of this book. The commands are below for each platform.

๐Ÿ–ฅA C compiler on macOS, Linux and Windows

On macOS, run xcode-select --install. It installs Apple's clang, which answers to the name cc. On Debian and Ubuntu, run sudo apt install build-essential, which installs gcc, make and the C library headers. On Windows there are two choices. The Microsoft way is to download the Build Tools for Visual Studio and select the "Desktop development with C++" workload. The compiler is then cl.exe, and you run it from a Developer Command Prompt, because the tools need many environment variables set. The other way is to install WSL and use the Ubuntu line above. The C code in this book is written for POSIX, the Portable Operating System Interface that macOS and Linux share. So WSL is the simpler choice on Windows.

Go is installed from a package. On macOS the package installs into /usr/local/go and adds it to your path. On Linux, download the archive from go.dev/dl and extract it:

$ rm -rf /usr/local/go && tar -C /usr/local -xzf go1.27.0.linux-amd64.tar.gz
$ export PATH=$PATH:/usr/local/go/bin
$ go version
go version go1.27.0 linux/amd64

On Windows, run the MSI installer and open a new command prompt. The check is the same, go version.

Rust is installed by rustup, which also manages versions later. On macOS, Linux and WSL the command is one line:

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ rustc --version
rustc 1.99.0 (b940084d7 2026-09-28)

On Windows, download and run rustup-init.exe. It may ask you to install the Visual Studio C++ Build Tools, because Rust uses the Microsoft linker there. Finally, install the Redis client. On macOS run brew install redis. On Debian and Ubuntu run sudo apt install redis-tools. Check with redis-cli --version.

โš ๏ธGo downloads a newer Go when a module asks for one

The Go you run may not be the Go you installed. A go.mod file names a Go version. Since Go 1.21, when that version is newer than the installed one, the go command downloads the newer toolchain and uses it. The author's machine had Go 1.23 installed, and go version printed 1.27 inside this book's code. That is convenient, and it is also surprising. The environment variable GOTOOLCHAIN=local turns it off.

C: from source to binary in four steps#

The C program is six lines:

#include <stdio.h>

int main(void) {
    printf("hello from C\n");
    return 0;
}
hello/c/hello.c · region hello

Compile and run it. The flags after -std=c17 turn on every useful warning and make each warning an error. Use them on every C file you ever compile:

$ cc -std=c17 -Wall -Wextra -Werror -o hello hello.c
$ ./hello
hello from C

The one command ran four tools, and it helps to see each one. The preprocessor copies stdio.h into the file, and expands macros. Those six lines become 571. The compiler turns C into assembly. The assembler turns assembly into an object file, which is machine code with holes where the addresses of other functions go. The linker fills the holes and joins the object file with the C library. You can stop after any step:

$ cc -std=c17 -E hello.c | wc -l        # preprocess only
     571
$ cc -std=c17 -S -O2 -o - hello.c       # compile to assembly, print it
_main:
        pushq   %rbp
        movq    %rsp, %rbp
        leaq    L_str(%rip), %rdi
        callq   _puts
        xorl    %eax, %eax
        popq    %rbp
        retq
$ cc -std=c17 -c -o hello.o hello.c     # assemble to an object file
$ nm hello.o                            # the names it defines (T) and needs (U)
0000000000000000 T _main
                 U _printf

Two details in that output matter for the rest of the book. The assembly calls _puts, not _printf. The compiler saw a format string with no arguments and chose the cheaper function. The compiler is allowed to do that, and much more, whenever the result is the same. The object file was built without -O2, so it lists _printf as undefined, marked U. The linker resolves it to the C library, which on macOS is the shared file libSystem.B.dylib. Your binary does not contain printf. It contains a note that says where to find it.

The -O2 is the optimisation level, and it is the reason for puts. At -O0, the default, the compiler translates each line as written, which is what you want under a debugger. At -O2 it may reorder, inline and replace, as long as the program behaves the same. At -O0 the hello program is 34 lines of assembly that call printf. At -O2 it is 29 that call puts. Every release build in this book uses -O2, and chapter 13 measures what it is worth.

๐Ÿ“œWhy C has header files

A C compiler reads one file at a time and knows nothing about the others. For a call to printf to compile, the file must have seen a declaration of printf first. Before header files, a declaration that several files needed was written again in each of them. A header file is that declaration written once and copied in by the preprocessor. The scheme was cheap on a PDP-11 and expensive. By 1984 one Unix program was found to include the same header 37 times. At Google in 2007 the same scheme turned 4.2 megabytes of source into 8 gigabytes of compiler input. Go and Rust both replaced headers with modules, which chapter 7 covers.

Exercise 02.1CWhat size is a word?

Add two lines to main. The first prints the size of an int, a long and a pointer. The second prints the compiler version. The operator sizeof gives a size in bytes, and the format %zu prints one. The macro __VERSION__ is a string that gcc and clang define. It is not part of the C standard, and cl.exe on Windows defines _MSC_VER instead. On the author's machine the output is int 4, long 8, pointer 8 bytes.

int main(void) {
    printf("hello from C\n");
    /* your two lines here */
    return 0;
}
Stuck?Show the full listing
#include <stdio.h>

int main(void) {
    printf("hello from C\n");
    /* sizeof gives a size in bytes. The %zu format prints a size_t. */
    printf("int %zu, long %zu, pointer %zu bytes\n",
           sizeof(int), sizeof(long), sizeof(void *));
    printf("compiler " __VERSION__ "\n");
    return 0;
}
hello/c/hello.c
โœ…Compile with -Wall -Wextra -Werror from the first day C

The C compiler accepts a great many programs that are wrong. The warnings catch a large share of them, and -Werror stops you from ignoring one. Every C file in this book is compiled with those three flags, and with -std=c17 so that the compiler applies one known standard.

C: two files, one binary#

kv will not fit in one file for long. Split the hello program in two and the four steps above explain how C programs of any size are built. The second file holds the greeting. Its header declares the one function other files may call:

#ifndef GREET_H
#define GREET_H

/* Declared here, defined in greet.c. Every file that includes this header can call it. */
void greet(const char *who);

#endif
hello/c/greet.h · region header
#include <stdio.h>

#include "greet.h"

/* static: only this file can see the name. The linker never hears of it. */
static const char *prefix(void) {
    return "hello from";
}

/* No static: this name has external linkage. units.c calls it through greet.h. */
void greet(const char *who) {
    printf("%s %s\n", prefix(), who);
}
hello/c/greet.c · region unit

The first file includes the header and calls the function:

#include "greet.h"

int main(void) {
    greet("two files");
    return 0;
}
hello/c/units.c · region unit

Each .c file, with the headers pasted in, is a translation unit. The compiler reads one unit at a time and knows nothing about the others. So units.c compiles because greet.h promised that greet exists somewhere. The Makefile compiles each unit to an object file and links the two object files:

# Two translation units. Each .c is compiled alone into a .o, and the linker joins them.
units: units.o greet.o
	$(CC) $(CFLAGS) -o $@ units.o greet.o

units.o: units.c greet.h
	$(CC) $(CFLAGS) -c units.c

greet.o: greet.c greet.h
	$(CC) $(CFLAGS) -c greet.c
hello/c/Makefile · region units

nm shows what each object file defines and what it still needs. The letters are the linkage of each name. T is a function this file defines and other files may use. U is a name this file uses and someone else must define:

$ make units
$ nm units.o
                 U _greet
0000000000000000 T _main
$ nm greet.o
0000000000000000 T _greet
                 U _printf

prefix is missing from greet.o. It was declared static, so its name never leaves the file, and at -O2 the compiler pasted its body into greet. A function without static has external linkage, which means every file in the program shares its name. That is why every public name in kv starts with kv_, and why every private one is static. Leave an object file out and the linker says what is missing:

$ cc -o units units.o
Undefined symbols for architecture x86_64:
  "_greet", referenced from:
      _main in units.o
ld: symbol(s) not found for architecture x86_64

Hold on to that error. In chapter 7 the same mechanism holds the modules of kv together. In chapter 14 it is what linking a library means. In chapter 15 it is how Go and Rust call C at all.

Go: one command, one binary#

A Go program is a package. The file below is the whole of package main, and the function main is where the program starts:

package main

import (
	"fmt"
	"runtime"
	"unsafe"
)

func main() {
	fmt.Println("hello from Go")
}
hello/go/main.go · region hello

Go needs one more file, go.mod, which names the module and the Go version it is written for:

module hello

go 1.27
hello/go/go.mod

Then one command compiles and links. There is no Makefile, because the go tool finds every file in the folder itself:

$ go build -o hello .
$ ./hello
hello from Go
$ go run .                  # compile and run in one step, for development
hello from Go

The binary is 2.5 megabytes, three hundred times the C one. It contains the Go runtime, which is the garbage collector, the scheduler for goroutines, and the code that talks to the operating system. On Linux it has no dependency on the C library at all. On macOS it calls libSystem for system calls and nothing else. That is why a Go binary can be copied to another machine of the same kind and run there with nothing installed. Chapter 14 uses that fact.

Every variable starts at zero#

Look at the two declarations in the solution below, var i int and var p *int. The declarations give no value, and both variables are usable. In Go every variable begins as its zero value. A number is 0, a string is empty, a bool is false. A pointer, a slice, a map or a function is nil. A struct is the zero value of each field. There is no uninitialised memory to read by mistake, which in C is one of the ways a program is wrong without a warning. The rule shapes how Go code is written. A type is designed so that its zero value is ready to use, and most types need no constructor.

The documentation is in the source#

The comment above an exported name is its documentation, and go doc prints it. Ask about the function the hello program calls:

$ go doc fmt.Println
package fmt // import "fmt"

func Println(a ...any) (n int, err error)
    Println formats using the default formats for its operands and writes to

The same text is what pkg.go.dev shows for every public module. When you write a package in chapter 7, a sentence above each exported name is all the documentation needs.

Exercise 02.2GoWhat does the runtime know?

Add two lines to main. The first prints the size of an int and of a pointer, with unsafe.Sizeof, which works like sizeof in C. The second prints runtime.Version(), runtime.GOOS and runtime.GOARCH. Import the packages runtime and unsafe. An import you do not use is an error in Go, so add each import together with the line that uses it.

func main() {
	fmt.Println("hello from Go")
	// your two lines here
}
Stuck?Show the full listing
package main

import (
	"fmt"
	"runtime"
	"unsafe"
)

func main() {
	fmt.Println("hello from Go")
	// unsafe.Sizeof is a compile-time constant, like sizeof in C.
	var i int
	var p *int
	fmt.Printf("int %d, pointer %d bytes\n", unsafe.Sizeof(i), unsafe.Sizeof(p))
	fmt.Println("compiler", runtime.Version(), runtime.GOOS, runtime.GOARCH)
}
hello/go/main.go
โœ…Run gofmt and go vet before you commit Go

gofmt -l . lists the files whose layout is not the standard one, and gofmt -w . fixes them. go vet ./... finds mistakes the compiler accepts, such as a wrong format verb. Both tools come with Go, and every Go project runs them. There is no setting for the style, which is the point.

Rust: cargo owns the project#

In Rust you rarely run the compiler, rustc, yourself. The tool cargo creates the project, builds it, runs it, tests it and fetches its dependencies. cargo new hello makes a folder with two files. The first is the project file:

[package]
name = "hello"
version = "0.1.0"
edition = "2024"

[dependencies]
hello/rust/Cargo.toml

The edition line names the version of the language rules. There are four, 2015, 2018, 2021 and 2024, and a new one comes every three years. An edition may change what code means, such as which names are keywords, so a crate says which edition it is written for. Crates of different editions link together, so a dependency written in 2018 compiles unchanged in a 2024 project. That is how Rust changes the language without breaking the world. The 2024 edition is the current one and this book uses it. The second file is src/main.rs:

fn main() {
    println!("hello from Rust");
}
hello/rust/src/main.rs · region hello

The exclamation mark means that println! is a macro, not a function. A macro can check the format string at compile time, and this one does. A build has two modes. The default is a debug build, which compiles quickly. The release build is slower to compile and runs faster:

$ cargo build --release
   Compiling hello v0.1.0
    Finished `release` profile [optimized] target(s)
$ ./target/release/hello
hello from Rust
$ cargo run --release       # build if needed, then run

Everything cargo makes goes under target/. cargo new already lists it in the project's .gitignore. The binary is 415 kilobytes. Like the Go one it contains its standard library. Like the C one it calls the C library of the operating system for a call such as write. Rust sits between C and Go.

Exercise 02.3RustWhat size is a usize?

Add two lines to main. The first prints the size of four types, with size_of::<T>(), which the compiler evaluates at compile time. The four types are i32, i64, usize and a reference. The second prints std::env::consts::OS and std::env::consts::ARCH. Then run cargo clippy and cargo fmt, and see whether either tool reports something.

fn main() {
    println!("hello from Rust");
    // your two lines here
}
Stuck?Show the full listing
fn main() {
    println!("hello from Rust");
    // size_of is evaluated at compile time, like sizeof in C.
    println!(
        "i32 {}, i64 {}, usize {}, pointer {} bytes",
        size_of::<i32>(),
        size_of::<i64>(),
        size_of::<usize>(),
        size_of::<&i32>()
    );
    println!("target {} {}", std::env::consts::OS, std::env::consts::ARCH);
}
hello/rust/src/main.rs
โœ…Run rustfmt and clippy before you commit Rust

cargo fmt formats the project. cargo clippy is a linter with more than 800 checks, from a needless clone to a loop that should be an iterator. Every Rust snapshot in this book passes cargo clippy -- -D warnings, which makes each warning an error, and cargo fmt --check.

The documentation is in the source, and it is tested#

A comment that starts with three slashes, ///, is a doc comment. cargo doc turns every doc comment in your crate and in every dependency into a website under target/doc, offline:

$ cargo doc
 Documenting hello v0.1.0
    Finished `dev` profile [unoptimized + debuginfo] target(s)
$ ls target/doc
crates.js  hello  help.html

A code block inside a doc comment is a test. cargo test compiles it and runs it, so an example in the documentation cannot go stale. Chapter 12 writes one.

What a binary is#

All three programs are files the operating system can load and run. On macOS the format is called Mach-O. On Linux it is called ELF, the Executable and Linkable Format. On Windows it is called PE, the Portable Executable. The file command names the format:

$ file hello
hello: Mach-O 64-bit executable x86_64

A binary holds machine code, the constant data such as the string to print, and a list of the shared libraries it needs. On macOS the list is printed by otool -L, on Linux by ldd, and on Windows by dumpbin /dependents. The three hello programs name the same library:

$ otool -L hello
hello:
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1356.0.0)

So the three binaries differ in what they carry, not in what they are. The C binary carries almost nothing and depends on the system for everything. The Go binary carries its own runtime. The Rust binary carries its standard library and depends on the system for the rest. The sizes in the table at the top of the chapter follow from that. The programs print the same word sizes too. An int is 4 bytes in C and 8 in Go. A pointer is 8 bytes in all three. A Rust usize is the size of a pointer. The pointer size comes from the machine. The size of int is a choice each language made, and C and Go chose differently.

๐Ÿ–ฅThe same program on Linux and Windows

On Linux the C binary is an ELF file and links libc.so.6, the GNU C library. The Go binary links nothing at all, because Go makes its system calls directly on Linux. On Windows the C program links the Windows C runtime, and Rust links the system libraries it needs. Go loads kernel32.dll and a few other system libraries at start, because Windows has no stable system call interface. The chapter on shipping explains what that means for copying a binary from one machine to another.

Build tools, and the lack of one#

C has no build tool of its own. The convention is make, which Stuart Feldman wrote at Bell Labs in 1976. It reads a file called Makefile. The file lists each output, the inputs it depends on, and the command that makes it. The Makefile for the hello program is short:

CC     ?= cc
CFLAGS ?= -std=c17 -Wall -Wextra -Werror -O2

hello: hello.c
	$(CC) $(CFLAGS) -o $@ hello.c

# Two translation units. Each .c is compiled alone into a .o, and the linker joins them.
units: units.o greet.o
	$(CC) $(CFLAGS) -o $@ units.o greet.o

units.o: units.c greet.h
	$(CC) $(CFLAGS) -c units.c

greet.o: greet.c greet.h
	$(CC) $(CFLAGS) -c greet.c

clean:
	rm -rf hello units *.o hello.dSYM units.dSYM

.PHONY: clean
hello/c/Makefile

The line hello: hello.c says that hello is made from hello.c, and the indented line below says how. make runs the command only when the input is newer than the output. The ?= sets a variable only when the environment has not set it, which is how the book's test script adds sanitizer flags later. Large C projects use CMake, which writes Makefiles for you from a description that works on every platform. The Redis source, by comparison, is plain Make.

โš ๏ธA Makefile recipe begins with a tab C

Eight spaces are not a tab. The command line under a rule must start with a tab character, and an editor that converts tabs to spaces produces the error missing separator. Feldman has said the tab was a workaround in the first version, kept so as not to break the first users. Every C programmer since has met it.

Go and Rust each come with a build tool, and each tool is the only one anyone uses. For Go the description of the project is go.mod, and the build is go build. For Rust it is Cargo.toml and cargo build. Both tools also run tests, fetch dependencies, and format code. The difference from C is not that these tools are cleverer than Make. It is that there is one of them, so every project looks the same.

Verdict##

TopicCGoRustWhy
InstallingOKStrongStrongA C compiler is one system command away on macOS and Linux. Go and Rust are one download each, on every platform.
BuildingWeakStrongStrongC has no build tool, only a convention. Go and Rust each have one tool that every project uses.
Seeing what you madeStrongOKOKC shows every step from preprocessor to linker. Go and Rust hide the steps, which is usually what you want.
Binary sizeStrongWeakOK8 KB for C, 2.5 MB for Go because the runtime is inside, 415 KB for Rust.
Running elsewhereWeakStrongOKThe C binary needs the right C library on the target, the Go binary needs nothing, and Rust is in between.
Carry forward
  • Translation unit. A C file with its headers pasted in, compiled alone, and joined to the others by the linker.
  • Zero value. A Go variable starts as zero, nil or empty, so a type is designed to be usable without a constructor.
  • Edition. A Rust crate names the version of the language rules it is written for, and crates of different editions link together.
Next

The three programs run and stop. A server must not stop. Chapter 3 opens a port, and until it does, the Redis client that will test every chapter has nobody to talk to.

Echo server#

Building kv · Chapter 03· 17 min read· 159 lines of code
▶Scene

You have a Redis client and no Redis. Ask it anyway:

$ redis-cli -p 6380 PING
Could not connect to Redis at 127.0.0.1:6380: Connection refused

Nothing is listening on the port. Every chapter from here on changes what the program on that port does. This one makes something listen.

A server that sends back what it receives. It is the smallest program that opens a port, accepts a client and moves bytes in both directions. Every later chapter adds to this one. The chapter teaches the socket calls that all three languages share, and how each language wraps them.

kv has to be reachable over the network. A client such as redis-cli connects to a port, sends bytes, and reads bytes back. Before kv can understand a command it must be able to hold that conversation at all. So the first version of kv is an echo server. Whatever the client sends, the server sends back, one client at a time.

The socket callsCGoRust
Open a portsocket, setsockopt, bind, listennet.Listen("tcp", ":6380")TcpListener::bind("0.0.0.0:6380")
Wait for a clientaccept, returns an intln.Accept(), returns a net.Connlistener.incoming(), yields a TcpStream
Read bytesread(fd, buf, n), returns a count or -1conn.Read(buf), returns a count and an errorstream.read(&mut buf), returns a Result
Copy in to outa loop you write, with a short-write loop insideio.Copy(conn, conn)io::copy(&mut reader, &mut writer)
Closeclose(fd), which you must rememberdefer conn.Close()dropped at the end of scope
An error-1 and a number in errnoa second return valuean Err you must handle or pass on

The four calls in the first row are all there is. A socket is a file descriptor that the operating system connects to the network instead of to a disk. To bind is to give the socket an address and a port. To listen is to tell the operating system that this socket accepts connections, and how many may wait. To accept is to block until a client connects, and to get a second socket that talks to that one client. The first socket keeps listening. The second one is read and written like a file, and closed like one.

๐Ÿ“œWhy every language has the same four calls

The calls come from 4.2BSD, the Berkeley version of Unix released in 1983, and are called the Berkeley sockets API. The operating system standard POSIX, the Portable Operating System Interface, adopted them. Windows copied them in 1992 under the name Winsock, with a few changes. Go and Rust do not replace these calls. They call them. Underneath, net.Listen and TcpListener::bind are the same four calls. So the C version of this chapter is also an explanation of the other two.

C: four calls, and a way to clean up#

The port comes from an environment variable, so the tests can pick one that is free. The C library function getenv returns a pointer to the value, or NULL when the variable is not set. The pointer belongs to the environment, not to you, so you do not free it:

/* The port comes from the environment, so the tests can pick a free one. */
static int kv_port(void) {
    const char *s = getenv("KV_PORT");
    if (s == NULL || *s == '\0') {
        return 6380;
    }
    return atoi(s);
}
c/ch03/kv.c · region port

Opening the port takes the four calls, and each of them can fail. The pattern below is the most common shape in C that does more than one thing that can fail. Each failure prints its reason and jumps to cleanup, which releases what was acquired so far:

/* Open a listening socket on every interface. Returns the socket, or -1 after
 * printing the reason. The caller owns the socket and must close() it. */
static int kv_listen(int port) {
    int fd = socket(AF_INET, SOCK_STREAM, 0);
    if (fd < 0) {
        perror("socket");
        return -1;
    }
    /* Without this, a restart within a minute fails with "address in use". */
    int yes = 1;
    if (setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes) < 0) {
        perror("setsockopt");
        goto cleanup;
    }
    struct sockaddr_in addr;
    memset(&addr, 0, sizeof addr);
    addr.sin_family = AF_INET;
    addr.sin_addr.s_addr = htonl(INADDR_ANY);
    addr.sin_port = htons((uint16_t)port);
    if (bind(fd, (struct sockaddr *)&addr, sizeof addr) < 0) {
        perror("bind");
        goto cleanup;
    }
    if (listen(fd, 16) < 0) {
        perror("listen");
        goto cleanup;
    }
    return fd;

cleanup:
    close(fd);
    return -1;
}
c/ch03/kv.c · region listen

Read the function from the bottom. There is one place where the socket is closed, and every failure after the socket exists reaches it. The goto is not a sign of bad code here. It is how C expresses "undo what I did" without exceptions and without destructors. perror prints the name you give it, a colon, and a message. The message is the text for the current value of errno. That is the global number which every failed system call sets.

โœ…One exit, through cleanup C

A function that acquires more than one resource has one label, cleanup, at the end. Each acquisition that fails jumps there. The cleanup code releases resources in the reverse order of acquisition. Each release must be safe to run when the resource was never acquired. For example, set a descriptor to -1 first, and skip close when it is still -1. The Linux kernel is written this way, and so is Redis.

โš ๏ธAddress already in use C

A server that you restart soon after it stopped fails to bind. When a connection over TCP, the Transmission Control Protocol, closes, the operating system keeps its address in a waiting state for a while. The wait stops late packets from reaching a new program. The option SO_REUSEADDR says that your server may bind anyway. Every server sets it. Forget it and the second start of kv prints bind: Address already in use.

errno: the number a failed call leaves behind#

Every system call in kv_listen reports failure the same way. It returns -1 and puts a number in errno, a variable that errno.h declares. The number says what went wrong. EADDRINUSE is the port being taken, and EINTR is a signal arriving mid-call. strerror turns the number into words, and perror prints your label, a colon and those words, which is what perror("bind") did above. Three rules keep errno honest. Read it right after the call that failed, because the next call may overwrite it. Do not read it after a call that succeeded, because a library function may set it even then and never resets it to zero. And since C11 it is thread-local, so a thread in chapter 11 cannot see another thread's number. Go and Rust carry the same number inside their error values, and chapter 4 shows the three designs side by side.

Serving a client is a loop. read returns the number of bytes it put in the buffer, zero when the client closed the connection, or -1 on an error. Writing those bytes back needs a loop of its own, because write may send fewer bytes than you asked when the network is busy. Both calls can also be interrupted by a signal, in which case errno is EINTR and the right response is to try again.

Exercise 03.1CServe one client

Write kv_serve. Read into a 4 KB buffer until read returns zero or an error. After each read, write everything back, and keep writing until every byte is out. Treat EINTR as "try again" in both calls. The caller closes the client socket.

static void kv_serve(int client) {
    char buf[4096];
    for (;;) {
        ssize_t n = read(client, buf, sizeof buf);
        /* your code: handle n < 0 and n == 0, then write all n bytes back */
    }
}
Stuck?Show the full listing
/* Copy everything the client sends back to it, until the client closes. */
static void kv_serve(int client) {
    char buf[4096];
    for (;;) {
        ssize_t n = read(client, buf, sizeof buf);
        if (n < 0) {
            if (errno == EINTR) {
                continue;
            }
            perror("read");
            return;
        }
        if (n == 0) {
            return; /* the client closed the connection */
        }
        /* write() may send fewer bytes than asked, so loop until all are out */
        ssize_t off = 0;
        while (off < n) {
            ssize_t w = write(client, buf + off, (size_t)(n - off));
            if (w < 0) {
                if (errno == EINTR) {
                    continue;
                }
                perror("write");
                return;
            }
            off += w;
        }
    }
}
c/ch03/kv.c · region serve

The main function ties the pieces together. It accepts one client, serves it to the end, closes it, and accepts the next. A second client waits in the operating system's queue until the first one is done. Chapter 9 removes that limit:

int main(void) {
    int port = kv_port();
    int fd = kv_listen(port);
    if (fd < 0) {
        return 1;
    }
    fprintf(stderr, "kv listening on port %d\n", port);
    for (;;) {
        int client = accept(fd, NULL, NULL);
        if (client < 0) {
            if (errno == EINTR) {
                continue;
            }
            perror("accept");
            break;
        }
        kv_serve(client);
        close(client);
    }
    close(fd);
    return 1;
}
c/ch03/kv.c · region main

Build and run it with the Makefile from chapter 2, which has grown a SANITIZE variable for the tests:

CC      ?= cc
CFLAGS  ?= -std=c17 -Wall -Wextra -Werror -O2 -g
SANITIZE ?=

kv: kv.c
	$(CC) $(CFLAGS) $(SANITIZE) -o $@ kv.c

clean:
	rm -rf kv kv.dSYM

.PHONY: clean
c/ch03/Makefile · region all

Then talk to it with nc, the netcat tool that comes with macOS and most Linux systems. Type a line and the server sends it back:

$ make
$ ./kv &
kv listening on port 6380
$ nc localhost 6380
hello
hello

Press Ctrl-C to stop nc, and kill %1 to stop the server.

Go: the loop you did not have to write#

The Go version has the same three parts. The port function reads the environment with os.Getenv, which returns the empty string when the variable is unset. There is no NULL to check, and nothing to free:

// The port comes from the environment, so the tests can pick a free one.
func port() string {
	if p := os.Getenv("KV_PORT"); p != "" {
		return p
	}
	return "6380"
}
go/ch03/main.go · region port

Serving a client is one call. io.Copy reads from its second argument and writes to its first until the reader returns end of file. Its signature is Copy(dst io.Writer, src io.Reader). An io.Reader is any type with a Read method. An io.Writer is any type with a Write method. A net.Conn has both, so copying a connection to itself is an echo. A file, a buffer and a compressed stream have them too, which is why one copy function serves them all. Chapter 7 names this mechanism. The short-write loop and the retry on interruption are inside io.Copy and the runtime:

Exercise 03.2GoServe one client with io.Copy

Write serve. It takes a net.Conn, closes it when the function returns, copies the connection to itself, and returns the error from the copy. Use defer for the close.

func serve(conn net.Conn) error {
	// your code here
}
Stuck?Show the full listing
// serve copies everything the client sends back to it, until the client closes.
func serve(conn net.Conn) error {
	defer conn.Close()
	_, err := io.Copy(conn, conn)
	return err
}
go/ch03/main.go · region serve
โœ…defer the close on the line after the open Go

defer conn.Close() runs when the function returns, by whichever path. Write it on the line after the connection is created, so that no later return can forget it. This is the Go form of the C cleanup label, and it needs no label.

Two rules of defer#

defer has two rules that surprise everyone once. Deferred calls run in the reverse order of their defer lines, last in first out, which is the order a cleanup label releases things. And the arguments of a deferred call are evaluated at the defer line, not when the call runs. A closure that reads a variable sees the value at exit, while an argument carries the value at the defer. The test below is in the chapter's snapshot, and go test runs it:

// Two rules of defer, as a test you can run with go test: the deferred calls run last in
// first out, and each one's arguments are evaluated at the defer line, not at the exit.
func deferOrder(log *[]string) {
	n := 1
	defer func() { *log = append(*log, "third, closure sees n =", itoa(n)) }()
	defer func(v int) { *log = append(*log, "second, argument was", itoa(v)) }(n)
	n = 2
	*log = append(*log, "first, the body")
}

func TestDeferOrder(t *testing.T) {
	var log []string
	deferOrder(&log)
	want := []string{"first, the body", "second, argument was", "1", "third, closure sees n =", "2"}
	for i := range want {
		if i >= len(log) || log[i] != want[i] {
			t.Fatalf("log = %q, want %q", log, want)
		}
	}
}
go/ch03/defer_test.go · region defer

The main function listens and loops. net.Listen makes the four calls of the C version for you. It also sets SO_REUSEADDR, which you can see in the file net/sockopt_bsd.go of the Go source. An error from Accept is printed and the loop continues, because one bad connection must not stop the server:

func main() {
	ln, err := net.Listen("tcp", ":"+port())
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	fmt.Fprintln(os.Stderr, "kv listening on port", port())
	for {
		conn, err := ln.Accept()
		if err != nil {
			fmt.Fprintln(os.Stderr, "accept:", err)
			continue
		}
		if err := serve(conn); err != nil {
			fmt.Fprintln(os.Stderr, "client:", err)
		}
	}
}
go/ch03/main.go · region main
$ go run .
kv listening on port 6380
โš ๏ธA missing error check is legal Go Go

Go lets you drop an error on the floor. serve(conn) on a line of its own compiles, and the error is lost. The compiler checks that a value you assign is used, but not that a value you never assign is looked at. go vet does not catch it either. A separate linter called errcheck does. This book does not add it, and writes every check out instead. In this chapter the check is written out, as if err := serve(conn); err != nil.

Rust: a Result at every step#

In Rust the environment variable is a Result. std::env::var returns Ok(value) or an error when the variable is unset or not valid text. unwrap_or_else turns the error case into the default:

/// The port comes from the environment, so the tests can pick a free one.
fn port() -> String {
    std::env::var("KV_PORT").unwrap_or_else(|_| "6380".to_string())
}
rust/ch03/src/main.rs · region port

Serving a client uses io::copy, which does the same as io.Copy in Go, with the arguments in the other order. There is one new idea. A TcpStream is one value, and the ownership rules allow one mutable use of it at a time. The simplest way to get two handles on it is to clone the stream. try_clone asks the operating system for a second descriptor to the same connection, and that can fail, so it returns a Result too:

Exercise 03.3RustServe one client with io::copy

Write serve. Clone the stream with try_clone, copy from the clone to the original with io::copy, and return Ok(()). Use ? after each call that returns a Result, so that an error returns from the function at once.

fn serve(stream: TcpStream) -> io::Result<()> {
    // your code here
}
Stuck?Show the full listing
/// Copy everything the client sends back to it, until the client closes.
fn serve(stream: TcpStream) -> io::Result<()> {
    let mut reader = stream.try_clone()?;
    let mut writer = stream;
    io::copy(&mut reader, &mut writer)?;
    Ok(())
}
rust/ch03/src/main.rs · region serve

A main that returns a Result#

The main function returns io::Result<()>, so ? works inside it and a failure to bind prints the error and exits with a non-zero status. The loop uses incoming(), which yields one Result per connection attempt. A match handles both arms, and the compiler would refuse the program if one arm were missing:

fn main() -> io::Result<()> {
    let port = port();
    let listener = TcpListener::bind(format!("0.0.0.0:{port}"))?;
    eprintln!("kv listening on port {port}");
    for stream in listener.incoming() {
        match stream {
            Ok(stream) => {
                if let Err(e) = serve(stream) {
                    eprintln!("client: {e}");
                }
            }
            Err(e) => eprintln!("accept: {e}"),
        }
    }
    Ok(())
}
rust/ch03/src/main.rs · region main
$ cargo run --release
kv listening on port 6380

Nothing closes the stream, and nothing needs to. When serve returns, the TcpStream it owns goes out of scope, and Rust runs its destructor, which calls close. The clone is closed the same way. This is the third way to do what cleanup did in C and defer did in Go. It is the only one of the three that cannot be forgotten.

โš ๏ธA Result you ignore is a warning, not an error Rust

The compiler warns about an unused Result and then compiles the program. serve(stream); on a line of its own produces warning: unused Result that must be used. The book's build turns warnings into errors with cargo clippy -- -D warnings, so in this book the line does not compile. Without that flag it does, and the error is lost as in Go. The difference from Go is that Rust told you.

?, expect and unwrap#

Three ways to take a value out of a Result appear in every Rust program, and each one is a decision. ? passes the error to the caller, and is the default. expect("the reason this cannot fail") stops the program with that message when it does fail. It is for a failure that would mean a bug, and the message says why you believed that. unwrap() is expect with no reason, and belongs in examples and tests. kv uses ? almost everywhere. In port above, unwrap_or_else is a fourth way, which gives a default instead of failing.

Errors, three ways#

The three versions of serve differ most in how they say that something went wrong, and the difference runs through the whole book. In C a function returns -1, and the reason is a number in the global errno. The caller must know to check, and must check before any other call overwrites the number. In Go a function returns the error as a second value, and nil means success. The caller must still choose to look. In Rust a function returns Result, which is either Ok(value) or Err(reason), and the value is inside. To use the value at all, the caller must take it out. The ways to take it out are match, ?, and unwrap. The last one crashes on an error and is for examples only. Chapter 4 builds a parser in all three styles, where the difference is larger.

๐Ÿ–ฅSockets on Windows

Winsock has the same calls with the same names, with three differences that matter. A program must call WSAStartup before any of them. A socket is a handle of its own type, not an int, and it is closed with closesocket. And the option SO_REUSEADDR means something stronger on Windows, where it allows two programs to bind the same port. The C version of kv is written for POSIX and does not build on Windows without changes. The Go and Rust versions build there unchanged, because their standard libraries contain the differences.

Verdict##

TopicCGoRustWhy
Opening a portWeakStrongStrongC needs four calls, a struct and an option. Go and Rust need one call that sets the option for you.
Closing it againWeakOKStrongC needs a cleanup label and Go needs a defer you must write, but Rust closes at the end of scope by itself.
The echo loopWeakStrongStrongC needs the short-write loop and the EINTR retry. Both standard libraries contain them.
A lost errorWeakWeakOKC and Go let a return value drop. Rust warns, and with the book's flags refuses.
Seeing what happensStrongOKOKThe C version shows every system call. The other two hide them, which is what you want on most days.
WindowsWeakStrongStrongThe C code needs Winsock changes. Go and Rust build unchanged.
What you must learn firstStrongOKWeakC is the calls themselves. Rust asks for Result, ?, ownership and try_clone before an echo works.
Carry forward
  • errno. A failed C call returns -1 and leaves a number you must read at once.
  • defer. Runs at function exit, last in first out, with its arguments taken at the defer line.
  • Result. The value is inside, and ?, expect and match are the ways out.
Next

The server answers. Point redis-cli at it and ask PING. The answer is not PONG.

Speaking RESP#

Building kv · Chapter 04· 33 min read· 545 lines of code
▶Scene

Start the chapter 3 server and point the real client at it:

$ redis-cli -p 6380 PING
1) "PING"
$ redis-cli -p 6380 SET a 1
1) "SET"
2) "a"
3) "1"

The client asked for PONG and got its own question back, as a list. The server echoed the bytes the client sent, and the client read them as a reply. Whatever those bytes are, kv has to understand them before it can answer.

Redis clients talk in a small wire format called RESP, the Redis serialization protocol. This chapter teaches kv to read it and write it. After this chapter the real redis-cli can say PING to your server and get PONG back. The chapter also compares how the three languages say "something went wrong", because a parser is where that question is asked most often.

The echo server sends bytes back. redis-cli does not want its bytes back. It wants an answer in the format it expects. When you type ECHO hi into redis-cli, it sends these 22 bytes to the server:

*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n

Written with one element per line, the request reads as follows. An array of 2 bulk strings. The first has 4 bytes, ECHO. The second has 2 bytes, hi. The server replies with a bulk string of its own, $2\r\nhi\r\n. Every message in RESP is built from five types, each marked by its first byte:

+OK\r\n                   a simple string, for short replies such as OK and PONG
-ERR unknown command\r\n  an error, which the client shows as an error
:42\r\n                   an integer
$5\r\nhello\r\n           a bulk string: the length, then exactly that many bytes
*2\r\n...                 an array: the count, then that many messages
$-1\r\n                   the null bulk string, which GET returns for a missing key

The parts are separated by \r\n, a carriage return and a line feed, which this chapter calls a line ending. A bulk string carries its length in front, so the bytes inside can be anything, including line endings. That one rule is what makes the format both readable and fast to parse. There is no quoting and no escaping.

๐Ÿ“œWhy RESP looks like this

The Redis protocol specification names its three aims as "Simple to implement. Fast to parse. Human readable." The length in front of a bulk string means the parser never scans the payload for special characters, as a JSON parser must. The second version of the protocol, RESP2, became the standard in Redis 2.0. A third version, RESP3, adds types such as maps and a null of its own. A client chooses it with the HELLO command. kv speaks RESP2, which every client still speaks. The specification also allows an inline command. That is a plain line of words, so that a person with only telnet can talk to a server. Those work in kv too, which is why nc still works after this chapter.

Reading a requestCGoRust
Where the bytes waita buffer you grow with realloca bufio.Readera BufReader
Reading a linememchr for the line feedr.ReadBytes('\n')r.read_until(b'\n', &mut line)
Reading n bytescheck the length, then take a pointerio.ReadFull(r, buf)r.read_exact(&mut buf)
The argumentspointers into the buffer, no copy[][]byte, one copyVec<Vec<u8>>, one copy
Not enough bytes yetthe status KV_PARSE_INCOMPLETEthe reader blocks until they arrivethe reader blocks until they arrive
Bytes that are not RESPthe status KV_PARSE_BADthe error value errProtocolan io::Error of kind InvalidData
The client went awayread returns 0io.EOFOk(None)

The table shows the first large difference between C and the other two. Go and Rust read from a buffered reader that blocks until the bytes arrive, so the parser can ask for a line and wait. C has stdio, a buffered reader built for text files, and kv does not use it. The event loop of chapter 9 needs to see the bytes that have arrived so far. The C parser works on whatever bytes have arrived so far, and says "incomplete" when the request is not all there. That design is more work now. In chapter 9 it is the design the C event loop needs.

C: a buffer, a parser and three answers#

C needs a place to keep bytes between reads, because one read may return half a request, or two and a half. The structure below is a growable byte buffer. The comment on data says who owns it. The book writes such a comment on every pointer in a structure, because nothing else in C will tell you:

/* A growable byte buffer. It owns data, and kv_buf_free releases it. */
typedef struct {
    char *data; /* owned; NULL until the first append */
    size_t len;
    size_t cap;
} kv_buf;

static bool kv_buf_reserve(kv_buf *b, size_t extra) {
    if (b->len + extra <= b->cap) {
        return true;
    }
    size_t cap = b->cap ? b->cap * 2 : 1024;
    while (cap < b->len + extra) {
        cap *= 2;
    }
    char *p = realloc(b->data, cap);
    if (p == NULL) {
        return false; /* the old block is still valid and still ours */
    }
    b->data = p;
    b->cap = cap;
    return true;
}

static bool kv_buf_append(kv_buf *b, const void *src, size_t n) {
    if (!kv_buf_reserve(b, n)) {
        return false;
    }
    memcpy(b->data + b->len, src, n);
    b->len += n;
    return true;
}

/* Drop the first n bytes. memmove allows the two ranges to overlap. */
static void kv_buf_consume(kv_buf *b, size_t n) {
    memmove(b->data, b->data + n, b->len - n);
    b->len -= n;
}

static void kv_buf_free(kv_buf *b) {
    free(b->data);
    b->data = NULL;
    b->len = b->cap = 0;
}
c/ch04/kv.c · region buf

Look at kv_buf_reserve. When realloc fails it returns NULL and leaves the old block alone, so the code must not write the result into b->data before checking it. Writing b->data = realloc(b->data, cap) is a common mistake that loses the buffer on failure. kv_buf_consume uses memmove, not memcpy, because the source and the destination overlap. memcpy on overlapping memory is undefined behaviour.

A parsed command is a small structure of pointers into the buffer. No bytes are copied. The cost of that choice is a rule. The pointers are valid only until the buffer changes, and the comment says so. The parser returns one of three statuses, and the names of the statuses are the whole design:

/* One parsed request. The arguments point into the request buffer, so they are valid
 * only until that buffer is consumed. A kv_cmd owns nothing. */
typedef struct {
    size_t argc;
    const char *argv[KV_MAX_ARGS];
    size_t arglen[KV_MAX_ARGS];
} kv_cmd;

typedef enum {
    KV_PARSE_OK,         /* cmd is filled in, and *used bytes were taken */
    KV_PARSE_INCOMPLETE, /* read more bytes and try again */
    KV_PARSE_BAD         /* the bytes are not RESP: close the connection */
} kv_parse_status;
c/ch04/kv.c · region cmd

Two helpers find a line ending and read a number. The number reader refuses more than 18 digits, so that the multiplication cannot overflow a 64-bit long. Signed overflow is undefined behaviour in C, and a length field is exactly where an attacker would put a large number. A third helper splits an inline line into words:

/* The '\n' that ends the line starting at p, or NULL when it has not arrived yet. */
static const char *kv_find_eol(const char *p, const char *end) {
    return memchr(p, '\n', (size_t)(end - p));
}

/* A decimal number between p and the line end. False when the text is not a number. */
static bool kv_parse_number(const char *p, const char *eol, long *out) {
    const char *stop = eol;
    if (stop > p && stop[-1] == '\r') {
        stop--;
    }
    if (p == stop || stop - p > 18) {
        return false;
    }
    long v = 0;
    for (; p < stop; p++) {
        if (*p < '0' || *p > '9') {
            return false;
        }
        v = v * 10 + (*p - '0');
    }
    *out = v;
    return true;
}

/* An inline command: words separated by spaces, as typed into nc or telnet. */
static kv_parse_status kv_parse_inline(const char *data, const char *eol, kv_cmd *cmd,
                                       size_t *used) {
    const char *p = data;
    const char *stop = eol;
    if (stop > p && stop[-1] == '\r') {
        stop--;
    }
    while (p < stop) {
        while (p < stop && *p == ' ') {
            p++;
        }
        if (p == stop) {
            break;
        }
        const char *word = p;
        while (p < stop && *p != ' ') {
            p++;
        }
        if (cmd->argc == KV_MAX_ARGS) {
            return KV_PARSE_BAD;
        }
        cmd->argv[cmd->argc] = word;
        cmd->arglen[cmd->argc] = (size_t)(p - word);
        cmd->argc++;
    }
    *used = (size_t)(eol + 1 - data);
    return KV_PARSE_OK;
}
c/ch04/kv.c · region parse-helpers

kv.c sets three limits. KV_MAX_ARGS is 64 arguments and KV_MAX_BULK is 1 MB for one bulk string. KV_MAX_REQUEST is twice that. The Go and Rust files set the same two limits under the names maxBulk and MAX_BULK.

โš ๏ธA length field is where an attacker writes

A header that says $999999999 asks the server to allocate a gigabyte before one byte of payload has arrived. A parser that trusts the number does it, and a few such connections empty the machine. All three parsers in this chapter refuse a length over the limit before they read anything. The protocol test 04-limits.test sends that header to each of them. Redis has the same limit, proto-max-bulk-len, set to 512 MB.

Exercise 04.1CParse the bulk strings

Write the loop in kv_parse that reads the n bulk strings after the array header. For each one, check that the next byte is $ and find its line ending. Read the length. Then check that the whole string and its line ending have arrived. Return KV_PARSE_INCOMPLETE when bytes are missing and KV_PARSE_BAD when a byte is wrong. Store a pointer and a length in cmd, and set *used to the number of bytes the request took. Treat a length above KV_MAX_BULK as bad.

static kv_parse_status kv_parse(const char *data, size_t len, kv_cmd *cmd, size_t *used) {
    const char *end = data + len;
    const char *p = data;
    cmd->argc = 0;
    if (len == 0) return KV_PARSE_INCOMPLETE;
    const char *eol = kv_find_eol(p, end);
    if (eol == NULL) return KV_PARSE_INCOMPLETE;
    if (*p != '*') return kv_parse_inline(data, eol, cmd, used);
    long n;
    if (!kv_parse_number(p + 1, eol, &n) || n > KV_MAX_ARGS) return KV_PARSE_BAD;
    p = eol + 1;
    for (long i = 0; i < n; i++) {
        /* your code here */
    }
    cmd->argc = (size_t)n;
    *used = (size_t)(p - data);
    return KV_PARSE_OK;
}
Stuck?Show the full listing
/* Parse one request from data[0..len). The request is an array of bulk strings, such as
 * "*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n", or one inline line. */
static kv_parse_status kv_parse(const char *data, size_t len, kv_cmd *cmd, size_t *used) {
    const char *end = data + len;
    const char *p = data;
    cmd->argc = 0;
    if (len == 0) {
        return KV_PARSE_INCOMPLETE;
    }
    const char *eol = kv_find_eol(p, end);
    if (eol == NULL) {
        return KV_PARSE_INCOMPLETE;
    }
    /* The first byte says what the client sent. A client sends arrays. The other four
     * type bytes are replies, which only a server sends, so they are not a request. */
    switch (*p) {
    case '*':
        break;
    case '+':
    case '-':
    case ':':
    case '$':
        return KV_PARSE_BAD;
    default:
        return kv_parse_inline(data, eol, cmd, used);
    }
    long n;
    if (!kv_parse_number(p + 1, eol, &n) || n > KV_MAX_ARGS) {
        return KV_PARSE_BAD;
    }
    p = eol + 1;
    for (long i = 0; i < n; i++) {
        if (p >= end) {
            return KV_PARSE_INCOMPLETE;
        }
        if (*p != '$') {
            return KV_PARSE_BAD;
        }
        eol = kv_find_eol(p, end);
        if (eol == NULL) {
            return KV_PARSE_INCOMPLETE;
        }
        long blen;
        if (!kv_parse_number(p + 1, eol, &blen) || blen > KV_MAX_BULK) {
            return KV_PARSE_BAD;
        }
        p = eol + 1;
        if (end - p < blen + 2) {
            return KV_PARSE_INCOMPLETE;
        }
        if (p[blen] != '\r' || p[blen + 1] != '\n') {
            return KV_PARSE_BAD;
        }
        cmd->argv[i] = p;
        cmd->arglen[i] = (size_t)blen;
        p += blen + 2;
    }
    cmd->argc = (size_t)n;
    *used = (size_t)(p - data);
    return KV_PARSE_OK;
}
c/ch04/kv.c · region parse

switch, and the cases that run into each other#

Look at the top of the solution. The first byte of a request decides what it is, and the decision is a switch on that byte. A switch compares one integer against a list of constant cases and jumps to the one that matches. The four reply markers are stacked with no break between them. Any one of the four reaches the same return. That is fallthrough, and it is the feature of switch that bites. Forget a break after a case that has a body, and the next case runs too. The compiler says nothing, which is why every case in kv ends in a break, a return or a continue.

The serving loop has a second switch, on the parse status, and that one has no default on purpose. The status is an enum, a set of named integers. With -Wall, the compiler reports a switch on an enum that leaves a name out. Add a fourth status next year and every switch that forgot it fails to build. A default would hide that. The same trick does not work for a chain of if statements, which is why the loop changed shape in this chapter.

โš ๏ธPointers into a buffer that moves C

A pointer into a buffer dies when the buffer grows. realloc may move the block, and every pointer into the old block then points at freed memory. The kv_cmd holds such pointers. kv is safe because it runs the command and consumes the buffer before the next read appends to it. Move one line and it is not safe, and the compiler will not say a word. Chapter 12 shows AddressSanitizer catching exactly this.

Replies are appended to a second buffer. Each reply function returns false when memory runs out, and the callers pass that on:

static bool kv_reply_line(kv_buf *out, char first, const char *s) {
    return kv_buf_append(out, &first, 1) && kv_buf_append(out, s, strlen(s)) &&
           kv_buf_append(out, "\r\n", 2);
}

static bool kv_reply_simple(kv_buf *out, const char *s) { return kv_reply_line(out, '+', s); }
static bool kv_reply_error(kv_buf *out, const char *s) { return kv_reply_line(out, '-', s); }

static bool kv_reply_bulk(kv_buf *out, const char *s, size_t n) {
    char head[32];
    int h = snprintf(head, sizeof head, "$%zu\r\n", n);
    return kv_buf_append(out, head, (size_t)h) && kv_buf_append(out, s, n) &&
           kv_buf_append(out, "\r\n", 2);
}

static bool kv_reply_array(kv_buf *out, size_t n) {
    char head[32];
    int h = snprintf(head, sizeof head, "*%zu\r\n", n);
    return kv_buf_append(out, head, (size_t)h);
}
c/ch04/kv.c · region reply

A function that takes any number of arguments#

Two error messages need a value inside them, the command name in one and the unknown word in the other. printf solves this with a format and any number of arguments, and C lets you write such a function yourself. The ... in the parameter list is the variadic part. va_list from stdarg.h walks the arguments. va_start points it after the last named parameter. And vsnprintf is the snprintf that takes a va_list instead of the arguments themselves:

/* An error reply with a format, as printf would write it. The ... takes any arguments,
 * va_list walks them, and vsnprintf is the printf that accepts a va_list. */
static bool kv_reply_errorf(kv_buf *out, const char *fmt, ...) {
    char msg[128];
    va_list ap;
    va_start(ap, fmt);
    vsnprintf(msg, sizeof msg, fmt, ap);
    va_end(ap);
    return kv_reply_error(out, msg);
}
c/ch04/kv.c · region errorf

The compiler cannot check the arguments of a variadic function against the format, because the function's type says nothing about them. Both gcc and clang check printf itself by name. An attribute, __attribute__((format)), asks for the same check on yours. It is a compiler extension, not part of the standard, and kv does without it. Every format in kv is a literal in the call, so the mistake the check would catch cannot happen.

Running a command is a chain of comparisons on the first argument. Command names in Redis are case-insensitive, so the comparison uses strncasecmp from strings.h, with the length, because the argument is not a C string. It has no terminating zero byte. That is a theme of the next chapter:

/* True when argument i of the command is name, ignoring case. */
static bool kv_arg_is(const kv_cmd *c, size_t i, const char *name) {
    size_t n = strlen(name);
    return c->arglen[i] == n && strncasecmp(c->argv[i], name, n) == 0;
}

static bool kv_reply_arity(kv_buf *out, const char *name) {
    return kv_reply_errorf(out, "ERR wrong number of arguments for '%s' command", name);
}

/* Run one command and append its reply to out. */
static bool kv_exec(const kv_cmd *c, kv_buf *out) {
    if (c->argc == 0) {
        return kv_reply_error(out, "ERR empty command");
    }
    if (kv_arg_is(c, 0, "PING")) {
        if (c->argc == 1) {
            return kv_reply_simple(out, "PONG");
        }
        if (c->argc == 2) {
            return kv_reply_bulk(out, c->argv[1], c->arglen[1]);
        }
        return kv_reply_arity(out, "ping");
    }
    if (kv_arg_is(c, 0, "ECHO")) {
        if (c->argc != 2) {
            return kv_reply_arity(out, "echo");
        }
        return kv_reply_bulk(out, c->argv[1], c->arglen[1]);
    }
    if (kv_arg_is(c, 0, "COMMAND")) {
        return kv_reply_array(out, 0); /* redis-cli asks on connect; an empty list is fine */
    }
    int n = c->arglen[0] < 40 ? (int)c->arglen[0] : 40;
    return kv_reply_errorf(out, "ERR unknown command '%.*s'", n, c->argv[0]);
}
c/ch04/kv.c · region exec

The serving loop ties it together. After each read it runs every complete request in the buffer, because a client may send several at once, which Redis calls a pipeline. Then it writes all the replies in one call. A request that grows past KV_MAX_REQUEST without ending is treated as bad, so that a client cannot make the server grow its buffer without limit. The write loop from chapter 3 becomes a function first:

/* write() may send fewer bytes than asked, so loop until all are out. */
static bool kv_write_all(int fd, const char *p, size_t n) {
    size_t off = 0;
    while (off < n) {
        ssize_t w = write(fd, p + off, n - off);
        if (w < 0) {
            if (errno == EINTR) {
                continue;
            }
            perror("write");
            return false;
        }
        off += (size_t)w;
    }
    return true;
}
c/ch04/kv.c · region write-all
/* Read requests, run them, send the replies, until the client closes. */
static void kv_serve(int client) {
    kv_buf in = {0};
    kv_buf out = {0};
    char tmp[4096];
    for (;;) {
        ssize_t n = read(client, tmp, sizeof tmp);
        if (n < 0) {
            if (errno == EINTR) {
                continue;
            }
            perror("read");
            break;
        }
        if (n == 0) {
            break; /* the client closed the connection */
        }
        if (!kv_buf_append(&in, tmp, (size_t)n)) {
            break;
        }
        /* A pipeline holds several requests in one read, so loop until one is incomplete. */
        bool bad = false;
        for (;;) {
            kv_cmd cmd;
            size_t used = 0;
            kv_parse_status st = kv_parse(in.data, in.len, &cmd, &used);
            /* No default: with -Wall the compiler reports a status this switch forgot. */
            switch (st) {
            case KV_PARSE_INCOMPLETE:
                bad = in.len > KV_MAX_REQUEST; /* a request that never ends is not a request */
                break;
            case KV_PARSE_BAD:
                bad = true;
                break;
            case KV_PARSE_OK:
                if (!kv_exec(&cmd, &out)) {
                    bad = true;
                }
                kv_buf_consume(&in, used);
                continue;
            }
            break;
        }
        if (bad) {
            kv_reply_error(&out, "ERR Protocol error");
        }
        if (out.len > 0 && !kv_write_all(client, out.data, out.len)) {
            break;
        }
        out.len = 0;
        if (bad) {
            break;
        }
    }
    kv_buf_free(&in);
    kv_buf_free(&out);
}
c/ch04/kv.c · region serve

Build it, run it, and talk to it with the real client:

$ make && ./kv &
$ redis-cli -p 6380 PING
PONG
$ redis-cli -p 6380 ECHO "hello world"
"hello world"
$ redis-cli -p 6380 FOO
(error) ERR unknown command 'FOO'
โœ…Say who owns every pointer C

A pointer in C carries no information about who frees it or how long it stays valid. So the comment must. kv_buf says "owned". kv_cmd says "valid only until the buffer is consumed". A function that returns a pointer says whether the caller must free it. Redis does this, the Linux kernel does this, and so does every C codebase that has survived long enough to be read by strangers.

Go: a reader that waits, and an error that names itself#

Go has bufio.Reader, which reads ahead into its own buffer and gives you lines or exact counts of bytes. The parser becomes a set of calls that block until the bytes are there. The "incomplete" status disappears, because the reader waits instead. The first helper reads a line and removes its ending:

// readLine returns one line without its line ending. A bare LF is accepted as well as
// CRLF, so that a command typed into nc works. io.EOF means the client closed the
// connection cleanly between commands.
func readLine(r *bufio.Reader) ([]byte, error) {
	line, err := r.ReadBytes('\n')
	if err != nil {
		if err == io.EOF && len(line) > 0 {
			return nil, io.ErrUnexpectedEOF
		}
		return nil, err
	}
	line = bytes.TrimSuffix(line, []byte("\n"))
	return bytes.TrimSuffix(line, []byte("\r")), nil
}
go/ch04/resp.go · region read-line

Two errors from the standard library carry meaning here. io.EOF means the connection closed, and readLine returns it unchanged when it happens between requests, which is the normal way for a client to leave. io.ErrUnexpectedEOF means the connection closed in the middle of something, which is not normal. A third error, errProtocol, is this file's own. It is a package-level variable, and a caller tests for it with errors.Is. Such a value is called a sentinel error.

An error is any value with an Error method#

There is nothing special about an error in Go. error is an interface with one method, Error() string, and any type with that method is an error. A sentinel is the smallest one, a value with a fixed message. Sometimes a caller needs more than the fact of a failure, such as which byte was wrong. Then the error becomes a type of your own with fields:

// errProtocol means the bytes were not RESP. The connection is closed after it. A caller
// that only needs to know that much tests errors.Is(err, errProtocol).
var errProtocol = errors.New("protocol error")

// A ProtocolError says which byte was wrong. Unwrap returns the sentinel, so errors.Is still
// finds errProtocol in it, and errors.As gives a caller the byte.
type ProtocolError struct {
	Byte byte
}

func (e *ProtocolError) Error() string { return fmt.Sprintf("protocol error at byte %q", e.Byte) }
func (e *ProtocolError) Unwrap() error { return errProtocol }

func protoErr(b byte) error { return &ProtocolError{Byte: b} }
go/ch04/resp.go · region errors

Three things in that block are the whole of Go error handling. Error makes *ProtocolError an error. Unwrap says that a ProtocolError contains the sentinel, so errors.Is(err, errProtocol) is still true for one. And errors.Is and errors.As walk that chain of Unwrap calls. Is asks whether a given value is in the chain. As asks whether a value of a given type is, and gives it to you. In the reader, every error from the standard library is wrapped with what kv was doing. The wrapping is fmt.Errorf with the %w verb, which adds a message and keeps the cause in the chain.

Exercise 04.2GoRead the bulk strings

Write the loop in readCommand that reads n bulk strings. For each one, read a line and check that it starts with $. Convert the rest to a size with strconv.Atoi. Then read exactly size+2 bytes with io.ReadFull. Check that the last two bytes are the line ending. Return errProtocol for a wrong byte, and turn an io.EOF inside a request into io.ErrUnexpectedEOF. Treat a size above maxBulk as bad. for range n runs the body n times, and is new in Go 1.22.

func readCommand(r *bufio.Reader) ([][]byte, error) {
	line, err := readLine(r)
	if err != nil {
		return nil, err
	}
	if len(line) == 0 || line[0] != '*' {
		return bytes.Fields(line), nil
	}
	n, err := strconv.Atoi(string(line[1:]))
	if err != nil || n < 0 || n > maxArgs {
		return nil, errProtocol
	}
	args := make([][]byte, 0, n)
	for range n {
		// your code here
	}
	return args, nil
}
Stuck?Show the full listing
// readCommand reads one request: an array of bulk strings such as
// "*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n", or one inline line of words. Every error from the
// reader is wrapped with what kv was doing, and the cause stays reachable with errors.Is.
func readCommand(r *bufio.Reader) ([][]byte, error) {
	line, err := readLine(r)
	if err != nil {
		return nil, fmt.Errorf("read command: %w", err)
	}
	if len(line) == 0 || line[0] != '*' {
		return bytes.Fields(line), nil
	}
	n, err := strconv.Atoi(string(line[1:]))
	if err != nil || n < 0 || n > maxArgs {
		return nil, protoErr('*')
	}
	args := make([][]byte, 0, n)
	for range n {
		line, err := readLine(r)
		if err != nil {
			return nil, fmt.Errorf("read argument: %w", unexpected(err))
		}
		if len(line) == 0 || line[0] != '$' {
			return nil, protoErr(first(line))
		}
		size, err := strconv.Atoi(string(line[1:]))
		if err != nil || size < 0 || size > maxBulk {
			return nil, protoErr('$')
		}
		buf := make([]byte, size+2)
		if _, err := io.ReadFull(r, buf); err != nil {
			return nil, fmt.Errorf("read argument: %w", unexpected(err))
		}
		if buf[size] != '\r' || buf[size+1] != '\n' {
			return nil, protoErr(buf[size])
		}
		args = append(args, buf[:size])
	}
	return args, nil
}

// unexpected turns a clean EOF in the middle of a request into an error that says so.
func unexpected(err error) error {
	if err == io.EOF {
		return io.ErrUnexpectedEOF
	}
	return err
}

func first(line []byte) byte {
	if len(line) == 0 {
		return '\n'
	}
	return line[0]
}
go/ch04/resp.go · region read-command

Writing replies uses a bufio.Writer, which collects small writes and sends them in one system call when it is flushed:

func writeSimple(w *bufio.Writer, s string) error {
	_, err := w.WriteString("+" + s + "\r\n")
	return err
}

func writeError(w *bufio.Writer, s string) error {
	_, err := w.WriteString("-" + s + "\r\n")
	return err
}

func writeBulk(w *bufio.Writer, b []byte) error {
	if _, err := w.WriteString("$" + strconv.Itoa(len(b)) + "\r\n"); err != nil {
		return err
	}
	if _, err := w.Write(b); err != nil {
		return err
	}
	_, err := w.WriteString("\r\n")
	return err
}

func writeArrayLen(w *bufio.Writer, n int) error {
	_, err := w.WriteString("*" + strconv.Itoa(n) + "\r\n")
	return err
}
go/ch04/resp.go · region write

The command switch is the C chain of comparisons, written as a switch on an upper-cased string. Go strings are immutable and can hold any bytes, so converting an argument to a string costs one copy and nothing else:

// execute runs one command and writes its reply.
func execute(args [][]byte, w *bufio.Writer) error {
	if len(args) == 0 {
		return writeError(w, "ERR empty command")
	}
	name := strings.ToUpper(string(args[0]))
	switch name {
	case "PING":
		switch len(args) {
		case 1:
			return writeSimple(w, "PONG")
		case 2:
			return writeBulk(w, args[1])
		}
		return writeArity(w, "ping")
	case "ECHO":
		if len(args) != 2 {
			return writeArity(w, "echo")
		}
		return writeBulk(w, args[1])
	case "COMMAND":
		return writeArrayLen(w, 0) // redis-cli asks on connect; an empty list is fine
	}
	shown := string(args[0])
	if len(shown) > 40 {
		shown = shown[:40]
	}
	return writeError(w, "ERR unknown command '"+shown+"'")
}

func writeArity(w *bufio.Writer, name string) error {
	return writeError(w, "ERR wrong number of arguments for '"+name+"' command")
}
go/ch04/exec.go · region exec

The serving loop reads a request, runs it, and flushes the writer when the reader has no bytes waiting. That last check is the Go form of the C pipeline loop. While a pipeline is still being read, replies collect in the writer, and go out together:

// serve reads requests, runs them and sends the replies, until the client closes.
func serve(conn net.Conn) error {
	defer conn.Close()
	r := bufio.NewReader(conn)
	w := bufio.NewWriter(conn)
	for {
		args, err := readCommand(r)
		// The error is wrapped, so == would miss io.EOF. errors.Is walks the chain.
		if errors.Is(err, io.EOF) {
			return nil
		}
		// errors.AsType finds the typed error in the chain and gives us its byte. Before
		// Go 1.26 the same check was errors.As(err, &pe) with a declared *ProtocolError.
		if pe, ok := errors.AsType[*ProtocolError](err); ok {
			writeError(w, "ERR Protocol error")
			w.Flush()
			return fmt.Errorf("closing client: %w", pe)
		}
		if err != nil {
			return err
		}
		if err := execute(args, w); err != nil {
			return err
		}
		// A pipeline holds several requests; flush once the reader has nothing waiting.
		if r.Buffered() == 0 {
			if err := w.Flush(); err != nil {
				return err
			}
		}
	}
}
go/ch04/main.go · region serve

The function returns nil on a clean close, and the error otherwise. On a protocol error it first tells the client why. The three tests on err are the Go way of handling three different failures, and the order matters. The specific checks come before the general err != nil. Two of them changed shape in this chapter. err == io.EOF became errors.Is(err, io.EOF), because the reader now wraps its errors, and a wrapped io.EOF is not equal to the bare one. And errors.AsType[*ProtocolError](err) pulls the typed error out of the chain so the log line can name the byte. AsType is new in Go 1.26. Before it the same check was errors.As(err, &pe) with a declared pe, and you will read that form in most code.

โš ๏ธA wrapped error is not == its cause Go

The moment any function in the chain wraps with %w, every err == io.EOF above it becomes false. The comparison sees the outer value. errors.Is sees the chain. In this book every comparison with a sentinel is errors.Is, including the ones that would work today with ==. The function below may start wrapping tomorrow. The replay in chapter 11 is where that caught the author.

โœ…A sentinel for a condition, a type for its details Go

A caller that only needs to know that something went wrong tests err != nil. A caller that must tell one failure from another compares with errors.Is, so define a sentinel for that failure. A caller that needs the details, a byte, a path, a count, gets a type with fields and finds it with errors.As. Give the type an Unwrap that returns the sentinel, and both callers are served by one value. errors.Join combines several errors into one that Is and As search through, for the case where a loop collects failures.

Rust: an Option for the end, a Result for the rest#

The Rust reader is the BufRead trait, and read_until does the same as ReadBytes in Go. The line function returns io::Result<Option<Vec<u8>>>, which is a mouthful and exactly right. The outer Result is "did the read fail". The inner Option is "was there a line at all". A clean close is Ok(None), and nobody has to remember that a particular error value means "not really an error":

/// One line without its line ending. A bare LF is accepted as well as CRLF, so that a
/// command typed into nc works. `Ok(None)` means the client closed the connection
/// cleanly between commands.
fn read_line(r: &mut impl BufRead) -> Result<Option<Vec<u8>>> {
    let mut line = Vec::new();
    if r.read_until(b'\n', &mut line)? == 0 {
        return Ok(None);
    }
    if line.pop() != Some(b'\n') {
        return Err(io::Error::from(io::ErrorKind::UnexpectedEof).into());
    }
    if line.last() == Some(&b'\r') {
        line.pop();
    }
    Ok(Some(line))
}

fn parse_len(digits: &[u8], type_byte: u8) -> Result<usize> {
    std::str::from_utf8(digits)
        .ok()
        .and_then(|s| s.parse().ok())
        .ok_or(KvError::Protocol(type_byte))
}
rust/ch04/src/resp.rs · region read-line

The ? after read_until returns any read error at once, and it does one more thing, which the next section explains. parse_len turns the digits into a number in two steps. Each step returns an Option, and and_then chains them. The final ok_or turns a missing number into the protocol error, with the type byte that was being read.

One enum for everything that can go wrong#

Two kinds of failure can come out of the reader, a failed read and bytes that are not RESP, and they need different answers. In Rust the two become one enum, and each variant carries what its caller needs. Io holds the io::Error itself. Protocol holds the byte that was wrong. A match on the enum must handle both, and the compiler refuses the program if a third variant is added and one match forgets it:

/// Everything that can go wrong while reading a request. Each variant carries what the
/// caller needs to know: the io error itself, or the byte that was not RESP.
#[derive(Debug)]
pub enum KvError {
    Io(io::Error),
    Protocol(u8),
}

/// How the error prints, with `{}`. `to_string` and `eprintln!` use this.
impl fmt::Display for KvError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            KvError::Io(e) => write!(f, "io error: {e}"),
            KvError::Protocol(b) => write!(f, "protocol error at byte {:?}", *b as char),
        }
    }
}

/// Joining the standard error trait lets a `KvError` sit in a `Box<dyn Error>`.
impl std::error::Error for KvError {}

/// `?` calls this when an `io::Error` must become a `KvError`. Without it, every read in
/// this file would need `.map_err(KvError::Io)`.
impl From<io::Error> for KvError {
    fn from(e: io::Error) -> Self {
        KvError::Io(e)
    }
}

/// The other direction, so a function that returns `io::Result` can use `?` on ours.
impl From<KvError> for io::Error {
    fn from(e: KvError) -> Self {
        match e {
            KvError::Io(e) => e,
            KvError::Protocol(_) => io::Error::new(io::ErrorKind::InvalidData, e.to_string()),
        }
    }
}

pub type Result<T> = std::result::Result<T, KvError>;
rust/ch04/src/resp.rs · region error

The four impl blocks are the standard shape of an error type, and every Rust error you meet has them. Display says how the error prints with {}. The empty Error block declares it an error, so it can live in a Box<dyn Error> with every other error. And From is the one that matters most. When ? meets an io::Error inside a function that returns our enum, it calls From::from to convert it. That is why no line in the reader says map_err. The second From goes the other way, so a function that still returns io::Result can use ? on ours. The last line gives the file its own Result, with the error type filled in, so the signatures below stay short.

Exercise 04.3RustRead the bulk strings

Write the loop in read_command that reads n bulk strings. Read a line with read_line, and treat None as an unexpected end of file. Check the first byte, parse the size, read exactly size + 2 bytes with read_exact, check the last two, then truncate and push. The let ... else at the top shows the shape for a missing line. Treat a size above MAX_BULK as bad.

pub fn read_command(r: &mut impl BufRead) -> io::Result<Option<Command>> {
    let Some(line) = read_line(r)? else {
        return Ok(None);
    };
    if line.first() != Some(&b'*') {
        let words = line.split(|b| *b == b' ').filter(|w| !w.is_empty());
        return Ok(Some(words.map(<[u8]>::to_vec).collect()));
    }
    let n = parse_len(&line[1..])?;
    if n > MAX_ARGS {
        return Err(protocol_error());
    }
    let mut args = Vec::with_capacity(n);
    for _ in 0..n {
        // your code here
    }
    Ok(Some(args))
}
Stuck?Show the full listing
/// One request: an array of bulk strings such as `*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n`,
/// or one inline line of words. `Ok(None)` means the client has gone.
pub fn read_command(r: &mut impl BufRead) -> Result<Option<Command>> {
    let Some(line) = read_line(r)? else {
        return Ok(None);
    };
    if line.first() != Some(&b'*') {
        let words = line.split(|b| *b == b' ').filter(|w| !w.is_empty());
        return Ok(Some(words.map(<[u8]>::to_vec).collect()));
    }
    let n = parse_len(&line[1..], b'*')?;
    if n > MAX_ARGS {
        return Err(KvError::Protocol(b'*'));
    }
    let mut args = Vec::with_capacity(n);
    for _ in 0..n {
        let line = read_line(r)?.ok_or(io::Error::from(io::ErrorKind::UnexpectedEof))?;
        if line.first() != Some(&b'$') {
            return Err(KvError::Protocol(line.first().copied().unwrap_or(b'\n')));
        }
        let size = parse_len(&line[1..], b'$')?;
        if size > MAX_BULK {
            return Err(KvError::Protocol(b'$'));
        }
        let mut buf = vec![0u8; size + 2];
        r.read_exact(&mut buf)?;
        if &buf[size..] != b"\r\n" {
            return Err(KvError::Protocol(buf[size]));
        }
        buf.truncate(size);
        args.push(buf);
    }
    Ok(Some(args))
}
rust/ch04/src/resp.rs · region read-command

The reply functions take impl Write, which means any writer, and the write! macro formats straight into it:

pub fn write_simple(w: &mut impl Write, s: &str) -> io::Result<()> {
    write!(w, "+{s}\r\n")
}

pub fn write_error(w: &mut impl Write, s: &str) -> io::Result<()> {
    write!(w, "-{s}\r\n")
}

pub fn write_bulk(w: &mut impl Write, b: &[u8]) -> io::Result<()> {
    write!(w, "${}\r\n", b.len())?;
    w.write_all(b)?;
    w.write_all(b"\r\n")
}

pub fn write_array_len(w: &mut impl Write, n: usize) -> io::Result<()> {
    write!(w, "*{n}\r\n")
}
rust/ch04/src/resp.rs · region write

Running a command is a match on the upper-cased name as bytes. The let ... else handles an empty request. A bulk string is Vec<u8> and not String, because a Rust String must be valid UTF-8 and a Redis value need not be. When the name must be shown in an error message, from_utf8_lossy replaces any invalid bytes with a marker:

/// Run one command and write its reply.
pub fn execute(args: &[Vec<u8>], w: &mut impl Write) -> io::Result<()> {
    let Some(name) = args.first() else {
        return resp::write_error(w, "ERR empty command");
    };
    match name.to_ascii_uppercase().as_slice() {
        b"PING" => match args.len() {
            1 => resp::write_simple(w, "PONG"),
            2 => resp::write_bulk(w, &args[1]),
            _ => write_arity(w, "ping"),
        },
        b"ECHO" => match args.len() {
            2 => resp::write_bulk(w, &args[1]),
            _ => write_arity(w, "echo"),
        },
        // redis-cli asks on connect; an empty list is fine
        b"COMMAND" => resp::write_array_len(w, 0),
        _ => {
            let shown = String::from_utf8_lossy(&name[..name.len().min(40)]);
            resp::write_error(w, &format!("ERR unknown command '{shown}'"))
        }
    }
}

fn write_arity(w: &mut impl Write, name: &str) -> io::Result<()> {
    resp::write_error(
        w,
        &format!("ERR wrong number of arguments for '{name}' command"),
    )
}
rust/ch04/src/cmd.rs · region exec

The serving loop matches on the three outcomes of read_command. The arm Err(e @ KvError::Protocol(_)) matches one variant and, with @, also binds the whole error to e so it can be returned. That is the Rust form of errors.As, checked by the compiler rather than searched at run time. The compiler also checks that the four arms cover every case:

/// Read requests, run them and send the replies, until the client closes. The function
/// returns a `KvError`, so `?` converts every `io::Error` on the way with `From`.
fn serve(stream: TcpStream) -> resp::Result<()> {
    let mut reader = BufReader::new(stream.try_clone()?);
    let mut writer = BufWriter::new(stream);
    loop {
        match resp::read_command(&mut reader) {
            Ok(Some(args)) => cmd::execute(&args, &mut writer)?,
            Ok(None) => return Ok(()),
            Err(e @ KvError::Protocol(_)) => {
                resp::write_error(&mut writer, "ERR Protocol error")?;
                writer.flush()?;
                return Err(e);
            }
            Err(e) => return Err(e),
        }
        // A pipeline holds several requests; flush once the reader has nothing waiting.
        if reader.buffer().is_empty() {
            writer.flush()?;
        }
    }
}
rust/ch04/src/main.rs · region serve
โš ๏ธA String is not a byte string Rust

String::from_utf8(bytes) fails on a value that is not text. Redis keys and values are bytes, and a client may store an image under a key. A parser that converts every argument to String rejects such a client with an error that looks like a bug in the client. Keep arguments as Vec<u8> and convert only for display, with from_utf8_lossy, or for commands that need text, such as a number.

Three ways to say what went wrong#

Each version of the parser had to express four different outcomes. The request is complete. The bytes are not RESP. The client closed the connection cleanly. The client closed it in the middle of a request. The table shows the three vocabularies.

The outcomeCGoRust
A value, or a failurea return code and an out-parametertwo return values, (v, err)one value, Result<T, E>
The reason for a failurethe global errno, or an enum of your ownan error value, found with errors.Is or errors.Asthe Err payload, an enum matched on its variant
Passing a failure upif (x < 0) return -1; on every callif err != nil { return err } on every callone character, ?
Ignoring a failurewrite nothing, and nothing warnswrite nothing, and nothing warnsthe compiler warns about an unused Result
"Not an error, but nothing"a special return value, such as 0 from reada special error value, io.EOFOption, which is not an error at all
Adding a reasona format string into a fixed arrayfmt.Errorf("...: %w", err)a variant of your enum, and From so that ? converts

C has one mechanism, the return value, and uses it for everything. The design works, and the whole of Unix is built on it, but it has two costs that the book will keep meeting. The first is that the check is optional and silent when missing. The second is that the value and the error share one channel. A function that returns a pointer uses NULL for failure. A function that returns a count uses -1. The caller must know which.

Go separates the value from the error, which fixes the second cost, and leaves the first in place by choice. The Go FAQ says the designers found try-catch-finally produced "convoluted code". The price is the three lines of if err != nil after every call, and a Go file of any size has many such checks. Rust puts the value inside the Result, so that the caller cannot reach the value without facing the failure. The ? operator makes the common case one character. The price is a type system that the programmer must learn before any of it makes sense. There is also a long error message from the compiler when a ? is used where the error types do not match.

Verdict##

TopicCGoRustWhy
Reading bytesWeakStrongStrongC has stdio, which blocks, so kv grows its own buffer. Both standard libraries have one.
Holding the argumentsStrongOKOKThe C parser copies nothing. Go and Rust copy each argument once, which is simpler and almost never matters.
Saying "not RESP"OKOKStrongC and Go use a value by convention. Rust uses a kind the compiler knows about and a match the compiler checks.
Saying "the client left"OKWeakStrongGo reuses an error value for a non-error. Rust has Option for exactly this.
Passing an error upWeakWeakStrongC and Go repeat the check at every call. Rust writes one character.
Binary valuesStrongStrongOKC and Go treat all bytes alike. Rust makes you choose between String and Vec<u8>, and the wrong choice rejects real data.
An error with detailsWeakOKStrongC has an enum and a fixed buffer, and Go has a type found by searching a chain. Rust has an enum variant the compiler matches.
Carry forward
  • switch. One integer against constant cases. A case with no break continues into the next, and an enum switch with no default is checked by the compiler.
  • Error chain. A Go error wraps its cause with %w. The functions errors.Is and errors.As search the chain, and == cannot.
  • From and ?. A Rust error enum with a From impl lets ? convert every lower error into yours.
Next

The server speaks the protocol and knows two commands. Ask it to store a value and it says so, in the error the protocol reserves for that. Chapter 5 gives it a store.

GET and SET#

Building kv · Chapter 05· 38 min read· 693 lines of code
▶Scene

Start the chapter 4 server and ask it to remember something:

$ redis-cli -p 6380 SET name kv
(error) ERR unknown command 'SET'
$ redis-cli -p 6380 GET name
(error) ERR unknown command 'GET'

It speaks the protocol and has nothing to say. Every command it will ever learn is a name, a number of arguments and a function. This chapter gives it a place to keep that list, and the first five entries.

kv starts to remember. SET stores a value under a key and GET returns it. Two commands take several keys, DEL and EXISTS. INCR treats a value as a number. The chapter is about strings and bytes, because a key and a value are bytes, and the three languages disagree about what a string is.

A Redis value is a sequence of bytes. It may be the text hello, the digits 42, or the first kilobyte of a JPEG. A Redis key is the same. The server must keep a copy of both and find a key again later. It must compare two keys for equality. For INCR it must read a value as an integer and write one back. Each of those steps asks the question "what is a string?", and each language answers it differently.

A stringCGoRust
What it isa pointer to bytes that end with a zero bytean immutable sequence of bytesa sequence of bytes that must be valid UTF-8
Bytes with a lengtha pointer and a size_t, by convention[]byteVec<u8>, or a borrowed &[u8]
Lengthstrlen walks to the zero, or you carry the lengthlen(s), stored with the valuev.len(), stored with the value
Equalitymemcmp with the lengths== on strings, bytes.Equal on slices== on both
As a map keya hash you write, in chapter 6string, because a slice cannot be a keyVec<u8>, because it is hashable
To an integeryou parse the digits, watching for overflowstrconv.ParseIntstr::parse::<i64>() after a UTF-8 check
Who owns the byteswhoever the comment saysthe garbage collectorexactly one binding, checked by the compiler
A command, as dataa function pointer in a table, built by a macroa map from the name to a funca fn pointer in a const table, built by a macro
๐Ÿ“œWhy a C string ends with a zero byte

In BCPL, the language before B, a string began with its length in one byte. B dropped the count and ended a string with a marker character instead, and C kept that rule with the zero byte as the marker. Ritchie gave two reasons. The one-byte count limited a string to 255 characters, and "maintaining the count seemed, in our experience, less convenient than using a terminator". Fifty years later, that marker is why a C string cannot hold a zero byte. It is also why every C string function walks the whole string to find its end. Go and Rust store the length with the bytes, as BCPL did.

C: bytes, a length and an owner#

The arguments that the parser produces are not C strings. They are pointers into the request buffer with a length beside them, and no zero byte ends them. Every function in kv that touches a key or a value takes a pointer and a length. The C string library is never called on them. The first helpers copy, compare and parse such bytes:

/* A copy of n bytes that the caller owns and must free. Never NULL on success, even
 * for n == 0, because malloc(0) may return NULL and that would look like failure. */
static char *kv_bytes_dup(const char *src, size_t n) {
    char *p = malloc(n ? n : 1);
    if (p == NULL) {
        return NULL;
    }
    memcpy(p, src, n);
    return p;
}

static bool kv_bytes_eq(const char *a, size_t alen, const char *b, size_t blen) {
    return alen == blen && memcmp(a, b, alen) == 0;
}

/* Parse a signed decimal integer from bytes that have no terminating zero. Strict, as
 * Redis is: no spaces, no plus sign, no leading zeros, and no overflow. */
static bool kv_parse_int(const char *s, size_t n, long long *out) {
    size_t i = 0;
    bool neg = false;
    if (n == 0) {
        return false;
    }
    if (s[0] == '-') {
        neg = true;
        i = 1;
        if (n == 1) {
            return false;
        }
    }
    if (s[i] == '0' && n - i > 1) {
        return false;
    }
    unsigned long long v = 0;
    for (; i < n; i++) {
        if (s[i] < '0' || s[i] > '9') {
            return false;
        }
        unsigned d = (unsigned)(s[i] - '0');
        if (v > (ULLONG_MAX - d) / 10) {
            return false;
        }
        v = v * 10 + d;
    }
    if (!neg && v > (unsigned long long)LLONG_MAX) {
        return false;
    }
    if (neg && v > (unsigned long long)LLONG_MAX + 1) {
        return false;
    }
    *out = neg ? -(long long)(v - 1) - 1 : (long long)v;
    return true;
}
c/ch05/kv.c · region bytes

Three details deserve a look. kv_bytes_dup asks for at least one byte. The reason is that malloc(0) is allowed to return NULL, which would look like an out-of-memory failure. The integer parser works in unsigned arithmetic and checks before every multiplication, because overflow of a signed integer is undefined behaviour in C. The last line turns the unsigned result into a negative number. It avoids computing -v when v is one more than the largest positive value. That value is exactly the smallest negative one, and the obvious expression would overflow on the way there.

โš ๏ธstrlen on bytes that are not a C string C

Calling a string function on a key reads past the key. The key hello in a request is followed by \r\n and the rest of the buffer, not by a zero byte. strlen walks on until it finds a zero somewhere, and strcmp compares bytes that belong to another argument. Most of the time there is no crash. Both are undefined behaviour, and AddressSanitizer, which chapter 12 introduces, reports such a read. The rule in kv is simple. A key or a value is always a pointer and a length, and string.h is for messages the program writes itself.

๐Ÿ“ฐHeartbleed, 2014

In April 2014 anyone on the internet could read 64 kilobytes of a web server's memory by sending it one message. The TLS heartbeat extension of OpenSSL, a C library, let a client send a payload and its length, and the server sent the payload back. The code allocated the reply "based on the length field in the requesting message, without regard to the actual size of that message's payload". A length of 65,535 and a payload of one byte returned that byte and whatever lay after it. Passwords, keys, other people's sessions. The bug is CVE-2014-0160 in the Common Vulnerabilities and Exposures list, and it had been in the library since the end of 2011. About 17 percent of the secure web servers on the internet were believed to be open to it on the day it was disclosed. It is the trusted length of chapter 4 and the pointer without a length of this chapter, in one line of C.

The store in this chapter is a list of entries. A lookup walks the list and compares each key, which is slow once there are thousands of keys. Chapter 6 replaces the list with a hash table. The entry owns copies of its key and value, and the comments say so:

/* One key and its value. Both are owned copies, freed with the entry. */
typedef struct {
    char *key;
    size_t klen;
    char *val;
    size_t vlen;
} kv_entry;

/* Every entry, in a list. A lookup walks the whole list, which chapter 6 fixes. */
typedef struct {
    kv_entry *items; /* owned */
    size_t len;
    size_t cap;
} kv_store;

static kv_entry *kv_store_find(kv_store *s, const char *key, size_t klen) {
    for (size_t i = 0; i < s->len; i++) {
        if (kv_bytes_eq(s->items[i].key, s->items[i].klen, key, klen)) {
            return &s->items[i];
        }
    }
    return NULL;
}
c/ch05/kv.c · region store
Exercise 05.1CStore a copy

Write kv_store_set. Copy the value first, because that can fail. If the key exists, free its old value and keep the new one. If not, grow the list when it is full, copy the key, and append an entry. On any failure, free what you copied and return false, leaving the store as it was.

static bool kv_store_set(kv_store *s, const char *key, size_t klen, const char *val,
                         size_t vlen) {
    char *v = kv_bytes_dup(val, vlen);
    if (v == NULL) return false;
    /* your code here */
}
Stuck?Show the full listing
/* Store a copy of val under a copy of key, replacing any old value. */
static bool kv_store_set(kv_store *s, const char *key, size_t klen, const char *val,
                         size_t vlen) {
    char *v = kv_bytes_dup(val, vlen);
    if (v == NULL) {
        return false;
    }
    kv_entry *e = kv_store_find(s, key, klen);
    if (e != NULL) {
        free(e->val); /* the old value, which nothing else points to */
        e->val = v;
        e->vlen = vlen;
        return true;
    }
    if (s->len == s->cap) {
        size_t cap = s->cap ? s->cap * 2 : 16;
        kv_entry *items = realloc(s->items, cap * sizeof *items);
        if (items == NULL) {
            free(v);
            return false;
        }
        s->items = items;
        s->cap = cap;
    }
    char *k = kv_bytes_dup(key, klen);
    if (k == NULL) {
        free(v);
        return false;
    }
    s->items[s->len++] = (kv_entry){k, klen, v, vlen};
    return true;
}

/* Remove a key. The last entry moves into the hole, so the order is not kept. */
static bool kv_store_del(kv_store *s, const char *key, size_t klen) {
    kv_entry *e = kv_store_find(s, key, klen);
    if (e == NULL) {
        return false;
    }
    free(e->key);
    free(e->val);
    *e = s->items[--s->len];
    return true;
}

static void kv_store_free(kv_store *s) {
    for (size_t i = 0; i < s->len; i++) {
        free(s->items[i].key);
        free(s->items[i].val);
    }
    free(s->items);
    s->items = NULL;
    s->len = s->cap = 0;
}
c/ch05/kv.c · region store-ops

kv_store_del moves the last entry into the hole that the deleted one leaves. That keeps the list packed without moving everything after it, at the cost of order, which a store does not promise. A value is freed in three places. kv_store_set frees the old one on replacement, kv_store_del frees it, and at exit kv_store_free walks the list once. Every malloc in this file has one free that matches it, and a reader can find the pair.

INCR is where bytes and numbers meet. The stored value is bytes. The command parses them, adds one, and stores bytes again. The error messages are the ones Redis uses, so that a client written for Redis understands kv. The check against LLONG_MAX comes before the add, and the section on integers at the end of the chapter says why it must:

/* INCR: the value is bytes that must parse as an integer, and the new value is bytes too.
 * The check against LLONG_MAX comes before the add, because overflowing a signed integer
 * is undefined behaviour, and the compiler may assume it never happens. */
static bool kv_cmd_incr(const kv_cmd *c, kv_store *s, kv_buf *out) {
    long long n = 0;
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
    if (e != NULL && !kv_parse_int(e->val, e->vlen, &n)) {
        return kv_reply_error(out, "ERR value is not an integer or out of range");
    }
    if (n == LLONG_MAX) {
        return kv_reply_error(out, "ERR increment or decrement would overflow");
    }
    n++;
    char text[32];
    int len = snprintf(text, sizeof text, "%lld", n);
    if (!kv_store_set(s, c->argv[1], c->arglen[1], text, (size_t)len)) {
        return false;
    }
    return kv_reply_int(out, n);
}
c/ch05/kv.c · region incr

A table of functions#

Chapter 4 ran a command through a chain of if statements, one per name. Five more commands make the chain long, and the arity check is repeated in every link. The shape underneath is the same each time. A name, how many arguments it takes, and what to do. That is a row of a table, and C can hold the "what to do" in a function pointer. A function pointer is the address of a function, with the signature it expects spelled out, so the compiler knows how to call through it. Every command becomes a function of one shape:

/* Each command is a function of this one shape, so a table can hold them all. */
typedef bool (*kv_cmd_fn)(const kv_cmd *c, kv_store *s, kv_buf *out);

static bool kv_cmd_ping(const kv_cmd *c, kv_store *s, kv_buf *out) {
    (void)s;
    if (c->argc == 1) {
        return kv_reply_simple(out, "PONG");
    }
    if (c->argc == 2) {
        return kv_reply_bulk(out, c->argv[1], c->arglen[1]);
    }
    return kv_reply_arity(out, "ping");
}

static bool kv_cmd_echo(const kv_cmd *c, kv_store *s, kv_buf *out) {
    (void)s;
    return kv_reply_bulk(out, c->argv[1], c->arglen[1]);
}

static bool kv_cmd_set(const kv_cmd *c, kv_store *s, kv_buf *out) {
    if (!kv_store_set(s, c->argv[1], c->arglen[1], c->argv[2], c->arglen[2])) {
        return false;
    }
    return kv_reply_simple(out, "OK");
}

static bool kv_cmd_get(const kv_cmd *c, kv_store *s, kv_buf *out) {
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
    return e ? kv_reply_bulk(out, e->val, e->vlen) : kv_reply_nil(out);
}

static bool kv_cmd_del(const kv_cmd *c, kv_store *s, kv_buf *out) {
    long long n = 0;
    for (size_t i = 1; i < c->argc; i++) {
        n += kv_store_del(s, c->argv[i], c->arglen[i]);
    }
    return kv_reply_int(out, n);
}

static bool kv_cmd_exists(const kv_cmd *c, kv_store *s, kv_buf *out) {
    long long n = 0;
    for (size_t i = 1; i < c->argc; i++) {
        n += kv_store_find(s, c->argv[i], c->arglen[i]) != NULL;
    }
    return kv_reply_int(out, n);
}

static bool kv_cmd_command(const kv_cmd *c, kv_store *s, kv_buf *out) {
    (void)c;
    (void)s;
    return kv_reply_array(out, 0); /* redis-cli asks on connect; an empty list is fine */
}
c/ch05/kv.c · region commands

The table is the list of rows. It is written once, as a macro that calls another macro, X, on each row. Each place that needs the list defines X to say what to make of a row, uses the list, and undefines X again. The first use builds the array of rows. The second counts them, by turning each row into +1. This is called an X-macro, and it is how C keeps one list in one place when several things have to agree with it:

/* The command list, written once. Each use of KV_COMMANDS defines X to say what to make
 * of a row. The first use below builds the table, the second counts it. A name is spelled
 * as the arity error must show it, and arity is counted as Redis counts it: a positive
 * number is the exact argc, name included, and a negative one is a minimum. */
#define KV_COMMANDS                     \
    X(ping, kv_cmd_ping, -1)            \
    X(echo, kv_cmd_echo, 2)             \
    X(set, kv_cmd_set, 3)               \
    X(get, kv_cmd_get, 2)               \
    X(del, kv_cmd_del, -2)              \
    X(exists, kv_cmd_exists, -2)        \
    X(incr, kv_cmd_incr, 2)             \
    X(append, kv_cmd_append, 3)         \
    X(strlen, kv_cmd_strlen, 2)         \
    X(getrange, kv_cmd_getrange, 4)     \
    X(command, kv_cmd_command, -1)

typedef struct {
    const char *name;
    kv_cmd_fn fn;
    int arity;
} kv_command;

static const kv_command kv_commands[] = {
#define X(name, fn, arity) {#name, fn, arity},
    KV_COMMANDS
#undef X
};

enum {
#define X(name, fn, arity) +1
    KV_COMMAND_COUNT = 0 KV_COMMANDS
#undef X
};
c/ch05/kv.c · region table

Running a command is now a walk down the table. The arity check happens once, in the loop, and the function is called through its pointer with the same arguments the chain used to pass:

/* Run one command against the store and append its reply to out: find the row, check
 * the arity, call the function through its pointer. */
static bool kv_exec(const kv_cmd *c, kv_store *s, kv_buf *out) {
    if (c->argc == 0) {
        return kv_reply_error(out, "ERR empty command");
    }
    for (size_t i = 0; i < KV_COMMAND_COUNT; i++) {
        const kv_command *cmd = &kv_commands[i];
        if (!kv_arg_is(c, 0, cmd->name)) {
            continue;
        }
        int argc = (int)c->argc;
        bool ok = cmd->arity < 0 ? argc >= -cmd->arity : argc == cmd->arity;
        if (!ok) {
            return kv_reply_arity(out, cmd->name);
        }
        return cmd->fn(c, s, out);
    }
    int n = c->arglen[0] < 40 ? (int)c->arglen[0] : 40;
    return kv_reply_errorf(out, "ERR unknown command '%.*s'", n, c->argv[0]);
}
c/ch05/kv.c · region exec
โœ…Behaviour in a table, not in a chain C

When several things of one shape are chosen by name, put them in a table of structs. Each struct holds a function pointer, and the code walks the table. Redis does this with its command table, and every interpreter, driver model and plugin system in C does the same. A chain of if grows with every feature, and a table grows by one row.

Growing, measuring and slicing bytes#

Three more commands work on the bytes of a value. APPEND makes a value longer, STRLEN measures it, and GETRANGE cuts a piece out by offset. Each one is a few lines, and each one meets a rule of C bytes:

/* APPEND key value: grow the stored bytes in place. A missing key starts empty. */
static bool kv_cmd_append(const kv_cmd *c, kv_store *s, kv_buf *out) {
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
    if (e == NULL) {
        if (!kv_store_set(s, c->argv[1], c->arglen[1], c->argv[2], c->arglen[2])) {
            return false;
        }
        return kv_reply_int(out, (long long)c->arglen[2]);
    }
    char *v = realloc(e->val, e->vlen + c->arglen[2] + 1);
    if (v == NULL) {
        return false; /* e->val is still valid: realloc leaves it alone on failure */
    }
    memcpy(v + e->vlen, c->argv[2], c->arglen[2]);
    e->val = v;
    e->vlen += c->arglen[2];
    return kv_reply_int(out, (long long)e->vlen);
}

/* STRLEN key: the length in bytes, which is the only length the store knows. */
static bool kv_cmd_strlen(const kv_cmd *c, kv_store *s, kv_buf *out) {
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
    return kv_reply_int(out, e ? (long long)e->vlen : 0);
}

/* GETRANGE key start end: a slice by byte offsets, both inclusive. A negative offset
 * counts from the end, and an offset past either end is clamped, as Redis does. */
static bool kv_cmd_getrange(const kv_cmd *c, kv_store *s, kv_buf *out) {
    long long start, end;
    if (!kv_parse_int(c->argv[2], c->arglen[2], &start) ||
        !kv_parse_int(c->argv[3], c->arglen[3], &end)) {
        return kv_reply_error(out, "ERR value is not an integer or out of range");
    }
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
    long long len = e ? (long long)e->vlen : 0;
    if (start < 0) {
        start += len;
    }
    if (end < 0) {
        end += len;
    }
    if (start < 0) {
        start = 0;
    }
    if (end >= len) {
        end = len - 1;
    }
    if (len == 0 || start > end) {
        return kv_reply_bulk(out, "", 0);
    }
    return kv_reply_bulk(out, e->val + start, (size_t)(end - start + 1));
}
c/ch05/kv.c · region strings

APPEND asks realloc to grow the block the value lives in. When the block cannot grow in place, realloc moves it, copies the bytes and frees the old one. So the pointer in the entry must be replaced. On failure it returns NULL and the old block is untouched, which is why the result goes into a new variable first. STRLEN answers in bytes, because the store knows no other length. GETRANGE is arithmetic on offsets, with four clamps, and then a pointer plus a length into the value. No byte is copied until the reply is written. The one function from the C string library in this chapter is snprintf, which turns the number in INCR into digits. It takes the size of its buffer, writes at most that many bytes with a terminator, and returns how many it wanted to write. Its older cousin sprintf takes no size and is how buffers overflow.

Run it, and the first real session with the real client works:

$ make && ./kv &
$ redis-cli -p 6380
127.0.0.1:6380> SET name kv
OK
127.0.0.1:6380> GET name
"kv"
127.0.0.1:6380> INCR visits
(integer) 1
127.0.0.1:6380> INCR name
(error) ERR value is not an integer or out of range
127.0.0.1:6380> DEL name visits
(integer) 2
โœ…A pointer and a length travel together C

Any function that takes bytes takes two parameters, the pointer and the length, in that order. A structure that holds bytes holds both fields side by side. The habit is older than kv. The read and write calls of POSIX, the Portable Operating System Interface, take a pointer and a length. Redis keeps a length in front of every string it stores. A C string is only for text the program itself writes, such as an error message, where the zero byte is guaranteed.

Go: two kinds of string, and a copy between them#

Go has both answers to the question. A string is immutable bytes with a length. A []byte is mutable bytes with a length. Converting one to the other copies the bytes, because the string must not change when the slice does. The store uses a string for the key, because a map key must support ==, and a slice does not. The value stays a slice:

// store keeps every key and its value. The parser allocated each slice, so the store can
// keep them without copying: nothing else holds them.
type store struct {
	m map[string][]byte
}

func newStore() *store {
	return &store{m: make(map[string][]byte)}
}

func (s *store) get(key []byte) ([]byte, bool) {
	v, ok := s.m[string(key)]
	return v, ok
}

func (s *store) set(key, val []byte) {
	s.m[string(key)] = val
}

func (s *store) del(key []byte) bool {
	_, ok := s.m[string(key)]
	delete(s.m, string(key))
	return ok
}

func (s *store) has(key []byte) bool {
	_, ok := s.m[string(key)]
	return ok
}
go/ch05/store.go · region store

The value is not copied on set. The key is converted to a string, which copies it once, because a map key must be comparable. The parser allocated a fresh slice for every argument, and nothing else holds it. The garbage collector keeps it alive for as long as the map refers to it. The question that took a paragraph of comments in C does not arise. The cost is in the collector, which must find out, from time to time, which slices are still referred to.

// The two ways INCR fails. Their text is what the client sees.
var (
	errNotInteger = errors.New("ERR value is not an integer or out of range")
	errOverflow   = errors.New("ERR increment or decrement would overflow")
)
go/ch05/store.go · region errors
Exercise 05.2GoAdd one to a value

Write the incr method. Read the current value if there is one, parse it with strconv.ParseInt, and return errNotInteger when parsing fails. Return errOverflow when the value is already math.MaxInt64. Otherwise store the new value, formatted with strconv.FormatInt, and return it.

func (s *store) incr(key []byte) (int64, error) {
	var n int64
	// your code here
	return n, nil
}
Stuck?Show the full listing
// incr adds one to the integer stored under key, which counts as 0 when absent.
func (s *store) incr(key []byte) (int64, error) {
	var n int64
	if cur, ok := s.get(key); ok {
		var err error
		n, err = strconv.ParseInt(string(cur), 10, 64)
		if err != nil {
			return 0, errNotInteger
		}
	}
	if n == math.MaxInt64 {
		return 0, errOverflow
	}
	n++
	s.set(key, []byte(strconv.FormatInt(n, 10)))
	return n, nil
}
go/ch05/store.go · region incr

The two errors are package-level values with the exact text the client should see. The command code does not look inside them. It writes err.Error() as the reply, which keeps the knowledge of what can go wrong next to the code that knows it.

A map of functions#

In Go a function is a value like any other. It can sit in a variable, in a struct field or in a map, and a func type names its shape. The command table is a map from the upper-cased name to a struct that holds the function and its arity:

// A handler is any function of this shape. The table maps the upper-cased name to one,
// with the arity as Redis counts it: a positive number is the exact argc, name included,
// and a negative one is a minimum.
type handler func(args [][]byte, s *store, w *bufio.Writer) error

type command struct {
	run   handler
	arity int
}

var commands = map[string]command{
	"PING":     {ping, -1},
	"ECHO":     {echo, 2},
	"SET":      {set, 3},
	"GET":      {get, 2},
	"DEL":      {del, -2},
	"EXISTS":   {exists, -2},
	"INCR":     {incr, 2},
	"APPEND":   {appendCmd, 3},
	"STRLEN":   {strlenCmd, 2},
	"GETRANGE": {getrange, 4},
	"COMMAND":  {commandCmd, -1},
}
go/ch05/exec.go · region table

Running a command is a map lookup, an arity check, and a call. The chain of cases is gone, and adding a command is one line in the map and one function:

// execute runs one command against the store and writes its reply: look the name up,
// check the arity, call the function the table holds.
func execute(args [][]byte, s *store, w *bufio.Writer) error {
	if len(args) == 0 {
		return writeError(w, "ERR empty command")
	}
	name := strings.ToUpper(string(args[0]))
	cmd, ok := commands[name]
	if !ok {
		shown := string(args[0])
		if len(shown) > 40 {
			shown = shown[:40]
		}
		return writeError(w, "ERR unknown command '"+shown+"'")
	}
	if n := len(args); (cmd.arity < 0 && n < -cmd.arity) || (cmd.arity > 0 && n != cmd.arity) {
		return writeArity(w, strings.ToLower(name))
	}
	return cmd.run(args, s, w)
}

func writeArity(w *bufio.Writer, name string) error {
	return writeError(w, "ERR wrong number of arguments for '"+name+"' command")
}
go/ch05/exec.go · region exec

Two of the handlers show what a function value can be. countKeys takes a function from a key to a bool and counts the keys for which it is true. DEL passes s.del, which is a method value. It is the method with its receiver already bound, so it can be called later with the key alone. EXISTS passes a closure, a function literal that captures s from the surrounding function and calls get on it. A closure captures by reference. It sees the variable as it is when the closure runs, not as it was when the closure was made. The defer test of chapter 3 showed the same rule:

func ping(args [][]byte, _ *store, w *bufio.Writer) error {
	switch len(args) {
	case 1:
		return writeSimple(w, "PONG")
	case 2:
		return writeBulk(w, args[1])
	}
	return writeArity(w, "ping")
}

func echo(args [][]byte, _ *store, w *bufio.Writer) error {
	return writeBulk(w, args[1])
}

func set(args [][]byte, s *store, w *bufio.Writer) error {
	s.set(args[1], args[2])
	return writeSimple(w, "OK")
}

func get(args [][]byte, s *store, w *bufio.Writer) error {
	if v, ok := s.get(args[1]); ok {
		return writeBulk(w, v)
	}
	return writeNil(w)
}

// countKeys counts the keys for which hit returns true. DEL and EXISTS differ only in
// which method they pass: s.del removes as it counts, s.has only looks.
func countKeys(keys [][]byte, hit func([]byte) bool) int64 {
	var n int64
	for _, key := range keys {
		if hit(key) {
			n++
		}
	}
	return n
}

func del(args [][]byte, s *store, w *bufio.Writer) error {
	return writeInt(w, countKeys(args[1:], s.del))
}

func exists(args [][]byte, s *store, w *bufio.Writer) error {
	return writeInt(w, countKeys(args[1:], s.has))
}

func incr(args [][]byte, s *store, w *bufio.Writer) error {
	n, err := s.incr(args[1])
	if err != nil {
		return writeError(w, err.Error())
	}
	return writeInt(w, n)
}

func commandCmd(_ [][]byte, _ *store, w *bufio.Writer) error {
	return writeArrayLen(w, 0) // redis-cli asks on connect; an empty list is fine
}
go/ch05/exec.go · region commands

Bytes, runes and a builder#

Three more commands work on the bytes of a value. APPEND uses the built-in append. It may write into the spare capacity of the old slice or allocate a new array, and the map keeps whichever slice comes back. STRLEN replies with len, which counts bytes. The character count of a string is a different question in Go. A string is bytes in UTF-8, and a character outside ASCII takes more than one byte. A rune is one character as an int32, range over a string walks runes, and utf8.RuneCount counts them. GETRANGE slices by byte offset, which is what Redis does and what may cut a rune in half. The test below is in the chapter's snapshot. It shows the three facts and strings.Builder, which grows a string without copying it at every step:

// A string is bytes. len counts them, and a character outside ASCII takes more than one.
func TestBytesAndRunes(t *testing.T) {
	s := "hรฉllo"
	if len(s) != 6 || utf8.RuneCountInString(s) != 5 {
		t.Fatalf("len %d, runes %d", len(s), utf8.RuneCountInString(s))
	}
	// range walks runes, not bytes: the index jumps from 1 to 3 over the two bytes of รฉ
	var idx []int
	for i := range s {
		idx = append(idx, i)
	}
	if idx[2] != 3 {
		t.Fatalf("indexes %v", idx)
	}
	// s[1:2] is one byte, half of รฉ, and not a valid string on its own
	if utf8.ValidString(s[1:2]) {
		t.Fatal("half a rune should not be valid")
	}
	// strings.Builder grows a string without copying it at every step
	var b strings.Builder
	for i := 0; i < 3; i++ {
		b.WriteString("ab")
	}
	if b.String() != "ababab" {
		t.Fatal(b.String())
	}
}
go/ch05/strings_test.go · region runes
// appendCmd grows the stored bytes. append may reuse the old slice's spare capacity or
// allocate a new array; either way the map is updated with the slice it returns.
func appendCmd(args [][]byte, s *store, w *bufio.Writer) error {
	old, _ := s.get(args[1])
	v := append(old, args[2]...)
	s.set(args[1], v)
	return writeInt(w, int64(len(v)))
}

// strlenCmd replies with the length in bytes. len counts bytes, and so does Redis; the
// number of characters is a different question, which utf8.RuneCount answers.
func strlenCmd(args [][]byte, s *store, w *bufio.Writer) error {
	v, _ := s.get(args[1])
	_ = utf8.RuneCount(v) // the count a user would expect for text, which STRLEN is not
	return writeInt(w, int64(len(v)))
}

// getrange slices by byte offsets, both inclusive. A negative offset counts from the end,
// and an offset past either end is clamped, as Redis does.
func getrange(args [][]byte, s *store, w *bufio.Writer) error {
	start, err1 := strconv.ParseInt(string(args[2]), 10, 64)
	end, err2 := strconv.ParseInt(string(args[3]), 10, 64)
	if err1 != nil || err2 != nil {
		return writeError(w, "ERR value is not an integer or out of range")
	}
	v, _ := s.get(args[1])
	n := int64(len(v))
	if start < 0 {
		start += n
	}
	if end < 0 {
		end += n
	}
	start = max(start, 0)
	end = min(end, n-1)
	if n == 0 || start > end {
		return writeBulk(w, nil)
	}
	return writeBulk(w, v[start:end+1])
}
go/ch05/exec.go · region strings
โš ๏ธParseInt is more generous than Redis Go

strconv.ParseInt accepts +5 and 007. Redis stores both and then refuses to INCR them, and the C version of kv does the same. The Go version accepts them, and so does the Rust version, because both standard libraries parse numbers the same generous way. The book leaves the difference in and tells you about it, because that is what happens in real ports. A strict parser in Go is a short function, and the C version shows what it must check.

Rust: a value has one owner#

Rust has the same two kinds as Go, String for text and Vec<u8> for bytes, with one more rule. A String must be valid UTF-8, and the compiler will not let you build one from bytes without a check. A Redis key is bytes, so the store is a map from Vec<u8> to Vec<u8>. A Vec can be a key because it supports hashing and equality:

/// Every key and its value. The store owns both: a key or value that is put in is moved,
/// not copied, and is freed when it is replaced, deleted or the store is dropped.
#[derive(Default)]
pub struct Store {
    map: HashMap<Vec<u8>, Vec<u8>>,
}

impl Store {
    pub fn new() -> Self {
        Self::default()
    }

    /// A borrowed view of the value, which lives as long as the borrow of the store.
    pub fn get(&self, key: &[u8]) -> Option<&[u8]> {
        self.map.get(key).map(Vec::as_slice)
    }

    pub fn set(&mut self, key: Vec<u8>, val: Vec<u8>) {
        self.map.insert(key, val);
    }

    pub fn del(&mut self, key: &[u8]) -> bool {
        self.map.remove(key).is_some()
    }

    /// Append to the value under key, which starts empty, and return the new length. The
    /// entry API finds or inserts in one lookup, and `extend_from_slice` grows the `Vec`
    /// in place when it has room.
    pub fn append(&mut self, key: Vec<u8>, val: Vec<u8>) -> usize {
        let v = self.map.entry(key).or_default();
        v.extend_from_slice(&val);
        v.len()
    }
}
rust/ch05/src/store.rs · region store

Two signatures carry the whole ownership story. get takes &self and returns Option<&[u8]>, a borrowed view that lives only as long as the borrow of the store. set takes Vec<u8> by value, which means the caller gives the bytes away. Nothing is copied, and nothing can be used twice.

The command function takes its arguments by value for the same reason. For SET, mem::take moves the key and the value out of the argument list and leaves empty vectors behind. The store receives them without a copy.

A table of fn pointers, built by a macro#

Rust has function pointers too, with the type fn(...) -> ..., and a const array of structs can hold them. The signature has one new thing in it. The writer is &mut dyn Write, which means any writer, chosen at run time. The helpers in resp take impl Write, which means any writer chosen at compile time. The compiler makes a copy of each helper per writer type. A table of pointers cannot hold a copy per type, so the handlers take the run-time form. Chapter 7 returns to the difference. The table itself is written through a macro_rules! macro, which turns each name => function, arity line into an Entry:

/// Every command is a function of this one shape, so a table can hold them. The writer is
/// `dyn Write`, any writer decided at run time, because a table cannot hold one function
/// per writer type. Chapter 7 explains `dyn`.
type Handler = fn(&mut Command, &mut Store, &mut dyn Write) -> io::Result<()>;

/// A row of the table: the name as the client spells it, the function, and the arity as
/// Redis counts it. Positive is the exact argc, name included; negative is a minimum.
struct Entry {
    name: &'static [u8],
    run: Handler,
    arity: i32,
}

/// The table, written once. The macro turns each `name => function, arity` line into an
/// `Entry`, so the list reads like a list and not like a struct literal eleven times.
macro_rules! commands {
    ($($name:literal => $run:ident, $arity:expr;)*) => {
        const COMMANDS: &[Entry] = &[$(Entry { name: $name, run: $run, arity: $arity }),*];
    };
}

commands! {
    b"PING" => ping, -1;
    b"ECHO" => echo, 2;
    b"SET" => set, 3;
    b"GET" => get, 2;
    b"DEL" => del, -2;
    b"EXISTS" => exists, -2;
    b"INCR" => incr, 2;
    b"APPEND" => append, 3;
    b"STRLEN" => strlen, 2;
    b"GETRANGE" => getrange, 4;
    b"COMMAND" => command, -1;
}
rust/ch05/src/cmd.rs · region table

A macro_rules! macro matches the tokens between its brackets against a pattern and writes code from a template. $name:literal captures a literal and $run:ident a name. $arity:expr captures an expression, and $( ... ;)* repeats for every line. The macro runs before type checking, so what it writes is checked as if you had typed it. The lookup is then a search of the array, and the call goes through the pointer:

/// Run one command against the store and write its reply: find the row, check the arity,
/// call the function. The command is taken by value, so that SET can move the key and
/// the value into the store instead of copying them.
pub fn execute(mut args: Command, store: &mut Store, w: &mut impl Write) -> io::Result<()> {
    let Some(name) = args.first() else {
        return resp::write_error(w, "ERR empty command");
    };
    let name = name.to_ascii_uppercase();
    let Some(entry) = COMMANDS.iter().find(|e| e.name == name.as_slice()) else {
        let shown = String::from_utf8_lossy(&args[0][..args[0].len().min(40)]);
        return resp::write_error(w, &format!("ERR unknown command '{shown}'"));
    };
    let argc = args.len() as i32;
    let ok = if entry.arity < 0 {
        argc >= -entry.arity
    } else {
        argc == entry.arity
    };
    if !ok {
        return write_arity(w, entry.name);
    }
    (entry.run)(&mut args, store, w)
}

fn write_arity(w: &mut dyn Write, name: &[u8]) -> io::Result<()> {
    let name = String::from_utf8_lossy(name).to_ascii_lowercase();
    resp::write_error(
        w,
        &format!("ERR wrong number of arguments for '{name}' command"),
    )
}
rust/ch05/src/cmd.rs · region exec
fn ping(args: &mut Command, _: &mut Store, w: &mut dyn Write) -> io::Result<()> {
    match args.as_slice() {
        [_] => resp::write_simple(w, "PONG"),
        [_, msg] => resp::write_bulk(w, msg),
        _ => write_arity(w, b"ping"),
    }
}

fn echo(args: &mut Command, _: &mut Store, w: &mut dyn Write) -> io::Result<()> {
    resp::write_bulk(w, &args[1])
}

fn set(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
    let [_, key, val] = args.as_mut_slice() else {
        unreachable!("arity checked")
    };
    store.set(mem::take(key), mem::take(val));
    resp::write_simple(w, "OK")
}

fn get(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
    match store.get(&args[1]) {
        Some(val) => resp::write_bulk(w, val),
        None => resp::write_nil(w),
    }
}

fn del(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
    let n = args[1..].iter().filter(|k| store.del(k)).count();
    resp::write_int(w, n as i64)
}

fn exists(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
    let n = args[1..].iter().filter(|k| store.get(k).is_some()).count();
    resp::write_int(w, n as i64)
}

fn incr(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
    match store.incr(&args[1]) {
        Ok(n) => resp::write_int(w, n),
        Err(e) => resp::write_error(w, e.message()),
    }
}

fn command(_: &mut Command, _: &mut Store, w: &mut dyn Write) -> io::Result<()> {
    resp::write_array_len(w, 0) // redis-cli asks on connect; an empty list is fine
}
rust/ch05/src/cmd.rs · region commands

Each handler is a match on the argument slice. One arm can say "PING with nothing", the next "PING with one message". A let ... else with a slice pattern takes the key and the value out of SET. The unreachable! says that the arity check above makes the other case impossible.

str, String, char and a slice of bytes#

Three more commands work on the bytes of a value. APPEND uses the map's entry API to find or insert in one lookup and extends the Vec in place. STRLEN is len, in bytes. GETRANGE slices the Vec by byte offset. None of them can be written on a String. A String is a Vec<u8> with a promise, that its bytes are valid UTF-8. A slice at a byte offset can break the promise. The test below is in the chapter's snapshot. A str has a length in bytes and a count of char, which are different. A slice of a str must land on a character boundary, and one that does not is a panic. The same bytes as a &[u8] can be cut anywhere, which is why kv stores values as bytes:

/// A `str` is bytes that are valid UTF-8. `len` counts the bytes, `chars` the
/// characters, and a slice must land on a character boundary.
#[test]
fn bytes_and_chars() {
    let s = "hรฉllo";
    assert_eq!(s.len(), 6);
    assert_eq!(s.chars().count(), 5);
    assert_eq!(&s[0..1], "h");
    assert_eq!(&s[1..3], "รฉ");
    assert!(s.is_char_boundary(3));
    assert!(!s.is_char_boundary(2));
    // the same bytes as a slice can be cut anywhere, which is what GETRANGE needs
    assert_eq!(&s.as_bytes()[1..2], &[0xC3]);
}

/// Slicing a `str` in the middle of a character is a panic, not a corrupt string.
#[test]
#[should_panic(expected = "byte index 2 is not a char boundary")]
fn half_a_char_panics() {
    let s = "hรฉllo";
    let _ = &s[0..2];
}
rust/ch05/src/store.rs · region text
/// APPEND key value: grow the stored bytes. The value is moved in when the key is new.
fn append(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
    let [_, key, val] = args.as_mut_slice() else {
        unreachable!("arity checked")
    };
    let n = store.append(mem::take(key), mem::take(val));
    resp::write_int(w, n as i64)
}

/// STRLEN key: the length in bytes. `len` on a `Vec<u8>` counts bytes, and so does Redis.
/// The number of characters is a different question, which `chars().count()` answers on
/// a `str`, and which the store cannot ask, because a value need not be text.
fn strlen(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
    let n = store.get(&args[1]).map_or(0, <[u8]>::len);
    resp::write_int(w, n as i64)
}

/// GETRANGE key start end: a slice by byte offsets, both inclusive. A negative offset
/// counts from the end, and an offset past either end is clamped, as Redis does.
fn getrange(args: &mut Command, store: &mut Store, w: &mut dyn Write) -> io::Result<()> {
    let (Some(start), Some(end)) = (parse_int(&args[2]), parse_int(&args[3])) else {
        return resp::write_error(w, "ERR value is not an integer or out of range");
    };
    let val = store.get(&args[1]).unwrap_or(b"");
    let len = val.len() as i64;
    let start = if start < 0 { start + len } else { start }.max(0);
    let end = if end < 0 { end + len } else { end }.min(len - 1);
    if len == 0 || start > end {
        return resp::write_bulk(w, b"");
    }
    resp::write_bulk(w, &val[start as usize..=end as usize])
}
rust/ch05/src/cmd.rs · region strings
/// Why INCR could not add one.
pub enum IncrError {
    NotInteger,
    Overflow,
}

impl IncrError {
    pub fn message(&self) -> &'static str {
        match self {
            Self::NotInteger => "ERR value is not an integer or out of range",
            Self::Overflow => "ERR increment or decrement would overflow",
        }
    }
}
/// The bytes as a signed 64-bit integer, if that is what they are.
pub fn parse_int(bytes: &[u8]) -> Option<i64> {
    std::str::from_utf8(bytes).ok()?.parse().ok()
}
rust/ch05/src/store.rs · region incr-types
Exercise 05.3RustAdd one to a value

Write the incr method. Use parse_int for the current value. Turn its None into IncrError::NotInteger with ok_or. Use checked_add, which returns None on overflow, and turn that into IncrError::Overflow. Store the new value with to_string and into_bytes, and return it.

pub fn incr(&mut self, key: &[u8]) -> Result<i64, IncrError> {
    let n = match self.get(key) {
        None => 0,
        Some(bytes) => parse_int(bytes).ok_or(IncrError::NotInteger)?,
    };
    // your code here
}
Stuck?Show the full listing
impl Store {
    /// Add one to the integer stored under key, which counts as 0 when absent.
    pub fn incr(&mut self, key: &[u8]) -> Result<i64, IncrError> {
        let n = match self.get(key) {
            None => 0,
            Some(bytes) => parse_int(bytes).ok_or(IncrError::NotInteger)?,
        };
        let n = n.checked_add(1).ok_or(IncrError::Overflow)?;
        self.set(key.to_vec(), n.to_string().into_bytes());
        Ok(n)
    }
}
rust/ch05/src/store.rs · region incr

IncrError is an enum with two cases, and the command code matches on it to write the reply. This is the Rust form of the two Go sentinel values. The difference is that a match on an enum must handle every case. So a third kind of failure, added later, is a compile error at every place that forgot it.

โš ๏ธYou cannot keep a value and change the store Rust

A borrowed value blocks every write to the store until it is dropped. The following three lines do not compile, and the compiler says why:

let old = store.get(b"a");
store.set(b"a".to_vec(), b"2".to_vec());
println!("{:?}", old);
error[E0502]: cannot borrow `store` as mutable because it is also borrowed as immutable
   |     let old = store.get(b"a");
   |               ----- immutable borrow occurs here
   |     store.set(b"a".to_vec(), b"2".to_vec());
   |     ^^^^^ mutable borrow occurs here
   |     println!("{:?}", old);
   |                      --- immutable borrow later used here

In C the same three lines compile, and old points at freed memory after the set. The Rust error is the C bug, caught before the program runs. When you need the old value after a write, copy it first with to_vec(). That is what the C code did with kv_bytes_dup, and it is what the compiler is asking for.

Three answers to the largest integer plus one#

Start any of the three servers and run SET big 9223372036854775807, then INCR big. All three reply with the Redis error, because each one checks before it adds. Remove the check and the three languages give three different answers, and that is the lesson of this section.

In C, adding one to LLONG_MAX is undefined behaviour. The standard does not say what happens, so the compiler is allowed to assume it never happens, and it does. A check written after the add, such as if (n + 1 < n), is removed by the optimiser as impossible. The check must come before, as kv_cmd_incr writes it. The parser of chapter 5 had the same rule and refused more than 18 digits for the same reason. Unsigned arithmetic is the exception. It wraps around by definition, which is why a hash function in chapter 6 is written with unsigned types.

In Go, integers wrap. The largest int64 plus one is the smallest, with no error, no panic and no flag. The test in the chapter's snapshot shows it:

// Go integers wrap. The largest int64 plus one is the smallest, with no error and no panic,
// which is why incr checks against math.MaxInt64 before it adds.
func TestOverflowWraps(t *testing.T) {
	n := int64(math.MaxInt64)
	n++
	if n != math.MinInt64 {
		t.Fatalf("got %d", n)
	}
}
go/ch05/strings_test.go · region overflow

In Rust, the answer depends on the build. A debug build panics on the overflow, with the message attempt to add with overflow. A release build wraps, as Go does. Three methods say what you want in their names, checked_add, wrapping_add and saturating_add. They behave the same in both builds. kv uses checked_add, which returns None on overflow, and turns that into the Redis error:

/// Three ways to add one to the largest integer: a debug build panics on plain `+`,
/// a release build wraps, and `checked_add` returns `None` in both. kv uses the last.
#[test]
fn overflow_three_ways() {
    let n = i64::MAX;
    assert_eq!(n.checked_add(1), None);
    assert_eq!(n.wrapping_add(1), i64::MIN);
    assert_eq!(n.saturating_add(1), i64::MAX);
}

#[test]
#[cfg(debug_assertions)]
#[should_panic(expected = "attempt to add with overflow")]
fn plain_add_panics_in_debug() {
    let n = std::hint::black_box(i64::MAX);
    let _ = n + 1;
}
rust/ch05/src/store.rs · region overflow
โš ๏ธThe compiler believes you C

A test for signed overflow that runs after the arithmetic may be deleted by the compiler. Because the overflow cannot happen in a correct program, a branch that only runs when it did happen is dead code to the optimiser. The same reasoning removes a NULL check after a pointer has already been used. UndefinedBehaviorSanitizer, in chapter 12, reports the overflow at run time, and -ftrapv turns it into a crash. Both are off in a release build.

Who owns the bytes#

Every version of SET had to answer one question. The bytes arrived in a request buffer. Who keeps them, and who frees them? C answered with a copy and a comment. The store makes its own copy with kv_bytes_dup and frees it on replacement. The comment on the field is the only record of the rule. Go answered with the collector. The store keeps the parser's slice. The collector frees it when nothing refers to it any more, at a time the program does not choose. Rust answered with a move. The store takes the bytes from the argument list, which is left empty. The compiler refuses any later use of what was moved.

All three answers work. The C answer costs a copy and a rule the compiler does not know. The Go answer costs memory until the next collection and a pause when it runs. The Rust answer costs nothing at run time. It costs a type signature that says Vec<u8> in place of the char * of C. Chapter 6 makes the costs larger, because the store becomes a structure that must free and move things itself.

Verdict##

TopicCGoRustWhy
A key is bytesOKStrongOKC carries a length beside every pointer, Go has []byte and string, and Rust makes you pick Vec<u8> because String rejects binary data.
Copying and owningWeakStrongStrongC copies and comments, Go lets the collector own everything, and Rust moves and checks.
Parsing a numberWeakOKOKC needs a careful parser to avoid undefined behaviour. The other two accept forms Redis rejects.
OverflowWeakOKStrongC must check before it adds and Go checks against a constant, while Rust has checked_add built in.
A missing caseWeakWeakStrongAn enum match in Rust must be complete. In C and Go, a new error kind can be forgotten at a call site.
Reading the codeOKStrongOKThe Go store is the shortest. The Rust signatures say more and take longer to read.
A table of commandsOKStrongOKC needs an X-macro and a function pointer type, and Go has a map of funcs. Rust needs dyn Write and a macro for the rows.
Text against bytesWeakOKStrongC has no text type at all, and Go has rune helpers. Rust makes a str keep its promise and a Vec<u8> make none.
Carry forward
  • Function pointer. A function is a value with an address, and a table of them replaces a chain of ifs in all three languages.
  • Bytes against text. A length in bytes is not a count of characters, and a slice at a byte offset can cut a character in half.
  • Signed overflow. Undefined in C, a silent wrap in Go, a panic in a Rust debug build, and a checked add in kv.
Next

The store is a list, and a lookup walks it. Fill it with twenty thousand keys, and a GET from one client takes a sixth of a millisecond, twenty times longer than Redis takes.

Your own hash table#

Building kv · Chapter 06· 36 min read· 606 lines of code
▶Scene

Start the chapter 5 server and fill it with twenty thousand keys. Then let the Redis benchmark tool ask for them, one client, one request at a time:

$ redis-cli -p 6380 --pipe < twenty-thousand-sets.txt
errors: 0, replies: 20000
$ redis-benchmark -p 6380 -c 1 -n 5000 -r 20000 -t get -q
GET: 6257.82 requests per second, p50=0.151 msec

Six thousand a second, from a program that does nothing but compare bytes. Real Redis answers the same question from one client about twenty times faster. Every GET is walking the list from chapter 5, and the list has twenty thousand entries.

The list from chapter 5 gets slower with every key. This chapter replaces it with a hash table that you write yourself, in all three languages. In C the table becomes the store. In Go and Rust the standard map stays, and your table stands beside it. The chapter is about memory, because a hash table must allocate, move and free its own parts.

With 20,000 keys in the store, the C server from chapter 5 answers about 6,300 GET requests a second from one client. Each lookup walks half the list on average. The same server with the hash table from this chapter answers about 17,700, and the median request takes 39 microseconds instead of 151. That is the measurement on the author's machine, and chapter 13 shows how to take it yourself. The difference grows with the number of keys, because the list takes time in proportion to the keys and the table does not.

A hash table turns a key into a number, called its hash. The low bits of that number are an index into an array of buckets. Two keys may land in the same bucket, so each bucket holds a chain of entries, and a lookup walks that short chain. When the table holds more entries than buckets, the array doubles and every entry moves to its new bucket. The chapter also adds two commands that ask about the whole table, DBSIZE and KEYS.

KEY "user:1" hash 0x9c1b...e3 & (cap-1) 3 BUCKETS, cap = 4 0 1 2 3 "user:1" = "ann" next "k7" = "1" next = NULL "order:1" = "x" a chain: the keys that share bucket 3
A hash table with separate chaining. The hash of the key, masked with the number of buckets, picks a bucket. The bucket holds a chain of the entries that share it, and a lookup walks that short chain.
The tableCGoRust
A bucketa pointer to the first entry of a chaina slice of entriesa Vec of entries
A chain linka next pointer the entry ownsthe next element of the slicethe next element of the Vec
Growingcalloc a bigger array, relink every entrymake a bigger slice, append every entrydrain every chain into a bigger Vec
Deletingunlink, then free the key, the value and the entrymove the last entry into the holeswap_remove, which drops the entry
What frees the old arrayyou, with freethe garbage collector, laterthe old Vec is dropped at the end of the function
The hashFNV-1a, written hereFNV-1a, written hereFNV-1a here, SipHash in the standard map
Who would use iteveryone, because C has no mapnobody, the map is built innobody, HashMap is built in
๐Ÿ“œWhy Redis rehashes a little at a time

Doubling the array is the simplest way to grow a table, and it is what this chapter does. It has a cost. The move of every entry takes a time proportional to the table, and a Redis server with ten million keys cannot stop for that long. So the Redis dictionary keeps two arrays during a resize and moves a few buckets on every operation. The function is dictRehash(dict *d, int n) in dict.c, and a field named rehashidx records how far the move has gone. The built-in map of Go also grows incrementally. It splits a large map into several small tables that grow one at a time. The server kv needs no such trick, and the simple version is easier to see.

C: an array of chains, and every pointer accounted for#

The hash is FNV-1a, a few lines that multiply and exclusive-or. It is fast and spreads keys well, and it is not safe against an attacker who chooses keys that collide. The Rust section says why that matters:

/* FNV-1a: a small, fast hash of bytes. Not safe against an attacker who chooses keys
 * to collide, which Redis guards against with a secret seed. Chapter 12 returns to it. */
static uint64_t kv_hash(const char *s, size_t n) {
    uint64_t h = 14695981039346656037ULL;
    for (size_t i = 0; i < n; i++) {
        h ^= (unsigned char)s[i];
        h *= 1099511628211ULL;
    }
    return h;
}
c/ch06/kv.c · region hash

Two details of the hash are rules of C integers. The multiply overflows on almost every step, and that is fine, because uint64_t is unsigned and unsigned arithmetic wraps by definition. A signed type here would be undefined behaviour. And each byte is cast to unsigned char before it is mixed in. A plain char may be signed or unsigned, by platform. On this machine it is signed, so the byte 0xC3 of an รฉ is the value -61. Mixing -61 into a 64-bit hash sets the top 56 bits. The cast makes it 195 on every platform, and the hash the same everywhere. The same cast belongs in front of every ctype.h call, for the same reason.

An entry now has a next pointer, and the comment says that the entry owns it. That one word decides who frees a chain. The table is an array of chain heads. The count of buckets is always a power of two, so that the index is the hash masked with cap - 1 instead of a division:

/* One key and its value, in a chain of entries that share a bucket. The key and the
 * value are owned copies. The next pointer is owned too: freeing an entry frees its
 * chain. */
typedef struct kv_entry {
    char *key;
    size_t klen;
    char *val;
    size_t vlen;
    struct kv_entry *next;
} kv_entry;

/* A hash table with separate chaining. buckets[i] is the head of a chain, or NULL. The
 * table grows when it holds more entries than buckets, so a chain stays short. */
typedef struct {
    kv_entry **buckets; /* owned array of cap pointers */
    size_t cap;         /* a power of two, or 0 before the first insert */
    size_t len;         /* how many entries, over all chains */
} kv_store;

static kv_entry *kv_store_find(kv_store *s, const char *key, size_t klen) {
    if (s->cap == 0) {
        return NULL;
    }
    size_t i = kv_hash(key, klen) & (s->cap - 1);
    for (kv_entry *e = s->buckets[i]; e != NULL; e = e->next) {
        if (kv_bytes_eq(e->key, e->klen, key, klen)) {
            return e;
        }
    }
    return NULL;
}
/* The command list, written once. Each use of KV_COMMANDS defines X to say what to make
 * of a row. The first use builds the table, the second counts it. A positive arity is
 * the exact argc, name included, and a negative one is a minimum, as Redis counts. */
#define KV_COMMANDS                     \
    X(ping, kv_cmd_ping, -1)            \
    X(echo, kv_cmd_echo, 2)             \
    X(set, kv_cmd_set, 3)               \
    X(get, kv_cmd_get, 2)               \
    X(del, kv_cmd_del, -2)              \
    X(exists, kv_cmd_exists, -2)        \
    X(incr, kv_cmd_incr, 2)             \
    X(append, kv_cmd_append, 3)         \
    X(strlen, kv_cmd_strlen, 2)         \
    X(getrange, kv_cmd_getrange, 4)     \
    X(dbsize, kv_cmd_dbsize, 1)         \
    X(keys, kv_cmd_keys, 2)             \
    X(command, kv_cmd_command, -1)

typedef struct {
    const char *name;
    kv_cmd_fn fn;
    int arity;
} kv_command;

static const kv_command kv_commands[] = {
#define X(name, fn, arity) {#name, fn, arity},
    KV_COMMANDS
#undef X
};

enum {
#define X(name, fn, arity) +1
    KV_COMMAND_COUNT = 0 KV_COMMANDS
#undef X
};
c/ch06/kv.c · region table

Growing allocates a new array with calloc, which returns memory set to zero, so every new bucket starts as an empty chain. Then it walks every old chain and pushes each entry onto the front of its new one. No key or value is copied. Only the next pointers change, which is the point of keeping entries as separate allocations:

/* Double the bucket array and move every entry to its new chain. No entry is copied or
 * freed: only the next pointers change. */
static bool kv_store_grow(kv_store *s) {
    size_t cap = s->cap ? s->cap * 2 : 16;
    kv_entry **buckets = calloc(cap, sizeof *buckets);
    if (buckets == NULL) {
        return false;
    }
    for (size_t i = 0; i < s->cap; i++) {
        kv_entry *e = s->buckets[i];
        while (e != NULL) {
            kv_entry *next = e->next; /* read it before the chain is changed */
            size_t j = kv_hash(e->key, e->klen) & (cap - 1);
            e->next = buckets[j];
            buckets[j] = e;
            e = next;
        }
    }
    free(s->buckets);
    s->buckets = buckets;
    s->cap = cap;
    return true;
}
c/ch06/kv.c · region grow
โš ๏ธRead next before you change it C

The line kv_entry *next = e->next must come first. The loop sets e->next to the head of the new chain, and after that the old chain is gone. Swap the two lines and the loop follows the new chain instead of the old one. It loses every entry after the first, and ends in a cycle or a crash. The compiler accepts both orders. Nothing but your reading of the code tells them apart.

Inserting has three allocations that can each fail, and the cleanup must free exactly the ones that succeeded. The code calls free on every pointer, because free(NULL) does nothing. That rule makes the failure path three lines instead of a tree of conditions:

/* Store a copy of val under a copy of key, replacing any old value. */
static bool kv_store_set(kv_store *s, const char *key, size_t klen, const char *val,
                         size_t vlen) {
    char *v = kv_bytes_dup(val, vlen);
    if (v == NULL) {
        return false;
    }
    kv_entry *e = kv_store_find(s, key, klen);
    if (e != NULL) {
        free(e->val);
        e->val = v;
        e->vlen = vlen;
        return true;
    }
    if (s->len >= s->cap && !kv_store_grow(s)) {
        free(v);
        return false;
    }
    e = malloc(sizeof *e);
    char *k = kv_bytes_dup(key, klen);
    if (e == NULL || k == NULL) {
        free(e);
        free(k);
        free(v);
        return false;
    }
    size_t i = kv_hash(key, klen) & (s->cap - 1);
    *e = (kv_entry){k, klen, v, vlen, s->buckets[i]};
    s->buckets[i] = e;
    s->len++;
    return true;
}
c/ch06/kv.c · region set
Exercise 06.1CUnlink an entry

Write kv_store_del. Walk the chain with a pointer to a pointer. Then removing the first entry of a chain and removing a middle one are the same operation. When the key matches, make the previous link skip the entry, free its key, its value and the entry itself, and reduce the count.

static bool kv_store_del(kv_store *s, const char *key, size_t klen) {
    if (s->cap == 0) return false;
    size_t i = kv_hash(key, klen) & (s->cap - 1);
    for (kv_entry **link = &s->buckets[i]; *link != NULL; link = &(*link)->next) {
        /* your code here */
    }
    return false;
}
Stuck?Show the full listing
/* Remove a key. The chain is walked with a pointer to the pointer that leads to each
 * entry, so that the head of the chain and the middle of it are the same case. */
static bool kv_store_del(kv_store *s, const char *key, size_t klen) {
    if (s->cap == 0) {
        return false;
    }
    size_t i = kv_hash(key, klen) & (s->cap - 1);
    for (kv_entry **link = &s->buckets[i]; *link != NULL; link = &(*link)->next) {
        kv_entry *e = *link;
        if (kv_bytes_eq(e->key, e->klen, key, klen)) {
            *link = e->next;
            free(e->key);
            free(e->val);
            free(e);
            s->len--;
            return true;
        }
    }
    return false;
}

static void kv_store_free(kv_store *s) {
    for (size_t i = 0; i < s->cap; i++) {
        kv_entry *e = s->buckets[i];
        while (e != NULL) {
            kv_entry *next = e->next;
            free(e->key);
            free(e->val);
            free(e);
            e = next;
        }
    }
    free(s->buckets);
    s->buckets = NULL;
    s->cap = s->len = 0;
}
c/ch06/kv.c · region del

The pointer-to-pointer walk is an old C idiom, and it is worth a moment. link points at whichever pointer currently leads to e. At the head of the chain that is the bucket itself. In the middle it is the next field of the entry before. Writing *link = e->next removes e in both cases without a special case for the head. Linus Torvalds has said that code which special-cases the head tells him "this person doesn't understand pointers". The version without the special case is also harder to read the first time.

KEYS walks every bucket and every chain. It needs a pattern matcher, and the one here supports * and ?. The Go and Rust files hold the same function in their glob regions. The reply is an array whose length is known only after the walk, so the keys are collected in a second buffer first:

/* KEYS pattern: every matching key, as an array. The count is known only after the
 * walk, so the keys are collected first and the header is written in front of them. */
static bool kv_cmd_keys(const kv_cmd *c, kv_store *s, kv_buf *out) {
    kv_buf keys = {0};
    size_t n = 0;
    for (size_t i = 0; i < s->cap; i++) {
        for (kv_entry *e = s->buckets[i]; e != NULL; e = e->next) {
            if (kv_glob(c->argv[1], c->arglen[1], e->key, e->klen)) {
                if (!kv_reply_bulk(&keys, e->key, e->klen)) {
                    kv_buf_free(&keys);
                    return false;
                }
                n++;
            }
        }
    }
    bool ok = kv_reply_array(out, n) && kv_buf_append(out, keys.data, keys.len);
    kv_buf_free(&keys);
    return ok;
}
c/ch06/kv.c · region keys
โœ…Every malloc has one free you can point to C

Open kv.c and find each malloc and calloc. For each one, find the free that releases it. The value of an entry is freed in kv_store_set on replacement. The key and the value are freed in kv_store_del and in kv_store_free. The bucket array is freed in kv_store_grow and in kv_store_free. If you cannot find the pair for an allocation, that is a leak, and chapter 12 shows how Valgrind finds it for you.

Where a value lives, and for how long#

Every object in C has a storage duration, and the table uses all three. A local variable such as i in kv_store_grow has automatic duration. It lives on the stack from its declaration to the end of its block, and its address must not be kept after that. A static variable, and anything declared outside a function, lives for the whole run, in memory the loader set up before main. And everything malloc returns has allocated duration. It lives on the heap until free, whatever block or function ends in between. The entries must outlive the function that made them, so they are allocated. The bucket index is needed for a few lines, so it is a local. Returning the address of a local is the mistake the three durations make possible, and the compiler warns about the plain form of it.

What C does instead of generics#

The table stores bytes under bytes, and nothing else. A table of integers under strings would be a second copy of the file with the types changed. Go and Rust write the table once with a type parameter, as the next two sections show. C has three ways to avoid the copy, each with a cost, and the file generic.c in the chapter's snapshot runs all three. make test builds it, and every claim in it is an assert. The first way is a void * and an element size. The function moves bytes without knowing what they are, and the caller casts them back:

/* 1. A void pointer and an element size. The caller says how big one element is, and the
 * function moves bytes without knowing what they are. qsort and bsearch work this way.
 * The compiler checks nothing: pass the wrong size and the sort is silently wrong. */
static int cmp_u32(const void *a, const void *b) {
    uint32_t x = *(const uint32_t *)a;
    uint32_t y = *(const uint32_t *)b;
    return (x > y) - (x < y);
}

static void sort_demo(void) {
    uint32_t xs[] = {30, 10, 20};
    qsort(xs, 3, sizeof xs[0], cmp_u32);
    assert(xs[0] == 10 && xs[1] == 20 && xs[2] == 30);
}
c/ch06/generic.c · region void

The cast (const uint32_t *)a is a claim the compiler cannot check. Pass qsort a size of 2 and it sorts garbage in silence. The second way is a macro that writes the typed code for each type. The macro is the template, and each use is a copy with T filled in. ## pastes two tokens into one name, so name##_push becomes vec_u32_push:

/* 2. A macro that writes a typed container for each element type. The macro is the
 * template, and each use is a copy with T filled in. A Rust generic compiles the same
 * way, one copy per type, with the difference that Rust checks the template once. */
#define DEFINE_VEC(name, T)                                                           \
    typedef struct {                                                                  \
        T *data; /* owned */                                                          \
        size_t len, cap;                                                              \
    } name;                                                                           \
    static bool name##_push(name *v, T x) {                                           \
        if (v->len == v->cap) {                                                       \
            size_t cap = v->cap ? v->cap * 2 : 8;                                     \
            T *p = realloc(v->data, cap * sizeof *p);                                 \
            if (p == NULL) {                                                          \
                return false;                                                         \
            }                                                                         \
            v->data = p;                                                              \
            v->cap = cap;                                                             \
        }                                                                             \
        v->data[v->len++] = x;                                                        \
        return true;                                                                  \
    }

DEFINE_VEC(vec_u32, uint32_t)
DEFINE_VEC(vec_str, const char *)

static void vec_demo(void) {
    vec_u32 nums = {0};
    vec_str words = {0};
    for (uint32_t i = 0; i < 100; i++) {
        assert(vec_u32_push(&nums, i * i));
    }
    assert(vec_str_push(&words, "hello") && vec_str_push(&words, "world"));
    assert(nums.len == 100 && nums.data[9] == 81);
    assert(strcmp(words.data[1], "world") == 0);
    free(nums.data);
    free(words.data);
}
c/ch06/generic.c · region macro

That is also how a Rust generic is compiled, one copy per type. The difference is that Rust checks the template once. A C macro is checked only when it is used, in error messages that point at the use. The third way is _Generic, a compile-time switch on the type of an expression, which lets one name reach several functions:

/* 3. _Generic: a compile-time switch on the type of an expression. It is how C11 gives
 * one name to several functions, which is as close as C gets to overloading. */
static const char *name_u32(uint32_t x) {
    (void)x;
    return "uint32_t";
}

static const char *name_str(const char *s) {
    (void)s;
    return "const char *";
}

#define type_name(x) _Generic((x), uint32_t: name_u32, const char *: name_str, char *: name_str)(x)

static void generic_demo(void) {
    uint32_t n = 7;
    const char *s = "seven";
    assert(strcmp(type_name(n), "uint32_t") == 0);
    assert(strcmp(type_name(s), "const char *") == 0);
}
c/ch06/generic.c · region generic

Redis uses the first way and the second. Its dictionary stores void * keys and values, with a table of function pointers that say how to hash, compare and free them. Chapter 7 returns to that table, because it is also how C does an interface.

What a struct is made of, and an arena#

A kv_entry is 40 bytes on this machine, five fields of 8. Change the order of fields in a struct, or mix sizes, and the compiler inserts padding. Every type has an alignment, and a member must start at a multiple of its own. offsetof reads where a member starts, and alignof reads the alignment. _Static_assert checks a claim about either when the file is compiled, so a wrong one never runs:

/* What a struct is made of. Every type has an alignment, and the compiler inserts padding
 * so that each member starts at a multiple of its own alignment. One byte of tag before
 * an eight-byte number costs seven bytes of padding, and reordering the members saves it.
 * offsetof reads the layout, and _Static_assert checks it when the program is compiled. */
typedef struct {
    uint8_t tag;
    uint64_t n;
    uint8_t flag;
} loose;

typedef struct {
    uint64_t n;
    uint8_t tag;
    uint8_t flag;
} tight;

_Static_assert(sizeof(loose) == 24, "two bytes of tags cost 14 bytes of padding");
_Static_assert(sizeof(tight) == 16, "the same members, reordered, need 6");
_Static_assert(offsetof(loose, n) == 8, "n starts at the next multiple of 8");
_Static_assert(alignof(uint64_t) == 8, "a 64-bit integer is aligned to 8 bytes");

static void layout_demo(void) {
    printf("loose %zu bytes, tight %zu bytes, char is %s\n", sizeof(loose), sizeof(tight),
           CHAR_MIN < 0 ? "signed" : "unsigned");
}
c/ch06/generic.c · region layout

Chapter 15 needs these numbers, because a struct that crosses into Go or Rust must have the same layout on both sides. The last demonstration is an arena, the other way to own memory in C. One block is given out from the front, each piece rounded up to the largest alignment, and the whole block is freed at once. A parser that builds a tree for one request frees a thousand things with one call. So does a game that throws away a frame, and no piece leaks:

/* An arena: one block, handed out from the front, freed all at once. Each piece starts
 * at a multiple of the largest alignment, so any type may live in it. A parser that
 * builds a tree per request, or a game that frees everything per frame, uses one. */
typedef struct {
    char *base; /* owned */
    size_t used;
    size_t cap;
} arena;

static void *arena_alloc(arena *a, size_t n) {
    size_t align = alignof(max_align_t);
    size_t start = (a->used + align - 1) / align * align; /* round up */
    if (start + n > a->cap) {
        return NULL;
    }
    a->used = start + n;
    return a->base + start;
}

static void arena_demo(void) {
    arena a = {malloc(4096), 0, 4096};
    assert(a.base != NULL);
    char *name = arena_alloc(&a, 5);
    uint64_t *count = arena_alloc(&a, sizeof *count);
    memcpy(name, "kv!!", 5);
    *count = 42;
    assert((uintptr_t)count % alignof(uint64_t) == 0); /* padded past the 5 bytes */
    assert(a.used == 16 + sizeof *count);
    free(a.base); /* one free for every piece */
}
c/ch06/generic.c · region arena
$ make test
loose 24 bytes, tight 16 bytes, char is signed
generic: all checks passed

Go: the same table, with nothing to free#

The Go version keeps the shape of the C one and drops the bookkeeping. A chain is a slice, so there is no next pointer. The entry has two fields, and the types of both are type parameters. K comparable says the key may be any type that supports ==, which is what a lookup needs. V any says the value may be anything at all. The table is written once, and the compiler produces the code for each pair of types a program uses. The hash comes from the caller as a function, because comparable promises equality and nothing about bytes:

// fnv1a hashes bytes the way kv does in C, so the two tables spread keys the same way.
func fnv1a(b []byte) uint64 {
	h := uint64(14695981039346656037)
	for _, c := range b {
		h ^= uint64(c)
		h *= 1099511628211
	}
	return h
}
go/ch06/table.go · region hash
// entry is one key and its value in a chain. There is no next pointer: a chain is a
// slice, and the garbage collector owns everything. K and V are type parameters: the
// table is written once, for any key that supports == and any value at all.
type entry[K comparable, V any] struct {
	key K
	val V
}

// Table is a hash table with separate chaining. It grows when it holds more entries
// than buckets, so a chain stays short. The hash function comes from the caller, because
// the constraint comparable promises == and nothing else.
type Table[K comparable, V any] struct {
	hash    func(K) uint64
	buckets [][]entry[K, V]
	n       int
}

func NewTable[K comparable, V any](hash func(K) uint64) *Table[K, V] {
	return &Table[K, V]{hash: hash}
}

func (t *Table[K, V]) Get(key K) (V, bool) {
	if len(t.buckets) == 0 {
		var zero V
		return zero, false
	}
	chain := t.buckets[t.hash(key)&uint64(len(t.buckets)-1)]
	for i := range chain {
		if chain[i].key == key {
			return chain[i].val, true
		}
	}
	var zero V
	return zero, false
}
go/ch06/table.go · region table

A constraint is an interface used as a bound. comparable and any are built in. cmp.Ordered covers the types with a less-than, and a union such as ~int | ~string names a set of types. Go had no generics until version 1.18 in 2022, thirteen years into its life, and the design kept them small on purpose. There are no generic methods, and a type parameter cannot be used as a receiver. The question of when to use them has a short answer. A container or an algorithm over several types, such as this table, is a generic. A function that takes behaviour is an interface, which chapter 7 covers.

Exercise 06.2GoGrow the table

Write the grow method. Make a slice of buckets twice as long. Walk every old chain, and append each entry to its new chain. Use fnv1a and the new length to find the index. Then replace t.buckets. Nothing is freed, because nothing in Go is freed by you.

func (t *table) grow() {
	buckets := make([][]entry, 2*len(t.buckets))
	// your code here
	t.buckets = buckets
}
Stuck?Show the full listing
func (t *Table[K, V]) Set(key K, val V) {
	if len(t.buckets) == 0 {
		t.buckets = make([][]entry[K, V], 16)
	}
	i := t.hash(key) & uint64(len(t.buckets)-1)
	chain := t.buckets[i]
	for j := range chain {
		if chain[j].key == key {
			chain[j].val = val
			return
		}
	}
	if t.n >= len(t.buckets) {
		t.grow()
		i = t.hash(key) & uint64(len(t.buckets)-1)
	}
	t.buckets[i] = append(t.buckets[i], entry[K, V]{key, val})
	t.n++
}

// grow doubles the bucket slice and moves every entry to its new chain. The old slices
// become garbage, and the collector frees them.
func (t *Table[K, V]) grow() {
	buckets := make([][]entry[K, V], 2*len(t.buckets))
	for _, chain := range t.buckets {
		for _, e := range chain {
			i := t.hash(e.key) & uint64(len(buckets)-1)
			buckets[i] = append(buckets[i], e)
		}
	}
	t.buckets = buckets
}
go/ch06/table.go · region set

The old slices become garbage when t.buckets is replaced, and the collector frees them at some later moment. Deleting moves the last entry of a chain into the hole, as the C list did in chapter 5. The removed entry is garbage too:

func (t *Table[K, V]) Del(key K) bool {
	if len(t.buckets) == 0 {
		return false
	}
	i := t.hash(key) & uint64(len(t.buckets)-1)
	chain := t.buckets[i]
	for j := range chain {
		if chain[j].key == key {
			// move the last entry into the hole, then shorten the chain by one
			chain[j] = chain[len(chain)-1]
			t.buckets[i] = chain[:len(chain)-1]
			t.n--
			return true
		}
	}
	return false
}

// Len is the number of entries, and Keys is every key in no particular order.
func (t *Table[K, V]) Len() int { return t.n }

func (t *Table[K, V]) Keys() []K {
	keys := make([]K, 0, t.n)
	for _, chain := range t.buckets {
		for _, e := range chain {
			keys = append(keys, e.key)
		}
	}
	return keys
}
go/ch06/table.go · region del

The table is never used by the server. It is checked by a test, which runs five thousand random operations against a built-in map and stops at the first disagreement. The test is in table_test.go, and go test finds it by the file name and the function name:

// TestTableAgainstMap runs random operations on the table and on a map, and checks
// after each one that the table agrees with the map. The table is instantiated with
// string keys and string values; the compiler writes that version of the code.
func TestTableAgainstMap(t *testing.T) {
	tb := NewTable[string, string](hashString)
	ref := map[string]string{}
	r := rand.New(rand.NewPCG(1, 2))
	for step := range 5000 {
		key := fmt.Sprintf("k%d", r.IntN(200))
		switch r.IntN(3) {
		case 0:
			val := fmt.Sprintf("v%d", step)
			tb.Set(key, val)
			ref[key] = val
		case 1:
			_, want := ref[key]
			if got := tb.Del(key); got != want {
				t.Fatalf("step %d: del %q = %v, want %v", step, key, got, want)
			}
			delete(ref, key)
		case 2:
			want, ok := ref[key]
			got, found := tb.Get(key)
			if found != ok || got != want {
				t.Fatalf("step %d: get %q = %q,%v, want %q,%v", step, key, got, found, want, ok)
			}
		}
		if tb.Len() != len(ref) {
			t.Fatalf("step %d: table has %d entries, map has %d", step, tb.Len(), len(ref))
		}
	}
	// the generic slices and maps packages: collect the map's keys, sort both, compare
	want := slices.Sorted(maps.Keys(ref))
	got := tb.Keys()
	slices.Sort(got)
	if !slices.Equal(got, want) {
		t.Fatalf("keys differ: %v vs %v", got, want)
	}
}

// TestTableWithIntKeys is the same table with another key type. Nothing in table.go
// changed; the compiler made a second copy for int keys.
func TestTableWithIntKeys(t *testing.T) {
	tb := NewTable[int, []byte](func(k int) uint64 { return uint64(k) * 11400714819323198485 })
	for i := range 100 {
		tb.Set(i, []byte{byte(i)})
	}
	if v, ok := tb.Get(42); !ok || v[0] != 42 {
		t.Fatalf("get 42 = %v, %v", v, ok)
	}
	if tb.Del(42) != true || tb.Len() != 99 {
		t.Fatal("del")
	}
}
go/ch06/table_test.go · region test
$ go test
PASS
ok  	kv	0.425s
$ go test -bench Get -benchmem
BenchmarkTableGet-12    	70199774	        17.49 ns/op	       0 B/op	       0 allocs/op
BenchmarkMapGet-12      	62301384	        20.14 ns/op	       0 B/op	       0 allocs/op

The test ends with the generic standard packages. maps.Keys walks the reference map, and slices.Sorted orders it. slices.Sort orders the table's list and slices.Equal compares both lists. Before 1.18 each of those was a loop you wrote. The second command runs the two benchmarks at the bottom of the test file. On the author's machine the hand-written table answers a lookup in 17 nanoseconds and the built-in map in 20. That is not a reason to use your own. The built-in map grows without a pause, and is what every Go programmer reads without thinking. The store keeps the map, and adds the two new commands:

// store keeps every key and its value in a map. The hand-written table in table.go
// would work too, and the tests show it does, but the map is faster and nobody would
// use anything else in Go.
type store struct {
	m map[string][]byte
}

func newStore() *store {
	return &store{m: make(map[string][]byte)}
}

func (s *store) get(key []byte) ([]byte, bool) {
	v, ok := s.m[string(key)]
	return v, ok
}

func (s *store) set(key, val []byte) {
	s.m[string(key)] = val
}

func (s *store) del(key []byte) bool {
	_, ok := s.m[string(key)]
	delete(s.m, string(key))
	return ok
}

func (s *store) size() int {
	return len(s.m)
}

// keys returns every key that matches the pattern, in no particular order.
func (s *store) keys(pattern []byte) [][]byte {
	var out [][]byte
	for k := range s.m {
		if glob(pattern, []byte(k)) {
			out = append(out, []byte(k))
		}
	}
	return out
}
go/ch06/store.go · region store

Where the compiler puts a value#

Nothing in the Go table says stack or heap, and the compiler decides. A value that no pointer outlives stays on the stack of its function, which is free. A value that a pointer could reach after the function returns moves to the heap, where the collector must find it later. The decision is called escape analysis, and the compiler will show it:

$ go build -gcflags=-m . 2>&1 | grep -E 'table.go|store.go' | head -4
./table.go:8:6: can inline fnv1a
./table.go:8:12: b does not escape
./store.go:30:21: key does not escape
./store.go:31:22: string(key) does not escape

b does not escape means the slice passed to fnv1a never leaves it, so no copy is made. string(key) does not escape in del is the compiler noticing something. The string made from the key is only used to look in the map, so it is built on the stack and never allocated. In set the same conversion escapes, because the map keeps it. This is the tool to open when a Go program allocates more than you expect, and chapter 13 does.

โš ๏ธA chain slice may share memory with its old self Go

append may or may not allocate. When a chain has room, the new entry goes into the existing array, and any other slice that pointed at that array sees it. In del, the shortened slice chain[:len(chain)-1] still points at the same array as before. The slot past the new end still holds a copy of the entry that was moved. Its key and value stay reachable until the next append writes over them. None of this is a bug here. It is the reason that a Go programmer must know what a slice is. A slice is a pointer, a length and a capacity, and not an array.

Rust: the table that drops itself#

In Rust the chain is a Vec<Entry>, and the entry owns its key and value. Dropping the table drops the outer Vec, which drops every chain, which drops every entry, which frees every byte. No function in table.rs frees anything, and nothing leaks. The table is generic too, in two type parameters. The where clause on the impl block is the bound. It says what the table needs from a key, a view of it as bytes to hash and equality to find it. AsRef<[u8]> and PartialEq are traits, which chapter 7 explains. A key type that has both compiles. One that lacks either is refused at the impl, not deep inside a method:

/// FNV-1a, the same hash the C version of kv uses.
fn fnv1a(bytes: &[u8]) -> u64 {
    bytes.iter().fold(0xcbf2_9ce4_8422_2325, |h, &b| {
        (h ^ u64::from(b)).wrapping_mul(0x0100_0000_01b3)
    })
}
rust/ch06/src/table.rs · region hash
/// One key and its value in a chain. The chain is a Vec, which owns its entries, and
/// the entries own their bytes. Dropping the table drops all of it. `K` and `V` are type
/// parameters: the table is written once for any key that can be seen as bytes and
/// compared, and any value at all.
struct Entry<K, V> {
    key: K,
    val: V,
}

/// A hash table with separate chaining. It grows when it holds more entries than
/// buckets, so a chain stays short.
pub struct Table<K, V> {
    buckets: Vec<Vec<Entry<K, V>>>,
    len: usize,
}

/// An empty table, for every `K` and `V`. `derive(Default)` would also demand that `K`
/// and `V` have defaults, which an empty table does not need, so this is written out.
impl<K, V> Default for Table<K, V> {
    fn default() -> Self {
        Self {
            buckets: Vec::new(),
            len: 0,
        }
    }
}

/// The bounds say what the table needs from a key: a view of it as bytes, to hash, and
/// equality, to find it. A key type with more is fine; one with less does not compile.
impl<K, V> Table<K, V>
where
    K: AsRef<[u8]> + PartialEq,
{
    fn index(&self, key: &K) -> usize {
        (fnv1a(key.as_ref()) & (self.buckets.len() as u64 - 1)) as usize
    }

    pub fn get(&self, key: &K) -> Option<&V> {
        if self.buckets.is_empty() {
            return None;
        }
        let chain = &self.buckets[self.index(key)];
        chain.iter().find(|e| e.key == *key).map(|e| &e.val)
    }
rust/ch06/src/table.rs · region table

Two shapes in this file have appeared since chapter 4 without a name. The hash is a fold, an iterator method that walks the bytes and carries a value along. get is iter().find(...).map(...), a chain where each call wraps the one before and nothing runs until the end asks for a value. And the |e| e.key == *key inside find is a closure, a function written in place that captures key from the method around it. The C loop with an index, the Go for range and this chain do the same work. Chapter 8 gives iterators and closures a section of their own, because SCAN needs both.

The compiler produces one copy of every method for each pair of types a program uses, which is called monomorphisation. The copy for byte keys and the copy for &str keys share no code at run time. So a generic costs nothing to call and something to compile. The Default block is written out rather than derived. The derive would also demand a default for the key and the value types, which an empty table does not need.

Exercise 06.3RustInsert, and grow when full

Write set. If the buckets are empty, create sixteen. Find the chain by index. If an entry with the key exists, replace its value. Otherwise grow when the table is full, compute the index again, because the length changed, and push a new entry. The compiler will tell you if you hold a borrow of a chain across the call to grow.

pub fn set(&mut self, key: Vec<u8>, val: Vec<u8>) {
    if self.buckets.is_empty() {
        self.buckets.resize_with(16, Vec::new);
    }
    // your code here
}
Stuck?Show the full listing
pub fn set(&mut self, key: K, val: V) {
    if self.buckets.is_empty() {
        self.buckets.resize_with(16, Vec::new);
    }
    let i = self.index(&key);
    if let Some(e) = self.buckets[i].iter_mut().find(|e| e.key == key) {
        e.val = val;
        return;
    }
    if self.len >= self.buckets.len() {
        self.grow();
    }
    let i = self.index(&key);
    self.buckets[i].push(Entry { key, val });
    self.len += 1;
}

/// Double the buckets and move every entry to its new chain. The entries are moved,
/// not copied: `drain` takes them out of the old chains.
fn grow(&mut self) {
    let mut buckets: Vec<Vec<Entry<K, V>>> = Vec::new();
    buckets.resize_with(self.buckets.len() * 2, Vec::new);
    let mask = buckets.len() as u64 - 1;
    for chain in &mut self.buckets {
        for e in chain.drain(..) {
            let i = (fnv1a(e.key.as_ref()) & mask) as usize;
            buckets[i].push(e);
        }
    }
    self.buckets = buckets;
}
rust/ch06/src/table.rs · region set

Look at grow. The call chain.drain(..) moves every entry out of the old chain and into the new one. There is no copy of a key or a value, as in C, and the old chains are empty when the loop ends. The last line replaces self.buckets, and the old outer Vec is dropped at that moment, not later. Deleting uses swap_remove, which does what the Go version did and drops the removed entry on the same line:

    pub fn del(&mut self, key: &K) -> bool {
        if self.buckets.is_empty() {
            return false;
        }
        let i = self.index(key);
        let chain = &mut self.buckets[i];
        match chain.iter().position(|e| e.key == *key) {
            Some(j) => {
                // move the last entry into the hole; the removed entry is dropped here
                chain.swap_remove(j);
                self.len -= 1;
                true
            }
            None => false,
        }
    }

    pub fn len(&self) -> usize {
        self.len
    }

    pub fn is_empty(&self) -> bool {
        self.len == 0
    }
}
rust/ch06/src/table.rs · region del

The test lives in the same file, in a module marked #[cfg(test)], which the compiler leaves out of the real binary:

#[cfg(test)]
mod tests {
    use super::Table;
    use std::collections::HashMap;

    /// A small deterministic random number generator, so the test is the same every run.
    fn next(state: &mut u64) -> u64 {
        *state ^= *state << 13;
        *state ^= *state >> 7;
        *state ^= *state << 17;
        *state
    }

    #[test]
    fn agrees_with_hashmap() {
        let mut table: Table<Vec<u8>, Vec<u8>> = Table::default();
        let mut reference: HashMap<Vec<u8>, Vec<u8>> = HashMap::new();
        let mut rng = 0x2545_f491_4f6c_dd1d_u64;
        for step in 0..5000 {
            let key = format!("k{}", next(&mut rng) % 200).into_bytes();
            match next(&mut rng) % 3 {
                0 => {
                    let val = format!("v{step}").into_bytes();
                    table.set(key.clone(), val.clone());
                    reference.insert(key, val);
                }
                1 => {
                    assert_eq!(
                        table.del(&key),
                        reference.remove(&key).is_some(),
                        "step {step}"
                    );
                }
                _ => {
                    assert_eq!(table.get(&key), reference.get(&key));
                }
            }
            assert_eq!(table.len(), reference.len());
        }
    }

    /// The same table with another key type. `&str` is bytes with `as_ref`, and a
    /// `u32` value needs nothing. The compiler made a second copy of every method.
    #[test]
    fn works_with_str_keys_and_integer_values() {
        let mut table: Table<&str, u32> = Table::default();
        for (i, word) in ["one", "two", "three"].iter().enumerate() {
            table.set(word, i as u32);
        }
        assert_eq!(table.get(&"two"), Some(&1));
        assert!(table.del(&"two"));
        assert_eq!(table.len(), 2);
    }
rust/ch06/src/table.rs · region test
$ cargo test
running 3 tests
test table::tests::agrees_with_hashmap ... ok
test table::tests::works_with_str_keys_and_integer_values ... ok
test table::tests::two_owners_with_rc_and_refcell ... ok

Two owners of one table#

Every value in Rust has one owner, and the table so far has had one. A program sometimes needs two parts to hold one thing, and no part is the owner. The standard library has two types for that in one thread. Rc is a reference count. Cloning an Rc makes a second owner and adds one to the count, and the value is dropped when the count reaches zero. RefCell moves the borrow rules from compile time to run time. borrow and borrow_mut give a shared or a mutable reference, and a second mutable borrow while one is live is a panic. The rule did not go away. It moved to where the compiler could not see it. This is called interior mutability. Cell is the same idea for a small copyable value, with a getter and a setter instead of references. The third test shows all three:

/// Two owners of one table, in one thread. `Rc` counts the owners and frees the table
/// when the last one goes; `RefCell` moves the borrow rules to run time, so each
/// owner can borrow it mutably in turn. A second mutable borrow at the same moment
/// is a panic, not a compile error: the check moved, it did not go away.
#[test]
fn two_owners_with_rc_and_refcell() {
    use std::cell::{Cell, RefCell};
    use std::rc::Rc;

    let shared: Rc<RefCell<Table<&str, u32>>> = Rc::new(RefCell::new(Table::default()));
    let writer = Rc::clone(&shared);
    let reader = Rc::clone(&shared);
    assert_eq!(Rc::strong_count(&shared), 3);

    writer.borrow_mut().set("hits", 1);
    assert_eq!(reader.borrow().get(&"hits"), Some(&1));
    drop(writer);
    assert_eq!(Rc::strong_count(&shared), 2);

    // Cell is the same idea for a Copy value: change it through a shared reference
    let counter = Cell::new(0);
    let bump = |c: &Cell<u32>| c.set(c.get() + 1);
    bump(&counter);
    bump(&counter);
    assert_eq!(counter.get(), 2);

    // the run-time check: a borrow_mut while a borrow is live
    let guard = shared.borrow();
    assert!(shared.try_borrow_mut().is_err());
    drop(guard);
    assert!(shared.try_borrow_mut().is_ok());
}
rust/ch06/src/table.rs · region shared

Both types refuse to cross a thread. Chapter 9 replaces them with Arc and Mutex, which are the same two ideas with a lock. A chain written as a linked list, the way the C table does it, would need one more type, Box. A struct cannot contain itself without a pointer, and chapter 8 writes one.

As in Go, the server keeps the standard HashMap. The standard map uses a hash called SipHash 1-3, which the documentation describes as "selected to provide resistance against HashDoS attacks". An attacker who knows the hash function can send thousands of keys that land in one bucket. Every lookup of such a key then walks that one chain, which holds every key the attacker sent. FNV-1a has no defence against that. Redis uses SipHash too, and seeds it with random bytes at startup, for the same reason.

๐Ÿ“ฐHash flooding, 2011

On 28 December 2011 one HTTP request could hold a web server at full CPU for hours. Alexander Klink and Julian Wรคlde showed where the hole was. Nearly every web platform kept its form parameters in a hash table with a known hash and no secret. A POST with tens of thousands of parameter names chosen to collide turned every insert into a walk of one long chain. The advisory listed Java, PHP, Python, Ruby, the V8 JavaScript engine and the servers built on them. The fix across the industry was a hash with a secret seed, and SipHash was designed in 2012 for that purpose. Your FNV-1a table has exactly this hole. The standard maps of Go and Rust, and Redis, do not.

โš ๏ธThe dead code warning is a feature Rust

A module nobody calls produces a warning for every function in it. The binary never calls Table, so the compiler reports each method as dead code, and the book's clippy -D warnings turns that into an error. The file opens with #![allow(dead_code)] and a comment that says why. In C, an unused function that is not static compiles in silence, because the compiler cannot know whether another file calls it. A static one gets a warning under -Wall, which the book's -Werror turns into an error.

Memory, three ways#

The chapter touched memory in every function. The table shows what each language asked of you.

The questionCGoRust
Who frees an entryyou, in three places that must agreethe collector, when nothing refers to itthe owner, at the end of its scope
Forgetting to freea leak that grows until the process diesimpossible for unreachable memoryimpossible, the compiler inserts the drop
Freeing twiceundefined behaviour, often a security holeimpossibledoes not compile
Using after freeundefined behaviour, often a security holeimpossible, the reference keeps it alivedoes not compile
Keeping a pointer across a moveyour responsibility, and grow relies on itsafe, the entry is reachable from the new slicechecked, and set recomputes the index after grow
The costyour attentionmemory held until the next collection, and a pausethe time to satisfy the borrow checker

The C table is about 140 lines, and about a third of them exist to allocate, check and free. The Go table is 75 lines with none of that. The price is a collector that runs in the background and holds freed memory until it does. The Rust table is also 75 lines, and it frees memory at the moment the C one does. The price is a compiler that refused several early drafts of set. Each table is the right answer for its language, and the differences between them are the differences between the languages.

Verdict##

TopicCGoRustWhy
Writing the tableWeakStrongOKC needs an owner comment on every pointer and three cleanup paths, Go is the shortest, and Rust argues about borrows in set.
Freeing memoryWeakStrongStrongC must free in three places that agree. Go and Rust free for you, in different ways.
Freeing on timeStrongWeakStrongC and Rust free on the line that removes an entry. Go frees at the next collection.
Testing itWeakStrongStrongC has no test runner in the box. go test and cargo test are built in.
Shipping unused codeWeakOKStrongC warns only for static functions and Go ignores unused functions, while Rust reports each one.
Needing to write oneStrongWeakWeakIn C the table is the store. In Go and Rust it is an exercise, because the map is built in.
Writing it once for any typeWeakOKStrongC has void pointers, macros and _Generic, each with a hole. Go has type parameters since 2022, and Rust has bounds the compiler checks at the impl.
Seeing where a value livesStrongOKStrongC says it in every line, and Rust says it in the types. Go decides for you and shows its decision with a flag.
Carry forward
  • Storage duration. A C value lives on the stack to the end of its block, in static memory for the run, or on the heap until freed.
  • Type parameter. A table written once for any key and value, with a bound that says what the type must support.
  • Rc and RefCell. Two owners of one value in one thread, with the borrow rules checked at run time instead of compile time.
Next

The C file has passed eight hundred lines. Chapter 7 cuts it into modules, in all three languages, and the first cut does not build.

Modules, packages and dependencies#

Building kv · Chapter 07· 33 min read· 500 lines of code
▶Scene

The chapter 6 file is 868 lines. Cut it the obvious way, by moving the hash table into store.c as it stands, and compile the two files together:

$ wc -l kv.c
     868 kv.c
$ cc -std=c17 -Wall -Wextra -Werror -o kv kv.c store.c
kv.c:357:44: error: unknown type name 'kv_store'
kv.c:359:42: error: unknown type name 'kv_store'
kv.c:370:42: error: unknown type name 'kv_store'
kv.c:375:41: error: unknown type name 'kv_store'
...

Twenty-five errors. Every function that uses the store has lost the word kv_store. The file that defines it is now a different file, and C compiles one file at a time. The cut needs a third file, which says what the second one offers.

kv is nearly nine hundred lines in one file, and the next chapters double it. This chapter splits the server into parts, in each language's own way. C gets header files and a Makefile that knows about them. Go gets packages under internal. Rust gets a library crate with a thin binary. It also covers dependencies, and the tools that keep code in one style.

The server does not change in this chapter. Every test from chapter 6 still passes, and no command is added. What changes is where the code lives, how one part sees another, and how a reader finds anything. That sounds like housekeeping. It is also where the three languages differ most in their daily feel. A module system is what you touch every time you open a file.

The unitCGoRust
A modulea .c file with a .h beside ita folder, one packagea file, declared with mod
A private namestatica lower-case first letterthe default, without pub
A public namedeclared in the header, prefixed kv_an upper-case first letterpub
Using another module#include "store.h", copied in by the preprocessorimport "kv/internal/store"use kv::store::Store
Building many filesa Makefile that lists them, or finds themgo build ./cmd/kvcargo build
A dependencycopy the source in, or a system packagego get, recorded in go.modcargo add, recorded in Cargo.toml
The exact versionswherever you wrote them downgo.sumCargo.lock
The registrynoneany Git host, through a proxycrates.io
The formatterclang-format, with a config filegofmt, with no optionsrustfmt, with few
Behaviour a caller may swapa struct of function pointers, and an opaque typean interface, satisfied by having the methodsa trait, implemented with impl Store for
๐Ÿ“œWhy Go has a compatibility promise and Rust has editions

A module system is a promise about what stays the same. Go 1 was released in 2012 with a written promise. Programs written to the Go 1 specification "will continue to compile and run correctly, unchanged, over the lifetime of that specification". Fourteen years later the promise holds, and a Go package from 2012 is meant to still build. Rust made a different promise. The language can change in ways that would break old code, but only in a new edition. A crate names its edition in Cargo.toml. The 2024 edition shipped with Rust 1.85 in February 2025. A crate on the 2015 edition still builds with today's compiler, next to one on the 2024 edition, in the same program. C made no promise at all, and relies on the fact that no compiler vendor dares to break the code that already exists.

C: headers, static, and a Makefile that knows about both#

C has no modules. It has files, and a convention. A module is a .c file that holds the code and a .h file that holds the declarations other files may use. The buffer from chapter 4 becomes two files. The header is the public part:

#ifndef KV_BUF_H
#define KV_BUF_H

#include <stdbool.h>
#include <stddef.h>

/* A growable byte buffer. It owns data, and kv_buf_free releases it. Start one with
 * kv_buf b = {0}. */
typedef struct {
    char *data; /* owned; NULL until the first append */
    size_t len;
    size_t cap;
} kv_buf;

bool kv_buf_append(kv_buf *b, const void *src, size_t n);
void kv_buf_consume(kv_buf *b, size_t n);
void kv_buf_free(kv_buf *b);

#endif
c/ch07/src/buf.h

Three conventions are at work. The first lines and the last line are a header guard. The preprocessor copies a header into every file that includes it. A file that includes buf.h twice, once directly and once through resp.h, would see the typedef twice. The guard makes the second copy empty. The second convention is the prefix. C has one namespace for the whole program. So every public name in kv begins with kv_, and a name from another library cannot collide with it. The third is that the header says what the caller must do, in comments, because nothing else will. The .c file holds the rest, and marks the one function that only it uses as static:

/* Private: only this file calls it, so it is static and not in the header. */
static bool kv_buf_reserve(kv_buf *b, size_t extra) {
    if (b->len + extra <= b->cap) {
        return true;
    }
    size_t cap = b->cap ? b->cap * 2 : 1024;
    while (cap < b->len + extra) {
        cap *= 2;
    }
    char *p = realloc(b->data, cap);
    if (p == NULL) {
        return false; /* the old block is still valid and still ours */
    }
    b->data = p;
    b->cap = cap;
    return true;
}
c/ch07/src/buf.c · region static

A static function is invisible outside its file. That is the whole of information hiding in C. There is no private keyword and no module boundary, only a function the linker never hears about. Every file in src/ follows the same shape. The header of the store is the most useful one to read. It says who owns every pointer, and how long a pointer from kv_store_find stays valid:

Exercise 07.1CWrite the header

Write store.h for the hash table from chapter 6. It needs a guard, the two structures, and a declaration for each of the four public functions. Put a comment on the entry that says what it owns, and a comment on kv_store_find that says how long the returned pointer is valid. Everything that only store.c uses stays out of the header and becomes static. That includes the hash function and the grow function.

#ifndef KV_STORE_H
#define KV_STORE_H

#include <stdbool.h>
#include <stddef.h>

/* your declarations here */

#endif
Stuck?Show the full listing
#ifndef KV_STORE_H
#define KV_STORE_H

#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>

/* The keys and values, in a hash table with separate chaining. */

/* One key and its value, in a chain of entries that share a bucket. The key, the value
 * and the next entry are owned. */
typedef struct kv_entry {
    char *key;
    size_t klen;
    char *val;
    size_t vlen;
    struct kv_entry *next;
} kv_entry;

/* How the table treats a key: a struct of function pointers, which is how C describes
 * behaviour a caller may choose. A store is made with one of the types below, and every
 * hash and comparison inside it goes through that type. Redis calls the same thing a
 * dictType. */
typedef struct {
    const char *name;
    uint64_t (*hash)(const char *key, size_t klen, uint64_t seed);
    bool (*eq)(const char *a, size_t alen, const char *b, size_t blen);
    bool seeded; /* kv_store_new draws a random seed, so keys cannot be chosen to collide */
} kv_store_type;

extern const kv_store_type kv_fnv_type;    /* FNV-1a, the same hash on every run */
extern const kv_store_type kv_seeded_type; /* FNV-1a mixed with a seed drawn at start */

/* The store itself is opaque: this header says it exists, store.c says what is in it.
 * Other files hold a pointer and call the functions below, and cannot reach inside. */
typedef struct kv_store kv_store;

/* A new, empty store of the given type, or NULL when out of memory. Free it with
 * kv_store_free, which also frees every entry. */
kv_store *kv_store_new(const kv_store_type *type);
void kv_store_free(kv_store *s);

size_t kv_store_len(const kv_store *s);

/* The entry for key, or NULL. The pointer is valid until the next set or del. */
kv_entry *kv_store_find(kv_store *s, const char *key, size_t klen);

/* Store a copy of val under a copy of key, replacing any old value. */
bool kv_store_set(kv_store *s, const char *key, size_t klen, const char *val, size_t vlen);
bool kv_store_del(kv_store *s, const char *key, size_t klen);

/* Call fn once for every entry, with ctx passed through, until fn returns false. The
 * callback must not set or del: the chain it is walking would move under it. */
void kv_store_each(kv_store *s, bool (*fn)(kv_entry *e, void *ctx), void *ctx);

#endif
c/ch07/src/store.h

An opaque type, and a struct of functions#

The header above still shows the fields of kv_store. So any file that includes it can reach inside and walk the buckets, as KEYS did in chapter 6. The second half of the header takes that away. It declares the struct without defining it, and store.c holds the definition. Every other file has a pointer to something it cannot look inside. That is called an opaque type, and it is the strongest form of hiding C has. FILE from stdio.h is the one you have used all along:

/* The store itself is opaque: this header says it exists, store.c says what is in it.
 * Other files hold a pointer and call the functions below, and cannot reach inside. */
typedef struct kv_store kv_store;

/* A new, empty store of the given type, or NULL when out of memory. Free it with
 * kv_store_free, which also frees every entry. */
kv_store *kv_store_new(const kv_store_type *type);
void kv_store_free(kv_store *s);

size_t kv_store_len(const kv_store *s);

/* The entry for key, or NULL. The pointer is valid until the next set or del. */
kv_entry *kv_store_find(kv_store *s, const char *key, size_t klen);

/* Store a copy of val under a copy of key, replacing any old value. */
bool kv_store_set(kv_store *s, const char *key, size_t klen, const char *val, size_t vlen);
bool kv_store_del(kv_store *s, const char *key, size_t klen);

/* Call fn once for every entry, with ctx passed through, until fn returns false. The
 * callback must not set or del: the chain it is walking would move under it. */
void kv_store_each(kv_store *s, bool (*fn)(kv_entry *e, void *ctx), void *ctx);
c/ch07/src/store.h · region opaque

Because nobody outside can walk the table, the table walks itself. kv_store_each takes a function pointer and a void *, and calls the function once per entry with that pointer passed back. The void * is the context, whatever the caller needs the callback to see. The callback casts it back to the real type on its first line. This pair, a function pointer and a context pointer, is how C passes a closure, and KEYS now uses it:

void kv_store_each(kv_store *s, bool (*fn)(kv_entry *e, void *ctx), void *ctx) {
    for (size_t i = 0; i < s->cap; i++) {
        for (kv_entry *e = s->buckets[i]; e != NULL; e = e->next) {
            if (!fn(e, ctx)) {
                return;
            }
        }
    }
}
c/ch07/src/store.c · region each
/* What the KEYS walk needs to know, passed to the callback through a void pointer. */
typedef struct {
    const char *pat;
    size_t plen;
    kv_buf keys; /* the matching keys, as replies, collected before the count is known */
    size_t n;
    bool ok;
} kv_keys_ctx;

/* Called once per entry by kv_store_each. The context comes back as a void pointer, and
 * the first line turns it into the real type again. */
static bool kv_keys_visit(kv_entry *e, void *ctx) {
    kv_keys_ctx *k = ctx;
    if (!kv_glob(k->pat, k->plen, e->key, e->klen)) {
        return true;
    }
    if (!kv_reply_bulk(&k->keys, e->key, e->klen)) {
        k->ok = false;
        return false; /* stop the walk */
    }
    k->n++;
    return true;
}

/* KEYS pattern: every matching key, as an array. The walk is the store's; this file only
 * supplies what to do with each entry. */
static bool kv_cmd_keys(const kv_cmd *c, kv_store *s, kv_buf *out) {
    kv_keys_ctx k = {c->argv[1], c->arglen[1], {0}, 0, true};
    kv_store_each(s, kv_keys_visit, &k);
    bool ok = k.ok && kv_reply_array(out, k.n) && kv_buf_append(out, k.keys.data, k.keys.len);
    kv_buf_free(&k.keys);
    return ok;
}
c/ch07/src/cmd.c · region keys

The header also lets a caller choose how the table treats a key. A kv_store_type is a struct whose fields are function pointers, one for the hash and one for equality, and a store is made with one. Inside store.c every hash is s->type->hash(...), a call through a pointer, so the table never names a hash function. This is the C form of an interface, and Redis builds its dictionary on exactly this struct, which it calls dictType:

/* How the table treats a key: a struct of function pointers, which is how C describes
 * behaviour a caller may choose. A store is made with one of the types below, and every
 * hash and comparison inside it goes through that type. Redis calls the same thing a
 * dictType. */
typedef struct {
    const char *name;
    uint64_t (*hash)(const char *key, size_t klen, uint64_t seed);
    bool (*eq)(const char *a, size_t alen, const char *b, size_t blen);
    bool seeded; /* kv_store_new draws a random seed, so keys cannot be chosen to collide */
} kv_store_type;

extern const kv_store_type kv_fnv_type;    /* FNV-1a, the same hash on every run */
extern const kv_store_type kv_seeded_type; /* FNV-1a mixed with a seed drawn at start */
c/ch07/src/store.h · region type
/* FNV-1a: a small, fast hash of bytes. The seed is mixed into the starting value, so a
 * seeded store hashes the same key differently on every run. */
static uint64_t kv_fnv1a(const char *s, size_t n, uint64_t seed) {
    uint64_t h = 14695981039346656037ULL ^ seed;
    for (size_t i = 0; i < n; i++) {
        h ^= (unsigned char)s[i];
        h *= 1099511628211ULL;
    }
    return h;
}

const kv_store_type kv_fnv_type = {"fnv", kv_fnv1a, kv_bytes_eq, false};
const kv_store_type kv_seeded_type = {"seeded", kv_fnv1a, kv_bytes_eq, true};

/* Eight random bytes from the operating system, or the clock when that fails. */
static uint64_t kv_random_seed(void) {
    uint64_t seed = 0;
    FILE *f = fopen("/dev/urandom", "rb");
    if (f != NULL) {
        size_t got = fread(&seed, sizeof seed, 1, f);
        fclose(f);
        if (got == 1) {
            return seed;
        }
    }
    return (uint64_t)(uintptr_t)&seed * 2654435761ULL;
}
c/ch07/src/store.c · region types

The two types here differ in one field. The seeded one mixes eight random bytes into the hash when the store is made. So the chain an attacker's keys land in cannot be predicted. That is the fix for the hash flooding of chapter 6, and the server uses it by default. KV_HASH=fnv picks the plain one, for a run that must be repeatable. kv_store_new is where the choice is applied:

kv_store *kv_store_new(const kv_store_type *type) {
    kv_store *s = calloc(1, sizeof *s);
    if (s == NULL) {
        return NULL;
    }
    s->type = type;
    s->seed = type->seeded ? kv_random_seed() : 0;
    return s;
}

size_t kv_store_len(const kv_store *s) {
    return s->len;
}
c/ch07/src/store.c · region new

The main file now includes four headers and nothing else of its own. Those four lines are the C form of an import list:

#include "buf.h"
#include "cmd.h"
#include "resp.h"
#include "store.h"
c/ch07/src/main.c · region includes
/* KV_HASH=fnv asks for the unseeded hash, which makes a run repeatable. The default
 * seeds the hash, so that nobody can choose keys that all land in one chain. */
const char *hash = getenv("KV_HASH");
const kv_store_type *type = hash && strcmp(hash, "fnv") == 0 ? &kv_fnv_type : &kv_seeded_type;
kv_store *store = kv_store_new(type);
if (store == NULL) {
    perror("kv_store_new");
    close(fd);
    return 1;
}
c/ch07/src/main.c · region store-new

With six source files, the Makefile from chapter 2 is not enough. The one below finds every .c file, compiles each into an object file under build/, and links them. The -MMD flag asks the compiler to write a small file beside each object that lists the headers it included. make reads those files, so a change to store.h rebuilds exactly the files that include it:

CC       ?= cc
CFLAGS   ?= -std=c17 -Wall -Wextra -Werror -O2 -g
SANITIZE ?=

# every .c under src/ becomes a .o under build/, and kv is linked from all of them
SRC := $(wildcard src/*.c)
OBJ := $(SRC:src/%.c=build/%.o)

kv: $(OBJ)
	$(CC) $(CFLAGS) $(SANITIZE) -o $@ $(OBJ)

# -MMD writes a .d file beside each .o that lists the headers it included, so that a
# change to a header rebuilds exactly the files that use it
build/%.o: src/%.c | build
	$(CC) $(CFLAGS) $(SANITIZE) -MMD -c -o $@ $<

build:
	mkdir -p build

-include $(OBJ:.o=.d)

clean:
	rm -rf kv kv.dSYM build

.PHONY: clean
c/ch07/Makefile
โš ๏ธA header without a guard C

Include a header twice and the compiler reports a redefinition. main.c includes buf.h directly and again through cmd.h. Without the guard the second copy defines kv_buf a second time, and the error names the header, not the file that included it twice. Every header gets a guard on the day it is created. Many compilers accept #pragma once as a shorter form, and it is not in the standard.

โœ…Include your own header first C

A .c file includes its own header before any other, as buf.c does. Suppose the header forgot an include it needs, such as stddef.h for size_t. The mistake then shows in the file that owns the header. It does not hide in another file that included things in a luckier order.

Dependencies in C: the package manager that is not there#

C has no package manager, and that sentence has to be said carefully, because it has several. The operating system has one. A C program that needs a library usually says apt install libssl-dev or brew install openssl. Then it asks a tool called pkg-config where the headers went. The second way is to copy the source into your own tree, which is called vendoring. Redis does this. Its deps/ folder holds its own copies of the libraries it needs, and they build with the rest. The third way is a package manager that is not part of the language, such as vcpkg. Microsoft describes vcpkg as "a free and open-source C/C++ package manager maintained by Microsoft and the C++ community". Conan is another. It works with CMake, Make and MSBuild. Both are used. There is no default.

Each way has a cost. A system package is whatever version the operating system has. A vendored copy must be updated yourself. A third-party manager is one more thing to install before the build works. The server kv has no dependency. That is not unusual for a C program, and it is one reason C programs so often write things themselves.

Go: a folder is a package, and a capital letter is an export#

A Go package is a folder. Every file in the folder begins with the same package line and sees everything in the other files. Another package sees only the names that begin with a capital letter. The spec puts it in one sentence. An identifier is exported if "the first character of the identifier's name is a Unicode uppercase letter". There is no keyword. The protocol code moves to internal/resp, and readCommand becomes ReadCommand:

// ReadCommand reads one request: an array of bulk strings, or one inline line of words.
// It returns io.EOF when the client closed the connection between commands.
func ReadCommand(r *bufio.Reader) ([][]byte, error) {
	line, err := readLine(r)
	if err != nil {
		return nil, fmt.Errorf("read command: %w", err)
	}
	if len(line) == 0 || line[0] != '*' {
		return bytes.Fields(line), nil
	}
	n, err := strconv.Atoi(string(line[1:]))
	if err != nil || n < 0 || n > maxArgs {
		return nil, protoErr('*')
	}
	args := make([][]byte, 0, n)
	for range n {
		line, err := readLine(r)
		if err != nil {
			return nil, fmt.Errorf("read argument: %w", unexpected(err))
		}
		if len(line) == 0 || line[0] != '$' {
			return nil, protoErr(first(line))
		}
		size, err := strconv.Atoi(string(line[1:]))
		if err != nil || size < 0 || size > maxBulk {
			return nil, protoErr('$')
		}
		buf := make([]byte, size+2)
		if _, err := io.ReadFull(r, buf); err != nil {
			return nil, fmt.Errorf("read argument: %w", unexpected(err))
		}
		if buf[size] != '\r' || buf[size+1] != '\n' {
			return nil, protoErr(buf[size])
		}
		args = append(args, buf[:size])
	}
	return args, nil
}
go/ch07/internal/resp/resp.go · region exported

The folder is named internal on purpose. The Go 1.4 release notes state the rule. A package with internal in its path "can be imported only by code in the directory tree rooted at" the parent of that folder. So kv/internal/store can be used by kv/cmd/kv and by nobody outside the module. The rule is in the go tool, not in the language, and it is how a Go program says "this is not an API". The import path begins with the module path from go.mod:

// Execute runs one command against the store and writes its reply. The two imports
// above are this module's own packages: kv is the module path from go.mod.
func Execute(args [][]byte, s Storer, w *bufio.Writer) error {
go/ch07/internal/cmd/cmd.go · region imports

The store exports a type and its methods, and keeps the map itself private, so that no other package can reach into it:

// Store keeps every key and its value in a map. Exported, so that package cmd can use
// it; its field is not, so that only this package can touch the map.
type Store struct {
	m map[string][]byte
}

func New() *Store {
	return &Store{m: make(map[string][]byte)}
}

func (s *Store) Get(key []byte) ([]byte, bool) {
	v, ok := s.m[string(key)]
	return v, ok
}

func (s *Store) Set(key, val []byte) {
	s.m[string(key)] = val
}

func (s *Store) Del(key []byte) bool {
	_, ok := s.m[string(key)]
	delete(s.m, string(key))
	return ok
}

func (s *Store) Len() int {
	return len(s.m)
}

// Keys returns every key that matches the pattern, in no particular order.
func (s *Store) Keys(pattern []byte) [][]byte {
	var out [][]byte
	for k := range s.m {
		if glob(pattern, []byte(k)) {
			out = append(out, []byte(k))
		}
	}
	return out
}
go/ch07/internal/store/store.go · region exported

An interface is a set of methods#

The commands do not need the map. They need six methods, and an interface is a type that lists methods and nothing else. Any type that has those methods satisfies the interface, with no declaration that says so. The interface lives in the cmd package, next to the code that calls it. Go puts an interface on the side that uses it. The package that implements it never hears of it:

// A Storer is whatever the commands need from a store: these six methods, and nothing
// about how they are done. Any type with these methods is a Storer, without saying so.
// The interface lives here, with the code that uses it, which is where Go puts one.
type Storer interface {
	Get(key []byte) ([]byte, bool)
	Set(key, val []byte)
	Del(key []byte) bool
	Len() int
	Keys(pattern []byte) [][]byte
	Incr(key []byte) (int64, error)
}
go/ch07/internal/cmd/cmd.go · region storer

Two types satisfy it. *store.Store is the map. *store.ByteTable is the table from chapter 6, wrapped so that its methods have the right shapes. Nothing in the file names cmd.Storer, and it could not, because store is imported by cmd and a Go import cannot go both ways:

// ByteTable is the hand-written table from chapter 6, used as a store. It has the same
// methods as Store, so it satisfies the cmd.Storer interface without naming it. Nothing
// here says "implements": the methods are the whole of the contract.
type ByteTable struct {
	t *Table[string, []byte]
}

func NewByteTable() *ByteTable {
	return &ByteTable{t: NewTable[string, []byte](func(s string) uint64 { return fnv1a([]byte(s)) })}
}

func (b *ByteTable) Get(key []byte) ([]byte, bool) { return b.t.Get(string(key)) }
func (b *ByteTable) Set(key, val []byte)           { b.t.Set(string(key), val) }
func (b *ByteTable) Del(key []byte) bool           { return b.t.Del(string(key)) }
func (b *ByteTable) Len() int                      { return b.t.Len() }

func (b *ByteTable) Keys(pattern []byte) [][]byte {
	var out [][]byte
	for _, k := range b.t.Keys() {
		if glob(pattern, []byte(k)) {
			out = append(out, []byte(k))
		}
	}
	return out
}

func (b *ByteTable) Incr(key []byte) (int64, error) {
	var n int64
	if cur, ok := b.Get(key); ok {
		var err error
		n, err = strconv.ParseInt(string(cur), 10, 64)
		if err != nil {
			return 0, ErrNotInteger
		}
	}
	if n == math.MaxInt64 {
		return 0, ErrOverflow
	}
	n++
	b.Set(key, []byte(strconv.FormatInt(n, 10)))
	return n, nil
}
go/ch07/internal/store/bytetable.go · region bytetable

The main function chooses. A variable of the interface type holds one or the other, and every call through it goes to whichever is inside:

// Either store satisfies cmd.Storer. The variable's type is the interface, and the
// value inside it is whichever the environment asked for.
var s cmd.Storer = store.New()
if os.Getenv("KV_STORE") == "table" {
	s = store.NewByteTable()
}
go/ch07/cmd/kv/main.go · region choose

This is why io.Copy in chapter 3 could copy a socket, a file or a buffer with one function. It is why error in chapter 4 was a type with one method. The small interfaces of the standard library are the ones most code accepts, such as io.Reader, io.Writer and error. A type that has a Read method joins them without asking.

Embedding, and the nil that is not nil#

A struct may hold a field with no name, only a type, which is called embedding. The methods of the embedded type are promoted to the outer struct, so a wrapper can replace one method and inherit the rest. The test below wraps any Storer and counts the writes:

// counting wraps any Storer and counts the writes. The embedded field lends the wrapped
// store's methods to counting, so only Set is written here: the rest are promoted.
type counting struct {
	Storer
	sets int
}

func (c *counting) Set(key, val []byte) {
	c.sets++
	c.Storer.Set(key, val)
}

func TestEmbeddedStorerPromotesMethods(t *testing.T) {
	c := &counting{Storer: store.New()}
	var buf bytes.Buffer
	w := bufio.NewWriter(&buf)
	for _, line := range [][]string{{"SET", "a", "1"}, {"SET", "b", "2"}, {"GET", "a"}} {
		var args [][]byte
		for _, s := range line {
			args = append(args, []byte(s))
		}
		if err := Execute(args, c, w); err != nil {
			t.Fatal(err)
		}
	}
	w.Flush()
	if c.sets != 2 || c.Len() != 2 || !bytes.Contains(buf.Bytes(), []byte("$1\r\n1\r\n")) {
		t.Fatalf("sets %d, len %d, out %q", c.sets, c.Len(), buf.String())
	}
}
go/ch07/internal/cmd/cmd_test.go · region embed

An interface value is two words, a type and a pointer. That shape has a trap, and the second test walks into it on purpose:

// A nil pointer inside an interface is not a nil interface. The interface holds a type
// and a pointer; the type is set, so the interface is not nil, and a method call reaches
// the pointer receiver with nil. This is the trap every Go programmer meets once.
func TestNilPointerInInterfaceIsNotNil(t *testing.T) {
	var p *store.Store // nil
	var s Storer = p   // a Storer holding (*store.Store, nil)
	if s == nil {
		t.Fatal("an interface holding a typed nil compares unequal to nil")
	}
	var none Storer // no type, no value
	if none != nil {
		t.Fatal("an interface with no type is nil")
	}
}
go/ch07/internal/cmd/cmd_test.go · region nil
โš ๏ธA typed nil in an interface is not nil Go

A function that returns an interface, and returns a nil pointer of a concrete type, returns something that is not nil. The caller's if err != nil is true, and the method call inside it runs on a nil receiver. The rule is to return the literal nil, never a nil pointer, from a function whose result type is an interface. The test above is the whole explanation, and most Go programmers learn it from a crash.

โœ…Accept interfaces, return structs, and define one late Go

A function that takes an argument asks for the smallest interface that has what it needs. That is why io.Copy takes a Reader and not a file. A function that returns a value returns the concrete type, so the caller has every method. And an interface is defined when the second implementation exists, not before. One implementation is a struct.

Exercise 07.2GoTest the pattern matcher

The function glob is now in its own file in the store package. Write TestGlob in glob_test.go. Make a table of cases with a pattern, a key and the expected answer, and a loop that calls t.Errorf for each mistake. Include a case where * must match an empty string, and one where ? must not. Run it with go test ./..., which tests every package in the module.

func TestGlob(t *testing.T) {
	cases := []struct {
		pat, key string
		want     bool
	}{
		// your cases here
	}
	for _, c := range cases {
		// your check here
	}
}
Stuck?Show the full listing
func TestGlob(t *testing.T) {
	cases := []struct {
		pat, key string
		want     bool
	}{
		{"*", "", true},
		{"*", "anything", true},
		{"user:*", "user:1", true},
		{"user:*", "order:1", false},
		{"us?r:1", "user:1", true},
		{"us?r:1", "usr:1", false},
		{"*:2", "user:2", true},
		{"a*b*c", "aXXbYYc", true},
		{"a*b*c", "aXXbYY", false},
	}
	for _, c := range cases {
		if got := glob([]byte(c.pat), []byte(c.key)); got != c.want {
			t.Errorf("glob(%q, %q) = %v, want %v", c.pat, c.key, got, c.want)
		}
	}
}
go/ch07/internal/store/glob_test.go · region test
$ go build -o kv ./cmd/kv
$ go test ./...
?   	kv/cmd/kv	[no test files]
?   	kv/internal/cmd	[no test files]
?   	kv/internal/resp	[no test files]
ok  	kv/internal/store	0.020s

kv still has no dependency, and Go makes that a normal state. The standard library has the network, the buffered reader, the map and the test runner, and most Go programs of this size use nothing else. When a dependency is needed, go get adds a line to go.mod and a hash to go.sum. The hash is checked on every later download, against the file and against a public log that Google runs. So a changed upload cannot replace what you tested. There is no central registry. A module path is a URL, and the go tool fetches through a proxy that caches every version it has seen.

โš ๏ธAn import you do not use is an error Go

Delete the last use of a package and the build fails. The error is "strings" imported and not used. Go does this on purpose, so that a dependency graph never holds a package that nothing needs, and the build stays fast. Editors remove the line for you, and new Go programmers curse it for a week.

Rust: a module is a file you must announce#

kv already had modules in Rust. Chapter 4 wrote mod resp; and mod cmd;, and each line told the compiler to read a file of that name. The new step is to put the modules in a library. Then a test or a benchmark can use the parser and the store without starting a server. A crate with both src/lib.rs and src/main.rs builds a library and a binary of the same name:

pub mod cmd;
pub mod resp;
pub mod store;
pub mod table;
rust/ch07/src/lib.rs · region mods

The pub on each mod line makes the module visible outside the library, and the pub on each function inside makes the function visible. Without either, the item is private to the file. The binary uses the library through its name:

use kv::store::{MapStore, Store};
use kv::table::Table;
use kv::{cmd, resp};
rust/ch07/src/main.rs · region use
Exercise 07.3RustMake the crate a library

Create src/lib.rs with four public mod lines. Remove the mod lines from main.rs. Replace them with use lines that name the library, such as kv::store::Store. Build, and run cargo clippy. It will tell you that Store has a public len and no is_empty. That is a convention it enforces for public types, and it never mentioned it while the type was private. Add a one-line is_empty that returns self.map.is_empty().

// src/lib.rs
// your four mod lines here

// src/main.rs
// your use lines here
Stuck?Show the full listing
//! kv as a library: the protocol, the store and the commands. The binary in main.rs is a
//! thin wrapper, and a test or a benchmark can use the crate without starting a server.

pub mod cmd;
pub mod resp;
pub mod store;
pub mod table;
rust/ch07/src/lib.rs
use kv::store::{MapStore, Store};
use kv::table::Table;
use kv::{cmd, resp};
rust/ch07/src/main.rs · region use

A trait is a promise of methods#

The Rust store is now a trait, a set of methods a type promises, and two types keep the promise. Unlike a Go interface, the promise is written down. A type says impl Store for MapStore and lists the methods, and the compiler checks that none is missing and none has the wrong shape:

/// What the commands need from a store. A trait is a set of methods a type promises, and
/// any type that writes an `impl Store for` block with these methods is a store. The
/// server is written against the trait, so the type behind it can change.
pub trait Store {
    /// A borrowed view of the value, which lives as long as the borrow of the store.
    fn get(&self, key: &[u8]) -> Option<&[u8]>;
    fn set(&mut self, key: Vec<u8>, val: Vec<u8>);
    fn del(&mut self, key: &[u8]) -> bool;
    fn len(&self) -> usize;
    /// Every key that matches the pattern, in no particular order.
    fn keys(&self, pattern: &[u8]) -> Vec<&[u8]>;

    fn is_empty(&self) -> bool {
        self.len() == 0
    }

    /// Add one to the integer stored under key, which counts as 0 when absent. A default
    /// method: written once here, in terms of `get` and `set`, and shared by every
    /// implementation unless one replaces it.
    fn incr(&mut self, key: &[u8]) -> Result<i64, IncrError> {
        let n = match self.get(key) {
            None => 0,
            Some(bytes) => parse_int(bytes).ok_or(IncrError::NotInteger)?,
        };
        let n = n.checked_add(1).ok_or(IncrError::Overflow)?;
        self.set(key.to_vec(), n.to_string().into_bytes());
        Ok(n)
    }
}
rust/ch07/src/store.rs · region trait

Two of the methods have bodies in the trait. is_empty and incr are default methods, written once in terms of the others and shared by every implementation unless one replaces them. The map store implements the five that remain:

/// The store the server uses: the standard HashMap. The hand-written table in table.rs
/// implements the same trait, and `KV_STORE=table` runs the server on it.
#[derive(Default)]
pub struct MapStore {
    map: HashMap<Vec<u8>, Vec<u8>>,
}

impl MapStore {
    pub fn new() -> Self {
        Self::default()
    }
}

impl Store for MapStore {
    fn get(&self, key: &[u8]) -> Option<&[u8]> {
        self.map.get(key).map(Vec::as_slice)
    }

    fn set(&mut self, key: Vec<u8>, val: Vec<u8>) {
        self.map.insert(key, val);
    }

    fn del(&mut self, key: &[u8]) -> bool {
        self.map.remove(key).is_some()
    }

    fn len(&self) -> usize {
        self.map.len()
    }

    fn keys(&self, pattern: &[u8]) -> Vec<&[u8]> {
        self.map
            .keys()
            .map(Vec::as_slice)
            .filter(|k| glob(pattern, k))
            .collect()
    }
}
rust/ch07/src/store.rs · region store

The table from chapter 6 implements the same five. The block lives in table.rs, next to the type. It could as well live next to the trait, because Rust allows an impl wherever the crate owns the trait or the type. That is the orphan rule. A crate may implement its own trait for any type, or any trait for its own type. It may never implement a foreign trait for a foreign type, so two crates cannot disagree about what Display means for Vec:

/// The hand-written table as a store: the same trait as MapStore, so the server can run
/// on either. The methods are those of the trait; the body of each is one line.
impl crate::store::Store for Table<Vec<u8>, Vec<u8>> {
    fn get(&self, key: &[u8]) -> Option<&[u8]> {
        Table::get(self, &key.to_vec()).map(Vec::as_slice)
    }

    fn set(&mut self, key: Vec<u8>, val: Vec<u8>) {
        Table::set(self, key, val);
    }

    fn del(&mut self, key: &[u8]) -> bool {
        Table::del(self, &key.to_vec())
    }

    fn len(&self) -> usize {
        Table::len(self)
    }

    fn keys(&self, pattern: &[u8]) -> Vec<&[u8]> {
        self.buckets
            .iter()
            .flatten()
            .map(|e| e.key.as_slice())
            .filter(|k| crate::store::glob(pattern, k))
            .collect()
    }
}
rust/ch07/src/table.rs · region impl-store

Static and dynamic dispatch#

There are two ways to call a method of a trait, and kv uses both. The main function builds a Box<dyn Store>, a box holding whichever type the environment asked for. dyn Store is a trait object. A pointer to one carries a second pointer to a table of the type's methods. Every call looks the method up in that table at run time:

// Either type implements Store. A Box<dyn Store> holds whichever one the environment
// asked for, and every call through it is decided at run time.
let mut store: Box<dyn Store> = match std::env::var("KV_STORE").as_deref() {
    Ok("table") => Box::new(Table::default()),
    _ => Box::new(MapStore::new()),
};
rust/ch07/src/main.rs · region choose

The test in store.rs takes the other road. fn behaves_like_a_store<S: Store> is generic with a bound, and the compiler writes one copy of it per type it is called with. Each copy calls the methods directly, which is static dispatch, the same mechanism as the generic table of chapter 6. impl Store in an argument position is the short spelling of the same bound:

#[cfg(test)]
mod tests {
    use super::{MapStore, Store};
    use crate::table::Table;

    /// One test for every store. `S: Store` is a bound, and the compiler writes a copy of
    /// this function for each type it is called with: static dispatch, no trait object.
    fn behaves_like_a_store<S: Store>(mut store: S) {
        assert!(store.is_empty());
        store.set(b"a".to_vec(), b"1".to_vec());
        assert_eq!(store.get(b"a"), Some(&b"1"[..]));
        assert_eq!(store.incr(b"a"), Ok(2));
        assert_eq!(store.keys(b"*").len(), 1);
        assert!(store.del(b"a"));
        assert!(store.get(b"a").is_none());
    }

    #[test]
    fn the_map_store_is_a_store() {
        behaves_like_a_store(MapStore::new());
    }

    #[test]
    fn the_table_is_a_store() {
        behaves_like_a_store(Table::<Vec<u8>, Vec<u8>>::default());
    }
}
rust/ch07/src/store.rs · region test

The handlers of chapter 5 took &mut dyn Write for the reason that appears again here. A table of function pointers holds one function, so the function must work for any writer at run time. The helpers in resp take impl Write + ?Sized, which is the generic form. The ?Sized is a note that the type may be a trait object, whose size the compiler does not know. One rule limits which traits can be objects. A trait with a generic method cannot be a dyn. The same holds for a method that returns Self. The method table could not hold either. Store has no such method, so it can.

One more word of visibility appeared in this chapter. parse_int is pub(crate), which every module in the crate may use and no user of the library can see. Between private and pub, it is the one that most helpers want.

A dependency in Rust is a line in Cargo.toml, which cargo add writes for you, and a crate on crates.io, which is the one registry. The Cargo book draws the line between the two files in one sentence each. "Cargo.toml is about describing your dependencies in a broad sense, and is written by you." "Cargo.lock contains exact information about your dependencies. It is maintained by Cargo and should not be manually edited." The first dependency of kv arrives in chapter 9. That is when the lock file appears. Two more Cargo words will come up before then. A feature is a named switch in Cargo.toml that turns part of a crate on. The code behind it is marked with a cfg attribute that names the feature. tokio in chapter 9 is built from features. Few programs need all of it. A workspace is a folder of crates that share one Cargo.lock and one target directory, for a project with a library and several binaries. Go has the same two ideas under other names. A build tag is a feature, and a go.work file is a workspace. Chapter 14 uses a workspace when a module must build with a local copy of another.

โš ๏ธA file is not a module until something declares it Rust

Add src/glob.rs and nothing happens. The compiler reads only the files that a mod line names, starting from lib.rs or main.rs. A forgotten mod line produces an error about an unresolved name somewhere else, and the file that holds the name is never mentioned. Go reads every file in the folder. C reads only what the Makefile lists. Rust reads what the tree of mod declarations reaches.

Three answers to "where does code come from"#

Set the three languages side by side and the pattern of the book repeats. C has a convention where the other two have a tool, and the convention has several competing forms. Go has one tool with no registry, which fits a company that hosts its own code. Rust has one tool with one registry, which fits a community that shares small crates. The number of dependencies a program of this size has in practice follows from that. A C program has few, because each one is work. A Go program has few, because the standard library is wide. A Rust program often has dozens, because adding one is a single command, and that is the weakness of the strongest answer. Chapter 14 shows what every dependency costs at the border, and chapter 15 what a C one costs.

One style, enforced by a tool#

All three languages now have a formatter, and the difference is who decides the style. clang-format takes a configuration file, and the one for kv is four settings:

# The layout every C file in kv keeps. clang-format -i src/*.c src/*.h applies it.
BasedOnStyle: LLVM
IndentWidth: 4
ColumnLimit: 100
AllowShortFunctionsOnASingleLine: Inline
c/ch07/.clang-format

It comes with clang, so it is installed on macOS with Xcode and on Linux as a package. A project that uses it must agree on the configuration first. gofmt has no configuration. The Go team decided the style when the language was released. The result is that every Go file anyone has ever read looks the same. rustfmt is between clang-format and gofmt. It has a configuration file, rustfmt.toml, and almost nobody writes one. The book runs all three formatters in make verify. A file that is not formatted fails the build.

Beside each formatter is a linter that finds mistakes the compiler allows. clang-tidy for C must be installed and configured. go vet comes with Go and runs in make verify, and staticcheck adds more. cargo clippy comes with Rust, has more than 800 checks, and is the tool that asked for is_empty above. The verdict for this chapter is mostly about these tools, because the day-to-day experience of a language is the experience of its tools.

Verdict##

TopicCGoRustWhy
Splitting codeWeakStrongOKC needs guards, prefixes and a Makefile that tracks headers, while Go needs a folder and Rust a mod line that is easy to forget.
Hiding a nameOKStrongStrongC has static, for functions only. Go and Rust hide by default or by a letter.
Adding a dependencyWeakOKStrongC has three competing ways and no default, while Go and Rust each have one command, and Rust also has one registry.
Having too manyStrongStrongWeakC and Go programs stay small by cost and by a wide standard library. Rust programs grow dozens of crates.
Keeping one styleOKStrongStrongclang-format needs a config everyone agrees on. gofmt and rustfmt need nothing.
Finding mistakesWeakOKStrongclang-tidy must be installed. go vet is built in. clippy is built in and has 800 checks.
Swapping an implementationOKStrongStrongC has a struct of function pointers that nobody checks. Go has implicit interfaces, and Rust has checked traits with two kinds of dispatch.
Hiding a structStrongStrongStrongC has the opaque type, and Go and Rust have unexported fields. All three keep the map out of reach.
Carry forward
  • Opaque type. A C header declares a struct it does not define, so other files hold a pointer and nothing more.
  • Interface. A Go type satisfies an interface by having its methods, and a nil pointer inside one is not nil.
  • Trait object. A dyn Store is dispatched at run time through a method table, and a generic bound is dispatched at compile time.
Next

A key holds a string, and only a string. Ask for every key in a million-key store, and a second client's GET waits most of a second for the answer to finish going out.

Data types#

Building kv · Chapter 08· 32 min read· 706 lines of code
▶Scene

Fill the chapter 7 server with a million keys. Then, from one terminal, ask for all of them, and fifty milliseconds later, from another, ask for one:

$ redis-cli -p 6380 --pipe < million-sets.txt
errors: 0, replies: 1000000
$ time redis-cli -p 6380 KEYS '*' > /dev/null         # terminal one
real    0m0.75s
$ time redis-cli -p 6380 GET user:1                     # terminal two, 50 ms later
"v"
real    0m0.68s

A GET takes no measurable time on its own. This one took most of a second, because it waited for another client's KEYS to finish walking a million entries. One question, and everyone waits. Redis answers it with a command that walks the table a page at a time, and with values that are more than strings.

A key can hold more than a string. This chapter gives kv lists, hashes and sets, and a TYPE command that says which. It adds SCAN, which walks the keys a page at a time. The chapter is about sum types, which each language builds in its own way, and about iteration.

Redis is not a map from strings to strings. Its documentation calls it a data structure server. The value under a key may be a string, a list, a hash of fields or a set of members. There are several more, and kv takes the first four. Each needs the same three things from the store. A value must say which kind it is. A command must take the value apart as that kind, and refuse any other kind with the error Redis uses, WRONGTYPE. And a command such as HGETALL or SMEMBERS must walk whatever is inside. The first need is a sum type, a value that is one of several shapes. The third is iteration. The three languages answer both, and the answers are the chapter.

One value, four kindsCGoRust
The valuea struct with a tag and a unionan any, holding one of four concrete typesan enum with data in each variant
Asking whichswitch (v->kind)a type switch, switch v := x.(type)match, which must cover every variant
Taking one kindread the union member the tag names, and nothing checksa type assertion, v.(*List), with an oka pattern, Value::List(l), in a match or let else
Adding a fifth kindthe compiler warns at each switch with no defaultnothing warns; a missed case falls to defaultthe compiler refuses every match that forgot it
A listan array of byte strings, shifted along for LPUSHa slice of byte slicesa VecDeque, with both ends cheap
A hash and a seta nested store, the set with empty valuesa map, and a generic Set[T]a HashMap and a HashSet
Walking a collectiona callback with a void * contextan iterator function that range callsan Iterator, with map, filter and collect
๐Ÿ“œWhy Redis has lists and sets, and not only strings

Salvatore Sanfilippo began Redis in 2009 while trying to improve the scalability of his startup, a real-time web log analyser. He had met "significant problems in scaling some types of workloads using traditional database systems". A log analyser keeps lists of recent events and sets of visitors. A list in a string store is a string the client must rewrite on every push. A list in the server is one command. The documentation calls Redis "a data structure server", and the name Remote Dictionary Server undersells it. The value is where the work is.

C: a tag and a union#

A C value that may be one of four things is a struct of two parts. The first is a tag, an enum that says which shape is live. The second is a union, a type whose members share the same bytes. So a value holds one string, or one list, or one table, and never more than one at a time. The pair is called a tagged union, and C checks nothing about it. Read the list member of a value whose tag says string and you read the string's pointer as an array length:

/* Which of the four shapes a value has. The number is the tag of the union below. */
typedef enum { KV_STRING, KV_LIST, KV_HASH, KV_SET } kv_kind;

/* A tagged union: the tag says which member of the union is live, and reading another
 * member is a bug the compiler will not catch. A hash and a set are each a nested store,
 * whose entries hold strings; a set keeps its values empty and uses only the keys. */
typedef struct {
    kv_kind kind;
    union {
        kv_bytes str;
        kv_list list;
        kv_store *table; /* owned, for KV_HASH and KV_SET */
    } u;
} kv_value;

const char *kv_kind_name(kv_kind kind); /* "string", "list", "hash" or "set" */
void kv_value_free(kv_value *v);
c/ch08/src/store.h · region value

A hash and a set are each a nested store. The store of chapter 7 is opaque, holds values under byte keys, and can be made with the same type as its parent. So a hash is a store whose values are strings. A set is a store whose values are empty. Nothing new was written for either. The list is new. It is an array of byte strings, and LPUSH shifts the whole array along by one, which costs the length of the list. Redis uses a linked structure there. kv keeps the array, because the chapter is about the union, and says what it costs:

/* A list: an array of owned byte strings. RPUSH appends; LPUSH shifts the array along by
 * one, which costs the length of the list. Redis keeps a linked structure instead, and
 * pays for that in other ways. */
typedef struct {
    kv_bytes *items; /* owned array of owned strings */
    size_t len;
    size_t cap;
} kv_list;
c/ch08/src/store.h · region list

Freeing a value is a switch on the tag with a case per kind and no default. The compiler reports a kind this switch forgot. That is the only check C gives a tagged union, and the reason the switch is written that way:

const char *kv_kind_name(kv_kind kind) {
    switch (kind) {
    case KV_STRING:
        return "string";
    case KV_LIST:
        return "list";
    case KV_HASH:
        return "hash";
    case KV_SET:
        return "set";
    }
    return "none";
}

bool kv_bytes_set(kv_bytes *b, const char *src, size_t n) {
    char *copy = kv_bytes_dup(src, n);
    if (copy == NULL) {
        return false;
    }
    free(b->p);
    b->p = copy;
    b->len = n;
    return true;
}

static void kv_list_free(kv_list *l) {
    for (size_t i = 0; i < l->len; i++) {
        free(l->items[i].p);
    }
    free(l->items);
}

bool kv_list_push(kv_list *l, const char *src, size_t n, bool front) {
    char *copy = kv_bytes_dup(src, n);
    if (copy == NULL) {
        return false;
    }
    if (l->len == l->cap) {
        size_t cap = l->cap ? l->cap * 2 : 4;
        kv_bytes *items = realloc(l->items, cap * sizeof *items);
        if (items == NULL) {
            free(copy);
            return false;
        }
        l->items = items;
        l->cap = cap;
    }
    if (front) {
        memmove(l->items + 1, l->items, l->len * sizeof *l->items);
        l->items[0] = (kv_bytes){copy, n};
    } else {
        l->items[l->len] = (kv_bytes){copy, n};
    }
    l->len++;
    return true;
}

/* Free whatever the tag says is live. The switch has no default, so a fifth kind would
 * be reported by the compiler here, with -Wall, until it is handled. */
void kv_value_free(kv_value *v) {
    switch (v->kind) {
    case KV_STRING:
        free(v->u.str.p);
        break;
    case KV_LIST:
        kv_list_free(&v->u.list);
        break;
    case KV_HASH:
    case KV_SET:
        kv_store_free(v->u.table);
        break;
    }
    *v = (kv_value){KV_STRING, {{NULL, 0}}};
}
c/ch08/src/store.c · region value-fns

Every command that takes a value apart starts in one of two helpers. The first returns the string of an entry, or writes the WRONGTYPE error and returns NULL. The second returns an entry as a given kind, making an empty one when the key is new. Each is the tag check written once:

#define KV_WRONGTYPE "WRONGTYPE Operation against a key holding the wrong kind of value"

/* The string of an entry, or NULL after writing the WRONGTYPE error. Every string
 * command starts here, so the check against the tag is in one place. */
static kv_bytes *kv_string_of(kv_entry *e, kv_buf *out) {
    if (e->val.kind != KV_STRING) {
        kv_reply_error(out, KV_WRONGTYPE);
        return NULL;
    }
    return &e->val.u.str;
}

/* The entry for key as the given kind, made empty when the key is new, or NULL after
 * writing the WRONGTYPE error when it holds another kind. */
static kv_entry *kv_entry_as(kv_store *s, const kv_cmd *c, kv_kind kind, kv_buf *out, bool *oom) {
    bool created;
    *oom = false;
    kv_entry *e = kv_store_insert(s, c->argv[1], c->arglen[1], &created);
    if (e == NULL) {
        *oom = true;
        return NULL;
    }
    if (created) {
        e->val.kind = kind;
        if (kind == KV_HASH || kind == KV_SET) {
            e->val.u.table = kv_store_new(kv_store_type_of(s));
            if (e->val.u.table == NULL) {
                kv_store_del(s, c->argv[1], c->arglen[1]);
                *oom = true;
                return NULL;
            }
        }
    } else if (e->val.kind != kind) {
        kv_reply_error(out, KV_WRONGTYPE);
        return NULL;
    }
    return e;
}
c/ch08/src/cmd.c · region wrongtype

The string commands of chapter 5 change by one line each, the call to kv_string_of. GETRANGE and LRANGE share a clamp, because a list is sliced by position exactly as a string is sliced by byte:

/* LPUSH and RPUSH: the same function, told which end by the table. */
static bool kv_push(const kv_cmd *c, kv_store *s, kv_buf *out, bool front) {
    bool oom;
    kv_entry *e = kv_entry_as(s, c, KV_LIST, out, &oom);
    if (e == NULL) {
        return !oom;
    }
    for (size_t i = 2; i < c->argc; i++) {
        if (!kv_list_push(&e->val.u.list, c->argv[i], c->arglen[i], front)) {
            return false;
        }
    }
    return kv_reply_int(out, (long long)e->val.u.list.len);
}

static bool kv_cmd_lpush(const kv_cmd *c, kv_store *s, kv_buf *out) {
    return kv_push(c, s, out, true);
}

static bool kv_cmd_rpush(const kv_cmd *c, kv_store *s, kv_buf *out) {
    return kv_push(c, s, out, false);
}

/* The list of an entry, or NULL after the WRONGTYPE error. */
static kv_list *kv_list_of(kv_entry *e, kv_buf *out) {
    if (e->val.kind != KV_LIST) {
        kv_reply_error(out, KV_WRONGTYPE);
        return NULL;
    }
    return &e->val.u.list;
}

static bool kv_cmd_llen(const kv_cmd *c, kv_store *s, kv_buf *out) {
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
    if (e == NULL) {
        return kv_reply_int(out, 0);
    }
    kv_list *l = kv_list_of(e, out);
    return l == NULL || kv_reply_int(out, (long long)l->len);
}

static bool kv_cmd_lrange(const kv_cmd *c, kv_store *s, kv_buf *out) {
    long long start, end;
    if (!kv_parse_int(c->argv[2], c->arglen[2], &start) ||
        !kv_parse_int(c->argv[3], c->arglen[3], &end)) {
        return kv_reply_error(out, "ERR value is not an integer or out of range");
    }
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
    if (e == NULL) {
        return kv_reply_array(out, 0);
    }
    kv_list *l = kv_list_of(e, out);
    if (l == NULL) {
        return true;
    }
    size_t from, to;
    if (!kv_clamp(start, end, (long long)l->len, &from, &to)) {
        return kv_reply_array(out, 0);
    }
    if (!kv_reply_array(out, to - from + 1)) {
        return false;
    }
    for (size_t i = from; i <= to; i++) {
        if (!kv_reply_bulk(out, l->items[i].p, l->items[i].len)) {
            return false;
        }
    }
    return true;
}
c/ch08/src/cmd.c · region lists
Exercise 08.1CAdd members to a set

Write kv_cmd_sadd. Get the entry as KV_SET with kv_entry_as, which writes the error for a key of another kind. Then insert each member into the nested store with kv_store_insert, and count the ones that were created. Reply with the count. Return false only when memory runs out, which is what oom reports.

static bool kv_cmd_sadd(const kv_cmd *c, kv_store *s, kv_buf *out) {
    bool oom;
    kv_entry *e = kv_entry_as(s, c, KV_SET, out, &oom);
    if (e == NULL) {
        return !oom;
    }
    /* your code here */
}
Stuck?Show the full listing
static bool kv_cmd_sadd(const kv_cmd *c, kv_store *s, kv_buf *out) {
    bool oom;
    kv_entry *e = kv_entry_as(s, c, KV_SET, out, &oom);
    if (e == NULL) {
        return !oom;
    }
    long long added = 0;
    for (size_t i = 2; i < c->argc; i++) {
        bool created;
        if (kv_store_insert(e->val.u.table, c->argv[i], c->arglen[i], &created) == NULL) {
            return false;
        }
        added += created;
    }
    return kv_reply_int(out, added);
}

static bool kv_cmd_sismember(const kv_cmd *c, kv_store *s, kv_buf *out) {
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
    if (e == NULL) {
        return kv_reply_int(out, 0);
    }
    kv_store *set = kv_table_of(e, KV_SET, out);
    return set == NULL || kv_reply_int(out, kv_store_find(set, c->argv[2], c->arglen[2]) != NULL);
}

static bool kv_cmd_smembers(const kv_cmd *c, kv_store *s, kv_buf *out) {
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
    if (e == NULL) {
        return kv_reply_array(out, 0);
    }
    kv_store *set = kv_table_of(e, KV_SET, out);
    if (set == NULL) {
        return true;
    }
    if (!kv_reply_array(out, kv_store_len(set))) {
        return false;
    }
    kv_members_ctx m = {out, false, true};
    kv_store_each(set, kv_members_visit, &m);
    return m.ok;
}
c/ch08/src/cmd.c · region sets

HGETALL and SMEMBERS walk the nested store with the callback of chapter 7. One callback serves both, told by its context whether to write the values as well as the keys:

/* HSET key field value [field value ...]: the fields go into the nested store. */
static bool kv_cmd_hset(const kv_cmd *c, kv_store *s, kv_buf *out) {
    if (c->argc % 2 != 0) {
        return kv_reply_arity(out, "hset");
    }
    bool oom;
    kv_entry *e = kv_entry_as(s, c, KV_HASH, out, &oom);
    if (e == NULL) {
        return !oom;
    }
    long long added = 0;
    for (size_t i = 2; i + 1 < c->argc; i += 2) {
        added += kv_store_find(e->val.u.table, c->argv[i], c->arglen[i]) == NULL;
        if (!kv_store_set(e->val.u.table, c->argv[i], c->arglen[i], c->argv[i + 1], c->arglen[i + 1])) {
            return false;
        }
    }
    return kv_reply_int(out, added);
}

/* The nested table of an entry, or NULL after the WRONGTYPE error. */
static kv_store *kv_table_of(kv_entry *e, kv_kind kind, kv_buf *out) {
    if (e->val.kind != kind) {
        kv_reply_error(out, KV_WRONGTYPE);
        return NULL;
    }
    return e->val.u.table;
}

static bool kv_cmd_hget(const kv_cmd *c, kv_store *s, kv_buf *out) {
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
    if (e == NULL) {
        return kv_reply_nil(out);
    }
    kv_store *h = kv_table_of(e, KV_HASH, out);
    if (h == NULL) {
        return true;
    }
    kv_entry *f = kv_store_find(h, c->argv[2], c->arglen[2]);
    return f ? kv_reply_bulk(out, f->val.u.str.p, f->val.u.str.len) : kv_reply_nil(out);
}

/* The context of the HGETALL and SMEMBERS walks: the reply so far, and whether a
 * reply failed to append. */
typedef struct {
    kv_buf *out;
    bool with_values;
    bool ok;
} kv_members_ctx;

static bool kv_members_visit(kv_entry *e, void *ctx) {
    kv_members_ctx *m = ctx;
    m->ok = kv_reply_bulk(m->out, e->key, e->klen) &&
            (!m->with_values || kv_reply_bulk(m->out, e->val.u.str.p, e->val.u.str.len));
    return m->ok;
}

static bool kv_cmd_hgetall(const kv_cmd *c, kv_store *s, kv_buf *out) {
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
    if (e == NULL) {
        return kv_reply_array(out, 0);
    }
    kv_store *h = kv_table_of(e, KV_HASH, out);
    if (h == NULL) {
        return true;
    }
    if (!kv_reply_array(out, kv_store_len(h) * 2)) {
        return false;
    }
    kv_members_ctx m = {out, true, true};
    kv_store_each(h, kv_members_visit, &m);
    return m.ok;
}
c/ch08/src/cmd.c · region hashes

SCAN costs C the least. The store's buckets are an array, and a cursor is an index into it. The walk starts at the cursor's bucket and visits whole buckets until it has seen enough entries. Then it returns the next bucket as the new cursor, or 0 at the end. A client that calls SCAN with each returned cursor sees every key that was there throughout, without the server ever holding the whole list:

size_t kv_store_scan(kv_store *s, size_t cursor, size_t count,
                     bool (*fn)(kv_entry *e, void *ctx), void *ctx) {
    size_t seen = 0;
    while (cursor < s->cap) {
        for (kv_entry *e = s->buckets[cursor]; e != NULL; e = e->next) {
            seen++;
            if (!fn(e, ctx)) {
                return 0;
            }
        }
        cursor++;
        if (seen >= count) {
            break;
        }
    }
    return cursor < s->cap ? cursor : 0;
}
c/ch08/src/store.c · region scan
static bool kv_cmd_type(const kv_cmd *c, kv_store *s, kv_buf *out) {
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1]);
    return kv_reply_simple(out, e ? kv_kind_name(e->val.kind) : "none");
}

/* The context of a SCAN walk: the pattern, and the matching keys collected so far. */
typedef struct {
    const char *pat;
    size_t plen;
    kv_buf keys;
    size_t n;
    bool ok;
} kv_scan_ctx;

static bool kv_scan_visit(kv_entry *e, void *ctx) {
    kv_scan_ctx *k = ctx;
    if (k->pat != NULL && !kv_glob(k->pat, k->plen, e->key, e->klen)) {
        return true;
    }
    if (!kv_reply_bulk(&k->keys, e->key, e->klen)) {
        k->ok = false;
        return false;
    }
    k->n++;
    return true;
}

/* SCAN cursor [MATCH pattern] [COUNT n]: a page of keys and the cursor for the next page.
 * The reply is an array of two things, the cursor as a string and the array of keys. */
static bool kv_cmd_scan(const kv_cmd *c, kv_store *s, kv_buf *out) {
    long long cursor;
    if (!kv_parse_int(c->argv[1], c->arglen[1], &cursor) || cursor < 0) {
        return kv_reply_error(out, "ERR invalid cursor");
    }
    kv_scan_ctx k = {NULL, 0, {0}, 0, true};
    long long count = 10;
    for (size_t i = 2; i + 1 < c->argc; i += 2) {
        if (kv_arg_is(c, i, "MATCH")) {
            k.pat = c->argv[i + 1];
            k.plen = c->arglen[i + 1];
        } else if (kv_arg_is(c, i, "COUNT")) {
            if (!kv_parse_int(c->argv[i + 1], c->arglen[i + 1], &count) || count < 1) {
                return kv_reply_error(out, "ERR value is not an integer or out of range");
            }
        } else {
            return kv_reply_error(out, "ERR syntax error");
        }
    }
    if (c->argc % 2 != 0) {
        return kv_reply_error(out, "ERR syntax error");
    }
    size_t next = kv_store_scan(s, (size_t)cursor, (size_t)count, kv_scan_visit, &k);
    char text[32];
    int len = snprintf(text, sizeof text, "%zu", next);
    bool ok = k.ok && kv_reply_array(out, 2) && kv_reply_bulk(out, text, (size_t)len) &&
              kv_reply_array(out, k.n) && kv_buf_append(out, k.keys.data, k.keys.len);
    kv_buf_free(&k.keys);
    return ok;
}
c/ch08/src/cmd.c · region type-scan
โš ๏ธThe union member the tag does not name C

Reading a union member that is not the live one is undefined behaviour in the general case, and the compiler will not say a word. The only defence is the habit the helpers above enforce. Check the tag, then read the member, and never pass a union member around without the tag beside it. A switch on the tag with no default keeps the check honest when a kind is added.

Go: any, and a switch on the type#

Go has no union. It has any, the interface with no methods, which every value satisfies. The store holds an any under each key, and the four kinds are four concrete types inside it. A type switch asks which. It is a switch whose cases are types, and Kind is one. The 'a in its name says that the reference it holds borrows the store, which chapter 10 explains:

// A value is one of four things, and the store holds it as `any`: an interface with no
// methods, which any value satisfies. The four concrete types are the cases, and a type
// switch takes the value apart. ErrWrongType is what a command replies when the case is
// not the one it needs.
var ErrWrongType = errors.New("WRONGTYPE Operation against a key holding the wrong kind of value")

// Kind names the case, as TYPE reports it. The switch has no default on purpose: a
// value of a fifth type would reach "none", and the tests would say so.
func Kind(v any) string {
	switch v.(type) {
	case []byte:
		return "string"
	case *List:
		return "list"
	case Hash:
		return "hash"
	case *Set[string]:
		return "set"
	}
	return "none"
}
go/ch08/internal/store/value.go · region kinds

A type assertion asks for one type and gets a second result that says whether the value had it. With that ok an assertion never panics. Without it, one that fails stops the program, which is the form to avoid on data from a client. The string commands start in stringOf, and the three collection kinds start in as, which is generic over the kind it returns:

// stringOf is the value as bytes, or ErrWrongType. The assertion v.([]byte) asks whether
// the value inside the interface has that type; the second result says. Every string
// command starts here, so the check is in one place.
func stringOf(v any) ([]byte, error) {
	b, ok := v.([]byte)
	if !ok {
		return nil, store.ErrWrongType
	}
	return b, nil
}

// as returns the value under key as T, making one with mk when the key is new. A key
// that holds another kind is ErrWrongType. T is a type parameter, so one function
// serves *store.List, store.Hash and *store.Set[string].
func as[T any](s Storer, key []byte, mk func() T) (T, error) {
	if v, ok := s.Get(key); ok {
		t, isT := v.(T)
		if !isT {
			var zero T
			return zero, store.ErrWrongType
		}
		return t, nil
	}
	t := mk()
	s.Set(key, t)
	return t, nil
}
go/ch08/internal/cmd/cmd.go · region as

Nothing warns when a type switch forgets a case. A fifth kind added to the store falls to the default of every switch that did not know it, and TYPE would answer none. That is the cost of a sum type built from an interface, and the tests are what catch it. The list, the hash and the set are three types of their own. The set is generic, because nothing in it depends on what the members are:

// A Hash is a map from field to value, with nothing to add: a map already does it all.
type Hash map[string][]byte

// A Set is a map with no values. It is generic, because nothing about it depends on
// what the members are, only that they can be compared.
type Set[T comparable] struct {
	m map[T]struct{}
}

func NewSet[T comparable]() *Set[T] { return &Set[T]{m: make(map[T]struct{})} }

// Add reports whether the member was new.
func (s *Set[T]) Add(v T) bool {
	_, had := s.m[v]
	s.m[v] = struct{}{}
	return !had
}

func (s *Set[T]) Has(v T) bool { _, ok := s.m[v]; return ok }
func (s *Set[T]) Len() int     { return len(s.m) }

// All yields every member, in no particular order.
func (s *Set[T]) All() iter.Seq[T] {
	return func(yield func(T) bool) {
		for v := range s.m {
			if !yield(v) {
				return
			}
		}
	}
}
go/ch08/internal/store/value.go · region hash-set

A function that range can call#

Since Go 1.23, range can loop over a function. The function takes the loop body, called yield, and calls it once per item until it returns false. The type iter.Seq[T] names that shape. The list's Range is one, and LRANGE loops over it without ever seeing the slice inside:

// A List is a slice of byte strings. PushBack appends; PushFront makes room at the
// start by copying the slice along, which costs the length of the list.
type List struct {
	items [][]byte
}

func (l *List) PushBack(v []byte)  { l.items = append(l.items, v) }
func (l *List) PushFront(v []byte) { l.items = append([][]byte{v}, l.items...) }
func (l *List) Len() int           { return len(l.items) }

// Range yields the items from start to end, both inclusive, with negative offsets
// counted from the end and out-of-range offsets clamped. It is an iterator: a function
// that takes the loop body, and range calls it. The caller never sees the slice.
func (l *List) Range(start, end int64) iter.Seq[[]byte] {
	return func(yield func([]byte) bool) {
		n := int64(len(l.items))
		if start < 0 {
			start += n
		}
		if end < 0 {
			end += n
		}
		start = max(start, 0)
		end = min(end, n-1)
		for i := start; i <= end; i++ {
			if !yield(l.items[i]) {
				return
			}
		}
	}
}
go/ch08/internal/store/value.go · region list
Exercise 08.2GoCollect a page from the iterator

Write lrange. Parse the two offsets with strconv.ParseInt, and reply with the Redis error when either fails. Get the list with listOf, and reply with its error when the key holds another kind. Then loop over l.Range(start, end) with range, append each item to a slice, and write the slice with writeBulks. The count must be known before the items, which is why the iterator is drained first.

func lrange(args [][]byte, s Storer, w *bufio.Writer) error {
	// your code here
}
Stuck?Show the full listing
func push(args [][]byte, s Storer, w *bufio.Writer, front bool) error {
	l, err := as(s, args[1], func() *store.List { return &store.List{} })
	if err != nil {
		return resp.WriteError(w, err.Error())
	}
	for _, v := range args[2:] {
		if front {
			l.PushFront(v)
		} else {
			l.PushBack(v)
		}
	}
	return resp.WriteInt(w, int64(l.Len()))
}

func lpush(args [][]byte, s Storer, w *bufio.Writer) error { return push(args, s, w, true) }
func rpush(args [][]byte, s Storer, w *bufio.Writer) error { return push(args, s, w, false) }

// listOf is the list under key, an empty one when the key is missing, or an error.
func listOf(s Storer, key []byte) (*store.List, error) {
	v, ok := s.Get(key)
	if !ok {
		return &store.List{}, nil
	}
	l, isList := v.(*store.List)
	if !isList {
		return nil, store.ErrWrongType
	}
	return l, nil
}

func llen(args [][]byte, s Storer, w *bufio.Writer) error {
	l, err := listOf(s, args[1])
	if err != nil {
		return resp.WriteError(w, err.Error())
	}
	return resp.WriteInt(w, int64(l.Len()))
}

// lrange collects the page with range over the list's iterator. The count is needed
// before the items, so the iterator is drained into a slice first.
func lrange(args [][]byte, s Storer, w *bufio.Writer) error {
	start, err1 := strconv.ParseInt(string(args[2]), 10, 64)
	end, err2 := strconv.ParseInt(string(args[3]), 10, 64)
	if err1 != nil || err2 != nil {
		return resp.WriteError(w, "ERR value is not an integer or out of range")
	}
	l, err := listOf(s, args[1])
	if err != nil {
		return resp.WriteError(w, err.Error())
	}
	var page [][]byte
	for item := range l.Range(start, end) {
		page = append(page, item)
	}
	return writeBulks(w, page)
}
go/ch08/internal/cmd/cmd.go · region lists

A Go map cannot be scanned from a position. Each range over a map starts in a new random order, by design, so that no program comes to depend on the order. The map store sorts its keys and uses the cursor as an index into the sorted list, which costs a sort per page. The table from chapter 6 owns its buckets and scans them as the C store does, in one walk. That difference is the reason Redis owns its table, and the one place in the book where the hand-written table is the better store:

// Scan returns a page of keys from the cursor on, and the cursor for the next page, or
// 0 when the walk is done. A Go map cannot be walked from a position: each range starts
// in a new random order. So the keys are sorted and the cursor is an index into that
// order, which costs a sort per page. The hand-written table scans by bucket instead.
func (s *Store) Scan(cursor uint64, count int) (uint64, [][]byte) {
	sorted := slices.Sorted(maps.Keys(s.m))
	if cursor >= uint64(len(sorted)) {
		return 0, nil
	}
	end := min(int(cursor)+count, len(sorted))
	var page [][]byte
	for _, k := range sorted[cursor:end] {
		page = append(page, []byte(k))
	}
	if end == len(sorted) {
		return 0, page
	}
	return uint64(end), page
}
go/ch08/internal/store/store.go · region scan
// Scan walks the buckets from cursor, collecting keys until at least count have been
// seen or the buckets end. The returned cursor is the next bucket, or 0 at the end: a
// cursor is a bucket index, which is why the walk can resume.
func (t *Table[K, V]) Scan(cursor uint64, count int) (uint64, []K) {
	var page []K
	for cursor < uint64(len(t.buckets)) {
		for _, e := range t.buckets[cursor] {
			page = append(page, e.key)
		}
		cursor++
		if len(page) >= count {
			break
		}
	}
	if cursor >= uint64(len(t.buckets)) {
		return 0, page
	}
	return cursor, page
}
go/ch08/internal/store/table.go · region scan
โš ๏ธA type assertion without the ok panics Go

v.(*List) on a value that is not a *List stops the program with interface conversion. On a value that came from a client, that is a crash any client can cause. The two-result form, l, ok := v.(*List), never panics, and every assertion in kv has the ok. A type switch has the same safety with no extra code, because an unmatched type takes the default branch.

Rust: an enum with data in each variant#

A Rust enum is the tagged union with the check the C version lacks. Each variant carries its own data, and the compiler knows which variant is live. The only way to read the data is a pattern that names the variant. A match must cover every variant, so a fifth kind added next year is a compile error at every match that forgot it:

/// The four kinds of value, as one enum. Each variant carries its own data, and a match
/// must handle every variant, so a fifth kind added later is a compile error at every
/// place that forgot it. This is the sum type that C builds by hand with a tag and a
/// union, and Go builds with an interface and a type switch.
pub enum Value {
    Str(Vec<u8>),
    List(VecDeque<Vec<u8>>),
    Hash(HashMap<Vec<u8>, Vec<u8>>),
    Set(HashSet<Vec<u8>>),
}

impl Value {
    /// The name TYPE reports.
    pub fn kind(&self) -> &'static str {
        match self {
            Value::Str(_) => "string",
            Value::List(_) => "list",
            Value::Hash(_) => "hash",
            Value::Set(_) => "set",
        }
    }
}

/// The error a command replies when the value is not the kind it needs.
#[derive(Debug, PartialEq)]
pub struct WrongType;

impl WrongType {
    pub const MESSAGE: &'static str =
        "WRONGTYPE Operation against a key holding the wrong kind of value";
}
rust/ch08/src/store.rs · region value

The trait gains five methods that work on a value of any kind, and the string commands become default methods on top of them. get returns a Result of an Option, because a string command has three outcomes, the string, no key, and a key of another kind. incr uses ? on that result. A From turns the wrong-type error into the IncrError it returns:

/// What the commands need from a store. Five methods work on values of any kind, and the
/// string commands are default methods written on top of them.
pub trait Store {
    fn value(&self, key: &[u8]) -> Option<&Value>;
    fn value_mut(&mut self, key: &[u8]) -> Option<&mut Value>;
    /// The value under key, made with `make` when the key is new.
    fn value_or_insert(&mut self, key: Vec<u8>, make: fn() -> Value) -> &mut Value;
    /// Store a value of any kind, replacing whatever kind was there.
    fn insert(&mut self, key: Vec<u8>, val: Value);
    fn del(&mut self, key: &[u8]) -> bool;
    fn len(&self) -> usize;
    /// Every key that matches the pattern, in no particular order.
    fn keys(&self, pattern: &[u8]) -> Vec<&[u8]>;
    /// A page of up to `count` keys from `cursor` on, and the cursor of the next page,
    /// or 0 when the walk is complete.
    fn scan(&self, cursor: usize, count: usize) -> (usize, Vec<&[u8]>);

    fn is_empty(&self) -> bool {
        self.len() == 0
    }

    /// The string under key: `None` for a missing key, an error for another kind.
    fn get(&self, key: &[u8]) -> Result<Option<&[u8]>, WrongType> {
        match self.value(key) {
            None => Ok(None),
            Some(Value::Str(s)) => Ok(Some(s)),
            Some(_) => Err(WrongType),
        }
    }

    fn set(&mut self, key: Vec<u8>, val: Vec<u8>) {
        self.insert(key, Value::Str(val));
    }

    /// Add one to the integer stored under key, which counts as 0 when absent.
    fn incr(&mut self, key: &[u8]) -> Result<i64, IncrError> {
        let n = match self.get(key)? {
            None => 0,
            Some(bytes) => parse_int(bytes).ok_or(IncrError::NotInteger)?,
        };
        let n = n.checked_add(1).ok_or(IncrError::Overflow)?;
        self.set(key.to_vec(), n.to_string().into_bytes());
        Ok(n)
    }
}
rust/ch08/src/store.rs · region trait

The commands take a value apart with let ... else, which matches one variant and leaves the function when the value is another. The list is a VecDeque, which pushes at either end in constant time. The arguments are moved into it with drain, so no byte of a pushed item is copied:

/// LPUSH and RPUSH: the same function, told which end by a flag. A `VecDeque` pushes at
/// either end in constant time, which is why the list is one and not a `Vec`.
fn push(
    args: &mut Command,
    store: &mut dyn Store,
    w: &mut dyn Write,
    front: bool,
) -> io::Result<()> {
    let key = mem::take(&mut args[1]);
    let Value::List(list) = store.value_or_insert(key, || Value::List(VecDeque::new())) else {
        return write_wrongtype(w);
    };
    for item in args.drain(2..) {
        if front {
            list.push_front(item);
        } else {
            list.push_back(item);
        }
    }
    resp::write_int(w, list.len() as i64)
}

fn lpush(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
    push(args, store, w, true)
}

fn rpush(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
    push(args, store, w, false)
}

/// The value under key as the kind the command needs: `None` for a missing key, an
/// error for another kind. `Kind<'a, T>` is a name for the shape all three helpers return.
type Kind<'a, T> = Result<Option<&'a T>, WrongType>;

/// The list under key, an empty one when the key is missing, or the wrong kind.
fn list_of<'a>(store: &'a dyn Store, key: &[u8]) -> Kind<'a, VecDeque<Vec<u8>>> {
    match store.value(key) {
        None => Ok(None),
        Some(Value::List(l)) => Ok(Some(l)),
        Some(_) => Err(WrongType),
    }
}

fn llen(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
    match list_of(store, &args[1]) {
        Ok(l) => resp::write_int(w, l.map_or(0, VecDeque::len) as i64),
        Err(WrongType) => write_wrongtype(w),
    }
}

/// LRANGE key start end: a slice of the list by position. `range` on a `VecDeque` is
/// an iterator over the items in that range, and `write_bulks` drains it.
fn lrange(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
    let (Some(start), Some(end)) = (parse_int(&args[2]), parse_int(&args[3])) else {
        return resp::write_error(w, "ERR value is not an integer or out of range");
    };
    let list = match list_of(store, &args[1]) {
        Ok(Some(l)) => l,
        Ok(None) => return resp::write_array_len(w, 0),
        Err(WrongType) => return write_wrongtype(w),
    };
    match clamp(start, end, list.len()) {
        Some(range) => write_bulks(w, list.range(range).map(Vec::as_slice)),
        None => resp::write_array_len(w, 0),
    }
}
rust/ch08/src/cmd.rs · region lists

Closures and iterators, named at last#

Rust has written iterators since chapter 4 without naming them. An iterator is any value with a next method that gives the next item or None. Every adaptor comes with that one method. map changes each item and filter keeps some. skip and take cut a range. flat_map turns each item into several, and collect builds a collection from what is left. Each adaptor wraps the one before and does nothing until something at the end asks for an item. That is laziness, and the scan below depends on it. skip(cursor) and take(count) describe a page of a million keys without touching the other keys:

/// A HashMap walks in one fixed order while it is not changed, so the cursor is a
/// count of keys already seen. `skip` and `take` are lazy: nothing runs until
/// `collect` asks, and the map is never copied. One more `next` tells whether a
/// page follows.
fn scan(&self, cursor: usize, count: usize) -> (usize, Vec<&[u8]>) {
    let mut rest = self.map.keys().skip(cursor);
    let page: Vec<&[u8]> = rest.by_ref().take(count).map(Vec::as_slice).collect();
    let next = if rest.next().is_some() {
        cursor + page.len()
    } else {
        0
    };
    (next, page)
}
rust/ch08/src/store.rs · region scan

The |k| glob(p, k) inside a filter is a closure, a function written where it is used, which can capture the variables around it. A closure has one of three traits, by what it does with what it captured. Fn only reads, FnMut changes something, and FnOnce consumes it and can run once. filter asks for an FnMut, so a closure that counts as it filters is allowed. sadd uses that to count the members that were new:

fn sadd(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
    let key = mem::take(&mut args[1]);
    let Value::Set(set) = store.value_or_insert(key, || Value::Set(HashSet::new())) else {
        return write_wrongtype(w);
    };
    let added = args.drain(2..).filter(|m| set.insert(m.clone())).count();
    resp::write_int(w, added as i64)
}

fn set_of<'a>(store: &'a dyn Store, key: &[u8]) -> Kind<'a, HashSet<Vec<u8>>> {
    match store.value(key) {
        None => Ok(None),
        Some(Value::Set(s)) => Ok(Some(s)),
        Some(_) => Err(WrongType),
    }
}

fn sismember(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
    match set_of(store, &args[1]) {
        Ok(s) => resp::write_int(w, i64::from(s.is_some_and(|s| s.contains(&args[2])))),
        Err(WrongType) => write_wrongtype(w),
    }
}

fn smembers(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
    match set_of(store, &args[1]) {
        Ok(Some(s)) => write_bulks(w, s.iter().map(Vec::as_slice)),
        Ok(None) => resp::write_array_len(w, 0),
        Err(WrongType) => write_wrongtype(w),
    }
}
rust/ch08/src/cmd.rs · region sets
Exercise 08.3RustFlatten a hash into one array

Write hgetall. Get the hash with hash_of. Reply with an empty array when the key is missing and with the error when it holds another kind. Then build a Vec of borrowed slices, two per field, with iter and flat_map. Collect it, and write it with write_bulks. The turbofish on collect is optional here, because the type of the variable says what to collect into.

fn hgetall(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
    let hash = match hash_of(store, &args[1]) {
        Ok(Some(h)) => h,
        Ok(None) => return resp::write_array_len(w, 0),
        Err(WrongType) => return write_wrongtype(w),
    };
    // your code here
}
Stuck?Show the full listing
fn hset(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
    if !args.len().is_multiple_of(2) {
        return write_arity(w, b"hset");
    }
    let key = mem::take(&mut args[1]);
    let Value::Hash(hash) = store.value_or_insert(key, || Value::Hash(HashMap::new())) else {
        return write_wrongtype(w);
    };
    let mut added = 0;
    let mut pairs = args.drain(2..);
    while let (Some(field), Some(val)) = (pairs.next(), pairs.next()) {
        added += usize::from(hash.insert(field, val).is_none());
    }
    resp::write_int(w, added as i64)
}

fn hash_of<'a>(store: &'a dyn Store, key: &[u8]) -> Kind<'a, HashMap<Vec<u8>, Vec<u8>>> {
    match store.value(key) {
        None => Ok(None),
        Some(Value::Hash(h)) => Ok(Some(h)),
        Some(_) => Err(WrongType),
    }
}

fn hget(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
    match hash_of(store, &args[1]) {
        Ok(h) => match h.and_then(|h| h.get(&args[2])) {
            Some(v) => resp::write_bulk(w, v),
            None => resp::write_nil(w),
        },
        Err(WrongType) => write_wrongtype(w),
    }
}

/// HGETALL: every field and value, as one flat array. `flat_map` turns each pair into
/// two items, and the count is twice the number of fields.
fn hgetall(args: &mut Command, store: &mut dyn Store, w: &mut dyn Write) -> io::Result<()> {
    let hash = match hash_of(store, &args[1]) {
        Ok(Some(h)) => h,
        Ok(None) => return resp::write_array_len(w, 0),
        Err(WrongType) => return write_wrongtype(w),
    };
    let flat: Vec<&[u8]> = hash
        .iter()
        .flat_map(|(f, v)| [f.as_slice(), v.as_slice()])
        .collect();
    write_bulks(w, flat.into_iter())
}
rust/ch08/src/cmd.rs · region hashes

Writing an iterator of your own is one method. The file chain.rs in the snapshot is the C chain of chapter 6 in Rust, kept for two lessons the server did not need. A type that contains itself must do so through a Box, a pointer to the heap that owns what it points to. The compiler must know the size of a value, and a Chain inside a Chain has none. And a type becomes an iterator by implementing next. Every adaptor then works on it, and so does for:

/// A recursive enum. Without the `Box`, a `Chain` would contain a `Chain` directly, and
/// the compiler could not say how large one is. `Box` puts the next link on the heap,
/// which makes the size one pointer, and it owns that link, so dropping the head drops
/// the whole chain.
pub enum Chain {
    Link(Vec<u8>, Box<Chain>),
    End,
}

impl Chain {
    pub fn push(self, item: Vec<u8>) -> Chain {
        Chain::Link(item, Box::new(self))
    }
}

/// An iterator over a chain: a value with one method, `next`, which gives the next item
/// or `None`. Everything else, `map`, `filter`, `count` and the `for` loop, comes with
/// the trait. The iterator takes the chain apart as it goes, so it owns it.
pub struct IntoIter(Chain);

impl Iterator for IntoIter {
    type Item = Vec<u8>;

    fn next(&mut self) -> Option<Vec<u8>> {
        match std::mem::replace(&mut self.0, Chain::End) {
            Chain::Link(item, rest) => {
                self.0 = *rest;
                Some(item)
            }
            Chain::End => None,
        }
    }
}

impl IntoIterator for Chain {
    type Item = Vec<u8>;
    type IntoIter = IntoIter;

    fn into_iter(self) -> IntoIter {
        IntoIter(self)
    }
}
rust/ch08/src/chain.rs · region chain

Three methods give three kinds of iterator over a collection. iter borrows each item, iter_mut borrows each mutably, and into_iter takes the collection apart and gives the items away. The chain only has the last, because its next consumes a link.

โš ๏ธAn iterator that nobody consumes does nothing Rust

keys.iter().map(|k| store.del(k)); deletes nothing. The adaptor builds a value that would delete on each next, and no one calls next. The compiler warns that the iterator is unused, and the book's build makes the warning an error. A side effect wants a for loop, or a call such as count() at the end, as del in this chapter writes it.

Iteration, three ways#

Every collection in this chapter is walked three times, once per language, and the three walks are the three answers to one question. Who holds the loop?

Walking a collectionCGoRust
Who holds the loopthe collection, which calls backthe collection, which calls yieldthe caller, which calls next
What the caller suppliesa function pointer and a void * contextthe loop body, through rangenothing, and adaptors between
Stopping earlythe callback returns falsethe body returns, and yield returns falsethe caller stops calling next
Composing two walksby hand, with a second callbackby hand, or with the iter packagefilter on map on skip, for free
Where the state livesin the context structin the closure the collection madein the iterator value
The cost of a stepan indirect callan indirect call, which the compiler often inlinesnothing: the chain compiles to a loop

The C and Go answers are the same shape, a collection that runs the loop and hands each item to a function. The Rust answer inverts it. The caller runs the loop, and the collection only knows how to give the next item. That inversion is why Rust iterators compose. A filter does not need to know what it filters, only how to ask for the next item and whether to pass it on. The slices and maps packages of Go now take and return an iterator function, a decade after Rust. C never will.

Verdict##

TopicCGoRustWhy
A value of several kindsWeakOKStrongC has a tag nothing checks, and Go has a type switch nothing checks. Rust has a match the compiler checks.
Adding a kind laterOKWeakStrongC warns at each switch with no default, and Go falls to default in silence. Rust refuses to build until every match knows.
A list, a hash, a setWeakStrongStrongC writes each, or nests the table. Go and Rust have them in the standard library.
Walking a collectionWeakOKStrongC needs a callback and a context struct, and Go has range over a function. Rust has adaptors that compose.
Scanning by cursorStrongOKOKC and the hand-written table scan buckets. The Go map sorts, and the Rust map counts from the start.
Reading the codeWeakStrongOKThirteen C commands need a helper each, and Go reads like prose. Rust reads well once match and let else are familiar.
Carry forward
  • Sum type. One value that is one of several shapes, with a tag that says which. C checks nothing, Go checks at a type switch, and Rust checks at compile time.
  • Cursor. A number the client sends back so the server can continue a walk where it stopped, without holding the whole list.
  • Iterator. A value with a next method, which every adaptor wraps and nothing runs until something asks.
Next

Fifty clients connect to the server at once and send a command each. One of them is answered. The other forty-nine are still waiting when it has finished.

Many clients#

Building kv · Chapter 09· 47 min read· 760 lines of code
▶Scene

Start the chapter 8 server and point redis-benchmark at it with its default fifty connections. Then, from a second terminal, ask it one question:

$ redis-benchmark -p 6380 -c 50 -n 1000 -t ping -q
PING_INLINE: rps=0.0 (overall: 262.2) avg_msec=0.000 (overall: 0.043)
$ redis-cli -p 6380 PING                              # terminal two
Could not connect to Redis at 127.0.0.1:6380: Operation timed out

The benchmark answered a few hundred requests in its first moment and then nothing. It never finishes. The second terminal waited fifteen seconds and could not even connect. One client is being served. Forty-nine are waiting, and the server does not know they exist.

kv serves one client at a time. This chapter makes it serve thousands. C gets an event loop on one thread, with the kernel's own list of ready sockets. Go gets a goroutine for each client and a load generator built from channels. Rust gets a thread for each client, the same tool built from scoped threads, and then the server on tokio.

The scene has a precise shape. The chapter 8 server accepts one connection and serves it until it closes. The benchmark keeps every connection open until the whole run ends, so the first client is never done. Behind it, the kernel keeps a queue of connections that are complete but not yet accepted. The server chose its length with listen(fd, 16) in chapter 3, so sixteen clients sit in that queue. The other thirty-three sent their first packet and got no answer at all. The kernel drops a connection it cannot queue, and the client tries again a second later, and again. That is the fifteen seconds.

There are two ways out, and the three languages take both. The first is to keep one thread and never block it. The thread asks the operating system which sockets have bytes waiting, reads from those, and asks again. That is an event loop, and it is what Redis does. The second is to give each client its own thread of control and let it block. The operating system or a runtime then switches between them. Go does that with goroutines. Rust does it with threads first, and then with tasks on an asynchronous runtime. That runtime is an event loop written as if each client had its own thread.

C: one thread client 1 client 2 client 3 client 50 ... poll() one thread "which socket is ready?" 1 core, 200 lines Go: goroutines g1 g2 g3 g50 one for each client, a few KB each the Go runtime: scheduler + kqueue/epoll thread thread thread one store, one RWMutex every core, 1 keyword Rust: tokio tasks t1 t2 t3 t50 one for each client, a few kilobytes tokio: scheduler + kqueue/epoll, a crate thread thread thread one store, Arc<Mutex>, checked every core, 1 dependency
Three ways to serve fifty clients. C asks poll which sockets are ready and serves them from one thread. Go gives each client a goroutine and the runtime spreads them over threads. Rust with tokio gives each client a task, and the runtime is a crate rather than part of the language.
ConcurrencyCGoRust
One client isa heap struct the poller hands backa goroutinea thread, then a task
Who waitsone call, for all clients at onceeach goroutine, on its own readeach thread on its read, then the tokio runtime
Which sockets are readypoll, kqueue or epoll, chosen by #ifdefthe runtime asks, you never see itthe runtime asks, you never see it
Platform code lives ina file per system, under #ifdefa file per system, with a //go:build lineone function per system, with #[cfg]
Memory for a clienttwo buffers you sizea stack of a few kilobytes that growsa 2 MiB stack, then a task of a few kilobytes, mostly its buffers
The store is shared bynobody, there is one threada sync.RWMutexan Arc<Mutex<Store>>
Forgetting the locknot possiblea data race, which go test -race may finddoes not compile
Fifty workers talk throughnothing: C has no load generator herechannels and a WaitGroupan mpsc channel and a scope
Using all the coresnoyes, the runtime spreads goroutinesyes, with threads or with tokio
๐Ÿ“œReadiness and completion: two shapes of the same question

The question "which socket can I use now?" got different answers on different systems, and every language has to hide the difference. Unix from Berkeley, the Berkeley Software Distribution (BSD), answered first with select. poll came later and takes an array instead of a fixed-size set of bits. FreeBSD 4.1 added kqueue, which is what macOS uses. Linux added epoll in October 2002. All of those report readiness. They say "this socket has bytes", and the program then reads. Windows NT 3.5 took the other shape with I/O completion ports. A program hands the operating system a buffer and a read request, and the port reports completion. It says "this read has finished, and the bytes are in your buffer". The Go runtime and tokio both contain a layer that makes the two shapes look the same. That is why the Go and Rust versions of kv run on Windows unchanged, and the C one does not.

C: one thread, and a poller to say who is ready#

The loop keeps a structure for each client, holding the socket, the bytes read so far, and the replies not yet written. The buffers are the ones from chapter 4. The clients live on the heap, one allocation each, and the loop keeps an array of pointers to them. The poller in the next section stores a pointer to each client, so a client must never move:

/* One client: its socket, the bytes read so far, and the replies not yet written. */
typedef struct {
    int fd;
    kv_buf in;
    kv_buf out;
    bool writing; /* the poller is watching for room to write */
    bool closing; /* after a protocol error: close once out is written */
    bool dead;    /* gone; removed once the batch of events it was part of is done */
} kv_conn;

/* Every client, beside the poller that watches their sockets. The array holds
 * pointers, so a client stays where it is in memory when another one is removed, and
 * the tag the poller hands back stays valid. */
typedef struct {
    kv_conn **conns;
    size_t len;
    size_t cap;
    kv_poller *poller;
} kv_clients;
c/ch09/src/server.c · region conn

Every socket in the loop is non-blocking. A blocking read on a socket with no bytes would stop the one thread, and every other client with it. A non-blocking one returns -1 with errno set to EAGAIN, which means "nothing right now", and the loop moves on:

/* A non-blocking socket returns EAGAIN instead of waiting, which the loop relies on. */
static bool kv_set_nonblocking(int fd) {
    int flags = fcntl(fd, F_GETFL, 0);
    return flags >= 0 && fcntl(fd, F_SETFL, flags | O_NONBLOCK) >= 0;
}
c/ch09/src/server.c · region nonblock

Accepting clients follows the same rule. When the poller says the listening socket is ready, there may be several connections waiting, so the loop accepts until accept itself says EAGAIN:

/* Accept every waiting client. The listening socket is non-blocking, so accept()
 * returns EAGAIN when the queue is empty. */
static void kv_accept_all(int listen_fd, kv_clients *cs) {
    for (;;) {
        int fd = accept(listen_fd, NULL, NULL);
        if (fd < 0) {
            if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) {
                return;
            }
            perror("accept");
            return;
        }
        if (!kv_set_nonblocking(fd) || !kv_clients_add(cs, fd)) {
            close(fd);
        }
    }
}
c/ch09/src/server.c · region accept

Reading from a ready client follows the pattern once more. Read until the socket is empty, then run every complete request in the buffer and queue the replies in the output buffer. The parser from chapter 4 was written for exactly this moment. Its KV_PARSE_INCOMPLETE status looked like extra work in a one-client server. Here it lets the loop stop in the middle of a request and come back later:

/* Read what has arrived, run every complete request, and queue the replies. Returns
 * false when the client has gone or must be dropped. */
static bool kv_conn_input(kv_conn *c, kv_store *store) {
    char tmp[4096];
    for (;;) {
        ssize_t n = read(c->fd, tmp, sizeof tmp);
        if (n < 0) {
            if (errno == EAGAIN || errno == EWOULDBLOCK) {
                break; /* nothing more for now */
            }
            if (errno == EINTR) {
                continue;
            }
            return false;
        }
        if (n == 0) {
            return false; /* the client closed */
        }
        if (!kv_buf_append(&c->in, tmp, (size_t)n)) {
            return false;
        }
    }
    for (;;) {
        kv_cmd cmd;
        size_t used = 0;
        kv_parse_status st = kv_parse(c->in.data, c->in.len, &cmd, &used);
        if (st == KV_PARSE_INCOMPLETE && c->in.len > KV_MAX_REQUEST) {
            st = KV_PARSE_BAD; /* a request that never ends is not a request */
        }
        /* No default: with -Wall the compiler reports a status this switch forgot. */
        switch (st) {
        case KV_PARSE_INCOMPLETE:
            return true; /* the poller says when more bytes arrive */
        case KV_PARSE_BAD:
            kv_reply_error(&c->out, "ERR Protocol error");
            c->closing = true;
            return true;
        case KV_PARSE_OK:
            if (!kv_exec(&cmd, store, &c->out)) {
                return false;
            }
            kv_buf_consume(&c->in, used);
            break;
        }
    }
}
c/ch09/src/server.c · region input
Exercise 09.1CWrite what the socket will take

Write kv_conn_output. Loop while the output buffer holds bytes. Call write with everything that is queued, and consume as many bytes as it took. When it returns EAGAIN, return true and stop, because the socket is full and the poller will say when it has room. Retry on EINTR. Return false on any other error.

static bool kv_conn_output(kv_conn *c) {
    while (c->out.len > 0) {
        /* your code here */
    }
    return true;
}
Stuck?Show the full listing
/* Write as much of the queued reply as the socket will take right now. */
static bool kv_conn_output(kv_conn *c) {
    while (c->out.len > 0) {
        ssize_t w = write(c->fd, c->out.data, c->out.len);
        if (w < 0) {
            if (errno == EAGAIN || errno == EWOULDBLOCK) {
                return true; /* the socket is full: the poller will say when to continue */
            }
            if (errno == EINTR) {
                continue;
            }
            return false;
        }
        kv_buf_consume(&c->out, (size_t)w);
    }
    return true;
}
c/ch09/src/server.c · region output

One function serves a client that the poller reported. It reads if there is input and writes if a reply is queued. Then it does two things that every event loop has to get right. It asks to be told about room to write only while a reply is waiting, because a socket is almost always writable. And it does not free a finished client. It marks it dead, because the batch of events may hold a second event for the same client:

/* One client had an event. Read if there is input, write if there is room or a reply is
 * queued, then ask to be told about room only while a reply is waiting. A socket is
 * almost always writable, so asking every time would make every wait return at once. */
static void kv_conn_ready(kv_clients *cs, kv_conn *c, const kv_event *ev, kv_store *store) {
    bool alive = true;
    if (ev->readable || ev->hangup) {
        alive = kv_conn_input(c, store);
    }
    if (alive && (ev->writable || c->out.len > 0)) {
        alive = kv_conn_output(c);
    }
    if (!alive || (c->closing && c->out.len == 0)) {
        c->dead = true; /* not freed yet: this batch may hold another event for it */
        return;
    }
    bool want = c->out.len > 0;
    if (want != c->writing && kv_poller_want_write(cs->poller, c->fd, c, want)) {
        c->writing = want;
    }
}
c/ch09/src/server.c · region ready

The loop itself is short. Wait, walk the batch, accept when the listening socket is ready and serve the client otherwise, then remove the dead. The removal walks backwards, so that moving the last client into a hole does not skip anyone:

int kv_server_run(int listen_fd, kv_store *store) {
    kv_clients cs = {0};
    cs.poller = kv_poller_new();
    /* the listening socket is watched too, with a null tag to tell it from a client */
    if (cs.poller == NULL || !kv_set_nonblocking(listen_fd) ||
        !kv_poller_add(cs.poller, listen_fd, NULL)) {
        kv_poller_free(cs.poller);
        return 1;
    }
    fprintf(stderr, "kv waits with %s\n", kv_poller_name());
    kv_event events[64];
    for (;;) {
        int n = kv_poller_wait(cs.poller, events, 64, -1);
        if (n < 0) {
            if (errno == EINTR) {
                continue;
            }
            perror("wait");
            break;
        }
        for (int i = 0; i < n; i++) {
            kv_conn *c = events[i].tag;
            if (c == NULL) {
                kv_accept_all(listen_fd, &cs);
            } else if (!c->dead) {
                kv_conn_ready(&cs, c, &events[i], store);
            }
        }
        /* walk backwards, because removing a client moves the last one into its slot */
        for (size_t i = cs.len; i-- > 0;) {
            if (cs.conns[i]->dead) {
                kv_clients_remove(&cs, i);
            }
        }
    }
    for (size_t i = cs.len; i-- > 0;) {
        kv_clients_remove(&cs, i);
    }
    free(cs.conns);
    kv_poller_free(cs.poller);
    return 1;
}
c/ch09/src/server.c · region loop

About two hundred lines of new code, and the server now serves fifty clients from one thread. It also uses one core, which is the limit of this design, and the one Redis still lives with for running commands. The main function shrinks to a call, with one addition. Every client is an open file, and a process starts with a soft limit on those, often 256 on macOS and 1,024 on Linux. The server raises its limit to the hard one, as Redis does, and says how many clients that leaves room for:

/* Every client is an open file, and a process starts with a soft limit on those, often
 * 256 on macOS and 1024 on Linux. Raise it to the hard limit, as Redis does, and say
 * how many clients that leaves room for. */
static void kv_raise_fd_limit(void) {
    struct rlimit lim;
    if (getrlimit(RLIMIT_NOFILE, &lim) < 0) {
        return;
    }
    if (lim.rlim_cur < lim.rlim_max && setrlimit(RLIMIT_NOFILE, &(struct rlimit){lim.rlim_max, lim.rlim_max}) == 0) {
        lim.rlim_cur = lim.rlim_max;
    }
    if (lim.rlim_cur == RLIM_INFINITY) {
        fprintf(stderr, "kv has no limit on open files\n");
    } else {
        /* the listener, the poller, stdio and some spare take the rest */
        fprintf(stderr, "kv can hold %llu clients\n", (unsigned long long)lim.rlim_cur - 32);
    }
}
c/ch09/src/main.c · region fdlimit
โš ๏ธOne blocking call stops every client C

Forget O_NONBLOCK on one socket and a slow client stops the server. The poller says the socket is readable, and the loop calls read. A client that sent half a request and went away keeps the one thread waiting for the other half. The other forty-nine clients see a server that has stopped answering. Nothing crashes and nothing is reported, which makes it one of the harder bugs to find in a C server. That is one reason the kv_set_nonblocking call is the first thing done with every new socket.

C again: poll, kqueue and epoll behind one header#

The loop above never names poll. It talks to a poller, which is six functions in a header, and the header decides which of three files provides them. The preprocessor decides, from macros the compiler defines for the target. There is no runtime check and no configuration file. The files that do not match compile to nothing, because everything in them sits under an #ifdef:

/* One of poller_poll.c, poller_kqueue.c and poller_epoll.c holds the code; the other two
 * compile to nothing. The compiler predefines a macro for the target: __linux__ on
 * Linux, __APPLE__ on macOS, __FreeBSD__ on FreeBSD. KV_POLL on the command line, as in
 * `make kv KV_POLL=1`, forces the portable poll() version on any system. */
#if defined(KV_POLL)
#define KV_POLLER_POLL 1
#elif defined(__linux__)
#define KV_POLLER_EPOLL 1
#elif defined(__APPLE__) || defined(__FreeBSD__)
#define KV_POLLER_KQUEUE 1
#else
#define KV_POLLER_POLL 1
#endif
c/ch09/src/poller.h · region choose

The six functions are the smallest shape the three system calls have in common. A socket is added with a tag, which comes back in every event for that socket, and the tag is the client pointer. A wait fills an array of events and says how many:

typedef struct kv_poller kv_poller;

/* What a wait reports for one socket: the tag given when it was added, and which of
 * the three things happened to it. */
typedef struct {
    void *tag;
    bool readable;
    bool writable;
    bool hangup;
} kv_event;

kv_poller *kv_poller_new(void);
void kv_poller_free(kv_poller *p);

/* Watch fd for input. The tag comes back in every event for fd. */
bool kv_poller_add(kv_poller *p, int fd, void *tag);

/* Also watch fd for room to write, or stop watching for it. */
bool kv_poller_want_write(kv_poller *p, int fd, void *tag, bool want);

/* Stop watching fd. Call it before close(). */
void kv_poller_remove(kv_poller *p, int fd);

/* Block until a socket is ready or timeout_ms passes, where -1 means wait for ever.
 * Fills up to max events and returns how many, 0 on timeout, or -1 with errno set. */
int kv_poller_wait(kv_poller *p, kv_event *events, int max, int timeout_ms);

/* "poll", "kqueue" or "epoll", for the startup line. */
const char *kv_poller_name(void);
c/ch09/src/poller.h · region api

poll is in POSIX, the Portable Operating System Interface, so its file builds anywhere, and make kv KV_POLL=1 asks for it on any system. Its cost is that every call gives the kernel the whole array to walk. So one wait costs more with every client. kqueue on macOS and epoll on Linux keep the list in the kernel. Adding a socket is one call. A wait sends nothing and gets back only the sockets that changed. The Linux file is the shorter one, and its state is one descriptor:

/* The whole state is one descriptor: the kernel holds the list. */
struct kv_poller {
    int ep;
};

kv_poller *kv_poller_new(void) {
    kv_poller *p = malloc(sizeof *p);
    if (p == NULL) {
        return NULL;
    }
    p->ep = epoll_create1(0);
    if (p->ep < 0) {
        free(p);
        return NULL;
    }
    return p;
}

void kv_poller_free(kv_poller *p) {
    if (p != NULL) {
        close(p->ep);
        free(p);
    }
}

/* One socket has one entry with one mask of events, so a change replaces the mask. */
static bool kv_ep_ctl(kv_poller *p, int op, int fd, uint32_t mask, void *tag) {
    struct epoll_event change = {.events = mask, .data.ptr = tag};
    return epoll_ctl(p->ep, op, fd, &change) == 0;
}

bool kv_poller_add(kv_poller *p, int fd, void *tag) {
    return kv_ep_ctl(p, EPOLL_CTL_ADD, fd, EPOLLIN, tag);
}

bool kv_poller_want_write(kv_poller *p, int fd, void *tag, bool want) {
    return kv_ep_ctl(p, EPOLL_CTL_MOD, fd, EPOLLIN | (want ? EPOLLOUT : 0), tag);
}

/* close() drops a socket from the list by itself, unless another descriptor refers to
 * the same socket, which is why the explicit call is the safe habit. */
void kv_poller_remove(kv_poller *p, int fd) {
    epoll_ctl(p->ep, EPOLL_CTL_DEL, fd, NULL);
}

int kv_poller_wait(kv_poller *p, kv_event *events, int max, int timeout_ms) {
    struct epoll_event ready[64];
    if (max > 64) {
        max = 64;
    }
    int n = epoll_wait(p->ep, ready, max, timeout_ms);
    if (n <= 0) {
        return n;
    }
    for (int i = 0; i < n; i++) {
        uint32_t mask = ready[i].events;
        events[i] = (kv_event){
            .tag = ready[i].data.ptr,
            .readable = (mask & EPOLLIN) != 0,
            .writable = (mask & EPOLLOUT) != 0,
            .hangup = (mask & (EPOLLHUP | EPOLLERR)) != 0,
        };
    }
    return n;
}

const char *kv_poller_name(void) {
    return "epoll";
}
c/ch09/src/poller_epoll.c · region epoll

The designated initialisers, .events = mask and .data.ptr = tag, name the fields they set and leave the rest zero. They are the C way to build a struct without remembering the order of its fields. Redis has the same three files and a fourth for select, and picks one the same way.

Exercise 09.2CWrite the kqueue poller

Write poller_kqueue.c for macOS and FreeBSD. The whole state is one descriptor from kqueue(). One call, kevent, both changes the list and waits on it. So write a helper that makes one change with EV_SET and asks for no results. A filter is one kind of event on one socket. Watching a socket for reading and for writing is two entries, and a wait can report the socket twice. The loop in the previous section already copes with that. Treat EV_DELETE of a filter that was never added as success.

struct kv_poller {
    int kq;
};

static bool kv_kq_change(kv_poller *p, int fd, int16_t filter, uint16_t flags, void *tag) {
    /* your code here: EV_SET, then kevent with one change and no results */
}
Stuck?Show the full listing
/* The whole state is one descriptor: the kernel holds the list. */
struct kv_poller {
    int kq;
};

kv_poller *kv_poller_new(void) {
    kv_poller *p = malloc(sizeof *p);
    if (p == NULL) {
        return NULL;
    }
    p->kq = kqueue();
    if (p->kq < 0) {
        free(p);
        return NULL;
    }
    return p;
}

void kv_poller_free(kv_poller *p) {
    if (p != NULL) {
        close(p->kq);
        free(p);
    }
}

/* kevent() both changes the list and waits on it. This call only changes it: one
 * entry, no room for results, no timeout. A filter is one kind of event on one socket,
 * so a socket watched for reading and for writing is two entries. */
static bool kv_kq_change(kv_poller *p, int fd, int16_t filter, uint16_t flags, void *tag) {
    struct kevent change;
    EV_SET(&change, (uintptr_t)fd, filter, flags, 0, 0, tag);
    if (kevent(p->kq, &change, 1, NULL, 0, NULL) < 0) {
        /* deleting a filter that was never added is not an error worth stopping for */
        return (flags & EV_DELETE) && errno == ENOENT;
    }
    return true;
}

bool kv_poller_add(kv_poller *p, int fd, void *tag) {
    return kv_kq_change(p, fd, EVFILT_READ, EV_ADD, tag);
}

bool kv_poller_want_write(kv_poller *p, int fd, void *tag, bool want) {
    return kv_kq_change(p, fd, EVFILT_WRITE, want ? EV_ADD : EV_DELETE, tag);
}

/* close() drops a socket from every kqueue by itself; this only makes it explicit. */
void kv_poller_remove(kv_poller *p, int fd) {
    kv_kq_change(p, fd, EVFILT_READ, EV_DELETE, NULL);
    kv_kq_change(p, fd, EVFILT_WRITE, EV_DELETE, NULL);
}

int kv_poller_wait(kv_poller *p, kv_event *events, int max, int timeout_ms) {
    struct kevent ready[64];
    if (max > 64) {
        max = 64;
    }
    struct timespec ts = {timeout_ms / 1000, (timeout_ms % 1000) * 1000000L};
    int n = kevent(p->kq, NULL, 0, ready, max, timeout_ms < 0 ? NULL : &ts);
    if (n <= 0) {
        return n;
    }
    /* A socket ready for both reading and writing is two entries, with the same tag.
     * The loop that gets them copes with that, so they are passed on as they are. */
    for (int i = 0; i < n; i++) {
        events[i] = (kv_event){
            .tag = ready[i].udata,
            .readable = ready[i].filter == EVFILT_READ,
            .writable = ready[i].filter == EVFILT_WRITE,
            .hangup = (ready[i].flags & (EV_EOF | EV_ERROR)) != 0,
        };
    }
    return n;
}

const char *kv_poller_name(void) {
    return "kqueue";
}
c/ch09/src/poller_kqueue.c · region kqueue

Build the server both ways, and the startup line names the poller. At fifty clients they run at the same speed, because fifty sockets are not enough for the cost of poll to show:

$ make kv && ./kv
kv waits with kqueue
$ make clean && make kv KV_POLL=1 && ./kv
kv waits with poll
๐Ÿ–ฅWindows, and the loop turned inside out

The three pollers all report readiness, so one loop fits all of them. Windows reports completion, so a fourth file would not be enough. The loop itself would change, and chapter 14 comes back to what that means for porting the C version.

Go: one word, and one lock#

The Go change in main is the keyword go. The call serve(conn, s) runs on a new goroutine, and the loop goes straight back to Accept. The serving function is unchanged. It still blocks on its reads, which is fine, because it blocks only its own goroutine, and the runtime runs another:

func main() {
	ln, err := net.Listen("tcp", ":"+port())
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	fmt.Fprintln(os.Stderr, "kv listening on port", port())
	if n, ok := sys.OpenFiles(); ok {
		fmt.Fprintln(os.Stderr, "kv can hold", n-32, "clients")
	}
	s := store.New()
	for {
		conn, err := ln.Accept()
		if err != nil {
			fmt.Fprintln(os.Stderr, "accept:", err)
			continue
		}
		// the one word that changes: the client is served on its own goroutine, and
		// the loop goes straight back to accept
		go func() {
			if err := serve(conn, s); err != nil {
				fmt.Fprintln(os.Stderr, "client:", err)
			}
		}()
	}
}
go/ch09/cmd/kv/main.go · region main

A goroutine is not a thread. The Go FAQ says one costs "little overhead beyond the memory for the stack", which is a few kilobytes. It also says it "is practical to create hundreds of thousands of goroutines in the same address space". The event loop that C needed is inside the runtime, and it talks to kqueue or epoll or a completion port on your behalf.

The runtime in one paragraph#

As many threads run Go code as GOMAXPROCS allows, which by default is the number of CPUs the process may use. Each thread takes a goroutine from a queue and runs it until it blocks, or for ten milliseconds. A goroutine that blocks on a socket is parked, and the runtime's poller queues it again when the socket is ready. One that blocks in a system call keeps its thread, and the runtime starts another. None of this appears in the code, and all of it appears in the numbers at the end of the chapter.

The price appears in the store. Fifty goroutines now call Set and Get at the same time, and a Go map is not safe for that. The language FAQ says that a mutex on every map operation "would slow down most programs and add safety to few". So the store adds its own, a read-write lock. Readers take RLock and may hold it together. Writers take Lock and hold it alone:

// Store keeps every key and its value in a map, behind a lock. Many goroutines read and
// write the store at once, and a Go map is not safe for that without one. The value is
// any of the four kinds in value.go.
type Store struct {
	mu sync.RWMutex
	m  map[string]any
}

func New() *Store {
	return &Store{m: make(map[string]any)}
}

// Get takes the read lock, which any number of readers may hold together.
func (s *Store) Get(key []byte) (any, bool) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	v, ok := s.m[string(key)]
	return v, ok
}

// Set takes the write lock, which excludes every other reader and writer.
func (s *Store) Set(key []byte, val any) {
	s.mu.Lock()
	defer s.mu.Unlock()
	s.m[string(key)] = val
}

func (s *Store) Del(key []byte) bool {
	s.mu.Lock()
	defer s.mu.Unlock()
	_, ok := s.m[string(key)]
	delete(s.m, string(key))
	return ok
}

func (s *Store) Len() int {
	s.mu.RLock()
	defer s.mu.RUnlock()
	return len(s.m)
}

// Keys returns every key that matches the pattern, in no particular order.
func (s *Store) Keys(pattern []byte) [][]byte {
	s.mu.RLock()
	defer s.mu.RUnlock()
	var out [][]byte
	for k := range s.m {
		if glob(pattern, []byte(k)) {
			out = append(out, []byte(k))
		}
	}
	return out
}

// Scan returns a page of keys from the cursor on, in sorted order, and the cursor for
// the next page, or 0 when the walk is done.
func (s *Store) Scan(cursor uint64, count int) (uint64, [][]byte) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	sorted := slices.Sorted(maps.Keys(s.m))
	if cursor >= uint64(len(sorted)) {
		return 0, nil
	}
	end := min(int(cursor)+count, len(sorted))
	var page [][]byte
	for _, k := range sorted[cursor:end] {
		page = append(page, []byte(k))
	}
	if end == len(sorted) {
		return 0, page
	}
	return uint64(end), page
}
go/ch09/internal/store/store.go · region store
Exercise 09.3GoMake INCR atomic

Rewrite Incr so that it holds the write lock from the moment it reads the old value to the moment it writes the new one. Do not call Get and Set inside it. Each of those takes the lock again, and a Go lock cannot be taken twice by the same goroutine. Read and write the map directly.

func (s *Store) Incr(key []byte) (int64, error) {
	// your code here: lock, read s.m, parse, add, write s.m, unlock
}
Stuck?Show the full listing
// Incr holds the write lock from the read to the write, so that two clients that
// increment the same key at the same moment cannot both read the same old value.
func (s *Store) Incr(key []byte) (int64, error) {
	s.mu.Lock()
	defer s.mu.Unlock()
	var n int64
	if cur, ok := s.m[string(key)]; ok {
		b, isStr := cur.([]byte)
		if !isStr {
			return 0, ErrWrongType
		}
		var err error
		n, err = strconv.ParseInt(string(b), 10, 64)
		if err != nil {
			return 0, ErrNotInteger
		}
	}
	if n == math.MaxInt64 {
		return 0, ErrOverflow
	}
	n++
	s.m[string(key)] = []byte(strconv.FormatInt(n, 10))
	return n, nil
}
go/ch09/internal/store/store.go · region incr

Without the single lock, two clients can both read 41, both write 42, and one increment is lost. The test below adds one to the same key from fifty goroutines, two hundred times each, and expects exactly ten thousand. The WaitGroup counts the goroutines, and its Go method, new in Go 1.25, starts one and counts it in one call. Wait blocks until every one of them has returned:

// TestIncrFromManyGoroutines adds one to the same key from 50 goroutines, 200 times
// each. With the lock the result is 10000. Without it, go test -race reports the data
// race, and the count comes out short.
func TestIncrFromManyGoroutines(t *testing.T) {
	s := New()
	var wg sync.WaitGroup
	for range 50 {
		wg.Go(func() {
			for range 200 {
				if _, err := s.Incr([]byte("n")); err != nil {
					t.Error(err)
				}
			}
		})
	}
	wg.Wait()
	got, _ := s.Get([]byte("n"))
	if b, _ := got.([]byte); string(b) != "10000" {
		t.Fatalf("n = %s, want 10000", got)
	}
}
go/ch09/internal/store/store_test.go · region test
$ go test -race ./...
ok  	kv/internal/store	1.514s

The -race flag turns on the race detector. It watches every memory access during the test, and reports two goroutines that touch the same byte without a lock between them. Remove the lock from Incr and the test fails twice. The count comes out short, and the detector prints both stack traces. Keep the flag on in every test run, which the book's make verify does.

โš ๏ธThe race detector finds races that happen, not races that exist Go

A race the test never provokes is a race the detector never reports. The detector watches real accesses, so a missing lock on a path the test does not run stays hidden until production runs it. It is a tool for finding races, not a proof that there are none. The Rust compiler makes the other kind of claim later in this chapter.

Go again: fifty workers and three channels#

The scene said that one client was served and forty-nine waited. A tool that counts who got answered can prove it, and it is the best place to meet the rest of Go's concurrency. kvload is a worker pool. One goroutine for each connection, a channel of jobs they all take from, and channels of answers they all send to:

// answer is one round trip: which connection made it and how long it took.
type answer struct {
	worker int
	took   time.Duration
}

// worker owns one connection. It takes job after job from the channel until the channel
// is closed and empty, and reports each round trip on answers, or its first error on
// errs, after which it gives up.
func worker(id int, addr string, jobs <-chan int, answers chan<- answer, errs chan<- error) {
	conn, err := net.Dial("tcp", addr)
	if err != nil {
		errs <- fmt.Errorf("worker %d: %w", id, err)
		return
	}
	defer conn.Close()
	r := bufio.NewReader(conn)
	for range jobs {
		start := time.Now()
		if _, err := conn.Write([]byte("PING\r\n")); err != nil {
			errs <- fmt.Errorf("worker %d: %w", id, err)
			return
		}
		if _, err := r.ReadString('\n'); err != nil {
			errs <- fmt.Errorf("worker %d: %w", id, err)
			return
		}
		answers <- answer{id, time.Since(start)}
	}
}
go/ch09/cmd/kvload/main.go · region worker

The for range jobs loop receives from the channel until the channel is closed and empty, and then ends. The channel types in the signature say which way each one flows, and the compiler holds the worker to that. The main function fills the jobs channel, starts the workers, and then waits on three things at once:

func main() {
	addr := flag.String("addr", "127.0.0.1:6380", "the server")
	clients := flag.Int("c", 50, "connections")
	n := flag.Int("n", 10000, "requests in total")
	timeout := flag.Duration("timeout", 5*time.Second, "give up after this long")
	flag.Parse()

	// A buffered channel holds every job up front. Closing it tells the workers that
	// nothing more is coming: their range loops end once the buffer is empty.
	jobs := make(chan int, *n)
	for i := range *n {
		jobs <- i
	}
	close(jobs)

	answers := make(chan answer, *n)
	errs := make(chan error, *clients)
	var wg sync.WaitGroup
	for id := range *clients {
		wg.Go(func() { worker(id, *addr, jobs, answers, errs) })
	}
	// Only the sender may close a channel, and here there are many senders, so a
	// goroutine of its own waits for all of them and closes both channels after.
	go func() {
		wg.Wait()
		close(answers)
		close(errs)
	}()

	start := time.Now()
	var took []time.Duration
	byWorker := map[int]int{}
	failed := 0
	deadline := time.After(*timeout)
	// select waits on three channels at once. A closed channel is set to nil, and a nil
	// channel never fires, which takes its case out of the select until the loop ends.
	for answers != nil || errs != nil {
		select {
		case a, ok := <-answers:
			if !ok {
				answers = nil
				continue
			}
			took = append(took, a.took)
			byWorker[a.worker]++
		case err, ok := <-errs:
			if !ok {
				errs = nil
				continue
			}
			failed++
			fmt.Fprintln(os.Stderr, err)
		case <-deadline:
			// the workers still blocked on a read are left behind: the process is
			// about to exit, so their goroutines go with it
			fmt.Printf("gave up after %v\n", *timeout)
			answers, errs = nil, nil
		}
	}
	report(took, byWorker, *clients, failed, time.Since(start))
}
go/ch09/cmd/kvload/main.go · region main

Four rules of channels are in that function. A channel with a buffer holds that many values without a receiver, so jobs can be filled before any worker exists. Closing a channel is how a sender says "no more". The language specification says that "sending to or closing a closed channel causes a run-time panic". So only the side that sends may close, and only once. That is why a goroutine of its own waits for every worker and then closes the two result channels. A select waits on several channels and takes the first that is ready, and with a default case it does not wait at all. And a nil channel "is never ready for communication", which sounds useless until a case must be switched off. Setting a closed channel to nil is the idiom, and the loop ends when both are nil.

The deadline case is the one to study. time.After returns a channel that delivers once, after the duration. When it fires, the workers still blocked on a read are abandoned, and the program exits with them still there. That is a goroutine leak, the Go bug that replaces the C memory leak. A goroutine that waits for a reply that never comes lives until the process ends, with its stack. Here the process ends at once. In a server that runs for a year it matters, and the cure is a timeout or a context, which chapter 10 adds. Run the tool against the chapter 8 server and then against this one:

$ go run ./cmd/kvload                        # the chapter 8 server, one client at a time
10000 answered by 17 of 50 connections, 9984 of them by one, 0 failed, 2.07 s in all
p50 0.045 ms, max 539.894 ms
$ go run ./cmd/kvload                        # the chapter 9 server
10000 answered by 50 of 50 connections, 232 of them by one, 0 failed, 0.11 s in all
p50 0.525 ms, max 1.861 ms

Seventeen is one plus sixteen. One connection was served and took nearly every job. Sixteen sat in the kernel's queue with their first PING already sent, and were answered once each when the first one closed. The other thirty-three connected after the jobs were gone. The second run is the chapter. Every connection was served, in a tenth of a second.

The rest of the sync package#

The rest of the sync package is short, and kv needs none of it yet. Once "will perform exactly one action", however many goroutines call it. Pool is a cache of temporary objects, such as buffers, that relieves "pressure on the garbage collector". sync/atomic gives a counter or a flag that many goroutines may change without a lock. Its own documentation says most code should use channels or a mutex instead. The Go proverb behind all of it says to share memory by communicating, and not to communicate by sharing memory. The channels in kvload are that proverb, and the lock in the store is its exception.

Go's platform files: a line at the top#

C chose its poller with #ifdef. Go has no preprocessor, so it chooses files. A comment on the first line, before the package clause, says which systems a file belongs to, and the compiler ignores the file everywhere else. The one place kv needs it is the open-file limit. Since Go 1.19, programs that import the os package "automatically increase the open file limit (RLIMIT_NOFILE) to the maximum allowed value". Reading it back is a system call that Windows does not have, so there are two files. One for every Unix:

// OpenFiles reports how many files, and so how many clients, the process may hold open.
// The Go runtime has already raised the soft limit to the hard one at startup.
func OpenFiles() (uint64, bool) {
	var lim syscall.Rlimit
	if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &lim); err != nil {
		return 0, false
	}
	return lim.Cur, true
}
go/ch09/internal/sys/limit_unix.go · region unix

And one for everything else, with the opposite constraint, so that exactly one of them is ever in the build:

// OpenFiles reports false where the system has no such limit to ask about.
func OpenFiles() (uint64, bool) {
	return 0, false
}
go/ch09/internal/sys/limit_other.go · region other

A file name ending in _linux.go is the same constraint without the comment. GOOS=windows go vet ./... checks the other file without a Windows machine.

Rust: a thread for each client, and a compiler that checks the sharing#

The thread version is the Go version with the sharing made explicit. The store is wrapped in two types. An Arc counts how many owners the store has. A Mutex lets one thread at a time use it. Each new thread gets a clone of the Arc, which is a new owner of the same store:

fn main() -> io::Result<()> {
    let port = port();
    let listener = TcpListener::bind(format!("0.0.0.0:{port}"))?;
    eprintln!("kv listening on port {port}");
    if let Some(n) = kv::sys::open_files() {
        eprintln!("kv can hold {} clients", n - 32);
    }
    let store = Arc::new(Mutex::new(MapStore::new()));
    for stream in listener.incoming() {
        match stream {
            Ok(stream) => {
                // a clone of the Arc is a new owner of the same store, moved into the thread
                let store = Arc::clone(&store);
                thread::spawn(move || {
                    if let Err(e) = serve(stream, store) {
                        eprintln!("client: {e}");
                    }
                });
            }
            Err(e) => eprintln!("accept: {e}"),
        }
    }
    Ok(())
}
rust/ch09/src/main.rs · region main

Inside the serving function, the lock is taken for one command. It is released at the end of the line, when the guard goes out of scope. The unwrap_or_else handles a poisoned lock, which is what a mutex becomes when a thread panics while holding it. The store is still usable, so the server takes it anyway:

/// The store is shared: an Arc counts the owners, and a Mutex lets one thread at a
/// time use it. The lock is taken for one command and released at the end of the line.
fn serve(stream: TcpStream, store: Arc<Mutex<MapStore>>) -> resp::Result<()> {
    let mut reader = BufReader::new(stream.try_clone()?);
    let mut writer = BufWriter::new(stream);
    loop {
        match resp::read_command(&mut reader) {
            Ok(Some(args)) => {
                let mut store = store.lock().unwrap_or_else(|e| e.into_inner());
                cmd::execute(args, &mut *store, &mut writer)?;
            }
            Ok(None) => return Ok(()),
            Err(e @ resp::KvError::Protocol(_)) => {
                resp::write_error(&mut writer, "ERR Protocol error")?;
                writer.flush()?;
                return Err(e);
            }
            Err(e) => return Err(e),
        }
        if reader.buffer().is_empty() {
            writer.flush()?;
        }
    }
}
rust/ch09/src/main.rs · region serve

What happens without the Arc and the Mutex? The compiler refuses. Try to move one Store into two threads and it says error[E0382]: use of moved value: store. Try to share a reference instead and the compiler refuses that too, because the thread may outlive main. The only way through is a type that is safe to share, and Arc<Mutex<T>> is the simplest one. The rule is the one the borrow checker applied in chapter 5. A value can have many readers or one writer, never both, and a thread is only another reader or writer. A data race in a Rust program that compiles without unsafe is not possible. The race detector that Go runs at test time is work the Rust compiler does at compile time.

Send and Sync#

The check has a name, and you met the single-threaded half of it in chapter 6. Replace the Arc with an Rc. Replace the Mutex with a RefCell, which shares and mutates a value as well on one thread. The compiler says:

error[E0277]: `Rc<RefCell<MapStore>>` cannot be sent between threads safely
   --> src/main.rs:57:31
    |
 57 |                 thread::spawn(move || {
    |                 ------------- ^------
    = help: within `{closure@src/main.rs:57:31: 57:38}`, the trait `Send` is not implemented
            for `Rc<RefCell<MapStore>>`
note: required by a bound in `spawn`
    |
128 |     F: Send + 'static,

Send and Sync are two marker traits with no methods. Send means a value may move to another thread. Sync means a reference to it may be shared between threads. The compiler derives both for every type from its parts. A type that holds an Rc loses Send. The reason is that Rc counts its owners without atomic operations, and two threads updating that count at once "is undefined behavior". The spawn function demands Send of its closure, as the note says. So thread safety in Rust is two traits, derived by the compiler and demanded where a value crosses between threads. There is nothing to remember, and no way to forget.

Rust again: the load generator with scoped threads#

The Rust kvload has the shape of the Go one, with each Go idea replaced by the Rust one. A worker owns one connection. The jobs are a counter, an AtomicUsize, rather than a channel. Its fetch_add returns the old value, so every job number is given out once without a lock. The answers go back on an mpsc channel, which has many senders and one receiver, as its name says. The timeout is on the socket, for a reason that comes in a moment:

/// One connection. It takes job numbers from the shared counter until they run out and
/// sends each round trip on the channel. An error ends the worker, and the caller
/// reports it. The read timeout is what lets a worker give up on a server that never
/// answers, because a scoped thread must finish before the scope ends.
fn worker(
    id: usize,
    addr: SocketAddr,
    jobs: &AtomicUsize,
    total: usize,
    timeout: Duration,
    answers: &mpsc::Sender<Answer>,
) -> io::Result<()> {
    let mut stream = TcpStream::connect_timeout(&addr, timeout)?;
    stream.set_read_timeout(Some(timeout))?;
    let mut reader = BufReader::new(stream.try_clone()?);
    let mut line = String::new();
    // fetch_add returns the old value, so every job number is handed out exactly once
    while jobs.fetch_add(1, Ordering::Relaxed) < total {
        let start = Instant::now();
        stream.write_all(b"PING\r\n")?;
        line.clear();
        if reader.read_line(&mut line)? == 0 {
            return Err(io::ErrorKind::UnexpectedEof.into());
        }
        // send fails only when main has stopped listening, and then there is nothing to do
        let _ = answers.send(Answer {
            worker: id,
            took: start.elapsed(),
        });
    }
    Ok(())
}
rust/ch09/src/bin/kvload.rs · region worker

The main function is the interesting half. The workers borrow the counter and the sender from main's own stack frame, with no Arc and no cloning of anything but the sender. That is what thread::scope is for. The standard library says that "the scope guarantees all threads will be joined at the end of the scope". No thread outlives the data it borrows, so the borrow checker allows it:

fn main() -> io::Result<()> {
    let (addr, clients, total, timeout) = args();
    let jobs = AtomicUsize::new(0);
    let (tx, rx) = mpsc::channel();
    let failed = AtomicUsize::new(0);
    let start = Instant::now();
    // The scope borrows jobs, failed and tx from this stack frame, and the threads it
    // spawns can use them without an Arc, because the scope joins every thread before
    // it returns. No thread outlives the data.
    let mut took: Vec<Duration> = thread::scope(|s| {
        for id in 0..clients {
            let tx = tx.clone();
            let (jobs, failed) = (&jobs, &failed);
            s.spawn(move || {
                if let Err(e) = worker(id, addr, jobs, total, timeout, &tx) {
                    failed.fetch_add(1, Ordering::Relaxed);
                    eprintln!("worker {id}: {e}");
                }
            });
        }
        // Each worker holds a clone of the sender. Once main drops its own, the receiver
        // ends when the last worker drops the last clone, which ends the for loop.
        drop(tx);
        let mut took = Vec::new();
        let mut by_worker = vec![0usize; clients];
        for answer in rx {
            took.push(answer.took);
            by_worker[answer.worker] += 1;
        }
        let answered = by_worker.iter().filter(|n| **n > 0).count();
        let most = by_worker.iter().max().copied().unwrap_or(0);
        println!(
            "{} answered by {answered} of {clients} connections, {most} of them by one, {} failed, {:.2} s in all",
            took.len(),
            failed.load(Ordering::Relaxed),
            start.elapsed().as_secs_f64()
        );
        took
    });
    if !took.is_empty() {
        took.sort();
        let ms = |d: Duration| d.as_secs_f64() * 1e3;
        println!(
            "p50 {:.3} ms, max {:.3} ms",
            ms(took[took.len() / 2]),
            ms(took[took.len() - 1])
        );
    }
    Ok(())
}
rust/ch09/src/bin/kvload.rs · region main

The channel ends the loop. Each worker holds a clone of the sender. When the last worker drops the last clone, the receiver reports that the channel is disconnected, and the for loop ends. The main thread has to drop its own sender first, or the loop would wait for ever. That replaces the WaitGroup, the closing goroutine and the nil channels of the Go version. The scope replaces the leak. A scoped thread must finish before the scope returns, so a stuck worker cannot be abandoned the way the Go version abandons it. The read timeout on the socket is what lets it give up and report an error. The result against the chapter 8 server is the same seventeen:

$ cargo run --release --bin kvload           # the chapter 8 server
10000 answered by 17 of 50 connections, 9984 of them by one, 0 failed, 2.06 s in all
p50 0.033 ms, max 410.443 ms
$ cargo run --release --bin kvload           # the chapter 9 server
10000 answered by 50 of 50 connections, 221 of them by one, 0 failed, 0.12 s in all
p50 0.540 ms, max 2.066 ms

Atomics#

An atomic is an integer that many threads may change without a lock, because the processor does each operation in one step. Every operation takes an Ordering, which says what else it promises about memory. Relaxed promises nothing beyond the counter itself, which is enough here and not enough for publishing data to another thread. A mutex or a channel promises that. Both are built from atomics, and so is Arc, which is why it may cross threads and Rc may not.

RwLock#

Go's store took a read-write lock, and Rust has the same type. A RwLock "allows a number of readers or at most one writer at any point in time". The server does not use it yet, because every command takes the store mutably. Chapter 13 makes the commands say which of them only read, and measures what the read lock gives.

๐Ÿ“ฐMars Pathfinder, 1997

A few days after it landed on Mars in July 1997, the Pathfinder lander began resetting itself. Its software ran on VxWorks, with tasks at fixed priorities and a shared area of memory behind a mutex. A low-priority task that published weather data took the mutex. A medium-priority communications task then preempted it and ran for a long time. The high-priority task that managed the shared memory waited for the mutex, behind a task that could not run. A watchdog noticed that the high-priority task had not run and reset the whole system. The engineers at the Jet Propulsion Laboratory reproduced it on the replica in their lab, with tracing on, and saw a classic priority inversion. The mutex had been created with priority inheritance turned off. They uploaded a short C program that changed the flag from FALSE to TRUE, and the resets stopped. A lock is a promise about memory, which Go checks at test time and Rust at compile time. It is also a promise about time, who holds it and for how long, and no language checks that one. Taking the lock for one command and dropping it at the end of the line is the answer kv has.

Rust's platform code: an attribute on the item#

Rust answers the platform question with an attribute, not a file. #[cfg(unix)] on a function, a module or a statement keeps it out of the build on every other system. #[cfg(target_os = "linux")] names one system. The open-file limit is the one place kv needs it, and the standard library has no call for it. So the function asks the C library, through the libc crate, which tokio already depends on. That call is the first unsafe block in the book. The word marks a promise the compiler cannot check, and chapter 15 is about it:

/// How many files, and so how many clients, the process may hold open. The standard
/// library has no call for it, so this asks the C library, which chapter 15 explains.
#[cfg(unix)]
pub fn open_files() -> Option<u64> {
    let mut lim = libc::rlimit {
        rlim_cur: 0,
        rlim_max: 0,
    };
    // SAFETY: getrlimit writes one struct through a valid pointer and reads nothing.
    let rc = unsafe { libc::getrlimit(libc::RLIMIT_NOFILE, &mut lim) };
    (rc == 0 && lim.rlim_cur != libc::RLIM_INFINITY).then_some(lim.rlim_cur)
}
rust/ch09/src/sys.rs · region unix

The other version carries the opposite condition, so that exactly one of them is compiled:

/// Where the system has no such limit to ask about, the answer is None.
#[cfg(not(unix))]
pub fn open_files() -> Option<u64> {
    None
}
rust/ch09/src/sys.rs · region other

Rust once more: tasks on tokio#

A thread for each client works for fifty clients and for five thousand. Each thread has a 2 MiB stack, on every main platform. So fifty thousand threads would reserve a hundred gigabytes of stack. The operating system limits how many threads a process may create long before that. The answer is the C answer, an event loop, with the Go answer on top of it. That is a cheap unit of concurrency that looks like a thread. In Rust that is an async task, and the runtime that schedules tasks is a library, not part of the language. The one almost everyone uses is tokio, and it becomes the first line in the [dependencies] section of Cargo.toml:

$ cargo add tokio@1 --features rt-multi-thread,net,io-util,macros
      Adding tokio v1.53.2 to dependencies

That command also writes Cargo.lock, with the exact version of tokio and of the fourteen crates it depends on. Every later build uses those versions until you ask for new ones, which is what the lock file is for. The server is a second binary in the same crate, src/bin/kv_async.rs. Its serving function looks like the thread version with one word added after every read and write:

/// The replies are written into a Vec first, with the ordinary synchronous command
/// code from the library, and then sent with one awaited write. The lock is never held
/// across an await, so a plain std Mutex is the right one.
async fn serve(stream: TcpStream, store: Arc<Mutex<MapStore>>) -> resp::Result<()> {
    let mut reader = BufReader::new(stream);
    let mut out = Vec::new();
    loop {
        match read_command(&mut reader).await {
            Ok(Some(args)) => {
                let mut store = store.lock().unwrap_or_else(|e| e.into_inner());
                cmd::execute(args, &mut *store, &mut out)?;
            }
            Ok(None) => return Ok(()),
            Err(e @ KvError::Protocol(_)) => {
                resp::write_error(&mut out, "ERR Protocol error")?;
                reader.get_mut().write_all(&out).await?;
                return Err(e);
            }
            Err(e) => return Err(e),
        }
        if reader.buffer().is_empty() {
            reader.get_mut().write_all(&out).await?;
            out.clear();
        }
    }
}
rust/ch09/src/bin/kv_async.rs · region serve

What a future is#

An async fn does not run when it is called. It returns a future, which the standard library defines as "a value that might not have finished computing yet". A future has one method, poll. It "attempts to resolve the future into a final value" and returns either Ready with the value or Pending. "Futures alone are inert", the documentation says. Nothing happens until a runtime polls them. A waker, a handle the future stores and the socket wakes, tells the runtime when to poll again. Each .await in the function above is a point where the task may return Pending and let another task use the thread. That is the C event loop with the bookkeeping done by the compiler. It turns the function into a state machine with one state per await.

Pin in one paragraph#

That state machine is why Pin appears in every async signature. Between two awaits, the function's local variables live inside the future, and one may be a reference to another. A value that points into itself must never move, or the pointer would point at the old address. Pin is a wrapper that promises a value will "remain located at the same place in memory" until it is dropped. You write the word only when you implement a future yourself, which this book never does.

The parser from chapter 4 has to be written again with .await after each read. A function that blocks must not be called from a task, because it stops the thread and every task on it:

/// The parser from chapter 4 again, with `.await` after every read. A blocking read
/// would stop the whole thread; an awaited one lets the thread run another task.
async fn read_line(r: &mut BufReader<TcpStream>) -> resp::Result<Option<Vec<u8>>> {
    let mut line = Vec::new();
    if r.read_until(b'\n', &mut line).await? == 0 {
        return Ok(None);
    }
    if line.pop() != Some(b'\n') {
        return Err(io::Error::from(io::ErrorKind::UnexpectedEof).into());
    }
    if line.last() == Some(&b'\r') {
        line.pop();
    }
    Ok(Some(line))
}

fn parse_len(digits: &[u8], type_byte: u8) -> resp::Result<usize> {
    std::str::from_utf8(digits)
        .ok()
        .and_then(|s| s.parse().ok())
        .ok_or(KvError::Protocol(type_byte))
}

async fn read_command(r: &mut BufReader<TcpStream>) -> resp::Result<Option<Command>> {
    let Some(line) = read_line(r).await? else {
        return Ok(None);
    };
    if line.first() != Some(&b'*') {
        let words = line.split(|b| *b == b' ').filter(|w| !w.is_empty());
        return Ok(Some(words.map(<[u8]>::to_vec).collect()));
    }
    let n = parse_len(&line[1..], b'*')?;
    if n > MAX_ARGS {
        return Err(KvError::Protocol(b'*'));
    }
    let mut args = Vec::with_capacity(n);
    for _ in 0..n {
        let line = read_line(r)
            .await?
            .ok_or(io::Error::from(io::ErrorKind::UnexpectedEof))?;
        if line.first() != Some(&b'$') {
            return Err(KvError::Protocol(line.first().copied().unwrap_or(b'\n')));
        }
        let size = parse_len(&line[1..], b'$')?;
        if size > MAX_BULK {
            return Err(KvError::Protocol(b'$'));
        }
        let mut buf = vec![0u8; size + 2];
        r.read_exact(&mut buf).await?;
        if &buf[size..] != b"\r\n" {
            return Err(KvError::Protocol(buf[size]));
        }
        buf.truncate(size);
        args.push(buf);
    }
    Ok(Some(args))
}
rust/ch09/src/bin/kv_async.rs · region read
Exercise 09.4RustAccept tasks instead of threads

Write the main of the tokio version. Mark it #[tokio::main] and make it async. Bind a tokio::net::TcpListener. In a loop, await accept and clone the Arc. Then tokio::spawn an async block that calls serve. The shape is the thread version with thread::spawn replaced and .await added.

#[tokio::main]
async fn main() -> io::Result<()> {
    // your code here
}
Stuck?Show the full listing
#[tokio::main]
async fn main() -> io::Result<()> {
    let port = port();
    let listener = TcpListener::bind(format!("0.0.0.0:{port}")).await?;
    eprintln!("kv (tokio) listening on port {port}");
    let store = Arc::new(Mutex::new(MapStore::new()));
    loop {
        let (stream, _) = listener.accept().await?;
        let store = Arc::clone(&store);
        // a task, not a thread: it costs a few hundred bytes and no kernel resources
        tokio::spawn(async move {
            if let Err(e) = serve(stream, store).await {
                eprintln!("client: {e}");
            }
        });
    }
}
rust/ch09/src/bin/kv_async.rs · region main

select!, cancellation and blocking#

Two more pieces of tokio appear in the next chapters. tokio::select! is the select of Go, for futures. It "waits on multiple concurrent branches, returning when the first branch completes, cancelling the remaining branches". Cancelling means dropping, and that is the whole of cancellation in Rust. A dropped future stops, because a future that is not polled does nothing. Chapter 10 uses it to wait for a client and for a shutdown signal at once. The other piece is the rule about blocking. A blocking call inside a task stops the thread from advancing every other task on it. The cure is spawn_blocking, which "runs the provided closure on a thread dedicated to blocking operations", and chapter 11 uses it for fsync.

โš ๏ธA lock held across an await Rust

Hold a std::sync::Mutex guard across .await and the task will not compile. The error is future cannot be sent between threads safely. The note says that MutexGuard is not Send, and that "await occurs here, with mut guard maybe used later". It is the same Send check as the thread version, on the future this time. A task may resume on another thread after an await, and a mutex must be unlocked by the thread that locked it. The serving function above takes the lock, runs the command into a buffer, and drops the guard before the write is awaited. The tokio crate has its own mutex for the cases where a lock must be held across an await. It is slower, which is why the code avoids needing it.

โœ…Keep the blocking code and call it from the task Rust

The command code in the library did not change for tokio. It writes replies into any Write, and a Vec<u8> is one. The task fills a vector with the synchronous code and sends the vector with one awaited write. That keeps the async surface of a program small, which makes the compiler's messages shorter and the code easier to test without a runtime.

What the four servers measure#

Each server was given fifty connections and two hundred thousand requests by redis-benchmark, driven from four threads, on a machine with twelve cores. The numbers are requests a second, and chapter 13 explains how to take and read such numbers properly.

C, kqueue, one thread    SET  72,700   GET  79,900
Go, goroutines           SET  88,700   GET  79,800
Rust, threads            SET  99,900   GET  88,800
Rust, tokio              SET  99,800   GET  88,800

Three things in the table are worth more than the numbers. The C server uses one core and is within a third of the others, which is why Redis stayed single-threaded for so long. The Go and Rust servers use every core. A lock on one store stops them from using the cores well. Chapter 13 returns to that. And the tokio version is no faster than the thread version at fifty clients, because fifty threads are cheap. Its advantage starts where threads run out, at tens of thousands of connections.

Verdict##

TopicCGoRustWhy
Serving many clientsWeakStrongOKC needs 200 lines of event loop and three pollers, Go one keyword, and Rust an Arc, a Mutex and a clone per thread.
Using every coreWeakStrongStrongThe C loop is one thread by design. Go and Rust spread clients over every core.
A forgotten lockn/aWeakStrongGo compiles it and the detector may catch it. Rust refuses to compile it.
Platform codeOKStrongStrongC needs a file per poller under ifdef. Go and Rust need one guarded function, and their runtimes hide the pollers.
Memory for a clientStrongStrongOKC sizes two buffers and a goroutine is a few kilobytes, while a Rust thread is 2 MiB until tokio fixes that at the cost of a dependency.
Reading the codeWeakStrongOKThe C loop is subtle and the Go version is the chapter 7 code, while the tokio version is a second copy of the parser.
WindowsWeakStrongStrongThe C loop reports readiness and Windows reports completion. Both runtimes hide that.
Carry forward
  • Event loop. One thread asks the kernel which sockets are ready and serves those. poll walks an array every time. The other two keep a list in the kernel.
  • Unit of concurrency. A C client is a struct. A Go client is a goroutine of a few kilobytes. A Rust client is a 2 MiB thread or a small tokio task.
  • Who checks the lock. Nobody in C, the race detector at test time in Go, and the Send and Sync traits at compile time in Rust. None of them checks how long you hold it.
Next

A client asks for a large value eight times and hangs up before reading any of it. The chapter 9 C server survives the first such client, and dies on the second.

Expiry#

Building kv · Chapter 10· 39 min read· 635 lines of code
▶Scene

Give the chapter 9 C server a value of a million bytes. Then be a bad client. Ask for it eight times, read sixteen bytes of the eight megabytes, and hang up. Do it in a loop:

$ redis-cli -p 6380 -x SET big < million-x.txt
OK
$ for i in 1 2 3; do
>   (exec 3<>/dev/tcp/127.0.0.1/6380; for _ in 1 2 3 4 5 6 7 8; do printf 'GET big\r\n'; done >&3
>    head -c 16 <&3 >/dev/null)
>   kill -0 $pid && echo "hang-up $i: the server is still here" || { wait $pid; echo "hang-up $i: gone, status $?"; break; }
> done
hang-up 1: the server is still here
hang-up 2: gone, status 141

Status 141 is 128 plus 13, and signal 13 is SIGPIPE. One client that did not want its reply has killed the server for every other client, and nothing was logged.

A cache forgets. This chapter gives kv keys that die on time, with two options on SET and the commands EXPIRE and TTL. It also teaches the server about signals, so that a hang-up no longer kills it and a kill stops it cleanly. And the Rust parser stops copying, which is where lifetimes come in.

Redis is used as a cache more than as anything else, and a cache entry is only useful with a lifetime. A session that lasts an hour is a key with a deadline. So is a rate limit that resets each minute, and a lock that must not outlive a crashed holder. After the deadline the key must behave as if it had been deleted, whether or not anybody asks for it.

That last clause is where the work is. A key that is read after its deadline is the simple case. The read notices and deletes it. A key that nobody reads again must still go, or the server fills with dead entries. So there are two mechanisms. Lazy expiry checks the deadline on every access. Active expiry runs on a timer and removes dead keys that nobody has touched. Redis does both, and so does kv.

Time, signals and background workCGoRust
A deadlinean int64_t of milliseconds, 0 for nonea time.Time, zero for nonean Option<Instant>
Reading the clockclock_gettime(CLOCK_MONOTONIC)time.Now(), which carries both clocksInstant::now()
Lazy expirykv_store_find unlinks a dead entryGet reports it absent, the sweep removes itget filters it out, the sweep removes it
The sweep runsin the event loop, between waitson a goroutine with a tickeron a thread with a sleep, then a tokio task with an interval
A hang-up mid-replySIGPIPE, ignored since this chapteran EPIPE error, by defaultan EPIPE error, by default
Ctrl-C or killa handler writes to a pipe the loop waits ona context that the signal cancelsa thread that waits for the signal, then a scope that waits for the threads
Stopping the clientsthe loop returnsAfterFunc closes each connectionthe socket is shut, or the task is dropped
The arguments of a requestpointers into the buffer, by conventioncopies, with the capacity cappedslices of the buffer, by lifetime

The whole chapter rests on one question, "is this entry dead yet?", and each language answers it in its own shape. The C entry stores a deadline as a number and treats zero as never. Go stores a time.Time and treats the zero time as never. Rust stores an Option, so "never" is a case the compiler makes every reader handle:

C
/* 0 means the key never expires */
typedef struct kv_entry {
    /* ... key, value ... */
    int64_t expires_at;
    struct kv_entry *next;
} kv_entry;

static bool kv_expired(const kv_entry *e, int64_t now) {
    return e->expires_at != 0 && e->expires_at <= now;
}
Go
// the zero time means the key never expires
type entry struct {
	val     []byte
	expires time.Time
}

func (e entry) expired(now time.Time) bool {
	return !e.expires.IsZero() && !now.Before(e.expires)
}
Rust
// None means the key never expires
struct Entry {
    val: Vec<u8>,
    expires: Option<Instant>,
}

impl Entry {
    fn expired(&self, now: Instant) -> bool {
        self.expires.is_some_and(|at| at <= now)
    }
}
๐Ÿ“œTwo clocks, and why a deadline uses the boring one

Every operating system offers two clocks. The wall clock says what time it is, and it can jump, when an administrator sets it or when the network corrects it. The monotonic clock says how much time has passed since some moment, and it only moves forward. A deadline measured on the wall clock expires at once when the clock jumps ahead an hour, and never when it jumps back. Redis stores expiry times as Unix timestamps, on the wall clock, because it must write them to disk and read them back on another machine. Its documentation warns that setting the computer time forward expires keys at once. The server kv keeps everything in memory, so it uses the monotonic clock. A deadline is then a reading of a clock that only moves forward, and the number means nothing to another process.

C: a deadline on the entry, and a sweep with a cursor#

The clock is one function, in a file of its own because chapter 11 will need it too. CLOCK_MONOTONIC is POSIX, the Portable Operating System Interface, and the call is clock_gettime:

/* Milliseconds on the monotonic clock, which only moves forward. It is not the time of
 * day: that clock jumps when the system time is set, and a deadline must not. */
int64_t kv_now_ms(void);
c/ch10/src/clock.h · region clock

An entry gains a field. The comment says what zero means, which matters because every check in the file depends on it:

/* One key and its value. expires_at is a deadline in milliseconds on the monotonic
 * clock, or 0 for a key that never expires. The key, the value and the next entry are
 * owned. */
typedef struct kv_entry {
    char *key;
    size_t klen;
    kv_value val;
    int64_t expires_at;
    struct kv_entry *next;
} kv_entry;
c/ch10/src/store.h · region entry

Lazy expiry goes into kv_store_find, which now takes the current time. An entry past its deadline is unlinked on the spot, which is why the walk uses the pointer-to-pointer form from chapter 6 again. The special value now = 0 turns the check off, for the callers that have already decided the key is alive:

static bool kv_expired(const kv_entry *e, int64_t now) {
    return e->expires_at != 0 && e->expires_at <= now;
}

/* Unlink and free the entry that *link points to. */
static void kv_unlink(kv_store *s, kv_entry **link) {
    kv_entry *e = *link;
    *link = e->next;
    kv_entry_free(e);
    s->len--;
}

kv_entry *kv_store_find(kv_store *s, const char *key, size_t klen, int64_t now) {
    if (s->cap == 0) {
        return NULL;
    }
    size_t i = kv_index(s, key, klen, s->cap);
    for (kv_entry **link = &s->buckets[i]; *link != NULL; link = &(*link)->next) {
        kv_entry *e = *link;
        if (s->type->eq(e->key, e->klen, key, klen)) {
            if (kv_expired(e, now)) {
                kv_unlink(s, link); /* lazy expiry: the key dies when it is touched */
                return NULL;
            }
            return e;
        }
    }
    return NULL;
}
c/ch10/src/store.c · region expired
Exercise 10.1CSweep a little at a time

Write kv_store_sweep. Start at the bucket the store's cursor names, walk its chain, unlink every expired entry, and move to the next bucket. Stop after looking at budget entries or after one full round of the buckets, and leave the cursor where the next call should start. Count what you removed. A bucket index wraps with the mask cap - 1, because cap is a power of two.

size_t kv_store_sweep(kv_store *s, int64_t now, size_t budget) {
    size_t removed = 0;
    size_t seen = 0;
    /* your code here */
    return removed;
}
Stuck?Show the full listing
size_t kv_store_sweep(kv_store *s, int64_t now, size_t budget) {
    size_t removed = 0;
    size_t seen = 0;
    for (size_t n = 0; n < s->cap && seen < budget; n++) {
        kv_entry **link = &s->buckets[s->cursor];
        while (*link != NULL && seen < budget) {
            seen++;
            if (kv_expired(*link, now)) {
                kv_unlink(s, link); /* *link now points at the next entry */
                removed++;
            } else {
                link = &(*link)->next;
            }
        }
        s->cursor = (s->cursor + 1) & (s->cap - 1);
    }
    return removed;
}
c/ch10/src/store.c · region sweep

The budget matters because the sweep runs on the one thread that serves every client. A sweep that walks ten million keys stops the server for the time that takes. The cursor makes a long walk into many short ones. Redis uses a different bound, a random sample, but the purpose is the same. The sweep is called from the event loop. The loop now waits at most a tenth of a second, so that keys expire on time even when no client is talking:

/* Wait at most 100 ms, so that keys expire even when no client is talking. */
int n = kv_poller_wait(cs.poller, events, 64, 100);
if (n < 0) {
    if (errno == EINTR) {
        continue;
    }
    perror("wait");
    break;
}
kv_store_sweep(store, kv_now_ms(), 200);
c/ch10/src/server.c · region tick

The SET command grows two options, and a plain SET removes any deadline, because that is what Redis does and what clients expect. A number that is not a number, and a lifetime of zero, get the two error texts Redis uses:

/* SET key value [EX seconds | PX milliseconds]. A plain SET removes any deadline, as
 * Redis does; a SET with EX or PX sets a new one. */
static bool kv_cmd_set(const kv_cmd *c, kv_store *s, kv_buf *out, int64_t now) {
    if (c->argc != 3 && c->argc != 5) {
        return kv_reply_arity(out, "set");
    }
    int64_t ttl_ms = 0;
    if (c->argc == 5) {
        long long n;
        bool ex = kv_arg_is(c, 3, "EX");
        if (!ex && !kv_arg_is(c, 3, "PX")) {
            return kv_reply_error(out, "ERR syntax error");
        }
        if (!kv_parse_int(c->argv[4], c->arglen[4], &n)) {
            return kv_reply_error(out, "ERR value is not an integer or out of range");
        }
        if (n <= 0 || n > INT64_MAX / 1000) {
            return kv_reply_error(out, "ERR invalid expire time in 'set' command");
        }
        ttl_ms = ex ? n * 1000 : n;
    }
    if (!kv_store_set(s, c->argv[1], c->arglen[1], c->argv[2], c->arglen[2])) {
        return false;
    }
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1], 0);
    e->expires_at = ttl_ms ? now + ttl_ms : 0;
    return kv_reply_simple(out, "OK");
}
c/ch10/src/cmd.c · region set

EXPIRE and TTL are short. The one surprise is that EXPIRE with zero or a negative number deletes the key at once. The Redis documentation says a non-positive timeout results in the key "being deleted rather than expired", and the test file checks for it:

/* EXPIRE key seconds: 1 if the key exists and got a deadline, 0 if it does not exist. */
static bool kv_cmd_expire(const kv_cmd *c, kv_store *s, kv_buf *out, int64_t now) {
    long long secs;
    if (!kv_parse_int(c->argv[2], c->arglen[2], &secs)) {
        return kv_reply_error(out, "ERR value is not an integer or out of range");
    }
    if (secs > INT64_MAX / 1000) { /* now + secs * 1000 must not overflow */
        return kv_reply_error(out, "ERR invalid expire time in 'expire' command");
    }
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1], now);
    if (e == NULL) {
        return kv_reply_int(out, 0);
    }
    if (secs <= 0) {
        kv_store_del(s, c->argv[1], c->arglen[1]); /* a deadline in the past is a DEL */
    } else {
        e->expires_at = now + secs * 1000;
    }
    return kv_reply_int(out, 1);
}

/* TTL key: the seconds left, -1 for a key with no deadline, -2 for a missing key. */
static bool kv_cmd_ttl(const kv_cmd *c, kv_store *s, kv_buf *out, int64_t now) {
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1], now);
    if (e == NULL) {
        return kv_reply_int(out, -2);
    }
    if (e->expires_at == 0) {
        return kv_reply_int(out, -1);
    }
    return kv_reply_int(out, (e->expires_at - now + 500) / 1000); /* nearest, as Redis */
}
c/ch10/src/cmd.c · region ttl
โš ๏ธA find that deletes C

kv_store_find can now free the entry it is asked about. Any pointer to that entry held across the call is a pointer to freed memory. Inside kv_exec_incr the code finds the entry and reads its deadline. Then it calls kv_store_set and finds the entry again. The first pointer must not be trusted after a call that can change the table. This rule was true in chapter 6. It became easier to break in this chapter, because a function named find no longer sounds like one that writes.

Go: a ticker, and a clock the test can hold still#

A Go time.Time carries both clocks. The documentation says a value from time.Now "contains both a wall clock reading and a monotonic clock reading". Sub and Before use the monotonic one. So a deadline is a plain time.Time, and a zero value means none:

// entry is a value of any kind and its deadline. A zero time means the key never
// expires. The deadline is a time.Time, which carries a monotonic reading, so a change
// to the system clock does not move it.
type entry struct {
	val     any
	expires time.Time
}

func (e entry) expired(now time.Time) bool {
	return !e.expires.IsZero() && !now.Before(e.expires)
}
go/ch10/internal/store/store.go · region entry

Every method takes now as a parameter instead of calling time.Now itself. That is a small decision with a large effect. The command code reads the clock once per command. So every check inside one command, such as the expiry test and the deadline it sets, sees the same moment. And a test can pass any time it likes, which is how the expiry test runs in a millisecond without sleeping:

// TestExpiry uses a clock the test controls, so it never sleeps.
func TestExpiry(t *testing.T) {
	s := New()
	t0 := time.Now()
	s.Set([]byte("k"), []byte("v"))
	s.SetExpiry([]byte("k"), t0.Add(time.Second), t0)
	if _, ok := s.Get([]byte("k"), t0.Add(999*time.Millisecond)); !ok {
		t.Fatal("key died early")
	}
	if _, ok := s.Get([]byte("k"), t0.Add(time.Second)); ok {
		t.Fatal("key outlived its deadline")
	}
	if n := s.Sweep(t0.Add(time.Second), 100); n != 1 {
		t.Fatalf("sweep removed %d keys, want 1", n)
	}
	if s.Len() != 0 {
		t.Fatalf("store holds %d keys after the sweep, want 0", s.Len())
	}
}
go/ch10/internal/store/store_test.go · region expiry-test

Get holds the read lock, and a dead key cannot be removed under a read lock. So Get reports the key absent and leaves the removal to the sweep. The sweep takes the write lock, walks the map, and stops after a budget. Go promises no order for a map walk, and the runtime starts each walk at a different place. That gives the sweep a fresh sample without any cursor:

// SetExpiry gives a live key a deadline. It reports whether the key was there.
func (s *Store) SetExpiry(key []byte, at time.Time, now time.Time) bool {
	s.mu.Lock()
	defer s.mu.Unlock()
	e, ok := s.m[string(key)]
	if !ok || e.expired(now) {
		return false
	}
	e.expires = at
	s.m[string(key)] = e
	return true
}

// TTL returns the time left, and a status: -2 for a missing key, -1 for a key with
// no deadline, 0 for a key with one.
func (s *Store) TTL(key []byte, now time.Time) (time.Duration, int) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	e, ok := s.m[string(key)]
	if !ok || e.expired(now) {
		return 0, -2
	}
	if e.expires.IsZero() {
		return 0, -1
	}
	return e.expires.Sub(now), 0
}

// Sweep removes keys past their deadline, looking at no more than budget of them. The
// map iterates in a different order each time, so each call sees a fresh sample.
func (s *Store) Sweep(now time.Time, budget int) int {
	s.mu.Lock()
	defer s.mu.Unlock()
	removed := 0
	for k, e := range s.m {
		if budget == 0 {
			break
		}
		budget--
		if e.expired(now) {
			delete(s.m, k)
			removed++
		}
	}
	return removed
}
go/ch10/internal/store/store.go · region expire
Exercise 10.2GoRun the sweep in the background

Write sweep, which main starts with go sweep(ctx, s). Make a time.NewTicker for a tenth of a second and stop it with defer. Loop over a select with two cases. When the context is done, return. When the ticker's channel delivers a time, call s.Sweep with it and a budget of 200. The goroutine takes the store's lock like any other, and the ticker makes it sleep in between.

func sweep(ctx context.Context, s *store.Store) {
	// your code here
}
Stuck?Show the full listing
// sweep removes keys past their deadline ten times a second, so that a key nobody reads
// still dies on time. It takes the store's lock like any other goroutine. The select
// waits on the ticker and on the context at once, and returns when the server stops.
func sweep(ctx context.Context, s *store.Store) {
	tick := time.NewTicker(100 * time.Millisecond)
	defer tick.Stop()
	for {
		select {
		case <-ctx.Done():
			return
		case now := <-tick.C:
			s.Sweep(now, 200)
		}
	}
}
go/ch10/cmd/kv/main.go · region sweeper
โœ…Pass the clock in Go

A function that calls time.Now can only be tested with real time, which means tests that sleep and tests that fail on a slow machine. A function that takes now can be tested with any time at all. The store does the second. The command layer reads the clock once and passes it down. The same shape works in Rust and in C, and the C version of kv does it too, in kv_exec, though without the test.

Rust: an Option for the deadline, a thread and then a task for the sweep#

Rust separates the clocks into two types. SystemTime is the wall clock. Instant is the monotonic one, and its documentation says it "will never go backwards". A deadline is Option<Instant>, so the "none" case is a case the compiler makes you handle rather than a zero you must remember:

/// A value and its deadline. `None` means the key never expires. An `Instant` is a
/// reading of the monotonic clock, so a change to the system time cannot move it.
struct Entry {
    val: Value,
    expires: Option<Instant>,
}

impl Entry {
    fn expired(&self, now: Instant) -> bool {
        self.expires.is_some_and(|at| at <= now)
    }
}
rust/ch10/src/store.rs · region entry

get borrows the store immutably, so it filters a dead key out and leaves it for the sweep, as the Go version does. TTL has three answers, and the enum Ttl gives each a name. The command code then matches on three names instead of on -2, -1 and a number:

fn set_expiry(&mut self, key: &[u8], at: Instant, now: Instant) -> bool {
    match self.map.get_mut(key) {
        Some(e) if !e.expired(now) => {
            e.expires = Some(at);
            true
        }
        _ => false,
    }
}

fn ttl(&self, key: &[u8], now: Instant) -> Ttl {
    match self.map.get(key) {
        None => Ttl::Missing,
        Some(e) if e.expired(now) => Ttl::Missing,
        Some(Entry { expires: None, .. }) => Ttl::Never,
        Some(Entry {
            expires: Some(at), ..
        }) => Ttl::Left(*at - now),
    }
}

/// `retain` walks the whole map, which is fine for a store of this size; Redis
/// samples instead, to bound the time.
fn sweep(&mut self, now: Instant) -> usize {
    let before = self.map.len();
    self.map.retain(|_, e| !e.expired(now));
    before - self.map.len()
}
rust/ch10/src/store.rs · region expire

The SET options are a second match arm with five elements. The inner match on the unit and the parsed number has one arm for each error text. The compiler checks that the arms are complete:

/// SET key value [EX seconds | PX milliseconds]. A plain SET removes any deadline, as
/// Redis does; a SET with EX or PX sets a new one.
fn set(args: &Command, store: &mut dyn Store, now: Instant, w: &mut dyn Write) -> io::Result<()> {
    match args.as_slice() {
        [_, key, val] => {
            // the one copy: the store must own its key and value, the buffer does not
            store.set(key.to_vec(), val.to_vec());
            resp::write_simple(w, "OK")
        }
        [_, key, val, unit, n] => {
            let ttl = match (unit.to_ascii_uppercase().as_slice(), parse_int(n)) {
                (b"EX" | b"PX", None) => {
                    return resp::write_error(w, "ERR value is not an integer or out of range");
                }
                (b"EX", Some(n)) if n > 0 && n <= i64::MAX / 1000 => Duration::from_secs(n as u64),
                (b"PX", Some(n)) if n > 0 => Duration::from_millis(n as u64),
                (b"EX" | b"PX", Some(_)) => {
                    return resp::write_error(w, "ERR invalid expire time in 'set' command");
                }
                _ => return resp::write_error(w, "ERR syntax error"),
            };
            let Some(at) = now.checked_add(ttl) else {
                return resp::write_error(w, "ERR invalid expire time in 'set' command");
            };
            store.set(key.to_vec(), val.to_vec());
            store.set_expiry(key, at, now);
            resp::write_simple(w, "OK")
        }
        _ => write_arity(w, b"set"),
    }
}
rust/ch10/src/cmd.rs · region set
Exercise 10.3RustSweep from a thread, then from a task

In the thread version, spawn a thread inside the scope. It sleeps 100 milliseconds, locks the store, and calls sweep with Instant::now(), until the stopping flag is raised. It borrows the store from the scope, with no clone. In the tokio version, spawn a task instead, with tokio::time::interval, and a select! between the tick and the stop channel. The tokio feature time must be added to Cargo.toml for the interval to exist.

s.spawn(|| {
    // your code here
});
Stuck?Show the full listing
// Ten times a second, a thread of its own removes keys past their deadline. It
// takes the lock like any other thread, and sleeps in between.
s.spawn(|| {
    while !stopping.load(Ordering::Relaxed) {
        thread::sleep(Duration::from_millis(100));
        let mut store = store.lock().unwrap_or_else(|e| e.into_inner());
        store.sweep(Instant::now());
    }
});
rust/ch10/src/main.rs · region sweeper
// The sweep is a task like the clients. The await on the interval costs no thread,
// and select! ends the task when the stop arrives.
let sweeper = Arc::clone(&store);
let mut stop_rx = stop.subscribe();
tokio::spawn(async move {
    let mut tick = tokio::time::interval(Duration::from_millis(100));
    loop {
        tokio::select! {
            _ = tick.tick() => {
                let mut store = sweeper.lock().unwrap_or_else(|e| e.into_inner());
                store.sweep(Instant::now());
            }
            _ = stop_rx.recv() => return,
        }
    }
});
rust/ch10/src/bin/kv_async.rs · region sweeper

The two sweepers differ in one way that matters. The thread version occupies a thread that spends almost all its time asleep, which is cheap at one sweeper and wasteful at a thousand timers. The task version occupies nothing while it waits. The await on the interval hands the thread back to the runtime, and the runtime wakes the task when the tick arrives. That is the same trade as chapter 9, seen from the other side.

โš ๏ธretain walks the whole map Rust

The Rust sweep has no budget. retain visits every entry under the lock, which is fine at ten thousand keys and a pause at ten million. The C version has a cursor, and the Go version has a budget on an iteration that starts at a random place. A Rust HashMap iterates in a fixed order for a given state, so taking the first 200 entries would sweep the same 200 every time. A bounded sweep in Rust needs its own cursor, or a random sample, as Redis takes. The book leaves it unbounded and says so, which is the honest version of a difference between the three standard libraries.

C: a signal, a flag and a pipe#

Back to the scene. A write to a socket whose other end has gone raises SIGPIPE, and the default action for SIGPIPE is to end the process. The kernel does not ask. That default is from the days when a pipeline such as cat file | head was the only pipe. A writer with nobody left to read should end, without a message. A server is the opposite case. One line in main changes the default to ignoring the signal. The write then fails with EPIPE instead, and the loop closes that one client:

/* A write to a client that has gone raises SIGPIPE, which kills the process by
 * default. Ignore it, and write() reports EPIPE instead, which closes that client.
 * Linux can ask for the same with MSG_NOSIGNAL on send(), and macOS with the
 * SO_NOSIGPIPE socket option. Ignoring the signal covers both. */
signal(SIGPIPE, SIG_IGN);
/* Ctrl-C and kill send SIGINT and SIGTERM, which also end the process by default.
 * From here on they end the event loop instead, through a pipe it waits on. */
int stop_fd = kv_stop_fd();
if (stop_fd < 0) {
    perror("kv_stop_fd");
    return 1;
}
c/ch10/src/main.c · region signals

The same line adds the second signal story. Ctrl-C sends SIGINT and kill sends SIGTERM, and both end the process at once by default, in the middle of whatever it was doing. A server would rather finish the command it is running, and in chapter 11 write the last bytes of its log. So the two signals get a handler. A handler is a function the kernel calls between any two instructions of the program, even in the middle of malloc. It may only do two things safely. It may set a flag of the one type made for that. And it may call the functions on the async-signal-safe list, which holds write and not printf:

/* Which signal asked the server to stop, or 0. sig_atomic_t is the one integer type
 * that a signal handler may write while the program reads it, without a torn value, and
 * volatile tells the compiler the value can change behind its back. */
extern volatile sig_atomic_t kv_stop_signal;
c/ch10/src/stop.h · region flag
/* A signal handler runs between any two instructions of the program, even in the middle
 * of malloc or printf. So it may only do what is safe at any moment: set the flag, and
 * call functions on the async-signal-safe list. write() is on the list. The byte wakes
 * the loop, which is otherwise asleep in its wait and would not look at the flag. */
static void kv_on_stop(int sig) {
    int saved = errno; /* write() may change errno, and the interrupted code needs it */
    kv_stop_signal = sig;
    if (write(kv_stop_write, "s", 1) < 0) {
        /* the pipe is full or closed: the loop is already awake, or already gone */
    }
    errno = saved;
}
c/ch10/src/stop.c · region handler

The flag alone is not enough, and the reason is a race. The loop is asleep in its wait when the signal arrives, and it checks the flag only when it wakes. A signal that arrives a moment before the wait is missed until the next event, which may be an hour away. The answer is the self-pipe trick, which Daniel Bernstein described in 1991. The handler writes one byte into a pipe, and the loop watches the pipe's other end like a socket. A signal is then an event, delivered through the same wait as everything else:

int kv_stop_fd(void) {
    int fds[2];
    if (pipe(fds) < 0) {
        return -1;
    }
    /* non-blocking, so that a handler can never wait on a full pipe */
    for (int i = 0; i < 2; i++) {
        int flags = fcntl(fds[i], F_GETFL, 0);
        if (flags < 0 || fcntl(fds[i], F_SETFL, flags | O_NONBLOCK) < 0) {
            return -1;
        }
    }
    kv_stop_write = fds[1];
    /* sigaction, not signal(): it says exactly what happens, on every Unix. SA_RESTART
     * lets a read or accept that the signal interrupted carry on, because the pipe,
     * not the interruption, is what tells the loop to stop. */
    struct sigaction sa = {0};
    sa.sa_handler = kv_on_stop;
    sigemptyset(&sa.sa_mask);
    sa.sa_flags = SA_RESTART;
    if (sigaction(SIGINT, &sa, NULL) < 0 || sigaction(SIGTERM, &sa, NULL) < 0) {
        return -1;
    }
    return fds[0];
}
c/ch10/src/stop.c · region install

sigaction is the call to install a handler. The older signal is for SIG_IGN only, because what it does to an interrupted call differs between systems. The flag SA_RESTART says that a read or an accept the signal interrupted should carry on as if nothing happened. The pipe, not the interruption, is what tells the loop to stop. In the loop, the byte is an event with a tag of its own:

/* A byte on the pipe is the signal, delivered to the loop as an event like
 * any other. Nothing was interrupted in the middle, so there is nothing to
 * repair: the loop ends, and main closes everything in order. */
if (c == (kv_conn *)&stop_tag) {
    rc = 0;
    break;
}
c/ch10/src/server.c · region stop

Nothing was interrupted in the middle, so nothing needs repair. The loop returns zero, and main frees the store and closes the socket in order, as it would have after any other return. The protocol tests gain a line, terminate, which sends SIGTERM and expects exit status 0 within two seconds. From this chapter on, every server in the book passes it.

โš ๏ธA handler that does too much C

A signal handler that calls printf, malloc or free can deadlock or corrupt the heap. Those functions take locks and keep state, and the handler may run while the main program is inside them. The symptom is a server that hangs once a month, on a signal that arrived at the wrong instruction. The rule is short. A handler sets a sig_atomic_t, writes a byte, or calls _exit, and everything else happens in the loop. The volatile on the flag is the other half. Without it the compiler may read the flag once and keep the value in a register for ever.

Go: a context that a signal cancels#

Go's runtime handles SIGPIPE on a socket by itself. The os/signal package says a write to a broken pipe on any descriptor but 1 and 2 "will fail with a syscall.EPIPE error". So the Go server survived the scene without a line of code. For SIGINT and SIGTERM, the same package offers a channel, and a context. One call turns the signal into a cancelled context, and every part of the server that holds the context can see it:

// Ctrl-C and kill send SIGINT and SIGTERM, which end a Go program at once by default.
// NotifyContext changes that: the signal cancels a context instead, and every part of
// the server that holds the context sees it. A second signal, after stop() has run,
// ends the program the default way, for a server that will not finish.
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
go/ch10/cmd/kv/main.go · region signals

context in one paragraph#

A Context "carries deadlines, cancellation signals, and other request-scoped values" down a chain of calls, as the first parameter of every function in the chain. WithCancel gives a child that a function call cancels. WithTimeout and WithDeadline give one that time cancels. Done is a channel that closes when the context is cancelled, which is what a select waits on, and Err says why. Values ride along with WithValue, for request-scoped data such as a trace identifier. The documentation warns against using them to pass ordinary parameters. kv needs only the cancel. The sweeper's select waits on the ticker and on ctx.Done() at once, and returns when the server stops.

A clean stop has three parts, and each is a few lines. Closing the listener makes a blocked Accept "unblocked and return errors", which ends the loop. context.AfterFunc runs a function when the context is cancelled, and the function closes a connection, which ends that client's serve. The WaitGroup from chapter 9 waits for every serve to return:

// Three parts to a clean stop. Closing the listener makes Accept return an error,
// which ends the loop. AfterFunc closes each client when the context is cancelled,
// which ends its serve. The WaitGroup waits for every serve to return.
var clients sync.WaitGroup
context.AfterFunc(ctx, func() { ln.Close() })
for {
	conn, err := ln.Accept()
	if err != nil {
		if ctx.Err() != nil {
			break
		}
		fmt.Fprintln(os.Stderr, "accept:", err)
		continue
	}
	clients.Go(func() {
		defer context.AfterFunc(ctx, func() { conn.Close() })()
		if err := serve(conn, s); err != nil && ctx.Err() == nil {
			fmt.Fprintln(os.Stderr, "client:", err)
		}
	})
}
fmt.Fprintln(os.Stderr, "kv stopping:", context.Cause(ctx))
clients.Wait()
go/ch10/cmd/kv/main.go · region shutdown

panic and recover#

One more defence goes into serve, and it is against the book's own bugs. A panic "stops the ordinary flow of control", unwinds the goroutine's stack running every deferred call, and then ends the whole process. An index out of range in one command handler would take down every client. recover "regains control of a panicking goroutine", and the specification says it works only when called directly by a deferred function. So the deferred function below catches the panic, turns it into an error for this one connection, and the connection closes. The bug is logged and is still a bug. The result is named, err, because a deferred function can set a named result and nothing else:

// A panic in one command would end the whole process, and every other client with
// it. recover, which only works inside a deferred function, turns it into an error
// for this one connection, which closes. The bug is still logged, and still a bug.
defer func() {
	if r := recover(); r != nil {
		err = fmt.Errorf("panic: %v", r)
	}
}()
go/ch10/cmd/kv/main.go · region recover

That is the Go rule for the two kinds of failure. An error is a value for the caller, returned and checked, as every err in this book has been. A panic is a bug in the program. recover is for the one place that must survive bugs in code below it, which is a server's per-connection loop. A library that panics at a caller's mistake is thought badly of. A library that returns an error for its own bugs hides them.

Rust: a thread that waits for the signal, then a scope that waits for the threads#

The Rust standard library ignores SIGPIPE before main runs, so the thread server of chapter 9 logged client: Broken pipe (os error 32) in the scene and went on. It has no signal handling of its own, so the stop goes through the C library again, with the pattern that fits threads best. The signals are blocked in main before any thread exists, so every thread inherits the block. One thread then asks for them with sigwait, which "suspends execution of the calling thread until one of the signals" arrives. No handler ever runs between two instructions. An ordinary thread gets the signal as a return value and may do anything:

/// Block SIGINT and SIGTERM in the calling thread. Every thread spawned afterwards
/// inherits the mask, so the one thread that calls `wait_for_stop` is the only one that
/// ever sees them, and no handler runs in the middle of anything.
#[cfg(unix)]
pub fn block_stop_signals() {
    // SAFETY: the set is initialised by sigemptyset before any use, and the mask call
    // reads it and writes nothing else.
    unsafe {
        let mut set = std::mem::zeroed::<libc::sigset_t>();
        libc::sigemptyset(&mut set);
        libc::sigaddset(&mut set, libc::SIGINT);
        libc::sigaddset(&mut set, libc::SIGTERM);
        libc::pthread_sigmask(libc::SIG_BLOCK, &set, std::ptr::null_mut());
    }
}

/// Wait until SIGINT or SIGTERM arrives, and say which. sigwait takes the signal off
/// the pending list and returns it to an ordinary thread, which may then do anything.
#[cfg(unix)]
pub fn wait_for_stop() -> i32 {
    // SAFETY: as above, and sigwait writes one int through a valid pointer.
    unsafe {
        let mut set = std::mem::zeroed::<libc::sigset_t>();
        libc::sigemptyset(&mut set);
        libc::sigaddset(&mut set, libc::SIGINT);
        libc::sigaddset(&mut set, libc::SIGTERM);
        let mut sig = 0;
        libc::sigwait(&set, &mut sig);
        sig
    }
}

/// Elsewhere there is nothing to block, and nothing to wait for: Ctrl-C ends the
/// process the default way.
#[cfg(not(unix))]
pub fn block_stop_signals() {}

#[cfg(not(unix))]
pub fn wait_for_stop() -> i32 {
    loop {
        std::thread::park();
    }
}
rust/ch10/src/sys.rs · region signals

With that, the whole server fits in one thread::scope. The store, the stop flag and the list of clients are locals of main, lent to every thread. The Arc from chapter 9 is gone. The sweeper and the signal thread run inside the scope. Each client runs inside it. And the scope ends only when all of them have returned, which is the waiting part of a clean stop, done by the type system:

// One thread waits for the signal. It raises the flag and then wakes the accept
// loop the only way a plain listener can be woken: by connecting to it.
s.spawn(|| {
    let sig = sys::wait_for_stop();
    eprintln!("kv stopping on signal {sig}");
    stopping.store(true, Ordering::Relaxed);
    let _ = TcpStream::connect(("127.0.0.1", port.parse().unwrap_or(6380)));
});
rust/ch10/src/main.rs · region stopper
// Shutting a socket makes its read return 0, and that client's serve returns.
// The scope then joins every thread, so nothing is still running when main ends.
for c in clients.lock().unwrap_or_else(|e| e.into_inner()).iter() {
    let _ = c.shutdown(Shutdown::Both);
}
rust/ch10/src/main.rs · region drain

Two details are unusual, and both are honest. A plain listener can only be woken by a connection. So the signal thread connects to its own server, and the accept loop sees the flag and breaks. And a client blocked in read is woken by shutting its socket from outside, through a second handle. After that, "all currently blocked and future reads will return Ok(0)". The tokio version has no such problem, because tokio owns the event loop:

/// Completes when Ctrl-C or kill asks the process to stop. Asking tokio for the signal
/// is what turns off the default, which would end the process at once.
async fn stop_signal() {
    #[cfg(unix)]
    {
        use tokio::signal::unix::{SignalKind, signal};
        let mut term = signal(SignalKind::terminate()).expect("SIGTERM handler");
        tokio::select! {
            _ = tokio::signal::ctrl_c() => {}
            _ = term.recv() => {}
        }
    }
    #[cfg(not(unix))]
    {
        let _ = tokio::signal::ctrl_c().await;
    }
}
rust/ch10/src/bin/kv_async.rs · region signal

Asking tokio for a signal is what turns off the default. From then on the accept loop waits on two futures with select!, the next connection and the stop. The stop reaches the tasks through the two channels from chapter 9's last section. A broadcast channel, where "each sent value is seen by all consumers", tells every task. An mpsc channel, where every task holds a sender it never uses, lets main wait until the last sender is dropped:

// Two channels for a clean stop. The broadcast channel tells: every task holds a
// receiver, and one send reaches all of them. The mpsc channel waits: every task
// holds a sender it never uses, and recv() returns None once the last one is dropped.
let (stop, _) = broadcast::channel::<()>(1);
let (done, mut all_done) = mpsc::channel::<()>(1);
rust/ch10/src/bin/kv_async.rs · region channels
loop {
    let stream = tokio::select! {
        accepted = listener.accept() => accepted?.0,
        _ = stop_signal() => break,
    };
    let store = Arc::clone(&store);
    let mut stop_rx = stop.subscribe();
    let done = done.clone();
    tokio::spawn(async move {
        // A task waits on its client and on the stop at once. When the stop wins,
        // the serve future is dropped where it stands, between two awaits, and that
        // is the whole of cancellation: nothing to flag, nothing to catch.
        tokio::select! {
            res = serve(stream, store) => {
                if let Err(e) = res {
                    eprintln!("client: {e}");
                }
            }
            _ = stop_rx.recv() => {}
        }
        drop(done);
    });
}
eprintln!("kv (tokio) stopping");
let _ = stop.send(());
drop(done);
all_done.recv().await; // None, once every task has dropped its sender
rust/ch10/src/bin/kv_async.rs · region accept

The line to look at is the select! inside the task. When the stop wins, the serve future is dropped where it stands, at whichever await it was parked on. That is cancellation in Rust, all of it. The lock is never held across an await, so a command is never cut in half. The socket closes when the future's locals are dropped. Go needed an AfterFunc and a closed connection to get the same effect, because a goroutine cannot be dropped, only persuaded.

The parser stops copying#

Every parser so far has copied each argument out of the buffer the bytes arrived in. Redis does not, and from this chapter the Rust version of kv does not either. A SET copies its key and value when it stores them, and only then. A GET copies nothing at all. The change is one type, and the type is a lifetime:

/// A parsed request: the command name and its arguments, as slices of the buffer the
/// request arrived in. Nothing is copied. The lifetime `'a` is the buffer's, and it says
/// that a `Command` cannot outlive the bytes it points into. A command that needs to
/// keep an argument, as SET keeps its value, copies it then, and only then.
pub type Command<'a> = Vec<&'a [u8]>;

/// What `parse` found in the bytes so far.
#[derive(Debug)]
pub enum Parsed<'a> {
    /// A whole request, and how many bytes of the buffer it used.
    Done(Command<'a>, usize),
    /// Not a whole request yet: read more bytes and call again.
    Incomplete,
}
rust/ch10/src/resp.rs · region command

Lifetimes, in the one place kv needs them#

A struct or a type alias that holds a reference must say what the reference borrows from, and 'a is that name. The Rust book puts it in one sentence. An instance "can't outlive the reference it holds". So a Command<'a> cannot be kept after the buffer it points into is changed or freed, and the compiler refuses a program that tries. The annotation changes nothing about how long anything lives. It only gives the compiler a name to check against, where it cannot infer one. In a function it usually can. The elision rules say that one reference in means the reference out borrows from it. So parse below writes '_, and the compiler fills in the rest:

/// One request from the front of `buf`: an array of bulk strings such as
/// `*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n`, or one inline line of words. The returned
/// `Command` borrows `buf`, which the signature says with the elided lifetime: one
/// reference in, so the one reference out lives as long as it.
pub fn parse(buf: &[u8]) -> Result<Parsed<'_>> {
    let Some((line, mut used)) = take_line(buf, 0) else {
        return Ok(Parsed::Incomplete);
    };
    if line.first() != Some(&b'*') {
        let words = line.split(|b| *b == b' ').filter(|w| !w.is_empty());
        return Ok(Parsed::Done(words.collect(), used));
    }
    let n = parse_len(&line[1..], b'*')?;
    if n > MAX_ARGS {
        return Err(KvError::Protocol(b'*'));
    }
    let mut args = Vec::with_capacity(n);
    for _ in 0..n {
        let Some((line, start)) = take_line(buf, used) else {
            return Ok(Parsed::Incomplete);
        };
        if line.first() != Some(&b'$') {
            return Err(KvError::Protocol(line.first().copied().unwrap_or(b'\n')));
        }
        let size = parse_len(&line[1..], b'$')?;
        if size > MAX_BULK {
            return Err(KvError::Protocol(b'$'));
        }
        let end = start + size;
        if buf.len() < end + 2 {
            return Ok(Parsed::Incomplete);
        }
        if &buf[end..end + 2] != b"\r\n" {
            return Err(KvError::Protocol(buf[end]));
        }
        args.push(&buf[start..end]);
        used = end + 2;
    }
    Ok(Parsed::Done(args, used))
}

/// The line that starts at `from`, without its ending, and where the next line starts.
/// A bare LF is accepted as well as CRLF, so that a command typed into nc works.
fn take_line(buf: &[u8], from: usize) -> Option<(&[u8], usize)> {
    let rest = &buf[from..];
    let nl = rest.iter().position(|b| *b == b'\n')?;
    let line = rest[..nl].strip_suffix(b"\r").unwrap_or(&rest[..nl]);
    Some((line, from + nl + 1))
}

fn parse_len(digits: &[u8], type_byte: u8) -> Result<usize> {
    std::str::from_utf8(digits)
        .ok()
        .and_then(|s| s.parse().ok())
        .ok_or(KvError::Protocol(type_byte))
}
rust/ch10/src/resp.rs · region parse

The serving loop is where the borrow is felt. It reads into a Vec, parses a command out of it, runs it, and then removes the used bytes with drain. The drain changes the vector, and a Command that still pointed into it would be a bug. The compiler allows the call because the borrow ended at the last use of args, not at the end of the block. That is the non-lexical rule from Rust 2018, and without it this loop would need an extra scope:

/// Bytes are read into a buffer and whole requests are parsed out of it without a
/// copy. The store is shared: the Mutex lets one thread at a time use it, and the scope
/// in main lends it to every client, so no Arc is needed.
fn serve(mut stream: TcpStream, store: &Mutex<MapStore>) -> resp::Result<()> {
    let mut writer = BufWriter::new(stream.try_clone()?);
    let mut buf: Vec<u8> = Vec::with_capacity(4096);
    let mut chunk = [0u8; 4096];
    loop {
        let n = stream.read(&mut chunk)?;
        if n == 0 {
            return Ok(()); // the client closed, or the server shut the socket
        }
        buf.extend_from_slice(&chunk[..n]);
        let mut used = 0;
        loop {
            match resp::parse(&buf[used..]) {
                Ok(Parsed::Done(args, n)) => {
                    let mut store = store.lock().unwrap_or_else(|e| e.into_inner());
                    cmd::execute(&args, &mut *store, &mut writer)?;
                    used += n;
                }
                Ok(Parsed::Incomplete) => break,
                Err(e @ resp::KvError::Protocol(_)) => {
                    resp::write_error(&mut writer, "ERR Protocol error")?;
                    writer.flush()?;
                    return Err(e);
                }
                Err(e) => return Err(e),
            }
        }
        // the arguments borrowed the buffer, and they are gone, so the buffer may change
        buf.drain(..used);
        if buf.len() > resp::MAX_REQUEST {
            return Err(resp::KvError::Protocol(b'*')); // a request that never ends
        }
        writer.flush()?;
    }
}
rust/ch10/src/main.rs · region serve

Three smaller things fall out of the same change. A &[u8] is Copy, so a handler can use an argument twice without a clone, where a Vec would have moved. The unknown-command reply uses String::from_utf8_lossy, which returns a Cow, "a clone-on-write smart pointer". It borrows when the bytes are valid text and allocates only when it must replace a byte. And thread::spawn demands 'static of its closure, as chapter 9's error message showed, which means "owns everything it holds". A Command can never be sent to a spawned thread, and a scoped thread may borrow it. The tokio server gained the most. Its second copy of the parser, with an await after every read, is gone. A parser that works on bytes already in memory has nothing to wait for.

Go cannot say any of this in a type, and so it chose to copy. What it can say is how much an argument may grow in place. Each argument is a slice of a buffer with two bytes of line ending after it. append "re-uses the underlying array" when there is room. So an append to such an argument would write over the line ending, or over whatever a later parser keeps there. The next command would see it. The full slice expression buf[:size:size] sets the capacity to the length, so an append must copy:

// buf[:size] would share its spare two bytes with the line ending. The full slice
// expression caps the capacity at the length, so an append to the argument copies.
args = append(args, buf[:size:size])
go/ch10/internal/resp/resp.go · region alias
// Every argument is a slice into a buffer the parser allocated, with two bytes of line
// ending after it. The full slice expression buf[:size:size] sets the capacity to the
// length, so that an append to an argument must copy instead of writing over the line
// ending, or over whatever a later version of the parser keeps there.
func TestArgumentsDoNotShareSpareCapacity(t *testing.T) {
	r := bufio.NewReader(strings.NewReader("*3\r\n$3\r\nSET\r\n$1\r\nk\r\n$2\r\nhi\r\n"))
	args, err := ReadCommand(r)
	if err != nil {
		t.Fatal(err)
	}
	val := args[2]
	if cap(val) != len(val) {
		t.Fatalf("cap %d, len %d: an append would write into the parser's buffer", cap(val), len(val))
	}
	grown := append(val, '!')
	if &grown[0] == &val[0] {
		t.Fatal("append reused the argument's memory")
	}
	if string(val) != "hi" || string(grown) != "hi!" {
		t.Fatalf("val %q, grown %q", val, grown)
	}
}
go/ch10/internal/resp/resp_test.go · region alias

C has had the borrowed parser since chapter 4, and the rule since then has been one comment. The const on each argument is the part of it the compiler can check. It is a promise not to write through the pointer, carried through every call that takes one:

/* One parsed request. The arguments point into the request buffer, so they are valid
 * only until that buffer is consumed. A kv_cmd owns nothing: the comment is the whole
 * of the rule, and const is the part of it the compiler can check. */
typedef struct {
    size_t argc;
    const char *argv[KV_MAX_ARGS];
    size_t arglen[KV_MAX_ARGS];
} kv_cmd;
c/ch10/src/resp.h · region cmd
๐Ÿ“ฐCloudbleed, 2017

On 18 February 2017 a security researcher found other people's cookies and passwords in pages served by Cloudflare. Cloudflare's edge servers rewrote HTML with a parser generated by Ragel, and the generated C checked for the end of the buffer with if (++p == pe). A pointer that had already stepped past the end never equalled it. The parser read on, into memory that held fragments of other customers' requests. The company's report says that "had the check been done using >= instead of == jumping over the buffer end would have been caught". About one request in three million leaked, over five months, and search engines had cached some of the leaked pages. A parser that returns pointers into a buffer is the fastest kind and the most dangerous kind. The C rule is a comment and a habit. The Rust rule is the lifetime on Command<'a>, and the compiler would have refused the bug.

Background work, three ways#

The sweep is the first work in kv that no client asked for. Each language had a natural place to put it. In C it went into the loop that already existed, with a timeout on the wait, and the loop does the sweep between clients. There is no second thread and no lock. In Go it went into a goroutine with a ticker, and the lock on the store made that safe without a thought. In Rust it went into a thread, and then into a task, and the compiler checked that the store was shared correctly both times.

Each placement reflects the chapter before. The C loop is one thread, so the sweep must share it and must be short, which is why it has a budget. The Go and Rust servers have many threads of control, so the sweep can be one more. The cost is a lock that every client also pays. Chapter 11 adds the second kind of background work, writing to disk, where the cost of sharing is higher and the three answers differ more.

Verdict##

TopicCGoRustWhy
A deadlineOKStrongStrongC uses zero to mean none, which the comments must say. Go's zero Time and Rust's Option say it in the type.
The clockOKStrongStrongC picks the clock by name, while Go carries both in one value and Rust gives each a type.
Lazy expiryStrongOKOKC deletes on find. Go and Rust must leave it to the sweep because of the read lock or the borrow.
The sweepStrongStrongWeakC has a cursor and Go a random sample. Rust's retain walks the whole map.
A client that hangs upWeakStrongStrongC dies unless one line ignores SIGPIPE. Both runtimes turn it into an error.
Stopping cleanlyOKStrongOKC needs a handler, a pipe and a rule about what a handler may do. Go needs a context, and Rust the C library or tokio.
A bug in one commandWeakStrongOKC corrupts or crashes, and Go recovers per connection. Rust panics per thread, and the scope reports it.
The parser's argumentsOKOKStrongC borrows by comment, Go copies and caps the capacity, Rust borrows by lifetime and the compiler checks it.
Testing timeWeakStrongStrongGo and Rust tests pass a clock in. The C store has no test runner yet.
Carry forward
  • Two expiries. Lazy expiry checks the deadline on every access. Active expiry sweeps on a timer, with a budget, so that a key nobody reads still dies.
  • A signal is an event. The C handler writes a byte into a pipe the loop waits on. Go turns it into a cancelled context. Rust gives it to one thread, or to a select! in tokio.
  • A lifetime is a name. Command<'a> says the arguments borrow the buffer and cannot outlive it. The annotation changes nothing about how long anything lives.
Next

Every key is gone when the server restarts. Chapter 11 writes each command to a file and pulls the plug in the middle of a write. Then it asks how much a cache may forget.

Persistence#

Building kv · Chapter 11· 32 min read· 534 lines of code
▶Scene

Start this chapter's C server with a log file, pour a million writes into it, and pull the plug a second and a half in. Then start it again:

$ KV_AOF=kv.aof ./kv &
$ redis-benchmark -p 6380 -t set -n 1000000 -r 100000 -q &
$ kill -9 $pid                        # no signal handler runs for this one
$ ls -l kv.aof
-rw-r--r--  1 you  staff  2686230 Oct 12 kv.aof
$ KV_AOF=kv.aof ./kv
kv replayed 59694 commands from kv.aof
$ redis-cli -p 6380 DBSIZE
45076

Fifty-nine thousand commands came back from a file the server never had time to force to disk. Now set KV_FSYNC=always and run the benchmark again. It manages 43 commands a second. The chapter is about the space between those two numbers.

Stop kv and every key is gone. This chapter gives it an append-only file, a log of every write in the bytes the client sent. A restart replays the log through the parser, and the store comes back. The chapter is about files and fsync, about the thread that waits for the disk, and about what a crash takes.

A cache may forget, but a store that forgets on every restart is not much use. The simplest way to remember is to write down every command that changed something, in order. At startup, run them all again. Redis calls this the append-only file, and its documentation says the commands "are logged using the same format as the Redis protocol itself". That one decision means the parser you already have is also the loader, and a text editor can read the file.

Two details make it work. The first is expiry. A command such as SET k v EX 60 means "sixty seconds from now", and "now" is gone by the time the file is replayed. So the file stores a deadline on the wall clock instead, as PEXPIREAT k 1760000000000, which is a Redis command too. The second is fsync. A write call hands bytes to the kernel, which keeps them in memory and writes them to the disk when it likes. fsync waits until they are on the disk. The first survives a crash of the process, which is what the scene showed. Only the second survives a crash of the machine, and it is slow.

The file, and the thread that waits for the diskCGoRust
Open for appendingopen(path, O_WRONLY | O_CREAT | O_APPEND)os.OpenFile with the same flagsOpenOptions::new().create(true).append(true)
One recordencode into a kv_buf, then writeencode into a bufio.Writer, then Flushencode into a BufWriter, then flush
Forcing it to diskfsync, or F_FULLFSYNC on macOSf.Sync(), which does the samefile.sync_all(), which does the same
Who waits for the diska pthread, asleep on a condition variablea goroutine, ranging over a channela scoped thread, ranging over an mpsc receiver, or spawn_blocking
How the loop askslock, set a flag, signal, unlocka send on a channel with room for onea send on the channel
What checks the sharingThreadSanitizer, at test timethe race detector, at test timeSend, at compile time
Replaykv_parse in a loop over the fileresp.ReadCommand on a bufio.Readerresp::parse on a buffer read from the file
Log and store togetherone thread, so no questionthe log has its own lockone lock around a Db with both
๐Ÿ“œWhy the log is in RESP, and why Redis rewrites it

The append-only file arrived in Redis 1.1, and it has kept the protocol format since. The persistence documentation gives the reason in practical terms. The file is "an easy to understand and parse format". If you flush the database by mistake, "you can still save your data set just by stopping the server, removing the latest command, and restarting". The cost is size. A counter incremented a hundred times is one key in memory and a hundred records on disk. Redis solves that by rewriting the file in the background, as "the minimal set of operations needed to create the current data set". The server kv does not rewrite, and the chapter says where that leaves it.

C: a file descriptor, a write, and a thread for the fsync#

The file is opened once, with O_APPEND, which makes every write go to the end of the file whatever else has happened to it. The policy is one of three, and they are the three that Redis offers under the names always, everysec and no:

/* How often the file is forced to disk. Redis has the same three settings. */
typedef enum {
    KV_FSYNC_ALWAYS,   /* after every command: slow and safe */
    KV_FSYNC_EVERYSEC, /* once a second: lose at most a second of writes */
    KV_FSYNC_NO        /* never: the operating system decides */
} kv_fsync_policy;

typedef struct {
    int fd; /* -1 when persistence is off */
    kv_fsync_policy policy;
    bool dirty;           /* bytes written since the last fsync was asked for */
    int64_t last_request; /* monotonic ms of the last request to the syncer */
    kv_syncer syncer;     /* the thread that waits for the disk */
} kv_aof;
c/ch11/src/aof.h · region policy
bool kv_aof_open(kv_aof *a, const char *path, kv_fsync_policy policy) {
    /* O_APPEND: every write goes to the end, even if another process appends too */
    int fd = open(path, O_WRONLY | O_CREAT | O_APPEND, 0644);
    if (fd < 0) {
        perror(path);
        return false;
    }
    *a = (kv_aof){.fd = fd, .policy = policy, .last_request = kv_now_ms()};
    if (!kv_syncer_start(&a->syncer, fd)) {
        perror("kv_syncer_start");
        close(fd);
        a->fd = -1;
        return false;
    }
    return true;
}
c/ch11/src/aof.c · region open

Recording a command means encoding it as an array of bulk strings, which is the format the client sent and the format kv_parse reads. The one rewrite turns a relative expiry into an absolute one, with the wall clock from a second function in clock.c. The record goes to the kernel with write at once. That is the system call, not fwrite, which would put a buffer of the C library's own in front of the kernel's. The Go gotcha below is what such a buffer does to a log. With the always policy it goes to the disk before the reply, on the one thread, and that is the 43 a second:

static bool kv_write_all(int fd, const char *p, size_t n) {
    size_t off = 0;
    while (off < n) {
        ssize_t w = write(fd, p + off, n - off);
        if (w < 0) {
            if (errno == EINTR) {
                continue;
            }
            perror("aof write");
            return false;
        }
        off += (size_t)w;
    }
    return true;
}

/* PEXPIREAT key <ms since 1970>, as the absolute form of EX, PX and EXPIRE. */
static bool kv_encode_pexpireat(kv_buf *out, const char *key, size_t klen, int64_t at) {
    char digits[32];
    int n = snprintf(digits, sizeof digits, "%lld", (long long)at);
    const char *argv[3] = {"PEXPIREAT", key, digits};
    size_t arglen[3] = {9, klen, (size_t)n};
    return kv_aof_encode(out, 3, argv, arglen);
}

bool kv_aof_record(kv_aof *a, const kv_cmd *c, int64_t wall_now) {
    if (a->fd < 0) {
        return true;
    }
    kv_buf out = {0};
    bool ok;
    long long n;
    if (kv_arg_is(c, 0, "SET") && c->argc == 5 && kv_parse_int(c->argv[4], c->arglen[4], &n) &&
        n > 0) {
        /* SET key value EX|PX n becomes SET key value, then PEXPIREAT key <absolute> */
        int64_t at = wall_now + (kv_arg_is(c, 3, "EX") ? n * 1000 : n);
        ok = kv_aof_encode(&out, 3, c->argv, c->arglen) &&
             kv_encode_pexpireat(&out, c->argv[1], c->arglen[1], at);
    } else if (kv_arg_is(c, 0, "EXPIRE") && c->argc == 3 &&
               kv_parse_int(c->argv[2], c->arglen[2], &n)) {
        if (n <= 0) {
            const char *argv[2] = {"DEL", c->argv[1]};
            size_t arglen[2] = {3, c->arglen[1]};
            ok = kv_aof_encode(&out, 2, argv, arglen);
        } else {
            ok = kv_encode_pexpireat(&out, c->argv[1], c->arglen[1], wall_now + n * 1000);
        }
    } else {
        ok = kv_aof_encode(&out, c->argc, c->argv, c->arglen);
    }
    ok = ok && kv_write_all(a->fd, out.data, out.len);
    kv_buf_free(&out);
    if (!ok) {
        return false;
    }
    a->dirty = true;
    if (a->policy == KV_FSYNC_ALWAYS) {
        /* always: wait for the disk here, before the reply, on the one thread */
        if (!kv_fsync(a->fd)) {
            perror("fsync");
        }
        a->dirty = false;
    }
    return true;
}
c/ch11/src/aof.c · region record

In the input function, a write command is recorded after it runs and before its reply is sent. So a client never sees OK for a write the file does not have:

/* The command ran; now it goes to the file, before the reply goes to the client. */
if (kv_cmd_is_write(&cmd) && !kv_aof_record(aof, &cmd, kv_wall_ms())) {
    return false;
}
c/ch11/src/server.c · region log

C again: the second thread#

With everysec, the fsync happens once a second, and the question is who waits for it. On this machine one takes 23 milliseconds, and the event loop cannot stop for 23 milliseconds without every client noticing. So the fsync gets a thread of its own, the first second thread in the C version of kv. The thread tools come from POSIX, the Portable Operating System Interface. A thread is started with pthread_create from a function and one void *, and waited for with pthread_join. The two threads share a struct, and the struct says which rule guards each field:

typedef struct {
    pthread_t thread;
    pthread_mutex_t mu; /* guards wanted and stop */
    pthread_cond_t cv;  /* signalled when either changes */
    bool wanted;        /* an fsync has been asked for */
    bool stop;          /* the thread should return */
    int fd;
    /* when the last fsync finished, in monotonic ms; written by the thread and read by
     * the loop without the lock, which is what an atomic is for */
    _Atomic int64_t last_sync;
} kv_syncer;
c/ch11/src/syncer.h · region struct

Three fields are guarded by a mutex, which one thread at a time may hold. A thread that reads or writes them without holding it has a data race. C says a data race is undefined behaviour, as chapter 6 said of an overrun. Nothing stops you forgetting the lock. The thread itself spends its life asleep on a condition variable, which is a place to wait until another thread says something has changed. The wait "atomically release[s] mutex and cause[s] the calling thread to block", and the mutex is held again when it returns:

/* The thread. It sleeps on the condition variable until the loop asks for an fsync or
 * for it to stop, and it never holds the mutex while it waits for the disk. */
static void *kv_syncer_run(void *arg) {
    kv_syncer *s = arg;
    pthread_mutex_lock(&s->mu);
    for (;;) {
        /* a while, not an if: a wait may return early, and the loop may change its mind */
        while (!s->wanted && !s->stop) {
            pthread_cond_wait(&s->cv, &s->mu); /* releases mu while asleep, takes it back after */
        }
        if (s->stop) {
            break;
        }
        s->wanted = false;
        pthread_mutex_unlock(&s->mu);
        if (!kv_fsync(s->fd)) {
            perror("fsync");
        }
        /* release: every write the fsync made visible is published with the time */
        atomic_store_explicit(&s->last_sync, kv_now_ms(), memory_order_release);
        pthread_mutex_lock(&s->mu);
    }
    pthread_mutex_unlock(&s->mu);
    return NULL;
}
c/ch11/src/syncer.c · region run

The wait is inside a while, never an if, and the manual says why. "Spurious wakeups from the pthread_cond_timedwait() or pthread_cond_wait() functions may occur", so the condition must be checked again on every return. The fourth field is an atomic. The thread writes the time the fsync finished. The loop reads it without taking the lock, to know whether the last request is still running. An atomic operation is one no other thread can see half done, and the memory order on it says what else it promises. The release on the store and the acquire on the load pair up. Everything the thread wrote before the store is visible to the loop after the load. The loop's side is three short functions:

bool kv_syncer_start(kv_syncer *s, int fd) {
    *s = (kv_syncer){.fd = fd};
    atomic_init(&s->last_sync, kv_now_ms());
    if (pthread_mutex_init(&s->mu, NULL) != 0 || pthread_cond_init(&s->cv, NULL) != 0) {
        return false;
    }
    /* A signal is delivered to any thread that does not block it. The stop handler must
     * run on the event loop's thread, so this thread is created with every signal
     * blocked, which it inherits, and the loop's mask is put back afterwards. */
    sigset_t all, before;
    sigfillset(&all);
    pthread_sigmask(SIG_BLOCK, &all, &before);
    int rc = pthread_create(&s->thread, NULL, kv_syncer_run, s);
    pthread_sigmask(SIG_SETMASK, &before, NULL);
    return rc == 0;
}

void kv_syncer_request(kv_syncer *s) {
    pthread_mutex_lock(&s->mu);
    s->wanted = true;
    pthread_cond_signal(&s->cv);
    pthread_mutex_unlock(&s->mu);
}

int64_t kv_syncer_last(const kv_syncer *s) {
    /* acquire: pairs with the release in the thread, so the reader sees the fsync */
    return atomic_load_explicit(&s->last_sync, memory_order_acquire);
}

void kv_syncer_stop(kv_syncer *s) {
    pthread_mutex_lock(&s->mu);
    s->stop = true;
    pthread_cond_signal(&s->cv);
    pthread_mutex_unlock(&s->mu);
    pthread_join(s->thread, NULL);
    pthread_cond_destroy(&s->cv);
    pthread_mutex_destroy(&s->mu);
}
c/ch11/src/syncer.c · region api

The event loop has woken ten times a second since chapter 10, for the sweep, and the request joins it. Once a second, when there is something to force and the last fsync has finished, the loop asks and goes straight back to its clients:

void kv_aof_tick(kv_aof *a, int64_t now) {
    if (a->fd < 0 || !a->dirty || a->policy != KV_FSYNC_EVERYSEC) {
        return;
    }
    if (now - a->last_request < 1000) {
        return; /* not yet a second since the last request */
    }
    if (kv_syncer_last(&a->syncer) < a->last_request) {
        return; /* the last fsync is still running: a slow disk gets one at a time */
    }
    a->dirty = false;
    a->last_request = now;
    kv_syncer_request(&a->syncer); /* returns at once; the thread waits for the disk */
}
c/ch11/src/aof.c · region tick

One line in the start function is about signals, and it answers a question chapter 10 left open. A signal is delivered to any thread that does not block it. The stop handler must run on the loop's thread, whose wait it is meant to interrupt. So the fsync thread is created with every signal blocked, a mask it inherits from the thread that created it. The loop's own mask is put back at once. ThreadSanitizer found the need for that line, in a way. Under it, a signal that landed on the sleeping thread was never acted on, and the server did not stop.

The thread is where the macOS surprise lives. On macOS, fsync only passes the bytes to the drive. The manual says that "the drive may not physically write the data to the platters for quite some time". The call that reaches the disk is fcntl(fd, F_FULLFSYNC). The Go and Rust standard libraries both make it when you ask for a sync on macOS. So the C version makes it too, behind an #ifdef:

bool kv_fsync(int fd) {
#ifdef __APPLE__
    /* On macOS fsync() only hands the bytes to the drive, which may hold them in its own
     * cache. F_FULLFSYNC asks the drive to write them, and is what Go and Rust call. */
    return fcntl(fd, F_FULLFSYNC) >= 0;
#else
    return fsync(fd) >= 0;
#endif
}
c/ch11/src/syncer.c · region fsync

Two threads sharing a struct is where the book's C tooling gains a sanitizer. ThreadSanitizer watches every access from every thread and reports two that touch the same memory without a lock or an atomic between them. It cannot run together with AddressSanitizer, so the build has a target of its own. make verify-c-tsan rebuilds every snapshot that calls pthread_create and runs the tests again:

$ make verify-c-tsan
## C (TSan) code/c/ch11
== c ch11: 207/207 passed

Thread safety of the C library#

POSIX requires every function in the standard to be thread-safe "except for the following functions". The list is short but has old friends on it. strtok keeps its place in a static. localtime and strerror return pointers to one shared buffer. getenv and rand share state. Each has a safe form with _r on the end, and the syncer uses perror, which is on the safe side. C11 standardised threads in <threads.h>, with thrd_create and mtx_t. But the header is optional and macOS does not provide it, so kv uses the POSIX names that every Unix has.

Exercise 11.1CReplay the file

Write kv_aof_load. Open the file read-only, and return 0 when it does not exist, because a missing file is an empty store. Read it in chunks into a kv_buf. After each chunk, run kv_parse in a loop. Execute each complete command with kv_exec into a reply buffer that nobody reads. Count the commands. Bytes left over at the end are a truncated last command. Report them and drop them.

long kv_aof_load(const char *path, kv_store *store) {
    int fd = open(path, O_RDONLY);
    if (fd < 0) {
        return errno == ENOENT ? 0 : -1;
    }
    /* your code here */
}
Stuck?Show the full listing
long kv_aof_load(const char *path, kv_store *store) {
    int fd = open(path, O_RDONLY);
    if (fd < 0) {
        return errno == ENOENT ? 0 : -1; /* no file yet is an empty store, not an error */
    }
    kv_buf in = {0};
    kv_buf out = {0}; /* the replies, which nobody reads */
    char tmp[65536];
    long applied = 0;
    for (;;) {
        ssize_t n = read(fd, tmp, sizeof tmp);
        if (n < 0 && errno == EINTR) {
            continue;
        }
        if (n <= 0) {
            break;
        }
        if (!kv_buf_append(&in, tmp, (size_t)n)) {
            applied = -1;
            break;
        }
        /* the same parser and the same executor as a live client */
        for (;;) {
            kv_cmd cmd;
            size_t used = 0;
            kv_parse_status st = kv_parse(in.data, in.len, &cmd, &used);
            if (st != KV_PARSE_OK) {
                break;
            }
            kv_exec(&cmd, store, &out);
            out.len = 0;
            kv_buf_consume(&in, used);
            applied++;
        }
    }
    if (in.len > 0) {
        fprintf(stderr, "aof: dropped %zu trailing bytes of a truncated command\n", in.len);
    }
    kv_buf_free(&in);
    kv_buf_free(&out);
    close(fd);
    return applied;
}
c/ch11/src/aof.c · region load

The absolute deadline needs a command of its own, PEXPIREAT, which turns a wall-clock time back into a monotonic one for the store. A deadline already in the past deletes the key, which is how an expired key stays expired across a restart:

/* PEXPIREAT key <ms since 1970>: the absolute form that the append-only file uses. The
 * wall-clock deadline is turned into a monotonic one for the store. */
static bool kv_cmd_pexpireat(const kv_cmd *c, kv_store *s, kv_buf *out, int64_t now) {
    long long at;
    if (!kv_parse_int(c->argv[2], c->arglen[2], &at)) {
        return kv_reply_error(out, "ERR value is not an integer or out of range");
    }
    kv_entry *e = kv_store_find(s, c->argv[1], c->arglen[1], now);
    if (e == NULL) {
        return kv_reply_int(out, 0);
    }
    int64_t left = at - kv_wall_ms();
    if (left <= 0) {
        kv_store_del(s, c->argv[1], c->arglen[1]);
    } else {
        e->expires_at = now + left;
    }
    return kv_reply_int(out, 1);
}
c/ch11/src/cmd.c · region pexpireat

Startup replays the file before it serves the first client, so that no client sees an empty store, and then opens the file for appending. At the other end, the clean stop from chapter 10 joins the thread, forces the last bytes to disk and closes the file:

/* KV_AOF names the file. Replay it first, then open it for appending. */
kv_aof aof = {.fd = -1, .policy = KV_FSYNC_EVERYSEC};
const char *path = getenv("KV_AOF");
if (path != NULL && *path != '\0') {
    kv_fsync_policy policy;
    if (!kv_fsync_policy_parse(getenv("KV_FSYNC"), &policy)) {
        fprintf(stderr, "KV_FSYNC must be always, everysec or no\n");
        return 1;
    }
    long applied = kv_aof_load(path, store);
    if (applied < 0) {
        perror(path);
        return 1;
    }
    fprintf(stderr, "kv replayed %ld commands from %s\n", applied, path);
    if (!kv_aof_open(&aof, path, policy)) {
        return 1;
    }
}
c/ch11/src/main.c · region startup
โš ๏ธwrite is not fsync C

After write returns, the bytes are in the kernel, not on the disk. If the process is killed, the kernel still has them and writes them out, which is why the scene lost nothing. If the machine loses power, they are gone. fsync makes the kernel write them now and wait. The Linux manual adds that even then the directory entry for a new file may not have reached the disk. On macOS, fsync itself is not enough, and the section above says what is. Most databases have a paragraph like this one in their source.

๐Ÿ–ฅfork, the other way to do work in the background

Redis takes its snapshots with fork, which "creates a new process by duplicating the calling process". The child writes the whole dataset to a file while the parent goes on serving. The kernel copies a page only when one of them writes to it. The child "is created with a single thread", the one that called fork. That is why a program with threads must be careful what the child does. Windows has no fork. After completion ports, that is the second reason why Redis on Windows was always a port and never the same program. The server kv uses a thread instead of a process, which is the choice Go and Rust make everywhere.

Go: the same reader, backwards#

The Go log is a struct with a lock, because many goroutines record into it. The policy is parsed from the environment into one of three constants:

// Policy says how often the file is forced to disk. Redis has the same three settings.
type Policy int

const (
	Always   Policy = iota // after every command: slow and safe
	EverySec               // once a second: lose at most a second of writes
	No                     // never: the operating system decides
)

func ParsePolicy(s string) (Policy, error) {
	switch strings.ToLower(s) {
	case "", "everysec":
		return EverySec, nil
	case "always":
		return Always, nil
	case "no":
		return No, nil
	}
	return 0, fmt.Errorf("KV_FSYNC must be always, everysec or no, not %q", s)
}

// Log is the open file. Many goroutines record into it, so it has a lock of its own.
type Log struct {
	mu     sync.Mutex
	f      *os.File
	w      *bufio.Writer
	policy Policy
	dirty  bool
	syncs chan struct{} // a request to the syncer goroutine; room for one, so a second waits for nobody
	done  sync.WaitGroup
}
go/ch11/internal/aof/aof.go · region policy

Recording uses the WriteArrayLen and WriteBulk functions from the protocol package, which were written to answer clients and serve for log records too. The buffer is flushed after every record. The book's own test found the bug that this line fixes, and the gotcha below tells the story:

// Record appends one command. A relative expiry is rewritten as an absolute PEXPIREAT,
// because the file may be replayed much later. The bytes reach the kernel before this
// returns: a policy delays only the fsync, never the write, so a crash of the process
// loses nothing and only a crash of the machine can lose the last second.
func (l *Log) Record(args [][]byte, now time.Time) error {
	l.mu.Lock()
	defer l.mu.Unlock()
	for _, cmd := range rewrite(args, now) {
		if err := resp.WriteArrayLen(l.w, len(cmd)); err != nil {
			return err
		}
		for _, a := range cmd {
			if err := resp.WriteBulk(l.w, a); err != nil {
				return err
			}
		}
	}
	if err := l.w.Flush(); err != nil {
		return err
	}
	l.dirty = true
	if l.policy == Always {
		return l.syncLocked()
	}
	return nil
}

// rewrite turns one command into the commands the file stores.
func rewrite(args [][]byte, now time.Time) [][][]byte {
	name := strings.ToUpper(string(args[0]))
	switch {
	case name == "SET" && len(args) == 5:
		n, err := strconv.ParseInt(string(args[4]), 10, 64)
		if err != nil || n <= 0 {
			break
		}
		d := time.Duration(n) * time.Millisecond
		if strings.ToUpper(string(args[3])) == "EX" {
			d = time.Duration(n) * time.Second
		}
		return [][][]byte{args[:3], pexpireat(args[1], now.Add(d))}
	case name == "EXPIRE" && len(args) == 3:
		n, err := strconv.ParseInt(string(args[2]), 10, 64)
		if err != nil {
			break
		}
		if n <= 0 {
			return [][][]byte{{[]byte("DEL"), args[1]}}
		}
		return [][][]byte{pexpireat(args[1], now.Add(time.Duration(n)*time.Second))}
	}
	return [][][]byte{args}
}

func pexpireat(key []byte, at time.Time) [][]byte {
	return [][]byte{[]byte("PEXPIREAT"), key, []byte(strconv.FormatInt(at.UnixMilli(), 10))}
}
go/ch11/internal/aof/aof.go · region record

Go again: a goroutine that waits for the disk#

The Go version of the fsync thread is a goroutine and a channel. It is shorter than the C version by the whole of the mutex and the condition variable. The channel has room for one request. A sender that finds it full is a request nobody needs, because one is already waiting. So the select with a default drops it without blocking. The goroutine ranges over the channel and ends when Close closes it, and the WaitGroup is how Close waits for that:

// syncer runs on a goroutine of its own and calls fsync whenever it is asked, so that no
// client ever waits for the disk while holding the log's lock. The range ends when Close
// closes the channel, and the WaitGroup lets Close wait for that.
func (l *Log) syncer() {
	for range l.syncs {
		if err := l.f.Sync(); err != nil {
			fmt.Fprintln(os.Stderr, "aof:", err)
		}
	}
}

// Tick is called once a second. With everysec it flushes the buffer to the kernel and
// asks the syncer for an fsync, without waiting for it. The select with a default never
// blocks: if a request is already waiting in the channel, this one is not needed.
func (l *Log) Tick() {
	l.mu.Lock()
	defer l.mu.Unlock()
	if !l.dirty || l.policy != EverySec {
		return
	}
	if err := l.w.Flush(); err != nil {
		fmt.Fprintln(os.Stderr, "aof:", err)
		return
	}
	l.dirty = false
	select {
	case l.syncs <- struct{}{}:
	default:
	}
}
go/ch11/internal/aof/aof.go · region syncer

The file is the one shared thing. The os package says that "the methods of File are safe for concurrent use". So a Sync on the goroutine and a Write under the lock may overlap. Go's Sync on macOS makes the same F_FULLFSYNC call as the C version, which is why the always policy measures the same 44 a second. The ticker goroutine from chapter 10 calls Tick once a second and returns when the context is cancelled.

Exercise 11.2GoReplay with the client's parser

Write Replay. Open the file, and return 0 and no error when it does not exist. Wrap it in a bufio.Reader and call resp.ReadCommand in a loop, handing each command to apply. Stop at io.EOF. Treat io.ErrUnexpectedEOF as a truncated last command, print a line about it, and return what you have.

func Replay(path string, apply func(args [][]byte)) (int, error) {
	f, err := os.Open(path)
	// your code here
}
Stuck?Show the full listing
// Replay reads the file with the same parser a client goes through and hands each
// command to apply. A truncated last command is dropped, as Redis drops it. A missing
// file is an empty store, not an error.
func Replay(path string, apply func(args [][]byte)) (int, error) {
	f, err := os.Open(path)
	if errors.Is(err, os.ErrNotExist) {
		return 0, nil
	}
	if err != nil {
		return 0, err
	}
	defer f.Close()
	r := bufio.NewReaderSize(f, 1<<16)
	n := 0
	for {
		args, err := resp.ReadCommand(r)
		if errors.Is(err, io.EOF) {
			return n, nil
		}
		if errors.Is(err, io.ErrUnexpectedEOF) {
			fmt.Fprintln(os.Stderr, "aof: dropped a truncated command at the end")
			return n, nil
		}
		if err != nil {
			return n, err
		}
		apply(args)
		n++
	}
}
go/ch11/internal/aof/aof.go · region replay

The command function takes the log as a parameter, and records a write before running it. During replay the log is nil, so nothing is recorded twice. The replies during replay go to a writer over io.Discard. Because the log and the store have different locks, two goroutines can record in one order and run in the other. A replay can then end with a different value for a key that both wrote. The Rust version below closes that gap with one lock:

// Execute runs one command against the store and writes its reply. A write command is
// also recorded in the log, when there is one. During replay the log is nil.
func Execute(args [][]byte, s Storer, log *aof.Log, w *bufio.Writer) error {
	now := time.Now()
	if len(args) == 0 {
		return resp.WriteError(w, "ERR empty command")
	}
	name := strings.ToUpper(string(args[0]))
	cmd, ok := commands[name]
	if !ok {
		shown := string(args[0])
		if len(shown) > 40 {
			shown = shown[:40]
		}
		return resp.WriteError(w, "ERR unknown command '"+shown+"'")
	}
	if n := len(args); (cmd.arity < 0 && n < -cmd.arity) || (cmd.arity > 0 && n != cmd.arity) {
		return arity(w, strings.ToLower(name))
	}
	if log != nil && aof.IsWrite(name) {
		if err := log.Record(args, now); err != nil {
			return err
		}
	}
	return cmd.run(args, s, w, now)
}
go/ch11/internal/cmd/cmd.go · region execute
// KV_AOF names the file. Replay it first, with no log, then open it for appending.
var log *aof.Log
if path := os.Getenv("KV_AOF"); path != "" {
	policy, err := aof.ParsePolicy(os.Getenv("KV_FSYNC"))
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	sink := bufio.NewWriter(io.Discard)
	n, err := aof.Replay(path, func(args [][]byte) { cmd.Execute(args, s, nil, sink) })
	if err != nil {
		fmt.Fprintln(os.Stderr, "aof:", err)
		os.Exit(1)
	}
	fmt.Fprintf(os.Stderr, "kv replayed %d commands from %s\n", n, path)
	if log, err = aof.Open(path, policy); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	go func() {
		tick := time.NewTicker(time.Second)
		defer tick.Stop()
		for {
			select {
			case <-ctx.Done():
				return
			case <-tick.C:
				log.Tick()
			}
		}
	}()
}
go/ch11/cmd/kv/main.go · region startup
โš ๏ธA buffered writer and a kill Go

The first Go version lost every write on restart, and the test caught it. The log used a bufio.Writer, as every Go file writer does, and flushed it once a second from the ticker. The test stops the server with a signal, which runs no Go code. So the bytes still in the buffer never reached the kernel. The C version had no such bug, because it has no buffer between the record and write. The fix is the Flush after every record. A policy decides when to fsync. It must never decide when to write.

Rust: the store and the log under one lock#

The Rust log is a struct around a BufWriter<File>, and it has no lock of its own. Instead the store and the log live in one struct, Db, and the one lock from chapter 9 now guards both. A command and its record happen together, and no other command can come between them:

/// Everything a command may touch. One Mutex around this struct means a command and its
/// record in the log happen together, with no other command in between.
pub struct Db {
    pub store: MapStore,
    pub aof: Option<Aof>,
}
rust/ch11/src/db.rs · region struct
Exercise 11.3RustRecord, then run

Write Db::execute. If there is a log and the command is a write, record it with the current wall-clock time, then run it with cmd::execute. Record first, so that a command the file holds is one the server accepted. Use a let chain, which the 2024 edition allows, one if with two conditions joined by &&.

pub fn execute(&mut self, args: &Command, w: &mut impl Write) -> io::Result<()> {
    // your code here
}
Stuck?Show the full listing
impl Db {
    /// Record a write command, then run it. Recording comes first, so that a command
    /// the file holds is one that was accepted, and a crash between the two loses a
    /// reply and never a write.
    pub fn execute(&mut self, args: &Command, w: &mut impl Write) -> io::Result<()> {
        if let Some(log) = self.aof.as_mut()
            && args.first().is_some_and(|name| aof::is_write(name))
        {
            log.record(args, aof::wall_ms())?;
        }
        crate::cmd::execute(args, &mut self.store, w)
    }
}
rust/ch11/src/db.rs · region execute

Recording mirrors the C version. The flush after each record is the same fix as in Go. The first Rust version had the same bug, and the same test found it:

/// Append one command. A relative expiry is rewritten as an absolute PEXPIREAT,
/// because the file may be replayed much later. The bytes reach the kernel before this
/// returns: a policy delays only the fsync, never the write, so a crash of the process
/// loses nothing and only a crash of the machine can lose the last second.
pub fn record(&mut self, args: &Command<'_>, now_ms: i64) -> io::Result<()> {
    for cmd in rewrite(args, now_ms) {
        resp::write_array_len(&mut self.file, cmd.len())?;
        for a in &cmd {
            resp::write_bulk(&mut self.file, a)?;
        }
    }
    self.file.flush()?;
    self.dirty = true;
    if self.policy == Policy::Always {
        self.sync()?;
    }
    Ok(())
}

/// Flush the buffer and force the file to disk, on the calling thread. The always
/// policy does this after every record, and main does it once, last of all.
pub fn sync(&mut self) -> io::Result<()> {
    self.file.flush()?;
    if mem::replace(&mut self.dirty, false) && self.policy != Policy::No {
        self.file.get_ref().sync_all()?;
    }
    Ok(())
}
rust/ch11/src/aof.rs · region record

Rust again: a scoped thread on a channel#

The Rust fsync thread needs the file and nothing else, so the log gives it a second handle. The standard library says a cloned File "shares the same underlying file handle", and an fsync through either forces what was written through both. The other method is the once-a-second question, and it uses mem::replace, which puts a new value in a place and returns the old one. Its cousin mem::take puts the default there instead, and mem::swap exchanges two places. All three exist because Rust will not let you move a value out of a &mut and leave a hole:

impl Aof {
    /// A second handle to the same file, for the thread that waits for the disk. An
    /// fsync through either handle forces everything written through both.
    pub fn handle(&self) -> io::Result<File> {
        self.file.get_ref().try_clone()
    }

    /// Once a second, with everysec: flush the buffer to the kernel and say whether an
    /// fsync is owed. `mem::replace` reads the flag and clears it in one expression.
    pub fn owes_sync(&mut self) -> io::Result<bool> {
        self.file.flush()?;
        Ok(self.policy == Policy::EverySec && mem::replace(&mut self.dirty, false))
    }
}
rust/ch11/src/aof.rs · region handle

The thread lives in the scope from chapter 10, beside the sweeper and the clients. It owns the handle and the receiving end of an mpsc channel, and it does one thing for each message. The sweeper owns the sending end. When the sweeper returns at shutdown, the sender is dropped and the channel reports that it is disconnected. The for loop ends, and the scope joins the thread. No flag and no stop message:

// The thread that waits for the disk. It receives a request on the channel,
// calls fsync through its own handle, and never touches the lock. The loop ends
// when every sender is dropped, which happens when the sweeper returns.
let (sync_tx, sync_rx) = mpsc::channel::<()>();
if let Some(file) = sync_file {
    s.spawn(move || {
        for () in sync_rx {
            if let Err(e) = file.sync_all() {
                eprintln!("aof: {e}");
            }
        }
    });
}
rust/ch11/src/main.rs · region syncer
// every tenth tick, once a second, the everysec policy flushes the log and
// asks the syncer for an fsync, without waiting for it
if tick.is_multiple_of(10) {
    match db.aof.as_mut().map(Aof::owes_sync) {
        Some(Ok(true)) => {
            let _ = sync_tx.send(());
        }
        Some(Err(e)) => eprintln!("aof: {e}"),
        _ => {}
    }
}
rust/ch11/src/main.rs · region sync

The tokio version cannot spawn a thread that blocks, and fsync blocks. This is the case chapter 9 promised spawn_blocking for. The call runs on a thread tokio keeps for exactly that, and the sweeper task goes on at once. The handle is in an Arc, because each blocking call needs its own owner:

// fsync blocks, and a task must not. spawn_blocking runs the call on
// a thread kept for exactly that, and the task goes on at once.
if owed && let Some(file) = sync_file.clone() {
    tokio::task::spawn_blocking(move || {
        if let Err(e) = file.sync_all() {
            eprintln!("aof: {e}");
        }
    });
}
rust/ch11/src/bin/kv_async.rs · region blocking

Replay reads the file in chunks into the same kind of buffer the server reads a socket into. The same parse takes whole commands out of it. The parser does not know whether it is reading a socket or a file, and the replies go into io::sink():

/// Replay the file into the store, through the same parser a client goes through, from
/// the same kind of buffer. A truncated last command is dropped, as Redis drops it. A
/// missing file is an empty store.
pub fn replay(path: &Path, store: &mut MapStore) -> io::Result<usize> {
    let mut file = match File::open(path) {
        Ok(f) => f,
        Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(0),
        Err(e) => return Err(e),
    };
    let mut sink = io::sink();
    let mut buf: Vec<u8> = Vec::new();
    let mut chunk = vec![0u8; 1 << 16];
    let mut n = 0;
    loop {
        let got = file.read(&mut chunk)?;
        if got == 0 {
            if !buf.is_empty() {
                eprintln!("aof: dropped a truncated command at the end");
            }
            return Ok(n);
        }
        buf.extend_from_slice(&chunk[..got]);
        let mut used = 0;
        while let Parsed::Done(args, len) = resp::parse(&buf[used..])? {
            crate::cmd::execute(&args, store, &mut sink)?;
            n += 1;
            used += len;
        }
        buf.drain(..used);
    }
}
rust/ch11/src/aof.rs · region replay
โš ๏ธDrop does not run when the process is killed Rust

The log has a Drop that syncs the file, and a signal never calls it. Drop runs when a value goes out of scope in a program that is still running. A process ended by kill -9, by the test runner, or by a power cut runs no more Rust code. So the Drop is a courtesy for a clean exit. The clean stop from chapter 10 does the same sync on purpose, last of all. The flush after every record is what the durability of the log rests on, exactly as in Go. Rust's ownership rules say nothing about this. The operating system does.

One thread, three ways#

The same thread was written three times in this chapter, and the three versions are a fair picture of the three languages with threads. The C version is a struct, a mutex, a condition variable and an atomic. A comment on each field says which rule guards it, and a sanitizer checks the comments. The Go version is a channel, and the channel is both the message and the lock. The Rust version is a channel too. The compiler checked that the handle could cross to the thread, and that the thread could not outlive the scope.

The fsync threadCGoRust
Starting itpthread_create with a function and a void *wg.Gos.spawn inside the scope
Waiting for workpthread_cond_wait in a whilefor range chfor () in rx
Asking for worklock, flag, signal, unlocka send, dropped when one is waitinga send
Telling it to stopa second flag under the same lockclose(ch)drop the sender
Waiting for it to endpthread_joinwg.Waitthe end of the scope
A forgotten lockundefined behaviour, found by TSan if the test provokes ita data race, found by the detector if the test provokes itdoes not compile

What a crash takes#

Three kinds of failure, and three policies, make a small table that is worth learning. Every version of kv behaves the same way, because all three call the same kernel.

After a crash ofCGoRust
The process, with any policynothing lost, the kernel has every recordthe samethe same
The machine, policy alwaysnothing lostthe samethe same
The machine, policy everysecup to one second of writes lostthe samethe same
The machine, policy nowhatever the kernel had not written, usually secondsthe samethe same
A truncated last recorddropped at replaythe samethe same

The price of always is the scene's number, and it is why Redis defaults to everysec. On the author's machine, with the disk really flushed, the C server answered 43 SET commands a second with one client. With fifty clients it answered 44, because the disk is the whole cost. With everysec it answered 14,700 from one client and 47,700 from fifty. The no policy was no faster, because one fsync a second on a thread of its own costs the clients nothing. The scene's file held 59,694 commands in 2,686,230 bytes, 45 bytes a command, which is the cost of a format a person can read. And a kill -9 never cut a record in half, because each record is one write call and the kernel either has it or does not. Only the machine's death can leave a truncated last record, and all three versions drop it at replay, as Redis does.

Verdict##

TopicCGoRustWhy
Writing the fileOKStrongStrongC encodes into a buffer and calls write. Go and Rust reuse the protocol writers and flush.
ReplayStrongStrongStrongAll three reuse the parser from chapter 4 unchanged, which is the point of a log in RESP.
A crash of the processStrongOKOKC writes straight through. Go and Rust needed a flush the book's test had to find.
The clock on diskOKStrongOKC converts between two clocks itself, while Go's time.Time holds both and Rust converts with a helper.
Log and store togetherStrongOKStrongOne C thread needs no lock, Go takes two locks, and Rust takes one around both.
The fsync threadWeakStrongStrongC needs a mutex, a condition variable, an atomic and a sanitizer. Go needs a channel, and Rust a channel the compiler checks.
Reaching the disk on macOSOKStrongStrongC must know about F_FULLFSYNC itself. Both standard libraries know.
Carry forward
  • write and fsync. write passes bytes to the kernel, which survives a dead process. fsync waits for the disk, which survives a dead machine. On this disk that is 43 a second.
  • Condition variable. A place a thread sleeps until another signals it, always inside a while, because a wait may return for no reason.
  • A thread's channel. In Go and Rust the channel is the message and the lock. Closing it, or dropping the last sender, is how the thread learns to stop.
Next

Every test so far was written by a person who knew what the parser should do. Chapter 12 gives the parser to a program that sends it random bytes, a hundred thousand times a second, and keeps whatever makes it crash.

Testing and fuzzing#

Quality and shipping · Chapter 12· 27 min read· 417 lines of code
▶Scene

Take the chapter 4 parser and remove the one check that compares a length field with the room it has. Give the result to a fuzzer, with a single valid request as its only hint:

$ make fuzz-trusting
INFO: seed corpus: files: 1 min: 22b max: 22b total: 22b rss: 35Mb
==15==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5030005a29d0
WRITE of size 41 at 0x5030005a29d0 thread T0
    #1 0x556e366fc450 in trusting_parse /w/bugs/trusting_parser.c:77:9
0x5030005a29d0 is located 0 bytes after 32-byte region
elapsed 2 s
$ od -c crash-8c6f20f4728eeea01d638af72038b7d1c045481f | head -1
0000000   *   2  \r  \n   $   4   1  \r  \n   E  \0  \0  \0  \0   (  \0

Two seconds. The fuzzer turned $4 into $41, padded the argument to match, and wrote nine bytes past a thirty-two byte buffer. No unit test in this book thought of that input, and no person would have.

Every chapter so far was checked by one script that talks to kv through redis-cli. This chapter adds the tests that live beside the code, in each language's own runner, and the tools that find what tests cannot. A fuzzer throws millions of inputs at the parser. A sanitizer or a race detector watches the program run. The chapter is about what each language can prove, and when.

The protocol tests in tests/protocol/ are the same for all three servers, and that is their value. They do not say which function is wrong, and they only send what a person thought to write. The parser from chapter 4 has five ways to fail on bad input, and the script tries one. A unit test can try all five, and a fuzzer can try five million things nobody thought of.

The toolsCGoRust
A unit testa main you write, with a CHECK macroa TestXxx function in a _test.go filea #[test] function in a mod tests
Running themmake test, a target you writego test ./...cargo test
A test that lives in the docsnoan Example function, whose output is checkeda code block in a doc comment, compiled and run
A test from outsidenoa _test packagethe tests/ directory, a crate of its own
A fuzzerlibFuzzer, with a clang that has itgo test -fuzz, built incargo-fuzz, on the nightly toolchain
Memory errorsAddressSanitizer, or Valgrind on Linuxnot possible in safe codenot possible in safe code
Data racesThreadSanitizergo test -racethe compiler, at compile time
Undefined behaviourUndefinedBehaviorSanitizerdefined, except under a data raceMiri, for unsafe code
What the book runsthe tests under ASan and UBSan, and TSan for the threadsgo vet, go test -racecargo test, clippy -D warnings
๐Ÿ“œWhy C needs tools the other two do not

C has no test runner in its standard library, and no way to find a bad pointer at run time. Both gaps were filled from outside. Valgrind, released by Julian Seward in 2002, runs a program on a simulated processor and watches every memory access. The program runs twenty to thirty times slower. AddressSanitizer, from Google in 2012, puts the checks into the compiled code instead, at about twice the cost. clang and gcc both carry it. Go and Rust were designed after both tools existed and built the answers in. Go's runtime can watch for races and its test tool can fuzz. Rust's compiler refuses the programs that the sanitizer would have caught. The C tools remain the reason that C is still written at all.

C: a runner you write, and a sanitizer that watches#

A C test is a program. The runner for kv is thirty lines. A counter, and a macro that prints the file and the line of a failed check and carries on. One assertion is checked by the compiler instead of the test. _Static_assert takes a constant expression, and if it is false "a compile-time error occurs and the compiler must display message". A test suite that assumes a limit should fail to build when the limit changes, not fail at run time:

static int failed = 0;
static int passed = 0;

/* Report a failure with its location, and carry on to the next check. */
#define CHECK(cond)                                                                                \
    do {                                                                                           \
        if (cond) {                                                                                \
            passed++;                                                                              \
        } else {                                                                                   \
            failed++;                                                                              \
            fprintf(stderr, "%s:%d: check failed: %s\n", __FILE__, __LINE__, #cond);               \
        }                                                                                          \
    } while (0)

/* The tests assume a request may carry at least four arguments. The compiler checks it,
 * so a smaller limit fails to build instead of failing at run time. */
_Static_assert(KV_MAX_ARGS >= 4, "the tests use requests of up to four arguments");
c/ch12/tests/test_resp.c · region harness

The tests for the parser cover the three answers from chapter 4, and the inline form. The one that matters most is the second. It cuts a valid request at every byte and checks that the parser says "incomplete" and never "bad". The event loop depends on exactly that:

Exercise 12.1CEvery prefix is incomplete

Write test_parse_incomplete_at_every_cut. Take a valid request, and for every length from 0 to one less than the whole, call kv_parse on that prefix. Each call must return KV_PARSE_INCOMPLETE. A single KV_PARSE_BAD would make the server drop a client in the middle of a slow request.

static void test_parse_incomplete_at_every_cut(void) {
    const char *req = "*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n";
    /* your loop here */
}
Stuck?Show the full listing
static void test_parse_complete(void) {
    const char *req = "*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n";
    kv_cmd cmd;
    size_t used = 0;
    CHECK(kv_parse(req, strlen(req), &cmd, &used) == KV_PARSE_OK);
    CHECK(used == strlen(req));
    CHECK(cmd.argc == 2);
    CHECK(cmd.arglen[0] == 4 && memcmp(cmd.argv[0], "ECHO", 4) == 0);
    CHECK(cmd.arglen[1] == 2 && memcmp(cmd.argv[1], "hi", 2) == 0);
}

static void test_parse_incomplete_at_every_cut(void) {
    /* every prefix of a valid request is incomplete, never bad */
    const char *req = "*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n";
    for (size_t n = 0; n < strlen(req); n++) {
        kv_cmd cmd;
        size_t used = 0;
        CHECK(kv_parse(req, n, &cmd, &used) == KV_PARSE_INCOMPLETE);
    }
}

static void test_parse_bad(void) {
    const char *bad[] = {"*x\r\n", "*1\r\n#4\r\nECHO\r\n", "*1\r\n$4\r\nECHOXX",
                         "*1\r\n$99999999\r\n", "*100\r\n"};
    for (size_t i = 0; i < sizeof bad / sizeof bad[0]; i++) {
        kv_cmd cmd;
        size_t used = 0;
        CHECK(kv_parse(bad[i], strlen(bad[i]), &cmd, &used) == KV_PARSE_BAD);
    }
}

static void test_parse_inline(void) {
    const char *req = "  SET  a   b\r\n";
    kv_cmd cmd;
    size_t used = 0;
    CHECK(kv_parse(req, strlen(req), &cmd, &used) == KV_PARSE_OK);
    CHECK(cmd.argc == 3);
    CHECK(cmd.arglen[2] == 1 && cmd.argv[2][0] == 'b');
    CHECK(used == strlen(req));
}
c/ch12/tests/test_resp.c · region parse

The list of tests is written once, as an X-macro, the trick from chapter 5's command table. Each use of the list gives X a new meaning. Here it makes a table of names and function pointers, and main walks the table. A name on the command line runs one test, and a new test is one line:

/* The list of tests, written once. Each use of the list defines X to mean something
 * different: a declaration, an entry in a table, or a call. Adding a test is one line
 * here, and nothing else to keep in step. */
#define KV_TESTS                                                                                   \
    X(parse_complete)                                                                              \
    X(parse_incomplete_at_every_cut)                                                               \
    X(parse_bad)                                                                                   \
    X(parse_inline)                                                                                \
    X(parse_int)                                                                                   \
    X(glob)

static const struct {
    const char *name;
    void (*run)(void);
} tests[] = {
#define X(name) {#name, test_##name},
    KV_TESTS
#undef X
};

/* Run every test, or only those whose name contains the argument: ./test_resp parse */
int main(int argc, char **argv) {
    const char *only = argc > 1 ? argv[1] : "";
    int ran = 0;
    for (size_t i = 0; i < sizeof tests / sizeof tests[0]; i++) {
        if (strstr(tests[i].name, only) == NULL) {
            continue;
        }
        int before = failed;
        tests[i].run();
        printf("%s %s\n", failed == before ? "ok  " : "FAIL", tests[i].name);
        ran++;
    }
    printf("%d tests, %d checks passed, %d failed\n", ran, passed, failed);
    return failed == 0 ? 0 : 1;
}
c/ch12/tests/test_resp.c · region table

The Makefile gains a test target. It compiles the test file with every source file except main.c, all under AddressSanitizer and UndefinedBehaviorSanitizer, and runs the result. The sanitizers are the point. A test that passes under them has also proved that it read no freed memory and overflowed no integer on the way:

# The unit tests: every source under src/ except main.c, plus the test file, compiled
# together under the sanitizers. make test fails when a check fails.

test: build/test_resp
	./build/test_resp

build/test_resp: tests/test_resp.c $(SRC) | build
	$(CC) $(CFLAGS) -fsanitize=address,undefined -o $@ tests/test_resp.c $(filter-out src/main.c,$(SRC))
c/ch12/Makefile · region test
$ make test
ok   parse_complete
ok   parse_incomplete_at_every_cut
ok   parse_bad
ok   parse_inline
ok   parse_int
ok   glob
6 tests, 53 checks passed, 0 failed
$ ./build/test_resp glob
ok   glob
1 tests, 7 checks passed, 0 failed

To see what the sanitizer does when there is something to find, the snapshot includes the bug from the chapter 4 gotcha, on purpose. It keeps a pointer into a buffer, grows the buffer, and reads through the pointer:

int main(void) {
    kv_buf b = {0};
    kv_buf_append(&b, "GET key\r\n", 9);
    const char *key = b.data + 4; /* points at "key", inside the buffer */
    char filler[2048];
    memset(filler, 'x', sizeof filler);
    kv_buf_append(&b, filler, sizeof filler); /* the buffer grows, and may move */
    printf("%.3s\n", key);                    /* reads through the old pointer */
    kv_buf_free(&b);
    return 0;
}
c/ch12/bugs/use_after_free.c · region bug
$ make bugs
==32154==ERROR: AddressSanitizer: heap-use-after-free on address 0x619000000584
READ of size 1 at 0x619000000584 thread T0
    #2 0x00010cec1930 in main use_after_free.c:16
0x619000000584 is located 4 bytes inside of 1024-byte region
freed by thread T0 here:
    #0 0x00010d3d9a47 in realloc+0x87
    #1 0x00010cec1aa1 in kv_buf_append buf.c:27
    #2 0x00010cec191e in main use_after_free.c:15

The report names the line that read the freed memory, the line that freed it, and the realloc inside kv_buf_append that did the freeing. Without the sanitizer the program prints key most of the time, because the old bytes are usually still there. That is the whole problem with undefined behaviour. On Linux, Valgrind gives the same report without recompiling, and also lists every block that was never freed. The book ran the test binary under it in a container:

$ valgrind --leak-check=full ./test_resp
53 checks passed, 0 failed
==215== All heap blocks were freed -- no leaks are possible
==215== ERROR SUMMARY: 0 errors from 0 contexts

A debugger session#

The third way to see the same bug is to watch it happen. A debugger stops the program at a line and shows what the variables hold. Build the bug without optimisation and with debug information, and give gdb a script of commands. The session below stops before the buffer grows, prints the buffer's address and the pointer into it, steps one line, and prints them again:

$ cc -O0 -g -o use_after_free bugs/use_after_free.c src/buf.c
$ gdb -q -batch -x session.gdb ./use_after_free
Breakpoint 1, main () at bugs/use_after_free.c:15
15	    kv_buf_append(&b, filler, sizeof filler); /* the buffer grows, and may move */
$1 = 0x7ffff7f55020 "GET key\r\n"
$2 = 0x7ffff7f55024 "key\r\n"
16	    printf("%.3s\n", key);                    /* reads through the old pointer */
$3 = 0x7ffff7f51030 "GET key\r\n", 'x' <repeats 191 times>...
$4 = 0x7ffff7f55024 <error: Cannot access memory at address 0x7ffff7f55024>
$5 = 4096

One next, and the buffer's data moved from one address to another while key kept the old one. The debugger cannot even read it now. On macOS the debugger is lldb, with the same commands in a different dialect. The book recorded this session in a Linux container, because gdb is what the container had. A session like this takes a minute, and it is the right tool when a sanitizer says what happened and you want to see why.

The same snapshot holds a second bug on purpose, for the sanitizer chapter 11 introduced. Two threads add to one counter with no lock, which is the simplest data race there is:

static long counter = 0;

static void *add(void *arg) {
    (void)arg;
    for (int i = 0; i < 100000; i++) {
        counter++; /* a read, an add and a write, and another thread between them */
    }
    return NULL;
}

int main(void) {
    pthread_t a, b;
    pthread_create(&a, NULL, add, NULL);
    pthread_create(&b, NULL, add, NULL);
    pthread_join(a, NULL);
    pthread_join(b, NULL);
    printf("%ld\n", counter); /* 200000 if nothing went wrong, which something did */
    return 0;
}
c/ch12/bugs/race.c · region race
$ make race
WARNING: ThreadSanitizer: data race (pid=28355)
  Write of size 8 at 0x00010c76e000 by thread T2:
    #0 add race.c:12
  Previous write of size 8 at 0x00010c76e000 by thread T1:
    #0 add race.c:12
  Location is global 'counter' at 0x00010c76e000
SUMMARY: ThreadSanitizer: data race race.c:12 in add
200000

The count came out right, at 200,000, and the race was reported anyway. That is the difference between a detector and a test. The test checks the answer, which was correct by luck. ThreadSanitizer "is a tool that detects data races", and it saw two threads write the same eight bytes with nothing between them, whatever the answer. The clang documentation puts its cost at "about 5x-15x" in time. That is why the book runs it on its own target, make verify-c-tsan, rather than on every build.

A fuzzer is the last tool, and the scene showed what it does. The one that comes with clang is libFuzzer, "an in-process, coverage-guided, evolutionary fuzzing engine". It calls one function of yours millions of times with inputs it mutates. It watches which branches each input reaches, so that it can mutate the interesting ones further. The target for kv feeds the input through the parser and the executor, as the server would:

int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
    kv_store store = {0};
    kv_buf out = {0};
    const char *p = (const char *)data;
    size_t len = size;
    /* run every complete command in the input, as the server would */
    for (;;) {
        kv_cmd cmd;
        size_t used = 0;
        if (kv_parse(p, len, &cmd, &used) != KV_PARSE_OK) {
            break;
        }
        kv_exec(&cmd, &store, &out);
        p += used;
        len -= used;
    }
    kv_buf_free(&out);
    kv_store_free(&store);
    return 0;
}
c/ch12/fuzz/fuzz_parse.c · region target

The scene's target is the same shape, pointed at the parser that trusts its length field. The parser is the chapter 4 one with a single line missing, and the line is the one the chapter 4 gotcha was about:

/* One request, with the arguments copied out. The bulk length is trusted: nothing
 * checks it against the room in the argument. A length of 40 writes 40 bytes into 32,
 * and the attacker chooses the 8 that land on whatever the heap keeps next. */
int trusting_parse(const char *data, size_t len, trusting_cmd *cmd) {
    const char *end = data + len;
    const char *p = data;
    const char *eol = find_eol(p, end);
    if (eol == NULL || *p != '*') {
        return -1;
    }
    long n;
    if (!parse_number(p + 1, eol, &n) || n > TRUSTING_MAX_ARGS) {
        return -1;
    }
    p = eol + 1;
    for (long i = 0; i < n; i++) {
        if (p >= end || *p != '$' || (eol = find_eol(p, end)) == NULL) {
            return -1;
        }
        long blen;
        if (!parse_number(p + 1, eol, &blen)) {
            return -1;
        }
        p = eol + 1;
        if (end - p < blen + 2) {
            return -1;
        }
        memcpy(cmd->argv[i], p, (size_t)blen); /* the missing check was here */
        cmd->arglen[i] = (size_t)blen;
        p += blen + 2;
    }
    cmd->argc = (size_t)n;
    return 0;
}
c/ch12/bugs/trusting_parser.c · region bug

Two things made the two seconds possible. Each argument's buffer is a heap allocation of its own, so AddressSanitizer sees the first byte past it. And the fuzzer started from one valid request, so it did not have to discover what a request looks like. Without the seed, and with the arguments inside one struct, the same fuzzer ran twenty-two million inputs in a minute and found nothing. A fuzzer is only as good as the signal it gets, and a seed corpus and a sanitizer are most of that signal.

๐Ÿ–ฅApple's clang has no libFuzzer

The clang that comes with Xcode cannot link a fuzzer. The error is library 'libclang_rt.fuzzer_osx.a' not found. The fuzz targets of the Makefile run in a Linux container instead, with the one-line docker run in their comment. The clang 19 in the container links them without complaint. Against the real kv parser the fuzzer found nothing. That is the usual result after the unit tests have done their work, and no reason to skip it.

โœ…Run the tests under the sanitizers, every time C

A C test suite without sanitizers checks the answers and nothing else. With -fsanitize=address,undefined it also checks every memory access and every arithmetic operation the tests perform. The cost is about twice the run time, which for a test suite is nothing. kv's make test always builds with them, and so should yours. The one sanitizer that cannot join them is ThreadSanitizer, which needs a build of its own.

Go: the runner is in the box, and so is the fuzzer#

Go tests live in files that end in _test.go, in the same package as the code, and go test finds them by name. The parser test is a table, which is the Go shape for "the same check on many inputs". Each row is a subtest, so that a failure names the case. Three calls in it are the rest of the testing package. t.Run "runs f as a subtest of t called name". t.Parallel "signals that this test is to be run in parallel with (and only with) other parallel tests". The rows share nothing, so they may. And the comparison moved into a helper that calls t.Helper, so that a failure reports the row's line and not the helper's:

func TestReadCommand(t *testing.T) {
	cases := []struct {
		name  string
		input string
		want  []string
		err   error
	}{
		{"array", "*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n", []string{"ECHO", "hi"}, nil},
		{"inline", "  SET  a   b\r\n", []string{"SET", "a", "b"}, nil},
		{"bare LF", "PING\n", []string{"PING"}, nil},
		{"empty value", "*3\r\n$3\r\nSET\r\n$1\r\nk\r\n$0\r\n\r\n", []string{"SET", "k", ""}, nil},
		{"closed", "", nil, io.EOF},
		{"cut short", "*2\r\n$4\r\nEC", nil, io.ErrUnexpectedEOF},
		{"not a number", "*x\r\n", nil, ErrProtocol},
		{"wrong marker", "*1\r\n#4\r\nECHO\r\n", nil, ErrProtocol},
		{"too many", "*100\r\n", nil, ErrProtocol},
	}
	for _, c := range cases {
		t.Run(c.name, func(t *testing.T) {
			t.Parallel() // the cases share nothing, so they may run at once
			got, err := ReadCommand(bufio.NewReader(strings.NewReader(c.input)))
			if !errors.Is(err, c.err) {
				t.Fatalf("err = %v, want %v", err, c.err)
			}
			expectArgs(t, got, c.want)
		})
	}
}

// expectArgs compares the parsed arguments with the expected strings. t.Helper makes a
// failure report the caller's line, not this one.
func expectArgs(t *testing.T, got [][]byte, want []string) {
	t.Helper()
	if len(got) != len(want) {
		t.Fatalf("got %q, want %q", got, want)
	}
	for i := range got {
		if string(got[i]) != want[i] {
			t.Fatalf("arg %d = %q, want %q", i, got[i], want[i])
		}
	}
}
go/ch12/internal/resp/resp_test.go · region table
Exercise 12.2GoFuzz the parser

Write FuzzReadCommand. Add three seed inputs with f.Add. Then call f.Fuzz with a function that parses the input and returns on an error. When the parse succeeds, write the arguments back out with the protocol writers, parse them again, and fail the test if anything changed. That round trip is a property the parser must keep for every input, and it is the kind of check a fuzzer is good at.

func FuzzReadCommand(f *testing.F) {
	f.Add([]byte("*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n"))
	f.Fuzz(func(t *testing.T, data []byte) {
		// your code here
	})
}
Stuck?Show the full listing
// FuzzReadCommand lets the fuzzer mutate the input. Whatever it sends, the parser must
// not panic, and a request it accepts must survive a round trip through the writers.
func FuzzReadCommand(f *testing.F) {
	f.Add([]byte("*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n"))
	f.Add([]byte("PING\r\n"))
	f.Add([]byte("*1\r\n$3\r\nGET\r\n"))
	f.Fuzz(func(t *testing.T, data []byte) {
		args, err := ReadCommand(bufio.NewReader(bytes.NewReader(data)))
		if err != nil {
			return
		}
		var buf bytes.Buffer
		w := bufio.NewWriter(&buf)
		WriteArrayLen(w, len(args))
		for _, a := range args {
			WriteBulk(w, a)
		}
		w.Flush()
		again, err := ReadCommand(bufio.NewReader(&buf))
		if err != nil {
			t.Fatalf("round trip failed: %v", err)
		}
		if len(again) != len(args) {
			t.Fatalf("round trip changed the argument count")
		}
		for i := range args {
			if !bytes.Equal(again[i], args[i]) {
				t.Fatalf("round trip changed argument %d", i)
			}
		}
	})
}
go/ch12/internal/resp/resp_test.go · region fuzz
$ go test ./...                         # the seeds run as ordinary tests
ok  	kv/internal/resp	0.021s
$ go test -fuzz=FuzzReadCommand -fuzztime=10s ./internal/resp
fuzz: elapsed: 11s, execs: 581307 (50220/sec), new interesting: 34 (total: 37)
PASS

The fuzzer ran half a million inputs in ten seconds. Had one failed, Go would have written it to testdata/fuzz/FuzzReadCommand/. From then on go test would run it with the seeds, as a regression test, until the bug was fixed. The race detector from chapter 9 is the other built-in tool, and it is worth seeing what it prints. A scratch program that runs the chapter 5 INCR from two goroutines with no lock prints this under the detector:

$ go run -race .
WARNING: DATA RACE
Write at 0x00c000186000 by goroutine 7:
  runtime.mapassign_faststr()
  main.main.func1()
Previous read at 0x00c000186000 by goroutine 6:
  runtime.mapaccess2_faststr()
Goroutine 7 (running) created at:
  sync.(*WaitGroup).Go()

errors.Join#

One Go change in this chapter is not a test but a thing tests find. The log's Close from chapter 11 must sync the file and then close it, and either may fail. Returning the first error would skip the close. errors.Join "returns an error that wraps the given errors. Any nil error values are discarded". It "returns nil if every value in errs is nil". So both calls run, and the caller hears about both. errors.Is and errors.As from chapter 4 look inside a joined error as they look inside a wrapped one:

// The file is closed even when the sync failed, and the caller hears about both.
// errors.Join drops the nils, so with no error at all it returns nil.
return errors.Join(sync, l.f.Close())
go/ch12/internal/aof/aof.go · region join
โš ๏ธA detector is not a proof Go

The race detector and the fuzzer report what they saw, not what exists. A race on a path the test never ran is not reported. An input the fuzzer never generated is not tried. Both tools are excellent and both are statistical. That is why kv runs the race detector in make verify on every change and keeps the fuzz corpus. The number of things seen then keeps growing. The next section shows the other kind of tool.

Rust: the compiler already ran, and the tools check the rest#

Rust has three places for a test, and kv now uses all three. The first is the file under test, in a module marked #[cfg(test)]. The attribute "tells Rust to compile and run the test code only when you run cargo test, not when you run cargo build". A helper in that module is compiled only for tests too, which is where a function that turns a typed line into a reply belongs:

#[cfg(test)]
mod tests {
    use super::*;
    use crate::store::MapStore;

    /// A command typed as one inline line, run against the store, and its reply as
    /// text. The helper is compiled only for tests, like the module around it.
    fn run(store: &mut MapStore, line: &str) -> String {
        let args: Command = line.split(' ').map(str::as_bytes).collect();
        let mut out = Vec::new();
        execute(&args, store, &mut out).unwrap();
        String::from_utf8(out).unwrap()
    }

    #[test]
    fn incr_counts_from_nothing() {
        let mut store = MapStore::new();
        assert_eq!(run(&mut store, "INCR n"), ":1\r\n");
        assert_eq!(run(&mut store, "INCR n"), ":2\r\n");
        assert_eq!(run(&mut store, "GET n"), "$1\r\n2\r\n");
    }

    #[test]
    fn an_expiry_in_the_past_deletes_the_key() {
        let mut store = MapStore::new();
        run(&mut store, "SET k v");
        assert_eq!(run(&mut store, "EXPIRE k 0"), ":1\r\n");
        assert_eq!(run(&mut store, "EXISTS k"), ":0\r\n");
    }
}
rust/ch12/src/cmd.rs · region test

The second place is the documentation. A code block in a doc comment is a test, because "rustdoc supports executing your documentation examples as tests", and cargo test runs them. The example in the comment on parse is the one a reader of the docs sees. It cannot go stale, because a stale example stops the build:

/// One request from the front of `buf`: an array of bulk strings such as
/// `*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n`, or one inline line of words. The returned
/// `Command` borrows `buf`, which the signature says with the elided lifetime: one
/// reference in, so the one reference out lives as long as it.
///
/// A code block in a doc comment is a test. `cargo test` compiles and runs it, and
/// `cargo doc` shows it, so an example that stops being true stops the build:
///
/// ```
/// use kv::resp::{Parsed, parse};
/// let Ok(Parsed::Done(args, used)) = parse(b"*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n") else {
///     unreachable!("a whole request")
/// };
/// assert_eq!(args, [&b"ECHO"[..], b"hi"]);
/// assert_eq!(used, 22);
/// assert!(matches!(parse(b"*2\r\n$4\r\nEC"), Ok(Parsed::Incomplete)));
/// ```
pub fn parse(buf: &[u8]) -> Result<Parsed<'_>> {
    let Some((line, mut used)) = take_line(buf, 0) else {
        return Ok(Parsed::Incomplete);
    };
    if line.first() != Some(&b'*') {
        let words = line.split(|b| *b == b' ').filter(|w| !w.is_empty());
        return Ok(Parsed::Done(words.collect(), used));
    }
    let n = parse_len(&line[1..], b'*')?;
    if n > MAX_ARGS {
        return Err(KvError::Protocol(b'*'));
    }
    let mut args = Vec::with_capacity(n);
    for _ in 0..n {
        let Some((line, start)) = take_line(buf, used) else {
            return Ok(Parsed::Incomplete);
        };
        if line.first() != Some(&b'$') {
            return Err(KvError::Protocol(line.first().copied().unwrap_or(b'\n')));
        }
        let size = parse_len(&line[1..], b'$')?;
        if size > MAX_BULK {
            return Err(KvError::Protocol(b'$'));
        }
        let end = start + size;
        if buf.len() < end + 2 {
            return Ok(Parsed::Incomplete);
        }
        if &buf[end..end + 2] != b"\r\n" {
            return Err(KvError::Protocol(buf[end]));
        }
        args.push(&buf[start..end]);
        used = end + 2;
    }
    Ok(Parsed::Done(args, used))
}

/// The line that starts at `from`, without its ending, and where the next line starts.
/// A bare LF is accepted as well as CRLF, so that a command typed into nc works.
fn take_line(buf: &[u8], from: usize) -> Option<(&[u8], usize)> {
    let rest = &buf[from..];
    let nl = rest.iter().position(|b| *b == b'\n')?;
    let line = rest[..nl].strip_suffix(b"\r").unwrap_or(&rest[..nl]);
    Some((line, from + nl + 1))
}

fn parse_len(digits: &[u8], type_byte: u8) -> Result<usize> {
    std::str::from_utf8(digits)
        .ok()
        .and_then(|s| s.parse().ok())
        .ok_or(KvError::Protocol(type_byte))
}
rust/ch12/src/resp.rs · region parse

The third place is the tests/ directory next to src/. Each file there is an integration test. "Cargo will compile each of the files as an individual crate", and a crate of its own sees only what the library makes pub. That is the point. An integration test cannot reach into private functions, so it tests the crate as a user would. The one in kv runs a request through the parser and the executor, the way the server does:

/// One request through the parser and the executor, as the server runs it, and the
/// reply bytes. A helper that every test in the file shares is an ordinary function.
fn round_trip(store: &mut MapStore, request: &[u8]) -> Vec<u8> {
    let mut out = Vec::new();
    match resp::parse(request) {
        Ok(Parsed::Done(args, used)) => {
            assert_eq!(used, request.len(), "the whole request is used");
            cmd::execute(&args, store, &mut out).expect("a Vec never fails to write");
        }
        other => panic!("not a whole request: {other:?}"),
    }
    out
}
rust/ch12/tests/roundtrip.rs · region helper
#[test]
fn set_then_get() {
    let mut store = MapStore::new();
    assert_eq!(
        round_trip(&mut store, b"*3\r\n$3\r\nSET\r\n$1\r\nk\r\n$2\r\nhi\r\n"),
        b"+OK\r\n"
    );
    assert_eq!(
        round_trip(&mut store, b"*2\r\n$3\r\nGET\r\n$1\r\nk\r\n"),
        b"$2\r\nhi\r\n"
    );
}

#[test]
fn the_wrong_kind_is_an_error_and_not_a_panic() {
    let mut store = MapStore::new();
    round_trip(&mut store, b"*3\r\n$5\r\nLPUSH\r\n$1\r\nl\r\n$1\r\nx\r\n");
    let reply = round_trip(&mut store, b"*2\r\n$3\r\nGET\r\n$1\r\nl\r\n");
    assert!(
        reply.starts_with(b"-WRONGTYPE"),
        "{}",
        String::from_utf8_lossy(&reply)
    );
}

#[test]
fn an_unknown_command_is_named_in_the_error() {
    let mut store = MapStore::new();
    assert_eq!(
        round_trip(&mut store, b"*1\r\n$4\r\nNOPE\r\n"),
        b"-ERR unknown command 'NOPE'\r\n"
    );
}
rust/ch12/tests/roundtrip.rs · region tests
Exercise 12.3RustA request survives a round trip

Write the unit test a_request_survives_a_round_trip in the parser's test module. Build a command with three arguments, one of which contains a line ending. Write it with write_array_len and write_bulk into a Vec<u8>. Parse the vector, and assert that the result equals the command. The line ending inside a value is the case that a length-prefixed format exists for.

#[test]
fn a_request_survives_a_round_trip() {
    let args = vec![b"SET".to_vec(), b"k".to_vec(), b"a value\r\nwith a line".to_vec()];
    // your code here
}
Stuck?Show the full listing
#[cfg(test)]
mod tests {
    use super::*;

    /// The parser works on bytes already in memory, so a test needs no socket.
    fn parse_one(input: &[u8]) -> Result<Parsed<'_>> {
        parse(input)
    }

    fn args_of(input: &[u8]) -> Vec<Vec<u8>> {
        match parse_one(input).unwrap() {
            Parsed::Done(args, used) => {
                assert_eq!(used, input.len());
                args.iter().map(|a| a.to_vec()).collect()
            }
            Parsed::Incomplete => panic!("incomplete"),
        }
    }

    #[test]
    fn an_array_of_bulk_strings() {
        assert_eq!(
            args_of(b"*2\r\n$4\r\nECHO\r\n$2\r\nhi\r\n"),
            vec![b"ECHO".to_vec(), b"hi".to_vec()]
        );
    }

    #[test]
    fn an_inline_line_and_a_bare_lf() {
        let got = args_of(b"  SET  a   b\n");
        assert_eq!(got.len(), 3);
        assert_eq!(got[2], b"b");
    }

    #[test]
    fn nothing_yet_is_incomplete() {
        for cut in [&b""[..], b"*2\r\n$4\r\nEC", b"*2\r\n$4\r\nECHO\r\n"] {
            assert!(matches!(parse_one(cut).unwrap(), Parsed::Incomplete));
        }
    }

    #[test]
    fn bad_bytes_are_a_protocol_error() {
        for bad in [&b"*x\r\n"[..], b"*1\r\n#4\r\nECHO\r\n", b"*100\r\n"] {
            assert!(matches!(parse_one(bad).unwrap_err(), KvError::Protocol(_)));
        }
    }

    #[test]
    fn a_request_survives_a_round_trip() {
        let args = vec![
            b"SET".to_vec(),
            b"k".to_vec(),
            b"a value\r\nwith a line".to_vec(),
        ];
        let mut bytes = Vec::new();
        write_array_len(&mut bytes, args.len()).unwrap();
        for a in &args {
            write_bulk(&mut bytes, a).unwrap();
        }
        assert_eq!(args_of(&bytes), args);
    }
}
rust/ch12/src/resp.rs · region test

cargo test, and what comes after the dashes#

cargo test runs every unit test, every integration test and every doc test, in that order, each set in a binary of its own. --lib runs only the unit tests, --test roundtrip only that integration file, and --doc only the documentation. A name after the command is a filter, so cargo test incr runs the tests with incr in their name. Everything after a lone -- goes to the test binary rather than to cargo. Two are worth knowing. -- --no-capture shows what a test prints, which the harness hides unless the test fails. -- --test-threads 1 runs the tests one at a time, because Rust runs them in parallel by default and Go only when asked.

Fuzzing in Rust is cargo-fuzz, which the fuzzing book calls "the recommended tool for fuzz testing Rust code". It drives the same libFuzzer that C used, through a crate called libfuzzer-sys, and it needs the nightly toolchain. The targets live in a small crate of their own under fuzz/, and the target for kv is the C one in Rust:

fuzz_target!(|data: &[u8]| {
    let mut input = data;
    let mut store = kv::store::MapStore::new();
    let mut out = Vec::new();
    // run every complete command in the input, as the server would
    while let Ok(kv::resp::Parsed::Done(args, used)) = kv::resp::parse(input) {
        let _ = kv::cmd::execute(&args, &mut store, &mut out);
        input = &input[used..];
    }
});
rust/ch12/fuzz/fuzz_targets/parse.rs · region target
$ cargo install cargo-fuzz
$ cargo +nightly fuzz run parse -- -max_total_time=20
#343809	DONE   cov: 335 ft: 1611 corp: 356/36Kb exec/s: 16371
Done 343809 runs in 21 second(s)

The last tool is Miri, an interpreter that runs a Rust program slowly and checks every memory access against the rules. Its README calls it "an Undefined Behavior detection tool for Rust", and it finds the same things AddressSanitizer finds in C. The server kv has no unsafe code of its own, so Miri has nothing to find until chapter 15. It still runs:

$ rustup +nightly component add miri
$ cargo +nightly miri test --lib
test result: ok. 12 passed; 0 failed; finished in 92.81s

Ninety seconds for tests that take a hundredth of a second natively. That is the cost of checking rules at run time that the compiler could not check at compile time. Safe Rust avoids that cost by having the compiler check them first.

โš ๏ธTwo toolchains, and a crate inside a crate Rust

cargo-fuzz and Miri both need nightly, and the fuzz crate must stay out of the main one. The nightly toolchain is a second compiler installed beside the stable one, chosen per command with +nightly. The fuzz/ directory is a separate package with an empty [workspace] table. The main Cargo.toml has one too. So a cargo command in one directory never reads the files of the other. Forget the one in fuzz/ and the first error names a workspace you did not know you had.

Fuzzing, three ways, one engine#

All three fuzzers in this chapter are coverage-guided. They watch which branches an input reaches and mutate the inputs that reach new ones. That is why a fuzzer finds the corner cases of a parser within seconds. Two of the three fuzzers are the same engine. The engine called libFuzzer is what clang links into the C target and what libfuzzer-sys links into the Rust one. Go wrote its own, with the same design, and built it into go test. The rates on the author's machine were 17,000 inputs a second for Rust and 50,000 for Go. The small C target in a container ran at 350,000. None found a bug in kv's parser. The unit tests had already covered the cases a fuzzer finds first, which is the order the book recommends. The fuzzer's own case is the one in the scene, the input that nobody writes down.

What finds what#

The bugCGoRust
Reading freed memoryASan at run time, if a test reaches itcannot be writtencannot be written
A data raceThreadSanitizer, in its own build-race at run time, if a test reaches itthe compiler, always
Integer overflowUBSan at run timewraps, by the language definitionpanics in a debug build, wraps in release
A panic or crash on bad inputa sanitizer, or a core dumpthe fuzzer, in go testthe fuzzer, on nightly
A wrong answeryour testsyour testsyour tests

The table is the chapter in four lines. C finds its bugs with tools that watch the program run, which means a test must reach the bug. Go has two such tools built in, and a language definition that removes the undefined cases. Rust removes two whole rows at compile time, and uses the same run-time tools as C for what remains. The last row is the same for all three, and it is where most bugs live.

Verdict##

TopicCGoRustWhy
A test runnerWeakStrongStrongC has none, so you write one. Go and Rust have one in the box, with no setup.
Where a test may liveWeakOKStrongC has the file you write, and Go the package and an Example. Rust has the module, the docs and a directory for outsiders.
FuzzingOKStrongOKlibFuzzer is one flag on Linux and a container on a Mac, Go has it built in, and Rust needs nightly and a second crate.
Memory errorsOKStrongStrongC finds them at run time with ASan or Valgrind. Go and Rust cannot write them in safe code.
Data racesOKOKStrongC and Go find the races a test provokes, with a sanitizer or the detector. Rust refuses them at compile time.
Undefined behaviourWeakStrongOKC has it everywhere and UBSan finds some. Go has almost none, a data race being the exception, and Rust confines it to unsafe, where Miri checks it.
Speed of the toolsStrongStrongWeakASan costs two times and the race detector two to twenty, while Miri costs nine thousand.
Carry forward
  • A detector sees what ran. A sanitizer, the race detector and a fuzzer report the bugs a test reached. The compiler's refusal is the only kind of proof.
  • Seed and signal. A fuzzer needs a valid input to start from and a sanitizer to tell it when it has won. With both, two seconds. Without, twenty-two million runs for nothing.
  • Three homes for a Rust test. The cfg(test) module sees everything, the doc comment is read by users, and the tests directory sees only what is pub.
Next

One request through the three parsers, ten million times each. Go takes three times as long as Rust and nine times as long as C. Chapter 13 finds out why, and whether it matters.

Benchmarking and profiling#

Quality and shipping · Chapter 13· 24 min read· 235 lines of code
▶Scene

Time the same request through the three parsers, ten million times each. Each language has its own way to say "run this loop and divide", and the chapter shows all three. The answers, on the author's machine:

C, kv_parse            56 ns      0 allocations
Rust, parse           179 ns      1 allocation
Go, ReadCommand       532 ns      9 allocations, 184 bytes

Go is three times slower than Rust and nine times slower than C, on the one piece of kv that is pure computation. The chapter finds out why, finds out whether it matters, and halves it. Then it turns to the servers, where a Rust number is the one that is wrong.

The book has quoted numbers since chapter 6. This chapter shows how to take one that means something. It also shows how to find where the time goes when the number is worse than you expected. It benchmarks and profiles all four servers, and follows two bad results from a profile to a fix. Measurement is the only argument that ends a discussion about speed.

Chapter 9 reported that the four servers answer about 70,000 to 100,000 requests a second from fifty clients. That number is true and nearly useless, because it measures a round trip on the loopback interface more than it measures kv. The Redis documentation says so in its own words. When the client and the server share a machine, "the CPU is the limiting factor with redis-benchmark". By default the tool "does not use pipelining or any parallelism at all". A benchmark is a question, and a question has to be chosen.

MeasuringCGoRust
The load generatorredis-benchmarkthe samethe same
A micro-benchmarka loop and the clockgo test -benchan example binary, or criterion
Counting allocationsnone to count-benchmema global allocator that counts
What the compiler decided-O0 to -O3, measured-gcflags=-mthe release profile, LTO
A CPU profilesample on macOS, perf on Linuxruntime/pprof, then go tool pprofsample or perf, with symbols kept in the release build
Memory knobsnone: you free itGOGC and GOMEMLIMITnone: you drop it
The surprisenonenine allocations a requesta wall at 200,000 requests a second with pipelining
The fixnone neededfive allocations, from a free list the reader already hada read lock, after four experiments that failed

Taking a number#

Every number in this chapter comes from one command shape. Fifty connections, four client threads so that the client is not the bottleneck, two hundred thousand requests, and two settings of the pipeline. A pipeline of 16 means each connection sends sixteen commands before it waits for a reply. The documentation calls pipelining "a feature often exploited by real world applications". It changes what is measured. Without it, the number is mostly the cost of a round trip. With it, the number is mostly the cost of the server:

$ redis-benchmark -p 6380 -c 50 -n 200000 --threads 4 -t set,get -q          # one at a time
$ redis-benchmark -p 6380 -c 50 -n 200000 --threads 4 -t set,get -q -P 16    # sixteen at a time
                         no pipeline              pipeline of 16
                         SET       GET            SET       GET
C, poll, one thread      88,700    79,900         796,800   800,000
Go, goroutines           88,700    88,700         800,000   800,000
Rust, threads            99,950    88,700         159,700   199,400
Rust, tokio              99,950    99,850         160,000   199,800

The left half says the four servers are the same, within noise, because the left half measures the loopback. The right half says something is wrong with the two Rust servers. Both stop at a fifth or a quarter of what C and Go reach. The second half of the chapter is about that column, and the tools that explain it. First, the shape of a single measurement. The full output of the benchmark, without -q, gives a distribution and not one number:

Summary:
  throughput summary: 79617.83 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        0.540     0.064     0.487     0.919     1.319     5.103

The median request took half a millisecond. One in a hundred took more than 1.3 milliseconds, and the slowest took five. A server is judged by the p99 column more often than by the average, because the average hides the requests that a user notices.

๐Ÿ“œFlame graphs, and what a profile is

A profiler interrupts a program many times a second and writes down what it was doing. The result is a list of stacks with a count against each, and the count says where the time went. In 2011 Brendan Gregg drew those stacks as a picture he called a flame graph. He describes it as "a visualization of hierarchical data that I created to visualize stack traces of profiled software". Each box is a function, and its width is the share of samples that passed through it. The boxes above it are the functions it called. The x axis "is not the passage of time". Every profiler in this chapter produces the same kind of data, and every one of them can draw that picture. The chapter reads the lists instead, because a list fits in a book.

C: a clock, a loop and five flags#

A server profile answers "where does the time go". A micro-benchmark answers "how long does this function take", which is a different question with a different tool. Go has one built in, and Rust has one on the nightly toolchain. C has a clock, which is enough:

Exercise 13.1CTime the parser

Write a benchmark for kv_parse. Build one SET request as a string and read the monotonic clock, with kv_now_ms from chapter 10. Parse the request ten million times in a loop. Read the clock again, and print the nanoseconds for one parse. Count the successful parses and print the count too, so that the compiler cannot remove the loop as unused work. The Makefile target bench builds it with the release flags and no sanitizer.

int main(void) {
    const char *req = "*3\r\n$3\r\nSET\r\n$16\r\nkey:000000001234\r\n$3\r\nxxx\r\n";
    /* your code here */
}
Stuck?Show the full listing
int main(void) {
    const char *req = "*3\r\n$3\r\nSET\r\n$16\r\nkey:000000001234\r\n$3\r\nxxx\r\n";
    size_t len = strlen(req);
    long runs = 10 * 1000 * 1000;
    kv_cmd cmd;
    size_t used = 0;
    long ok = 0;
    int64_t start = kv_now_ms();
    for (long i = 0; i < runs; i++) {
        ok += kv_parse(req, len, &cmd, &used) == KV_PARSE_OK;
    }
    int64_t elapsed = kv_now_ms() - start;
    /* ok is printed so that the compiler cannot remove the loop as unused work */
    printf("%ld parses in %lld ms: %.1f ns each (%ld ok)\n", runs, (long long)elapsed,
           elapsed * 1e6 / (double)runs, ok);
    return 0;
}
c/ch13/bench/bench_parse.c · region bench

The optimisation levels, measured#

The one number that changes a C benchmark most is a compiler flag. The book has built with -O2 since chapter 2, and the Makefile has a target that builds the same benchmark at every level. The gcc manual describes -O0 as "Reduce compilation time and make debugging produce the expected results. This is the default." At -O2 the compiler "performs nearly all supported optimizations that do not involve a space-speed tradeoff". -O3 adds the ones that do, such as unrolling loops. -Os optimises for size:

$ make bench-levels
-O0  10000000 parses in 998 ms: 99.8 ns each
-O1  10000000 parses in 560 ms: 56.0 ns each
-O2  10000000 parses in 586 ms: 58.6 ns each
-O3  10000000 parses in 551 ms: 55.1 ns each
-Os  10000000 parses in 622 ms: 62.2 ns each

The whole of the gain is between -O0 and -O1, and the rest is noise. That is the usual shape. A parser is small functions and short loops, and the first level already inlines the small functions, such as kv_find_eol, into their one caller. The keyword inline would ask for the same thing by name. But its intent "is to serve as a hint", and "the compilers can (and usually do) ignore presence or absence of the inline specifier". At -O1 and above the compiler has already made that decision from the size of the function, and kv has no inline anywhere. A second keyword makes a promise rather than a request. restrict on a pointer parameter says that nothing else reaches the memory it points to, so the compiler may reorder loads and stores freely. The benchmark below adds two arrays with and without it:

/* Without restrict, the compiler must allow for dst and src overlapping, so each store
 * may change the next load, and the loop is kept in order or guarded by a check. With
 * the promise, the loop may add eight or sixteen numbers in one instruction. */
static void add_plain(int32_t *dst, const int32_t *src, size_t n) {
    for (size_t i = 0; i < n; i++) {
        dst[i] += src[i];
    }
}

static void add_restrict(int32_t *restrict dst, const int32_t *restrict src, size_t n) {
    for (size_t i = 0; i < n; i++) {
        dst[i] += src[i];
    }
}
c/ch13/bench/bench_copy.c · region copy
$ make bench-copy
plain    62 ms for 100000 passes over 4096 ints
restrict 62 ms for 100000 passes over 4096 ints (100% of the time, sum 1677312000000)

No difference, and the reason is worth more than a difference would have been. clang vectorises both loops. For the plain one it adds a run-time check that the arrays do not overlap, and takes the fast path when they do not. They never do. The promise bought nothing that the compiler could not check itself. A broken promise is undefined behaviour, as the C reference says in its own words. The lesson is the chapter's. A keyword that is supposed to make code faster is a hypothesis, and the benchmark is the experiment.

A sampler and a short answer#

macOS ships a sampling profiler called sample. Given a process and a number of seconds, it writes every stack it saw. It adds a summary sorted by the function at the top of the stack. The C server was sampled for five seconds while the benchmark ran:

$ sample $(pgrep kv) 5 -file kv.sample
$ grep -A12 "Sort by top of stack" kv.sample
        read   (in libsystem_kernel.dylib)        1741
        write  (in libsystem_kernel.dylib)        1561
        poll   (in libsystem_kernel.dylib)         427
        _platform_memmove                           67
        kv_server_run  (in kv)                      31
        kv_parse  (in kv)                           17
        strncasecmp_l                               16
        kv_exec  (in kv)                            13

Three system calls hold ninety percent of the samples. The parser and the command code together hold one percent. The C server is not spending its time in C. It is spending it in the kernel, moving bytes in and out of sockets, and no change to kv_parse would be visible in the benchmark. That is the most common result of profiling a server, and the most useful, because it stops you from optimising the wrong thing. On Linux the same question is asked with perf record, which the manual describes as "Run a command and record its profile into perf.data". It is answered with perf report.

โœ…Measure before you optimise, and measure after C

A profile is cheap and a guess is expensive. The sampler above took five seconds to run and ten seconds to read, and it closed a question that could have cost a day. The second half of the rule matters as much. Every change made for speed is measured again with the same command. A change that does not move the number is reverted, however clever it was.

Go: nine allocations, and the flag that names them#

The Go benchmark is a function in a test file, and the testing package runs it. Functions "of the form func BenchmarkXxx(*testing.B) are considered benchmarks". b.Loop runs the body for as long as the package needs to get a stable number. The flag -benchmem adds two columns that the scene's number came from:

// BenchmarkReadCommand times the parser on one SET request. go test -bench runs the loop
// for as long as it needs to get a stable number. The reader is reused, so that the
// number is the parser and not the cost of a 4 KB buffer per request.
func BenchmarkReadCommand(b *testing.B) {
	req := []byte("*3\r\n$3\r\nSET\r\n$16\r\nkey:000000001234\r\n$3\r\nxxx\r\n")
	r := bufio.NewReader(nil)
	for b.Loop() {
		r.Reset(bytes.NewReader(req))
		if _, err := ReadCommand(r); err != nil {
			b.Fatal(err)
		}
	}
}
go/ch13/internal/resp/bench_test.go · region bench
$ go test -run xxx -bench ReadCommand -benchmem ./internal/resp/
BenchmarkReadCommand-12    	 2267204	       532.2 ns/op	     184 B/op	       9 allocs/op

Escape analysis#

Nine allocations for a request with three arguments. Go decides for each value whether it may live on the goroutine's stack, which is free, or must go on the heap. There the garbage collector (GC) will find it later. The guide to the collector says a value escapes to the heap when "the Go compiler cannot determine its lifetime". The compiler will say which, for every line, with one flag. The manual describes -m as "Print optimization decisions. Higher values or repetition produce more detail":

$ go build -gcflags=-m ./internal/resp/ 2>&1 | grep "escapes to heap"
internal/resp/resp.go:54:14: make([][]byte, 0, n) escapes to heap
internal/resp/resp.go:67:14: make([]byte, size + 2) escapes to heap
internal/resp/resp.go:77:16: append escapes to heap

Those three are the arguments and the slice that holds them, four allocations the parser must make, because its caller keeps them. The other five were a surprise. bufio.Reader.ReadBytes returns a fresh slice for each line, four lines in this request, and strconv.Atoi wants a string, which the conversion from bytes allocates. Both values are dead by the next line of the parser. The reader has a second method, ReadSlice, which returns a view into the reader's own buffer, valid until the next read. That is exactly as long as the parser needs it. The number is parsed from the bytes without a string:

// readLine returns one line without its line ending. ReadSlice hands back a view into
// the reader's own buffer, valid until the next read, so no allocation is made for a
// line. Chapter 13 measured the difference. A line longer than the buffer is an error,
// which for a length field it should be.
func readLine(r *bufio.Reader) ([]byte, error) {
	line, err := r.ReadSlice('\n')
	if err != nil {
		if err == io.EOF && len(line) > 0 {
			return nil, io.ErrUnexpectedEOF
		}
		if err == bufio.ErrBufferFull {
			return nil, protoErr(line[0])
		}
		return nil, err
	}
	line = bytes.TrimSuffix(line, []byte("\n"))
	return bytes.TrimSuffix(line, []byte("\r")), nil
}

// atoi parses a non-negative decimal from bytes without making a string of them, which
// strconv.Atoi would need and the compiler would allocate.
func atoi(b []byte) (int, bool) {
	if len(b) == 0 || len(b) > 18 {
		return 0, false
	}
	n := 0
	for _, c := range b {
		if c < '0' || c > '9' {
			return 0, false
		}
		n = n*10 + int(c-'0')
	}
	return n, true
}
go/ch13/internal/resp/resp.go · region readline
$ go test -run xxx -bench ReadCommand -benchmem ./internal/resp/
BenchmarkReadCommand-12    	 3196206	       369.9 ns/op	     162 B/op	       5 allocs/op

From 532 to 370 nanoseconds, and from nine allocations to five, for a change of fifteen lines that the flag pointed at. The five that remain are the ones the caller needs, plus the error values that are never made on the happy path. The Go parser is still twice as slow as the Rust one, which allocates once. It is six times slower than the C one, which allocates nothing. Each of those is a design, not a flaw. Go copies so that the arguments are safe to keep. The rest of the gap is the collector's accounting on every make, which is the price of not writing free.

GOGC, GOMEMLIMIT and the Pool#

The collector has two knobs, both environment variables, and both are worth one experiment. "The GOGC variable sets the initial garbage collection target percentage", and "the default is GOGC=100". A collection runs when the heap has grown by that percentage since the last one. The guide puts the trade in one sentence. "Doubling GOGC will double heap memory overheads and roughly halve GC CPU cost." The second knob is a ceiling. "The GOMEMLIMIT variable sets a soft memory limit for the runtime", and with GOGC=off the collector runs only when the limit is near. The server was run under each setting with the pipelined SET benchmark, and the collections counted with GODEBUG=gctrace=1:

setting                      SET a second   collections   resident memory
GOGC=100 (the default)       796,800        11            10 MB
GOGC=25                      800,000        41             8 MB
GOGC=400                     800,000         2            21 MB
GOGC=off GOMEMLIMIT=32MiB    800,000         1            26 MB

The throughput did not move, because this server allocates little and the collector is not where its time goes. The memory did, in the direction the guide says. A server that must fit in a container with a fixed memory should set the limit. The guide recommends it when "the Go program is the only program with access to some set of resources". sync.Pool, from chapter 9, is the third tool. It is "a set of temporary objects that may be individually saved and retrieved", so that a buffer is reused instead of collected. The five remaining allocations are not worth one. A pool pays when the object is large and the rate is high. The reader's 4 KB buffer is the one object in kv that fits, and it lives for the whole connection already.

The profiler is a package#

Go's profiler is in the standard library. The package runtime/pprof "writes runtime profiling data in the format expected by the pprof visualization tool", and starting it is one call. The chapter 13 server starts a CPU profile when an environment variable names a file, and the clean stop from chapter 10 ends it:

Exercise 13.2GoProfile on request

At the start of main, if KV_CPUPROFILE names a file, create the file and call pprof.StartCPUProfile. Then arrange for the profile to be stopped and the file closed when main returns. That is after the clean stop has waited for every client. Two defer calls do it, in the order that runs last first. Without the stop, the profile file stays empty.

if path := os.Getenv("KV_CPUPROFILE"); path != "" {
	// your code here
}
Stuck?Show the full listing
// KV_CPUPROFILE names a file. The profile runs until the server is told to stop, with
// Ctrl-C or kill, so that a benchmark can be profiled from start to end.
if path := os.Getenv("KV_CPUPROFILE"); path != "" {
	f, err := os.Create(path)
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	if err := pprof.StartCPUProfile(f); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	defer f.Close()
	defer pprof.StopCPUProfile()
}
go/ch13/cmd/kv/main.go · region profile
$ KV_CPUPROFILE=cpu.prof ./kv &
$ redis-benchmark -p 6380 -c 50 -n 300000 --threads 4 -t get -q
$ kill -INT %1
$ go tool pprof -top kv cpu.prof
Duration: 4.40s, Total samples = 11.61s (263.58%)
      flat  flat%   sum%        cum   cum%
     6.51s 56.07% 56.07%      6.51s 56.07%  syscall.rawsyscalln
     1.89s 16.28% 72.35%      1.89s 16.28%  runtime.pthread_cond_wait
     1.20s 10.34% 82.69%      1.20s 10.34%  runtime.kevent
     0.82s  7.06% 89.75%      0.82s  7.06%  runtime.pthread_cond_signal
     0.01s 0.086% 97.59%      3.66s 31.52%  internal/poll.(*FD).Read
     0.01s 0.086% 97.67%      2.88s 24.81%  internal/poll.(*FD).Write

The total is 263 percent, because the profile adds up the time of every thread, and the server used almost three cores. More than half of that time was in system calls. A tenth was in kevent, which is the Go runtime asking the kernel which sockets are ready. Most of the rest was in the scheduler, parking and waking threads. The cum column says a third of all time was spent somewhere under Read. Again, the code that kv wrote is not in the list, and the nine allocations that the micro-benchmark found are invisible at the server's scale. Both measurements are true. The server one says which to act on.

Rust: one allocation, a profile, four experiments, and a lock#

Stable Rust has no benchmark runner in the box. The one most projects use is criterion, "a statistics-driven micro-benchmarking tool". It runs under cargo bench and compares each run with the last. The server kv keeps to the standard library, and its benchmark is an example binary with the C loop in it. It adds one thing the C version could not. A program may name its own allocator, with "a static item whose type implements the GlobalAlloc trait", and this one counts:

/// Every allocation in the program goes through the global allocator, and a program may
/// name its own. This one counts the calls and passes them on to the system allocator.
struct Counting;

static ALLOCS: AtomicU64 = AtomicU64::new(0);

// SAFETY: every call is passed through to System unchanged, which is itself a correct
// allocator, and the counter has no effect on the memory handed back.
unsafe impl GlobalAlloc for Counting {
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        ALLOCS.fetch_add(1, Ordering::Relaxed);
        unsafe { System.alloc(layout) }
    }

    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
        unsafe { System.dealloc(ptr, layout) }
    }
}

#[global_allocator]
static GLOBAL: Counting = Counting;
rust/ch13/examples/bench_parse.rs · region counting
fn main() {
    let req: &[u8] = b"*3\r\n$3\r\nSET\r\n$16\r\nkey:000000001234\r\n$3\r\nxxx\r\n";
    let runs = 10_000_000u32;
    let mut ok = 0u32;
    let before = ALLOCS.load(Ordering::Relaxed);
    let start = Instant::now();
    for _ in 0..runs {
        // the parser takes a slice and copies nothing, so no allocation is made per argument
        ok += u32::from(matches!(
            kv::resp::parse(req),
            Ok(kv::resp::Parsed::Done(..))
        ));
    }
    let elapsed = start.elapsed();
    let allocs = ALLOCS.load(Ordering::Relaxed) - before;
    println!(
        "{runs} parses in {} ms: {:.1} ns each ({ok} ok), {} allocations a parse",
        elapsed.as_millis(),
        elapsed.as_nanos() as f64 / f64::from(runs),
        allocs / u64::from(runs)
    );
}
rust/ch13/examples/bench_parse.rs · region bench

The release profile#

The counter says one allocation a parse, the Vec that holds the borrowed arguments. The time depends on the profile. The dev profile of Cargo has no optimisation and full debug information. The release profile is opt-level = 3. The same example, both ways, and then with two settings the release profile leaves off by default. lto = "fat", link-time optimisation (LTO), optimises across every crate at once. codegen-units = 1 gives the compiler the whole crate in one piece, at the cost of a slower build:

cargo run --example bench_parse                        1460.9 ns   (dev: opt-level 0)
cargo run --release --example bench_parse               179.0 ns   (release: opt-level 3)
  with lto = "fat" and codegen-units = 1                149.2 ns
  with #[inline(never)] on the two helpers              185.2 ns

Eight times between the two profiles, which is why a Rust benchmark run without --release measures nothing. The last line is the Rust form of the C inline experiment. The attribute #[inline] "suggests whether a copy of the attributed function's code should be placed in the caller". The reference says that "in every form the attribute is a hint". Forbidding it on the two helpers cost three percent, which is what the compiler's own decision was worth, and kv does not write the attribute. The release build keeps line information when Cargo.toml says so, which costs nothing at run time and lets a profiler name inlined functions:

# Keep function names in the release build, so a profiler can show them. The code is
# as fast as without this; only the binary is larger.
[profile.release]
debug = 1
rust/ch13/Cargo.toml · region profile

The Rust servers stop at 200,000 requests a second with pipelining, and the first question is where the time goes. The sampler names Rust functions, and its summary for the thread server under load is this:

__recvfrom          (in libsystem_kernel.dylib)    67711
__accept            (in libsystem_kernel.dylib)     1411
__semwait_signal    (in libsystem_kernel.dylib)     1409
__sendto            (in libsystem_kernel.dylib)     1252
__psynch_mutexwait  (in libsystem_kernel.dylib)      689
core::slice::memchr::memchr_aligned  (in kv)         256
kv::cmd::execute    (in kv)                           25
kv::resp::read_line (in kv)                           24
โš ๏ธA wall-clock sampler counts sleeping threads Rust

Sixty-seven thousand samples in recvfrom are not CPU time. The sampler records every thread on every tick, and the thread server has fifty threads that spend most of their lives blocked in a read. The C server has one thread, so its list was honest. The Rust list is dominated by waiting. The lines that describe work are lower down. The one that matters is __psynch_mutexwait, which is a thread waiting for a mutex that another thread holds. On Linux, perf samples CPU time by default and does not have this problem.

One line of a profile is a hypothesis, not an answer. The author tested it with experiments, each a few minutes, and the table is the record:

experiment                                      SET        GET      conclusion
one client, pipeline 16                                  201,000   the client limits one connection
50 clients, GET only / SET only                 200,000    253,000   the count of clients changes nothing
parking_lot's user-space mutex                  200,000    199,800   not the kind of mutex
TCP_NODELAY on the socket                       159,700    199,200   not Nagle's algorithm
one socket handle instead of try_clone          199,600    199,400   not the clone
each thread with a private store, no lock       800,000    796,800   the lock

The last line is not a server anyone would ship, because the threads no longer share data, but it answers the question. Without the shared lock the Rust server is as fast as the C one. With it, every command from every thread passes through one mutex. A contended mutex on macOS is a pthread mutex, which passes the lock from one thread to the next through the kernel. The Rust standard library uses a futex on Linux and the pthread mutex on other Unix systems. The same server in a Linux container reached 398,000. Go has the same lock around its store and reaches 800,000. The comment above the Go mutex explains why. A waiting goroutine is parked on a semaphore of the runtime. In what the comment calls "starvation mode", ownership passes "from the unlocking goroutine to the waiter at the front of the queue". The kernel is not involved.

The first fix was to shrink what the lock covers. The command code now returns the reply as a value, and the server writes it after the lock is released:

/// A reply as a value, so that a command can be run under a lock and written after
/// the lock is released. Formatting and the write to the socket cost more than the
/// store operation itself, and neither needs the store.
pub enum Reply {
    Simple(&'static str),
    Error(String),
    Bulk(Vec<u8>),
    Int(i64),
    Nil,
    Array(Vec<Reply>),
}

pub fn write_reply(w: &mut (impl Write + ?Sized), reply: &Reply) -> io::Result<()> {
    match reply {
        Reply::Simple(s) => write_simple(w, s),
        Reply::Error(s) => write_error(w, s),
        Reply::Bulk(b) => write_bulk(w, b),
        Reply::Int(n) => write_int(w, *n),
        Reply::Nil => write_nil(w),
        Reply::Array(items) => {
            write_array_len(w, items.len())?;
            items.iter().try_for_each(|item| write_reply(w, item))
        }
    }
}
rust/ch13/src/resp.rs · region reply

It made no difference. Both servers stayed at 200,000, because the cost was the handoff and not the hold. The change stays, because formatting under a lock is a habit worth losing, and because the second fix needed it. The second fix is the reader-writer lock chapter 9 promised. Reads of the store take a shared lock that any number of threads may hold together. Writes take the exclusive one:

Exercise 13.3RustThe read-only commands

Write execute_read. It takes the arguments by reference and the store by shared reference. It returns Some(reply) for the commands that only read the store, and None for anything else. The table of commands says which is which, with Run::Read and Run::Write, so the function is a lookup and a call.

pub fn execute_read(args: &Command, store: &dyn Store) -> Option<Reply> {
    // your code here
}
Stuck?Show the full listing
/// The reply of a command that only reads the store, or `None` when the command writes
/// and the caller must take the exclusive lock and call `execute`.
pub fn execute_read(args: &Command, store: &dyn Store) -> Option<Reply> {
    match lookup(args) {
        Ok(Entry {
            run: Run::Read(f), ..
        }) => Some(f(args, store, Instant::now())),
        Ok(_) => None,
        Err(reply) => Some(reply),
    }
}
rust/ch13/src/cmd.rs · region execute-read

The serving loop tries the shared lock first and takes the exclusive one only for a write:

// A read takes the shared lock, which many threads hold at once. A
// write takes the exclusive one. Either way the lock is released
// before the reply is formatted or written.
let reply = {
    let db = db.read().unwrap_or_else(|e| e.into_inner());
    db.execute_read(&args)
};
let reply = match reply {
    Some(reply) => reply,
    None => {
        let mut db = db.write().unwrap_or_else(|e| e.into_inner());
        db.execute(&args)?
    }
};
resp::write_reply(&mut writer, &reply)?;
rust/ch13/src/main.rs · region unlock-first
                         no pipeline              pipeline of 16
                         SET       GET            SET       GET
Rust, threads, before    99,950    88,700         159,700   199,400
Rust, threads, after     99,800    99,850         264,900   796,800
Rust, tokio, after       99,850    99,255         199,600   800,000

Reads are fixed. GET with pipelining went from 200,000 to 800,000, the C and Go figure, because fifty readers no longer queue. Writes are not fixed, because fifty writers still do, and 265,000 is where the pthread handoff leaves them. The next step would be to split the store into shards with a lock each, so that writers to different keys never meet. That is what every fast Rust store does, and what kv does not need. The chapter stops here, with a number that is honest about what was fixed and what was not.

The method#

The two halves of the chapter are the same method. A benchmark gave a number that looked wrong. A tool gave one line to suspect, the escape flag in Go and a profile in Rust. Each experiment changed one thing and measured again with the same command. The Go fix was kept because it moved the number. The first Rust fix was kept and measured, found to do nothing, and said so. The second was measured, found to fix half the problem, and said so. The C flags were measured and found to be noise, and said so. Nothing in that sequence needed a particular language. The same steps find the same kind of answer in all three, with their own tools. The "golden rule" of the benchmark documentation applies to all of it. "Only compare apples and apples."

๐Ÿ–ฅProfilers by platform

On macOS, sample needs nothing installed, and Instruments, started from the command line with xctrace record, gives a time profile with a graphical viewer. Both are macOS tools and do not run on Linux. There, perf is the tool for C and Rust, and its output feeds the flame graph scripts directly. Go's pprof works everywhere Go does, because the profiler is inside the runtime and not in the operating system. On Windows, Visual Studio has a CPU profiler for native code, and pprof again covers Go.

Verdict##

TopicCGoRustWhy
A micro-benchmarkOKStrongOKC has a clock and a loop, and Go has go test -bench with allocation counts. Rust needs an example binary or a crate.
Seeing what the compiler didOKStrongOKGo prints its decisions for every line. C and Rust measure each flag, profile and attribute yourself.
A profiler in the boxOKStrongOKC and Rust use the operating system's profiler. Go carries its own, and it works the same everywhere.
Reading the profileStrongOKWeakOne C thread gives an honest list and Go needs the cum column, while the fifty Rust threads drown the list in waiting.
The cost of memoryStrongOKStrongC and Rust allocate what the code says. Go allocates what escapes, and a flag has to tell you which.
A fast serverStrongStrongOKC and Go reach 800,000 with pipelining. Rust needs a read lock to match on reads and stays behind on writes.
Why it was slowOKStrongWeakThe Go runtime hides lock handoff from the kernel. Rust's standard mutex is the platform's, and macOS makes it expensive.
Honest numbersStrongStrongStrongredis-benchmark and the same command shape treat all three alike.
Carry forward
  • Two questions, two tools. A micro-benchmark says how long a function takes. A profile says where a server's time goes. The parser was nine times slower in Go and one percent of the server in all three.
  • Escape analysis. Go puts a value on the heap when it cannot prove its lifetime, and -gcflags=-m names each one. Four of nine allocations were the caller's, and five were the parser's habits.
  • A flag is a hypothesis. -O3, inline, restrict, LTO and #[inline] each moved the number by a few percent or not at all. The benchmark decides, not the keyword.
Next

Copy the Linux binary to another Linux machine and it refuses to start, over a C library that is six years too old. Chapter 14 ships the three servers for real, to machines that are not yours.

Shipping#

Quality and shipping · Chapter 14· 24 min read· 254 lines of code
▶Scene

Build the C server on a current Linux, copy the binary to a Linux that is six years older, and run it:

$ docker run --rm -v "$PWD":/w -w /w gcc:14 sh -c 'ldd --version | head -1; gcc -O2 -pthread -o kv src/*.c'
ldd (Debian GLIBC 2.41-12+deb13u4) 2.41
$ docker run --rm -v "$PWD":/w -w /w debian:buster sh -c 'ldd --version | head -1; ./kv'
ldd (Debian GLIBC 2.28-10+deb10u3) 2.28
./kv: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./kv)

Nothing in kv changed between the two machines. The C library did, and the binary carries a note that names the version it was built against. This chapter is about that note, and about everything else a program has to settle before it leaves the machine it was written on.

kv works on the machine it was written on. This chapter builds it for other machines, Linux on two kinds of processor and Windows, from a Mac. It gives each server the options, the configuration and the version a shipped program needs. It covers linking, cross-compilation, the three operating systems, and what the newest C standard changes.

To ship a program, three questions need answers. What does the binary carry with it, and what does it expect to find on the target machine? Which machines is it built for, and from where? And how does a user get a particular version, know which one they have, and tell it where to listen? Each language has settled answers. They differ enough that the same questions produce three different make dist targets.

ShippingCGoRust
A plain build linksthe system C library, dynamicallynothing, with cgo offthe system C library, dynamically
A static binary-static with musl, through zigthe default on Linuxthe musl target
Cross-compilingzig cc -targetGOOS= and GOARCH=rustup target add, then cargo zigbuild
Optionsgetopt, and a die that never returnsthe flag packagestd::env::args, or the clap crate
A config filenone here, by hand elsewhereencoding/json into a structnone here, the serde crate elsewhere
Loggingfprintf(stderr, ...)log/slog, with fieldseprintln!
A version in the binarya macro you definego:embed of a generated fileCARGO_PKG_VERSION at build time
Platform code#ifdef and a file per systema build constraint per file#[cfg] per item
A Linux server, stripped45 KB2.5 MB521 KB
Windowsno, the C code is Unix-onlyGOOS=windows, unchangedx86_64-pc-windows-gnu, unchanged

Chapter 2 showed what the three hello programs linked, and the picture has not changed. A C program calls the C library for everything and links it dynamically by default. The binary then holds a note that says "find libc.so.6 on this machine", and the scene showed the rest of the note. Each function it uses is tagged with the library version that introduced it. A library too old for the tag refuses to load the program. On Linux, a Go program with CGO_ENABLED=0 makes its system calls itself and links nothing. On macOS it still links libSystem, as chapter 2 showed. A Rust program carries its own standard library and still calls the C library for the system. So it links libc.so.6 as the C one does, with the same note.

The old answer was to build on the oldest system you support. The newer answer is to link statically against musl, a small C library written for this. The binary then carries its own copy and asks for nothing. Go made that the default. The other two can choose it.

๐Ÿ“œWhy static binaries came back

Shared libraries exist to save memory and disk. One copy of the C library serves every program on the machine, which mattered when memory was small and expensive. The cost was that a program depended on the exact library that was there. A program moved to another machine often met a different one. Containers reversed the trade. A container image holds one program, so there is nothing to share, and a binary that needs nothing can run in an empty image. Go chose to link nothing from the start, and the choice fit the container world that arrived a few years later.

C: zig as the cross-compiler#

Cross-compiling C has always been possible and has rarely been pleasant. The compiler must be built for the target, and so must the C library and its headers. Those pieces must be found and told about each other. The Zig project changed that. The project overview says that Zig "is also a C compiler" and carries a C library for 97 targets. As a result, "Zig builds for all supported targets independently of the host", with "no separate cross toolchain". One download, and zig cc -target names the machine:

Exercise 14.1CBuild for two kinds of Linux

Write the dist target. Loop over two targets, x86_64-linux-musl and aarch64-linux-musl. For each, run zig cc over every source file with the usual flags and three more. -target names the machine, -static puts the C library inside the binary, and -s strips the symbols. Name each output after its target under dist/.

ZIG ?= zig
TARGETS := x86_64-linux-musl aarch64-linux-musl

dist: $(SRC)
	mkdir -p dist
	# your loop here
Stuck?Show the full listing
# Cross-compile with zig cc, which carries a C compiler, a linker and the C libraries of
# every target in one download. -static with musl gives a binary that needs no libc file,
# and -s strips the symbols, which is most of the size.
ZIG ?= zig
TARGETS := x86_64-linux-musl aarch64-linux-musl

dist: $(SRC)
	mkdir -p dist
	for t in $(TARGETS); do \
	  $(ZIG) cc -target $$t -static -s -std=c17 -Wall -Wextra -Werror -O2 -o dist/kv-$$t $(SRC) || exit 1; \
	done
	ls -l dist
c/ch14/Makefile · region dist
$ make dist
-rwxr-xr-x  53952 kv-aarch64-linux-musl
-rwxr-xr-x  44824 kv-x86_64-linux-musl
$ file dist/kv-x86_64-linux-musl
ELF 64-bit LSB executable, x86-64, statically linked

Forty-five kilobytes, for a server with a hash table, an event loop and a log. The static musl library costs almost nothing, because only the functions kv calls are linked in. Before the -s flag the same file was 1.5 megabytes, and the difference was the symbol table and the debugging information. The binary was then run in a Linux container that had never seen a compiler:

$ docker run --rm -v "$PWD/dist":/dist redis:8 bash -c \
    'KV_PORT=6380 /dist/kv-x86_64-linux-musl & sleep 1; redis-cli -p 6380 PING'
PONG
$ docker run --rm --platform linux/arm64 -v "$PWD/dist":/dist redis:8 bash -c \
    'KV_PORT=6380 /dist/kv-aarch64-linux-musl & sleep 1; redis-cli -p 6380 PING'
PONG

The second command ran the arm64 binary under emulation on an Intel machine, which is slow and is enough to prove the build. Both answered, and both ran without a file from the system. The scene's binary would have answered on the old Debian too, had it been built this way.

C again: options, and a function that does not come back#

A shipped server is started by a script or a service manager, which passes options. The environment variables kv has used since chapter 3 are not enough. The same standard gives C one function for the job. getopt "parses the command-line arguments", from a string that lists the option letters, and a letter followed by a colon "requires an argument". It returns the letters one at a time, leaves an argument in optarg, and "returns -1" when the options are done:

/* The settings, from the command line first and the environment second. */
typedef struct {
    int port;
    const char *aof;   /* NULL for no persistence */
    const char *fsync; /* NULL for the default */
    const char *hash;  /* "fnv" for the unseeded hash */
} kv_options;

static const char USAGE[] = "usage: kv [-p port] [-a file] [-f always|everysec|no]\n"
                            "  -p port   the port to listen on (KV_PORT), 6380 by default\n"
                            "  -a file   append every write to this file (KV_AOF)\n"
                            "  -f policy how often to fsync the file (KV_FSYNC)\n";

/* getopt walks argv for the options in its string. A letter followed by a colon takes an
 * argument, which getopt leaves in optarg. It returns -1 when the options are done, and
 * optind then says where the rest of the arguments start. */
static kv_options kv_options_parse(int argc, char **argv) {
    const char *port = getenv("KV_PORT");
    kv_options o = {
        .port = port && *port ? atoi(port) : 6380,
        .aof = getenv("KV_AOF"),
        .fsync = getenv("KV_FSYNC"),
        .hash = getenv("KV_HASH"),
    };
    int c;
    while ((c = getopt(argc, argv, "p:a:f:h")) != -1) {
        switch (c) {
        case 'p':
            o.port = atoi(optarg);
            break;
        case 'a':
            o.aof = optarg;
            break;
        case 'f':
            o.fsync = optarg;
            break;
        case 'h':
            fputs(USAGE, stdout);
            exit(0);
        default: /* getopt has already printed which option it did not know */
            die("%s", USAGE);
        }
    }
    if (optind < argc) {
        die("kv: unexpected argument '%s'", argv[optind]);
    }
    if (o.port <= 0 || o.port > 65535) {
        die("kv: not a port: %d", o.port);
    }
    if (o.aof != NULL && *o.aof == '\0') {
        o.aof = NULL;
    }
    return o;
}
c/ch14/src/main.c · region options

The function that reports a bad option is the one to look at. It never returns, and it says so with _Noreturn. That is the C11 specifier that "specifies that the function does not return to its point of invocation". The compiler then knows that the code after a call to die is unreachable, and the reader knows it too. C23 deprecates the keyword for the attribute [[noreturn]], and the section below says why. The other word on the line is older than both. __attribute__ is a GNU extension, outside the standard but shared by gcc and clang. The format attribute says a function takes printf-style "arguments that should be type-checked against a format string". So a wrong %d in a call to die is an error at compile time, as it is for printf itself:

/* Print a message the way printf would, and exit. _Noreturn tells the compiler, and
 * the reader, that nothing follows a call. The attribute is a GNU extension that gcc
 * and clang share, outside the standard: it checks the arguments against the format
 * string at compile time, as the compiler does for printf itself. */
__attribute__((format(printf, 1, 2))) _Noreturn static void die(const char *fmt, ...) {
    va_list ap;
    va_start(ap, fmt);
    vfprintf(stderr, fmt, ap);
    va_end(ap);
    fputc('\n', stderr);
    exit(2);
}
c/ch14/src/main.c · region die
โš ๏ธOne source tree, one platform C

The C version of kv does not build for Windows. It uses poll, kqueue or epoll and the socket calls of POSIX, the Portable Operating System Interface. Chapter 9 explained that Windows wants a different loop. A C program that must run on both keeps two versions of the networking code behind one header. It chooses between them with #ifdef _WIN32, as chapter 9 chose a poller. kv does not, and says so. The other two languages carry that #ifdef inside their standard libraries, which is the whole difference in the next two sections.

C17 against C23#

Every C file in this book is built with -std=c17, because every compiler the book names accepts it. The newer standard, ISO/IEC 9899:2024, is in clang 17 and gcc 14, and the snapshot has a tour of it in one file. Most of the list is C catching up with the other two languages, feature by feature. nullptr is a null with a type of its own, so that it cannot be passed where an integer is wanted. constexpr is a constant the compiler evaluates, for the places a #define was used. typeof and auto take a type from an expression, as the other two languages do. Attributes in double brackets replace __attribute__ for the common cases. The booleans are keywords without a header. #embed puts the bytes of a file into an array at compile time, which Go does with go:embed and Rust with include_bytes!:

/* nullptr has a type of its own, so it cannot be mistaken for an int, as 0 or NULL can */
static const char *first_or(const char *s, const char *fallback) {
    return s != nullptr ? s : fallback;
}

/* [[nodiscard]] makes the compiler warn when a caller ignores the result */
[[nodiscard]] static int parse_digit(char c) {
    return c >= '0' && c <= '9' ? c - '0' : -1;
}

/* #embed pastes the bytes of a file into an array at compile time */
static const unsigned char banner[] = {
#embed "banner.txt"
};

int main(void) {
    /* constexpr: a true compile-time constant, usable where a #define was needed before */
    constexpr int limit = 64;
    /* typeof and auto: the type comes from the expression, as in the other two languages */
    typeof(limit) half = limit / 2;
    auto digit = parse_digit('7');
    /* binary literals and digit separators */
    unsigned mask = 0b1111'0000;
    /* true and false are keywords, and bool needs no header */
    bool ok = digit == 7;
    static_assert(limit > 0); /* no message needed any more */
    printf("%s %d %d %u %d %.*s", first_or(nullptr, "default"), half, digit, mask, ok,
           (int)sizeof banner, banner);
    return 0;
}
c/ch14/extras/c23.c · region c23
$ make c23
default 32 7 240 1 kv, three languages

The standard also removes something. A function defined the way Kernighan and Ritchie wrote them in 1978, with the parameter types on separate lines, is no longer C. That is the first removal of a feature from the language in its history. The server kv stays on C17 for now. The one line that would change is the _Noreturn above.

Go: two variables#

Go cross-compiles by design. The compiler is one program that contains the code generators for every target. The standard library is compiled for the target as part of the build. go help environment describes the two variables. GOOS is "The operating system for which to compile code", and GOARCH is "The architecture, or processor, for which to compile code". A third, CGO_ENABLED, says "Whether the cgo command is supported", and setting it to 0 keeps the C library out:

Exercise 14.2GoBuild for four targets

Write a dist target. Loop over the four targets below. Split each into the two variables, add .exe for Windows, and set CGO_ENABLED=0. Build with -ldflags="-s -w", which drops the symbol table and the debugging information.

TARGETS := linux/amd64 linux/arm64 windows/amd64 darwin/arm64

dist:
	mkdir -p dist
	# your loop here
Stuck?Show the full listing
# Cross-compile by setting two variables. CGO_ENABLED=0 keeps the binary free of the C
# library, so that it runs on any Linux of the right architecture, and in a scratch
# container. -s -w drop the symbol table and debug information, which halves the size.
TARGETS := linux/amd64 linux/arm64 windows/amd64 darwin/arm64

dist:
	mkdir -p dist
	for t in $(TARGETS); do \
	  os=$${t%/*}; arch=$${t#*/}; ext=""; [ $$os = windows ] && ext=.exe; \
	  CGO_ENABLED=0 GOOS=$$os GOARCH=$$arch go build -ldflags="-s -w" -o dist/kv-$$os-$$arch$$ext ./cmd/kv || exit 1; \
	done
	ls -l dist
go/ch14/Makefile · region dist
$ make dist
-rwxr-xr-x  2541858 kv-darwin-arm64
-rwxr-xr-x  2556064 kv-linux-amd64
-rwxr-xr-x  2490528 kv-linux-arm64
-rwxr-xr-x  2677760 kv-windows-amd64.exe

Four binaries for three operating systems, from one command, with nothing installed beyond Go. Each is about 2.5 megabytes, because each carries the runtime, and the Linux ones link nothing. They answered PING in the same containers as the C ones. The Windows one was not run, because this book has no Windows machine. The two Windows binaries in this chapter are the one case that was built and not tested. The file command at least agrees that it is "PE32+ executable (console) x86-64, for MS Windows".

โœ…Turn cgo off unless you need it Go

A Go program that imports a package which uses cgo links the C library and loses the freedom to run anywhere. The usual sources are the net package's name resolver and the os/user package, both of which have pure Go paths that CGO_ENABLED=0 selects. Build with it off, and turn it on only for a package that has no other way, which chapter 15 shows.

Go again: flags, a file, a log and a version#

The Go server reads its settings in three layers, each over the last. The environment comes first, for the test runner. A JSON file comes second, if a flag names one. The flags on the command line come last. The flag package "implements command-line flag parsing". Each call such as flag.StringVar registers a flag with its default and its help text. flag.Parse reads the arguments, and -h prints the help that the package wrote from those calls. The file is read with encoding/json, which "parses the JSON-encoded data and stores the result in the value pointed to by v". The tag on each field of the struct "gives the name of the field" in the file. A field must be exported to be filled:

// config is every setting the server takes. The json tags name the keys in a config
// file, and the same struct is filled from the environment and the flags.
type config struct {
	Port  string `json:"port"`
	AOF   string `json:"aof"`
	Fsync string `json:"fsync"`
}

func env(name, fallback string) string {
	if v := os.Getenv(name); v != "" {
		return v
	}
	return fallback
}

// loadConfig reads the settings in order: the environment, then a JSON file if -config
// names one, then the flags given on the command line, each layer over the last.
func loadConfig() config {
	cfg := config{Port: env("KV_PORT", "6380"), AOF: os.Getenv("KV_AOF"), Fsync: os.Getenv("KV_FSYNC")}
	file := flag.String("config", "", "a JSON file with port, aof and fsync")
	flag.StringVar(&cfg.Port, "port", cfg.Port, "the port to listen on")
	flag.StringVar(&cfg.AOF, "aof", cfg.AOF, "append every write to this file")
	flag.StringVar(&cfg.Fsync, "fsync", cfg.Fsync, "how often to fsync the file: always, everysec or no")
	showVersion := flag.Bool("version", false, "print the version and exit")
	flag.Parse()
	if *showVersion {
		fmt.Print(version)
		os.Exit(0)
	}
	if *file != "" {
		data, err := os.ReadFile(*file)
		if err == nil {
			err = json.Unmarshal(data, &cfg)
		}
		if err != nil {
			slog.Error("config", "file", *file, "err", err)
			os.Exit(2)
		}
		// a flag given on the command line still wins over the file
		flag.Visit(func(f *flag.Flag) {
			switch f.Name {
			case "port":
				cfg.Port = f.Value.String()
			case "aof":
				cfg.AOF = f.Value.String()
			case "fsync":
				cfg.Fsync = f.Value.String()
			}
		})
	}
	return cfg
}
go/ch14/cmd/kv/main.go · region config
$ cat kv.json
{"port": "6395", "fsync": "no"}
$ ./kv -config kv.json -aof kv.aof
2026/10/12 20:33:04 INFO listening port=6395 version=8f75db5-dirty
2026/10/12 20:33:04 INFO open files clients=122848
2026/10/12 20:33:04 INFO replayed commands=0 file=kv.aof
2026/10/12 20:33:05 INFO stopping cause="interrupt signal received"

The lines it printed come from the fourth piece, log/slog. It "provides structured logging, in which log records include a message, a severity level" and key-value pairs. The text handler writes key=value. The JSON handler, "whose output is line-delimited JSON", is one line to switch to, for a collector that wants fields and not sentences. The version in the first line is the fifth piece, and it took two directives:

// go generate runs the command in the directive and writes version.txt, which is checked
// in. go:embed then compiles the file's contents into the binary as a string, so that a
// shipped kv can say which commit it is.
//
//go:generate sh -c "git describe --always --dirty > version.txt"
//go:embed version.txt
var version string
go/ch14/cmd/kv/main.go · region version

go generate "runs commands described by directives within existing files". The manual is firm that it "is never run automatically by go build, go test, and so on. It must be run explicitly". So the generated file is checked in. go:embed then does what the C23 #embed does. The embed package "provides access to files embedded in the running Go program". The files are read "from the package directory or subdirectories at compile time", into a string, a byte slice or a file system.

๐Ÿ“ฐGo 1.22 changes the meaning of a loop, 2024

For thirteen years, a goroutine started inside a Go loop saw the loop's last value, not its own. The variable was "created once and updated by each iteration", so every closure captured the same one. The bug was so common that go vet had a check for it, and it reached production often enough that the Go team changed the language. In Go 1.22, "each iteration of the loop creates new variables, to avoid accidental sharing bugs". A change to what existing code means is the thing a language with a compatibility promise cannot do, and Go found a way. The new meaning applies "only in packages contained in modules that declare go 1.22 or later in their go.mod files". The line go 1.27 in kv's module is a shipping decision, then, and not only a minimum. It says which language the code was written in. The preview ran for four months before the release and produced "zero reports of any problems in production code".

Rust: a target, a linker, and zig again#

Rust is in between. The compiler can produce code for any target, and rustup target add fetches the standard library for one. The rustup book is precise about what that gives you. "rustup target add only installs the Rust standard library for a given target", and "There are typically other tools necessary to cross-compile, particularly a linker". The linker, and the C library it links against, are the C problem again. The common answer is the same tool as in C. cargo-zigbuild describes itself as "Compile Cargo project with zig as linker for easier cross compiling":

Exercise 14.3RustA profile for shipping, and three targets

Add a dist profile to Cargo.toml that inherits from release, turns debugging information off, and strips the symbols. Then write a dist target in a Makefile. It runs cargo zigbuild --profile dist for each of the three targets below and copies each binary under dist/. Each target must first be added with rustup target add.

x86_64-unknown-linux-musl
aarch64-unknown-linux-gnu
x86_64-pc-windows-gnu

The profile comes first.

[profile.dist]
# your settings here
Stuck?Show the full listing
# Keep function names in the release build, so a profiler can show them. The code is
# as fast as without this; only the binary is larger.
[profile.release]
debug = 1

# The profile for shipping: the release settings, without the symbols.
[profile.dist]
inherits = "release"
debug = 0
strip = true
rust/ch14/Cargo.toml · region profile
# Cross-compile with cargo-zigbuild, which lets zig link for the target. Each target is
# a rustup component: rustup target add <triple>. The musl target is static. The dist
# profile in Cargo.toml strips the symbols that the release profile keeps for profiling.
TARGETS := x86_64-unknown-linux-musl aarch64-unknown-linux-gnu x86_64-pc-windows-gnu

dist:
	mkdir -p dist
	for t in $(TARGETS); do \
	  cargo zigbuild --profile dist -q --target $$t || exit 1; \
	  cp target/$$t/dist/kv dist/kv-$$t 2>/dev/null || cp target/$$t/dist/kv.exe dist/kv-$$t.exe; \
	done
	ls -l dist
rust/ch14/Makefile · region dist
$ cargo install --locked cargo-zigbuild
$ rustup target add x86_64-unknown-linux-musl aarch64-unknown-linux-gnu x86_64-pc-windows-gnu
$ make dist
-rwxr-xr-x   444416 kv-aarch64-unknown-linux-gnu
-rwxr-xr-x  1020416 kv-x86_64-pc-windows-gnu.exe
-rwxr-xr-x   521024 kv-x86_64-unknown-linux-musl

The musl binary is static and the gnu one is not. In the arm64 container, ldd lists three shared libraries for it, starting with libc.so.6. That is the note from the first section, and it ran because the container had them. The musl one ran because it needed nothing. The chapter 13 profile setting, debug = 1, is why a separate profile exists. It keeps symbols for the profiler, and a shipped binary does not want them. Stripping took the musl build from 7.3 megabytes to 521 kilobytes.

โš ๏ธA warning from a linker you did not choose Rust

The gnu target prints a warning on every build. It reads warning: linker stderr: ignoring deprecated linker optimization setting "1", under the lint linker_messages. It comes from zig's linker, which is being driven with a flag meant for another, and the binary is fine. Cross-compiling in Rust means rustup, cargo and the linker cooperating. A message from one of them that the others do not understand is the normal cost of that.

Options from the standard library, and the crate everyone uses#

The Rust standard library gives a program its arguments and nothing more. std::env::args "returns the arguments that this program was started with", as an iterator, and "the first element is traditionally the path of the executable". Everything after that is a loop with a match. The server kv writes it itself, because the book has one dependency and will not add a second for this. The function that reports a bad option returns !, the never type, which is what _Noreturn is in C with the compiler's full attention. A call to it can stand where a value is expected, because no value will ever be needed:

/// Every setting the server takes, from the environment first and the command line
/// second. The standard library gives an iterator over the arguments and nothing else.
/// Most Rust programs reach the same result with the clap crate, which writes the
/// parser and the help text from a struct like this one.
struct Options {
    port: u16,
    aof: Option<String>,
    fsync: String,
}

const USAGE: &str = "usage: kv [-p port] [-a file] [-f always|everysec|no]
  -p port   the port to listen on (KV_PORT), 6380 by default
  -a file   append every write to this file (KV_AOF)
  -f policy how often to fsync the file (KV_FSYNC)";

fn env(name: &str) -> Option<String> {
    std::env::var(name).ok().filter(|v| !v.is_empty())
}

/// A usage error is reported and the process ends with status 2, as getopt programs do.
/// The return type `!` is Rust's _Noreturn: this function never comes back.
fn usage_error(msg: &str) -> ! {
    eprintln!("kv: {msg}\n{USAGE}");
    std::process::exit(2)
}

fn options() -> Options {
    let mut o = Options {
        port: 6380,
        aof: env("KV_AOF"),
        fsync: env("KV_FSYNC").unwrap_or_default(),
    };
    let mut port = env("KV_PORT").unwrap_or_else(|| "6380".to_string());
    let mut args = std::env::args().skip(1);
    while let Some(flag) = args.next() {
        let mut value = || {
            args.next()
                .unwrap_or_else(|| usage_error(&format!("{flag} needs a value")))
        };
        match flag.as_str() {
            "-p" => port = value(),
            "-a" => o.aof = Some(value()).filter(|v| !v.is_empty()),
            "-f" => o.fsync = value(),
            "-h" => {
                println!("{USAGE}");
                std::process::exit(0)
            }
            other => usage_error(&format!("unknown option '{other}'")),
        }
    }
    o.port = match port.parse() {
        Ok(port) => port,
        Err(_) => usage_error(&format!("not a port: {port}")),
    };
    o
}
rust/ch14/src/main.rs · region options

Most Rust programs do not write this loop. They add clap, "Command Line Argument Parser for Rust", and put a #[derive(Parser)] on a struct like Options. The parser, the help text and --version are written for them. A config file goes the same way, through the serde crate. It derives the reading of JSON or TOML into a struct, as the Go package does with tags. Both crates are in nearly every shipped Rust program, and the standard library has no such thing. Go put the same two things in its standard library, which is the same decision made in opposite directions. The version needs no crate at all. Cargo sets CARGO_PKG_VERSION from Cargo.toml at build time, and env!("CARGO_PKG_VERSION") puts it in the binary.

Three sizes, three reasons#

                      Linux x86_64, stripped    what is inside
C, musl, static              44,824 bytes       kv, and the parts of musl it calls
Rust, musl, static          521,024 bytes       kv, the Rust standard library, and musl
Go, CGO_ENABLED=0         2,556,064 bytes       kv, and the Go runtime with its collector

The C binary is small because C has no runtime and the linker keeps only what is referenced. The Rust binary is ten times larger, because its standard library is inside and is itself written in Rust, with formatting, collections and the allocator. The Go binary is fifty times larger, because the garbage collector, the scheduler and the network poller are inside. They are there whether a program uses them or not. None of the three sizes is a problem for a server. All three are a fact a reader should know before choosing a language for a device with a megabyte of flash.

The three operating systems, in one place#

The platform boxes in earlier chapters each covered one difference. Set together, they are the list of what a program must know when it leaves its first machine. The last three rows are how each language says "this code is for that system". The book used all three in chapters 9 and 10.

The differenceCGoRust
Readiness and completionpoll and kqueue on macOS, epoll on Linux, completion ports on Windowsthe runtime hides itthe runtime hides it
Socketsone API from BSD, with Winsock variants on Windowshiddenhidden
Threads and lockspthreads, or Windows threadshiddenhidden, but the mutex differs underneath
The C librarylibc.so.6, libSystem, or the Windows C runtimenot usedused through a thin layer
Line endings and paths\n and /, or \r\n and \path/filepath and runtime.GOOSstd::path and cfg!(windows)
Choosing code per system#ifdef __linux__, from macros the compiler predefines//go:build unix on the first line, or _linux.go in the name#[cfg(unix)] or #[cfg(target_os = "linux")] on an item
The unit it choosesa region of a file, or a whole file that compiles to nothinga whole file, alwaysone item: a function, a module, a statement
Where it shows in kvthe pollers, the fsync call, every #includeinternal/syssys.rs, and the mutex in chapter 13

The last row is the honest one. The Go version of kv has one file that knows which operating system it runs on, and so does the Rust version. The Rust one still ran twice as fast on Linux as on macOS under contention, because the standard mutex is a different object on each. Hiding a platform is not the same as making it disappear.

Versions and releases#

A version is a promise about change, and the three ecosystems make the promise the same way, with three numbers. Semantic versioning, which Go modules and cargo both read, gives each number a meaning. Increase the "MAJOR version when you make incompatible API changes". Increase the "MINOR version when you add functionality in a backward compatible manner". Increase the "PATCH version when you make backward compatible bug fixes". kv's Rust crate has carried a version in Cargo.toml since chapter 2. Its Go module carries one as a Git tag, which go install reads, and the binary now carries the commit. Its C version carries one wherever you choose to write it.

How a user gets the program differs more. A Go program with a public module path is installed with go install path@version, which fetches and builds it. Or it is shipped as the one binary this chapter made. A Rust program is installed with cargo install from crates.io or a Git URL, or shipped as a binary. A C program is shipped as source, and becomes a package when a distribution packages it, or as a binary built as above. Redis ships as source and as distribution packages, and that is the C norm. The one binary with nothing to install is the Go norm, and increasingly the Rust one.

Verdict##

TopicCGoRustWhy
Cross-compilingOKStrongOKC needs zig and gets a small static binary. Go needs two variables, and Rust needs a target, a linker and zig.
Static linkingOKStrongOKC and Rust choose musl. Go links nothing by default.
Binary sizeStrongWeakOK45 KB, 2.5 MB and 521 KB for the same server.
Options and configOKStrongOKC has getopt and nothing for a file, and Go has flag, json and slog in the box. Rust has the arguments and a crate for the rest.
WindowsWeakStrongStrongThe C code is Unix-only. Go and Rust build for Windows unchanged.
Knowing the platformWeakStrongOKC sees every difference and Go hides all of them. Rust hides most and leaks one through the mutex.
A releaseWeakStrongStrongC has no mechanism. Go and Rust read a tag or a version field and install from a path.
Carry forward
  • The note in the binary. A dynamic binary names the C library version it needs, and an older machine refuses it. A static musl binary, or a Go binary, carries everything and asks for nothing.
  • Three ways to choose code per system. C uses the preprocessor and macros the compiler predefines. Go uses a constraint on a whole file. Rust uses an attribute on one item.
  • A function that never returns. C says _Noreturn or [[noreturn]] and the compiler trusts it. Rust says ! and the compiler knows it. Go has os.Exit and no way to say so.
Next

The Rust pattern matcher is called from C, and Go calls the C matcher through cgo. Chapter 15 crosses the border between the languages, and the same function costs three times more through cgo than in Go itself.

Talking to C#

Quality and shipping · Chapter 15· 24 min read· 264 lines of code
▶Scene

Take the chapter 6 pattern matcher, which Go and C both have. Time the Go version against the C version called through cgo, on the same input, from the same benchmark:

$ go test -run xxx -bench Glob -benchmem ./internal/cmatch/
BenchmarkGlobGo-12     	49871018	        24.75 ns/op	       0 B/op	       0 allocs/op
BenchmarkGlobCgo-12    	15145387	        76.56 ns/op	       0 B/op	       0 allocs/op

The C function does the same work in the same number of instructions, and the call takes three times as long. The difference is the border, and the chapter is about what stands on it. The Rust call to the same function costs nothing, and the chapter says why.

Every operating system, database driver and codec exposes a C interface, so Go and Rust both have a door to C. This chapter builds a small C library from kv and calls it from Go through cgo and from Rust through an extern block. It passes a struct and a union across, where the layout of memory is the interface. Then it calls Rust from C.

The hash and the pattern matcher from chapter 6 are the library. They are small, they take bytes and a length, and they allocate nothing, which makes them the easiest kind of C to call from anywhere. The same two functions are already written in Go and Rust, so each call can be checked against a native version, and timed against it.

Talking to CCGoRust
Declaring a C functiona headera C comment before import "C"an unsafe extern "C" block
Passing bytesa pointer and a size_tunsafe.Pointer and C.size_tas_ptr() and len()
A struct across the borderits layout, by the rules of the ABIC.struct_name, laid out as C did#[repr(C)], laid out as C did
A union across the bordera tag and the same bytesa byte array, read with unsafea union, read with unsafe
Who compiles the C fileyou, with makethe go tool, from the package directorybuild.rs, with the cc crate
A trivial call costsnothingabout three times a Go callthe same as a C call
Being called from Cthe normal case//export#[unsafe(no_mangle)] pub unsafe extern "C"
What it does to the buildnothingneeds a C compiler and cgo onneeds a C compiler for the C part

The border is a C function#

A compiled function has a name, a place in memory, and a convention for where its arguments and its result go. That convention is the application binary interface (ABI), and the C one is the one every operating system defines and documents. A Go or Rust function has its own. A caller in either language cannot reach a C function without agreeing to the C convention for that one call. The Rust book puts it in one sentence. "The "C" part defines which application binary interface (ABI) the external function uses".

Go cmatch.Glob Rust cmatch::glob the C ABI a name a pointer a length an int back C kvmatch_glob Rust, with a C name kv_rust_glob cgo extern from C
Whatever the languages on each side, the border itself is always shaped like a C function. The C program at the bottom right calls a Rust function and cannot tell.

The border has a second property. Nothing on it is checked. The C compiler trusts the header, Go trusts the comment above import "C", and Rust trusts the extern block. A wrong type in any of them compiles and misbehaves at run time. That is why the library below is written to make the declarations hard to get wrong.

๐Ÿ“œWhy everything speaks C

Unix was written in C, and its system calls were given C wrappers. So the first interface every other language had to reach was a C one. Every operating system since has kept the habit, and so has every library that wants to be used from more than one language. Go got its door early. The Go blog described cgo in March 2011, a year before Go 1, with C as a "pseudo-package" for "C's name space". Rust had extern blocks before 1.0, and the 2024 edition made them say unsafe, because that is what they always were. In October 2022 the border was crossed in the other direction at the largest scale there is. Linus Torvalds accepted the Rust for Linux work into the kernel, released as Linux 6.1, and the first drivers written in Rust followed in 6.8. A kernel is C that calls everything, and it now calls Rust.

C: a library that is easy to call#

A C library is a header and an archive. The header is the only thing the other side reads, so its shape decides how pleasant the border is. The rules are short. Take a pointer and a length, never a C string, because Go and Rust bytes have no terminating zero. Return an int, not a bool, because every language agrees on what a C int is. Allocate nothing, so that no one has to ask which side frees it.

Exercise 15.1CWrite the header

Declare both functions. kvmatch_hash takes a pointer and a byte count and returns a uint64_t. kvmatch_glob takes a pattern and a key, each as a pointer and a length, and returns 1 for a match and 0 otherwise. Include the headers the types need.

#ifndef KVMATCH_H
#define KVMATCH_H
// your declarations here
#endif
Stuck?Show the full listing
/* FNV-1a over n bytes. */
uint64_t kvmatch_hash(const char *s, size_t n);

/* 1 when the key matches the Redis-style pattern (* and ? only), else 0. An int, not a
 * bool, because every language agrees on what a C int is. */
int kvmatch_glob(const char *pat, size_t plen, const char *key, size_t klen);
c/ch15/lib/kvmatch.h · region api

The bodies are the chapter 6 code with the lengths made explicit. The archive is made with ar, the tool every C library is built with:

# The hash and the matcher as a static library, for chapter 15. ar bundles object files
# into an archive that a linker can search; this is what every C library is.
lib: build/libkvmatch.a

build/libkvmatch.a: lib/kvmatch.c lib/kvmatch.h | build
	$(CC) $(CFLAGS) -c -o build/kvmatch.o lib/kvmatch.c
	ar rcs $@ build/kvmatch.o
c/ch15/Makefile · region lib
$ make lib
cc -std=c17 -Wall -Wextra -Werror -O2 -g -c -o build/kvmatch.o lib/kvmatch.c
ar rcs build/libkvmatch.a build/kvmatch.o
$ ls -l build/libkvmatch.a
-rw-r--r--  4264 build/libkvmatch.a

A struct and a union on the border#

Two more functions pass something larger than a number. One fills a struct with both answers at once. The other fills a tagged union, the shape chapter 8 used for a value of several kinds. On the border, the shape of a struct is not a detail of the compiler. It is the interface. Every C compiler lays out a struct by the same rule, the one the ABI writes down. Each field sits at the first offset after the previous one that its alignment allows. The struct's size is rounded up to its own alignment. A uint8_t followed by a uint64_t is therefore sixteen bytes, with seven of them padding:

/* Both answers in one call. The struct is part of the interface, and so is its layout:
 * the one-byte flag is followed by seven bytes of padding, so that the hash sits at an
 * offset of 8, and every caller on every side of the border must agree. */
typedef struct kvmatch_stats {
    uint8_t matched; /* 0 or 1 */
    uint64_t hash;   /* of the key */
} kvmatch_stats;

void kvmatch_glob_stats(const char *pat, size_t plen, const char *key, size_t klen,
                        kvmatch_stats *out);

/* A number parsed from bytes. A union holds the integer or the double in the same eight
 * bytes, and the tag is the only record of which one is there. */
enum { KVMATCH_NONE = 0, KVMATCH_INT = 1, KVMATCH_FLOAT = 2 };

typedef struct kvmatch_number {
    int kind; /* KVMATCH_INT or KVMATCH_FLOAT, or KVMATCH_NONE for not a number */
    union {
        int64_t i;
        double f;
    } as;
} kvmatch_number;

/* Parse up to 63 bytes as a decimal integer or, with a point, a double. Returns the kind. */
int kvmatch_parse_number(const char *s, size_t n, kvmatch_number *out);
c/ch15/lib/kvmatch.h · region layout

The C reference defines the alignment requirement as "the number of bytes between successive addresses at which objects of this type can be allocated". offsetof gives the offset of a field "from the beginning of an object of specified type", padding included. A small program asks the compiler for all of the numbers. The Go and Rust tests below assert the same ones:

int main(void) {
    printf("kvmatch_stats:  size %zu, alignment %zu, matched at %zu, hash at %zu\n",
           sizeof(kvmatch_stats), alignof(kvmatch_stats), offsetof(kvmatch_stats, matched),
           offsetof(kvmatch_stats, hash));
    printf("kvmatch_number: size %zu, alignment %zu, kind at %zu, as at %zu, as is %zu bytes\n",
           sizeof(kvmatch_number), alignof(kvmatch_number), offsetof(kvmatch_number, kind),
           offsetof(kvmatch_number, as), sizeof(((kvmatch_number *)0)->as));
    return 0;
}
c/ch15/lib/layout.c · region layout
$ make layout
kvmatch_stats:  size 16, alignment 8, matched at 0, hash at 8
kvmatch_number: size 16, alignment 8, kind at 0, as at 8, as is 8 bytes

Four kilobytes, and the C side is done. C has no door to build, because C is the door. The rest of the chapter is the other two languages arriving at it.

Go: cgo#

cgo is part of the go tool. A Go file that has import "C" is a cgo file. The comment directly above that import is the preamble. The documentation says it "is used as a header when compiling the C parts of the package". Every C name in the preamble appears in Go under the C prefix, so C.size_t is the C type and C.kvmatch_glob is the function. Any .c file in the package directory is compiled with the system C compiler and linked in, which is why kvmatch.c sits beside cmatch.go.

Exercise 15.2GoWrap both functions

Write Glob(pat, key []byte) bool and Hash(b []byte) uint64 over the C functions. Get the pointer to a slice with unsafe.SliceData and convert it to *C.char through unsafe.Pointer. Decide what to pass for an empty slice, whose pointer may be nil.

/*
#include "kvmatch.h"
*/
import "C"

import "unsafe"

func Glob(pat, key []byte) bool {
	// your code here
}
Stuck?Show the full listing
/*
#include "kvmatch.h"
*/
import "C"

import "unsafe"

// Glob reports whether key matches the pattern, by calling the C function. The slices
// are passed as a pointer and a length, and C must not keep the pointer: after the call
// returns, the garbage collector may free the bytes, or move them if they live on a stack.
func Glob(pat, key []byte) bool {
	return C.kvmatch_glob(cstr(pat), C.size_t(len(pat)), cstr(key), C.size_t(len(key))) != 0
}

// Hash returns the FNV-1a hash of b, from the C code.
func Hash(b []byte) uint64 {
	return uint64(C.kvmatch_hash(cstr(b), C.size_t(len(b))))
}

// cstr gives C a pointer to the first byte of a slice, or a pointer to a static byte for
// an empty slice, because unsafe.SliceData of an empty slice may be nil and C must not
// receive nil for a length of zero.
func cstr(b []byte) *C.char {
	if len(b) == 0 {
		return (*C.char)(unsafe.Pointer(&empty))
	}
	return (*C.char)(unsafe.Pointer(unsafe.SliceData(b)))
}

var empty byte
go/ch15/internal/cmatch/cmatch.go · region cgo

The pointer is the delicate part, because Go has a garbage collector and C does not know about it. The cgo documentation gives the rule. "All Go pointers passed to C must point to pinned Go memory". A pointer passed as a function argument has "the memory they point to implicitly pinned for the duration of the call". After that, "C code may not keep a copy of a Go pointer after the call returns". kvmatch_glob reads and returns, so it obeys without trying. A C library that keeps a buffer after the call needs runtime.Pinner, or a copy made with C.CBytes. The copy is freed with C.free, in the C way.

The struct and the union come across under their C names, with the prefix the documentation describes. "To access a struct, union, or enum type directly, prefix it with struct_, union_, or enum_". Go lays the struct out as the C compiler did, padding and all. The test asks the unsafe package for the numbers. The union is the odd one. Go has no unions, so "C's union types are represented as a Go byte array with the same length". The value is read out of those bytes with a conversion that the tag chooses:

// cStats is the C struct, under its cgo name. Go lays it out exactly as the C compiler
// did, padding included, because the bytes cross the border as they are. The type stays
// private: only a cgo file can see the fields of a C struct.
type cStats = C.struct_kvmatch_stats

// Layout asks the Go compiler what it decided about the C struct, with the questions
// sizeof, offsetof and alignof answer in C. The offset is pointer arithmetic, because
// unsafe.Offsetof does not accept a field of a type that cgo generated.
func Layout() (size, hashOffset, align uintptr) {
	var s cStats
	at := uintptr(unsafe.Pointer(&s.hash)) - uintptr(unsafe.Pointer(&s))
	return unsafe.Sizeof(s), at, unsafe.Alignof(s)
}

// GlobStats fills the C struct through a pointer to a Go value, which the cgo rules
// allow because the struct holds no Go pointers, and takes the two answers out of it.
func GlobStats(pat, key []byte) (matched bool, hash uint64) {
	var out cStats
	C.kvmatch_glob_stats(cstr(pat), C.size_t(len(pat)), cstr(key), C.size_t(len(key)), &out)
	return out.matched != 0, uint64(out.hash)
}

// Number is the parsed value, as Go would rather have it: an interface holding an int64
// or a float64, instead of the C union and its tag.
type Number any

// ParseNumber calls the C parser. cgo represents the union as a byte array of its size,
// so the value is read out of those bytes with an unsafe conversion, chosen by the tag.
func ParseNumber(b []byte) Number {
	var n C.kvmatch_number
	switch C.kvmatch_parse_number(cstr(b), C.size_t(len(b)), &n) {
	case C.KVMATCH_INT:
		return int64(*(*C.int64_t)(unsafe.Pointer(&n.as)))
	case C.KVMATCH_FLOAT:
		return float64(*(*C.double)(unsafe.Pointer(&n.as)))
	}
	return nil
}
go/ch15/internal/cmatch/cmatch.go · region layout
// The layout Go sees must be the layout C wrote: a one-byte flag, seven bytes of
// padding, and the hash at offset 8. Layout asks the compiler through unsafe.
func TestLayoutAgreesWithC(t *testing.T) {
	if size, at, align := Layout(); size != 16 || at != 8 || align != 8 {
		t.Fatalf("size %d, hash at %d, align %d", size, at, align)
	}
	matched, hash := GlobStats([]byte("user:*"), []byte("user:1"))
	if !matched || hash != store.Fnv1a([]byte("user:1")) {
		t.Fatalf("GlobStats = %v, %d", matched, hash)
	}
	if n, ok := ParseNumber([]byte("42")).(int64); !ok || n != 42 {
		t.Fatalf("42 parsed as %v", ParseNumber([]byte("42")))
	}
	if f, ok := ParseNumber([]byte("4.5")).(float64); !ok || f != 4.5 {
		t.Fatalf("4.5 parsed as %v", ParseNumber([]byte("4.5")))
	}
	if ParseNumber([]byte("x")) != nil {
		t.Fatal("x is not a number")
	}
}
go/ch15/internal/cmatch/cmatch_test.go · region layout-test

The first test calls both matchers on the same inputs, and the benchmark from the scene times them:

// The price of the border: the same match, in Go and through cgo.
var sink bool

func BenchmarkGlobGo(b *testing.B) {
	pat, key := []byte("user:*"), []byte("user:1")
	for b.Loop() {
		sink = store.Glob(pat, key)
	}
}

func BenchmarkGlobCgo(b *testing.B) {
	pat, key := []byte("user:*"), []byte("user:1")
	for b.Loop() {
		sink = Glob(pat, key)
	}
}
go/ch15/internal/cmatch/cmatch_test.go · region bench

The C function does the same work as the Go one and takes three times as long. The difference, about fifty nanoseconds, is the crossing. A goroutine runs on a small stack that can grow, and C expects a large fixed one. So the runtime switches stacks on the way in and back on the way out. It also tells the scheduler that this goroutine is away. Fifty nanoseconds is nothing for a call that then decodes a video frame. It is everything for a call made once per byte.

โš ๏ธOne cgo import changes the whole binary Go

A program that imports a cgo package cannot be built with CGO_ENABLED=0. The chapter 14 dist target still works here, because the kv server does not import cmatch. The moment it did, the build would need a C compiler for every target, and a cross-compiler for the foreign ones. The binary would then link the C library. The one-file freedom of chapter 14 is paid for with the rule that the program contains no C.

โœ…Keep the C in one package Go

Put every cgo call in one small package with a Go interface. Pass a pointer and a length, nothing that holds a Go pointer, and keep the calls coarse. One call that processes a buffer costs the same crossing as one call that processes a byte. Check the standard library first. The net resolver and os/user both have a pure Go path, and most of what a program needs from C already has one.

Rust: extern and unsafe#

Rust has no separate tool. An extern "C" block declares functions that live somewhere else, in Rust syntax with C types. The 2024 edition requires the word unsafe in front of it. The edition guide explains why. "It is the responsibility of the author of the extern block to ensure that the signatures are correct". A wrong one "may result in undefined behavior". The compiler checks that the Rust side of each call matches the declaration. It cannot check that the declaration matches the C.

Exercise 15.3RustDeclare, then wrap

Declare both functions in an unsafe extern "C" block. Use *const u8 for the pointers and usize for the lengths. The C int is std::ffi::c_int. Then write safe glob(pat: &[u8], key: &[u8]) -> bool and hash(bytes: &[u8]) -> u64 around them, each with a SAFETY comment that says why the call is sound.

unsafe extern "C" {
    // your declarations here
}

pub fn glob(pat: &[u8], key: &[u8]) -> bool {
    // your code here
}
Stuck?Show the full listing
// What the C header declares, in Rust spelling. The compiler takes these on trust: a
// wrong type here is undefined behaviour, which is why the block is unsafe to call.
unsafe extern "C" {
    fn kvmatch_hash(s: *const u8, n: usize) -> u64;
    fn kvmatch_glob(pat: *const u8, plen: usize, key: *const u8, klen: usize) -> c_int;
    fn kvmatch_glob_stats(
        pat: *const u8,
        plen: usize,
        key: *const u8,
        klen: usize,
        out: *mut Stats,
    );
    fn kvmatch_parse_number(s: *const u8, n: usize, out: *mut CNumber) -> c_int;
}
rust/ch15/src/cmatch.rs · region extern
/// Does the key match the pattern? The C function reads `plen` bytes from `pat` and
/// `klen` from `key` and keeps neither pointer, so a slice is safe to pass for the
/// duration of the call.
pub fn glob(pat: &[u8], key: &[u8]) -> bool {
    // SAFETY: both pointers come from live slices and the lengths are their lengths;
    // the C code does not write through them and does not keep them.
    unsafe { kvmatch_glob(pat.as_ptr(), pat.len(), key.as_ptr(), key.len()) != 0 }
}

/// FNV-1a, from the C code.
pub fn hash(bytes: &[u8]) -> u64 {
    // SAFETY: as above, a live slice and its length.
    unsafe { kvmatch_hash(bytes.as_ptr(), bytes.len()) }
}
rust/ch15/src/cmatch.rs · region safe

The five powers, and the pointer they are for#

The wrapper is the whole idea of unsafe in Rust. The Rust book lists five things that only an unsafe block may do. Dereference a raw pointer. Call an unsafe function or method. Access or modify a mutable static variable. Implement an unsafe trait. Access fields of a union. A call into C is the second, and the block around it is a boundary, not a switch. Inside it, the borrow checker still runs and every other rule still holds. Only the five powers are added, and the SAFETY comment records the argument that they were used correctly. Everything that calls glob is ordinary safe Rust and cannot break the rule the comment states.

The first power is about the type that chapter 10's lifetimes never touched. A raw pointer, *const T or *mut T, is a reference with the checks removed. The book says raw pointers "are allowed to ignore the borrowing rules" and "aren't guaranteed to point to valid memory". They "are allowed to be null" and "don't implement any automatic cleanup". Making one is safe, which is what as_ptr does. Reading through one is the first of the five powers. The only place kv does it is on the way back from C, in the exported function below.

The struct is declared again in Rust, with one attribute. A plain Rust struct has no promised layout. The reference says the order of the fields "does not have to be the same as the order in which the fields are specified". #[repr(C)] gives it C's rule instead, field by field in declaration order, so that the bytes mean the same on both sides. The union is a union, a type Rust has for exactly this case. Reading a field of it is the fifth power, "because Rust can't guarantee the type of the data currently being stored in the union instance". The tag the C code set is the argument that the read is sound:

/// The C struct, declared again in Rust. `repr(C)` makes the compiler lay it out as C
/// would, fields in order with C's padding, instead of reordering them as it may for a
/// plain Rust struct. The test below checks the numbers against the C ones.
#[repr(C)]
#[derive(Clone, Copy, Debug, Default)]
pub struct Stats {
    pub matched: u8,
    pub hash: u64,
}

/// The C union: eight bytes that are an integer or a double. Reading a field of a union
/// is unsafe, because the compiler cannot know which one was last written.
#[repr(C)]
#[derive(Clone, Copy)]
union NumberBits {
    i: i64,
    f: f64,
}

#[repr(C)]
#[derive(Clone, Copy)]
struct CNumber {
    kind: c_int,
    as_: NumberBits,
}

/// What the parser found, as a Rust enum: the tag and the bytes become one value that
/// a match must take apart, and no wrong field can be read.
#[derive(Debug, PartialEq)]
pub enum Number {
    Int(i64),
    Float(f64),
}

pub fn glob_stats(pat: &[u8], key: &[u8]) -> Stats {
    let mut out = Stats::default();
    // SAFETY: live slices and their lengths, and `out` is a valid Stats the C code fills.
    unsafe { kvmatch_glob_stats(pat.as_ptr(), pat.len(), key.as_ptr(), key.len(), &mut out) };
    out
}

pub fn parse_number(bytes: &[u8]) -> Option<Number> {
    let mut n = CNumber {
        kind: 0,
        as_: NumberBits { i: 0 },
    };
    // SAFETY: a live slice and its length, and `n` is a valid CNumber the C code fills.
    // The union field read below is chosen by the tag the C code set, which is the
    // whole contract of a tagged union.
    unsafe {
        match kvmatch_parse_number(bytes.as_ptr(), bytes.len(), &mut n) {
            1 => Some(Number::Int(n.as_.i)),
            2 => Some(Number::Float(n.as_.f)),
            _ => None,
        }
    }
}
rust/ch15/src/cmatch.rs · region layout

The test asks for the same numbers that the C program printed, with size_of, align_of and offset_of!. The three agree, as they must. A program that declares a C struct itself does this for every struct. That is why most Rust projects with a large C library do not declare them at all. They run bindgen, which "automatically generates Rust FFI bindings to C and C++ libraries". FFI is the foreign function interface. It reads the header, writes the repr(C) structs and the extern blocks, and runs from a build script like the one below.

Someone must compile the C file. A build script does, before the crate compiles, using the cc crate, which finds the system compiler and links the result into the crate:

// A build script runs before the crate compiles. This one compiles the C library with
// the cc crate, which finds the system C compiler and links the result into the crate.
fn main() {
    cc::Build::new()
        .file("../../c/ch15/lib/kvmatch.c")
        .include("../../c/ch15/lib")
        .warnings(true)
        .compile("kvmatch");
    println!("cargo:rerun-if-changed=../../c/ch15/lib/kvmatch.c");
    println!("cargo:rerun-if-changed=../../c/ch15/lib/kvmatch.h");
}
rust/ch15/build.rs · region build
# The build script needs a C compiler driver; cc finds the system one.
[build-dependencies]
cc = "1"

# Besides the Rust library, produce a static library a C program can link: libkv.a.
[lib]
crate-type = ["lib", "staticlib"]
rust/ch15/Cargo.toml · region ffi
$ cargo test -q cmatch
running 4 tests
....
test result: ok. 4 passed; 0 failed

A crossing from Rust to C costs what a C to C call costs. Rust threads use the system stack, and the Rust ABI differs from the C one only in what the compiler is free to choose. So the compiler emits a plain call. The benchmark that the Go section needed does not have a result to show here, because there is no difference to measure.

โš ๏ธmiri stops at the border Rust

cargo miri test cannot run a test that calls into C. The tool interprets Rust and knows nothing about the compiled C code, so it reports an unsupported foreign function and stops. It can still check the Rust side of the border. The test for the exported function below passes under miri, because the raw pointers it builds slices from come from Rust. Test the C side with the C tools from chapter 12 and the Rust side with miri. No tool sees both at once.

๐Ÿ“ฐactix-web, 2020

In January 2020 the most used Rust web framework was taken down by its author for a weekend, over the word unsafe. Sergey Davidoff had published an analysis of the framework's internals. He found code that "violates memory safety" by giving out "multiple mutable references to the same data". He wrote that the maintainers "have refused to investigate it". The discussion that followed on Reddit was large and unkind. The maintainer, Nikolay Kim, wrote "I am done with open source" and threatened to "make [Actix] repos private and then delete them". He restored them within days, writing that "a lot of people depend on actix". He passed the project to a new leader, under whom it continues. The Register noted at the time that "there is extensive use of unsafe code in Actix, leading to debate about what should be fixed". The lesson is not about one maintainer. A Rust project is safe exactly to the extent of its unsafe blocks, and the word draws reviewers the way an open port draws scanners. Every such block in kv has a SAFETY comment that states the argument, which is the habit the dispute was about.

The other way: C calls Rust#

Rust can also be the library. A function marked extern "C" uses the C convention, and #[unsafe(no_mangle)] keeps its name as written, so that a C linker can find it. The function below is the chapter 6 matcher behind a C signature:

/// The other direction: a Rust function with a C name and a C signature, for a C program
/// to call. It is the Rust matcher from chapter 6. `#[unsafe(no_mangle)]` keeps the name
/// as written, and `extern "C"` uses the C calling convention.
///
/// # Safety
/// `pat` must point to `plen` readable bytes and `key` to `klen`, as a C caller would
/// promise in a comment. Rust cannot check it, and a wrong length here is the same bug
/// it would be in C.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn kv_rust_glob(
    pat: *const u8,
    plen: usize,
    key: *const u8,
    klen: usize,
) -> c_int {
    if pat.is_null() || key.is_null() {
        return 0;
    }
    // SAFETY: the contract above, which the caller accepted by calling an unsafe function.
    let (pat, key) = unsafe {
        (
            std::slice::from_raw_parts(pat, plen),
            std::slice::from_raw_parts(key, klen),
        )
    };
    c_int::from(crate::store::glob(pat, key))
}
rust/ch15/src/cmatch.rs · region export

It is marked unsafe because calling it is unsafe. The caller promises the lengths, and nothing checks the promise. That is the contract every C function has always had, now written in the signature instead of in a comment. With crate-type = ["lib", "staticlib"] in Cargo.toml, cargo also produces libkv.a, and a C program links it like any archive:

/* The declaration a header would hold. Rust exported this name with #[unsafe(no_mangle)]. */
int kv_rust_glob(const char *pat, size_t plen, const char *key, size_t klen);

int main(void) {
    const char *pat = "user:*";
    const char *key = "user:1";
    int ok = kv_rust_glob(pat, strlen(pat), key, strlen(key));
    printf("kv_rust_glob(\"%s\", \"%s\") = %d\n", pat, key, ok);
    return ok ? 0 : 1;
}
c/ch15/from_rust/main.c · region call
# A C program calling Rust: build the Rust crate as a static library, then link it like
# any other. The flags after -lkv are what the Rust standard library needs from the
# system on each platform.
RUST := ../../rust/ch15
from-rust: build/from_rust
	./build/from_rust

build/from_rust: from_rust/main.c | build
	cargo build --release -q --manifest-path $(RUST)/Cargo.toml
	$(CC) $(CFLAGS) -o $@ from_rust/main.c -L$(RUST)/target/release -lkv $(RUST_LIBS)

# What the Rust standard library needs from the system. On Linux, set RUST_LIBS to the
# list that `cargo rustc --release -- --print native-static-libs` prints.
ifeq ($(shell uname),Darwin)
RUST_LIBS ?= -framework CoreFoundation
endif
c/ch15/Makefile · region from-rust
$ make from-rust
cargo build --release -q --manifest-path ../../rust/ch15/Cargo.toml
cc ... -o build/from_rust from_rust/main.c -L../../rust/ch15/target/release -lkv -framework CoreFoundation
./build/from_rust
kv_rust_glob("user:*", "user:1") = 1
$ ls -l ../../rust/ch15/target/release/libkv.a build/from_rust
-rw-r--r--  19629264 libkv.a
-rwxr-xr-x   1498400 build/from_rust

The archive is nineteen megabytes because it holds the whole crate, tokio included. The C program is 1.5 megabytes, because the linker kept only what one matcher needs. That is still the Rust runtime and its allocator. The -framework CoreFoundation at the end is what the Rust standard library asks of macOS. On Linux the list is different and longer, and cargo rustc --release -- --print native-static-libs prints it. The Rust side knows nothing of this. The C side, as always, must.

Go can be called from C as well, with a //export comment and -buildmode=c-archive. But the archive carries the Go runtime, which starts its scheduler and collector inside the host program. It is done, and it is rarely a comfortable fit. The Rust case has no runtime to start. That is why Rust libraries are the ones you find inside C programs, Python packages and browsers, and now inside the kernel.

What a crossing costs#

The cost of the borderCGoRust
Per callnothingabout 50 nanoseconds and a stack switchnothing
Memory rulesyours, as everpin for the call, copy to keepthe borrow checker stops at the border
A structlaid out by the ABIlaid out by the ABI, fields under their C nameslaid out by the ABI, with repr(C)
A uniona tag you must honourbytes you must converta field read that needs unsafe
What the compiler checksthe header, against the bodythe preamble, against the Go sidethe extern block, against the Rust side
What nothing checksthe header, against the callerthe preamble, against the Cthe extern block, against the C
Tools that crossthe sanitizersthe race detector does notmiri does not
The buildmakea C compiler, per targeta C compiler, per target, for the C part

The row that matters most is the one that nothing checks. In all three languages the declaration is a promise. The chapter 6 tests, run on both sides of the border, are the only thing that keeps it. That is the C way of working, and at the border, every language works the C way.

Verdict##

TopicCGoRustWhy
Calling CStrongOKStrongGo pays a stack switch per call and loses static builds. C and Rust pay nothing per call.
Being called from CStrongWeakStrongA Go archive carries a runtime that must start. A Rust archive is a plain library.
Safety at the borderWeakOKOKC checks nothing anywhere. Go and Rust check their own side and trust the declaration.
A struct on the borderStrongOKStrongC owns the rule. Go follows it under C names, and Rust follows it with repr(C) and can assert the numbers.
A union on the borderOKWeakOKC reads the field the tag names. Go converts the bytes itself, and Rust names the field and marks the read unsafe.
Making the rules visibleWeakOKStrongC has comments and Go has a documented pointer rule. Rust puts unsafe in the signature.
Build cost of C codeStrongWeakOKGo needs cgo on for every target. Rust needs the cc crate and a compiler for the C part only.
Carry forward
  • The border is a C function. A name, a pointer, a length and an int, laid out by the ABI. The declaration on each side is a promise that no compiler checks.
  • Layout is the interface. A struct's offsets, padding and size are fixed by the ABI. C, Go and Rust compute the same sixteen bytes, and repr(C) is how Rust promises to.
  • Five powers. An unsafe block adds raw pointer reads, unsafe calls, mutable statics, unsafe traits and union fields, and nothing else. The SAFETY comment is the argument that they were used correctly.
Next

Fifteen chapters, one server three times over, and a scorecard with more rows than any one language wins. Chapter 16 adds them up, and says which language to pick for what.

The verdict#

The verdict · Chapter 16· 17 min read

Thirteen chapters each ended with a small scorecard. This chapter gathers every row into one table, reads what the pattern says about each language, and turns it into advice. When to choose C, when Go, when Rust, and when the right answer is a language this book did not cover.

One warning before the table. The rows score one project, a key-value server, built by one person who had to learn each language's way as they went. A different project moves rows. A device with sixty-four kilobytes of memory has no row for a garbage collector, because it cannot afford one. A web service has no row for binary size, because it does not care. Read the table as evidence about this kind of program, not as a ranking of languages.

The scorecard#

TopicCGoRustWhy
02  Toolchains and hello, three times
InstallingOKStrongStrongA C compiler is one system command away on macOS and Linux. Go and Rust are one download each, on every platform.
BuildingWeakStrongStrongC has no build tool, only a convention. Go and Rust each have one tool that every project uses.
Seeing what you madeStrongOKOKC shows every step from preprocessor to linker. Go and Rust hide the steps, which is usually what you want.
Binary sizeStrongWeakOK8 KB for C, 2.5 MB for Go because the runtime is inside, 415 KB for Rust.
Running elsewhereWeakStrongOKThe C binary needs the right C library on the target, the Go binary needs nothing, and Rust is in between.
03  Echo server
Opening a portWeakStrongStrongC needs four calls, a struct and an option. Go and Rust need one call that sets the option for you.
Closing it againWeakOKStrongC needs a cleanup label and Go needs a defer you must write, but Rust closes at the end of scope by itself.
The echo loopWeakStrongStrongC needs the short-write loop and the EINTR retry. Both standard libraries contain them.
A lost errorWeakWeakOKC and Go let a return value drop. Rust warns, and with the book's flags refuses.
Seeing what happensStrongOKOKThe C version shows every system call. The other two hide them, which is what you want on most days.
WindowsWeakStrongStrongThe C code needs Winsock changes. Go and Rust build unchanged.
What you must learn firstStrongOKWeakC is the calls themselves. Rust asks for Result, ?, ownership and try_clone before an echo works.
04  Speaking RESP
Reading bytesWeakStrongStrongC has stdio, which blocks, so kv grows its own buffer. Both standard libraries have one.
Holding the argumentsStrongOKOKThe C parser copies nothing. Go and Rust copy each argument once, which is simpler and almost never matters.
Saying "not RESP"OKOKStrongC and Go use a value by convention. Rust uses a kind the compiler knows about and a match the compiler checks.
Saying "the client left"OKWeakStrongGo reuses an error value for a non-error. Rust has Option for exactly this.
Passing an error upWeakWeakStrongC and Go repeat the check at every call. Rust writes one character.
Binary valuesStrongStrongOKC and Go treat all bytes alike. Rust makes you choose between String and Vec<u8>, and the wrong choice rejects real data.
An error with detailsWeakOKStrongC has an enum and a fixed buffer, and Go has a type found by searching a chain. Rust has an enum variant the compiler matches.
05  GET and SET
A key is bytesOKStrongOKC carries a length beside every pointer, Go has []byte and string, and Rust makes you pick Vec<u8> because String rejects binary data.
Copying and owningWeakStrongStrongC copies and comments, Go lets the collector own everything, and Rust moves and checks.
Parsing a numberWeakOKOKC needs a careful parser to avoid undefined behaviour. The other two accept forms Redis rejects.
OverflowWeakOKStrongC must check before it adds and Go checks against a constant, while Rust has checked_add built in.
A missing caseWeakWeakStrongAn enum match in Rust must be complete. In C and Go, a new error kind can be forgotten at a call site.
Reading the codeOKStrongOKThe Go store is the shortest. The Rust signatures say more and take longer to read.
A table of commandsOKStrongOKC needs an X-macro and a function pointer type, and Go has a map of funcs. Rust needs dyn Write and a macro for the rows.
Text against bytesWeakOKStrongC has no text type at all, and Go has rune helpers. Rust makes a str keep its promise and a Vec<u8> make none.
06  Your own hash table
Writing the tableWeakStrongOKC needs an owner comment on every pointer and three cleanup paths, Go is the shortest, and Rust argues about borrows in set.
Freeing memoryWeakStrongStrongC must free in three places that agree. Go and Rust free for you, in different ways.
Freeing on timeStrongWeakStrongC and Rust free on the line that removes an entry. Go frees at the next collection.
Testing itWeakStrongStrongC has no test runner in the box. go test and cargo test are built in.
Shipping unused codeWeakOKStrongC warns only for static functions and Go ignores unused functions, while Rust reports each one.
Needing to write oneStrongWeakWeakIn C the table is the store. In Go and Rust it is an exercise, because the map is built in.
Writing it once for any typeWeakOKStrongC has void pointers, macros and _Generic, each with a hole. Go has type parameters since 2022, and Rust has bounds the compiler checks at the impl.
Seeing where a value livesStrongOKStrongC says it in every line, and Rust says it in the types. Go decides for you and shows its decision with a flag.
07  Modules, packages and dependencies
Splitting codeWeakStrongOKC needs guards, prefixes and a Makefile that tracks headers, while Go needs a folder and Rust a mod line that is easy to forget.
Hiding a nameOKStrongStrongC has static, for functions only. Go and Rust hide by default or by a letter.
Adding a dependencyWeakOKStrongC has three competing ways and no default, while Go and Rust each have one command, and Rust also has one registry.
Having too manyStrongStrongWeakC and Go programs stay small by cost and by a wide standard library. Rust programs grow dozens of crates.
Keeping one styleOKStrongStrongclang-format needs a config everyone agrees on. gofmt and rustfmt need nothing.
Finding mistakesWeakOKStrongclang-tidy must be installed. go vet is built in. clippy is built in and has 800 checks.
Swapping an implementationOKStrongStrongC has a struct of function pointers that nobody checks. Go has implicit interfaces, and Rust has checked traits with two kinds of dispatch.
Hiding a structStrongStrongStrongC has the opaque type, and Go and Rust have unexported fields. All three keep the map out of reach.
08  Data types
A value of several kindsWeakOKStrongC has a tag nothing checks, and Go has a type switch nothing checks. Rust has a match the compiler checks.
Adding a kind laterOKWeakStrongC warns at each switch with no default, and Go falls to default in silence. Rust refuses to build until every match knows.
A list, a hash, a setWeakStrongStrongC writes each, or nests the table. Go and Rust have them in the standard library.
Walking a collectionWeakOKStrongC needs a callback and a context struct, and Go has range over a function. Rust has adaptors that compose.
Scanning by cursorStrongOKOKC and the hand-written table scan buckets. The Go map sorts, and the Rust map counts from the start.
Reading the codeWeakStrongOKThirteen C commands need a helper each, and Go reads like prose. Rust reads well once match and let else are familiar.
09  Many clients
Serving many clientsWeakStrongOKC needs 200 lines of event loop and three pollers, Go one keyword, and Rust an Arc, a Mutex and a clone per thread.
Using every coreWeakStrongStrongThe C loop is one thread by design. Go and Rust spread clients over every core.
A forgotten lockn/aWeakStrongGo compiles it and the detector may catch it. Rust refuses to compile it.
Platform codeOKStrongStrongC needs a file per poller under ifdef. Go and Rust need one guarded function, and their runtimes hide the pollers.
Memory for a clientStrongStrongOKC sizes two buffers and a goroutine is a few kilobytes, while a Rust thread is 2 MiB until tokio fixes that at the cost of a dependency.
Reading the codeWeakStrongOKThe C loop is subtle and the Go version is the chapter 7 code, while the tokio version is a second copy of the parser.
WindowsWeakStrongStrongThe C loop reports readiness and Windows reports completion. Both runtimes hide that.
10  Expiry
A deadlineOKStrongStrongC uses zero to mean none, which the comments must say. Go's zero Time and Rust's Option say it in the type.
The clockOKStrongStrongC picks the clock by name, while Go carries both in one value and Rust gives each a type.
Lazy expiryStrongOKOKC deletes on find. Go and Rust must leave it to the sweep because of the read lock or the borrow.
The sweepStrongStrongWeakC has a cursor and Go a random sample. Rust's retain walks the whole map.
A client that hangs upWeakStrongStrongC dies unless one line ignores SIGPIPE. Both runtimes turn it into an error.
Stopping cleanlyOKStrongOKC needs a handler, a pipe and a rule about what a handler may do. Go needs a context, and Rust the C library or tokio.
A bug in one commandWeakStrongOKC corrupts or crashes, and Go recovers per connection. Rust panics per thread, and the scope reports it.
The parser's argumentsOKOKStrongC borrows by comment, Go copies and caps the capacity, Rust borrows by lifetime and the compiler checks it.
Testing timeWeakStrongStrongGo and Rust tests pass a clock in. The C store has no test runner yet.
11  Persistence
Writing the fileOKStrongStrongC encodes into a buffer and calls write. Go and Rust reuse the protocol writers and flush.
ReplayStrongStrongStrongAll three reuse the parser from chapter 4 unchanged, which is the point of a log in RESP.
A crash of the processStrongOKOKC writes straight through. Go and Rust needed a flush the book's test had to find.
The clock on diskOKStrongOKC converts between two clocks itself, while Go's time.Time holds both and Rust converts with a helper.
Log and store togetherStrongOKStrongOne C thread needs no lock, Go takes two locks, and Rust takes one around both.
The fsync threadWeakStrongStrongC needs a mutex, a condition variable, an atomic and a sanitizer. Go needs a channel, and Rust a channel the compiler checks.
Reaching the disk on macOSOKStrongStrongC must know about F_FULLFSYNC itself. Both standard libraries know.
12  Testing and fuzzing
A test runnerWeakStrongStrongC has none, so you write one. Go and Rust have one in the box, with no setup.
Where a test may liveWeakOKStrongC has the file you write, and Go the package and an Example. Rust has the module, the docs and a directory for outsiders.
FuzzingOKStrongOKlibFuzzer is one flag on Linux and a container on a Mac, Go has it built in, and Rust needs nightly and a second crate.
Memory errorsOKStrongStrongC finds them at run time with ASan or Valgrind. Go and Rust cannot write them in safe code.
Data racesOKOKStrongC and Go find the races a test provokes, with a sanitizer or the detector. Rust refuses them at compile time.
Undefined behaviourWeakStrongOKC has it everywhere and UBSan finds some. Go has almost none, a data race being the exception, and Rust confines it to unsafe, where Miri checks it.
Speed of the toolsStrongStrongWeakASan costs two times and the race detector two to twenty, while Miri costs nine thousand.
13  Benchmarking and profiling
A micro-benchmarkOKStrongOKC has a clock and a loop, and Go has go test -bench with allocation counts. Rust needs an example binary or a crate.
Seeing what the compiler didOKStrongOKGo prints its decisions for every line. C and Rust measure each flag, profile and attribute yourself.
A profiler in the boxOKStrongOKC and Rust use the operating system's profiler. Go carries its own, and it works the same everywhere.
Reading the profileStrongOKWeakOne C thread gives an honest list and Go needs the cum column, while the fifty Rust threads drown the list in waiting.
The cost of memoryStrongOKStrongC and Rust allocate what the code says. Go allocates what escapes, and a flag has to tell you which.
A fast serverStrongStrongOKC and Go reach 800,000 with pipelining. Rust needs a read lock to match on reads and stays behind on writes.
Why it was slowOKStrongWeakThe Go runtime hides lock handoff from the kernel. Rust's standard mutex is the platform's, and macOS makes it expensive.
Honest numbersStrongStrongStrongredis-benchmark and the same command shape treat all three alike.
14  Shipping
Cross-compilingOKStrongOKC needs zig and gets a small static binary. Go needs two variables, and Rust needs a target, a linker and zig.
Static linkingOKStrongOKC and Rust choose musl. Go links nothing by default.
Binary sizeStrongWeakOK45 KB, 2.5 MB and 521 KB for the same server.
Options and configOKStrongOKC has getopt and nothing for a file, and Go has flag, json and slog in the box. Rust has the arguments and a crate for the rest.
WindowsWeakStrongStrongThe C code is Unix-only. Go and Rust build for Windows unchanged.
Knowing the platformWeakStrongOKC sees every difference and Go hides all of them. Rust hides most and leaks one through the mutex.
A releaseWeakStrongStrongC has no mechanism. Go and Rust read a tag or a version field and install from a path.
15  Talking to C
Calling CStrongOKStrongGo pays a stack switch per call and loses static builds. C and Rust pay nothing per call.
Being called from CStrongWeakStrongA Go archive carries a runtime that must start. A Rust archive is a plain library.
Safety at the borderWeakOKOKC checks nothing anywhere. Go and Rust check their own side and trust the declaration.
A struct on the borderStrongOKStrongC owns the rule. Go follows it under C names, and Rust follows it with repr(C) and can assert the numbers.
A union on the borderOKWeakOKC reads the field the tag names. Go converts the bytes itself, and Rust names the field and marks the read unsafe.
Making the rules visibleWeakOKStrongC has comments and Go has a documented pointer rule. Rust puts unsafe in the signature.
Build cost of C codeStrongWeakOKGo needs cgo on for every target. Rust needs the cc crate and a compiler for the C part only.

One hundred and one rows. C was strong in 28, ok in 29 and weak in 43. Go was strong in 58, ok in 30 and weak in 13. Rust was strong in 57, ok in 37 and weak in 7. One row for C had no mark, because a forgotten lock cannot happen on one thread. The thorough edition added rows for abstraction, generics, sum types and iteration in chapters 7 and 8. It added rows for signals, background work, platform code and the structs on the border in the chapters after. Those sums are less interesting than where the marks fall, because each language's weak rows belong to one or two families.

C: the price of seeing everything#

Almost every weak row for C is the same row. Memory that must be freed by the right code at the right time. An error that must be checked by the next line or is lost. A platform difference that the program must know about, because nothing stands between it and the operating system. A value of several kinds that is a tag and a union the compiler never checks against each other. A lock, a condition variable and an atomic that a comment, not a type, says how to use. Those are not five weaknesses. They are one property, that C hides nothing, seen from five sides.

The same property fills C's strong rows. The binary was forty-five kilobytes. The profile read like the source, because the source is what ran. The sanitizers found the use-after-free in chapter 12 in a second, because they watch the real memory operations. And at the border in chapter 15, C was not a language calling another. It was the border. Every other language's door is shaped like a C function, and that will not change in your career.

Choose C when the program must be small, or must run where no runtime fits. Choose it when the program is itself the thing other languages will call, or when the platform is the subject. An operating system, a C library, a device driver, a codec, an interpreter that other languages embed. Choose it knowing that the compiler checks types and nothing else. The chapter 12 tools are not optional in C. They are the part of the compiler that was left out.

Go: the server language#

Go's strong rows cluster around the task this book set. A server that talks to many clients, in one word, go, with a runtime that schedules them across every core. A test runner, a fuzzer, a race detector and a profiler in the box, with no decision to make about any of them. A channel that is both the message and the lock, which made the fsync thread of chapter 11 the shortest of its three versions. One command to build for any machine, and a binary that needs nothing. The chapter 9 server was the shortest of the four. It was also the one that used every core without a line of code written for the purpose.

Go's weak rows are the cost of the runtime and the cost of simplicity. The binary is 2.5 megabytes before the program starts. Calling C costs a stack switch and the one-file freedom. The language has no way to say that a function can fail except a second return value. Chapter 3 showed what happens when the caller ignores it. The forgotten lock in chapter 9 compiled and ran, and only the race detector, run on purpose, said anything. A value of several kinds is an interface and a type switch, and the compiler does not say when a case is missing. The parser allocated nine times a request until a flag said where. What lives on the heap is the compiler's decision and not the programmer's. Go trusts the programmer to check every error and every lock. It gives better tools than C for finding the ones they missed, and no more protection at compile time.

Choose Go when the program is a network service, a command-line tool, or anything a team must read, build and deploy with the least effort. Choose it when the people who will maintain it are many, or new, or both. Go has one way to do most things, and the formatter enforces it. Do not choose it for a library that C or Python will embed, or for a device with a megabyte of flash. Do not choose it when correctness depends on the compiler catching a race.

Rust: the compiler as the reviewer#

Rust is strong in the rows that were weak for C. The hash table freed itself. The lost error could not be lost, because a Result that is dropped is a warning and a ? is a line. The forgotten lock did not compile. The data race did not compile. The value of several kinds was an enum, and a match that forgot a kind did not compile either. The parser that borrowed its arguments from the buffer could not outlive it, because a lifetime said so. At the border in chapter 15, the unsafe block was a labelled boundary with a written argument, not a mood. Rust is what C would be if the compiler checked the comments.

Rust's weak rows are the cost of that checking, and one surprise. The compiler is slower. A clean release build of chapter 15 took 19 seconds in Rust, 8 in Go and 1 in C. The chapter 12 tools took the longest to run. The chapter 13 profile was the hardest to read, because the sampler counted fifty sleeping threads and buried the one line that mattered. The mutex in chapter 13 was a different object on macOS and on Linux, and the standard library did not say so. And in chapter 6, writing a hash table was harder than in Go, and harder than the rest of Rust had been. A structure that owns its entries and lends references to them is exactly what the borrow checker was built to question. That one is a real cost. Rust is at its most awkward when writing the data structures it is best at using.

Choose Rust when the program must be fast and must be correct. Choose it when the people who will run it are not the people who wrote it. A database engine, a proxy, a parser for untrusted input, a command-line tool that must never crash. A library that will be embedded in C, in Python or in a browser. Choose it knowing that the first month is spent arguing with the compiler, and that the argument is the point. Do not choose it for a program that will be rewritten next quarter, or for a team that has no time to learn it. A half-learned Rust is slower to write than Go, and a program full of unwrap() crashes when Go would have returned an error.

When the answer is none of them#

This book chose a server so that all three languages would be in their element. Many programs are not servers. A script that runs once, a report, a prototype that exists to be thrown away. A program whose whole cost is the time of the person writing it. For those, a language with a garbage collector, a large library and no build step is the right choice. That it runs many times slower than C does not appear in any row that matters. Python, JavaScript, Ruby and their relatives exist for this, and a systems programmer who refuses them is choosing the wrong tool out of pride.

The honest version of the question is not "which language is best". It is "what does this program need to be true, and which compiler will check it". C checks the types. Go checks the types and watches the running program. Rust checks the types, the lifetimes and the sharing. Each check has a price in time and in freedom, and the right language is the one whose price buys a guarantee the program needs.

What this book did not do#

A list, so that the scorecard is read with it. No chapter ran a Windows binary, so every Windows row rests on documentation and a file command. The Rust chapters used tokio and did not compare it with another runtime. The C server ran kqueue and epoll but never completion ports, because the book has no Windows machine. ThreadSanitizer ran the C tests on Linux only, because on macOS it defers a signal handler for ever. The garbage collector was measured by its count of collections and its memory, not by its pauses, because kv never allocated enough to show one. No chapter sharded the Rust store, so its write number in chapter 13 stayed where the mutex left it. And the three appendices teach the long tail of each language in two sentences each, which is a reminder and not a lesson. Each of those would change a row or add one.

Where to go from here#

kv is small enough to understand and large enough to extend. The next features, in rough order of difficulty, are more data types, replication to a second server, and a cluster that shares keys by hash. Each is a chapter this book did not write. Each is a chance to learn whether the scorecard above survives contact with a harder problem. Pick the language whose weak rows worried you least and build the feature. Then build it again in the language whose weak rows worried you most. The second build is where the learning is.

Appendix A: Rosetta cheat card#

Appendices · Appendix A· 9 min read· 297 lines of code

The constructs kv used most, side by side. Each block shows the same small thing in C, Go and Rust, in the form the chapters settled on. Use the Follow buttons at the top of the page to put your language first.

A value and a type#

C
int n = 42;
const char *name = "kv";
size_t len = strlen(name);
uint64_t big = 1ULL << 40;
bool ok = n > 0;           /* stdbool.h */
Go
n := 42
name := "kv"
length := len(name)
var big uint64 = 1 << 40
ok := n > 0
Rust
let n = 42;
let name = "kv";
let len = name.len();
let big: u64 = 1 << 40;
let ok = n > 0;
let mut count = 0; // mutable

A function that can fail#

C
/* 0 on success, -1 and errno on failure */
int kv_open(const char *path, int *fd_out) {
    int fd = open(path, O_RDONLY);
    if (fd < 0) return -1;
    *fd_out = fd;
    return 0;
}
int fd;
if (kv_open("kv.aof", &fd) < 0) {
    perror("open");
}
Go
func open(path string) (*os.File, error) {
	f, err := os.Open(path)
	if err != nil {
		return nil, fmt.Errorf("open: %w", err)
	}
	return f, nil
}
f, err := open("kv.aof")
if err != nil {
	log.Fatal(err)
}
Rust
fn open(path: &str) -> io::Result<File> {
    let f = File::open(path)?;
    Ok(f)
}
let f = match open("kv.aof") {
    Ok(f) => f,
    Err(e) => {
        eprintln!("open: {e}");
        std::process::exit(1);
    }
};

A struct with a function on it#

C
typedef struct {
    char *data;
    size_t len;
} kv_buf;

void kv_buf_clear(kv_buf *b) {
    b->len = 0;
}
kv_buf b = {0};
kv_buf_clear(&b);
Go
type Buf struct {
	data []byte
}

func (b *Buf) Clear() {
	b.data = b.data[:0]
}

var b Buf
b.Clear()
Rust
struct Buf {
    data: Vec<u8>,
}

impl Buf {
    fn clear(&mut self) {
        self.data.clear();
    }
}
let mut b = Buf { data: Vec::new() };
b.clear();

Bytes and strings#

C
/* a string is bytes with a zero at the end */
char s[] = "PING";
size_t n = strlen(s);        /* 4 */
if (memcmp(s, "PING", 4) == 0) { }
char copy[16];
snprintf(copy, sizeof copy, "%s", s);
Go
s := "PING"            // immutable
b := []byte(s)         // a copy, mutable
if string(b) == "PING" { }
if bytes.Equal(b, []byte("PING")) { }
n := len(b)            // 4, in bytes
Rust
let s = "PING";                 // &str, UTF-8
let b: &[u8] = s.as_bytes();    // no copy
let owned: Vec<u8> = b.to_vec(); // a copy
if b == b"PING" { }
let n = b.len();                // 4

A growable array#

C
size_t cap = 4, len = 0;
int *xs = malloc(cap * sizeof *xs);
if (len == cap) {
    int *p = realloc(xs, 2 * cap * sizeof *xs);
    if (p == NULL) { free(xs); return -1; }
    xs = p;
    cap *= 2;
}
xs[len++] = 7;
free(xs);
Go
xs := make([]int, 0, 4)
xs = append(xs, 7)
for i, x := range xs {
	fmt.Println(i, x)
}
// nothing to free
Rust
let mut xs = Vec::with_capacity(4);
xs.push(7);
for (i, x) in xs.iter().enumerate() {
    println!("{i} {x}");
}
// freed when xs goes out of scope

A map#

C
/* no map in the standard library:
   write one (chapter 6) or use a library */
kv_store_set(&s, key, klen, val, vlen);
kv_entry *e = kv_store_find(&s, key, klen, now);
if (e == NULL) { /* missing */ }
kv_store_del(&s, key, klen);
Go
m := map[string][]byte{}
m["k"] = []byte("v")
if v, ok := m["k"]; ok { use(v) }
delete(m, "k")
Rust
let mut m: HashMap<Vec<u8>, Vec<u8>> = HashMap::new();
m.insert(b"k".to_vec(), b"v".to_vec());
match m.get(b"k".as_slice()) {
    Some(v) => { /* found */ }
    None => { /* missing */ }
}
m.remove(b"k".as_slice());

Heap memory, and who frees it#

C
kv_entry *e = malloc(sizeof *e);
if (e == NULL) return -1;
e->key = malloc(klen);
/* ... */
free(e->key);     /* in the same order */
free(e);          /* you decided, you free */
Go
e := &entry{key: make([]byte, klen)}
// the collector frees e when nothing
// points to it any more
Rust
let e = Box::new(Entry { key: vec![0; klen] });
// freed at the end of the scope that owns it
let moved = e;   // ownership moves, e is gone
drop(moved);     // or free it early

A thread and a shared counter#

C
static int count;
static pthread_mutex_t mu = PTHREAD_MUTEX_INITIALIZER;
void *work(void *arg) {
    pthread_mutex_lock(&mu);
    count++;
    pthread_mutex_unlock(&mu);
    return NULL;
}
pthread_t t;
pthread_create(&t, NULL, work, NULL);
pthread_join(t, NULL);
Go
var mu sync.Mutex
var count int
var wg sync.WaitGroup
wg.Go(func() {
	mu.Lock()
	count++
	mu.Unlock()
})
wg.Wait()
Rust
let count = Arc::new(Mutex::new(0));
let c = Arc::clone(&count);
let t = thread::spawn(move || {
    *c.lock().unwrap() += 1;
    // unlocked here, when the guard drops
});
t.join().unwrap();

A value that is one of several kinds#

C
typedef struct {
    enum { KV_STR, KV_LIST } kind;
    union { kv_bytes str; kv_list list; } as;
} kv_value;
switch (v->kind) {          /* the compiler does not check kind against as */
case KV_STR:  use(v->as.str); break;
case KV_LIST: use(v->as.list); break;
}
Go
var v any = []byte("hi")     // any value, with its type inside
switch val := v.(type) {
case []byte:
	use(val)
case *List:
	use(val)
default:                     // no error if a case is missing
}
Rust
enum Value { Str(Vec<u8>), List(VecDeque<Vec<u8>>) }
match v {                   // every variant, or it does not compile
    Value::Str(s) => use_str(s),
    Value::List(l) => use_list(l),
}

Calling a function chosen at run time#

C
typedef bool (*kv_handler)(const kv_cmd *, kv_store *, kv_buf *);
static const struct { const char *name; kv_handler fn; } commands[] = {
    {"GET", kv_cmd_get}, {"SET", kv_cmd_set},
};
for (size_t i = 0; i < 2; i++)
    if (strcmp(commands[i].name, name) == 0)
        return commands[i].fn(&cmd, store, &out);
Go
type Storer interface {       // satisfied by any type with these methods
	Get(key []byte) ([]byte, bool)
	Set(key, val []byte)
}
var handlers = map[string]func(args [][]byte, s Storer) error{
	"GET": get, "SET": set,
}
return handlers[name](args, store)
Rust
trait Store {                 // implemented by name, not by shape
    fn get(&self, key: &[u8]) -> Option<&[u8]>;
    fn set(&mut self, key: Vec<u8>, val: Vec<u8>);
}
type Handler = fn(&Command, &mut dyn Store) -> Reply;  // dyn: a vtable
const COMMANDS: &[(&[u8], Handler)] = &[(b"GET", get), (b"SET", set)];

Walking every key#

C
static void print_key(const char *k, size_t n, void *ctx) {
    fprintf(ctx, "%.*s\n", (int)n, k);
}
kv_store_each(store, print_key, stdout);   /* a callback and a void * */
Go
for k := range s.Keys() {    // Keys returns an iter.Seq[[]byte]
	fmt.Println(string(k))
}
Rust
for k in store.keys() {      // keys returns impl Iterator<Item = &[u8]>
    println!("{}", String::from_utf8_lossy(k));
}
let n = store.keys().filter(|k| k.starts_with(b"user:")).count();  // lazy

Handing work to another thread#

C
pthread_mutex_lock(&s->mu);           /* a flag under a mutex, and a signal */
s->wanted = true;
pthread_cond_signal(&s->cv);
pthread_mutex_unlock(&s->mu);
/* the thread: while (!s->wanted) pthread_cond_wait(&s->cv, &s->mu); */
Go
jobs := make(chan int, 10)
go func() {
	for j := range jobs {      // ends when jobs is closed and empty
		work(j)
	}
}()
jobs <- 1
close(jobs)
Rust
let (tx, rx) = mpsc::channel();
thread::scope(|s| {
    s.spawn(move || for j in rx { work(j) });  // ends when tx is dropped
    tx.send(1).unwrap();
    drop(tx);
});

Stopping on Ctrl-C#

C
static volatile sig_atomic_t stopped;
static void on_signal(int sig) { stopped = sig; write(pipe_w, "s", 1); }
struct sigaction sa = {.sa_handler = on_signal, .sa_flags = SA_RESTART};
sigaction(SIGINT, &sa, NULL);
/* the loop: poll on pipe_r as well, and stop when it is readable */
Go
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()
select {
case <-ctx.Done():       // the signal arrived
case <-work:
}
Rust
tokio::select! {
    _ = tokio::signal::ctrl_c() => break,   // the signal arrived
    accepted = listener.accept() => serve(accepted?),
}
// with std: block the signals, and sigwait on one thread

Reading a whole file#

C
FILE *f = fopen("kv.aof", "rb");
if (f == NULL) return -1;
char buf[4096];
size_t n;
while ((n = fread(buf, 1, sizeof buf, f)) > 0) {
    /* use buf[0..n] */
}
fclose(f);
Go
data, err := os.ReadFile("kv.aof")
if err != nil {
	return err
}
// data is the whole file
Rust
let data = std::fs::read("kv.aof")?;
// data: Vec<u8>, the whole file

Printing#

C
printf("%s=%zu (%d)\n", key, len, n);
fprintf(stderr, "kv: %s\n", strerror(errno));
Go
fmt.Printf("%s=%d (%d)\n", key, len(key), n)
fmt.Fprintf(os.Stderr, "kv: %v\n", err)
log.Printf("listening on %s", addr)
Rust
println!("{key}={} ({n})", key.len());
eprintln!("kv: {err}");
println!("{:?}", bytes);   // debug form

Build, test, check#

C, with make
make                        # build
make test                   # your runner
make SANITIZE=-fsanitize=address,undefined
clang-format -i src/*.c
make dist                   # zig cc, per target
Go
go build ./cmd/kv
go test ./...
go test -race ./...
go vet ./... && gofmt -l .
GOOS=linux GOARCH=arm64 go build ./cmd/kv
Rust, with cargo
cargo build --release
cargo test
cargo clippy -- -D warnings
cargo fmt
cargo zigbuild --target x86_64-unknown-linux-musl

Appendix B: Glossary#

Appendices · Appendix B· 5 min read

The terms the book uses, each with the section that explains it. The first use of a term in each chapter links here.

TermMeaningSee
ABIApplication binary interface. The convention for how a compiled function receives its arguments and returns its result.The border is a C function
AddressSanitizerA compiler mode that watches every memory access and reports a use after free or an access outside a block.C: a runner you write, and a sanitizer that watches
AOFAppend-only file. kv's log of every write, replayed at start to rebuild the store.C: a file descriptor, a write, and a thread for the fsync
asyncRust code that pauses at an await point instead of blocking a thread, run by a runtime such as tokio.Rust once more: tasks on tokio
atomicAn operation on a shared variable that no other thread can see half done, with a memory order that says what else it promises.C again: the second thread
borrow checkerThe part of the Rust compiler that tracks who may read or change a value and for how long.Rust: a value has one owner
build constraintThe comment on the first line of a Go file, //go:build, that says which systems or settings the file is for.Go's platform files: a line at the top
bulk stringThe RESP form of a byte string. A dollar sign, a length, a line break, the bytes, a line break.C: a buffer, a parser and three answers
cfgThe Rust attribute, #[cfg(...)], that keeps an item out of the build on every system or setting it does not name.Rust's platform code: an attribute on the item
cgoThe part of the go tool that lets Go code call C and be called from it.Go: cgo
chainingA hash table design where each bucket holds a linked list of the entries that hash to it.C: an array of chains, and every pointer accounted for
channelA queue that goroutines or threads send values through, which in Go and Rust is also the lock.Go again: fifty workers and three channels
clippyThe Rust linter. It reports patterns that compile but are slower, longer or riskier than they need to be.One style, enforced by a tool
closureA function literal that captures variables from the scope around it.Go: two kinds of string, and a copy between them
condition variableA place a thread sleeps until another thread signals it, always inside a loop that checks the condition again.C again: the second thread
contextThe Go value that carries a cancellation, a deadline and request values down a chain of calls.context in one paragraph
crateA Rust package. A library crate, a binary crate, or both, described by Cargo.toml.Rust: a module is a file you must announce
cross-compilingBuilding on one machine a binary that runs on another kind of machine.C: zig as the cross-compiler
cursorA number the client sends back so that the server can continue a walk where it stopped.C: a tag and a union
data raceTwo threads touching the same memory at the same time, one of them writing, with no lock between them.Go: one word, and one lock
deadlineThe moment a key expires, stored with the key as a reading of the monotonic clock.C: a deadline on the entry, and a sweep with a cursor
dynamic linkingLinking where the binary holds only the name of a library and finds it on the machine at start.What a binary carries
epollThe Linux call that waits for many sockets at once. kqueue on macOS and completion ports on Windows do the same job.C: one thread, and a poller to say who is ready
escape analysisThe Go compiler's decision, for each value, whether it may live on the stack or must go on the heap.Escape analysis
event loopOne thread that asks which sockets are ready and serves each in turn.C: one thread, and a poller to say who is ready
extern blockThe Rust declaration of functions that live in another language, marked unsafe since the 2024 edition.Rust: extern and unsafe
file descriptorThe small integer a Unix system gives a program for an open file or socket.C: four calls, and a way to clean up
flame graphA picture of a profile in which each box is a function and its width is the share of samples through it.Taking a number
fsyncThe system call that makes the operating system write a file's bytes to the disk before it returns.What a crash takes
fuzzingTesting with generated inputs that a tool mutates toward the paths it has not yet run.Fuzzing, three ways, one engine
garbage collectorThe part of the Go runtime that finds memory nothing points to and frees it.Go: the same table, with nothing to free
genericA function or type written once for many types, with type parameters that the compiler fills in.Go: the same table, with nothing to free
goroutineA function the Go runtime runs on its own small stack, a few kilobytes, scheduled over the real threads.Go: one word, and one lock
goroutine leakA goroutine that waits for a message or a reply that never comes, and lives until the process ends.Go again: fifty workers and three channels
hash tableAn array of buckets, with a hash of the key choosing the bucket. The store under GET and SET.C: an array of chains, and every pointer accounted for
headerA C file of declarations, included by every file that uses them.C: headers, static, and a Makefile that knows about both
header guardThe #ifndef at the top of a header that stops it being read twice.C: headers, static, and a Makefile that knows about both
interfaceA Go type that names methods, satisfied by any type that has them, without being told.Go: a folder is a package, and a capital letter is an export
iteratorA value with a next method that yields one item at a time, which adaptors wrap and nothing runs until asked.Rust: an enum with data in each variant
lazy expiryChecking a key's deadline when the key is read, and deleting it then.C: a deadline on the entry, and a sweep with a cursor
libcThe C standard library, which also wraps the system calls. glibc and musl are two of them.What a binary carries
libFuzzerThe in-process, coverage-guided fuzzing engine in clang, which cargo-fuzz drives for Rust.C: a runner you write, and a sanitizer that watches
lifetimeThe name, written 'a, that says how long a reference in a type is valid, which the compiler checks and never extends.Lifetimes, in the one place kv needs them
linkerThe tool that joins compiled files and libraries into one binary.What a binary is
miriAn interpreter for Rust that checks unsafe code for undefined behaviour while it runs.Rust: the compiler already ran, and the tools check the rest
moduleIn Go, a set of packages with a version. In Rust, a file or block of code with its own names.Go: a folder is a package, and a capital letter is an export
monotonic clockA clock that only goes forward, unaffected by the user changing the time.C: a deadline on the entry, and a sweep with a cursor
muslA small C library for Linux that is designed to be linked statically.What a binary carries
mutexA lock. One thread holds it at a time, and the others wait.Go: one word, and one lock
ownershipThe Rust rule that every value has one owner, and is freed when the owner goes away.Rust: a value has one owner
packageIn Go, a folder of files that share a name. In Rust, a crate with a Cargo.toml.Go: a folder is a package, and a capital letter is an export
pipeliningSending many commands before reading any reply, so that the network wait is paid once.Taking a number
pollThe portable call that waits until one of a set of sockets is ready.C: one thread, and a poller to say who is ready
pollerThe loop's question to the kernel, which socket is ready, asked with poll, kqueue or epoll.C again: poll, kqueue and epoll behind one header
profileA record of where a running program spent its time, taken by sampling the stack.C: a clock, a loop and five flags
race detectorThe Go tool, go test -race, that watches memory access and reports a data race it sees.Go: the runner is in the box, and so is the fuzzer
raw pointerA Rust pointer, *const T or *mut T, with the borrow checker's guarantees removed, which only an unsafe block may read through.The five powers, and the pointer they are for
regionA marked part of a listing that the book pulls in by name, so the text and the code cannot drift.The code
repr(C)The Rust attribute that lays a struct or union out as a C compiler would, so that the bytes mean the same on both sides.The five powers, and the pointer they are for
RESPThe Redis serialization protocol. Arrays of bulk strings in, and simple strings, errors, integers, bulk strings or nil out.C: a buffer, a parser and three answers
ResultThe Rust type that is either a value or an error, and must be looked at.Rust: a Result at every step
sanitizerA compiler mode that adds checks for memory errors, undefined behaviour or data races.C: a runner you write, and a sanitizer that watches
scoped threadA Rust thread spawned inside thread::scope, which may borrow local data because the scope joins it before returning.Rust again: the load generator with scoped threads
self-pipe trickA signal handler that writes one byte into a pipe the event loop waits on, so that a signal becomes an event.C: a signal, a flag and a pipe
semantic versioningThree numbers, major, minor and patch, that say what kind of change a release holds.Versions and releases
Send and SyncThe two Rust marker traits that say a value may move to another thread, or be shared between threads, derived by the compiler.Send and Sync
static linkingLinking where the binary carries its libraries inside and needs nothing from the machine.What a binary carries
sum typeA value that is one of several kinds, with a tag that says which. A tagged union in C, an interface in Go, an enum in Rust.C: a tag and a union
sweepThe background pass that deletes expired keys nobody has asked for.C: a deadline on the entry, and a sweep with a cursor
tokioThe Rust async runtime that the book's second Rust server runs on.Rust once more: tasks on tokio
trait objectA value of unknown concrete type behind dyn Trait, called through a table of function pointers at run time.Rust: a module is a file you must announce
type switchThe Go switch on the dynamic type inside an interface value, one case per type.Go: any, and a switch on the type
undefined behaviourA C or unsafe Rust operation the standard does not define, after which the compiler may do anything.C: a runner you write, and a sanitizer that watches
unionA C or Rust type whose fields share the same bytes, so that only one is meaningful at a time.A struct and a union on the border
unsafeThe Rust keyword that marks a block where the five extra powers are allowed, and the author takes responsibility.Rust: extern and unsafe
X-macroA list written once as a macro, reused with a different meaning of X for a table, a declaration or a call.C: a runner you write, and a sanitizer that watches
zigA toolchain that includes a C compiler and the C libraries of many targets, used here to cross-compile.C: zig as the cross-compiler

Appendix C: C in one chapter#

Appendices · Appendix C· 7 min read

Every C concept the book teaches, in the groups of the concept index, with one to three sentences each. A concept that a chapter teaches links to that section. A concept kv never needed is taught here, and links to its official reference. Open this appendix six months after the book, when a construct looks familiar and its name has gone.

Types and values#

ConceptWhat to rememberWhere
integer types and stdint.hint, long and friends have platform sizes; int32_t and uint64_t say exactly what they are.C: an array of chains, and every pointer accounted for c/ch06/kv.c
char signednessPlain char may be signed or unsigned by platform, so a byte over 127 can become negative.C: an array of chains, and every pointer accounted for c/ch06/kv.c
integer promotion and the usual conversionsSmall types widen to int before arithmetic, so two chars add as ints. When a signed and an unsigned operand meet, the signed one becomes unsigned, and -1 is suddenly the largest value. Compare sizes as size_t and counts as int, and never mix them in one expression.reference
signed overflow is undefinedOverflowing a signed integer is undefined behaviour, so the compiler may assume it never happens.Three answers to the largest integer plus one c/ch05/kv.c
unsigned wrapUnsigned arithmetic wraps around modulo 2^n, by definition, which is how a hash function is allowed to overflow.Three answers to the largest integer plus one c/ch05/kv.c
boolstdbool.h since C99, a keyword in C23. Any non-zero value is true.C17 against C23 c/ch14/extras/c23.c
enumNamed integer constants, with no check that a value is one of the names.switch, and the cases that run into each other c/ch04/kv.c
sizeofThe size of a type or an object in bytes, computed at compile time, with no parentheses needed for an object.What a struct is made of, and an arena c/ch06/generic.c
size_t, ssize_t and ptrdiff_tsize_t is the unsigned type of a size and an index, and sizeof yields it. ssize_t is the signed one that read and write return, so that -1 can mean failure. ptrdiff_t is the type of one pointer minus another.reference
castsAn explicit conversion the compiler will not question, so each one is a claim you must be able to defend.What C does instead of generics c/ch06/generic.c
const and const-correctnessA promise not to write through a pointer, checked by the compiler and carried through every call.The parser stops copying c/ch10/src/resp.h
volatileEvery read and write must happen, because something outside the program can change the value.C: a signal, a flag and a pipe c/ch10/src/stop.h
static, extern and linkagestatic hides a name inside one file; extern says a name is defined elsewhere.C: two files, one binary hello/c/greet.c
translation unitsEach .c file compiles alone, with the headers pasted in, and the linker joins the results.C: two files, one binary hello/c/units.c
storage durationAutomatic variables live until the block ends, static ones for the whole run, allocated ones until free.Where a value lives, and for how long c/ch06/kv.c
stack and heapLocals live on the stack and vanish with the function; malloc gives memory that outlives it.Where a value lives, and for how long c/ch06/kv.c
zero initialisation= {0} sets every member to zero, and a static variable is zero without it.C: an array of chains, and every pointer accounted for c/ch06/kv.c
designated initialisers{.fd = -1, .len = 0} names the members, so the order of the struct does not matter.C again: poll, kqueue and epoll behind one header c/ch09/src/poller_epoll.c
compound literals(struct pollfd){fd, POLLIN, 0} builds a value in place without a named variable.C: an array of chains, and every pointer accounted for c/ch06/kv.c

Pointers and memory#

ConceptWhat to rememberWhere
pointersAn address plus a type, and the type says how many bytes one step is.C: an array of chains, and every pointer accounted for c/ch06/kv.c
pointer arithmeticp + 1 moves by one element, not one byte, so the type of p decides the step. The result is only defined inside the same object, or one past its end, and the chapter 4 parser stays inside its buffer by comparing against end.reference
arrays decay to pointersAn array name used as a value becomes a pointer to its first element, and the length is gone. That is why a function takes a pointer and a length, and why sizeof an array parameter is the size of a pointer.reference
pointer to pointerHow a function gives its caller a new pointer, as in a grow function that may move a buffer.C: an array of chains, and every pointer accounted for c/ch06/kv.c
void pointerA pointer to anything, which malloc returns and a callback context uses.What C does instead of generics c/ch06/generic.c
NULL and nullptrThe null pointer. C23 adds nullptr with its own type.C17 against C23 c/ch14/extras/c23.c
malloc, calloc, realloc and freeAsk for bytes, get them zeroed, grow them in place or elsewhere, give them back.Growing, measuring and slicing bytes c/ch05/kv.c
ownership conventionsA comment on every pointer says who frees it, because the compiler will not.The parser stops copying c/ch10/src/resp.h
alignment and offsetofEach type has an alignment, structs get padding to keep it, and offsetof finds a member's position.C: an array of chains, and every pointer accounted for c/ch06/generic.c
flexible array membersA struct whose last member is an array of unknown size, declared as data[], is allocated in one block with room after its header. Redis stores its strings this way, with the length and the bytes in one allocation.reference
restrictA promise that two pointers do not overlap, which lets the compiler optimise.The optimisation levels, measured c/ch13/bench/bench_copy.c
strict aliasingReading an object through a pointer of a different type is undefined, except through char or unsigned char. The compiler may assume an int and a float never overlap, which is why a memcpy is the way to reinterpret bytes.reference
use after free, double free and leaksThe three memory bugs C lets you write, and what each looks like under AddressSanitizer.C: an array of chains, and every pointer accounted for c/ch06/kv.c
arenasAllocate many small things from one block and free them all at once.What a struct is made of, and an arena c/ch06/generic.c

Structs and unions#

ConceptWhat to rememberWhere
structMembers laid out in order, with padding, passed by value unless you pass a pointer.C: an array of chains, and every pointer accounted for c/ch06/kv.c
typedefA new name for a type, which is how kv_store stands for struct kv_store.C: an array of chains, and every pointer accounted for c/ch06/kv.c
opaque pointersA header declares struct kv_store without its members, so callers cannot touch them.An opaque type, and a struct of functions c/ch07/src/store.h
unionMembers that share the same bytes, so only one is meaningful at a time.C: a tag and a union c/ch08/src/store.h
tagged unionA union plus an enum that says which member is live: C's hand-built sum type.C: a tag and a union c/ch08/src/store.h
bitfieldsMembers with a width in bits, packed by the compiler in an order it chooses.C: a tag and a union

Functions and control#

ConceptWhat to rememberWhere
switch and fallthroughA jump table on an integer, where a case without break runs into the next one.switch, and the cases that run into each other c/ch04/kv.c
do whileA loop that runs its body once before it tests the condition. The test runner's CHECK macro is built on one with a condition of 0, so that it runs exactly once.reference
goto cleanupOne exit path at the end of a function that frees everything, reached by goto from each failure.C: four calls, and a way to clean up c/ch03/kv.c
function pointersA pointer to code, declared with the signature it points to, which is how C passes behaviour.A table of functions c/ch05/kv.c
callbacks with a void contextA function pointer plus a void * that the callee passes back, which is C's closure.An opaque type, and a struct of functions c/ch07/src/cmd.c
variadic functionsFunctions that take any number of arguments, read with va_list, as printf does.A function that takes any number of arguments c/ch04/kv.c
inlineA hint to paste the function body at each call, with linkage rules that surprise.The optimisation levels, measured c/ch13/Makefile
static functionsA function only this file can call, which the compiler can warn about when unused.C: headers, static, and a Makefile that knows about both c/ch07/src/buf.c
_NoreturnMarks a function that never returns, such as one that prints an error and exits.C again: options, and a function that does not come back c/ch14/src/main.c

Preprocessor#

ConceptWhat to rememberWhere
#includePastes a file in place. Angle brackets search the system, quotes search here first.C: headers, static, and a Makefile that knows about both c/ch07/src/main.c
header guards and #pragma onceThe #ifndef at the top of a header that stops it being pasted twice.C: headers, static, and a Makefile that knows about both c/ch07/src/buf.h
object and function macrosText replacement before compilation, with or without arguments, with no types.A table of functions c/ch05/kv.c
do { } while (0)The wrapper that makes a macro of several statements behave like one statement, so that it can follow an if without braces and still take a semicolon. The CHECK macro in chapter 12 is one.reference
X-macrosOne list of entries that expands into a table, an enum and a set of names at once.C: bytes, a length and an owner c/ch05/kv.c
#ifdef platform guardsChoose code at compile time by platform, as in __linux__ against __APPLE__.C again: poll, kqueue and epoll behind one header c/ch09/src/poller.h
_Static_assertA check the compiler runs, such as that a struct has the size you think.C: a runner you write, and a sanitizer that watches c/ch12/tests/test_resp.c
_GenericA compile-time switch on the type of an expression, the nearest C gets to overloading.What C does instead of generics c/ch06/generic.c
predefined macros__FILE__, __LINE__, __func__ and the platform and compiler names.C again: poll, kqueue and epoll behind one header c/ch09/src/poller.h
__attribute__ and compiler extensionsNon-standard annotations such as format checking and cleanup, understood by gcc and clang.C again: options, and a function that does not come back c/ch14/src/main.c

Strings and I/O#

ConceptWhat to rememberWhere
NUL-terminated stringsA string is bytes up to the first zero, so a string cannot hold a zero and its length is a walk.C: bytes, a length and an owner c/ch05/kv.c
strlen, memcpy and memcmpThe three calls behind most string code, and what each assumes about its arguments.C: bytes, a length and an owner c/ch05/kv.c
the strncpy trapstrncpy fills the whole destination and may leave no terminating zero when the source is as long as the limit. A bounded copy of a C string wants snprintf, and a copy of bytes with a length wants memcpy, which kv uses throughout.reference
snprintfFormats into a buffer of a known size and never overruns it, returning what it wanted to write.Growing, measuring and slicing bytes c/ch05/kv.c
pointer and length disciplineEvery buffer travels with its length, which is how kv handles bytes that may contain zero.C: a buffer, a parser and three answers c/ch04/kv.c
stdio against read and writeFILE * buffers in user space; read and write are the system calls underneath.C: a file descriptor, a write, and a thread for the fsync c/ch11/src/aof.c
errno, strerror and perrorA failed call returns -1 and sets a thread-local number, which strerror turns into words.errno: the number a failed call leaves behind c/ch03/kv.c

Concurrency and signals#

ConceptWhat to rememberWhere
pthreads and C11 threadsCreate a thread with a function and a void *, join it to wait.C again: the second thread c/ch11/src/syncer.c
mutexpthread_mutex_lock and unlock around every access to shared data, with nothing to stop you forgetting.C again: the second thread c/ch11/src/syncer.c
condition variableSleep until another thread signals, always inside a loop that rechecks the condition.C again: the second thread c/ch11/src/syncer.c
atomics and memory orderstdatomic.h gives operations no other thread can see half done, with an ordering you choose.C again: the second thread c/ch11/src/syncer.c
thread safety of libcMost of libc is safe to call from threads; strtok, localtime and a few others are not.Thread safety of the C library
signals and sigactionThe kernel interrupts the program to run a handler, which may only do a few safe things.C: a signal, a flag and a pipe c/ch10/src/stop.c
sig_atomic_t and the self-pipe trickA handler sets a flag or writes one byte to a pipe that the poll loop watches.C: a signal, a flag and a pipe c/ch10/src/stop.c
SIGPIPEA write to a peer that has gone raises a signal that kills the process unless ignored.C: a signal, a flag and a pipe c/ch10/src/main.c
forkClone the process. The Unix way to start a program, which Windows does not have.C again: the second thread

Build and tooling#

ConceptWhat to rememberWhere
compiler flags and -O levels-O0 for debugging, -O2 for release, and what the optimiser is allowed to assume.The optimisation levels, measured c/ch13/Makefile
warnings as errors-Wall -Wextra -Werror turns the compiler's advice into a gate.C: from source to binary in four steps hello/c/Makefile
static librariesAn archive of object files that the linker copies from, made with ar.C: a library that is easy to call c/ch15/Makefile
MakeTargets, prerequisites and recipes, with tabs, and a rule for how a .o comes from a .c.Build tools, and the lack of one hello/c/Makefile
CMakeA generator of build files for larger projects, which most C libraries now use.Build tools, and the lack of one
pkg-configAsks an installed library for the flags to compile and link against it.Dependencies in C: the package manager that is not there
AddressSanitizer and UndefinedBehaviorSanitizerCompile-time instrumentation that reports memory errors and undefined behaviour as they happen.C: a runner you write, and a sanitizer that watches c/ch12/Makefile
ThreadSanitizerThe sanitizer that reports data races, which cannot run together with AddressSanitizer.C again: the second thread c/ch11/src/syncer.c
valgrindRuns the unmodified binary under an emulator and reports leaks and bad accesses, on Linux.C: a runner you write, and a sanitizer that watches
a debugger sessionlldb or gdb: break, step, print, backtrace, and the -g flag that makes them useful.A debugger session
clang-formatThe formatter, driven by a .clang-format file, since C has no single style.Build tools, and the lack of one
C17 against C23nullptr, constexpr, typeof, auto, #embed, attributes and true keywords, and how far clang and gcc support them.C17 against C23 c/ch14/extras/c23.c

Conventions#

ConceptWhat to rememberWhere
name prefixeskv_ on every public name, because C has one namespace for the whole program.C: headers, static, and a Makefile that knows about both c/ch07/src/buf.h
header hygieneA header declares, includes only what it needs, and never defines a variable.C: headers, static, and a Makefile that knows about both c/ch07/src/buf.h
error returnsReturn a code or a bool and put the result in an out parameter, consistently.C: bytes, a length and an owner c/ch05/kv.c
ownership commentsWho frees it, written next to the pointer, every time.C: bytes, a length and an owner c/ch05/kv.c
const parametersA const char * parameter promises the caller the bytes come back unchanged.C: bytes, a length and an owner c/ch05/kv.c

Appendix D: Go in one chapter#

Appendices · Appendix D· 6 min read

Every Go concept the book teaches, in the groups of the concept index, with one to three sentences each. A concept that a chapter teaches links to that section. A concept kv never needed is taught here, and links to its official reference. Open this appendix six months after the book, when a construct looks familiar and its name has gone.

Basics#

ConceptWhat to rememberWhere
packages and importsA folder is a package, an import is a path, and an unused import is an error.Go: one command, one binary hello/go/main.go
zero valuesEvery variable starts as zero, nil or empty, so a struct is usable without a constructor.Every variable starts at zero hello/go/main.go
:= and varInside a function, := declares and assigns in one step and infers the type. var works anywhere and takes an explicit type or a zero value. A := in an inner scope can declare a new variable that shadows one outside, which is the usual way an err goes missing.reference
const and iotaA const is untyped until it is used, so 1 << 40 is fine where an int would overflow. Inside one const block, iota counts from 0 on each line, which is how the aof package names its three policies.reference
basic types and conversionsint is the machine word, every conversion is written out, and there is no implicit widening.Three answers to the largest integer plus one go/ch05/strings_test.go
runes, strings, bytes and UTF-8A string is immutable bytes, range gives runes, and len counts bytes.Bytes, runes and a builder go/ch05/strings_test.go
arrays against slicesAn array has a fixed length in its type; a slice is a view of one with a length and a capacity.Go: the same table, with nothing to free go/ch06/table.go
len, cap, append and aliasingappend writes into spare capacity, so two slices can share and overwrite the same array.Bytes, runes and a builder go/ch05/exec.go
copy and the full slice expressioncopy duplicates bytes, and s[a:b:c] caps the capacity to stop append from aliasing.The parser stops copying go/ch10/internal/resp/resp.go
mapsReading a nil map is fine and writing one panics; iteration order is random on purpose.Go: two kinds of string, and a copy between them go/ch05/store.go
comma-okv, ok := m[k] separates a missing key from a zero value, and the same form works for assertions and channels.Go: two kinds of string, and a copy between them go/ch05/store.go
structs and composite literalsFields in order or by name, and a pointer to a literal with &.Go: the same table, with nothing to free go/ch06/table.go
pointers against valuesEverything is passed by value, so a function that must change the caller's struct takes a pointer. A method on a pointer receiver can change the value and a method on a value receiver gets a copy. The store's methods all take *Store.reference
the forms of rangeOver a slice, a map, a string, a channel, an integer since 1.22, and a function since 1.23.A function that range can call go/ch08/internal/store/value.go
switch and type switchNo fallthrough by default, cases can be expressions, and switch v := x.(type) branches on a type.Go: any, and a switch on the type go/ch08/internal/store/value.go
labeled break and continueThe way out of a nested loop, with a label on the outer one.A function that range can call

Functions and methods#

ConceptWhat to rememberWhere
multiple returns and named resultsA function returns a value and an error together, and named results document them.Go: any, and a switch on the type go/ch08/internal/cmd/cmd.go
variadic functionsf(args ...string) takes any number, and f(slice...) spreads one.Bytes, runes and a builder go/ch05/exec.go
closures and captureA function literal captures variables by reference, and since 1.22 each loop iteration has its own.Go: two kinds of string, and a copy between them go/ch05/exec.go
deferRuns at function exit, last in first out, with its arguments evaluated at the defer line.Two rules of defer go/ch03/defer_test.go
panic and recoverA panic unwinds the stack; a deferred recover stops it, which is how one bad client does not kill a server.panic and recover go/ch10/cmd/kv/main.go
methods and receiversA function with a receiver, on a pointer when it changes the value, on a value when it does not.Go: a folder is a package, and a capital letter is an export go/ch07/internal/store/store.go
method setsA value of type T has the methods with value receivers, and a *T has those and the pointer-receiver ones too. That decides which of the two satisfies an interface: a *Store satisfies Storer, and a Store does not.reference
method valuess.Get without the call is a function bound to s, which a dispatch table can hold.A map of functions go/ch05/exec.go
embedding and promotionAn unnamed field lends its methods to the outer struct, as an embedded sync.Mutex does.Embedding, and the nil that is not nil go/ch07/internal/cmd/cmd_test.go

Interfaces#

ConceptWhat to rememberWhere
interfaces and implicit satisfactionA set of method signatures, and any type with those methods satisfies it without saying so.An interface is a set of methods go/ch07/internal/cmd/cmd.go
small interfacesio.Reader, io.Writer, fmt.Stringer and error have one method each, and most code accepts them.An interface is a set of methods go/ch07/internal/cmd/cmd.go
anyThe empty interface, which every value satisfies, and which a type switch takes apart.Go: any, and a switch on the type go/ch08/internal/store/value.go
type assertionv, ok := x.(T) asks whether the dynamic value is a T, and without ok it panics.Go: any, and a switch on the type go/ch08/internal/cmd/cmd.go
the nil interface trapAn interface holding a nil pointer is not nil, which makes a returned *MyErr look like an error.Embedding, and the nil that is not nil go/ch07/internal/cmd/cmd_test.go
interface embeddingAn interface may list other interfaces, and gets their methods. io.ReadWriter is io.Reader and io.Writer in one declaration, and a type that has both methods satisfies all three.reference
when not to use an interfaceAccept interfaces, return structs, and define one only when a second implementation exists.Embedding, and the nil that is not nil go/ch07/internal/cmd/cmd.go

Generics#

ConceptWhat to rememberWhere
type parametersfunc Max[T cmp.Ordered](a, b T) T, since Go 1.18, with the type inferred at the call.Go: the same table, with nothing to free go/ch06/table.go
constraints, comparable and anyAn interface used as a bound, comparable for map keys, and a union of types with |.Go: the same table, with nothing to free go/ch06/table.go
the slices, maps and cmp packagesSort, search, clone and compare, written once with generics instead of per type.Go: the same table, with nothing to free go/ch06/table_test.go
when to use genericsA container or an algorithm over many types, not a method that takes an interface.Go: the same table, with nothing to free go/ch06/table.go

Errors#

ConceptWhat to rememberWhere
the error interfaceAnything with an Error() string method, and nil means no error.An error is any value with an Error method go/ch04/resp.go
custom error typesA struct with fields the caller can read, returned as an error.An error is any value with an Error method go/ch04/resp.go
wrapping with %wfmt.Errorf("read config: %w", err) adds context and keeps the cause reachable.An error is any value with an Error method go/ch04/resp.go
errors.Is, errors.As and errors.JoinWalk the wrapped chain for a sentinel, for a type, or combine several errors.errors.Join go/ch12/internal/aof/aof.go
sentinel against typed errorsvar ErrNotFound = errors.New(...) for a condition, a type when the caller needs fields.An error is any value with an Error method go/ch04/resp.go
panics against errorsAn error is a value for the caller; a panic is a bug in the program.panic and recover

Concurrency#

ConceptWhat to rememberWhere
goroutinesgo f() runs f on a stack of a few kilobytes, multiplexed over the real threads.Go: one word, and one lock go/ch09/cmd/kv/main.go
channelsA typed queue between goroutines; unbuffered ones hand off, buffered ones hold a few.Go again: fifty workers and three channels go/ch09/cmd/kvload/main.go
close and range over a channelThe sender closes, the receiver's range ends, and a receive from a closed channel gives zero.Go again: fifty workers and three channels go/ch09/cmd/kvload/main.go
nil channelsA nil channel blocks forever, which is a way to switch a select case off.Go again: fifty workers and three channels go/ch09/cmd/kvload/main.go
selectWait on several channels at once, with default for a non-blocking try and a timer for a timeout.Go again: fifty workers and three channels go/ch09/cmd/kvload/main.go
sync.Mutex and RWMutexLock around shared data, with RLock for readers, and nothing stops you forgetting.Go: one word, and one lock go/ch09/internal/store/store.go
sync.WaitGroupAdd before starting, Done when finished, Wait for all, and wg.Go since 1.25.Go: one word, and one lock go/ch09/internal/store/store_test.go
sync.Once and sync.PoolRun something exactly once, and reuse buffers instead of allocating.The rest of the sync package
sync/atomicCounters and flags that need no lock, as atomic.Int64.The rest of the sync package
contextCancellation, a deadline and request values that flow down every call.context in one paragraph go/ch10/cmd/kv/main.go
worker poolsA fixed number of goroutines reading jobs from one channel.Go again: fifty workers and three channels go/ch09/cmd/kvload/main.go
data races and the race detectorTwo goroutines, one writing, no sync, and go test -race finds it.Go: one word, and one lock go/ch09/internal/store/store_test.go
GOMAXPROCS and the schedulerGoroutines run on P logical processors over M threads, and a blocking system call hands its P over.The runtime in one paragraph
goroutine leaksA goroutine blocked forever on a channel nobody will touch again.Go again: fifty workers and three channels go/ch09/cmd/kvload/main.go

Runtime and memory#

ConceptWhat to rememberWhere
the garbage collectorConcurrent mark and sweep, with short pauses and memory held until the next cycle.Go: the same table, with nothing to free go/ch06/table.go
escape analysisgo build -gcflags=-m says which values move to the heap, and why.Escape analysis go/ch13/internal/resp/resp.go
GOGC and GOMEMLIMITHow often the collector runs, and a ceiling on the heap.GOGC, GOMEMLIMIT and the Pool
pprofCPU and memory profiles from a running program, read as a graph or a flamegraph.The profiler is a package go/ch13/cmd/kv/main.go

Tooling and packaging#

ConceptWhat to rememberWhere
go.mod and go.sumThe module path and its dependencies, and the hashes that pin them.Go: one command, one binary hello/go/go.mod
semantic import versioningA v2 module has /v2 in its import path, so two majors can coexist.Go: a folder is a package, and a capital letter is an export
go get and replaceAdd or upgrade a dependency, or point a module at a local folder.Go: a folder is a package, and a capital letter is an export
workspacesgo.work lets several modules build together while you change them.Rust: a module is a file you must announce
internal packagesA package under internal/ can only be imported from inside its parent.Go: a folder is a package, and a capital letter is an export go/ch07/internal/store/store.go
go vet, gofmt and staticcheckThe checker, the one formatter, and the linter most teams add.Go: one command, one binary
build tags and platform files//go:build linux, or a _linux.go suffix, picks a file by platform.Go's platform files: a line at the top go/ch09/internal/sys/limit_unix.go
go generate and go:embedRun a tool from a comment, and compile a file into the binary.Go again: flags, a file, a log and a version go/ch14/cmd/kv/main.go
table-driven testsA slice of cases and one loop, the idiom every Go test file uses.Go: a folder is a package, and a capital letter is an export go/ch07/internal/store/glob_test.go
subtests, t.Parallel and helperst.Run names a case, t.Parallel runs them together, t.Helper hides a helper from the report.Go: the runner is in the box, and so is the fuzzer go/ch12/internal/resp/resp_test.go
benchmarksfunc BenchmarkX(b *testing.B) with b.Loop, and go test -bench.Go: nine allocations, and the flag that names them go/ch13/internal/resp/bench_test.go
fuzzingfunc FuzzX(f *testing.F) with go test -fuzz, built into the toolchain.Go: the runner is in the box, and so is the fuzzer go/ch12/internal/resp/resp_test.go
cross-compilingGOOS and GOARCH, with CGO_ENABLED=0 for a static binary.Go: two variables go/ch14/Makefile
cgoimport "C" and a preamble, with a cost on every call.Go: cgo go/ch15/internal/cmatch/cmatch.go
the module proxy and vendoringproxy.golang.org serves modules, and go mod vendor copies them into the repository.Go: a folder is a package, and a capital letter is an export
go docDoc comments on exported names, read with go doc and on pkg.go.dev.The documentation is in the source

Standard library tour#

ConceptWhat to rememberWhere
fmtPrintf and its verbs: %v for anything, %q for a quoted string, %d and %s as in C, and %w to wrap an error in Errorf. Println adds spaces and a newline, and a type with a String method prints as it chooses.reference
strconvAtoi, ParseInt and Itoa, with errors, not panics.Go: a reader that waits, and an error that names itself go/ch04/resp.go
strings and bytesThe same functions twice, for string and for []byte, plus strings.Builder.Bytes, runes and a builder go/ch05/strings_test.go
bufioA buffered reader and writer over any io.Reader or io.Writer.Go: a reader that waits, and an error that names itself go/ch04/resp.go
io and osThe interfaces, and files, environment and arguments.Go: the loop you did not have to write go/ch03/main.go
netnet.Listen, Accept and net.Conn, which is an io.ReadWriteCloser.Go: the loop you did not have to write go/ch03/main.go
timetime.Time and Duration, the monotonic reading hidden inside Now, Timer and Ticker.Go: a ticker, and a clock the test can hold still go/ch10/internal/store/store.go
encoding/jsonStruct tags and Marshal, the most used package after fmt.Go again: flags, a file, a log and a version go/ch14/cmd/kv/main.go
flagCommand-line flags with defaults and a generated usage.Go again: flags, a file, a log and a version go/ch14/cmd/kv/main.go
log/slogStructured logging with levels and key-value pairs.Go again: flags, a file, a log and a version go/ch14/cmd/kv/main.go
os/signalsignal.Notify delivers a signal on a channel, and NotifyContext cancels a context.Go: a context that a signal cancels go/ch10/cmd/kv/main.go

Appendix E: Rust in one chapter#

Appendices · Appendix E· 7 min read

Every Rust concept the book teaches, in the groups of the concept index, with one to three sentences each. A concept that a chapter teaches links to that section. A concept kv never needed is taught here, and links to its official reference. Open this appendix six months after the book, when a construct looks familiar and its name has gone.

Basics#

ConceptWhat to rememberWhere
let, mut and shadowingA binding is immutable unless it says mut, which the compiler enforces. A second let with the same name shadows the first, and may change its type, which is how a parsed value replaces the string it came from.reference
integer overflow rulesDebug builds panic on overflow, release builds wrap, and checked_add and wrapping_add say what you mean.Three answers to the largest integer plus one rust/ch05/src/store.rs
as and From castsas converts between numbers and may truncate or wrap, so n as u8 of 300 is 44. From and Into convert where the conversion cannot fail, and the ? operator uses From to turn one error type into another.reference
tuples, arrays and slicesA tuple groups a fixed set of values of mixed types, and a pattern takes it apart. An array [T; N] has a length in its type. A slice &[T] is a borrowed view of a run of elements, with its length beside the pointer, and it is what every function in kv takes.reference
VecThe growable array that owns its elements and frees them when it drops.Rust: a value has one owner rust/ch05/src/store.rs
String, str, char and bytesString owns valid UTF-8, &str borrows it, char is a code point, and as_bytes gives the bytes.str, String, char and a slice of bytes rust/ch05/src/store.rs
structs and impl blocksData in a struct, behaviour in impl, and a value is built with a literal or a new function.Rust: the table that drops itself rust/ch06/src/table.rs
methods and the forms of self&self reads, &mut self changes, self consumes.Rust: the table that drops itself rust/ch06/src/table.rs
enums with dataEach variant can carry its own fields, which is a sum type the compiler checks.One enum for everything that can go wrong rust/ch04/src/resp.rs
Option and ResultSome or None for a value that may be absent, Ok or Err for one that may fail.Rust: an Option for the end, a Result for the rest rust/ch04/src/resp.rs
match and patternsEvery case must be handled, with guards, bindings, slice and struct patterns, and @.Rust: an Option for the end, a Result for the rest rust/ch04/src/main.rs
if let, let else and while letMatch one pattern without the full match, or leave early when it does not match.Rust: an Option for the end, a Result for the rest rust/ch04/src/resp.rs
loops and labelsloop runs until break, and break may carry a value out of it. while and for are the usual ones, and for takes anything that is an iterator. A label such as 'outer: lets a break or continue name the loop it means.reference
expressions against statementsA block is an expression, and its value is the last line without a semicolon. So if, match and a plain block all produce values, and a function returns the value of its final expression. A semicolon turns an expression into a statement that yields nothing.reference
const and staticA const is inlined, a static has one address, and a mutable static needs unsafe.A table of fn pointers, built by a macro rust/ch05/src/cmd.rs
modules and visibilitymod, pub, pub(crate), use and paths, with files as modules.Rust: a module is a file you must announce rust/ch07/src/store.rs
CargoCrates, features, workspaces, profiles, build.rs and the lockfile, with semantic versioning.Rust: cargo owns the project hello/rust/Cargo.toml
doc comments and doc tests/// becomes documentation, and a code block in it is compiled and run by cargo test.Rust: the compiler already ran, and the tools check the rest rust/ch12/src/resp.rs
rustfmt and clippyThe one formatter and the linter with hundreds of checks, both standard.Rust: cargo owns the project
editionsA crate picks 2015, 2018, 2021 or 2024, and crates of different editions link together.Rust: cargo owns the project hello/rust/Cargo.toml

Ownership#

ConceptWhat to rememberWhere
moveAssigning or passing a value gives it away, and the old name cannot be used again.Rust: a value has one owner rust/ch05/src/cmd.rs
Copy against CloneSmall plain values copy on assignment; everything else moves unless you call clone.Lifetimes, in the one place kv needs them
borrows& lends a value for reading, &mut for changing, and the owner waits until the borrow ends.Rust: a value has one owner rust/ch05/src/store.rs
mutable or shared, never bothAny number of readers or one writer at a time, which is the rule that prevents data races.Rust: a value has one owner
non-lexical lifetimesA borrow ends at its last use, not at the end of the block.Lifetimes, in the one place kv needs them rust/ch10/src/main.rs
lifetime annotations and elision'a names how long a reference lives, and the compiler fills in the common cases.Lifetimes, in the one place kv needs them rust/ch10/src/resp.rs
'staticLives for the whole program, as a string literal does, or owns everything it holds.Lifetimes, in the one place kv needs them
structs that hold referencesCommand<'a> borrows from a buffer and cannot outlive it.The parser stops copying rust/ch10/src/resp.rs
Drop and RAIIA value is cleaned up when its owner goes out of scope, in reverse order of declaration.Rust: the table that drops itself rust/ch06/src/table.rs
mem::take, replace and swapMove a value out of a place you only have &mut to, leaving a default behind.Rust again: a scoped thread on a channel rust/ch11/src/aof.rs
interior mutabilityCell, RefCell and Mutex let you change a value through a shared reference, checked at runtime.Two owners of one table rust/ch06/src/table.rs
Rc and ArcShared ownership by reference count, Arc when threads are involved.Two owners of one table rust/ch06/src/table.rs
BoxOne owner of a value on the heap, which recursive types and trait objects need.Closures and iterators, named at last rust/ch08/src/chain.rs
DerefA type that implements Deref is treated as the type it dereferences to where a reference is wanted. That is why a &String is accepted as a &str and a &Vec<T> as a &[T], and why a Box<T> can be used as a T.reference
borrowing from a VecA slice of a Vec blocks every push until the slice is dropped.Lifetimes, in the one place kv needs them rust/ch10/src/main.rs
CowBorrowed or owned, decided at runtime, for a value that usually needs no copy.Lifetimes, in the one place kv needs them rust/ch10/src/cmd.rs

Traits and generics#

ConceptWhat to rememberWhere
defining and implementing a traitA set of methods a type promises, implemented in an impl Trait for Type block.A trait is a promise of methods rust/ch07/src/store.rs
default methodsA trait method with a body that implementations may keep or replace.A trait is a promise of methods rust/ch07/src/store.rs
derive#[derive(Debug, Clone, PartialEq)] asks the compiler to write the usual implementations of those traits from the fields. The book derives Debug on every error type, so that a test can print one, and Default on the stores.reference
the standard traitsDebug and Display for printing, Clone and Copy for copying, Default for an empty value, PartialEq, Eq, Hash and Ord for comparing and hashing, From and Into for converting, AsRef for borrowing in any form, Iterator for walking, Read and Write for bytes, and Error for errors. Most are derived, and the rest are an impl block each.reference
generics and boundsfn largest<T: Ord>(xs: &[T]) works for any T that can be ordered, and where clauses spell long bounds out.Rust: the table that drops itself rust/ch06/src/table.rs
impl Trait in argument and return positionfn f(w: impl Write) is a generic in disguise, and -> impl Iterator hides a type.Static and dynamic dispatch rust/ch07/src/store.rs
trait objects and dynBox<dyn Store> picks the implementation at runtime through a vtable.Static and dynamic dispatch rust/ch07/src/main.rs
object safetyA trait with generic methods or Self returns cannot be a dyn.Static and dynamic dispatch rust/ch07/src/store.rs
static against dynamic dispatchGenerics copy the code per type and inline; dyn calls through a pointer and keeps one copy.Static and dynamic dispatch rust/ch07/src/store.rs
monomorphisationEach use of a generic with a new type compiles to its own code, which is fast and slow to build.Rust: the table that drops itself rust/ch06/src/table.rs
the orphan ruleYou can implement a trait for a type only if you own one of them.A trait is a promise of methods rust/ch07/src/table.rs
operator overloadingAn operator is a trait: + is Add, [] is Index, == is PartialEq. A type that implements the trait gets the operator, and nothing else is allowed to redefine one.reference
associated types and constantsIterator's Item is a type chosen by the implementation, not by the caller.Closures and iterators, named at last rust/ch08/src/chain.rs

Closures and iterators#

ConceptWhat to rememberWhere
closures|x| x + 1 captures its surroundings by reference, by mutable reference, or by move.Closures and iterators, named at last rust/ch08/src/cmd.rs
Fn, FnMut and FnOnceWhat a closure may do with what it captured, which decides where it can be passed.Closures and iterators, named at last rust/ch08/src/cmd.rs
iterators and adaptorsmap, filter, enumerate, zip, take and the rest, each a lazy wrapper around the last.Closures and iterators, named at last rust/ch08/src/store.rs
collect and the turbofishcollect::<Vec<_>>() builds any collection, and the type says which.Closures and iterators, named at last rust/ch08/src/cmd.rs
implementing IteratorOne method, next, and every adaptor works on your type.Closures and iterators, named at last rust/ch08/src/chain.rs
iter, iter_mut and into_iterBorrow the elements, borrow them mutably, or take them.Closures and iterators, named at last rust/ch08/src/chain.rs
lazinessNothing runs until something consumes the chain, so an unused map is a warning.Closures and iterators, named at last rust/ch08/src/store.rs

Errors#

ConceptWhat to rememberWhere
custom error enumsOne enum with a variant per failure, matched exhaustively by the caller.One enum for everything that can go wrong rust/ch04/src/resp.rs
Display and ErrorImplement Display for the message and Error to join the ecosystem.One enum for everything that can go wrong rust/ch04/src/resp.rs
From and ?? converts the error with From, so an io::Error becomes your KvError without a line of code.One enum for everything that can go wrong rust/ch04/src/resp.rs
Box<dyn Error>A boxed trait object that can hold any error type at all. A main or a tool that only reports errors returns Result<(), Box<dyn Error>> and uses ? on every error it meets, at the cost of no longer being able to match on the kind.reference
thiserror and anyhowThe two crates most projects use for errors. thiserror derives Display and From for an error enum like the chapter 4 KvError, so that the impl blocks are one attribute each. anyhow is the Box<dyn Error> idea with context, for programs that report errors and never match on them.reference
panics and the unwrap policyunwrap and expect are for bugs that cannot happen, with expect saying why.?, expect and unwrap rust/ch03/src/main.rs
Result from mainfn main() -> Result<(), Box<dyn Error>> prints the error and exits non-zero.A main that returns a Result rust/ch03/src/main.rs

Concurrency#

ConceptWhat to rememberWhere
threadsthread::spawn takes a closure that must be 'static, and join waits.Rust: a thread for each client, and a compiler that checks the sharing rust/ch09/src/main.rs
scoped threadsthread::scope lets threads borrow from the stack, because all are joined before it returns.Rust again: the load generator with scoped threads rust/ch09/src/bin/kvload.rs
Send and SyncMarker traits the compiler checks: safe to move to a thread, safe to share between threads.Send and Sync
Arc<Mutex<T>>Shared ownership plus one writer at a time, the shape most shared state takes.Rust: a thread for each client, and a compiler that checks the sharing rust/ch09/src/main.rs
RwLockMany readers or one writer, when reads dominate.Rust: one allocation, a profile, four experiments, and a lock rust/ch13/src/main.rs
mpsc channelsMany senders, one receiver, and the receiver's loop ends when every sender drops.Rust again: the load generator with scoped threads rust/ch09/src/bin/kvload.rs
atomicsAtomicBool and AtomicUsize with an Ordering, for flags and counters.Atomics rust/ch09/src/bin/kvload.rs
async and awaitAn async fn returns a Future, and await pauses it instead of blocking the thread.Rust once more: tasks on tokio rust/ch09/src/bin/kv_async.rs
what a Future isA state machine with a poll method that a runtime drives.What a future is
PinA future that borrows itself cannot move, and Pin is the promise that it will not.Pin in one paragraph
tokiospawn, tasks, the multi-threaded runtime, and its own Mutex and channels.Rust once more: tasks on tokio rust/ch09/src/bin/kv_async.rs
select!Wait on several futures and take the first, with cancellation of the rest.select!, cancellation and blocking
cancellationDropping a future stops it, which is both convenient and a trap for half-done work.select!, cancellation and blocking
blocking inside asyncA std Mutex held across await, or a long computation, stalls the whole runtime.select!, cancellation and blocking

Unsafe and FFI#

ConceptWhat to rememberWhere
raw pointers*const T and *mut T, with no borrow checking, dereferenced only in unsafe.The five powers, and the pointer they are for rust/ch15/src/cmatch.rs
unsafe blocks and functionsThe keyword that marks where the compiler's rules stop and yours start.Rust: extern and unsafe rust/ch15/src/cmatch.rs
the five powersDereference a raw pointer, call unsafe code, touch a mutable static, implement an unsafe trait, read a union.The five powers, and the pointer they are for
repr(C)Lay a struct out as C would, so both sides agree on the bytes.The five powers, and the pointer they are for rust/ch15/src/cmatch.rs
extern blocksDeclare C functions, unsafe since the 2024 edition, and export your own with no_mangle.Rust: extern and unsafe rust/ch15/src/cmatch.rs
miriThe interpreter that runs tests and reports undefined behaviour in unsafe code.Rust: the compiler already ran, and the tools check the rest
bindgenGenerates the extern declarations from a C header.The five powers, and the pointer they are for

Macros#

ConceptWhat to rememberWhere
macro_rules!Pattern-based macros that write code from a template, for tables and repetition.A table of fn pointers, built by a macro rust/ch05/src/cmd.rs
derive and attribute macros#[derive(...)] and #[tokio::main] are procedural macros, which run Rust code at compile time to generate code. They come from crates, and the book uses them without writing one. A macro_rules! macro, like the command table in chapter 5, is the simpler kind.reference
println!, format!, vec! and assert!The standard macros. println! and format! take a format string checked at compile time with a variable number of arguments, which a function cannot. vec! builds a Vec from a list, and assert! and assert_eq! are the test checks that print both sides when they fail.reference

Testing and tooling#

ConceptWhat to rememberWhere
unit, integration and doc tests#[cfg(test)] modules in the file, the tests/ directory, and examples in doc comments.Rust: the compiler already ran, and the tools check the rest rust/ch12/tests/roundtrip.rs
cargo test flagsFilter by name, show output, run one thread, and ignore the slow ones.cargo test, and what comes after the dashes
benchmarks with criterionStable Rust has no built-in bench, and criterion is the crate that fills the gap.Rust: one allocation, a profile, four experiments, and a lock rust/ch13/examples/bench_parse.rs
cargo-fuzzlibFuzzer targets in a fuzz/ directory, run on nightly.Rust: the compiler already ran, and the tools check the rest rust/ch12/fuzz/fuzz_targets/parse.rs
cargo docBuilds the documentation of your crate and every dependency, offline.Rust: the compiler already ran, and the tools check the rest rust/ch12/src/resp.rs

Appendix F: Concept index#

Appendices · Appendix F· 20 min read

Every concept the book claims to teach, per language and per group, with the section that teaches it and the kv file that uses it. The build generates this index from the three files under concepts/ and fails when a concept has no section.

C#

Types and values#

ConceptIn one lineTaught inIn kv
integer types and stdint.hint, long and friends have platform sizes; int32_t and uint64_t say exactly what they are.C: an array of chains, and every pointer accounted forc/ch06/kv.c
char signednessPlain char may be signed or unsigned by platform, so a byte over 127 can become negative.C: an array of chains, and every pointer accounted forc/ch06/kv.c
integer promotion and the usual conversionsSmall types widen to int before arithmetic, so two chars add as ints. When a signed and an unsigned operand meet, the signed one becomes unsigned, and -1 is suddenly the largest value. Compare sizes as size_t and counts as int, and never mix them in one expression.Appendix C
signed overflow is undefinedOverflowing a signed integer is undefined behaviour, so the compiler may assume it never happens.Three answers to the largest integer plus onec/ch05/kv.c
unsigned wrapUnsigned arithmetic wraps around modulo 2^n, by definition, which is how a hash function is allowed to overflow.Three answers to the largest integer plus onec/ch05/kv.c
boolstdbool.h since C99, a keyword in C23. Any non-zero value is true.C17 against C23c/ch14/extras/c23.c
enumNamed integer constants, with no check that a value is one of the names.switch, and the cases that run into each otherc/ch04/kv.c
sizeofThe size of a type or an object in bytes, computed at compile time, with no parentheses needed for an object.What a struct is made of, and an arenac/ch06/generic.c
size_t, ssize_t and ptrdiff_tsize_t is the unsigned type of a size and an index, and sizeof yields it. ssize_t is the signed one that read and write return, so that -1 can mean failure. ptrdiff_t is the type of one pointer minus another.Appendix C
castsAn explicit conversion the compiler will not question, so each one is a claim you must be able to defend.What C does instead of genericsc/ch06/generic.c
const and const-correctnessA promise not to write through a pointer, checked by the compiler and carried through every call.The parser stops copyingc/ch10/src/resp.h
volatileEvery read and write must happen, because something outside the program can change the value.C: a signal, a flag and a pipec/ch10/src/stop.h
static, extern and linkagestatic hides a name inside one file; extern says a name is defined elsewhere.C: two files, one binaryhello/c/greet.c
translation unitsEach .c file compiles alone, with the headers pasted in, and the linker joins the results.C: two files, one binaryhello/c/units.c
storage durationAutomatic variables live until the block ends, static ones for the whole run, allocated ones until free.Where a value lives, and for how longc/ch06/kv.c
stack and heapLocals live on the stack and vanish with the function; malloc gives memory that outlives it.Where a value lives, and for how longc/ch06/kv.c
zero initialisation= {0} sets every member to zero, and a static variable is zero without it.C: an array of chains, and every pointer accounted forc/ch06/kv.c
designated initialisers{.fd = -1, .len = 0} names the members, so the order of the struct does not matter.C again: poll, kqueue and epoll behind one headerc/ch09/src/poller_epoll.c
compound literals(struct pollfd){fd, POLLIN, 0} builds a value in place without a named variable.C: an array of chains, and every pointer accounted forc/ch06/kv.c

Pointers and memory#

ConceptIn one lineTaught inIn kv
pointersAn address plus a type, and the type says how many bytes one step is.C: an array of chains, and every pointer accounted forc/ch06/kv.c
pointer arithmeticp + 1 moves by one element, not one byte, so the type of p decides the step. The result is only defined inside the same object, or one past its end, and the chapter 4 parser stays inside its buffer by comparing against end.Appendix C
arrays decay to pointersAn array name used as a value becomes a pointer to its first element, and the length is gone. That is why a function takes a pointer and a length, and why sizeof an array parameter is the size of a pointer.Appendix C
pointer to pointerHow a function gives its caller a new pointer, as in a grow function that may move a buffer.C: an array of chains, and every pointer accounted forc/ch06/kv.c
void pointerA pointer to anything, which malloc returns and a callback context uses.What C does instead of genericsc/ch06/generic.c
NULL and nullptrThe null pointer. C23 adds nullptr with its own type.C17 against C23c/ch14/extras/c23.c
malloc, calloc, realloc and freeAsk for bytes, get them zeroed, grow them in place or elsewhere, give them back.Growing, measuring and slicing bytesc/ch05/kv.c
ownership conventionsA comment on every pointer says who frees it, because the compiler will not.The parser stops copyingc/ch10/src/resp.h
alignment and offsetofEach type has an alignment, structs get padding to keep it, and offsetof finds a member's position.C: an array of chains, and every pointer accounted forc/ch06/generic.c
flexible array membersA struct whose last member is an array of unknown size, declared as data[], is allocated in one block with room after its header. Redis stores its strings this way, with the length and the bytes in one allocation.Appendix C
restrictA promise that two pointers do not overlap, which lets the compiler optimise.The optimisation levels, measuredc/ch13/bench/bench_copy.c
strict aliasingReading an object through a pointer of a different type is undefined, except through char or unsigned char. The compiler may assume an int and a float never overlap, which is why a memcpy is the way to reinterpret bytes.Appendix C
use after free, double free and leaksThe three memory bugs C lets you write, and what each looks like under AddressSanitizer.C: an array of chains, and every pointer accounted forc/ch06/kv.c
arenasAllocate many small things from one block and free them all at once.What a struct is made of, and an arenac/ch06/generic.c

Structs and unions#

ConceptIn one lineTaught inIn kv
structMembers laid out in order, with padding, passed by value unless you pass a pointer.C: an array of chains, and every pointer accounted forc/ch06/kv.c
typedefA new name for a type, which is how kv_store stands for struct kv_store.C: an array of chains, and every pointer accounted forc/ch06/kv.c
opaque pointersA header declares struct kv_store without its members, so callers cannot touch them.An opaque type, and a struct of functionsc/ch07/src/store.h
unionMembers that share the same bytes, so only one is meaningful at a time.C: a tag and a unionc/ch08/src/store.h
tagged unionA union plus an enum that says which member is live: C's hand-built sum type.C: a tag and a unionc/ch08/src/store.h
bitfieldsMembers with a width in bits, packed by the compiler in an order it chooses.C: a tag and a union

Functions and control#

ConceptIn one lineTaught inIn kv
switch and fallthroughA jump table on an integer, where a case without break runs into the next one.switch, and the cases that run into each otherc/ch04/kv.c
do whileA loop that runs its body once before it tests the condition. The test runner's CHECK macro is built on one with a condition of 0, so that it runs exactly once.Appendix C
goto cleanupOne exit path at the end of a function that frees everything, reached by goto from each failure.C: four calls, and a way to clean upc/ch03/kv.c
function pointersA pointer to code, declared with the signature it points to, which is how C passes behaviour.A table of functionsc/ch05/kv.c
callbacks with a void contextA function pointer plus a void * that the callee passes back, which is C's closure.An opaque type, and a struct of functionsc/ch07/src/cmd.c
variadic functionsFunctions that take any number of arguments, read with va_list, as printf does.A function that takes any number of argumentsc/ch04/kv.c
inlineA hint to paste the function body at each call, with linkage rules that surprise.The optimisation levels, measuredc/ch13/Makefile
static functionsA function only this file can call, which the compiler can warn about when unused.C: headers, static, and a Makefile that knows about bothc/ch07/src/buf.c
_NoreturnMarks a function that never returns, such as one that prints an error and exits.C again: options, and a function that does not come backc/ch14/src/main.c

Preprocessor#

ConceptIn one lineTaught inIn kv
#includePastes a file in place. Angle brackets search the system, quotes search here first.C: headers, static, and a Makefile that knows about bothc/ch07/src/main.c
header guards and #pragma onceThe #ifndef at the top of a header that stops it being pasted twice.C: headers, static, and a Makefile that knows about bothc/ch07/src/buf.h
object and function macrosText replacement before compilation, with or without arguments, with no types.A table of functionsc/ch05/kv.c
do { } while (0)The wrapper that makes a macro of several statements behave like one statement, so that it can follow an if without braces and still take a semicolon. The CHECK macro in chapter 12 is one.Appendix C
X-macrosOne list of entries that expands into a table, an enum and a set of names at once.C: bytes, a length and an ownerc/ch05/kv.c
#ifdef platform guardsChoose code at compile time by platform, as in __linux__ against __APPLE__.C again: poll, kqueue and epoll behind one headerc/ch09/src/poller.h
_Static_assertA check the compiler runs, such as that a struct has the size you think.C: a runner you write, and a sanitizer that watchesc/ch12/tests/test_resp.c
_GenericA compile-time switch on the type of an expression, the nearest C gets to overloading.What C does instead of genericsc/ch06/generic.c
predefined macros__FILE__, __LINE__, __func__ and the platform and compiler names.C again: poll, kqueue and epoll behind one headerc/ch09/src/poller.h
__attribute__ and compiler extensionsNon-standard annotations such as format checking and cleanup, understood by gcc and clang.C again: options, and a function that does not come backc/ch14/src/main.c

Strings and I/O#

ConceptIn one lineTaught inIn kv
NUL-terminated stringsA string is bytes up to the first zero, so a string cannot hold a zero and its length is a walk.C: bytes, a length and an ownerc/ch05/kv.c
strlen, memcpy and memcmpThe three calls behind most string code, and what each assumes about its arguments.C: bytes, a length and an ownerc/ch05/kv.c
the strncpy trapstrncpy fills the whole destination and may leave no terminating zero when the source is as long as the limit. A bounded copy of a C string wants snprintf, and a copy of bytes with a length wants memcpy, which kv uses throughout.Appendix C
snprintfFormats into a buffer of a known size and never overruns it, returning what it wanted to write.Growing, measuring and slicing bytesc/ch05/kv.c
pointer and length disciplineEvery buffer travels with its length, which is how kv handles bytes that may contain zero.C: a buffer, a parser and three answersc/ch04/kv.c
stdio against read and writeFILE * buffers in user space; read and write are the system calls underneath.C: a file descriptor, a write, and a thread for the fsyncc/ch11/src/aof.c
errno, strerror and perrorA failed call returns -1 and sets a thread-local number, which strerror turns into words.errno: the number a failed call leaves behindc/ch03/kv.c

Concurrency and signals#

ConceptIn one lineTaught inIn kv
pthreads and C11 threadsCreate a thread with a function and a void *, join it to wait.C again: the second threadc/ch11/src/syncer.c
mutexpthread_mutex_lock and unlock around every access to shared data, with nothing to stop you forgetting.C again: the second threadc/ch11/src/syncer.c
condition variableSleep until another thread signals, always inside a loop that rechecks the condition.C again: the second threadc/ch11/src/syncer.c
atomics and memory orderstdatomic.h gives operations no other thread can see half done, with an ordering you choose.C again: the second threadc/ch11/src/syncer.c
thread safety of libcMost of libc is safe to call from threads; strtok, localtime and a few others are not.Thread safety of the C library
signals and sigactionThe kernel interrupts the program to run a handler, which may only do a few safe things.C: a signal, a flag and a pipec/ch10/src/stop.c
sig_atomic_t and the self-pipe trickA handler sets a flag or writes one byte to a pipe that the poll loop watches.C: a signal, a flag and a pipec/ch10/src/stop.c
SIGPIPEA write to a peer that has gone raises a signal that kills the process unless ignored.C: a signal, a flag and a pipec/ch10/src/main.c
forkClone the process. The Unix way to start a program, which Windows does not have.C again: the second thread

Build and tooling#

ConceptIn one lineTaught inIn kv
compiler flags and -O levels-O0 for debugging, -O2 for release, and what the optimiser is allowed to assume.The optimisation levels, measuredc/ch13/Makefile
warnings as errors-Wall -Wextra -Werror turns the compiler's advice into a gate.C: from source to binary in four stepshello/c/Makefile
static librariesAn archive of object files that the linker copies from, made with ar.C: a library that is easy to callc/ch15/Makefile
MakeTargets, prerequisites and recipes, with tabs, and a rule for how a .o comes from a .c.Build tools, and the lack of onehello/c/Makefile
CMakeA generator of build files for larger projects, which most C libraries now use.Build tools, and the lack of one
pkg-configAsks an installed library for the flags to compile and link against it.Dependencies in C: the package manager that is not there
AddressSanitizer and UndefinedBehaviorSanitizerCompile-time instrumentation that reports memory errors and undefined behaviour as they happen.C: a runner you write, and a sanitizer that watchesc/ch12/Makefile
ThreadSanitizerThe sanitizer that reports data races, which cannot run together with AddressSanitizer.C again: the second threadc/ch11/src/syncer.c
valgrindRuns the unmodified binary under an emulator and reports leaks and bad accesses, on Linux.C: a runner you write, and a sanitizer that watches
a debugger sessionlldb or gdb: break, step, print, backtrace, and the -g flag that makes them useful.A debugger session
clang-formatThe formatter, driven by a .clang-format file, since C has no single style.Build tools, and the lack of one
C17 against C23nullptr, constexpr, typeof, auto, #embed, attributes and true keywords, and how far clang and gcc support them.C17 against C23c/ch14/extras/c23.c

Conventions#

ConceptIn one lineTaught inIn kv
name prefixeskv_ on every public name, because C has one namespace for the whole program.C: headers, static, and a Makefile that knows about bothc/ch07/src/buf.h
header hygieneA header declares, includes only what it needs, and never defines a variable.C: headers, static, and a Makefile that knows about bothc/ch07/src/buf.h
error returnsReturn a code or a bool and put the result in an out parameter, consistently.C: bytes, a length and an ownerc/ch05/kv.c
ownership commentsWho frees it, written next to the pointer, every time.C: bytes, a length and an ownerc/ch05/kv.c
const parametersA const char * parameter promises the caller the bytes come back unchanged.C: bytes, a length and an ownerc/ch05/kv.c

Go#

Basics#

ConceptIn one lineTaught inIn kv
packages and importsA folder is a package, an import is a path, and an unused import is an error.Go: one command, one binaryhello/go/main.go
zero valuesEvery variable starts as zero, nil or empty, so a struct is usable without a constructor.Every variable starts at zerohello/go/main.go
:= and varInside a function, := declares and assigns in one step and infers the type. var works anywhere and takes an explicit type or a zero value. A := in an inner scope can declare a new variable that shadows one outside, which is the usual way an err goes missing.Appendix D
const and iotaA const is untyped until it is used, so 1 << 40 is fine where an int would overflow. Inside one const block, iota counts from 0 on each line, which is how the aof package names its three policies.Appendix D
basic types and conversionsint is the machine word, every conversion is written out, and there is no implicit widening.Three answers to the largest integer plus onego/ch05/strings_test.go
runes, strings, bytes and UTF-8A string is immutable bytes, range gives runes, and len counts bytes.Bytes, runes and a buildergo/ch05/strings_test.go
arrays against slicesAn array has a fixed length in its type; a slice is a view of one with a length and a capacity.Go: the same table, with nothing to freego/ch06/table.go
len, cap, append and aliasingappend writes into spare capacity, so two slices can share and overwrite the same array.Bytes, runes and a buildergo/ch05/exec.go
copy and the full slice expressioncopy duplicates bytes, and s[a:b:c] caps the capacity to stop append from aliasing.The parser stops copyinggo/ch10/internal/resp/resp.go
mapsReading a nil map is fine and writing one panics; iteration order is random on purpose.Go: two kinds of string, and a copy between themgo/ch05/store.go
comma-okv, ok := m[k] separates a missing key from a zero value, and the same form works for assertions and channels.Go: two kinds of string, and a copy between themgo/ch05/store.go
structs and composite literalsFields in order or by name, and a pointer to a literal with &.Go: the same table, with nothing to freego/ch06/table.go
pointers against valuesEverything is passed by value, so a function that must change the caller's struct takes a pointer. A method on a pointer receiver can change the value and a method on a value receiver gets a copy. The store's methods all take *Store.Appendix D
the forms of rangeOver a slice, a map, a string, a channel, an integer since 1.22, and a function since 1.23.A function that range can callgo/ch08/internal/store/value.go
switch and type switchNo fallthrough by default, cases can be expressions, and switch v := x.(type) branches on a type.Go: any, and a switch on the typego/ch08/internal/store/value.go
labeled break and continueThe way out of a nested loop, with a label on the outer one.A function that range can call

Functions and methods#

ConceptIn one lineTaught inIn kv
multiple returns and named resultsA function returns a value and an error together, and named results document them.Go: any, and a switch on the typego/ch08/internal/cmd/cmd.go
variadic functionsf(args ...string) takes any number, and f(slice...) spreads one.Bytes, runes and a buildergo/ch05/exec.go
closures and captureA function literal captures variables by reference, and since 1.22 each loop iteration has its own.Go: two kinds of string, and a copy between themgo/ch05/exec.go
deferRuns at function exit, last in first out, with its arguments evaluated at the defer line.Two rules of defergo/ch03/defer_test.go
panic and recoverA panic unwinds the stack; a deferred recover stops it, which is how one bad client does not kill a server.panic and recovergo/ch10/cmd/kv/main.go
methods and receiversA function with a receiver, on a pointer when it changes the value, on a value when it does not.Go: a folder is a package, and a capital letter is an exportgo/ch07/internal/store/store.go
method setsA value of type T has the methods with value receivers, and a *T has those and the pointer-receiver ones too. That decides which of the two satisfies an interface: a *Store satisfies Storer, and a Store does not.Appendix D
method valuess.Get without the call is a function bound to s, which a dispatch table can hold.A map of functionsgo/ch05/exec.go
embedding and promotionAn unnamed field lends its methods to the outer struct, as an embedded sync.Mutex does.Embedding, and the nil that is not nilgo/ch07/internal/cmd/cmd_test.go

Interfaces#

ConceptIn one lineTaught inIn kv
interfaces and implicit satisfactionA set of method signatures, and any type with those methods satisfies it without saying so.An interface is a set of methodsgo/ch07/internal/cmd/cmd.go
small interfacesio.Reader, io.Writer, fmt.Stringer and error have one method each, and most code accepts them.An interface is a set of methodsgo/ch07/internal/cmd/cmd.go
anyThe empty interface, which every value satisfies, and which a type switch takes apart.Go: any, and a switch on the typego/ch08/internal/store/value.go
type assertionv, ok := x.(T) asks whether the dynamic value is a T, and without ok it panics.Go: any, and a switch on the typego/ch08/internal/cmd/cmd.go
the nil interface trapAn interface holding a nil pointer is not nil, which makes a returned *MyErr look like an error.Embedding, and the nil that is not nilgo/ch07/internal/cmd/cmd_test.go
interface embeddingAn interface may list other interfaces, and gets their methods. io.ReadWriter is io.Reader and io.Writer in one declaration, and a type that has both methods satisfies all three.Appendix D
when not to use an interfaceAccept interfaces, return structs, and define one only when a second implementation exists.Embedding, and the nil that is not nilgo/ch07/internal/cmd/cmd.go

Generics#

ConceptIn one lineTaught inIn kv
type parametersfunc Max[T cmp.Ordered](a, b T) T, since Go 1.18, with the type inferred at the call.Go: the same table, with nothing to freego/ch06/table.go
constraints, comparable and anyAn interface used as a bound, comparable for map keys, and a union of types with |.Go: the same table, with nothing to freego/ch06/table.go
the slices, maps and cmp packagesSort, search, clone and compare, written once with generics instead of per type.Go: the same table, with nothing to freego/ch06/table_test.go
when to use genericsA container or an algorithm over many types, not a method that takes an interface.Go: the same table, with nothing to freego/ch06/table.go

Errors#

ConceptIn one lineTaught inIn kv
the error interfaceAnything with an Error() string method, and nil means no error.An error is any value with an Error methodgo/ch04/resp.go
custom error typesA struct with fields the caller can read, returned as an error.An error is any value with an Error methodgo/ch04/resp.go
wrapping with %wfmt.Errorf("read config: %w", err) adds context and keeps the cause reachable.An error is any value with an Error methodgo/ch04/resp.go
errors.Is, errors.As and errors.JoinWalk the wrapped chain for a sentinel, for a type, or combine several errors.errors.Joingo/ch12/internal/aof/aof.go
sentinel against typed errorsvar ErrNotFound = errors.New(...) for a condition, a type when the caller needs fields.An error is any value with an Error methodgo/ch04/resp.go
panics against errorsAn error is a value for the caller; a panic is a bug in the program.panic and recover

Concurrency#

ConceptIn one lineTaught inIn kv
goroutinesgo f() runs f on a stack of a few kilobytes, multiplexed over the real threads.Go: one word, and one lockgo/ch09/cmd/kv/main.go
channelsA typed queue between goroutines; unbuffered ones hand off, buffered ones hold a few.Go again: fifty workers and three channelsgo/ch09/cmd/kvload/main.go
close and range over a channelThe sender closes, the receiver's range ends, and a receive from a closed channel gives zero.Go again: fifty workers and three channelsgo/ch09/cmd/kvload/main.go
nil channelsA nil channel blocks forever, which is a way to switch a select case off.Go again: fifty workers and three channelsgo/ch09/cmd/kvload/main.go
selectWait on several channels at once, with default for a non-blocking try and a timer for a timeout.Go again: fifty workers and three channelsgo/ch09/cmd/kvload/main.go
sync.Mutex and RWMutexLock around shared data, with RLock for readers, and nothing stops you forgetting.Go: one word, and one lockgo/ch09/internal/store/store.go
sync.WaitGroupAdd before starting, Done when finished, Wait for all, and wg.Go since 1.25.Go: one word, and one lockgo/ch09/internal/store/store_test.go
sync.Once and sync.PoolRun something exactly once, and reuse buffers instead of allocating.The rest of the sync package
sync/atomicCounters and flags that need no lock, as atomic.Int64.The rest of the sync package
contextCancellation, a deadline and request values that flow down every call.context in one paragraphgo/ch10/cmd/kv/main.go
worker poolsA fixed number of goroutines reading jobs from one channel.Go again: fifty workers and three channelsgo/ch09/cmd/kvload/main.go
data races and the race detectorTwo goroutines, one writing, no sync, and go test -race finds it.Go: one word, and one lockgo/ch09/internal/store/store_test.go
GOMAXPROCS and the schedulerGoroutines run on P logical processors over M threads, and a blocking system call hands its P over.The runtime in one paragraph
goroutine leaksA goroutine blocked forever on a channel nobody will touch again.Go again: fifty workers and three channelsgo/ch09/cmd/kvload/main.go

Runtime and memory#

ConceptIn one lineTaught inIn kv
the garbage collectorConcurrent mark and sweep, with short pauses and memory held until the next cycle.Go: the same table, with nothing to freego/ch06/table.go
escape analysisgo build -gcflags=-m says which values move to the heap, and why.Escape analysisgo/ch13/internal/resp/resp.go
GOGC and GOMEMLIMITHow often the collector runs, and a ceiling on the heap.GOGC, GOMEMLIMIT and the Pool
pprofCPU and memory profiles from a running program, read as a graph or a flamegraph.The profiler is a packagego/ch13/cmd/kv/main.go

Tooling and packaging#

ConceptIn one lineTaught inIn kv
go.mod and go.sumThe module path and its dependencies, and the hashes that pin them.Go: one command, one binaryhello/go/go.mod
semantic import versioningA v2 module has /v2 in its import path, so two majors can coexist.Go: a folder is a package, and a capital letter is an export
go get and replaceAdd or upgrade a dependency, or point a module at a local folder.Go: a folder is a package, and a capital letter is an export
workspacesgo.work lets several modules build together while you change them.Rust: a module is a file you must announce
internal packagesA package under internal/ can only be imported from inside its parent.Go: a folder is a package, and a capital letter is an exportgo/ch07/internal/store/store.go
go vet, gofmt and staticcheckThe checker, the one formatter, and the linter most teams add.Go: one command, one binary
build tags and platform files//go:build linux, or a _linux.go suffix, picks a file by platform.Go's platform files: a line at the topgo/ch09/internal/sys/limit_unix.go
go generate and go:embedRun a tool from a comment, and compile a file into the binary.Go again: flags, a file, a log and a versiongo/ch14/cmd/kv/main.go
table-driven testsA slice of cases and one loop, the idiom every Go test file uses.Go: a folder is a package, and a capital letter is an exportgo/ch07/internal/store/glob_test.go
subtests, t.Parallel and helperst.Run names a case, t.Parallel runs them together, t.Helper hides a helper from the report.Go: the runner is in the box, and so is the fuzzergo/ch12/internal/resp/resp_test.go
benchmarksfunc BenchmarkX(b *testing.B) with b.Loop, and go test -bench.Go: nine allocations, and the flag that names themgo/ch13/internal/resp/bench_test.go
fuzzingfunc FuzzX(f *testing.F) with go test -fuzz, built into the toolchain.Go: the runner is in the box, and so is the fuzzergo/ch12/internal/resp/resp_test.go
cross-compilingGOOS and GOARCH, with CGO_ENABLED=0 for a static binary.Go: two variablesgo/ch14/Makefile
cgoimport "C" and a preamble, with a cost on every call.Go: cgogo/ch15/internal/cmatch/cmatch.go
the module proxy and vendoringproxy.golang.org serves modules, and go mod vendor copies them into the repository.Go: a folder is a package, and a capital letter is an export
go docDoc comments on exported names, read with go doc and on pkg.go.dev.The documentation is in the source

Standard library tour#

ConceptIn one lineTaught inIn kv
fmtPrintf and its verbs: %v for anything, %q for a quoted string, %d and %s as in C, and %w to wrap an error in Errorf. Println adds spaces and a newline, and a type with a String method prints as it chooses.Appendix D
strconvAtoi, ParseInt and Itoa, with errors, not panics.Go: a reader that waits, and an error that names itselfgo/ch04/resp.go
strings and bytesThe same functions twice, for string and for []byte, plus strings.Builder.Bytes, runes and a buildergo/ch05/strings_test.go
bufioA buffered reader and writer over any io.Reader or io.Writer.Go: a reader that waits, and an error that names itselfgo/ch04/resp.go
io and osThe interfaces, and files, environment and arguments.Go: the loop you did not have to writego/ch03/main.go
netnet.Listen, Accept and net.Conn, which is an io.ReadWriteCloser.Go: the loop you did not have to writego/ch03/main.go
timetime.Time and Duration, the monotonic reading hidden inside Now, Timer and Ticker.Go: a ticker, and a clock the test can hold stillgo/ch10/internal/store/store.go
encoding/jsonStruct tags and Marshal, the most used package after fmt.Go again: flags, a file, a log and a versiongo/ch14/cmd/kv/main.go
flagCommand-line flags with defaults and a generated usage.Go again: flags, a file, a log and a versiongo/ch14/cmd/kv/main.go
log/slogStructured logging with levels and key-value pairs.Go again: flags, a file, a log and a versiongo/ch14/cmd/kv/main.go
os/signalsignal.Notify delivers a signal on a channel, and NotifyContext cancels a context.Go: a context that a signal cancelsgo/ch10/cmd/kv/main.go

Rust#

Basics#

ConceptIn one lineTaught inIn kv
let, mut and shadowingA binding is immutable unless it says mut, which the compiler enforces. A second let with the same name shadows the first, and may change its type, which is how a parsed value replaces the string it came from.Appendix E
integer overflow rulesDebug builds panic on overflow, release builds wrap, and checked_add and wrapping_add say what you mean.Three answers to the largest integer plus onerust/ch05/src/store.rs
as and From castsas converts between numbers and may truncate or wrap, so n as u8 of 300 is 44. From and Into convert where the conversion cannot fail, and the ? operator uses From to turn one error type into another.Appendix E
tuples, arrays and slicesA tuple groups a fixed set of values of mixed types, and a pattern takes it apart. An array [T; N] has a length in its type. A slice &[T] is a borrowed view of a run of elements, with its length beside the pointer, and it is what every function in kv takes.Appendix E
VecThe growable array that owns its elements and frees them when it drops.Rust: a value has one ownerrust/ch05/src/store.rs
String, str, char and bytesString owns valid UTF-8, &str borrows it, char is a code point, and as_bytes gives the bytes.str, String, char and a slice of bytesrust/ch05/src/store.rs
structs and impl blocksData in a struct, behaviour in impl, and a value is built with a literal or a new function.Rust: the table that drops itselfrust/ch06/src/table.rs
methods and the forms of self&self reads, &mut self changes, self consumes.Rust: the table that drops itselfrust/ch06/src/table.rs
enums with dataEach variant can carry its own fields, which is a sum type the compiler checks.One enum for everything that can go wrongrust/ch04/src/resp.rs
Option and ResultSome or None for a value that may be absent, Ok or Err for one that may fail.Rust: an Option for the end, a Result for the restrust/ch04/src/resp.rs
match and patternsEvery case must be handled, with guards, bindings, slice and struct patterns, and @.Rust: an Option for the end, a Result for the restrust/ch04/src/main.rs
if let, let else and while letMatch one pattern without the full match, or leave early when it does not match.Rust: an Option for the end, a Result for the restrust/ch04/src/resp.rs
loops and labelsloop runs until break, and break may carry a value out of it. while and for are the usual ones, and for takes anything that is an iterator. A label such as 'outer: lets a break or continue name the loop it means.Appendix E
expressions against statementsA block is an expression, and its value is the last line without a semicolon. So if, match and a plain block all produce values, and a function returns the value of its final expression. A semicolon turns an expression into a statement that yields nothing.Appendix E
const and staticA const is inlined, a static has one address, and a mutable static needs unsafe.A table of fn pointers, built by a macrorust/ch05/src/cmd.rs
modules and visibilitymod, pub, pub(crate), use and paths, with files as modules.Rust: a module is a file you must announcerust/ch07/src/store.rs
CargoCrates, features, workspaces, profiles, build.rs and the lockfile, with semantic versioning.Rust: cargo owns the projecthello/rust/Cargo.toml
doc comments and doc tests/// becomes documentation, and a code block in it is compiled and run by cargo test.Rust: the compiler already ran, and the tools check the restrust/ch12/src/resp.rs
rustfmt and clippyThe one formatter and the linter with hundreds of checks, both standard.Rust: cargo owns the project
editionsA crate picks 2015, 2018, 2021 or 2024, and crates of different editions link together.Rust: cargo owns the projecthello/rust/Cargo.toml

Ownership#

ConceptIn one lineTaught inIn kv
moveAssigning or passing a value gives it away, and the old name cannot be used again.Rust: a value has one ownerrust/ch05/src/cmd.rs
Copy against CloneSmall plain values copy on assignment; everything else moves unless you call clone.Lifetimes, in the one place kv needs them
borrows& lends a value for reading, &mut for changing, and the owner waits until the borrow ends.Rust: a value has one ownerrust/ch05/src/store.rs
mutable or shared, never bothAny number of readers or one writer at a time, which is the rule that prevents data races.Rust: a value has one owner
non-lexical lifetimesA borrow ends at its last use, not at the end of the block.Lifetimes, in the one place kv needs themrust/ch10/src/main.rs
lifetime annotations and elision'a names how long a reference lives, and the compiler fills in the common cases.Lifetimes, in the one place kv needs themrust/ch10/src/resp.rs
'staticLives for the whole program, as a string literal does, or owns everything it holds.Lifetimes, in the one place kv needs them
structs that hold referencesCommand<'a> borrows from a buffer and cannot outlive it.The parser stops copyingrust/ch10/src/resp.rs
Drop and RAIIA value is cleaned up when its owner goes out of scope, in reverse order of declaration.Rust: the table that drops itselfrust/ch06/src/table.rs
mem::take, replace and swapMove a value out of a place you only have &mut to, leaving a default behind.Rust again: a scoped thread on a channelrust/ch11/src/aof.rs
interior mutabilityCell, RefCell and Mutex let you change a value through a shared reference, checked at runtime.Two owners of one tablerust/ch06/src/table.rs
Rc and ArcShared ownership by reference count, Arc when threads are involved.Two owners of one tablerust/ch06/src/table.rs
BoxOne owner of a value on the heap, which recursive types and trait objects need.Closures and iterators, named at lastrust/ch08/src/chain.rs
DerefA type that implements Deref is treated as the type it dereferences to where a reference is wanted. That is why a &String is accepted as a &str and a &Vec<T> as a &[T], and why a Box<T> can be used as a T.Appendix E
borrowing from a VecA slice of a Vec blocks every push until the slice is dropped.Lifetimes, in the one place kv needs themrust/ch10/src/main.rs
CowBorrowed or owned, decided at runtime, for a value that usually needs no copy.Lifetimes, in the one place kv needs themrust/ch10/src/cmd.rs

Traits and generics#

ConceptIn one lineTaught inIn kv
defining and implementing a traitA set of methods a type promises, implemented in an impl Trait for Type block.A trait is a promise of methodsrust/ch07/src/store.rs
default methodsA trait method with a body that implementations may keep or replace.A trait is a promise of methodsrust/ch07/src/store.rs
derive#[derive(Debug, Clone, PartialEq)] asks the compiler to write the usual implementations of those traits from the fields. The book derives Debug on every error type, so that a test can print one, and Default on the stores.Appendix E
the standard traitsDebug and Display for printing, Clone and Copy for copying, Default for an empty value, PartialEq, Eq, Hash and Ord for comparing and hashing, From and Into for converting, AsRef for borrowing in any form, Iterator for walking, Read and Write for bytes, and Error for errors. Most are derived, and the rest are an impl block each.Appendix E
generics and boundsfn largest<T: Ord>(xs: &[T]) works for any T that can be ordered, and where clauses spell long bounds out.Rust: the table that drops itselfrust/ch06/src/table.rs
impl Trait in argument and return positionfn f(w: impl Write) is a generic in disguise, and -> impl Iterator hides a type.Static and dynamic dispatchrust/ch07/src/store.rs
trait objects and dynBox<dyn Store> picks the implementation at runtime through a vtable.Static and dynamic dispatchrust/ch07/src/main.rs
object safetyA trait with generic methods or Self returns cannot be a dyn.Static and dynamic dispatchrust/ch07/src/store.rs
static against dynamic dispatchGenerics copy the code per type and inline; dyn calls through a pointer and keeps one copy.Static and dynamic dispatchrust/ch07/src/store.rs
monomorphisationEach use of a generic with a new type compiles to its own code, which is fast and slow to build.Rust: the table that drops itselfrust/ch06/src/table.rs
the orphan ruleYou can implement a trait for a type only if you own one of them.A trait is a promise of methodsrust/ch07/src/table.rs
operator overloadingAn operator is a trait: + is Add, [] is Index, == is PartialEq. A type that implements the trait gets the operator, and nothing else is allowed to redefine one.Appendix E
associated types and constantsIterator's Item is a type chosen by the implementation, not by the caller.Closures and iterators, named at lastrust/ch08/src/chain.rs

Closures and iterators#

ConceptIn one lineTaught inIn kv
closures|x| x + 1 captures its surroundings by reference, by mutable reference, or by move.Closures and iterators, named at lastrust/ch08/src/cmd.rs
Fn, FnMut and FnOnceWhat a closure may do with what it captured, which decides where it can be passed.Closures and iterators, named at lastrust/ch08/src/cmd.rs
iterators and adaptorsmap, filter, enumerate, zip, take and the rest, each a lazy wrapper around the last.Closures and iterators, named at lastrust/ch08/src/store.rs
collect and the turbofishcollect::<Vec<_>>() builds any collection, and the type says which.Closures and iterators, named at lastrust/ch08/src/cmd.rs
implementing IteratorOne method, next, and every adaptor works on your type.Closures and iterators, named at lastrust/ch08/src/chain.rs
iter, iter_mut and into_iterBorrow the elements, borrow them mutably, or take them.Closures and iterators, named at lastrust/ch08/src/chain.rs
lazinessNothing runs until something consumes the chain, so an unused map is a warning.Closures and iterators, named at lastrust/ch08/src/store.rs

Errors#

ConceptIn one lineTaught inIn kv
custom error enumsOne enum with a variant per failure, matched exhaustively by the caller.One enum for everything that can go wrongrust/ch04/src/resp.rs
Display and ErrorImplement Display for the message and Error to join the ecosystem.One enum for everything that can go wrongrust/ch04/src/resp.rs
From and ?? converts the error with From, so an io::Error becomes your KvError without a line of code.One enum for everything that can go wrongrust/ch04/src/resp.rs
Box<dyn Error>A boxed trait object that can hold any error type at all. A main or a tool that only reports errors returns Result<(), Box<dyn Error>> and uses ? on every error it meets, at the cost of no longer being able to match on the kind.Appendix E
thiserror and anyhowThe two crates most projects use for errors. thiserror derives Display and From for an error enum like the chapter 4 KvError, so that the impl blocks are one attribute each. anyhow is the Box<dyn Error> idea with context, for programs that report errors and never match on them.Appendix E
panics and the unwrap policyunwrap and expect are for bugs that cannot happen, with expect saying why.?, expect and unwraprust/ch03/src/main.rs
Result from mainfn main() -> Result<(), Box<dyn Error>> prints the error and exits non-zero.A main that returns a Resultrust/ch03/src/main.rs

Concurrency#

ConceptIn one lineTaught inIn kv
threadsthread::spawn takes a closure that must be 'static, and join waits.Rust: a thread for each client, and a compiler that checks the sharingrust/ch09/src/main.rs
scoped threadsthread::scope lets threads borrow from the stack, because all are joined before it returns.Rust again: the load generator with scoped threadsrust/ch09/src/bin/kvload.rs
Send and SyncMarker traits the compiler checks: safe to move to a thread, safe to share between threads.Send and Sync
Arc<Mutex<T>>Shared ownership plus one writer at a time, the shape most shared state takes.Rust: a thread for each client, and a compiler that checks the sharingrust/ch09/src/main.rs
RwLockMany readers or one writer, when reads dominate.Rust: one allocation, a profile, four experiments, and a lockrust/ch13/src/main.rs
mpsc channelsMany senders, one receiver, and the receiver's loop ends when every sender drops.Rust again: the load generator with scoped threadsrust/ch09/src/bin/kvload.rs
atomicsAtomicBool and AtomicUsize with an Ordering, for flags and counters.Atomicsrust/ch09/src/bin/kvload.rs
async and awaitAn async fn returns a Future, and await pauses it instead of blocking the thread.Rust once more: tasks on tokiorust/ch09/src/bin/kv_async.rs
what a Future isA state machine with a poll method that a runtime drives.What a future is
PinA future that borrows itself cannot move, and Pin is the promise that it will not.Pin in one paragraph
tokiospawn, tasks, the multi-threaded runtime, and its own Mutex and channels.Rust once more: tasks on tokiorust/ch09/src/bin/kv_async.rs
select!Wait on several futures and take the first, with cancellation of the rest.select!, cancellation and blocking
cancellationDropping a future stops it, which is both convenient and a trap for half-done work.select!, cancellation and blocking
blocking inside asyncA std Mutex held across await, or a long computation, stalls the whole runtime.select!, cancellation and blocking

Unsafe and FFI#

ConceptIn one lineTaught inIn kv
raw pointers*const T and *mut T, with no borrow checking, dereferenced only in unsafe.The five powers, and the pointer they are forrust/ch15/src/cmatch.rs
unsafe blocks and functionsThe keyword that marks where the compiler's rules stop and yours start.Rust: extern and unsaferust/ch15/src/cmatch.rs
the five powersDereference a raw pointer, call unsafe code, touch a mutable static, implement an unsafe trait, read a union.The five powers, and the pointer they are for
repr(C)Lay a struct out as C would, so both sides agree on the bytes.The five powers, and the pointer they are forrust/ch15/src/cmatch.rs
extern blocksDeclare C functions, unsafe since the 2024 edition, and export your own with no_mangle.Rust: extern and unsaferust/ch15/src/cmatch.rs
miriThe interpreter that runs tests and reports undefined behaviour in unsafe code.Rust: the compiler already ran, and the tools check the rest
bindgenGenerates the extern declarations from a C header.The five powers, and the pointer they are for

Macros#

ConceptIn one lineTaught inIn kv
macro_rules!Pattern-based macros that write code from a template, for tables and repetition.A table of fn pointers, built by a macrorust/ch05/src/cmd.rs
derive and attribute macros#[derive(...)] and #[tokio::main] are procedural macros, which run Rust code at compile time to generate code. They come from crates, and the book uses them without writing one. A macro_rules! macro, like the command table in chapter 5, is the simpler kind.Appendix E
println!, format!, vec! and assert!The standard macros. println! and format! take a format string checked at compile time with a variable number of arguments, which a function cannot. vec! builds a Vec from a list, and assert! and assert_eq! are the test checks that print both sides when they fail.Appendix E

Testing and tooling#

ConceptIn one lineTaught inIn kv
unit, integration and doc tests#[cfg(test)] modules in the file, the tests/ directory, and examples in doc comments.Rust: the compiler already ran, and the tools check the restrust/ch12/tests/roundtrip.rs
cargo test flagsFilter by name, show output, run one thread, and ignore the slow ones.cargo test, and what comes after the dashes
benchmarks with criterionStable Rust has no built-in bench, and criterion is the crate that fills the gap.Rust: one allocation, a profile, four experiments, and a lockrust/ch13/examples/bench_parse.rs
cargo-fuzzlibFuzzer targets in a fuzz/ directory, run on nightly.Rust: the compiler already ran, and the tools check the restrust/ch12/fuzz/fuzz_targets/parse.rs
cargo docBuilds the documentation of your crate and every dependency, offline.Rust: the compiler already ran, and the tools check the restrust/ch12/src/resp.rs

End of the book