GitLab CI/CD, Decoded#

Read, trace and fix any enterprise pipeline

Compiled bykodebot

GitLab 19.3 GitLab.com and self-managed Worked examples on AWS and Azure

Preface#

Your team's pipeline file is twenty lines long. The pipeline it produces is not, and keeping it working is now your job.

In a large organisation, a central team writes shared CI/CD templates, and each team writes its own pipeline file on top of them. Your file includes a template, which includes eight more. A job inherits from a job that inherits from another, in a repository you have never opened. Variables arrive from settings pages nobody on your team can see. When something breaks, the answer is often not in the file in front of you. And when your team needs a new job, a new rule or a deploy, you are the one who writes it.

GitLab's documentation explains each feature on its own. This book teaches you to do two things with a real pipeline. You learn to write and change your team's own file. You also learn to read everything the central team adds to it: to take any job and find where it came from, why it ran, what it was given and what it did. Everything is taught on MegaCorp, a fictional company whose GitLab is as tangled as the real ones, and on small practice projects that you build on your own GitLab.

Who it is for#

You are a developer who looks after your team's pipeline. Some of it comes from a central team's templates, and the rest your team writes. The book assumes that you know what CI/CD is, can use a terminal, and can create a project on a GitLab: GitLab.com's free tier, or your organisation's own. It assumes nothing more:

  • You don't need to have written a .gitlab-ci.yml, or any YAML.
  • You don't need to have used GitLab.
  • You don't need to have run a container.

Every term is explained where it first appears. The short chapter Before you start gives you the Git and GitLab words, enough YAML to read any pipeline file, and a first pipeline, in about ten minutes. Your first pipeline then has you run it on your own GitLab. Wherever an idea has parts, an order or a decision in it, a diagram shows it.

What reading it gets you#

LevelYou canWhat gets you there
0 · Zerolittle: the YAML looks like noise
1 · Vocabularysay what every keyword doesPart II and Part IV
2 · Recognitionopen any pipeline and name the patterns in itthe rest of the book, read once
3 · Tracingtake any job back to where it was defined, why it ran and what it was giventhe exercise at the end of each chapter from Part IV to Part X
4 · Diagnosisgo from a symptom to its cause in minutesthe worksheet on your own pipeline, then a few real incidents with the symptom cards open
5 · Writing and maintainingwrite and change your team's pipeline, and know what each change will dothe labs in Part III, and the "When you write it" boxes
6 · Template-library designdesign templates for a whole organisationnot this book's goal

Level 5 is the goal. It rests on level 4, because you can't safely change a pipeline that you can't diagnose. Reading gets you to level 2, and the exercises to level 3. The worksheet, which you fill in privately against your own organisation's pipeline, and the symptom cards take you to level 4. The labs take you to level 5. The final test tells you when you're there.

How to read it#

There are three paths, and the contents shows what each one costs on a clock set for a technical reader: prose at 150 words a minute, code at 80, a minute for each diagram.

PathWhat it isTime
The HandbookPart I: every keyword on one annotated project, every word explained in one line, and the cardabout 58 min of reading
The Pipeline textbookfrom Part II to Part XI, in order, with its labs and exercises7 h 59 min of reading, and 2 h 13 min of doing
After the pipelinePart X: from the image in the registry to software running in a cluster2 h 27 min of reading, and 1 h 38 min of doing

Read the Handbook first, whoever you are. If you have a broken pipeline in front of you, its last chapter, the card, sends you to the right symptom card. If your team's pipeline has just become yours, go on to Before you start and Part III, doing each lab. Then read the rest in order.

There is a third way in: press ⌘K and type what you see. A symptom ("blocked"), a keyword (!reference), a variable or an error message takes you to the answer. Deep dives, legacy asides, exercises and labs fold and unfold one at a time; printing opens them all.

Conventions#

  • PremiumUltimate The lowest GitLab tier that has the feature. No tier badge means every tier has it.
  • Self-managedGitLab.com The feature exists only on that offering.
  • ExperimentalBetaDeprecated GitLab's own status for the feature.
  • Since 17.11 The release that introduced it. If your organisation runs its own GitLab, check which version it is on.

Examples come in a few shapes. A You wrote pane shows configuration as it sits in its file. A GitLab builds pane beside it shows what GitLab made of it, marked captured from GitLab when it is GitLab's own output. A symptom card goes from what you see to the cause, how to confirm it and the fix. A spot the bug exercise ends every chapter from Part IV to Part X. A lab is something to do on your own GitLab, with its steps and what you should see. A When you write it box says what to write, why, and what to watch out for, when you write that mechanism in your team's file.

What it targets#

ScopeBaselineNote
GitLabGitLab 19.3features newer than this are left out; recent ones carry a since badge
OfferingsGitLab.com and self-manageddifferences are called out where they exist
LabsGitLab.com's Free tier, or your own GitLabno lab needs a paid feature or a cloud account
CloudWorked examples on AWS and Azurethe ideas transfer; the commands are AWS's and Azure's, side by side
Delivery toolsKubernetes 1.36, Helm 4.3, Kustomize 5.8, Flux 2.9, Argo CD 3.5, OpenTofu 1.12 and External Secrets Operator 2.10MegaCorp's charts, overlays, manifests and modules were rendered and validated with these versions

On accuracy#

GitLab releases every month, and search results do not sort by version. Every version-sensitive claim here was checked against GitLab's documentation. Every "GitLab builds" pane marked as captured is real output from a GitLab 19.3 instance, loaded with MegaCorp's projects exactly as this book shows them. Every lab was run end to end on that instance, and what each lab says you should see is what it printed. Where a behaviour needs a tier that instance doesn't have, the book says described, not captured.

It will still age. When this book and GitLab's documentation disagree, the documentation is right.

What it does not cover#

Not coveredWhy
Designing an organisation's template libraryyou learn why central teams build what they build, and how to use it in your own file, not how to design one
Running GitLabinstance settings, upgrades and runner fleets belong to administrators; you see what their choices do to your jobs
Triaging security findingsyou learn what each scanner does and why it blocks you, not whether a vulnerability matters
Other CI systemsthere are no comparisons with GitHub Actions, Jenkins or others
Other cloudsthe worked examples use AWS and Azure; Google Cloud and the rest are left out, and the ideas carry over
Operating clusters and controllersyou read and change what your service needs: values, overlays and manifests; installing and upgrading Kubernetes, Flux or Argo CD belongs to the platform team
Choosing an observability vendorproducts are named where they fit, not compared
GitLab beyond pipelinesPages, wikis, planning and the AI features appear only where a pipeline touches them

Before you start#

This book assumes you know what CI/CD is: code gets built, tested and shipped automatically whenever it changes. It assumes nothing else. This short chapter gives you the three things the rest of the book leans on:

  • the Git and GitLab words
  • enough YAML to read any pipeline file
  • a first pipeline, from a push to a result

If you already know all three, skip to chapter 4.

Git and GitLab words#

Git keeps the history of a project's files as a chain of saved changes. GitLab hosts that history and runs pipelines on it. Every word below appears on almost every page of this book:

A project's history. Commits on main, a feature branch with its merge request, the merge back into main, and a tag on a release commit. merge request: asks to merge the branch into main main main: the default branch branch feature/login a commit the merge v1.4.0 a tag
Figure A project's history. Commits on main, a feature branch with its merge request, the merge back into main, and a tag on a release commit.#
WordWhat it isWhy a pipeline cares
commitone saved change to the files, with an ID like 4f2c9a1, called its SHAevery pipeline runs for exactly one commit
brancha named line of commits, such as feature/loginpushing to a branch starts a branch pipeline
default branchthe main line of the project, usually called maindeployments usually start from it
pushsending your new commits from your computer to GitLabthe most common way a pipeline starts
merge requesta request to merge one branch into another, with review and discussion; other tools call it a pull requestGitLab can run a pipeline for it, called a merge request pipeline
mergecombining a branch's commits into another branchmerging to main usually starts the pipeline that deploys
taga permanent name for one commit, usually a release number such as v1.4.0pushing a tag starts a tag pipeline, often the one that releases
protected branch or tagone that only chosen people may push tosecrets and deploy permissions are often limited to protected branches
projectGitLab's home for one repository, with its settings, pipelines and merge requestsa pipeline belongs to a project
groupa folder of projects, which can contain further groupssettings made on a group, such as CI/CD variables, reach every project inside it

YAML in five minutes#

A pipeline is described in a file called .gitlab-ci.yml, written in YAML. YAML is a way of writing data as indented text. Almost everything you will see is one of these few shapes:

# A comment: YAML ignores everything after #

name: payments-api        # a key and its value
retries: 2                # a number
enabled: true             # true or false

owner:                    # a map: the keys indented under "owner" belong to it
  team: payments
  email: payments@example.com

stages:                   # a list: one item per line, each starting with "-"
  - build
  - test

tags: [shared, linux]     # the same kind of list, written on one line

script: |                 # a block of text, kept line by line
  echo "first line"
  echo "second line"

Four rules explain nearly every YAML mistake you will meet:

  • Indentation is meaning. Keys indented under another key belong to it. Use spaces, never tabs; two spaces per level is the convention.
  • A dash starts a list item. - build is one item in a list. A key followed by a colon, such as stage:, starts a key in a map.
  • Numbers and true/false are not text. 21 is read as a number and true as a yes-or-no value. When you mean text, put it in quotes: JAVA_VERSION: "21". GitLab's variables are always text, which is why you will see so many quotes in pipeline files.
  • Some characters change the meaning. A value that starts with *, &, !, [ or {, or contains a colon followed by a space, means something special to YAML. Put such values in quotes when you mean plain text. Chapter 13 explains what the special ones do.

Numbers can even change as they are read. On the local GitLab 19.3, a variable written VERSION: 1.10 without quotes reached a rule as 1.1, so $VERSION == "1.10" never matched. Written as "1.10", it did.

In a .gitlab-ci.yml, every top-level key is one of two things. It is either a GitLab keyword that sets something for the whole pipeline, such as stages, or the name of a job. Here is the shape of the pipeline file you are about to read:

The shape of a small pipeline file. The file is a map. Each top-level key is either a keyword or a job. A job is a map of its own settings, and its script is a list of commands. the file its top-level keys what each key holds .gitlab-ci.yml stages: a keyword build-app: a job unit-tests: a job a list: build, then test stage: build image: node:22 script: a list of commands the same three keys, for tests
Figure The shape of a small pipeline file. The file is a map. Each top-level key is either a keyword or a job. A job is a map of its own settings, and its script is a list of commands.#

Your first pipeline file#

This is a complete .gitlab-ci.yml for a small Node.js project. It has two jobs: one builds the code, and one tests it.

practice · first-pipelinesteps/01-first.ymlpipeline
stages:
  - build
  - test

build-app:
  stage: build
  image: node:24.21.0-alpine
  script:
    - npm ci
    - npm run build

unit-tests:
  stage: test
  image: node:24.21.0-alpine
  script:
    - npm ci
    - npm test
LineWhat it means
stages:the stages of this pipeline, in the order they run: first build, then test
build-app:a job. Its name is up to you, as long as it isn't a GitLab keyword
stage: buildthis job belongs to the build stage
image: node:24.21.0-alpinethe job runs inside a container started from the node:24.21.0-alpine image, which has Node.js 24 installed
script:the commands the job runs, in order. If one fails, the job stops and fails
unit-tests:a second job, in the test stage, with the same image and its own commands

An image is a packaged set of tools and files. A container is a running copy of it, fresh for every job and thrown away afterwards. So nothing a job does on its own disk reaches the next job, unless the job saves it on purpose (chapter 31). That is why both jobs above run npm ci.

When you push a commit, GitLab turns this file into a pipeline. The pipeline graph shows each stage as a column and each job as a box:

The pipeline this file produces: build-app in the build stage, then unit-tests in the test stagebuildbuild-apptestunit-tests
Pipeline The pipeline this file produces: build-app in the build stage, then unit-tests in the test stage#
As text
  1. build: build-app
  2. test: unit-tests

Here is what happens between your push and the result:

From a push to a result. Every job runs in a fresh container, and a failed job stops the stages after it.noyesYou push a commit to GitLabGitLab reads .gitlab-ci.yml at thatcommitGitLab creates a pipeline with itsstages and jobsA runner takes the next job and startsa container from the job's imageThe runner runs the job's script, onecommand at a timeDid every command succeed?The job fails, and later stagesdon't startThe next stage starts. When every jobhas passed, the pipeline passes
Figure From a push to a result. Every job runs in a fresh container, and a failed job stops the stages after it.#
As text
  1. You push a commit to GitLab
  2. GitLab reads .gitlab-ci.yml at that commit
  3. GitLab creates a pipeline with its stages and jobs
  4. A runner takes the next job and starts a container from the job's image
  5. The runner runs the job's script, one command at a time
  6. Did every command succeed? No: The job fails, and later stages don't start. Yes: the next step.
  7. The next stage starts. When every job has passed, the pipeline passes

A runner is a program, running somewhere your organisation chose, that asks GitLab for jobs and runs them. Chapter 4 takes each of these words further.

Every pipeline in this book is built from the same pieces: keywords, jobs, stages, images and scripts. The difficulty in a large organisation is not the pieces. It is that a real pipeline is assembled from many files, settings and teams you can't see from your own project. Finding those, and reading them, is what the rest of the book teaches.

Reading about a pipeline only goes so far. The next chapter, Your first pipeline, has you run this file on your own GitLab, break it three ways and fix it, in about a quarter of an hour.

Your first pipeline#

Before you start showed you a pipeline file. Now you run it on your own GitLab, then break it on purpose, three ways, so that you know what each kind of mistake looks like. You need about twenty minutes, a terminal with Git, and an account on a GitLab where you can create a project. GitLab.com's Free tier works, and so does your organisation's own GitLab.

The project#

The project is a tiny Node.js one. It has a function that builds a greeting, a test for it, and a build step that copies the source into dist/, the way a real build would bundle it. It has no dependencies, so npm ci has nothing to download. These are its files; each pane has a copy button.

practice · first-pipelinepackage.json
{
  "name": "first-pipeline",
  "version": "1.0.0",
  "private": true,
  "description": "The project of the first-pipeline lab: a greeting, built and tested.",
  "type": "module",
  "scripts": {
    "build": "node build.js",
    "test": "node --test"
  }
}
practice · first-pipelinepackage-lock.json
{
  "name": "first-pipeline",
  "version": "1.0.0",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "first-pipeline",
      "version": "1.0.0"
    }
  }
}
practice · first-pipelinebuild.js
// The build: copies src/ into dist/, the way a real build would bundle it.
import { cpSync, mkdirSync } from "node:fs";

mkdirSync("dist", { recursive: true });
cpSync("src", "dist", { recursive: true });
console.log("built dist/greet.js and dist/cli.js");
practice · first-pipelinesrc/greet.js
// The first-pipeline lab's one function.
export function greet(name) {
  return `Hello, ${name}!`;
}
practice · first-pipelinesrc/cli.js
// Prints a greeting: node src/cli.js <name>
import { greet } from "./greet.js";

console.log(greet(process.argv[2] ?? "world"));
practice · first-pipelinetest/greet.test.js
// The first-pipeline lab's test, run with node --test.
import { test } from "node:test";
import assert from "node:assert/strict";
import { greet } from "../src/greet.js";

test("greet says hello", () => {
  assert.equal(greet("GitLab"), "Hello, GitLab!");
});
LabRun itabout 8 min#
You need

An account on a GitLab where you can create a project, a terminal with Git, and the six files above.

Steps
  1. In GitLab's upper-right corner, select Create new (the plus sign), then New project/repository, then Create blank project. Name it first-pipeline, leave everything else as it is, and select Create project.
  2. Clone the project with the address under its Code button. Copy the six files into the folder, keeping their paths.
  3. Add a file named .gitlab-ci.yml, with the pipeline from Before you start:
    practice · first-pipelinesteps/01-first.ymlpipeline
    stages:
      - build
      - test
    
    build-app:
      stage: build
      image: node:24.21.0-alpine
      script:
        - npm ci
        - npm run build
    
    unit-tests:
      stage: test
      image: node:24.21.0-alpine
      script:
        - npm ci
        - npm test
  4. Commit and push:
    git add .
    git commit -m "Add the first pipeline"
    git push
  5. In GitLab, open Build › Pipelines, then the newest pipeline, then its build-app job.
What you should see

The pipeline passes: build-app in the build stage, then unit-tests in the test stage. The job's log shows what the runner did, then each of your commands and what it printed. Here it is, cut to its main lines:

GitLab ranjob logcaptured from GitLab
Running with gitlab-runner 19.3.0 (9cbf0074)
⋯ lines omitted
Using Docker executor with image node:24.21.0-alpine ...
⋯ lines omitted
Getting source from Git repository
⋯ lines omitted
Executing "step_script" stage of the job script
⋯ lines omitted
$ npm ci
up to date, audited 1 package in 5s
⋯ lines omitted
$ npm run build
> first-pipeline@1.0.0 build
> node build.js
built dist/greet.js and dist/cli.js
⋯ lines omitted
Job succeeded
Note
On GitLab.com, a new account's first pipeline can stop with the message "Identity verification is required in order to run CI jobs". Select Verify my account in the banner, verify a phone number or add a payment method, then run the pipeline again. On your organisation's own GitLab, if the jobs just wait, ask your platform team: your project may have no runner it is allowed to use.
LabBreak it three waysabout 10 min#
You need

The project from the previous lab, with its pipeline passing.

Steps
  1. Bad indentation. In build-app, move script: one space to the left, then commit and push:
    practice · first-pipelinesteps/02-bad-indent.ymlbroken
    build-app:
      stage: build
      image: node:24.21.0-alpine
     script:
        - npm ci
        - npm run build
    Look at the new pipeline, then put the space back and push again.
  2. A command that fails. Add a line to unit-tests that runs a script the project doesn't have, and push:
    practice · first-pipelinesteps/03-failing-command.ymlbroken
    unit-tests:
      stage: test
      image: node:24.21.0-alpine
      script:
        - npm ci
        - npm run lint
        - npm test
    Read the failed job's log, then take the line out again.
  3. A file from an earlier job. Add a job that runs what build-app built, and push:
    practice · first-pipelinesteps/04-missing-file.ymlsmoke-test
    smoke-test:
      stage: test
      image: node:24.21.0-alpine
      script:
        - node dist/cli.js "smoke test"
    When it fails, make build-app keep dist/ as an artifact, and push again:
    practice · first-pipelinesteps/05-artifacts.ymlbuild-app
    build-app:
      stage: build
      image: node:24.21.0-alpine
      script:
        - npm ci
        - npm run build
      artifacts:
        paths:
          - dist/
What you should see

Bad indentation. The pipeline fails at once, with no jobs. Its badge says error, and its tooltip gives GitLab's reason: "The pipeline failed due to an error on the CI/CD configuration file." Paste the file into CI Lint (chapter 4) for the detail:

(<unknown>): did not find expected key while parsing a block mapping at line 1 column 1

Line 1 is where GitLab began the map it couldn't finish, not the line you changed. Look further down, where the indentation shifts.

A command that fails. unit-tests fails. npm says what went wrong, and the runner says why the job stopped:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ npm ci
⋯ lines omitted
$ npm run lint
npm error Missing script: "lint"
⋯ lines omitted
ERROR: Job failed: exit code 1

A file from an earlier job. smoke-test fails, because its fresh container has no dist/:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ node dist/cli.js "smoke test"
⋯ lines omitted
Error: Cannot find module '/builds/practice/first-pipeline/dist/cli.js'
⋯ lines omitted
ERROR: Job failed: exit code 1

With the artifact, the job downloads build-app's files first, and passes:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
Downloading artifacts for build-app (192)...
⋯ lines omitted
$ node dist/cli.js "smoke test"
Hello, smoke test!
⋯ lines omitted
Job succeeded

The third break is the one that surprises people most. Every job starts in a fresh container, so nothing a job leaves on its own disk reaches the next job unless the job keeps it on purpose. Chapter 31 covers artifacts, and their cousin, the cache.

LabA job with nowhere to runabout 4 min#
Steps
  1. Add a job that asks for a runner with a gpu tag, and push:
    practice · first-pipelinesteps/06-stuck.ymlstuck
    train-model:
      stage: test
      image: node:24.21.0-alpine
      tags: [gpu]
      script:
        - echo "no runner has the gpu tag, so this job waits"
  2. Open train-model's page from the pipeline.
  3. Cancel the pipeline, then delete the job from the file and push.
What you should see

The other jobs pass, but train-model stays pending. Its page says why, and lists the tag that no runner has:

A job's page while no online runner has all of its tags: the message, the job's tags, and links to the project and its CI settings. your-group / first-pipeline › Build › Jobs › #312 pending train-model This job is stuck because of one of the following problems. There are no active runners online, no runners for the protected branch, or no runners that match all of the job's tags: gpu Go to project CI settings Job details Queued waiting for a runner Source Push Tags gpu no online runner has every tag
Figure A job's page while no online runner has all of its tags: the message, the job's tags, and links to the project and its CI settings.#

On your organisation's GitLab, runners have tags that the platform team chooses. A job that asks for any other tag waits like this (stuck on tags).

That is the whole engine: a push, a file GitLab reads, jobs in stages, containers started from images, and runners that take the jobs whose tags they have. Chapter 4 takes each of these words further, and Part III grows this into a real service's pipeline.

GitLab CI/CD in an hour#

Part I · Handbook · Chapter 01 · 51 min read

One small project holds every keyword of a GitLab pipeline file, in the order this book teaches them, with the explanation written as comments on the lines. Each comment says what the keyword does, why you would use it, how it differs from the ones that look like it, and what an expert knows about it. Read it from the top and you can read, write and debug any pipeline file. Nothing here is typed into the book: every pane is quoted from the project, which runs on a real GitLab.

This is the dense reading. It assumes only that you know what CI/CD is, and it explains each word where it first appears, so you never have to jump ahead. The textbook after the Handbook covers the same ground as a story, with worked examples, labs and more depth. Each part below ends with a link to where that story starts.

A pipeline is what GitLab runs when you push: jobs, grouped in stages, each job a script in a fresh container. Everything below is about the file that describes it, .gitlab-ci.yml, and the few things that live outside any file. Words in the comments that are new to you are one line each in What is what.

1. The file#

Three top-level keys set up the whole pipeline. Everything else is a job.

practice · in-45-minutes.gitlab-ci.ymlm1-file
# ---- 1. The file -----------------------------------------------------------
# When you push a commit, GitLab looks for a file named .gitlab-ci.yml at the
# root of your repository. What it finds there becomes a PIPELINE: a set of
# JOBS, each a small script that runs on a machine GitLab controls, called a
# RUNNER. A few top-level keys in this file are settings for the whole
# pipeline. Every other top-level key is the name of a job.

# `spec` is optional, and if you use it, it has to be the very first thing in
# the file, in its own YAML document that ends with a line of three dashes.
spec:
  # "Inputs" are values that whoever starts the pipeline can choose, like the
  # arguments of a function. Use them for the few things that change from one
  # run to the next, such as which environment to deploy to. GitLab fills
  # them in while it is reading this file, before any job exists, so an input
  # can't hold something that is only known later, like a build number.
  inputs:
    greeting:
      default: "hello"              # the value used when nobody chooses one
      description: "What the jobs say"   # shown next to the field
      type: string                  # or number, boolean, array
      options: ["hello", "hi"]      # the only values allowed
      # regex: ^h                   # or a pattern the value must match
---
# The STAGES, in order. A stage is a step of the pipeline, such as "build" or
# "test". Every job belongs to one stage, and a stage starts only when every
# job in the stage before it has passed. That is how you say "test after
# building". Two stages always exist without being listed: `.pre` runs first
# and `.post` runs last.
stages:
  - build
  - test
  - package
  - deploy

# `default` holds settings that every job gets, unless a job sets its own.
# Put here what you would otherwise repeat in every job. (Older files put
# `image:` or `before_script:` directly at the top level instead; that is the
# old way of writing the same thing.)
default:
  # Every job runs inside a CONTAINER: a fresh, throwaway Linux box. The IMAGE
  # is the snapshot that box starts from, with the tools already installed.
  # Always pin it to a version, so that the job behaves the same next month.
  image: alpine:3.22
  # Commands that run before the job's own script, in the same shell. Use it
  # for setup that every job needs, such as logging in.
  before_script:
    - echo "before_script runs first, in every job"
  # Commands that run after the script, in a separate shell, even when the
  # script failed. Use it for clean-up, or for saving a report of a failure.
  after_script:
    - echo "after_script runs last, even when the script failed"
  # If a newer commit is pushed while this job is waiting or running, GitLab
  # may cancel it, because the newer pipeline will run it again anyway.
  interruptible: true
  # Run the job again if it fails: at most once, and only when the runner
  # itself broke, not when the script returned an error. Retrying on every
  # failure would hide real bugs.
  retry:
    max: 1
    when: [runner_system_failure]
    # exit_codes: [137]              # or only for these exit codes

# Here is a job. Its name is the key, and you choose it. `stage` says when it
# runs, and `script` is the list of shell commands it runs, one after the
# other, in a fresh container that holds a fresh copy of the repository. If
# any command fails, the job fails.
say-hello:
  stage: build
  # Stop the job, as failed, if it runs longer than this. `timeout` is set per
  # job; it is one of the few settings `default` can't hold.
  timeout: 10m
  script:
    # An input from `spec` above, filled in while the file was being read.
    - echo "$[[ inputs.greeting ]] from a job"
    # VARIABLES are named values a job can read, written as $NAME. GitLab
    # sets these two, and many more, for every job: they are called
    # predefined variables. SHORT_SHA is the commit's id, and REF_NAME the
    # branch or tag.
    - echo "commit $CI_COMMIT_SHORT_SHA on $CI_COMMIT_REF_NAME"

Deeper: Part II.

2. Reuse: where most of a real pipeline comes from#

In a large organisation your file is short, because most of its jobs are copied in from files a central team owns. Six keywords do the copying.

practice · in-45-minutes.gitlab-ci.ymlm2-reuse
# ---- 2. Reuse: where most of a real pipeline comes from --------------------
# In a large organisation, your file is short, because most of your jobs are
# written once by a central team and copied into your pipeline. This part
# shows every way of copying, and when to use which:
#   include     brings whole FILES into this one, as if pasted here.
#   extends     copies a whole JOB (a template job) into one of yours.
#   !reference  copies one SECTION of a job, such as its list of commands.
#   an anchor   copies a piece of YAML, but only inside the same file.

# `include` pulls other YAML files into this one. GitLab reads them first,
# and joins everything into one big configuration, as if you had pasted the
# files here. Jobs in those files become your jobs, and template jobs in
# them become available to `extends` and `!reference` below.
include:
  # `local`: a file in this same repository, at the same commit.
  - local: ci/base.yml
    # An include can carry `rules`: conditions that say when to include the
    # file at all. This one says: only if the file exists.
    rules:
      - exists: [ci/base.yml]
  # `project`: a file in another GitLab project. This is how a central team
  # shares its templates. `ref` pins a tag or commit, so that the central
  # team's next change can't alter your pipeline until you choose it.
  - project: practice/central-templates
    ref: 1.0.0
    file: templates/node.yml
  # `component`: a packaged, versioned job with typed inputs, published by a
  # project. It differs from a plain `project` include in that you set its
  # inputs here, and GitLab checks them. The address is the GitLab server,
  # the project, the component's name, and its version after the @.
  - component: $CI_SERVER_FQDN/practice/central-templates/secret-scan@1.0.0
    inputs:
      stage: test
  # `template`: a file that GitLab itself ships, such as its scanners:
  # - template: Jobs/Build.gitlab-ci.yml
  # `remote`: a file at any URL. `integrity` pins its checksum, so that a
  # changed file is refused:
  # - remote: https://example.com/ci.yml
  #   integrity: sha256-L5/GAbeXGkPeEtIuOJhiJP+5xgrgdlwFEqm1O3EJwHM=
#
# What an expert knows about includes: GitLab joins the files in the order
# listed, and if two files define a job of the same name, the later one wins.
# Your own file is joined last, so it beats them all. To see the joined
# result, open Build > Pipeline editor > Full configuration: that view shows
# every job with its includes, `extends` and `!reference` already resolved,
# and it is where you trace any job you did not write.

# A YAML ANCHOR is plain YAML, not a GitLab feature: `&login` gives this list
# a name, and `*login` further down pastes a copy of it. Anchors are the
# oldest way of avoiding repetition, and you will meet them in older files.
# Their limit: an anchor only works inside the one file it is written in. It
# can't reach anything that came in through `include`. The two GitLab
# keywords below can, which is why they replaced anchors.
.login: &login
  - echo "logging in"

# A job whose name starts with a dot is HIDDEN: GitLab never runs it. Hidden
# jobs exist to be copied. `.base-build` is one, written in ci/base.yml.
#
# `extends` copies a hidden job into this job, and then lays this job's own
# settings on top. Use it when a central team has written the job you need
# and you want to change one or two things. How the two are combined:
#   - settings that are maps, such as `variables` and `cache`, are merged
#     key by key, so you can add or change one key and keep the rest;
#   - settings that are lists, such as `script`, `rules`, `before_script`,
#     `tags` and `services`, are replaced whole: if you write one line of
#     `script`, the template's whole script is gone.
# Experts also know: `extends` can chain, a job extending a job that extends
# another, across files, and GitLab advises three levels at most. To trace a
# chain, search the included files for each parent's name, one hop at a time.
build:
  extends: .base-build
  stage: build
  # The anchor from above, pasted in. It works because both are in this file.
  before_script: *login
  script:
    # `!reference` copies ONE section of another job, from any file, and
    # pastes it exactly here. Compare it with `extends`: `extends` takes a
    # whole job and merges it underneath yours; `!reference` takes one piece
    # and puts it where you say. Use it to keep a template's list and add
    # your own lines around it, which `extends` can't do for lists. Here it
    # pastes the `make` list from `.snippets` in ci/base.yml.
    - !reference [.snippets, make]
    - echo "built at $CI_COMMIT_SHORT_SHA" > build.txt
  # ARTIFACTS are files GitLab keeps when the job ends and hands to later
  # jobs. Without this, build.txt would vanish with the container. Part 5
  # shows all the settings.
  artifacts:
    paths: [build.txt]

# `parallel: matrix` turns one job into several, one per combination of the
# values, each with the variable set. GitLab names them "test: [unit]" and
# "test: [integration]". Use it to run the same job for several versions,
# platforms or suites without copying the job.
test:
  stage: test
  parallel:
    matrix:
      - SUITE: [unit, integration]
  script:
    - echo "running the $SUITE tests"

# A TRIGGER job has no script. It starts another pipeline and reports how that
# went. With `include`, the other pipeline is a CHILD: it is described by
# another file in this repository and shown nested inside this pipeline.
# Use a child pipeline to keep a big pipeline in several files, or to
# generate a pipeline from a script.
child:
  stage: test
  trigger:
    include:
      - local: ci/child.yml
        # The child file has a `spec` with inputs of its own, filled in here.
        inputs:
          who: "the parent pipeline"
    # `mirror` makes this job show the child's status: it waits for the child
    # and fails if the child fails. Without it, the job passes the moment the
    # child is created, whatever happens afterwards, so a broken child would
    # show green. Older files say `strategy: depend`, which GitLab no longer
    # recommends.
    strategy: mirror
    # Pass this pipeline's variables down to the child.
    forward:
      pipeline_variables: true
  # The other kind of trigger: with `project`, GitLab runs ANOTHER PROJECT's
  # own pipeline, called a multi-project pipeline. Use it to kick off a
  # deploy repository, for example. Its inputs go under `trigger:` itself:
  # trigger:
  #   project: practice/node-service
  #   inputs: { who: "another project" }

The two files this one borrows from:

practice · in-45-minutesci/base.ymlbase
# A job whose name starts with a dot is HIDDEN: GitLab never runs it. It is a
# template, written so that other jobs can copy it with `extends`. This one
# lives in a separate file, which .gitlab-ci.yml brings in with `include`.
# A YAML anchor could never reach it from there; `extends` can.
.base-build:
  image: alpine:3.22
  cache:
    key: base
    paths: [.cache/]
    policy: pull

# A hidden job can also hold keys that are not CI/CD keywords at all, like
# `make` here. They are snippets: pieces of YAML kept in one place, for jobs
# to paste with `!reference [.snippets, make]`. Central teams keep their
# shared script lines and rule lists this way.
.snippets:
  make:
    - echo "a make step, pasted from ci/base.yml with !reference"
practice · in-45-minutesci/child.ymlchild
# A child pipeline is a complete pipeline of its own, described by this file
# and shown nested inside the pipeline that triggered it. It has its own
# `spec`, so the parent can pass it inputs.
spec:
  inputs:
    who:
      default: "nobody"          # the parent's trigger fills this in
---
child-job:
  image: alpine:3.22
  # Inside a child, $CI_PIPELINE_SOURCE is "parent_pipeline". A job with no
  # rules runs only for branches and tags, and a merge request's pipeline is
  # neither, so without this rule the child would have no jobs there, and
  # GitLab would refuse to create it.
  rules:
    - if: $CI_PIPELINE_SOURCE == "parent_pipeline"
  script:
    - echo "hello from $[[ inputs.who ]], in a child pipeline"
    # The parent's `forward: pipeline_variables` passed its variables down.
    - echo "the parent forwarded LOG_LEVEL=$LOG_LEVEL"

Deeper: Part IV, and chapter 20 for finding where any job came from.

3. Conditions: which pipelines exist, and which jobs are in them#

GitLab decides twice. First whether this push gets a pipeline at all, with workflow. Then, for every job, whether it is in that pipeline, with its rules. Both are lists of questions, asked in order, and the first yes decides.

practice · in-45-minutes.gitlab-ci.ymlm3-conditions
# ---- 3. Conditions: which pipelines exist, and which jobs are in them ------
# GitLab decides twice. First, does this push get a pipeline at all? That is
# `workflow`. Then, for every job, is it in this pipeline? That is the job's
# `rules`. Both are lists of conditions that GitLab tries from the top, and
# the first one that matches decides. Nothing else is looked at.
#
# The conditions use predefined variables. The ones you will see most:
#   $CI_PIPELINE_SOURCE   what caused the pipeline: "push", "schedule",
#                         "merge_request_event", "web" and others
#   $CI_COMMIT_BRANCH     the branch name, set only in a branch pipeline
#   $CI_COMMIT_TAG        the tag name, set only in a tag pipeline
#   $CI_OPEN_MERGE_REQUESTS   set when the branch has an open merge request
#
# The pattern experts use: put the cases to exclude first, each with
# `when: never`, then the cases to allow, and let everything else fall off
# the end, which means no pipeline. A workflow decides only whether the
# pipeline exists; it can't put a job in or out.
workflow:
  # The pipeline's title in the pipeline list. Variables are filled in.
  name: "$KIND pipeline for $CI_COMMIT_REF_NAME"
  # When a newer commit is pushed, cancel the older pipeline's jobs that are
  # marked `interruptible`, since the newer pipeline covers them.
  auto_cancel:
    on_new_commit: interruptible
    # on_job_failure: all           # or cancel everything when one job fails
  rules:
    # A MERGE REQUEST is a request to merge one branch into another, with a
    # review. GitLab can run a pipeline for the merge request itself, and
    # show the result on it. This rule says: create that pipeline.
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      variables:
        # A rule can also set a variable for the whole pipeline.
        KIND: "merge request"
    # A push to a branch normally gets a branch pipeline too. While a merge
    # request is open for that branch, the merge request pipeline already
    # covers the push, so don't make a second one. Without this rule, every
    # push would run everything twice.
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    # Any other push to a branch: create the pipeline.
    - if: $CI_COMMIT_BRANCH
      variables:
        KIND: "branch"              # every job can read KIND
    # A push of a tag: create the pipeline. Anything not matched here, such
    # as a schedule, gets no pipeline.
    - if: $CI_COMMIT_TAG
      variables:
        KIND: "tag"

# A job's `rules` decide whether the job is part of this pipeline, and how it
# runs. The first rule that matches decides. If no rule matches, the job is
# simply left out, with no message anywhere: that is the most common reason a
# job "didn't run". Order the rules from the most specific case to the most
# general, and put `when: never` rules first to exclude cases. A job can't mix
# `rules` with the older `only`/`except`. To see which jobs a push would get
# before pushing, use Build > Pipeline editor > Validate.
lint:
  stage: test
  script: echo "lint, in $LINT_MODE mode"
  rules:
    # `if` tests variables. This one says: always, on the default branch
    # (usually main).
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      variables:
        LINT_MODE: "full"           # a rule can set a variable for this job
    # `changes` tests which files the commit changed. Only if a file under
    # src/ changed does this rule match. Two surprises: on the first push of
    # a new branch it is always true, and in a merge request pipeline it
    # compares against the target branch, not the previous commit.
    - changes:
        paths: [src/**/*]
        # regexp: src/.*\.js$       # or a Ruby regular expression instead
        compare_to: refs/heads/main
      variables:
        LINT_MODE: "changed-files"  # a different value from a different rule
    # Before `rules` existed, jobs used `only:` and `except:` for the same
    # purpose. Older templates still have them; they mean "only in these
    # cases" and "except in these cases".

# A MANUAL job is a play button: it sits in the pipeline until a person
# starts it. Two things to know. A manual job that a rule adds counts as
# "not finished", so the pipeline shows as blocked until someone runs it,
# unless the rule also says `allow_failure: true`, as here. And a job that
# `needs` a manual job is skipped until the manual job has been run.
approve-docs:
  stage: test
  script: echo "the docs were approved by hand"
  rules:
    # `exists` tests whether a file is in the repository at all.
    - exists: [docs/notes.md]
      when: manual
      allow_failure: true

# By default a job waits for the whole previous stage. `needs` lists the jobs
# this one really depends on, and the job starts as soon as those have
# finished, whatever their stage. It also downloads their artifacts. Use it
# to make pipelines faster. A job may need jobs from earlier stages or its
# own; a job that needs a job that isn't in the pipeline fails to be created,
# unless the need is `optional`.
package:
  stage: package
  needs:
    - job: build
    # One instance of a matrix job. `artifacts: false` means: wait for it,
    # but don't download its files.
    - job: test
      parallel:
        matrix:
          - SUITE: unit
      artifacts: false
    # `optional` means: if this job isn't in the pipeline at all, as on a
    # branch where lint's rules didn't match, don't wait for it.
    - job: lint
      optional: true
    # Premium: a job in another project's pipeline.
    # - project: group/other
    #   job: build
    #   ref: main
    #   artifacts: true
  script:
    - cat build.txt
    - echo "this is a $KIND pipeline"

# `when` says under what condition a job runs, once it is in the pipeline:
#   on_success   when no job in an earlier stage failed (the default)
#   on_failure   only when a job in an earlier stage failed
#   always       whatever happened earlier
#   never        not at all
#   manual       when a person presses play
#   delayed      after a wait
cleanup:
  stage: .post
  when: always
  # `dependencies` chooses which earlier jobs' artifacts to download. By
  # default a job downloads all of them. An empty list downloads none, which
  # makes a job like this faster.
  dependencies: []
  script: echo "cleanup"

# A DELAYED job waits for a set time after its stage is ready, then starts on
# its own, unless someone cancels it. Use it for a pause before a rollout.
warm-up:
  stage: deploy
  when: delayed
  # How long to wait before starting.
  start_in: 10 seconds
  script: echo "warmed up"

Deeper: Part V.

4. What a job is given#

Before a job's first line runs, it receives variables from several places, and the places have an order. Here is the order, and the shape of each kind.

practice · in-45-minutes.gitlab-ci.ymlm4-given
# ---- 4. What a job is given: variables, secrets and identity ---------------
# A job can read values from several places, and when two places set the
# same name, the higher one on this ladder wins:
#   1. a security policy (set by a security team, outside any file)
#   2. values given when the pipeline was started by hand or by a trigger
#   3. the project's settings page, then its groups', then the instance's
#   4. dotenv reports from earlier jobs (part 5)
#   5. the job's own `variables`
#   6. the top-level `variables` below
#   7. GitLab's predefined variables
# The most common surprise: a value set on a settings page (3) silently beats
# the value you wrote in the file (5 or 6). When a value isn't what you wrote,
# check Settings > CI/CD > Variables on the project, then on each of its
# groups, then ask about the instance: the job's page does not show them.
#
# Top-level `variables` reach every job.
variables:
  APP: "in-45-minutes"          # the plain form: a name and a value
  # The long form, with a description and a list of allowed values. GitLab
  # then shows the variable as a field on its "Run pipeline" page.
  LOG_LEVEL:
    value: "info"
    description: "How much the jobs print"
    options: ["info", "debug"]
  # Normally a `$` inside a value refers to another variable. `expand: false`
  # keeps the `$` as plain text.
  RAW_VALUE:
    value: "$APP-literal"
    expand: false

show-variables:
  stage: build
  # The job's own `variables` win over the top-level ones (5 beats 6). A value
  # that came in through `extends` counts as the job's own.
  variables:
    LOG_LEVEL: "debug"
  # `inherit` limits what this job takes from `default:` and from the
  # top-level `variables`. Here it takes only `image`, and only these three
  # variables. The workflow's KIND is not in the list, so this job can't see
  # it. Use `inherit` for a job that must be isolated from the rest.
  inherit:
    default: [image]
    variables: [APP, LOG_LEVEL, RAW_VALUE]
  # An ID TOKEN is a short-lived, signed statement from GitLab that says which
  # project, branch and job this is. A cloud such as AWS or Azure can be set
  # up to trust it, and give the job access, so that no long-lived password
  # or key has to be stored anywhere. `aud` (audience) is who the token is
  # meant for.
  id_tokens:
    CLOUD_TOKEN:
      aud: https://cloud.example.com
  script:
    - echo "APP=$APP LOG_LEVEL=$LOG_LEVEL RAW_VALUE=$RAW_VALUE KIND=$KIND"
    - test -n "$CLOUD_TOKEN" && echo "an ID token was issued for this job"
    # DEPLOY_TOKEN is not in this file at all. It was typed into the project's
    # settings page (rung 3 of the ladder), which is where secrets belong.
    # There it was marked MASKED, so a log shows [MASKED] instead of the
    # value, and PROTECTED, so only pipelines on protected branches and tags,
    # such as main, receive it. On any other branch it is simply empty, which
    # is why a job can work on main and fail on a feature branch. GitLab can
    # only mask a value that is one line, without spaces, and at least eight
    # characters long; anything else it refuses to mask.
    - echo "DEPLOY_TOKEN=$DEPLOY_TOKEN"
  # Premium: `secrets` fetches a value at run time from an outside store, and
  # by default writes it to a file whose path is in the variable.
  # secrets:
  #   DB_PASSWORD:
  #     vault: kv/db/password@secret
  #     file: false                  # the value in the variable, not a file
  #     token: $CLOUD_TOKEN          # which ID token to present to the store
  #   API_KEY:
  #     aws_secrets_manager: { secret_id: api-key }
  #   CERT:
  #     azure_key_vault: { name: cert }
  #   DSN:
  #     gcp_secret_manager: { name: dsn, version: latest }
  #   PEPPER:
  #     gitlab_secrets_manager: { name: pepper }
  # Sign the job in to Google Cloud with its ID token (GitLab.com only, beta):
  # identity: google_cloud

The job above, as it ran on a protected branch and then in a merge request. The settings variable is masked on one and empty on the other:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ echo "APP=$APP LOG_LEVEL=$LOG_LEVEL RAW_VALUE=$RAW_VALUE KIND=$KIND"
APP=in-45-minutes LOG_LEVEL=debug RAW_VALUE=$APP-literal KIND=
$ test -n "$CLOUD_TOKEN" && echo "an ID token was issued for this job"
an ID token was issued for this job
$ echo "DEPLOY_TOKEN=$DEPLOY_TOKEN"
DEPLOY_TOKEN=[MASKED]
⋯ lines omitted
Job succeeded
GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ echo "APP=$APP LOG_LEVEL=$LOG_LEVEL RAW_VALUE=$RAW_VALUE KIND=$KIND"
APP=in-45-minutes LOG_LEVEL=debug RAW_VALUE=$APP-literal KIND=
$ test -n "$CLOUD_TOKEN" && echo "an ID token was issued for this job"
an ID token was issued for this job
$ echo "DEPLOY_TOKEN=$DEPLOY_TOKEN"
DEPLOY_TOKEN=
⋯ lines omitted
Job succeeded

Deeper: Part VI.

5. Execution and data#

Where a job runs, what runs beside it, what it keeps, and what it hands to the jobs after it.

practice · in-45-minutes.gitlab-ci.ymlm5-execution
# ---- 5. Execution and data: where a job runs, and what it passes on --------
with-service:
  stage: test
  # `image` in its long form. `name` is the image; the other keys adjust how
  # it is used.
  image:
    name: alpine:3.22
    # An image can have a default command, its ENTRYPOINT. A tool's own image
    # often makes that the tool itself, and then your script never runs.
    # An empty entrypoint switches that off.
    entrypoint: [""]
    # Reuse an already downloaded copy of the image instead of pulling it
    # again every time.
    pull_policy: if-not-present
    # Docker-only settings, such as the platform:
    # docker: { platform: linux/amd64 }
  # A SERVICE is a second container started beside the job, for something the
  # job needs to talk to, such as a database. The job reaches it by the
  # `alias` as a host name. Here it is a small registry, reached as "reg".
  # A service must stay running on its own; when a job can't reach one, read
  # the service's own log, which the job page shows separately.
  services:
    - name: registry:2
      alias: reg
      variables:
        REGISTRY_LOG_LEVEL: "warn"   # set inside the service container only
  # `tags` sends the job only to runners that carry the same tag, for example
  # a runner with a GPU or one inside a private network. If no online runner
  # has the tag, the job waits forever.
  # tags: [linux]
  # `hooks` runs commands at fixed points of the job's life. This one runs
  # before the repository is even fetched, for example to set up Git access.
  hooks:
    pre_get_sources_script:
      - echo "about to clone"
  variables:
    # GIT_DEPTH controls how much history the job fetches. Jobs get a shallow
    # copy of their own branch only, which is fast, but a script that reads
    # older commits or tags finds nothing there.
    GIT_DEPTH: "5"
  script:
    - wget -qO- http://reg:5000/v2/ && echo "the service answered"
    - echo "coverage lines 87.5%"
  # `coverage` is a pattern GitLab searches the log with. The number it finds
  # becomes the job's test coverage, shown in badges and merge requests.
  coverage: '/coverage lines (\d+\.\d+)%/'

keep-files:
  stage: package
  # An empty `needs` means: depend on nothing, start at once.
  needs: []
  script:
    - mkdir -p out && echo "a report" > out/report.txt
    # A DOTENV file is a text file of NAME=value lines. Reported as an
    # artifact (below), its lines become variables in every later job.
    - echo "VERSION=1.4.0" > build.env
  # ARTIFACTS, in full. Files GitLab uploads when the job ends, keeps for a
  # while, and gives to later jobs. They are guaranteed to arrive, which is
  # what makes them different from the cache further down.
  artifacts:
    # The name of the download, as shown on the job page.
    name: "$CI_JOB_NAME-$CI_COMMIT_SHORT_SHA"
    # Which files and folders to keep.
    paths: [out/]
    # Which of those to leave out.
    exclude: [out/**/*.tmp]
    # How long GitLab keeps them before deleting them. Without it, the
    # instance's default applies.
    expire_in: 1 week
    # Keep the files even when the job failed: that is when you need them.
    when: always
    # Show a link to them on the merge request.
    expose_as: "report"
    # Who may download them from GitLab: all, developer, maintainer or none.
    # The older `public: false` means the same as `developer`.
    access: developer
    # REPORTS are artifacts in a format GitLab understands and shows: test
    # results (junit), coverage, security findings, and dotenv for variables.
    reports:
      dotenv: build.env
  # The CACHE is for speed, and nothing else. It keeps files such as
  # downloaded packages between pipelines, so the next job doesn't download
  # them again.
  # Unlike artifacts, a cache is not guaranteed: it may be missing on another
  # runner or after a clean-up, so a job must still work without it. The
  # first pipeline after a key changes always finds nothing, and caches of
  # protected branches are kept apart from the rest (the key gets a
  # "-protected" suffix) unless `unprotect` says otherwise.
  cache:
    # The key names the cache. `files` makes the key change when these files
    # change, so a changed dependency list never restores a stale cache.
    key:
      files: [ci/base.yml]
      # files_commits: [ci/base.yml]   # or the files' last commits instead
      prefix: "$CI_JOB_NAME"
    # Which folders to keep. They must be inside the project's folder.
    paths: [.cache/]
    # `pull` restores the cache, `push` saves it, `pull-push` does both.
    policy: pull-push
    # Save the cache only when the job passed (or `always`, or `on_failure`).
    when: on_success
    # Other keys to try when this one has no cache yet.
    fallback_keys: [default]
    # Caches of protected branches are normally kept apart. This shares them.
    unprotect: true
    # `true` would also cache every file Git doesn't track, such as build
    # output.
    untracked: false

use-files:
  stage: deploy
  needs: [keep-files]
  # A RESOURCE GROUP lets only one job of the group run at a time, even across
  # pipelines. Use it so that two deploys to the same place never overlap.
  resource_group: staging
  script:
    # out/report.txt arrived as an artifact, and VERSION from the dotenv.
    - cat out/report.txt
    - echo "VERSION=$VERSION, from the dotenv report"

The package job from part 3, as it ran: its artifact from build arrived, and the workflow's variable named the pipeline's kind:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
Downloading artifacts for build (695)...
⋯ lines omitted
$ echo "before_script runs first, in every job"
⋯ lines omitted
$ cat build.txt
built at a043c660
$ echo "this is a $KIND pipeline"
this is a branch pipeline
⋯ lines omitted
$ echo "after_script runs last, even when the script failed"
⋯ lines omitted
Job succeeded

Deeper: Part VII.

6. Not in any file#

Some of what shapes your pipeline is a setting on a GitLab page, or a policy set by another team. None of it shows in your file, and some of it shows in no file at all.

You seeIt isWhere to lookDeeper
a value no file setsa CI/CD variable from the project's, a group's or the instance's settings; it beats the YAMLSettings › CI/CD › Variables, on the project and on each groupchapter 18
a secret that is empty on your brancha protected variable: only protected branches and tags receive itSettings › CI/CD › Variables, and Settings › Repository › Protected brancheschapter 27
a job that waits with no runnerrunners are registered on the instance, a group or the project, and take jobs whose tags they carrySettings › CI/CD › Runnerschapter 29
a job in no file, often in a stage you never wrote <badge>ultimate</badge>a pipeline execution policy or a scan execution policy, set by a security team, which no toggle in your file can switch offSecure › Policieschapter 36
a merge that needs an approval you can't give <badge>ultimate</badge>a merge request approval policySecure › Policies, and the merge request's approvalschapter 18
sast, secret_detection, container_scanning jobsGitLab's own scanners, from templates GitLab ships; they skip merge request pipelines unless AST_ENABLE_MR_PIPELINES is "true"the included template, then Secure › Security configurationchapter 34
a pipeline with no .gitlab-ci.yml in the projectthe file lives at another path or in another project, or Auto DevOps is onSettings › CI/CD › General pipelines, CI/CD configuration filechapter 18
a template you never included <badge>premium</badge>the instance template repository, or a required include the instance adds to every pipelineask the platform teamchapter 18
another project refuses your job's tokenjob token permissions: which projects may use this project's job tokensSettings › CI/CD › Job token permissionschapter 28
secrets: in a job <badge>premium</badge>a secret fetched at run time from Vault or a cloud's secret managerthe job's YAML, and the store it nameschapter 27
needs: project: <badge>premium</badge>artifacts from a job in another project's pipelinethe job's YAMLchapter 24
dast_configuration: <badge>ultimate</badge>which site and scanner profiles a DAST job usesSecure › Security configurationchapter 34
run: instead of script: <badge>experimental</badge>GitLab Functions, formerly CI/CD Steps: reusable steps in place of shell linesthe job's YAMLchapter 19
release:a job in a tag pipeline that records a release for the tagDeploy › Releaseschapter 40
pages: true on a joba job that publishes its public/ folder as a website; naming the job pages is the old wayDeploy › Pageschapter 64

Deeper: Part VIII, and chapter 18.

7. After the pipeline#

A deploy job names an environment, and GitLab records what went where. In an organisation that uses GitOps, the job's real work is a commit to another repository, and a controller in the cluster does the rest, later.

practice · in-45-minutes.gitlab-ci.ymlm7-after
# ---- 7. After the pipeline: deploys, environments and releases -------------
# An ENVIRONMENT is a place your software runs, such as staging or production.
# A job that names one is a deploy job, and GitLab records every run of it:
# which commit went where, and when. That record is what "Operate >
# Environments" shows, and its Rollback button reruns an earlier deploy job
# with that older commit, which is the quickest way back.
deploy-staging:
  stage: deploy
  script:
    # Deploy the image this pipeline built, tagged with the commit, and deploy
    # that same tag everywhere. A pipeline that builds a fresh image for each
    # environment ships something nobody tested.
    - echo "deploying $APP:$CI_COMMIT_SHORT_SHA to staging"
  environment:
    name: staging
    # A link to the running service, shown beside the environment.
    url: https://staging.example.com
    # What kind of environment this is: production, staging, testing,
    # development or other. GitLab groups and reports them by tier.
    deployment_tier: staging
    # The job that tears the environment down, and when to run it on its own.
    on_stop: stop-staging
    auto_stop_in: 1 day
    # Which agent for Kubernetes and which namespace the environment's
    # dashboard shows. (`kubernetes: namespace:` on its own is the old form.)
    # kubernetes:
    #   agent: platform/agents:staging
    #   dashboard: { namespace: staging }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

# The stop job named above. `action: stop` tells GitLab that this job ends the
# environment rather than deploying to it.
stop-staging:
  stage: deploy
  script: echo "stopping staging"
  environment:
    name: staging
    action: stop
  when: manual
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-production:
  stage: deploy
  script:
    # Many organisations deploy with GITOPS: the job doesn't touch the cluster
    # at all. It writes the new image tag into a Git repository that describes
    # the environment, and a controller running in the cluster notices the
    # change and applies it, minutes later. So a green job here means
    # "written to Git", not "running". To know whether the change is really
    # running, look at the controller, Argo CD or Flux, and at the pods, not
    # at this job.
    - echo "writing tag $CI_COMMIT_SHORT_SHA to the environment repository"
  environment:
    name: production
    url: https://example.com
  # Production waits for a person: `manual` gives the job a play button, and
  # `manual_confirmation` adds a box with this text that must be confirmed
  # before the job starts.
  when: manual
  manual_confirmation: "Deploy to production?"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

# A RELEASE is GitLab's record of a version: a tag with a title, notes and
# download links, shown under "Deploy > Releases". A job with `release:` in a
# tag pipeline creates it. The job needs GitLab's `glab` command in its image
# (registry.gitlab.com/gitlab-org/cli), so it stays a comment here.
# release:
#   stage: deploy
#   script: echo "releasing $CI_COMMIT_TAG"
#   release:
#     tag_name: $CI_COMMIT_TAG
#     tag_message: "Version $CI_COMMIT_TAG"
#     name: "Release $CI_COMMIT_TAG"
#     description: "Built from $CI_COMMIT_SHORT_SHA"
#     milestones: ["1.4"]
#     released_at: "2026-09-26T09:00:00Z"
#     assets:
#       links: [{ name: "binary", url: "https://example.com/app.tar.gz" }]
#   rules:
#     - if: $CI_COMMIT_TAG

# GITLAB PAGES hosts static websites straight from a repository. A job with
# `pages: true` publishes its `public/` folder as the site; naming the job
# `pages` used to be what did it, and is now deprecated. Pages is switched
# off on this GitLab, so it stays a comment too.
# publish-site:
#   stage: deploy
#   script: mkdir -p public && echo "hello" > public/index.html
#   pages: true
#   artifacts:
#     paths: [public/]

Deeper: Part IX, then Part X.

8. The traps, and where you just saw them#

Every trap on the card is a comment on a line above. When one of them bites, the matching card takes you from the symptom to the fix.

TrapWhere the tour shows itCard
Your own rules, script or before_script replaced the template's whole listbuild, in part 2your rules replaced the template's
A settings variable beat the value in your YAMLDEPLOY_TOKEN, in part 4your change is ignored
A protected secret is empty on a branchDEPLOY_TOKEN, in part 4fails only on branches
An include or image on a moving branch or tag changed under youinclude, in part 2the pipeline changed overnight
One push made two pipelinesworkflow, in part 3duplicate pipelines
changes was true on a new branch's first pushlint, in part 3runs on a new branch
A manual job blocked the pipeline, or a job that needed it was skippedapprove-docs, in part 3a play button that waits
No rule matched, so the job silently vanished, and a pipeline with no jobs isn't createdlint, in part 3no jobs, no pipeline
A child pipeline of a merge request had no jobs, because its jobs had no ruleschild-job, in part 2no jobs, no pipeline
The cache missed, and the job had counted on itkeep-files, in part 5a job restores stale files
A script needed history the shallow clone didn't haveGIT_DEPTH, in part 5history missing
The image's entrypoint ran instead of your scriptwith-service, in part 5the script never runs
No online runner carries the job's tagstags, in part 5no runner matches its tags
Green meant written to Git, not runningdeploy-production, in part 7deploys appear minutes later

When something breaks and it isn't in this table, start from the four kinds of symptom on the card: didn't run, failed, wrong result, stuck or drifting. Deeper: Part XI.

What is what#

Part I · Handbook · Chapter 02 · 28 min read

Every word you will meet in a pipeline file or on a GitLab page, one line each. That is every keyword of .gitlab-ci.yml, every way configuration gets shared, every term this book uses, and the old names of things GitLab has renamed. Search it with ⌘K, or read one group at a time. Each line links to where the book explains it.

This table is built from the keyword index, the reuse catalogue and the glossary, so it says exactly what they say.

You seeKindIt meansWhere
Keyword
after_scriptKeywordcommands that run last, in a separate shellchapter 29
allow_failureKeywordlets the pipeline continue if the job failschapter 24
artifactsKeywordfiles kept after the job, for later jobs and peoplechapter 31
artifacts:access, :public, :expose_asKeywordwho can download them, and links in merge requestschapter 31
artifacts:paths, :exclude, :name, :untrackedKeywordwhich files to keepchapter 31
artifacts:reportsKeywordfiles GitLab reads, such as test results and security reportschapter 31
artifacts:when, :expire_inKeywordwhen to keep them, and for how longchapter 31
before_scriptKeywordcommands that run before script, in the same shellchapter 29
cacheKeywordfiles reused between jobs and pipelines, to save timechapter 31
cache:key:files, :files_commits, :prefixKeyworda key that changes when the files' content changes, or when the files get a new commit, with an optional prefixchapter 31
cache:key, :paths, :policy, :when, :fallback_keys, :unprotect, :untrackedKeywordwhich cache, what goes in it, and when it is savedchapter 31
coverageKeyworda regular expression that finds the coverage figure in the logchapter 37
dast_configurationKeywordDAST site and scanner profiles–
defaultKeywordsettings every job gets unless it sets its own: image, before_script, tags and otherschapter 14
dependenciesKeywordwhich earlier jobs' artifacts to downloadchapter 31
environmentKeywordwhere the job deploys, which makes it a deployment jobchapter 40
environment:name, :url, :on_stop, :action, :auto_stop_in, :deployment_tier, :kubernetesKeywordthe environment's name, address, stopping and tierchapter 40
extendsKeywordcopies settings from other jobs, usually hidden oneschapter 14
hooks:pre_get_sources_scriptKeywordcommands on the runner before the Git fetchchapter 29
identityKeywordidentity federation with a cloud provider, in beta–
id_tokensKeywordsigned ID tokens for other systems, such as AWS and Azurechapter 28
imageKeywordthe container image the job runs inchapter 30
image:name, :entrypoint, :pull_policy, :docker, :kubernetesKeywordthe image's name, entrypoint, pull policy and executor optionschapter 30
image, services, cache, before_script, after_script at the top levelKeywordthe same keywords inside default:chapter 14
includeKeywordcopies other YAML files into this configurationchapter 15
include:componentKeyworda versioned CI/CD componentchapter 16
include:inputsKeywordvalues for an included file's inputschapter 16
include:integrity, include:cacheKeywordcheck or cache a remote file–
include:localKeyworda file from the same repository and branchchapter 15
include:projectKeyworda file from another project, at a refchapter 15
include:remoteKeyworda file from any URLchapter 15
include:rulesKeywordincludes the file only when the rules matchchapter 15
include:templateKeyworda template that ships with GitLabchapter 15
inheritKeywordwhich default: settings and top-level variables the job takeschapter 14
inherit:default, inherit:variablesKeywordtake all, none, or a listchapter 14
inputs, :type, :regex and othersKeywordtyped inputs for one job, which can be changed when the job is run by hand or retried–
interruptibleKeywordlets a newer pipeline cancel this jobchapter 24
manual_confirmationKeyworda message to confirm before running a manual jobchapter 24
needsKeywordstarts the job as soon as named jobs finish, ignoring stageschapter 24
needs:artifactsKeywordwhether to download the needed job's artifactschapter 31
needs:optionalKeyworda need that may be missing from the pipelinechapter 24
needs:parallel:matrixKeywordneeds one variant of a matrix jobchapter 14
needs:project, needs:pipeline, needs:pipeline:jobKeywordartifacts or status from other pipelineschapter 17
only, exceptKeywordruleschapter 23
pagesKeyworda GitLab Pages publishing job–
parallelKeywordruns several copies of the jobchapter 14
parallel:matrixKeywordone copy for each combination of variable valueschapter 14
publish, and a Pages job named pagesKeywordthe pages keyword–
release, :tag_name, :tag_message, :name, :description, :ref, :milestones, :released_at, :assets:linksKeywordcreates a release, and describes it–
resource_groupKeywordonly one job in the group runs at a timechapter 24
retryKeywordretries the job on failurechapter 24
retry:max, :when, :exit_codesKeywordhow often, and for which failureschapter 24
rulesKeyworddecides whether the job is in the pipeline, and how it runschapter 23
rules:changes:paths, :compare_to, :regexp, and rules:exists:regexpKeywordthe files to check, the ref to compare against, or a Ruby regular expression in place of glob patternschapter 23
rules:changes, rules:existsKeywordconditions on changed or existing fileschapter 23
rules:ifKeyworda condition on variableschapter 23
rules:when, :allow_failure, :needs, :variables, :interruptibleKeywordwhat a matching rule changeschapter 23
runKeyworda sequence of GitLab Functions stepschapter 19
scriptKeywordthe commands the job runschapter 29
secretsKeywordsecrets fetched from a secrets managerchapter 27
secrets:file, secrets:tokenKeywordfile or value, and which ID token to usechapter 27
secrets:vault, :aws_secrets_manager, :gcp_secret_manager, :azure_key_vault, :gitlab_secrets_managerKeywordthe provider and the secretchapter 27, and for AWS and Azure chapter 38
servicesKeywordextra containers beside the job, such as a databasechapter 30
services:name, :alias, :entrypoint, :command, :variables, :pull_policyKeywordthe service's image, hostname, start command and settingschapter 30
specKeywordthe header of a file, before its --- linechapter 16
spec:include, spec:component, spec:descriptionKeywordshared input definitions, component context, and a descriptionchapter 16
spec:inputs, :default, :type, :options, :regex, :rules, :descriptionKeywordtyped parameters for a component, an included file or a pipeline, with defaults, allowed values and checkschapter 16
stageKeywordthe stage the job belongs tochapter 4
stagesKeywordthe stages, in orderchapter 24
start_inKeywordthe delay for when: delayedchapter 24
tagsKeywordwhich runners may take the jobchapter 29
timeoutKeywordthe job's own time limitchapter 29
triggerKeywordstarts a downstream pipelinechapter 17
trigger:forward, trigger:inputsKeywordwhat the downstream pipeline receiveschapter 17
trigger:includeKeyworda child pipeline from a file in this projectchapter 17
trigger:projectKeyworda pipeline in another projectchapter 17
trigger:strategyKeywordmakes the trigger job wait for, and mirror, the downstream pipelinechapter 17
variablesKeywordvariables for every job, or for one jobchapter 26
variables:expandKeywordwhether $ references in the value are expandedchapter 26
variables:value, :description, :optionsKeyworddefaults and choices shown when someone runs a pipeline by handchapter 26
whenKeywordwhen the job runs: on_success, manual, delayed, always, neverchapter 24
workflowKeywordwhether a pipeline is created, and its settingschapter 22
workflow:auto_cancel, :on_new_commit, :on_job_failureKeywordwhich jobs to cancel when a newer commit starts a pipeline, or when a job failschapter 22
workflow:nameKeywordthe pipeline's namechapter 22
workflow:rulesKeyworddecide whether the pipeline exists at allchapter 22
workflow:rules:variablesKeywordvariables set when a workflow rule matcheschapter 22
Mechanism
a compliance framework label on the project, and jobs from a file the project does not includeMechanismCompliance pipelines: Organisation-wideCompliance pipelines
a form of inputs or variables on the Run pipeline pageMechanismPipeline inputs: ParametersPipeline inputs
a .gitlab-ci.yml created from the Web Editor's template list, matching a file in the administrators' templates projectMechanismInstance template repository: Organisation-wideInstance template repository
a .gitlab-ci.yml that matches a starter template, while the central copy has moved onMechanismProject and file templates: Organisation-wideProject and file templates
a job name that starts with a dot, such as .maven-base:MechanismHidden jobs: Inside one YAML fileHidden jobs
a job sees a variable that no file definesMechanismInstance and group CI/CD variables: Organisation-wideInstance and group CI/CD variables
a job with run: and a list of steps using func:, and no script:MechanismGitLab Functions: Outside the YAMLGitLab Functions
a merge request needs extra approvals because of security findings or licencesMechanismMerge request approval policies: Organisation-wideMerge request approval policies
a spec: inputs: header above a --- lineMechanismspec: inputs: Parametersspec: inputs
a template reads $JAVA_VERSION, and your file sets it under variables:MechanismVariables as parameters: ParametersVariables as parameters
a top-level default: block setting image, before_script, tags or retryMechanismdefault:: Sharing between jobsdefault:
a top-level variables: blockMechanismTop-level variables: Sharing between jobsTop-level variables
behaviour that changes with the runner: environment variables, mounted files or cloud permissions that no file setsMechanismRunner configuration: Outside the YAMLRunner configuration
curl --request POST …/trigger/pipeline with a trigger token, inside a scriptMechanismPipeline trigger API: Whole pipelinesPipeline trigger API
curl -sSL … | bash, or a git clone of a scripts repository, inside script:MechanismScripts fetched at run time: Outside the YAMLScripts fetched at run time
&defaults after a key, and *defaults later in the same fileMechanismYAML anchors and aliases: Inside one YAML fileYAML anchors and aliases
<<: *defaults inside a jobMechanismYAML merge key: Inside one YAML fileYAML merge key
extends: .maven-base or extends: [.base, .with-cache]Mechanismextends: Sharing between jobsextends
image: registry.example.com/devops/ci-tools:3.2 and a script that calls a command no repository definesMechanismToolbox images: Outside the YAMLToolbox images
include: - component: $CI_SERVER_FQDN/devops/components/ecr-push@1.4.0Mechanisminclude: component: Pulling in filesinclude: component
include: local: ci/build.yml, a wildcard such as ci/*.yml, or a bare path under include:Mechanisminclude: local: Pulling in filesinclude: local
include: - project: devops/ci-templates with ref: and file:Mechanisminclude: project: Pulling in filesinclude: project
include: remote: https://…/pipeline.ymlMechanisminclude: remote: Pulling in filesinclude: remote
include: template: Security/SAST.gitlab-ci.ymlMechanisminclude: template: Pulling in filesinclude: template
no .gitlab-ci.yml, yet jobs appear with names such as build, test, code_quality and container_scanningMechanismAuto DevOps: Organisation-wideAuto DevOps
parallel: matrix: in a job, and job names like test: [21] in the pipelineMechanismparallel: matrix: Sharing between jobsparallel: matrix
pipelines run but the repository has no .gitlab-ci.ymlMechanismCustom CI/CD configuration file: Organisation-wideCustom CI/CD configuration file
- !reference [.snippets, aws_login] inside a script or rules listMechanism!reference tags: Sharing between jobs!reference tags
scanner jobs named with a hyphen and a number, such as secret-detection-1MechanismScan execution policies: Organisation-wideScan execution policies
script: mc deploy --env prod: a company tool, not a public oneMechanismInternal command-line tools: Outside the YAMLInternal command-line tools
script: mvn -P ci verify with a <parent> POM owned by another teamMechanismBuild-tool reuse: Outside the YAMLBuild-tool reuse
source /opt/ci/lib.shMechanismShell function libraries: Outside the YAMLShell function libraries
stages named .pipeline-policy-pre or .pipeline-policy-postMechanismPipeline execution policies: Organisation-widePipeline execution policies
trigger: include: - artifact: generated.yml with job: generateMechanismDynamic child pipelines: Whole pipelinesDynamic child pipelines
trigger: include: ci/deploy.ymlMechanismParent–child pipelines: Whole pipelinesParent–child pipelines
trigger: project: platform/deployerMechanismMulti-project pipelines: Whole pipelinesMulti-project pipelines
variables such as SAST_DISABLED, SKIP_SONAR or DEPLOY_ENABLED, tested in rules: - if:MechanismToggle variables: ParametersToggle variables
Term
ACRTermAzure Container Registry, where MegaCorp keeps the images that aks-prod runschapter 38
agent for KubernetesTermGitLab's program inside a cluster, which connects the cluster to GitLabchapter 45
anchor, alias, merge keyTermYAML's own copy and paste, with & and *; it works only inside one filechapter 13
Application, ApplicationSetTermArgo CD's objects: an Application says what to deploy, from where and to where; an ApplicationSet makes many from one templatechapter 45
Argo CDTerma GitOps controller with a web UI, which syncs Applications from Git into a clusterchapter 41
artifactTermfiles a job keeps when it ends, for later jobs and for peoplechapter 31
AssumeRoleWithWebIdentityTermthe AWS call that swaps an ID token for temporary AWS keyschapter 38
Auto DevOpsTermGitLab's built-in pipeline, for projects with no pipeline file when the setting is onchapter 18
cacheTermfiles kept between jobs to save time; they may be missing at any timechapter 31
chartTerma Helm package: templates plus default values, which render Kubernetes objectschapter 43
child pipelineTerma pipeline that a trigger job starts from a file in the same projectchapter 17
CI LintTerma tool that checks a piece of pipeline YAMLchapter 4
componentTerma versioned piece of configuration with typed inputs, included with include: componentchapter 16
container registryTerma store for a project's container images, under Deploy › Container Registrychapter 8
container scanningTerma scanner that looks for known vulnerabilities in an imagechapter 33
controllerTerma program in the cluster that watches objects, and changes the cluster to match themchapter 41
default:Termsettings every job gets unless it sets its ownchapter 14
dependency scanningTerma scanner that looks for known vulnerabilities in the libraries a project useschapter 33
deployment jobTerma job with environment:, which records what it deployed, and wherechapter 40
Docker-in-DockerTerma Docker engine that runs as a service beside a job, so that the job can build imageschapter 8
dotenv reportTerma file of NAME=value lines that a job leaves as variables for later jobschapter 31
downstream pipelineTerma pipeline that another pipeline started: a child or a multi-project pipelinechapter 17
dynamic child pipelineTerma child pipeline whose YAML an earlier job wrotechapter 17
ECRTermAmazon Elastic Container Registry, where AWS keeps container imageschapter 38
Entra IDTermMicrosoft's identity service for Azure, formerly Azure Active Directorychapter 38
environmentTerma place a job deploys to, such as staging, with its history of deploymentschapter 40
environment repositoryTermthe Git repository that says what runs in each environment: image tags and settingschapter 45
executorTermhow a runner runs jobs: in Docker containers, in Kubernetes pods, in a shell, and otherschapter 29
extendsTermcopies another job's settings: maps are merged, lists are replacedchapter 14
External Secrets OperatorTerma controller that copies secrets from a cloud's vault into Kubernetes Secretschapter 50
federated credentialTerma rule on an Entra ID application or managed identity that says which outside tokens it accepts, matched exactlychapter 38
FluxTerma GitOps tool made of controllers, configured with objects such as GitRepository, Kustomization and HelmReleasechapter 45
Full configurationTermthe pipeline editor's view of every file, merged into onechapter 4
GitOpsTermdeploying by writing the wanted state to a Git repository, which a tool in the cluster applieschapter 41
hidden jobTerma job whose name starts with a dot; it never runs, and exists to be copiedchapter 13
IAM role, trust policyTerman AWS identity a job can take on; its trust policy says who maychapter 38
ID tokenTerma note about the job, signed by GitLab, that another system such as AWS can checkchapter 28
includeTermcopies other YAML files into the configurationchapter 15
inputTerma typed parameter of a component, an included file or a pipeline, filled in when the pipeline is createdchapter 16
instance, group and project runnersTermrunners for every project, for one group's projects, or for chosen projectschapter 29
jobTermone piece of work in a pipeline, run by one runnerchapter 4
job tokenTermCI_JOB_TOKEN, which lets a job call GitLab while it runschapter 28
Kustomize, overlayTerma tool that lays changes over a base set of objects, one overlay for each environment or clusterchapter 44
masked, hidden, protectedTermhow a variable is guarded: masked in logs, hidden in the settings, passed only to protected branches and tagschapter 27
merged results pipelineTerma merge request pipeline that runs on the source branch merged into the targetchapter 21
merge request approval policyTerma security policy that requires approval when scans find problemschapter 36
merge request pipelineTerma pipeline for a merge request, rather than for a branchchapter 21
merge trainTerma queue of merge requests, each tested together with those ahead of itchapter 21
multi-project pipelineTerma pipeline in another project, started by a trigger jobchapter 17
namespaceTerma named group of objects inside a cluster, with its own permissions and quotaschapter 54
needsTermstarts a job as soon as the named jobs finish, ignoring stageschapter 24
OIDCTermOpenID Connect, the standard that ID tokens followchapter 28
OpenTelemetryTermthe common standard for producing logs, metrics and traceschapter 52
OpenTofuTerman open-source tool that creates and changes infrastructure from code; it began as a copy of Terraformchapter 10
pipelineTermall the jobs GitLab runs for one commit, for one reason such as a pushchapter 4
pipeline execution policyTerma security policy that adds jobs to projects' pipelineschapter 36
pipeline sourceTermwhat started a pipeline, held in CI_PIPELINE_SOURCEchapter 21
plan, applyTermOpenTofu's two steps: work out what would change, then change itchapter 10
predefined variableTerma variable GitLab sets, such as CI_COMMIT_BRANCHpredefined variables
promotionTermmoving a version from one environment to the nextchapter 45
protected branch, protected tagTermbranches and tags that only some people may push to; only their pipelines get protected variableschapter 27
readiness probeTermthe check a new pod must pass before Kubernetes sends it traffic, or counts it in a rolloutchapter 51
!referenceTerma YAML tag that pastes one section of another job, such as its rules or scriptchapter 14
resource_groupTermlets only one job of the group run at a timechapter 24
rulesTermdecide whether a job is in a pipeline, and how it runschapter 23
runnerTermthe program that takes jobs from GitLab and runs themchapter 29
SASTTermstatic application security testing: a scanner that reads source codechapter 33
SBOMTermsoftware bill of materials: a list of every component in a buildchapter 33
scan execution policyTerma security policy that adds scanner jobs to pipelines, or runs them on a schedulechapter 36
secret detectionTerma scanner that looks for passwords and keys committed to the repositorychapter 33
security policy projectTermthe project that holds a group's security policieschapter 36
serviceTerman extra container beside the job, such as a databasechapter 30
settings variableTerma variable set in the CI/CD settings of a project, a group or the instancechapter 26
shallow cloneTerma copy of the repository with only the newest commitschapter 32
signalsTerma running service's logs, metrics and traceschapter 52
stageTerma group of jobs; by default, a stage starts only when the stage before it has succeededchapter 24
stateTermOpenTofu's record of what it has made, which GitLab can keep for a projectchapter 10
state, state lockTermOpenTofu's record of what it created, and the lock that stops two runs at oncechapter 48
syncTerma controller making the cluster match what Git sayschapter 45
tagTerma Git tag names a commit; a runner tag is a label that matches jobs to runnerschapter 29
templateTermYAML that GitLab ships, included with include: templatechapter 15
toggleTerma variable a template checks to switch a job off, such as SAST_DISABLEDchapter 16
trigger jobTerma job that starts a downstream pipeline instead of running a scriptchapter 17
truth tableTermthis book's method for predicting which jobs run in which pipelinechapter 25
values fileTerma Helm chart's settings for one service or environmentchapter 43
workflow:rulesTermdecide whether a pipeline is created at allchapter 22
workload identityTerma pod's own cloud identity, taken from its Kubernetes service accountchapter 49
Former name
CI/CD StepsFormer nameGitLab Functions, an experiment–
CI_JOB_JWT, CI_JOB_JWT_V2Former nameID tokens, requested with id_tokensremoved in 17.0
Code Quality's CodeClimate templateFormer nameany tool's results, imported as a codequality reportthe template is deprecated
compliance pipelinesFormer namepipeline execution policiesdeprecated in 17.3; removal planned for 20.0
Gemnasium, the dependency scanning analyzerFormer namedependency scanning using an SBOMGemnasium deprecated in 17.9; removal proposed for 20.0
image, services, cache, before_script or after_script at the top of the fileFormer namethe same keywords inside default:deprecated
inject_ci, in a pipeline execution policyFormer nameinject_policy17.9; inject_ci is deprecated
kaniko, for building imagesFormer nameDocker, Buildah, Podman or rootless BuildKitGitLab's page marks kaniko removed
Limit access to this project, then Authorized groups and projectsFormer nameCI/CD job token allowlist17.2, then 17.3
Merge when pipeline succeedsFormer nameauto-merge–
newly_detected, in a policy's vulnerability_statesFormer namenew_needs_triage and new_dismissed17.0
only, exceptFormer namerulesdeprecated
scan result policies, scan_result_policyFormer namemerge request approval policies, approval_policyonly approval_policy is accepted since 17.0
Security/SAST.gitlab-ci.yml and the other Security/ templatesFormer namewrappers: each only includes its Jobs/ twin, such as Jobs/SAST.gitlab-ci.yml (chapter 34)–
Token Access, in Settings › CI/CDFormer nameJob token permissions17.2

The card#

Part I · Handbook · Chapter 03 · 7 min read

The book on one page: what to ask about any job, where to click, which cards to open when something breaks, and the traps. Print it and keep it next to your screen.

Five questions for any job#

Ask them in this order (chapter 55):

QuestionLook at
Where is it defined?Full configuration, then its extends chain and includes; if it's in no file, policies
When does it run?workflow:rules, the job's final rules, and needs
Where does it run?the job page's runner and tags, and the job's image
What does it execute?its scripts, and the tools and functions those call
What goes in and out?variables and secrets; artifacts, caches, reports, images and deploys

Five questions for a deployment#

The same five, asked after the pipeline (chapter 55):

QuestionLook at
Where is it defined?the environment repository: the chart, its values, overlays and inline values
When does it deploy?the commit or merge request, then the controller's sync
Where does it run?the cluster and namespace, in the Application or Kustomization
What does it run?the rendered image reference, and whether that tag is in the cluster's registry
What goes in and out?settings and secrets in; logs, metrics and traces out

Four kinds of symptom#

SymptomCheck firstCards
Didn't run, or ran when it shouldn'tthe pipeline exists; the rules matched; everything it needs existsdidn't run
Failedthe log's first error; the image and runner; what went infailed
Passed, but did the wrong thingwhich value won; which image or artifact; what was deployedwrong result
Stuck, slow, or broke with no commitrunners online; tokens expired; central files or images changedstuck or drifting

Writing your team's pipeline#

  1. Start with a workflow: merge request pipelines while a merge request is open, branch pipelines otherwise (chapter 7).
  2. Pin every image, include and component to a version.
  3. Keep what later jobs need as artifacts; use the cache only for speed.
  4. Put secrets in settings, masked and protected, and expect them to be empty on branches.
  5. Tag each image with its commit, and deploy that one tag everywhere.
  6. Give every deploy job an environment, and make production manual.

Which variable value wins, highest first#

Security policies, then values given when the pipeline started, then the project's settings, then the groups', closest first, then the instance's. Below those come dotenv reports, then the job's own YAML, then top-level YAML, then deployment and predefined variables (chapter 26).

Where to look in GitLab#

PlaceShows
Build › Pipeline editor, Full configurationevery file and template, merged
Build › Pipeline editor, Validatethe jobs a pipeline would get, and problems with them
The pipeline graphwhich jobs exist, and any downstream pipelines
The job pagethe runner, the image, the timeout, and the log
Settings › CI/CD › Variables, Runners, General pipelinessettings variables, runners, clone depth and the pipeline file's location
Settings › CI/CD › Job token permissionswhich projects may use your job tokens
Secure › Policiesthe security policies that apply
Operate › Environmentsdeployments, approvals and rollback
Operate › Kubernetes clusters, Terraform statesthe agents that connect clusters; OpenTofu's states and their locks

Where to look after the pipeline#

PlaceShows
The environment repository's history and merge requestswhat should run where, and what waits for a merge
Argo CD's Application, or Flux's Kustomizationwhat the controller applied, and whether it's healthy
kubectl describe pod, and its eventswhy a pod won't pull, start or become ready
The service's dashboardswhether the deploy worked, after its mark

Commands worth remembering#

aws sts get-caller-identity                              # which AWS role is really in use
echo "$MC_ID_TOKEN" | cut -d '.' -f2 | base64 -d | jq .  # an ID token's claims, not the token
env | cut -d= -f1 | sort                                 # which variables exist: names only
git fetch --unshallow                                    # full history, for scripts that need it
az account show                                          # who the Azure CLI is signed in as
helm template payments-api ./service -f values.yaml      # what a chart renders
kustomize build overlays/staging                         # what an overlay builds
kubectl rollout status deployment/payments-api -n payments-staging   # did the rollout finish?
kubectl logs <pod> --previous                            # a crashed container's last output

Ten traps#

  1. A job's own rules, before_script, tags or services replace the inherited list completely.
  2. A settings variable beats anything in the YAML.
  3. A job's variables, including those from extends, beat top-level ones.
  4. Toggles never stop a security policy's jobs.
  5. GitLab's scanners skip merge request pipelines unless AST_ENABLE_MR_PIPELINES is "true".
  6. Protected values reach only protected branches and tags.
  7. A cache can miss at any time; only artifacts are guaranteed.
  8. Jobs get a shallow clone of their own ref only.
  9. With GitOps, green means written to Git, not running.
  10. A pipeline that rebuilds an image ships something nobody tested.

Four traps after the pipeline#

  1. Healthy means the pods pass their probes, not that requests succeed.
  2. A running pod keeps the settings and secrets it started with.
  3. Entra ID matches a federated credential's subject exactly, with no wildcards.
  4. In Argo CD, inline values beat value files.

How GitLab thinks#

Part II · Orientation · Chapter 04 · 24 min read

A pipeline is one run of your configuration: jobs, grouped in stages, each job a script that a runner executes in a container. In a large organisation a job is rarely written in one place. It is assembled from your file, the central team's files, settings that live in no file, and the image it runs in, at four moments in turn. The first two you can see on one page, Full configuration. Nearly every question about a job is one of five: where is it defined, when does it run, where does it run, what does it execute, and what goes in and out.

Tuesday, 09:12. Your first week owning payments-api.

You add a job called smoke-test to the end of the file, give it stage: smoke, and push. Nothing runs. The commit page shows a red pipeline with no jobs in it. At the top, one line: smoke-test job: chosen stage smoke does not exist; available stages are .pre, build, test, package, scan, deploy, .post. You never wrote that list of stages. Where did it come from, and why does one wrong word stop everything? (the card)

The answer is the shape of the whole system, so this chapter gives you the shape. Each piece gets its own chapter later; here you only need to recognise it.

The engine: six words#

When you push, GitLab starts a pipeline: one run of the configuration, for one commit and one event, such as a push or a merge request. A pipeline is made of jobs, each a script with its settings. Jobs are grouped in stages, which run in the order listed, while the jobs inside one stage run at the same time. GitLab schedules jobs but doesn't run them. A runner, a program registered with GitLab, keeps asking for work. It runs each job with its executor, usually a fresh container started from the job's image. On the pipeline's page, stages are the columns and jobs the boxes; a job's own page holds its log, and names the runner that took it.

Where each word shows up#

TermWhat it isWhere you see it
Pipelineone run of the configuration, for one commit and one eventthe pipeline's page, with its graph
Stagea named group of jobs; stages run in the order they are listedthe columns of the pipeline graph
Joba script to run, with its settingsa box in the graph; the job's own page holds its log
Runnera program that takes jobs from GitLab and runs themthe job's page names it
Executorhow the runner runs a job: a container, a Kubernetes pod, or a plain shellthe first lines of the job log
Imagethe container image a job runs inthe first lines of the job log

Three things about names catch people. Any top-level key in the configuration is a job, except GitLab's own keywords such as stages, variables and include, so you can't have a job called services. A name that starts with a dot, such as .maven-base, is a hidden job that never runs: it exists to be copied by others (chapter 14). And two definitions of the same name, one in your file and one in an included file, merge into one job. Names are at most 255 characters, and true, false and nil work as names only in quotes.

One more word means two things. A Git tag, such as v1.4.0, names a commit and starts tag pipelines. A runner tag, written with the tags: keyword, chooses which runner takes a job. In the file, tags: always means runner tags.

Stages, and the two mistakes that stop a pipeline#

If a configuration never lists its stages, GitLab uses five: .pre, build, test, deploy and .post. A job that names no stage lands in test. A project can declare its own list with stages:, and MegaCorp's shared pipeline does: build, test, package, scan, deploy. That list is what your Tuesday error quoted, and it lives in a file your project pulls in, not in your file. When a job fails, the stages after it usually don't start:

A small pipeline where a test failed, so the deploy stage never startedbuild✓compiletest✓lint✕unit-testsdeploy»deploy-review
Pipeline A small pipeline where a test failed, so the deploy stage never started#
As text
  1. build: compile (passed)
  2. test: lint (passed), unit-tests (failed)
  3. deploy: deploy-review (skipped)

Two stage mistakes stop GitLab from creating a pipeline at all, and both produce your Tuesday message. A job names a stage that isn't in the list, as smoke-test did. Or a job names no stage, lands in test, and the project's list leaves test out. The fix for yours: use a listed stage, or add smoke to the list, in the file that owns the list.

Where a job comes from#

Picture a job as an onion. Your project's file is at the centre, and each layer around it can add to it or change it:

The layers that make up one job, from your own file outwards. Each outer layer can add to the ones inside it, or override them. 4 · Run time: the runner's configuration and the job's image 3 · The organisation: security policies, and instance, group and project variables 2 · The central team's templates: include, components, extends, !reference, default: 1 · Your project's .gitlab-ci.yml the job in your pipeline every layer, merged
Figure The layers that make up one job, from your own file outwards. Each outer layer can add to the ones inside it, or override them.#
  1. Your project's .gitlab-ci.yml. The file your team writes. In a big organisation it is often twenty lines: an include, a few settings, and perhaps one job adjusted.
  2. The central team's shared files. YAML files that your file pulls in with include. Most of your jobs are really defined there, as hidden jobs that your jobs copy with extends, snippets pasted with !reference, and a default: block that applies to every job. Your stage list came from here.
  3. Organisation settings. CI/CD variables set on the settings pages of the project, its groups, and the whole installation, called the instance. They beat anything written in YAML. On the top tier, security policies can add jobs from here too, to every pipeline, from outside any file.
  4. Run time. The runner's own configuration, and the image the job runs in. The real logic of a script often lives inside the image.

Four moments, and what you can see#

GitLab doesn't apply the layers all at once. It reads each file, joins the files, creates the pipeline, and then runs each job. The moment decides where you can look:

When GitLab resolves each kind of reuse, and the boundary of what the Full configuration view shows. earlier later Parse inside one YAML file Compile assembling the files Create creating the pipeline Run running each job &anchor, *alias <<: merge keys .hidden-jobs include (every kind) components, inputs extends !reference default: top-level variables workflow: and rules: parallel: matrix settings variables pipeline inputs execution policies scan policies the job's image runner configuration fetched scripts shell libraries build-tool config child pipelines approval policies the Full configuration view shows these trace these another way: graph, log, settings
Figure When GitLab resolves each kind of reuse, and the boundary of what the Full configuration view shows.#
MomentWhat happensWhere to see the result
Parse: each file is readYAML's own copy-and-paste is applied: anchors, aliases and merge keysBuild › Pipeline editor › Full configuration
Compile: the files are joinedincluded files are copied in; extends and !reference copy settings between jobsthe same view
Create: the pipeline is maderules decide which jobs exist; a matrix job becomes several; settings variables and policies applythe pipeline graph, and the settings pages
Run: a runner runs one jobthe image, the runner's configuration and any downloaded scripts do their partthe job log

So if what you need happens in the first two moments, one page shows it: Full configuration, with every include, extends and !reference already resolved. Two details of that view surprise people. A component's jobs appear with their inputs already filled in. And the default: block appears on its own, not copied into each job, so a job's final image may be there rather than in the job.

Five questions for any job#

When a job surprises you, the question you are asking is almost always one of these, and each has a usual place to look:

QuestionWhere the answer usually isCovered in
Where is it defined?Full configuration, then the extends chain, then the included file; if it is in no file, a policyPart IV
When does it run?workflow:rules, the job's rules, needs and stagesPart V
Where does it run?the job's runner tags, the runner's executor and the job's imagePart VII
What does it execute?before_script and script, and whatever those call inside the imagereuse outside the YAML
What goes in and out?variables and secrets in; artifacts, caches, reports, images and deployments outPart VI and Part VII

Here is the idea as a real pipeline: a merge request pipeline for payments-api, a service at MegaCorp, the made-up company this book uses throughout. The project's own file defines none of these jobs. Three are copied from the central team's hidden jobs, one comes from a component, one from a file GitLab itself ships, and policy-sbom is in no file at all: a security policy adds it.

A payments-api merge request pipeline. On GitLab Ultimate, MegaCorp's security policy also adds policy-sbom.buildmaven-buildtestmaven-test: [17]maven-test: [21]!sonar-scanpackageimage-buildscan!container_scanning.pipeline-policy-postpolicy-sbompolicy
Pipeline A payments-api merge request pipeline. On GitLab Ultimate, MegaCorp's security policy also adds policy-sbom.#
As text
  1. build: maven-build
  2. test: maven-test: [17], maven-test: [21], sonar-scan (allow failure)
  3. package: image-build
  4. scan: container_scanning (allow failure)
  5. .pipeline-policy-post: policy-sbom (policy)

Needs: image-build waits only for maven-build; container_scanning waits only for image-build.

From a push to a pipeline#

Between your push and the first job, GitLab does a fixed sequence of things, and each step can stop the pipeline in a different way. Your Tuesday stopped at the validity check, which is why you got a red pipeline with an error. A workflow that says no stops silently instead, and so does a pipeline in which no job's rules matched.

What GitLab does between a push and the first job. Each question is a point where a pipeline can stop, and each stop looks different.noyesnoyesnoyesYou push a commitGitLab finds the configuration:.gitlab-ci.yml, or the file thesettings nameIt fetches every include, and mergeseverything into one configurationIt resolves extends, !reference andanchorsIs the configuration valid?A failed pipeline, with the errorshown at the topDoes workflow: rules allow thispipeline?No pipeline, and after a push noerrorIt checks each job's rules, expandsmatrices, and adds any policy jobsIs at least one job left?No pipelineGitLab creates the pipeline, andrunners start taking its jobs
Figure What GitLab does between a push and the first job. Each question is a point where a pipeline can stop, and each stop looks different.#
As text
  1. You push a commit
  2. GitLab finds the configuration: .gitlab-ci.yml, or the file the settings name
  3. It fetches every include, and merges everything into one configuration
  4. It resolves extends, !reference and anchors
  5. Is the configuration valid? No: A failed pipeline, with the error shown at the top. Yes: the next step.
  6. Does workflow: rules allow this pipeline? No: No pipeline, and after a push no error. Yes: the next step.
  7. It checks each job's rules, expands matrices, and adds any policy jobs
  8. Is at least one job left? No: No pipeline. Yes: the next step.
  9. GitLab creates the pipeline, and runners start taking its jobs

Here is the whole assembly for one job. payments-api's file is short, and the job GitLab builds from it is not:

You wrotemegacorp/payments/payments-api · .gitlab-ci.yml
include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file: pipelines/java-service.yml
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api
You wrotemegacorp/devops/ci-templates · pipelines/java-service.yml
maven-build:
  extends: .maven-build

maven-test:
  extends: .maven-test

image-build:
  extends: .image-build
  needs: [maven-build]
GitLab buildsmerged configurationcaptured from GitLab
maven-build:
  variables:
    MC_TEAM: unknown
    JAVA_VERSION: '21'
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  artifacts:
    expire_in: 7 days
    paths:
    - target/*.jar
  rules:
  - - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
  - - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  - - if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
  - - if: "$CI_COMMIT_TAG =~ /^v\\d+\\.\\d+\\.\\d+$/"
  extends: ".maven-build"
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
  cache:
    key:
      files:
      - pom.xml
    paths:
    - ".m2/repository"
  before_script:
  - - source /opt/megacorp/lib/ci-lib.sh
  - - mkdir -p .m2
    - mc_maven_settings > .m2/settings.xml
  stage: build
  script:
  - mvn $MAVEN_CLI_OPTS -DskipTests package

Every line of the result came from a file payments-api never opens. The variables, the image, the cache, the rules and the before_script all arrived through the include and the extends chain, which chapter 14 walks one link at a time.

The pipeline editor, and CI Lint#

You can watch the first four steps without pushing anything. The pipeline editor is at Build › Pipeline editor. It edits the project's file, and its tabs show what GitLab would do with it:

A sketch of the pipeline editor, at Build › Pipeline editor, with the Full configuration tab selected. It shows payments-api's configuration with every include, extends and !reference resolved. megacorp / payments / payments-api › Build › Pipeline editor Code Build Pipelines Pipeline editor Pipeline schedules Settings Branch: main ▾ Edit Visualize Validate Full configuration every include, extends and !reference resolved maven-build: variables: MC_TEAM: unknown extends: ".maven-build" image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION} … Commit message, branch, and Commit changes
Figure A sketch of the pipeline editor, at Build › Pipeline editor, with the Full configuration tab selected. It shows payments-api's configuration with every include, extends and !reference resolved.#
TabWhat it showsUse it to
Editthe file, checked against GitLab's schema as you typecatch syntax mistakes before you commit
Visualizeevery stage and job, with needs drawn as lines between jobssee the shape of the pipeline
Validatea simulated pipeline for a push to the branch you choosefind problems with rules and needs before you push
Full configurationthe whole configuration as one file: includes copied in, extends merged, anchors and !reference replacedfind where a line of a job really comes from

Full configuration shows one scenario: anything conditional is decided as if the pipeline were a push to the default branch. Content pasted by !reference can appear as a list inside a list, with lines that start - -; GitLab documents that as expected. And the view can't show what happens later: rules, matrices, policies, settings variables and everything the runner does come after it.

CI Lint#

On the same Validate tab, Lint CI/CD sample checks configuration you paste in, so you can test a change without touching the project's file. Select Simulate pipeline creation for the default branch to go past syntax: the simulation evaluates the rules as a push to the default branch, and finds the problems a real push would hit. Every error message quoted in this book came from that check, run on the local GitLab 19.3. Your Tuesday error would have shown here before you pushed.

Reading a job log#

A job's log is written top to bottom in a fixed order, and the job page lets you fold each part away. It starts with the runner's name and the executor. Then come Preparing environment, which says where the job runs, and Getting source, with the fetch and the commit, and Restoring cache. Then the script itself, each line echoed after a $ followed by its output. Then after_script, saving the cache, uploading artifacts, and finally Job succeeded or ERROR: Job failed. On the right, Job details names the runner, the job's timeout and where that came from, and its tags.

Every job shows a status, and the graph is read through them. A job is created while its stage hasn't started, pending while it waits for a runner, preparing and then running once one has it, and ends success or failed. Around those come five more. Canceled means someone or something stopped it, and skipped that an earlier stage failed. Manual waits for a person, scheduled is a delayed job counting down, and waiting_for_resource means another job holds the resource it needs. In a stage, jobs are sorted alphabetically, not by when they started or where they were defined.

When a job fails, read the log from the end backwards: the last command echoed is the one that failed, because a failing command stops the script. When a job does the wrong thing, read the log from the top, because the first lines tell you where it really ran. A job that sits in pending for ever is almost always a runner problem: no runner has the tags the job asks for (chapter 29).

Which tool answers which question#

You want to knowUseIt can't tell you
what all the files add up toFull configurationanything decided when the pipeline is created
which jobs a push to a branch would createValidate, choosing the branchmerge request or scheduled pipelines
whether a snippet is valid, or what it would doCI Lint, with simulationanything about pipelines other than a default-branch push
which jobs exist, and which downstream pipelines ranthe pipeline graphwhich file a job came from
what ran, where, and what it printedthe job logvalues set in settings, or what a policy added
values no file showsSettings › CI/CD › Variables, on the project, its groups and the instancenothing: this is where they live
What you can now do
  1. Read your Tuesday error as GitLab meant it: the stage list lives in the central team's file, and you either use one of its stages or add yours there.
  2. Open Full configuration and see every job as GitLab assembled it, and say which of the four moments produced any line you don't recognise.
  3. Ask the five questions about any job, and know which page answers each.

Meet MegaCorp#

Part II · Orientation · Chapter 05 · 9 min read

MegaCorp is a made-up company, and so is everything in its GitLab. It is built the way large organisations build theirs:

  • A central DevOps team writes the shared pipeline files, the ready-made building blocks called components, and the container image most jobs run in.
  • The security team writes policies that add scans to every pipeline.
  • The platform team runs every deployment.
  • Each application team keeps a short file that pulls in everything else.

Every example in this book is a real file from MegaCorp's projects, labelled with its project and path. The same files are loaded into a real GitLab 19.3 instance, and every piece of GitLab output in the book comes from there.

Tuesday, 11:40. Still your first week.

A deploy job in payments-api calls the deployer project through the API and gets 404 Not Found. The project exists: you can open it in the browser. A colleague says "ask platform, it's their project". Who owns what at MegaCorp, and which project is allowed to talk to which? (the card)

The map#

Each box is a GitLab project, and the arrows show who uses whom:

MegaCorp's projects: who includes, triggers or runs what, and what reaches every project without appearing in any file. application projects payments/payments-api one include and one component web/web-portal no .gitlab-ci.yml at all data/mono local includes, child pipelines legacy/billing-batch anchors, remote include, trigger API the central DevOps team devops/ci-templates pipelines/ java-service.yml · web.yml templates/ base · snippets · rules · workflow java-maven · node · container security · deploy devops/ci-tools toolbox image: ci-lib.sh and mc devops/components sonar-scan, ecr-push, gitops-deploy platform/deployer deploys every service reach every project, in no file security/policies scans, SBOM job, approvals instance, group settings variables, Auto DevOps runner configuration config.toml on the runner hosts include: project include: component config-file setting include: remote image: trigger: project
Figure MegaCorp's projects: who includes, triggers or runs what, and what reaches every project without appearing in any file.#

The projects#

You don't need to follow every word in this table yet. The last column says where each project is explained.

ProjectOwned byHoldsYou'll meet it in
megacorp/devops/ci-templatescentral DevOpsthe shared pipeline files: job templates, script snippets, rules, and two complete pipelinesPart IV
megacorp/devops/componentscentral DevOpscomponents, which are versioned building blocks with settings: sonar-scan, ecr-push and gitops-deploycomponents and inputs
megacorp/devops/ci-toolscentral DevOpsthe toolbox image most jobs run in, its shell functions, and MegaCorp's own mc commandreuse outside the YAML
megacorp/security/policiessecuritythe security policies that add scans and gates to other projects' pipelinespolicies and gates
megacorp/platform/deployerplatformthe deployment pipeline that every service startsGitOps repositories and promotion
megacorp/platform/chartsplatformthe Helm chart that every service is deployed withHelm
megacorp/platform/gitops-configplatformthe environment repository: what runs on eks-prod and aks-prod, and each environment's valuesGitOps repositories and promotion
megacorp/platform/infra-modules, infra-liveplatformthe OpenTofu that builds both clusters, their registries and their identitiesOpenTofu and Terraform
megacorp/payments/payments-apithe payments teama Java service: one include, one component, two adjustmentsthroughout
megacorp/web/web-portalthe web teama Node front end with no .gitlab-ci.yml at allorg-wide reuse
megacorp/data/monothe data teama monorepo, one repository holding many services, that generates its own child pipelinespipelines as building blocks
megacorp/legacy/billing-batchnobody, any morean old file from 2019, using older techniques: YAML anchors, a file included from a URL, and a trigger tokenYAML-level reuse

A service's whole pipeline file#

This is all the configuration the payments team maintains, apart from two small adjustments further down the same file:

megacorp/payments/payments-api.gitlab-ci.ymlinclude
include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file: pipelines/java-service.yml
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api

include pulls another file's contents into this one, as if they had been typed here. The first include is MegaCorp's "golden pipeline" for Java services: a complete, ready-made pipeline that any Java project can take whole. This is what it pulls in:

megacorp/devops/ci-templatespipelines/java-service.ymlincludes
include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file:
      - templates/base.yml
      - templates/snippets.yml
      - templates/rules.yml
      - templates/workflow.yml
      - templates/java-maven.yml
      - templates/container.yml
      - templates/security.yml
      - templates/deploy.yml

That is eight shared files, all pinned to the tag v4.2.0. Pinned means GitLab reads them as they were at that tag. A later change by the central team doesn't reach payments-api until someone moves the tag in this file.

Those files define hidden jobs. Their names start with a dot, and they never run by themselves. The golden pipeline turns three of them into real jobs:

megacorp/devops/ci-templatespipelines/java-service.ymljobs
maven-build:
  extends: .maven-build

maven-test:
  extends: .maven-test

image-build:
  extends: .image-build
  needs: [maven-build]

So there are four files between payments-api and the script its maven-test job runs. Chapter 14 walks that chain one link at a time.

What no file in the project shows#

Some of MegaCorp's configuration isn't in any repository the payments team can read:

  • Settings. Variables set on the instance and on the megacorp group, and web-portal's pointer to a pipeline file in another project. The book shows them in files called SETTINGS.yml, which GitLab never reads.
  • Policies. On GitLab Ultimate, the security team's policy project adds a secret-detection scan to every pipeline. It also adds a job that lists everything the build contains, known as an SBOM (software bill of materials).
  • The runners. Their configuration sets environment variables and cloud permissions for every job.

Here are the variables set on the megacorp group. Every project in the group receives them:

settingsgroup-megacorp.ymlgroup-variables
# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
  - key: MAVEN_CLI_OPTS
    value: "--batch-mode --errors --show-version -s .m2/settings.xml"
  - key: AWS_ACCOUNT_ID
    value: "123456789012"
  - key: MC_AZURE_TENANT_ID
    value: "00000000-0000-0000-0000-000000000000"
  - key: MC_ACR_REGISTRY
    value: megacorpprod.azurecr.io
  - key: MC_ACR_PUSH_CLIENT_ID
    value: "00000000-0000-0000-0000-000000000000"
    # the gitlab-acr-push application in Entra ID, which mc_acr_push signs in as
  - key: SONAR_HOST_URL
    value: https://sonar.example.com
  - key: SONAR_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true

Keep MAVEN_CLI_OPTS in mind. It matters in chapter 26.

Reading the specimen#

  • Every file opens with a comment saying what it is and which chapters use it.
  • Every quote is labelled with its project and path. When only part of a file is shown, the label also names the part, called a region.
  • Files named SETTINGS.yml describe settings made in GitLab's web pages. GitLab never reads them.
  • Every pipeline file in the specimen is valid. On the local GitLab, each one creates its pipelines. The deployer creates one only when a service passes it an image to deploy, by design. Like any real setup, the specimen also hides a few surprises, and the exercises find them.

The full, annotated listing is in the appendix.

What you can now do
  1. Name the seven MegaCorp projects a payments-api pipeline touches, and which team owns each.
  2. Read a 404 from another project as a permissions question, not a missing project, and know whose settings page to ask about.
  3. Find in the specimen the file behind any job you meet in the rest of the book.

Build and test#

Part III · Build your team's pipeline · Chapter 06 · 8 min read

Your team's pipeline starts with two jobs: one builds the service, and one tests it. Each job runs in a fresh container. So the build job keeps what it made as an artifact, which later jobs receive, and both jobs keep downloaded dependencies in a cache, which later pipelines reuse. The test job hands its results to GitLab as a report, so failing tests show on the pipeline and in merge requests.

Wednesday, 10:05. greeting-service's first pipeline.

A one-line change to the README, and the build job downloads every dependency again, from scratch, for four minutes. Yesterday's run had them. Its log says Checking cache for default-protected... and then nothing was found. Yesterday the key had the lockfile's name in it. What is default, and where did yesterday's cache go? (the card)

This part grows one service's pipeline, a chapter at a time. The service is greeting-service, a small web service that answers GET /greet?name=Ada with a greeting. It comes in Node and in Java: pick yours with any Node or Java tab, and every tab in the book follows. Each chapter ends with a lab that you do on your own GitLab. Nothing here needs the rest of the book; where a chapter uses something that a later chapter explains fully, it links there.

Two jobs, two stages#

Here is the pipeline this chapter builds. Each column is a stage, and a stage starts only when the stage before it has passed:

greeting-service at the end of this chapterbuildbuildtestunit-tests
Pipeline greeting-service at the end of this chapter#
As text
  1. build: build
  2. test: unit-tests
practice · node-servicesteps/1-build-and-test.ymlpipeline
stages:
  - build
  - test

default:
  image: node:24.21.0-alpine

build:
  stage: build
  script:
    - npm ci --cache .npm --prefer-offline
    - npm run build
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
  artifacts:
    paths:
      - dist/

unit-tests:
  stage: test
  script:
    - npm ci --cache .npm --prefer-offline
    - npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=junit.xml
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
    policy: pull
  artifacts:
    when: always
    reports:
      junit: junit.xml
practice · java-servicesteps/1-build-and-test.ymlpipeline
stages:
  - build
  - test

default:
  image: maven:3.9.16-eclipse-temurin-25

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  MAVEN_CLI_OPTS: "--batch-mode --no-transfer-progress"

build:
  stage: build
  script:
    - mvn $MAVEN_CLI_OPTS package -DskipTests
  cache:
    key:
      files:
        - pom.xml
    paths:
      - .m2/repository/
  artifacts:
    paths:
      - target/greeting-service.jar

unit-tests:
  stage: test
  script:
    - mvn $MAVEN_CLI_OPTS test
  cache:
    key:
      files:
        - pom.xml
    paths:
      - .m2/repository/
    policy: pull
  artifacts:
    when: always
    reports:
      junit: target/surefire-reports/TEST-*.xml

From the top:

  • stages lists the stages in order.
  • default: image gives every job the same image unless a job names its own: Node 24, on Alpine Linux. The version is pinned, so the job behaves the same next month.
  • build installs the dependencies and builds. npm ci installs exactly what the lockfile, package-lock.json, lists. npm run build then writes the service to dist/.
  • unit-tests installs the dependencies again, because it runs in a new container, and runs the tests with npm test.
  • stages lists the stages in order.
  • default: image gives every job the same image unless a job names its own: Maven 3.9 with Java 25. The version is pinned, so the job behaves the same next month.
  • variables sets two values for Maven. MAVEN_CLI_OPTS holds the options that every mvn command in the scripts is given: --batch-mode runs Maven without prompts, and --no-transfer-progress keeps download progress out of the log. MAVEN_OPTS is different: no script mentions it, because Maven itself reads it from the environment. It tells Maven to keep its local repository, the folder where downloaded dependencies go, inside the project, which is what the cache below keeps.
  • build compiles and packages. Maven's package compiles the code and makes the jar, target/greeting-service.jar, and -DskipTests leaves the tests to the next job.
  • unit-tests downloads the dependencies again, because it runs in a new container, and runs the tests with mvn test.

Artifacts: what later jobs receive#

artifacts: paths tells GitLab which files to keep when the job ends.

The build keeps dist/, the folder that npm run build writes.

The build keeps target/greeting-service.jar, the jar that Maven packages.

GitLab uploads them, and every job in a later stage downloads them before its script starts. That is how a later job, such as one that builds an image, gets the build's output without building again.

Without artifacts, the files vanish with the container, as the third break in your first pipeline showed. Chapter 31 covers artifacts in full.

Cache: what later pipelines reuse#

Every job downloads its dependencies again. A cache keeps them between pipelines, so that the next job starts from what the last one downloaded:

  • cache: key: files names the file whose content decides the cache: package-lock.json. Change the dependencies, and the key changes, so no job restores a stale cache.
  • cache: paths is what is kept: npm's download folder, .npm/. The jobs point npm at it with --cache .npm, and --prefer-offline makes npm use a package it already has there before asking the registry for it.
  • policy: pull on unit-tests means it reads the cache but never writes it back. Only build saves it.
  • cache: key: files names the file whose content decides the cache: pom.xml. Change the dependencies, and the key changes, so no job restores a stale cache.
  • cache: paths is what is kept: Maven's local repository, .m2/repository/. That is the folder MAVEN_OPTS points Maven at. Without that variable, Maven would download into the container's home folder, and a cache can only keep files inside the project.
  • policy: pull on unit-tests means it reads the cache but never writes it back. Only build saves it.

A pipeline's first run finds no cache. The next one restores it, and its log says so. A cache can still miss at any time, for example on a different runner, so a job must work without it: a cache makes jobs faster, never correct. Chapter 31 explains why.

Test results GitLab can read#

artifacts: reports: junit hands GitLab a file of test results in the JUnit format. GitLab shows them on the pipeline's Tests tab, and in a merge request's test summary.

The test runner writes the file itself: --test-reporter=junit with junit.xml as its destination, beside the usual output to the terminal.

Maven's test plugin, Surefire, writes one file per test class under target/surefire-reports/, and the pattern collects them all.

when: always keeps the report even when tests fail, which is when you need it most.

When you write it
Do this
Key the dependency cache on your lockfile, with cache: key: files, and give every job that only reads the cache policy: pull.
Why
The key changes whenever the dependencies change, so a job never restores an old cache by mistake, and jobs that only read never overwrite what the build saved.
Watch out
One fixed key, shared by every branch and every job, hands jobs files that are old or half-written. The card a job restores stale files shows what that looks like.
What you can now do
  1. Write a two-job pipeline that builds, tests and keeps its results, in your stack.
  2. Read a build job's log and say whether the cache was restored, and under which key.
  3. Tell an artifact from a cache, and choose the right one for a file a later job needs.
LabBuild and test greeting-serviceabout 15 min#
You need

An account on your GitLab, and the service's files for your stack. Copy them from the panes below, keeping their paths.

practice · node-servicepackage.json
{
  "name": "greeting-service",
  "version": "1.0.0",
  "private": true,
  "description": "The Node service whose pipeline the Build your team's pipeline labs grow.",
  "type": "module",
  "scripts": {
    "build": "node build.js",
    "start": "node dist/server.js",
    "test": "node --test"
  },
  "dependencies": {
    "zod": "4.6.4"
  }
}
practice · node-servicesrc/greet.js
// Builds the greeting. zod checks the name before it is used.
import { z } from "zod";

const Name = z.string().trim().min(1).max(40);

export function greet(name) {
  return `Hello, ${Name.parse(name)}!`;
}
practice · node-servicesrc/server.js
// A tiny HTTP service: GET /greet?name=Ada answers {"message":"Hello, Ada!"}.
import { createServer } from "node:http";
import { greet } from "./greet.js";

const port = Number(process.env.PORT ?? 8080);

createServer((req, res) => {
  const url = new URL(req.url, "http://localhost");
  if (url.pathname !== "/greet") {
    res.writeHead(404).end();
    return;
  }
  try {
    const body = JSON.stringify({ message: greet(url.searchParams.get("name") ?? "world") });
    res.writeHead(200, { "content-type": "application/json" }).end(body);
  } catch {
    res.writeHead(400).end();
  }
}).listen(port, () => console.log(`listening on ${port}`));
practice · node-servicebuild.js
// The build: copies src/ into dist/, the way a real build would bundle it.
import { cpSync, mkdirSync } from "node:fs";

mkdirSync("dist", { recursive: true });
cpSync("src", "dist", { recursive: true });
console.log("built dist/");
practice · node-servicetest/greet.test.js
// The service's tests, run with node --test.
import { test } from "node:test";
import assert from "node:assert/strict";
import { greet } from "../src/greet.js";

test("greet says hello", () => {
  assert.equal(greet("Ada"), "Hello, Ada!");
});

test("greet refuses an empty name", () => {
  assert.throws(() => greet("  "));
});
practice · node-serviceDockerfile
# The greeting service's image: its built files and its production dependencies.
FROM node:24.21.0-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY dist/ dist/
EXPOSE 8080
CMD ["node", "dist/server.js"]

Then run npm install --package-lock-only once, to create package-lock.json.

practice · java-servicepom.xml
<?xml version="1.0" encoding="UTF-8"?>
<!-- The Java service whose pipeline the Build your team's pipeline labs grow. -->
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example</groupId>
  <artifactId>greeting-service</artifactId>
  <version>1.0.0</version>
  <packaging>jar</packaging>

  <properties>
    <maven.compiler.release>25</maven.compiler.release>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter</artifactId>
      <version>6.1.3</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <build>
    <finalName>greeting-service</finalName>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.16.0</version>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.6.0</version>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>3.5.1</version>
        <configuration>
          <archive>
            <manifest>
              <mainClass>com.example.greeting.Server</mainClass>
            </manifest>
          </archive>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>
practice · java-servicesrc/main/java/com/example/greeting/Greeter.java
package com.example.greeting;

/** Builds the greeting, after checking the name. */
public final class Greeter {
    private Greeter() {
    }

    public static String greet(String name) {
        String n = name == null ? "" : name.strip();
        if (n.isEmpty() || n.length() > 40) {
            throw new IllegalArgumentException("a name is 1 to 40 characters");
        }
        return "Hello, " + n + "!";
    }
}
practice · java-servicesrc/main/java/com/example/greeting/Server.java
package com.example.greeting;

import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.URLDecoder;
import java.nio.charset.StandardCharsets;

/** A tiny HTTP service: GET /greet?name=Ada answers {"message":"Hello, Ada!"}. */
public final class Server {
    private Server() {
    }

    public static void main(String[] args) throws IOException {
        int port = Integer.parseInt(System.getenv().getOrDefault("PORT", "8080"));
        HttpServer server = HttpServer.create(new InetSocketAddress(port), 0);
        server.createContext("/greet", exchange -> {
            String query = exchange.getRequestURI().getRawQuery();
            String name = "world";
            if (query != null && query.startsWith("name=")) {
                name = URLDecoder.decode(query.substring(5), StandardCharsets.UTF_8);
            }
            int status = 200;
            String body;
            try {
                String message = Greeter.greet(name).replace("\\", "\\\\").replace("\"", "\\\"");
                body = "{\"message\":\"" + message + "\"}";
            } catch (IllegalArgumentException e) {
                status = 400;
                body = "";
            }
            byte[] bytes = body.getBytes(StandardCharsets.UTF_8);
            exchange.getResponseHeaders().set("content-type", "application/json");
            exchange.sendResponseHeaders(status, bytes.length == 0 ? -1 : bytes.length);
            try (OutputStream out = exchange.getResponseBody()) {
                out.write(bytes);
            }
        });
        server.start();
        System.out.println("listening on " + port);
    }
}
practice · java-servicesrc/test/java/com/example/greeting/GreeterTest.java
package com.example.greeting;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;

import org.junit.jupiter.api.Test;

class GreeterTest {
    @Test
    void greetsByName() {
        assertEquals("Hello, Ada!", Greeter.greet("Ada"));
    }

    @Test
    void refusesAnEmptyName() {
        assertThrows(IllegalArgumentException.class, () -> Greeter.greet("  "));
    }
}
practice · java-serviceDockerfile
# The greeting service's image: the jar the build job made, on a Java runtime.
FROM eclipse-temurin:25.0.4_7-jre-alpine
COPY target/greeting-service.jar /app/greeting-service.jar
EXPOSE 8080
CMD ["java", "-jar", "/app/greeting-service.jar"]

Both stacks also have docs/README.md, any short text about the service.

Steps
  1. Create a blank project named greeting-service, clone it, and copy the files in.
  2. Add .gitlab-ci.yml with the pipeline above, for your stack. Commit and push.
  3. Open the pipeline, then build's log, and find where it saved the cache and uploaded the artifacts.
  4. Change a line in docs/README.md, and push again. Open the new pipeline's build log and look for the cache.
  5. Open the pipeline's Tests tab.
What you should see

Both pipelines pass. In the second, build finds the cache that the first saved, saves it again, and uploads its artifact:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
Restoring cache
Checking cache for 0_package-lock-ec7e7bfbf81f3b7ffde77ecb457c6376d0e9c398-protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
⋯ lines omitted
$ npm ci --cache .npm --prefer-offline
⋯ lines omitted
$ npm run build
⋯ lines omitted
Saving cache for successful job
Creating cache 0_package-lock-ec7e7bfbf81f3b7ffde77ecb457c6376d0e9c398-protected...
.npm/: found 24 matching artifact files and directories
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally.
Created cache
Uploading artifacts for successful job
Uploading artifacts...
dist/: found 3 matching artifact files and directories
Uploading artifacts as "archive" to coordinator... 201 Created  correlation_id=01M2DK95M6VDYK1HBZNF3X30P2 id=266 status=201 token=[REDACTED]
⋯ lines omitted
Job succeeded
GitLab ranjob logcaptured from GitLab
⋯ lines omitted
Restoring cache
Checking cache for 0_pom-2336814c62c17d1db9b38c969d72005f6a7e387d-protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
⋯ lines omitted
$ mvn $MAVEN_CLI_OPTS package -DskipTests
⋯ lines omitted
[INFO] BUILD SUCCESS
⋯ lines omitted
Saving cache for successful job
Creating cache 0_pom-2336814c62c17d1db9b38c969d72005f6a7e387d-protected...
.m2/repository/: found 724 matching artifact files and directories
⋯ lines omitted
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally.
Created cache
Uploading artifacts for successful job
Uploading artifacts...
target/greeting-service.jar: found 1 matching artifact files and directories
Uploading artifacts as "archive" to coordinator... 201 Created  correlation_id=01M2DSHQ77BPCPRXVJ0WVNJV9X id=318 status=201 token=[REDACTED]
⋯ lines omitted
Job succeeded
  • The key ends in -protected because main is a protected branch. GitLab keeps the caches of protected pipelines apart from the rest.
  • "No URL provided" means this runner keeps its cache on its own machine. GitLab.com's runners share a cache stored in Google Cloud Storage, so their log reads differently here.
  • unit-tests restores the same cache, and at the end says "Not uploading cache … due to policy". That is policy: pull.
  • The Tests tab counts the service's 2 tests, both passed.
Break it on purpose

In cache: key: files:, change the file's name to one that doesn't exist, and push. GitLab ignores a missing file, and when none of the named files exists the key is default. So the job looks for a different cache, and the one it saved before is no longer found. Put the name back.

Merge request pipelines#

Part III · Build your team's pipeline · Chapter 07 · 7 min read

So far, every push to any branch runs the whole pipeline. Once your team works in merge requests, you want one pipeline for each push to a merge request, with its result shown on the merge request, and no second pipeline for the same push. A workflow at the top of the file decides which kind of pipeline a push gets. Each job's rules then decide whether the job is in it. A rule on changed files lets a documentation change skip the build.

Thursday, 14:20. Your first merge request.

You open a merge request for the longer-names branch and push a fix to it. Two pipelines appear for the same commit, one on the branch and one on the merge request, and every job runs twice. The team lead asks why the runners are busy. What made the second pipeline, and which one is the right one? (the card)

Two kinds of pipeline#

GitLab can run two kinds of pipeline for the same commit:

  • A branch pipeline runs when you push to a branch. Its $CI_PIPELINE_SOURCE is push, and $CI_COMMIT_BRANCH holds the branch's name.
  • A merge request pipeline runs for a merge request, when it opens and on every push to its source branch. Its $CI_PIPELINE_SOURCE is merge_request_event, and $CI_COMMIT_BRANCH is not set at all. Its result shows on the merge request.

A job with no rules runs only in branch pipelines. As soon as jobs have rules, they can run in merge request pipelines too, and one push can then create both kinds. Chapter 21 covers every kind of pipeline.

A workflow: one pipeline per push#

A workflow is checked before any job. GitLab tries its rules in order, and the first that matches decides. If none matches, there is no pipeline:

practice · node-servicesteps/2-merge-requests.ymlworkflow
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH
practice · java-servicesteps/2-merge-requests.ymlworkflow
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH
  1. A merge request pipeline: create it.
  2. A branch pipeline for a branch that has an open merge request: don't. The merge request pipeline covers that push.
  3. Any other branch pipeline: create it. That includes main.
A push to a branch with an open merge request. Without a workflow, jobs whose rules allow both kinds of pipeline run twice. With the workflow, the branch pipeline is dropped and each job runs once. WITHOUT A WORKFLOW a push to a branch with an open merge request branch pipeline push merge request pipeline merge_request_event Two pipelines: every job runs twice. WITH THE WORKFLOW a push to a branch with an open merge request merge request pipeline merge_request_event no branch pipeline when: never One pipeline: each job runs once. The workflow's second rule, $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS with when: never, drops the branch pipeline.
Figure A push to a branch with an open merge request. Without a workflow, jobs whose rules allow both kinds of pipeline run twice. With the workflow, the branch pipeline is dropped and each job runs once.#

The workflow is the same for both stacks. Chapter 22 explains workflows in full, including why the second rule matters (duplicate pipelines).

Skipping work when only the docs changed#

Each job now has rules of its own:

practice · node-servicesteps/2-merge-requests.ymlbuild
build:
  stage: build
  script:
    - npm ci --cache .npm --prefer-offline
    - npm run build
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
  artifacts:
    paths:
      - dist/
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - changes:
        - src/**/*
        - test/**/*
        - package.json
        - package-lock.json
practice · java-servicesteps/2-merge-requests.ymlbuild
build:
  stage: build
  script:
    - mvn $MAVEN_CLI_OPTS package -DskipTests
  cache:
    key:
      files:
        - pom.xml
    paths:
      - .m2/repository/
  artifacts:
    paths:
      - target/greeting-service.jar
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - changes:
        - src/**/*
        - pom.xml
  • On the default branch, main, the job always runs, so main always has a fresh build.
  • Anywhere else, changes decides. In a merge request pipeline, GitLab compares the merge request's changes with its target branch. The job runs only if a file matching one of the patterns changed.

So a merge request whose changes, taken together, touch only docs/README.md matches no rule in any job. A pipeline with no jobs isn't created, so that merge request gets no pipeline at all (no jobs, no pipeline). One surprise: on a branch pipeline, changes is always true for the first push of a new branch (runs on a new branch).

unit-tests carries the same rules, repeated word for word. Repeated rules drift apart, and Chapter 14 shows the usual way to write them once. Chapter 23 covers rules in full.

greeting-service at the end of this chapter: the same two jobs, now with rules. A docs-only merge request gets no pipeline.buildbuildtestunit-tests
Pipeline greeting-service at the end of this chapter: the same two jobs, now with rules. A docs-only merge request gets no pipeline.#
As text
  1. build: build
  2. test: unit-tests
What you can now do
  1. Write a workflow that gives each push exactly one pipeline, of the right kind.
  2. Say which kind of pipeline a push will get before you push it.
  3. Skip the build when only documentation changed, and explain the new-branch surprise.
LabOne pipeline per pushabout 15 min#
You need

greeting-service from the previous lab.

Steps
  1. Replace .gitlab-ci.yml with this chapter's pipeline for your stack, and push to main.
    practice · node-servicesteps/2-merge-requests.yml
    # greeting-service (Node): the pipeline at the end of "Merge request pipelines".
    workflow:
      rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
          when: never
        - if: $CI_COMMIT_BRANCH
    
    stages:
      - build
      - test
    
    default:
      image: node:24.21.0-alpine
    
    build:
      stage: build
      script:
        - npm ci --cache .npm --prefer-offline
        - npm run build
      cache:
        key:
          files:
            - package-lock.json
        paths:
          - .npm/
      artifacts:
        paths:
          - dist/
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - test/**/*
            - package.json
            - package-lock.json
    
    unit-tests:
      stage: test
      script:
        - npm ci --cache .npm --prefer-offline
        - npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=junit.xml
      cache:
        key:
          files:
            - package-lock.json
        paths:
          - .npm/
        policy: pull
      artifacts:
        when: always
        reports:
          junit: junit.xml
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - test/**/*
            - package.json
            - package-lock.json
    practice · java-servicesteps/2-merge-requests.yml
    # greeting-service (Java): the pipeline at the end of "Merge request pipelines".
    workflow:
      rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
          when: never
        - if: $CI_COMMIT_BRANCH
    
    stages:
      - build
      - test
    
    default:
      image: maven:3.9.16-eclipse-temurin-25
    
    variables:
      MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
      MAVEN_CLI_OPTS: "--batch-mode --no-transfer-progress"
    
    build:
      stage: build
      script:
        - mvn $MAVEN_CLI_OPTS package -DskipTests
      cache:
        key:
          files:
            - pom.xml
        paths:
          - .m2/repository/
      artifacts:
        paths:
          - target/greeting-service.jar
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - pom.xml
    
    unit-tests:
      stage: test
      script:
        - mvn $MAVEN_CLI_OPTS test
      cache:
        key:
          files:
            - pom.xml
        paths:
          - .m2/repository/
        policy: pull
      artifacts:
        when: always
        reports:
          junit: target/surefire-reports/TEST-*.xml
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - pom.xml
  2. Create a branch, longer-names. In the greeting code, allow names of up to 60 characters instead of 40, and push the branch.
  3. Open a merge request from longer-names into main.
  4. Change the limit to 80 and push again. Count the new pipelines, then close the merge request.
  5. From main, create another branch, docs-only, that changes only docs/README.md, and push it. Then open a merge request from it, and look for its pipeline.
  6. Close that merge request too.
What you should see

What each step created, when this lab ran on GitLab 19.3:

YouPipelinesJobs
push the new file to mainone branch pipelinebuild, unit-tests
push longer-names, before the merge requestone branch pipelinebuild, unit-tests
open the merge requestone merge request pipelinebuild, unit-tests
push again, with the merge request openone merge request pipeline, and no branch pipelinebuild, unit-tests
push docs-onlyone branch pipeline: changes is true for a new branch's first pushbuild, unit-tests
open the docs-only merge requestnonenone

The merge request pipelines are listed on the merge request, under Pipelines. Their jobs ran because the merge request changed the greeting code. The docs-only merge request changed no file that a changes rule names, so it got no pipeline.

Break it on purpose

Delete the workflow's second rule and push a code change to the branch again. Two pipelines appear for the one push, one of each kind, and each job runs twice.

Images, the registry and variables#

Part III · Build your team's pipeline · Chapter 08 · 7 min read

Production runs greeting-service as a container image, so the pipeline builds one and pushes it to the project's container registry, tagged with the commit it was built from. The values that jobs need come from variables: ordinary ones in the YAML, and secrets in the project's settings. A secret is masked, so logs never show it, and protected, so only protected branches such as main receive it.

Friday, 09:30. The first feature branch since the secret went in.

build-image passes on main. On your branch, the same job fails at docker login with unauthorized, and the token it prints is empty. Nothing in the file mentions a branch. Where does the token come from, and why does main get it and your branch not? (the card)

The project's container registry#

Every GitLab project can hold container images in its container registry, under Deploy › Container Registry. On a self-managed GitLab, an administrator must turn the registry on first. A job reaches the registry through variables that GitLab sets for it:

VariableHolds
CI_REGISTRYthe registry's address
CI_REGISTRY_IMAGEthe registry's address followed by the project's path: this project's image name
CI_REGISTRY_USER, CI_REGISTRY_PASSWORDa user name and password that can push to this project's registry
CI_COMMIT_SHORT_SHAthe first eight characters of the commit's SHA

Building an image in a job#

A new job builds the image from the service's Dockerfile and pushes it:

practice · node-servicesteps/3-image.ymlbuild-image
build-image:
  stage: package
  image: docker:29.8.0-cli
  services:
    - docker:29.8.0-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
    - docker build --provenance=false --tag "$IMAGE" .
    - docker push "$IMAGE"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
practice · java-servicesteps/3-image.ymlbuild-image
build-image:
  stage: package
  image: docker:29.8.0-cli
  services:
    - docker:29.8.0-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
    - docker build --provenance=false --tag "$IMAGE" .
    - docker push "$IMAGE"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  • The job runs in docker:29.8.0-cli, which has the Docker command but no Docker engine. The engine runs beside it as a service, a second container: docker:29.8.0-dind, short for Docker-in-Docker.
  • DOCKER_TLS_CERTDIR makes the job and the engine talk over TLS.
  • The job logs in with the registry's variables, builds the image, and pushes it as $IMAGE. --provenance=false leaves out the build's provenance record, so the push is a single image rather than an index with an extra attestation manifest. On the local GitLab this book was checked on, the push failed without it.
  • The rule builds images only on main.

Docker-in-Docker needs a runner that allows privileged containers. GitLab.com's own Linux runners do. On your organisation's GitLab, ask your platform team: many allow it only on some runners, or offer another way to build images (chapter 30).

The Dockerfile starts from a pinned runtime image, and copies in what the build job kept as an artifact:

practice · node-serviceDockerfile
# The greeting service's image: its built files and its production dependencies.
FROM node:24.21.0-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY dist/ dist/
EXPOSE 8080
CMD ["node", "dist/server.js"]
practice · java-serviceDockerfile
# The greeting service's image: the jar the build job made, on a Java runtime.
FROM eclipse-temurin:25.0.4_7-jre-alpine
COPY target/greeting-service.jar /app/greeting-service.jar
EXPOSE 8080
CMD ["java", "-jar", "/app/greeting-service.jar"]

Variables in the YAML#

practice · node-servicesteps/3-image.ymlimage-variable
variables:
  IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
practice · java-servicesteps/3-image.ymlvariables
variables:
  MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  MAVEN_CLI_OPTS: "--batch-mode --no-transfer-progress"
  IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

The two MAVEN_ variables are the ones from Build and test. IMAGE is the new line.

A top-level variables: block gives every job a value. Here IMAGE is built from two variables that GitLab sets: the project's image name, and the commit's short SHA. Tagging with the commit means that each image says exactly which code it holds, and a later deploy can name that one image. Values that aren't secret belong here, in the file, where reviewers see them (chapter 26).

Secrets in settings: masked and protected#

Secrets never go in the file. You add them under Settings › CI/CD › Variables, where two settings guard them:

  • Masked replaces the value with [MASKED] wherever a job prints it. GitLab can only mask a value it can find reliably, so the value must be a single line, with no spaces, at least 8 characters long. Masked is the default for a new variable.
  • Protect variable gives the value only to pipelines on protected branches and tags.

This chapter's pipeline has a job that exists only to show both, and that you would never keep:

practice · node-servicesteps/3-image.ymlsecret-check
secret-check:
  stage: test
  script:
    - echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'"

On main, which is protected, its log shows [MASKED]. On a feature branch the value is empty. That second fact is behind a common failure: a job that works on main and fails on every branch (fails only on branches). Chapter 27 covers secrets in full.

greeting-service at the end of this chapter, on mainbuildbuildtestsecret-checkunit-testspackagebuild-image
Pipeline greeting-service at the end of this chapter, on main#
As text
  1. build: build
  2. test: secret-check, unit-tests
  3. package: build-image
What you can now do
  1. Build an image in a job and push it to the project's registry, tagged with its commit.
  2. Put a secret where it belongs, masked and protected, and predict which pipelines will see it empty.
  3. Explain the difference between a variable in the YAML and one in the settings.
LabPush an image, and watch a secretabout 20 min#
You need

greeting-service from the previous lab.

Steps
  1. Under Settings › CI/CD › Variables, add a variable. Name it DEPLOY_TOKEN, give it any value of 8 or more characters without spaces, such as lab-deploy-token-4711, make it Masked, and tick Protect variable.
  2. Replace .gitlab-ci.yml with this chapter's pipeline, and push to main.
    practice · node-servicesteps/3-image.yml
    # greeting-service (Node): the pipeline at the end of "Images, the registry and variables".
    workflow:
      rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
          when: never
        - if: $CI_COMMIT_BRANCH
    
    stages:
      - build
      - test
      - package
    
    default:
      image: node:24.21.0-alpine
    
    variables:
      IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    
    build:
      stage: build
      script:
        - npm ci --cache .npm --prefer-offline
        - npm run build
      cache:
        key:
          files:
            - package-lock.json
        paths:
          - .npm/
      artifacts:
        paths:
          - dist/
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - test/**/*
            - package.json
            - package-lock.json
    
    unit-tests:
      stage: test
      script:
        - npm ci --cache .npm --prefer-offline
        - npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=junit.xml
      cache:
        key:
          files:
            - package-lock.json
        paths:
          - .npm/
        policy: pull
      artifacts:
        when: always
        reports:
          junit: junit.xml
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - test/**/*
            - package.json
            - package-lock.json
    
    secret-check:
      stage: test
      script:
        - echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'"
    
    build-image:
      stage: package
      image: docker:29.8.0-cli
      services:
        - docker:29.8.0-dind
      variables:
        DOCKER_TLS_CERTDIR: "/certs"
      script:
        - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
        - docker build --provenance=false --tag "$IMAGE" .
        - docker push "$IMAGE"
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    practice · java-servicesteps/3-image.yml
    # greeting-service (Java): the pipeline at the end of "Images, the registry and variables".
    workflow:
      rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
          when: never
        - if: $CI_COMMIT_BRANCH
    
    stages:
      - build
      - test
      - package
    
    default:
      image: maven:3.9.16-eclipse-temurin-25
    
    variables:
      MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
      MAVEN_CLI_OPTS: "--batch-mode --no-transfer-progress"
      IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    
    build:
      stage: build
      script:
        - mvn $MAVEN_CLI_OPTS package -DskipTests
      cache:
        key:
          files:
            - pom.xml
        paths:
          - .m2/repository/
      artifacts:
        paths:
          - target/greeting-service.jar
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - pom.xml
    
    unit-tests:
      stage: test
      script:
        - mvn $MAVEN_CLI_OPTS test
      cache:
        key:
          files:
            - pom.xml
        paths:
          - .m2/repository/
        policy: pull
      artifacts:
        when: always
        reports:
          junit: target/surefire-reports/TEST-*.xml
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - pom.xml
    
    secret-check:
      stage: test
      script:
        - echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'"
    
    build-image:
      stage: package
      image: docker:29.8.0-cli
      services:
        - docker:29.8.0-dind
      variables:
        DOCKER_TLS_CERTDIR: "/certs"
      script:
        - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
        - docker build --provenance=false --tag "$IMAGE" .
        - docker push "$IMAGE"
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  3. Open secret-check's log, then build-image's.
  4. Open Deploy › Container Registry.
  5. Push any small change to a new branch, and open that pipeline's secret-check log.
What you should see

On main, which is protected, secret-check is given the value, and the log masks it:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'"
DEPLOY_TOKEN is '[MASKED]'
⋯ lines omitted
Job succeeded
GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'"
DEPLOY_TOKEN is '[MASKED]'
⋯ lines omitted
Job succeeded

build-image logs in, builds the image, names it with the commit's short SHA, and pushes it:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
Login Succeeded
⋯ lines omitted
$ docker build --provenance=false --tag "$IMAGE" .
⋯ lines omitted
#10 naming to gitlab.example.com:5050/practice/node-service:0648d3dd done
⋯ lines omitted
$ docker push "$IMAGE"
The push refers to repository [gitlab.example.com:5050/practice/node-service]
⋯ lines omitted
0648d3dd: digest: sha256:df19c9e8709250340d0fa909605eef0681c9e4448045eadc8cdd74b1ef89e46d size: 1809
⋯ lines omitted
Job succeeded

Deploy › Container Registry lists the project's image with one tag: that same short SHA, 0648d3dd here.

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
Login Succeeded
⋯ lines omitted
$ docker build --provenance=false --tag "$IMAGE" .
⋯ lines omitted
#7 naming to gitlab.example.com:5050/practice/java-service:d74ed00d done
⋯ lines omitted
$ docker push "$IMAGE"
The push refers to repository [gitlab.example.com:5050/practice/java-service]
⋯ lines omitted
d74ed00d: digest: sha256:e25c1d658c4a8bd0fcfd3829bc7235929bb93beb20f6aec4ca37aa6f33363c73 size: 1432
⋯ lines omitted
Job succeeded

Deploy › Container Registry lists the project's image with one tag: that same short SHA, d74ed00d here.

On the branch, the rule keeps build-image out of the pipeline, and secret-check's value is empty:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'"
DEPLOY_TOKEN is ''
⋯ lines omitted
Job succeeded
GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'"
DEPLOY_TOKEN is ''
⋯ lines omitted
Job succeeded

Environments and deploys#

Part III · Build your team's pipeline · Chapter 09 · 4 min read

A deploy job names an environment, such as staging or production, and GitLab records each deployment: which commit went where, and when. greeting-service deploys to staging on every push to main, and to production only when a person starts the job. Until someone does, the pipeline shows as blocked.

Monday, 08:50. The first deploy to production.

Friday's pipeline on main is still marked blocked. Every job passed except one, which shows a play button nobody pressed. Staging was deployed on Friday evening; production was not. Was something broken over the weekend, or is this what waiting looks like? (the card)

A deploy job names its environment#

practice · node-servicesteps/4-deploy.ymldeploy
deploy-staging:
  stage: deploy
  image: alpine:3.22
  script:
    - echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN"
  environment:
    name: staging
    url: https://staging.greeting.example.com
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-production:
  stage: deploy
  image: alpine:3.22
  script:
    - echo "Deploying $IMAGE to production, with the token $DEPLOY_TOKEN"
  environment:
    name: production
    url: https://greeting.example.com
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
practice · java-servicesteps/4-deploy.ymldeploy
deploy-staging:
  stage: deploy
  image: alpine:3.22
  script:
    - echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN"
  environment:
    name: staging
    url: https://staging.greeting.example.com
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-production:
  stage: deploy
  image: alpine:3.22
  script:
    - echo "Deploying $IMAGE to production, with the token $DEPLOY_TOKEN"
  environment:
    name: production
    url: https://greeting.example.com
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

environment: name makes a job a deployment job. GitLab lists every environment under Operate › Environments, each with its deployments, and url puts a link to the running service beside it.

These deploy scripts only print what they would do, with the token masked. A real deploy job runs whatever your organisation deploys with: a cloud's command-line tool, or a commit to a configuration repository that a tool in the cluster applies. Chapter 40 covers the rest.

Production waits for a person#

The production job's rule adds when: manual. The job appears in the pipeline with a play button, and runs only when someone selects it. A manual job added by a rule is not allowed to fail, so the pipeline waits for it and shows as blocked (a play button that waits).

Two more guards exist on paid tiers. A protected environment Premium lets only chosen people deploy to it, and deployment approvals Premium make a deploy wait for someone to approve it. Neither is needed for this lab.

greeting-service at the end of this chapter, on mainbuildbuildtestunit-testspackagebuild-imagedeploy▶deploy-productiondeploy-staging
Pipeline greeting-service at the end of this chapter, on main#
As text
  1. build: build
  2. test: unit-tests
  3. package: build-image
  4. deploy: deploy-production (manual), deploy-staging
What you can now do
  1. Give a deploy job an environment, so that GitLab records what went where.
  2. Make production wait for a person, and read a blocked pipeline as a job waiting for its button.
  3. Find every deployment of an environment under Operate › Environments.
LabDeploy to staging, then to productionabout 10 min#
You need

greeting-service from the previous lab, with its DEPLOY_TOKEN variable.

Steps
  1. Replace .gitlab-ci.yml with this chapter's pipeline, and push to main. The temporary secret-check job is gone.
    practice · node-servicesteps/4-deploy.yml
    # greeting-service (Node): the pipeline at the end of "Environments and deploys".
    workflow:
      rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
          when: never
        - if: $CI_COMMIT_BRANCH
    
    stages:
      - build
      - test
      - package
      - deploy
    
    default:
      image: node:24.21.0-alpine
    
    variables:
      IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    
    build:
      stage: build
      script:
        - npm ci --cache .npm --prefer-offline
        - npm run build
      cache:
        key:
          files:
            - package-lock.json
        paths:
          - .npm/
      artifacts:
        paths:
          - dist/
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - test/**/*
            - package.json
            - package-lock.json
    
    unit-tests:
      stage: test
      script:
        - npm ci --cache .npm --prefer-offline
        - npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=junit.xml
      cache:
        key:
          files:
            - package-lock.json
        paths:
          - .npm/
        policy: pull
      artifacts:
        when: always
        reports:
          junit: junit.xml
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - test/**/*
            - package.json
            - package-lock.json
    
    build-image:
      stage: package
      image: docker:29.8.0-cli
      services:
        - docker:29.8.0-dind
      variables:
        DOCKER_TLS_CERTDIR: "/certs"
      script:
        - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
        - docker build --provenance=false --tag "$IMAGE" .
        - docker push "$IMAGE"
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    
    deploy-staging:
      stage: deploy
      image: alpine:3.22
      script:
        - echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN"
      environment:
        name: staging
        url: https://staging.greeting.example.com
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    
    deploy-production:
      stage: deploy
      image: alpine:3.22
      script:
        - echo "Deploying $IMAGE to production, with the token $DEPLOY_TOKEN"
      environment:
        name: production
        url: https://greeting.example.com
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
          when: manual
    practice · java-servicesteps/4-deploy.yml
    # greeting-service (Java): the pipeline at the end of "Environments and deploys".
    workflow:
      rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
          when: never
        - if: $CI_COMMIT_BRANCH
    
    stages:
      - build
      - test
      - package
      - deploy
    
    default:
      image: maven:3.9.16-eclipse-temurin-25
    
    variables:
      MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
      MAVEN_CLI_OPTS: "--batch-mode --no-transfer-progress"
      IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    
    build:
      stage: build
      script:
        - mvn $MAVEN_CLI_OPTS package -DskipTests
      cache:
        key:
          files:
            - pom.xml
        paths:
          - .m2/repository/
      artifacts:
        paths:
          - target/greeting-service.jar
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - pom.xml
    
    unit-tests:
      stage: test
      script:
        - mvn $MAVEN_CLI_OPTS test
      cache:
        key:
          files:
            - pom.xml
        paths:
          - .m2/repository/
        policy: pull
      artifacts:
        when: always
        reports:
          junit: target/surefire-reports/TEST-*.xml
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - pom.xml
    
    build-image:
      stage: package
      image: docker:29.8.0-cli
      services:
        - docker:29.8.0-dind
      variables:
        DOCKER_TLS_CERTDIR: "/certs"
      script:
        - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
        - docker build --provenance=false --tag "$IMAGE" .
        - docker push "$IMAGE"
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    
    deploy-staging:
      stage: deploy
      image: alpine:3.22
      script:
        - echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN"
      environment:
        name: staging
        url: https://staging.greeting.example.com
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    
    deploy-production:
      stage: deploy
      image: alpine:3.22
      script:
        - echo "Deploying $IMAGE to production, with the token $DEPLOY_TOKEN"
      environment:
        name: production
        url: https://greeting.example.com
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
          when: manual
  2. Open the pipeline. Note its status, and deploy-staging's log.
  3. Open Operate › Environments.
  4. Back in the pipeline, run deploy-production with its play button.
What you should see

deploy-staging runs on its own:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN"
Deploying gitlab.example.com:5050/practice/node-service:7d625fae to staging, with the token [MASKED]
⋯ lines omitted
Job succeeded
GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN"
Deploying gitlab.example.com:5050/practice/java-service:fbb6b2d6 to staging, with the token [MASKED]
⋯ lines omitted
Job succeeded

Then the pipeline stops with the badge Blocked. Every job has passed except deploy-production, which waits for its play button. Operate › Environments lists staging, with its URL and the deployment you just made.

After you run deploy-production, it prints the same line for production, and the pipeline passes. The environments list then shows both environments, each with its URL. Production's URL can take a minute to appear.

OpenTofu in your pipeline#

Part III · Build your team's pipeline · Chapter 10 · 9 min read

Infrastructure that your team owns, such as a queue or a storage bucket, is often written as code with OpenTofu and changed through the pipeline. A merge request shows the plan, which is what would change, and merging to main applies it. GitLab can keep OpenTofu's state for you, so there is nothing else to set up.

Tuesday, 15:10. The first infrastructure change.

You add a job that runs tofu plan in OpenTofu's own image. It fails at once, before any of your commands, with the tool's usage text in the log. The same script works on your laptop. What ran instead of your script? (the card)

OpenTofu in two minutes#

OpenTofu is an open-source tool that creates and changes infrastructure from files of code. It began as a copy of Terraform: its command is tofu where Terraform's is terraform, and it reads the same kind of files, written in a language called HCL. The words you need:

WordMeans
configurationthe .tf files: what should exist
providera plugin that talks to one kind of system, such as a cloud
planthe changes OpenTofu would make: what it would add, change and destroy
applymaking those changes
stateOpenTofu's record of what it has made, which the next plan compares with

This lab's configuration makes a random name for the team's queue. Its provider, random, needs no cloud account, so every plan and apply is real:

practice · team-inframain.tfsettings
terraform {
  required_version = ">= 1.12"
  backend "http" {}
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "3.9.1"
    }
  }
}
practice · team-inframain.tfresources
variable "team" {
  type    = string
  default = "payments"
}

resource "random_pet" "queue" {
  prefix = var.team
  length = 2
}

output "queue_name" {
  value = random_pet.queue.id
}

State that GitLab keeps#

The empty backend "http" {} block says the state lives on a web server, here GitLab. Variables tell OpenTofu where, and how to sign in:

practice · team-infra.gitlab-ci.ymlvariables
variables:
  TF_STATE_NAME: default
  TF_HTTP_ADDRESS: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/$TF_STATE_NAME
  TF_HTTP_LOCK_ADDRESS: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/$TF_STATE_NAME/lock
  TF_HTTP_UNLOCK_ADDRESS: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/$TF_STATE_NAME/lock
  TF_HTTP_LOCK_METHOD: POST
  TF_HTTP_UNLOCK_METHOD: DELETE
  TF_HTTP_USERNAME: gitlab-ci-token
  TF_HTTP_PASSWORD: $CI_JOB_TOKEN
VariableTells OpenTofu
TF_STATE_NAMEthe state's name, default. A project can keep several states, one for each name
TF_HTTP_ADDRESSwhere to read and write the state: this project's state, under GitLab's API
TF_HTTP_LOCK_ADDRESS, TF_HTTP_UNLOCK_ADDRESSwhere to lock the state before a change and unlock it after, so that two jobs never write it at once
TF_HTTP_LOCK_METHOD, TF_HTTP_UNLOCK_METHODwhich HTTP method GitLab expects for each: POST to lock, DELETE to unlock
TF_HTTP_USERNAME, TF_HTTP_PASSWORDhow to sign in: the fixed user name gitlab-ci-token, and the job's own token

OpenTofu reads every TF_HTTP_ variable from the environment, which is why the backend block can stay empty and no script mentions them. The state's address is under the project's API, and the job signs in with its own job token. GitLab lists the states under Operate › Terraform states. A Developer can plan against the state, but applying a change needs the Maintainer role, so merge requests can plan, and only the pipeline on main applies.

Plan in the merge request, apply on merge#

practice · team-infra.gitlab-ci.ymldefault
default:
  image:
    name: ghcr.io/opentofu/opentofu:1.12.6
    entrypoint: [""]
  before_script:
    - tofu init -input=false
practice · team-infra.gitlab-ci.ymlvalidate
validate:
  stage: validate
  script:
    - tofu fmt -check
    - tofu validate
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
practice · team-infra.gitlab-ci.ymlplan
plan:
  stage: plan
  script:
    - apk add --no-cache jq
    - tofu plan -input=false -out=plan.cache
    - tofu show -json plan.cache | jq -r '([.resource_changes[]?.change.actions?]|flatten)|{"create":(map(select(.=="create"))|length),"update":(map(select(.=="update"))|length),"delete":(map(select(.=="delete"))|length)}' > plan.json
  artifacts:
    paths:
      - plan.cache
    reports:
      terraform: plan.json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
practice · team-infra.gitlab-ci.ymlapply
apply:
  stage: apply
  script:
    - tofu apply -input=false plan.cache
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  • default runs tofu init before every job, with the pinned ghcr.io/opentofu/opentofu:1.12.6 image. Its entrypoint is emptied, because the image would otherwise run tofu instead of your script (the script never runs).
  • validate checks the formatting and the configuration.
  • validate and plan carry rules for merge request pipelines and for main. A job without rules runs only in branch pipelines, and the workflow drops those while a merge request is open, so without the rules the merge request would get no pipeline at all.
  • plan saves the plan in plan.cache, then turns it into a summary with jq, GitLab's documented command. The summary is a report, so the merge request shows how many things the plan adds, changes and destroys.
  • apply runs only on main, and applies the plan saved by that pipeline's plan job.
Plan in the merge request, apply on merge: what each pipeline does with OpenTofu, and the state GitLab keepsnoyesPush a change to a branch, and open amerge requestThe merge request pipeline validatesand plansThe merge request shows the plan'scountsMerged into main?Nothing changes: the plan was apreviewThe pipeline on main validates andplans againapply runs that plan, and GitLab savesthe new state
Figure Plan in the merge request, apply on merge: what each pipeline does with OpenTofu, and the state GitLab keeps#
As text
  1. Push a change to a branch, and open a merge request
  2. The merge request pipeline validates and plans
  3. The merge request shows the plan's counts
  4. Merged into main? No: Nothing changes: the plan was a preview. Yes: the next step.
  5. The pipeline on main validates and plans again
  6. apply runs that plan, and GitLab saves the new state

Central teams often wrap these jobs in GitLab's OpenTofu component, gitlab.com/components/opentofu/validate-plan-apply, included at a version such as 4.8.1. You will recognise its jobs from these. On a self-managed GitLab, an administrator mirrors the component's project first.

LegacyGitLab's Terraform templates
Older pipelines include GitLab's Terraform CI/CD templates, and run their jobs in GitLab's terraform images. GitLab no longer distributes either. Use OpenTofu jobs like these, or the OpenTofu component.
team-infra's pipeline on mainvalidatevalidateplanplanapplyapply
Pipeline team-infra's pipeline on main#
As text
  1. validate: validate
  2. plan: plan
  3. apply: apply
What you can now do
  1. Write a pipeline that plans in a merge request and applies on merge, with the state kept by GitLab.
  2. Empty a tool image's entrypoint when its default command swallows your script.
  3. Read a plan's counts on the merge request before anyone merges it.
LabOpenTofu: plan in a merge request, apply on mergeabout 15 min#
You need

An account on your GitLab.

Steps
  1. Create a blank project named team-infra. Add main.tf and .gitlab-ci.yml, both shown in full here:
    practice · team-inframain.tf
    # team-infra: what the "OpenTofu in your pipeline" lab manages. random_pet needs no cloud
    # account, so every plan and apply is real.
    terraform {
      required_version = ">= 1.12"
      backend "http" {}
      required_providers {
        random = {
          source  = "hashicorp/random"
          version = "3.9.1"
        }
      }
    }
    
    variable "team" {
      type    = string
      default = "payments"
    }
    
    resource "random_pet" "queue" {
      prefix = var.team
      length = 2
    }
    
    output "queue_name" {
      value = random_pet.queue.id
    }
    practice · team-infra.gitlab-ci.yml
    # team-infra: the pipeline of the "OpenTofu in your pipeline" lab. GitLab keeps the
    # state, a merge request shows the plan, and merging to main applies it.
    workflow:
      rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
          when: never
        - if: $CI_COMMIT_BRANCH
    
    stages:
      - validate
      - plan
      - apply
    
    variables:
      TF_STATE_NAME: default
      TF_HTTP_ADDRESS: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/$TF_STATE_NAME
      TF_HTTP_LOCK_ADDRESS: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/$TF_STATE_NAME/lock
      TF_HTTP_UNLOCK_ADDRESS: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/$TF_STATE_NAME/lock
      TF_HTTP_LOCK_METHOD: POST
      TF_HTTP_UNLOCK_METHOD: DELETE
      TF_HTTP_USERNAME: gitlab-ci-token
      TF_HTTP_PASSWORD: $CI_JOB_TOKEN
    
    default:
      image:
        name: ghcr.io/opentofu/opentofu:1.12.6
        entrypoint: [""]
      before_script:
        - tofu init -input=false
    
    validate:
      stage: validate
      script:
        - tofu fmt -check
        - tofu validate
      rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    
    plan:
      stage: plan
      script:
        - apk add --no-cache jq
        - tofu plan -input=false -out=plan.cache
        - tofu show -json plan.cache | jq -r '([.resource_changes[]?.change.actions?]|flatten)|{"create":(map(select(.=="create"))|length),"update":(map(select(.=="update"))|length),"delete":(map(select(.=="delete"))|length)}' > plan.json
      artifacts:
        paths:
          - plan.cache
        reports:
          terraform: plan.json
      rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    
    apply:
      stage: apply
      script:
        - tofu apply -input=false plan.cache
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  2. Push to main. When the pipeline finishes, open apply's log, then Operate › Terraform states.
  3. Create a branch. On it, change length = 2 to length = 3, and push. Then open a merge request from the branch into main.
  4. Look at the merge request once its pipeline finishes.
  5. Merge it, and open the apply log of the pipeline on main.
What you should see

The first pipeline on main creates the queue's name, and GitLab saves the state:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ tofu apply -input=false plan.cache
Acquiring state lock. This may take a few moments...
random_pet.queue: Creating...
random_pet.queue: Creation complete after 0s [id=payments-helped-sunfish]
Releasing state lock. This may take a few moments...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Outputs:
queue_name = "payments-helped-sunfish"
⋯ lines omitted
Job succeeded

Operate › Terraform states lists one state, default. In the merge request, plan shows what the change would do. The length can't be changed in place, so OpenTofu would replace the resource:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ tofu plan -input=false -out=plan.cache
Acquiring state lock. This may take a few moments...
random_pet.queue: Refreshing state... [id=payments-helped-sunfish]
OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
OpenTofu will perform the following actions:
  # random_pet.queue must be replaced
-/+ resource "random_pet" "queue" {
      ~ id        = "payments-helped-sunfish" -> (known after apply)
      ~ length    = 2 -> 3 # forces replacement
        # (2 unchanged attributes hidden)
    }
Plan: 1 to add, 0 to change, 1 to destroy.
Changes to Outputs:
  ~ queue_name = "payments-helped-sunfish" -> (known after apply)
⋯ lines omitted
Saved the plan to: plan.cache
⋯ lines omitted
Releasing state lock. This may take a few moments...
⋯ lines omitted
plan.json: found 1 matching artifact files and directories
Uploading artifacts as "terraform" to coordinator... 201 Created  correlation_id=01M2DR15CECXP68NQ0SWZNPXZ2 id=303 status=201 token=[REDACTED]
⋯ lines omitted
Job succeeded

The last lines upload plan.json as the terraform report, which is how the merge request shows the same counts. Merging runs the pipeline on main, and its apply destroys the old name and creates the new one:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ tofu apply -input=false plan.cache
Acquiring state lock. This may take a few moments...
random_pet.queue: Destroying... [id=payments-helped-sunfish]
random_pet.queue: Destruction complete after 0s
random_pet.queue: Creating...
random_pet.queue: Creation complete after 0s [id=payments-likely-careful-dane]
Releasing state lock. This may take a few moments...
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.
Outputs:
queue_name = "payments-likely-careful-dane"
⋯ lines omitted
Job succeeded

The state's serial, which goes up with each change, went from 1 to 2. Your names will differ, because random_pet picks them at random.

The central team's templates#

Part III · Build your team's pipeline · Chapter 11 · 7 min read

Your file doesn't have to hold every job. A central team can publish templates, which are files of hidden jobs that you extend, and components, which are packaged jobs with inputs. Your file includes them at a version you choose. You then change only what your team needs, and the pipeline editor's Full configuration shows the result.

Wednesday, 11:15. The central team's templates arrive.

You replace your unit-tests job with two lines: extends: .node-test and a rule of your own. The job now runs in pipelines it never used to, on every tag. You added one rule; the template had three. Where did the other two go? (the card)

Play the central team#

In this lab you are both teams. A second project, central-templates, holds the central team's files:

  • templates/node.yml and templates/java.yml, each with one hidden job: a job whose name starts with a dot, which never runs itself and exists to be copied.
  • templates/secret-scan/template.yml, a component: a job with typed inputs at the top, before a --- line. Its job has rules of its own, so it runs in merge request pipelines as well as branch pipelines.
practice · central-templatestemplates/node.ymlnode-test
.node-test:
  stage: test
  image: node:24.21.0-alpine
  script:
    - npm ci --cache .npm --prefer-offline
    - npm test
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
    policy: pull
practice · central-templatestemplates/java.ymlmaven-test
.maven-test:
  stage: test
  image: maven:3.9.16-eclipse-temurin-25
  variables:
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  script:
    - mvn --batch-mode --no-transfer-progress test
  cache:
    key:
      files:
        - pom.xml
    paths:
      - .m2/repository/
    policy: pull
practice · central-templatestemplates/secret-scan/template.ymlspec
spec:
  inputs:
    stage:
      default: test
    paths:
      default: "."
      description: The directory to look in
practice · central-templatestemplates/secret-scan/template.ymljob
secret-scan:
  stage: $[[ inputs.stage ]]
  image: alpine:3.22
  script:
    - echo "Looking for private keys in $[[ inputs.paths ]]"
    - if grep -rl "BEGIN [A-Z ]*PRIVATE KEY" $[[ inputs.paths ]]; then echo "A private key is committed"; exit 1; fi
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH

The central team tags the project 1.0.0. That tag is the version your file asks for.

Include at a version#

practice · node-servicesteps/5-central-templates.ymlinclude
include:
  - project: practice/central-templates
    ref: 1.0.0
    file: templates/node.yml
  - component: $CI_SERVER_FQDN/practice/central-templates/secret-scan@1.0.0
    inputs:
      stage: test
practice · java-servicesteps/5-central-templates.ymlinclude
include:
  - project: practice/central-templates
    ref: 1.0.0
    file: templates/java.yml
  - component: $CI_SERVER_FQDN/practice/central-templates/secret-scan@1.0.0
    inputs:
      stage: test
  • project and file name a file in another project, and ref: 1.0.0 fixes which version of it.
  • component names the component by the instance, the project and its own name, then @1.0.0. $CI_SERVER_FQDN is your GitLab's address, so the line works on any instance. inputs fill in the component's inputs.

These files use the group practice. Change it to the group or user that owns your two projects. Chapter 15 and chapter 16 cover both kinds of include.

Extend, and change only what you must#

practice · node-servicesteps/5-central-templates.ymlunit-tests
unit-tests:
  extends: .node-test
  script:
    - npm ci --cache .npm --prefer-offline
    - npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=junit.xml
  artifacts:
    when: always
    reports:
      junit: junit.xml
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - changes:
        - src/**/*
        - test/**/*
        - package.json
        - package-lock.json
practice · java-servicesteps/5-central-templates.ymlunit-tests
unit-tests:
  extends: .maven-test
  artifacts:
    when: always
    reports:
      junit: target/surefire-reports/TEST-*.xml
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - changes:
        - src/**/*
        - pom.xml

extends copies the hidden job's settings into unit-tests, and then the job's own settings are laid on top. Lists are replaced whole, and maps are merged key by key.

So the job's own script replaces the template's, and its artifacts and rules are added. The stage, the image and the cache still come from the template.

So the job's own artifacts and rules are added, and everything else still comes from the template: the stage, the image, the MAVEN_OPTS variable, the script and the cache.

Your file and the central team's: your project includes the template and the component at 1.0.0; unit-tests extends the hidden job and replaces its script, and secret-scan comes from the component. central-templates the central team's project, tagged 1.0.0 templates/node.yml hidden job .node-test: image, cache, script templates/secret-scan/template.yml a component, with inputs stage and paths ref: 1.0.0 @1.0.0 your project .gitlab-ci.yml unit-tests: extends .node-test your script replaces its script secret-scan from the component, in stage test Pinned to 1.0.0: the central team's next change reaches you only when you move the version. Full configuration shows the merged unit-tests: .node-test's image and cache, with your script.
Figure Your file and the central team's: your project includes the template and the component at 1.0.0; unit-tests extends the hidden job and replaces its script, and secret-scan comes from the component.#

To see what GitLab made of it, open Build › Pipeline editor, then Full configuration. Chapter 14 covers extends in full, including why changing a template job's rules is the classic trap (your rules replaced the template's).

greeting-service at the end of this part, on mainbuildbuildtestsecret-scanunit-testspackagebuild-imagedeploy▶deploy-productiondeploy-staging
Pipeline greeting-service at the end of this part, on main#
As text
  1. build: build
  2. test: secret-scan, unit-tests
  3. package: build-image
  4. deploy: deploy-production (manual), deploy-staging
What you can now do
  1. Include a central team's template at a pinned version, and a component with its inputs.
  2. Extend a hidden job and change only what you must, then check the result in Full configuration.
  3. Explain why your own rules replaced the template's, and keep theirs with a reference.
LabInclude the central team's templatesabout 15 min#
You need

greeting-service from the previous labs.

Steps
  1. Create a blank project named central-templates, beside greeting-service. Add the files above, plus this README.md:
    practice · central-templatesREADME.md
    # central-templates
    
    The central team's project in the "central team's templates" lab: a Node and a Java
    template with hidden jobs, and one CI/CD component. Tag it 1.0.0 before a service
    includes it.
  2. Under Code › Tags, select New tag. Name it 1.0.0, create it from main, and select Create tag.
  3. In greeting-service, replace .gitlab-ci.yml with this chapter's pipeline, with practice changed to your group or user, and push to main.
    practice · node-servicesteps/5-central-templates.yml
    # greeting-service (Node): the pipeline at the end of "The central team's templates".
    include:
      - project: practice/central-templates
        ref: 1.0.0
        file: templates/node.yml
      - component: $CI_SERVER_FQDN/practice/central-templates/secret-scan@1.0.0
        inputs:
          stage: test
    
    workflow:
      rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
          when: never
        - if: $CI_COMMIT_BRANCH
    
    stages:
      - build
      - test
      - package
      - deploy
    
    default:
      image: node:24.21.0-alpine
    
    variables:
      IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    
    build:
      stage: build
      script:
        - npm ci --cache .npm --prefer-offline
        - npm run build
      cache:
        key:
          files:
            - package-lock.json
        paths:
          - .npm/
      artifacts:
        paths:
          - dist/
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - test/**/*
            - package.json
            - package-lock.json
    
    unit-tests:
      extends: .node-test
      script:
        - npm ci --cache .npm --prefer-offline
        - npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=junit.xml
      artifacts:
        when: always
        reports:
          junit: junit.xml
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - test/**/*
            - package.json
            - package-lock.json
    
    build-image:
      stage: package
      image: docker:29.8.0-cli
      services:
        - docker:29.8.0-dind
      variables:
        DOCKER_TLS_CERTDIR: "/certs"
      script:
        - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
        - docker build --provenance=false --tag "$IMAGE" .
        - docker push "$IMAGE"
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    
    deploy-staging:
      stage: deploy
      image: alpine:3.22
      script:
        - echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN"
      environment:
        name: staging
        url: https://staging.greeting.example.com
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    
    deploy-production:
      stage: deploy
      image: alpine:3.22
      script:
        - echo "Deploying $IMAGE to production, with the token $DEPLOY_TOKEN"
      environment:
        name: production
        url: https://greeting.example.com
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
          when: manual
    practice · java-servicesteps/5-central-templates.yml
    # greeting-service (Java): the pipeline at the end of "The central team's templates".
    include:
      - project: practice/central-templates
        ref: 1.0.0
        file: templates/java.yml
      - component: $CI_SERVER_FQDN/practice/central-templates/secret-scan@1.0.0
        inputs:
          stage: test
    
    workflow:
      rules:
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
          when: never
        - if: $CI_COMMIT_BRANCH
    
    stages:
      - build
      - test
      - package
      - deploy
    
    default:
      image: maven:3.9.16-eclipse-temurin-25
    
    variables:
      MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
      MAVEN_CLI_OPTS: "--batch-mode --no-transfer-progress"
      IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    
    build:
      stage: build
      script:
        - mvn $MAVEN_CLI_OPTS package -DskipTests
      cache:
        key:
          files:
            - pom.xml
        paths:
          - .m2/repository/
      artifacts:
        paths:
          - target/greeting-service.jar
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - pom.xml
    
    unit-tests:
      extends: .maven-test
      artifacts:
        when: always
        reports:
          junit: target/surefire-reports/TEST-*.xml
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
        - changes:
            - src/**/*
            - pom.xml
    
    build-image:
      stage: package
      image: docker:29.8.0-cli
      services:
        - docker:29.8.0-dind
      variables:
        DOCKER_TLS_CERTDIR: "/certs"
      script:
        - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
        - docker build --provenance=false --tag "$IMAGE" .
        - docker push "$IMAGE"
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    
    deploy-staging:
      stage: deploy
      image: alpine:3.22
      script:
        - echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN"
      environment:
        name: staging
        url: https://staging.greeting.example.com
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    
    deploy-production:
      stage: deploy
      image: alpine:3.22
      script:
        - echo "Deploying $IMAGE to production, with the token $DEPLOY_TOKEN"
      environment:
        name: production
        url: https://greeting.example.com
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
          when: manual
  4. Open Build › Pipeline editor, then Full configuration, and find unit-tests.
  5. Open the pipeline and secret-scan's log.
What you should see

The Full configuration shows unit-tests as GitLab runs it, with the template's settings copied in:

GitLab buildsmerged configurationcaptured from GitLab
unit-tests:
  stage: test
  image: node:24.21.0-alpine
  script:
  - npm ci --cache .npm --prefer-offline
  - npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit
    --test-reporter-destination=junit.xml
  cache:
    key:
      files:
      - package-lock.json
    paths:
    - ".npm/"
    policy: pull
  extends: ".node-test"
  artifacts:
    when: always
    reports:
      junit:
      - junit.xml
  rules:
  - if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
  - changes:
    - src/**/*
    - test/**/*
    - package.json
    - package-lock.json

The stage, the image and the cache came from .node-test. The script is the job's own, and replaced the template's.

GitLab buildsmerged configurationcaptured from GitLab
unit-tests:
  stage: test
  image: maven:3.9.16-eclipse-temurin-25
  variables:
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  script:
  - mvn --batch-mode --no-transfer-progress test
  cache:
    key:
      files:
      - pom.xml
    paths:
    - ".m2/repository/"
    policy: pull
  extends: ".maven-test"
  artifacts:
    when: always
    reports:
      junit:
      - target/surefire-reports/TEST-*.xml
  rules:
  - if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
  - changes:
    - src/**/*
    - pom.xml

The job's own settings are only its artifacts and rules. Everything else came from .maven-test.

GitLab writes the merged file its own way: list items sit level with their key, and a long line wraps onto the next. extends is still listed, but its settings are already copied in.

secret-scan came from the component, with its inputs filled in, and found no key:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ echo "Looking for private keys in ."
Looking for private keys in .
$ if grep -rl "BEGIN [A-Z ]*PRIVATE KEY" .; then echo "A private key is committed"; exit 1; fi
⋯ lines omitted
Job succeeded

Read a new check's log once, to see that it really looked. In an Alpine image, grep is BusyBox's, which lacks some of GNU grep's options, such as --exclude-dir. Given one, it prints its usage and fails, and inside if that failure reads as "no key found": the job passes without looking.

Break it on purpose

Add a file to greeting-service containing the line -----BEGIN PRIVATE KEY-----, and push. secret-scan fails and names the file. Delete the file again.

Reuse at a glance#

Part IV · Reuse · Chapter 12 · 17 min read

Reuse means writing something once and using it in many places. In a large organisation, most of what your job does was written somewhere else. This book counts 37 ways that happens, in seven families. They run from YAML shortcuts inside one file to scripts built into an image.

Four questions sort them:

  • How far does it reach?
  • Is it pinned to a version?
  • Did your project choose it?
  • Can the merged view show it?

This chapter is the index to Part IV. The first table lists every mechanism, and the second turns what you see into the mechanism responsible. The chapters that follow give each mechanism its own card.

Monday, 09:00. Your first look at the real thing.

payments-api's file is nineteen lines. Its pipeline has eleven jobs, in five stages, and a job named policy-sbom that no file in the project mentions. Someone asks you to "just add a step after the build". Eleven jobs came from somewhere; you need to know from where before you touch anything. (the card)

You don't need to remember 37 of anything. You need to recognise which family a thing belongs to when you meet it, and know where that family is explained. This is how to use the chapter when a job surprises you:

Using this index when a job surprises you: the fingerprint table names the mechanism, and the mechanism's visibility says where to lookyesnoyesnoFind what you see in the fingerprinttable belowIt names the mechanism, and the chapterthat explains itDoes the merged view show it?Read it in Full configurationIs it applied when the pipeline iscreated?Check the pipeline graph and thesettings pagesIt happens while the job runs: read thejob log
Figure Using this index when a job surprises you: the fingerprint table names the mechanism, and the mechanism's visibility says where to look#
As text
  1. Find what you see in the fingerprint table below
  2. It names the mechanism, and the chapter that explains it
  3. Does the merged view show it? Yes: Read it in Full configuration. No: the next step.
  4. Is it applied when the pipeline is created? Yes: Check the pipeline graph and the settings pages. No: the next step.
  5. It happens while the job runs: read the job log

Seven families#

The families are ordered by distance. The first lives inside your own file, and the last lives outside YAML altogether. The further away a mechanism is, the harder it is to see from your project:

The seven families of reuse, ordered from closest to your file to furthest from it, with the number of mechanisms in each. close to your file far from it Inside one YAML file 3 mechanisms &anchor, *alias <<: merge keys .hidden-jobs Sharing between jobs 5 mechanisms extends !reference default: variables: parallel: matrix Pulling in files 5 mechanisms include: local include: project include: remote include: template include: component Parameters 4 mechanisms spec: inputs variables as inputs toggle variables pipeline inputs Whole pipelines 4 mechanisms parent–child dynamic child multi-project trigger API Organisation-wide 9 mechanisms custom config file settings variables security policies Auto DevOps and five more Outside the YAML 7 mechanisms toolbox images shell libraries build-tool reuse runner configuration and three more
Figure The seven families of reuse, ordered from closest to your file to furthest from it, with the number of mechanisms in each.#
FamilyMechanismsWhat they have in commonChapter
Inside one YAML file3plain YAML shortcuts that GitLab never sees as such13
Sharing between jobs5one job borrowing from another once the files are joined14
Pulling in files5the kinds of include, each fetching configuration from somewhere15
Parameters4values that change what shared configuration does16
Whole pipelines4one pipeline starting another17
Organisation-wide9settings and policies that reach projects without the project asking18
Outside the YAML7logic in images, scripts, build tools and runners19

Four questions for every mechanism#

For each mechanism, four answers tell you where to look, and whom to ask:

QuestionThe answersWhy it matters when something breaks
Scope: how far does it reach?one file · one project · cross-project · organisation-widetells you which repository or settings page to open
Binding: is it pinned?inline · copied once · pinned or floating · always livetells you whether a change somewhere else can break you without a commit of yours
Control: did you choose it?opt-in · inherited · enforcedtells you whether you can change it yourself, or have to ask someone
Visibility: does the merged view show it?shown · keyword only · not showntells you whether Full configuration answers the question, or you need another tool

All 37#

Hover over any cell for the long form of its answer.

MechanismScopeBindingControlPhaseMerged view
Inside one YAML file
YAML anchors and aliasesOne fileInlineOpt-inParseShown
YAML merge keyOne fileInlineOpt-inParseShown
Hidden jobsOne projectInlineOpt-inCompileShown
Sharing between jobs
extendsOne projectInlineOpt-inCompileShown
!reference tagsOne projectInlineOpt-inCompileShown
default:One projectInlineOpt-inCompileKeyword only
Top-level variablesOne projectInlineOpt-inCompileKeyword only
parallel: matrixOne projectInlineOpt-inCreateKeyword only
Pulling in files
include: localOne projectInlineOpt-inCompileShown
include: projectCross-projectPinned or floatingOpt-inCompileShown
include: remoteCross-projectPinned or floatingOpt-inCompileShown
include: templateOrganisation-wideAlways liveOpt-inCompileShown
include: componentCross-projectPinned or floatingOpt-inCompileShown
Parameters
spec: inputsCross-projectInlineOpt-inCompileShown
Variables as parametersOne projectInlineOpt-inRunKeyword only
Toggle variablesOne projectInlineOpt-inCreateKeyword only
Pipeline inputsOne projectInlineOpt-inCreateKeyword only
Whole pipelines
Parent–child pipelinesOne projectInlineOpt-inRunKeyword only
Dynamic child pipelinesOne projectInlineOpt-inRunNot shown
Multi-project pipelinesCross-projectAlways liveOpt-inRunNot shown
Pipeline trigger APICross-projectAlways liveOpt-inRunNot shown
Organisation-wide
Custom CI/CD configuration fileCross-projectPinned or floatingEnforcedCompileNot shown
Auto DevOpsOrganisation-wideAlways liveInheritedCompileNot shown
Instance and group CI/CD variablesOrganisation-wideAlways liveInheritedCreateNot shown
Instance template repositoryOrganisation-wideCopied onceOpt-inCompileShown
Pipeline execution policiesOrganisation-wideAlways liveEnforcedCreateNot shown
Scan execution policiesOrganisation-wideAlways liveEnforcedCreateNot shown
Compliance pipelinesOrganisation-widePinned or floatingEnforcedCompileNot shown
Merge request approval policiesOrganisation-wideAlways liveEnforcedRunNot shown
Project and file templatesOrganisation-wideCopied onceOpt-inCompileShown
Outside the YAML
Toolbox imagesOrganisation-widePinned or floatingOpt-inRunKeyword only
Scripts fetched at run timeCross-projectPinned or floatingOpt-inRunKeyword only
Shell function librariesCross-projectPinned or floatingOpt-inRunKeyword only
Build-tool reuseCross-projectPinned or floatingOpt-inRunNot shown
Internal command-line toolsOrganisation-widePinned or floatingOpt-inRunKeyword only
Runner configurationOrganisation-wideAlways liveEnforcedRunNot shown
GitLab FunctionsCross-projectPinned or floatingOpt-inRunKeyword only

From what you see to the mechanism#

Search this table for the thing in front of you. Every mechanism is in it at least once.

You seeMechanismFamily
&defaults after a key, and *defaults later in the same fileYAML anchors and aliasesInside one YAML file
<<: *defaults inside a jobYAML merge keyInside one YAML file
a job name that starts with a dot, such as .maven-base:Hidden jobsInside one YAML file
extends: .maven-base or extends: [.base, .with-cache]extendsSharing between jobs
- !reference [.snippets, aws_login] inside a script or rules list!reference tagsSharing between jobs
a top-level default: block setting image, before_script, tags or retrydefault:Sharing between jobs
inherit: default: false in a jobdefault:Sharing between jobs
a top-level variables: blockTop-level variablesSharing between jobs
inherit: variables: false in a jobTop-level variablesSharing between jobs
parallel: matrix: in a job, and job names like test: [21] in the pipelineparallel: matrixSharing between jobs
include: local: ci/build.yml, a wildcard such as ci/*.yml, or a bare path under include:include: localPulling in files
include: - project: devops/ci-templates with ref: and file:include: projectPulling in files
include: remote: https://…/pipeline.ymlinclude: remotePulling in files
include: template: Security/SAST.gitlab-ci.ymlinclude: templatePulling in files
include: - component: $CI_SERVER_FQDN/devops/components/ecr-push@1.4.0include: componentPulling in files
a spec: inputs: header above a --- linespec: inputsParameters
$[[ inputs.jdk ]] inside a templatespec: inputsParameters
a template reads $JAVA_VERSION, and your file sets it under variables:Variables as parametersParameters
variables such as SAST_DISABLED, SKIP_SONAR or DEPLOY_ENABLED, tested in rules: - if:Toggle variablesParameters
a form of inputs or variables on the Run pipeline pagePipeline inputsParameters
spec: inputs: at the top of a project's own .gitlab-ci.ymlPipeline inputsParameters
trigger: include: ci/deploy.ymlParent–child pipelinesWhole pipelines
a downstream box labelled as a child pipelineParent–child pipelinesWhole pipelines
trigger: include: - artifact: generated.yml with job: generateDynamic child pipelinesWhole pipelines
trigger: project: platform/deployerMulti-project pipelinesWhole pipelines
a downstream pipeline that belongs to another projectMulti-project pipelinesWhole pipelines
curl --request POST …/trigger/pipeline with a trigger token, inside a scriptPipeline trigger APIWhole pipelines
pipelines run but the repository has no .gitlab-ci.ymlCustom CI/CD configuration fileOrganisation-wide
the project's CI/CD settings name a path such as ci/web.yml@devops/ci-templatesCustom CI/CD configuration fileOrganisation-wide
no .gitlab-ci.yml, yet jobs appear with names such as build, test, code_quality and container_scanningAuto DevOpsOrganisation-wide
a job sees a variable that no file definesInstance and group CI/CD variablesOrganisation-wide
a variable listed in a group's or the instance's CI/CD settingsInstance and group CI/CD variablesOrganisation-wide
a .gitlab-ci.yml created from the Web Editor's template list, matching a file in the administrators' templates projectInstance template repositoryOrganisation-wide
stages named .pipeline-policy-pre or .pipeline-policy-postPipeline execution policiesOrganisation-wide
jobs that appear in the pipeline but in no file you can findPipeline execution policiesOrganisation-wide
a job name ending in :policy- followed by two numbersPipeline execution policiesOrganisation-wide
scanner jobs named with a hyphen and a number, such as secret-detection-1Scan execution policiesOrganisation-wide
scanner jobs run although no file includes a scanner templateScan execution policiesOrganisation-wide
security scans that run on a schedule nobody in the project createdScan execution policiesOrganisation-wide
a compliance framework label on the project, and jobs from a file the project does not includeCompliance pipelinesOrganisation-wide
a merge request needs extra approvals because of security findings or licencesMerge request approval policiesOrganisation-wide
a .gitlab-ci.yml that matches a starter template, while the central copy has moved onProject and file templatesOrganisation-wide
image: registry.example.com/devops/ci-tools:3.2 and a script that calls a command no repository definesToolbox imagesOutside the YAML
curl -sSL … | bash, or a git clone of a scripts repository, inside script:Scripts fetched at run timeOutside the YAML
source /opt/ci/lib.shShell function librariesOutside the YAML
functions defined in a hidden job and pulled into before_script with !referenceShell function librariesOutside the YAML
script: mvn -P ci verify with a <parent> POM owned by another teamBuild-tool reuseOutside the YAML
npm run ci driven by a shared configuration packageBuild-tool reuseOutside the YAML
script: mc deploy --env prod: a company tool, not a public oneInternal command-line toolsOutside the YAML
behaviour that changes with the runner: environment variables, mounted files or cloud permissions that no file setsRunner configurationOutside the YAML
a job with run: and a list of steps using func:, and no script:GitLab FunctionsOutside the YAML

Pinned, floating, copied#

Binding is the question that most often explains "it broke and nobody changed anything". Every shared file is read at some ref: a branch, a tag or a commit:

  • A commit never changes.
  • A tag normally stays where it was put, although whoever owns it can move it.
  • A branch moves every time someone merges into it.

MegaCorp shows all three kinds of binding:

megacorp/payments/payments-api.gitlab-ci.ymlinclude
include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file: pipelines/java-service.yml
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api
  • Pinned. payments-api includes the golden pipeline at the tag v4.2.0, and the golden pipeline pins everything it includes to the same tag. Nothing upstream can change what payments-api gets until someone edits a version number.
  • Floating. web-portal's settings point at a pipeline file in ci-templates without naming a ref. It follows whatever that project's default branch holds today.
  • Copied. billing-batch's file was created from a starter template in 2019, and has never heard from the template since.
What you can now do
  1. Sort any reuse mechanism you meet into one of five families, and know which chapter covers it.
  2. Recognise a mechanism from what it looks like in the file, with the fingerprints table.
  3. Say which of the four moments each mechanism acts at, and so where to look for its result.
Spot the bugName the mechanism#

Four things a developer found while investigating MegaCorp pipelines:

  1. A job in billing-batch has a line <<: *defaults.
  2. A pipeline has a stage called .pipeline-policy-post, holding a job that no file in the project mentions.
  3. web-portal's pipelines run, but the repository has no .gitlab-ci.yml.
  4. A before_script contains - !reference [.snippets, aws_login].
QuestionWhich mechanism explains each one, and where would you look next?
Show the answer
  1. A YAML merge key. Look for &defaults in the same file.
  2. A pipeline execution policy. Look at the group's security policies.
  3. A custom CI/CD configuration file, set in the project's CI/CD settings. Auto DevOps can also produce pipelines without a file, but its jobs have GitLab's names.
  4. A !reference tag. Search the included files for .snippets:.

YAML-level reuse: anchors, merge keys and hidden jobs#

Part IV · Reuse · Chapter 13 · 12 min read

YAML has its own copy and paste, which happens before GitLab reads a single keyword:

  • An anchor, written &name, gives a piece of YAML a name.
  • An alias, written *name, pastes that piece somewhere else.
  • A merge key, written <<:, pastes the keys of a named map into another map.

All three work only inside the file that defines them. Hidden jobs, whose names start with a dot, are GitLab's own addition: jobs that never run and exist only to be copied from. You'll meet all of this in older files, and in any file whose author never needed to reach across an include.

Monday, 14:30. A merge key that didn't merge.

An older template in billing-batch uses <<: [*base, *java]. You add a variable to *java, and the job never sees it. The same change to *base works. Both anchors are in the same file. Why does the order matter? (There is no symptom card for this one: the answer is in what wins in a merge.)

Anchors, aliases and merge keys#

Here is the whole idea in seven lines:

.base: &base              # "&base" gives this map the name base
  image: node:22
  tags: [linux]

test:
  <<: *base               # paste the keys of the map named base
  script: npm test

GitLab never sees &base or *base. The YAML reader has already replaced them, so GitLab reads test as if it had been written out in full:

test:
  image: node:22
  tags: [linux]
  script: npm test

The dot in front of .base matters too. Without it, GitLab would try to run base as a job of its own.

MegaCorp's oldest project still shares configuration this way:

megacorp/legacy/billing-batch.gitlab-ci.ymlanchors
.defaults: &defaults
  image: registry.example.com/megacorp/devops/ci-tools:2.9
  tags: [megacorp-shared]
  only:
    - branches
    - tags

build:
  <<: *defaults
  stage: build
  script:
    - ./gradlew assemble

The same three pieces are at work:

  • &defaults names the map under .defaults.
  • <<: *defaults copies that map's keys into build.
  • The dot in front of .defaults stops GitLab from treating the anchor's home as a job.

By the time GitLab sees the file, build simply has an image, tags and only of its own. That is why anchors are the one kind of reuse that leaves no trace in the merged configuration. It shows the result, with the anchors already replaced. Here is another of billing-batch's jobs, next to what GitLab made of it:

You wrotemegacorp/legacy/billing-batch · .gitlab-ci.yml
notify-reports:
  <<: *defaults
  stage: deploy
  only: [tags]
  script:
    - >
      curl --fail --request POST
      --form "token=$REPORTS_TRIGGER_TOKEN"
      --form "ref=main"
      --form "variables[BILLING_VERSION]=$CI_COMMIT_TAG"
      "https://gitlab.example.com/api/v4/projects/4242/trigger/pipeline"
GitLab buildsmerged configurationcaptured from GitLab
notify-reports:
  image: registry.example.com/megacorp/devops/ci-tools:2.9
  tags:
  - megacorp-shared
  only:
  - tags
  stage: deploy
  script:
  - 'curl --fail --request POST --form "token=$REPORTS_TRIGGER_TOKEN" --form "ref=main"
    --form "variables[BILLING_VERSION]=$CI_COMMIT_TAG" "https://gitlab.example.com/api/v4/projects/4242/trigger/pipeline"

    '

notify-reports merges *defaults, which says only: [branches, tags], and then writes only: [tags] itself. A key written in the job always beats a merged one, so the job runs only for tags.

Anchors have one hard limit: they work only inside the file that defines them.

An alias finds its anchor only inside the same file. Across an include it can't, which is why central teams use !reference instead. templates/base.yml .defaults: &defaults build: <<: *defaults works: same file .gitlab-ci.yml, which includes base.yml test: <<: *defaults fails: anchors don't cross an include
Figure An alias finds its anchor only inside the same file. Across an include it can't, which is why central teams use !reference instead.#

What wins in a merge#

Merge keys follow two rules, and both differ from extends, which chapter 14 covers:

SituationMerge key <<:extends
the job sets a key the source also setsthe job winsthe job wins
both sides have a map, such as variablesthe job's map replaces the source's whole map: merge keys are shallowthe maps merge key by key
two sources set the same key<<: [*a, *b]: the earlier one, *a, winsextends: [.a, .b]: the later one, .b, wins
the source is in an included fileimpossible: anchors stop at the file boundaryworks
Gotcha

The third row catches people who know one mechanism and assume the other behaves the same way. When you see a list after <<:, the first entry is the one that sticks.

The local GitLab 19.3 confirmed both rules. A job written <<: [*a, *b] took its image and its variables from a, the first in the list. A job that merged *a and wrote its own variables: kept only its own variable. Every variable from the anchor was gone.

Inside one YAML file#
YAML anchors and aliases
Scope
One file
Binding
Inline
Control
Opt-in
Phase
Parse
Merged view
Shown
Looks like
.ci_image: &ci_image registry.example.com/megacorp/devops/ci-tools:2.9

build:
  image: *ci_image
What GitLab does

The YAML reader resolves anchors before GitLab reads the configuration. An alias becomes a copy of whatever its anchor names: a string, a list or a map. GitLab never sees the anchor, only the copy, so the Full configuration view shows the result with every alias replaced.

How to trace it

An alias always has its anchor in the same file. Search that file for &name, and nowhere else.

Gotchas
  • An anchor in an included file is invisible to the file that includes it. Central teams use !reference instead (chapter 14).
  • Editing the anchored block changes every alias at once, with no sign at the places that use it.
Inside one YAML file#
YAML merge key
Scope
One file
Binding
Inline
Control
Opt-in
Phase
Parse
Merged view
Shown
Looks like
build:
  <<: *defaults

test:
  <<: [*defaults, *java21]
What GitLab does

The merge key copies the keys of the named map into the current map, except for keys the current map already has. The copy is shallow: a key whose value is a map, such as variables, is copied or ignored whole, never merged. With a list of maps, earlier maps win over later ones.

How to trace it

Find the anchor in the same file, then compare keys. Any key the job writes itself wins entirely. Anything else comes from the first map in the list that has it.

Gotchas
  • Adding one variable to a job that merges *defaults deletes all of the anchored variables for that job.
  • The order in a list is the opposite of extends.

Hidden jobs#

Any top-level key that starts with a dot is ignored as a job. That makes hidden jobs the raw material of nearly every template library:

  • They hold anchors, as in billing-batch.
  • They are the parents that extends copies from.
  • They hold the snippets that !reference pastes.

MegaCorp's base template is one:

megacorp/devops/ci-templatestemplates/base.ymlmegacorp-base
.megacorp-base:
  variables:
    MC_TEAM: unknown
  artifacts:
    expire_in: 7 days
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]
    - !reference [.rules, release-tag]
Inside one YAML file#
Hidden jobs
Scope
One project
Binding
Inline
Control
Opt-in
Phase
Compile
Merged view
Shown
Looks like
.megacorp-base:
  artifacts:
    expire_in: 7 days

.snippets:
  functions:
    - source /opt/megacorp/lib/ci-lib.sh
What GitLab does

GitLab doesn't process a job whose name begins with a dot. It never runs, and it never appears in a pipeline. Its content is still there to borrow, through extends, !reference or YAML anchors. Because it is never treated as a job, it can also hold keys that aren't CI/CD keywords at all, like functions above.

How to trace it

Search every included file for the name, with its dot and colon: .megacorp-base:. With several includes, check each one. The name alone doesn't tell you which file defines it.

Gotchas
  • A dot is also the quickest way to switch a job off. A job that "disappeared" from the pipeline may simply have been hidden.
  • If two included files define the same hidden job, they merge like any duplicated key, and the file included last wins where they clash.
When you write it
Do this
When two jobs should share settings, write a hidden job and copy it with extends. Use a YAML anchor only inside one file.
Why
extends works across every included file. An anchor stops at the end of the file it is written in, so it cannot reach a job in another file.
Watch out
If you do use a merge key, remember two rules. A key the job writes itself replaces the merged one completely, and when you merge a list of aliases, the first alias wins.
What you can now do
  1. Read anchors, aliases and merge keys in an older file, and say which key wins when they collide.
  2. Explain why an anchor cannot reach a job in another file, and what replaced it.
  3. Recognise a hidden job, and know that it is the base of every modern template.
Spot the bugThe Gradle job on the wrong JDK#

Someone modernised billing-batch's build job so that it builds on JDK 21. This is billing-batch/.gitlab-ci.yml, modified:

.defaults: &defaults
  image: registry.example.com/megacorp/devops/ci-tools:2.9
  variables:
    GRADLE_OPTS: "-Dorg.gradle.daemon=false"

.java21: &java21
  image: registry.example.com/megacorp/devops/ci-tools/gradle:8-jdk21

build:
  <<: [*defaults, *java21]
  variables:
    BILLING_ENV: ci
  script:
    - ./gradlew assemble

The job still reports JDK 11, and the Gradle daemon is starting again.

QuestionWhich image does build use, and why has GRADLE_OPTS gone?
Show the answer

It uses ci-tools:2.9. In a merge-key list the earlier map wins, and *defaults comes first. GRADLE_OPTS is gone because the job writes its own variables:. Merge keys are shallow, so that map replaces the anchored one outright.

Put *java21 first and repeat GRADLE_OPTS in the job. Or move both anchors into hidden jobs and use extends: [.defaults, .java21], which merges maps key by key and lets the later parent win. See merge keys.

extends, !reference and default#

Part IV · Reuse · Chapter 14 · 27 min read

Five keywords let one job borrow configuration from somewhere else. Between them they explain most of the "where did that line come from?" moments in an enterprise pipeline: extends, !reference, default:, top-level variables: and parallel: matrix.

One rule explains most of their surprises: maps merge key by key, but lists are replaced whole. A map is a set of named keys, such as variables. A list is a sequence of items, such as script (YAML in five minutes). If you override one line of a template's rules, script or before_script, you have replaced all of it.

Tuesday, 10:20. One line, three jobs changed.

You add before_script: [echo starting] to maven-test. The job now fails at once: mc_log: command not found. The function comes from the central team's default. Your one line was meant to add, not remove. What did GitLab do with it? (the card)

The five, side by side#

In MegaCorp's template library, almost nothing is a real job. The templates are hidden jobs, whose names start with a dot, and they borrow from each other through the five keywords below. The project's own file then turns a few of them into real jobs.

KeywordWhat it sharesHow it combines with the jobWorks across included files
extendsa whole job, from one or more parentsmaps merge key by key, lists are replaced, and the later parent winsyes
!referenceone section of another job, such as its script or rulespastes exactly what it points at, where you put ityes
default:eleven keywords, such as image and before_script, for every jobnone: a job's own keyword replaces the default outrightyes, wherever it is defined
top-level variables:variables for every joba job's own variables outrank themyes
parallel: matrixone job definition, run once per combination of valueseach generated job gets its own copy of the variablesnot applicable

Before tracing anything, open the one view that resolves all five for you: Build › Pipeline editor, then the Full configuration tab. It shows the configuration with included files copied in, extends merged, YAML anchors replaced and !reference tags replaced.

extends: a job inherits a job#

Take the maven-test job in payments-api's pipeline. The project's own file only adds one variable to it. The job itself is defined three files away: pipelines/java-service.yml says maven-test: extends: .maven-test. .maven-test extends .maven-base, which extends .megacorp-base. Here is the middle of that chain:

megacorp/devops/ci-templatestemplates/java-maven.ymlmaven-base
.maven-base:
  extends: .megacorp-base
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
  variables:
    JAVA_VERSION: "21"
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  cache:
    key:
      files: [pom.xml]
    paths: [.m2/repository]
  before_script:
    # a job's own before_script replaces default:before_script, so re-add the functions
    - !reference [.snippets, functions]
    - !reference [.snippets, maven_settings]

Here is what GitLab builds from the whole chain, next to three of the four files that feed it:

You wrotemegacorp/payments/payments-api · .gitlab-ci.yml
maven-test:
  variables:
    MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"
You wrotemegacorp/devops/ci-templates · pipelines/java-service.yml
maven-build:
  extends: .maven-build

maven-test:
  extends: .maven-test

image-build:
  extends: .image-build
  needs: [maven-build]
You wrotemegacorp/devops/ci-templates · templates/java-maven.yml
.maven-test:
  extends: .maven-base
  stage: test
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JDK}
  parallel:
    matrix:
      - JDK: ["17", "21"]
  script:
    - mvn $MAVEN_CLI_OPTS verify
  artifacts:
    when: always
    reports:
      junit: target/surefire-reports/TEST-*.xml
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]
GitLab buildsmerged configurationcaptured from GitLab
maven-test:
  variables:
    MC_TEAM: unknown
    JAVA_VERSION: '21'
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
    MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"
  artifacts:
    expire_in: 7 days
    when: always
    reports:
      junit:
      - target/surefire-reports/TEST-*.xml
  rules:
  - - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
  - - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  - - if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
  extends: ".maven-test"
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JDK}
  cache:
    key:
      files:
      - pom.xml
    paths:
    - ".m2/repository"
  before_script:
  - - source /opt/megacorp/lib/ci-lib.sh
  - - mkdir -p .m2
    - mc_maven_settings > .m2/settings.xml
  stage: test
  parallel:
    matrix:
    - JDK:
      - '17'
      - '21'
  script:
  - mvn $MAVEN_CLI_OPTS verify

Five things to notice, and each is the merge rule at work:

  • Variables from three levels become one map. MC_TEAM comes from .megacorp-base, JAVA_VERSION and MAVEN_OPTS from .maven-base, and MAVEN_CLI_OPTS from the project's file.
  • So do the artifacts. expire_in: 7 days comes from .megacorp-base, and when and reports come from .maven-test. artifacts is a map, so the two merged.
  • The image is .maven-test's, not .maven-base's. When the same key appears at two levels, the later one wins.
  • The rules are .maven-test's alone. .megacorp-base had a fourth rule, for release tags, but a list never merges. That is why maven-test does not run in tag pipelines while maven-build does.
  • Pasted snippets show as lists inside the list. The rules and before_script entries that start with - - were pasted in by !reference, covered in the next section. GitLab flattens them into one list when the job runs.

Whenever you wonder whether your own key merged with a template's or replaced it, ask these questions in order:

What extends does with each key a job sets: maps merge, lists are replaced, and anything else is simply overwrittenyesnoyesnoIs the key's value a map, such asvariables, cache or artifacts?The two maps merge key by key; ona clash, the job's value winsIs it a list, such as script,before_script or rules?The job's list replaces thetemplate's whole listIt is a single value, such as image orstage: the job's value replaces thetemplate's
Figure What extends does with each key a job sets: maps merge, lists are replaced, and anything else is simply overwritten#
As text
  1. Is the key's value a map, such as variables, cache or artifacts? Yes: The two maps merge key by key; on a clash, the job's value wins. No: the next step.
  2. Is it a list, such as script, before_script or rules? Yes: The job's list replaces the template's whole list. No: the next step.
  3. It is a single value, such as image or stage: the job's value replaces the template's
How extends combines a template with the job that extends it: maps merge key by key, lists are replaced whole. templates/java-maven.yml .maven-base: image: maven:3.9-jdk${JAVA_VERSION} variables: JAVA_VERSION: "21" MAVEN_OPTS: … before_script: - source ci-lib.sh - mc_maven_settings … your .gitlab-ci.yml integration-test: extends: .maven-base variables: SPRING_PROFILE: it before_script: - ./start-db.sh what GitLab builds integration-test: image: maven:3.9-jdk${JAVA_VERSION} inherited variables: JAVA_VERSION: "21" MAVEN_OPTS: … SPRING_PROFILE: it maps merge key by key before_script: - ./start-db.sh - source ci-lib.sh - mc_maven_settings … lists are replaced whole Struck-through lines came from the template and are gone: the job's own before_script replaced the whole list.
Figure How extends combines a template with the job that extends it: maps merge key by key, lists are replaced whole.#
Sharing between jobs#
extends
Scope
One project
Binding
Inline
Control
Opt-in
Phase
Compile
Merged view
Shown
Looks like
maven-test:
  extends: .maven-test

integration-test:
  extends: [.maven-base, .with-database]
What GitLab does

GitLab resolves the chain of parents first, then merges each level into the next. Maps such as variables, cache and artifacts merge key by key. Where the same key appears twice, the later definition wins: the job over its parent, and the second parent over the first. Lists such as script, before_script and rules are replaced whole. A job can extend a hidden job defined in any included file. Chains can be eleven levels deep, though GitLab's own guidance is to stay within three.

How to trace it

Open Full configuration in the pipeline editor, where the job appears with everything it inherited already merged in. To see which file contributed a line, follow the extends names upwards. Search the included files for each parent's name, such as .maven-test:. Expect every hop to be in a different file.

Gotchas
  • Setting rules, script or before_script on your job throws away the template's version entirely (symptom card).
  • Setting one variable is safe, because maps merge. People who have been bitten by lists often copy a whole variables: block when a single key would do.
  • A job's variables inherited through extends outrank the project's top-level variables (see top-level variables).
Didn't run, or ran when it shouldn'tMerging and inheritance#
You gave a template job one rule of your own. Now it runs in pipelines it never used to, or has vanished from ones it always ran in.
Cause

Your rules: replaced the template's entire list. Lists never merge, so every rule the template had is gone, including the ones that skipped scheduled pipelines or added tag pipelines.

Confirm

In Build › Pipeline editor › Full configuration, find the job. Its rules: shows only your lines.

Fix

Rebuild the list: paste the template's rules back with !reference and put your rule where it belongs. The first matching rule wins, so order matters.

maven-test:
  rules:
    - if: $SKIP_TESTS == "true"
      when: never
    - !reference [.maven-test, rules]

!reference: paste one section#

extends takes a whole job. !reference takes one piece of one, and pastes it exactly where you write it. MegaCorp keeps its script fragments in a hidden job whose keys aren't CI/CD keywords at all. GitLab's own documentation uses the same pattern.

megacorp/devops/ci-templatestemplates/snippets.ymlsnippets
.snippets:
  functions:
    - source /opt/megacorp/lib/ci-lib.sh
  aws_login:
    - mc_aws_login "$AWS_ROLE_ARN"
  acr_push:
    - mc_acr_push "$IMAGE:$CI_COMMIT_SHA"
  maven_settings:
    - mkdir -p .m2
    - mc_maven_settings > .m2/settings.xml

The rules library works the same way. Named lists of rules are assembled job by job, which is how .maven-test got its three rules:

megacorp/devops/ci-templatestemplates/rules.ymlrules-library
.rules:
  mr:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  default-branch:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  release-tag:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
  never-on-schedule:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never

You will meet the older way of doing this in long-lived projects: a YAML anchor. The two look alike, but they don't reach equally far:

.login: &login
  - mc_aws_login "$AWS_ROLE_ARN"

image-build:
  before_script: *login

This works only if .login is in the same file as image-build.

image-build:
  before_script:
    - !reference [.snippets, aws_login]

This works wherever .snippets is defined, including in an included file.

Anchors stop at the file boundary

A YAML anchor exists only inside the file that defines it. An included file's anchors are invisible to the file that includes it. The workaround that central teams use is !reference, which GitLab resolves after it has assembled every included file.

Sharing between jobs#
!reference tags
Scope
One project
Binding
Inline
Control
Opt-in
Phase
Compile
Merged view
Shown
Looks like
before_script:
  - !reference [.snippets, functions]
rules:
  - !reference [.rules, mr]
variables:
  NEXUS_URL: !reference [.vars, variables, NEXUS_URL]
What GitLab does

GitLab replaces the tag with the configuration it names: a job, then a key path inside it. A list pasted into a list is flattened. The path can name any key in a hidden job, not only a CI/CD keyword. The tags can be nested up to ten levels deep in script, before_script and after_script. They work across included files, but inputs can't be used inside the path.

How to trace it

The tag names its source: the first item is the job to search for, such as .snippets:. In the Full configuration view the pasted content is already in place. Nested tags sometimes show extra leading hyphens there, which GitLab documents as expected and harmless.

Gotchas
  • The paste happens when GitLab assembles the configuration, but the pasted lines run inside your job. Functions a snippet defines exist only in the job that pasted it.
  • A snippet pasted into before_script disappears the moment a job redefines before_script without pasting it again.

default: the layer you never see#

default: sets a keyword for every job that doesn't set it itself. It lives in a template, so no project file mentions it, yet it is in every job.

megacorp/devops/ci-templatestemplates/base.ymldefault
default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  tags: [megacorp-shared]
  interruptible: true
  retry:
    max: 2
    when: [runner_system_failure, stuck_or_timeout_failure]
  before_script:
    - !reference [.snippets, functions]
    - mc_log "job $CI_JOB_NAME on $CI_COMMIT_REF_NAME"

Eleven keywords can have defaults: after_script, artifacts, before_script, cache, hooks, id_tokens, image, interruptible, retry, services and tags. Unlike extends, a default never merges with anything. If a job has the keyword, the default for that keyword simply doesn't apply.

Gotcha

MegaCorp's default before_script loads the shell function library that every other template relies on. The moment a job defines its own before_script, even a single line, the library is no longer loaded. That is why .maven-base pastes the functions back in, and says so in a comment.

MegaCorp uses !reference inside default: as well as inside jobs. GitLab's documentation shows it only in jobs, but the local GitLab 19.3 accepts a default: before_script built from !reference without complaint.

Opting out with inherit#

A job can refuse some or all of the defaults with inherit. inherit: default: takes true, false, or a list of the keywords to keep. inherit: variables: does the same for top-level variables.

megacorp/payments/payments-api.gitlab-ci.ymlinherit
publish-docs:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
  inherit:
    default: [tags, retry]
    variables: false
  script:
    - mkdocs build --strict
  rules:
    - !reference [.rules, default-branch]

publish-docs keeps the organisation's runner tags and retry policy. It drops the default image, which it overrides anyway, and the default before_script, so none of the shell functions are loaded. It also receives none of the top-level variables.

Legacyimage, services, cache, before_script and after_script at the top level

Older files set these five keywords at the top level, outside any job, as a way of applying them to every job. GitLab has deprecated that form. The replacement is the same keywords under default:. When you meet the old form, read it exactly as you would a default: block.

Sharing between jobs#
default:
Scope
One project
Binding
Inline
Control
Opt-in
Phase
Compile
Merged view
Keyword only
Looks like
default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  before_script:
    - !reference [.snippets, functions]

publish-docs:
  inherit:
    default: [tags, retry]
What GitLab does

For each of the eleven keywords it can hold, default: supplies a value to every job that doesn't have one. A job's own value replaces the default outright; the two are never merged. inherit: default: turns the defaults off, entirely or keyword by keyword. Defaults are not passed to downstream pipelines, which start again from their own configuration.

How to trace it

Search the included files for default:. Then compare it with the job in the Full configuration view: a keyword the job doesn't show was never overridden, so the default applies. Check for inherit: on the job before concluding anything.

Gotchas
  • A single line of before_script on a job removes the whole default before_script, including anything the organisation put there.
  • A child or multi-project pipeline doesn't get the parent's defaults. The same template can behave differently one pipeline downstream.

Top-level variables#

A variables: block at the top of any file applies to every job. Blocks from included files and from the project's own file merge into one map:

megacorp/devops/ci-templatestemplates/base.ymlvariables
variables:
  GIT_DEPTH: "20"
  AWS_REGION: eu-west-2
  MC_TEMPLATES_VERSION: "4.2.0"
megacorp/payments/payments-api.gitlab-ci.ymlvariables
variables:
  JAVA_VERSION: "21"
  MC_TEAM: payments

Here, payments-api sets MC_TEAM: payments, but every one of its jobs still sees unknown. The reason is in the base template:

megacorp/devops/ci-templatestemplates/base.ymlmegacorp-base
.megacorp-base:
  variables:
    MC_TEAM: unknown
  artifacts:
    expire_in: 7 days
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]
    - !reference [.rules, release-tag]

.megacorp-base sets MC_TEAM as a job variable, and every template extends it. Job variables outrank top-level variables, so the project's value never reaches a job. The ranking goes further than YAML: a variable set in the project's, group's or instance's CI/CD settings outranks everything in YAML. Chapter 26 gives the full ladder.

Sharing between jobs#
Top-level variables
Scope
One project
Binding
Inline
Control
Opt-in
Phase
Compile
Merged view
Keyword only
Looks like
variables:
  GIT_DEPTH: "20"
  AWS_REGION: eu-west-2

publish-docs:
  inherit:
    variables: false
What GitLab does

Top-level variables become default variables for every job. Blocks from every included file and the project's own file merge into one map, with the project's file winning on a clash. A job's own variables, including those it inherits through extends, outrank them. inherit: variables: can turn them off for a job, entirely or by name.

How to trace it

Look for the variable in three places, in this order:

  • the job's own variables: in the Full configuration view
  • the top-level variables: blocks
  • the project's, group's and instance's CI/CD settings, which outrank both of the above
Gotchas
  • A project cannot override a variable that a template sets at job level by setting it at the top of its own file.
  • A top-level variable is not the last word. A CI/CD variable of the same name set in the UI replaces it without a trace in any file.

parallel: matrix: one definition, several jobs#

.maven-test, quoted in the comparison earlier, asks for a matrix of two JDK versions. The pipeline shows one job per value, named after the job and the value:

Jobs built from java-maven.yml and container.yml in a payments-api merge request pipelinebuildmaven-buildtestmaven-test: [17]maven-test: [21]packageimage-build
Pipeline Jobs built from java-maven.yml and container.yml in a payments-api merge request pipeline#
As text
  1. build: maven-build
  2. test: maven-test: [17], maven-test: [21]
  3. package: image-build

Needs: image-build waits only for maven-build.

Each generated job gets its own JDK variable, which the image name uses. A matrix can produce at most 200 jobs. When another job needs only some of them, needs: parallel: matrix selects them by value.

Sharing between jobs#
parallel: matrix
Scope
One project
Binding
Inline
Control
Opt-in
Phase
Create
Merged view
Keyword only
Looks like
.maven-test:
  parallel:
    matrix:
      - JDK: ["17", "21"]
What GitLab does

GitLab creates one job for every combination of the matrix values. It names each <job>: [<value>, …], for example maven-test: [17], and gives each its own copy of the variables. The matrix is expanded when the pipeline is created, not when the configuration is assembled.

How to trace it

A job name ending in square brackets is a matrix job. Strip the brackets and search for the name; here, maven-test leads to .maven-test and its parallel: block. The Full configuration view shows the matrix definition, not the generated jobs; the pipeline graph shows those.

Gotchas
  • needs: [maven-test] depends on every job in the matrix. To depend on one, use needs: parallel: matrix.
  • Matrix values are plain variables. A value that collides with a variable set elsewhere follows the usual precedence rules.

Which of these jobs run where#

Put the rules from this chapter together and you can predict, without running anything, which of payments-api's jobs exist in each kind of pipeline:

jobfeature branch pushmerge requestmaintag v1.4.0nightly on main
maven-buildnot in the pipelinerunsrunsrunsnot in the pipeline
maven-test: [17] and [21]not in the pipelinerunsrunsnot in the pipelinenot in the pipeline
image-buildnot in the pipelinerunsrunsrunsnot in the pipeline
publish-docsnot in the pipelinenot in the pipelinerunsnot in the pipelineruns
payments-api: the jobs in this chapter, by pipeline·not in the pipeline✓runs

Two cells are worth a second look:

  • Tag pipelines have no tests. maven-test's own rules replaced the base template's list, which was the only one with a tag rule.
  • publish-docs runs every night. Its only rule is "on the default branch", and nothing tells it to skip scheduled pipelines.
How GitLab resolves an extends chaindeep dive

GitLab assembles the configuration first: every included file is copied in, and the project's file is merged on top. Only then does it resolve extends. For each job it walks up the chain of parents, .megacorp-base then .maven-base then .maven-test, and merges downwards, each level into the next.

At every level the same reverse deep merge applies. Maps are merged key by key, the later level wins a clash, and a list replaces the list above it whole. With several parents, extends: [.a, .b], .b is applied after .a and wins any clash between them.

!reference tags are resolved in the same assembly pass, which is why they can point into included files when YAML anchors cannot. By the time the pipeline is created, none of these keywords exists any more. There are only complete jobs, which is exactly what the Full configuration view shows.

When you write it
Do this
When you extend a template job, change only what you must, and then open the Full configuration to see the result.
Why
Settings that are lists, such as rules, script and before_script, replace the template's whole list. Settings that are maps, such as variables, merge key by key. So one line of yours can throw away everything the template had.
Watch out
To keep a template's rules and add your own, paste its rules into your list with !reference. The card your rules replaced the template's shows what happens when you forget.
What you can now do
  1. Trace a job through its extends chain to the file that defines each setting.
  2. Predict what a job's own key does to the inherited one: merge for maps, replace for lists.
  3. Keep a template's list and add your own line to it with a reference.
Spot the bugThe team name that never changes#

A dashboard groups jobs by the MC_TEAM variable. It files every payments-api job under unknown, even though payments-api's own file sets it:

megacorp/payments/payments-api.gitlab-ci.ymlvariables
variables:
  JAVA_VERSION: "21"
  MC_TEAM: payments
QuestionWhy do the jobs never see payments, and what is the least invasive fix?
Show the answer

.megacorp-base sets MC_TEAM: unknown as a job variable, and every job reaches it through extends. Job variables outrank top-level variables, so the project's value is overridden in every job.

The least invasive fix is a project CI/CD variable MC_TEAM=payments in the project's settings, because settings variables outrank anything in YAML. The better long-term fix is for the template owners to move MC_TEAM to top-level variables, so that projects can override it. See top-level variables.

Spot the bugOne line that broke Maven#

The payments team needed a database for maven-test, so they added one line to their file. This is payments-api/.gitlab-ci.yml, modified:

maven-test:
  before_script:
    - ./scripts/start-db.sh
  variables:
    MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"

Now both maven-test jobs fail at once. Maven complains that .m2/settings.xml does not exist.

QuestionThey only added a line. Where did settings.xml go, and how should they have added their script?
Show the answer

before_script is a list, and lists are replaced whole. The job's one-line list replaced .maven-base's list. That list was the one that loaded the function library and wrote .m2/settings.xml.

Keep the template's lines by pasting them back first:

maven-test:
  before_script:
    - !reference [.maven-base, before_script]
    - ./scripts/start-db.sh

See the symptom card for the same trap with rules, and extends.

include: pulling in files#

Part IV · Reuse · Chapter 15 · 14 min read

include copies configuration from other files into yours, before anything else happens. There are five kinds, and they differ only in where the file comes from:

  • local: your own repository
  • project: another project
  • remote: a URL
  • template: GitLab itself
  • component: a versioned component

Every included file is merged into one configuration, and your own file wins any clash. The most important thing to read on an include is not the file name but the ref: the branch, tag or commit it reads from. The ref decides whether a change somewhere else can reach you without a commit of yours.

Wednesday, 08:45. The pipeline changed overnight.

Nobody committed to payments-api since Friday, but this morning's pipeline has a new stage and a job you have never seen. Your file has one include line, and it hasn't changed either. Something you point at moved. What, and how do you stop it moving? (the card)

Five kinds#

KindFetches fromBindingYou'll see it as
localthe same repository, at the same commitinlineinclude: ci/build.yml, or a wildcard such as ci/*.yml
projecta file in another project, at a refpinned or floating, depending on refproject: with ref: and file:
remoteany public HTTP(S) URLwhatever the URL serves todayremote: https://…
templateGitLab's own library of templateschanges when your GitLab is upgradedtemplate: Security/SAST.gitlab-ci.yml
componenta component project, at a versionpinned or floating, depending on the versioncomponent: $CI_SERVER_FQDN/…@1.4.0
Where each kind of include fetches its file. Green always matches your own commit. Blue is pinned or floating, depending on the ref or version. Amber is whatever the URL serves today. Grey changes when GitLab is upgraded. Everything is merged into one configuration. local your repository, same commit project another project, at a ref remote whatever a URL serves template shipped with GitLab itself component a component, at a version one merged configuration every file joined into one; the including file wins a clash
Figure Where each kind of include fetches its file. Green always matches your own commit. Blue is pinned or floating, depending on the ref or version. Amber is whatever the URL serves today. Grey changes when GitLab is upgraded. Everything is merged into one configuration.#

MegaCorp uses all five. payments-api includes a project file and a component:

megacorp/payments/payments-api.gitlab-ci.ymlinclude
include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file: pipelines/java-service.yml
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api

The security template includes GitLab's own templates:

megacorp/devops/ci-templatestemplates/security.ymlsecurity-includes
include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml
  - template: Security/Container-Scanning.gitlab-ci.yml

The monorepo includes a file from its own repository, and billing-batch still pulls one in by URL:

megacorp/data/mono.gitlab-ci.ymllocal-include
include:
  - local: ci/common.yml
megacorp/legacy/billing-batch.gitlab-ci.ymlremote-include
include:
  - remote: https://gitlab.example.com/megacorp/devops/ci-templates/-/raw/main/legacy/notify.yml

How included files merge#

An included file can include more files, so payments-api's single include becomes ten files. GitLab then merges everything into one configuration, using the same rule as extends (chapter 14):

  • maps merge at any depth
  • lists are replaced whole
  • where two files set the same key, the including file beats the included one

The same file may be included twice, and the last inclusion wins.

That is how MegaCorp adjusts GitLab's container-scanning job without copying it. The security template includes GitLab's template, then defines a job with the same name:

You wrotemegacorp/devops/ci-templates · templates/security.yml
variables:
  CS_IMAGE: $IMAGE_REF

container_scanning:
  stage: scan
  needs: [image-build]
  rules:
    - if: $CONTAINER_SCANNING_DISABLED == "true"
      when: never
    - !reference [.rules, never-on-schedule]
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
GitLab buildsmerged configurationcaptured from GitLab
container_scanning:
  image: "$CS_ANALYZER_IMAGE$CS_IMAGE_SUFFIX"
  stage: scan
  variables:
    GIT_STRATEGY: none
  allow_failure: true
  artifacts:
    access: developer
    reports:
      container_scanning:
      - gl-container-scanning-report.json
      cyclonedx:
      - "**/gl-sbom-*.cdx.json"
    paths:
    - gl-container-scanning-report.json
    - gl-dependency-scanning-report.json
    - "**/gl-sbom-*.cdx.json"
  dependencies: []
  script:
  - gtcs scan
  rules:
  - if: $CONTAINER_SCANNING_DISABLED == "true"
    when: never
  - - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
  - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  - if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
  needs:
  - image-build

Read the merged job from the bottom up. stage, rules and needs are MegaCorp's. Everything else is GitLab's: the image, the script, the artifacts, and allow_failure: true. MegaCorp's rules replaced GitLab's list outright, because lists never merge. From here on, GitLab's own rules for the job, whatever they were in this version, no longer apply.

Reading a ref#

Every include except local reads its file at some ref. That one word decides whether a change somewhere else can reach you:

Will a change upstream reach you without a commit of yours? Read the ref on the include.yesnoyesnoyesnoIs the ref a commit SHA?Never: a commit can't changeIs it a tag, such as v4.2.0?Only if someone changes theversion, or moves the tagIs it a branch, such as main?Yes, on the very next pipelineThere is no ref at all: yes, it followsthe other project's default branch
Figure Will a change upstream reach you without a commit of yours? Read the ref on the include.#
As text
  1. Is the ref a commit SHA? Yes: Never: a commit can't change. No: the next step.
  2. Is it a tag, such as v4.2.0? Yes: Only if someone changes the version, or moves the tag. No: the next step.
  3. Is it a branch, such as main? Yes: Yes, on the very next pipeline. No: the next step.
  4. There is no ref at all: yes, it follows the other project's default branch
The include saysYou getUpstream changes reach you
ref: v4.2.0, a tagthat tag's contentonly when someone edits the version, or moves the tag
ref: with a commit SHAthat exact commitnever
ref: main, a branchwhatever the branch holds when the pipeline is createdat once, on the next pipeline
no ref at allthe head of the other project's default branchat once, on the next pipeline

A missing ref is the easiest time bomb to plant, because it looks tidy and floats silently. MegaCorp pins everything to v4.2.0, both where payments-api includes the golden pipeline and inside the golden pipeline itself. A project that pinned only the first file would still get whatever the nested includes point at.

What an include can depend on#

An include can carry rules, with if, exists or changes, and GitLab skips it when they don't match. The variables it can test are limited, because GitLab evaluates include rules before any job exists:

Usable in include rulesNot usable
project, group and instance CI/CD variablesvariables defined in a job
predefined CI_PROJECT_* variablestop-level variables: in any file
variables of triggered, scheduled and manually run pipelines
CI_PIPELINE_SOURCE, CI_PIPELINE_TRIGGERED, CI_COMMIT_REF_NAME

One more trap: in a file included from another project, rules: exists checks for files in that project, not yours. Use rules: exists: project to point it back. Included files can also take parameters, called inputs, which chapter 16 covers.

A nested include: local is read from the project that holds the file doing the including, not from yours. On the local GitLab 19.3, a file in ci-templates included local: templates/rules.yml while being included into payments-api. GitLab found ci-templates' copy, although payments-api has no such file.

Pulling in files#
include: local
Scope
One project
Binding
Inline
Control
Opt-in
Phase
Compile
Merged view
Shown
Looks like
include:
  - local: ci/common.yml
  - local: ci/jobs/*.yml
What GitLab does

GitLab copies the file from the same repository, at the same commit as the pipeline, into the configuration. A wildcard pulls in every match. A single * doesn't reach into subfolders; ** does.

How to trace it

Open the path in the same repository, at the commit the pipeline ran for. With a wildcard, list the matching files at that commit, because a file added later changes what the include means.

Gotchas
  • A new file that matches a wildcard joins the pipeline without anyone editing the include.
  • Inside a file that was itself included from another project, "local" means that project, not yours (see the note above).
Pulling in files#
include: project
Scope
Cross-project
Binding
Pinned or floating
Control
Opt-in
Phase
Compile
Merged view
Shown
Looks like
include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file:
      - templates/base.yml
      - templates/rules.yml
What GitLab does

GitLab fetches each listed file from the other project, at ref, and merges it into the configuration. Without a ref, it uses the head of that project's default branch.

How to trace it

Open the other project at the ref shown, not at its default branch, and read the file there. Check whether that file includes more files, and at which refs.

Gotchas
  • A branch ref, or no ref at all, floats. The pipeline can change without a commit in your project.
  • Pinning the top-level include pins nothing below it, unless the included file pins its own includes too.
Pulling in files#
include: remote
Scope
Cross-project
Binding
Pinned or floating
Control
Opt-in
Phase
Compile
Merged view
Shown
Looks like
include:
  - remote: https://gitlab.example.com/megacorp/devops/ci-templates/-/raw/main/legacy/notify.yml
What GitLab does

GitLab downloads the URL when it assembles the configuration, and merges the result. The URL must be publicly reachable. GitLab's guidance is to avoid rate-limited hosts, such as GitLab Pages.

How to trace it

Open the URL. What it serves now is what the next pipeline gets, which is not necessarily what an earlier pipeline got.

Gotchas
  • Nothing pins a remote include unless the URL itself names a fixed version. Here it names the main branch.
  • The file's owner can change every consumer's pipeline, without any of them knowing the file exists.
Pulling in files#
include: template
Scope
Organisation-wide
Binding
Always live
Control
Opt-in
Phase
Compile
Merged view
Shown
Looks like
include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml
What GitLab does

GitLab inserts one of the templates it ships with. They cover security scanners, Auto DevOps and common build jobs. The content belongs to the GitLab version your instance runs.

How to trace it

The Full configuration view shows the template's jobs copied in. GitLab's templates are published in GitLab's own source repository, under lib/gitlab/ci/templates/. Read the version that matches your instance.

Gotchas
  • Upgrading GitLab changes what the template contains, including job names and rules, without any commit in any of your projects.
  • Overriding a template job by name merges with it. Overriding its rules replaces them entirely.
When you write it
Do this
Pin every include to a tag or a commit SHA with ref:. Move to a new version on purpose, in a merge request of your own.
Why
Your pipeline then changes only when your team decides to change it, and a reviewer sees the change.
Watch out
An include of a branch, such as main, changes your pipeline whenever the central team pushes to that branch. The card the pipeline changed overnight shows how that looks.
What you can now do
  1. Read every kind of include and say which file it brings in, from where, and at which version.
  2. Pin an include to a tag or commit, and move it on purpose in a merge request.
  3. Find the included file behind any job, using the editor's file tree.
Spot the bugThe deploy jobs that never appear#

A team wanted deploy jobs only in projects that opt in, so their file says:

include:
  - local: ci/deploy.yml
    rules:
      - if: $DEPLOY_ENABLED == "true"

build:
  variables:
    DEPLOY_ENABLED: "true"
  script: make build

The deploy jobs from ci/deploy.yml never appear, on any branch.

QuestionWhy is the include always skipped, and how can the project opt in?
Show the answer

Include rules are evaluated while GitLab assembles the configuration, before any job exists, so they can't see variables defined in a job. $DEPLOY_ENABLED is empty there, and the rule never matches.

Set DEPLOY_ENABLED as a CI/CD variable in the project's settings, which include rules can see. Another option is a rule on a predefined variable such as $CI_COMMIT_REF_NAME. See include: local and what an include can depend on.

Components, inputs and toggles#

Part IV · Reuse · Chapter 16 · 15 min read

Shared configuration usually needs settings of its own, called parameters, and there are two ways to pass them:

  • Inputs are declared by the shared file, with a type. GitLab writes each value into the file, in place of $[[ inputs.name ]], when the pipeline is created. After that they never change.
  • Variables are the older way. The shared file reads $SOMETHING, and whoever uses it sets the value. Variable precedence decides who really wins.

A component is shared configuration packaged for this: a versioned file with declared inputs, included with component:. A toggle is a variable that a rule tests, turning a job on or off.

Wednesday, 15:40. A toggle that did nothing.

A colleague sets SONAR_ENABLED: "false" to skip the code-quality scan on a hotfix. The scan still runs. Another colleague sets SAST_DISABLED: 1 and the scan vanishes from every pipeline, including main. Two toggles, two opposite surprises. What decides whether a toggle works? (the card)

Components#

A component lives in a component project, as templates/<name>.yml or templates/<name>/template.yml. The include names the project, the component and a version. Here is MegaCorp's Sonar component, which runs a code-quality scan:

megacorp/devops/componentstemplates/sonar-scan.ymlspec
spec:
  inputs:
    stage:
      default: test
    project_key:
      description: SonarQube project key
    quality_gate:
      type: boolean
      default: true
---
megacorp/devops/componentstemplates/sonar-scan.ymljob
sonar-scan:
  stage: $[[ inputs.stage ]]
  image: registry.example.com/megacorp/devops/ci-tools/sonar-scanner:6
  variables:
    SONAR_PROJECT_KEY: $[[ inputs.project_key ]]
    SONAR_QUALITYGATE_WAIT: "$[[ inputs.quality_gate ]]"
  script:
    - sonar-scanner -Dsonar.projectKey="$SONAR_PROJECT_KEY" -Dsonar.host.url="$SONAR_HOST_URL"
  rules:
    - if: $SKIP_SONAR == "true"
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  allow_failure: true

payments-api includes it, with two inputs:

You wrotemegacorp/payments/payments-api · .gitlab-ci.yml
include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file: pipelines/java-service.yml
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api
GitLab buildsmerged configurationcaptured from GitLab
sonar-scan:
  stage: test
  image: registry.example.com/megacorp/devops/ci-tools/sonar-scanner:6
  variables:
    SONAR_PROJECT_KEY: payments-api
    SONAR_QUALITYGATE_WAIT: true
  script:
  - sonar-scanner -Dsonar.projectKey="$SONAR_PROJECT_KEY" -Dsonar.host.url="$SONAR_HOST_URL"
  rules:
  - if: $SKIP_SONAR == "true"
    when: never
  - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  - if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
  allow_failure: true

The inputs are gone from the result. $[[ inputs.stage ]] became test, and $[[ inputs.project_key ]] became payments-api. quality_gate wasn't passed, so it took its default, true.

The version after @ decides what you get:

VersionExampleResolves to
a commit SHA@e3262fdd…exactly that commit
a tag@2.1.0that tag; a tag beats a branch with the same name
a branch@mainthe branch's head when the pipeline is created
the latest release@~latestthe newest version published to the CI/CD Catalog
a partial version@2 or @2.1the newest matching version published to the CI/CD Catalog

Components work from any project, but the last two forms work only for components published to the CI/CD Catalog, GitLab's searchable list of shared components. Publishing needs a release, created with the release: keyword, on a tag that is a version number such as 2.1.0.

Inputs#

Inputs are declared in a header above a --- line. Each has a type: string (the default), number, boolean or array. It can also have a default, a description, a fixed list of options, or a regex the value must match. This is what happens to them:

How an input reaches a job. GitLab checks every value against the header before it creates the pipeline.noyesThe include passes the inputs it wants,such as project_key: payments-apiGitLab reads the shared file's spec:headerDoes every required input have a value?No pipeline. GitLab names themissing inputGitLab writes each value in place of$[[ inputs.name ]]The result is ordinary YAML, mergedlike any other include
Figure How an input reaches a job. GitLab checks every value against the header before it creates the pipeline.#
As text
  1. The include passes the inputs it wants, such as project_key: payments-api
  2. GitLab reads the shared file's spec: header
  3. Does every required input have a value? No: No pipeline. GitLab names the missing input. Yes: the next step.
  4. GitLab writes each value in place of $[[ inputs.name ]]
  5. The result is ordinary YAML, merged like any other include

A value can be transformed on the way in, by adding one or more of four functions after the name: expand_vars, truncate(offset,length), posix_escape and split(separator). At most three can be used at once.

Components aren't the only files that can declare inputs. Any included file can, and the include passes values with inputs:.

InputsVariables
Declared bythe shared file, in its spec: headernobody: any file or setting can define them
Resolvedonce, when the pipeline is createdwhen rules are evaluated, and again when the job runs
Can change during the pipelineno: fixed for the whole runyes: dotenv reports and scripts can set new values
Typed and checkedyes: type, options, regexno
Who wins a clashthe value passed in, else the defaultthe precedence ladder in chapter 26

Two things to know when you trace an input:

  • The merged view shows the result, not the input. To see where payments-api came from, open the component's own file at the version the include names, and read the include's inputs:.
  • A missing required input stops the whole pipeline, before any job runs. Leave out project_key, and the local GitLab 19.3 says:
`gitlab.example.com/megacorp/devops/components/sonar-scan@2.1.0`: `project_key` input: required value has not been provided

Toggles#

A toggle is a variable that a rule checks. MegaCorp's Sonar component has one (SKIP_SONAR), and so does its container-scanning override:

megacorp/devops/ci-templatestemplates/security.ymlcontainer-scan
variables:
  CS_IMAGE: $IMAGE_REF

container_scanning:
  stage: scan
  needs: [image-build]
  rules:
    - if: $CONTAINER_SCANNING_DISABLED == "true"
      when: never
    - !reference [.rules, never-on-schedule]
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Setting CONTAINER_SCANNING_DISABLED to "true" in a project's variables makes the first rule match, with when: never, so the job never exists. GitLab's own scanner templates use the same pattern, with variables named after the scanner.

Gotcha

Rules compare text. A toggle tested with == "true" ignores 1, yes and TRUE. And a toggle only removes a job that the file itself defines. A scan that a security policy adds doesn't read your toggles (chapter 36).

GitLab's own templates at GitLab 19.3 name their toggles after the scanner, and accept either "true" or "1". payments-api's merged configuration has three: SAST_DISABLED, SECRET_DETECTION_DISABLED and DEPENDENCY_SCANNING_DISABLED. MegaCorp's container-scanning override replaced GitLab's rules with its own, so its CONTAINER_SCANNING_DISABLED accepts only "true".

Pipeline inputs Since 17.11#

A project's own .gitlab-ci.yml can declare inputs for the whole pipeline. They are filled in when someone runs the pipeline by hand, or passed by the API, a schedule, a Git push option, or a trigger from another pipeline. MegaCorp's deployer asks which environment to deploy to:

megacorp/platform/deployer.gitlab-ci.ymlinputs
spec:
  inputs:
    environment:
      type: string
      default: staging
      options: [staging, production]
---

A pipeline can take at most 20 inputs. Give each one a default, because pipelines that start automatically have nobody to fill in the form.

Pulling in filesSince 17.0#
include: component
Scope
Cross-project
Binding
Pinned or floating
Control
Opt-in
Phase
Compile
Merged view
Shown
Looks like
include:
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      project_key: payments-api
What GitLab does

GitLab fetches templates/sonar-scan.yml, or templates/sonar-scan/template.yml, from the component project at the version given. It writes in the inputs and merges the result like any other include.

How to trace it

Everything before the @ is the project path plus the component name. Open that project at the version given, and read the file under templates/. For ~latest or a partial version, the CI/CD Catalog shows which release it resolved to.

Gotchas
  • A component's job merges with any job of the same name in your pipeline.
  • @~latest fails with "content not found" if the component has never been published to the catalog.
  • Global keywords in a component, such as default:, affect every job in your pipeline.
ParametersSince 17.0#
spec: inputs
Scope
Cross-project
Binding
Inline
Control
Opt-in
Phase
Compile
Merged view
Shown
Looks like
spec:
  inputs:
    stage:
      default: test
    quality_gate:
      type: boolean
      default: true
---
sonar-scan:
  stage: $[[ inputs.stage ]]
What GitLab does

The header declares what the file accepts. When the pipeline is created, GitLab checks the values passed in against the declarations, and writes them in place of every $[[ inputs.… ]]. The result is ordinary configuration, fixed for the rest of the pipeline.

How to trace it

Read the header at the top of the included file for the defaults. Then read the inputs: under the include that pulled it in, for the values actually passed.

Gotchas
  • A file with inputs needs the --- line between its header and its jobs.
  • Inputs can't be used inside a !reference path.
  • An input that holds $CI_COMMIT_SHA inserts that text. The variable is expanded later, when the job runs.
Parameters#
Variables as parameters
Scope
One project
Binding
Inline
Control
Opt-in
Phase
Run
Merged view
Keyword only
Looks like
# in the template
image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}

# in the project
variables:
  JAVA_VERSION: "21"
What GitLab does

The template reads an ordinary CI/CD variable, and the consumer sets it. Nothing declares the variable or checks its value. GitLab resolves it when it evaluates rules and when the job runs, from whichever source ranks highest in variable precedence.

How to trace it

Find where the template reads the variable. Then find every place that sets it: the job, its extends parents, the top-level blocks, and the CI/CD settings of the project, group and instance.

Gotchas
  • A template that sets the same variable at job level overrides the project's top-level value.
  • A settings variable with the same name overrides both, and appears in no file.
Parameters#
Toggle variables
Scope
One project
Binding
Inline
Control
Opt-in
Phase
Create
Merged view
Keyword only
Looks like
rules:
  - if: $SKIP_SONAR == "true"
    when: never
  - if: $CI_PIPELINE_SOURCE == "merge_request_event"
What GitLab does

A rule tests the variable when the pipeline is created, and the first matching rule decides whether the job exists. Setting the variable in the project, a schedule or a manual run switches the job off or on without editing any file.

How to trace it

Search the templates' rules for the variable's name. Then check every place the variable can be set, including CI/CD settings and schedules, which no file records.

Gotchas
  • Rules compare text, so "true" and 1 are different values.
  • A toggle can't remove a job that a security policy adds.
ParametersSince 17.11#
Pipeline inputs
Scope
One project
Binding
Inline
Control
Opt-in
Phase
Create
Merged view
Keyword only
Looks like
spec:
  inputs:
    environment:
      default: staging
      options: [staging, production]
---
variables:
  ENVIRONMENT: $[[ inputs.environment ]]
What GitLab does

Inputs declared in the project's own configuration become the pipeline's parameters. They are offered on the New pipeline form, and accepted from the API, schedules, push options and trigger. As with any input, their values are fixed when the pipeline is created.

How to trace it

Open the top of the project's .gitlab-ci.yml for the declarations. The values a particular pipeline used come from whoever started it: a person, a schedule, an API call or an upstream pipeline.

Gotchas
  • An input without a default breaks every pipeline that starts automatically.
  • Pipeline variables sent by a trigger outrank YAML variables. In the deployer, an upstream job's ENVIRONMENT variable beats the value built from the input.
When you write it
Do this
Include a component at an exact version, such as @2.1.0, and pass every input that its spec requires.
Why
An exact version keeps your pipeline the same until you change it. A required input that is missing stops GitLab creating the pipeline at all.
Watch out
Never use ~latest or a branch name as the version. And remember that inputs are filled in when GitLab reads the file, so an input cannot hold a value that only exists once a job is running.
What you can now do
  1. Include a component at a version, pass its inputs, and read what it expands to.
  2. Tell an input from a variable, and know which one can hold a run-time value.
  3. Predict whether a toggle will remove a job, and explain why a policy's job ignores it.
Spot the bugThe upgrade to ~latest#

To stop bumping versions by hand, the payments team changed their include to:

include:
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@~latest
    inputs:
      stage: test
      project_key: payments-api

Now no pipeline can be created. The error says the component's content was not found, yet @2.1.0 still works.

QuestionWhat does ~latest need that 2.1.0 doesn't?
Show the answer

~latest and partial versions such as @2 resolve only against versions published to the CI/CD Catalog. MegaCorp's components project has a 2.1.0 tag, but has never been published, so there is no "latest" to find. A tag works without publishing.

Either keep an explicit tag, or ask the component owners to publish releases to the catalog. See include: component.

Pipelines as building blocks#

Part IV · Reuse · Chapter 17 · 17 min read

A trigger job is a job whose only work is to start another pipeline. The pipeline that starts it is called upstream, and the one it starts downstream. There are four ways it happens:

  • A child pipeline runs in the same project, on the same commit.
  • A multi-project pipeline runs in another project, with your permissions.
  • A dynamic child runs YAML that a job wrote moments earlier.
  • A script can do the same thing through the trigger API, with a token.

Two things decide what you see. The first is the trigger job's strategy: without one, it goes green as soon as the other pipeline exists, whatever happens next. The second is what crosses the boundary. Variables and inputs do; default: does not.

Thursday, 09:10. Green, and the child failed.

The monorepo's trigger job shows green. The child pipeline it started, the docs build, failed ten minutes later, and nobody noticed until the site went stale. In the merge request, the same trigger job fails outright with downstream pipeline can not be created. One trigger job, two different lies. What does a trigger job actually report? (the card)

Four kinds#

Four ways one pipeline starts another. Child and dynamic child pipelines run in the same project and on the same commit. A multi-project trigger, or a script calling the trigger API, starts a pipeline in another project, which runs that project's own configuration. In one project, on one commit megacorp/data/mono parent pipeline generate-pipeline service-pipelines docs writes YAML dynamic child runs that YAML child pipeline ci/docs.yml Into another project payments-api deploy-staging platform/deployer its own pipeline on its main branch multi- project billing-batch notify-reports project 4242 its own pipeline source: trigger trigger API
Figure Four ways one pipeline starts another. Child and dynamic child pipelines run in the same project and on the same commit. A multi-project trigger, or a script calling the trigger API, starts a pipeline in another project, which runs that project's own configuration.#
KindWritten asRuns inIts configuration comes from
Parent–childtrigger: include: ci/docs.ymlthe same project, ref and commita file in the same repository
Dynamic childtrigger: include: - artifact: … job: …the same project, ref and commitYAML a job wrote during this pipeline
Multi-projecttrigger: project: …another project, on the branch namedthat project's own configuration
Trigger APIcurl …/trigger/pipeline in a scriptwhichever project the call namesthat project's own configuration

Parent and child#

MegaCorp's monorepo builds its documentation in a child pipeline, and only when the docs change:

megacorp/data/mono.gitlab-ci.ymlstatic-child
docs:
  stage: build
  trigger:
    include: ci/docs.yml
  rules:
    - changes: [docs/**/*]

The child runs under the same project, ref and commit as its parent. It inherits the parent's top-level variables, unless the trigger job says inherit: variables: false. It does not inherit the parent's default: block, because defaults never cross into a downstream pipeline. Children can have children, but only two levels deep.

A child pipeline's jobs follow their own rules. In a merge request pipeline, the child still sees CI_PIPELINE_SOURCE as parent_pipeline, and jobs with no rules at all don't join it. On the local GitLab 19.3, mono's docs trigger failed in every merge request pipeline for exactly that reason (chapter 22).

Dynamic child pipelines#

The monorepo's services are built by a pipeline that doesn't exist until the pipeline runs. A job writes it, and saves the file as an artifact: a file a job keeps for later jobs to use.

megacorp/data/mono.gitlab-ci.ymlgenerate
generate-pipeline:
  stage: generate
  script:
    - ci/generate.sh > generated-pipeline.yml
  artifacts:
    paths: [generated-pipeline.yml]
megacorp/data/monoci/generate.sh
#!/usr/bin/env bash
# ci/generate.sh in megacorp/data/mono
# Writes one child-pipeline job per service that changed. Its output is the YAML
# that the service-pipelines trigger job runs. Used by the chapter "Pipelines as building
# blocks".
set -euo pipefail

base="${CI_MERGE_REQUEST_DIFF_BASE_SHA:-HEAD~1}"
changed=$(git diff --name-only "$base" HEAD -- services/ | cut -d/ -f2 | sort -u)

echo "stages: [build]"
for svc in $changed; do
  cat <<EOF
build-${svc}:
  stage: build
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  script:
    - make -C services/${svc} build
EOF
done

# a child pipeline needs at least one job, even when nothing changed
if [ -z "$changed" ]; then
  printf 'no-changes:\n  stage: build\n  script: [echo "no service changed"]\n'
fi

Then a trigger job runs whatever the generator wrote:

megacorp/data/mono.gitlab-ci.ymldynamic-child
service-pipelines:
  stage: build
  trigger:
    include:
      - artifact: generated-pipeline.yml
        job: generate-pipeline
    strategy: depend

There is no file to read in advance. The configuration that actually ran is the generated-pipeline.yml artifact of that one job, in that one pipeline. The generator script is the only thing you can read beforehand.

Multi-project pipelines#

Every MegaCorp service deploys by triggering the platform team's project:

megacorp/devops/ci-templatestemplates/deploy.ymldeploy-trigger
.deploy:
  stage: deploy
  trigger:
    project: megacorp/platform/deployer
    branch: main
    strategy: depend
  variables:
    APP: $CI_PROJECT_NAME
    IMAGE_REF: $IMAGE_REF
payments-api on main: deploy-staging starts a pipeline in the deployer project (test and scan jobs left out)buildmaven-buildpackageimage-builddeploydeploy-stagingdownstreammegacorp/platform/deployermulti-project pipeline
Pipeline payments-api on main: deploy-staging starts a pipeline in the deployer project (test and scan jobs left out)#
As text
  1. build: maven-build
  2. package: image-build
  3. deploy: deploy-staging

Needs: image-build waits only for maven-build; deploy-staging waits only for image-build.

Downstream: deploy-staging triggers megacorp/platform/deployer (multi-project pipeline).

The downstream pipeline runs the deployer's own configuration, at the head of its main branch. The person who triggered the upstream pipeline must be allowed to start pipelines in the deployer project, or the trigger fails.

By default, the downstream pipeline receives the trigger job's variables, including the top-level variables it inherits. They arrive as trigger variables, and take precedence over the downstream project's own variables of the same name.

Some variables stay behind, unless the trigger job sets forward: pipeline_variables: true:

  • values typed into a manual run
  • variables set on a schedule
  • dotenv variables from the jobs the trigger job needs

Nothing forwarded is passed on again by the next trigger down, unless that trigger job also sets forward.

That list is about passing values on automatically. Naming a dotenv variable in the trigger job's own variables is different, and it works. On the local GitLab 19.3, a trigger job with IMAGE_REF: $IMAGE_REF passed the value from an earlier job's dotenv report to the downstream pipeline. It did so whether the trigger job listed that job in needs, or only came in a later stage. MegaCorp's .deploy relies on this.

What the trigger job's status means#

A green trigger job does not always mean the downstream pipeline passed. It depends on one key:

What a trigger job's colour tells you, depending on its strategynoyesyesnoDoes the trigger job set a strategy?It goes green once the downstreampipeline exists, whatever happensnextIs the strategy mirror?It copies the downstreampipeline's status exactlyThe strategy is depend: green when thedownstream succeeds, and "running"while it waits for a manual job
Figure What a trigger job's colour tells you, depending on its strategy#
As text
  1. Does the trigger job set a strategy? No: It goes green once the downstream pipeline exists, whatever happens next. Yes: the next step.
  2. Is the strategy mirror? Yes: It copies the downstream pipeline's status exactly. No: the next step.
  3. The strategy is depend: green when the downstream succeeds, and "running" while it waits for a manual job
The trigger job hasIt turns green whenSo a green trigger job means
no strategythe downstream pipeline has been createdonly that the other pipeline started
strategy: mirrorthe downstream pipeline succeeds; it copies its status throughoutthe downstream pipeline succeeded
strategy: dependthe downstream pipeline finishesan older setting; GitLab's documentation now recommends mirror, because this status doesn't always match

MegaCorp still uses depend everywhere, as many real template libraries do. The first row is the trap: a pipeline can be entirely green while the deployment it started failed.

The trigger API#

Before trigger jobs existed, pipelines started each other with a token and a script, and some still do:

megacorp/legacy/billing-batch.gitlab-ci.ymltrigger-api
notify-reports:
  <<: *defaults
  stage: deploy
  only: [tags]
  script:
    - >
      curl --fail --request POST
      --form "token=$REPORTS_TRIGGER_TOKEN"
      --form "ref=main"
      --form "variables[BILLING_VERSION]=$CI_COMMIT_TAG"
      "https://gitlab.example.com/api/v4/projects/4242/trigger/pipeline"

The started pipeline has CI_PIPELINE_SOURCE set to trigger. Variables sent with the call have the highest precedence of all. The token acts with the access of the user it belongs to.

Whole pipelines#
Parent–child pipelines
Scope
One project
Binding
Inline
Control
Opt-in
Phase
Run
Merged view
Keyword only
Looks like
docs:
  trigger:
    include: ci/docs.yml
What GitLab does

GitLab starts a child pipeline from the file, in the same project and on the same ref and commit. The child inherits the parent's top-level variables but not its default:. It appears as a card to the right of the parent's graph. Children nest at most two levels deep.

How to trace it

The trigger job names the child's file. Open the card to the right of the parent's graph to see the child's jobs, then read the file at the same commit.

Gotchas
  • Without strategy: mirror, the trigger job passes as soon as the child is created, even if every job in it then fails.
  • The parent's default:, such as its image or before_script, doesn't reach the child's jobs.
  • In a merge request pipeline, a child whose jobs have no rules gets no jobs, and the trigger job fails.
Whole pipelines#
Dynamic child pipelines
Scope
One project
Binding
Inline
Control
Opt-in
Phase
Run
Merged view
Not shown
Looks like
service-pipelines:
  trigger:
    include:
      - artifact: generated-pipeline.yml
        job: generate-pipeline
What GitLab does

One job writes pipeline configuration as an artifact, and a trigger job starts a child pipeline from it. The configuration exists only for that pipeline. An include inside the generated configuration can't use CI/CD variables.

How to trace it

Download the artifact from the generator job of the pipeline you are investigating: that YAML is exactly what ran. Then read the generator script, to understand why it wrote what it wrote.

Gotchas
  • No merged view ever shows the child's configuration, because it didn't exist before the pipeline ran.
  • The generator must always write at least one job. A generator that sometimes writes nothing breaks the pipeline only sometimes.
Whole pipelines#
Multi-project pipelines
Scope
Cross-project
Binding
Always live
Control
Opt-in
Phase
Run
Merged view
Not shown
Looks like
deploy-staging:
  trigger:
    project: megacorp/platform/deployer
    branch: main
    strategy: depend
What GitLab does

GitLab starts a pipeline in the other project, on the branch named, using that project's own configuration. The user who triggered the upstream pipeline must be allowed to run pipelines in the downstream project.

How to trace it

Open the card to the right of the graph to reach the downstream pipeline. Its jobs come from the downstream project's configuration at the head of the named branch, not from anything in your project.

Gotchas
  • The downstream configuration floats with that branch. The platform team can change your deployment without a commit in your project.
  • Someone without access to the downstream project can run your pipeline, but not its deployment.
Whole pipelines#
Pipeline trigger API
Scope
Cross-project
Binding
Always live
Control
Opt-in
Phase
Run
Merged view
Not shown
Looks like
curl --fail --request POST \
  --form "token=$REPORTS_TRIGGER_TOKEN" \
  --form "ref=main" \
  "https://gitlab.example.com/api/v4/projects/4242/trigger/pipeline"
What GitLab does

The API creates a pipeline in the project named by its number, on the ref given. Variables sent with the call outrank every other variable. The pipeline's CI_PIPELINE_SOURCE is trigger. The token acts with its owner's project access.

How to trace it

Search scripts for /trigger/pipeline. Look up the project number in the URL to find which project is being started, and find out who owns the token variable.

Gotchas
  • A number in a URL is the only sign of the relationship. No file names the target project.
  • Because the token acts with its owner's access, the call can break when that person's access changes, far from any pipeline file.
When you write it
Do this
Add strategy: mirror to a trigger job when your pipeline's result should include the downstream pipeline's result.
Why
With it, the trigger job shows the downstream pipeline's status: it waits for that pipeline and fails if it fails. Older files use strategy: depend, which GitLab no longer recommends because its status doesn't always match.
Watch out
Without it, the trigger job passes as soon as the downstream pipeline has been created, whatever happens to that pipeline afterwards. Your pipeline can show green while the real work failed.
What you can now do
  1. Read a parent, child or multi-project pipeline and say which file each part came from.
  2. Make a trigger job wait for its child and fail with it.
  3. Explain why a child of a merge request pipeline can end up with no jobs, and fix it.
Spot the bugGreen pipeline, nothing deployed#

A team copied MegaCorp's deploy template into their own file, and trimmed it:

deploy-staging:
  stage: deploy
  trigger:
    project: megacorp/platform/deployer
    branch: main
  variables:
    APP: $CI_PROJECT_NAME
    IMAGE_REF: $IMAGE_REF

Their pipelines are green, but staging hasn't changed in a week. The deployer project shows a failed pipeline for every one of their merges.

QuestionHow can a pipeline be green when its deployment failed?
Show the answer

With no strategy, a trigger job succeeds as soon as the downstream pipeline is created. It doesn't wait for that pipeline, or reflect how it ends. The deployer's pipelines failed, but the trigger job had already passed.

Add strategy: mirror, so that the trigger job takes the downstream pipeline's status. See multi-project pipelines and what the trigger job's status means.

Organisation-wide reuse: settings and policies#

Part IV · Reuse · Chapter 18 · 24 min read

Nine mechanisms reach a project without any of its files asking for them. Four are available on every tier (GitLab's paid plans are Free, Premium and Ultimate):

  • a setting that points the project at a pipeline file somewhere else
  • Auto DevOps, GitLab's own ready-made pipeline
  • CI/CD variables set in the settings of the instance, group and project
  • templates that are copied in when a file or project is created

Four more come with GitLab Ultimate. They are security policies, rules a security team sets once for many projects: pipeline execution policies, scan execution policies and merge request approval policies. The fourth is compliance pipelines, which those policies replace. A ninth, the instance template repository, is a copy-once template list on self-managed Premium.

None of them shows up in the project's files. You find them in settings pages, and in the pipeline itself.

Thursday, 16:00. The value that wouldn't change.

You change MAVEN_CLI_OPTS in payments-api's file, push, and the job still uses the old options. Full configuration shows your new value. The log shows the old one. Nothing in the project explains it. Where else can a variable come from? (the card)

Where each one lives#

Two words first. The instance is the whole GitLab installation. Self-managed means your organisation runs that installation itself, rather than using GitLab.com.

MechanismWho sets itWhere you lookTier
Custom CI/CD configuration filea project maintainerthe project's CI/CD settings, under General pipelinesall
Auto DevOpsan administrator or group owner, or the projectthe Auto DevOps section of the CI/CD settingsall
Instance, group and project variablesadministrators, group owners, project maintainersthe CI/CD variables settings at each levelall
Project and file templateswhoever created the project or filenowhere afterwards: the copy is the only traceall; templates kept in a group need Premium
Instance template repositoryan administratorAdmin › Settings › TemplatesPremium, self-managed and Dedicated
Pipeline execution policiesthe security teamthe group's security policiesUltimate
Scan execution policiesthe security teamthe group's security policiesUltimate
Merge request approval policiesthe security teamthe group's security policies, and the merge request itselfUltimate
Compliance pipelinescompliance ownersthe project's compliance framework labelUltimate, deprecated

In the UI, settings variables are under Settings › CI/CD › Variables: in the project, in each group, and in the Admin area for the instance. Security policies are under Secure › Policies, in a project or a group.

The most common puzzle these mechanisms cause is a project that has pipelines but no pipeline file. Check these places, in this order:

A project has pipelines but no .gitlab-ci.yml. Each question points to a mechanism in this chapter.yesnoyesnoyesnoDo the project's CI/CD settings name aconfiguration file?A custom configuration file,perhaps in another projectIs Auto DevOps on for the project, itsgroup or the instance?GitLab's own Auto DevOps pipelineDoes a scan execution policy apply tothe project?The policy creates a scan pipelineitselfAsk the group's owners about complianceframeworks and other policies
Figure A project has pipelines but no .gitlab-ci.yml. Each question points to a mechanism in this chapter.#
As text
  1. Do the project's CI/CD settings name a configuration file? Yes: A custom configuration file, perhaps in another project. No: the next step.
  2. Is Auto DevOps on for the project, its group or the instance? Yes: GitLab's own Auto DevOps pipeline. No: the next step.
  3. Does a scan execution policy apply to the project? Yes: The policy creates a scan pipeline itself. No: the next step.
  4. Ask the group's owners about compliance frameworks and other policies

A pipeline file that isn't in the repository#

web-portal's repository has no .gitlab-ci.yml, yet it has pipelines. Its settings point at a file in the templates project:

megacorp/web/web-portalSETTINGS.ymlconfig-path
ci_cd_configuration_file: pipelines/web.yml@megacorp/devops/ci-templates

The setting accepts three forms:

  • a path in the same repository
  • a file in another project, written as path@namespace/project, optionally with :ref at the end
  • a URL ending in .yml

The project's own pipeline editor can't edit a file that lives somewhere else. That is often the first sign this setting is in play. On the local GitLab 19.3, web-portal's pipelines ran the three jobs defined in ci-templates' pipelines/web.yml.

Organisation-wide#
Custom CI/CD configuration file
Scope
Cross-project
Binding
Pinned or floating
Control
Enforced
Phase
Compile
Merged view
Not shown
Looks like
pipelines/web.yml@megacorp/devops/ci-templates
pipelines/web.yml@megacorp/devops/ci-templates:v4.2.0
What GitLab does

GitLab reads the pipeline configuration from the path the setting names, instead of .gitlab-ci.yml at the root of the repository. The file can be in the same project, in another project at an optional ref, or at a URL.

How to trace it

Open the project's CI/CD settings and expand General pipelines. If the file is elsewhere, read it in that project at the ref given. The project's own pipeline editor won't show it.

Gotchas
  • Without a ref, GitLab reads the file from the other project's default branch. Every change there reaches every consuming project at once.
  • A repository with no pipeline file isn't necessarily a project without a pipeline.

Variables from settings#

CI/CD variables can be set on the instance, on each group, and on the project. Every job in every project below them receives them, and they outrank anything in YAML:

settingsinstance.ymlinstance-variables
# Every project on the instance receives these, unless a group or project
# variable with the same name takes precedence.
instance_variables:
  - key: MEGACORP_REGISTRY
    value: registry.example.com
  - key: HTTPS_PROXY
    value: http://proxy.example.com:3128
settingsgroup-megacorp.ymlgroup-variables
# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
  - key: MAVEN_CLI_OPTS
    value: "--batch-mode --errors --show-version -s .m2/settings.xml"
  - key: AWS_ACCOUNT_ID
    value: "123456789012"
  - key: MC_AZURE_TENANT_ID
    value: "00000000-0000-0000-0000-000000000000"
  - key: MC_ACR_REGISTRY
    value: megacorpprod.azurecr.io
  - key: MC_ACR_PUSH_CLIENT_ID
    value: "00000000-0000-0000-0000-000000000000"
    # the gitlab-acr-push application in Entra ID, which mc_acr_push signs in as
  - key: SONAR_HOST_URL
    value: https://sonar.example.com
  - key: SONAR_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true

Among themselves, the project beats its groups, the closest group beats the ones above it, and groups beat the instance. Only policy variables, variables typed into a manual job, and pipeline variables rank higher. Chapter 26 gives the whole ladder.

Organisation-wide#
Instance and group CI/CD variables
Scope
Organisation-wide
Binding
Always live
Control
Inherited
Phase
Create
Merged view
Not shown
Looks like
Group megacorp › CI/CD variables
  MAVEN_CLI_OPTS = --batch-mode --errors --show-version -s .m2/settings.xml
What GitLab does

Every pipeline in every project under the group, or on the instance, receives the variable. It outranks variables written anywhere in the configuration, including job variables. It can also be protected, masked or hidden.

How to trace it

A job's log never lists its variables. Check the CI/CD variables settings of the project, then each parent group, then the instance. Where you lack access to a level, ask someone who has it.

Gotchas
  • Setting the same variable in YAML does nothing: the settings value wins silently.
  • A protected variable exists only in pipelines on protected branches and tags.

Auto DevOps#

Auto DevOps is GitLab's own complete pipeline. It runs for a project only if the project has no .gitlab-ci.yml, and only if it finds a Dockerfile, or a buildpack that matches the code. A project's own configuration always wins. On a self-managed instance, the setting for all projects starts switched on: the local GitLab 19.3 had it on before any setting was changed. MegaCorp left it on:

settingsinstance.ymlauto-devops
# A project with no CI/CD configuration file gets GitLab's Auto DevOps pipeline.
auto_devops:
  default_to_auto_devops: true
Organisation-wide#
Auto DevOps
Scope
Organisation-wide
Binding
Always live
Control
Inherited
Phase
Compile
Merged view
Not shown
Looks like
Settings › CI/CD › Auto DevOps
  [x] Default to Auto DevOps pipeline
What GitLab does

When a project has no CI/CD configuration and Auto DevOps applies to it, GitLab runs its built-in Auto DevOps pipeline, which builds, tests, scans and can deploy. The setting can come from the project, a group or the instance.

How to trace it

If a project without a pipeline file has pipelines, check the project's Auto DevOps setting, then its groups'. Then check the custom configuration file setting, which is the other explanation.

Gotchas
  • Adding any .gitlab-ci.yml, even a small one, switches Auto DevOps off for that project.
  • Its jobs change with GitLab upgrades, like any GitLab template.

Its jobs are easy to recognise. They have names such as build, test and code_quality, scanner jobs such as container_scanning and secret_detection, and deploy jobs such as review, staging and production.

Templates copied at creation#

Two mechanisms hand a project configuration once, when a file or a project is created, and never again:

  • Project templates. A new project is created as a copy of a template project, including its .gitlab-ci.yml. GitLab ships some on every tier; keeping your own in a group, as MegaCorp does, needs Premium Premium.
  • The instance template repository Premium Self-managed Dedicated. An administrator chooses a project whose gitlab-ci folder fills the template list in the Web Editor, GitLab's in-browser file editor. Picking one copies it into the new file. Despite the name, these templates can't be used with include: template.
settingsgroup-megacorp.ymlproject-templates
# New projects can be created from these. The copy happens once, at creation.
group_project_templates:
  source_group: megacorp/templates
  templates:
    - megacorp/templates/java-service-starter
    - megacorp/templates/batch-job-starter
settingsinstance.ymltemplate-repo
# Self-managed only: the .yml files in this project's gitlab-ci/ folder appear in the
# Web Editor's template list when anyone creates a CI/CD file. Choosing one copies it
# into the new file; include: template cannot use them.
instance_template_repository:
  project: megacorp/devops/instance-templates
  offers:
    - gitlab-ci/MegaCorp-Legacy-Java.yml
Organisation-wide#
Project and file templates
Scope
Organisation-wide
Binding
Copied once
Control
Opt-in
Phase
Compile
Merged view
Shown
Looks like
New project › Create from template › Group › megacorp/templates/java-service-starter
What GitLab does

GitLab copies the template project, including its pipeline file, into the new project when it is created. After that, the copy is an ordinary file in the new project, with no link back to the template.

How to trace it

Compare the project's first commit with the template project's history. A .gitlab-ci.yml that arrived fully formed in the first commit came from somewhere.

Gotchas
  • Fixes to the template never reach projects created from it. Copies drift apart silently.
Organisation-widePremiumSelf-managed#
Instance template repository
Scope
Organisation-wide
Binding
Copied once
Control
Opt-in
Phase
Compile
Merged view
Shown
Looks like
Admin › Settings › Templates
  Templates project: megacorp/devops/instance-templates
    gitlab-ci/MegaCorp-Legacy-Java.yml
What GitLab does

The .yml files in the chosen project's gitlab-ci folder appear in the template list whenever someone creates a CI/CD file in the Web Editor. Choosing one copies its content into the new file.

How to trace it

If a project's pipeline file looks like one of the organisation's templates, compare it with the file in the administrators' templates project. Only administrators can see which project that is.

Gotchas
  • It is a copy, not an include: the template can change without the project changing.
  • include: template reaches only GitLab's own templates, never these.

Security policies Ultimate#

On GitLab Ultimate, a security team keeps policies in a policy project linked to a group, and they apply to every project under it. MegaCorp's policy project defines one of each kind:

megacorp/security/policies.gitlab/security-policies/policy.ymlpipeline-execution
pipeline_execution_policy:
  - name: MegaCorp guardrails
    description: Add an SBOM job to every pipeline.
    enabled: true
    pipeline_config_strategy: inject_policy
    content:
      include:
        - project: megacorp/security/policies
          file: pipeline-policies/megacorp-guardrails.yml
megacorp/security/policiespipeline-policies/megacorp-guardrails.ymlguardrails
policy-sbom:
  stage: .pipeline-policy-post
  image: registry.example.com/megacorp/devops/ci-tools/syft:1
  script:
    - syft dir:. -o cyclonedx-json=sbom.cdx.json
  artifacts:
    reports:
      cyclonedx: sbom.cdx.json

A pipeline execution policy puts its own CI/CD configuration into every pipeline. With inject_policy, its jobs are added to the project's own jobs. With override_project_ci, the policy's configuration replaces the project's entirely. Two reserved stages frame the pipeline:

  • .pipeline-policy-pre runs before everything else. Every other job waits for it, and if it fails, the rest is skipped.
  • .pipeline-policy-post runs after everything else.
megacorp/security/policies.gitlab/security-policies/policy.ymlscan-execution
scan_execution_policy:
  - name: Secret detection everywhere
    description: Run secret detection in every pipeline on every branch.
    enabled: true
    rules:
      - type: pipeline
        branches: ["*"]
    actions:
      - scan: secret_detection
  - name: Nightly dependency scan
    description: Scan dependencies on the default branch every night.
    enabled: true
    rules:
      - type: schedule
        branches: [main]
        cadence: "0 2 * * *"
    actions:
      - scan: dependency_scanning

A scan execution policy adds GitLab's scanners to pipelines, or runs them on a schedule. Its jobs get a hyphen and a number added to their names, such as secret-detection-1, so that they can't clash with the project's own scanner jobs. Scheduled scans run as a bot user, not as a member of your team.

payments-api on main, on GitLab Ultimate: jobs from both policies appear beside the project's own (described, not captured; other jobs left out)buildmaven-buildtestmaven-test: [17]maven-test: [21]secret-detection-1policy.pipeline-policy-postpolicy-sbompolicy
Pipeline payments-api on main, on GitLab Ultimate: jobs from both policies appear beside the project's own (described, not captured; other jobs left out)#
As text
  1. build: maven-build
  2. test: maven-test: [17], maven-test: [21], secret-detection-1 (policy)
  3. .pipeline-policy-post: policy-sbom (policy)
megacorp/security/policies.gitlab/security-policies/policy.ymlapproval
approval_policy:
  - name: Block new critical vulnerabilities
    description: A merge into main that adds a critical finding needs security's approval.
    enabled: true
    rules:
      - type: scan_finding
        branches: [main]
        scanners: [sast, secret_detection, dependency_scanning, container_scanning]
        vulnerabilities_allowed: 0
        severity_levels: [critical]
        vulnerability_states: [new_needs_triage]
    actions:
      - type: require_approval
        approvals_required: 1
        group_approvers: [megacorp/security]

A merge request approval policy adds no jobs. It reads the scanners' results, and when its rule matches, the merge request needs the approvals the policy names. If a required scanner produced no report, the policy by default requires the approval anyway: it fails closed.

LegacyCompliance pipelines

Before pipeline execution policies, a compliance framework could name a pipeline file, written path@group/project, for every project carrying its label. A compliance framework is a label an administrator attaches to a project, to mark the rules it must follow. By default the framework's file ran instead of the project's own configuration, unless it included the project's .gitlab-ci.yml explicitly. Compliance pipelines were deprecated in 17.3 and are planned for removal in 20.0. Use pipeline execution policies instead.

settingsgroup-megacorp.ymlcompliance
# Legacy: a compliance framework with its own pipeline configuration, the
# predecessor of pipeline execution policies.
compliance_frameworks:
  - name: SOX
    pipeline_configuration: compliance/sox.yml@megacorp/security/policies
    applied_to:
      - megacorp/payments/payments-api
Organisation-wideUltimateSince 17.3#
Pipeline execution policies
Scope
Organisation-wide
Binding
Always live
Control
Enforced
Phase
Create
Merged view
Not shown
Looks like
pipeline_execution_policy:
  - name: MegaCorp guardrails
    pipeline_config_strategy: inject_policy
    content:
      include:
        - project: megacorp/security/policies
          file: pipeline-policies/megacorp-guardrails.yml
What GitLab does

When a pipeline is created, GitLab adds the policy's configuration to it, or, with override_project_ci, uses the policy's configuration instead of the project's. Policy jobs can run in the reserved stages .pipeline-policy-pre and .pipeline-policy-post. Policy variables outrank the project's and the groups'. A policy project holds at most five of these policies. Generally available in 17.3; inject_policy from 17.9.

How to trace it

A job in no project file, or in a .pipeline-policy-* stage, points here. Open the group's security policies, find the policy's content, and read that file in the policy project.

Gotchas
  • If a policy job has the same name as a project job, the policy job gets :policy-<project id>-<index> added to its name by default.
  • A failure in .pipeline-policy-pre skips every other job in the pipeline.
  • With override_project_ci, editing your .gitlab-ci.yml may change nothing at all.
Organisation-wideUltimate#
Scan execution policies
Scope
Organisation-wide
Binding
Always live
Control
Enforced
Phase
Create
Merged view
Not shown
Looks like
scan_execution_policy:
  - name: Secret detection everywhere
    rules:
      - type: pipeline
        branches: ["*"]
    actions:
      - scan: secret_detection
What GitLab does

GitLab adds the named scanners to matching pipelines, or runs them on a schedule. Added jobs are renamed with a hyphen and a number. They run in the test stage, or in scan-policies when there is no test stage; DAST runs in dast. For projects without a pipeline file, the policy creates the configuration itself.

How to trace it

A scanner job with a numbered name, or a scan in a project that includes no scanner template, points here. Open the group's security policies.

Gotchas
  • Toggles such as SAST_DISABLED skip only the project's own scanner jobs. They can't switch off a scan the policy enforces.
  • Scheduled scans ignore [skip ci], and run as the security_policy_bot user.
Organisation-wideUltimate#
Merge request approval policies
Scope
Organisation-wide
Binding
Always live
Control
Enforced
Phase
Run
Merged view
Not shown
Looks like
approval_policy:
  - name: Block new critical vulnerabilities
    rules:
      - type: scan_finding
        severity_levels: [critical]
        vulnerabilities_allowed: 0
    actions:
      - type: require_approval
        approvals_required: 1
What GitLab does

GitLab checks the rule against the scanners' results for the merge request. When it matches, the merge request needs the approvals the policy names. Rules can also look at licence findings, or apply to every merge request. By default, a missing scanner report makes the policy require approval: it fails closed.

How to trace it

A merge request blocked "because of a policy", with every pipeline job green, points here. The merge request page names the policy. The policy file in the policy project shows what it checks.

Gotchas
  • Formerly called scan result policies, under the key scan_result_policy. From 17.0, only approval_policy is accepted.
  • Removing a scanner job to get past a failing scan can make the merge request need approval, because the report is now missing.
Organisation-wideUltimateDeprecated#
Compliance pipelines
Scope
Organisation-wide
Binding
Pinned or floating
Control
Enforced
Phase
Compile
Merged view
Not shown
Looks like
Compliance framework: SOX
  Compliance pipeline configuration: compliance/sox.yml@megacorp/security/policies
What GitLab does

For every project with the framework's label, GitLab runs the compliance pipeline configuration instead of the project's own, unless that configuration includes the project's file. Deprecated in 17.3, planned for removal in 20.0, and replaced by pipeline execution policies.

How to trace it

Check the project for a compliance framework label, then read the file its framework names.

Gotchas
  • A project whose own .gitlab-ci.yml seems to be ignored may have a compliance pipeline that doesn't include it.
When you write it
Do this
Before you set a variable in your YAML, look for a variable with the same name in the settings of the project, its groups and the instance.
Why
A settings variable always wins over a value in YAML, whatever you write. The card your change is ignored shows the symptom.
Watch out
Settings belong to whoever owns them. If a setting is in your way, ask its owner to change it. Do not work around it in your file, because the next person will not know why.
What you can now do
  1. Find a variable set in the project's, a group's or the instance's settings, and know that it beats the YAML.
  2. Recognise a job that a policy added, and know that no toggle in your file removes it.
  3. Tell the organisation-wide mechanisms apart: settings, policies, templates, the instance's own configuration.
Spot the bugThe scan that won't switch off#

A payments engineer, fed up with a slow secret-detection job, added a toggle to the project's CI/CD variables:

SECRET_DETECTION_DISABLED = true

The project's own secret-detection job disappeared. A job named secret-detection-1 still runs in every pipeline.

QuestionWhere does secret-detection-1 come from, and why does the toggle not affect it?
Show the answer

The numbered name is the mark of a scan execution policy. MegaCorp's security team enforces secret detection on every branch. SECRET_DETECTION_DISABLED, which GitLab's template accepts as "true" or "1", skips only the scanner job the project defines itself. A scan that a policy enforces ignores it.

Whether the scan should run is the security team's decision, not the project's. Raise the slowness with them; don't look for a way round it.

Reuse outside the YAML#

Part IV · Reuse · Chapter 19 · 16 min read

YAML says what to run. In a large organisation, what actually happens is often decided somewhere else:

  • by the job's image
  • by a shell library, a company command-line tool, or a downloaded script
  • by a build tool's shared configuration
  • by the runner the job landed on

None of these shows in the merged configuration. A script line that says mc deploy tells you nothing about what mc does. The job log is where you start, and the question to ask of every command is "where did this come from?"

Friday, 10:30. The tools changed and the file didn't.

The image build fails today with a flag that mc no longer accepts. Your YAML has not changed in a month. The job runs in ci-tools:latest, the central team's toolbox image, and last night they pushed a new one. The real logic was never in your file. Where is it? (the card)

Take any command from a job log, and ask these questions in order:

Where a command in a job log comes from. Each answer is one section of this chapter.yesnoyesnoyesnoDid the log download it first, withcurl or git clone?A fetched script: read what thatURL served on that dayIs it a build tool, such as mvn, npm orgradle?Shared build configuration, suchas a parent POMIs it loaded by a source line inbefore_script?A shell library, usually shippedin the imageOtherwise it is a program inside theimage: read the image's DockerfileBehaves differently on differentrunners? Compare the runners'configuration
Figure Where a command in a job log comes from. Each answer is one section of this chapter.#
As text
  1. Did the log download it first, with curl or git clone? Yes: A fetched script: read what that URL served on that day. No: the next step.
  2. Is it a build tool, such as mvn, npm or gradle? Yes: Shared build configuration, such as a parent POM. No: the next step.
  3. Is it loaded by a source line in before_script? Yes: A shell library, usually shipped in the image. No: the next step.
  4. Otherwise it is a program inside the image: read the image's Dockerfile
  5. Behaves differently on different runners? Compare the runners' configuration

Toolbox images#

A toolbox image is an image a central team builds with every tool the organisation's jobs need. Every MegaCorp job runs in the ci-tools image unless it names another, because the base template sets it as the default:

megacorp/devops/ci-templatestemplates/base.ymldefault
default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  tags: [megacorp-shared]
  interruptible: true
  retry:
    max: 2
    when: [runner_system_failure, stuck_or_timeout_failure]
  before_script:
    - !reference [.snippets, functions]
    - mc_log "job $CI_JOB_NAME on $CI_COMMIT_REF_NAME"

The image carries more than tools. It carries MegaCorp's own logic, copied in by its Dockerfile, the recipe that builds the image:

megacorp/devops/ci-toolsDockerfilelibrary
COPY lib/ci-lib.sh /opt/megacorp/lib/ci-lib.sh
COPY bin/mc /usr/local/bin/mc
RUN chmod +x /usr/local/bin/mc
Outside the YAML#
Toolbox images
Scope
Organisation-wide
Binding
Pinned or floating
Control
Opt-in
Phase
Run
Merged view
Keyword only
Looks like
default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2
What GitLab does

The runner starts the job in a container from the image. Every command the script calls, and every file it reads that isn't in your repository, comes from the image's contents at the moment it was pulled.

How to trace it

The first lines of the job log name the image that was used. Find the project that builds it, here megacorp/devops/ci-tools, and read its Dockerfile for what gets copied in. Read the copied files at the version the image was built from.

Gotchas
  • An image tag like 3.2 can be pushed again with new contents. Only an image digest, a fixed ID for its exact contents such as @sha256:…, is truly pinned.
  • Upgrading the image can change what every job in the organisation does, with no commit in any pipeline file.

Shell function libraries#

A shell function is a named group of commands, like a small script inside the shell. Central teams package functions in a library file rather than repeating shell code in YAML. A source command loads the file, and MegaCorp's snippet library does nothing but that:

megacorp/devops/ci-templatestemplates/snippets.ymlsnippets
.snippets:
  functions:
    - source /opt/megacorp/lib/ci-lib.sh
  aws_login:
    - mc_aws_login "$AWS_ROLE_ARN"
  acr_push:
    - mc_acr_push "$IMAGE:$CI_COMMIT_SHA"
  maven_settings:
    - mkdir -p .m2
    - mc_maven_settings > .m2/settings.xml

The functions themselves live in the image:

megacorp/devops/ci-toolslib/ci-lib.shaws-login
# Exchange the job's OIDC ID token for temporary AWS credentials.
mc_aws_login() {
  local role="${1:?role ARN required}"
  local creds
  creds=$(aws sts assume-role-with-web-identity \
    --role-arn "$role" \
    --role-session-name "gitlab-${CI_PROJECT_ID}-${CI_JOB_ID}" \
    --web-identity-token "$MC_ID_TOKEN" \
    --duration-seconds 3600 \
    --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
    --output text) || { mc_log "AssumeRoleWithWebIdentity failed for $role"; return 1; }
  read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< "$creds"
  export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
  mc_log "assumed $role"
}
Outside the YAML#
Shell function libraries
Scope
Cross-project
Binding
Pinned or floating
Control
Opt-in
Phase
Run
Merged view
Keyword only
Looks like
before_script:
  - !reference [.snippets, functions]   # source /opt/megacorp/lib/ci-lib.sh
script:
  - mc_aws_login "$AWS_ROLE_ARN"
What GitLab does

A source line loads shell functions into the job's shell. They are then available to everything that runs later in the same shell: before_script and script share one. The library usually ships in the image, or arrives through a snippet or a download.

How to trace it

A command in the log that isn't a real program is a function. Search the snippets for the source line that loads it, then read the library file in the project that builds the image.

Gotchas
  • A job that redefines before_script loses the source line, and fails with "command not found" at the first function call.
  • Functions are written for one shell. A library written for bash fails in an image that only has sh.

Company command-line tools#

Some organisations wrap their deployment steps in a command of their own. MegaCorp's is called mc:

megacorp/devops/ci-toolsbin/mcdeploy
deploy)
  env="${1:?usage: mc deploy <environment> <image-ref>}"
  image="${2:?usage: mc deploy <environment> <image-ref>}"
  tag="${image##*:}"
  mc_log "promoting ${APP} ${tag} to ${env} through the GitOps repository"
  git clone --depth 1 \
    "https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
  yq -i ".image.tag = \"${tag}\"" "gitops/envs/${env}/${APP}.yaml"
  git -C gitops commit -am "deploy ${APP} ${tag} to ${env}"
  if [ "$env" = production ]; then
    # production changes wait in a merge request until someone merges them
    git -C gitops push origin "HEAD:deploy/${APP}-${tag}" \
      -o merge_request.create -o merge_request.target=main \
      -o merge_request.title="Deploy ${APP} ${tag} to production"
  else
    git -C gitops push origin HEAD:main
  fi
  ;;

The deployer's whole job is mc deploy "$ENVIRONMENT" "$IMAGE_REF". That one line clones a GitOps repository with a token, edits a values file and pushes a commit. A GitOps repository holds the desired state of the running systems, and a tool in the cluster applies whatever it says (chapter 45). Nothing in any pipeline file says any of this.

Outside the YAML#
Internal command-line tools
Scope
Organisation-wide
Binding
Pinned or floating
Control
Opt-in
Phase
Run
Merged view
Keyword only
Looks like
script:
  - mc deploy "$ENVIRONMENT" "$IMAGE_REF"
What GitLab does

The script calls a company tool, installed in the image or downloaded by an earlier command. All the behaviour is in the tool's own code, at whatever version the image contains.

How to trace it

Find how the tool got into the image, here COPY bin/mc in the Dockerfile, and read its source. Many such tools print their version. Add that command to a job to see which version you are running.

Gotchas
  • A tool that reads variables such as APP or GITOPS_TOKEN depends on names that no pipeline file documents.

Scripts fetched at run time#

The oldest trick of all is to download a script and run it, in one line:

megacorp/legacy/billing-batch.gitlab-ci.ymlfetched
test:
  <<: *defaults
  stage: test
  script:
    - curl -sSL https://gitlab.example.com/megacorp/devops/scripts/-/raw/main/run-tests.sh | bash
Outside the YAML#
Scripts fetched at run time
Scope
Cross-project
Binding
Pinned or floating
Control
Opt-in
Phase
Run
Merged view
Keyword only
Looks like
curl -sSL https://gitlab.example.com/megacorp/devops/scripts/-/raw/main/run-tests.sh | bash
What GitLab does

The job downloads a script from a URL or a repository while it runs, and executes it. What runs is whatever the URL served at that moment.

How to trace it

Open the URL from the job log. For a branch URL like …/raw/main/…, check the file's history, to see what it said when the job ran.

Gotchas
  • A URL on a branch is the most floating kind of reuse there is: unpinned, invisible to every GitLab view, and able to change mid-afternoon.
  • If the host is down, the job fails, even though nothing about your project changed.

Build-tool reuse#

Build tools share configuration too. A Maven project can inherit its build settings from a parent POM: a shared pom.xml published by another team. Java services at MegaCorp do:

megacorp/payments/payments-apipom.xmlparent
<parent>
  <groupId>com.example.megacorp</groupId>
  <artifactId>megacorp-parent</artifactId>
  <version>12.3.0</version>
</parent>
Outside the YAML#
Build-tool reuse
Scope
Cross-project
Binding
Pinned or floating
Control
Opt-in
Phase
Run
Merged view
Not shown
Looks like
script:
  - mvn $MAVEN_CLI_OPTS verify
What GitLab does

The pipeline calls the build tool. The build tool reads configuration that is shared through its own mechanism: a Maven parent POM, a Gradle convention plugin, a shared npm configuration package, or a Makefile included from elsewhere. Plugins, profiles and checks can all arrive that way.

How to trace it

Look in the build file for what it inherits: <parent> in a POM, or the equivalent in your build tool. Then read that shared configuration at the version named.

Gotchas
  • A new check in the parent POM can fail every build that uses it. The pipeline file won't have changed.

Runner configuration#

The runner itself can change every job it runs, through a file called config.toml that projects can't see:

runnersconfig.tomlrunner
[[runners]]
  name = "megacorp-eks-shared"
  # registered in GitLab as a group runner of megacorp, with the tag megacorp-shared
  url = "https://gitlab.example.com"
  executor = "kubernetes"
  environment = ["MAVEN_OPTS=-Xmx2g", "HTTPS_PROXY=http://proxy.example.com:3128"]
  pre_build_script = "echo 'runner: megacorp-eks-shared'"
  [runners.kubernetes]
    namespace = "gitlab-runners"
    image = "registry.example.com/megacorp/devops/ci-tools:3.2"
    # jobs run as this service account, which is bound to an AWS IAM role
    service_account = "gitlab-runner-jobs"
Outside the YAML#
Runner configuration
Scope
Organisation-wide
Binding
Always live
Control
Enforced
Phase
Run
Merged view
Not shown
Looks like
[[runners]]
  environment = ["MAVEN_OPTS=-Xmx2g"]
  pre_build_script = "echo 'runner: megacorp-eks-shared'"
What GitLab does

A runner's config.toml can add or overwrite environment variables for every job. It can run commands before fetching the source, before the job and after it. It chooses the executor and shell, can restrict which images jobs may use, and caps the log size, 4 MB by default. On Kubernetes, the pod's service account can carry cloud permissions.

How to trace it

The job log names the runner that ran the job. If a job behaves differently depending on which runner took it, compare the runners' configuration. Only the runners' administrators can read it, so ask them.

Gotchas
  • A job log longer than the runner's limit is cut off, so the end of a failing job may be missing.
  • Moving a job to a different runner, by changing its tags, can change its environment, its permissions and its image policy all at once.

GitLab Functions Experimental#

GitLab's experiment in reusable steps, formerly called CI/CD Steps, replaces a job's script with a list of steps. Each step either calls a published function or runs a script:

megacorp/data/monoci/docs.ymlsteps
lint-docs:
  image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
  run:
    - name: markdown_lint
      func: registry.example.com/megacorp/devops/functions/markdownlint:1.2.0
      inputs:
        path: docs
    - name: report
      script: echo "markdown lint finished"
Outside the YAMLExperimental#
GitLab Functions
Scope
Cross-project
Binding
Pinned or floating
Control
Opt-in
Phase
Run
Merged view
Keyword only
Looks like
lint-docs:
  run:
    - name: markdown_lint
      func: registry.example.com/megacorp/devops/functions/markdownlint:1.2.0
      inputs:
        path: docs
What GitLab does

The job runs its run: steps in order. A func: step loads a function and passes it inputs. The function can come from an OCI registry (a server that stores images and similar packages), a local path or, deprecated, a Git repository. A job that uses functions can't also have before_script, script or after_script.

How to trace it

Follow the function's reference to its registry or repository, and read it at the version given.

Gotchas
  • It is an experiment. Its keywords have changed before, from step: to func:, and may change again.
  • Loading functions from OCI registries needs a recent runner. An older runner fails the job, not the pipeline.
When you write it
Do this
Keep the job logic that your team owns in scripts in your own repository, and call those scripts from script:.
Why
You can then run the same scripts on your laptop, and a change to them goes through your team's review.
Watch out
When you use the central team's toolbox image, pin its tag. A moving tag, such as latest, changes your tools without any commit in your project. The card tools changed overnight shows the symptom.
What you can now do
  1. Follow any command in a job log to where its logic lives: the image, a fetched script, a shell library, or a build tool's own configuration.
  2. Pin a toolbox image to a tag, and explain what a moving tag does to a pipeline.
  3. Say which parts of a job Full configuration can never show you.
Spot the bugThe command no repository defines#

The deployer's job log shows it running mc deploy "staging" "123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:4f2c…". It then pushes a commit to a repository called gitops-config. A developer searches every repository they can see for a definition of mc, and finds nothing under that name.

QuestionWhere does mc come from, and how would you read exactly what mc deploy did in that job?
Show the answer

The deploy job runs in the ci-tools image, and its log names the image. That image's Dockerfile, in megacorp/devops/ci-tools, copies bin/mc into /usr/local/bin/mc. The command is a script in that repository.

Read bin/mc at the version the image was built from, not at the head of the branch. The deploy case shows the clone, the yq edit and the push. See toolbox images and company command-line tools.

Tracing a job to its source#

Part IV · Reuse · Chapter 20 · 8 min read

You now know all 37 mechanisms. This chapter puts them in order, as a procedure. Start from a job in the pipeline graph, and ask one question at a time:

  1. Is it in the merged configuration? If yes, walk its extends chain and its !reference tags back to the files.
  2. If it isn't, it was made when the pipeline was created (by a policy, a matrix or a downstream pipeline), or when it ran.
  3. Either way, finish in the job log. That is where you find the image, the runner, and the commands whose logic lives outside YAML.
Friday, 14:00. Where does this line come from?

maven-test fails with mc_log: command not found, again, in a different project. This time nobody changed before_script. You have the merged view, five included files, and a chain of three extends. You need a method, not a guess. (the card)

Here is the whole procedure on one page. The steps below take each branch in turn:

Tracing any job to where it came from. Whichever branch you take, finish in the job log (step 5).yesnoyesnoyesnoRead the job's name and stage in thepipeline graph (step 1)Is it in Full configuration, under itsname without any [ ] suffix?Walk its extends chain, !referencetags and includes (step 2)Is it in a .pipeline-policy stage, ornumbered like secret-detection-1?A security policy added it (step4)Is it in a card to the right of thegraph?A downstream pipeline: find itstrigger job (step 3)None of these: check the project'ssettings for a config file or AutoDevOps (steps 3 and 4)
Figure Tracing any job to where it came from. Whichever branch you take, finish in the job log (step 5).#
As text
  1. Read the job's name and stage in the pipeline graph (step 1)
  2. Is it in Full configuration, under its name without any [ ] suffix? Yes: Walk its extends chain, !reference tags and includes (step 2). No: the next step.
  3. Is it in a .pipeline-policy stage, or numbered like secret-detection-1? Yes: A security policy added it (step 4). No: the next step.
  4. Is it in a card to the right of the graph? Yes: A downstream pipeline: find its trigger job (step 3). No: the next step.
  5. None of these: check the project's settings for a config file or Auto DevOps (steps 3 and 4)

Step 1: read the name#

The job's name and stage in the pipeline graph often tell you which branch of the procedure you are on, before you open anything:

The name or stage looks likeIt is probablyGo to
an ordinary namea job in the configurationstep 2
a name ending in […], such as maven-test: [17]one job of a parallel: matrixstep 2, with the name before the colon
a stage called .pipeline-policy-pre or .pipeline-policy-posta pipeline execution policy jobstep 4
a scanner name with a hyphen and a number, such as secret-detection-1a scan execution policy jobstep 4
a name ending in :policy- and two numbersa policy job renamed to avoid a clashstep 4
a card to the right of the grapha downstream pipelinestep 3

Step 2: find it in the merged configuration#

Open Build › Pipeline editor, then Full configuration, and search for the job name followed by a colon. If it is there, everything the file-based mechanisms contributed has already been merged in. What remains is to find which file each part came from:

  1. Follow extends upward. Search the included files for each parent's name, such as .maven-test:, then its parent's, until there is none.
  2. Follow every !reference. Its first item names a hidden job, which you search for in the same way.
  3. Look for default: and top-level variables: in every included file, and check the job for inherit:.
  4. Note the ref of every include you passed through. A branch, or no ref at all, means the answer can change without a commit of yours.

Here is that walk for payments-api's maven-test. It starts from the only line that mentions the job by name:

megacorp/devops/ci-templatespipelines/java-service.ymljobs
maven-build:
  extends: .maven-build

maven-test:
  extends: .maven-test

image-build:
  extends: .image-build
  needs: [maven-build]
HopFileWhat it contributes
1payments-api .gitlab-ci.ymla MAVEN_CLI_OPTS variable, and the include of the golden pipeline at v4.2.0
2ci-templates pipelines/java-service.ymlthe job itself: maven-test: extends: .maven-test
3ci-templates templates/java-maven.yml.maven-test: stage, image, matrix, script, artifacts, rules
4ci-templates templates/java-maven.yml.maven-base: cache, variables, and a before_script built from !reference
5ci-templates templates/snippets.ymlthe snippets that before_script pastes in
6ci-templates templates/base.yml.megacorp-base: MC_TEAM, artifact expiry, and the rules that .maven-test replaced

If the project's configuration file lives in another project, its own pipeline editor can't show it. Open the file in that project, at the ref the setting names. GitLab can still assemble the whole configuration for you. On the local GitLab 19.3, the CI Lint API for web-portal, GET /api/v4/projects/:id/ci/lint?dry_run=true, returned web.yml's jobs with every include from ci-templates already merged in.

Step 3: if it's in no file#

A job that isn't in the merged configuration was added when the pipeline was created, or belongs to another pipeline:

  • Matrix jobs don't exist in the configuration under their bracketed names. Search for the name without the brackets.
  • Policy jobs are added when the pipeline is created. Go to step 4.
  • Jobs in a downstream card belong to another pipeline: a child from a file or a generated artifact, or another project's own pipeline. Open the card. Then find the trigger job in the upstream pipeline, which names the file, the artifact or the project.
  • A project without a .gitlab-ci.yml takes its whole configuration from a custom configuration file setting, from Auto DevOps, or from a scan execution policy that creates one (chapter 18).

Step 4: settings and policies#

These places hold configuration that no file in the project shows:

  • the project's CI/CD settings: its configuration file path, Auto DevOps, and variables
  • the CI/CD variables of every parent group, and of the instance
  • the group's security policies, on GitLab Ultimate
  • the project's compliance framework label, if it has one

If you can't open a level yourself, the question for its owner is specific: "Which variables, policies or settings at your level apply to this project?"

Step 5: finish in the job log#

Configuration tells you what GitLab asked for. The job log tells you what happened. Its first lines name the runner and the image. After that, every command the script ran is echoed. For each command that isn't a standard tool, ask where it came from:

  • a function loaded from a library in the image, like MegaCorp's mc_aws_login
  • a tool installed in the image, like MegaCorp's mc
  • a script downloaded during the job, with curl or git clone
  • behaviour configured in a build tool, like a parent POM or a Makefile

Chapter 19 covers each one. Behaviour that changes with the runner the job landed on, and that nothing in the job explains, belongs to the runner's configuration, which only its administrators can read.

What you can now do
  1. Take any job back to the file and line that set each of its settings, one hop at a time.
  2. Find a job that is in no file, and know that a policy put it there.
  3. See the merged configuration even when the project's file lives in another project.
Spot the bugTrace a production deploy#

web-portal's pipelines on main include a job called deploy-site that publishes to production. The repository has no .gitlab-ci.yml. You are asked three things: where is deploy-site defined, which file decides the command it runs, and which AWS role does it use?

QuestionTrace deploy-site to its source and answer all three.
Show the answer
  1. The project's CI/CD settings name pipelines/web.yml@megacorp/devops/ci-templates, a custom configuration file. That file defines deploy-site: extends: .publish-site, with its own rules.
  2. .publish-site is in templates/node.yml, included at v4.2.0. Its script runs aws s3 sync and a CloudFront invalidation. Its before_script pastes two snippets from templates/snippets.yml, which call mc_aws_login from the function library in the ci-tools image.
  3. .publish-site sets AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-web-publish, unless a settings variable of the same name overrides it. Check the project's and the groups' variables before you trust the YAML (extends).

Pipeline sources and types#

Part V · Conditions · Chapter 21 · 16 min read

Every pipeline is created by an event, and GitLab records which one in CI_PIPELINE_SOURCE: push, merge_request_event, schedule, web and a dozen more. The kind of pipeline then decides which variables exist:

  • CI_COMMIT_BRANCH is set in branch pipelines and schedules, and not in merge request pipelines
  • CI_COMMIT_TAG is set only for tags
  • CI_MERGE_REQUEST_* is set only in merge request pipelines

So the first question about any job that ran, or didn't, is "what kind of pipeline was this?" One push to a branch with an open merge request can start two pipelines.

Monday, 09:25. The job that only ran half the time.

contract-test runs on every push to a branch, and never in a merge request pipeline. Its rule says if: $CI_COMMIT_BRANCH, which looks right: it is a branch. The merge request's pipeline is for the same branch and the same commit. Why is one of them not "a branch"? (the card)

Where pipelines come from#

Seven kinds of event account for almost every pipeline you will meet in an enterprise project:

Each kind of event sets CI_PIPELINE_SOURCE, and the kind of pipeline decides which ref variables are set. A push to a branch with an open merge request can start both of the first two. What happened CI_PIPELINE_SOURCE The pipeline, and what it sets Push to a branch push Branch pipeline CI_COMMIT_BRANCH Merge request opened, or its branch pushed to merge_request_event Merge request pipeline CI_MERGE_REQUEST_* one push Push a tag push Tag pipeline CI_COMMIT_TAG A schedule comes due schedule Scheduled pipeline CI_COMMIT_BRANCH or CI_COMMIT_TAG New pipeline, the API, or a trigger token web · api · trigger A pipeline on the ref chosen CI_COMMIT_BRANCH or CI_COMMIT_TAG A trigger job in another pipeline pipeline · parent_pipeline Downstream pipeline CI_UPSTREAM_PIPELINE_ID A security policy's schedule comes due security_orchestration_policy Scan pipeline GitLab Ultimate
Figure Each kind of event sets CI_PIPELINE_SOURCE, and the kind of pipeline decides which ref variables are set. A push to a branch with an open merge request can start both of the first two.#

The full list of values, which is also what the pipelines API reports as a pipeline's source:

CI_PIPELINE_SOURCEWhat created the pipeline
pusha push to a branch, or a new tag
merge_request_eventa merge request being created, a push to its source branch, or Run pipeline on its Pipelines tab
schedulea pipeline schedule coming due, or someone selecting Run on it
webNew pipeline, under Build › Pipelines
apithe pipelines API
triggerthe pipeline trigger API, called with a trigger token
pipelinea trigger job in another project: a multi-project pipeline
parent_pipelinea trigger job in the same project: a child pipeline
security_orchestration_policya scan execution policy's schedule, on GitLab Ultimate
webidethe Web IDE
chata ChatOps command, typed into a chat tool connected to GitLab
externala CI service other than GitLab
external_pull_request_eventa pull request on GitHub, for a project that mirrors it
ondemand_dast_scanan on-demand DAST scan, which tests a running web application for security problems
ondemand_dast_validationa check that the site a DAST scan targets belongs to you

GitLab Duo Agent Platform also runs workload pipelines, on refs named refs/workloads/…, which pipeline lists show with the source duo_workflow. They are not part of your CI/CD configuration.

The pipeline list labels each pipeline by type: branch, tag, merge request, merged results or merge train. Look at the label before you look at any rule.

What each kind of pipeline sets#

Rules test variables, and the kind of pipeline decides which of those variables exist. GitLab documents these combinations:

VariableBranch pipelineTag pipelineMerge request pipelineScheduled pipeline
CI_PIPELINE_SOURCEpushpushmerge_request_eventschedule
CI_COMMIT_BRANCHthe branchnot setnot setthe schedule's branch
CI_COMMIT_TAGnot setthe tagnot setonly if the schedule runs on a tag
CI_MERGE_REQUEST_*not setnot setset while the merge request is opennot set
CI_PIPELINE_SCHEDULE_DESCRIPTIONnot setnot setnot setthe schedule's description

Three consequences catch almost everyone:

  • A merge request pipeline has no branch. Any rule that tests CI_COMMIT_BRANCH is false there. Test CI_MERGE_REQUEST_SOURCE_BRANCH_NAME or CI_MERGE_REQUEST_TARGET_BRANCH_NAME instead.
  • A tag pipeline has no branch either, even when the tagged commit is on main.
  • A schedule on a branch sets CI_COMMIT_BRANCH exactly as a push does. A rule meant for "merged to main" also matches the nightly schedule on main, unless it tests CI_PIPELINE_SOURCE. That is why payments-api's docs job runs every night (chapter 14).

A few other variables describe the pipeline rather than the commit:

  • CI_COMMIT_REF_NAME is the branch or tag the pipeline is built for.
  • CI_OPEN_MERGE_REQUESTS lists up to four merge requests whose source is the current branch. It is set only in branch and merge request pipelines, and only when such a merge request exists. Chapter 22 uses it to stop duplicate pipelines.
  • CI_COMMIT_BEFORE_SHA, the commit the branch pointed at before the push, is all zeros in merge request pipelines, scheduled pipelines, manual runs, and the first pipeline of a new branch or tag.

MegaCorp names these tests once, in its rules library, and every template picks from it:

megacorp/devops/ci-templatestemplates/rules.ymlrules-library
.rules:
  mr:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  default-branch:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  release-tag:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
  never-on-schedule:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
RuleMatchesWatch out for
mrevery merge request pipelinenothing else: branch, tag and scheduled pipelines never match it
default-branchpushes to main, and scheduled or manual runs on mainit can't tell a merge to main from the nightly schedule on main
release-tagtag pipelines whose tag looks like v1.4.0v1.4.0-rc1 matches nothing, so a release candidate's tag pipeline gets none of these jobs
never-on-scheduleremoves the job from scheduled pipelinesonly if GitLab reaches it before a rule that matches; see chapter 23

Merge request pipelines, in three flavours#

A merge request pipeline runs:

  • when a merge request is created from a branch with at least one commit
  • on every push to the merge request's source branch
  • when someone selects Run pipeline on the merge request's Pipelines tab

It tests the source branch alone, not the result of merging it. GitLab has two stricter variants, and CI_MERGE_REQUEST_EVENT_TYPE tells you which one you are in:

FlavourWhat it testsCI_MERGE_REQUEST_EVENT_TYPETier and list label
Merge request pipelinethe source branch onlydetachedevery tier; merge request
Merged results pipelinea temporary commit that merges the source into the latest target; if the two conflict, GitLab runs a plain merge request pipeline insteadmerged_resultPremium; merged results
Merge trainthe merge request together with every merge request queued ahead of itmerge_trainPremium; merge train

Merge request pipelines don't exist until the configuration asks for them: some rule has to match merge_request_event. Merged results pipelines and merge trains also need a project setting, under Settings › Merge requests.

Note
GitLab's documentation for GitLab 19.3 warns that a merge_request_event rule arriving through include: does not satisfy this requirement, and that the rule must be in the project's own .gitlab-ci.yml. On the local GitLab, payments-api gets every rule through includes. It still got a merge request pipeline when its merge request was opened, on every push after that, and whenever someone selected Run pipeline. If a project's merge request pipelines go missing, the documented caveat is one more thing to check.

Two situations change what a merge request pipeline can reach:

  • Merge requests from forks. The pipeline runs in the fork, with the fork's configuration and CI/CD variables, not the parent's. A member of the parent project can choose to run it in the parent instead. It then uses the fork's configuration with the parent's variables and runners, so GitLab warns first: the fork's code could try to read the parent's secrets.
  • Protected variables and runners Since 18.1. A merge request pipeline gets them only when both branches are protected and in the same project, the person running it can push or merge to the target, and a project setting allows it. Otherwise a job that needs a protected secret finds it empty.

One push, two pipelines#

Push to a branch that has an open merge request and GitLab receives two events: the push, and an update to the merge request. It tries to create a pipeline for each: a branch pipeline with source push, and a merge request pipeline with source merge_request_event. Each is created only if its rules add at least one job. If both qualify, the same commit shows up twice in the merge request's Pipelines tab, and runs every shared job twice.

The two can disagree, and it matters which one counts:

  • Pipelines must succeed checks the merge request pipeline, not the branch pipeline. GitLab documents a race condition in which pipeline's result blocks or passes the merge request, so a green branch pipeline beside a red merge request pipeline can go either way.
  • An invalid configuration pushed to a merge request's branch produces two failed pipelines, one of each type.

Which of the two you get is decided by workflow:rules and job rules, in chapter 22. Since 19.2 a project setting can also decide it: Settings › CI/CD › General pipelines › Skip branch pipelines for merge requests Beta. With it on, a push to a merge request's source branch creates only the merge request pipeline. The push that opens the merge request still makes a branch pipeline, and jobs with no rules at all start appearing in merge request pipelines.

Pipelines nobody pushed#

Several kinds of pipeline start without a commit. Each has a trait that confuses people:

KindSourceWhat to know
Scheduledscheduleruns with its owner's permissions; turns Inactive if the owner is blocked or leaves, until a Maintainer takes ownership; its variables exist only in its own pipelines; rules: changes is always true in it
New pipeline in the UIwebvalues typed into the form override every other variable of the same name, and are not masked
APIapithe ref and variables come from the call
Trigger tokentriggerCI_PIPELINE_TRIGGERED is true; the token acts with its owner's access (chapter 17)
Multi-projectpipelineevery job in it sees pipeline; it runs on the target's default branch unless the trigger job names one
Childparent_pipelineevery job in it sees parent_pipeline, even when the parent is a merge request pipeline
Policy scansecurity_orchestration_policycreated by a scheduled scan execution policy, not by anything in the project

A scheduled pipeline execution policy Since 19.2 also creates pipelines on a schedule. They run only the policy's jobs, never the project's (chapter 36).

The child pipeline row is the one that bites. A job in a child pipeline that tests $CI_PIPELINE_SOURCE == "merge_request_event" never runs, because the source is always parent_pipeline. The parent's CI_MERGE_REQUEST_* variables are passed down, so a child job that should run only for merge requests tests $CI_MERGE_REQUEST_ID instead.

A pipeline can also be created and then skipped. A commit message containing [ci skip] or [skip ci], in any capitalisation, still creates a pipeline, with no jobs and the status Skipped. The ci.skip push option does the same, except for merge request pipelines, which it doesn't skip. A security policy can forbid skipping.

When a variable exists#

GitLab sets predefined variables in three waves, and a rule can only test variables from the waves before it runs:

WaveSet whenExamplesCan be tested by
Pre-pipelinebefore GitLab starts building the pipelineCI_COMMIT_BRANCH, CI_COMMIT_TAG, CI_PIPELINE_SOURCE, CI_OPEN_MERGE_REQUESTS, CI_DEFAULT_BRANCH, CI_MERGE_REQUEST_*include: rules, workflow: rules, job rules, scripts
Pipelinewhile GitLab builds the pipelineCI_PIPELINE_IID, CI_JOB_NAME, CI_JOB_STAGE, CI_NODE_INDEX, CI_ENVIRONMENT_NAMEjob rules, scripts
Job-onlywhen a runner picks up the jobCI_JOB_ID, CI_JOB_TOKEN, CI_PIPELINE_ID, CI_PROJECT_DIR, CI_RUNNER_TAGSscripts only: never workflow, include, rules or trigger jobs

Three more exclusions follow from the same timing:

  • Dotenv variables, which a job writes for later jobs, never reach any rule. Every rule is evaluated before the first job runs.
  • rules: if can't use CI_ENVIRONMENT_SLUG, or the variables GitLab calls persisted, such as CI_PIPELINE_ID, CI_JOB_TOKEN and CI_REGISTRY_PASSWORD.
  • include can't use top-level variables: or job variables, because includes are resolved before either exists. It can use settings variables, CI_PROJECT_*, CI_PIPELINE_SOURCE, CI_COMMIT_REF_NAME, and variables from triggers, schedules and manual runs.

When a rule refers to a variable and never matches, look for that variable in these three lists before you look anywhere else.

When you write it
Do this
In rules, test $CI_PIPELINE_SOURCE to tell a merge request pipeline from a branch pipeline.
Why
That variable is set in every kind of pipeline, so a rule on it always has something to compare.
Watch out
Use $CI_COMMIT_BRANCH only when you mean a branch pipeline. In a merge request pipeline it is not set at all, so a rule on it never matches there. The card missing from merge request pipelines shows the symptom.
What you can now do
  1. Name the seven kinds of pipeline you will meet, and the variable that tells them apart.
  2. Say which predefined variables exist in each kind, and so which rules can match there.
  3. Read a rule and predict in which kinds of pipeline the job will appear.
Spot the bugThe check that never runs on merge requests#

payments-api adds a slow contract test. It should run only in merge requests that target main:

contract-test:
  stage: test
  script:
    - mvn $MAVEN_CLI_OPTS verify -Pcontract
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

It never appears in any merge request pipeline, including merge requests into main. It never appears in any other pipeline either.

QuestionWhy does the rule never match, and what should it say?
Show the answer

CI_COMMIT_BRANCH is not set in merge request pipelines, so the second comparison is false whenever the first is true. The branch a merge request is heading for is in CI_MERGE_REQUEST_TARGET_BRANCH_NAME:

rules:
  - if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH

See what each kind of pipeline sets.

workflow: whether a pipeline exists#

Part V · Conditions · Chapter 22 · 13 min read

workflow: rules decides whether a pipeline is created at all. GitLab checks it after assembling the configuration and before looking at a single job:

  • If no rule matches, there is no pipeline. For a push, you get no error anywhere.
  • The syntax is the same as job rules, but when can only be always or never.

Most organisations use it for one thing: stopping the duplicate branch and merge request pipelines from chapter 21. It can also name pipelines, set a variable for every job, and decide what a newer commit cancels.

Monday, 13:45. No pipeline, and no error.

You push a tag, v1.5.0-rc1, to cut a release candidate. Nothing happens. No pipeline, no red mark, nothing in the pipeline list. The jobs all have rules that allow tags. Something above the jobs said no before any of them was asked. (the card)

Checked before any job#

Whatever the jobs say, workflow has the first word. A job whose rules allow tag pipelines never runs if workflow refuses tag pipelines, because the pipeline it would have run in is never created.

A workflow rule may use if, changes, exists, variables and auto_cancel. Its when is limited to always and never. A rule that matches without a when lets the pipeline be created. GitLab accepts start_in, allow_failure and needs in a workflow rule without complaint, and ignores them.

Workflow rules can only test variables that exist before any job does: the pre-pipeline variables of chapter 21, and variables from settings, triggers, schedules and manual runs. A workflow rule that tests a job-only variable can't work.

What you see when no rule matches depends on how the pipeline was asked for. After a push, nothing appears. GitLab gives no error because nothing failed; the pipeline was simply never wanted. When someone asks for a pipeline through the API, GitLab 19.3 refuses it with this error:

The pipeline did not run. Review the workflow:rules configuration for the pipeline.

A different error means workflow let the pipeline through, but every job's rules left the job out:

The resulting pipeline would have been empty. Review the rules configuration.

MegaCorp's workflow, rule by rule#

Every MegaCorp project that uses a golden pipeline includes this file:

megacorp/devops/ci-templatestemplates/workflow.ymlworkflow
workflow:
  name: "$CI_PIPELINE_SOURCE: $CI_COMMIT_REF_NAME"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH
    - if: $CI_COMMIT_TAG

Read it top to bottom, stopping at the first rule that matches:

The pipeline GitLab is asked forFirst rule that matchesCreated?
a merge request pipeline1: merge_request_eventyes
a push to a branch that has an open merge request2: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTSno
a push to any other branch, main included3: $CI_COMMIT_BRANCHyes
a tag4: $CI_COMMIT_TAGyes
a schedule on main3yes
a schedule, API call or trigger on a branch that has an open merge request2no

Rules 1 to 3 are GitLab's documented switch pattern: branch pipelines until a merge request is opened, merge request pipelines after that. Rule 4 adds tags.

Rule 2 is broader than it looks. It doesn't ask whether the pipeline came from a push, so it refuses every pipeline on a branch that has an open merge request, unless it is a merge request pipeline. That includes pipelines started by an API call, a schedule, or a trigger from another project. GitLab's documentation gives the guard for exactly this, && $CI_PIPELINE_SOURCE == "push", and MegaCorp's rule doesn't have it. The exercise at the end of this chapter is what that costs.

Duplicate pipelines#

One push to a branch with an open merge request makes GitLab try to create two pipelines (chapter 21). You get both whenever some job's rules accept both. The configurations that cause it:

  • A final rule with no if, such as - when: on_success or - when: always, which matches every kind of pipeline. CI Lint and the New pipeline page show the warning Job may allow multiple pipelines to run for a single action for it.
  • Rules that list both push and merge_request_event, with no workflow to choose between them.
  • Jobs without rules mixed with jobs with rules. A job with no rules at all gets the legacy default: branches and tags, never merge requests. Put it beside a job whose rules match merge request pipelines and each push makes two pipelines, one for each job.

MegaCorp's monorepo has the third problem. It has no workflow section, its generator jobs have no rules, and its docs trigger has a rule that tests only which files changed:

megacorp/data/mono.gitlab-ci.ymlstatic-child
docs:
  stage: build
  trigger:
    include: ci/docs.yml
  rules:
    - changes: [docs/**/*]

A rule with only changes matches in any kind of pipeline, merge request pipelines included. When an open merge request touches docs/, every push to its branch makes two pipelines: a branch pipeline with the generator jobs, and a merge request pipeline containing only docs.

The local GitLab showed both pipelines, and a second problem. In every merge request pipeline, docs failed, with the reason downstream_pipeline_creation_failed. The child pipeline's own jobs have no rules, and GitLab documents that such jobs don't join merge request pipelines, child pipelines included. With none of its jobs added, the child pipeline couldn't be created (chapter 21).

Teams stop duplicates in one of four ways. Recognise which one a project uses and you know which pipelines to expect:

PatternWhat the workflow rules sayWhat you get
Switchmerge request events; never a branch with an open merge request; any other branchbranch pipelines until a merge request opens, then merge request pipelines only
Merge requests, main and tagsmerge request events; the default branch; tags; sometimes protected branchesno pipeline at all for a branch without a merge request
Branches only$CI_PIPELINE_SOURCE == "push", and no rule for merge request eventsno merge request pipelines, and no merged results or merge trains
The project settingnothing; the setting Skip branch pipelines for merge requests is onas the switch, for pushes only; beta since 19.2

Names and variables for the whole pipeline#

workflow: name gives every pipeline a title in the pipeline list, and the value is available to jobs as CI_PIPELINE_NAME. MegaCorp's name shows the source and the ref, such as push: main or schedule: main. That tells you the kind of pipeline before you open it. If the name comes out empty, because every variable in it is empty, the pipeline gets no name.

workflow: rules: variables sets variables when a rule matches:

  • They become default variables for every job, just like top-level variables:, and they replace top-level variables of the same name.
  • A variable set in the job itself still wins.
  • Trigger jobs pass them downstream, where they replace the downstream project's own values of the same name. A generic name like ENVIRONMENT set here can silently change another team's pipeline.

To stop that, give such variables unique names, or have the trigger job list what it passes with inherit: variables.

What a newer commit cancels#

Push twice in quick succession and GitLab can cancel the older pipeline for you, but only if the project setting Settings › CI/CD › General pipelines › Auto-cancel redundant pipelines is on. It applies only to a newer commit on the same branch; selecting New pipeline again for the same commit cancels nothing. With the setting off, interruptible does nothing.

workflow: auto_cancel: on_new_commit takes one of three values. conservative, the default, cancels the older pipeline unless a job with interruptible: false has already started; jobs that haven't started yet count as interruptible. interruptible cancels only the jobs marked interruptible: true, and the rest carry on. none cancels nothing.

The companion key on_job_failure: all cancels every running job as soon as one job fails, instead of letting its stage finish. Both keys can be overridden per rule with workflow: rules: auto_cancel, for example so that pipelines on protected branches are never cancelled.

Under the default, a single non-interruptible job decides for the whole pipeline. Once it starts, the pipeline can no longer be auto-cancelled. A downstream pipeline is cancelled along with its parent if none of its own non-interruptible jobs has started yet.

MegaCorp sets interruptible: true in default:, so every job that inherits it can be cancelled by a newer push. That is fine for builds and tests. Deployment jobs are different: GitLab advises against making them interruptible, because cancelling one halfway can leave a partial deployment. When a template puts interruptible: true in default:, check that deploy jobs set it back to false.

When no pipeline appears#

Work down this list. It runs from the most common cause to the least:

What you seeLikely causeHow to confirm
no pipeline at all after a pushworkflow: rules matched nothingrun the rules by hand against the pipeline's variables (chapter 21's table)
no pipeline, and workflow would have allowed itevery job's rules left it out; a pipeline with no jobs isn't createdCI Lint, simulating the branch
no pipeline, although jobs should matchthe only jobs left are in .pre or .post, which can't form a pipeline alonecheck which stages the matching jobs are in
a pipeline marked Skipped, with no jobs[ci skip] or [skip ci] in the commit message, or the ci.skip push optionthe commit message
a failed pipeline with an error and no jobsthe configuration is invalidthe error on the pipeline page, or the pipeline editor
jobs or variables missing for no visible reasona byte-order mark at the start of a YAML file, which the pipeline editor can't showa hex viewer, or a tool that shows invisible characters
a merge request stuck on Checking pipeline statusPipelines must succeed is on, and nothing lets a pipeline run for its latest committhe merge request's Pipelines tab
When you write it
Do this
As soon as any job has a merge request rule, add a workflow that allows one kind of pipeline per push. Allow a merge request pipeline while a merge request is open, and a branch pipeline otherwise.
Why
GitLab checks the workflow before it looks at any job, so it is the one place that decides how many pipelines a push gets.
Watch out
Without it, one push can create two pipelines, and every job runs twice. The card duplicate pipelines shows how that looks.
What you can now do
  1. Write a workflow that names which pushes get a pipeline, and know that it is asked first.
  2. Explain a push that produced no pipeline, and find the rule that refused it.
  3. Give each push exactly one pipeline, and set a variable for the whole pipeline from a workflow rule.
Spot the bugThe pipeline nobody can start#

A payments-api developer is working on feature/retry-fix, which has an open merge request. They want a one-off branch pipeline with an extra variable, so they open Build › Pipelines › New pipeline, choose their branch, add the variable and submit. No pipeline is created. The same happens when a colleague uses the API.

megacorp/devops/ci-templatestemplates/workflow.ymlworkflow
workflow:
  name: "$CI_PIPELINE_SOURCE: $CI_COMMIT_REF_NAME"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH
    - if: $CI_COMMIT_TAG
QuestionWhich rule refuses the pipeline, and what are the two ways round it?
Show the answer

Rule 2. A pipeline started on a branch from the UI or the API has CI_COMMIT_BRANCH set, and CI_OPEN_MERGE_REQUESTS is set because the branch has an open merge request. The rule doesn't check the source, so it refuses the pipeline.

  • Now: run a merge request pipeline instead. On the merge request's Pipelines tab, Run pipeline with modified values lets you set variables and inputs.
  • For good: ask the template owners to narrow the rule to pushes, as GitLab documents: if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push".

See MegaCorp's workflow, rule by rule.

Job rules: is this job in the pipeline?#

Part V · Conditions · Chapter 23 · 17 min read

When GitLab creates a pipeline, it reads each job's rules from the top. The first rule that matches decides the outcome:

  • the job is added: normally, as a manual job, or delayed
  • or the rule says when: never and the job is left out

If no rule matches, the job is left out. A matching rule can also change the job's allow_failure, needs, variables and interruptible.

A rule's conditions are if (variables), changes (files changed) and exists (files present). A job with no rules at all gets an older default: branches and tags, never merge requests.

Tuesday, 10:00. A job that ran when nothing changed.

docs-build has rules: changes: [docs/**/*]. You push a new branch with a code fix and no docs change, and the docs build runs anyway. On the next push to the same branch it stays away, as it should. What is different about the first push? (the card)

Four gates#

A job passes four gates before it runs, and GitLab checks each at a different moment:

The condition funnel. Four gates, checked while GitLab assembles the configuration, creates the pipeline and runs it. Each gate says no in its own way. assembling the configuration creating the pipeline running it include: rules which files join the configuration? workflow: rules is there a pipeline at all? job rules is the job in it, and as what? when, needs does it start, and when? runs its jobs never exist no pipeline; after a push, no error not in the pipeline skipped, or waiting
Figure The condition funnel. Four gates, checked while GitLab assembles the configuration, creates the pipeline and runs it. Each gate says no in its own way.#
GateCheckedCan testWhen it says no
include: ruleswhile GitLab assembles the configurationpre-pipeline variables, settings variables, and filesthe included file's jobs never exist, and nothing in the pipeline mentions them
workflow: rulesbefore any job is consideredvariables that exist before any job, and filesthere is no pipeline; after a push there is no error either
job rulesonce for each job while the pipeline is created, and once for each matrix jobthose, plus pipeline variables and the job's own variablesthe job is not in the pipeline
when, needs, allow_failurewhile the pipeline runsthe outcome of earlier jobsthe job is skipped, or waits for a person or a timer

Chapter 22 covered the first two gates, and chapter 24 covers the last. This chapter is about the third.

First match wins#

A job's rules is a list, and GitLab reads it from the top, stopping at the first rule whose conditions all hold:

  • A rule needs at least one of if, changes, exists or when. When a rule has several conditions, all of them must be true.
  • A rule with only when has no conditions, so it matches whenever GitLab reaches it. It is the "otherwise" at the end of a list.
  • A matching rule with when: never leaves the job out. Any other matching rule adds it.
  • A matching rule without when uses the job's own when, which defaults to on_success. A when written in the rule overrides the job's.
  • If GitLab reaches the end of the list without a match, the job is left out.

Order is everything. MegaCorp's two deploy jobs show both common styles:

megacorp/devops/ci-templatespipelines/java-service.ymldeploys
deploy-staging:
  extends: .deploy
  needs: [image-build]
  variables:
    ENVIRONMENT: staging
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, default-branch]

deploy-prod:
  extends: .deploy
  needs: [image-build]
  variables:
    ENVIRONMENT: production
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
      when: manual

deploy-staging pastes two lists from the rules library with !reference. GitLab flattens them into one list, in the order they are written (chapter 14). Read it for four pipelines:

PipelineRule 1: never-on-scheduleRule 2: default-branchdeploy-staging
the nightly schedule on mainmatches, with when: nevernot reachedleft out
a push to mainnomatchesadded, on_success
a merge requestnonoleft out
tag v1.4.0nonoleft out

Swap the two rules and the nightly schedule matches default-branch first, because a schedule on main sets CI_COMMIT_BRANCH. Nothing in the list looks wrong; only its order is.

deploy-prod has a single rule, with when: manual. In a tag pipeline whose tag looks like a release, it is added as a manual job. Everywhere else it is left out. Because the when: manual is inside a rule, the job blocks its pipeline until someone runs it; chapter 24 explains why.

Writing if conditions#

An if is an expression over variables. Some forms use a regular expression: a pattern for matching text, written between slashes. For example, /^release/ means "starts with release". These forms cover almost everything you will read:

ExpressionTrue whenWatch out for
$VARVAR is set and not emptyany non-empty value counts, including "false" and "0"
$VAR == "text"the value is exactly textcase matters; the variable goes on the left, and only the string is quoted
$VAR != "text"the value is anything elsealso true when VAR isn't set at all
$VAR == nullVAR is not seta variable set to "" is not null
$VAR == ""VAR is set, and empty
$A == $Bboth have the same value
$VAR =~ /re/the regular expression matches part of the valueunanchored: /release/ matches pre-release-notes; write /^release/
$VAR =~ /re/ithe same, ignoring case
$VAR !~ /re/the regular expression doesn't match
$VAR =~ $PATTERNthe value matches the regular expression stored in PATTERN, slashes includedvariables inside the stored expression are not expanded
a && b, a || bboth, or either&& binds tighter than ||; add parentheses to be sure
!$VAR, !(a == b)the opposite; since 18.11! tests for empty or unset, not for "false"

The surprising rows were checked on the local GitLab 19.3:

  • "false" counts as true.
  • != is true for a variable that isn't set.
  • An empty variable is not null.
  • && binds before ||.

The same test turned up one more trap. Put a negated expression in quotes: if: '!($VAR == "x")'. Without the quotes, GitLab rejects it with invalid expression syntax, even though GitLab's own documentation shows it unquoted.

Four syntax traps make GitLab reject a configuration, or accept it and mean something else:

  • Quote strings, never variables. $ENV == production (unquoted string), "$ENV" == "production" (quoted variable) and ${ENV} == "production" (braces) are all invalid, and GitLab refuses the whole configuration.
  • A regular expression is one whole /…/. issue-/.*/ doesn't work; /issue-.*/ does. A pattern of a single character, such as /./, is rejected as invalid expression syntax. Matching uses RE2 syntax.
  • A right-hand side without slashes is not a regular expression. $A =~ "text" is accepted, but tests whether A's value appears inside text: "23" =~ "1234" is true. Always write the slashes.
  • Values are not expanded twice. If a variable's value is $OTHER, the condition sees that literal text, not OTHER's value. This holds for matrix values too.

A condition can only test a variable that exists when the pipeline is created. If a rule never matches, check the variable against chapter 21's list: job-only variables, persisted variables and dotenv variables are never there.

changes and exists#

Two conditions test files instead of variables:

ConditionTrue whenWatch out for
changes: [paths]a changed file matches one of the paths"changed" depends on the kind of pipeline; see the next table
changes: paths with compare_toa file differs from the branch, tag or commit namedin a merged results pipeline, the comparison also includes the target branch's own changes
changes: regexpa changed path matches a Ruby regular expression; since 19.2anchor it with \A and \z, not ^ and $
exists: [paths]a file or directory matching one of the paths is in the repositorya directory needs a trailing slash; artifacts are never seen
exists: paths with project and refthe file exists in another project, at a ref

"Changed" is the trap. What a file is compared against depends on the kind of pipeline:

Pipelinechanges compares withSo it is true
merge requestthe target branchwhen the merge request as a whole touches the files
push to an existing branchthe branch's previous commitwhen this push touches the files
the first push of a new branch, or a new tagnothingalways
scheduled, New pipeline, API or trigger: anything without a pushnothingalways

MegaCorp's monorepo shows the third row. Its docs trigger job has a rule with changes: [docs/**/*] and nothing else. When the local GitLab received mono's very first push to main, the pipeline included docs, although the repository had no docs/ directory at all.

GitLab's advice follows from the table: pair changes with an if that limits it to merge request or branch pipelines, or give it a compare_to.

Four smaller rules about paths:

  • Don't write ./ or //. Paths are compared as plain text, not resolved like a shell path.
  • A variable ending in a slash turns $DIR/* into dir//*.
  • Root-level wildcards need quotes in YAML: "*.json".
  • Each list holds at most 50 paths. After 50,000 checks a pattern counts as matched, so in a huge change or repository every changes or exists rule is true.

One more difference: in include: rules, exists looks in the project and ref of the file that contains the include. In a job, it looks in the project running the pipeline. A template that works in its own repository can test the wrong project once someone else includes it.

What a matching rule can change#

Key in the ruleEffect, only when this rule is the one that matched
whenon_success, on_failure, always, manual or delayed add the job; never leaves it out
allow_failurereplaces the job's value; the default is false, even with when: manual
needsreplaces the job's entire needs list; [] makes it start at once
variablesadds variables, or overrides the job's, for this case only
interruptiblereplaces the job's value
Gotcha
when: manual means two different things depending on where it is written. On the job itself, it makes an optional manual job, with allow_failure: true. Inside a rule, it makes a blocking one, with allow_failure: false: the pipeline waits at that stage until someone runs it. Add allow_failure: true to the rule to get the optional kind.

Rules and matrix jobs#

With parallel: matrix, GitLab evaluates the rules separately for each generated job, using that job's values. In MegaCorp's matrix, a rule if: $JDK == "17" would keep maven-test: [17] and drop maven-test: [21]. Matrix values can also appear in changes and exists paths, which lets a monorepo run each service's job only when that service's files change.

Legacyonly and except

Older templates control jobs with only and except, which GitLab has deprecated; use rules instead. They list branch names, regular expressions and kinds of pipeline, such as branches, tags, merge_requests, schedules, pushes, web, api, triggers and pipelines.

  • A job with no only, except or rules behaves as only: [branches, tags]. That is why such a job never appears in merge request pipelines.
  • only: branches matches scheduled pipelines too, because a schedule runs on a branch. except: schedules removes them.
  • Mixing only/except jobs with rules jobs is a classic cause of duplicate pipelines (chapter 22).

To translate a job to rules, write one if rule for each only entry. For each except entry, write one when: never rule, then finish the list with - when: on_success.

Why was this job left out?#

  1. Name the pipeline. Its label and source tell you which variables exist (chapter 21).
  2. Get the job's real rules from the Full configuration view. extends may have replaced the rules you expected (the card), and every !reference is already expanded there.
  3. Walk the list from the top with those variables, and stop at the first match.
  4. For each variable a rule tests, check that it exists when the pipeline is created, and find where its value is set (chapter 26).
  5. For each changes, check which comparison applies in this kind of pipeline.
  6. If the job is in the pipeline but never ran, you have moved on to the fourth gate: chapter 24.
When you write it
Do this
Order a job's rules from the most specific case to the most general, and end with the case you want by default.
Why
GitLab stops at the first rule that matches. If a general rule comes first, the specific ones below it are never reached.
Watch out
A job with rules, none of which match, is left out of the pipeline without a message. And a changes rule is always true on the first push of a new branch. The card runs on a new branch explains why.
What you can now do
  1. Read a job's rules top to bottom and say whether the job is in a given pipeline, and how it will run.
  2. Order rules so that the specific cases come first and the default comes last.
  3. Explain the new-branch surprise of changes, and the silence of a job with no matching rule.
Spot the bugThe smoke test that runs every night#

A team adds a smoke test that should run after each merge to main, using MegaCorp's rules library:

smoke-test-staging:
  stage: deploy
  script:
    - ./scripts/smoke-test.sh staging
  rules:
    - !reference [.rules, default-branch]
    - !reference [.rules, never-on-schedule]

It does run after merges. It also runs at 02:00 every night, in the scheduled pipeline on main, against a staging environment nobody has changed.

QuestionThe list includes never-on-schedule. Why doesn't it stop the nightly run?
Show the answer

First match wins. A schedule on main sets CI_COMMIT_BRANCH to main, so default-branch matches first and adds the job. GitLab never reaches never-on-schedule.

Put the exclusion first, as MegaCorp's deploy-staging does:

rules:
  - !reference [.rules, never-on-schedule]
  - !reference [.rules, default-branch]

See first match wins.

Order and gates: when a job starts, and what stops it#

Part V · Conditions · Chapter 24 · 17 min read

A job that made it into the pipeline still has to wait its turn:

  • Stages run in order. A job starts when every job in the earlier stages has succeeded, been allowed to fail, or been left waiting as an optional manual job.
  • needs replaces that barrier with a list of jobs. The job starts when those finish, whatever else is still running.
  • when decides the rest: on_success (the default), on_failure, always, manual or delayed.

Then come the gates:

  • allow_failure stops a failure from counting.
  • A manual job added by a rule holds the pipeline until someone runs it.
  • resource_group makes jobs wait their turn, across pipelines.
  • retry, timeout and interruptible decide what happens when things go wrong.
Tuesday, 16:30. Blocked, with nothing failed.

The pipeline on main says blocked. Every job is green except approve-release, which a rule added with when: manual. Deploy is waiting for it; so is everyone. Yesterday the same job was there and the pipeline passed without anyone pressing anything. What changed the meaning of the button? (the card)

Stages: the default barrier#

Without needs, a stage starts only when the stage before it has finished. For that barrier, three outcomes count as finished well:

  • success
  • a failure of a job marked allow_failure: true
  • an optional manual job that nobody has started, which counts as skipped

If any other job fails, the pipeline is marked failed and later stages never start. Jobs already running in the failing stage carry on to the end.

The stage list has fixed ends. .pre runs first and .post last, and neither needs listing. A pipeline execution policy can add .pipeline-policy-pre before .pre and .pipeline-policy-post after .post (chapter 36).

Three smaller rules explain some odd-looking graphs:

  • A job with no stage lands in test.
  • A listed stage that no job uses is not shown at all.
  • A pipeline whose only jobs are in .pre or .post is not created.

needs: a graph instead of a barrier#

A job with needs ignores the stage barrier. It starts as soon as every job it lists has finished, even while other jobs in earlier stages are still running. MegaCorp's golden pipeline uses it three times: image-build needs maven-build, and both container_scanning and deploy-staging need image-build. Here is what that does to a pipeline on main:

The same payments-api jobs on main, ordered by stages alone, then with MegaCorp's needs. Durations are illustrative. With needs, deploy-staging waits only for image-build. Stages only: each stage waits for every job before it build test package scan deploy maven-build maven-test: [17] maven-test: [21] sonar-scan image-build container_scanning deploy-staging With MegaCorp's needs: each job waits only for what it lists the tests finish maven-build maven-test: [17] maven-test: [21] sonar-scan image-build container_scanning deploy-staging starts before the tests have finished
Figure The same payments-api jobs on main, ordered by stages alone, then with MegaCorp's needs. Durations are illustrative. With needs, deploy-staging waits only for image-build.#

The pipeline is much shorter, and one bar is in the wrong place. deploy-staging lists only image-build. So it starts as soon as the image exists, while maven-test and container_scanning may still be running. A test that fails afterwards turns the pipeline red, but staging already has the build. Nobody wrote "deploy without testing"; the missing entries in a needs list say it for them.

The details that matter when you read a needs list:

  • needs: [] starts the job as soon as the pipeline is created, as .pre jobs do. It suits linters and scanners that read only the source.
  • A job can need jobs in its own stage. A job can list at most 50 jobs.
  • Needing a matrix job means needing all of it. needs: [maven-test] waits for every maven-test: […] job. needs: parallel: matrix picks particular ones.
  • Artifacts follow needs. A job with needs downloads artifacts only from the jobs it lists. artifacts: false on an entry downloads nothing from that job. Don't combine needs with dependencies.
  • Seeing the graph: on the pipeline page, select Job dependencies. The default view groups jobs by stage and hides the lines.

needs and rules collide when a needed job is left out of a pipeline by its own rules. GitLab then refuses to create the pipeline at all, with an error naming both jobs. There are two cures:

  • Give the needing job rules that keep it out whenever the needed job is out.
  • Mark the entry optional: true. The job then waits for the needed job when it exists, and ignores it when it doesn't. If every entry is optional and none of those jobs exists, the job starts at once, as if it had needs: [].

Here is the error from the local GitLab 19.3, for a unit-tests job that needs a compile job its rules had left out:

'unit-tests' job needs 'compile' job, but 'compile' does not exist in the pipeline. This might be because of the only, except, or rules keywords. To need a job that sometimes does not exist in the pipeline, use needs:optional.

Three other keys are spelt needs but do different jobs:

KeyWhat it doesWatch out for
needs: projectdownloads artifacts from the latest successful run of a job in another project, at a ref; Premiumit doesn't wait: if that project's pipeline is still running, you get the previous run's artifacts
needs: pipeline with joblets a child pipeline download artifacts from a job in its parent or a siblingthe job must have succeeded
needs: pipeline alonecopies the latest status of another project's default-branch pipeline into this jobit is a status mirror, not a dependency

when: the job's own condition#

whenThe job runs
on_success (the default)when every job in earlier stages succeeded, was allowed to fail, or is an unstarted manual job
on_failureonly when at least one job in an earlier stage failed; for cleanup and notifications
alwayswhatever happened earlier
manualwhen a person starts it
delayedafter the time in start_in
nevernever; allowed only in rules and workflow: rules

One consequence surprises people: a failed job with allow_failure: true counts as a success. An on_failure notification job therefore stays silent when the only failure was allowed.

allow_failure: failures that don't count#

A job with allow_failure: true can fail without failing the pipeline. The pipeline passes, and the job carries an orange warning. The default depends on how the job was made manual:

The default for allow_failure depends on where when: manual was written. On the job itself, the default is true: an optional manual job. From a rule, the default is false: a blocking manual job. For anything else it is false.

allow_failure: exit_codes: [137, 255] allows a failure only for those exit codes. Any other failure still counts.

Scanners are the usual place to find it. In MegaCorp's pipeline on main, the local GitLab created container_scanning, secret_detection and sonar-scan all with allow_failure: true. That comes from GitLab's scanner templates and from the component. A scanner that crashes, or finds something, doesn't turn the pipeline red unless a policy makes it (chapter 36).

Manual and delayed jobs#

A manual job is shown as skipped until someone runs it, and there are two kinds:

An optional manual job, with allow_failure: true, lets the pipeline carry on: it can pass without the job ever running. A blocking one, with allow_failure: false, stops the pipeline at the job's stage with the status blocked. Later stages wait, and a merge request set to "Pipelines must succeed" won't merge a blocked pipeline.

Running a manual job needs permission to merge to the pipeline's branch. On Premium, a protected environment narrows that to a named list of people. If you are not on it, the job shows You are not authorized to run this manual job. manual_confirmation adds an "are you sure?" message. Variables typed in when starting a manual job are visible to other members and are not masked.

A release of payments-api passes through two manual gates. deploy-prod is manual by a rule, so the tag pipeline is blocked until someone runs it:

payments-api tag v1.4.0: the release waits at deploy-prod, which then starts the deployer's own pipelinebuildmaven-buildpackageimage-builddeploy▶deploy-proddownstreammegacorp/platform/deployermulti-project pipeline
Pipeline payments-api tag v1.4.0: the release waits at deploy-prod, which then starts the deployer's own pipeline#
As text
  1. build: maven-build
  2. package: image-build
  3. deploy: deploy-prod (manual)

Needs: image-build waits only for maven-build; deploy-prod waits only for image-build.

Downstream: deploy-prod triggers megacorp/platform/deployer (multi-project pipeline).

The deployer then asks again. Its deploy job is manual whenever ENVIRONMENT is production:

megacorp/platform/deployer.gitlab-ci.ymldeploy
variables:
  ENVIRONMENT: $[[ inputs.environment ]]

deploy:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  resource_group: $APP-$ENVIRONMENT
  environment:
    name: $ENVIRONMENT/$APP
  script:
    - mc deploy "$ENVIRONMENT" "$IMAGE_REF"
  rules:
    - if: $IMAGE_REF == null
      when: never
    - if: $ENVIRONMENT == "production"
      when: manual
    - when: on_success

So a production release takes two clicks in two projects. Between them, deploy-prod uses strategy: depend, and with it the trigger job shows running for as long as the deployer waits for its manual job. "The deploy has been running for three hours" usually means "nobody has pressed the second button". strategy: mirror would copy the deployer's own status instead.

A delayed job waits start_in, anything from one second to one week, counted from when its previous stage completes. Its stage doesn't finish until it has run. A bare number needs quotes ('30' means 30 seconds). To start one early, select Unschedule, then Run.

resource_group: one at a time#

The deployer's deploy job carries resource_group: $APP-$ENVIRONMENT. Jobs with the same resource group never run at the same time, even in different pipelines of the same project. A second payments-api staging deploy waits for the first, showing Waiting for resource: payments-api-staging and the status waiting_for_resource. Deploys of other services, with other names, go ahead.

The order in which waiting jobs are released is the group's process mode, and it isn't in any YAML file. It is set through the API:

Process modeWhen the resource frees up, GitLab starts
unordered (the default)whichever waiting job is ready
oldest_firstthe job from the oldest pipeline
newest_firstthe job from the newest pipeline, so older deploys are skipped over
newest_ready_firstthe newest job that is ready to run

With oldest_first, a parent pipeline and its child that use the same resource group can deadlock, each waiting for the other. The documented fix is to put the resource group on the trigger job instead.

When things go wrong: retry, timeout, interruptible#

retry reruns a failed job up to twice. With when, it retries only certain kinds of failure; with exit_codes, only certain exit codes. MegaCorp sets it for every job in default::

megacorp/devops/ci-templatestemplates/base.ymldefault
default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  tags: [megacorp-shared]
  interruptible: true
  retry:
    max: 2
    when: [runner_system_failure, stuck_or_timeout_failure]
  before_script:
    - !reference [.snippets, functions]
    - mc_log "job $CI_JOB_NAME on $CI_COMMIT_REF_NAME"

That configuration has aged twice without anyone touching it:

  • stuck_or_timeout_failure was deprecated in 19.1, in favour of more precise reasons. The local GitLab 19.3 accepts it, but warns on every lint:
    retry uses deprecated `when` value(s): stuck_or_timeout_failure. These match the more specific failure reasons that replaced them; migrate to those reasons.
  • 19.1 also moved some failures out of runner_system_failure, into the new runner_external_dependency_failure (such as a registry that couldn't be reached) and runner_interrupted. A job that used to be retried after a network blip may now fail at once.

timeout sets how long the job may run, in words: 90 minutes, 3h 30m. It can be longer than the project's timeout, but never longer than the runner's. It can't be set in default:. Jobs that stop making progress are dropped, with a failure reason that says why:

The job wasDropped afterFailure reason
pending, with a runner that could take it24 hoursstuck_pending_with_matching_runners
pending, with no runner that could take it1 hourstuck_pending_no_matching_runners
running, with no updates from the runner30 minutesno_updates_running
running past its timeoutthe timeout plus 15 minutesserver_timeout_running

interruptible decides whether a newer commit may cancel the job; chapter 22 covers it.

When you write it
Do this
Use needs to start a job as soon as the jobs it depends on have finished, and list every job it takes artifacts from.
Why
Without needs, a job waits for the whole previous stage, even for jobs it never uses.
Watch out
A manual job that a rule adds is not allowed to fail, so the pipeline shows as blocked until someone runs it. If the job is optional, add allow_failure: true to that rule. The card a play button that waits shows the symptom.
What you can now do
  1. Say when a job starts: after its stage, or as soon as its needs are done.
  2. Tell a manual job that blocks from one that doesn't, and choose which you want.
  3. Read a stuck or blocked pipeline and name the gate that holds it.
Spot the bugThe nightly pipeline that never starts#

An earlier version of MegaCorp's security.yml gave the container scan these rules:

container_scanning:
  stage: scan
  needs: [image-build]
  rules:
    - if: $CONTAINER_SCANNING_DISABLED == "true"
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

When the payments team added a nightly schedule on main, every nightly run failed before a single job started. Pushes to main and merge requests were fine.

QuestionWhat stops the scheduled pipeline from being created, and what are the two ways to fix it?
Show the answer

In a schedule on main, CI_COMMIT_BRANCH is main, so the last rule adds container_scanning. But image-build inherits .megacorp-base's rules, which start with never-on-schedule, so it is left out. The pipeline would contain a job that needs a job that doesn't exist, so GitLab refuses to create it.

  • Keep the scan out of schedules too, by putting - !reference [.rules, never-on-schedule] before the other rules. That is what MegaCorp's current security.yml does.
  • Or make the need optional: true, if a scan without a fresh image makes sense.

See needs: a graph instead of a barrier.

Truth tables: predicting which jobs run#

Part V · Conditions · Chapter 25 · 10 min read

A truth table lists every job down the side and every kind of pipeline your project runs across the top. Each cell says what the job does in that pipeline: runs, waits for a person, is allowed to fail, or is absent.

You derive it from the merged configuration by applying the earlier chapters of Part V in order. It answers "why didn't this run?" at a glance. It also finds the jobs that run where nobody meant them to, and the checks that never run where everybody assumes they do.

Wednesday, 09:15. Which jobs will the schedule get?

The nightly schedule is about to be switched on for payments-api, and the release manager asks which jobs it will run. Eleven jobs, three rule lists, a workflow, and no way to try it without running it. You need the answer on paper, before tonight. (the card)

The method#

  1. Pick the scenarios: the kinds of pipeline the project really gets (chapter 21).
  2. Apply workflow. Cross out the scenarios it refuses, and note when a scenario stops applying (chapter 22).
  3. Find each job's final rules in the Full configuration view, after extends, !reference, templates and components have done their work (chapter 14).
  4. Walk each job's rules for each scenario. The first match fills the cell (chapter 23).
  5. Mark how the job runs: manual, delayed, or allowed to fail (chapter 24).
  6. Check every column's needs. Each needed job must be in the same column, or GitLab won't create that pipeline.
  7. Check the table against real pipelines of each kind.

Here is the whole method, applied to payments-api.

Steps 1 and 2: scenarios and workflow#

ScenarioSourceWhat the rules will seeMegaCorp's workflow
feature branch pushpushCI_COMMIT_BRANCH is the feature branchallowed by rule 3, but only until a merge request is opened; after that, rule 2 refuses it
merge requestmerge_request_eventCI_MERGE_REQUEST_* is set, and CI_COMMIT_BRANCH is notallowed by rule 1
mainpushCI_COMMIT_BRANCH equals CI_DEFAULT_BRANCHallowed by rule 3
tag v1.4.0pushCI_COMMIT_TAG is set, and CI_COMMIT_BRANCH is notallowed by rule 4
nightly on mainscheduleCI_COMMIT_BRANCH is main, and the source is scheduleallowed by rule 3

All five scenarios get a pipeline, provided at least one job is added to it.

Step 3: each job's final rules#

Most of payments-api's jobs inherit their rules from the base template:

megacorp/devops/ci-templatestemplates/base.ymlmegacorp-base
.megacorp-base:
  variables:
    MC_TEAM: unknown
  artifacts:
    expire_in: 7 days
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]
    - !reference [.rules, release-tag]

Not all of them, though. Here is where each job's final list comes from:

JobIts rules come fromThe list, in order
maven-build.megacorp-base, through .maven-base and .maven-buildnever-on-schedule, mr, default-branch, release-tag
maven-test.maven-test, whose own list replaces the base'snever-on-schedule, mr, default-branch
image-build.megacorp-base, through .image-buildnever-on-schedule, mr, default-branch, release-tag
container_scanningMegaCorp's security.yml, whose list replaces the one in GitLab's templatedisabled means never; never-on-schedule; merge request event; default branch
secret_detectionGitLab's Secret-Detection templatedisabled means never; two rules that apply only when AST_ENABLE_MR_PIPELINES is "true"; $CI_COMMIT_BRANCH
sonar-scanthe sonar-scan componentSKIP_SONAR means never; merge request event; default branch
deploy-stagingjava-service.ymlnever-on-schedule, default-branch
deploy-prodjava-service.ymla tag matching the release pattern, as a manual job
publish-docspayments-api's own filedefault-branch

Two of these lists replaced a list the job would otherwise have inherited: the one on maven-test, and the one on container_scanning. Replaced lists are where most surprises in a truth table come from. For secret_detection, the rules are in a template nobody at MegaCorp wrote. The only way to see them is the Full configuration view.

Steps 4 and 5: the table#

Walk each list for each scenario, stopping at the first match:

jobfeature branch pushmerge requestmaintag v1.4.0nightly on main
maven-buildnot in the pipelinerunsrunsrunsnot in the pipeline
maven-test: [17] and [21]not in the pipelinerunsrunsnot in the pipelinenot in the pipeline
secret_detectionruns, allowed to failnot in the pipelineruns, allowed to failnot in the pipelineruns, allowed to fail
sonar-scannot in the pipelineruns, allowed to failruns, allowed to failnot in the pipelineruns, allowed to fail
image-buildnot in the pipelinerunsrunsrunsnot in the pipeline
container_scanningnot in the pipelineruns, allowed to failruns, allowed to failnot in the pipelinenot in the pipeline
deploy-stagingnot in the pipelinenot in the pipelinerunsnot in the pipelinenot in the pipeline
deploy-prodnot in the pipelinenot in the pipelinenot in the pipelinemanual: waits for someone to run itnot in the pipeline
publish-docsnot in the pipelinenot in the pipelinerunsnot in the pipelineruns
payments-api: every job, by pipeline. Derived from the rules, and checked against real pipelines on a local GitLab 19.3.·not in the pipeline✓runs!runs, allowed to fail▶manual: waits for someone to run it

Two columns take a moment's thought:

  • Merge request, secret_detection. None of its four rules matches. The two merge request rules need AST_ENABLE_MR_PIPELINES, which nobody set, and the last rule needs CI_COMMIT_BRANCH, which a merge request pipeline doesn't have.
  • Nightly, sonar-scan. Its last rule is $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH, a schedule on main matches it, and nothing in its list excludes schedules.

Step 6: needs in every column#

JobNeedsIn the same columns?
image-buildmaven-buildyes: merge request, main, tag
container_scanningimage-buildyes: merge request, main
deploy-stagingimage-buildyes: main
deploy-prodimage-buildyes: tag

The nightly column is safe only because container_scanning leaves schedules out, just as image-build does. Before MegaCorp added that rule, the nightly column held a scan that needed an image the column didn't have, and GitLab refused to create the pipeline at all (chapter 24's exercise).

Step 7: check it against GitLab#

The table above was checked cell by cell against pipelines that a local GitLab 19.3 created for the same five scenarios. You can do the same with less effort:

  • Pushes to a branch or tag can be simulated before you push. GitLab's CI Lint runs the rules as if a pipeline were being created (chapter 4).
  • Merge request and scheduled pipelines can't be simulated. Find a recent one of each kind in the pipeline list, by its label, and compare its jobs with your column.
  • If you use the API, remember that it lists trigger jobs separately from other jobs. A pipeline's jobs endpoint doesn't show deploy-staging or deploy-prod; its bridges endpoint does.

What the table says about payments-api#

Five findings, none of them visible in any single file:

  • Merge requests get no secret detection. Once a merge request is open, MegaCorp's workflow refuses branch pipelines, and GitLab's template skips merge request pipelines. A secret committed on a merge request branch is first scanned after the merge. GitLab's documented switch is AST_ENABLE_MR_PIPELINES: "true", typically set once for the whole group.
  • A feature branch with no merge request gets only secret detection. No build and no tests run until someone opens the merge request.
  • A release tag is built but not tested or scanned. It relies on the tag sitting on a commit that main's pipeline already tested.
  • The nightly pipeline doesn't build or test anything. It republishes the docs and runs two scanners.
  • Every "allowed to fail" cell is a scanner. A finding, or a crashed scanner, never turns these pipelines red; only a policy can make it (chapter 36).
What you can now do
  1. Build a truth table for a pipeline: every job against every kind of pipeline, from the rules alone.
  2. Check the table against what GitLab actually created, and find the rule that disagrees.
  3. Answer "what will run if" without pushing anything.
Spot the bugThe release candidate with no pipeline#

The payments team starts tagging release candidates. They push the tag v1.5.0-rc1 on a commit of main. No pipeline appears, and GitLab shows no error. The next day, v1.5.0 on the same commit gets a pipeline as usual.

QuestionAdd a column for v1.5.0-rc1 to the table. Why is there no pipeline?
Show the answer

Workflow rule 4 lets the tag pipeline through, but every job's rules then leave it out:

  • maven-build and image-build reach release-tag, whose pattern /^v\d+\.\d+\.\d+$/ doesn't match -rc1.
  • maven-test, container_scanning, sonar-scan, deploy-staging and publish-docs have no rule for tags at all.
  • secret_detection needs CI_COMMIT_BRANCH, which a tag pipeline doesn't have.
  • deploy-prod repeats the release pattern.

The column is empty, and GitLab doesn't create a pipeline with no jobs. After a push it says nothing. To build release candidates, widen the pattern, for example to /^v\d+\.\d+\.\d+(-rc\d+)?$/, in two places: the rules library, and deploy-prod, which copies the pattern instead of using the library. See each job's final rules and when no pipeline appears.

Variables and precedence#

Part VI · What a job is given · Chapter 26 · 13 min read

A CI/CD variable is a name and a value that GitLab hands to each job as an environment variable. Variables come from many places:

  • your YAML
  • the settings pages of the project, its groups and the instance
  • whoever started the pipeline, and earlier jobs
  • security policies, and GitLab itself

When one name is set in several places, a fixed ladder decides which value wins. Settings beat YAML, and a value given when the pipeline started beats settings. Your YAML is near the bottom. Most "my change had no effect" problems are this ladder at work.

Wednesday, 14:05. Two values, one name.

You set LOG_LEVEL: debug at the top of the file to see more output. Three jobs print debug. Two print info, as before. Nothing in those two jobs mentions LOG_LEVEL, but the template they extend does. Which value wins, and by what rule? (the card)

What a variable is#

Here is the smallest example:

variables:
  DEPLOY_ENV: staging

show-env:
  script:
    - echo "Deploying to $DEPLOY_ENV"     # prints: Deploying to staging

GitLab passes DEPLOY_ENV to the job as an environment variable: a named value that every program the script runs can read. In a script you write it as $DEPLOY_ENV. Variables can also appear in many keywords, such as image, and in rules, although rules can only see the variables that exist when the pipeline is created (chapter 21).

Variables in YAML#

A variables: block can sit at the top of a file or inside a job:

  • At the top, it sets a default for every job.
  • In a job, it sets a value for that job only, and a job's value beats a default of the same name.

Everyone who can read the repository can read these values, so YAML is for ordinary settings, never for passwords or tokens. variables: {} in a job turns every default off for that job, and inherit: variables: does the same more selectively (chapter 14).

Gotcha
A job can't read a default into a variable of the same name. MY_VAR: $MY_VAR in a job doesn't pick up the top-level MY_VAR. The job's own definition hides the default, so the value is the literal text $MY_VAR. Use a different name.

Variables in settings#

Anything secret belongs in settings, under Settings › CI/CD › Variables, on the project, on a group, or for the whole instance. Every job below that level receives the variable. Each one has a few options:

OptionWhat it does
Keythe name: letters, digits and underscores only
Valueup to 10,000 characters
TypeVariable, the usual kind, or File: the value is written to a temporary file, and the variable holds the file's path, for tools that need a file
Environment scopelimits the variable to jobs deploying to certain environments (Premium, on groups)
Protect variablethe variable exists only in pipelines on protected branches and tags
VisibilityVisible; Masked, which hides the value in job logs and has been the default since 18.3; or Masked and hidden, which also hides it in the settings page, for ever
Expand variable referencelets the value refer to other variables with $; off by default since 18.6

Here are MegaCorp's group variables. Every project under megacorp/ gets them:

settingsgroup-megacorp.ymlgroup-variables
# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
  - key: MAVEN_CLI_OPTS
    value: "--batch-mode --errors --show-version -s .m2/settings.xml"
  - key: AWS_ACCOUNT_ID
    value: "123456789012"
  - key: MC_AZURE_TENANT_ID
    value: "00000000-0000-0000-0000-000000000000"
  - key: MC_ACR_REGISTRY
    value: megacorpprod.azurecr.io
  - key: MC_ACR_PUSH_CLIENT_ID
    value: "00000000-0000-0000-0000-000000000000"
    # the gitlab-acr-push application in Entra ID, which mc_acr_push signs in as
  - key: SONAR_HOST_URL
    value: https://sonar.example.com
  - key: SONAR_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true

Changing a project variable needs the Maintainer role, a group variable the Owner role, and an instance variable an administrator. If you can't see a level's settings, you can't see its variables either. That is the first reason variables are hard to trace.

Who wins: the precedence ladder#

When the same name is set in more than one place, the job gets the value from the highest place on this ladder:

The variable precedence ladder in GitLab 19.3. When a name is set in several places, the highest rung wins. Everything you write in .gitlab-ci.yml sits below every settings page. higher wins 1 Pipeline execution policy variables 2 Scan execution policy variables 3 Pipeline variables: manual runs, schedules, triggers, the API, downstream, manual jobs 4 Project variables 5 Group variables: the closest subgroup wins 6 Instance variables 7 Dotenv variables from earlier jobs 8 Job variables in .gitlab-ci.yml 9 Top-level (default) variables in .gitlab-ci.yml 10 Deployment variables 11 Predefined variables security policies (Ultimate) set when the pipeline starts CI/CD settings pages: in no file at all written by a job's script your YAML: lowest of all that you set yourself GitLab's own
Figure The variable precedence ladder in GitLab 19.3. When a name is set in several places, the highest rung wins. Everything you write in .gitlab-ci.yml sits below every settings page.#
RungWhere the value comes fromBeats
1a pipeline execution policyeverything
2a scan execution policyeverything below
3pipeline variables: a manual run, a schedule, a trigger, the API, an upstream pipeline, or a manual job's formall settings and all YAML
4the project's CI/CD settingsgroups, instance, and all YAML
5a group's CI/CD settings; the closest subgroup winsthe instance, and all YAML
6the instance's CI/CD settingsall YAML
7a dotenv report written by an earlier jobyour YAML
8a job's own variables:, including those it gets through extendstop-level YAML
9top-level variables:GitLab's own values
10–11deployment and predefined variablesnothing

Two of MegaCorp's variables show the ladder at work:

  • MC_TEAM is set at the top of payments-api's file (rung 9), and in .megacorp-base, a job that every template extends (rung 8). The job value wins, so every job reports unknown (chapter 14).
  • MAVEN_CLI_OPTS is set in payments-api's maven-test job (rung 8), and on the megacorp group (rung 5). The group wins. The option payments-api added is never used, and nothing in any file shows why.
megacorp/payments/payments-api.gitlab-ci.ymloverride
maven-test:
  variables:
    MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"

Values given when the pipeline starts#

Rung 3 holds pipeline variables: values given to one pipeline when it starts. They come from the New pipeline form, a schedule, the API, a trigger token, an upstream trigger job, or the form on a manual job. They beat every settings page, and even GitLab's predefined variables, so they are powerful and easy to misuse.

Because of that, a project can limit who may use them, with Minimum role to use pipeline variables. On self-managed GitLab the default is Maintainer; new projects on GitLab.com default to nobody. Someone below the minimum gets Insufficient permissions to set pipeline variables. The setting also covers the variables a trigger job sends downstream, including top-level ones it inherits. So a downstream project with the restriction on can refuse your trigger with:

Failed - (downstream pipeline can not be created, Insufficient permissions to set pipeline variables)

GitLab now recommends pipeline inputs instead of pipeline variables (chapter 16), because inputs are typed and checked.

When a $ is expanded#

A value can mention another variable. What happens depends on where the value was set:

  • In YAML, references are expanded: LS_CMD: 'ls "$FLAGS"' uses the value of FLAGS. The variables: expand keyword can turn that off for one variable.
  • In settings, a value is taken literally unless its Expand variable reference box is ticked. That box has been off by default since 18.6, and a masked variable can't have it.
  • To keep a literal dollar sign in YAML, write $$.

Inside the job#

Every variable reaches the job as an environment variable. Three details trip people up:

  • before_script and script share one shell; after_script doesn't. A value set with export in the script is gone by after_script, and never reaches another job.
  • To pass a value to a later job, write it to a dotenv report: a small file of NAME=value lines that later jobs receive as variables (rung 7; chapter 31). Rules can't see such values.
  • Service containers, the helper containers a job starts beside itself, get only the variables written in YAML, not those from settings (chapter 30).

Finding where a value came from#

A job log never lists where each variable came from. Work down the ladder, from the top, and stop at the first place that sets the name:

Finding where a variable's value came from. Ask from the top; the first yes is the value your job sees.yesnoyesnoyesnoyesnoyesnoDoes a security policy set it?The policy's value wins; ask thesecurity teamWas it given when this pipelinestarted: by hand, a schedule, a triggeror the API?That value wins over everysettings pageIs it in the CI/CD settings of theproject, a group or the instance?The closest level wins: project,then the nearest groupDid an earlier job write it to a dotenvreport?The dotenv value wins over yourYAMLIs it in the job's own variables, or ina job it extends?The job's value wins over thetop-level oneOtherwise it comes from top-levelvariables, or from GitLab itself
Figure Finding where a variable's value came from. Ask from the top; the first yes is the value your job sees.#
As text
  1. Does a security policy set it? Yes: The policy's value wins; ask the security team. No: the next step.
  2. Was it given when this pipeline started: by hand, a schedule, a trigger or the API? Yes: That value wins over every settings page. No: the next step.
  3. Is it in the CI/CD settings of the project, a group or the instance? Yes: The closest level wins: project, then the nearest group. No: the next step.
  4. Did an earlier job write it to a dotenv report? Yes: The dotenv value wins over your YAML. No: the next step.
  5. Is it in the job's own variables, or in a job it extends? Yes: The job's value wins over the top-level one. No: the next step.
  6. Otherwise it comes from top-level variables, or from GitLab itself

To look at a value without leaking it, print the variable's name and length, never a secret's value. Masked values appear in logs as [MASKED]. A command such as export lists every variable the job has, and GitLab's debug tracing lists even more (chapter 27).

When you write it
Do this
Put values that are not secret in the YAML, beside the jobs that use them, and put them in quotes, for example JAVA_VERSION: "21". Put secrets in the settings, never in the YAML.
Why
Reviewers see a value in the YAML, and quotes stop YAML from reading a number or a word such as yes as something else.
Watch out
If a change to a YAML value seems to be ignored, look for a settings variable with the same name. The card the old value wins shows what happened.
What you can now do
  1. Recite the ladder: which source of a variable's value beats which, from a policy down to the predefined ones.
  2. Find where a job's value really came from, in the YAML or on a settings page.
  3. Write a variable in the right place for what it is: a plain value, a choice, or a secret.
Spot the bugThe retry option that never retries#

payments-api's tests are flaky, so the team told Maven to rerun failing tests twice, in their own file:

megacorp/payments/payments-api.gitlab-ci.ymloverride
maven-test:
  variables:
    MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"

The job log shows no reruns at all. The merged configuration shows their MAVEN_CLI_OPTS exactly as they wrote it.

QuestionWhy does Maven never see -Dsurefire.rerunFailingTestsCount=2, and what are the ways to fix it?
Show the answer

The megacorp group sets MAVEN_CLI_OPTS in its CI/CD settings (rung 5). Settings variables beat every value in YAML, including a job's own (rung 8). The merged configuration shows only YAML, so it can't show the override.

Either give the option another name that the template adds to the Maven command, or set a project variable MAVEN_CLI_OPTS (rung 4) containing the group's options plus the new one. The cleanest fix is for the group owners to stop setting a value that projects are meant to adjust. See who wins.

Secrets#

Part VI · What a job is given · Chapter 27 · 12 min read

A secret is a value that must not leak: a password, a token, a private key. There are three ways to get one into a job:

  • a CI/CD variable set in settings, which can be protected, masked or hidden
  • the secrets: keyword, which fetches the value from a secrets manager as the job starts (Premium)
  • the job logging in to a secrets manager or cloud itself, with an ID token (chapter 28)

Each has limits. Most "the secret is empty" problems come from protection rules deciding that this pipeline doesn't get the value. Most leaks come from a script printing the value in a form that masking doesn't recognise.

Thursday, 11:20. It works on main and fails everywhere else.

The deploy job signs in to the registry on main and fails with unauthorized on every feature branch and in every merge request. The password is a settings variable, marked protected, as the security team asked. The job is the same. The branches are not. (the card)

Three ways a secret reaches a job#

WayWhere the value livesHow the job gets itTier
A CI/CD variablethe settings of the project, a group or the instanceas an environment variable, or a file if its type is Fileall
The secrets: keyworda secrets manager: HashiCorp Vault, AWS, Google Cloud or Azure, or GitLab's own Secrets ManagerGitLab fetches it as the job starts, and hands it over as a file by defaultPremium
The job logs in itselfanywhere that trusts GitLab's ID tokensthe script exchanges an ID token for access, then reads what it needsall

Settings variables are the commonest, and the easiest to get wrong, so most of this chapter is about them. Never put a secret in .gitlab-ci.yml: everyone who can read the repository can read it.

Protect, mask, hide: what each one does#

A settings variable has three safety options. They protect against different things, and none of them protects against everything:

OptionWhat it doesWhat it doesn't do
Protectthe value is delivered only to pipelines running on protected branches and tagsstop code in those pipelines from using or sending the value
Maskthe exact value is replaced by [MASKED] in job logscatch the value if a program prints it changed, for example with a backslash added before a special character
Mask and hideas masked, and nobody can read the value in the settings page againanything more at run time; it can only be chosen when the variable is created
File typethe value is written to a temporary file, and the variable holds its pathstop a script that reads the file from printing it

GitLab's own documentation is blunt about masking: it "is not a guaranteed way to prevent malicious users from accessing variable values". A masked value must also be a single line with no spaces and at least 8 characters, which rules out most private keys.

Who gets a protected value#

Protection decides which pipelines receive a value at all. This is the rule behind most "works on main, empty on my branch" problems:

  • Pipelines on protected branches and tags get protected values. MegaCorp protects main and tags matching v*.
  • Pipelines on any other branch don't. The variable is simply not set.
  • Merge request pipelines get them only when all four of these are true:
    • the source and target branches are both protected
    • both branches are in the same project
    • the person running the pipeline can push or merge to the target
    • the project setting Allow merge request pipelines to access protected variables and runners is on
    A merge request from an ordinary feature branch never qualifies.
  • Pipelines from forks run in the fork and get none of the parent project's variables. A parent-project member can choose to run a fork's merge request pipeline in the parent instead. That pipeline gets the parent's variables, though not protected ones, and runs the fork's pipeline file. GitLab shows a warning that must be accepted first.

payments-api protects main and tags matching v*, and keeps one secret of its own:

megacorp/payments/payments-apiSETTINGS.ymlprotection
protected_branches: [main]
protected_tags: ["v*"]
megacorp/payments/payments-apiSETTINGS.ymlproject-variables
project_variables:
  - key: MC_NEXUS_URL
    value: https://nexus.example.com/repository/maven-public/
  - key: PAYMENTS_DB_PASSWORD
    value: "(set in the UI)"
    masked: true
    protected: true

The megacorp group adds one more secret, for every project:

settingsgroup-megacorp.ymlgroup-variables
# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
  - key: MAVEN_CLI_OPTS
    value: "--batch-mode --errors --show-version -s .m2/settings.xml"
  - key: AWS_ACCOUNT_ID
    value: "123456789012"
  - key: MC_AZURE_TENANT_ID
    value: "00000000-0000-0000-0000-000000000000"
  - key: MC_ACR_REGISTRY
    value: megacorpprod.azurecr.io
  - key: MC_ACR_PUSH_CLIENT_ID
    value: "00000000-0000-0000-0000-000000000000"
    # the gitlab-acr-push application in Entra ID, which mc_acr_push signs in as
  - key: SONAR_HOST_URL
    value: https://sonar.example.com
  - key: SONAR_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true

SONAR_TOKEN is protected. sonar-scan runs in every merge request pipeline, and feature branches aren't protected, so in those pipelines the token is empty. The scan can't sign in to the Sonar server. The job is allowed to fail, so the merge request stays green, and nobody notices that the quality gate never ran.

The secrets: keyword Premium#

With secrets:, the value never sits in GitLab's settings at all. GitLab fetches it from a secrets manager when the job starts:

deploy:
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://vault.example.com
  secrets:
    DATABASE_PASSWORD:
      vault: production/db/password@ops     # engine ops, secret production/db, field password
  script:
    - ./deploy.sh --password-file "$DATABASE_PASSWORD"

Four things to know when you read one:

  • It arrives as a file. The variable holds the path to a temporary file containing the value, which is why the script above passes it as --password-file. Write file: false under the variable name to get the value itself.
  • It needs a way to prove who is asking, usually an ID token (chapter 28). For Vault, a job's only ID token is used automatically. With several, name one with token:, or which is used can't be predicted. AWS Secrets Manager looks for a token named AWS_ID_TOKEN, or uses the runner's own AWS role (chapter 38).
  • The provider decides who may read. A Vault role, for example, lists bound claims: values the ID token must carry, such as the project's ID or ref_protected: "true". GitLab's documentation warns that a role not tied to a project or group accepts a token from any job on the instance.
  • The providers are HashiCorp Vault, AWS Secrets Manager, Google Cloud Secret Manager, Azure Key Vault, and GitLab's own Secrets Manager.

GitLab Secrets Manager Beta is GitLab's own store, under Secure › Secrets manager, and has been in public beta since 19.0. Jobs read it with secrets: … gitlab_secrets_manager:, which needs GitLab Runner 19.0 or later. A secret can be limited to a branch or an environment. Its value is masked if a job prints it, and unlike a masked variable it can contain spaces and new lines. Switching the manager off deletes every secret in it, permanently.

How secrets leak anyway#

None of the options above stops a determined script. These are the usual ways values escape:

  • Printing them changed. Base64-encoding a token, or a shell adding escape characters, produces text that masking doesn't recognise.
  • Debug tracing. Setting CI_DEBUG_TRACE: "true" writes every variable the job has into the log. Masked values still show as [MASKED], but everything else appears in full. Only Developers and above can read such logs, unless the variable was set in a runner's own configuration, where the check doesn't apply.
  • Listing the environment. export, env or printenv print every variable. Masked values show as [MASKED], but unmasked ones don't. File-type variables show only a path.
  • Artifacts and dotenv files. Anything a job writes into an artifact can be downloaded by people who can see the pipeline.
  • Changed pipeline files. A merge request can change .gitlab-ci.yml to send a variable anywhere. Protection limits which pipelines have the value, but review of pipeline changes is what actually stops this.

When a secret is empty#

Work through these questions in order. The first yes is almost always the cause:

A secret is empty, or wrong, in one pipeline but fine in another. Ask these in order.yesnoyesnoyesnoyesnoyesnoIs the variable protected, and thisbranch or tag not?Protected values reach onlyprotected branches and tagsIs this a merge request pipeline?Both branches must be protected,and a setting must allow itIs the pipeline running in a fork?Forks don't receive the parentproject's variablesDoes the variable have an environmentscope?Only jobs deploying to thatenvironment receive itDoes the job get it through secrets:?Check the ID token and theprovider's role (chapter 21)Check the name matches exactly, andthat a higher rung doesn't override it(chapter 19)
Figure A secret is empty, or wrong, in one pipeline but fine in another. Ask these in order.#
As text
  1. Is the variable protected, and this branch or tag not? Yes: Protected values reach only protected branches and tags. No: the next step.
  2. Is this a merge request pipeline? Yes: Both branches must be protected, and a setting must allow it. No: the next step.
  3. Is the pipeline running in a fork? Yes: Forks don't receive the parent project's variables. No: the next step.
  4. Does the variable have an environment scope? Yes: Only jobs deploying to that environment receive it. No: the next step.
  5. Does the job get it through secrets:? Yes: Check the ID token and the provider's role (chapter 21). No: the next step.
  6. Check the name matches exactly, and that a higher rung doesn't override it (chapter 19)
When you write it
Do this
Mask every secret, so that logs show [MASKED]. Protect it as well if only protected branches and tags need it. Prefer a short-lived ID token to a stored credential wherever the other side accepts one.
Why
A masked value never appears in a log. A protected value never reaches a branch that anyone can push to.
Watch out
A protected secret is empty on a feature branch, so a job there that needs it fails. The card fails only on branches shows the symptom.
What you can now do
  1. Put a secret in the settings, masked and protected, and predict where it will be empty.
  2. Read [MASKED] in a log and know what GitLab can and cannot mask.
  3. Choose an ID token over a stored key wherever the other side accepts one.
Spot the bugThe quality gate that never ran#

MegaCorp's platform team notices that Sonar has had no analysis of any payments-api merge request for months. Analyses of main arrive after every merge. In merge request pipelines, sonar-scan shows an orange warning, and its log ends in an authentication error.

settingsgroup-megacorp.ymlgroup-variables
# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
  - key: MAVEN_CLI_OPTS
    value: "--batch-mode --errors --show-version -s .m2/settings.xml"
  - key: AWS_ACCOUNT_ID
    value: "123456789012"
  - key: MC_AZURE_TENANT_ID
    value: "00000000-0000-0000-0000-000000000000"
  - key: MC_ACR_REGISTRY
    value: megacorpprod.azurecr.io
  - key: MC_ACR_PUSH_CLIENT_ID
    value: "00000000-0000-0000-0000-000000000000"
    # the gitlab-acr-push application in Entra ID, which mc_acr_push signs in as
  - key: SONAR_HOST_URL
    value: https://sonar.example.com
  - key: SONAR_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true
QuestionWhy does the scan fail only in merge request pipelines, and why has nobody noticed?
Show the answer

SONAR_TOKEN is a protected group variable. Protected values reach only pipelines on protected branches and tags. A merge request pipeline from a feature branch gets them only if both branches are protected, which a feature branch isn't. So the token is empty, and the scanner can't authenticate.

Nobody noticed because sonar-scan has allow_failure: true: a failure turns the job orange, not the pipeline red. The owners of the token must decide whether an unprotected token, a separate token for merge requests, or a different design is acceptable. See who gets a protected value.

Identity and tokens#

Part VI · What a job is given · Chapter 28 · 18 min read

Many jobs must prove who they are before another system lets them in: GitLab's own API and registries, AWS, or a secrets manager. A job can do that in four ways:

  • the job token, which GitLab creates for every job, for calls back to GitLab
  • an ID token, a signed note about the job that other systems can check
  • a stored token, such as a project access token, kept in a CI/CD variable
  • the runner's own identity, which every job on that runner shares

The first two are made fresh for each job and stop working when it ends. Stored tokens expire on a date nobody remembers, which is why a pipeline can break with no commit to blame.

Thursday, 15:50. The job that stopped after a year.

A job that has published release notes through the GitLab API every week since last September fails today with 401 Unauthorized. Nobody committed anything. Nobody changed a setting. The token it uses was created by someone who left in May, and its page now says Expired. Four kinds of credential can be in play in a job; which one is this, and why did it die today? (the card)

Four ways a job proves who it is#

The four identities a job can use, and what each one reaches. The job token and ID tokens are made for each job. Stored tokens and the runner's role last much longer. a job payments-api, on main CI_JOB_TOKEN made for this job, ends with it GitLab API, registries, packages, clones an ID token, from id_tokens: made for this job, short-lived AWS, Vault, other clouds they check GitLab's signature a stored token, in a variable made once, expires on a date whatever it was made for GitLab, a GitOps repository, a vendor the runner's own role set by the platform team anything that role can reach the same for every job on the runner
Figure The four identities a job can use, and what each one reaches. The job token and ID tokens are made for each job. Stored tokens and the runner's role last much longer.#
IdentityMade byWorks untilThe usual failure
The job token, CI_JOB_TOKENGitLab, for every jobthe job ends404 Not Found from a project that hasn't allowed yours
An ID tokenGitLab, when the job asks with id_tokens:the job's timeout, or 5 minutesthe other system refuses it, because its details don't match the trust rule
A stored token in a variablea person, onceits expiry date, often a year after it was madean authentication error, starting the day it expires
The runner's own rolethe platform teamsomeone changes the runnerevery project on that runner has the same access

When a call from a job is refused, first work out which of the four it used. Each fails for different reasons, and each is fixed in a different place.

The job token: calling back to GitLab#

Every job gets a variable called CI_JOB_TOKEN. GitLab creates it as the job starts and cancels it when the job finishes. It lets the job call GitLab as the person who started the pipeline, but it reaches far less than that person could:

  • the container, package and Terraform module registries
  • downloading job artifacts, and reading files from a repository
  • releases, environments and deployments
  • cloning a repository, with gitlab-ci-token as the user name
  • starting a pipeline in another project through the API

It doesn't work with GitLab's GraphQL API, and GitLab masks it in job logs. Here a job downloads a package that another project published:

fetch-test-data:
  script:
    - >
      curl --fail --header "JOB-TOKEN: $CI_JOB_TOKEN" --output data.zip
      "$CI_API_V4_URL/projects/megacorp%2Fdevops%2Fci-tools/packages/generic/test-data/1.0.0/data.zip"

Calling another project needs that project's permission. Each project keeps a job token allowlist: the groups and projects whose job tokens it accepts. At first it holds only the project itself. So for the job above to work:

  • ci-tools must list payments-api, or the megacorp group, under Settings › CI/CD › Job token permissions.
  • The person who started the pipeline must be a member of ci-tools, with a role that allows the action. Being on the allowlist grants nothing by itself.

Public and internal projects are a partial exception. Some of their features, such as artifacts and registries, answer any project's job token unless the project limits them to members.

A refusal rarely says "not allowed". GitLab answers 404 Not Found, as if the project didn't exist. A refused clone says:

remote: The project you were looking for could not be found or you don't have permission to view it.
A job's call to another GitLab project fails with 404. Ask these in order.noyesyesnoyesnoyesnoIs the call made with CI_JOB_TOKEN?The job token isn't involved; seestored tokens belowIs your project, or its group, missingfrom the target's allowlist?A Maintainer of the target adds itunder Job token permissionsIs the person who started the pipelinenot a member of the target?The job token acts as that person,so they need accessIs it a GraphQL call, or an API the jobtoken can't reach?That call needs a differentidentityCheck the job was still running; thetoken stops working when the job ends
Figure A job's call to another GitLab project fails with 404. Ask these in order.#
As text
  1. Is the call made with CI_JOB_TOKEN? No: The job token isn't involved; see stored tokens below. Yes: the next step.
  2. Is your project, or its group, missing from the target's allowlist? Yes: A Maintainer of the target adds it under Job token permissions. No: the next step.
  3. Is the person who started the pipeline not a member of the target? Yes: The job token acts as that person, so they need access. No: the next step.
  4. Is it a GraphQL call, or an API the job token can't reach? Yes: That call needs a different identity. No: the next step.
  5. Check the job was still running; the token stops working when the job ends

ID tokens: a signed note about the job#

Systems outside GitLab, such as AWS and Vault, don't accept the job token. For them, a job asks GitLab for an ID token. This is a short piece of text, called a JWT, that lists facts about the job and carries GitLab's signature. The receiver checks the signature against keys that GitLab publishes, so nobody can forge or change the facts. A job asks for one with id_tokens::

megacorp/devops/ci-templatestemplates/container.ymlimage-build
.image-build:
  extends: .megacorp-base
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
    IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
    - buildah push "$IMAGE:$CI_COMMIT_SHA"
    - !reference [.snippets, acr_push]
    - echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
  artifacts:
    reports:
      dotenv: build.env

MC_ID_TOKEN becomes a variable that holds the token. aud, short for audience, says who the token is for, and a receiver can refuse a token meant for someone else. MegaCorp's mc_aws_login function then hands the token to AWS. On main, mc_acr_push hands the same token to Azure (chapter 38).

The facts in a token are called claims. These matter most. The examples are for image-build in a payments-api pipeline on main:

ClaimExampleWhat it says
subproject_path:megacorp/payments/payments-api:ref_type:branch:ref:mainthe subject: project, kind of ref, and ref, in one string
audhttps://gitlab.example.comwho the token is for, from aud: in the YAML
project_pathmegacorp/payments/payments-apithe project running the job
ref, ref_typemain, branchthe branch or tag; in a merge request pipeline, the source branch
ref_protectedtruewhether that branch or tag is protected
pipeline_sourcepushwhat started the pipeline
environmentproductionthe environment, only when the job has one
expa timewhen the token expires

A probe job on the local GitLab 19.3 showed the same shape in four pipelines. On main, ref_protected was "true", as a string. On a feature branch it was "false", and the branch's merge request pipeline had the same sub and ref, with pipeline_source set to merge_request_event. A tag put ref_type:tag in sub.

A token expires when the job's timeout runs out, or after 5 minutes if no timeout is specified. In the probe, each token lasted one hour, the default job timeout. So exchange it early. MegaCorp logs in to AWS in before_script, before the long build starts.

How AWS checks an ID token#

Here is the whole exchange for MegaCorp's image push. GitLab signs the token, and the job sends it to AWS's token service, STS. STS makes three checks before it hands back keys that last one hour:

The identity flow. The job trades its ID token for temporary AWS keys, then pushes the image to ECR. Below: how the trust policy's subject pattern matches three of payments-api's pipelines. GitLab signs the token the job image-build AWS STS the token service Amazon ECR the image registry 1 an ID token, signed sub, aud, ref and more 2 swap it for keys role ARN + ID token 3 three checks signed by our GitLab? aud is the expected one? sub allowed by the role? 4 temporary keys valid for one hour 5 log in and push the image How the role's trust policy matches sub pattern project_path:megacorp/*:ref_type:branch:ref:* ✓ main project_path:megacorp/payments/payments-api:ref_type:branch:ref:main ✓ merge request project_path:megacorp/payments/payments-api:ref_type:branch:ref:feature/retry ✗ tag v1.4.0 project_path:megacorp/payments/payments-api:ref_type:tag:ref:v1.4.0
Figure The identity flow. The job trades its ID token for temporary AWS keys, then pushes the image to ECR. Below: how the trust policy's subject pattern matches three of payments-api's pipelines.#

The third check is where most refusals happen. A role's trust policy says which subjects may use it. MegaCorp's push role accepts any branch of any project under megacorp/:

awsiam.ymlecr-push-role
# Assumed by image-build, and by the ecr-push component, through mc_aws_login.
ecr_push_role:
  arn: arn:aws:iam::123456789012:role/gitlab-ecr-push
  trust_policy:
    Effect: Allow
    Principal:
      Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
    Action: sts:AssumeRoleWithWebIdentity
    Condition:
      StringEquals:
        gitlab.example.com:aud: https://gitlab.example.com
      StringLike:
        gitlab.example.com:sub: project_path:megacorp/*:ref_type:branch:ref:*
  permissions: push and pull images in every ECR repository in the account

Read the pattern against each pipeline's sub:

  • A pipeline on main matches.
  • A merge request pipeline matches too, because its ref is the source branch, such as feature/retry.
  • A pipeline for tag v1.4.0 doesn't match. Its subject says ref_type:tag, and the pattern demands branch.

On self-managed GitLab, AWS can check only sub and aud. On GitLab.com it can also check claims such as project_id and ref_protected.

Stored tokens, and the day they expire#

Some jobs use a token that a person made once and saved as a masked CI/CD variable. MegaCorp's deploy tool pushes to the GitOps repository with one:

megacorp/devops/ci-toolsbin/mcdeploy
deploy)
  env="${1:?usage: mc deploy <environment> <image-ref>}"
  image="${2:?usage: mc deploy <environment> <image-ref>}"
  tag="${image##*:}"
  mc_log "promoting ${APP} ${tag} to ${env} through the GitOps repository"
  git clone --depth 1 \
    "https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
  yq -i ".image.tag = \"${tag}\"" "gitops/envs/${env}/${APP}.yaml"
  git -C gitops commit -am "deploy ${APP} ${tag} to ${env}"
  if [ "$env" = production ]; then
    # production changes wait in a merge request until someone merges them
    git -C gitops push origin "HEAD:deploy/${APP}-${tag}" \
      -o merge_request.create -o merge_request.target=main \
      -o merge_request.title="Deploy ${APP} ${tag} to production"
  else
    git -C gitops push origin HEAD:main
  fi
  ;;
megacorp/platform/deployerSETTINGS.ymlproject-variables
project_variables:
  - key: GITOPS_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true
    # A project access token created in megacorp/platform/gitops-config, with the
    # Maintainer role and the write_repository scope. Nobody entered an expiry date,
    # so GitLab set one 365 days after the token was created.

GitLab has several kinds of stored token:

TokenActs asExpires
Personal access tokenone person, with that person's accesson its expiry date, 365 days after creation if none was entered
Project or group access tokena bot user that GitLab creates for the project or groupthe same as a personal access token
Deploy tokena project or group, for repositories and registriesnever, unless a date was set

Access tokens stop working at midnight UTC on their expiry date. The job that uses one runs unchanged and then fails with an authentication error. Nothing in the repository changed, so nothing in its history explains the failure. GitLab emails the Maintainers and Owners of the token's project 60, 30 and 7 days before a project access token expires. They aren't always the people who look after the pipeline that uses it.

Note
A deploy token named exactly gitlab-deploy-token reaches the project's jobs automatically, as CI_DEPLOY_USER and CI_DEPLOY_PASSWORD. No variable needs to be set.

The runner's own role#

The machine or pod that runs a job can have an identity of its own. MegaCorp's shared runner starts every job as a Kubernetes service account that AWS maps to a role:

runnersconfig.tomlrunner
[[runners]]
  name = "megacorp-eks-shared"
  # registered in GitLab as a group runner of megacorp, with the tag megacorp-shared
  url = "https://gitlab.example.com"
  executor = "kubernetes"
  environment = ["MAVEN_OPTS=-Xmx2g", "HTTPS_PROXY=http://proxy.example.com:3128"]
  pre_build_script = "echo 'runner: megacorp-eks-shared'"
  [runners.kubernetes]
    namespace = "gitlab-runners"
    image = "registry.example.com/megacorp/devops/ci-tools:3.2"
    # jobs run as this service account, which is bound to an AWS IAM role
    service_account = "gitlab-runner-jobs"
awsiam.ymlrunner-role
# The role behind the gitlab-runner-jobs service account on the megacorp-eks-shared
# runner. Every job on that runner can use it without asking. The EKS cluster is what
# AWS trusts for this role, not GitLab.
runner_role:
  arn: arn:aws:iam::123456789012:role/gitlab-runner-jobs
  permissions: pull images from every ECR repository in the account

Any job on that runner can use this role without asking. The access is the same for every project, it appears in no .gitlab-ci.yml, and the platform team controls it. When a job's AWS calls act as the wrong identity, run aws sts get-caller-identity. It prints the role the job is really using, and needs no permissions (chapter 38).

When you write it
Do this
Give each job that talks to a cloud its own id_tokens entry, with the audience the cloud role expects. Let the role's trust policy decide which branches and tags may use it.
Why
An ID token is made for one job and expires minutes later. Nothing has to be stored, and nothing can leak from a settings page.
Watch out
Do not store long-lived cloud keys as variables. They expire, or they outlive the people who made them. The card it worked for months shows what that failure looks like.
What you can now do
  1. Tell the four credentials a job can use apart: job token, ID token, stored key, and the runner's own identity.
  2. Read an authentication failure and say which credential failed and why.
  3. Replace a stored cloud key with an ID token and a trust policy.
Spot the bugDeploys that stopped on a Tuesday#

No MegaCorp service has deployed since Tuesday morning. Every deployer pipeline fails in mc deploy, at the clone of the GitOps repository, with an authentication error. Nothing was merged into deployer, ci-tools or gitops-config for a week, and the same pipelines deployed on Monday.

megacorp/platform/deployerSETTINGS.ymlproject-variables
project_variables:
  - key: GITOPS_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true
    # A project access token created in megacorp/platform/gitops-config, with the
    # Maintainer role and the write_repository scope. Nobody entered an expiry date,
    # so GitLab set one 365 days after the token was created.
QuestionWhat most likely changed, why does the history show nothing, and what would stop it happening again?
Show the answer

GITOPS_TOKEN is a project access token. Access tokens stop working at midnight UTC on their expiry date, which is 365 days after creation unless someone chose another. The clone sends the expired token, and GitLab refuses it. No file changed, so no commit explains the failure.

Create a new token and update the variable. To stop it recurring, track the expiry date. Also make sure the people GitLab emails, the Maintainers and Owners of gitops-config, know that the deploys depend on this token. See stored tokens.

Runners and executors#

Part VII · Execution and data · Chapter 29 · 11 min read

GitLab only schedules jobs; a runner runs them. Which runner takes a job depends on three things:

  • which runners the project may use
  • whether a runner has every tag the job asks for
  • whether a protected runner accepts this branch

Once a runner has the job, the runner's own configuration shapes what happens, and your project can't see it. A job stuck in pending is almost always a problem with the first part. A job that behaves differently from run to run is often the second.

Friday, 09:40. Pending, for ever.

load-test has sat in pending for forty minutes. Its page says it is stuck because no runner matches all of its tags: linux and big-memory. Yesterday it ran in three minutes. Nobody touched the job. Somewhere, a runner went away. (the card)

Which runners a project can use#

Every runner belongs to one of three scopes:

ScopeAvailable toWhich waiting job it takes next
Instance runnerevery project on the instance, unless a project or group turns instance runners offa fair-usage queue: jobs from projects with the fewest jobs already running come first
Group runnerevery project and subgroup in its groupthe job that has waited longest
Project runneronly the projects it has been turned on for; a fork doesn't get itthe job that has waited longest

To see the runners your project can use, open Settings › CI/CD and expand Runners. Each runner shows its tags and its status. A runner is online if it has contacted GitLab in the last two hours, offline after that, and stale after seven days.

MegaCorp has one shared runner, megacorp-eks-shared. It is a group runner of megacorp with the tag megacorp-shared, and every job asks for that tag through default: in the base template (chapter 14).

How a job finds a runner#

When a job's turn comes, it waits in pending until a runner that fits asks for work. Work through these checks in order:

Why is my job stuck in pending? Ask these in order.noyesyesnoyesnoyesnoyesnoIs any runner available to the projectonline?Nothing can take the job; checkthe runners' statusDoes the job list tags that no onlinerunner has all of?A runner needs every tag the joblistsDoes the job list no tags, and norunner takes untagged jobs?Add a tag, or tick Run untaggedjobs on a runnerAre the only runners that fitprotected, and this branch isn't?Protected runners take jobs onlyfrom protected branches and tagsAre the runners that fit paused, oralready running as many jobs as theyallow?The job waits its turn; it isn'tstuckA runner takes the job, and it moves torunning
Figure Why is my job stuck in pending? Ask these in order.#
As text
  1. Is any runner available to the project online? No: Nothing can take the job; check the runners' status. Yes: the next step.
  2. Does the job list tags that no online runner has all of? Yes: A runner needs every tag the job lists. No: the next step.
  3. Does the job list no tags, and no runner takes untagged jobs? Yes: Add a tag, or tick Run untagged jobs on a runner. No: the next step.
  4. Are the only runners that fit protected, and this branch isn't? Yes: Protected runners take jobs only from protected branches and tags. No: the next step.
  5. Are the runners that fit paused, or already running as many jobs as they allow? Yes: The job waits its turn; it isn't stuck. No: the next step.
  6. A runner takes the job, and it moves to running

The job's page says which problem GitLab suspects. These are its messages in GitLab 19.3:

The job page says which. "This job is stuck because of one of the following problems. There are no active runners online, no runners for the protected branch, or no runners that match all of the job's tags" means the job has tags, and the page lists them after the message. "This job is stuck because the project doesn't have any runners online assigned to it" means the project's runners exist, but none is online. "This job is stuck because you don't have any active runners that can run this job" means no runner available to the project can take it.

Note
A protected runner can also take jobs from merge request pipelines, under the same four conditions as protected variables (chapter 27).

What a runner does with a job#

A runner takes every job through the same steps. Some come from your YAML, some from the runner's configuration, and the runner does the rest by itself:

The steps a runner takes for every job, from top to bottom. The steps marked config.toml come from the runner's configuration, which your project can't see. The job timeout covers every step. your YAML the runner's config.toml the runner itself pre_get_sources_script get the sources: clone or fetch the repository restore the cache, then download artifacts from earlier jobs pre_build_script before_script script post_build_script after_script, in a new shell: exported variables are gone save the cache, then upload artifacts one shell RUNNER_SCRIPT_TIMEOUT can stop it early RUNNER_AFTER_SCRIPT_TIMEOUT 5 minutes by default the job timeout covers it all
Figure The steps a runner takes for every job, from top to bottom. The steps marked config.toml come from the runner's configuration, which your project can't see. The job timeout covers every step.#

Two things surprise people:

  • after_script runs in a new shell. Variables exported in script are gone. It gets 5 minutes, unless RUNNER_AFTER_SCRIPT_TIMEOUT gives it another limit.
  • The runner can add commands of its own. pre_build_script runs just before your before_script, in the same shell, and nothing in your project shows it.

What the runner's configuration decides#

A runner's settings live in a file called config.toml, on the runner's machine. This is MegaCorp's:

runnersconfig.tomlrunner
[[runners]]
  name = "megacorp-eks-shared"
  # registered in GitLab as a group runner of megacorp, with the tag megacorp-shared
  url = "https://gitlab.example.com"
  executor = "kubernetes"
  environment = ["MAVEN_OPTS=-Xmx2g", "HTTPS_PROXY=http://proxy.example.com:3128"]
  pre_build_script = "echo 'runner: megacorp-eks-shared'"
  [runners.kubernetes]
    namespace = "gitlab-runners"
    image = "registry.example.com/megacorp/devops/ci-tools:3.2"
    # jobs run as this service account, which is bound to an AWS IAM role
    service_account = "gitlab-runner-jobs"

These settings change jobs most often:

SettingWhat it does to your job
executorwhere the job runs: a Kubernetes pod here; elsewhere a container, or the machine itself
environmentadds or overwrites environment variables in every job; here, a Maven memory setting and a proxy
pre_build_scriptruns before your before_script, in the same shell
imagethe image for jobs that don't set one
output_limitthe largest log the runner sends, 4096 KB by default; the rest is cut off
concurrent and limithow many jobs run at once; the others wait in pending
allowed_images, privileged, pull_policywhich images jobs may use, whether containers get extra powers, and when images are downloaded again
CPU and memory settingshow much the job's container gets, on Kubernetes

Executors#

The executor decides where a job's commands run, which changes what a job can rely on:

ExecutorEach job runsA clean start for every job?
Kubernetesin a new pod in a cluster: one container for the job, and one for each serviceyes
Dockerin a new container, from the job's imageyes
Docker Autoscalerlike Docker, on machines created on demandyes
Instanceon a whole machine created on demand, with no containerit depends on the setup
Shelldirectly on the runner's machineno: files and tools from earlier jobs may still be there

The Shell, SSH, VirtualBox, Parallels and Custom executors are in maintenance mode: they get security fixes, but no new features. The shell and instance executors don't support image: or services:. A job that works on one runner but not on another often landed on runners with different executors.

Time limits#

Three limits can stop a job:

  • The job timeout. This is the job's own timeout:, or the project's setting if the job has none. A runner's maximum job timeout cuts it down when the runner's is shorter.
  • RUNNER_SCRIPT_TIMEOUT, a job variable that stops script early, so artifacts can still upload before the job timeout.
  • RUNNER_AFTER_SCRIPT_TIMEOUT, the same for after_script. Its default is 5 minutes.

So a job with timeout: 2h, on a runner whose maximum is 30 minutes, stops after 30 minutes.

When you write it
Do this
Add tags to a job only when it needs a particular kind of runner, and use only the tags your platform team publishes.
Why
A job with no tags can run on any shared runner. Every tag you add narrows the runners that can take it.
Watch out
A tag that no online runner has leaves the job pending forever, and GitLab marks it as stuck. The card no runner matches its tags shows the symptom.
What you can now do
  1. Say which runners can take a job, from the job's tags and the runners' scopes.
  2. Read the first lines of a log and know which runner and executor ran the job.
  3. Diagnose a pending job in one look: tags, protection, or a queue.
Spot the bugThe job that waits for ever#

A payments-api developer copies a job from a blog post into the project's .gitlab-ci.yml:

integration-test:
  stage: test
  tags: [docker]
  image: maven:3.9-eclipse-temurin-21
  script:
    - mvn verify -Pintegration

Every other job runs. integration-test stays in pending, and its page says it is stuck, listing the tag docker.

QuestionWhy does no runner take it, and what is the smallest fix?
Show the answer

A job's own tags replace the default: tags completely. So this job asks only for docker, and no longer for megacorp-shared. MegaCorp's only runner has the tag megacorp-shared, and a runner must have every tag a job lists. No runner qualifies, so the job waits for ever.

Delete the tags line, so that the job inherits megacorp-shared again. It still runs in a container, because the runner uses the Kubernetes executor. See how a job finds a runner.

Images, services and Docker#

Part VII · Execution and data · Chapter 30 · 12 min read

Most jobs run inside a container. The job's image: decides which tools its script can use. services: start extra containers beside it, such as a database for tests. Building container images inside a job is a problem of its own, because the classic way, Docker-in-Docker, needs a privileged runner.

The usual surprises are an image whose entrypoint swallows the script, a service that never gets its password, and a registry login that happens in the wrong place.

Friday, 13:30. The script that never ran.

You change a job's image to amazon/aws-cli:2.27.0. The job now fails before your first command, with the CLI's usage text in the log. Your script is still in the file. The image ran something else instead. (the card)

The job's image#

image: names the container image a job runs in. It can be a name, which means the latest tag, a name and a tag, or a name and a digest:

unit-test:
  image: node:22          # a name and a tag; with no registry named, from Docker Hub
  script:
    - npm test

MegaCorp's base template sets default: image to its toolbox image, ci-tools (chapter 19). Four things to know about any image:

  • It needs a shell. The image must contain sh or bash, and grep, because the runner sends the script to the container's shell.
  • Its entrypoint must not get in the way. The Docker executor starts the container with the image's own entrypoint and hands it a shell. An image whose entrypoint runs a program, such as amazon/aws-cli, never reaches your script, and entrypoint: [""] removes it. The Kubernetes executor ignores entrypoints unless the runner turns on FF_KUBERNETES_HONOR_ENTRYPOINT. So the same image can work on one runner and fail on another.
  • Scripts run in /builds/ followed by the project's path, not in the image's own working directory.
  • The runner decides when to download it again. Its pull policy is always, if-not-present or never. A job can ask for one with image: pull_policy:, if the runner allows it.
aws-version:
  image:
    name: amazon/aws-cli
    entrypoint: [""]      # needed on the Docker executor
  script:
    - aws --version

Images from private registries#

The runner downloads the image before the job starts, so the job's script can't log in for it. The runner looks for registry credentials in this order:

  1. a config.json file in /root/.docker on the runner
  2. a DOCKER_AUTH_CONFIG CI/CD variable
  3. a DOCKER_AUTH_CONFIG set in the runner's config.toml
  4. a config.json in the home directory of the user that runs the runner

Images in the same GitLab's container registry need none of this: the runner uses the job token (chapter 28). The person who started the pipeline needs at least the Reporter role in the image's project. That project must also let your project's job token in, which is off by default.

Every pull from Docker Hub counts against Docker Hub's rate limits. GitLab's dependency proxy is a cache in front of it. Put a prefix on the image name, and the runner signs in to the proxy by itself:

unit-test:
  image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/node:22

Services: containers beside the job#

services: starts more containers for the length of the job, most often a database for tests. They start before before_script, and stop when the job ends, even if it fails:

integration-test:
  image: maven:3.9-eclipse-temurin-21
  services:
    - name: postgres:16
      alias: db
  variables:
    POSTGRES_DB: payments_test
    POSTGRES_PASSWORD: test-only-password   # the database lives only for this job
  script:
    - mvn verify -Ddb.url=jdbc:postgresql://db:5432/payments_test
What runs beside a job. The job's container reaches each service by its hostname, and every container shares /builds. Variables from settings reach only the job's container. ONE JOB: one Kubernetes pod, or one Docker network the job's container started from the job's image runs before_script, script and after_script gets every variable, from YAML and settings reaches each service by its hostname the runner's helper clones the code, handles cache and artifacts service: postgres alias db, listening on port 5432 gets YAML variables, not settings db:5432 service: docker:dind a Docker daemon; needs a privileged runner alias docker, port 2376 with TLS docker:2376 variables from settings reach the job's container only, never a service /builds: the project's files, shared by every container
Figure What runs beside a job. The job's container reaches each service by its hostname, and every container shares /builds. Variables from settings reach only the job's container.#

What trips people up:

  • A service is another machine on the network. Connect to its hostname, such as db, not to a local socket. On Kubernetes, the services share the job's network, so localhost works too, but two services can't use the same port.
  • A service adds no programs to the job. Listing node:22 as a service doesn't give your script a node command.
  • The hostname comes from the image name, unless you set an alias. postgres:16 is reachable as postgres.
  • Services get only YAML variables. GitLab doesn't trust service containers by default, so variables from settings don't reach them. To pass one, re-assign it in the YAML under a different name.
  • A job's services replace default: services, like every list (chapter 14).
  • A service that fails to start leaves a line in the job log: *** WARNING: Service XYZ probably didn't start properly. Setting CI_DEBUG_SERVICES: "true" adds the services' own logs, but can reveal masked values.

Building images inside a job#

A job that runs docker build needs a Docker daemon, and the job's container doesn't have one. There are four ways round that, and the runner decides which of them are possible:

MethodHow it worksPrivileged runner?The catch
Docker-in-Dockera docker:dind service runs a daemon for the jobyesprivileged containers can break out to the host; no layer cache between jobs
Socket bindingthe runner mounts the host's Docker socket into the jobnothe job controls the host's daemon, and can remove other jobs' containers
Buildahbuilds images with no daemon at allnoa different command line from docker
Rootless BuildKitDocker's own build engine, run without a daemonnothe runner must still allow the system calls it uses
How can this job build a container image? It depends on the runner.yesnoyesnoyesnoDoes the runner run privilegedcontainers?Docker-in-Docker works: adocker:dind service, with TLSDoes the runner mount the host's Dockersocket?docker build works, but the jobcontrols the host's daemonDoes the runner allow the system callsthat rootless builds need?Rootless BuildKit, or BuildahRootless Buildah, which GitLab suggestswhen the runner can't be changed
Figure How can this job build a container image? It depends on the runner.#
As text
  1. Does the runner run privileged containers? Yes: Docker-in-Docker works: a docker:dind service, with TLS. No: the next step.
  2. Does the runner mount the host's Docker socket? Yes: docker build works, but the job controls the host's daemon. No: the next step.
  3. Does the runner allow the system calls that rootless builds need? Yes: Rootless BuildKit, or Buildah. No: the next step.
  4. Rootless Buildah, which GitLab suggests when the runner can't be changed

kaniko, once the usual answer, is no longer maintained, and GitLab has marked its page as removed. MegaCorp builds with Buildah, so no job needs a Docker daemon:

megacorp/devops/ci-templatestemplates/container.ymlimage-build
.image-build:
  extends: .megacorp-base
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
    IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
    - buildah push "$IMAGE:$CI_COMMIT_SHA"
    - !reference [.snippets, acr_push]
    - echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
  artifacts:
    reports:
      dotenv: build.env

The job runs in a Buildah image rather than the toolbox. It logs in to ECR with the job's ID token (chapter 28), builds and pushes. On main, it pushes the same image to Azure's registry too. Then it writes the image's name to a dotenv report for later jobs (chapter 31).

Docker-in-Docker, set up correctly#

GitLab recommends Docker-in-Docker with TLS. The runner must be privileged, and must share a certificates directory with the service. The job then looks like this:

build-image:
  image: docker:24.0.5-cli
  services:
    - docker:24.0.5-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .

On the Kubernetes executor, also set DOCKER_HOST: tcp://docker:2376, DOCKER_TLS_VERIFY: 1 and DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client". The image's entrypoint would normally set them, but that executor doesn't run it. Without TLS, the port is 2375 and DOCKER_TLS_CERTDIR is empty.

Pin the image versions, because docker:latest can change under you. Log in to registries in before_script: the daemon in the service is new for every job, and doesn't know the runner's credentials.

When you write it
Do this
Pin every image to a version tag, such as node:24.21.0-alpine, and never to latest.
Why
A pinned image makes the job behave the same next month. The card tools changed overnight shows what a moving tag does.
Watch out
When you use a tool's own image, check its entrypoint, the command the image runs by default. If the entrypoint is the tool itself, your script never runs, and you need entrypoint: [""]. The card the script never runs shows the symptom.
What you can now do
  1. Choose a job's image and pin it, and empty an entrypoint that swallows your script.
  2. Run a service beside a job and reach it by its alias.
  3. Build an image inside a job, and know what the runner must allow for that.
Spot the bugThe test database that never started#

payments-api adds integration tests with a PostgreSQL service. Following the advice in chapter 27, the team keeps the database password out of the YAML. They save it in the project's settings as a masked variable named POSTGRES_PASSWORD:

integration-test:
  image: maven:3.9-eclipse-temurin-21
  services:
    - name: postgres:16
      alias: db
  script:
    - mvn verify -Ddb.url=jdbc:postgresql://db:5432/postgres

The job log warns that the service probably didn't start properly, and the tests can't connect to db. Someone adds POSTGRES_PASSWORD: $POSTGRES_PASSWORD to the job's variables, and nothing changes.

QuestionWhy does the database get no password, and why didn't the added line help?
Show the answer

Variables from settings don't reach service containers, which GitLab doesn't trust by default. The postgres image needs POSTGRES_PASSWORD to start, so it doesn't.

A re-assignment only works under a different name. POSTGRES_PASSWORD: $POSTGRES_PASSWORD uses the same name, so it isn't expanded. This is the trap from chapter 26. Rename the setting, for example to TEST_DB_PASSWORD, and write POSTGRES_PASSWORD: $TEST_DB_PASSWORD in the YAML. Or, because this database lives only for one job, a fixed test-only password in the YAML protects nothing worth hiding. See services.

Artifacts, cache and dotenv#

Part VII · Execution and data · Chapter 31 · 13 min read

Every job starts in a fresh place, so anything a job makes is lost unless it is saved. GitLab has two ways to keep files, and they are easy to confuse:

  • Artifacts are a job's results, such as a built .jar. GitLab stores them and hands them to later jobs in the same pipeline. They are guaranteed.
  • A cache is a speed-up, such as downloaded libraries. The runner may reuse it in later jobs and pipelines, but it can be missing at any time.

A third tool, the dotenv report, passes small values, such as an image name, from one job to later ones as variables.

Monday, 10:10. The file that was there yesterday.

The package job reads version.txt, which the build job wrote. On Monday it finds last Thursday's file, with last Thursday's version. Both jobs passed. The file is in the cache, keyed on the branch, and the build job had nothing to write today. Which of the two ways of passing files should this have been? (the card)

Two ways to keep files#

Artifacts versus cache. Artifacts carry results forward inside one pipeline, and always arrive. A cache carries speed-ups to later runs, and may miss. ARTIFACTS: results that later jobs need maven-build pipeline 1 uploads stored in GitLab target/*.jar downloads maven-test pipeline 1, a later stage ✓ Guaranteed. Same pipeline: from earlier stages, or from the jobs named in needs or dependencies. Deleted after expire_in, except the latest successful pipeline's. Downloadable from the job page. CACHE: files that later runs can reuse maven-build pipeline 1 saves cache.zip, on the runner or in S3 key: made from pom.xml if found maven-build a later pipeline ✗ Not guaranteed. Another runner, a changed pom.xml or a full disk means a miss, and a slower job. Protected and unprotected branches get separate caches. Caches are restored before artifacts. Artifacts pass results forward. A cache only saves time.
Figure Artifacts versus cache. Artifacts carry results forward inside one pipeline, and always arrive. A cache carries speed-ups to later runs, and may miss.#

The rule that follows: never rely on the cache for something a job must have. If a later job needs a file, make it an artifact. If losing the file only makes a job slower, cache it.

Artifacts#

MegaCorp's Maven templates keep both kinds of artifact. maven-build keeps the built jar, and maven-test keeps a test report:

megacorp/devops/ci-templatestemplates/java-maven.ymlmaven-build
.maven-build:
  extends: .maven-base
  stage: build
  script:
    - mvn $MAVEN_CLI_OPTS -DskipTests package
  artifacts:
    paths: [target/*.jar]
megacorp/devops/ci-templatestemplates/java-maven.ymlmaven-test
.maven-test:
  extends: .maven-base
  stage: test
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JDK}
  parallel:
    matrix:
      - JDK: ["17", "21"]
  script:
    - mvn $MAVEN_CLI_OPTS verify
  artifacts:
    when: always
    reports:
      junit: target/surefire-reports/TEST-*.xml
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]
KeywordWhat it decidesDefault
artifacts:pathswhich files to keep, relative to the project directorynothing
artifacts:whenkeep them when the job succeeds, fails, or alwayson_success
artifacts:expire_inhow long GitLab keeps themthe instance's setting, but the latest successful pipeline's are kept anyway
artifacts:reportsfiles that GitLab reads and shows, such as test resultsuploaded even when the job fails
artifacts:accesswho can download them in the UI and APIall
dependencieswhich earlier jobs' artifacts a job downloads; [] means noneevery job in earlier stages

Three behaviours cause most artifact surprises:

  • A failed job keeps nothing, unless artifacts:when says otherwise. That is why maven-test has when: always: the output of a failed test run is the output you want. Reports are uploaded even without it.
  • needs and dependencies narrow the download. A job with either gets artifacts only from the jobs it names (chapter 24).
  • Artifacts expire. Every job that extends MegaCorp's .megacorp-base keeps them for 7 days. A job that needs expired artifacts fails before its script starts.

That failure, and the log line of a job that saved nothing, look like this:

This job could not start because it could not retrieve the needed artifacts.
No files to upload

The first means a missing dependency, expired artifacts or missing permissions. The second, in the earlier job's log, means its path was wrong, or the file was never made.

Cache#

MegaCorp's Maven jobs cache the local Maven repository, so that each job doesn't download every library again:

megacorp/devops/ci-templatestemplates/java-maven.ymlmaven-base
.maven-base:
  extends: .megacorp-base
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
  variables:
    JAVA_VERSION: "21"
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  cache:
    key:
      files: [pom.xml]
    paths: [.m2/repository]
  before_script:
    # a job's own before_script replaces default:before_script, so re-add the functions
    - !reference [.snippets, functions]
    - !reference [.snippets, maven_settings]

The key decides which saved cache a job gets. Jobs with the same key share one cache, in this pipeline and in later ones. Here the key is made from the contents of pom.xml, so a change to the dependencies starts a new cache.

KeywordWhat it decidesDefault
cache:keywhich cache to use; key: files: makes one from up to two files' contentsdefault
cache:pathswhich files to save and restorenone, so nothing is cached
cache:policypull only restores, push only saves, pull-push does bothpull-push
cache:whensave on success, on failure, or alwayson_success
cache:fallback_keysother keys to try when the key has no cache yetnone
cache:unprotectshare caches between protected and unprotected branchesfalse

MegaCorp's Node templates use the policy well. The base sets pull, so jobs only read the cache, and the build job alone saves it:

megacorp/devops/ci-templatestemplates/node.ymlnode-base
.node-base:
  extends: .megacorp-base
  image: registry.example.com/megacorp/devops/ci-tools/node:22
  cache:
    key:
      files: [package-lock.json]
    paths: [.npm/]
    policy: pull
  before_script:
    - npm ci --cache .npm --prefer-offline
megacorp/devops/ci-templatestemplates/node.ymlnode-build
.node-build:
  extends: .node-base
  stage: build
  cache:
    policy: pull-push
  script:
    - npm run build
  artifacts:
    paths: [dist/]

A cache that never seems to hit is the most common complaint. Check these in order:

The cache never seems to be used. Check these in order.yesnoyesnoyesnoyesnoyesnoDo the jobs run on different runners,with no shared cache storage?Each runner keeps its own cache;shared storage, such as S3, fixesitIs this branch protected when the onethat saved the cache wasn't, or theother way round?Protected and unprotected branchesget separate caches, by designDoes the key change more often than youthink?A key made from a file changeswhenever that file doesDo two jobs use the same key fordifferent paths?Each overwrites the other's cache;give them different keysDid the job that saves the cache fail?Caches are saved only on success,by defaultRead the cache lines in the job log;they name the key it tried
Figure The cache never seems to be used. Check these in order.#
As text
  1. Do the jobs run on different runners, with no shared cache storage? Yes: Each runner keeps its own cache; shared storage, such as S3, fixes it. No: the next step.
  2. Is this branch protected when the one that saved the cache wasn't, or the other way round? Yes: Protected and unprotected branches get separate caches, by design. No: the next step.
  3. Does the key change more often than you think? Yes: A key made from a file changes whenever that file does. No: the next step.
  4. Do two jobs use the same key for different paths? Yes: Each overwrites the other's cache; give them different keys. No: the next step.
  5. Did the job that saves the cache fail? Yes: Caches are saved only on success, by default. No: the next step.
  6. Read the cache lines in the job log; they name the key it tried
Note
On the Kubernetes executor, every job runs in a new pod. A cache survives between jobs only if the runner keeps it somewhere lasting, such as S3, or a volume that outlives the pod. MegaCorp's runner excerpt doesn't show its cache settings, so ask the platform team.

Why one branch can have two caches#

GitLab adds a suffix to every cache key. Pipelines for protected branches and tags get -protected. So, since 18.4.5, does any pipeline started by someone with the Maintainer or Owner role. Every other pipeline gets -non_protected.

So when a Developer and a Maintainer both push to the same feature branch, their pipelines use two different caches, and each misses the other's files. The project setting Use separate caches for protected branches, under Settings › CI/CD › General pipelines, turns the separation off. GitLab advises that only where everyone with the Developer role is highly trusted.

Also: caches are restored before artifacts. If a job caches and keeps the same path, the artifact overwrites what the cache restored.

dotenv: passing values to later jobs#

Variables set in a script vanish when the job ends. To hand a value to later jobs, write it to a file of NAME=value lines, and declare the file as a dotenv report. MegaCorp's image build passes on the image it pushed:

megacorp/devops/ci-templatestemplates/container.ymlimage-build
.image-build:
  extends: .megacorp-base
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
    IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
    - buildah push "$IMAGE:$CI_COMMIT_SHA"
    - !reference [.snippets, acr_push]
    - echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
  artifacts:
    reports:
      dotenv: build.env

Later jobs receive IMAGE_REF as an ordinary variable. The rules:

  • Who gets it: jobs in later stages, or only the jobs that name this one in needs or dependencies.
  • Not in rules. Rules are decided when the pipeline is created, before any job has written anything (chapter 21).
  • Its rung on the ladder: a dotenv value beats the job's YAML variables, but not settings or pipeline variables (chapter 26).
  • A strict format: UTF-8, no blank lines, no comments, no quotes, and no multi-line values, in at most 5 KB. On self-managed GitLab, at most 20 of these variables are passed on by default.
  • Not for secrets: anyone who can see the pipeline can read the file.
When you write it
Do this
Use artifacts for everything a later job needs, and use the cache only to make jobs faster. Key a dependency cache on the lockfile: key: files: [package-lock.json].
Why
An artifact is guaranteed to arrive. A cache can miss at any time. A key on the lockfile changes whenever the dependencies change, so nobody restores an old cache.
Watch out
One fixed key, shared by every branch and every job, hands jobs files that are old or half-written. The card a job restores stale files shows what that looks like.
What you can now do
  1. Choose artifacts for what a later job needs and the cache for speed, and never confuse the two.
  2. Key a cache so that it can't serve stale files, and read a log to see whether it hit.
  3. Pass a variable from one job to the next with a dotenv report.
Spot the bugThe retry that couldn't start#

The v1.5.0 release pipeline failed at image-build, because of the trust policy in chapter 38. Eight days later the role is fixed, and someone retries image-build. It fails at once, before its script runs:

This job could not start because it could not retrieve the needed artifacts.
megacorp/devops/ci-templatestemplates/base.ymlmegacorp-base
.megacorp-base:
  variables:
    MC_TEAM: unknown
  artifacts:
    expire_in: 7 days
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]
    - !reference [.rules, release-tag]
QuestionWhat is missing, why wasn't it kept, and what should they do instead?
Show the answer

image-build needs maven-build, so it must download the jar that job saved. Every MegaCorp job extends .megacorp-base, whose artifacts expire after 7 days, and eight days have passed. GitLab keeps the artifacts of the latest successful pipeline on each ref no matter what, but this pipeline failed.

Retry maven-build first, so that a new jar exists, and then image-build. Or start a new pipeline for the tag. See artifacts.

Git in CI#

Part VII · Execution and data · Chapter 32 · 8 min read

Before a job's script runs, the runner fetches the code. What it fetches is not the full repository you have on your laptop:

  • only the pipeline's own branch or tag, and no others
  • only the newest commits, 20 by default in new projects
  • the pipeline's commit, checked out directly, so there is no current branch

Most scripts never notice. Scripts that compare with main, count commits or read tags do notice, and they often fail only on big changes.

Monday, 15:20. History missing.

A job runs git describe --tags to name the build. In CI it prints fatal: No names found. On your laptop it prints the version. The job's clone has the commit, but not the tag before it, nor the fifty commits in between. How much of the repository does a job actually get? (the card)

What a job's copy contains#

GitLab tells the runner exactly which refs to fetch, and they depend on the kind of pipeline:

PipelineWhat is fetchedSo the job has
Branch pipelinethe pipeline's commit, and its own branchorigin/<branch>, but no other branch; on a feature branch, no origin/main
Tag pipelinethe pipeline's commit, and that tagthat one tag, so git describe can't see older ones
Merge request pipelinethe pipeline's commit only, as refs/pipelines/<id>no branch names at all

The runner then checks out the pipeline's commit, not a branch, so Git has no current branch. Git calls this a detached HEAD. Take names from variables instead:

  • CI_COMMIT_REF_NAME is the branch or tag being built.
  • CI_COMMIT_BRANCH is set only in branch pipelines.
  • CI_MERGE_REQUEST_TARGET_BRANCH_NAME is the target, in merge request pipelines.

Shallow clones#

A shallow clone fetches only the newest commits. New projects fetch 20, set under Settings › CI/CD › General pipelines › Git shallow clone, and the GIT_DEPTH variable overrides that. MegaCorp's base template sets GIT_DEPTH: "20" for every job that includes it.

A job's clone with GIT_DEPTH 20. The newest 20 commits are fetched and older ones aren't. So a script that needs an older commit works on small changes and fails on big ones. A job's clone with GIT_DEPTH 20: oldest on the left, newest on the right fetched: the newest 20 commits never fetched a big merge request's base not in the clone a small merge request's base in the clone HEAD git diff "$base" HEAD works for the small merge request, and fails for the big one.
Figure A job's clone with GIT_DEPTH 20. The newest 20 commits are fetched and older ones aren't. So a script that needs an older commit works on small changes and fails on big ones.#

Three kinds of script need more history than that:

  • Comparing with an older commit, such as git diff against a merge request's base, fails when that commit is older than the clone.
  • Reading tags, with git describe or a version made from tags, needs the tags and the commits between them.
  • Running old commits. GitLab's documentation warns that a retried or queued job may find its commit outside the clone, and the log then says unresolved reference. A depth of 1 makes this likely.

Fix it in the one job that needs history. Raise GIT_DEPTH for that job, or fetch what the script needs, for example with git fetch --unshallow.

A script fails on Git history in CI but works on a laptop. Ask these in order.yesnoyesnoyesnoyesnoDoes the script use another branch,such as main?That branch isn't in the job'scopy; fetch it firstDoes it use an older commit, such as amerge request's base?The commit may be older thanGIT_DEPTH; fetch more historyDoes it read tags, or run git describe?A tag pipeline fetches one tag,and other pipelines noneDoes it ask Git for the current branch?There is none; useCI_COMMIT_REF_NAMECheck GIT_STRATEGY; with none or empty,the job gets no fresh code at all
Figure A script fails on Git history in CI but works on a laptop. Ask these in order.#
As text
  1. Does the script use another branch, such as main? Yes: That branch isn't in the job's copy; fetch it first. No: the next step.
  2. Does it use an older commit, such as a merge request's base? Yes: The commit may be older than GIT_DEPTH; fetch more history. No: the next step.
  3. Does it read tags, or run git describe? Yes: A tag pipeline fetches one tag, and other pipelines none. No: the next step.
  4. Does it ask Git for the current branch? Yes: There is none; use CI_COMMIT_REF_NAME. No: the next step.
  5. Check GIT_STRATEGY; with none or empty, the job gets no fresh code at all

Variables that shape the clone#

VariableWhat it doesDefault
GIT_STRATEGYclone starts fresh; fetch reuses the last copy where the executor keeps one; none and empty skip Git, for jobs that only use artifactsthe project's setting
GIT_DEPTHhow many commits to fetchthe project's setting: 20 in new projects
GIT_CHECKOUT"false" fetches without checking out the pipeline's commit"true"
GIT_CLEAN_FLAGShow git clean tidies a reused copy; none skips it-ffdx
GIT_FETCH_EXTRA_FLAGSflags added to git fetch--prune --quiet
GIT_SUBMODULE_STRATEGYnormal fetches submodules, recursive their submodules toosubmodules aren't fetched

On the Kubernetes executor each job gets a new pod, so fetch usually has no earlier copy to reuse, and clones anyway.

Submodules#

A submodule is another repository placed inside yours, pinned to one commit. The runner fetches submodules only when GIT_SUBMODULE_STRATEGY is normal or recursive. For them to work in CI:

  • In .gitmodules, point at submodules on the same GitLab with a relative URL, such as ../../devops/ci-tools.git, or with an HTTPS URL. A relative URL can resolve wrongly in forks.
  • For SSH URLs, set GIT_SUBMODULE_FORCE_HTTPS, so the runner fetches them over HTTPS.
  • The job token fetches them. The person who started the pipeline needs at least the Reporter role in the submodule's project, and that project must allow your project's job token (chapter 28).
  • GIT_SUBMODULE_DEPTH sets their depth separately from GIT_DEPTH.

If a job with GIT_STRATEGY: fetch fails with fatal: run_command returned non-zero status, GitLab's documentation suggests switching to GIT_STRATEGY: clone.

When you write it
Do this
If a job reads Git history, for example the commits between two points or a version from tags, set GIT_DEPTH for that job. Set it to 0 for a full clone.
Why
By default a job gets a shallow copy of the repository: only the latest commits, and no tags from before them.
Watch out
A command that needs older history fails or gives a wrong answer in that shallow copy, and the error does not say why. The card history missing shows the symptom.
What you can now do
  1. Say what a job's clone contains, for each kind of pipeline.
  2. Give a job the history it needs, and no more.
  3. Explain why a merge request pipeline may be testing a commit that is on no branch.
Spot the bugThe generator that fails on big merge requests#

In megacorp/data/mono, generate-pipeline works in most merge request pipelines. It fails in a few, always on long-lived branches with dozens of commits, and the log shows that git diff can't find a commit. mono's clone depth is the project default of 20:

megacorp/data/monoci/generate.sh
#!/usr/bin/env bash
# ci/generate.sh in megacorp/data/mono
# Writes one child-pipeline job per service that changed. Its output is the YAML
# that the service-pipelines trigger job runs. Used by the chapter "Pipelines as building
# blocks".
set -euo pipefail

base="${CI_MERGE_REQUEST_DIFF_BASE_SHA:-HEAD~1}"
changed=$(git diff --name-only "$base" HEAD -- services/ | cut -d/ -f2 | sort -u)

echo "stages: [build]"
for svc in $changed; do
  cat <<EOF
build-${svc}:
  stage: build
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  script:
    - make -C services/${svc} build
EOF
done

# a child pipeline needs at least one job, even when nothing changed
if [ -z "$changed" ]; then
  printf 'no-changes:\n  stage: build\n  script: [echo "no service changed"]\n'
fi
QuestionWhy do only the big merge requests fail, and what other mistake hides in the same line?
Show the answer

A merge request pipeline fetches only the pipeline's commit, and only the newest 20 commits of its history. CI_MERGE_REQUEST_DIFF_BASE_SHA is the base of the merge request's diff. On a branch with 20 or more commits, that base is older than the clone, so git diff can't find it. Give this job more history: raise GIT_DEPTH for it, or run git fetch --unshallow before the diff.

The other mistake is the fallback, HEAD~1, used in every other pipeline. It compares with the previous commit only, so a push of three commits builds just the services changed in the last one. For pushes, CI_COMMIT_BEFORE_SHA holds the branch's tip before the push. It is all zeros for new branches, schedules and manual runs, so the script still needs a fallback. See shallow clones.

The scanning landscape#

Part VIII · Scanners and gates · Chapter 33 · 7 min read

A scanner is a job that reads something your pipeline has, looking for known problems. It might read the source code, the libraries the code uses, the image the pipeline built, or the running application. Enterprise pipelines often run five or six scanners. Each one writes a security report, and GitLab shows the findings in merge requests and on security pages, if your tier includes them.

Three questions explain almost any scanner: what does it read, who put it in my pipeline, and what can it block?

Tuesday, 09:05. Twelve scanners, no findings.

A merge request touches the payment code, and its pipeline ran six scanner jobs. The merge request page shows no findings at all, and the reviewer asks whether the scans found nothing or reported nothing. You can't tell which. Where do a scanner's results go, and who reads them? (the card)

What each scanner reads#

The scanner map. Each scanner reads one thing along the build path and writes a security report. Where the findings appear depends on the tier. WHAT EACH SCANNER READS, ALONG THE BUILD PATH source code the repository dependencies lock files, an SBOM container image built by the pipeline running app a test environment SAST Free Secret detection Free IaC scanning Free Dependency scanning Ultimate Container scanning Free DAST, API testing Ultimate each scanner job writes a security report: a JSON artifact WHERE THE FINDINGS APPEAR the job's log and artifacts every tier the pipeline's Security tab, the merge request's Reports tab Ultimate the vulnerability report the default branch only Ultimate
Figure The scanner map. Each scanner reads one thing along the build path and writes a security report. Where the findings appear depends on the tier.#
ScannerReadsLooks forTier
SASTthe source coderisky code, such as a database query built from user inputFree
Secret detectionthe source codepasswords, keys and tokens committed by mistakeFree
IaC scanninginfrastructure files, such as Terraforminsecure cloud and cluster settingsFree
Dependency scanningthe dependencies, as an SBOMlibraries with known vulnerabilitiesUltimate
Container scanningthe built imagevulnerable packages inside the imageFree
DAST and API testingthe running applicationweaknesses that show only when the app runsUltimate

An SBOM, a software bill of materials, is a list of every library a project uses, with versions. A free tier can run a scanner and still not show its findings anywhere except the job itself.

Who put it in your pipeline#

A scanner job arrives in one of four ways, and the way decides who can change it:

Arrives throughHow you recognise itWho can change it
A GitLab templatean include: template: line, in your file or a central onewhoever owns that file; a project can adjust a job by redefining it under the same name
A central template or componentan include: of your platform team's filesthe platform team
A scan execution policyjobs that are in no file, with names such as secret-detection-1only the security team, through the policy project
A pipeline execution policyjobs in the stages .pipeline-policy-pre and .pipeline-policy-postonly the security team

MegaCorp uses all four. Its central template includes GitLab's scanners, and its security policy project adds more to every pipeline:

megacorp/devops/ci-templatestemplates/security.ymlsecurity-includes
include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml
  - template: Security/Container-Scanning.gitlab-ci.yml
megacorp/security/policies.gitlab/security-policies/policy.ymlscan-execution
scan_execution_policy:
  - name: Secret detection everywhere
    description: Run secret detection in every pipeline on every branch.
    enabled: true
    rules:
      - type: pipeline
        branches: ["*"]
    actions:
      - scan: secret_detection
  - name: Nightly dependency scan
    description: Scan dependencies on the default branch every night.
    enabled: true
    rules:
      - type: schedule
        branches: [main]
        cadence: "0 2 * * *"
    actions:
      - scan: dependency_scanning

Where the findings appear#

PlaceWhat it showsTier
The job's log and artifactsthe raw report, as a JSON fileevery tier
The pipeline's Security tabevery finding from that pipelineUltimate
The merge request's Reports tabwhat the merge would add or fixUltimate
The vulnerability reportvulnerabilities on the default branchUltimate

GitLab calls a problem on a branch a finding. It becomes a vulnerability when the branch is merged into the default branch. Findings expire with the job's artifacts, or after 90 days by default on self-managed GitLab and 30 days on GitLab.com.

What a scanner can block#

Three different things can stop a change, and they are easy to mix up:

  • The scanner job fails. The scanner crashed, or couldn't read what it needed. GitLab's scanner jobs are allowed to fail, so this shows only as a warning, and the pipeline stays green.
  • The merge request needs extra approval. A merge request approval policy reads the findings and demands approval, for example from the security team (chapter 36).
  • A third-party gate fails the job on purpose, such as a quality gate that isn't met (chapter 35).
What you can now do
  1. Name what each kind of scanner reads and produces, and which tier shows its findings where.
  2. Tell a scan that found nothing from one whose report never arrived.
  3. Read a merge request's security widget and know what it can and cannot show.
Spot the bugWhose job is it?#

On MegaCorp's production GitLab, which has Ultimate, a main pipeline of payments-api shows these jobs, among others:

secret_detection
container_scanning
secret-detection-1
policy-sbom

Use the specimen files above, and chapter 18.

QuestionFor each job, where is it defined, and who could change it?
Show the answer
  • secret_detection comes from GitLab's secret detection template, which MegaCorp's security.yml includes. The platform team owns that include. A project could redefine the job by name.
  • container_scanning comes from GitLab's container scanning template, and security.yml replaces its rules and needs. The platform team owns it.
  • secret-detection-1 comes from the scan execution policy "Secret detection everywhere". The numbered suffix gives it away. Only the security team can change it, in the policy project.
  • policy-sbom comes from the pipeline execution policy "MegaCorp guardrails", which puts it in .pipeline-policy-post. Only the security team can change it.

See who put it in your pipeline.

GitLab's own scanners#

Part VIII · Scanners and gates · Chapter 34 · 7 min read

GitLab's scanners arrive as templates. Each template adds jobs whose rules decide when they appear, and four behaviours explain most surprises:

  • By default they run in branch pipelines, not merge request pipelines.
  • Some add a job only when matching files exist, or only on Ultimate.
  • They are all allowed to fail, so a broken scanner shows only a warning.
  • Their jobs sit in the test stage, so a pipeline must have one.
Tuesday, 14:35. The scan that scanned the wrong thing.

container_scanning runs in every merge request pipeline and always passes. Today you notice it scans alpine:latest, not the image the pipeline built. The template was included months ago and adjusted by the central team. Somewhere between their adjustment and GitLab's template, the image name got lost. (the card)

What the templates add#

Each Security/ template that MegaCorp includes contains a single line: an include of its Jobs/ twin, which holds the jobs.

TemplateJobs it addsWhen a job appears
Jobs/SAST.gitlab-ci.ymlone per analyzer, such as semgrep-sastonly if files that analyzer reads exist, such as *.java or *.py; gitlab-advanced-sast also needs Ultimate
Jobs/Secret-Detection.gitlab-ci.ymlsecret_detectionin branch pipelines
Jobs/Dependency-Scanning.gitlab-ci.ymlone per package manager, such as gemnasium-maven-dependency_scanningonly on Ultimate; its Gemnasium analyzer is deprecated
Jobs/Container-Scanning.gitlab-ci.ymlcontainer_scanningin branch pipelines; it scans the image named in CS_IMAGE

Every job's first rule checks its toggle, such as SAST_DISABLED. Every job has allow_failure: true. The secret detection and container scanning reports can be downloaded only by people with the Developer role or above.

On the local GitLab, which is the Free edition, these templates add just two jobs to payments-api's main pipeline: secret_detection and container_scanning. There is no SAST job, because the specimen repository holds only a pom.xml, and no .java files. There is no dependency scanning job either, because those jobs need Ultimate.

The merge request gap#

The scanner jobs' rules follow the same three steps:

  1. In a merge request pipeline, run only if AST_ENABLE_MR_PIPELINES is "true".
  2. If that variable is on and the branch has an open merge request, skip the job in the branch pipeline, so that nothing is scanned twice.
  3. Otherwise, run in the branch pipeline.

So by default the scanners run only in branch pipelines. On the local GitLab, a push to a payments-api merge request branch produced both pipelines. The branch pipeline had secret_detection; the merge request pipeline didn't. Anyone reading only the merge request pipeline sees no secret detection at all.

One of GitLab's scanner jobs is missing from a pipeline. Ask these in order.yesnoyesnoyesnoyesnoIs this a merge request pipeline?Scanners skip merge requestpipelines unlessAST_ENABLE_MR_PIPELINES is "true"Is the scanner's toggle set, in YAML orin settings?A toggle such as SAST_DISABLEDremoves the jobIs it SAST, and the repository has nofiles that the analyzer reads?SAST adds jobs only for thelanguages it findsIs it dependency scanning, on Free orPremium?Those jobs appear only on UltimateCheck whether someone redefined the joband replaced its rules
Figure One of GitLab's scanner jobs is missing from a pipeline. Ask these in order.#
As text
  1. Is this a merge request pipeline? Yes: Scanners skip merge request pipelines unless AST_ENABLE_MR_PIPELINES is "true". No: the next step.
  2. Is the scanner's toggle set, in YAML or in settings? Yes: A toggle such as SAST_DISABLED removes the job. No: the next step.
  3. Is it SAST, and the repository has no files that the analyzer reads? Yes: SAST adds jobs only for the languages it finds. No: the next step.
  4. Is it dependency scanning, on Free or Premium? Yes: Those jobs appear only on Ultimate. No: the next step.
  5. Check whether someone redefined the job and replaced its rules

A pipeline without a test stage fails in a different way. It can't be created at all, because the scanner jobs name a stage that doesn't exist (chapter 24).

Changing a template's job#

To change a template's job, define a job with the same name. GitLab merges the two, and your keys win. MegaCorp moves container scanning into its own stage, points it at the image it has just built, and replaces its rules:

megacorp/devops/ci-templatestemplates/security.ymlcontainer-scan
variables:
  CS_IMAGE: $IMAGE_REF

container_scanning:
  stage: scan
  needs: [image-build]
  rules:
    - if: $CONTAINER_SCANNING_DISABLED == "true"
      when: never
    - !reference [.rules, never-on-schedule]
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Replacing rules replaces all of them, because lists are never merged (chapter 14). MegaCorp's rules keep the toggle, and add merge request pipelines, which the template alone would skip. That's why the local captures show container_scanning in payments-api's merge request pipeline, but not secret_detection.

Gotcha
One key from the template survives the merge and undoes MegaCorp's intent. GitLab's job has dependencies: [], and MegaCorp's version doesn't replace it. An empty dependencies blocks every artifact download, dotenv reports included, even with needs: [image-build]. On the local GitLab 19.3, a probe job with both keys got an empty IMAGE_REF, and so an empty CS_IMAGE. So MegaCorp's container scan never receives the image it was meant to scan. Adding dependencies: [image-build] fixed the probe. See the symptom card.
When you write it
Do this
Include GitLab's scanner templates and leave their jobs alone where you can. To change one, set only the variables that the template documents.
Why
The templates change with every GitLab release. A job you copied and edited stops getting those changes.
Watch out
The templates' jobs run in branch pipelines. If your workflow uses merge request pipelines, check that your merge requests are scanned too. The card no scans in merge requests shows the gap.
What you can now do
  1. Include a GitLab scanner template and know which of its jobs will run, in which pipelines.
  2. Adjust a template's job without losing what the template set.
  3. Find why a scanner ran against the wrong target, or not in merge requests at all.
Spot the bugThe scan the reviewers never saw#

A reviewer opens a payments-api merge request and checks its pipeline. There is no secret_detection job. A colleague finds one in a different pipeline, for the same commit, on the merge request's branch.

megacorp/devops/ci-templatestemplates/security.ymlsecurity-includes
include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml
  - template: Security/Container-Scanning.gitlab-ci.yml
QuestionWhy does the scan run in one pipeline and not the other, and how could it run in the merge request pipeline instead?
Show the answer

GitLab's secret detection template runs its job in branch pipelines, and in merge request pipelines only when AST_ENABLE_MR_PIPELINES is "true". Nobody set it, so the scan ran in the branch pipeline only.

Set AST_ENABLE_MR_PIPELINES: "true", in the YAML or in settings. The job then runs in merge request pipelines, and is skipped in the branch pipeline while a merge request is open. Alternatively, redefine secret_detection with rules of your own, as MegaCorp did for container scanning. See the merge request gap.

Third-party scanners#

Part VIII · Scanners and gates · Chapter 35 · 5 min read

Many organisations also run tools that GitLab doesn't ship, such as SonarQube, Snyk or Trivy. They all follow one pattern: a job runs the tool, the tool's exit code decides whether the job fails, and a report file may carry the findings into GitLab. Find those three things, and the job's allow_failure and needs, and you know what the tool can really stop.

Wednesday, 10:45. The gate that let it through.

The SonarQube quality gate failed on Monday's merge request. It merged anyway, and deployed. The job showed a red cross, and a small "allowed to fail" beside it that nobody read. A scanner that cannot block is a scanner that reports. Which did the team want? (the card)

Tool, exit code, report, gate#

What can a third-party scanner job actually stop? Follow it from start to finish.noyesyesnonoyesThe job runs the tool's command-lineprogramDoes the tool exit with an error whenit finds problems?The job passes whatever it finds;many tools need an option for thisIs the job allowed to fail?A failure shows only as a warning,and blocks nothingDo the jobs that matter wait for it?With needs, a deploy can startwithout waiting for the scanA failure fails the pipeline, and jobsthat wait for the scan don't run
Figure What can a third-party scanner job actually stop? Follow it from start to finish.#
As text
  1. The job runs the tool's command-line program
  2. Does the tool exit with an error when it finds problems? No: The job passes whatever it finds; many tools need an option for this. Yes: the next step.
  3. Is the job allowed to fail? Yes: A failure shows only as a warning, and blocks nothing. No: the next step.
  4. Do the jobs that matter wait for it? No: With needs, a deploy can start without waiting for the scan. Yes: the next step.
  5. A failure fails the pipeline, and jobs that wait for the scan don't run

A report file lets GitLab show the tool's findings, instead of leaving them in the job log or on the tool's own website:

ReportForWhere GitLab shows it
codequalitylint and code-quality findings, in a simple JSON formatmerge requests, and the pipeline's Code Quality tab (Premium)
sarifsecurity findings from any tool that writes SARIF 2.1.0the Security tab and the vulnerability report (Ultimate, since 19.2)
junittest resultsthe pipeline's Tests tab, and the merge request's test summary

For code quality, GitLab's preferred route is exactly this: run your own linter and import its report. GitLab's built-in CodeClimate template is deprecated.

MegaCorp's SonarQube job#

MegaCorp's platform team ships SonarQube as a component (chapter 16):

megacorp/devops/componentstemplates/sonar-scan.ymljob
sonar-scan:
  stage: $[[ inputs.stage ]]
  image: registry.example.com/megacorp/devops/ci-tools/sonar-scanner:6
  variables:
    SONAR_PROJECT_KEY: $[[ inputs.project_key ]]
    SONAR_QUALITYGATE_WAIT: "$[[ inputs.quality_gate ]]"
  script:
    - sonar-scanner -Dsonar.projectKey="$SONAR_PROJECT_KEY" -Dsonar.host.url="$SONAR_HOST_URL"
  rules:
    - if: $SKIP_SONAR == "true"
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  allow_failure: true

Read it with the pattern:

  • Tool: sonar-scanner, which sends the code to the Sonar server for analysis.
  • Exit code: the component's quality_gate input sets SONAR_QUALITYGATE_WAIT. It is meant to make the scanner wait for Sonar's quality gate, and fail the job when the gate fails.
  • Report: none. The findings stay on the Sonar server.
  • Gate: allow_failure: true, so even a failed gate is only a warning.

It also needs SONAR_TOKEN, which is protected, so in merge request pipelines from feature branches the scan can't sign in (chapter 27).

When you write it
Do this
Make a scanner tool exit with an error on the findings you care about, and keep its report as an artifact.
Why
A scanner job can only block a merge if the job fails. The report lets people see why it failed.
Watch out
Do not set allow_failure: true on a scanner unless its result is only advice. With it, the job can fail and the merge still goes through. The card the gate didn't stop a deploy shows the result.
What you can now do
  1. Add a third-party scanner as a job whose failure means what the team wants it to mean.
  2. Publish its report so GitLab shows the findings where reviewers look.
  3. Say whether a scanner job can block a merge, and make it block when it should.
Spot the bugThe quality gate that couldn't stop a deploy#

A change that fails MegaCorp's Sonar quality gate is merged into payments-api's main. The pipeline goes green, and deploy-staging ships the change. sonar-scan shows an orange warning.

megacorp/devops/ci-templatespipelines/java-service.ymldeploys
deploy-staging:
  extends: .deploy
  needs: [image-build]
  variables:
    ENVIRONMENT: staging
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, default-branch]

deploy-prod:
  extends: .deploy
  needs: [image-build]
  variables:
    ENVIRONMENT: production
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
      when: manual
QuestionName the two separate reasons the failed gate didn't stop the deploy.
Show the answer
  • sonar-scan has allow_failure: true, so its failure never fails the pipeline.
  • deploy-staging has needs: [image-build]. It starts as soon as the image is built, without waiting for jobs in the test stage, so it doesn't wait for sonar-scan at all (chapter 24).

To make the gate stop deploys, both would have to change: the scan must be allowed to fail the pipeline, and the deploy must wait for it. Whether the gate should block is a decision for the teams that own the pipeline. See the pattern.

Policies and gates#

Part VIII · Scanners and gates · Chapter 36 · 7 min read

Security policies are rules the security team writes once for many projects. They need Ultimate, and come in three kinds:

  • scan execution policies add scanner jobs
  • pipeline execution policies add any jobs, or replace the pipeline
  • merge request approval policies demand approval when scans find problems

Policies win over anything a project writes. A toggle such as SAST_DISABLED skips a project's own scanner jobs, never a policy's. Turning a mandated scan off is a decision for the security team, not a change to the YAML.

Wednesday, 16:10. Jobs from nowhere.

This morning every pipeline in the group has two new jobs, secret-detection-1 and policy-sbom, in a stage called .pipeline-policy-post. No file in any project mentions them. Your toggles don't remove them. Someone with a different kind of power put them there. (the card)

Three kinds of policy#

PolicyWhat it doesHow you recognise it
Scan executionadds scanner jobs to pipelines, or runs scans on a schedulejobs such as secret-detection-1, in test, or in scan-policies when there is no test stage
Pipeline executionadds any jobs, or replaces the project's pipelinejobs in .pipeline-policy-pre, which runs first, and .pipeline-policy-post, which runs last
Merge request approvalrequires approval when scans find problemsan approval rule on the merge request, named after the policy

Pipeline execution policies have a scheduled variant Since 19.2. It creates pipelines of its own on a schedule, in every project it covers, and runs only the policy's jobs, never the project's.

All three live in one file, .gitlab/security-policies/policy.yml, in a separate security policy project linked to the group. A project can hold at most five policies of each kind. Chapter 18 shows MegaCorp's first two. This is the third:

megacorp/security/policies.gitlab/security-policies/policy.ymlapproval
approval_policy:
  - name: Block new critical vulnerabilities
    description: A merge into main that adds a critical finding needs security's approval.
    enabled: true
    rules:
      - type: scan_finding
        branches: [main]
        scanners: [sast, secret_detection, dependency_scanning, container_scanning]
        vulnerabilities_allowed: 0
        severity_levels: [critical]
        vulnerability_states: [new_needs_triage]
    actions:
      - type: require_approval
        approvals_required: 1
        group_approvers: [megacorp/security]

Toggles against policies#

GitLab's scanner templates switch off with variables: SAST_DISABLED, SECRET_DETECTION_DISABLED, CONTAINER_SCANNING_DISABLED and others. GitLab's documentation is exact about their limit: "Skipping jobs does not prevent any security jobs defined by scan execution policies from running." Two more rules follow:

  • A project can't override a policy's scanner jobs, even by using the same job name.
  • In a project under a scan execution policy, the scanners' exclude settings, such as SAST_EXCLUDED_PATHS, are preset, and only a policy can change them.

So when a team wants a mandated scan off, the answer isn't in their YAML. The security team decides, for example by narrowing the policy's scope.

When a merge request needs the security team#

MegaCorp's approval policy looks at merges into main. If a scan finds a new critical vulnerability, one member of megacorp/security must approve. The rule also applies when GitLab can't check it:

Why does my merge request need the security team's approval? Ask these in order.yesnoyesnonoyesDid a scan find a new criticalvulnerability?The rule is working; fix thefinding, or ask for approvalDid a scanner the rule names produce noreport?The rule can't be checked, and bydefault that also requiresapprovalDoes the policy target this branch?Look for another policy; eachshows as its own approval ruleOpen Secure › Policies, and read thepolicy's rule
Figure Why does my merge request need the security team's approval? Ask these in order.#
As text
  1. Did a scan find a new critical vulnerability? Yes: The rule is working; fix the finding, or ask for approval. No: the next step.
  2. Did a scanner the rule names produce no report? Yes: The rule can't be checked, and by default that also requires approval. No: the next step.
  3. Does the policy target this branch? No: Look for another policy; each shows as its own approval rule. Yes: the next step.
  4. Open Secure › Policies, and read the policy's rule

The second case surprises people. By default a policy fails closed: "Invalid or unenforceable rules of a policy require approval." So a scanner that was skipped or crashed doesn't let a merge request slip through. It blocks it until someone approves. The policy can say fallback_behavior: fail: open instead.

Finding the policies#

Anyone who can see a project can open its policies at Secure › Policies. The list shows each policy and whether it is enforced. Changes go through a merge request in the policy project:

A simplified sketch of the Policies page and the policy editor, not a screenshot. A green checkmark marks an enabled policy. The editor saves changes as a merge request in the security policy project. payments-api Build Secure Policies Deploy Settings Policies New policy ✓ Secret detection everywhere scan execution ✓ MegaCorp guardrails pipeline execution ✓ Block new critical vulnerabilities approval ✓ green: enabled and enforced · grey: not enabled · select a row to open its drawer, then Edit policy Policy editor Rule mode YAML mode rules and actions, or the policy's YAML Configure with a merge request
Figure A simplified sketch of the Policies page and the policy editor, not a screenshot. A green checkmark marks an enabled policy. The editor saves changes as a merge request in the security policy project.#
What you can now do
  1. Recognise a job that a policy added, and find the policy under Secure › Policies.
  2. Tell the four kinds of policy apart, and say what each can add, require or block.
  3. Know what your file can and cannot change about a policy's jobs.
Spot the bugThe toggle that only half worked#

payments-api's pipelines run two secret detection jobs, and the team wants to save time. They add this to their .gitlab-ci.yml:

variables:
  SECRET_DETECTION_DISABLED: "true"

secret_detection disappears from their pipelines, but secret-detection-1 still runs.

QuestionWhy did only one job go, and what should the team do if they want the other one gone too?
Show the answer

secret_detection comes from GitLab's template, included by MegaCorp's security.yml, and the template honours the toggle. secret-detection-1 comes from the scan execution policy "Secret detection everywhere". Toggles skip only a project's own jobs, never a policy's.

Removing the second job isn't the team's decision. They can ask the security team to change the policy's scope. Having the job twice is expected: GitLab runs both, because the two can use different settings. See toggles against policies.

Building Java, Node and front ends#

Part IX · Delivery · Chapter 37 · 8 min read

A build job is an ordinary job with the right image. What makes builds confusing in a big organisation is everything around the compiler:

  • where the dependencies come from: a company mirror, and a cache
  • what the build produces, and where it goes: an artifact, a package registry, an image or a bucket
  • which version number the result carries

Most "works on my machine" failures come from the first. Most "which version is running?" confusion comes from the last.

Thursday, 09:30. A release with SNAPSHOT in its name.

The release job publishes payments-api 1.8.0-SNAPSHOT. The tag was v1.8.0. The Maven build took the version from pom.xml, which nobody bumps, and the pipeline never told it otherwise. Where does a build's version number come from, and where should it? (the card)

A Maven build at MegaCorp#

MegaCorp's Maven jobs start from one hidden job. It picks the JDK image, caches the local repository, and writes a Maven settings file before the script runs:

megacorp/devops/ci-templatestemplates/java-maven.ymlmaven-base
.maven-base:
  extends: .megacorp-base
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
  variables:
    JAVA_VERSION: "21"
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  cache:
    key:
      files: [pom.xml]
    paths: [.m2/repository]
  before_script:
    # a job's own before_script replaces default:before_script, so re-add the functions
    - !reference [.snippets, functions]
    - !reference [.snippets, maven_settings]

The settings file comes from a function in the ci-tools image. It sends every download through MegaCorp's Nexus server, a company copy of the public repositories:

megacorp/devops/ci-toolslib/ci-lib.shmaven-settings
mc_maven_settings() {
  cat <<EOF
<settings><mirrors><mirror><id>nexus</id><mirrorOf>*</mirrorOf><url>${MC_NEXUS_URL:-https://nexus.example.com/repository/maven-public/}</url></mirror></mirrors></settings>
EOF
}

Three variables decide how Maven runs, and only one is in the YAML:

VariableSet inEffect
MAVEN_CLI_OPTSthe megacorp group's CI/CD settingsbatch mode, and the settings file above; it overrides any value a project writes in YAML (chapter 26)
MAVEN_OPTSthe templatekeeps Maven's local repository inside the project, so that it can be cached
MC_NEXUS_URLpayments-api's CI/CD settingswhich Nexus the mirror points at

maven-build packages the jar without running the tests, and keeps it as an artifact. maven-test runs the tests twice, on JDK 17 and 21, with parallel: matrix (chapter 14).

A Node front end at MegaCorp#

The Node templates install with npm ci, from a cache keyed on package-lock.json, and build into dist/ (chapter 31). web-portal's whole pipeline is three jobs:

megacorp/devops/ci-templatespipelines/web.ymljobs
node-build:
  extends: .node-build

node-test:
  extends: .node-test

deploy-site:
  extends: .publish-site
  needs: [node-build]
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, default-branch]

deploy-site copies dist/ to an S3 bucket and clears the CloudFront cache (chapter 38). web-portal has no .gitlab-ci.yml of its own: its settings point at this file (chapter 18).

A build works on your machine but fails in CI. Ask these in order.yesnoyesnoyesnoyesnoDoes it download dependencies?CI downloads through the companymirror and proxy, which may lackor block a packageDoes it need a file that isn't in Git,such as a local settings file?The job has only the repository,artifacts and variablesIs the tool's version different fromyours?The job uses the image's version;check the image's tagDid it restore an old cache?Check the cache key, or clear thecache (chapter 24)Compare the job's variables with yourshell's; settings variables may differ
Figure A build works on your machine but fails in CI. Ask these in order.#
As text
  1. Does it download dependencies? Yes: CI downloads through the company mirror and proxy, which may lack or block a package. No: the next step.
  2. Does it need a file that isn't in Git, such as a local settings file? Yes: The job has only the repository, artifacts and variables. No: the next step.
  3. Is the tool's version different from yours? Yes: The job uses the image's version; check the image's tag. No: the next step.
  4. Did it restore an old cache? Yes: Check the cache key, or clear the cache (chapter 24). No: the next step.
  5. Compare the job's variables with your shell's; settings variables may differ

Which version is it?#

Three numbers can describe one build, and they often disagree:

NumberComes frompayments-api's example
The version in the build filepom.xml or package.json, as committed1.8.0-SNAPSHOT
The Git tagthe release tag, in CI_COMMIT_TAG, which is set only in tag pipelinesv1.5.0
The image tagthe commit, in CI_COMMIT_SHAa 40-character commit hash

MegaCorp's mc tool can work out a version in any pipeline: the tag if there is one, otherwise 0.0.0- followed by the short commit hash.

megacorp/devops/ci-toolsbin/mcversion
version)
  echo "${CI_COMMIT_TAG:-0.0.0-${CI_COMMIT_SHORT_SHA}}"
  ;;

Nothing in the Maven templates uses it, though. The exercise shows the result.

Test results and coverage#

  • Test reports in JUnit XML, saved as artifacts: reports: junit, fill the pipeline's Tests tab and the merge request's test summary. The summary lists newly failed tests. A report never changes the job's status: the test command's exit code does.
  • A coverage percentage comes from the job log. The coverage keyword holds a regular expression, and GitLab shows the first number it matches in the merge request.
  • Line-by-line coverage comes from a Cobertura or JaCoCo file saved as artifacts: reports: coverage_report. GitLab marks the lines in the merge request's diff.

Publishing packages to GitLab's registries#

Some teams publish libraries to GitLab's own package registry, using the job token as the password.

Maven. A settings file sends the job token as a header:

<settings>
  <servers>
    <server>
      <id>gitlab-maven</id>
      <configuration>
        <httpHeaders>
          <property>
            <name>Job-Token</name>
            <value>${CI_JOB_TOKEN}</value>
          </property>
        </httpHeaders>
      </configuration>
    </server>
  </servers>
</settings>

The POM's distribution repository is ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/maven, and a job runs mvn deploy. To refuse a version that is already published, a group Owner turns off Allow duplicates under Settings › Packages and registries.

npm. The job writes an .npmrc that points the package's scope at the project's registry, with the job token:

echo "@megacorp:registry=https://${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/" > .npmrc
echo "//${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN}" >> .npmrc
npm publish

The scope is the top-level group of the project that hosts the package, in lower case: @megacorp. A package name can belong to only one project per top-level group. Publishing it from a second project fails, even with a new version.

When you write it
Do this
Build once, in one job, and pass the result to later jobs as an artifact. Test and package that one result. Publish test results with artifacts: reports: junit.
Why
Every later job then works on the same build, and test results show in the merge request.
Watch out
Use npm ci, not npm install, and give Maven --batch-mode. Otherwise a build can install different versions from one run to the next, and Maven fills the log with progress lines.
What you can now do
  1. Read MegaCorp's build jobs for Java, Node and a front end, and say what each caches, keeps and reports.
  2. Give a build its version from the pipeline, not from a file nobody bumps.
  3. Build once and pass the result on, instead of rebuilding in every job.
Spot the bugThe release that called itself a snapshot#

After payments-api's v1.5.0 release, the support team checks the running service. Its version page, which reads the jar's version, says 1.8.0-SNAPSHOT. The image is the one that the v1.5.0 tag pipeline built.

megacorp/payments/payments-apipom.xmlversion
<artifactId>payments-api</artifactId>
<version>1.8.0-SNAPSHOT</version>
megacorp/devops/ci-templatestemplates/java-maven.ymlmaven-build
.maven-build:
  extends: .maven-base
  stage: build
  script:
    - mvn $MAVEN_CLI_OPTS -DskipTests package
  artifacts:
    paths: [target/*.jar]
QuestionWhere does 1.8.0-SNAPSHOT come from, and what would make the jar carry the release's version?
Show the answer

Maven takes the version from pom.xml, which says 1.8.0-SNAPSHOT. The build job runs mvn package without changing it, so every pipeline, the tag pipeline included, builds a jar with that version. The tag names the commit, but never reaches Maven.

Pass the version into the build in tag pipelines, for example from mc version, which returns CI_COMMIT_TAG there. Or make each release a commit that sets the POM's version, and tag that commit. See which version is it.

GitLab, AWS and Azure#

Part IX · Delivery · Chapter 38 · 25 min read

Jobs that build and deploy to AWS or Azure need permissions there. The modern way, and MegaCorp's way, stores no password anywhere. Each job trades its ID token for short-lived access, as chapter 28 showed: temporary keys for an AWS role, or a sign-in as an application in Microsoft Entra ID, Azure's identity service.

This chapter covers:

  • what must match on each side, and who owns each part
  • which credentials each cloud's command-line tool really uses
  • working with several AWS accounts or Azure subscriptions
  • ECR and ACR, secrets from each cloud, and the usual deploy targets
  • what to check, in order, when AWS or Azure says no
Thursday, 15:00. Refused, but only on tags.

The AWS login works on every branch pipeline and fails on every tag pipeline with Not authorized to perform sts:AssumeRoleWithWebIdentity. Same job, same role, same token keyword. The role's trust policy names branches. A tag is not a branch, and the token says so. (the card)

Cloud words in two minutes#

If a cloud is new to you, these are the only words this chapter needs:

WordMeaning
Accounta separate AWS space with its own resources and bill, named by a 12-digit number such as 123456789012
Regiona location where resources live, such as eu-west-2 (London); jobs set it with AWS_REGION
IAMAWS's identity service: who may do what
Rolean identity with permissions and no password; a person or a system assumes it for a while
Trust policythe rules on a role that say who may assume it
Permission policythe rules on a role that say what it may do once assumed
ARNthe full name of anything in AWS, such as arn:aws:iam::123456789012:role/gitlab-ecr-push
STSthe Security Token Service, which hands out temporary keys for a role
ECRthe Elastic Container Registry, where images are stored
S3 and CloudFrontfile storage, and the network that serves those files quickly worldwide
ECS, EKS and Lambdathree places to run code: AWS's container service, Kubernetes clusters, and single functions
Secrets Manager and Parameter Storetwo AWS services that store secrets and settings
WordMeaning
Tenantan organisation's directory in Microsoft Entra ID, named by a GUID such as 00000000-0000-0000-0000-000000000000
Subscriptiona separate Azure space with its own resources and bill, also named by a GUID
Resource groupa named folder for resources inside a subscription, such as rg-delivery-prod
Microsoft Entra IDAzure's identity service: who may sign in, and as what; it used to be called Azure Active Directory
App registrationan identity for software, named by its client ID; its service principal is what receives permissions
Managed identityan identity that Azure creates and looks after for a resource, with no secret to store
Federated credentiala rule on an app registration or managed identity that says which outside tokens it accepts instead of a secret
Role assignmenta role, such as AcrPush, given to an identity at a scope: a subscription, a resource group or one resource
ACRAzure Container Registry, where images are stored
AKS and Container Appstwo places to run containers: Kubernetes clusters, and a service that runs containers without a cluster to manage
Key VaultAzure's store for secrets, keys and certificates

What must match#

A job gets into a cloud only when several pairs line up. Each pair has one half in GitLab and the other half in the cloud, and a different team often owns each half.

On AWS, five pairs must match:

The five pairs that must match before a job can use an AWS role. The first four decide whether STS hands out keys at all. The fifth decides what the keys can do. IN GITLAB the job and its template IN AWS IAM, in the account 1 who signed the token https://gitlab.example.com must match the OIDC identity provider's URL and AWS must be able to reach it 2 aud, from id_tokens: https://gitlab.example.com must match the provider's audience and the trust policy's aud condition 3 sub: project, ref type, ref …:ref_type:branch:ref:main must match the trust policy's sub pattern StringLike, when it has a * 4 the role the job asks for AWS_ROLE_ARN must exist a role with that exact ARN role/gitlab-ecr-push 5 what the script then does log in to ECR, push an image must be allowed the role's permission policy push and pull in ECR
Figure The five pairs that must match before a job can use an AWS role. The first four decide whether STS hands out keys at all. The fifth decides what the keys can do.#

At MegaCorp the GitLab halves live in the templates, which the platform team owns:

megacorp/devops/ci-templatestemplates/container.ymlimage-build
.image-build:
  extends: .megacorp-base
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
    IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
    - buildah push "$IMAGE:$CI_COMMIT_SHA"
    - !reference [.snippets, acr_push]
    - echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
  artifacts:
    reports:
      dotenv: build.env

The AWS halves are set in the AWS account itself. This file describes MegaCorp's:

awsiam.ymloidc-provider
# AWS accepts ID tokens signed by this GitLab, for this audience.
oidc_provider:
  arn: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
  url: https://gitlab.example.com
  audiences: [https://gitlab.example.com]
awsiam.ymlecr-push-role
# Assumed by image-build, and by the ecr-push component, through mc_aws_login.
ecr_push_role:
  arn: arn:aws:iam::123456789012:role/gitlab-ecr-push
  trust_policy:
    Effect: Allow
    Principal:
      Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
    Action: sts:AssumeRoleWithWebIdentity
    Condition:
      StringEquals:
        gitlab.example.com:aud: https://gitlab.example.com
      StringLike:
        gitlab.example.com:sub: project_path:megacorp/*:ref_type:branch:ref:*
  permissions: push and pull images in every ECR repository in the account

A mismatch in the first four pairs stops the login: STS refuses, and no keys come back. A mismatch in the fifth lets the login succeed, and a later command fails with an access-denied error. That difference tells you where to look.

On Azure, the pairs are much the same:

In GitLabIn Entra ID and AzureMatched how
the instance's URL, which becomes the token's issthe federated credential's issuer, https://gitlab.example.comexactly
aud in the job's id_tokens:the credential's single audienceexactly
the project, the kind of ref and the ref, which make the token's subthe credential's subjectexactly, with no wildcards
the client ID and tenant ID that az login namesthe app registrationexactly
what the job does once signed inrole assignments, such as AcrPush on the registryby scope

This file describes MegaCorp's application for image pushes, and what it may do:

azureentra.ymlapp-registration
# The identity that image jobs sign in as. Each federated credential names one kind of
# pipeline that may use it.
acr_push_app:
  display_name: gitlab-acr-push
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: 00000000-0000-0000-0000-000000000000
  federated_credentials:
    - name: payments-api-main
      issuer: https://gitlab.example.com
      subject: project_path:megacorp/payments/payments-api:ref_type:branch:ref:main
      audiences: [https://gitlab.example.com]
azureentra.ymlrole-assignments
# What the app may do once signed in, on one registry only. AcrPush pushes and pulls
# images. az acr login also has to look the registry up in Azure Resource Manager, which
# AcrPush doesn't allow; Microsoft documents the second role for that, and it can also
# change or delete the registry, which is one more reason to trust main only.
role_assignments:
  - principal: gitlab-acr-push
    role: AcrPush
    scope: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-delivery-prod/providers/Microsoft.ContainerRegistry/registries/megacorpprod
  - principal: gitlab-acr-push
    role: Container Registry Contributor and Data Access Configuration Administrator
    scope: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-delivery-prod/providers/Microsoft.ContainerRegistry/registries/megacorpprod

The big difference from AWS is the subject. Entra ID matches it exactly, with no wildcards. So each project and branch that may sign in needs its own federated credential, and one application holds at most 20 of them. MegaCorp pushes images to Azure only from each service's main, which needs one credential per service. A wrong subject is saved without complaint: the error shows only when a job tries to sign in.

How a MegaCorp job logs in#

The template runs snippets that call functions from the ci-tools image:

megacorp/devops/ci-templatestemplates/snippets.ymlsnippets
.snippets:
  functions:
    - source /opt/megacorp/lib/ci-lib.sh
  aws_login:
    - mc_aws_login "$AWS_ROLE_ARN"
  acr_push:
    - mc_acr_push "$IMAGE:$CI_COMMIT_SHA"
  maven_settings:
    - mkdir -p .m2
    - mc_maven_settings > .m2/settings.xml
megacorp/devops/ci-toolslib/ci-lib.shaws-login
# Exchange the job's OIDC ID token for temporary AWS credentials.
mc_aws_login() {
  local role="${1:?role ARN required}"
  local creds
  creds=$(aws sts assume-role-with-web-identity \
    --role-arn "$role" \
    --role-session-name "gitlab-${CI_PROJECT_ID}-${CI_JOB_ID}" \
    --web-identity-token "$MC_ID_TOKEN" \
    --duration-seconds 3600 \
    --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
    --output text) || { mc_log "AssumeRoleWithWebIdentity failed for $role"; return 1; }
  read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< "$creds"
  export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
  mc_log "assumed $role"
}

Step by step:

  1. aws sts assume-role-with-web-identity sends the role's ARN and the ID token to STS.
  2. STS checks the token and the trust policy. It returns three values: an access key, a secret key and a session token.
  3. The function exports them as AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. Every later aws command in the job uses them.
  4. They expire after 3600 seconds, which is one hour.

To see which identity a job is really using, add this line. It needs no permissions, and it prints the ARN of the role in use:

aws sts get-caller-identity
megacorp/devops/ci-toolslib/ci-lib.shazure-login
# Sign in to Azure with the job's OIDC ID token, as the Entra application that trusts it.
mc_azure_login() {
  local client_id="${1:?application (client) ID required}"
  az login --service-principal -u "$client_id" -t "$MC_AZURE_TENANT_ID" \
    --federated-token "$MC_ID_TOKEN" --output none \
    || { mc_log "az login failed for $client_id"; return 1; }
  mc_log "signed in to Azure as $client_id"
}

# Push an image that is already built to Azure's registry as well. Only main's images go
# there: Entra matches a federated credential's subject exactly, and aks-prod deploys main.
mc_acr_push() {
  local image="${1:?image required}"
  if [ "${CI_COMMIT_BRANCH:-}" != "$CI_DEFAULT_BRANCH" ]; then
    mc_log "not on $CI_DEFAULT_BRANCH, so not pushing to $MC_ACR_REGISTRY"
    return 0
  fi
  mc_azure_login "$MC_ACR_PUSH_CLIENT_ID"
  az acr login --name "${MC_ACR_REGISTRY%%.*}" --expose-token --query accessToken --output tsv \
    | buildah login --username 00000000-0000-0000-0000-000000000000 --password-stdin "$MC_ACR_REGISTRY"
  buildah push "$image" "docker://$MC_ACR_REGISTRY/${image##*/}"
  mc_log "pushed $MC_ACR_REGISTRY/${image##*/}"
}

Step by step:

  1. On any branch but main, mc_acr_push stops at once. Merge request and tag pipelines never sign in to Azure.
  2. az login --service-principal --federated-token sends the ID token to Entra ID, with the application's client ID and the tenant ID. Both IDs come from the megacorp group's CI/CD variables.
  3. Entra ID compares the token's issuer, subject and audience with the application's federated credentials. If one matches, the CLI is signed in as the application.
  4. az acr login --expose-token trades that sign-in for a registry token, valid for 3 hours. Buildah logs in with it, under the all-zero user name that ACR expects, and pushes.

To see who a job is signed in as, add this line. It prints the subscription in use, and the identity that signed in:

az account show

Logging in to AWS without a script#

The AWS CLI can do the exchange by itself. Put the ID token in a file, and set two variables that name the role and the file:

publish:
  id_tokens:
    AWS_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-web-publish
    AWS_ROLE_SESSION_NAME: gitlab-$CI_JOB_ID
  script:
    - echo "$AWS_ID_TOKEN" > /tmp/web-identity-token
    - export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/web-identity-token
    - aws sts get-caller-identity

Every aws command then assumes the role on its own. This works only when no access keys are set, because keys come first in the order below.

Which credentials each CLI uses#

A job can have credentials from several places at once:

  • credentials that a login step created
  • keys that someone stored in CI/CD settings long ago
  • an identity that belongs to the runner's machine

The two clouds' command-line tools deal with this differently.

The AWS CLI doesn't combine them. It takes the first it finds, in a fixed order:

Which identity will an aws command use? The CLI stops at the first yes.yesnoyesnoyesnoyesnoDoes the command pass --profile, orkeys, on its command line?Those are usedAre AWS_ACCESS_KEY_ID andAWS_SECRET_ACCESS_KEY set?Those keys are used, whether alogin step exported them or theycame from settingsAre AWS_ROLE_ARN andAWS_WEB_IDENTITY_TOKEN_FILE both set?The CLI assumes that role, withthe token in the fileDoes the image contain AWSconfiguration files?Their profile is usedOtherwise the role of the machine orpod is used, often the runner's ownrole
Figure Which identity will an aws command use? The CLI stops at the first yes.#
As text
  1. Does the command pass --profile, or keys, on its command line? Yes: Those are used. No: the next step.
  2. Are AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set? Yes: Those keys are used, whether a login step exported them or they came from settings. No: the next step.
  3. Are AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE both set? Yes: The CLI assumes that role, with the token in the file. No: the next step.
  4. Does the image contain AWS configuration files? Yes: Their profile is used. No: the next step.
  5. Otherwise the role of the machine or pod is used, often the runner's own role

Two failures follow from this order:

  • Old keys in settings win silently. A leftover AWS_ACCESS_KEY_ID in a group's CI/CD variables beats any role, until a login step overwrites it. A job that skips the login then acts as whoever owns those keys.
  • A skipped login falls through to the runner. A job that sets its own before_script replaces the template's, so the login never runs (chapter 14). The job then acts as the runner's role, which at MegaCorp can pull images but not push them.

The Azure CLI doesn't search in an order. It uses the sign-in that az login left in its local cache, under its configuration directory: $HOME/.azure, unless AZURE_CONFIG_DIR names another. Each command runs against the current subscription, which az account set --subscription changes. A machine's managed identity is used only after az login --identity.

Two failures follow:

  • A skipped sign-in usually fails loudly. In a fresh container the cache is empty, so the first az command stops and asks for a sign-in.
  • On a shell runner, the last sign-in lingers. The cache lives in the runner user's home directory, so it survives from one job to the next. A job that skips its own sign-in can act as whoever signed in before it. Give each job its own directory with AZURE_CONFIG_DIR, such as one inside the build folder.

Several accounts and subscriptions#

Large organisations split each cloud into several spaces, often one for shared tools and one for each environment.

A job can reach another AWS account in two ways:

Reaching several AWS accounts. Each account's role can trust the ID token directly. Role chaining, from one role to the next, also works, but its keys last at most one hour. a job with an ID token tooling account · 123456789012 IAM role gitlab-ecr-push Amazon ECR the images ID token or chain roles: 1 hour at most staging account · 111122223333 IAM role deploy-staging ECS or EKS the staging app ID token production account · 444455556666 IAM role deploy-production ECS or EKS the production app ID token
Figure Reaching several AWS accounts. Each account's role can trust the ID token directly. Role chaining, from one role to the next, also works, but its keys last at most one hour.#
  • Direct trust. Each account has a role whose trust policy accepts the ID token, and the job logs in to the account it needs. Each account checks the token's sub itself, so production can accept only release tags, for example.
  • Role chaining. The job logs in to one role, then uses it to assume a role in the other account. AWS limits a chained session to one hour, whatever the role allows, and asking for longer fails.

In Azure, one sign-in reaches every subscription in the tenant where the identity has a role assignment. az account list shows them, and az account set --subscription picks one. There's no chaining to worry about: what the identity may do in each subscription depends only on its role assignments there.

MegaCorp needs none of this. Its pipelines touch only the tooling account, 123456789012, where ECR and the website's bucket live, and one registry in Azure. Deploys go through the GitOps repository, and a controller inside each cluster pulls the new image (chapter 45). The pipeline never holds a production cluster's credentials at all.

Registries: pushing images, and pulling job images#

An ECR registry's address contains the account and the region: 123456789012.dkr.ecr.eu-west-2.amazonaws.com. To push, a job logs in to AWS first, then to the registry:

aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"

The user name is always AWS, and the password this prints is valid for 12 hours.

An ACR registry's address is its name followed by .azurecr.io: megacorpprod.azurecr.io. To push, a job signs in to Azure first, then asks for a registry token:

az acr login --name megacorpprod --expose-token --query accessToken --output tsv \
  | buildah login --username 00000000-0000-0000-0000-000000000000 --password-stdin megacorpprod.azurecr.io

The user name is always the all-zero GUID, and the token is valid for 3 hours. Without --expose-token, az acr login hands the token to a Docker daemon itself, which a Buildah job doesn't have. The registry's admin user is off by default, and Microsoft advises leaving it off.

Tagging and promoting images is the subject of chapter 39.

Pulling a job's own image: from either registry is different. The runner pulls it before the job starts, so nothing in the job's script can help. The runner needs its own way in.

Job images from ECR or ACR#

For ECR, GitLab's documented way is the ECR credential helper. It is a small program that gets a registry password from AWS whenever the runner needs one. It must be installed on the runner, and the runner needs AWS credentials of its own. A DOCKER_AUTH_CONFIG variable then tells the runner to use it:

{ "credHelpers": { "123456789012.dkr.ecr.eu-west-2.amazonaws.com": "ecr-login" } }

GitLab's documentation notes that credential helpers aren't available on instance runners, or on any runner whose machine you can't configure.

For ACR, a common way is a registry token: a name and password that ACR creates, tied to a scope map that allows pulls from chosen repositories only. Tokens exist in every ACR tier, and work with an ordinary registry login, so they fit in DOCKER_AUTH_CONFIG like any other password.

MegaCorp avoids the question: its job images come from registry.example.com, not from either cloud.

Secrets from AWS and Azure#

A secret in a cloud's vault can reach software in three ways. The best one keeps it out of the pipeline completely:

RouteWho reads the secretWhat the job sees
The secrets: keyword, with aws_secrets_manager or azure_key_vault (Premium)the runner, as the job startsa file holding the value; for AWS, the value itself with file: false
The job's script calls the cloud's CLIthe jobthe value, which GitLab can't mask because it never saw it
The running service reads it: through an ECS task definition, or the External Secrets Operator on a clusterthe service, when it startsnothing at all

With the keyword, the runner does the fetching. It logs in with an ID token named AWS_ID_TOKEN and the role in AWS_ROLE_ARN. The region comes from AWS_REGION, which MegaCorp's base template sets for every job:

db-migrate:
  id_tokens:
    AWS_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/payments-db-migrate
  secrets:
    DB_PASSWORD:
      aws_secrets_manager:
        secret_id: payments/db
        field: password
  script:
    - ./migrate.sh --password-file "$DB_PASSWORD"

Without an ID token, the runner uses its own AWS role instead. On the Kubernetes executor, that role must be on the runner manager, the part that starts the job pods. If it is only on the pods, the job fails with no EC2 IMDS role found.

From a script, the commands look like this. Capture the value in a variable, and never print it:

DB_PASSWORD=$(aws secretsmanager get-secret-value --secret-id payments/db --query SecretString --output text)
DB_URL=$(aws ssm get-parameter --name /payments/db-url --with-decryption --query Parameter.Value --output text)

The third route is the cleanest. An ECS task definition can name the secret, and ECS reads it when the container starts, using the task's execution role:

"secrets": [
  { "name": "DB_PASSWORD",
    "valueFrom": "arn:aws:secretsmanager:eu-west-2:123456789012:secret:payments/db-AbCdEf" }
]

The pipeline deploys a task definition that holds only the secret's name. The value never passes through GitLab.

With the keyword, the runner signs in as the application in AZURE_CLIENT_ID, with the ID token that token: names, and reads the vault at AZURE_KEY_VAULT_SERVER_URL:

db-migrate:
  id_tokens:
    AZURE_JWT:
      aud: https://gitlab.example.com
  variables:
    AZURE_KEY_VAULT_SERVER_URL: https://megacorp-prod-kv.vault.azure.net
    AZURE_CLIENT_ID: 00000000-0000-0000-0000-000000000000
    AZURE_TENANT_ID: 00000000-0000-0000-0000-000000000000
  secrets:
    DB_PASSWORD:
      token: $AZURE_JWT
      azure_key_vault:
        name: payments-db-password
        version: 00000000000000000000000000000000
  script:
    - ./migrate.sh --password-file "$DB_PASSWORD"

The runner writes the value to a temporary file, and DB_PASSWORD holds the file's path. version names one version of the secret: a GUID written without dashes.

From a script, after az login:

DB_PASSWORD=$(az keyvault secret show --vault-name megacorp-prod-kv --name payments-db-password --query value --output tsv)

The third route is the cleanest here too. On MegaCorp's clusters, the External Secrets Operator copies the secret from Key Vault into the service's namespace, with its own identity (the External Secrets Operator). The value never passes through GitLab.

The usual deploy targets#

Most deploys are one of these. Each is a few CLI commands, run with the short-lived access the job got:

TargetWhat the job runsWhere to look when it fails
Static site on S3 and CloudFrontaws s3 sync, then aws cloudfront create-invalidationif visitors still see old files, check that the invalidation ran
ECSa new task definition that names the new image, then aws ecs update-service and aws ecs wait services-stablethe wait checks every 15 seconds and gives up after 40 checks, with exit code 255; the service's events show why new tasks fail
EKSaws eks update-kubeconfig, then kubectl or helm, or a GitOps handoffbesides its AWS permissions, the role needs access inside the cluster, which the cluster's owners grant
Lambdaaws lambda update-function-code, with a new image or zip filethe role's permission on that function, then the function's own logs
Infrastructure as codeOpenTofu or Terraform, with state stored in GitLab; or CloudFormationthe role needs every permission the change needs, so guard it with protected branches and environments
TargetWhat the job runsWhere to look when it fails
AKSaz aks get-credentials, then kubelogin convert-kubeconfig -l azurecli, then kubectl or helm; or a GitOps handoffbesides its Azure roles, the identity needs access inside the cluster, which the cluster's owners grant
Container Appsaz containerapp update --name … --resource-group … --image …, which starts a new revision of the appthe app's revisions, then the new revision's logs
Infrastructure as codeOpenTofu or Terraform with the azurerm provider, with state stored in GitLabthe identity needs every permission the change needs, so guard it with protected branches and environments

MegaCorp's website job is the first AWS kind:

megacorp/devops/ci-templatestemplates/node.ymlpublish-site
.publish-site:
  extends: .megacorp-base
  stage: deploy
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-web-publish
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - aws s3 sync dist/ "s3://$SITE_BUCKET/" --delete
    - aws cloudfront create-invalidation --distribution-id "$CF_DISTRIBUTION_ID" --paths '/*'
  environment:
    name: production
    url: https://www.example.com

GitLab's ready-made AWS templates#

GitLab ships templates for AWS, and some older pipelines use them:

  • AWS/Deploy-ECS.gitlab-ci.yml builds an image into GitLab's registry, creates a new task definition revision, and updates an ECS service. You set CI_AWS_ECS_CLUSTER, CI_AWS_ECS_SERVICE and a task definition variable. The deploy waits for the rollout unless CI_AWS_ECS_WAIT_FOR_ROLLOUT_COMPLETE_DISABLED is set.
  • AWS/CF-Provision-and-Deploy-EC2.gitlab-ci.yml creates a CloudFormation stack, pushes the build to S3, and deploys it to EC2.
  • Both expect access keys stored as the CI/CD variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION. GitLab's own page says ID tokens are more secure, but don't work with these templates.
  • GitLab's image registry.gitlab.com/gitlab-org/cloud-deploy/aws-base contains the AWS CLI. If its output fails on a non-ASCII character, set LANG: "UTF-8".

Small things that bite#

  • The amazon/aws-cli image. Its entrypoint is the aws command itself. On the Docker executor, which keeps an image's entrypoint, it can't run your script unless you set entrypoint: [""]. The Kubernetes executor ignores entrypoints by default, so the same job can work on one runner and fail on another (chapter 30).
  • The region. AWS commands without --region use AWS_REGION, which MegaCorp sets to eu-west-2. A resource in another region looks as if it doesn't exist.
  • AWS keys expire after an hour. MegaCorp asks for 3600 seconds. A job that runs longer after logging in gets expired-token errors part-way through.
  • The ID token can expire first. Log in at the start of the job, not after a long build.
  • The cloud must reach GitLab. AWS's STS and Microsoft Entra ID both download GitLab's signing keys from the instance. A self-managed GitLab that the internet can't reach gets InvalidIdentityToken from AWS. GitLab documents a workaround for AWS that publishes the keys elsewhere; for Azure, its page says the instance must be publicly accessible.
  • A new federated credential takes a few minutes. Until it has spread through Entra ID, sign-ins can fail with AADSTS70021, as if it didn't exist.

When AWS or Azure says no#

Read the error first, because it names the step that failed. Then ask these in order:

A job's AWS call is refused. Ask these in order; the first yes is usually the cause.yesnoyesnoyesnonoyesyesnoyesnoDoes the login fail withInvalidIdentityToken?AWS can't fetch GitLab's signingkeys; can the internet reach theinstance?Does it say "Not authorized to performsts:AssumeRoleWithWebIdentity"?Compare the token's sub and audwith the trust policyDoes the trust policy put a * insideStringEquals?Wildcards only work withStringLikeDid the login step run at all?Something replaced before_script,so another identity is in useDoes get-caller-identity show anunexpected role?Other credentials come first; seewhich credentials the CLI usesDid the job run for over an hour afterlogging in?The keys expired; log in again, orsplit the jobThe role lacks permission for thataction, and its owners must add it
Figure A job's AWS call is refused. Ask these in order; the first yes is usually the cause.#
As text
  1. Does the login fail with InvalidIdentityToken? Yes: AWS can't fetch GitLab's signing keys; can the internet reach the instance?. No: the next step.
  2. Does it say "Not authorized to perform sts:AssumeRoleWithWebIdentity"? Yes: Compare the token's sub and aud with the trust policy. No: the next step.
  3. Does the trust policy put a * inside StringEquals? Yes: Wildcards only work with StringLike. No: the next step.
  4. Did the login step run at all? No: Something replaced before_script, so another identity is in use. Yes: the next step.
  5. Does get-caller-identity show an unexpected role? Yes: Other credentials come first; see which credentials the CLI uses. No: the next step.
  6. Did the job run for over an hour after logging in? Yes: The keys expired; log in again, or split the job. No: the next step.
  7. The role lacks permission for that action, and its owners must add it
A job's Azure sign-in or push is refused. Ask these in order; the first yes is usually the cause.yesnoyesnoyesnonoyesyesnoIs the pipeline on a branch other thanmain?mc_acr_push skips Azure there bydesign; nothing was refusedDoes az login fail with AADSTS70021?No federated credential matchedthe token's issuer and subject;compare them character bycharacter, or wait a few minutesafter adding oneDoes the error blame the audience, asAADSTS700212 does?The credential's audience differsfrom aud in the job's id_tokensCan the internet reach the GitLabinstance?Entra ID can't fetch GitLab'ssigning keysDoes az acr login fail to find theregistry?The identity can't read theregistry in Azure ResourceManager; it needs the roleMicrosoft documents for az acrloginThe push itself is refused: theidentity lacks AcrPush on the registry
Figure A job's Azure sign-in or push is refused. Ask these in order; the first yes is usually the cause.#
As text
  1. Is the pipeline on a branch other than main? Yes: mc_acr_push skips Azure there by design; nothing was refused. No: the next step.
  2. Does az login fail with AADSTS70021? Yes: No federated credential matched the token's issuer and subject; compare them character by character, or wait a few minutes after adding one. No: the next step.
  3. Does the error blame the audience, as AADSTS700212 does? Yes: The credential's audience differs from aud in the job's id_tokens. No: the next step.
  4. Can the internet reach the GitLab instance? No: Entra ID can't fetch GitLab's signing keys. Yes: the next step.
  5. Does az acr login fail to find the registry? Yes: The identity can't read the registry in Azure Resource Manager; it needs the role Microsoft documents for az acr login. No: the next step.
  6. The push itself is refused: the identity lacks AcrPush on the registry

To compare sub with the trust policy or the federated credential, print the token's claims, not the token. A token has three parts separated by dots, and the middle part holds the claims. GitLab's documentation suggests this command:

echo "$MC_ID_TOKEN" | cut -d '.' -f2 | base64 -d | jq .

The claims include the email address of the person who started the pipeline, so remove the line when you have your answer. base64 may also print base64: invalid input and return an error, because a token leaves out the padding that base64 expects. It happened with two tokens in four on the local GitLab 19.3, and the claims still printed in full.

When you write it
Do this
For each AWS role, write the trust policy's subject condition for exactly the refs that need it, for example project_path:megacorp/payments/payments-api:ref_type:branch:ref:main. In Entra ID, add one federated credential for each project and branch that must sign in, spelled exactly.
Why
The cloud compares the token's subject with that condition, character by character. Anything that does not match is refused.
Watch out
Test the role from tag pipelines and merge request pipelines too, because their subjects differ from a branch's. The card refused in some pipelines shows the symptom.
What you can now do
  1. Sign a job in to AWS or Azure with an ID token and no stored key, and read the token's claims.
  2. Write a trust policy or federated credential that matches exactly the pipelines that should sign in.
  3. Read a refused sign-in and say which claim didn't match.
Spot the bugThe release that couldn't push#

payments-api tags v1.5.0. In the tag pipeline, image-build fails during the login:

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

The same job pushed an image from main an hour earlier, and from a merge request pipeline this morning. This is the push role:

awsiam.ymlecr-push-role
# Assumed by image-build, and by the ecr-push component, through mc_aws_login.
ecr_push_role:
  arn: arn:aws:iam::123456789012:role/gitlab-ecr-push
  trust_policy:
    Effect: Allow
    Principal:
      Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
    Action: sts:AssumeRoleWithWebIdentity
    Condition:
      StringEquals:
        gitlab.example.com:aud: https://gitlab.example.com
      StringLike:
        gitlab.example.com:sub: project_path:megacorp/*:ref_type:branch:ref:*
  permissions: push and pull images in every ECR repository in the account
QuestionWhy is only the tag pipeline refused, and how would you change the role without opening it to every tag?
Show the answer

A tag pipeline's ID token has a subject that ends in ref_type:tag:ref:v1.5.0. The pattern accepts only ref_type:branch, so STS refuses. Pipelines on main, and merge request pipelines, match because both run on branches.

Add a second pattern for release tags, such as project_path:megacorp/*:ref_type:tag:ref:v*. IAM accepts a list of values for one condition, and any match is enough. payments-api protects tags matching v*, which limits who can create them. On self-managed GitLab, AWS can't check ref_protected, so the pattern is the only guard.

While you are there, notice that ref:* lets any branch push images, including unreviewed feature branches. See how AWS checks an ID token.

Spot the bugThe branch Azure didn't know#

The payments team starts releasing from branches such as release/2026-09. At its request, the platform team changes mc_acr_push so that release/* branches push to ACR too. The first pipeline on release/2026-09 fails in image-build:

AADSTS70021: No matching federated identity record found for presented assertion.

Pushes from main still work, and nobody has touched Entra ID.

azureentra.ymlapp-registration
# The identity that image jobs sign in as. Each federated credential names one kind of
# pipeline that may use it.
acr_push_app:
  display_name: gitlab-acr-push
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: 00000000-0000-0000-0000-000000000000
  federated_credentials:
    - name: payments-api-main
      issuer: https://gitlab.example.com
      subject: project_path:megacorp/payments/payments-api:ref_type:branch:ref:main
      audiences: [https://gitlab.example.com]
QuestionWhy is the release branch refused, and what are the choices?
Show the answer

The job's token has the subject project_path:megacorp/payments/payments-api:ref_type:branch:ref:release/2026-09. The application's only federated credential names ref:main. Entra ID compares subjects exactly, with no wildcards, so nothing matches and the sign-in fails.

A credential for release/2026-09 would work, but only for that branch, and each release would need another, up to 20 per application. The simpler choice is to keep pushing to Azure only from main, and release the image that main built (from build to deploy). See what must match.

See no matching federated credential.

Images to ECR and ACR#

Part IX · Delivery · Chapter 39 · 7 min read

MegaCorp builds each service's image in the pipeline, tags it with the commit's SHA, and pushes it to Amazon ECR. On main, it pushes the same image to Azure Container Registry too, for aks-prod. The deploy jobs receive that exact reference. The goal is build once, promote many: the image that staging tested is the image that production runs.

The common way to break that goal is to build again later, for example in a tag pipeline. A rebuild is a new image, even from the same commit.

Friday, 10:00. The tag already exists.

A retried push to ECR fails with ImageTagAlreadyExistsException. The first attempt had pushed the image and then timed out on the last step. The tag is right. The registry is protecting you from something; what? (the card)

How an image is named#

An image reference has three parts: the registry, the repository and a tag. MegaCorp's look like this:

123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:3f9c1e2…
└──────────── registry ─────────────────────┘ └─ repository ─┘ └ tag ┘
megacorpprod.azurecr.io/payments-api:3f9c1e2…
└───── registry ──────┘ └─ repository ─┘ └ tag ┘

A tag is only a label, and labels can move. A commit SHA as the tag ties the image to exactly one commit, which is why MegaCorp uses $CI_COMMIT_SHA. A tag such as latest says nothing about what is inside, and changes with every push.

From build to deploy#

image-build logs in to AWS and ECR, builds with Buildah, and pushes. On main, it pushes the same image to ACR as well, under the same repository and tag (chapter 38). Then it writes the ECR reference to a dotenv report as IMAGE_REF (chapter 31). The deploy jobs pass it on to the deployer:

megacorp/devops/ci-templatestemplates/deploy.ymldeploy-trigger
.deploy:
  stage: deploy
  trigger:
    project: megacorp/platform/deployer
    branch: main
    strategy: depend
  variables:
    APP: $CI_PROJECT_NAME
    IMAGE_REF: $IMAGE_REF

The deployer writes only the tag into the environment repository. Each cluster's values name its own registry, so eks-prod pulls the image from ECR, and aks-prod pulls the same tag from ACR (from tag to pod).

Build once, promote many. Top: one image, tagged by commit, moves from staging to production. Bottom: MegaCorp's tag pipeline builds the same commit again, so production can get an image that staging never tested. BUILD ONCE, PROMOTE MANY main pipeline image-build ECR: payments-api tag 3f9c1e2… staging deploy-staging production deploy-prod ✓ Production runs the exact image that staging tested: the same tag, and the same contents. WHAT MEGACORP'S TAG PIPELINE DOES main pipeline image-build makes image A tag v1.5.0 pipeline image-build makes image B ECR: payments-api tag 3f9c1e2… the same tag, pushed twice staging tested image A production gets image B, never tested Mutable tags: B silently replaces A. Immutable tags: the second push fails with ImageTagAlreadyExistsException.
Figure Build once, promote many. Top: one image, tagged by commit, moves from staging to production. Bottom: MegaCorp's tag pipeline builds the same commit again, so production can get an image that staging never tested.#

payments-api's tag pipeline runs maven-build and image-build again before deploy-prod. The release tag usually points at a commit that main already built and staging already tested. The rebuild pushes a second image under the same SHA tag. Base images and libraries may have changed in the meantime, so it is not the image that was tested. What happens next depends on the ECR repository:

  • Mutable tags: the new image quietly replaces the tested one.
  • Immutable tags: the push fails with ImageTagAlreadyExistsException.

ACR never sees the rebuild. A tag pipeline isn't on main, so image-build skips the push to Azure, and aks-prod keeps the image that main built. With mutable tags in ECR, the two clusters would then run different images under the same tag.

Promoting without rebuilding#

To promote, deploy the reference that was already built and tested. To give the same image a friendly name as well, add a tag in ECR without pulling or pushing anything:

MANIFEST=$(aws ecr batch-get-image --repository-name payments-api \
  --image-ids imageTag="$CI_COMMIT_SHA" --query 'images[].imageManifest' --output text)
aws ecr put-image --repository-name payments-api \
  --image-tag "$CI_COMMIT_TAG" --image-manifest "$MANIFEST"

Now v1.5.0 and the commit's SHA name the same image. With immutable tags, ECR refuses any attempt to move either of them.

ACR's tags are mutable by default. Its owners can lock a released tag, so that nobody can overwrite or delete it:

az acr repository update --name megacorpprod --image payments-api:3f9c1e2… --write-enabled false
Which image is production really running? Ask these in order.yesnoyesnonoyesDoes the deploy use a tag such aslatest?You can't tell from the tag; lookup the image's digest in theregistryWas the same tag pushed by more thanone pipeline?With mutable tags, the last pushwon; compare the pipelines' timesDid the deploy receive IMAGE_REF fromthe build it followed?It may be deploying an olderreference; check the trigger'svariablesThe tag names one image; check thecluster really runs that tag
Figure Which image is production really running? Ask these in order.#
As text
  1. Does the deploy use a tag such as latest? Yes: You can't tell from the tag; look up the image's digest in the registry. No: the next step.
  2. Was the same tag pushed by more than one pipeline? Yes: With mutable tags, the last push won; compare the pipelines' times. No: the next step.
  3. Did the deploy receive IMAGE_REF from the build it followed? No: It may be deploying an older reference; check the trigger's variables. Yes: the next step.
  4. The tag names one image; check the cluster really runs that tag
When you write it
Do this
Tag each image with the commit it was built from, such as $CI_COMMIT_SHORT_SHA, and deploy that one tag to every environment.
Why
One image, tested once, is what runs everywhere. Rebuilding for each environment or each cloud ships something nobody tested.
Watch out
With immutable tags in ECR, pushing the same tag again fails, so a retried job fails. The card the tag already exists shows it. In ACR, ask for the tags you release to be locked.
What you can now do
  1. Push an image to ECR or ACR from a job, named and tagged so that a deploy can find it.
  2. Explain immutable tags, and make a retry safe.
  3. Deploy one tested tag everywhere instead of rebuilding per environment.
Spot the bugTwo images, one tag#

Commit 3f9c1e2 was built on main, deployed to staging, and tested for a week. Then it was tagged v1.5.0. The tag pipeline built and pushed payments-api:3f9c1e2… again, and deploy-prod shipped it. In production, a bug appears that staging never showed. At the time, payments-api's ECR repository had mutable tags. The platform team has made them immutable since.

megacorp/devops/ci-templatespipelines/java-service.ymljobs
maven-build:
  extends: .maven-build

maven-test:
  extends: .maven-test

image-build:
  extends: .image-build
  needs: [maven-build]
QuestionHow can production run code that staging never tested, when both used the same tag?
Show the answer

image-build runs in tag pipelines too, so the release built the commit a second time. That rebuild picked up whatever base image and libraries were current, made a different image, and pushed it under the same tag. With mutable tags it replaced the tested image. Production then deployed the new one.

Build once: skip image-build in tag pipelines, and deploy the image that main built. Add the release tag with put-image. Turning on tag immutability in ECR makes a rebuild fail loudly instead of replacing the image. See from build to deploy.

Environments and deployments#

Part IX · Delivery · Chapter 40 · 6 min read

An environment is GitLab's record of a place you deploy to: a name, a URL, and a history of deployments. A job with environment: becomes a deployment job, and that unlocks five safeguards:

  • who may deploy there
  • approvals before a deploy runs
  • one deploy at a time
  • refusing outdated deploys
  • re-deploying an earlier version

At MegaCorp every environment lives in the deployer project, so that is where to look when a deploy won't start.

Friday, 14:25. A deploy that waits for nothing.

deploy-staging sits in waiting for resource. Nothing else in its pipeline is running. Another pipeline, from a colleague's merge, finished deploying to staging ten minutes ago, and its job is green. The resource it waits for is held by a job that has already ended. (the card)

Environments and deployments#

MegaCorp's deployer names each environment after the target and the service:

megacorp/platform/deployer.gitlab-ci.ymldeploy
variables:
  ENVIRONMENT: $[[ inputs.environment ]]

deploy:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  resource_group: $APP-$ENVIRONMENT
  environment:
    name: $ENVIRONMENT/$APP
  script:
    - mc deploy "$ENVIRONMENT" "$IMAGE_REF"
  rules:
    - if: $IMAGE_REF == null
      when: never
    - if: $ENVIRONMENT == "production"
      when: manual
    - when: on_success

Each run of this job records a deployment of staging/payments-api or production/payments-api. Its history appears under Operate › Environments. The names come from variables, so one job serves every service. The resource_group lets only one deploy per service and environment run at a time (chapter 24).

An environment also scopes variables. A CI/CD variable limited to the production/* environments reaches only jobs deploying there (chapter 26).

A deployment here records the deploy job, not the running software. At MegaCorp the job succeeds once mc deploy has written the new tag into the environment repository. The cluster rolls it out afterwards, so a green deployment can still end in pods that never start. Chapter 41 follows the rest of the way, and chapter 52 shows how to tell whether it worked.

Who may deploy, and approvals Premium#

  • Protected environments. A Maintainer can protect an environment, and list who is Allowed to deploy: roles, users or groups. Anyone else's deploy job to it can't run.
  • Deployment approvals. A protected environment can require approvals. Its deployments are blocked until all of them are given, under Operate › Environments, on the deployment's status badge. Approving doesn't start the job: someone must still run it.
Note
A trigger job, such as MegaCorp's .deploy, can use deployment-only access to a protected environment only if it has an environment: of its own. Without one, GitLab treats it as an ordinary job, with ordinary permissions.

One at a time, newest wins#

Two pipelines can race. The newer one deploys first, and then the older one finishes and puts the old version back. The project setting Prevent outdated deployment jobs stops that. An older deployment job then fails with:

The deployment job is older than the latest deployment, and therefore failed.

GitLab judges a job's age by when it started, not by its commit. So running an old pipeline's manual deploy blocks a newer pipeline's manual deploy.

Rolling back#

Under Operate › Environments, each earlier successful deployment offers Rollback environment, and the latest offers Re-deploy to environment. Both simply run that deployment's job again. That works only if the job holds everything the deploy needs. At MegaCorp it does: re-running the job writes the old IMAGE_REF's tag into the environment repository, straight to main for staging, and as a merge request for production (promotion by merge request). Reverting the deploy's commit in that repository does the same, and leaves a clearer history (drift, and going back).

A deploy job won't run. Ask these in order.yesnoyesnoyesnoyesnoyesnoIs it waiting for approvals?Approve it under Operate ›Environments, then run it;approval doesn't start itIs the environment protected, and youaren't allowed to deploy?Ask a Maintainer to add you, orsomeone allowed to run itIs another deploy in the same resourcegroup running?It waits its turnIs it older than the latest deployment?Prevent outdated deployment jobsblocks it; run a newer pipelineIs there a deploy freeze?GitLab blocks deployments untilthe freeze endsCheck its rules: a manual job waits forsomeone to run it
Figure A deploy job won't run. Ask these in order.#
As text
  1. Is it waiting for approvals? Yes: Approve it under Operate › Environments, then run it; approval doesn't start it. No: the next step.
  2. Is the environment protected, and you aren't allowed to deploy? Yes: Ask a Maintainer to add you, or someone allowed to run it. No: the next step.
  3. Is another deploy in the same resource group running? Yes: It waits its turn. No: the next step.
  4. Is it older than the latest deployment? Yes: Prevent outdated deployment jobs blocks it; run a newer pipeline. No: the next step.
  5. Is there a deploy freeze? Yes: GitLab blocks deployments until the freeze ends. No: the next step.
  6. Check its rules: a manual job waits for someone to run it
When you write it
Do this
Give every deploy job an environment, make the production job when: manual, and add a resource_group. On Premium, protect the production environment as well.
Why
GitLab then records what runs where, a person decides when production changes, and two deploys to the same environment never run at the same time.
Watch out
With a resource group, a second deploy waits until the first has finished, and the pipeline looks stuck while it does. The card a deploy waits explains it.
What you can now do
  1. Give a deploy job an environment and a resource group, and read what GitLab records for each deployment.
  2. Roll back from Operate › Environments, and know what a rollback reruns.
  3. Explain a deploy that waits, and find the job holding its resource.
Spot the bugApproved, but not deployed#

The production environment in megacorp/platform/deployer needs one approval. A release manager approves a payments-api deployment at 17:55 and goes home. At 09:00 the next day, production still runs the old version. The deployer pipeline shows the deploy job waiting.

megacorp/platform/deployer.gitlab-ci.ymldeploy
variables:
  ENVIRONMENT: $[[ inputs.environment ]]

deploy:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  resource_group: $APP-$ENVIRONMENT
  environment:
    name: $ENVIRONMENT/$APP
  script:
    - mc deploy "$ENVIRONMENT" "$IMAGE_REF"
  rules:
    - if: $IMAGE_REF == null
      when: never
    - if: $ENVIRONMENT == "production"
      when: manual
    - when: on_success
QuestionWhy didn't the deploy happen after the approval?
Show the answer

Production deploys are when: manual in the deployer's rules. A deployment approval unblocks the job, but never starts it. Someone allowed to deploy must still select Run. The approval was needed, but it wasn't enough.

Make running the job part of the release manager's routine. See who may deploy, and approvals.

The delivery map#

Part X · After the pipeline · Chapter 41 · 15 min read

Your pipeline ends when the image is in the registry. Four more steps happen before anyone uses the new version, and GitLab sees only the first. The new tag is written into a GitOps repository. A controller in the cluster notices, renders the chart or overlay, and applies it. Kubernetes replaces the old pods with new ones. Then the running service sends back signals: logs, metrics and traces. This part follows those steps, one chapter each, and this chapter is the map. When a deploy is missing, walk the map from left to right.

Monday, 09:00. Green, and still the old version.

The deploy pipeline is green. Staging serves last week's build. The job that "deployed" ran mc deploy, printed done, and exited zero. Somewhere between that job and a running pod is a whole system that GitLab never sees. You need the map of it. (the card)

From commit to cluster, and back#

The delivery map. Your pipeline pushes an image, and the new tag is written into a GitOps repository. A controller renders and applies it, and the cluster swaps old pods for new ones. The service's signals come back to you. GitLab shows the steps up to the commit. GitLab shows these, up to the commit GitLab can't see these your team the cloud platform, your values platform team cloud and platform your team your pipeline builds, tests, pushes an image the registry image:tag ECR or ACR GitOps repo the new tag, in a values file the controller Flux or Argo CD: renders, applies the cluster swaps old pods for new ones your service runs, and reports signals: logs, metrics and traces to dashboards and alerts you and on-call A deploy that's missing is stuck at one of these steps: walk them from left to right.
Figure The delivery map. Your pipeline pushes an image, and the new tag is written into a GitOps repository. A controller renders and applies it, and the cluster swaps old pods for new ones. The service's signals come back to you. GitLab shows the steps up to the commit.#
StepWhat happensWho owns it
build and pushyour pipeline builds the image and pushes it, tagged, to a registryyour team
request the deploya job writes the new tag into a values file in the GitOps repositoryyour team, often through a central job
synca controller in the cluster sees the change, renders it, and applies the resultthe platform team runs the controller; the values are yours
roll outKubernetes starts pods from the new image, checks them, and stops the old onesthe cluster: the cloud and the platform team
run and reportthe service runs, and sends logs, metrics and traces to where people watchyour team

Your team owns the two ends: the service and its pipeline, and what the service reports. The platform team owns the middle: the clusters, the controllers, the add-ons they install, and the shape of the GitOps repository. The cloud provider runs the managed Kubernetes, the registry and the identities. So a failure in the middle is usually found by you and fixed with them. The chapters that follow give you enough to find it.

Push and pull#

A pipeline can deploy in two ways:

  • Push: a job changes the target itself, for example by running helm upgrade against a cluster, or updating a cloud service.
  • Pull, or GitOps: a job writes the new version into a Git repository, and a controller running inside the cluster notices and applies it.

The two patterns ask different things of the pipeline, and tell it different things back. With push, the pipeline needs credentials for the cluster or the cloud service, and when the job finishes you know the change was applied, and ready, if the job waited for it. With pull, the pipeline needs write access to the GitOps repository and nothing else, and when the job finishes you know only that the change was committed.

Pull keeps production credentials out of the pipeline, since only the controller touches the cluster. It also means that Git holds the truth: to see what should be running, read the repository; to go back, revert a commit. The price is that the pipeline can't see the deploy happen. MegaCorp pulls.

MegaCorp's delivery side#

MegaCorp's delivery repositories. payments-api's pipeline pushes images to ECR and ACR and triggers the deployer, which writes the tag into gitops-config. Both clusters pull their configuration from gitops-config and the chart from charts; infra-modules creates the clusters themselves. payments/payments-api builds and pushes images ECR ACR platform/deployer mc deploy tag platform/gitops-config clusters/ apps/ infrastructure/ envs/ values platform/charts the service chart platform/infra-modules OpenTofu modules eks-prod, on AWS Flux: add-ons, config Argo CD: payments-api in staging and production aks-prod, on Azure Flux: add-ons, config Argo CD: payments-api in staging and production the chart, pulled by Argo CD both clusters pull these creates clusters, registries and identities, in both clouds
Figure MegaCorp's delivery repositories. payments-api's pipeline pushes images to ECR and ACR and triggers the deployer, which writes the tag into gitops-config. Both clusters pull their configuration from gitops-config and the chart from charts; infra-modules creates the clusters themselves.#
RepositoryOwned byHoldsReaches
megacorp/payments/payments-apithe payments teamthe service and its pipeline fileECR and ACR, with images; the deployer, with a trigger
megacorp/platform/deployerthe platform teamthe deploy pipeline, which runs mc deploygitops-config: a commit for staging, a merge request for production
megacorp/platform/gitops-configthe platform team; teams change their own valueseach cluster's Flux setup, the add-ons, the ApplicationSets, and each service's valueseks-prod and aks-prod, which pull from it
megacorp/platform/chartsthe platform teamthe shared service chartevery ApplicationSet, at a tagged version
megacorp/platform/infra-modulesthe platform teamOpenTofu modules for clusters, registries and identitiesthe AWS account and the Azure subscription

The last step that GitLab shows is the deployer's job. It runs mc deploy, which writes the new tag into the environment's values file in gitops-config: straight to main for staging, and in a merge request for production:

megacorp/devops/ci-toolsbin/mcdeploy
deploy)
  env="${1:?usage: mc deploy <environment> <image-ref>}"
  image="${2:?usage: mc deploy <environment> <image-ref>}"
  tag="${image##*:}"
  mc_log "promoting ${APP} ${tag} to ${env} through the GitOps repository"
  git clone --depth 1 \
    "https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
  yq -i ".image.tag = \"${tag}\"" "gitops/envs/${env}/${APP}.yaml"
  git -C gitops commit -am "deploy ${APP} ${tag} to ${env}"
  if [ "$env" = production ]; then
    # production changes wait in a merge request until someone merges them
    git -C gitops push origin "HEAD:deploy/${APP}-${tag}" \
      -o merge_request.create -o merge_request.target=main \
      -o merge_request.title="Deploy ${APP} ${tag} to production"
  else
    git -C gitops push origin HEAD:main
  fi
  ;;

Both clusters run the same way. The platform team's Flux installs the add-ons, including Argo CD, from gitops-config. The payments team's services are then deployed by Argo CD, one Application for each environment. The chapters on chapter 43 and chapter 44 take the files apart.

Five questions for a deployment#

The decoding method asks five questions of a job (chapter 55). They work for a deployment too, with different answers:

The five questions, asked of a deployment: where it is defined, when it deploys, where it runs, what it runs, and what goes in and out. Where is it defined? the chart, the values and the overlays, in the GitOps repository When does it deploy? after a merge, when the controller next syncs Where does it run? a cluster, a namespace, and some nodes What does it run? an image, named by tag and digest, its command and its config What goes in and out? in: config and secrets out: logs, metrics and traces
Figure The five questions, asked of a deployment: where it is defined, when it deploys, where it runs, what it runs, and what goes in and out.#
QuestionFor a jobFor a deployment
Where is it defined?the merged configurationthe chart, the values and the overlays, in the GitOps repository
When does it run?the pipeline's source and the job's rulesafter a merge, when the controller next syncs
Where does it run?a runner, with its tagsa cluster, a namespace, some nodes
What does it run?its image and scriptan image, named by tag and digest, with its command and config
What goes in and out?variables and artifactsin: config and secrets; out: logs, metrics and traces

When a merged change never reaches production, ask the first two in order. Read the GitOps repository, then the controller's sync status, then the pods' events. The rest of this part shows where each of those is.

What you can now do
  1. Draw the path from an image in the registry to a pod that serves requests, and name who owns each piece.
  2. Say what a green deploy job actually proves, for push and for pull deployments.
  3. Know where to look next when the pipeline is green and the service is old.
Spot the bugGreen, then nothing#

payments-api's pipeline on main passed at 10:02, including the trigger to the deployer. At 10:20, staging still serves the old version.

QuestionWhich steps of the map could the deploy be stuck at, and which do you check first?
Show the answer

Any step after the pipeline: the deployer's commit to gitops-config, the controller's sync, or the rollout. Check them from left to right. First, does the staging values file in gitops-config name the new tag? If it does, has the controller synced since that commit? If it has, did the new pods start and stay up? The pipeline can't tell you any of this, because GitLab sees nothing after the commit.

See green pipeline, old version.

The lab kit: tools, runtimes, networks and tokens#

The labs in this part come in two kinds. Render-and-read labs need only the helm and kustomize commands, and no cluster. Cluster labs run a small Kubernetes cluster on your computer with kind, which runs each cluster node as a container. Every cluster lab also has a render-and-read path.

ToolThe labs useFor
helm4.3rendering charts
kustomize5.8, or kubectl 1.36, which has it built inbuilding overlays
kubectl1.36reading the cluster: within one minor version of the cluster
kind0.33, with the Kubernetes 1.36.4 node imagethe local cluster
flux2.9installing Flux, and reading its objects
argocd3.5reading Argo CD's applications
a container runtimesee belowrunning kind's node

A container runtime. kind uses Docker by default. Docker Desktop is free only for personal use, education, non-commercial open source, and businesses with fewer than 250 employees and less than US$10 million in revenue; everyone else needs a paid subscription. On Linux, Docker Engine is open source and free. Elsewhere, open-source runtimes such as Rancher Desktop and Podman Desktop are alternatives. kind's support for Podman is experimental, and needs extra setup.

Company networks. kind passes HTTP_PROXY, HTTPS_PROXY and NO_PROXY into its nodes, so a proxy that works for Docker usually works for the cluster. A proxy that inspects HTTPS re-signs every certificate, so its certificate authority must be trusted inside the nodes too; ask your IT team for it. If a registry such as ghcr.io is blocked, pull the image wherever you can and load it with kind load docker-image <image>. The first run downloads several hundred megabytes of images. On a slow connection, a chart's install can run out of time while they download (a HelmRelease that timed out).

Tokens. The cluster labs use two deploy tokens. A project's deploy token works only in that project, and only for what its scopes allow. Flux and Argo CD read the environment repository with one that has the read_repository scope. The cluster pulls the image with one that has read_registry on the project that built it. A token on the environment repository can't pull another project's images, however it's scoped (pods that can't pull their image). Deploy tokens are free on every GitLab offering. Project access tokens need Premium or Ultimate on GitLab.com. Avoid personal access tokens: they act as you, in every project you can reach. A lab that needs more says why.

Containers and Kubernetes, enough to read#

Part X · After the pipeline · Chapter 42 · 12 min read

A container runs one image: a program packaged with everything it needs. Kubernetes runs containers across a cluster of machines, from objects that describe what should run. You'll read a handful of them. A Deployment keeps a number of pods running, each with the service's container, and replaces them when their template changes. A Service gives those pods one stable name, and an Ingress brings outside traffic to it. Secrets hold settings that must stay hidden, and a ServiceAccount gives pods an identity. You rarely write these objects: MegaCorp's chart writes them from your values. This chapter shows how to read what it writes.

Monday, 13:30. Running, and not ready.

The new pods show Running, and 0/1 ready, and the rollout waits. The platform team says "check your probe". You have never opened a Kubernetes manifest. You need enough of it to read this one. (the card)

Images and containers#

An image is a program packaged with everything it needs to run. It's stored in a registry, under a repository and a tag, such as payments-api:3f9c2e1a. A container is one running copy of an image. Chapter 30 covered how pipelines build images; here, they're what runs.

A tag names an image, but tags can move. A digest, sha256: followed by 64 hexadecimal characters, names exactly one image and never moves. When you need to know exactly what ran, the digest is the answer.

The objects you'll read#

Kubernetes objects are YAML, like pipelines, and each one names its apiVersion and kind. A controller in the cluster watches each kind, and makes the cluster match what the objects ask for. MegaCorp's chart makes these for each service:

ObjectWhat it isWhat you change, in the service's values
Deploymentkeeps a number of pods running from one pod template, and replaces them when the template changesreplicaCount, image.tag, env, probes and resources
ReplicaSetmade by the Deployment, one for each version of the pod template; it keeps that version's pods runningnothing: you read it, to see which versions exist
Podthe smallest thing Kubernetes runs: one or more containers that share a network and storagenothing directly: pods come and go
Serviceone stable name and address for a set of pods, which it finds by their labelsusually nothing
Ingressroutes outside HTTP and HTTPS traffic to a Service, by host and path; an ingress controller must be runningingress.host
ExternalSecret and Secretthe Secret holds the service's secrets, which the External Secrets Operator copies from the cloud's vaultsecrets.keys
ServiceAccountthe pods' identity inside the cluster and, through its annotations, in the cloudserviceAccount.annotations
Namespacethe named group that holds one service's objects in one environment, such as payments-stagingnothing: the platform decides
What a Deployment creates. The Deployment makes a ReplicaSet for each version of its pod template, and the ReplicaSet keeps the pods running. A Service finds the pods by their labels, an Ingress sends outside traffic to the Service, and the pods take settings, secrets and an identity from other objects. Deployment payments-api replicas: 3 ReplicaSet one for each version of the pod template Pod container app Pod container app Pod container app Ingress outside traffic, by host Service finds pods by their labels Secret, ServiceAccount settings, secrets, identity
Figure What a Deployment creates. The Deployment makes a ReplicaSet for each version of its pod template, and the ReplicaSet keeps the pods running. A Service finds the pods by their labels, an Ingress sends outside traffic to the Service, and the pods take settings, secrets and an identity from other objects.#

Reading a Deployment#

This is payments-api's Deployment in staging, as MegaCorp's chart renders it:

helm templateoutputcaptured at the baseline
---
# Source: service/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  labels:
    app.kubernetes.io/name: payments-api
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: service-3.4.0
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: payments-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: payments-api
    spec:
      serviceAccountName: payments-api
      containers:
        - name: app
          image: "123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:3f9c2e1a"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: PAYMENTS_PROVIDER_URL
              value: "https://sandbox.pay.example.com"
            - name: SPRING_PROFILES_ACTIVE
              value: "staging"
          envFrom:
            - secretRef:
                name: payments-api
          readinessProbe:
            httpGet:
              path: /actuator/health
              port: http
          livenessProbe:
            httpGet:
              path: /actuator/health
              port: http
            initialDelaySeconds: 20
          resources:
            limits:
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 256Mi
  • replicas is how many pods run. Staging runs one; production runs three.
  • selector and the template's labels must agree. They're how the Deployment, and the Service, know which pods are theirs.
  • template is the pod template. Change anything under it, such as the image, and the Deployment makes a new ReplicaSet and moves the pods over to it (rolling, the default).
  • image is the full reference: registry, repository and tag. mc deploy changes only the tag.
  • env holds plain settings. envFrom adds every key of the payments-api Secret as a variable (the External Secrets Operator).
  • readinessProbe decides when a pod may receive traffic. livenessProbe restarts a container that stops answering.
  • resources.requests is what the pod is given, and what Kubernetes uses to choose a node. limits.memory is the most it may use; a container that goes over it can be killed.

None of it was typed into the Deployment. The port, the probe's path and the resources are the chart's defaults, which every service starts from:

megacorp/platform/chartsservice/values.yamlcontainer
port: 8080
env: {}                   # plain settings, as NAME: value
probes:
  path: /actuator/health
resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    memory: 512Mi

The image, the settings and the replica count come from staging's values file, which overrides those defaults for payments-api (values).

What you see with kubectl#

You read a running service with a few kubectl commands. Each one needs the namespace, such as -n payments-staging:

CommandShowsLook for
kubectl get podseach pod, whether it's ready, its status and its restarts1/1 ready and Running; restarts that keep growing
kubectl describe pod <pod>a pod's settings, with its events at the endwhy it can't pull its image, start, or become ready
kubectl logs <pod>what the container wrote to standard outputthe service's own errors; add --previous for a run that crashed
kubectl get deployment payments-apihow many pods are ready, up to date and availablea rollout that hasn't finished
kubectl rollout status deployment/payments-apiwaits until a rollout finishes, or failssuccessfully rolled out, or a progress-deadline error

This is payments-api in staging, on the local cluster the book was checked on:

kubectloutputcaptured at the baseline
NAME                            READY   STATUS    RESTARTS   AGE
payments-api-796d6777f9-stwjv   1/1     Running   0          14m
kubectloutputcaptured at the baseline
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
payments-api   1/1     1            1           14m

And the parts of kubectl describe pod you'll read most: the pod's status, its image, its readiness check, and the events at the end. The image was already present because the local cluster can't pull from ECR, so a stand-in was loaded into it under the same name. On a real cluster, the kubelet pulls the image first, and says so.

kubectloutputcaptured at the baseline
Name:             payments-api-796d6777f9-stwjv
Namespace:        payments-staging
⋯ lines omitted
Status:           Running
⋯ lines omitted
    Image:          123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:3f9c2e1a
⋯ lines omitted
    Ready:          True
⋯ lines omitted
    Readiness:  http-get http://:http/actuator/health delay=0s timeout=1s period=10s #success=1 #failure=3
⋯ lines omitted
  Type                        Status
⋯ lines omitted
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  14m   default-scheduler  Successfully assigned payments-staging/payments-api-796d6777f9-stwjv to delivery-lab-control-plane
  Normal  Pulled     14m   kubelet            spec.containers{app}: Container image "123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:3f9c2e1a" already present on machine and can be accessed by the pod
  Normal  Created    14m   kubelet            spec.containers{app}: Container created
  Normal  Started    14m   kubelet            spec.containers{app}: Container started

Most of what you read about a pod comes from other objects. Its image comes from the Deployment, which comes from the chart and your values. Its secrets come from the ExternalSecret, and its traffic from the Service and the Ingress. When something is wrong, the object at fault is often not the pod.

When you write it
Do this
Change a service through its values, never by editing its objects in the cluster. Point the readiness probe at something that shows the service can do its work. Give each pod the CPU and memory it really needs in resources.requests, and a memory limit it will not reach in normal use.
Why
The controller puts back whatever Git says, so a change made with kubectl is undone. A good probe keeps traffic away from a pod that is not ready.
Watch out
A memory limit that is too low makes the cluster kill the pod, and the restart looks like a crash in your code.
What you can now do
  1. Read a Deployment, a Service and an Ingress, and say what each does for your service.
  2. Read a pod's status and its events, and tell an image problem from a probe problem from a crash.
  3. Ask kubectl the four questions that explain most rollouts that stall.
Spot the bugRead production on aks-prod#

This is payments-api's Deployment for production on aks-prod, rendered from the same chart:

helm templateoutputcaptured at the baseline
---
# Source: service/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  labels:
    app.kubernetes.io/name: payments-api
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: service-3.4.0
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: payments-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: payments-api
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: payments-api
      containers:
        - name: app
          image: "megacorpprod.azurecr.io/payments-api:1c7d4b09"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: PAYMENTS_PROVIDER_URL
              value: "https://pay.example.com"
            - name: SPRING_PROFILES_ACTIVE
              value: "production"
          envFrom:
            - secretRef:
                name: payments-api
          readinessProbe:
            httpGet:
              path: /actuator/health
              port: http
          livenessProbe:
            httpGet:
              path: /actuator/health
              port: http
            initialDelaySeconds: 20
          resources:
            limits:
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 256Mi
QuestionHow many pods run, from which image? Where does DB_PASSWORD come from, and when does a new pod start receiving traffic?
Show the answer

Three pods, from replicas: 3. The image is megacorpprod.azurecr.io/payments-api:1c7d4b09: Azure's registry, with production's tag. DB_PASSWORD isn't in the Deployment at all. envFrom loads every key of the payments-api Secret, which the External Secrets Operator fills from Key Vault. A new pod receives traffic once its readiness probe on /actuator/health passes (0/1 ready).

Helm#

Part X · After the pipeline · Chapter 43 · 15 min read

Every service needs the same handful of Kubernetes objects, with different names, images and settings. Helm writes those objects once, as a chart: YAML templates with gaps, and a default for every gap. A values file fills the gaps for one service in one environment, and helm template prints the result. That is how a new image tag travels from a values file into the Deployment that runs the pods. MegaCorp has one chart for every service, and each service's values live in gitops-config.

Tuesday, 10:15. The value that only one cluster ignored.

You change the memory limit in payments-api's production values file. Two clusters pick it up. The third keeps the old limit. Same chart, same values file, same tag. The third cluster's Application sets the limit inline, and inline wins. Who gets to say which value wins? (the card)

One chart, many services#

A service on Kubernetes needs the same few objects. A Deployment runs its pods, a Service sits in front of them, and an Ingress brings in traffic from outside. A ServiceAccount carries its identity, and something brings in its secrets. Written by hand, that is about a hundred lines of YAML for each service in each environment, and the copies drift apart.

Helm writes the objects once, with gaps where services differ. MegaCorp's platform team keeps one chart, service, in megacorp/platform/charts. Teams don't edit it. They write values.

WordMeans
charta folder of templates and default values, versioned as one package
templatea YAML file with gaps, each written {{ … }}
valueswhat fills the gaps: the chart's defaults, then yours
renderfill every gap and print plain YAML, which is what helm template does
releaseone installation of a chart in a cluster, with a numbered revision for each change

What's in a chart#

A chart is a folder with three kinds of file. Chart.yaml names the chart and its version:

megacorp/platform/chartsservice/Chart.yaml
# service/Chart.yaml in megacorp/platform/charts: the shared Helm chart that every
# MegaCorp HTTP service deploys with. The charts repository tags each release, such as
# service-3.4.0, and the ApplicationSets name that tag.
# Used by the chapters "Helm" and "Argo CD".
apiVersion: v2
name: service
description: One MegaCorp HTTP service, with its Service, Ingress and secrets
type: application
version: 3.4.0

values.yaml holds a default for every gap. The image's repository and tag are empty, because only a service and an environment can answer them:

megacorp/platform/chartsservice/values.yamlimage
image:
  repository: ""          # set for each service
  tag: ""                 # set for each environment; mc deploy writes it
  pullPolicy: IfNotPresent

The templates folder holds YAML with gaps: deployment.yaml, service.yaml (with the ServiceAccount and the Ingress) and externalsecret.yaml. A file whose name starts with an underscore, such as _helpers.tpl, makes no objects: it holds named pieces that the other templates reuse.

The charts repository tags each release, such as service-3.4.0, and each ApplicationSet names the tag it uses. A new chart version reaches a service only when someone changes that line.

Reading a template#

Here is the line that sets the image, in the Deployment template:

megacorp/platform/chartsservice/templates/deployment.yamlimage
image: "{{ .Values.image.repository }}:{{ required "image.tag is set for each environment" .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}

Everything between {{ and }} is an action, which Helm replaces with text. Everything else is copied as it is. This book shades actions, so you can tell them from the YAML around them. The actions you will meet most:

You seeIt means
{{ .Values.image.tag }}the value at image.tag
{{ required "…" .Values.image.tag }}the same, but stop with that message if it's empty
{{ include "service.name" . }}the text of a named template from _helpers.tpl
{{- toYaml .Values.resources | nindent 12 }}a whole block of values, indented 12 spaces. The - trims the spaces before the action
{{- if .Values.ingress.enabled }} … {{- end }}the lines between, only when the value is true
{{- range .Values.secrets.keys }} … {{- end }}the lines between, once for each item in the list
{{- with .Values.podLabels }} … {{- end }}the lines between, only when the value is set, with . meaning that value
Where the name comes fromdeep dive

{{ include "service.name" . }} is the release name, cut to 63 characters: the first word after helm template or helm install. Argo CD sets it from releaseName in the ApplicationSet, so every object is called payments-api, whatever the environment. The namespace, not the name, keeps staging and production apart.

Values: the defaults, then yours#

Helm starts from the chart's values.yaml, then lays each values file over it, in order. Maps merge key by key, and anything else is replaced, lists included. So a values file only says what differs:

megacorp/platform/gitops-configenvs/staging/payments-api.yamlimage
image:
  repository: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api
  tag: 3f9c2e1a

The staging file sets the repository and the tag, and keeps the chart's defaults for everything it doesn't mention, such as the probes and the resources.

Argo CD adds layers of its own. From weakest to strongest: the chart's values.yaml, value files, inline values, valuesObject, and single parameters, which beat everything. aks-prod uses that: its ApplicationSet adds inline values that switch the image to Azure's registry and give the pods their Azure identity:

megacorp/platform/gitops-configapps/aks-prod/applicationset.yamlazure-values
valuesObject:
  image:
    repository: megacorpprod.azurecr.io/payments-api
  serviceAccount:
    annotations:
      azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
  podLabels:
    azure.workload.identity/use: "true"

Render it yourself#

You never need a cluster to see what a chart makes. helm template takes a release name, the chart and your values, and prints the manifests. --show-only keeps one template's output:

helm template payments-api ./service -n payments-staging \
  -f envs/staging/payments-api.yaml --show-only templates/deployment.yaml
helm templateoutputcaptured at the baseline
---
# Source: service/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  labels:
    app.kubernetes.io/name: payments-api
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: service-3.4.0
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: payments-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: payments-api
    spec:
      serviceAccountName: payments-api
      containers:
        - name: app
          image: "123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:3f9c2e1a"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: PAYMENTS_PROVIDER_URL
              value: "https://sandbox.pay.example.com"
            - name: SPRING_PROFILES_ACTIVE
              value: "staging"
          envFrom:
            - secretRef:
                name: payments-api
          readinessProbe:
            httpGet:
              path: /actuator/health
              port: http
          livenessProbe:
            httpGet:
              path: /actuator/health
              port: http
            initialDelaySeconds: 20
          resources:
            limits:
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 256Mi

Read it from the top. The name comes from the release name, and the labels from _helpers.tpl. The replicas, the image and the settings come from the staging file. envFrom is there because the file lists a secret. Everything the cluster will run is on the page.

A required value that's missing stops the render, with the chart's own message:

helm templateoutputcaptured at the baseline
Error: execution error at (service/templates/deployment.yaml:28:52): image.tag is set for each environment

Use --debug flag to render out invalid YAML

helm lint checks a chart's structure in the same way, also without a cluster.

Helm rendering. The chart's templates and defaults, and an environment's values file, go into helm template, which fills every gap and prints the manifests. Argo CD applies them; Flux installs them as a Helm release. the chart: service 3.4.0 Chart.yaml values.yaml the defaults templates/*.yaml YAML with {{ }} gaps an environment's values envs/staging/payments-api.yaml tag: 3f9c2e1a helm template fills every gap later values win the manifests ServiceAccount Service Deployment image: …:3f9c2e1a Ingress ExternalSecret the cluster Argo CD applies them, or Flux installs them as a Helm release
Figure Helm rendering. The chart's templates and defaults, and an environment's values file, go into helm template, which fills every gap and prints the manifests. Argo CD applies them; Flux installs them as a Helm release.#

From a tag to a pod#

When a deploy seems not to have happened, follow the tag through three places, in order:

Where is the new tag? Follow it from the values file to the pods.noyesnoyesnoyesDoes the environment's values file ingitops-config name the new tag?The deploy never reachedgitops-config. Read the deployer'sjob logDoes the rendered Deployment name it?Something overrides the file:inline values or parametersDo the running pods report it?The controller hasn't synced yet,or the new pods can't startThe new version is running
Figure Where is the new tag? Follow it from the values file to the pods.#
As text
  1. Does the environment's values file in gitops-config name the new tag? No: The deploy never reached gitops-config. Read the deployer's job log. Yes: the next step.
  2. Does the rendered Deployment name it? No: Something overrides the file: inline values or parameters. Yes: the next step.
  3. Do the running pods report it? No: The controller hasn't synced yet, or the new pods can't start. Yes: the next step.
  4. The new version is running

The first two need no cluster: read the file, and render it. The third asks the cluster which image each pod was started from:

kubectloutputcaptured at the baseline
NAME                            IMAGE
payments-api-6c854bc48b-lzhss   123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:1c7d4b09
payments-api-6c854bc48b-nsd64   123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:1c7d4b09
payments-api-6c854bc48b-qpdqw   123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:1c7d4b09

That is production on the local cluster: three pods, as replicaCount: 3 in production's values file asks, each started from the tag 1c7d4b09 that the same file names.

Releases, and who installs them#

helm install and helm upgrade render a chart and apply the result, and keep a release: a record in the cluster of what was applied, with a revision number for each change. helm list shows releases, helm history shows a release's revisions, and helm rollback goes back to one.

With GitOps, people rarely run these commands. What happens to a chart depends on the controller:

Flux, through a HelmRelease, installs and upgrades a real release, so helm list shows it. Argo CD renders the chart with helm template, then applies and tracks the output itself, so helm list shows nothing.

So on MegaCorp's clusters, helm list shows the add-ons that Flux installed, and none of the services that Argo CD deploys, even though their objects carry the label app.kubernetes.io/managed-by: Helm.

helmoutputcaptured at the baseline
NAME            	NAMESPACE       	REVISION	UPDATED                                	STATUS  	CHART                  	APP VERSION
argo-cd         	argocd          	2       	2026-09-14 06:40:20.289277196 +0000 UTC	deployed	argo-cd-10.9.0         	v3.5.2     
external-secrets	external-secrets	1       	2026-09-14 06:33:33.744648979 +0000 UTC	deployed	external-secrets-2.10.0	v2.10.0    
⋯ lines omitted

On the local cluster, Flux installed both of these. payments-api runs in staging and production, yet it has no release: Argo CD deployed it. The line left out is the lab cluster's own ingress controller, installed with helm directly.

Note
The book targets Helm 4.3. Charts written for Helm 3 work unchanged. Two flags in older pipelines were renamed: --atomic is now --rollback-on-failure, and --force is now --force-replace. The old names still work, with a warning. New releases are applied with server-side apply.
When you write it
Do this
Put everything that differs by environment in that environment's values file, and nothing environment-specific in the chart.
Why
The chart then stays one thing, and reviewers can see every difference between environments in one file each.
Watch out
Never type an image tag into a chart or a template. The deploy writes tags into the values files. Mark the tag as required in the template, so that a render with no tag stops with an error instead of deploying nothing.
What you can now do
  1. Read a chart and its values files, and say what a template will render.
  2. Render the chart yourself with helm template and compare it with what runs.
  3. Recite the order in which Helm's value sources win.
Spot the bugWhich image runs on aks-prod?#

payments-api's production values file names the tag 1c7d4b09 and an ECR repository. aks-prod's ApplicationSet adds the inline values shown above.

QuestionWhich image do payments-api's production pods run on aks-prod, and why?
Show the answer

megacorpprod.azurecr.io/payments-api:1c7d4b09. The tag comes from the production values file. The repository comes from the inline values, which beat value files. Rendered with the same values, the Deployment says so:

helm templateoutputcaptured at the baseline
⋯ lines omitted
kind: Deployment
⋯ lines omitted
  name: payments-api
⋯ lines omitted
        azure.workload.identity/use: "true"
⋯ lines omitted
          image: "megacorpprod.azurecr.io/payments-api:1c7d4b09"
⋯ lines omitted

See a value that loses to another.

LabRender the shared chart with your valuesabout 10 min#
You need

Helm 4.3 on your computer, and the image you pushed in chapter 8. No cluster.

Steps
  1. In a new folder, save MegaCorp's chart as service, keeping these paths:
    megacorp/platform/chartsservice/Chart.yaml
    # service/Chart.yaml in megacorp/platform/charts: the shared Helm chart that every
    # MegaCorp HTTP service deploys with. The charts repository tags each release, such as
    # service-3.4.0, and the ApplicationSets name that tag.
    # Used by the chapters "Helm" and "Argo CD".
    apiVersion: v2
    name: service
    description: One MegaCorp HTTP service, with its Service, Ingress and secrets
    type: application
    version: 3.4.0
    megacorp/platform/chartsservice/values.yaml
    # service/values.yaml in megacorp/platform/charts: the defaults every service starts
    # from. A service's file in gitops-config sets what differs.
    # Used by the chapters "Helm" and "GitOps repositories and promotion".
    
    image:
      repository: ""          # set for each service
      tag: ""                 # set for each environment; mc deploy writes it
      pullPolicy: IfNotPresent
    
    replicaCount: 2
    
    port: 8080
    env: {}                   # plain settings, as NAME: value
    probes:
      path: /actuator/health
    resources:
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        memory: 512Mi
    
    serviceAccount:
      annotations: {}         # the pod's cloud identity, where the cloud needs one
    podLabels: {}
    
    ingress:
      enabled: false
      className: nginx
      host: ""
    
    secrets:
      store: ""               # the ClusterSecretStore to read from
      keys: []                # each one: {name: DB_PASSWORD, remote: payments-api/staging/db-password}
    megacorp/platform/chartsservice/templates/_helpers.tpl
    {{- /*
    service/templates/_helpers.tpl in megacorp/platform/charts: named templates that the
    other templates share, so every object gets the same name and labels.
    Used by the chapter "Helm".
    */}}
    
    {{- define "service.name" -}}
    {{- .Release.Name | trunc 63 | trimSuffix "-" -}}
    {{- end -}}
    
    {{- define "service.selectorLabels" -}}
    app.kubernetes.io/name: {{ include "service.name" . }}
    {{- end -}}
    
    {{- define "service.labels" -}}
    {{ include "service.selectorLabels" . }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
    {{- end -}}
    megacorp/platform/chartsservice/templates/deployment.yaml
    {{- /*
    service/templates/deployment.yaml in megacorp/platform/charts: the Deployment that runs a
    service's pods. Used by the chapters "Containers and Kubernetes, enough to read" and "Helm".
    */}}
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: {{ include "service.name" . }}
      labels:
        {{- include "service.labels" . | nindent 4 }}
    spec:
      replicas: {{ .Values.replicaCount }}
      selector:
        matchLabels:
          {{- include "service.selectorLabels" . | nindent 6 }}
      template:
        metadata:
          labels:
            {{- include "service.selectorLabels" . | nindent 8 }}
            {{- with .Values.podLabels }}
            {{- toYaml . | nindent 8 }}
            {{- end }}
        spec:
          serviceAccountName: {{ include "service.name" . }}
          containers:
            - name: app
              image: "{{ .Values.image.repository }}:{{ required "image.tag is set for each environment" .Values.image.tag }}"
              imagePullPolicy: {{ .Values.image.pullPolicy }}
              ports:
                - name: http
                  containerPort: {{ .Values.port }}
              {{- with .Values.env }}
              env:
                {{- range $name, $value := . }}
                - name: {{ $name }}
                  value: {{ $value | quote }}
                {{- end }}
              {{- end }}
              {{- if .Values.secrets.keys }}
              envFrom:
                - secretRef:
                    name: {{ include "service.name" . }}
              {{- end }}
              readinessProbe:
                httpGet:
                  path: {{ .Values.probes.path }}
                  port: http
              livenessProbe:
                httpGet:
                  path: {{ .Values.probes.path }}
                  port: http
                initialDelaySeconds: 20
              resources:
                {{- toYaml .Values.resources | nindent 12 }}
    megacorp/platform/chartsservice/templates/service.yaml
    {{- /*
    service/templates/service.yaml in megacorp/platform/charts: the service's ServiceAccount,
    the Service in front of its pods, and the Ingress that brings outside traffic in.
    Used by the chapters "Containers and Kubernetes, enough to read", "Helm" and "AWS and
    Azure for delivery".
    */}}
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: {{ include "service.name" . }}
      labels:
        {{- include "service.labels" . | nindent 4 }}
      {{- with .Values.serviceAccount.annotations }}
      annotations:
        {{- toYaml . | nindent 4 }}
      {{- end }}
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: {{ include "service.name" . }}
      labels:
        {{- include "service.labels" . | nindent 4 }}
    spec:
      selector:
        {{- include "service.selectorLabels" . | nindent 4 }}
      ports:
        - name: http
          port: 80
          targetPort: http
    {{- if .Values.ingress.enabled }}
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: {{ include "service.name" . }}
      labels:
        {{- include "service.labels" . | nindent 4 }}
    spec:
      ingressClassName: {{ .Values.ingress.className }}
      rules:
        - host: {{ required "ingress.host is needed when ingress.enabled is true" .Values.ingress.host }}
          http:
            paths:
              - path: /
                pathType: Prefix
                backend:
                  service:
                    name: {{ include "service.name" . }}
                    port:
                      name: http
    {{- end }}
    megacorp/platform/chartsservice/templates/externalsecret.yaml
    {{- /*
    service/templates/externalsecret.yaml in megacorp/platform/charts: asks the External Secrets
    Operator to copy a service's secrets from the cloud's vault into a Kubernetes Secret, which
    the Deployment reads. Used by the chapters "Helm" and "Secrets and config after the pipeline".
    */}}
    {{- if .Values.secrets.keys }}
    apiVersion: external-secrets.io/v1
    kind: ExternalSecret
    metadata:
      name: {{ include "service.name" . }}
      labels:
        {{- include "service.labels" . | nindent 4 }}
    spec:
      refreshInterval: 1h
      secretStoreRef:
        kind: ClusterSecretStore
        name: {{ required "secrets.store names the ClusterSecretStore" .Values.secrets.store }}
      target:
        name: {{ include "service.name" . }}
      data:
        {{- range .Values.secrets.keys }}
        - secretKey: {{ .name }}
          remoteRef:
            key: {{ .remote }}
        {{- end }}
    {{- end }}
  2. Beside the service folder, save your values as greeting-values.yaml, with your own image's repository and tag:
    practice · deliverygreeting-values.yaml
    # practice/delivery/greeting-values.yaml: your values for MegaCorp's shared chart, for the
    # greeting-service image you pushed in "Images, the registry and variables". Change the
    # repository to your project's image, and the tag to one of its tags.
    image:
      repository: gitlab.example.com:5050/practice/node-service
      tag: 0648d3dd
    replicaCount: 1
    port: 8080
    probes:
      path: /greet
    resources:
      requests:
        cpu: 50m
        memory: 64Mi
      limits:
        memory: 128Mi
  3. Render the Deployment:
    helm template greeting ./service -f greeting-values.yaml --show-only templates/deployment.yaml
  4. Find the image line and the probe path, and say which file each came from.
What you should see

The Deployment is called greeting, after the release name. Its image and probe path come from your file, and its labels and the rest from the chart:

helm templateoutputcaptured at the baseline
---
# Source: service/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: greeting
  labels:
    app.kubernetes.io/name: greeting
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: service-3.4.0
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: greeting
  template:
    metadata:
      labels:
        app.kubernetes.io/name: greeting
    spec:
      serviceAccountName: greeting
      containers:
        - name: app
          image: "gitlab.example.com:5050/practice/node-service:0648d3dd"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /greet
              port: http
          livenessProbe:
            httpGet:
              path: /greet
              port: http
            initialDelaySeconds: 20
          resources:
            limits:
              memory: 128Mi
            requests:
              cpu: 50m
              memory: 64Mi
Break it on purpose

Delete the tag line from your values file and render again. Helm stops with the chart's message:

helm templateoutputcaptured at the baseline
Error: execution error at (service/templates/deployment.yaml:28:52): image.tag is set for each environment

Use --debug flag to render out invalid YAML

Kustomize#

Part X · After the pipeline · Chapter 44 · 10 min read

Kustomize changes plain YAML without templates. A base is ordinary YAML that works on its own. An overlay names the base and lists only what differs for one environment: a namespace, an image tag, a patch. kustomize build prints the base with the overlay's changes applied, and kubectl and Flux have Kustomize built in. MegaCorp uses it to set up each cluster's add-ons, and many teams use it for their services. Always read the build's output: a change that matches nothing is skipped without a word.

Tuesday, 15:00. The tag that changed nothing.

An overlay sets a new image tag for the add-on, and the cluster keeps running the old one. The build succeeds. The tag is spelled right. The overlay's images entry names an image that the base doesn't use by that name, and Kustomize changes nothing without a word. (the card)

Plain YAML, then changes#

Helm fills gaps in templates. Kustomize has no gaps: its input is the same YAML you would apply by hand. It edits that YAML with changes that you list in a file called kustomization.yaml.

WordMeans
basea folder of plain YAML, with a kustomization.yaml listing its files, that works on its own
overlaya folder whose kustomization.yaml names a base and lists changes
patchone change to one object: set a field, add one, or remove one
buildapply an overlay's changes to its base and print the result

A base and an overlay#

The lab at the end of this chapter uses greeting-service, the Node service from Part III. Its base is a Deployment and a Service, with the image tagged latest. Staging's overlay changes three things:

practice · deliverykustomize/overlays/staging/kustomization.yaml
# practice/delivery/kustomize/overlays/staging/kustomization.yaml: staging's overlay. It takes
# the base, puts it in its own namespace, pins the image's tag, and runs two replicas.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: greeting-staging
resources:
  - ../../base
images:
  - name: gitlab.example.com:5050/practice/node-service
    newTag: 0648d3dd
patches:
  - target:
      kind: Deployment
      name: greeting
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 2
  • resources names the base, as a path to its folder.
  • namespace puts every object in greeting-staging.
  • images finds each container whose image has the name given, and sets its tag. The name is the image without its tag, and it must match exactly.
  • patches changes one object. This patch is a list of operations, each with a path into the object: replace the value at /spec/replicas with 2.

kustomize build overlays/staging prints the result:

kustomize buildoutputcaptured at the baseline
⋯ lines omitted
kind: Service
⋯ lines omitted
  name: greeting
  namespace: greeting-staging
⋯ lines omitted
kind: Deployment
⋯ lines omitted
  name: greeting
  namespace: greeting-staging
⋯ lines omitted
  replicas: 2
⋯ lines omitted
      - image: gitlab.example.com:5050/practice/node-service:0648d3dd
⋯ lines omitted

Kustomize prints every map with its keys in alphabetical order, so image now comes before name. The meaning is the same.

Kustomize: a base and its overlays. The base is plain YAML that works on its own. Each overlay names the base and lists only its changes, and a build prints the base with those changes applied. base plain YAML, valid on its own Deployment greeting image: …:latest replicas: 1 overlays/staging resources: ../../base images: newTag 0648d3dd patch: replicas 2 overlays/production resources: ../../base images: another tag patch: replicas 3 build build staging's manifests image: …:0648d3dd replicas: 2 production's manifests the base, with production's changes
Figure Kustomize: a base and its overlays. The base is plain YAML that works on its own. Each overlay names the base and lists only its changes, and a build prints the base with those changes applied.#

A patch as part of an object#

A patch can also be written as a piece of the object, and Kustomize merges it in. aks-prod's add-ons overlay does that. It keeps MegaCorp's base of add-ons, and gives the External Secrets Operator the Azure identity it needs to read Key Vault:

megacorp/platform/gitops-configinfrastructure/add-ons/aks/kustomization.yamloverlay
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base
patches:
  - target:
      kind: HelmRelease
      name: external-secrets
    patch: |-
      apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: external-secrets
      spec:
        values:
          serviceAccount:
            annotations:
              azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
          podLabels:
            azure.workload.identity/use: "true"

The patch names the object by kind and name, and gives only the fields to add. Maps merge key by key, so the values the base already set, such as the service account's name, stay:

kustomize buildoutputcaptured at the baseline
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: external-secrets
  namespace: external-secrets
spec:
  chart:
    spec:
      chart: external-secrets
      sourceRef:
        kind: HelmRepository
        name: external-secrets
        namespace: flux-system
      version: 2.10.0
  install:
    crds: CreateReplace
  interval: 30m
  upgrade:
    crds: CreateReplace
  values:
    podLabels:
      azure.workload.identity/use: "true"
    serviceAccount:
      annotations:
        azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
      name: external-secrets

eks-prod needs none of this, because EKS Pod Identity is set up in AWS, not in the cluster. So eks-prod's Flux builds the base itself.

Where Kustomize runs#

WhereHow
your terminalkustomize build <folder>, or kubectl kustomize <folder>: kubectl 1.36 has Kustomize 5.8.1 built in
kubectl apply -k <folder>builds the folder and applies the result
Fluxeach Kustomization object names a path in a Git source, and Flux builds and applies it
Argo CDan Application whose path holds a kustomization.yaml is built with Kustomize

So the file you read in a GitOps repository is never what the cluster gets. The build's output is.

A change that matches nothing#

When an images entry names an image that isn't there, Kustomize doesn't complain. It builds as usual, and the tag stays as it was. Here is the same overlay with node-service misspelt as node-servce. The namespace and the replicas still change, which makes the miss easy to overlook:

kustomize buildoutputcaptured at the baseline
⋯ lines omitted
kind: Service
⋯ lines omitted
  namespace: greeting-staging
⋯ lines omitted
kind: Deployment
⋯ lines omitted
  namespace: greeting-staging
⋯ lines omitted
  replicas: 2
⋯ lines omitted
      - image: gitlab.example.com:5050/practice/node-service:latest
⋯ lines omitted

Before merging a change to an overlay, build it and read the line you meant to change.

LegacypatchesStrategicMerge and commonLabels
Kustomize 5 replaced several fields. patchesStrategicMerge and patchesJson6902 became patches, commonLabels became labels, and a base is listed under resources, not bases. Old files still build, with a warning at the top of the output:
kustomize buildoutputcaptured at the baseline
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
⋯ lines omitted
kustomize edit fix rewrites the file for you.
When you write it
Do this
Keep the base valid on its own, and put each environment's differences in its overlay. Change image tags with images, not by editing the base.
Why
Every overlay then starts from the same tested base, and a tag change is one line that reviewers can see.
Watch out
Build the overlay and read the output before you merge. A patch or an images entry whose name matches nothing changes nothing, and Kustomize gives no warning.
What you can now do
  1. Read a base and its overlays, and say what kustomize build will produce.
  2. Change an image tag in an overlay so that it actually applies.
  3. Build the overlay and read the output before you merge.
Spot the bugWhat does aks-prod's overlay change?#

eks-prod's Flux builds infrastructure/add-ons/base. aks-prod's builds infrastructure/add-ons/aks, the overlay shown above.

QuestionWhich objects differ between the two clusters' add-ons, and how?
Show the answer

Only the External Secrets Operator's HelmRelease. On aks-prod its values gain the azure.workload.identity/client-id annotation for the service account, and the azure.workload.identity/use: "true" label for the pods. The namespaces, the Helm repositories and Argo CD's HelmRelease are the base's, unchanged, on both clusters.

Without that patch, the Operator's pods would have no Azure identity (a pod without its Azure identity).

LabBuild an overlayabout 10 min#
You need

Kustomize 5.8, or kubectl 1.36, which has it built in. No cluster.

Steps
  1. Save the base, with your own image in place of gitlab.example.com:5050/practice/node-service:
    practice · deliverykustomize/base/kustomization.yaml
    # practice/delivery/kustomize/base/kustomization.yaml: the base for the Kustomize lab:
    # greeting-service as plain YAML, the same in every environment.
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    resources:
      - greeting.yaml
    practice · deliverykustomize/base/greeting.yaml
    # practice/delivery/kustomize/base/greeting.yaml: greeting-service's Deployment and Service,
    # with the image your Build labs pushed. Change the image to your project's. The overlays
    # set its tag.
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: greeting
      labels:
        app.kubernetes.io/name: greeting
    spec:
      replicas: 1
      selector:
        matchLabels:
          app.kubernetes.io/name: greeting
      template:
        metadata:
          labels:
            app.kubernetes.io/name: greeting
        spec:
          containers:
            - name: app
              image: gitlab.example.com:5050/practice/node-service:latest
              ports:
                - name: http
                  containerPort: 8080
              readinessProbe:
                httpGet:
                  path: /greet
                  port: http
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: greeting
      labels:
        app.kubernetes.io/name: greeting
    spec:
      selector:
        app.kubernetes.io/name: greeting
      ports:
        - name: http
          port: 80
          targetPort: http
  2. Save staging's overlay, shown above, as overlays/staging/kustomization.yaml, with your image's name and one of its tags.
  3. Run kustomize build overlays/staging, or kubectl kustomize overlays/staging.
  4. Find the namespace, the replicas and the image, and say which line of the overlay set each.
What you should see

Every object is in greeting-staging, the Deployment has 2 replicas, and the image has your tag:

kustomize buildoutputcaptured at the baseline
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: greeting
  name: greeting
  namespace: greeting-staging
spec:
  ports:
  - name: http
    port: 80
    targetPort: http
  selector:
    app.kubernetes.io/name: greeting
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: greeting
  name: greeting
  namespace: greeting-staging
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: greeting
  template:
    metadata:
      labels:
        app.kubernetes.io/name: greeting
    spec:
      containers:
      - image: gitlab.example.com:5050/practice/node-service:0648d3dd
        name: app
        ports:
        - containerPort: 8080
          name: http
        readinessProbe:
          httpGet:
            path: /greet
            port: http
Break it on purpose

In the overlay's images entry, misspell the image's name, and build again. The build succeeds, and the image is back to latest.

GitOps repositories and promotion#

Part X · After the pipeline · Chapter 45 · 13 min read

With GitOps, a deploy is a commit. A service's application repository holds its code and pipeline, and an environment repository says what runs where. The pipeline's last job writes the new image tag into the environment repository, and a controller in each cluster applies whatever that repository says. Promotion to production is a merge request in the environment repository, so merging it is the approval, and going back is a revert. A change made in the cluster by hand doesn't last: the controller puts back what the repository says.

Wednesday, 09:20. Ten minutes late.

At 10:02 the pipeline goes green. At 10:05 staging still serves the old version, and you open an incident. At 10:12 the new version is live, and nobody did anything. The pipeline's last act was a commit to gitops-config. Something else did the deploying, on its own clock. (the card)

Two kinds of repository#

The application repository holds the service's code, its Dockerfile and its pipeline file. The team changes it in merge requests, and its pipeline builds and pushes images. The environment repository holds what runs in each cluster and environment: which chart or overlay, which values, which image tag. Deploy jobs change it, and so do people, in merge requests.

Keeping them apart makes the environment repository's history the history of deploys: one commit for each change to what runs. Its permissions can differ too: many people change code, and fewer may change production. MegaCorp's environment repository is gitops-config. Some teams keep their deployment files in the application repository instead, in a folder such as deploy/, and point the controller at that folder.

Inside an environment repository#

Folder in gitops-configHoldsOwned by
clusters/each cluster's Flux setup: where it syncs from, and what it runs, in orderthe platform team
infrastructure/the add-ons every cluster runs, and each cloud's changes to themthe platform team
apps/each cluster's ApplicationSets: which services it deploys, from which chartthe platform team, with the teams
envs/each service's values in each environmentthe teams, and the deploy job

A service's team mostly touches one file for each environment. This is payments-api's in production:

megacorp/platform/gitops-configenvs/production/payments-api.yamlimage
image:
  repository: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api
  tag: 1c7d4b09

The job that bumps a tag#

MegaCorp's service pipelines don't write to gitops-config themselves. A service's deploy job triggers the platform team's deployer pipeline, whose job runs mc deploy:

megacorp/devops/ci-toolsbin/mcdeploy
deploy)
  env="${1:?usage: mc deploy <environment> <image-ref>}"
  image="${2:?usage: mc deploy <environment> <image-ref>}"
  tag="${image##*:}"
  mc_log "promoting ${APP} ${tag} to ${env} through the GitOps repository"
  git clone --depth 1 \
    "https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
  yq -i ".image.tag = \"${tag}\"" "gitops/envs/${env}/${APP}.yaml"
  git -C gitops commit -am "deploy ${APP} ${tag} to ${env}"
  if [ "$env" = production ]; then
    # production changes wait in a merge request until someone merges them
    git -C gitops push origin "HEAD:deploy/${APP}-${tag}" \
      -o merge_request.create -o merge_request.target=main \
      -o merge_request.title="Deploy ${APP} ${tag} to production"
  else
    git -C gitops push origin HEAD:main
  fi
  ;;
  • It clones gitops-config with the deployer's token, and changes one line: the tag, in the service's file for that environment.
  • For staging, it pushes the commit to main, and the controllers pick it up on their next sync.
  • For production, it pushes a branch, and asks GitLab to open a merge request. Nothing changes in production until someone merges it.
The GitOps handoff. The pipelines end when the new image is committed to gitops-config. The controller's sync and the cluster's rollout happen later, and GitLab doesn't see them. GitLab shows these: green once the commit is pushed GitLab can't see these service pipeline deploy-staging deployer pipeline mc deploy gitops-config new image written controller in the cluster cluster new pods, new image IMAGE_REF git push sync, on a timer apply Green in GitLab means the new version was written to Git. To know it is running, check the controller and the cluster.
Figure The GitOps handoff. The pipelines end when the new image is committed to gitops-config. The controller's sync and the cluster's rollout happen later, and GitLab doesn't see them.#

The trigger jobs use strategy: depend, so a service's pipeline turns red if the deployer's pipeline fails (chapter 17). But the deployer's pipeline is green as soon as the push succeeds. Anything that goes wrong after that never reaches GitLab: a controller that hasn't synced, or pods that crash.

Promotion by merge request#

To promote is to move a version that is already running in one environment into the next. In an environment repository, that is a change to the next environment's file, and a merge request is the safest way to make it:

  • it shows exactly what will change: one tag, from one value to another;
  • whoever may merge into main may deploy, so the branch's protection is the approval rule;
  • the merge is the record of who promoted what, and when.

With push options, the deploy job opens the merge request in the same push, and GitLab answers with its link:

git pushoutputcaptured at the baseline
remote: 
remote: View merge request for deploy/greeting-7d625fae:        
remote:   https://gitlab.example.com/practice/greeting-gitops/-/merge_requests/3        
remote:

Without the options, GitLab only offers a link to create one, and nobody is asked to review anything:

git pushoutputcaptured at the baseline
remote: 
remote: To create a merge request for deploy/greeting-7d625fae-plain, visit:        
remote:   https://gitlab.example.com/practice/greeting-gitops/-/merge_requests/new?merge_request%5Bsource_branch%5D=deploy%2Fgreeting-7d625fae-plain        
remote:
Promotion in the environment repository. Staging's new tag is committed straight to main. Production's waits on a branch, in a merge request, until someone merges it. The controllers sync each cluster to main. the deploy job mc deploy staging a commit on main envs/staging/…: new tag production a merge request deploy/payments-api-1c7d4b09 someone merges it: the approval main what should run the controllers sync each cluster to main To go back, revert the commit on main.
Figure Promotion in the environment repository. Staging's new tag is committed straight to main. Production's waits on a branch, in a merge request, until someone merges it. The controllers sync each cluster to main.#

Drift, and going back#

Drift is any difference between the cluster and the repository. It usually comes from a change made by hand, such as kubectl edit or kubectl scale. Both controllers put the repository's version back:

What happens to a change made by hand depends on the controller. Flux reverts it at the Kustomization's next reconciliation, every ten minutes at MegaCorp. Argo CD with selfHeal: true reverts it a few seconds after noticing. Argo CD without selfHeal shows the Application as OutOfSync until someone syncs.

So the way to change a running service is always a commit, and that includes going back. kubectl rollout undo and helm rollback change the cluster, and the controller changes it straight back. Revert the deploy's commit in the environment repository instead:

git revert <the deploy's commit>

The revert is itself a commit, so it deploys like any other change, and the history shows it.

Automatic image updates#

Some setups have no deploy job. A controller watches the registry, picks new tags by a rule, and writes them into the environment repository itself. You can tell by what's in the repository:

You can recognise each from what it leaves behind. A comment such as {"$imagepolicy": "flux-system:greeting"} at the end of an image: line, with ImagePolicy and ImageUpdateAutomation objects, is Flux's image automation, which commits to the branch its ImageUpdateAutomation names. An ImageUpdater object, or the annotation argocd-image-updater.argoproj.io/image-list on an Application, is Argo CD Image Updater, which writes either the Application's parameters or commits to Git.

If a tag changed and no pipeline ran, look for these. MegaCorp doesn't use them: it wants a pipeline, with its tests, to decide what gets deployed.

GitLab's agent for Kubernetes#

The agent for Kubernetes is a program installed in a cluster that keeps a connection open to GitLab. It is available in every tier, and does two jobs:

  • Push, through the agent. Projects and groups listed under ci_access in the agent's configuration get a kubeconfig file in every job, at $KUBECONFIG, with a context for each agent they may use. kubectl and helm then reach the cluster without any cluster credentials stored in GitLab.
  • Pull, with Flux. When the agent sees a push to a repository that a Flux GitRepository watches, it can tell Flux to sync at once. Otherwise Flux checks on its own interval.
Note
The agent can trigger an immediate sync only for its own configuration project and for public projects. For a private environment repository, such as gitops-config, Flux syncs on its interval, so a deploy can appear minutes after the pipeline goes green.
The pipeline is green, but the new version isn't running. Ask these in order.noyesnoyesnoyesnoyesnoyesDid the deployer's job run, or is it amanual job waiting?A waiting manual job deploysnothingDid the commit reach the environmentrepository?Read the deploy job's log; thepush may have been refusedFor production, was the merge requestmerged?Nothing deploys until someonemerges itHas the controller synced since thatcommit?It syncs on its interval; wait, orask for a syncDid the new pods start and stay up?The cluster's events show why; thepipeline never willCheck which image the environmentrepository now names
Figure The pipeline is green, but the new version isn't running. Ask these in order.#
As text
  1. Did the deployer's job run, or is it a manual job waiting? No: A waiting manual job deploys nothing. Yes: the next step.
  2. Did the commit reach the environment repository? No: Read the deploy job's log; the push may have been refused. Yes: the next step.
  3. For production, was the merge request merged? No: Nothing deploys until someone merges it. Yes: the next step.
  4. Has the controller synced since that commit? No: It syncs on its interval; wait, or ask for a sync. Yes: the next step.
  5. Did the new pods start and stay up? No: The cluster's events show why; the pipeline never will. Yes: the next step.
  6. Check which image the environment repository now names
When you write it
Do this
When your pipeline hands off to GitOps, name the job for what it does, such as request-deploy, and give its environment the URL where the deploy can be watched.
Why
The job ends when the change is committed to the environment repository, not when the service is running. The name says so, and the URL shows where to look next.
Watch out
People read a green job as a finished deploy. The card deploys appear minutes later shows what the gap looks like.
What you can now do
  1. Read an environment repository and say what runs where, from its files alone.
  2. Follow a tag bump from the pipeline's commit to the controller's sync.
  3. Promote a version from staging to production the way the repository expects.
Spot the bugGreen, but still the old version#

At 10:02 a payments-api pipeline on main goes green, including deploy-staging and the deployer pipeline it triggered. At 10:05, staging still serves the old version. MegaCorp's clusters run Flux, connected through GitLab's agent, and gitops-config is a private project. By 10:12 the new version is live, and nobody changed anything.

QuestionWhat happened between 10:02 and 10:12, and where would you have looked at 10:05?
Show the answer

The pipelines ended when mc deploy pushed the new tag to gitops-config. Because that repository is private, the agent couldn't trigger an immediate sync, so Flux picked the change up on its next scheduled check. The pods then rolled out. None of this shows in GitLab.

At 10:05, check the latest commit in gitops-config, then the controller's last sync, then the cluster's rollout. See the job that bumps a tag.

See deploys that appear minutes later.

LabPromote by merge requestabout 10 min#
You need

An account on your GitLab, git, and two tags of the image you pushed in chapter 8. No cluster.

Steps
  1. Create a blank project named greeting-gitops. Add envs/production/greeting.yaml: your values from the Helm lab (render the shared chart), with the older tag. Push to main.
  2. Change the tag to the newer one, and commit.
  3. Push the commit to a new branch, and ask for a merge request:
    git push origin HEAD:deploy/greeting-<tag> -o merge_request.create \
      -o merge_request.target=main -o merge_request.title="Deploy greeting <tag> to production"
  4. Open the merge request, and read its changes.
What you should see

GitLab answers on the push with the merge request's link:

git pushoutputcaptured at the baseline
remote: 
remote: View merge request for deploy/greeting-7d625fae:        
remote:   https://gitlab.example.com/practice/greeting-gitops/-/merge_requests/3        
remote:

The merge request changes one line of envs/production/greeting.yaml: the tag. Merging it is the deploy's approval.

Break it on purpose

Push the same commit to another new branch, without the -o options. GitLab only offers a link to create a merge request, and nobody is asked:

git pushoutputcaptured at the baseline
remote: 
remote: To create a merge request for deploy/greeting-7d625fae-plain, visit:        
remote:   https://gitlab.example.com/practice/greeting-gitops/-/merge_requests/new?merge_request%5Bsource_branch%5D=deploy%2Fgreeting-7d625fae-plain        
remote:

Flux#

Part X · After the pipeline · Chapter 46 · 11 min read

Flux is a set of controllers that keep a cluster in step with Git. A GitRepository says what to fetch, and how often. A Kustomization says which folder to apply, how often, and what to do about drift. A HelmRelease installs a chart with its values. MegaCorp's platform team uses Flux to install each cluster's add-ons, in order, from gitops-config. GitLab's agent for Kubernetes can tell Flux about each push, so that it syncs at once instead of at its next check, but only for public projects and the agent's own project.

Wednesday, 14:40. The change that never arrived.

A merged change to the monitoring folder never reaches the cluster. Flux's other Kustomizations reconcile every minute. This one's row says dependency 'flux-system/add-ons' is not ready, and has for an hour. It waits for another Kustomization, which itself waits for a HelmRelease that is failing its health check. (the card)

Flux's objects#

KindapiVersionWhat it does
GitRepositorysource.toolkit.fluxcd.io/v1fetches a repository at a branch or tag, every interval
HelmRepositorysource.toolkit.fluxcd.io/v1fetches a chart repository's index
Kustomizationkustomize.toolkit.fluxcd.io/v1builds one folder of a source with Kustomize, and applies the result, every interval
HelmReleasehelm.toolkit.fluxcd.io/v2installs or upgrades a chart, with values, as a Helm release
Receivernotification.toolkit.fluxcd.io/v1a webhook that tells Flux to fetch now
Provider and Alertnotification.toolkit.fluxcd.io/v1beta3send Flux's events elsewhere, such as to Grafana or a chat
ImagePolicy and ImageUpdateAutomationimage.toolkit.fluxcd.io/v1pick new image tags, and commit them to Git (automation)

Two things share a name. Flux's Kustomization is an object in the cluster that applies a folder. Kustomize's kustomization.yaml is a file in that folder, which says how to build it (chapter 44). A folder without one still works: Flux generates it, for all the manifests in the folder.

Flux's objects on eks-prod. A GitRepository fetches gitops-config. The flux-system Kustomization applies the cluster's folder, which holds three more Kustomizations that go in order: the add-ons, then their settings, then the ApplicationSets that Argo CD deploys. GitLab's agent can tell Flux about a push through a Receiver. GitRepository main, every minute Kustomization ./clusters/eks-prod GitLab's agent a push, via a Receiver add-ons waits for their health config after add-ons is ready apps after config is ready HelmReleases External Secrets, Argo CD ClusterSecretStore and other settings ApplicationSets which Argo CD deploys
Figure Flux's objects on eks-prod. A GitRepository fetches gitops-config. The flux-system Kustomization applies the cluster's folder, which holds three more Kustomizations that go in order: the add-ons, then their settings, then the ApplicationSets that Argo CD deploys. GitLab's agent can tell Flux about a push through a Receiver.#

How eks-prod's Flux starts#

flux bootstrap gitlab installs Flux in a cluster, and commits two objects that point Flux at the repository it came from:

megacorp/platform/gitops-configclusters/eks-prod/flux-system/gotk-sync.yamlsource
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: flux-system
  url: https://gitlab.example.com/megacorp/platform/gitops-config.git
megacorp/platform/gitops-configclusters/eks-prod/flux-system/gotk-sync.yamlsync
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./clusters/eks-prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

Flux fetches main every minute. At least every ten minutes, it applies ./clusters/eks-prod, and with prune: true it deletes whatever that folder no longer holds. Everything else on the cluster follows from that folder.

Add-ons, then settings, then applications#

eks-prod's folder holds three more Kustomizations. The first installs the add-ons:

megacorp/platform/gitops-configclusters/eks-prod/platform.yamladd-ons
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: add-ons
  namespace: flux-system
spec:
  interval: 10m
  retryInterval: 1m
  path: ./infrastructure/add-ons/base
  prune: true
  wait: true
  sourceRef:
    kind: GitRepository
    name: flux-system

With wait: true, the Kustomization is ready only once every object it applied is healthy. retryInterval retries a failure after a minute, instead of ten. The second waits for the first:

megacorp/platform/gitops-configclusters/eks-prod/platform.yamlconfig
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: config
  namespace: flux-system
spec:
  dependsOn:
    - name: add-ons
  interval: 10m
  path: ./infrastructure/config/eks
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

dependsOn applies config only after add-ons is ready. The order matters: the ClusterSecretStore in config is a kind that the External Secrets Operator's chart adds, so it can't be applied before that chart is installed. The third, apps, waits for config in the same way, and applies the ApplicationSets that Argo CD turns into applications (chapter 47).

While a dependency isn't ready, flux get kustomizations says so. This is the local cluster, where apps waits directly on add-ons, a minute after Flux started installing the add-ons:

fluxoutputcaptured at the baseline
NAME   	REVISION	SUSPENDED	READY  	MESSAGE                                       
add-ons	        	False    	Unknown	Reconciliation in progress                   	
apps   	        	False    	False  	dependency 'flux-system/add-ons' is not ready

Charts through Flux#

An add-on is a HelmRelease, which names a chart, its version, where to fetch it, and its values:

megacorp/platform/gitops-configinfrastructure/add-ons/base/add-ons.yamlexternal-secrets
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: external-secrets
  namespace: external-secrets
spec:
  interval: 30m
  chart:
    spec:
      chart: external-secrets
      version: 2.10.0
      sourceRef:
        kind: HelmRepository
        name: external-secrets
        namespace: flux-system
  install:
    crds: CreateReplace
  upgrade:
    crds: CreateReplace
  values:
    serviceAccount:
      name: external-secrets

The HelmRepository it names is in the same file. crds: CreateReplace makes Flux create the chart's custom resource definitions, and replace them when a new chart version changes them. By default, Flux creates them on install and skips them on upgrade. Flux installs a real Helm release, so helm list shows it, unlike an Argo CD application (no release in helm list).

Each Helm action has five minutes by default, including the chart's hook Jobs. An install that runs out of time stays failed: Flux tries again only if install.remediation.retries says so, and the default is 0. On the local cluster, Argo CD's first install timed out while its image was still downloading:

fluxoutputcaptured at the baseline
NAMESPACE       	NAME            	REVISION	SUSPENDED	READY	MESSAGE                                                                                                                                                                               
argocd          	argo-cd         	10.9.0  	False    	False	Helm install failed for release argocd/argo-cd with chart argo-cd@10.9.0: failed pre-install: timeout waiting for: [Job/argocd/argo-cd-argocd-redis-secret-init status: 'InProgress']	
external-secrets	external-secrets	2.10.0  	False    	True 	Helm install succeeded for release external-secrets/external-secrets.v1 with chart external-secrets@2.10.0

With the image in place, flux reconcile helmrelease argo-cd -n argocd --force installed it (a HelmRelease that timed out).

Drift and pruning#

  • Drift. Every ten minutes, a Kustomization compares the cluster with Git, and puts back what differs. The annotation kustomize.toolkit.fluxcd.io/ssa changes that for one object: Override reverts every change, and is the default; Merge keeps fields that other tools added; IfNotPresent creates the object and then leaves it alone.
  • Pruning. With prune: true, objects that Flux applied earlier, but that the current revision no longer holds, are deleted from the cluster.

GitLab's agent, and when Flux syncs#

Left alone, Flux sees a new commit only at its next fetch. GitLab's agent for Kubernetes, in every tier, can shorten that. It finds the GitRepository objects that point at GitLab projects, sets up a Receiver for each, and calls it on every push, so Flux fetches at once. The cluster needs Flux's source and notification controllers as well as the agent.

There's a catch. The agent can reach only its own configuration project and public projects. A private environment repository, like gitops-config, is still synced on Flux's own schedule (deploys appear minutes later).

On the local cluster, once everything had synced, Flux showed this:

fluxoutputcaptured at the baseline
NAME   	REVISION          	SUSPENDED	READY	MESSAGE                              
add-ons	main@sha1:9f3c2960	False    	True 	Applied revision: main@sha1:9f3c2960	
apps   	main@sha1:9f3c2960	False    	True 	Applied revision: main@sha1:9f3c2960
fluxoutputcaptured at the baseline
NAMESPACE       	NAME            	REVISION	SUSPENDED	READY	MESSAGE                                                                                                    
argocd          	argo-cd         	10.9.0  	False    	True 	Helm upgrade succeeded for release argocd/argo-cd.v2 with chart argo-cd@10.9.0                            	
external-secrets	external-secrets	2.10.0  	False    	True 	Helm install succeeded for release external-secrets/external-secrets.v1 with chart external-secrets@2.10.0

The local cluster applies add-ons and apps only: its secret store is a stand-in, so it has no config. Argo CD's release is at revision 2, because its first install timed out and was forced again.

What you can now do
  1. Read Flux's sources, Kustomizations and HelmReleases, and say what each watches and applies.
  2. Ask Flux why a change hasn't arrived, with three commands.
  3. Trace a dependency chain to the object that is really stuck.
LabLet Flux deploy greeting-serviceabout 20 min#
You need

A cluster from the lab kit, with Flux installed by flux install; the kustomize folder from the Kustomize lab; and the image you pushed in chapter 8. No cluster? Do the first step, then run kustomize build kustomize/overlays/staging: that is exactly what Flux applies, and the expected output shows what it did with it.

Steps
  1. On your GitLab, create a project called greeting-gitops, and push the kustomize folder to it.
  2. Add two deploy tokens, each under Settings › Repository, in Deploy tokens. In greeting-gitops, add one with the read_repository scope, for Flux. In the project that built your image, add one with the read_registry scope, for the cluster. Neither can reach any other project.
  3. Make the namespace, and let its pods pull your image with the second token:
    kubectl create namespace greeting-staging
    kubectl create secret docker-registry gitlab-registry -n greeting-staging \
      --docker-server=<your registry> --docker-username=<pull token's username> --docker-password=<pull token>
    kubectl patch serviceaccount default -n greeting-staging -p '{"imagePullSecrets": [{"name": "gitlab-registry"}]}'
  4. Give Flux the first token, then point it at the repository and at staging's overlay:
    flux create secret git greeting --url=https://<your GitLab>/<your group>/greeting-gitops.git \
      --username=<token's username> --password=<token>
    flux create source git greeting --url=https://<your GitLab>/<your group>/greeting-gitops.git \
      --branch=main --secret-ref=greeting --interval=1m
    flux create kustomization greeting --source=GitRepository/greeting \
      --path=./kustomize/overlays/staging --prune=true --interval=1m --wait=true
    If your GitLab's certificate comes from a private authority, add --ca-crt-file=<its certificate> to the first command.
  5. List the pods: kubectl get pods -n greeting-staging.
  6. In the overlay, change newTag to another tag of your image, then commit and push. Wait a minute or two, run flux get kustomizations greeting, and list the pods with their images.
What you should see

Flux applies the overlay, and waits until everything it applied is healthy:

fluxoutputcaptured at the baseline
✚ generating Kustomization
► applying Kustomization
✔ Kustomization created
◎ waiting for Kustomization reconciliation
✔ Kustomization greeting is ready
✔ applied revision main@sha1:dde381ae59c419f817c765433d13371b651adf72

Two pods run, as the overlay's patch says:

kubectloutputcaptured at the baseline
NAME                       READY   STATUS    RESTARTS   AGE
greeting-b7f6b6f75-fhvgq   1/1     Running   0          17s
greeting-b7f6b6f75-x8cgt   1/1     Running   0          17s

After the push, the Kustomization shows the new commit, and the pods run the new tag:

fluxoutputcaptured at the baseline
NAME    	REVISION          	SUSPENDED	READY	MESSAGE                              
greeting	main@sha1:fd092043	False    	True 	Applied revision: main@sha1:fd092043
kubectloutputcaptured at the baseline
NAME                        IMAGE
greeting-744b5b7bc4-lnq5g   gitlab.example.com:5050/practice/node-service:7d625fae
greeting-744b5b7bc4-wwtwm   gitlab.example.com:5050/practice/node-service:7d625fae
When you write it
Do this
For each Flux Kustomization you add, set prune: true. Add dependsOn when it needs another Kustomization's objects first, and wait: true when others depend on it.
Why
With prune, a file deleted from Git is deleted from the cluster too. Without it, old objects stay behind. The two ordering settings make Flux apply things in the right order.
Watch out
Keep the interval at one minute or more. Flux accepts nothing shorter, and a very short interval only makes the controller busy.
Spot the bugThe setting that waited#

A merge request adds a second ClusterSecretStore to infrastructure/config/eks. Twenty minutes after it was merged, the store still isn't on eks-prod. The add-ons Kustomization shows the External Secrets Operator's HelmRelease failing, since a chart upgrade that someone merged the day before.

megacorp/platform/gitops-configclusters/eks-prod/platform.yamlconfig
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: config
  namespace: flux-system
spec:
  dependsOn:
    - name: add-ons
  interval: 10m
  path: ./infrastructure/config/eks
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
QuestionWhy hasn't the new store appeared, and what do you fix first?
Show the answer

config depends on add-ons, and add-ons waits for the health of everything it applied. While the HelmRelease fails, add-ons isn't ready, so Flux never applies config, whatever it holds. Fix the failing HelmRelease first; config follows by itself (a Kustomization waiting for another).

Argo CD#

Part X · After the pipeline · Chapter 47 · 12 min read

Argo CD keeps applications in a cluster in step with Git, and shows them in a web UI. An Application names what to deploy, from which repositories, to which cluster and namespace. An ApplicationSet makes Applications from a template, such as one for each environment. Every Application has two statuses: sync, which compares the cluster with Git, and health, which says whether what runs works. MegaCorp's teams deploy with Argo CD: the payments team's ApplicationSet makes payments-api's Applications, for staging and production, on each cluster.

Thursday, 10:30. Not permitted.

Argo CD's sync fails with resource :Namespace is not permitted in project. The Application is fine. The manifests are fine. The AppProject the Application belongs to does not allow it to create namespaces, and the new chart version tries to. (the card)

Argo CD's objects#

KindWhat it saysAt MegaCorp
AppProjectwhich repositories a team's Applications may use, where they may deploy, and which cluster-wide kinds they may createpayments: the charts and gitops-config, into payments-* namespaces, which it may create (a namespace the project doesn't allow)
Applicationone set of objects: its sources, its destination, and how to syncmade by the ApplicationSet, one for each environment
ApplicationSeta template, and a generator that fills it in to make Applicationspayments-api, with a list of two environments

All three are argoproj.io/v1alpha1 objects in the argocd namespace, where Argo CD runs.

payments-api's ApplicationSet#

megacorp/platform/gitops-configapps/eks-prod/applicationset.yamlproject
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payments
  namespace: argocd
spec:
  description: The payments team's services
  sourceRepos:
    - https://gitlab.example.com/megacorp/platform/charts.git
    - https://gitlab.example.com/megacorp/platform/gitops-config.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: payments-*
  # CreateNamespace makes each environment's Namespace, which is cluster-wide, so the
  # project must allow that kind, and only with the team's names
  clusterResourceWhitelist:
    - group: ""
      kind: Namespace
      name: payments-*
megacorp/platform/gitops-configapps/eks-prod/applicationset.yamlapplicationset
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payments-api
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - list:
        elements:
          - env: staging
          - env: production
  template:
    metadata:
      name: "payments-api-{{ .env }}"
      annotations:
        notifications.argoproj.io/subscribe.on-deployed.grafana: "payments-api|{{ .env }}"
    spec:
      project: payments
      sources:
        - repoURL: https://gitlab.example.com/megacorp/platform/charts.git
          targetRevision: service-3.4.0
          path: service
          helm:
            releaseName: payments-api
            valueFiles:
              - "$values/envs/{{ .env }}/payments-api.yaml"
        - repoURL: https://gitlab.example.com/megacorp/platform/gitops-config.git
          targetRevision: main
          ref: values
      destination:
        server: https://kubernetes.default.svc
        namespace: "payments-{{ .env }}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
  • The list generator gives two elements, staging and production. The template is filled in once for each, with {{ .env }} replaced.
  • There are two sources. The first is the shared chart, at the tag service-3.4.0. The second is gitops-config itself, named values, so that the chart's valueFiles can start with $values and read each environment's file from it.
  • The destination, https://kubernetes.default.svc, is the cluster Argo CD runs in, and the namespace is made if it's missing.
  • The sync policy is automated, with pruning and self-healing.

Sync and health#

StatusMeans
Syncedthe cluster matches Git, at the revision shown
OutOfSyncsomething in the cluster differs from Git: a new commit, or a change by hand
Healthyevery object works; a Deployment is healthy once all its pods are updated and ready
Progressingnot healthy yet, but still making progress
Degradedsomething has failed, such as a Deployment whose rollout missed its deadline
Suspendedwaiting for something outside to resume it
Missingthe object isn't in the cluster
UnknownArgo CD can't judge the object's health

The two are separate. An Application can be Synced and Degraded: Git and the cluster agree, and what they agree on doesn't work. An Application takes the worst health of its objects.

Argo CD's two statuses. Argo CD compares what Git says should run with what runs in the cluster: the sync status. It also judges whether what runs works: the health status. Automated sync turns OutOfSync back into Synced. Git: what should run the chart and the values the cluster: what runs the objects, and their pods Argo CD compares every two minutes or so sync status Synced: the cluster matches Git OutOfSync: something differs automated sync applies Git again health status Healthy · Progressing · Degraded Suspended · Missing · Unknown the worst of its objects wins
Figure Argo CD's two statuses. Argo CD compares what Git says should run with what runs in the cluster: the sync status. It also judges whether what runs works: the health status. Automated sync turns OutOfSync back into Synced.#

How it syncs#

Argo CD checks Git about every two minutes, with up to a minute's random delay added. With automated sync, it syncs any Application that is OutOfSync. Two options, both off by default, make it stricter: prune deletes objects that Git no longer holds, and selfHeal puts back a change made by hand, about five seconds after it happens.

Within a sync, objects can go in order. The annotation argocd.argoproj.io/sync-wave gives an object a wave number, 0 by default; lower waves go first. argocd.argoproj.io/hook turns an object into a hook that runs at a phase of the sync, such as a Job that migrates a database before anything else changes:

metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync

Values, and what wins#

Argo CD renders a chart itself, with helm template, and applies the result, so no Helm release exists (no release in helm list). When the same value is set in several places, the order is: parameters, then inline valuesObject, then inline values, then value files, then the chart's own values.yaml. On aks-prod, the ApplicationSet sets the image's repository inline, so it beats the values file's:

megacorp/platform/gitops-configapps/aks-prod/applicationset.yamlazure-values
valuesObject:
  image:
    repository: megacorpprod.azurecr.io/payments-api
  serviceAccount:
    annotations:
      azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
  podLabels:
    azure.workload.identity/use: "true"

In the UI, and from the command line#

The UI shows each Application with its sync and health, and a tree of the objects it made, down to the pods. From there you can sync by hand, see what differs from Git, and run built-in actions such as restarting a Deployment. Argo CD's notifications can also mark each deploy on your dashboards (chapter 52).

A sketch of payments-api's staging Application in Argo CD. The buttons sync by hand, show what differs from Git, and roll back; DIFF is greyed out while nothing differs. The panel shows the application's health and sync, and the tree shows each object it made, down to the pod. A heart marks health and a tick marks sync. The ReplicaSet, the pod and the Secret aren't in Git, so they have no sync status, and a Secret has no health. Applications / payments-api-staging APPLICATION DETAILS TREE DETAILS DIFF SYNC SYNC STATUS HISTORY AND ROLLBACK DELETE REFRESH APP HEALTH Healthy SYNC STATUS Synced to service-3.4.0 and (1) more Auto sync is enabled. LAST SYNC Sync OK to 56f428c and (1) more Succeeded 17 minutes ago payments-api-staging payments-api svc payments-api sa payments-api deploy payments-api externalsecret payments-api ing payments-api-796d6777f9 rs payments-api secret payments-api-796d6777f9-… pod 1/1 running not in Git, so no sync status
Figure A sketch of payments-api's staging Application in Argo CD. The buttons sync by hand, show what differs from Git, and roll back; DIFF is greyed out while nothing differs. The panel shows the application's health and sync, and the tree shows each object it made, down to the pod. A heart marks health and a tick marks sync. The ReplicaSet, the pod and the Secret aren't in Git, so they have no sync status, and a Secret has no health.#

argocd app get shows the same from the command line. This is payments-api in staging on the local cluster, with each object's sync and health:

argocdoutputcaptured at the baseline
Name:               argocd/payments-api-staging
Project:            payments
Server:             https://kubernetes.default.svc
Namespace:          payments-staging
URL:                https://argocd.example.com/applications/payments-api-staging
Sources:
- Repo:             https://gitlab.example.com/megacorp/platform/charts.git
  Target:           service-3.4.0
  Path:             service
  Helm Values:      $values/envs/staging/payments-api.yaml
- Repo:             https://gitlab.example.com/megacorp/platform/gitops-config.git
  Target:           main
  Ref:              values
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to service-3.4.0
Health Status:      Healthy

GROUP                KIND            NAMESPACE         NAME              STATUS   HEALTH   HOOK  MESSAGE
                     Namespace                         payments-staging  Running  Synced         namespace/payments-staging created
                     ServiceAccount  payments-staging  payments-api      Synced                  serviceaccount/payments-api created
                     Service         payments-staging  payments-api      Synced   Healthy        service/payments-api created
apps                 Deployment      payments-staging  payments-api      Synced   Healthy        deployment.apps/payments-api created
networking.k8s.io    Ingress         payments-staging  payments-api      Synced   Healthy        ingress.networking.k8s.io/payments-api created
external-secrets.io  ExternalSecret  payments-staging  payments-api      Synced   Healthy        externalsecret.external-secrets.io/payments-api created
What you can now do
  1. Read an Application, an ApplicationSet and an AppProject, and say what each allows and deploys.
  2. Read sync and health status, and know what OutOfSync, Progressing and Degraded each mean.
  3. Find why a sync failed from the Application's own status, before asking anyone.
LabThe same service, with Argo CDabout 15 min#
You need

The cluster and the project from the Flux lab, and Argo CD on the cluster. Install it from the chart MegaCorp's platform team uses:

helm install argo-cd argo-cd --repo https://argoproj.github.io/argo-helm --version 10.9.0 \
  --namespace argocd --create-namespace

No cluster? Read the steps and the expected output: Argo CD applies the same overlay that kustomize build shows.

Steps
  1. Let Flux let go of greeting-service. With pruning on, deleting the Kustomization deletes what it applied:
    flux delete kustomization greeting --silent
    flux delete source git greeting --silent
  2. Use the argocd command through your kube context, without logging in to a server:
    kubectl config set-context --current --namespace=argocd
    argocd login --core
  3. Give Argo CD the repository's deploy token, then create the application:
    argocd repo add https://<your GitLab>/<your group>/greeting-gitops.git \
      --username <token's username> --password <token>
    argocd app create greeting --repo https://<your GitLab>/<your group>/greeting-gitops.git \
      --path kustomize/overlays/staging --dest-server https://kubernetes.default.svc \
      --dest-namespace greeting-staging --sync-policy automated --auto-prune --self-heal
    If your GitLab's certificate comes from a private authority, run argocd cert add-tls <its host name> --from <its certificate> first.
  4. Wait a minute, then run argocd app get greeting.
What you should see
argocdoutputcaptured at the baseline
application 'greeting' created

The application is Synced and Healthy, with the objects it made:

argocdoutputcaptured at the baseline
Name:               argocd/greeting
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          greeting-staging
URL:                https://argocd.example.com/applications/greeting
Source:
- Repo:             https://gitlab.example.com/practice/greeting-gitops.git
  Target:           
  Path:             kustomize/overlays/staging
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to  (fd09204)
Health Status:      Healthy

GROUP  KIND        NAMESPACE         NAME      STATUS  HEALTH   HOOK  MESSAGE
       Service     greeting-staging  greeting  Synced  Healthy        service/greeting created
apps   Deployment  greeting-staging  greeting  Synced  Healthy        deployment.apps/greeting created
When you write it
Do this
For an Application you own, turn on automated sync with prune and selfHeal. Keep settings in value files.
Why
Git then stays the truth: a change in Git is applied, a deleted object is removed, and a change made by hand in the cluster is undone.
Watch out
Use inline values only for what really differs between clusters. An inline value beats the same setting in a value file, and reviewers looking at the value files will not see it.
Spot the bugSynced, and Degraded#

After a deploy, payments-api-staging shows Synced and Progressing for ten minutes. Then it shows Synced and Degraded. Nobody has changed anything since the deploy's commit.

QuestionWhat do the two statuses tell you, and where do you look next?
Show the answer

Synced means the cluster matches the deploy's commit: Argo CD did its part. Degraded means what it applied doesn't work. The Deployment was Progressing while its new pods weren't ready, and after its ten-minute deadline it counts as failed. Look at the new pods: their events and logs say why they never became ready (a rollout stuck half-way).

OpenTofu and Terraform#

Part X · After the pipeline · Chapter 48 · 13 min read

The clusters, registries and identities that delivery runs on are made from code, with OpenTofu or Terraform. The platform team writes modules; a root configuration calls them with inputs; plan shows what would change, and apply makes it so. The state records what exists, and a lock lets only one run write it at a time. You met plan-in-a-merge-request and apply-on-merge in Part III. This chapter adds what surrounds them: modules, reading a plan, the state's lock, and who changes what.

Thursday, 15:15. Error acquiring the state lock.

Every plan fails at once: Error acquiring the state lock. A colleague's apply was cancelled mid-way an hour ago, and the lock it took is still there. Someone suggests -lock=false. Someone else says that is how you corrupt a state. Who is right? (the card)

Two names, one language#

OpenTofu is a community fork of Terraform. It is open source, under the Mozilla Public License, and has been a CNCF Sandbox project since April 2025. Terraform is under the Business Source License. The two read the same language, HCL, and most configurations work in both. Where they differ, you will notice:

WhatOpenTofuTerraform
the commandtofuterraform
where hashicorp/aws comes fromregistry.opentofu.orgregistry.terraform.io
.tofu filesread; a .tofu file wins over a .tf file of the same namenot read
the licenceMozilla Public License 2.0Business Source License 1.1

OpenTofu can also encrypt state and plan files. The rest of this chapter says OpenTofu, and means both.

Modules, and the root configuration#

A module is a folder of .tf files with inputs, called variables, and outputs. MegaCorp's platform team keeps one for each cloud in infra-modules. This is what the AWS one asks for:

megacorp/platform/infra-modulesaws/main.tfinputs
variable "cluster_name" {
  type = string
}

variable "kubernetes_version" {
  type    = string
  default = "1.36"
}

variable "subnet_ids" {
  type = list(string)
}

variable "repositories" {
  description = "ECR repositories to create, such as payments-api"
  type        = set(string)
}

variable "workloads" {
  description = "Pods that get an AWS role, keyed by name, each with its namespace and service account"
  type = map(object({
    namespace       = string
    service_account = string
  }))
}

For each workload, it makes an IAM role that EKS Pod Identity hands to that namespace's service account:

megacorp/platform/infra-modulesaws/main.tfidentity
resource "aws_iam_role" "workload" {
  for_each = var.workloads
  name     = "${var.cluster_name}-${each.key}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "pods.eks.amazonaws.com" }
      Action    = ["sts:AssumeRole", "sts:TagSession"]
    }]
  })
}

resource "aws_eks_pod_identity_association" "workload" {
  for_each        = var.workloads
  cluster_name    = aws_eks_cluster.this.name
  namespace       = each.value.namespace
  service_account = each.value.service_account
  role_arn        = aws_iam_role.workload[each.key].arn
}

A module does nothing on its own. A root configuration, the folder where you run tofu plan, calls it with inputs. MegaCorp's is infra-live. It names each module's Git repository and a tag, so a module's new version reaches production only when someone changes the tag:

megacorp/platform/infra-livemain.tfeks
module "eks_prod" {
  source       = "git::https://gitlab.example.com/megacorp/platform/infra-modules.git//aws?ref=aws-2.3.0"
  cluster_name = "eks-prod"
  subnet_ids   = ["subnet-0a1b2c3d4e5f60718", "subnet-0b2c3d4e5f6071829"]

  repositories = ["payments-api", "web-portal"]
  workloads = {
    payments-api-staging    = { namespace = "payments-staging", service_account = "payments-api" }
    payments-api-production = { namespace = "payments-production", service_account = "payments-api" }
  }
}
OpenTofu in the delivery path. A root configuration calls the platform team's modules. Plan compares it with the state; apply makes the changes, holding the state's lock, and saves the new state. What it makes is what the rest of delivery runs on. modules the platform team's, at a pinned version a root configuration calls modules with a team's inputs tofu plan what would change in a merge request tofu apply makes the changes, on main the state, and its lock kept by GitLab or a cloud; one writer at a time reads locks, writes what delivery runs on clusters and registries identities for pipelines and pods, queues, buckets, databases
Figure OpenTofu in the delivery path. A root configuration calls the platform team's modules. Plan compares it with the state; apply makes the changes, holding the state's lock, and saves the new state. What it makes is what the rest of delivery runs on.#

Reading a plan in a merge request#

A plan starts each change with a symbol. + creates, - destroys, and ~ changes in place. -/+ destroys and creates again, because something that can't change in place did. Here is team-infra's plan from chapter 10:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ tofu plan -input=false -out=plan.cache
Acquiring state lock. This may take a few moments...
random_pet.queue: Refreshing state... [id=payments-helped-sunfish]
OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
OpenTofu will perform the following actions:
  # random_pet.queue must be replaced
-/+ resource "random_pet" "queue" {
      ~ id        = "payments-helped-sunfish" -> (known after apply)
      ~ length    = 2 -> 3 # forces replacement
        # (2 unchanged attributes hidden)
    }
Plan: 1 to add, 0 to change, 1 to destroy.
Changes to Outputs:
  ~ queue_name = "payments-helped-sunfish" -> (known after apply)
⋯ lines omitted
Saved the plan to: plan.cache
⋯ lines omitted
Releasing state lock. This may take a few moments...
⋯ lines omitted
plan.json: found 1 matching artifact files and directories
Uploading artifacts as "terraform" to coordinator... 201 Created  correlation_id=01M2DR15CECXP68NQ0SWZNPXZ2 id=303 status=201 token=[REDACTED]
⋯ lines omitted
Job succeeded

Read every must be replaced line before you merge. For a random name, a replacement costs nothing. For a database, a registry or a cluster, it can mean losing what it held. # forces replacement marks the line that caused it, and the counts at the end are what the merge request shows.

State, and its lock#

The state is OpenTofu's record of what it has made. A team's state lives remotely: in GitLab, as in Part III, or in cloud storage. While a run writes it, the backend locks it, so a second run can't write at the same time. That second run stops at once:

GitLab ranjob logcaptured from GitLab
⋯ lines omitted
$ tofu plan -input=false -out=plan.cache
Acquiring state lock. This may take a few moments...
╷
│ Error: Error acquiring the state lock
│
│ Error message: HTTP remote state already locked:
│ ID=5b0e8c3a-7d2f-4c1e-9a64-3f8b2d1c0e97
│ Lock Info:
│   ID:        5b0e8c3a-7d2f-4c1e-9a64-3f8b2d1c0e97
│   Path:
│   Operation: OperationTypePlan
│   Who:       root
│   Version:   1.12.6
│   Created:   2026-09-13 20:23:16.966 +0000 UTC
│   Info:
│
│
│ OpenTofu acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵
⋯ lines omitted
ERROR: Job failed: exit code 1

Only three lines describe the lock that's in the way: ID, Who and Created. With GitLab-managed state, Who is the GitLab user whose job took the lock. The other lines describe your own run: GitLab fills Operation and Version in from the request that failed, which is why this blocked plan says OperationTypePlan.

Usually the other run is still going, and waiting is the fix. A lock can outlive its run, though, when an apply is cancelled or its runner dies. Then:

  1. Make sure no job is still running against this state.
  2. Unlock it. In GitLab, open Operate › Terraform states, and use the state's Actions menu; this needs the Maintainer or Owner role. Or run tofu force-unlock <ID>, which removes the lock without changing any resources.
  3. Run the pipeline again, and read its plan: the interrupted apply may have done part of its work.

Don't reach for -lock=false. It lets two runs write the same state, and the second overwrites what the first recorded.

What your team changes, and what the platform team owns#

You wantWhere the change goesWho applies it
a registry repository for a new servicean entry in infra-live's repositoriesthe platform team, after reviewing your merge request
a cloud identity for your podsan entry in infra-live's workloadsthe platform team, likewise
a queue, bucket or database of your ownyour team's root configuration, if your organisation has one, calling the platform's modulesyour team's pipeline, as in the Build part
the cluster, its nodes, its networknowhere you can changethe platform team

So a team's usual change is a small merge request to someone else's repository. Read the plan in it as carefully as your own.

When you write it
Do this
Pin every module to a tag, never to a branch. Before you merge, read every line in the plan that says must be replaced.
Why
A pinned module changes only when you change it. A replaced resource is destroyed and created again, which can lose data or cause downtime.
Watch out
Never edit a state file by hand, and never apply with -lock=false. If a lock is stale, unlock it deliberately with the unlock command, so that two runs never write the state at once.
What you can now do
  1. Read a plan and say what will be created, changed, destroyed or replaced.
  2. Unlock a stale state safely, and know why applying without a lock is never the answer.
  3. Pin modules, and read a plan that wants to change something nobody edited.
Spot the bugLocked for forty minutes#

team-infra's plan job fails with the lock error above. Its Created time was forty minutes ago. The project's pipelines show an apply job that was cancelled forty minutes ago, and nothing running now.

QuestionWhat happened, and what do you do, in which order?
Show the answer

The cancelled apply took the lock and never released it, so the lock outlived its run. With nothing running, unlock the state: Operate › Terraform states, the state's Actions menu, Unlock, or tofu force-unlock with the lock's ID. Then run the pipeline again, and read the new plan before anything applies: the cancelled apply may have made some of its changes. Don't use -lock=false to get past it.

See state locked.

AWS and Azure for delivery#

Part X · After the pipeline · Chapter 49 · 8 min read

EKS and AKS are Kubernetes run for you by AWS and by Azure. For a pipeline owner, what differs between them is identity: how a pod gets the right to read a secret or a bucket, and how a pipeline signs in when it must. Neither needs a stored password. A pod proves who it is with its service account's token, and a job with its GitLab ID token; the cloud checks that token against a trust set up once. MegaCorp runs the same service on both clouds, so this chapter shows each thing side by side.

Friday, 09:50. The pod with no identity.

A pod on AKS can't reach Key Vault. Its log says no AZURE_FEDERATED_TOKEN_FILE. The same service on EKS reaches Secrets Manager fine. The pod's service account carries the annotation; the pod doesn't carry the label. Two clouds, two ways of giving a pod an identity. (the card)

The same service on two clouds#

WhatAWSAzure
the clusterEKSAKS
the registryECRACR
a job's identityan IAM role that trusts GitLab's ID tokensan Entra app with a federated credential for GitLab
a pod's identityEKS Pod Identity, or IRSA on older setupsAKS workload identity
secretsSecrets ManagerKey Vault
logs and metricsCloudWatchAzure Monitor
Pipeline and pod identity on AWS and Azure. Both clouds trust a signed token instead of a stored secret: a GitLab ID token for a job, and a service account's token for a pod. AWS Azure a job GitLab ID token names project and ref IAM OIDC provider for GitLab, and a role whose trust policy checks aud and sub an Entra app, with a federated credential matching issuer, subject and audience a pod service account namespace/name EKS Pod Identity an association in AWS: this namespace's service account gets this role; nothing in the cluster (older: IRSA, an annotation) AKS workload identity the service account's client-id annotation, the pod's label, and a federated credential system:serviceaccount:ns:name api://AzureADTokenExchange
Figure Pipeline and pod identity on AWS and Azure. Both clouds trust a signed token instead of a stored secret: a GitLab ID token for a job, and a service account's token for a pod.#

How a pod gets a cloud identity#

On both clouds, the identity belongs to a Kubernetes service account, in its namespace. Every pod that runs as that service account gets it.

With EKS Pod Identity, the link is made in AWS, not in the cluster: an association says that one namespace's service account gets one IAM role. MegaCorp's AWS module makes a role and an association for each workload:

megacorp/platform/infra-modulesaws/main.tfidentity
resource "aws_iam_role" "workload" {
  for_each = var.workloads
  name     = "${var.cluster_name}-${each.key}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "pods.eks.amazonaws.com" }
      Action    = ["sts:AssumeRole", "sts:TagSession"]
    }]
  })
}

resource "aws_eks_pod_identity_association" "workload" {
  for_each        = var.workloads
  cluster_name    = aws_eks_cluster.this.name
  namespace       = each.value.namespace
  service_account = each.value.service_account
  role_arn        = aws_iam_role.workload[each.key].arn
}

The role trusts pods.eks.amazonaws.com, and an agent on each node hands its credentials to the pods. Nothing in the chart changes. Older setups use IRSA instead. You can tell by an eks.amazonaws.com/role-arn annotation on the service account, and a role that trusts the cluster's own OIDC provider, checking system:serviceaccount:<namespace>:<name>.

AKS workload identity needs three things to line up:

  • the service account's azure.workload.identity/client-id annotation, naming the managed identity;
  • the pod's label azure.workload.identity/use: "true", without which AKS doesn't give the pod its token;
  • a federated credential on that identity, with the cluster's issuer, the subject system:serviceaccount:<namespace>:<name>, and the audience api://AzureADTokenExchange.
megacorp/platform/infra-modulesazure/main.tfidentity
resource "azurerm_user_assigned_identity" "workload" {
  for_each            = var.workloads
  name                = "${var.cluster_name}-${each.key}"
  resource_group_name = var.resource_group_name
  location            = var.location
}

resource "azurerm_federated_identity_credential" "workload" {
  for_each  = var.workloads
  name      = each.key
  parent_id = azurerm_user_assigned_identity.workload[each.key].id
  issuer    = azurerm_kubernetes_cluster.this.oidc_issuer_url
  subject   = "system:serviceaccount:${each.value.namespace}:${each.value.service_account}"
  audience  = ["api://AzureADTokenExchange"]
}

In MegaCorp's chart, aks-prod's ApplicationSet sets the first two, through its inline values (values: the defaults, then yours).

So payments-api in payments-staging and in payments-production get different identities, even from the same chart: the namespace is part of the match.

How a pipeline signs in#

A job presents its GitLab ID token, and assumes a role whose trust policy checks the token's audience and subject. Chapter 38 covers it, with the five things that must match (five things that must match).

The job declares an ID token and hands it to the Azure CLI:

id_tokens:
  GITLAB_OIDC_TOKEN:
    aud: https://gitlab.example.com
script:
  - az login --service-principal -u "$AZURE_CLIENT_ID" -t "$AZURE_TENANT_ID" --federated-token "$GITLAB_OIDC_TOKEN"

Entra ID accepts the token when one of the app's federated credentials matches its issuer, subject and audience:

azureentra.ymlapp-registration
# The identity that image jobs sign in as. Each federated credential names one kind of
# pipeline that may use it.
acr_push_app:
  display_name: gitlab-acr-push
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: 00000000-0000-0000-0000-000000000000
  federated_credentials:
    - name: payments-api-main
      issuer: https://gitlab.example.com
      subject: project_path:megacorp/payments/payments-api:ref_type:branch:ref:main
      audiences: [https://gitlab.example.com]

The match is exact: Entra allows no wildcards in a credential. So each kind of pipeline that signs in needs a credential of its own, one for main, another for each tag or environment, and an app can have at most 20. A token that matches none fails with AADSTS70021: No matching federated identity record found for presented assertion. So does one that arrives just after its credential was added, before the credential has spread through Entra.

When a pipeline reaches a cluster#

With GitOps, a pipeline never touches the cluster. When a job must, for a push deploy or to run kubectl, it needs three things:

StepAWSAzure
sign inassume the job's roleaz login with the ID token
get a kubeconfigaws eks update-kubeconfig --name <cluster>az aks get-credentials, then kubelogin convert-kubeconfig -l azurecli
permissions in the clusteran EKS access entry for the role, with an access policy or a Kubernetes groupKubernetes roles, or Azure roles if the cluster uses them, for the identity

GitLab's agent for Kubernetes is the other way in: its ci_access gives jobs a kubeconfig without any cloud credentials for the cluster (GitLab's agent for Kubernetes).

When you write it
Do this
Give each environment's service account its own cloud identity, and keep the namespace in every trust. Use an ID token instead of cloud keys in CI/CD variables whenever the cloud accepts one.
Why
One identity per environment means staging can never act as production, and a token that expires in minutes cannot leak for long.
Watch out
On Azure, each branch, tag or environment that signs in needs its own federated credential, spelled exactly, and an application can have at most 20 of them. Plan the names before you run out.
What you can now do
  1. Say how a pod gets a cloud identity on EKS and on AKS, and what each needs on the pod, the service account and the cloud.
  2. Read an access-denied error from a pod and name the missing piece.
  3. Keep environments apart with one identity per namespace.
Spot the bugKey Vault works on one cluster only#

ExternalSecrets sync on eks-prod, but none sync on aks-prod: the External Secrets Operator there can't sign in to Key Vault. Its HelmRelease comes from aks-prod's add-ons overlay.

QuestionWhich things must line up for the Operator to get its Azure identity, and where is each one set?
Show the answer

Three things, from the Operator's side, and one more in Azure:

  • the service account's azure.workload.identity/client-id annotation, and
  • the pods' azure.workload.identity/use: "true" label, both set by the overlay's patch (a patch as part of an object);
  • a federated credential on the managed identity for system:serviceaccount:external-secrets:external-secrets, with the audience api://AzureADTokenExchange, made with the platform team's OpenTofu;
  • and a role on the vault, such as Key Vault Secrets User, for that identity.

Check them in that order. A missing label is the quietest: the pod starts, but gets no token.

See a pod without its Azure identity.

Secrets and config after the pipeline#

Part X · After the pipeline · Chapter 50 · 8 min read

A pipeline's secrets live in CI/CD variables. A running service's secrets can't: they must reach its pods without ever being written into Git in plain text. There are three common ways. An operator copies them from the cloud's vault into the cluster, which is MegaCorp's way. The environment repository holds them encrypted. Or the service reads the vault itself. Settings that aren't secret go in the environment's values file. Either way, a running pod keeps the values it started with: a new value needs new pods.

Friday, 13:20. Rotated, and still the old one.

The database password was rotated at noon. The ExternalSecret updated the Secret within a minute. The service kept failing with the old password until someone restarted the pods at three. A running pod keeps the values it started with. Who should have restarted it? (the card)

How a secret reaches a pod#

WayGit holdsWho fetches or decryptsYou'll see
copied from the cloud's vaultan ExternalSecret: which keys, from which storethe External Secrets Operator, with its own cloud identityExternalSecret and ClusterSecretStore objects
encrypted with SOPSthe Secret, with its values encryptedFlux, with a key it holdsENC[AES256_GCM,… values and a sops: block
sealed with Sealed Secretsa SealedSecretthe Sealed Secrets controller in the clusterkind: SealedSecret, from bitnami.com/v1alpha1
read by the service itselfnothingthe service, with its own cloud identitythe vault's address in the service's settings

Whichever way it arrives, a Kubernetes Secret is not encrypted by default. It is stored base64-encoded, and anyone who may read Secrets in the namespace can read it. So who may read Secrets there matters as much as how they got there.

The External Secrets Operator#

Each MegaCorp cluster has one store, called cloud-secrets on both, so a service's values don't change between clouds:

megacorp/platform/gitops-configinfrastructure/config/eks/secret-store.yamlstore
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: cloud-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-2
megacorp/platform/gitops-configinfrastructure/config/aks/secret-store.yamlstore
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: cloud-secrets
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: https://megacorp-prod-kv.vault.azure.net
      serviceAccountRef:
        name: external-secrets
        namespace: external-secrets

A service's values file names the keys it needs, and the shared chart turns them into an ExternalSecret:

megacorp/platform/chartsservice/templates/externalsecret.yamlexternalsecret
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: {{ include "service.name" . }}
  labels:
    {{- include "service.labels" . | nindent 4 }}
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: {{ required "secrets.store names the ClusterSecretStore" .Values.secrets.store }}
  target:
    name: {{ include "service.name" . }}
  data:
    {{- range .Values.secrets.keys }}
    - secretKey: {{ .name }}
      remoteRef:
        key: {{ .remote }}
    {{- end }}

The Operator reads each key from the vault, and writes a Secret with the service's name in its namespace. It reads again every refreshInterval, an hour here. The Deployment reads that Secret as environment variables, through envFrom.

Secrets from a cloud vault to a pod. The External Secrets Operator reads the vault with its own cloud identity, as the store says, and writes a Kubernetes Secret, which the pod reads when it starts. Git holds only the ExternalSecret: names, never values. the cloud's vault Secrets Manager or Key Vault reads External Secrets Operator signs in with its own cloud identity, as the ClusterSecretStore says; refreshes hourly Git holds the ExternalSecret: names only writes a Kubernetes Secret in the service's namespace the pod reads it as variables when it starts
Figure Secrets from a cloud vault to a pod. The External Secrets Operator reads the vault with its own cloud identity, as the store says, and writes a Kubernetes Secret, which the pod reads when it starts. Git holds only the ExternalSecret: names, never values.#

On the local cluster, a Fake store stands in for the vault, under the same name, cloud-secrets. The ExternalSecret reports that it synced, and the Secret it made holds one key:

kubectloutputcaptured at the baseline
NAME           STORETYPE            STORE           REFRESH INTERVAL   STATUS         READY   LAST SYNC
payments-api   ClusterSecretStore   cloud-secrets   1h                 SecretSynced   True    14m
kubectloutputcaptured at the baseline
NAME           TYPE     DATA   AGE
payments-api   Opaque   1      14m

Secrets encrypted in Git#

Some teams keep secrets in the environment repository, encrypted, so that Git holds everything the cluster needs.

  • SOPS encrypts the values in a file and leaves the keys readable, with a key from age, OpenPGP or a cloud's key service. A .sops.yaml file says which files and fields to encrypt, and with which key. Flux decrypts them when a Kustomization has decryption: provider: sops.
  • Sealed Secrets encrypts a whole Secret with the public key of a controller running in the cluster, using kubeseal. Only that controller can decrypt it. By default, a SealedSecret works only under the name and namespace it was sealed for, so it can't be copied into another namespace and read there.

Config for each environment#

Settings that aren't secret belong in the environment's values file, where reviewers can read them:

megacorp/platform/gitops-configenvs/staging/payments-api.yamlsettings
env:
  SPRING_PROFILES_ACTIVE: staging
  PAYMENTS_PROVIDER_URL: https://sandbox.pay.example.com
secrets:
  store: cloud-secrets
  keys:
    - name: DB_PASSWORD
      remote: payments-api/staging/db-password
ingress:
  enabled: true
  host: payments.staging.example.com

MegaCorp's chart writes env into the Deployment's pod template. So a changed setting changes the template, and Kubernetes rolls out new pods with it. A ConfigMap works differently. A pod that reads one as environment variables keeps the old values until it restarts. Mounted as files, a ConfigMap is refreshed after a while. Secrets behave the same way: a rotated secret reaches a running pod's variables only when the pod restarts.

When you write it
Do this
Name secrets in an ExternalSecret and keep their values out of Git. Give the secret store the same name on every cluster. Keep settings in ConfigMaps and secrets in Secrets.
Why
The same ExternalSecret then works on every cluster, and nobody has to read a secret to review a change.
Watch out
A running pod keeps the secret it started with. After a secret is rotated, make sure the pods restart, or have the service read the secret from a mounted file that it reads again.
What you can now do
  1. Read an ExternalSecret and say where its values come from, and where they end up.
  2. Tell a ConfigMap from a Secret, and know which a running pod will notice changing.
  3. Rotate a secret so that the pods actually pick it up.
Spot the bugThe old password, an hour later#

At 09:00, payments-api's database password is rotated in Key Vault. At 09:40, and again at 10:15, payments-api's staging pods on aks-prod still use the old password. Nobody has restarted anything.

QuestionWhy, and what gets the new password to the pods?
Show the answer

Two waits add up. The ExternalSecret reads the vault once an hour, so the Secret in the namespace can change up to an hour after the rotation. Then the pods read that Secret as environment variables, which never change in a running pod: they keep the old value until they restart. Once the Secret holds the new password, new pods get it. They start on the next deploy that changes the pod template, or when someone restarts the Deployment. With Argo CD, use its built-in restart action on the Deployment, from the UI or with argocd app actions run (config for each environment).

See old values in running pods.

Releasing safely#

Part X · After the pipeline · Chapter 51 · 7 min read

A release is the moment new pods start taking real traffic. Kubernetes' default is a rolling update: it swaps pods a few at a time, and only counts a new pod once its readiness probe passes, so a broken version stalls instead of taking everything down. Blue-green starts a whole copy of the new version and switches traffic at once. Canary sends a small share of traffic to the new version and grows it while checks pass. Both need a tool such as Argo Rollouts or Flagger. Approvals come before a release, and going back is a revert in Git.

Monday, 10:40. Half old, half new.

The rollout stopped with three new pods and two old ones. After ten minutes the Deployment reports ProgressDeadlineExceeded. The new pods pass their readiness probe. They answer requests with 500s, because the probe checks only that the process is up. (the card)

Rolling, the default#

MegaCorp's shared chart doesn't name a strategy, so every Deployment rolls. Kubernetes starts new pods and stops old ones in steps. By default it runs at most a quarter more pods than it should, and keeps at least three quarters of them available. A new pod counts only when its readiness probe passes:

megacorp/platform/chartsservice/templates/deployment.yamlprobes
readinessProbe:
  httpGet:
    path: {{ .Values.probes.path }}
    port: http
livenessProbe:
  httpGet:
    path: {{ .Values.probes.path }}
    port: http
  initialDelaySeconds: 20
resources:
  {{- toYaml .Values.resources | nindent 12 }}

So a version whose pods never become ready doesn't replace the old one: the old pods keep serving, and the rollout waits. After ten minutes without progress, the Deployment reports ProgressDeadlineExceeded. It does not roll back by itself. With GitOps, you go back by reverting the deploy's commit (drift, and going back).

A readiness probe that checks too little, such as one that always answers, takes that safety away. A pod then counts as ready while it can't serve anyone.

Release strategies. Rolling replaces pods a few at a time. Blue-green starts a whole copy of the new version and switches traffic at once. Canary sends a share of traffic to the new version, and grows it while the checks pass. rolling a few pods at a time old and new both serve while it rolls blue-green a whole copy, then a switch old: all traffic new: tested, no traffic then the Service switches to the new copy at once twice the pods, briefly; going back is a switch canary a share of traffic, growing 10% 30%, if the checks pass 100%, or back to 0% when a check fails
Figure Release strategies. Rolling replaces pods a few at a time. Blue-green starts a whole copy of the new version and switches traffic at once. Canary sends a share of traffic to the new version, and grows it while the checks pass.#

Blue-green and canary#

Kubernetes has no blue-green or canary of its own. Two tools add them, one from each side of GitOps:

With Argo Rollouts, from the Argo project, you'll see a Rollout object in place of the Deployment, and the canary's share is set by setWeight in its steps. With Flagger, from the Flux project, you'll see a Canary object beside the Deployment, and the share climbs by stepWeight up to maxWeight in its analysis.

A canary in Argo Rollouts is a list of steps: send some traffic, wait, check, send more.

strategy:
  canary:
    steps:
      - setWeight: 20
      - pause: {duration: 5m}
      - analysis:
          templates:
            - templateName: success-rate
      - setWeight: 50
      - pause: {}
  • setWeight: 20 sends a fifth of the traffic to the new version. With a traffic router, such as a service mesh or an ingress controller, that share is exact. Without one, Argo Rollouts gets close by running a fifth of the pods on the new version.
  • analysis runs the checks an AnalysisTemplate describes, such as a success rate from Prometheus. If they fail, the rollout aborts, and all traffic goes back to the stable version.
  • pause: {} waits for a person, who continues with kubectl argo rollouts promote.

Blue-green in Argo Rollouts uses two Services: activeService for real traffic, and previewService, which points at the new version while it is tested. The switch happens on promotion. Flagger does the same with a Canary object, and needs a traffic router for its weights.

Approvals, and going back#

Three gates put a person in the way. Merging production's merge request in the environment repository, guarded by that repository's branch protection (promotion by merge request). Promoting a paused canary or blue-green rollout, with kubectl argo rollouts promote. And approving a GitLab deployment job, in GitLab environments, where approvals are Premium (environments and deployments).

Going back depends on how far the release got. A canary that fails its checks goes back by itself. Anything already released goes back by reverting its commit in the environment repository, so the controller deploys the old version again, and the history says so. kubectl rollout undo would be undone by the controller.

When you write it
Do this
Give every service a readiness probe that checks what the service needs in order to serve, not only that the process is running. Keep the default rolling update unless your team runs Argo Rollouts or Flagger.
Why
Kubernetes sends traffic to a pod as soon as the probe passes. A probe that only checks the process lets traffic reach a pod that cannot answer.
Watch out
When a release goes wrong, revert the commit in the environment repository. Do not fix the cluster by hand: the controller will put the broken version back.
What you can now do
  1. Read a rolling update's progress and say why it stopped.
  2. Write a readiness probe that checks what the service needs to serve.
  3. Roll back a bad release the GitOps way: revert the commit.
Spot the bugHalf a release#

payments-api's new version reaches staging with 4 replicas. Ten minutes later, 3 pods run the old version and 1 runs the new one, which restarts again and again. The service still answers.

QuestionWhat is Kubernetes doing, why does the service still work, and what do you do?
Show the answer

A rolling update is stuck. The new pod never passes its readiness probe, so Kubernetes won't count it or stop more old pods, and the old pods keep serving. After ten minutes the Deployment reports ProgressDeadlineExceeded, but it doesn't roll back by itself. Find out why the new pod restarts from its events and logs, then revert the deploy's commit in the environment repository, or fix forward with a new commit (rolling, the default).

See a rollout stuck half-way.

Observability#

Part X · After the pipeline · Chapter 52 · 13 min read

Once a deploy is out, its signals say whether it works. Logs record what happened, one event at a time. Metrics count things over time, such as requests, errors and restarts. Traces follow one request through every service it touches. OpenTelemetry is the common way to produce all three, and a collector sends them to the backends that keep them. In the open, those are Prometheus, Loki and Tempo. On AWS, they're CloudWatch and X-Ray; on Azure, Azure Monitor and Application Insights. You read them on dashboards, where each deploy is marked. A controller that says "synced and healthy" only means the new pods started. The signals say whether they work.

Monday, 15:30. CrashLoopBackOff.

The new pods restart every forty seconds. kubectl logs shows nothing, because the container dies before it logs. The dashboard shows the error rate climbing since the deploy marker at 15:02. The previous container's log has the answer, if you know to ask for it. (the card)

Three signals#

SignalWhat it isIt answersYou'll see
logslines a service writes as things happen, one event eachwhat happened to this request, and why it failedlines on standard output, often as JSON
metricsnumbers counted over time, each with labelshow often, how fast, how manya /metrics page for Prometheus to read, or numbers sent over OTLP
tracesone request's path through the services: a span for each step, all sharing a trace IDwhere the time went, and which service faileda waterfall of spans, one bar per step

A pod's logs are what its containers write to standard output and standard error. Kubernetes keeps them on the node for a while, and kubectl logs reads them from there. To keep them longer and search them, an agent on each node sends them to a backend. Metrics and traces need the service to produce them, usually through a library.

Signals, from a pod to people. The pod writes logs, counts metrics and records traces. A collector sends each signal to a backend that keeps it. People read them on dashboards, where each deploy is marked, and alerts watch them. your pod writes logs, counts metrics, records traces a collector OpenTelemetry Collector, or the cloud's agent where each signal is kept metrics Prometheus · CloudWatch · Azure Monitor logs Loki · CloudWatch Logs · Log Analytics traces Tempo · X-Ray · Application Insights dashboards and alerts Grafana, or the cloud's own; a deploy marked on the graphs
Figure Signals, from a pod to people. The pod writes logs, counts metrics and records traces. A collector sends each signal to a backend that keeps it. People read them on dashboards, where each deploy is marked, and alerts watch them.#

OpenTelemetry and the collector#

OpenTelemetry is a project of the Cloud Native Computing Foundation. It gives most languages one standard set of APIs, SDKs and tools for producing traces, metrics and logs. It isn't a backend, and stores nothing. Its protocol, OTLP, carries signals to whatever does, on port 4317 over gRPC or 4318 over HTTP.

The OpenTelemetry Collector sits between services and backends. It receives signals, processes them, for example by adding the pod's name, and exports them to one or more backends. So each service sends everything to one address, and the platform team decides where it goes. Each cloud's own agent does a similar job for that cloud's backends.

An OpenTelemetry SDK reads its settings from environment variables, so these are the ones you'll see in a service's values:

Three variables do most of the work. OTEL_SERVICE_NAME sets the service's name on every signal it sends. OTEL_RESOURCE_ATTRIBUTES adds more names and values, such as service.version=3f9c2e1a,deployment.environment.name=staging. OTEL_EXPORTER_OTLP_ENDPOINT says where to send the signals, usually the collector.

The open example: Prometheus, Grafana, Loki and Tempo#

  • Prometheus keeps metrics. It pulls them: every so often, it reads each service's metrics page over HTTP, which is called scraping. It stores each number with its time and labels, and its query language, PromQL, reads them back. Its rules raise alerts, and Alertmanager sends them to people. A cluster that runs the Prometheus Operator has ServiceMonitor objects, from monitoring.coreos.com/v1. Each one tells Prometheus which Services to scrape, and on which port. A service's chart may include one.
  • Loki keeps logs. It indexes only each stream's labels, such as the namespace and pod, not the text of the lines, which keeps it cheap. Its query language is LogQL.
  • Tempo keeps traces.
  • Grafana shows all three on dashboards, side by side, and can alert on them.

Where each signal lives on AWS and Azure#

SignalOpen exampleAWSAzure
metricsPrometheusCloudWatch, through Container Insights; or Amazon Managed Service for PrometheusAzure Monitor managed service for Prometheus
logsLokiCloudWatch Logsa Log Analytics workspace, through Container insights
tracesTempoX-Ray, sent with OpenTelemetryApplication Insights
dashboardsGrafanaCloudWatch dashboards, or Amazon Managed Grafanathe Azure portal, or Azure Managed Grafana

On EKS, the amazon-cloudwatch-observability add-on installs Container Insights, which collects the cluster's metrics and logs. On AKS, Container insights collects the containers' logs, and managed Prometheus collects metrics. Both clouds point you to OpenTelemetry for traces. AWS put X-Ray's own SDKs and daemon into maintenance mode in February 2026, and recommends OpenTelemetry instead; X-Ray still keeps and shows the traces. Application Insights accepts OpenTelemetry too.

MegaCorp's platform team runs one Grafana, at grafana.example.com, that reads the signals from both clouds, and teams build their dashboards there. GitLab also has tracing, metrics and logs of its own, as an experiment since GitLab 18.1. This book doesn't use them.

Marking each deploy#

A graph that jumps is easier to read when you can see what changed just before. So MegaCorp marks each deploy on Grafana's graphs, as what Grafana calls an annotation. The pipeline can't do this well. With GitOps, it only commits to the environment repository, and never learns when the new pods are ready. The controller does. So Argo CD sends the mark, through its notifications controller, which the platform team sets up in Argo CD's chart values:

megacorp/platform/gitops-configinfrastructure/add-ons/base/add-ons.yamlnotifications
notifications:
  secret:
    create: false
  notifiers:
    service.grafana: |
      apiUrl: https://grafana.example.com/api
      apiKey: $grafana-api-key
  triggers:
    trigger.on-deployed: |
      - description: Application is synced and healthy.
        send: [app-deployed]
        oncePer: app.status.operationState?.syncResult?.revision
        when: >-
          app.status.operationState != nil
          and app.status.operationState.phase in ['Succeeded']
          and app.status.health.status == 'Healthy'
          and (!time.Parse(app.status.health.lastTransitionTime).Add(1 * time.Minute).Before(time.Parse(app.status.operationState.finishedAt))
          or time.Parse(app.status.health.lastTransitionTime).Before(time.Parse(app.status.operationState.startedAt)))
  templates:
    template.app-deployed: |
      message: "{{ .app.metadata.name }} is synced and healthy"
  • service.grafana says where to send marks. $grafana-api-key names a key in the notifications Secret, which an ExternalSecret copies from the cloud's vault. That's why secret.create is false: the chart mustn't make the Secret too.
  • trigger.on-deployed is copied from Argo CD's catalog of triggers. It fires once a sync has succeeded and the Application is healthy.
  • template.app-deployed is the mark's text.

Nothing is sent until an Application subscribes. MegaCorp's ApplicationSet subscribes each payments-api Application, with two tags for dashboards to pick marks by:

megacorp/platform/gitops-configapps/eks-prod/applicationset.yamldeploy-marker
annotations:
  notifications.argoproj.io/subscribe.on-deployed.grafana: "payments-api|{{ .env }}"

The key comes from the same store as every other secret:

megacorp/platform/gitops-configinfrastructure/config/eks/secret-store.yamlnotifications-secret
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: argocd-notifications-secret
  namespace: argocd
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: cloud-secrets
  target:
    name: argocd-notifications-secret
  data:
    - secretKey: grafana-api-key
      remoteRef:
        key: platform-grafana-api-key

A dashboard shows the marks by asking for annotations with those tags. Flux can send the same marks, through a Provider of type grafana and an Alert that picks which events to send.

Did my deploy work?#

Go through these in order. Each step needs the one before it, so the first one that fails is where to look.

  1. The change is in the environment repository. For staging, the deploy job's commit is on main. For production, its merge request is merged (promotion by merge request). No commit means the pipeline didn't deploy: read the deploy job's log.
  2. The controller applied that commit. Argo CD shows the Application as Synced to that commit, and Healthy. With Flux, the Kustomization is Ready, and its last applied revision is that commit. If not, the controller's message says why.
  3. The rollout finished. kubectl rollout status deployment/payments-api -n payments-staging exits with 0 once every new pod is ready. If the rollout stalls, it ends with error: deployment "payments-api" exceeded its progress deadline, and exits with 1. Then the pods' events and logs say why (rolling, the default).
  4. The pods run the new version. The Deployment's image tag is the one the deploy wrote.
  5. The signals agree. After the deploy's mark, compare the error rate and the response times with before it. Look for restarts, and for new errors in the logs. With service.version set, you can split metrics and traces by version.

Step 2 looks different with each controller. This is greeting-service from the cluster labs, once each controller had applied the pushed commit:

fluxoutputcaptured at the baseline
NAME    	REVISION          	SUSPENDED	READY	MESSAGE                              
greeting	main@sha1:fd092043	False    	True 	Applied revision: main@sha1:fd092043

READY is True, and the revision is the commit that was pushed. While Flux is still applying it, or waiting for what it applied to become healthy, READY is Unknown, with Reconciliation in progress.

argocdoutputcaptured at the baseline
Name:               argocd/greeting
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          greeting-staging
URL:                https://argocd.example.com/applications/greeting
Source:
- Repo:             https://gitlab.example.com/practice/greeting-gitops.git
  Target:           
  Path:             kustomize/overlays/staging
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to  (fd09204)
Health Status:      Healthy

GROUP  KIND        NAMESPACE         NAME      STATUS  HEALTH   HOOK  MESSAGE
       Service     greeting-staging  greeting  Synced  Healthy        service/greeting created
apps   Deployment  greeting-staging  greeting  Synced  Healthy        deployment.apps/greeting created

Sync Status is Synced, to the commit in brackets, and Health Status is Healthy. Each object has its own row, with its own sync and health.

If only the last step fails, the deploy didn't work, even though the controller calls it healthy. Healthy means the pods pass their probes, not that users' requests succeed. Revert the deploy's commit in the environment repository (drift, and going back), or fix forward with a new one.

What you can now do
  1. Read a crashed container's previous log, and a pod's events, in the right order.
  2. Find your deploy on the dashboards, and read logs, metrics and traces for one request.
  3. Run the after-deploy checklist to the signals, not just to the controller's green.
LabBreak a deploy three waysabout 20 min#
You need

The application from the Argo CD lab, synced and healthy. No cluster? Make each change, run kustomize build kustomize/overlays/staging to see what the cluster would get, and read the signs below, which are what the cluster showed.

Steps
  1. An image that can't be pulled. Set newTag to does-not-exist, then commit and push. Once Argo CD has synced, run kubectl get pods -n greeting-staging, then kubectl describe pod on the new pod. Revert the commit, and push.
  2. A value that makes the pod crash. Add this to the overlay's patches, then commit and push:
    - target:
        kind: Deployment
        name: greeting
      patch: |-
        - op: add
          path: /spec/template/spec/containers/0/env
          value:
            - name: PORT
              value: eight-thousand
    Run kubectl get pods -n greeting-staging, then kubectl logs on the new pod with --previous. Revert the commit, and push.
  3. A change made by hand. Run kubectl scale deployment greeting -n greeting-staging --replicas=5. Run kubectl get deployment greeting -n greeting-staging at once, and again 30 seconds later.
What you should see

An image that can't be pulled. The new pod never starts, and the old ones keep serving: a rolling update removes old pods only once new ones are ready.

kubectloutputcaptured at the baseline
NAME                        READY   STATUS             RESTARTS   AGE
greeting-675f65c675-bzkfg   0/1     ImagePullBackOff   0          15s
greeting-744b5b7bc4-cw8rs   1/1     Running            0          3m35s
greeting-744b5b7bc4-scqmm   1/1     Running            0          3m35s

The pod's events say why. The registry answered not found: it has no such tag.

kubectloutputcaptured at the baseline
Name:             greeting-675f65c675-bzkfg
⋯ lines omitted
Status:           Pending
⋯ lines omitted
    Image:          gitlab.example.com:5050/practice/node-service:does-not-exist
⋯ lines omitted
    State:          Waiting
      Reason:       ImagePullBackOff
⋯ lines omitted
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  15s               default-scheduler  Successfully assigned greeting-staging/greeting-675f65c675-bzkfg to delivery-lab-control-plane
  Normal   BackOff    14s               kubelet            spec.containers{app}: Back-off pulling image "gitlab.example.com:5050/practice/node-service:does-not-exist"
  Warning  Failed     14s               kubelet            spec.containers{app}: Error: ImagePullBackOff
  Normal   Pulling    3s (x2 over 14s)  kubelet            spec.containers{app}: Pulling image "gitlab.example.com:5050/practice/node-service:does-not-exist"
  Warning  Failed     3s (x2 over 14s)  kubelet            spec.containers{app}: Failed to pull image "gitlab.example.com:5050/practice/node-service:does-not-exist": rpc error: code = NotFound desc = failed to pull and unpack image "gitlab.example.com:5050/practice/node-service:does-not-exist": failed to resolve reference "gitlab.example.com:5050/practice/node-service:does-not-exist": gitlab.example.com:5050/practice/node-service:does-not-exist: not found
  Warning  Failed     3s (x2 over 14s)  kubelet            spec.containers{app}: Error: ErrImagePull

Argo CD did its part: the application is Synced, and it stays Progressing while the Deployment waits for a pod that will never be ready. After the Deployment's ten-minute deadline, it turns Degraded (pods that can't pull their image).

argocdoutputcaptured at the baseline
Name:               argocd/greeting
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          greeting-staging
URL:                https://argocd.example.com/applications/greeting
Source:
- Repo:             https://gitlab.example.com/practice/greeting-gitops.git
  Target:           
  Path:             kustomize/overlays/staging
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to  (37af6e9)
Health Status:      Progressing

GROUP  KIND        NAMESPACE         NAME      STATUS  HEALTH       HOOK  MESSAGE
       Service     greeting-staging  greeting  Synced  Healthy            service/greeting unchanged
apps   Deployment  greeting-staging  greeting  Synced  Progressing        deployment.apps/greeting configured

A value that makes the pod crash. This time the image pulls, and the container starts, but it exits at once. Kubernetes starts it again, and waits longer before each try: CrashLoopBackOff. The old pods keep serving here too.

kubectloutputcaptured at the baseline
NAME                        READY   STATUS             RESTARTS     AGE
greeting-744b5b7bc4-cw8rs   1/1     Running            0            8m56s
greeting-744b5b7bc4-scqmm   1/1     Running            0            8m56s
greeting-8544d877ff-b72bq   0/1     CrashLoopBackOff   1 (3s ago)   5s

The last run's output names the value. The service read PORT as a number, got NaN, and couldn't listen on it (pods that restart again and again).

kubectloutputcaptured at the baseline
node:net:2551
    validatePort(options.port, 'options.port');
    ^

RangeError [ERR_SOCKET_BAD_PORT]: options.port should be >= 0 and < 65536. Received type number (NaN).
    at Server.listen (node:net:2551:5)
    at file:///app/dist/server.js:19:4
    at ModuleJob.run (node:internal/modules/esm/module_job:561:25)
    at async node:internal/modules/esm/loader:647:26
    at async asyncRunEntryPointWithESMLoader (node:internal/modules/run_main:101:5) {
  code: 'ERR_SOCKET_BAD_PORT'
}

Node.js v24.21.0

A change made by hand. Right after the scale, the Deployment wants five pods, and only two are ready:

kubectloutputcaptured at the baseline
deployment.apps/greeting scaled
kubectloutputcaptured at the baseline
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
greeting   2/5     5            2           11m

Half a minute later, it's back to two. Argo CD saw that the cluster no longer matched Git and, with self-healing on, put the Deployment back as Git says. To run more pods, change the overlay and push (how it syncs).

kubectloutputcaptured at the baseline
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
greeting   2/2     2            2           12m
When you write it
Do this
Ask for your Applications to be subscribed to on-deployed, so that each deploy is marked on your dashboards. Set OTEL_SERVICE_NAME, and set service.version to the image tag. Write logs to standard output, one event per line.
Why
Every signal then says which service and which version sent it, and each deploy is visible on the same graph as its effects.
Watch out
After each deploy, go through the checklist all the way to the signals. A deploy that is healthy in the controller can still be failing in the dashboards.
Spot the bugHealthy, and failing#

At 14:02, payments-api 1c7d4b09 reaches production. Argo CD shows payments-api-production as Synced and Healthy, and all its pods are ready. On the service's Grafana dashboard, a mark tagged payments-api and production appears at 14:03. Just after it, the share of failed requests rises from 0.1% to 4%, and stays there.

QuestionDid the deploy work? What do you check, and what do you do?
Show the answer

No. Healthy means the new pods pass their readiness probes, not that they serve every request. The mark shows the failures started with this deploy. Check the logs and traces of failed requests, split by service.version, to confirm they come from 1c7d4b09. Then revert the deploy's commit in the environment repository, so Argo CD deploys the old version again, or fix forward with a new commit (did my deploy work?). A canary with an analysis step would have stopped this at a small share of the traffic (blue-green and canary).

Around your service#

Part X · After the pipeline · Chapter 53 · 10 min read

Some tools shape how your service runs without being in its code or its pipeline. A service mesh puts a proxy in the path of every request between services. It encrypts that traffic, retries failed requests, and can split traffic between two versions. Feature flags turn code on for some users without a deploy: GitLab has its own, and OpenFeature is a standard way to read any tool's flags. A developer portal such as Backstage lists every service, and its templates create new repositories with a pipeline already in them. Each of these leaves fingerprints you can learn to spot.

Tuesday, 09:45. The template that made your repository.

Your repository was created from the platform team's portal template last spring. Since then the template gained a security scan on every merge request. Yours never got it, because a portal template is a copy, not a link, and nobody noticed until an audit asked when the scans had last run on your code. Meanwhile a feature flag turned on in production first. Three things around your service, each with its own rules. (the card)

A service mesh, as you see it#

A service mesh adds a proxy between your service and every service it calls. Your code still calls the other service by its usual address. The proxy adds what the platform team configured: mutual TLS, retries, timeouts, traffic splits and metrics. Istio and Linkerd are the common meshes.

You seeWhat it means
a second container in your pod, istio-proxy or linkerd-proxy, so the pod shows 2/2 readythe mesh runs as a sidecar: a proxy container added to each pod
the namespace label istio-injection=enabled, or the annotation linkerd.io/inject: enabledpods created in that namespace get the sidecar
the label istio.io/dataplane-mode=ambient, and no extra containerIstio's ambient mode: a proxy on each node handles mutual TLS, and a separate waypoint proxy handles HTTP features
a VirtualService, from networking.istio.io/v1Istio's routes, retries and timeouts for a service
an HTTPRoute, from gateway.networking.k8s.io/v1the Kubernetes Gateway API's routes, a standard that many meshes and ingress controllers read

A sidecar is added when a pod is created, so pods that were already running need a restart to get one. To read your service's own logs in a pod with a sidecar, name its container: kubectl logs with -c.

Mutual TLS means both ends of a connection prove who they are with certificates, and the traffic between them is encrypted. The mesh does it for you. Linkerd, for example, turns it on for all TCP traffic between meshed pods. Each certificate is tied to the pod's service account, and is replaced every day. Your service still speaks plain HTTP to its own proxy.

Retries and timeouts are set in the mesh's routes. By default, Istio retries a failed HTTP request twice, and sets no timeout. A retry is only safe for a request that can be repeated. For a service that takes payments, find out how the mesh retries, and make a repeated request harmless, for example with an idempotency key.

Where a canary's share is set#

Splitting traffic between two versions is a job for the mesh or the ingress controller. The share is a pair of weights on a route. In the Gateway API, a route that sends a tenth of payments-api's traffic to a canary looks like this:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payments-api
spec:
  parentRefs:
    - name: shop
  rules:
    - backendRefs:
        - name: payments-api-stable
          port: 80
          weight: 90
        - name: payments-api-canary
          port: 80
          weight: 10

Weights are shares, not percentages: each backend gets its weight divided by the total. Who changes them depends on the setup:

SetupWhere the share is setWho changes it
by handweight in the route, in the environment repositorya person, through a merge request
Argo RolloutssetWeight in the Rollout's stepsArgo Rollouts, which edits the route as each step runs (blue-green and canary)
FlaggerstepWeight and maxWeight in the Canary's analysisFlagger, which edits the route while the checks pass
a feature flag's percent rolloutthe flag's strategy, in GitLabwhoever owns the flag, with no deploy

The last row is different. It splits users, not requests between two versions: both code paths are in the same deploy, and the flag picks one.

Feature flags#

A feature flag lets a deploy carry code that stays off until someone turns it on, for everyone or for some users. So releasing a feature no longer has to happen at the moment you deploy it.

GitLab has feature flags in every tier, under Deploy › Feature flags. Each flag has one or more strategies, and each strategy applies to some environments:

StrategyTurns the flag on for
All userseveryone
Percent rollouta share of page views; you choose whether a user or session keeps the same answer
Percent of usersa share of signed-in users; the application must pass each user's ID
User IDsthe users you list in the strategy
User lista named list of users, kept in GitLab and shared between flags

Your application reads its flags with an Unleash client library, because GitLab's flag API is compatible with Unleash, a feature-flag tool. The client needs the connection details that the Feature flags page shows. It also needs an application name, which must be the environment's name, such as production, not the service's. A self-managed GitLab allows 200 flags per project; GitLab.com allows 50, 150 or 200, by tier.

OpenFeature, an incubating project of the Cloud Native Computing Foundation, is a standard API for reading flags from any flag tool. Your code calls OpenFeature, and a provider translates each call for the tool in use. So changing tools doesn't change your code.

Old flags are debt. Once a feature is on for everyone, remove its flag and the code path it turned off.

Developer portals, and where a pipeline came from#

A developer portal is a website that lists every service and who owns it, with links to its repository, pipelines and dashboards. Backstage is the common one. Spotify created it, and it is now an incubating project of the Cloud Native Computing Foundation. MegaCorp's portal knows payments-api from this file at the root of its repository:

megacorp/payments/payments-apicatalog-info.yamlcatalog
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payments-api
  description: Takes card payments for MegaCorp's shop
  annotations:
    backstage.io/source-template: template:default/java-service
    backstage.io/source-location: url:https://gitlab.example.com/megacorp/payments/payments-api/
spec:
  type: service
  lifecycle: production
  owner: group:default/payments

The portal's software templates create new repositories. A template copies a skeleton of files and fills in the names you type. Its publish:gitlab step pushes the new repository to GitLab, and its catalog:register step adds the new catalog-info.yaml to the portal. The skeleton usually includes a .gitlab-ci.yml, so a new service has a pipeline from its first commit.

A template copies files once. It isn't an include: when the platform team improves a template, repositories made from it earlier don't change. That's why MegaCorp's templates write a .gitlab-ci.yml that includes the golden pipeline, instead of copying its jobs (include: pulling in files). A pipeline that came from a template has these fingerprints:

  • a catalog-info.yaml at the root, with a backstage.io/source-template annotation that names the template
  • a first commit called initial commit, by the author Scaffolder, unless the template changed those defaults
  • a pipeline file that matches other services' files, comments and all
When you write it
Do this
If a portal template made your repository, change its pipeline in your own repository. If the fix belongs in the template too, tell the platform team. Turn a new feature flag on in staging before production, and remove it once its feature is fully on.
Why
The template does not push changes into repositories it has already made, so your copy is yours to fix. A flag that stays behind is one more thing nobody understands later.
Watch out
Before you rely on the service mesh's retries, check that your requests can be repeated safely. A retried payment is a double payment.
What you can now do
  1. Say what a portal template gives you once, and what a shared template keeps giving you.
  2. Turn a feature flag on in the right order, and remove it when its feature is fully on.
  3. Know when the mesh's retries are safe for your service, and when they double a payment.
Spot the bugThe fix that never arrived#

billing-api's .gitlab-ci.yml has a sonar job written out in full. Last month, the platform team fixed a bug in that job in the portal's "Java service" template. billing-api's pipeline still runs the old job. Its repository has a catalog-info.yaml with backstage.io/source-template: template:default/java-service.

QuestionWhy didn't the fix reach billing-api, and where do you fix it?
Show the answer

The annotation shows that the portal's template created billing-api. A template copies its files once, so billing-api's .gitlab-ci.yml has been its own copy since then, and later changes to the template don't reach it. Fix the job in billing-api's own file. Better still, have the job come from the golden pipeline or a component that the file includes, so later fixes arrive with a version bump (developer portals, and where a pipeline came from).

Platform words, decoded#

Part X · After the pipeline · Chapter 54 · 19 min read

The words a platform team uses, one row each: what the thing is, why you care as a developer, and where you'll meet it. Where a chapter covers a word, the last column links there. Search finds every row.

Traffic, names and certificates#

TermWhat it isWhy you careWhere you'll see it
ingress controllera program in the cluster that reads Ingress objects and sends outside traffic to the right Serviceyour service's public address and TLS go through itan Ingress in your service's chart (Helm); Ingress NGINX, a popular one, is retired, with no fixes after March 2026
Gateway APIKubernetes' newer routing API, with Gateway and HTTPRoute objects from gateway.networking.k8s.ioroutes, traffic splits and canaries are written with itHTTPRoute objects (where a canary's share is set)
cert-managera controller that gets TLS certificates, from Let's Encrypt or a private authority, and renews them before they expireyour HTTPS certificate is renewed without anyone askingCertificate and ClusterIssuer objects; the annotation cert-manager.io/cluster-issuer on an Ingress
ExternalDNSa controller that creates DNS records for Services and Ingresses, in Route 53, Azure DNS and other providersyour service's hostname appears in DNS when it's deployedthe annotation external-dns.kubernetes.io/hostname, or an Ingress's hosts
service meshproxies in the path of every request between services, adding mutual TLS, retries and traffic splitsyour calls get retries and encryption you didn't writeIstio and Linkerd (a service mesh, as you see it)
sidecaran extra container that runs beside your service's container, in the same podthe pod shows 2/2, and kubectl logs needs -cistio-proxy or linkerd-proxy in your pod (a service mesh)
mutual TLS, mTLSboth ends of a connection prove who they are with certificates, and the traffic is encrypteda service can check which service is calling itthe mesh's settings (a service mesh)
NetworkPolicyrules for which pods may talk to which; the cluster's network plugin must support thema call that times out may be blocked, not brokenNetworkPolicy objects, from networking.k8s.io/v1, in your namespace

Scaling and capacity#

TermWhat it isWhy you careWhere you'll see it
HorizontalPodAutoscaler, HPAchanges a workload's number of pods to match its use of CPU, memory or another measurethe number of pods changes without a deployHorizontalPodAutoscaler objects, from autoscaling/v2; kubectl get hpa
VerticalPodAutoscaler, VPAan add-on, not built into Kubernetes, that changes pods' CPU and memory requestsyour pods' requests may differ from the values fileVerticalPodAutoscaler objects, where the add-on is installed
KEDAscales workloads on events, such as a queue's length, down to zero and back, working with the HPAa service with no pods starts when work arrivesScaledObject objects
requests and limitsthe CPU and memory a container asks for, and the most it may usetoo little memory and the container is killed; ask for too much and it waits for a node with roomresources in a chart's values (reading a Deployment)
Cluster Autoscaleradds nodes when pods can't be scheduled for lack of room, and removes nodes that stay underuseda pod may wait in Pending for a new nodethe cloud's node groups
Karpenterlaunches nodes that fit the pods the scheduler can't place, and removes nodes that aren't neededa pod may wait in Pending for a node chosen to fit itNodePool objects, from karpenter.sh/v1; on AKS, node auto-provisioning runs Karpenter
node pool, node groupa set of the cluster's machines of one kindyour pods may be placed on one by labels and taintsAKS node pools; EKS managed node groups
PodDisruptionBudget, PDBlimits how many of an app's pods may be down at once during planned disruptions, such as a node upgradewithout one, an upgrade may take down more of your pods at once than you can sparePodDisruptionBudget objects, from policy/v1
ResourceQuotalimits a namespace's total CPU, memory and number of objectsa deploy can fail because the namespace is fullResourceQuota objects; errors that say exceeded quota

Kubernetes, extended#

TermWhat it isWhy you careWhere you'll see it
namespacea named group of objects in a cluster, with its own permissions and quotasyour service's objects live in one for each environmentpayments-staging and payments-production at MegaCorp (the objects you'll read)
custom resource, CRDa new kind of object added to Kubernetes, such as HelmRelease or ExternalSecret; a CRD defines itmost tools in this chapter are custom resources plus a controllerapiVersions you don't recognise; kubectl get crd
controllera loop that watches objects, and changes the cluster to match what they ask fora change you make by hand can be undone by oneFlux, Argo CD and the External Secrets Operator (push and pull)
operatora controller, with its own custom resources, that runs an application such as a database for youa database your service uses may be run by onecustom resources named after the application
admission webhookcode that checks each object sent to the Kubernetes API, and may change it, before it's storeda deploy can be refused, or changed on its way inerrors that say admission webhook "…" denied the request
reconcilewhat a controller does: compare what's asked for with what exists, and change what exists to matchthe controllers' messages use the worddrift, and going back
driftthe cluster differing from Git, usually after a change by handthe controller puts it back, or reports itdrift, and going back

Policy and the supply chain#

TermWhat it isWhy you careWhere you'll see it
policy enginechecks objects against the organisation's rules as they reach the clustera deploy can be refused for breaking a rule, not for a bugKyverno and OPA Gatekeeper; GitLab's own policies are something else (policies and gates)
Kyvernoa policy engine whose policies are Kubernetes objects: they check, change, create or clean up objects, and can verify image signaturesa refused deploy's message names the policyClusterPolicy and Policy objects, from kyverno.io/v1; a graduated CNCF project since March 2026
OPA Gatekeeperan admission webhook that enforces policies written in Rego, Open Policy Agent's language; its audit lists what already breaks a rulea refused deploy's message names the constraintConstraintTemplate objects, and the constraints made from them
image signinga signature that proves who signed an image, and that it hasn't changed sincethe cluster may refuse an image that isn't signedcosign
cosignSigstore's tool to sign and verify images and other artifacts; keyless signing ties a short-lived certificate to an OIDC identity, such as a CI job's, and logs it in Rekora pipeline signs, and the cluster's policy checks the signaturecosign sign and cosign verify in jobs (ID tokens); signatures in the registry, beside the image
SBOMa software bill of materials: every component in an image or build, in a format such as CycloneDX or SPDXscanners and auditors read it to find vulnerable partsGitLab's dependency scanning writes CycloneDX files (GitLab's scanners)

Delivery#

TermWhat it isWhy you careWhere you'll see it
GitOpsGit holds what should run, and a controller in the cluster pulls it ina deploy is a commit, and going back is a revertpush and pull
environment repositorythe repository that says what runs in each environment: image tags and settingsyour deploy job commits theremegacorp/platform/gitops-config (the repositories)
promotionmoving a version from one environment to the nextproduction's deploy is a merge requestpromotion by merge request
Helm charta package of templates and default values that renders Kubernetes objectsyour service's objects come from oneChart.yaml, values.yaml and templates/ (Helm)
Kustomize overlaychanges laid over a base set of objects, one overlay for each environment or clustera setting may come from the overlay, not the basekustomization.yaml files (Kustomize)
Fluxa GitOps tool made of several controllers, configured with objects such as GitRepository, Kustomization and HelmReleaseon MegaCorp's clusters, it installs the platform's add-onsclusters/eks-prod/platform.yaml (Flux)
Argo CDa GitOps controller with a web UI; an Application says what to deploy, from where, and to which namespaceyour service's sync and health show on its UIApplication and ApplicationSet objects (Argo CD)
ApplicationSetmakes many Argo CD Applications from one template, such as one for each environmentchanging it changes every environment's Applicationapps/eks-prod/applicationset.yaml at MegaCorp
image automationa controller that finds a new image tag in the registry, and writes it into Gita commit in the environment repository may come from a botFlux's image automation; Argo CD Image Updater (automation)
progressive deliveryreleasing to a growing share of traffic, while checks passa bad version reaches few usersblue-green and canary
Argo Rolloutsa controller for canary and blue-green releases, with steps and analysis; a Rollout object takes the Deployment's placeyour service's pods are managed by a Rollout, not a DeploymentRollout and AnalysisTemplate objects (blue-green and canary)
Flaggera controller for canary and blue-green releases, from the Flux project; a Canary object sits beside the Deploymentchecks decide whether the new version goes onCanary objects (blue-green and canary)
feature flaga switch that turns code on for some or all users, without a deployreleasing can happen after deployingGitLab's **Deploy › Feature flags**; OpenFeature (feature flags)

Identity, secrets and infrastructure#

TermWhat it isWhy you careWhere you'll see it
workload identitya pod's own cloud identity, from its Kubernetes service account, with no stored keyyour service reaches the cloud without a passwordpods and their cloud identity
IRSA, EKS Pod Identitythe two ways a pod on EKS gets an IAM rolethey decide your pod's AWS permissionsthe annotation eks.amazonaws.com/role-arn, or a Pod Identity association (pods and their cloud identity)
External Secrets Operatorcopies secrets from the cloud's vault into Kubernetes Secretsyour secrets' values never go into GitExternalSecret and ClusterSecretStore objects (the External Secrets Operator)
SOPS, Sealed Secretstwo ways to keep secrets in Git, encryptedyou'll see secret values in Git that you can't readENCAES256_GCM,… values, or kind: SealedSecret ([secrets encrypted in Git)
OpenTofu, Terraformtools that create cloud resources from code, with a plan to review before each applyyour cluster, registry and roles came from them.tf files, and plans in merge requests (OpenTofu and Terraform)
state, state lockthe tool's record of what it created, and the lock that stops two runs at oncea stale lock blocks every applystate and its lock
Crossplanea control plane framework: cloud resources become Kubernetes objects, and a platform team's compositions turn one request into many resourcesyou may ask for a database with a Kubernetes object, not a Terraform modulecomposite resources that the platform team defines; a graduated CNCF project since October 2025
landing zonean organisation's standard cloud setup: accounts or subscriptions made to one pattern, with rules applied to all of themyour team's AWS account or Azure subscription came from itAWS Control Tower and its Account Factory; Azure's platform and application landing zones

Signals and reliability#

TermWhat it isWhy you careWhere you'll see it
OpenTelemetry Collectorreceives signals from services, processes them, and sends them on to backendsyour service sends everything to one addressOTEL_EXPORTER_OTLP_ENDPOINT in your settings (OpenTelemetry and the collector)
Prometheuskeeps metrics, pulling them from each service's metrics page; PromQL queries themyour service's graphs and alerts come from itServiceMonitor objects (the open example)
Grafanadashboards over metrics, logs and traceswhere you look after a deploymarking each deploy
SLIa service level indicator: a measure of the service, such as the share of requests that succeedit's what your service is judged bydashboards and alerts
SLOa service level objective: a target for an SLI, such as 99.9% of requests succeeding over 30 daysa service can miss it while every pipeline is greenthe team's agreed targets
SLAa service level agreement: a contract with users, with consequences if its SLOs are missedit's why some SLOs are strictcontracts, not code
error budgethow much an SLO allows to fail: at 99.9%, one request in a thousanda bad deploy uses it upSLO dashboards
runbookthe written steps for handling a known problemthe alert that wakes someone should link to onealerts, and the on-call team's pages

Platforms and portals#

TermWhat it isWhy you careWhere you'll see it
platform, internal developer platform"an integrated collection of capabilities defined and presented according to the needs of the platform's users", in the CNCF's words: the tools, templates and services an organisation's teams build and run onthe platform team builds it for teams like yoursa portal, templates, and self-service APIs
golden patha supported way to build a kind of service: a project template and its docs, with a pipeline that builds, scans, tests, deploys and observes itthe easy way is also the supported onesoftware templates (developer portals)
developer portala website that lists every service and its owner, with links to its repository, pipelines and dashboardsit tells you who owns a serviceBackstage (developer portals)
Backstagean open-source developer portal, created by Spotify, now an incubating CNCF projectits templates may have made your repositorycatalog-info.yaml (developer portals)
software templatea portal's recipe for a new repository: a skeleton of files, filled in and publishedyour pipeline file may be a copy of onethe annotation backstage.io/source-template (developer portals)

The decoding method#

Part XI · Decode and debug · Chapter 55 · 8 min read

Everything about a pipeline job comes down to five questions: where it is defined, when it runs, where it runs, what it executes, and what goes in and out. Chapter 4 named them. This chapter turns them into a procedure you can follow on any job in any organisation, and shows it on one of MegaCorp's jobs from start to finish. A deploy gets the same five questions, asked of the cluster instead of the pipeline.

When something is already wrong, start from the symptom instead. The decision tree at the end sends you to the right chapter of symptom cards.

Tuesday, 14:10. A pipeline you have never seen.

The team next door is on leave, and their nightly schedule ran nothing at 03:00: no pipeline, no error. You have never opened the project. Forty jobs, six included files, a policy stage, and a workflow someone wrote last year. You have twenty minutes before the stand-up. You need an order to ask questions in. (the card)

The five questions, in order#

QuestionLook first atThenChapters
Where is it defined?the job's name in the pipeline graph, then Full configurationits extends chain, !reference tags and includes; if it's in no file, policies and settings12–20
When does it run?workflow:rules, then the job's own final rulesneeds and stages; compare pipeline types in a truth table21–25
Where does it run?the job's page: the runner's name and tagsthe runner's executor, and the job's image29–30
What does it execute?before_script, script and after_script in Full configurationthe scripts, functions and tools those lines call inside the image19 and 29
What goes in and out?the variables and secrets it needs, from YAML and from settingsits artifacts, caches, reports, and what it pushes or deploys26–32

Answer them in this order, because each answer narrows the next. You can't read a job's rules until you know where they come from. You can't judge its inputs until you know which runner and image it had.

Each question has one trap that catches most people:

  • Defined: a policy's job is in no file at all (chapter 20).
  • When: a job's own rules replace a template's list completely (chapter 14).
  • Where: the runner's configuration changes jobs, and no project can see it (chapter 29).
  • What: the script calls a tool or function from the image, whose code is somewhere else (chapter 19).
  • In and out: a settings variable beats anything in the YAML (chapter 26).

One job, decoded#

Here is payments-api's image-build, decoded using nothing but the specimen:

QuestionThe answer for image-buildFound in
Where is it defined?image-build in java-service.yml extends .image-build in container.yml, which extends .megacorp-base in base.ymlthe Full configuration view, and chapter 20's steps
When does it run?.megacorp-base's rules: merge requests, main and release tags, never schedules; it needs maven-buildthe .rules library, and chapter 25's truth table
Where does it run?on the megacorp-shared runner, in a Kubernetes pod, from the buildah imagedefault: tags, the runner's configuration, and the job's image
What does it execute?the ci-tools functions and the AWS login from the snippets, then Buildah's login, build and push, and on main a second push, to Azurethe !reference lines in before_script, and ci-lib.sh
What goes in and out?in: an ID token, AWS_ROLE_ARN, ECR_REGISTRY, AWS_REGION and the jar from maven-build; out: an image in ECR, on main in ACR too, and IMAGE_REF in a dotenv reportits variables and needs, and its artifacts

After five short answers, the job holds no more surprises.

A deployment, decoded#

A deploy has the same five questions, with different places to look (five questions for a deployment). Here they are for payments-api 1c7d4b09, deployed to production on eks-prod:

QuestionThe answer for payments-api in productionFound in
Where is it defined?the shared chart service at service-3.4.0, with the values in envs/production/payments-api.yaml, through Argo CD's ApplicationSetthe ApplicationSet's sources, then the values file (chapter 45)
When does it deploy?when the deploy's merge request is merged in the environment repository, and Argo CD next syncsthe merge request, then the Application's sync status
Where does it run?eks-prod, in the payments-production namespacethe ApplicationSet's destination
What does it run?123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:1c7d4b09, in the Deployment that the chart rendershelm template with the same values (chapter 43)
What goes in and out?in: settings from the values file, and secrets that the External Secrets Operator copies from Secrets Manager; out: logs, metrics and traces, and a mark on the dashboardsthe ExternalSecret, and the service's dashboards (chapter 52)

When a deploy is missing or wrong, answer them in the same order. The first answer that surprises you is where to look (chapter 52).

Starting from a symptom#

The debugging decision tree. Start from what you can see, pick one of four groups, make its first three checks, then open that group's chapter of symptom cards. Something is wrong with a job start from what you can see It didn't run or ran when it shouldn't It failed red, with an error It passed, but did the wrong thing Stuck or slow, or broke with no commit 1 Does the pipeline exist? 2 Did its rules match? 3 Did everything it needs exist? 1 Read the first error 2 Check the image and runner 3 Check what went in 1 Which value won? 2 Which image or artifact? 3 What did it really deploy? 1 Is a runner online? 2 Did a token expire? 3 Did a central file change? cards: chapter 39 cards: chapter 40 cards: chapter 41 cards: chapter 42 Each group has its own chapter of symptom cards, with the most common causes first.
Figure The debugging decision tree. Start from what you can see, pick one of four groups, make its first three checks, then open that group's chapter of symptom cards.#

Each symptom card names a symptom, then its cause, how to confirm it, and how to fix it. The cards come in four chapters, one per group, with the most common causes first: didn't run, failed, wrong result, and stuck or drifting. For your own pipelines, the worksheet is a printable form of the five questions.

What you can now do
  1. Decode any pipeline in a fixed order: the five questions, each narrowing the next.
  2. Go from a symptom to a card in one step, using the four kinds of symptom.
  3. Write down what you found so the next person doesn't start from zero.
Spot the bugDecode publish-docs#

payments-api defines one job of its own. Answer the five questions for it:

megacorp/payments/payments-api.gitlab-ci.ymlinherit
publish-docs:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
  inherit:
    default: [tags, retry]
    variables: false
  script:
    - mkdocs build --strict
  rules:
    - !reference [.rules, default-branch]
QuestionWhere is it defined, when does it run, where does it run, what does it execute, and what goes in and out? Which answer should worry its owners?
Show the answer
  • Defined in payments-api's own file. Nothing extends it, and inherit takes only tags and retry from default:.
  • When: in pipelines for main only, through the default-branch rule.
  • Where: on the megacorp-shared runner, because it inherits tags, in the docs:1.4 image.
  • What: mkdocs build --strict.
  • In and out: no YAML variables, because variables: false, but settings variables still arrive. Out: nothing at all.

The last answer is the worry. The job builds the documentation site and then ends. With no artifacts and no deploy step, the site is thrown away with the job's pod. Its name promises a publication that never happens. See the five questions.

Field guide to enterprise setups#

Part XI · Decode and debug · Chapter 56 · 7 min read

Large organisations build their pipelines in a handful of recognisable ways. Each leaves fingerprints: signs in a repository, a pipeline graph or a job log that tell you which setup you are looking at. Recognise the setup first, and you know where the answers live before you start reading YAML. This chapter lists ten pipeline setups and eight delivery setups, with their fingerprints and where to confirm each one.

Wednesday, 10:00. Ten shapes of enterprise pipeline.

Your organisation is not MegaCorp. This morning its pipeline gained a stage nobody on your team added, and your project's history shows no change: the golden pipeline you include moved under you. Every large organisation builds from the same ten shapes, and each shape has its own way of changing overnight. Name yours, and the rest of the book tells you where its traps are. (the card)

Ten setups and their fingerprints#

SetupFingerprintsConfirm itChapters
The golden pipelinea .gitlab-ci.yml of a few lines: one include: project: of a central file at a tag, a few variablesFull configuration shows dozens of jobs that the file never names14–15
Componentsinclude: component: lines ending in @1.2.0, with inputs:the component's project, and its spec: inputs16
The generated monorepoa job that writes YAML, then a trigger that runs it; child pipelines to the right of the graphthe generator script, and the trigger's include: artifact:17
The central deployerdeploy jobs that are trigger: project: jobs; environments that live in another projectthe deployer project's pipeline and environments17, 40
GitOps deliverydeploy jobs that only commit to a configuration repositorythat repository's history, and the controller's sync status45
Policy-injected jobsjobs named like secret-detection-1, or in .pipeline-policy-pre and .pipeline-policy-post stages**Secure › Policies**18, 36
The invisible runner layerbehaviour that differs by runner; environment variables nobody set in YAMLthe runner's name in the job log, and the runner's configuration19, 29
The toolbox imagescripts calling commands such as mc or shell functions that the repository doesn't containthe job's image, and the image's own repository19
Cloud access by identityid_tokens: in templates, a login snippet, and no cloud keys in settingsthe role's trust policy, and aws sts get-caller-identity28, 38
The 2019 legacy fileanchors and <<: merge keys, only: and except:, include: remote:, trigger tokensthe parse phase: anchors are resolved in Full configuration13, 15, 23

MegaCorp shows all ten: payments-api is a golden pipeline with a component, mono is a generated monorepo, the deployer is central and GitOps-based, and billing-batch is the 2019 legacy file.

Eight delivery setups and their fingerprints#

After the pipeline, deploys follow a handful of setups of their own. These are the ones you'll meet, with where to confirm each:

SetupFingerprintsConfirm itChapters
Push deploysa deploy job runs kubectl apply, helm upgrade --install or a cloud's CLI, and holds the cluster's credentialsthe job's script, and where its kubeconfig or cloud identity comes from41, 38
GitOps with Argo CDan environment repository with Application or ApplicationSet files; deploy jobs that only committhe Application's sync status and revision, in Argo CD's UI47
GitOps with Fluxa clusters/<name>/flux-system/ folder in a repository, with Kustomization and HelmRelease objectsthe Kustomization's Ready condition, and the revision it last applied46
GitLab's agent for Kubernetesa .gitlab/agents/<name>/config.yaml file in some project; jobs whose $KUBECONFIG has contexts for agent connections**Operate › Kubernetes clusters**, and the agent's configuration file46
Image automationcommits in the environment repository by a bot, changing only image tagsImagePolicy and ImageUpdateAutomation objects, or an Argo CD ImageUpdater45
Infrastructure from modules.tf or .tofu files that call modules at ?ref= tags; plans in merge requests; backend "http"**Operate › Terraform states**, and the plan job's output48
Progressive deliveryRollout objects where Deployments would be, or Canary objects beside them; AnalysisTemplatesthe Rollout's or the Canary's status, step by step51
A repository from a portal templatea catalog-info.yaml with a backstage.io/source-template annotation; a first commit called initial commit, by Scaffolderthe portal's template, and whether the pipeline file copies jobs or includes a central file53

MegaCorp shows five of them. Flux installs its platform add-ons, Argo CD deploys its services, the platform team's OpenTofu builds both clusters from modules, and payments-api's repository came from the portal's template. Its deploy marks come from Argo CD too (chapter 52).

No .gitlab-ci.yml, but pipelines run#

This is the most disorienting fingerprint of all. Five mechanisms can produce it, and each has a place to confirm it:

A project has no .gitlab-ci.yml, yet pipelines run. Check these in order.yesnoyesnoyesnoyesnoyesnoDoes Settings › CI/CD › Generalpipelines name a configuration fileelsewhere?The pipeline comes from that file,perhaps in another projectAre the only jobs scanners with nameslike secret-detection-1?A scan execution policy creates apipeline file for the projectimplicitlyIs a pipeline execution policy set tooverride_project_ci?The policy's configurationreplaces the project's entirelyIs Auto DevOps turned on for theproject, group or instance?GitLab's Auto DevOps pipeline runsDoes the project belong to a complianceframework with its own pipeline?A compliance pipeline runs; theseare deprecated in favour ofpoliciesCheck the pipeline's source, and itsfirst job's origin (chapter 13)
Figure A project has no .gitlab-ci.yml, yet pipelines run. Check these in order.#
As text
  1. Does Settings › CI/CD › General pipelines name a configuration file elsewhere? Yes: The pipeline comes from that file, perhaps in another project. No: the next step.
  2. Are the only jobs scanners with names like secret-detection-1? Yes: A scan execution policy creates a pipeline file for the project implicitly. No: the next step.
  3. Is a pipeline execution policy set to override_project_ci? Yes: The policy's configuration replaces the project's entirely. No: the next step.
  4. Is Auto DevOps turned on for the project, group or instance? Yes: GitLab's Auto DevOps pipeline runs. No: the next step.
  5. Does the project belong to a compliance framework with its own pipeline? Yes: A compliance pipeline runs; these are deprecated in favour of policies. No: the next step.
  6. Check the pipeline's source, and its first job's origin (chapter 13)

MegaCorp's web-portal is the first case. Its settings point at pipelines/web.yml in ci-templates, so its repository needs no pipeline file (chapter 18):

megacorp/web/web-portalSETTINGS.ymlconfig-path
ci_cd_configuration_file: pipelines/web.yml@megacorp/devops/ci-templates
What you can now do
  1. Recognise the ten shapes of enterprise setup, and name which ones your organisation uses.
  2. Say where each shape hides its logic, and which chapter decodes it.
  3. Predict the traps a shape brings before you meet them.
Spot the bugWhich setup is this?#

You join a team and open their newest pipeline. You notice three things:

  • the project's .gitlab-ci.yml is nine lines long, and includes one file from platform/ci-library at ref: v7.1.0
  • one job, deploy-prod, is a trigger job pointing at platform/release
  • two jobs are called sast-1 and secret-detection-1
QuestionWhich three setups are in play, and where would you look first for each?
Show the answer
  • A golden pipeline: read platform/ci-library at v7.1.0, starting from Full configuration.
  • A central deployer: the real deploy happens in platform/release's pipeline and environments.
  • Policy-injected jobs: the numbered scanners come from a scan execution policy, so check Secure › Policies.

See ten setups and their fingerprints.

Idioms#

Part XI · Decode and debug · Chapter 57 · 5 min read

Central teams repeat a small number of patterns, called idioms, in every organisation. Each one solves a real problem for the team that writes the templates. Each also hides something from the people who read the pipelines. Learn to spot them, and an unfamiliar pipeline stops looking unfamiliar. For each idiom, this chapter shows what it looks like, why central teams use it, and how it confuses a reader.

Wednesday, 15:20. A rule that only works on main.

A job's rule says if: $DEPLOY_KEY. It matches on main and never on a branch. The variable is protected, so on a branch it is empty, and an empty variable is false. MegaCorp's templates are full of idioms like this: patterns that look like magic until you have seen each one once. (the card)

A rules library, pulled in with !reference#

This is the idiom that confuses most readers, so here it is in full. MegaCorp keeps its conditions in one hidden job:

megacorp/devops/ci-templatestemplates/rules.ymlrules-library
.rules:
  mr:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  default-branch:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  release-tag:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
  never-on-schedule:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never

Jobs assemble their rules from it: - !reference [.rules, mr].

  • Why central teams use it: "which pipelines does this job run in" is decided once, with one name for each condition. Every template agrees on what a release tag looks like.
  • What it hides: the conditions live in another file, so a job's rules read like a list of names. The names say nothing about the order, and in rules the order decides the outcome, because the first match wins. If you replace the job's rules:, every referenced rule disappears with them (chapter 14).

The catalogue#

IdiomLooks likeWhy central teams use itHow it confuses youChapters
The hidden base jobextends: .megacorp-baseshared rules, variables and expiry in one placesettings arrive from files you never opened; its variables beat your top-level ones14
Script snippets- !reference [.snippets, aws_login]the same few script lines in every templatea job's own before_script drops them; the functions they call live in the image14, 19
Variables as parametersimage: maven:3.9-jdk${JAVA_VERSION}one template serves many projectsthe default is in the template; a settings variable beats your YAML14, 26
Togglesif: $SKIP_SONAR == "true" then when: neveran escape hatch without editing the templatea toggle set in settings is invisible; policy jobs ignore toggles34, 36
The pinned includeinclude: project: … ref: v4.2.0projects upgrade when they choose towhat runs is that tag's content, not the central repo's current version; a tag can be moved15
Components with inputscomponent: …/sonar-scan@2.1.0 with inputs:typed, versioned settingsinputs are filled in before anything else, even into job names16
The dotenv hand-offecho "IMAGE_REF=…" >> build.envpasses a build's result to later jobsno file shows the value; needs and dependencies can block it; rules can't see it31
The trigger to a deployertrigger: project: …/deployer with strategy: dependone team controls every environmentthe deploy happens in another project's pipeline17, 40
Never on schedulethe first rule excludes schedulenightly pipelines run only what they shouldjobs are silently absent from scheduled pipelines23, 25
Opting out with inheritinherit: default: [tags, retry]a job that shouldn't get the shared image or scriptswhich defaults still apply is easy to misread14
No duplicate pipelinesa workflow rule with $CI_OPEN_MERGE_REQUESTSone pipeline per push, not twoonce a merge request opens, branch pipelines stop appearing22
Build once, deploy by referencethe image tagged with $CI_COMMIT_SHA, then passed onwhat is tested is what shipsa pipeline that rebuilds breaks the promise quietly39

Most real pipelines combine several idioms in a single job. Recognising each one tells you which file to open next.

What you can now do
  1. Read the dozen idioms MegaCorp's templates use, and say what each is for.
  2. Recognise a rule that tests a protected variable, and predict where it matches.
  3. Write your own rules and snippets the way the central team will expect to read them.
Spot the bugCount the idioms#

This is the template behind payments-api's image-build:

megacorp/devops/ci-templatestemplates/container.ymlimage-build
.image-build:
  extends: .megacorp-base
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
    IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
    - buildah push "$IMAGE:$CI_COMMIT_SHA"
    - !reference [.snippets, acr_push]
    - echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
  artifacts:
    reports:
      dotenv: build.env
QuestionWhich idioms from this chapter can you find, directly or through what the job extends?
Show the answer
  • The hidden base job: extends: .megacorp-base, which also brings in the rules library and never on schedule.
  • Script snippets: the three !reference [.snippets, …] lines, two before the script and one in it.
  • Variables as parameters: AWS_ROLE_ARN and ECR_REGISTRY, which a project could override.
  • The dotenv hand-off: IMAGE_REF written to build.env.
  • Build once, deploy by reference: the image tagged with $CI_COMMIT_SHA.

The job reaches payments-api through a pinned include as well: java-service.yml includes it at v4.2.0. See the catalogue.

Worksheet: decode your own pipeline#

Part XI · Decode and debug · Chapter 58 · 5 min read

Print this page, or copy it into a private note, and fill it in for one of your own projects. It is the five questions, for a job and for a deployment, plus the few facts about your organisation that make every answer easier. Keep your answers private, because they describe your organisation's systems.

Once for your organisation#

Most of these answers are the same for every project, so you only find them once:

QuestionWhere to find the answerYour answer
Where do the central templates and components live?the include: lines of any project, and the platform team's group…
Which versions do projects pin, and who moves them?the ref: of each include, and the templates' release notes…
Which runners run your jobs, and which tags do they need?Settings › CI/CD › Runners, and default: tags…
Which executor do those runners use?the first lines of any job log…
Which CI/CD variables do your groups and the instance set?each group's Settings › CI/CD › Variables, if you can see it…
Which security policies apply?Secure › Policies…
How do jobs reach the cloud?id_tokens: in the templates, and the login script…
Where do deployments really happen?trigger jobs, environments, and any GitOps repository…

For each job you need to understand#

QuestionWhere to lookYour answer
Where is it defined?Full configuration; its extends chain; if it's in no file, policies…
When does it run?workflow:rules, its final rules, needs; compare pipeline types…
Where does it run?the job page's runner and tags; the job's image…
What does it execute?its scripts, and the tools and functions they call…
What goes in and out?variables, secrets, artifacts, caches, reports, images, deployments…

For each deployment you need to understand#

The same five questions, asked after the pipeline (a deployment, decoded):

QuestionWhere to lookYour answer
Where is it defined?the environment repository; the chart and its version; values files, overlays and inline values…
When does it deploy?the commit or merge request that changed it; the controller and its sync policy…
Where does it run?the cluster and namespace, in the controller's Application or Kustomization…
What does it run?the image reference, as rendered; whether that tag is in the cluster's registry…
What goes in and out?settings and secrets, and where each comes from; its logs, metrics and traces, and where they're kept…

When something goes wrong#

Write down, in this order:

  1. The symptom, in one sentence, and its group: didn't run, failed, wrong result, or stuck or drifting (chapter 55).
  2. The pipeline and job, with links, and what changed since the last good run: a commit, a setting, a template version, a runner, a date.
  3. The cards you tried, and what each confirmation showed.
  4. The fix, and who had to make it: you, the platform team, the security team or the cloud team.

Questions for your platform team#

Some answers aren't visible from a project at all. These are worth asking once:

  • What do the runners add to every job: environment variables, a pre_build_script, limits?
  • Which of the group's CI/CD variables are protected, and why?
  • Which stored tokens do the pipelines depend on, who owns them, and when do they expire?
  • How are template versions released, and can a tag ever move?
  • Which security policies are enforced, and on which projects?
  • Which controller deploys each environment, how often does it sync, and who may merge into the environment repository?
  • Where do the services' logs, metrics and traces go, and is each deploy marked on the dashboards?

Cards: didn't run#

Part XI · Decode and debug · Chapter 59 · 14 min read

Use these cards when a job is missing from a pipeline, when the pipeline wasn't created, or when a job ran somewhere it shouldn't have. The most common causes come first. Each card gives the symptom, the cause, how to confirm it, and the fix. One more card for this group is in chapter 14: your rules replaced the template's. The last cards cover what happens after the pipeline.

Didn't run, or ran when it shouldn'tRules and conditions#
A job runs in branch pipelines but is missing from merge request pipelines, or the other way round.
Cause

Merge request pipelines and branch pipelines are different pipelines, with different variables. CI_COMMIT_BRANCH, for example, isn't set in merge request pipelines, so a rule that tests it is false there. The job has no rule for $CI_PIPELINE_SOURCE == "merge_request_event".

Confirm

Open both pipelines and compare their job lists. Each pipeline's page shows its source. Then read the job's final rules in Build › Pipeline editor › Full configuration.

Fix

Add a rule for merge request pipelines, or reuse your organisation's rule from its rules library with !reference. See merge request pipelines.

Didn't run, or ran when it shouldn'tRules and conditions#
A push creates no pipeline at all, or Run pipeline refuses to start one.
Cause

workflow:rules decide whether a pipeline exists. None of them matched this push, or the one that matched said when: never.

Confirm

Run pipeline and the API answer with:

The pipeline did not run. Review the workflow:rules configuration for the pipeline.

Read workflow: in Full configuration.

Fix

Add the missing pipeline source or branch to the workflow rules, or push where the workflow allows. See the first matching rule.

Didn't run, or ran when it shouldn'tRules and conditions#
The workflow allows the pipeline, yet it isn't created, because no job would have been in it.
Cause

Every job's rules excluded it for this source and branch. MegaCorp's deployer does this deliberately: all its jobs need IMAGE_REF, which only a trigger supplies.

Confirm

GitLab says:

The resulting pipeline would have been empty. Review the rules configuration.

The Validate tab, run for the same branch, shows no jobs.

Fix

Provide what the rules expect, such as the variable a trigger would send, or change the rules. See when a pipeline is missing.

Didn't run, or ran when it shouldn'tRules and conditions#
Pushing a tag such as v1.5.0-rc1 creates no pipeline, and shows no error anywhere.
Cause

The workflow allows tag pipelines, but no job's rules match this tag. MegaCorp's release rule accepts only tags shaped like v1.5.0, exactly ^v\d+\.\d+\.\d+$. A pipeline with no jobs isn't created, and on a push GitLab doesn't say why.

Confirm

Run Validate for the tag: it shows no jobs. Starting a pipeline for the tag by hand gives The resulting pipeline would have been empty. Review the rules configuration.

Fix

Name tags the way the rules expect, or widen the rule if release candidates should build. See what the truth table says.

Didn't run, or ran when it shouldn'tScanners and policies#
GitLab's scanner jobs, such as secret_detection, appear in branch pipelines but never in merge request pipelines.
Cause

GitLab's scanner templates run their jobs in merge request pipelines only when AST_ENABLE_MR_PIPELINES is "true". Otherwise the jobs run in branch pipelines only.

Confirm

In Full configuration, the job's rules contain the AST_ENABLE_MR_PIPELINES conditions. The branch pipeline for the same commit has the job.

Fix

Set AST_ENABLE_MR_PIPELINES: "true", in the YAML or in settings. The job then moves to the merge request pipeline. Or redefine the job's rules. See the merge request gap.

Didn't run, or ran when it shouldn'tRules and conditions#
A job runs on every push, but never in scheduled pipelines.
Cause

Its rules start with a rule that excludes schedules. MegaCorp's .megacorp-base begins with !reference [.rules, never-on-schedule], so every job that extends it skips scheduled pipelines.

Confirm

The job's final rules in Full configuration start with the schedule rule, and the pipeline's source is schedule.

Fix

If the job should run on a schedule, give it rules without that line, or ask the template's owners to change the base. See each job's final rules.

Didn't run, or ran when it shouldn'tRules and conditions#
A job shows a play button and never starts. The stages after it may wait too.
Cause

The job is manual, through when: manual on the job or in the rule that matched. Manual jobs run only when someone starts them. A manual job added by a rule blocks the later stages, unless it has allow_failure: true.

Confirm

The job page offers Run, and its rules or when in Full configuration say manual.

Fix

Run it. If nothing should wait for it, add allow_failure: true to that rule. See manual jobs.

Didn't run, or ran when it shouldn'tRules and conditions#
A job with rules: changes runs on the first push of a new branch, or on a tag, although none of its files changed.
Cause

For new branches, for tags, and for pipelines without a push event, rules: changes is always true. There is nothing to compare with. On the local GitLab, mono's first push ran its docs job although there was no docs/ folder.

Confirm

The pipeline is a tag pipeline, or the branch's first pipeline.

Fix

Compare against a fixed branch with changes: compare_to:, or add a rule that excludes tags. See rules about files.

Didn't run, or ran when it shouldn'tRules and conditions#
Every push to a merge request's branch creates two pipelines, and every job runs twice.
Cause

The workflow allows both a branch pipeline and a merge request pipeline for the same push.

Confirm

The pipelines list shows two pipelines for the same commit, one of them marked as a merge request pipeline.

Fix

Add a workflow rule that skips branch pipelines while a merge request is open, using $CI_OPEN_MERGE_REQUESTS. See duplicate pipelines.

Didn't run, or ran when it shouldn'tScanners and policies#
A scanner job has vanished from every pipeline, and nobody changed the YAML.
Cause

Someone set its toggle, such as CONTAINER_SCANNING_DISABLED, in the project's or a group's CI/CD settings. The template's first rule removes the job. MegaCorp's own container scanning rule accepts only "true".

Confirm

Look for the variable in the project's and each group's CI/CD settings. Its rule is the job's first.

Fix

Remove the setting if the scan should run. A scan execution policy's jobs are never affected by toggles. See toggles against policies.

Didn't run, or ran when it shouldn'tScanners and policies#
The SAST template is included, but no SAST job appears.
Cause

Each SAST analyzer's job exists only when files that it reads are in the repository, such as *.java or *.py for semgrep-sast.

Confirm

In Full configuration, compare the analyzer's exists patterns with the repository's files.

Fix

None is needed if the repository has no code in those languages. Otherwise check that the source files are where the patterns look. See what the templates add.

Didn't run, or ran when it shouldn'tScanners and policies#
The dependency scanning template is included, but its jobs never appear.
Cause

Those jobs' rules require the dependency_scanning feature, which comes with Ultimate. On Free and Premium, the template adds nothing, without an error.

Confirm

The jobs' rules in Full configuration test $GITLAB_FEATURES.

Fix

Use Ultimate, or run another dependency scanner as an ordinary job (chapter 35).

Didn't run, or ran when it shouldn'tRules and conditions#
The pipeline isn't created, and the error names a job that another job needs.
Cause

The needed job's rules left it out of this pipeline, while the job that needs it is still in.

Confirm

GitLab's message, captured here with other job names, says:

'unit-tests' job needs 'compile' job, but 'compile' does not exist in the pipeline.
This might be because of the only, except, or rules keywords. To need a job that
sometimes does not exist in the pipeline, use needs:optional.
Fix

Mark the need as optional, or give both jobs matching rules:

unit-tests:
  needs:
    - job: compile
      optional: true

See needs.

Didn't run, or ran when it shouldn'tYAML itself#
The pipeline isn't created, and the error says a stage doesn't exist.
Cause

A job names a stage missing from stages:. Often a project redefines stages: without test, which GitLab's scanner templates use.

Confirm

The message names the job, its stage and the stages that exist:

orphan job: chosen stage nowhere does not exist; available stages are .pre, build, .post
Fix

Add the stage to stages:, or move the job to one that exists. See stages.

Didn't run, or ran when it shouldn'tYAML itself#
The pipeline isn't created, because of a rule written in a way that looks reasonable.
Cause

Two common forms fail. A negation written as !( … ) is invalid. A quoted variable, as in "$VAR" == "x", breaks the YAML itself.

Confirm

CI Lint shows one of these, captured on the local GitLab:

jobs:job:rules:rule if invalid expression syntax
(): did not find expected key while parsing a block mapping at line 5 column 7
Fix

Use != for negation, and never quote the variable: if: $CI_COMMIT_BRANCH != "nope". See writing conditions.

Didn't run, or ran when it shouldn'tVariables#
A rule tests a variable that an earlier job sets, and never matches.
Cause

Rules are decided when the pipeline is created, before any job runs. Values from a dotenv report, or from export in a script, don't exist yet.

Confirm

Find where the variable is set. If it is set by a job, no rule can see it.

Fix

Make the decision inside the job's script instead, or in a child pipeline that a later job generates. See when each thing is decided.

Didn't run, or ran when it shouldn'tVariables#
A rule such as if: $DEPLOY_KEY matches on main, but never on feature branches.
Cause

The variable is protected. Protected variables exist only in pipelines on protected branches and tags, so elsewhere the rule sees an unset variable.

Confirm

In the CI/CD settings where it is defined, the variable has Protect variable ticked.

Fix

Don't gate jobs on secrets. Test the branch, or an ordinary variable, instead. See who gets a protected value.

The cards from here on are for what happens after the pipeline: the environment repository, the controllers, the cluster and the clouds.

Didn't run, or ran when it shouldn'tGitOps sync and drift#
A change reached staging, but production still runs the old version, and no job failed.
Cause

Production deploys wait for a person. mc deploy pushes production's change to a branch of the environment repository and opens a merge request, which nobody has merged yet. The controller deploys only what is on main.

Confirm

The environment repository has an open merge request called Deploy payments-api 1c7d4b09 to production, or similar.

Fix

Review it and merge it. See promotion by merge request.

Didn't run, or ran when it shouldn'tCluster and cloud access#
An image is in ECR, but not in Azure's registry.
Cause

MegaCorp's image-build pushes to ACR only in pipelines on main. This image was built in a merge request, tag or feature-branch pipeline, where Entra ID wouldn't accept the job's token anyway.

Confirm

The job log says:

[megacorp] not on main, so not pushing to megacorpprod.azurecr.io
Fix

Deploy images built on main, which reach both registries. See how a MegaCorp job logs in.

Cards: failed#

Part XI · Decode and debug · Chapter 60 · 21 min read

Use these cards when a job, or the pipeline, turned red. Start with the job log's first error, not its last line. The most common causes come first. Each card gives the symptom, the cause, how to confirm it, and the fix. The last cards cover what happens after the pipeline.

FailedMerging and inheritance#
A job fails at once, saying a company function or tool, such as mc_log, can't be found. The same template works in other projects.
Cause

The job sets its own before_script. That replaces the template's or the default's list completely, including the line that loaded the functions.

Confirm

In Full configuration, the job's before_script shows only your lines.

Fix

Put the template's lines back with !reference, then add yours. See the before_script exercise.

FailedImages and Docker#
The job fails before its script, while the runner is pulling the job's image.
Cause

The image is in a private registry, and the runner has no credentials for it. The job's own script can't help, because the runner pulls the image before the job starts.

Confirm

The error appears in the job log's preparation lines, before any of your commands.

Fix

Give the runner credentials with DOCKER_AUTH_CONFIG, or a credential helper on the runner. For images in the same GitLab, allow your project's job token in the image's project. See private registries.

FailedArtifacts, cache and data#
A job fails before its script starts.
Cause

It needs artifacts from an earlier job, and can't get them. They may have expired, the job may not be a dependency, or the user may lack permission.

Confirm

The job says:

This job could not start because it could not retrieve the needed artifacts.
Fix

Retry the earlier job first, so that new artifacts exist, or run a new pipeline. Lengthen expire_in if jobs routinely run later. See artifacts.

FailedSecrets, tokens and identity#
A job that signs in to something fails with an authentication error, but only on feature branches or in merge request pipelines.
Cause

Its token is a protected variable. Protected values reach only pipelines on protected branches and tags, so elsewhere the variable is empty.

Confirm

In the CI/CD settings, the variable is protected. In the job, print the variable's length, never its value.

Fix

Decide with the token's owners: an unprotected token with fewer rights, a separate token for merge requests, or no sign-in there. See who gets a protected value.

FailedSecrets, tokens and identity#
The AWS login step fails in some pipelines, for example only tag pipelines, with this error:
Cause

The role's trust policy doesn't accept this token. Usually the sub pattern allows branches but not tags, or the audience is wrong. A wildcard written with StringEquals never matches.

Confirm
An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

Print the token's claims, not the token, and compare sub and aud with the trust policy.

Fix

Ask the account's owners to add a pattern for the missing pipelines, using StringLike. See when AWS says no.

FailedVariables#
A trigger job fails, and the downstream pipeline is never created.
Cause

The downstream project limits who may set pipeline variables. The variables the trigger job sends, including top-level ones it inherits, count as pipeline variables.

Confirm

The trigger job shows:

Failed - (downstream pipeline can not be created, Insufficient permissions to set pipeline variables)
Fix

Send inputs instead of variables, or ask the downstream project to allow the role. See values given when the pipeline starts.

FailedSecrets, tokens and identity#
A call to another GitLab project fails with 404 Not Found, although the project exists.
Cause

The job used CI_JOB_TOKEN. The other project hasn't added yours to its job token allowlist, or the person who started the pipeline isn't a member there.

Confirm

A refused clone says:

remote: The project you were looking for could not be found or you don't have permission to view it.
Fix

A Maintainer of the target adds your project, or its group, under Settings › CI/CD › Job token permissions. See the job token.

FailedGit in CI#
A script that uses Git history fails in CI, often only on big merge requests. Or a retried old job reports unresolved reference.
Cause

The job has a shallow clone: only the newest commits, 20 by default, and only the pipeline's own ref.

Confirm

The commit the script needs is older than the clone's depth, or is on a branch the job didn't fetch.

Fix

Raise GIT_DEPTH for that job, or fetch what the script needs, for example with git fetch --unshallow. See shallow clones.

FailedImages and Docker#
With a tool's own image, such as amazon/aws-cli, the job fails without running your script. The same job works on another runner.
Cause

The image's entrypoint runs the tool, not a shell. The Docker executor keeps the entrypoint. The Kubernetes executor ignores it by default.

Confirm

Compare the runners' executors. The failing one uses Docker.

Fix

Set entrypoint: [""] under image:. See the job's image.

FailedImages and Docker#
Tests can't reach a service, such as a database, and the log warns: *** WARNING: Service XYZ probably didn't start properly.
Cause

The service lacked a setting it needs to start, such as its password. That value is a CI/CD variable from settings, and those never reach service containers.

Confirm

The variable is defined in settings, not in the YAML. CI_DEBUG_SERVICES: "true" shows the service's own log, but can reveal masked values.

Fix

Re-assign it in the YAML under a different name, such as POSTGRES_PASSWORD: $TEST_DB_PASSWORD. See services.

FailedRunners and execution#
A job stops at, say, 30 minutes, although its timeout says 2 hours.
Cause

The runner has a shorter maximum job timeout. A runner's maximum wins over the job's own timeout and the project's.

Confirm

The runner's settings show its maximum job timeout.

Fix

Use a runner with a longer maximum, or split the job. See time limits.

FailedRules and conditions#
A trigger job for a child pipeline fails in merge request pipelines, but works in branch pipelines.
Cause

The child configuration's jobs have no rules. In a merge request pipeline, a child whose jobs have no rules gets no jobs at all, so it can't be created. On the local GitLab, mono's docs trigger failed this way in every merge request pipeline.

Confirm

The trigger job's failure reason is downstream_pipeline_creation_failed.

Fix

Give the child's jobs rules that allow merge request pipelines. See child pipelines.

FailedVariables#
A command receives the literal text $MY_VAR instead of a value.
Cause

The job defines MY_VAR: $MY_VAR. The job's own definition hides the default of the same name, so there's nothing to expand.

Confirm

The job's variables in Full configuration re-use the name.

Fix

Use a different name for the job's variable. See variables in YAML.

FailedArtifacts, cache and data#
A later job can't find a file that an earlier job should have made.
Cause

The earlier job's artifacts: paths matched nothing: the path is wrong, or the file wasn't created.

Confirm

The earlier job's log says No files to upload.

Fix

Fix the path, which is relative to the project directory, or the command that makes the file. See artifacts.

FailedSecrets, tokens and identity#
Every AWS login fails, in every project, with an error about a verification key.
Cause

AWS must download GitLab's signing keys from the instance, and can't reach it. This happens with a self-managed GitLab behind a firewall.

Confirm
An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn't retrieve verification key from your identity provider
Fix

Make the instance's OIDC keys reachable, or publish them elsewhere, as GitLab's documentation describes. See small things that bite.

FailedDeployments and delivery#
A deploy job fails straight away, in an older pipeline.
Cause

Prevent outdated deployment jobs is on, and a newer deployment already ran. GitLab judges a job's age by when it started.

Confirm
The deployment job is older than the latest deployment, and therefore failed.
Fix

Deploy from a newer pipeline, or use Rollback environment to go back deliberately. See one at a time, newest wins.

FailedDeployments and delivery#
An image push to ECR fails with ImageTagAlreadyExistsException.
Cause

The repository has immutable tags, and this tag was already pushed, often by an earlier pipeline that built the same commit.

Confirm

ECR already lists the tag, with an earlier push time.

Fix

Don't rebuild what was already built: deploy the existing image, and add new tags with put-image. See promoting without rebuilding.

FailedImages and Docker#
A job fails at once with a tool's small image, before any of your commands.
Cause

A job's image must contain sh or bash, and grep, because the runner sends the script to a shell. Very small images leave them out.

Confirm

Run the image locally and look for sh.

Fix

Use a variant of the image that includes a shell, or build one. See the job's image.

The cards from here on are for what happens after the pipeline: the environment repository, the controllers, the cluster and the clouds.

FailedPods and Kubernetes objects#
New pods never start. Their status is ErrImagePull, then ImagePullBackOff.
Cause

Kubernetes can't pull the image: its name or tag doesn't exist in the registry that the cluster uses, or the nodes aren't allowed to pull from it. Kubernetes keeps trying, and waits longer each time, up to five minutes.

Confirm

The pod's events, in kubectl describe pod, name the image and the registry's answer. An answer that ends in not found means the registry has no such tag: check the tag, and the registry. On aks-prod, the usual cause is a tag that exists only in ECR (an image not in Azure's registry). An answer that ends like this means the tag may well exist, but the cluster isn't allowed to pull it:

pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Fix

Deploy a tag that exists in the cluster's registry. If it exists and the pull is refused, check what the cluster pulls with. On aks-prod, the nodes need AcrPull. A pull secret made from a GitLab deploy token needs read_registry on the project that built the image; a token from any other project is refused. See from tag to pod.

FailedPods and Kubernetes objects#
Pods restart again and again, and their status shows CrashLoopBackOff.
Cause

The container starts and then exits, over and over. Kubernetes waits longer before each restart: 10 seconds, then 20, then 40, up to five minutes. The cause is in the program, or in what it was given, such as a missing setting or secret.

Confirm

Read the last run's output with kubectl logs <pod> --previous. The pod's events show the restarts.

Fix

Fix what the logs show, in the code or in the environment's values. See config for each environment.

FailedCharts, values and overlays#
Rendering the chart, or Argo CD's sync, fails with image.tag is set for each environment.
Cause

MegaCorp's chart refuses to render without an image tag. Its Deployment template uses Helm's required, and the environment's values file has no image.tag.

Confirm
Error: execution error at (service/templates/deployment.yaml:28:52): image.tag is set for each environment

The values file for that environment has no tag under image.

Fix

Set the tag. mc deploy normally writes it; a values file made by hand needs one too. See values.

FailedCluster and cloud access#
A job's Azure sign-in fails with AADSTS70021: No matching federated identity record found for presented assertion.
Cause

No federated credential on the application matches the token's issuer and subject. Entra ID compares them exactly, with no wildcards, so each new branch or tag needs a credential of its own. A credential added in the last few minutes may not have spread yet.

Confirm

Print the token's claims, and compare sub with each federated credential's subject, character by character.

Fix

Add a credential for that exact subject, or sign in only from refs that have one. See what must match.

FailedCluster and cloud access#
An Azure sign-in fails with AADSTS700212, about the token's audience.
Cause

The federated credential's single audience differs from the token's aud. For a pod on AKS, the audience is api://AzureADTokenExchange. For a GitLab job, it is whatever the job's id_tokens: asks for.

Confirm

Compare the credential's audience with aud in the token's claims.

Fix

Make them the same. See pods and their cloud identity.

FailedCluster and cloud access#
az acr login fails, although the identity may push to the registry.
Cause

az acr login needs to read the registry through Azure Resource Manager. AcrPush gives data-plane access only, to push and pull images, and no control-plane access.

Confirm

The identity's role assignments on the registry show AcrPush and nothing else.

Fix

The registry's owners add the role that Microsoft documents for az acr login: Container Registry Contributor and Data Access Configuration Administrator. See what must match.

FailedCluster and cloud access#
A pod on AKS can't reach Key Vault or another Azure service, and has no AZURE_FEDERATED_TOKEN_FILE variable.
Cause

AKS gives a pod its Azure identity only when the pod template has the label azure.workload.identity/use: "true", and its service account names the identity in the azure.workload.identity/client-id annotation. Without the label, the webhook leaves the pod alone.

Confirm

kubectl get pod <pod> -o yaml shows neither the label nor the variable.

Fix

Add both through the chart's values, as MegaCorp's aks-prod values do. See pods and their cloud identity.

FailedCluster and cloud access#
A pod on EKS gets AccessDenied naming the nodes' role, or finds no AWS credentials at all.
Cause

The pod has no identity of its own: its service account has neither the eks.amazonaws.com/role-arn annotation nor a Pod Identity association. The AWS SDK then falls back to whatever the node allows, often the node's own role.

Confirm

The service account's annotations, and the cluster's Pod Identity associations for that namespace and service account.

Fix

Annotate the service account, or ask for an association, through the platform team's OpenTofu. See pods and their cloud identity.

FailedInfrastructure state and plans#
A plan or apply fails straight away with Error acquiring the state lock.
Cause

Another run holds the state's lock, or a cancelled or killed job took it and never released it.

Confirm

The log shows the lock:

│ Error: Error acquiring the state lock
│
│ Error message: HTTP remote state already locked:
│ ID=5b0e8c3a-7d2f-4c1e-9a64-3f8b2d1c0e97

In the Lock Info that follows, ID, Who and Created describe the lock. Operate › Terraform states shows the state as locked.

Fix

If a run is still going, wait for it. If nothing is running, a Maintainer unlocks the state under Operate › Terraform states, or runs tofu force-unlock with the lock's ID. Then read a fresh plan before anything applies. See state and its lock.

FailedCharts, values and overlays#
A HelmRelease stays failed with Helm install failed and timeout waiting for, and nothing tries again.
Cause

A Helm action has five minutes by default, and that includes the chart's hook Jobs. The install ran out of time, often because an image was still downloading. Flux retries a failed install only if install.remediation.retries allows it, and the default is 0, so the release stays failed until its settings or its chart change.

Confirm

flux get helmreleases -A names what the install was waiting for:

Helm install failed for release argocd/argo-cd with chart argo-cd@10.9.0: failed pre-install: timeout waiting for: [Job/argocd/argo-cd-argocd-redis-secret-init status: 'InProgress']

A Kustomization that applied it with wait: true fails its health check too.

Fix

Once the cause has passed, flux reconcile helmrelease argo-cd -n argocd --force makes one more attempt. To avoid it next time, give the HelmRelease a longer timeout, or some install.remediation.retries. See charts through Flux.

FailedGitOps sync and drift#
Argo CD's sync fails with resource :Namespace is not permitted in project, and nothing is deployed.
Cause

The Application syncs with CreateNamespace=true, so Argo CD makes the destination's Namespace. A Namespace is a cluster-wide object, and a project other than default allows no cluster-wide kinds unless its clusterResourceWhitelist lists them.

Confirm

In argocd app get, the Namespace's row shows SyncFailed, with this message:

resource :Namespace is not permitted in project payments

The AppProject has no clusterResourceWhitelist, or one without Namespace.

Fix

Allow the kind in the project, only with the team's names:

clusterResourceWhitelist:
  - group: ""
    kind: Namespace
    name: payments-*

Or have the platform team create the namespaces, and drop CreateNamespace. See Argo CD's objects.

FailedCluster and cloud access#
An Argo CD Application's sync status is Unknown, and its error ends with x509: certificate signed by unknown authority.
Cause

The Git server's certificate comes from a private authority, as a self-managed GitLab's often does, and Argo CD's repo server doesn't trust it.

Confirm

The Application's condition names the repository it couldn't read:

ComparisonError: Failed to load target state: failed to generate manifest for source 1 of 2: rpc error: code = Unknown desc = failed to list refs: Get "https://gitlab.example.com/megacorp/platform/charts.git/info/refs?service=git-upload-pack": tls: failed to verify certificate: x509: certificate signed by unknown authority
Fix

Give Argo CD the authority's certificate: in the argocd-tls-certs-cm ConfigMap, under the server's name, or with argocd cert add-tls. Restarting the repo server makes sure it has read it; then refresh the Application. See how it syncs.

Cards: wrong result#

Part XI · Decode and debug · Chapter 61 · 14 min read

Use these cards when everything passed, but the result is wrong: the wrong value, the wrong image, the wrong version, or nothing at all. These are the hardest problems, because no log line points at them. The most common causes come first. The last cards cover what happens after the pipeline.

Passed, but did the wrong thingVariables#
You change a variable in the YAML, and the job still uses the old value.
Cause

The same name is set in CI/CD settings, on the project, a group or the instance. A settings variable beats every value in YAML, even a job's own. MegaCorp's group sets MAVEN_CLI_OPTS this way.

Confirm

The job log never says where a value came from, and Full configuration shows only YAML. Look for the name in the project's CI/CD variables, then each group's.

Fix

Change or remove the settings variable, or use a name the settings don't set. See who wins.

Passed, but did the wrong thingVariables#
A variable set at the top of your .gitlab-ci.yml is ignored in some jobs, which use a template's value instead.
Cause

A job's own variables, including those it gets through extends, beat top-level ones. MegaCorp's .megacorp-base sets MC_TEAM: unknown, so every job that extends it reports unknown.

Confirm

In Full configuration, the job's own variables list the name.

Fix

Set the value on the job, or ask the template's owners to remove the job-level default. See the MC_TEAM exercise.

Passed, but did the wrong thingDeployments and delivery#
Production shows a bug that staging never had, although both deployed the same image tag.
Cause

A later pipeline built the same commit again and pushed the same tag. With mutable tags, the rebuild replaced the image that staging tested.

Confirm

ECR shows that the tag was pushed after the staging deploy, and the release pipeline contains an image build.

Fix

Build once, and promote the existing image. Make tags immutable, so that a rebuild fails loudly. See from build to deploy.

Passed, but did the wrong thingDeployments and delivery#
The deploy pipeline is green, but the old version is still running.
Cause

With GitOps, green means only that the new version was written to Git. The controller may not have synced yet, or the rollout failed in the cluster. A manual deploy job that is still waiting deploys nothing, either.

Confirm

Check the GitOps repository's latest commit, then the controller's last sync, then the cluster's rollout.

Fix

Wait for the sync or ask for one, and fix any failed rollout. See MegaCorp's handoff.

Passed, but did the wrong thingScanners and policies#
A failed quality or security gate didn't stop a deploy.
Cause

The gate's job is allowed to fail. Or the deploy's needs don't include it, so the deploy never waits for it.

Confirm

The gate job shows an orange warning, and the deploy job's needs name other jobs only.

Fix

Decide with the owners whether the gate should block. If it should, remove allow_failure and make the deploy wait for it. See the quality gate exercise.

Passed, but did the wrong thingScanners and policies#
Security scans ran, but the merge request shows no findings.
Cause

The merge request's Reports tab and the pipeline's Security tab need Ultimate. Also, GitLab's scanners run in the branch pipeline by default, not in the merge request pipeline.

Confirm

Check your tier, and which pipeline has the scanner jobs. The raw report is in the scanner job's artifacts.

Fix

Below Ultimate, read the report artifact. To scan in merge request pipelines, set AST_ENABLE_MR_PIPELINES: "true". See where the findings appear.

Passed, but did the wrong thingArtifacts, cache and data#
A variable that an earlier job wrote to a dotenv report is empty in a later job.
Cause

The later job doesn't receive that job's artifacts. Its needs or dependencies leave the job out, or it uses needs with artifacts: false. An empty dependencies: [] blocks them even when needs names the job.

Confirm

Read the later job's needs and dependencies in Full configuration, and check that the earlier job saved a dotenv report.

Fix

Add the earlier job to needs, and to dependencies too if the job has that list. See dotenv, and the container scan that gets no image.

Passed, but did the wrong thingArtifacts, cache and data#
A job finds files that shouldn't be there: stale ones, or another job's.
Cause

Two jobs share a cache key but cache different paths, so each overwrites the other's cache. Or a job caches and keeps the same path, and the cache is restored before the artifacts.

Confirm

Compare the jobs' cache keys and paths in Full configuration.

Fix

Give each set of paths its own key, and don't cache what you keep as an artifact. See cache.

Passed, but did the wrong thingGit in CI#
After a push of several commits, some changed services weren't built.
Cause

The script compares with HEAD~1, which is only the previous commit. Changes in the push's earlier commits are missed.

Confirm

Read the base the script uses in push pipelines.

Fix

For pushes, compare with CI_COMMIT_BEFORE_SHA, with a fallback for when it is all zeros. See the monorepo exercise.

Passed, but did the wrong thingDeployments and delivery#
A release reports a development version, such as 1.8.0-SNAPSHOT.
Cause

The build takes its version from the committed build file. Nothing passes the release tag into the build.

Confirm

Read the version in pom.xml or package.json, and the build command.

Fix

Pass CI_COMMIT_TAG into the build in tag pipelines, or commit the release version before tagging. See which version is it.

Passed, but did the wrong thingSecrets, tokens and identity#
AWS commands act as the wrong identity. For example, pushes are denied, although the job's role is allowed to push.
Cause

The login step didn't run, often because the job's own before_script replaced the template's. The AWS CLI then used the next credentials it found, such as the runner's own role.

Confirm

aws sts get-caller-identity prints the role really in use.

Fix

Restore the login step. See which credentials the AWS CLI uses.

Passed, but did the wrong thingSecrets, tokens and identity#
A job acts as an old user or account in AWS, although it has an ID token and a role.
Cause

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are still set in CI/CD settings, from before OIDC. Keys in environment variables come first, unless a login step overwrites them.

Confirm

Look for the two variables in the project's and groups' settings, and run aws sts get-caller-identity.

Fix

Remove the old keys, after checking that nothing else still needs them. See which credentials the AWS CLI uses.

Passed, but did the wrong thingArtifacts, cache and data#
A job passes, but whatever it built is nowhere to be found.
Cause

The job has no artifacts and no step that publishes its output, so the output is deleted with the job's environment. MegaCorp's publish-docs builds its site this way.

Confirm

The job has no artifacts: and no deploy command.

Fix

Keep the output as an artifact, or add a step that publishes it. See the publish-docs exercise.

Passed, but did the wrong thingYAML itself#
A version such as 1.10 turns into 1.1, and a rule comparing it never matches.
Cause

YAML reads an unquoted 1.10 as a number, and the number loses its trailing zero.

Confirm

The value is unquoted in the YAML. On the local GitLab, VERSION: 1.10 reached a rule as 1.1.

Fix

Quote it: VERSION: "1.10". See YAML in five minutes.

Passed, but did the wrong thingSecrets, tokens and identity#
A scanner has checked only the main branch for months, and nobody noticed.
Cause

Its token is protected, so merge request pipelines on feature branches get an empty token, and the scan can't sign in. The job is allowed to fail, so each failure is only a warning.

Confirm

The scan job shows warnings in merge request pipelines, and the token is protected in the settings.

Fix

Decide on a token for merge requests, and make the failures visible. See the quality gate that never ran.

Passed, but did the wrong thingRunners and execution#
The same job behaves differently depending on which runner takes it.
Cause

Runners differ in their executor and their configuration: extra environment variables, a pre_build_script, resource limits. No project can see any of it.

Confirm

The job log names the runner. Compare the runners' settings with the platform team.

Fix

Pin the job to the right runners with tags, or make the runners consistent. See what the runner's configuration decides.

Passed, but did the wrong thingScanners and policies#
The container scan runs in every merge request pipeline, yet it never scans the image that the pipeline built.
Cause

GitLab's container scanning template gives the job dependencies: []. An empty list blocks every artifact download, dotenv reports included, and adding needs doesn't lift it. So a value such as IMAGE_REF, from the build job's dotenv report, is empty in the scan, and so is the CS_IMAGE built from it.

Confirm

In Full configuration, the merged container_scanning job has both dependencies: [] and needs. On the local GitLab 19.3, a job with both keys printed an empty value.

Fix

Add dependencies: [image-build] to your version of the job, naming the job that writes the report. See changing a template's job.

The cards from here on are for what happens after the pipeline: the environment repository, the controllers, the cluster and the clouds.

Passed, but did the wrong thingCharts, values and overlays#
An overlay's new image tag has no effect: the Deployment keeps the base's image.
Cause

An images: entry changes only images whose name matches its name exactly. A misspelt name matches nothing, and Kustomize says nothing.

Confirm

kustomize build on the overlay still shows the base's image, such as …/node-service:latest.

Fix

Copy the image's name from the base. See what Kustomize does silently.

Passed, but did the wrong thingCharts, values and overlays#
A change to a service's values file has no effect on one cluster.
Cause

Something with higher precedence sets the same value there. In Argo CD, parameters beat inline valuesObject, which beats value files. MegaCorp's aks-prod ApplicationSet sets the image's repository inline, for example.

Confirm

Read the Application's sources, and look for valuesObject, values or parameters that set the same key.

Fix

Change the value where it wins, or remove the inline one. See values.

Passed, but did the wrong thingGitOps sync and drift#
A change made with kubectl disappears within seconds.
Cause

Argo CD's automated sync, with selfHeal: true, puts back what Git says a few seconds after the cluster drifts. Flux corrects drift at its next reconciliation too.

Confirm

The Application's syncPolicy, and a sync in its history just after your change.

Fix

Change the environment repository instead. See drift, and going back.

Passed, but did the wrong thingGitOps sync and drift#
A resource removed from the environment repository is still running.
Cause

Deleting a file from Git deletes its objects only when the controller prunes. Argo CD prunes only with prune: true, which is off by default, and a Flux Kustomization only with prune: true.

Confirm

The Application's or the Kustomization's settings. Argo CD also marks the leftover object as needing to be pruned.

Fix

Turn pruning on, or delete the object by hand once. See drift, and going back.

Passed, but did the wrong thingPods and Kubernetes objects#
A changed setting, or a rotated secret, doesn't reach the running pods.
Cause

Pods read ConfigMaps and Secrets as environment variables when they start, and never again. A rotated secret also waits for the External Secrets Operator's next refresh, hourly at MegaCorp.

Confirm

The Secret or ConfigMap holds the new value, and the pods started before it changed.

Fix

Restart the pods, with a new deploy or Argo CD's restart action. See config for each environment.

Passed, but did the wrong thingCharts, values and overlays#
helm list shows no release for a service that runs in the cluster.
Cause

Argo CD renders charts with helm template and applies the objects itself, so Helm never records a release. Only charts installed by Helm, or by Flux's HelmRelease, appear.

Confirm

The service has an Application in Argo CD.

Fix

Nothing is broken. Read the release in Argo CD instead. See releases.

Cards: stuck or drifting#

Part XI · Decode and debug · Chapter 62 · 12 min read

Use these cards when jobs wait for ever, run slowly, or break although nobody changed anything. Drift has no commit to blame, so look outside the repository: runners, tokens, included files, images and policies. The most common causes come first. The last cards cover what happens after the pipeline.

Stuck, slow, or broke without a commitRunners and execution#
A job stays pending, and its page says it is stuck because no runner matches all of its tags.
Cause

No online runner has every tag the job lists. A job's own tags replace the default tags completely.

Confirm

The page says:

This job is stuck because of one of the following problems. There are no active runners online, no runners for the protected branch, or no runners that match all of the job's tags:

The job's tags are listed after the message.

Fix

Fix the job's tags or the runner's. See how a job finds a runner.

Stuck, slow, or broke without a commitRunners and execution#
Every job in a project waits in pending.
Cause

The project's runners are offline: none has contacted GitLab for more than two hours.

Confirm

The job page says:

This job is stuck because the project doesn't have any runners online assigned to it.

Settings › CI/CD › Runners shows their status.

Fix

The runners' owners restart or replace them. See which runners a project can use.

Stuck, slow, or broke without a commitRunners and execution#
Jobs run on main, but stay pending on feature branches.
Cause

The only runners that fit are protected. Protected runners take jobs only from protected branches and tags.

Confirm

The runners' settings show Protected.

Fix

Provide an unprotected runner for other branches, or protect the branch. See how a job finds a runner.

Stuck, slow, or broke without a commitRunners and execution#
Jobs wait a long time in pending at busy times, then run normally.
Cause

The runners are already running as many jobs as they allow. Instance runners also share a fair-usage queue between projects.

Confirm

Many jobs are pending at once. The platform team can confirm the runners' concurrent and limit settings.

Fix

Add runner capacity, or run fewer jobs in parallel. See what the runner's configuration decides.

Stuck, slow, or broke without a commitTime bombs and drift#
A job that has worked for months suddenly fails with an authentication error, and nobody committed anything.
Cause

A stored access token expired, at midnight UTC on its expiry date. Unless someone chose a date, that is 365 days after it was created.

Confirm

The token's page lists it as expired.

Fix

Create a new token, update the variable, and track the next expiry date. See stored tokens.

Stuck, slow, or broke without a commitTime bombs and drift#
The pipeline changed overnight, but your project's history shows no change.
Cause

Your configuration includes a file from another project at a branch, or at a tag that someone moved. When that file changes, so does your pipeline.

Confirm

Check the include's ref in Full configuration, then the other project's recent commits.

Fix

Pin includes to tags, and treat tags as never moving. See refs in includes.

Stuck, slow, or broke without a commitTime bombs and drift#
A job's tools behave differently from one day to the next, with no change in the project.
Cause

The job's image tag, such as node:22, now points at a newer image. Anyone who can push to that repository can move a tag.

Confirm

Compare the image's digest in the registry with the digest it had before.

Fix

Name the image by digest, as name@sha256:…, or use tags that their owners never overwrite. See the job's image.

Stuck, slow, or broke without a commitTime bombs and drift#
A long job's AWS calls start failing part-way through.
Cause

The temporary keys last one hour, because MegaCorp asks for 3600 seconds. Chained roles are limited to one hour, whatever you ask for.

Confirm

The failures start more than an hour after the login step.

Fix

Log in again before the late steps, or split the job. See small things that bite.

Stuck, slow, or broke without a commitDeployments and delivery#
A deploy waits and never starts.
Cause

The environment needs approvals, or the job is manual. An approval doesn't start the job.

Confirm

Operate › Environments shows the deployment waiting, and the job offers Run.

Fix

Approve it, then run it. See who may deploy, and approvals.

Stuck, slow, or broke without a commitDeployments and delivery#
A deploy job waits, although nothing else seems to be running in its pipeline.
Cause

Another job in the same resource_group is running, in another pipeline. Only one job per resource group runs at a time.

Confirm

The job is waiting for its resource. Look for the same deploy in other pipelines.

Fix

Let the other job finish, or find what is holding the resource. See one at a time.

Stuck, slow, or broke without a commitTime bombs and drift#
New jobs appear in every pipeline overnight, with names like secret-detection-1, or in .pipeline-policy stages.
Cause

The security team added or changed a security policy.

Confirm

Secure › Policies lists the policies that apply.

Fix

Nothing in your YAML can remove them. Talk to the security team. See three kinds of policy.

Stuck, slow, or broke without a commitArtifacts, cache and data#
Jobs are slow, because they download everything every time.
Cause

The cache is never found. The runners may have no shared cache storage, the key may keep changing, or protected and unprotected branches may have separate caches.

Confirm

The cache lines in the job log name the key it tried, and whether it was found.

Fix

Work through the cache flow. See cache.

Stuck, slow, or broke without a commitTime bombs and drift#
Pulls from Docker Hub fail at busy times, and work when the job is retried.
Cause

Docker Hub limits how many pulls it allows, counting each request for an image's manifest.

Confirm

The failing images all come from Docker Hub.

Fix

Pull through GitLab's dependency proxy, or keep copies in your own registry. See private registries.

Stuck, slow, or broke without a commitDeployments and delivery#
Deployments are blocked for a period, and the pipeline hasn't changed.
Cause

A deploy freeze is set for the project, and GitLab blocks deployments during it.

Confirm

The environment's deployments list shows the next freeze.

Fix

Wait for the freeze to end, or ask whoever set it. See one at a time, newest wins.

Stuck, slow, or broke without a commitDeployments and delivery#
Deploys appear several minutes after the pipeline goes green.
Cause

The GitOps controller syncs on a timer. GitLab's agent can trigger an immediate sync only for its own configuration project and for public projects.

Confirm

Compare the GitOps commit's time with the controller's last sync.

Fix

Shorten the controller's sync interval, or accept the delay. See GitLab's agent for Kubernetes.

The cards from here on are for what happens after the pipeline: the environment repository, the controllers, the cluster and the clouds.

Stuck, slow, or broke without a commitGitOps sync and drift#
Argo CD shows an Application as OutOfSync, and it stays that way.
Cause

The Application has no automated sync, so it waits for someone to sync it. With automated sync, Argo CD checks Git about every two minutes, and syncs whatever is OutOfSync; a sync that fails shows as failed, not as waiting.

Confirm

The Application's syncPolicy has no automated block, and its last sync is older than the commit.

Fix

Sync it, or ask for automated sync. See drift, and going back.

Stuck, slow, or broke without a commitPods and Kubernetes objects#
A rollout stops half-way, with some pods new and some old. After ten minutes, the Deployment reports ProgressDeadlineExceeded.
Cause

The new pods never become ready, so Kubernetes won't replace more old ones. It doesn't roll back by itself.

Confirm

kubectl rollout status ends with this, and exits with 1:

error: deployment "payments-api" exceeded its progress deadline
Fix

Find out why the new pods aren't ready, then revert or fix forward. See rolling, the default.

Stuck, slow, or broke without a commitPods and Kubernetes objects#
Pods are Running but show 0/1 ready, and the rollout waits.
Cause

The readiness probe keeps failing. MegaCorp's chart probes /actuator/health unless a service's values say otherwise, and a service that doesn't serve that path is never ready.

Confirm

The pod's events show failed readiness probes, with the path and the status code.

Fix

Set probes.path in the service's values. See rolling, the default.

Stuck, slow, or broke without a commitCharts, values and overlays#
kustomize build prints warnings that commonLabels or patchesStrategicMerge is deprecated.
Cause

The overlay uses older fields. Kustomize 5.8 still builds them, and asks you to move to the new ones.

Confirm
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
Fix

Run kustomize edit fix, or change to labels and patches by hand, and check that the build's output hasn't changed. See patches.

Stuck, slow, or broke without a commitInfrastructure state and plans#
A plan wants to change infrastructure that nobody edited.
Cause

Something the configuration relies on has changed: a module's ?ref= tag was moved to another commit, or someone changed the resources by hand, outside OpenTofu.

Confirm

The plan's changes, the module tag's history in its repository, and the cloud's activity log.

Fix

Protect the module repositories' tags so they can't move, and bring a hand-made change into code or undo it. See modules.

Stuck, slow, or broke without a commitGitOps sync and drift#
A merged change in one Flux Kustomization's folder never reaches the cluster.
Cause

The Kustomization depends on another through dependsOn, and that one isn't ready, often because one of its objects is failing its health check. Flux doesn't apply a Kustomization until everything it depends on is ready.

Confirm

flux get kustomizations shows the waiting one's message, such as:

dependency 'flux-system/add-ons' is not ready

The dependency's own row, or flux get helmreleases -A, shows what is failing there.

Fix

Fix the dependency; the waiting Kustomization follows by itself. See add-ons, then settings, then applications.

The final test#

Part XI · Decode and debug · Chapter 63 · 2 min read

One file, ten mistakes. The file is payments-api's pipeline after a week of well-meant changes. Each mistake is one this book has explained, and each has a symptom card. Read the scenario, find all ten, and for each one say what will go wrong and which card describes it. Then open the answer key. Next, review a change to MegaCorp's delivery configuration, with five more mistakes. Last, write one file of your own from a short brief, and check it the way its answer shows.

The scenario#

It's Friday afternoon. payments-api's team asks you to review their pipeline changes before they merge them. They tell you what they meant to do:

  • Integration tests must run on every merge request, against a PostgreSQL service. The database password is in the project's CI/CD settings, as POSTGRES_PASSWORD.
  • changed-files should list the files a merge request changes, for the reviewers.
  • smoke-test checks the image that the pipeline has just built.
  • The API contract version is 1.10.
  • Everything else should behave exactly as before.

MegaCorp's templates, runners and settings are the ones described in this book.

Spot the bugTen mistakes in one file#
final test · broken on purposepayments-api/.gitlab-ci.yml
# final-test/payments-api/.gitlab-ci.yml
# The final test: payments-api's pipeline file after a busy week of changes. It contains
# exactly ten mistakes, planted on purpose. Used by the chapter "The final test". GitLab
# never runs it.

include:
  - project: megacorp/devops/ci-templates
    ref: main
    file: pipelines/java-service.yml
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api

variables:
  JAVA_VERSION: "21"
  MC_TEAM: payments
  API_VERSION: 1.10

maven-test:
  variables:
    MC_TEAM: $MC_TEAM
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

image-build:
  before_script:
    - echo "Building $CI_PROJECT_NAME for $MC_TEAM"

integration-test:
  stage: test
  tags: [docker]
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk21
  services:
    - name: postgres:16
      alias: db
  script:
    - mvn verify -Pintegration -Dapi.version=$API_VERSION
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

changed-files:
  stage: test
  script:
    - git diff --name-only origin/main...HEAD > changed.txt
  artifacts:
    paths: [changed.txt]
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

smoke-test:
  stage: deploy
  needs: [maven-build]
  script:
    - ./smoke.sh "$IMAGE_REF"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
QuestionFind the ten mistakes. For each, say what will go wrong, and which symptom card describes it.
Show the answer
#The mistakeWhat goes wrongCard
1ref: main on the central includethe pipeline changes whenever ci-templates' main changes, with no commit herea moving include
2API_VERSION: 1.10, unquotedYAML reads a number, so the tests get 1.11.10 becomes 1.1
3MC_TEAM: $MC_TEAM in maven-testthe job gets the literal text $MC_TEAMa literal variable
4maven-test's own rulesthey replace the template's list: the job leaves merge request pipelines, and now runs in schedulesyour rules replaced the template's
5image-build's own before_scriptthe functions and the AWS login are gone, so the build fails at mc_retrybefore_script replaced
6tags: [docker] on integration-testno MegaCorp runner has that tag, so the job waits for everstuck on tags
7the database password only in settingssettings variables never reach services, so PostgreSQL doesn't starta service without its variables
8integration-test runs only on mainit never runs in merge request pipelines, as it was meant tono merge request rule
9git diff … origin/main in a merge request pipelinethat pipeline fetches only its own commit, so origin/main doesn't existshallow history
10smoke-test needs maven-build, not image-buildit doesn't receive IMAGE_REF, so it tests nothingan empty dotenv variable

Scoring: seven or more, and you're ready for real incidents: you can decode pipelines you have never seen. Five or six: reread the cards you missed, then try again tomorrow. Fewer than five: work through chapter 55 and the four card chapters once more. Every mistake here is in real pipelines somewhere, usually more than one at a time.

After the pipeline: a change that never reached aks-prod#

It's Monday. payments-api 1c7d4b09 went live on eks-prod last week, but production on aks-prod still runs the old version. A colleague's merge request to gitops-config says it fixes that. It changes two files. You're asked to review them before anyone merges. MegaCorp's chart, controllers and clusters are the ones that Part X describes.

Spot the bugFive mistakes on the way to aks-prod#
final test · broken on purposeplatform/gitops-config/apps/aks-prod/applicationset.yaml
# final-test/platform/gitops-config/apps/aks-prod/applicationset.yaml
# The final test's delivery part: aks-prod's ApplicationSet as a colleague's merge request
# would leave it. Together with envs/production/payments-api.yaml, it holds exactly five
# mistakes, planted on purpose. Used by the chapter "The final test". No cluster reads it.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payments-api
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - list:
        elements:
          - env: staging
          - env: production
  template:
    metadata:
      name: "payments-api-{{ .env }}"
      annotations:
        notifications.argoproj.io/subscribe.on-deployed.grafana: "payments-api|{{ .env }}"
    spec:
      project: payments
      sources:
        - repoURL: https://gitlab.example.com/megacorp/platform/charts.git
          targetRevision: service-3.4.0
          path: service
          helm:
            releaseName: payments-api
            valueFiles:
              - "$values/envs/{{ .env }}/payments-api.yaml"
            valuesObject:
              image:
                repository: megacorpprod.azurecr.io/payment-api
              serviceAccount:
                annotations:
                  azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
              podLabels: {}
        - repoURL: https://gitlab.example.com/megacorp/platform/gitops-config.git
          targetRevision: main
          ref: values
      destination:
        server: https://kubernetes.default.svc
        namespace: "payments-{{ .env }}"
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
final test · broken on purposeplatform/gitops-config/envs/production/payments-api.yaml
# final-test/platform/gitops-config/envs/production/payments-api.yaml
# The final test's delivery part: payments-api's production values as a colleague's merge
# request would leave them. Together with apps/aks-prod/applicationset.yaml, they hold
# exactly five mistakes, planted on purpose. Used by the chapter "The final test".
image:
  repository: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api
  tags: 1c7d4b09
replicaCount: 3
probes:
  path: /healthz
env:
  SPRING_PROFILES_ACTIVE: production
  PAYMENTS_PROVIDER_URL: https://pay.example.com
secrets:
  store: cloud-secrets
  keys:
    - name: DB_PASSWORD
      remote: payments-api/production/db-password
ingress:
  enabled: true
  host: payments.example.com
QuestionFind the five mistakes. For each, say what will go wrong, and which symptom card describes it.
Show the answer
#The mistakeWhat goes wrongCard
1no automated: under the ApplicationSet's syncPolicyaks-prod's Applications never sync by themselves, so every merged change waits for someone to press SyncOutOfSync, and it stays that way
2megacorpprod.azurecr.io/payment-api in the inline valuesthat repository doesn't exist, so the new pods can't pull their imagean image that can't be pulled
3podLabels: {}, without azure.workload.identity/use: "true"the pods get no Azure identity, although their service account names onea pod without its Azure identity
4tags: instead of tag: under imagethe chart finds no image.tag and refuses to render, on eks-prod as well as aks-prodthe chart needs a tag
5probes: path: /healthzpayments-api answers health checks on /actuator/health, so the new pods never become ready, and the rollout stalls0/1 ready

The mistakes hide each other. Mistake 1 stops anything from syncing, and mistake 4 stops the chart from rendering, so mistakes 2, 3 and 5 would appear one at a time, each after the one before was fixed. A review catches all five at once. That's why deploy changes go through merge requests (promotion by merge request).

Scoring: four or five, and you can follow a change from Git to a running pod. Fewer: work through chapter 55 and the delivery cards again.

Then write one#

Finding mistakes is half of owning a pipeline. The other half is writing rules that do what your team means. This task uses only what Part III and Part V teach.

Spot the buginvoice-api's rules#

MegaCorp's billing team is starting a new Java service, invoice-api. Its pipeline has five jobs: build, unit-tests, deploy-staging, deploy-production and nightly-report. The team asks you for a .gitlab-ci.yml whose rules do this:

  • A push creates one pipeline: a merge request pipeline while a merge request is open, and a branch pipeline otherwise.
  • build and unit-tests run for every push to main and for every release tag, such as v1.2.3. In a merge request or on another branch, they run only when src/ or pom.xml changed.
  • deploy-staging runs after every push to main.
  • deploy-production runs only in a release tag's pipeline, and waits for a person to start it.
  • nightly-report runs only in scheduled pipelines, and nothing else runs in them.
QuestionWrite the workflow and each job's rules. Then fill in a table: which jobs does each kind of pipeline get?
Show the answer

Here is one answer that meets the brief. Yours may differ and still be right: what matters is that its table matches.

practice · authoring-taskmodel-answer.ymlworkflow
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH
    - if: $CI_COMMIT_TAG
practice · authoring-taskmodel-answer.ymlbuild-and-test
build:
  stage: build
  script:
    - mvn --batch-mode --no-transfer-progress package -DskipTests
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
    - if: $CI_COMMIT_TAG
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - changes:
        - src/**/*
        - pom.xml

unit-tests:
  stage: test
  script:
    - mvn --batch-mode --no-transfer-progress test
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
    - if: $CI_COMMIT_TAG
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - changes:
        - src/**/*
        - pom.xml
practice · authoring-taskmodel-answer.ymldeploys
deploy-staging:
  stage: deploy
  image: alpine:3.22
  script:
    - echo "Deploying $CI_COMMIT_SHORT_SHA to staging"
  environment:
    name: staging
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE == "push"

deploy-production:
  stage: deploy
  image: alpine:3.22
  script:
    - echo "Deploying $CI_COMMIT_TAG to production"
  environment:
    name: production
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
      when: manual
practice · authoring-taskmodel-answer.ymlnightly
nightly-report:
  stage: test
  image: alpine:3.22
  script:
    - echo "The nightly report for $CI_COMMIT_BRANCH"
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

Three details trip people up:

  • The workflow's last rule, $CI_COMMIT_TAG. Without it, a tag creates no pipeline at all (a tag with no pipeline).
  • deploy-staging also checks $CI_PIPELINE_SOURCE == "push", because a scheduled pipeline on main has $CI_COMMIT_BRANCH set to main too (what each kind of pipeline sets).
  • A manual job added by a rule holds the tag's pipeline at blocked until someone runs it. That is what "waits for a person" means here (a play button that waits).
jobfeature branch, codemerge request, codemerge request, docs onlymaintag v1.2.3schedule
buildrunsrunsnot in the pipelinerunsrunsnot in the pipeline
unit-testsrunsrunsnot in the pipelinerunsrunsnot in the pipeline
deploy-stagingnot in the pipelinenot in the pipelinenot in the pipelinerunsnot in the pipelinenot in the pipeline
deploy-productionnot in the pipelinenot in the pipelinenot in the pipelinenot in the pipelinemanual: waits for someone to run itnot in the pipeline
nightly-reportnot in the pipelinenot in the pipelinenot in the pipelinenot in the pipelinenot in the pipelineruns
invoice-api: which jobs each kind of pipeline got, on the local GitLab✓runs·not in the pipeline▶manual: waits for someone to run it

In the docs-only merge request no job matches, so GitLab creates no pipeline at all.

Checking your own answer. Paste it into Build › Pipeline editor in a project of yours, and validate it with main selected: the jobs shown should match the main column. The Validate tab simulates only a push, so check the other columns for real. Push a tag, open a merge request with a code change, open another whose only change is to the docs, and run a schedule once. The docs-only branch's first push still gets a branch pipeline with build and unit-tests, because changes is true for a new branch; the column is about its merge request. If every column matches, you're ready to own a team's pipeline.

Keyword index#

Appendix · Appendices · Chapter 64 · 8 min read

Every .gitlab-ci.yml keyword in GitLab 19.3, what it does, and where this book explains it. Keywords written as parent:child are settings inside the parent keyword. A dash means the book doesn't cover that keyword.

Global keywords#

KeywordWhat it doesCovered in
defaultsettings every job gets unless it sets its own: image, before_script, tags and otherschapter 14
includecopies other YAML files into this configurationchapter 15
include:locala file from the same repository and branchchapter 15
include:projecta file from another project, at a refchapter 15
include:remotea file from any URLchapter 15
include:templatea template that ships with GitLabchapter 15
include:componenta versioned CI/CD componentchapter 16
include:inputsvalues for an included file's inputschapter 16
include:rulesincludes the file only when the rules matchchapter 15
include:integrity, include:cachecheck or cache a remote file–
stagesthe stages, in orderchapter 24
workflowwhether a pipeline is created, and its settingschapter 22
workflow:rulesdecide whether the pipeline exists at allchapter 22
workflow:namethe pipeline's namechapter 22
workflow:rules:variablesvariables set when a workflow rule matcheschapter 22
workflow:auto_cancel, :on_new_commit, :on_job_failurewhich jobs to cancel when a newer commit starts a pipeline, or when a job failschapter 22

Header keywords#

KeywordWhat it doesCovered in
specthe header of a file, before its --- linechapter 16
spec:inputs, :default, :type, :options, :regex, :rules, :descriptiontyped parameters for a component, an included file or a pipeline, with defaults, allowed values and checkschapter 16
spec:include, spec:component, spec:descriptionshared input definitions, component context, and a descriptionchapter 16

Job keywords#

KeywordWhat it doesCovered in
after_scriptcommands that run last, in a separate shellchapter 29
allow_failurelets the pipeline continue if the job failschapter 24
artifactsfiles kept after the job, for later jobs and peoplechapter 31
artifacts:paths, :exclude, :name, :untrackedwhich files to keepchapter 31
artifacts:when, :expire_inwhen to keep them, and for how longchapter 31
artifacts:reportsfiles GitLab reads, such as test results and security reportschapter 31
artifacts:access, :public, :expose_aswho can download them, and links in merge requestschapter 31
before_scriptcommands that run before script, in the same shellchapter 29
cachefiles reused between jobs and pipelines, to save timechapter 31
cache:key, :paths, :policy, :when, :fallback_keys, :unprotect, :untrackedwhich cache, what goes in it, and when it is savedchapter 31
cache:key:files, :files_commits, :prefixa key that changes when the files' content changes, or when the files get a new commit, with an optional prefixchapter 31
coveragea regular expression that finds the coverage figure in the logchapter 37
dast_configurationDAST site and scanner profiles–
dependencieswhich earlier jobs' artifacts to downloadchapter 31
environmentwhere the job deploys, which makes it a deployment jobchapter 40
environment:name, :url, :on_stop, :action, :auto_stop_in, :deployment_tier, :kubernetesthe environment's name, address, stopping and tierchapter 40
extendscopies settings from other jobs, usually hidden oneschapter 14
hooks:pre_get_sources_scriptcommands on the runner before the Git fetchchapter 29
identityidentity federation with a cloud provider, in beta–
id_tokenssigned ID tokens for other systems, such as AWS and Azurechapter 28
imagethe container image the job runs inchapter 30
image:name, :entrypoint, :pull_policy, :docker, :kubernetesthe image's name, entrypoint, pull policy and executor optionschapter 30
inputs, :type, :regex and otherstyped inputs for one job, which can be changed when the job is run by hand or retried–
inheritwhich default: settings and top-level variables the job takeschapter 14
inherit:default, inherit:variablestake all, none, or a listchapter 14
interruptiblelets a newer pipeline cancel this jobchapter 24
needsstarts the job as soon as named jobs finish, ignoring stageschapter 24
needs:optionala need that may be missing from the pipelinechapter 24
needs:artifactswhether to download the needed job's artifactschapter 31
needs:project, needs:pipeline, needs:pipeline:jobartifacts or status from other pipelineschapter 17
needs:parallel:matrixneeds one variant of a matrix jobchapter 14
pagesa GitLab Pages publishing job–
parallelruns several copies of the jobchapter 14
parallel:matrixone copy for each combination of variable valueschapter 14
release, :tag_name, :tag_message, :name, :description, :ref, :milestones, :released_at, :assets:linkscreates a release, and describes it–
resource_grouponly one job in the group runs at a timechapter 24
retryretries the job on failurechapter 24
retry:max, :when, :exit_codeshow often, and for which failureschapter 24
rulesdecides whether the job is in the pipeline, and how it runschapter 23
rules:ifa condition on variableschapter 23
rules:changes, rules:existsconditions on changed or existing fileschapter 23
rules:changes:paths, :compare_to, :regexp, and rules:exists:regexpthe files to check, the ref to compare against, or a Ruby regular expression in place of glob patternschapter 23
rules:when, :allow_failure, :needs, :variables, :interruptiblewhat a matching rule changeschapter 23
runa sequence of GitLab Functions stepschapter 19
scriptthe commands the job runschapter 29
secretssecrets fetched from a secrets managerchapter 27
secrets:vault, :aws_secrets_manager, :gcp_secret_manager, :azure_key_vault, :gitlab_secrets_managerthe provider and the secretchapter 27, and for AWS and Azure chapter 38
secrets:file, secrets:tokenfile or value, and which ID token to usechapter 27
servicesextra containers beside the job, such as a databasechapter 30
services:name, :alias, :entrypoint, :command, :variables, :pull_policythe service's image, hostname, start command and settingschapter 30
stagethe stage the job belongs tochapter 4
tagswhich runners may take the jobchapter 29
timeoutthe job's own time limitchapter 29
triggerstarts a downstream pipelinechapter 17
trigger:includea child pipeline from a file in this projectchapter 17
trigger:projecta pipeline in another projectchapter 17
trigger:strategymakes the trigger job wait for, and mirror, the downstream pipelinechapter 17
trigger:forward, trigger:inputswhat the downstream pipeline receiveschapter 17
whenwhen the job runs: on_success, manual, delayed, always, neverchapter 24
start_inthe delay for when: delayedchapter 24
manual_confirmationa message to confirm before running a manual jobchapter 24
variablesvariables for every job, or for one jobchapter 26
variables:value, :description, :optionsdefaults and choices shown when someone runs a pipeline by handchapter 26
variables:expandwhether $ references in the value are expandedchapter 26

Deprecated keywords#

KeywordUse insteadCovered in
only, exceptruleschapter 23
image, services, cache, before_script, after_script at the top levelthe same keywords inside default:chapter 14
publish, and a Pages job named pagesthe pages keyword–

Predefined variables#

Appendix · Appendices · Chapter 65 · 4 min read

GitLab sets these variables in every pipeline, and each one exists only from a certain moment. This appendix lists the variables you will meet most, when each exists, and which kinds of pipeline set the ones that differ. GitLab's documentation has the full list.

When a variable exists#

PhaseUsable inExamples
Pre-pipelineeverything, including include:rulesCI_COMMIT_*, CI_PIPELINE_SOURCE, CI_DEFAULT_BRANCH, CI_OPEN_MERGE_REQUESTS, CI_MERGE_REQUEST_*, CI_PROJECT_ID, CI_SERVER_HOST
Pipelinejob rules and scripts, but not include:rulesCI_PIPELINE_IID, CI_JOB_NAME, CI_JOB_STAGE, CI_NODE_INDEX, CI_ENVIRONMENT_NAME, GITLAB_USER_LOGIN
Job-onlyscripts only: not workflow, include, rules or trigger jobsCI_PIPELINE_ID, CI_PIPELINE_URL, CI_JOB_ID, CI_JOB_TOKEN, CI_PROJECT_DIR, CI_REGISTRY_PASSWORD

Which pipelines set them#

VariableBranchTagMerge requestScheduled
CI_COMMIT_BRANCHyes––yes
CI_COMMIT_TAG–yes–only if the schedule runs on a tag
CI_PIPELINE_SOURCE is pushyesyes––
CI_PIPELINE_SOURCE is merge_request_event––yes–
CI_PIPELINE_SOURCE is schedule–––yes
CI_MERGE_REQUEST_*––yes–
CI_OPEN_MERGE_REQUESTSif the branch has an open merge request–yes–

CI_COMMIT_BEFORE_SHA is all zeros in merge request pipelines, scheduled pipelines, manual runs, and the first pipeline of a branch or tag.

The variables you meet most#

VariableHoldsPhase
CI_PIPELINE_SOURCEwhat started the pipeline: push, merge_request_event, schedule, web, api, trigger, pipeline, parent_pipeline, and otherspre-pipeline
CI_COMMIT_BRANCHthe branch, in branch pipelines onlypre-pipeline
CI_COMMIT_TAGthe tag, in tag pipelinespre-pipeline
CI_COMMIT_REF_NAMEthe branch or tag being builtpre-pipeline
CI_COMMIT_REF_SLUGthe ref name in lower case, at most 63 bytes, with anything but 0-9 and a-z replaced by -, for URLs and host namespre-pipeline
CI_COMMIT_REF_PROTECTEDtrue for a protected branch or tagpre-pipeline
CI_COMMIT_SHA, CI_COMMIT_SHORT_SHAthe commit, and its first eight characterspre-pipeline
CI_COMMIT_BEFORE_SHAthe branch's latest commit before this pushpre-pipeline
CI_DEFAULT_BRANCHthe project's default branchpre-pipeline
CI_OPEN_MERGE_REQUESTSup to four open merge requests from this branchpre-pipeline
CI_MERGE_REQUEST_IID, …_SOURCE_BRANCH_NAME, …_TARGET_BRANCH_NAME, …_DIFF_BASE_SHA, …_EVENT_TYPEthe merge request, in merge request pipelinespre-pipeline
CI_PROJECT_ID, CI_PROJECT_PATH, CI_PROJECT_NAMEthe projectpre-pipeline
CI_SERVER_HOST, CI_SERVER_FQDN, CI_API_V4_URLthe GitLab instance, and its APIpre-pipeline
CI_REGISTRY, CI_REGISTRY_IMAGEthe container registry, and the project's image path in itpre-pipeline
CI_PIPELINE_IIDthe pipeline's number within the projectpipeline
CI_JOB_NAME, CI_ENVIRONMENT_NAMEthe job's name, and its environmentpipeline
GITLAB_USER_LOGINwho started the pipeline, or the manual jobpipeline
CI_PIPELINE_ID, CI_JOB_IDinstance-wide IDs of the pipeline and the jobjob-only
CI_PROJECT_DIRwhere the repository is cloned, and where the job runsjob-only
CI_JOB_TOKENthe job's token, valid only while the job runsjob-only
CI_REGISTRY_USER, CI_REGISTRY_PASSWORDcredentials for the project's registryjob-only

Where variables can't be used#

  • rules:if can't use CI_ENVIRONMENT_SLUG, or variables GitLab stores with the job: CI_PIPELINE_ID, CI_PIPELINE_URL, CI_JOB_ID, CI_JOB_TOKEN, CI_JOB_URL, CI_REGISTRY_USER, CI_REGISTRY_PASSWORD and others like them.
  • include can use settings variables, CI_PROJECT_*, CI_PIPELINE_SOURCE, CI_COMMIT_REF_NAME, and variables given when the pipeline starts. It can't use variables defined in the YAML.
  • No rule can use a value from a dotenv report (chapter 21).

Rules syntax#

Appendix · Appendices · Chapter 66 · 4 min read

A reference for rules: and its conditions in GitLab 19.3: how rules are evaluated, how to write if: expressions, and what changes: and exists: really check. Every line comes from GitLab's documentation, or was captured on a local 19.3 instance.

How rules are evaluated#

  • Rules are evaluated when the pipeline is created, before any job runs.
  • They are checked in order, and the first rule that matches decides. If none matches, the job isn't added.
  • Every rule needs if, changes, exists or when, and every clause in one rule must hold.
  • A matching rule without when uses the job's when, which defaults to on_success. A when in the rule overrides the job's.
  • A rule can also set allow_failure, needs, which replaces the job's list, variables and interruptible.
  • A rule with when but no if brings the warning Job may allow multiple pipelines to run for a single action, in CI Lint and on manual runs.
  • GitLab's documentation says jobs with no rules "default to except: merge_requests": they don't join merge request pipelines.

if expressions#

WriteMeansNotes
$VAR == "value"equalthe variable on the left; strings quoted, variables not
$VAR != "value"not equal
$VARset, and not empty
$VAR == nullnot set
$VAR == ""set, but empty
$VAR =~ /pattern/matches the regular expressionRE2 syntax, case-sensitive, /i to ignore case; a pattern without anchors matches anywhere in the value
$VAR !~ /pattern/doesn't match
a && b, a || band, or&& binds before ||; brackets group
!$VARempty, or not setsince 18.11; so !"false" is false

Traps, each confirmed in the documentation or on a local 19.3 instance:

  • ${VAR} and "$VAR" don't work in if. A quoted variable breaks the YAML itself: (): did not find expected key while parsing a block mapping at line 5 column 7.
  • An unquoted !( … ) around a comparison is rejected with jobs:job:rules:rule if invalid expression syntax. Use !=.
  • A one-character regular expression, /./, is invalid.
  • A variable on the right of =~ is used as the regular expression, so its value must include the slashes. Variables inside a /…/ pattern aren't expanded.
  • A right side that isn't a /…/ pattern turns into a substring test, the other way round. $CI_COMMIT_BRANCH =~ "xmainx" was true for main.
  • A variable whose value mentions another variable isn't expanded in if.
  • An unquoted YAML number changes: 1.10 arrives as 1.1.

changes and exists#

ClauseChecksTraps
changesin merge request pipelines, files changed against the target branch; in branch pipelines, against the previous commitalways true for new branches, tags, and pipelines without a push: schedules, manual runs; compare_to sets another base; at most 50 patterns
existsfiles in the repository, relative to the project directorydirectories need a trailing slash (since 18.2); artifacts are invisible to it; beyond 50,000 files it always matches

In include and workflow#

  • include:rules accepts only if, exists and changes, and can use only the variables that exist before the pipeline is created (predefined variables). Its exists searches the project that holds the include.
  • workflow:rules decide whether the pipeline exists at all (chapter 22).

Error messages#

Appendix · Appendices · Chapter 67 · 9 min read

Search this page for the exact text you see. Each message is quoted as GitLab, the runner, a cloud or a delivery tool prints it: captured on a local GitLab 19.3 instance, or copied from the documentation. Each links to the symptom card or section that explains it. Messages that contain job or stage names were captured with the names shown; yours will differ.

When the pipeline is created#

MessageMeansGo to
The pipeline did not run. Review the workflow:rules configuration for the pipeline.no workflow:rules matchedworkflow refused
The resulting pipeline would have been empty. Review the rules configuration.the workflow allowed it, but no job's rules matchedempty pipeline
'unit-tests' job needs 'compile' job, but 'compile' does not exist in the pipeline.a needed job was left out by its rules; the message goes on to suggest needs:optionalneeds a missing job
orphan job: chosen stage nowhere does not exist; available stages are .pre, build, .posta job names a stage missing from stages:stage not listed
jobs:job:rules:rule if invalid expression syntaxan expression GitLab can't parse, such as !( … )rule syntax
(): did not find expected key while parsing a block mapping at line 5 column 7broken YAML, for example a quoted variable in if:rule syntax
Insufficient permissions to set pipeline variablesyour role may not set variables when starting a pipelinevalues given when the pipeline starts
Failed - (downstream pipeline can not be created, Insufficient permissions to set pipeline variables)the downstream project refuses the trigger job's variablespipeline variables refused
downstream_pipeline_creation_faileda trigger job's failure reason: the downstream pipeline couldn't be createdchild pipeline in merge requests

CI Lint also warns. Job may allow multiple pipelines to run for a single action means a rule has when but no if (rules syntax). On MegaCorp's templates, it also prints this deprecation warning, which suggests its own fix:

retry uses deprecated `when` value(s): stuck_or_timeout_failure. These match the more
specific failure reasons that replaced them; migrate to those reasons. See
https://docs.gitlab.com/ci/yaml/#retrywhen

On the job page, or in the job log#

MessageMeansGo to
This job is stuck because of one of the following problems. There are no active runners online, no runners for the protected branch, or no runners that match all of the job's tags:no online runner fits the job's tags, or only protected runners fitstuck on tags
This job is stuck because the project doesn't have any runners online assigned to it.the project's runners are offlinerunners offline
This job is stuck because you don't have any active runners that can run this job.no runner available to the project can take the jobhow a job finds a runner
This job could not start because it could not retrieve the needed artifacts.needed artifacts expired, or aren't reachableartifacts unavailable
No files to uploadartifacts: paths matched nothingno files to upload
*** WARNING: Service XYZ probably didn't start properlya service container didn't open its port in timea service without its variables
The deployment job is older than the latest deployment, and therefore failed.Prevent outdated deployment jobs stopped an old deployoutdated deploy
This deployment job does not run automatically and must be started manually, but it's older than the latest deployment, and therefore can't run.the same, for a manual deployoutdated deploy
unresolved referencethe commit isn't in the shallow cloneshallow history
remote: The project you were looking for could not be found or you don't have permission to view it.a clone with the job token was refusedjob token 404
fatal: run_command returned non-zero statussubmodules with GIT_STRATEGY: fetch; GitLab suggests clonesubmodules
fatal: unable to access 'https://gitlab.example.com/…/….git/': Could not resolve proxy: proxy.example.coma proxy variable from settings, or from the runner's configuration, names a proxy the runner can't reach, and Git obeys it; a job's YAML can't unset a settings variablevariables in settings
ERROR: Job failed (system failure): resolving secrets: operation error Secrets Manager: GetSecretValue, …, no EC2 IMDS role found, …the runner manager has no AWS role for secrets:secrets from AWS

From Azure#

MessageMeansGo to
AADSTS70021: No matching federated identity record found for presented assertion.no federated credential matches the token's issuer and subject, or one was added minutes agono matching federated credential
AADSTS700212the federated credential's audience differs from the token's audaudience mismatch
[megacorp] not on main, so not pushing to megacorpprod.azurecr.ioMegaCorp's image-build skipped the push to ACR outside main, by designan image not in Azure's registry

After the pipeline#

MessageMeansGo to
ErrImagePull, then ImagePullBackOffKubernetes can't pull the pod's imagean image that can't be pulled
CrashLoopBackOffthe container keeps exiting, and Kubernetes waits longer before each restarta crash loop
error: deployment "payments-api" exceeded its progress deadlinethe new pods didn't become ready within ten minutesa rollout stuck half-way
Error: execution error at (service/templates/deployment.yaml:28:52): image.tag is set for each environmentthe environment's values file has no image tagthe chart needs a tag
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. …an overlay uses an older Kustomize fielddeprecated Kustomize fields
Error acquiring the state lockanother run holds the state's lock, or a cancelled one left it behindstate locked
Helm install failed for release … failed pre-install: timeout waiting for: [Job/…]a HelmRelease's install ran out of time, five minutes by default, and Flux won't retry it by itselfa HelmRelease that timed out
dependency 'flux-system/add-ons' is not readya Flux Kustomization waits for another one that isn't readya Kustomization waiting for another
resource :Namespace is not permitted in project paymentsArgo CD may not create the namespace, because the project allows no cluster-wide kindsa namespace the project doesn't allow
tls: failed to verify certificate: x509: certificate signed by unknown authorityArgo CD, or another client, doesn't trust the Git server's certificatea certificate Argo CD doesn't trust

From AWS#

MessageMeansGo to
An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentitythe role's trust policy doesn't accept this tokenAssumeRole denied
An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn't retrieve verification key from your identity providerAWS can't reach GitLab's signing keysinvalid identity token
ImageTagAlreadyExistsExceptiona push to an immutable tag that already existstag already exists

The annotated specimen#

Appendix · Appendices · Chapter 68 · 82 min read

Every file of MegaCorp's specimen, in full. The chapters quote parts of these files; here you can read each one from top to bottom. The files are grouped by the project that owns them. Each group starts with what is worth noticing, linked to the pages that explain it.

A line such as # catalog: include-project marks one of the 37 reuse mechanisms: the construct just below it is an example. Files named SETTINGS.yml, and those under _settings, _runners, _aws and _azure, describe configuration kept in GitLab's settings, on the runners, or in the clouds. GitLab doesn't read them.

Where each mechanism appears#

Each marker links to its mechanism card. To see a mechanism at work, find it here, then open the file.

ChapterMarkers, and the file each is in
13yaml-anchors and yaml-merge-key in billing-batch; hidden-jobs in base.yml
14default-section and global-variables in base.yml; extends, reference-tag and parallel-matrix in java-maven.yml
15include-local in mono; include-project in payments-api; include-remote in billing-batch; include-template in security.yml
16include-component in payments-api; spec-inputs in sonar-scan.yml; variables-as-parameters in java-maven.yml; toggle-variables in security.yml; pipeline-inputs in the deployer
17parent-child and dynamic-child in mono; multi-project in deploy.yml; trigger-api in billing-batch
18custom-config-path in web-portal's settings; instance-group-variables in the instance and group settings; auto-devops and instance-template-repo in the instance settings; project-templates and compliance-pipeline in the group settings; scan-execution-policy, pipeline-execution-policy and mr-approval-policy in policy.yml
19toolbox-image in base.yml and the Dockerfile; shell-libraries in snippets.yml; internal-cli in mc; fetched-scripts in billing-batch; build-tool-reuse in pom.xml; runner-config in config.toml; ci-steps in mono's docs.yml

payments-api: one service#

Start here: this is everything the service team wrote. Worth noticing:

  • Its jobs come from the golden pipeline, pinned at v4.2.0, and from a component pinned at 2.1.0 (chapter 15).
  • maven-test: has no extends. It merges into the golden pipeline's job of the same name, and adds a variable (chapter 15).
  • That MAVEN_CLI_OPTS loses to the group variable of the same name, because settings beat YAML (chapter 26).
  • MC_TEAM: payments never reaches a job. .megacorp-base sets MC_TEAM as a job variable, which beats top-level ones (chapter 14).
  • publish-docs runs every night, because its only rule is the default branch. Its script only builds the site: nothing publishes it (chapter 55).
  • Only main and v* tags are protected, so only their pipelines receive PAYMENTS_DB_PASSWORD (chapter 27).
  • The POM's version stays 1.8.0-SNAPSHOT: nothing sets it from the Git tag (chapter 37).
megacorp/payments/payments-api.gitlab-ci.yml
# .gitlab-ci.yml in megacorp/payments/payments-api
# A Java service's entire pipeline file: include the golden pipeline and a
# component, set a few variables, adjust two jobs.
# Used by the chapters "Meet MegaCorp", "extends, !reference and default", "include:
# pulling in files", "Tracing a job to its source" and "Truth tables: predicting which jobs
# run".

# catalog: include-project
include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file: pipelines/java-service.yml
  # catalog: include-component
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api

variables:
  JAVA_VERSION: "21"
  MC_TEAM: payments

maven-test:
  variables:
    MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"

publish-docs:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
  inherit:
    default: [tags, retry]
    variables: false
  script:
    - mkdocs build --strict
  rules:
    - !reference [.rules, default-branch]
megacorp/payments/payments-apiSETTINGS.yml
# SETTINGS.yml for megacorp/payments/payments-api
# This file DESCRIBES the project's settings in the GitLab UI. GitLab does not read it.
# Used by the chapters "Variables and precedence" and "Secrets".

protected_branches: [main]
protected_tags: ["v*"]

project_variables:
  - key: MC_NEXUS_URL
    value: https://nexus.example.com/repository/maven-public/
  - key: PAYMENTS_DB_PASSWORD
    value: "(set in the UI)"
    masked: true
    protected: true
megacorp/payments/payments-apipom.xml
<?xml version="1.0" encoding="UTF-8"?>
<!-- pom.xml in megacorp/payments/payments-api (excerpt). The service inherits its
     build, including the "ci" profile the pipeline calls, from MegaCorp's parent POM.
     Used by the chapters "Reuse outside the YAML" and "Building Java, Node and front ends".
     -->
<project xmlns="http://maven.apache.org/POM/4.0.0">
  <modelVersion>4.0.0</modelVersion>
  <!-- catalog: build-tool-reuse -->
  <parent>
    <groupId>com.example.megacorp</groupId>
    <artifactId>megacorp-parent</artifactId>
    <version>12.3.0</version>
  </parent>
  <artifactId>payments-api</artifactId>
  <version>1.8.0-SNAPSHOT</version>
</project>
megacorp/payments/payments-apicatalog-info.yaml
# catalog-info.yaml in megacorp/payments/payments-api: how MegaCorp's developer portal, a
# Backstage, knows the service. The portal's "Java service" template wrote this file and the
# first .gitlab-ci.yml when it created the repository. Used by the chapters "Around your
# service" and "Field guide to enterprise setups".
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payments-api
  description: Takes card payments for MegaCorp's shop
  annotations:
    backstage.io/source-template: template:default/java-service
    backstage.io/source-location: url:https://gitlab.example.com/megacorp/payments/payments-api/
spec:
  type: service
  lifecycle: production
  owner: group:default/payments

devops/ci-templates: the central templates#

The central team's templates, at the v4.2.0 tag that payments-api pins. Worth noticing:

  • Only the two files in pipelines/ create jobs. The files in templates/ hold defaults, hidden jobs and snippets, except security.yml, which pulls in GitLab's scanner jobs (chapter 34).
  • .megacorp-base's rules decide when almost every job runs: merge requests, the default branch and release tags, never schedules (chapter 25).
  • .maven-test brings its own rules, which replace that list, so tag pipelines have no tests (chapter 14).
  • .maven-base and .image-build set their own before_script, which replaces the default one, so each loads the functions again (chapter 14).
  • .image-build runs in tag pipelines too, so a release builds a new image instead of promoting the one staging tested (chapter 39). With today's trust policy, it can't even push it (chapter 38).
  • .deploy hands IMAGE_REF to the deployer. The value comes from image-build's dotenv report (chapter 31).
  • security.yml's container_scanning adds needs, but keeps the template's dependencies: []. So IMAGE_REF, and the CS_IMAGE built from it, are empty in the scan (the symptom card).
  • legacy/notify.yml still uses only, and an older toolbox image. Every project that includes it by URL gets each change at once (include: remote).
megacorp/devops/ci-templatespipelines/java-service.yml
# pipelines/java-service.yml in megacorp/devops/ci-templates
# The "golden pipeline" for Java services. One include gives a project a build,
# tests, scanners, an image in ECR and deployments.
# Used by the chapters "Meet MegaCorp", "Reuse at a glance", "include: pulling in files",
# "Tracing a job to its source" and "Truth tables: predicting which jobs run".

include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file:
      - templates/base.yml
      - templates/snippets.yml
      - templates/rules.yml
      - templates/workflow.yml
      - templates/java-maven.yml
      - templates/container.yml
      - templates/security.yml
      - templates/deploy.yml

stages: [build, test, package, scan, deploy]

maven-build:
  extends: .maven-build

maven-test:
  extends: .maven-test

image-build:
  extends: .image-build
  needs: [maven-build]

deploy-staging:
  extends: .deploy
  needs: [image-build]
  variables:
    ENVIRONMENT: staging
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, default-branch]

deploy-prod:
  extends: .deploy
  needs: [image-build]
  variables:
    ENVIRONMENT: production
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
      when: manual
megacorp/devops/ci-templatespipelines/web.yml
# pipelines/web.yml in megacorp/devops/ci-templates
# The whole pipeline for static web front ends. Projects never include this
# file: their CI/CD settings point straight at it, so their repositories carry
# no .gitlab-ci.yml at all. Used by the chapters "Organisation-wide reuse: settings and
# policies", "Tracing a job to its source" and "Building Java, Node and front ends".

include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file:
      - templates/base.yml
      - templates/snippets.yml
      - templates/rules.yml
      - templates/workflow.yml
      - templates/node.yml

stages: [build, test, deploy]

node-build:
  extends: .node-build

node-test:
  extends: .node-test

deploy-site:
  extends: .publish-site
  needs: [node-build]
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, default-branch]
megacorp/devops/ci-templatestemplates/base.yml
# templates/base.yml in megacorp/devops/ci-templates
# The foundation: defaults for every job, organisation-wide variables, and the
# hidden base job that almost every template extends.
# Used by the chapters "Meet MegaCorp", "extends, !reference and default", "Job rules: is
# this job in the pipeline?", "Variables and precedence" and "Runners and executors".

# catalog: default-section, toolbox-image
default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  tags: [megacorp-shared]
  interruptible: true
  retry:
    max: 2
    when: [runner_system_failure, stuck_or_timeout_failure]
  before_script:
    - !reference [.snippets, functions]
    - mc_log "job $CI_JOB_NAME on $CI_COMMIT_REF_NAME"

# catalog: global-variables
variables:
  GIT_DEPTH: "20"
  AWS_REGION: eu-west-2
  MC_TEMPLATES_VERSION: "4.2.0"

# catalog: hidden-jobs
.megacorp-base:
  variables:
    MC_TEAM: unknown
  artifacts:
    expire_in: 7 days
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]
    - !reference [.rules, release-tag]
megacorp/devops/ci-templatestemplates/rules.yml
# templates/rules.yml in megacorp/devops/ci-templates
# A rules library: named lists of rules that jobs assemble with !reference.
# Used by the chapters "extends, !reference and default", "Job rules: is this job in the
# pipeline?" and "Idioms".

.rules:
  mr:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  default-branch:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  release-tag:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
  never-on-schedule:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
megacorp/devops/ci-templatestemplates/workflow.yml
# templates/workflow.yml in megacorp/devops/ci-templates
# Which pipelines exist at all: merge request pipelines, the default branch and
# tags, with no second branch pipeline while a merge request is open.
# Used by the chapters "Pipeline sources and types" and "workflow: whether a pipeline
# exists".

workflow:
  name: "$CI_PIPELINE_SOURCE: $CI_COMMIT_REF_NAME"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH
    - if: $CI_COMMIT_TAG
megacorp/devops/ci-templatestemplates/snippets.yml
# templates/snippets.yml in megacorp/devops/ci-templates
# Reusable script fragments. Jobs pull them in with !reference; the functions
# they call live in the ci-tools image, in /opt/megacorp/lib/ci-lib.sh.
# Used by the chapters "extends, !reference and default", "Reuse outside the YAML",
# "Identity and tokens" and "GitLab, AWS and Azure".

# catalog: shell-libraries
.snippets:
  functions:
    - source /opt/megacorp/lib/ci-lib.sh
  aws_login:
    - mc_aws_login "$AWS_ROLE_ARN"
  acr_push:
    - mc_acr_push "$IMAGE:$CI_COMMIT_SHA"
  maven_settings:
    - mkdir -p .m2
    - mc_maven_settings > .m2/settings.xml
megacorp/devops/ci-templatestemplates/java-maven.yml
# templates/java-maven.yml in megacorp/devops/ci-templates
# Build and test for Maven services. Everything here is hidden: a pipeline entry
# file turns these templates into real jobs.
# Used by the chapters "extends, !reference and default", "Variables and precedence",
# "Artifacts, cache and dotenv" and "Building Java, Node and front ends".

# catalog: extends, variables-as-parameters
.maven-base:
  extends: .megacorp-base
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
  variables:
    JAVA_VERSION: "21"
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  cache:
    key:
      files: [pom.xml]
    paths: [.m2/repository]
  before_script:
    # a job's own before_script replaces default:before_script, so re-add the functions
    - !reference [.snippets, functions]
    - !reference [.snippets, maven_settings]

.maven-build:
  extends: .maven-base
  stage: build
  script:
    - mvn $MAVEN_CLI_OPTS -DskipTests package
  artifacts:
    paths: [target/*.jar]

# catalog: reference-tag, parallel-matrix
.maven-test:
  extends: .maven-base
  stage: test
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JDK}
  parallel:
    matrix:
      - JDK: ["17", "21"]
  script:
    - mvn $MAVEN_CLI_OPTS verify
  artifacts:
    when: always
    reports:
      junit: target/surefire-reports/TEST-*.xml
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]
megacorp/devops/ci-templatestemplates/node.yml
# templates/node.yml in megacorp/devops/ci-templates
# Install, test and build Node front ends, then publish the static files to S3
# behind CloudFront. Used by the chapters "Artifacts, cache and dotenv", "Building Java,
# Node and front ends" and "GitLab, AWS and Azure".

.node-base:
  extends: .megacorp-base
  image: registry.example.com/megacorp/devops/ci-tools/node:22
  cache:
    key:
      files: [package-lock.json]
    paths: [.npm/]
    policy: pull
  before_script:
    - npm ci --cache .npm --prefer-offline

.node-build:
  extends: .node-base
  stage: build
  cache:
    policy: pull-push
  script:
    - npm run build
  artifacts:
    paths: [dist/]

.node-test:
  extends: .node-base
  stage: test
  script:
    - npm test -- --ci
  artifacts:
    when: always
    reports:
      junit: junit.xml

.publish-site:
  extends: .megacorp-base
  stage: deploy
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-web-publish
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - aws s3 sync dist/ "s3://$SITE_BUCKET/" --delete
    - aws cloudfront create-invalidation --distribution-id "$CF_DISTRIBUTION_ID" --paths '/*'
  environment:
    name: production
    url: https://www.example.com
megacorp/devops/ci-templatestemplates/container.yml
# templates/container.yml in megacorp/devops/ci-templates
# Build the service image without Docker-in-Docker and push it to Amazon ECR, tagged by
# commit, using the job's OIDC identity; on main, push the same image to Azure Container
# Registry too. Used by the chapters "Identity and tokens", "Images, services and Docker",
# "GitLab, AWS and Azure" and "Images to ECR and ACR".

.image-build:
  extends: .megacorp-base
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
    IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
    - buildah push "$IMAGE:$CI_COMMIT_SHA"
    - !reference [.snippets, acr_push]
    - echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
  artifacts:
    reports:
      dotenv: build.env
megacorp/devops/ci-templatestemplates/security.yml
# templates/security.yml in megacorp/devops/ci-templates
# GitLab's own scanners, pulled in as GitLab-shipped templates, plus MegaCorp's
# adjustments to the container scan. Used by the chapters "The scanning landscape",
# "GitLab's own scanners" and "Policies and gates".

# catalog: include-template
include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml
  - template: Security/Container-Scanning.gitlab-ci.yml

# catalog: toggle-variables
variables:
  CS_IMAGE: $IMAGE_REF

container_scanning:
  stage: scan
  needs: [image-build]
  rules:
    - if: $CONTAINER_SCANNING_DISABLED == "true"
      when: never
    - !reference [.rules, never-on-schedule]
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
megacorp/devops/ci-templatestemplates/deploy.yml
# templates/deploy.yml in megacorp/devops/ci-templates
# Hand the built image to the platform team's deployer project, which owns every
# environment. Used by the chapters "Pipelines as building blocks", "Environments and
# deployments" and "GitOps repositories and promotion".

# catalog: multi-project
.deploy:
  stage: deploy
  trigger:
    project: megacorp/platform/deployer
    branch: main
    strategy: depend
  variables:
    APP: $CI_PROJECT_NAME
    IMAGE_REF: $IMAGE_REF
megacorp/devops/ci-templateslegacy/notify.yml
# legacy/notify.yml in megacorp/devops/ci-templates
# An old notification job that billing-batch still pulls in by URL with
# include: remote, straight from this repository's raw file view.
# Used by the chapters "include: pulling in files" and "Reuse outside the YAML".

notify-chat:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools:2.9
  script:
    - curl --fail -X POST -d "text=billing-batch $CI_COMMIT_REF_NAME finished" "$CHAT_WEBHOOK_URL"
  only:
    - tags

devops/ci-tools: the toolbox image#

None of this is YAML, yet every job runs in this image unless it picks another one. Worth noticing:

  • The default before_script sources ci-lib.sh, so every job can call its functions (chapter 19).
  • mc_aws_login swaps the job's ID token for AWS keys that last one hour (chapter 38).
  • mc_retry runs a command up to three times, waiting longer each time.
  • mc deploy changes one line in the GitOps repository, using GITOPS_TOKEN (chapter 45).
  • mc version prints the Git tag, or 0.0.0- and the short commit when there is no tag (chapter 37).
megacorp/devops/ci-toolsDockerfile
# Dockerfile in megacorp/devops/ci-tools: MegaCorp's toolbox image.
# Every job runs in it unless it picks another image, and the shell functions
# and the "mc" tool that pipelines call live inside it.
# Used by the chapters "Reuse outside the YAML", "Identity and tokens" and "Images,
# services and Docker".

FROM public.ecr.aws/amazonlinux/amazonlinux:2023
RUN dnf install -y --allowerasing git jq make tar gzip unzip curl-minimal && dnf clean all
RUN curl -sSLo /tmp/awscli.zip https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip \
 && unzip -q /tmp/awscli.zip -d /tmp && /tmp/aws/install && rm -rf /tmp/aws /tmp/awscli.zip
RUN curl -sSLo /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.44.3/yq_linux_amd64 \
 && chmod +x /usr/local/bin/yq

# catalog: toolbox-image
COPY lib/ci-lib.sh /opt/megacorp/lib/ci-lib.sh
COPY bin/mc /usr/local/bin/mc
RUN chmod +x /usr/local/bin/mc
megacorp/devops/ci-toolslib/ci-lib.sh
#!/usr/bin/env bash
# lib/ci-lib.sh in megacorp/devops/ci-tools: MegaCorp's shell function library.
# It is baked into the ci-tools image as /opt/megacorp/lib/ci-lib.sh, and jobs
# source it through the .snippets functions snippet.
# Used by the chapters "extends, !reference and default", "Reuse outside the YAML",
# "Identity and tokens" and "GitLab, AWS and Azure".

mc_log() { printf '[megacorp] %s\n' "$*"; }

mc_retry() {
  local n=0
  until "$@"; do
    n=$((n + 1))
    [ "$n" -ge 3 ] && return 1
    mc_log "retry $n: $*"
    sleep $((n * 5))
  done
}

# Exchange the job's OIDC ID token for temporary AWS credentials.
mc_aws_login() {
  local role="${1:?role ARN required}"
  local creds
  creds=$(aws sts assume-role-with-web-identity \
    --role-arn "$role" \
    --role-session-name "gitlab-${CI_PROJECT_ID}-${CI_JOB_ID}" \
    --web-identity-token "$MC_ID_TOKEN" \
    --duration-seconds 3600 \
    --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
    --output text) || { mc_log "AssumeRoleWithWebIdentity failed for $role"; return 1; }
  read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< "$creds"
  export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
  mc_log "assumed $role"
}

# Sign in to Azure with the job's OIDC ID token, as the Entra application that trusts it.
mc_azure_login() {
  local client_id="${1:?application (client) ID required}"
  az login --service-principal -u "$client_id" -t "$MC_AZURE_TENANT_ID" \
    --federated-token "$MC_ID_TOKEN" --output none \
    || { mc_log "az login failed for $client_id"; return 1; }
  mc_log "signed in to Azure as $client_id"
}

# Push an image that is already built to Azure's registry as well. Only main's images go
# there: Entra matches a federated credential's subject exactly, and aks-prod deploys main.
mc_acr_push() {
  local image="${1:?image required}"
  if [ "${CI_COMMIT_BRANCH:-}" != "$CI_DEFAULT_BRANCH" ]; then
    mc_log "not on $CI_DEFAULT_BRANCH, so not pushing to $MC_ACR_REGISTRY"
    return 0
  fi
  mc_azure_login "$MC_ACR_PUSH_CLIENT_ID"
  az acr login --name "${MC_ACR_REGISTRY%%.*}" --expose-token --query accessToken --output tsv \
    | buildah login --username 00000000-0000-0000-0000-000000000000 --password-stdin "$MC_ACR_REGISTRY"
  buildah push "$image" "docker://$MC_ACR_REGISTRY/${image##*/}"
  mc_log "pushed $MC_ACR_REGISTRY/${image##*/}"
}

mc_maven_settings() {
  cat <<EOF
<settings><mirrors><mirror><id>nexus</id><mirrorOf>*</mirrorOf><url>${MC_NEXUS_URL:-https://nexus.example.com/repository/maven-public/}</url></mirror></mirrors></settings>
EOF
}
megacorp/devops/ci-toolsbin/mc
#!/usr/bin/env bash
# bin/mc in megacorp/devops/ci-tools: MegaCorp's internal CI command-line tool,
# shipped in the ci-tools image as /usr/local/bin/mc. The deployer pipeline calls
# "mc deploy"; release jobs call "mc version". Used by the chapters "Reuse outside the
# YAML", "Building Java, Node and front ends" and "GitOps repositories and promotion".
# catalog: internal-cli
set -euo pipefail
source /opt/megacorp/lib/ci-lib.sh

cmd="${1:-help}"
shift || true
case "$cmd" in
  deploy)
    env="${1:?usage: mc deploy <environment> <image-ref>}"
    image="${2:?usage: mc deploy <environment> <image-ref>}"
    tag="${image##*:}"
    mc_log "promoting ${APP} ${tag} to ${env} through the GitOps repository"
    git clone --depth 1 \
      "https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
    yq -i ".image.tag = \"${tag}\"" "gitops/envs/${env}/${APP}.yaml"
    git -C gitops commit -am "deploy ${APP} ${tag} to ${env}"
    if [ "$env" = production ]; then
      # production changes wait in a merge request until someone merges them
      git -C gitops push origin "HEAD:deploy/${APP}-${tag}" \
        -o merge_request.create -o merge_request.target=main \
        -o merge_request.title="Deploy ${APP} ${tag} to production"
    else
      git -C gitops push origin HEAD:main
    fi
    ;;
  version)
    echo "${CI_COMMIT_TAG:-0.0.0-${CI_COMMIT_SHORT_SHA}}"
    ;;
  *)
    echo "usage: mc deploy <environment> <image-ref> | mc version" >&2
    exit 2
    ;;
esac

devops/components: CI/CD components#

Three components. A project includes each at a version it chooses. Worth noticing:

  • The spec: header ends at ---. Each $[[ inputs.… ]] is replaced when the pipeline is created, before any variable is expanded (chapter 16).
  • sonar-scan has allow_failure: true and a SKIP_SONAR toggle, so even when the scan fails, the pipeline carries on (chapter 35).
  • ecr-push puts the repository input in its job's name, so each include with a different repository makes its own job (chapter 16).
  • gitops-deploy deploys only from the default branch, and waits for a person when deploy_when is manual (chapter 45).
megacorp/devops/componentstemplates/sonar-scan.yml
# templates/sonar-scan.yml in megacorp/devops/components: the sonar-scan CI/CD component.
# Projects include it with
#   component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@<version>
# Used by the chapters "Components, inputs and toggles" and "Third-party scanners".

# catalog: spec-inputs
spec:
  inputs:
    stage:
      default: test
    project_key:
      description: SonarQube project key
    quality_gate:
      type: boolean
      default: true
---

sonar-scan:
  stage: $[[ inputs.stage ]]
  image: registry.example.com/megacorp/devops/ci-tools/sonar-scanner:6
  variables:
    SONAR_PROJECT_KEY: $[[ inputs.project_key ]]
    SONAR_QUALITYGATE_WAIT: "$[[ inputs.quality_gate ]]"
  script:
    - sonar-scanner -Dsonar.projectKey="$SONAR_PROJECT_KEY" -Dsonar.host.url="$SONAR_HOST_URL"
  rules:
    - if: $SKIP_SONAR == "true"
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  allow_failure: true
megacorp/devops/componentstemplates/ecr-push/template.yml
# templates/ecr-push/template.yml in megacorp/devops/components: the ecr-push component.
# Builds an image with buildah and pushes it to Amazon ECR, authenticating with the
# job's OIDC ID token. Used by the chapters "Components, inputs and toggles" and "Images to
# ECR".

spec:
  inputs:
    repository:
      description: ECR repository name
    context:
      default: .
    role_arn:
      default: arn:aws:iam::123456789012:role/gitlab-ecr-push
    tag:
      default: $CI_COMMIT_SHA
---

ecr-push-$[[ inputs.repository ]]:
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: $[[ inputs.role_arn ]]
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
  script:
    - source /opt/megacorp/lib/ci-lib.sh
    - mc_aws_login "$AWS_ROLE_ARN"
    - aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$ECR_REGISTRY/$[[ inputs.repository ]]:$[[ inputs.tag ]]" "$[[ inputs.context ]]"
    - buildah push "$ECR_REGISTRY/$[[ inputs.repository ]]:$[[ inputs.tag ]]"
megacorp/devops/componentstemplates/gitops-deploy/template.yml
# templates/gitops-deploy/template.yml in megacorp/devops/components: the gitops-deploy component.
# It wraps the multi-project trigger to the platform team's deployer, so a project can
# deploy with one include instead of extending the .deploy template.
# Used by the chapters "Components, inputs and toggles" and "GitOps repositories and promotion".

spec:
  inputs:
    environment:
      options: [staging, production]
    app:
      default: $CI_PROJECT_NAME
    deploy_when:
      default: on_success
      options: [on_success, manual]
---

deploy-$[[ inputs.environment ]]:
  stage: deploy
  trigger:
    project: megacorp/platform/deployer
    branch: main
    strategy: depend
  variables:
    APP: $[[ inputs.app ]]
    ENVIRONMENT: $[[ inputs.environment ]]
    IMAGE_REF: $IMAGE_REF
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: $[[ inputs.deploy_when ]]

data/mono: a monorepo#

A monorepo that doesn't use the golden pipeline. Worth noticing:

  • generate-pipeline writes a pipeline file, and service-pipelines runs it as a child pipeline (chapter 17).
  • generate.sh fails on merge requests with a long history, and builds too little after a push of several commits (chapter 32).
  • docs starts its child pipeline only when files under docs/ changed. For a new branch, changes is always true (chapter 23).
  • lint-docs runs GitLab Functions instead of a script, an experimental feature (chapter 19).
megacorp/data/mono.gitlab-ci.yml
# .gitlab-ci.yml in megacorp/data/mono
# A monorepo. A generator job writes one child-pipeline job per changed service,
# and a static child pipeline builds the docs. Used by the chapters "include: pulling in
# files", "Pipelines as building blocks" and "Field guide to enterprise setups".

# catalog: include-local
include:
  - local: ci/common.yml

stages: [generate, build]

generate-pipeline:
  stage: generate
  script:
    - ci/generate.sh > generated-pipeline.yml
  artifacts:
    paths: [generated-pipeline.yml]

# catalog: dynamic-child
service-pipelines:
  stage: build
  trigger:
    include:
      - artifact: generated-pipeline.yml
        job: generate-pipeline
    strategy: depend

# catalog: parent-child
docs:
  stage: build
  trigger:
    include: ci/docs.yml
  rules:
    - changes: [docs/**/*]
megacorp/data/monoci/common.yml
# ci/common.yml in megacorp/data/mono
# Defaults shared by the monorepo's own pipeline, pulled in with include: local.
# Used by the chapter "include: pulling in files".

default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  tags: [megacorp-shared]
megacorp/data/monoci/generate.sh
#!/usr/bin/env bash
# ci/generate.sh in megacorp/data/mono
# Writes one child-pipeline job per service that changed. Its output is the YAML
# that the service-pipelines trigger job runs. Used by the chapter "Pipelines as building
# blocks".
set -euo pipefail

base="${CI_MERGE_REQUEST_DIFF_BASE_SHA:-HEAD~1}"
changed=$(git diff --name-only "$base" HEAD -- services/ | cut -d/ -f2 | sort -u)

echo "stages: [build]"
for svc in $changed; do
  cat <<EOF
build-${svc}:
  stage: build
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  script:
    - make -C services/${svc} build
EOF
done

# a child pipeline needs at least one job, even when nothing changed
if [ -z "$changed" ]; then
  printf 'no-changes:\n  stage: build\n  script: [echo "no service changed"]\n'
fi
megacorp/data/monoci/docs.yml
# ci/docs.yml in megacorp/data/mono
# The child pipeline that builds the monorepo's documentation. It also carries an
# experimental job that uses GitLab Functions (formerly CI/CD Steps) instead of a script.
# Used by the chapters "Pipelines as building blocks" and "Reuse outside the YAML".

build-docs:
  image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
  script:
    - mkdocs build --strict

# catalog: ci-steps
lint-docs:
  image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
  run:
    - name: markdown_lint
      func: registry.example.com/megacorp/devops/functions/markdownlint:1.2.0
      inputs:
        path: docs
    - name: report
      script: echo "markdown lint finished"

web/web-portal: a project with no pipeline file#

web-portal's repository has no .gitlab-ci.yml. Its settings point at pipelines/web.yml in the templates project, shown above, and that file is its whole pipeline (chapter 18). For the other ways a project without the file still runs pipelines, see chapter 56.

megacorp/web/web-portalSETTINGS.yml
# SETTINGS.yml for megacorp/web/web-portal
# This file DESCRIBES the project's CI/CD settings in the GitLab UI. GitLab does
# not read it. The repository itself has no .gitlab-ci.yml: its pipeline comes
# from the central templates project. Used by the chapters "Organisation-wide reuse:
# settings and policies", "Tracing a job to its source" and "Field guide to enterprise
# setups".

# catalog: custom-config-path
ci_cd_configuration_file: pipelines/web.yml@megacorp/devops/ci-templates

project_variables:
  - key: SITE_BUCKET
    value: www-example-com-site
  - key: CF_DISTRIBUTION_ID
    value: E2EXAMPLE1234

platform/deployer: every deploy goes through here#

The platform team's project. Service pipelines trigger it, and people can run it by hand. Worth noticing:

  • Someone running it by hand picks an environment from the input's options. When a service triggers it, the trigger's ENVIRONMENT beats the value built from the input (pipeline inputs).
  • resource_group: $APP-$ENVIRONMENT lets only one deploy of an app to an environment run at a time (chapter 24).
  • Without an IMAGE_REF there is no deploy, and production waits for a person.
  • GITOPS_TOKEN expires a year after it was created. From that day, every deploy fails (chapter 28).
megacorp/platform/deployer.gitlab-ci.yml
# .gitlab-ci.yml in megacorp/platform/deployer
# The platform team's deployment pipeline. Service pipelines trigger it as a
# multi-project pipeline, and people can also run it by hand from the UI.
# Used by the chapters "Pipelines as building blocks", "Environments and deployments" and
# "GitOps repositories and promotion".

# catalog: pipeline-inputs
spec:
  inputs:
    environment:
      type: string
      default: staging
      options: [staging, production]
---

variables:
  ENVIRONMENT: $[[ inputs.environment ]]

deploy:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  resource_group: $APP-$ENVIRONMENT
  environment:
    name: $ENVIRONMENT/$APP
  script:
    - mc deploy "$ENVIRONMENT" "$IMAGE_REF"
  rules:
    - if: $IMAGE_REF == null
      when: never
    - if: $ENVIRONMENT == "production"
      when: manual
    - when: on_success
megacorp/platform/deployerSETTINGS.yml
# SETTINGS.yml for megacorp/platform/deployer
# This file DESCRIBES the project's settings in the GitLab UI. GitLab does not read it.
# Used by the chapters "Identity and tokens" and "GitOps repositories and promotion".

project_variables:
  - key: GITOPS_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true
    # A project access token created in megacorp/platform/gitops-config, with the
    # Maintainer role and the write_repository scope. Nobody entered an expiry date,
    # so GitLab set one 365 days after the token was created.

platform/charts: the shared chart#

Every MegaCorp service is deployed with this chart; each environment's values file says what differs. Worth noticing:

  • The image's tag has no default. Rendering without one stops with the chart's own message (the chart needs a tag).
  • The ExternalSecret exists only when a service's values name secret keys, and the Deployment reads the Secret it makes through envFrom (chapter 50).
  • The probes' path defaults to /actuator/health, which suits the Java services and nothing else (0/1 ready).
  • Region markers in the templates are Helm comments, so helm template never prints them.
megacorp/platform/chartsservice/Chart.yaml
# service/Chart.yaml in megacorp/platform/charts: the shared Helm chart that every
# MegaCorp HTTP service deploys with. The charts repository tags each release, such as
# service-3.4.0, and the ApplicationSets name that tag.
# Used by the chapters "Helm" and "Argo CD".
apiVersion: v2
name: service
description: One MegaCorp HTTP service, with its Service, Ingress and secrets
type: application
version: 3.4.0
megacorp/platform/chartsservice/values.yaml
# service/values.yaml in megacorp/platform/charts: the defaults every service starts
# from. A service's file in gitops-config sets what differs.
# Used by the chapters "Helm" and "GitOps repositories and promotion".

image:
  repository: ""          # set for each service
  tag: ""                 # set for each environment; mc deploy writes it
  pullPolicy: IfNotPresent

replicaCount: 2

port: 8080
env: {}                   # plain settings, as NAME: value
probes:
  path: /actuator/health
resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    memory: 512Mi

serviceAccount:
  annotations: {}         # the pod's cloud identity, where the cloud needs one
podLabels: {}

ingress:
  enabled: false
  className: nginx
  host: ""

secrets:
  store: ""               # the ClusterSecretStore to read from
  keys: []                # each one: {name: DB_PASSWORD, remote: payments-api/staging/db-password}
megacorp/platform/chartsservice/templates/_helpers.tpl
{{- /*
service/templates/_helpers.tpl in megacorp/platform/charts: named templates that the
other templates share, so every object gets the same name and labels.
Used by the chapter "Helm".
*/}}

{{- define "service.name" -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{- define "service.selectorLabels" -}}
app.kubernetes.io/name: {{ include "service.name" . }}
{{- end -}}

{{- define "service.labels" -}}
{{ include "service.selectorLabels" . }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
{{- end -}}
megacorp/platform/chartsservice/templates/deployment.yaml
{{- /*
service/templates/deployment.yaml in megacorp/platform/charts: the Deployment that runs a
service's pods. Used by the chapters "Containers and Kubernetes, enough to read" and "Helm".
*/}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "service.name" . }}
  labels:
    {{- include "service.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "service.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "service.selectorLabels" . | nindent 8 }}
        {{- with .Values.podLabels }}
        {{- toYaml . | nindent 8 }}
        {{- end }}
    spec:
      serviceAccountName: {{ include "service.name" . }}
      containers:
        - name: app
          image: "{{ .Values.image.repository }}:{{ required "image.tag is set for each environment" .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: {{ .Values.port }}
          {{- with .Values.env }}
          env:
            {{- range $name, $value := . }}
            - name: {{ $name }}
              value: {{ $value | quote }}
            {{- end }}
          {{- end }}
          {{- if .Values.secrets.keys }}
          envFrom:
            - secretRef:
                name: {{ include "service.name" . }}
          {{- end }}
          readinessProbe:
            httpGet:
              path: {{ .Values.probes.path }}
              port: http
          livenessProbe:
            httpGet:
              path: {{ .Values.probes.path }}
              port: http
            initialDelaySeconds: 20
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
megacorp/platform/chartsservice/templates/service.yaml
{{- /*
service/templates/service.yaml in megacorp/platform/charts: the service's ServiceAccount,
the Service in front of its pods, and the Ingress that brings outside traffic in.
Used by the chapters "Containers and Kubernetes, enough to read", "Helm" and "AWS and
Azure for delivery".
*/}}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ include "service.name" . }}
  labels:
    {{- include "service.labels" . | nindent 4 }}
  {{- with .Values.serviceAccount.annotations }}
  annotations:
    {{- toYaml . | nindent 4 }}
  {{- end }}
---
apiVersion: v1
kind: Service
metadata:
  name: {{ include "service.name" . }}
  labels:
    {{- include "service.labels" . | nindent 4 }}
spec:
  selector:
    {{- include "service.selectorLabels" . | nindent 4 }}
  ports:
    - name: http
      port: 80
      targetPort: http
{{- if .Values.ingress.enabled }}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ include "service.name" . }}
  labels:
    {{- include "service.labels" . | nindent 4 }}
spec:
  ingressClassName: {{ .Values.ingress.className }}
  rules:
    - host: {{ required "ingress.host is needed when ingress.enabled is true" .Values.ingress.host }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ include "service.name" . }}
                port:
                  name: http
{{- end }}
megacorp/platform/chartsservice/templates/externalsecret.yaml
{{- /*
service/templates/externalsecret.yaml in megacorp/platform/charts: asks the External Secrets
Operator to copy a service's secrets from the cloud's vault into a Kubernetes Secret, which
the Deployment reads. Used by the chapters "Helm" and "Secrets and config after the pipeline".
*/}}
{{- if .Values.secrets.keys }}
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: {{ include "service.name" . }}
  labels:
    {{- include "service.labels" . | nindent 4 }}
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: {{ required "secrets.store names the ClusterSecretStore" .Values.secrets.store }}
  target:
    name: {{ include "service.name" . }}
  data:
    {{- range .Values.secrets.keys }}
    - secretKey: {{ .name }}
      remoteRef:
        key: {{ .remote }}
    {{- end }}
{{- end }}

platform/gitops-config: the environment repository#

What runs on eks-prod and aks-prod. Flux reads the clusters folders, and Argo CD reads apps and envs. Worth noticing:

  • Each cluster applies three Kustomizations in order: add-ons, then their settings, then the applications (chapter 46).
  • aks-prod's add-ons are eks-prod's, with one patch that gives the External Secrets Operator its Azure identity (chapter 44).
  • Both clusters name their secret store cloud-secrets, so a service's values don't change between clouds (chapter 50).
  • One ApplicationSet makes payments-api's Application for each environment, from the chart at a tag and the values in envs (chapter 47). aks-prod's sets the image's repository inline, which beats the values files (chapter 47).
  • Argo CD marks each deploy on the dashboards, with a key that an ExternalSecret copies from the vault (chapter 52).
megacorp/platform/gitops-configclusters/eks-prod/flux-system/gotk-sync.yaml
# clusters/eks-prod/flux-system/gotk-sync.yaml in megacorp/platform/gitops-config: written by
# "flux bootstrap gitlab". It points eks-prod's Flux at this repository and at this folder.
# The bootstrap also wrote gotk-components.yaml, Flux's own install, which is left out here.
# Used by the chapters "GitOps repositories and promotion" and "Flux".
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: flux-system
  url: https://gitlab.example.com/megacorp/platform/gitops-config.git
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./clusters/eks-prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
megacorp/platform/gitops-configclusters/eks-prod/platform.yaml
# clusters/eks-prod/platform.yaml in megacorp/platform/gitops-config: what eks-prod runs, as
# three Flux Kustomizations that go in order: the add-ons, then their settings, then the
# teams' applications. Used by the chapters "GitOps repositories and promotion" and "Flux".
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: add-ons
  namespace: flux-system
spec:
  interval: 10m
  retryInterval: 1m
  path: ./infrastructure/add-ons/base
  prune: true
  wait: true
  sourceRef:
    kind: GitRepository
    name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: config
  namespace: flux-system
spec:
  dependsOn:
    - name: add-ons
  interval: 10m
  path: ./infrastructure/config/eks
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  dependsOn:
    - name: config
  interval: 10m
  path: ./apps/eks-prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
megacorp/platform/gitops-configclusters/aks-prod/flux-system/gotk-sync.yaml
# clusters/aks-prod/flux-system/gotk-sync.yaml in megacorp/platform/gitops-config: written by
# "flux bootstrap gitlab" for aks-prod. The same as eks-prod's, apart from the folder.
# Used by the chapter "Flux".
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: flux-system
  url: https://gitlab.example.com/megacorp/platform/gitops-config.git
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./clusters/aks-prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
megacorp/platform/gitops-configclusters/aks-prod/platform.yaml
# clusters/aks-prod/platform.yaml in megacorp/platform/gitops-config: what aks-prod runs. The
# same three steps as eks-prod, from the Azure folders.
# Used by the chapters "Flux" and "AWS and Azure for delivery".
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: add-ons
  namespace: flux-system
spec:
  interval: 10m
  retryInterval: 1m
  path: ./infrastructure/add-ons/aks
  prune: true
  wait: true
  sourceRef:
    kind: GitRepository
    name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: config
  namespace: flux-system
spec:
  dependsOn:
    - name: add-ons
  interval: 10m
  path: ./infrastructure/config/aks
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  dependsOn:
    - name: config
  interval: 10m
  path: ./apps/aks-prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
megacorp/platform/gitops-configinfrastructure/add-ons/base/kustomization.yaml
# infrastructure/add-ons/base/kustomization.yaml in megacorp/platform/gitops-config: the
# add-ons every MegaCorp cluster runs, as Kustomize's base. eks-prod uses it as it is, and
# aks-prod's overlay changes it. Used by the chapters "Kustomize" and "Flux".
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - add-ons.yaml
megacorp/platform/gitops-configinfrastructure/add-ons/base/add-ons.yaml
# infrastructure/add-ons/base/add-ons.yaml in megacorp/platform/gitops-config: the External
# Secrets Operator and Argo CD, which Flux installs from their Helm charts at pinned versions.
# Used by the chapters "Kustomize", "Flux", "Argo CD", "Secrets and config after the
# pipeline" and "Observability".
apiVersion: v1
kind: Namespace
metadata:
  name: external-secrets
---
apiVersion: v1
kind: Namespace
metadata:
  name: argocd
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: external-secrets
  namespace: flux-system
spec:
  interval: 1h
  url: https://charts.external-secrets.io
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: argo
  namespace: flux-system
spec:
  interval: 1h
  url: https://argoproj.github.io/argo-helm
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: external-secrets
  namespace: external-secrets
spec:
  interval: 30m
  chart:
    spec:
      chart: external-secrets
      version: 2.10.0
      sourceRef:
        kind: HelmRepository
        name: external-secrets
        namespace: flux-system
  install:
    crds: CreateReplace
  upgrade:
    crds: CreateReplace
  values:
    serviceAccount:
      name: external-secrets
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: argo-cd
  namespace: argocd
spec:
  interval: 30m
  chart:
    spec:
      chart: argo-cd
      version: 10.9.0
      sourceRef:
        kind: HelmRepository
        name: argo
        namespace: flux-system
  values:
    configs:
      params:
        server.insecure: true
    notifications:
      secret:
        create: false
      notifiers:
        service.grafana: |
          apiUrl: https://grafana.example.com/api
          apiKey: $grafana-api-key
      triggers:
        trigger.on-deployed: |
          - description: Application is synced and healthy.
            send: [app-deployed]
            oncePer: app.status.operationState?.syncResult?.revision
            when: >-
              app.status.operationState != nil
              and app.status.operationState.phase in ['Succeeded']
              and app.status.health.status == 'Healthy'
              and (!time.Parse(app.status.health.lastTransitionTime).Add(1 * time.Minute).Before(time.Parse(app.status.operationState.finishedAt))
              or time.Parse(app.status.health.lastTransitionTime).Before(time.Parse(app.status.operationState.startedAt)))
      templates:
        template.app-deployed: |
          message: "{{ .app.metadata.name }} is synced and healthy"
megacorp/platform/gitops-configinfrastructure/add-ons/aks/kustomization.yaml
# infrastructure/add-ons/aks/kustomization.yaml in megacorp/platform/gitops-config: aks-prod's
# overlay. It takes the base, and gives the External Secrets Operator the Azure identity that
# it reads Key Vault with. Used by the chapters "Kustomize" and "AWS and Azure for delivery".
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base
patches:
  - target:
      kind: HelmRelease
      name: external-secrets
    patch: |-
      apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: external-secrets
      spec:
        values:
          serviceAccount:
            annotations:
              azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
          podLabels:
            azure.workload.identity/use: "true"
megacorp/platform/gitops-configinfrastructure/config/eks/secret-store.yaml
# infrastructure/config/eks/secret-store.yaml in megacorp/platform/gitops-config: eks-prod's
# secret store. Services name "cloud-secrets"; on this cluster it reads AWS Secrets Manager,
# as the External Secrets Operator's own identity, through EKS Pod Identity. Also the key
# Argo CD's notifications use to mark deploys on Grafana's dashboards.
# Used by the chapters "Secrets and config after the pipeline", "AWS and Azure for
# delivery" and "Observability".
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: cloud-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-2
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: argocd-notifications-secret
  namespace: argocd
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: cloud-secrets
  target:
    name: argocd-notifications-secret
  data:
    - secretKey: grafana-api-key
      remoteRef:
        key: platform-grafana-api-key
megacorp/platform/gitops-configinfrastructure/config/aks/secret-store.yaml
# infrastructure/config/aks/secret-store.yaml in megacorp/platform/gitops-config: aks-prod's
# secret store. The same name, "cloud-secrets", so a service's values don't change; on this
# cluster it reads Azure Key Vault through AKS workload identity. Also the key Argo CD's
# notifications use to mark deploys on Grafana's dashboards.
# Used by the chapters "Secrets and config after the pipeline", "AWS and Azure for
# delivery" and "Observability".
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: cloud-secrets
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: https://megacorp-prod-kv.vault.azure.net
      serviceAccountRef:
        name: external-secrets
        namespace: external-secrets
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: argocd-notifications-secret
  namespace: argocd
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: cloud-secrets
  target:
    name: argocd-notifications-secret
  data:
    - secretKey: grafana-api-key
      remoteRef:
        key: platform-grafana-api-key
megacorp/platform/gitops-configapps/eks-prod/applicationset.yaml
# apps/eks-prod/applicationset.yaml in megacorp/platform/gitops-config: the payments team's
# Argo CD project on eks-prod, and the ApplicationSet that makes one payments-api
# Application for each environment. Used by the chapters "GitOps repositories and
# promotion", "Argo CD" and "Observability".
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payments
  namespace: argocd
spec:
  description: The payments team's services
  sourceRepos:
    - https://gitlab.example.com/megacorp/platform/charts.git
    - https://gitlab.example.com/megacorp/platform/gitops-config.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: payments-*
  # CreateNamespace makes each environment's Namespace, which is cluster-wide, so the
  # project must allow that kind, and only with the team's names
  clusterResourceWhitelist:
    - group: ""
      kind: Namespace
      name: payments-*
---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payments-api
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - list:
        elements:
          - env: staging
          - env: production
  template:
    metadata:
      name: "payments-api-{{ .env }}"
      annotations:
        notifications.argoproj.io/subscribe.on-deployed.grafana: "payments-api|{{ .env }}"
    spec:
      project: payments
      sources:
        - repoURL: https://gitlab.example.com/megacorp/platform/charts.git
          targetRevision: service-3.4.0
          path: service
          helm:
            releaseName: payments-api
            valueFiles:
              - "$values/envs/{{ .env }}/payments-api.yaml"
        - repoURL: https://gitlab.example.com/megacorp/platform/gitops-config.git
          targetRevision: main
          ref: values
      destination:
        server: https://kubernetes.default.svc
        namespace: "payments-{{ .env }}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
megacorp/platform/gitops-configapps/aks-prod/applicationset.yaml
# apps/aks-prod/applicationset.yaml in megacorp/platform/gitops-config: the same project and
# ApplicationSet on aks-prod. The inline values point payments-api at Azure's registry, and
# give its pods their Azure identity. Used by the chapters "Argo CD" and "AWS and Azure for
# delivery".
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payments
  namespace: argocd
spec:
  description: The payments team's services
  sourceRepos:
    - https://gitlab.example.com/megacorp/platform/charts.git
    - https://gitlab.example.com/megacorp/platform/gitops-config.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: payments-*
  # CreateNamespace makes each environment's Namespace, which is cluster-wide, so the
  # project must allow that kind, and only with the team's names
  clusterResourceWhitelist:
    - group: ""
      kind: Namespace
      name: payments-*
---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payments-api
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - list:
        elements:
          - env: staging
          - env: production
  template:
    metadata:
      name: "payments-api-{{ .env }}"
      annotations:
        notifications.argoproj.io/subscribe.on-deployed.grafana: "payments-api|{{ .env }}"
    spec:
      project: payments
      sources:
        - repoURL: https://gitlab.example.com/megacorp/platform/charts.git
          targetRevision: service-3.4.0
          path: service
          helm:
            releaseName: payments-api
            valueFiles:
              - "$values/envs/{{ .env }}/payments-api.yaml"
            valuesObject:
              image:
                repository: megacorpprod.azurecr.io/payments-api
              serviceAccount:
                annotations:
                  azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
              podLabels:
                azure.workload.identity/use: "true"
        - repoURL: https://gitlab.example.com/megacorp/platform/gitops-config.git
          targetRevision: main
          ref: values
      destination:
        server: https://kubernetes.default.svc
        namespace: "payments-{{ .env }}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
megacorp/platform/gitops-configenvs/staging/payments-api.yaml
# envs/staging/payments-api.yaml in megacorp/platform/gitops-config: payments-api's values in
# staging. mc deploy writes the tag; people change the rest by merge request.
# Used by the chapters "Helm", "GitOps repositories and promotion" and "Argo CD".
image:
  repository: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api
  tag: 3f9c2e1a
replicaCount: 1
env:
  SPRING_PROFILES_ACTIVE: staging
  PAYMENTS_PROVIDER_URL: https://sandbox.pay.example.com
secrets:
  store: cloud-secrets
  keys:
    - name: DB_PASSWORD
      remote: payments-api/staging/db-password
ingress:
  enabled: true
  host: payments.staging.example.com
megacorp/platform/gitops-configenvs/production/payments-api.yaml
# envs/production/payments-api.yaml in megacorp/platform/gitops-config: payments-api's values
# in production. A new tag arrives in a merge request that mc deploy opens, and goes live
# when someone merges it. Used by the chapters "GitOps repositories and promotion" and
# "Releasing safely".
image:
  repository: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api
  tag: 1c7d4b09
replicaCount: 3
env:
  SPRING_PROFILES_ACTIVE: production
  PAYMENTS_PROVIDER_URL: https://pay.example.com
secrets:
  store: cloud-secrets
  keys:
    - name: DB_PASSWORD
      remote: payments-api/production/db-password
ingress:
  enabled: true
  host: payments.example.com

platform/infra-modules and infra-live: OpenTofu#

The platform team's modules for each cloud, and the root configuration that calls them. Worth noticing:

  • infra-live calls each module at a tag, so a module's change reaches the clusters only when the tag in infra-live moves (chapter 48).
  • A team asks for a registry or a pod identity by adding an entry to infra-live, in a merge request the platform team reviews and applies (chapter 48).
  • ECR's tags are immutable, and the AKS cluster's nodes may pull from ACR.
megacorp/platform/infra-modulesaws/main.tf
# aws/main.tf in megacorp/platform/infra-modules: the AWS side of MegaCorp's delivery
# platform, as one OpenTofu module: an EKS cluster, an ECR repository for each service, and
# an EKS Pod Identity role for each workload. The platform team applies it.
# Used by the chapters "OpenTofu and Terraform" and "AWS and Azure for delivery".

terraform {
  required_version = ">= 1.12.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

variable "cluster_name" {
  type = string
}

variable "kubernetes_version" {
  type    = string
  default = "1.36"
}

variable "subnet_ids" {
  type = list(string)
}

variable "repositories" {
  description = "ECR repositories to create, such as payments-api"
  type        = set(string)
}

variable "workloads" {
  description = "Pods that get an AWS role, keyed by name, each with its namespace and service account"
  type = map(object({
    namespace       = string
    service_account = string
  }))
}

resource "aws_iam_role" "cluster" {
  name = "${var.cluster_name}-cluster"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "eks.amazonaws.com" }
      Action    = ["sts:AssumeRole", "sts:TagSession"]
    }]
  })
}

resource "aws_eks_cluster" "this" {
  name     = var.cluster_name
  version  = var.kubernetes_version
  role_arn = aws_iam_role.cluster.arn

  vpc_config {
    subnet_ids = var.subnet_ids
  }

  access_config {
    authentication_mode = "API"
  }
}

resource "aws_eks_addon" "pod_identity_agent" {
  cluster_name = aws_eks_cluster.this.name
  addon_name   = "eks-pod-identity-agent"
}

resource "aws_ecr_repository" "service" {
  for_each             = var.repositories
  name                 = each.value
  image_tag_mutability = "IMMUTABLE"
}

resource "aws_iam_role" "workload" {
  for_each = var.workloads
  name     = "${var.cluster_name}-${each.key}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "pods.eks.amazonaws.com" }
      Action    = ["sts:AssumeRole", "sts:TagSession"]
    }]
  })
}

resource "aws_eks_pod_identity_association" "workload" {
  for_each        = var.workloads
  cluster_name    = aws_eks_cluster.this.name
  namespace       = each.value.namespace
  service_account = each.value.service_account
  role_arn        = aws_iam_role.workload[each.key].arn
}

output "repository_urls" {
  value = { for name, repo in aws_ecr_repository.service : name => repo.repository_url }
}

output "workload_role_arns" {
  value = { for name, role in aws_iam_role.workload : name => role.arn }
}
megacorp/platform/infra-modulesazure/main.tf
# azure/main.tf in megacorp/platform/infra-modules: the Azure side of MegaCorp's delivery
# platform, as one OpenTofu module: an AKS cluster with workload identity, an Azure
# Container Registry that the cluster pulls from, and a managed identity for each workload.
# Used by the chapters "OpenTofu and Terraform" and "AWS and Azure for delivery".

terraform {
  required_version = ">= 1.12.0"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
  }
}

variable "resource_group_name" {
  type = string
}

variable "location" {
  type    = string
  default = "westeurope"
}

variable "cluster_name" {
  type = string
}

variable "registry_name" {
  description = "Globally unique, letters and digits only, such as megacorpprod"
  type        = string
}

variable "workloads" {
  description = "Pods that get an Azure identity, keyed by name, each with its namespace and service account"
  type = map(object({
    namespace       = string
    service_account = string
  }))
}

resource "azurerm_kubernetes_cluster" "this" {
  name                      = var.cluster_name
  location                  = var.location
  resource_group_name       = var.resource_group_name
  dns_prefix                = var.cluster_name
  kubernetes_version        = "1.36"
  oidc_issuer_enabled       = true
  workload_identity_enabled = true

  default_node_pool {
    name       = "system"
    node_count = 3
    vm_size    = "Standard_D4s_v5"
  }

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_container_registry" "this" {
  name                = var.registry_name
  resource_group_name = var.resource_group_name
  location            = var.location
  sku                 = "Standard"
  admin_enabled       = false
}

resource "azurerm_role_assignment" "cluster_pulls" {
  scope                = azurerm_container_registry.this.id
  role_definition_name = "AcrPull"
  principal_id         = azurerm_kubernetes_cluster.this.kubelet_identity[0].object_id
}

resource "azurerm_user_assigned_identity" "workload" {
  for_each            = var.workloads
  name                = "${var.cluster_name}-${each.key}"
  resource_group_name = var.resource_group_name
  location            = var.location
}

resource "azurerm_federated_identity_credential" "workload" {
  for_each  = var.workloads
  name      = each.key
  parent_id = azurerm_user_assigned_identity.workload[each.key].id
  issuer    = azurerm_kubernetes_cluster.this.oidc_issuer_url
  subject   = "system:serviceaccount:${each.value.namespace}:${each.value.service_account}"
  audience  = ["api://AzureADTokenExchange"]
}

output "workload_client_ids" {
  value = { for name, id in azurerm_user_assigned_identity.workload : name => id.client_id }
}
megacorp/platform/infra-livemain.tf
# main.tf in megacorp/platform/infra-live: the platform team's root configuration for both
# clouds. It calls the modules in infra-modules at pinned tags. A team asks for a registry or
# a pod identity by adding an entry here, in a merge request that the platform team reviews
# and applies. Used by the chapters "OpenTofu and Terraform" and "AWS and Azure for delivery".

terraform {
  required_version = ">= 1.12.0"
  backend "http" {}
}

provider "aws" {
  region = "eu-west-2"
}

provider "azurerm" {
  features {}
  subscription_id = "00000000-0000-0000-0000-000000000000"
}

module "eks_prod" {
  source       = "git::https://gitlab.example.com/megacorp/platform/infra-modules.git//aws?ref=aws-2.3.0"
  cluster_name = "eks-prod"
  subnet_ids   = ["subnet-0a1b2c3d4e5f60718", "subnet-0b2c3d4e5f6071829"]

  repositories = ["payments-api", "web-portal"]
  workloads = {
    payments-api-staging    = { namespace = "payments-staging", service_account = "payments-api" }
    payments-api-production = { namespace = "payments-production", service_account = "payments-api" }
  }
}

module "aks_prod" {
  source              = "git::https://gitlab.example.com/megacorp/platform/infra-modules.git//azure?ref=azure-1.8.0"
  resource_group_name = "rg-delivery-prod"
  cluster_name        = "aks-prod"
  registry_name       = "megacorpprod"

  workloads = {
    payments-api-staging    = { namespace = "payments-staging", service_account = "payments-api" }
    payments-api-production = { namespace = "payments-production", service_account = "payments-api" }
  }
}

Settings, runners, AWS and Azure#

What MegaCorp keeps outside every repository. Projects can't see most of it, yet all of it reaches their jobs. Worth noticing:

  • A group variable such as MAVEN_CLI_OPTS beats anything a project writes in YAML (chapter 26).
  • SONAR_TOKEN is protected, so a merge request pipeline from an unprotected branch gets it empty (chapter 27).
  • The instance runs Auto DevOps for any project without a pipeline file (chapter 18).
  • The runner adds variables, a default image and an AWS role to every job it runs (chapter 29).
  • The ECR push role trusts branches only, so a tag pipeline can't push (chapter 38). The web role trusts web-portal's main and nothing else.
  • The runner's role lets every job on that runner pull from ECR without logging in (chapter 28).
  • Azure trusts one exact pipeline per federated credential: payments-api's main, with no wildcards (chapter 38).
settingsinstance.yml
# _settings/instance.yml
# This file DESCRIBES settings made in the GitLab UI of MegaCorp's self-managed
# instance, gitlab.example.com. GitLab does not read it: it exists so the book can
# show what those settings are and where they bite.
# Used by the chapters "Organisation-wide reuse: settings and policies", "Variables and
# precedence" and "Runners and executors".

# catalog: instance-group-variables
# Every project on the instance receives these, unless a group or project
# variable with the same name takes precedence.
instance_variables:
  - key: MEGACORP_REGISTRY
    value: registry.example.com
  - key: HTTPS_PROXY
    value: http://proxy.example.com:3128

# catalog: instance-template-repo
# Self-managed only: the .yml files in this project's gitlab-ci/ folder appear in the
# Web Editor's template list when anyone creates a CI/CD file. Choosing one copies it
# into the new file; include: template cannot use them.
instance_template_repository:
  project: megacorp/devops/instance-templates
  offers:
    - gitlab-ci/MegaCorp-Legacy-Java.yml

# catalog: auto-devops
# A project with no CI/CD configuration file gets GitLab's Auto DevOps pipeline.
auto_devops:
  default_to_auto_devops: true
settingsgroup-megacorp.yml
# _settings/group-megacorp.yml
# This file DESCRIBES settings made in the GitLab UI on the top-level group
# "megacorp". GitLab does not read it.
# Used by the chapters "Organisation-wide reuse: settings and policies", "Variables and
# precedence", "Policies and gates" and "GitLab, AWS and Azure".

# catalog: instance-group-variables
# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
  - key: MAVEN_CLI_OPTS
    value: "--batch-mode --errors --show-version -s .m2/settings.xml"
  - key: AWS_ACCOUNT_ID
    value: "123456789012"
  - key: MC_AZURE_TENANT_ID
    value: "00000000-0000-0000-0000-000000000000"
  - key: MC_ACR_REGISTRY
    value: megacorpprod.azurecr.io
  - key: MC_ACR_PUSH_CLIENT_ID
    value: "00000000-0000-0000-0000-000000000000"
    # the gitlab-acr-push application in Entra ID, which mc_acr_push signs in as
  - key: SONAR_HOST_URL
    value: https://sonar.example.com
  - key: SONAR_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true

# catalog: project-templates
# New projects can be created from these. The copy happens once, at creation.
group_project_templates:
  source_group: megacorp/templates
  templates:
    - megacorp/templates/java-service-starter
    - megacorp/templates/batch-job-starter

# catalog: compliance-pipeline
# Legacy: a compliance framework with its own pipeline configuration, the
# predecessor of pipeline execution policies.
compliance_frameworks:
  - name: SOX
    pipeline_configuration: compliance/sox.yml@megacorp/security/policies
    applied_to:
      - megacorp/payments/payments-api
runnersconfig.toml
# _runners/config.toml
# An excerpt of the GitLab Runner configuration on MegaCorp's shared runners.
# Projects cannot see this file, yet everything in it reaches their jobs.
# Used by the chapters "Reuse outside the YAML", "Identity and tokens" and "Runners and
# executors".

concurrent = 20

# catalog: runner-config
[[runners]]
  name = "megacorp-eks-shared"
  # registered in GitLab as a group runner of megacorp, with the tag megacorp-shared
  url = "https://gitlab.example.com"
  executor = "kubernetes"
  environment = ["MAVEN_OPTS=-Xmx2g", "HTTPS_PROXY=http://proxy.example.com:3128"]
  pre_build_script = "echo 'runner: megacorp-eks-shared'"
  [runners.kubernetes]
    namespace = "gitlab-runners"
    image = "registry.example.com/megacorp/devops/ci-tools:3.2"
    # jobs run as this service account, which is bound to an AWS IAM role
    service_account = "gitlab-runner-jobs"
awsiam.yml
# _aws/iam.yml
# This file DESCRIBES how MegaCorp's AWS account 123456789012 trusts GitLab. AWS holds
# the real configuration, as JSON policies. GitLab does not read this file.
# Used by the chapters "Identity and tokens" and "GitLab, AWS and Azure".

# AWS accepts ID tokens signed by this GitLab, for this audience.
oidc_provider:
  arn: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
  url: https://gitlab.example.com
  audiences: [https://gitlab.example.com]

# Assumed by image-build, and by the ecr-push component, through mc_aws_login.
ecr_push_role:
  arn: arn:aws:iam::123456789012:role/gitlab-ecr-push
  trust_policy:
    Effect: Allow
    Principal:
      Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
    Action: sts:AssumeRoleWithWebIdentity
    Condition:
      StringEquals:
        gitlab.example.com:aud: https://gitlab.example.com
      StringLike:
        gitlab.example.com:sub: project_path:megacorp/*:ref_type:branch:ref:*
  permissions: push and pull images in every ECR repository in the account

# Assumed by web-portal's deploy-site job.
web_publish_role:
  arn: arn:aws:iam::123456789012:role/gitlab-web-publish
  trust_policy:
    Effect: Allow
    Principal:
      Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
    Action: sts:AssumeRoleWithWebIdentity
    Condition:
      StringEquals:
        gitlab.example.com:aud: https://gitlab.example.com
        gitlab.example.com:sub: project_path:megacorp/web/web-portal:ref_type:branch:ref:main
  permissions: write to the www.example.com bucket, and invalidate its CloudFront cache

# The role behind the gitlab-runner-jobs service account on the megacorp-eks-shared
# runner. Every job on that runner can use it without asking. The EKS cluster is what
# AWS trusts for this role, not GitLab.
runner_role:
  arn: arn:aws:iam::123456789012:role/gitlab-runner-jobs
  permissions: pull images from every ECR repository in the account
azureentra.yml
# _azure/entra.yml
# This file DESCRIBES how MegaCorp's Azure tenant trusts GitLab, and what MegaCorp's
# pipelines may do in Azure. Entra ID and Azure hold the real configuration. GitLab does
# not read this file. Tenant and subscription 00000000-0000-0000-0000-000000000000 are
# placeholders. Used by the chapters "GitLab, AWS and Azure" and "AWS and Azure for
# delivery".

# The identity that image jobs sign in as. Each federated credential names one kind of
# pipeline that may use it.
acr_push_app:
  display_name: gitlab-acr-push
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: 00000000-0000-0000-0000-000000000000
  federated_credentials:
    - name: payments-api-main
      issuer: https://gitlab.example.com
      subject: project_path:megacorp/payments/payments-api:ref_type:branch:ref:main
      audiences: [https://gitlab.example.com]

# What the app may do once signed in, on one registry only. AcrPush pushes and pulls
# images. az acr login also has to look the registry up in Azure Resource Manager, which
# AcrPush doesn't allow; Microsoft documents the second role for that, and it can also
# change or delete the registry, which is one more reason to trust main only.
role_assignments:
  - principal: gitlab-acr-push
    role: AcrPush
    scope: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-delivery-prod/providers/Microsoft.ContainerRegistry/registries/megacorpprod
  - principal: gitlab-acr-push
    role: Container Registry Contributor and Data Access Configuration Administrator
    scope: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-delivery-prod/providers/Microsoft.ContainerRegistry/registries/megacorpprod

security/policies: rules no project shows#

MegaCorp's security policy project. Nothing here appears in any project's configuration, and no toggle stops it (chapter 36). Worth noticing:

  • Secret detection runs in every pipeline, on every branch.
  • The guardrails policy adds policy-sbom to every pipeline, in the reserved stage .pipeline-policy-post.
  • The approval policy doesn't set fallback_behavior, so it fails closed. A merge request whose required scans produced no report needs the security team's approval too (chapter 36).
megacorp/security/policies.gitlab/security-policies/policy.yml
# .gitlab/security-policies/policy.yml in megacorp/security/policies
# MegaCorp's security policy project, linked to the top-level group. These rules
# reach every project without appearing in any project's CI/CD configuration.
# Used by the chapters "Organisation-wide reuse: settings and policies", "Tracing a job to
# its source", "GitLab's own scanners" and "Policies and gates".

# catalog: scan-execution-policy
scan_execution_policy:
  - name: Secret detection everywhere
    description: Run secret detection in every pipeline on every branch.
    enabled: true
    rules:
      - type: pipeline
        branches: ["*"]
    actions:
      - scan: secret_detection
  - name: Nightly dependency scan
    description: Scan dependencies on the default branch every night.
    enabled: true
    rules:
      - type: schedule
        branches: [main]
        cadence: "0 2 * * *"
    actions:
      - scan: dependency_scanning

# catalog: pipeline-execution-policy
pipeline_execution_policy:
  - name: MegaCorp guardrails
    description: Add an SBOM job to every pipeline.
    enabled: true
    pipeline_config_strategy: inject_policy
    content:
      include:
        - project: megacorp/security/policies
          file: pipeline-policies/megacorp-guardrails.yml

# catalog: mr-approval-policy
approval_policy:
  - name: Block new critical vulnerabilities
    description: A merge into main that adds a critical finding needs security's approval.
    enabled: true
    rules:
      - type: scan_finding
        branches: [main]
        scanners: [sast, secret_detection, dependency_scanning, container_scanning]
        vulnerabilities_allowed: 0
        severity_levels: [critical]
        vulnerability_states: [new_needs_triage]
    actions:
      - type: require_approval
        approvals_required: 1
        group_approvers: [megacorp/security]
megacorp/security/policiespipeline-policies/megacorp-guardrails.yml
# pipeline-policies/megacorp-guardrails.yml in megacorp/security/policies
# The CI/CD configuration that the "MegaCorp guardrails" pipeline execution
# policy injects into every project's pipeline. Used by the chapters "Organisation-wide
# reuse: settings and policies", "Tracing a job to its source" and "Policies and gates".

policy-sbom:
  stage: .pipeline-policy-post
  image: registry.example.com/megacorp/devops/ci-tools/syft:1
  script:
    - syft dir:. -o cyclonedx-json=sbom.cdx.json
  artifacts:
    reports:
      cyclonedx: sbom.cdx.json

legacy/billing-batch: the older ways#

Created in 2019 from a starter template, and never updated. Worth noticing:

  • Anchors and merge keys do the work of extends, and only does the work of rules (chapter 13).
  • Its include follows another project's main by URL, so any change there reaches it at once (chapter 12).
  • test downloads a script and runs it. What it does is in no file of the project (chapter 19).
  • notify-reports starts a pipeline in project 4242 with a trigger token (chapter 17).
megacorp/legacy/billing-batch.gitlab-ci.yml
# .gitlab-ci.yml in megacorp/legacy/billing-batch
# A 2019-era project, created from a starter template and never updated since.
# It shows the older ways configuration was shared.
# Used by the chapters "YAML-level reuse: anchors, merge keys and hidden jobs", "include:
# pulling in files", "Pipelines as building blocks" and "Reuse outside the YAML".
# catalog: project-templates

# catalog: include-remote
include:
  - remote: https://gitlab.example.com/megacorp/devops/ci-templates/-/raw/main/legacy/notify.yml

# catalog: yaml-anchors, yaml-merge-key
.defaults: &defaults
  image: registry.example.com/megacorp/devops/ci-tools:2.9
  tags: [megacorp-shared]
  only:
    - branches
    - tags

build:
  <<: *defaults
  stage: build
  script:
    - ./gradlew assemble

# catalog: fetched-scripts
test:
  <<: *defaults
  stage: test
  script:
    - curl -sSL https://gitlab.example.com/megacorp/devops/scripts/-/raw/main/run-tests.sh | bash

# catalog: trigger-api
notify-reports:
  <<: *defaults
  stage: deploy
  only: [tags]
  script:
    - >
      curl --fail --request POST
      --form "token=$REPORTS_TRIGGER_TOKEN"
      --form "ref=main"
      --form "variables[BILLING_VERSION]=$CI_COMMIT_TAG"
      "https://gitlab.example.com/api/v4/projects/4242/trigger/pipeline"

Glossary and former names#

Appendix · Appendices · Chapter 69 · 12 min read

First, the features GitLab has renamed or replaced. When an older template, an older page or a colleague uses a name you can't find, look it up here. Then, every term this book uses, in one line each, with a link to where it's explained.

Former names#

You may meetToday, at 19.3When it changed
scan result policies, scan_result_policymerge request approval policies, approval_policyonly approval_policy is accepted since 17.0
newly_detected, in a policy's vulnerability_statesnew_needs_triage and new_dismissed17.0
compliance pipelinespipeline execution policiesdeprecated in 17.3; removal planned for 20.0
inject_ci, in a pipeline execution policyinject_policy17.9; inject_ci is deprecated
CI_JOB_JWT, CI_JOB_JWT_V2ID tokens, requested with id_tokensremoved in 17.0
Token Access, in Settings › CI/CDJob token permissions17.2
Limit access to this project, then Authorized groups and projectsCI/CD job token allowlist17.2, then 17.3
Merge when pipeline succeedsauto-merge–
CI/CD StepsGitLab Functions, an experiment–
only, exceptrulesdeprecated
image, services, cache, before_script or after_script at the top of the filethe same keywords inside default:deprecated
Security/SAST.gitlab-ci.yml and the other Security/ templateswrappers: each only includes its Jobs/ twin, such as Jobs/SAST.gitlab-ci.yml (chapter 34)–
Gemnasium, the dependency scanning analyzerdependency scanning using an SBOMGemnasium deprecated in 17.9; removal proposed for 20.0
Code Quality's CodeClimate templateany tool's results, imported as a codequality reportthe template is deprecated
kaniko, for building imagesDocker, Buildah, Podman or rootless BuildKitGitLab's page marks kaniko removed

Terms#

TermMeansExplained in
ACRAzure Container Registry, where MegaCorp keeps the images that aks-prod runschapter 38
Application, ApplicationSetArgo CD's objects: an Application says what to deploy, from where and to where; an ApplicationSet makes many from one templatechapter 45
Argo CDa GitOps controller with a web UI, which syncs Applications from Git into a clusterchapter 41
charta Helm package: templates plus default values, which render Kubernetes objectschapter 43
controllera program in the cluster that watches objects, and changes the cluster to match themchapter 41
Entra IDMicrosoft's identity service for Azure, formerly Azure Active Directorychapter 38
environment repositorythe Git repository that says what runs in each environment: image tags and settingschapter 45
External Secrets Operatora controller that copies secrets from a cloud's vault into Kubernetes Secretschapter 50
federated credentiala rule on an Entra ID application or managed identity that says which outside tokens it accepts, matched exactlychapter 38
Fluxa GitOps tool made of controllers, configured with objects such as GitRepository, Kustomization and HelmReleasechapter 45
Kustomize, overlaya tool that lays changes over a base set of objects, one overlay for each environment or clusterchapter 44
namespacea named group of objects inside a cluster, with its own permissions and quotaschapter 54
OpenTelemetrythe common standard for producing logs, metrics and traceschapter 52
promotionmoving a version from one environment to the nextchapter 45
readiness probethe check a new pod must pass before Kubernetes sends it traffic, or counts it in a rolloutchapter 51
!referencea YAML tag that pastes one section of another job, such as its rules or scriptchapter 14
agent for KubernetesGitLab's program inside a cluster, which connects the cluster to GitLabchapter 45
anchor, alias, merge keyYAML's own copy and paste, with & and *; it works only inside one filechapter 13
artifactfiles a job keeps when it ends, for later jobs and for peoplechapter 31
AssumeRoleWithWebIdentitythe AWS call that swaps an ID token for temporary AWS keyschapter 38
Auto DevOpsGitLab's built-in pipeline, for projects with no pipeline file when the setting is onchapter 18
cachefiles kept between jobs to save time; they may be missing at any timechapter 31
child pipelinea pipeline that a trigger job starts from a file in the same projectchapter 17
CI Linta tool that checks a piece of pipeline YAMLchapter 4
componenta versioned piece of configuration with typed inputs, included with include: componentchapter 16
container registrya store for a project's container images, under Deploy › Container Registrychapter 8
container scanninga scanner that looks for known vulnerabilities in an imagechapter 33
default:settings every job gets unless it sets its ownchapter 14
dependency scanninga scanner that looks for known vulnerabilities in the libraries a project useschapter 33
deployment joba job with environment:, which records what it deployed, and wherechapter 40
Docker-in-Dockera Docker engine that runs as a service beside a job, so that the job can build imageschapter 8
dotenv reporta file of NAME=value lines that a job leaves as variables for later jobschapter 31
downstream pipelinea pipeline that another pipeline started: a child or a multi-project pipelinechapter 17
dynamic child pipelinea child pipeline whose YAML an earlier job wrotechapter 17
ECRAmazon Elastic Container Registry, where AWS keeps container imageschapter 38
environmenta place a job deploys to, such as staging, with its history of deploymentschapter 40
executorhow a runner runs jobs: in Docker containers, in Kubernetes pods, in a shell, and otherschapter 29
extendscopies another job's settings: maps are merged, lists are replacedchapter 14
Full configurationthe pipeline editor's view of every file, merged into onechapter 4
GitOpsdeploying by writing the wanted state to a Git repository, which a tool in the cluster applieschapter 41
hidden joba job whose name starts with a dot; it never runs, and exists to be copiedchapter 13
IAM role, trust policyan AWS identity a job can take on; its trust policy says who maychapter 38
ID tokena note about the job, signed by GitLab, that another system such as AWS can checkchapter 28
includecopies other YAML files into the configurationchapter 15
inputa typed parameter of a component, an included file or a pipeline, filled in when the pipeline is createdchapter 16
instance, group and project runnersrunners for every project, for one group's projects, or for chosen projectschapter 29
jobone piece of work in a pipeline, run by one runnerchapter 4
job tokenCI_JOB_TOKEN, which lets a job call GitLab while it runschapter 28
masked, hidden, protectedhow a variable is guarded: masked in logs, hidden in the settings, passed only to protected branches and tagschapter 27
merge request approval policya security policy that requires approval when scans find problemschapter 36
merge request pipelinea pipeline for a merge request, rather than for a branchchapter 21
merge traina queue of merge requests, each tested together with those ahead of itchapter 21
merged results pipelinea merge request pipeline that runs on the source branch merged into the targetchapter 21
multi-project pipelinea pipeline in another project, started by a trigger jobchapter 17
needsstarts a job as soon as the named jobs finish, ignoring stageschapter 24
OIDCOpenID Connect, the standard that ID tokens followchapter 28
OpenTofuan open-source tool that creates and changes infrastructure from code; it began as a copy of Terraformchapter 10
pipelineall the jobs GitLab runs for one commit, for one reason such as a pushchapter 4
pipeline execution policya security policy that adds jobs to projects' pipelineschapter 36
pipeline sourcewhat started a pipeline, held in CI_PIPELINE_SOURCEchapter 21
plan, applyOpenTofu's two steps: work out what would change, then change itchapter 10
predefined variablea variable GitLab sets, such as CI_COMMIT_BRANCHpredefined variables
protected branch, protected tagbranches and tags that only some people may push to; only their pipelines get protected variableschapter 27
resource_grouplets only one job of the group run at a timechapter 24
rulesdecide whether a job is in a pipeline, and how it runschapter 23
runnerthe program that takes jobs from GitLab and runs themchapter 29
SASTstatic application security testing: a scanner that reads source codechapter 33
SBOMsoftware bill of materials: a list of every component in a buildchapter 33
scan execution policya security policy that adds scanner jobs to pipelines, or runs them on a schedulechapter 36
secret detectiona scanner that looks for passwords and keys committed to the repositorychapter 33
security policy projectthe project that holds a group's security policieschapter 36
servicean extra container beside the job, such as a databasechapter 30
settings variablea variable set in the CI/CD settings of a project, a group or the instancechapter 26
shallow clonea copy of the repository with only the newest commitschapter 32
signalsa running service's logs, metrics and traceschapter 52
stagea group of jobs; by default, a stage starts only when the stage before it has succeededchapter 24
stateOpenTofu's record of what it has made, which GitLab can keep for a projectchapter 10
state, state lockOpenTofu's record of what it created, and the lock that stops two runs at oncechapter 48
synca controller making the cluster match what Git sayschapter 45
taga Git tag names a commit; a runner tag is a label that matches jobs to runnerschapter 29
templateYAML that GitLab ships, included with include: templatechapter 15
togglea variable a template checks to switch a job off, such as SAST_DISABLEDchapter 16
trigger joba job that starts a downstream pipeline instead of running a scriptchapter 17
truth tablethis book's method for predicting which jobs run in which pipelinechapter 25
values filea Helm chart's settings for one service or environmentchapter 43
workflow:rulesdecide whether a pipeline is created at allchapter 22
workload identitya pod's own cloud identity, taken from its Kubernetes service accountchapter 49

End of the book