GitLab CI/CD, Decoded#
Read, trace and fix any enterprise pipeline
Preface#
Your team's pipeline file is twenty lines long. The pipeline it produces is not, and keeping it working is now your job.
In a large organisation, a central team writes shared CI/CD templates, and each team writes its own pipeline file on top of them. Your file includes a template, which includes eight more. A job inherits from a job that inherits from another, in a repository you have never opened. Variables arrive from settings pages nobody on your team can see. When something breaks, the answer is often not in the file in front of you. And when your team needs a new job, a new rule or a deploy, you are the one who writes it.
GitLab's documentation explains each feature on its own. This book teaches you to do two things with a real pipeline. You learn to write and change your team's own file. You also learn to read everything the central team adds to it: to take any job and find where it came from, why it ran, what it was given and what it did. Everything is taught on MegaCorp, a fictional company whose GitLab is as tangled as the real ones, and on small practice projects that you build on your own GitLab.
Who it is for#
You are a developer who looks after your team's pipeline. Some of it comes from a central team's templates, and the rest your team writes. The book assumes that you know what CI/CD is, can use a terminal, and can create a project on a GitLab: GitLab.com's free tier, or your organisation's own. It assumes nothing more:
- You don't need to have written a
.gitlab-ci.yml, or any YAML. - You don't need to have used GitLab.
- You don't need to have run a container.
Every term is explained where it first appears. The short chapter Before you start gives you the Git and GitLab words, enough YAML to read any pipeline file, and a first pipeline, in about ten minutes. Your first pipeline then has you run it on your own GitLab. Wherever an idea has parts, an order or a decision in it, a diagram shows it.
What reading it gets you#
| Level | You can | What gets you there |
|---|---|---|
| 0 · Zero | little: the YAML looks like noise | |
| 1 · Vocabulary | say what every keyword does | Part II and Part IV |
| 2 · Recognition | open any pipeline and name the patterns in it | the rest of the book, read once |
| 3 · Tracing | take any job back to where it was defined, why it ran and what it was given | the exercise at the end of each chapter from Part IV to Part X |
| 4 · Diagnosis | go from a symptom to its cause in minutes | the worksheet on your own pipeline, then a few real incidents with the symptom cards open |
| 5 · Writing and maintaining | write and change your team's pipeline, and know what each change will do | the labs in Part III, and the "When you write it" boxes |
| 6 · Template-library design | design templates for a whole organisation | not this book's goal |
Level 5 is the goal. It rests on level 4, because you can't safely change a pipeline that you can't diagnose. Reading gets you to level 2, and the exercises to level 3. The worksheet, which you fill in privately against your own organisation's pipeline, and the symptom cards take you to level 4. The labs take you to level 5. The final test tells you when you're there.
How to read it#
There are three paths, and the contents shows what each one costs on a clock set for a technical reader: prose at 150 words a minute, code at 80, a minute for each diagram.
| Path | What it is | Time |
|---|---|---|
| The Handbook | Part I: every keyword on one annotated project, every word explained in one line, and the card | about 58 min of reading |
| The Pipeline textbook | from Part II to Part XI, in order, with its labs and exercises | 7 h 59 min of reading, and 2 h 13 min of doing |
| After the pipeline | Part X: from the image in the registry to software running in a cluster | 2 h 27 min of reading, and 1 h 38 min of doing |
Read the Handbook first, whoever you are. If you have a broken pipeline in front of you, its last chapter, the card, sends you to the right symptom card. If your team's pipeline has just become yours, go on to Before you start and Part III, doing each lab. Then read the rest in order.
There is a third way in: press ⌘K and type what you see. A symptom
("blocked"), a keyword (!reference), a variable or an error message takes
you to the answer. Deep dives, legacy asides, exercises and labs fold and unfold one at a
time; printing opens them all.
Conventions#
- PremiumUltimate The lowest GitLab tier that has the feature. No tier badge means every tier has it.
- Self-managedGitLab.com The feature exists only on that offering.
- ExperimentalBetaDeprecated GitLab's own status for the feature.
- Since 17.11 The release that introduced it. If your organisation runs its own GitLab, check which version it is on.
Examples come in a few shapes. A You wrote pane shows configuration as it sits in its file. A GitLab builds pane beside it shows what GitLab made of it, marked captured from GitLab when it is GitLab's own output. A symptom card goes from what you see to the cause, how to confirm it and the fix. A spot the bug exercise ends every chapter from Part IV to Part X. A lab is something to do on your own GitLab, with its steps and what you should see. A When you write it box says what to write, why, and what to watch out for, when you write that mechanism in your team's file.
What it targets#
| Scope | Baseline | Note |
|---|---|---|
| GitLab | GitLab 19.3 | features newer than this are left out; recent ones carry a since badge |
| Offerings | GitLab.com and self-managed | differences are called out where they exist |
| Labs | GitLab.com's Free tier, or your own GitLab | no lab needs a paid feature or a cloud account |
| Cloud | Worked examples on AWS and Azure | the ideas transfer; the commands are AWS's and Azure's, side by side |
| Delivery tools | Kubernetes 1.36, Helm 4.3, Kustomize 5.8, Flux 2.9, Argo CD 3.5, OpenTofu 1.12 and External Secrets Operator 2.10 | MegaCorp's charts, overlays, manifests and modules were rendered and validated with these versions |
On accuracy#
GitLab releases every month, and search results do not sort by version. Every version-sensitive claim here was checked against GitLab's documentation. Every "GitLab builds" pane marked as captured is real output from a GitLab 19.3 instance, loaded with MegaCorp's projects exactly as this book shows them. Every lab was run end to end on that instance, and what each lab says you should see is what it printed. Where a behaviour needs a tier that instance doesn't have, the book says described, not captured.
It will still age. When this book and GitLab's documentation disagree, the documentation is right.
What it does not cover#
| Not covered | Why |
|---|---|
| Designing an organisation's template library | you learn why central teams build what they build, and how to use it in your own file, not how to design one |
| Running GitLab | instance settings, upgrades and runner fleets belong to administrators; you see what their choices do to your jobs |
| Triaging security findings | you learn what each scanner does and why it blocks you, not whether a vulnerability matters |
| Other CI systems | there are no comparisons with GitHub Actions, Jenkins or others |
| Other clouds | the worked examples use AWS and Azure; Google Cloud and the rest are left out, and the ideas carry over |
| Operating clusters and controllers | you read and change what your service needs: values, overlays and manifests; installing and upgrading Kubernetes, Flux or Argo CD belongs to the platform team |
| Choosing an observability vendor | products are named where they fit, not compared |
| GitLab beyond pipelines | Pages, wikis, planning and the AI features appear only where a pipeline touches them |
Before you start#
This book assumes you know what CI/CD is: code gets built, tested and shipped automatically whenever it changes. It assumes nothing else. This short chapter gives you the three things the rest of the book leans on:
- the Git and GitLab words
- enough YAML to read any pipeline file
- a first pipeline, from a push to a result
If you already know all three, skip to chapter 4.
Git and GitLab words#
Git keeps the history of a project's files as a chain of saved changes. GitLab hosts that history and runs pipelines on it. Every word below appears on almost every page of this book:
| Word | What it is | Why a pipeline cares |
|---|---|---|
| commit | one saved change to the files, with an ID like 4f2c9a1, called its SHA | every pipeline runs for exactly one commit |
| branch | a named line of commits, such as feature/login | pushing to a branch starts a branch pipeline |
| default branch | the main line of the project, usually called main | deployments usually start from it |
| push | sending your new commits from your computer to GitLab | the most common way a pipeline starts |
| merge request | a request to merge one branch into another, with review and discussion; other tools call it a pull request | GitLab can run a pipeline for it, called a merge request pipeline |
| merge | combining a branch's commits into another branch | merging to main usually starts the pipeline that deploys |
| tag | a permanent name for one commit, usually a release number such as v1.4.0 | pushing a tag starts a tag pipeline, often the one that releases |
| protected branch or tag | one that only chosen people may push to | secrets and deploy permissions are often limited to protected branches |
| project | GitLab's home for one repository, with its settings, pipelines and merge requests | a pipeline belongs to a project |
| group | a folder of projects, which can contain further groups | settings made on a group, such as CI/CD variables, reach every project inside it |
YAML in five minutes#
A pipeline is described in a file called .gitlab-ci.yml, written in YAML.
YAML is a way of writing data as indented text. Almost everything you will see is one
of these few shapes:
# A comment: YAML ignores everything after #
name: payments-api # a key and its value
retries: 2 # a number
enabled: true # true or false
owner: # a map: the keys indented under "owner" belong to it
team: payments
email: payments@example.com
stages: # a list: one item per line, each starting with "-"
- build
- test
tags: [shared, linux] # the same kind of list, written on one line
script: | # a block of text, kept line by line
echo "first line"
echo "second line"Four rules explain nearly every YAML mistake you will meet:
- Indentation is meaning. Keys indented under another key belong to it. Use spaces, never tabs; two spaces per level is the convention.
- A dash starts a list item.
- buildis one item in a list. A key followed by a colon, such asstage:, starts a key in a map. - Numbers and true/false are not text.
21is read as a number andtrueas a yes-or-no value. When you mean text, put it in quotes:JAVA_VERSION: "21". GitLab's variables are always text, which is why you will see so many quotes in pipeline files. - Some characters change the meaning. A value that starts with
*,&,!,[or{, or contains a colon followed by a space, means something special to YAML. Put such values in quotes when you mean plain text. Chapter 13 explains what the special ones do.
Numbers can even change as they are read. On the local GitLab 19.3, a variable
written VERSION: 1.10 without quotes reached a rule as 1.1, so
$VERSION == "1.10" never matched. Written as "1.10", it
did.
In a .gitlab-ci.yml, every top-level key is one of two things. It is
either a GitLab keyword that sets something for the whole pipeline, such as
stages, or the name of a job. Here is the shape of the pipeline file you are
about to read:
Your first pipeline file#
This is a complete .gitlab-ci.yml for a small Node.js project. It has two
jobs: one builds the code, and one tests it.
stages:
- build
- test
build-app:
stage: build
image: node:24.21.0-alpine
script:
- npm ci
- npm run build
unit-tests:
stage: test
image: node:24.21.0-alpine
script:
- npm ci
- npm test| Line | What it means |
|---|---|
stages: | the stages of this pipeline, in the order they run: first build, then test |
build-app: | a job. Its name is up to you, as long as it isn't a GitLab keyword |
stage: build | this job belongs to the build stage |
image: node:24.21.0-alpine | the job runs inside a container started from the node:24.21.0-alpine image, which has Node.js 24 installed |
script: | the commands the job runs, in order. If one fails, the job stops and fails |
unit-tests: | a second job, in the test stage, with the same image and its own commands |
An image is a packaged set of tools and files. A
container is a running copy of it, fresh for every job and thrown away
afterwards. So nothing a job does on its own disk reaches the next job, unless the job
saves it on purpose (chapter 31). That is why
both jobs above run npm ci.
When you push a commit, GitLab turns this file into a pipeline. The pipeline graph shows each stage as a column and each job as a box:
As text
- build: build-app
- test: unit-tests
Here is what happens between your push and the result:
As text
- You push a commit to GitLab
- GitLab reads .gitlab-ci.yml at that commit
- GitLab creates a pipeline with its stages and jobs
- A runner takes the next job and starts a container from the job's image
- The runner runs the job's script, one command at a time
- Did every command succeed? No: The job fails, and later stages don't start. Yes: the next step.
- The next stage starts. When every job has passed, the pipeline passes
A runner is a program, running somewhere your organisation chose, that asks GitLab for jobs and runs them. Chapter 4 takes each of these words further.
Every pipeline in this book is built from the same pieces: keywords, jobs, stages, images and scripts. The difficulty in a large organisation is not the pieces. It is that a real pipeline is assembled from many files, settings and teams you can't see from your own project. Finding those, and reading them, is what the rest of the book teaches.
Reading about a pipeline only goes so far. The next chapter, Your first pipeline, has you run this file on your own GitLab, break it three ways and fix it, in about a quarter of an hour.
Your first pipeline#
Before you start showed you a pipeline file. Now you run it on your own GitLab, then break it on purpose, three ways, so that you know what each kind of mistake looks like. You need about twenty minutes, a terminal with Git, and an account on a GitLab where you can create a project. GitLab.com's Free tier works, and so does your organisation's own GitLab.
The project#
The project is a tiny Node.js one. It has a function that builds a greeting, a test
for it, and a build step that copies the source into dist/, the way a real
build would bundle it. It has no dependencies, so npm ci has nothing to
download. These are its files; each pane has a copy button.
{
"name": "first-pipeline",
"version": "1.0.0",
"private": true,
"description": "The project of the first-pipeline lab: a greeting, built and tested.",
"type": "module",
"scripts": {
"build": "node build.js",
"test": "node --test"
}
}{
"name": "first-pipeline",
"version": "1.0.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "first-pipeline",
"version": "1.0.0"
}
}
}// The build: copies src/ into dist/, the way a real build would bundle it.
import { cpSync, mkdirSync } from "node:fs";
mkdirSync("dist", { recursive: true });
cpSync("src", "dist", { recursive: true });
console.log("built dist/greet.js and dist/cli.js");// The first-pipeline lab's one function.
export function greet(name) {
return `Hello, ${name}!`;
}// Prints a greeting: node src/cli.js <name>
import { greet } from "./greet.js";
console.log(greet(process.argv[2] ?? "world"));// The first-pipeline lab's test, run with node --test.
import { test } from "node:test";
import assert from "node:assert/strict";
import { greet } from "../src/greet.js";
test("greet says hello", () => {
assert.equal(greet("GitLab"), "Hello, GitLab!");
});LabRun itabout 8 min#
An account on a GitLab where you can create a project, a terminal with Git, and the six files above.
- In GitLab's upper-right corner, select Create new (the plus sign),
then New project/repository, then Create blank project.
Name it
first-pipeline, leave everything else as it is, and select Create project. - Clone the project with the address under its Code button. Copy the six files into the folder, keeping their paths.
- Add a file named
.gitlab-ci.yml, with the pipeline from Before you start:practice · first-pipelinesteps/01-first.ymlpipelinestages: - build - test build-app: stage: build image: node:24.21.0-alpine script: - npm ci - npm run build unit-tests: stage: test image: node:24.21.0-alpine script: - npm ci - npm test - Commit and push:
git add . git commit -m "Add the first pipeline" git push - In GitLab, open Build › Pipelines, then the newest pipeline, then its
build-appjob.
The pipeline passes: build-app in the build stage, then
unit-tests in the test stage. The job's log shows what the runner did, then
each of your commands and what it printed. Here it is, cut to its main lines:
Running with gitlab-runner 19.3.0 (9cbf0074)Using Docker executor with image node:24.21.0-alpine ...Getting source from Git repositoryExecuting "step_script" stage of the job script$ npm ci
up to date, audited 1 package in 5s$ npm run build
> first-pipeline@1.0.0 build
> node build.js
built dist/greet.js and dist/cli.jsJob succeededLabBreak it three waysabout 10 min#
The project from the previous lab, with its pipeline passing.
- Bad indentation. In
build-app, movescript:one space to the left, then commit and push:Look at the new pipeline, then put the space back and push again.practice · first-pipelinesteps/02-bad-indent.ymlbrokenbuild-app: stage: build image: node:24.21.0-alpine script: - npm ci - npm run build - A command that fails. Add a line to
unit-teststhat runs a script the project doesn't have, and push:Read the failed job's log, then take the line out again.practice · first-pipelinesteps/03-failing-command.ymlbrokenunit-tests: stage: test image: node:24.21.0-alpine script: - npm ci - npm run lint - npm test - A file from an earlier job. Add a job that runs what
build-appbuilt, and push:When it fails, makepractice · first-pipelinesteps/04-missing-file.ymlsmoke-testsmoke-test: stage: test image: node:24.21.0-alpine script: - node dist/cli.js "smoke test"build-appkeepdist/as an artifact, and push again:practice · first-pipelinesteps/05-artifacts.ymlbuild-appbuild-app: stage: build image: node:24.21.0-alpine script: - npm ci - npm run build artifacts: paths: - dist/
Bad indentation. The pipeline fails at once, with no jobs. Its badge says error, and its tooltip gives GitLab's reason: "The pipeline failed due to an error on the CI/CD configuration file." Paste the file into CI Lint (chapter 4) for the detail:
(<unknown>): did not find expected key while parsing a block mapping at line 1 column 1Line 1 is where GitLab began the map it couldn't finish, not the line you changed. Look further down, where the indentation shifts.
A command that fails. unit-tests fails. npm says what went
wrong, and the runner says why the job stopped:
$ npm ci$ npm run lint
npm error Missing script: "lint"ERROR: Job failed: exit code 1A file from an earlier job. smoke-test fails, because its
fresh container has no dist/:
$ node dist/cli.js "smoke test"Error: Cannot find module '/builds/practice/first-pipeline/dist/cli.js'ERROR: Job failed: exit code 1With the artifact, the job downloads build-app's files first, and
passes:
Downloading artifacts for build-app (192)...$ node dist/cli.js "smoke test"
Hello, smoke test!Job succeededThe third break is the one that surprises people most. Every job starts in a fresh container, so nothing a job leaves on its own disk reaches the next job unless the job keeps it on purpose. Chapter 31 covers artifacts, and their cousin, the cache.
LabA job with nowhere to runabout 4 min#
- Add a job that asks for a runner with a
gputag, and push:practice · first-pipelinesteps/06-stuck.ymlstucktrain-model: stage: test image: node:24.21.0-alpine tags: [gpu] script: - echo "no runner has the gpu tag, so this job waits" - Open
train-model's page from the pipeline. - Cancel the pipeline, then delete the job from the file and push.
The other jobs pass, but train-model stays pending. Its page says
why, and lists the tag that no runner has:
On your organisation's GitLab, runners have tags that the platform team chooses. A job that asks for any other tag waits like this (stuck on tags).
That is the whole engine: a push, a file GitLab reads, jobs in stages, containers started from images, and runners that take the jobs whose tags they have. Chapter 4 takes each of these words further, and Part III grows this into a real service's pipeline.
GitLab CI/CD in an hour#
One small project holds every keyword of a GitLab pipeline file, in the order this book teaches them, with the explanation written as comments on the lines. Each comment says what the keyword does, why you would use it, how it differs from the ones that look like it, and what an expert knows about it. Read it from the top and you can read, write and debug any pipeline file. Nothing here is typed into the book: every pane is quoted from the project, which runs on a real GitLab.
This is the dense reading. It assumes only that you know what CI/CD is, and it explains each word where it first appears, so you never have to jump ahead. The textbook after the Handbook covers the same ground as a story, with worked examples, labs and more depth. Each part below ends with a link to where that story starts.
A pipeline is what GitLab runs when you push: jobs, grouped in
stages, each job a script in a fresh container. Everything below is about the
file that describes it, .gitlab-ci.yml, and the few things that live outside
any file. Words in the comments that are new to you are one line each in
What is what.
1. The file#
Three top-level keys set up the whole pipeline. Everything else is a job.
# ---- 1. The file -----------------------------------------------------------
# When you push a commit, GitLab looks for a file named .gitlab-ci.yml at the
# root of your repository. What it finds there becomes a PIPELINE: a set of
# JOBS, each a small script that runs on a machine GitLab controls, called a
# RUNNER. A few top-level keys in this file are settings for the whole
# pipeline. Every other top-level key is the name of a job.
# `spec` is optional, and if you use it, it has to be the very first thing in
# the file, in its own YAML document that ends with a line of three dashes.
spec:
# "Inputs" are values that whoever starts the pipeline can choose, like the
# arguments of a function. Use them for the few things that change from one
# run to the next, such as which environment to deploy to. GitLab fills
# them in while it is reading this file, before any job exists, so an input
# can't hold something that is only known later, like a build number.
inputs:
greeting:
default: "hello" # the value used when nobody chooses one
description: "What the jobs say" # shown next to the field
type: string # or number, boolean, array
options: ["hello", "hi"] # the only values allowed
# regex: ^h # or a pattern the value must match
---
# The STAGES, in order. A stage is a step of the pipeline, such as "build" or
# "test". Every job belongs to one stage, and a stage starts only when every
# job in the stage before it has passed. That is how you say "test after
# building". Two stages always exist without being listed: `.pre` runs first
# and `.post` runs last.
stages:
- build
- test
- package
- deploy
# `default` holds settings that every job gets, unless a job sets its own.
# Put here what you would otherwise repeat in every job. (Older files put
# `image:` or `before_script:` directly at the top level instead; that is the
# old way of writing the same thing.)
default:
# Every job runs inside a CONTAINER: a fresh, throwaway Linux box. The IMAGE
# is the snapshot that box starts from, with the tools already installed.
# Always pin it to a version, so that the job behaves the same next month.
image: alpine:3.22
# Commands that run before the job's own script, in the same shell. Use it
# for setup that every job needs, such as logging in.
before_script:
- echo "before_script runs first, in every job"
# Commands that run after the script, in a separate shell, even when the
# script failed. Use it for clean-up, or for saving a report of a failure.
after_script:
- echo "after_script runs last, even when the script failed"
# If a newer commit is pushed while this job is waiting or running, GitLab
# may cancel it, because the newer pipeline will run it again anyway.
interruptible: true
# Run the job again if it fails: at most once, and only when the runner
# itself broke, not when the script returned an error. Retrying on every
# failure would hide real bugs.
retry:
max: 1
when: [runner_system_failure]
# exit_codes: [137] # or only for these exit codes
# Here is a job. Its name is the key, and you choose it. `stage` says when it
# runs, and `script` is the list of shell commands it runs, one after the
# other, in a fresh container that holds a fresh copy of the repository. If
# any command fails, the job fails.
say-hello:
stage: build
# Stop the job, as failed, if it runs longer than this. `timeout` is set per
# job; it is one of the few settings `default` can't hold.
timeout: 10m
script:
# An input from `spec` above, filled in while the file was being read.
- echo "$[[ inputs.greeting ]] from a job"
# VARIABLES are named values a job can read, written as $NAME. GitLab
# sets these two, and many more, for every job: they are called
# predefined variables. SHORT_SHA is the commit's id, and REF_NAME the
# branch or tag.
- echo "commit $CI_COMMIT_SHORT_SHA on $CI_COMMIT_REF_NAME"Deeper: Part II.
2. Reuse: where most of a real pipeline comes from#
In a large organisation your file is short, because most of its jobs are copied in from files a central team owns. Six keywords do the copying.
# ---- 2. Reuse: where most of a real pipeline comes from --------------------
# In a large organisation, your file is short, because most of your jobs are
# written once by a central team and copied into your pipeline. This part
# shows every way of copying, and when to use which:
# include brings whole FILES into this one, as if pasted here.
# extends copies a whole JOB (a template job) into one of yours.
# !reference copies one SECTION of a job, such as its list of commands.
# an anchor copies a piece of YAML, but only inside the same file.
# `include` pulls other YAML files into this one. GitLab reads them first,
# and joins everything into one big configuration, as if you had pasted the
# files here. Jobs in those files become your jobs, and template jobs in
# them become available to `extends` and `!reference` below.
include:
# `local`: a file in this same repository, at the same commit.
- local: ci/base.yml
# An include can carry `rules`: conditions that say when to include the
# file at all. This one says: only if the file exists.
rules:
- exists: [ci/base.yml]
# `project`: a file in another GitLab project. This is how a central team
# shares its templates. `ref` pins a tag or commit, so that the central
# team's next change can't alter your pipeline until you choose it.
- project: practice/central-templates
ref: 1.0.0
file: templates/node.yml
# `component`: a packaged, versioned job with typed inputs, published by a
# project. It differs from a plain `project` include in that you set its
# inputs here, and GitLab checks them. The address is the GitLab server,
# the project, the component's name, and its version after the @.
- component: $CI_SERVER_FQDN/practice/central-templates/secret-scan@1.0.0
inputs:
stage: test
# `template`: a file that GitLab itself ships, such as its scanners:
# - template: Jobs/Build.gitlab-ci.yml
# `remote`: a file at any URL. `integrity` pins its checksum, so that a
# changed file is refused:
# - remote: https://example.com/ci.yml
# integrity: sha256-L5/GAbeXGkPeEtIuOJhiJP+5xgrgdlwFEqm1O3EJwHM=
#
# What an expert knows about includes: GitLab joins the files in the order
# listed, and if two files define a job of the same name, the later one wins.
# Your own file is joined last, so it beats them all. To see the joined
# result, open Build > Pipeline editor > Full configuration: that view shows
# every job with its includes, `extends` and `!reference` already resolved,
# and it is where you trace any job you did not write.
# A YAML ANCHOR is plain YAML, not a GitLab feature: `&login` gives this list
# a name, and `*login` further down pastes a copy of it. Anchors are the
# oldest way of avoiding repetition, and you will meet them in older files.
# Their limit: an anchor only works inside the one file it is written in. It
# can't reach anything that came in through `include`. The two GitLab
# keywords below can, which is why they replaced anchors.
.login: &login
- echo "logging in"
# A job whose name starts with a dot is HIDDEN: GitLab never runs it. Hidden
# jobs exist to be copied. `.base-build` is one, written in ci/base.yml.
#
# `extends` copies a hidden job into this job, and then lays this job's own
# settings on top. Use it when a central team has written the job you need
# and you want to change one or two things. How the two are combined:
# - settings that are maps, such as `variables` and `cache`, are merged
# key by key, so you can add or change one key and keep the rest;
# - settings that are lists, such as `script`, `rules`, `before_script`,
# `tags` and `services`, are replaced whole: if you write one line of
# `script`, the template's whole script is gone.
# Experts also know: `extends` can chain, a job extending a job that extends
# another, across files, and GitLab advises three levels at most. To trace a
# chain, search the included files for each parent's name, one hop at a time.
build:
extends: .base-build
stage: build
# The anchor from above, pasted in. It works because both are in this file.
before_script: *login
script:
# `!reference` copies ONE section of another job, from any file, and
# pastes it exactly here. Compare it with `extends`: `extends` takes a
# whole job and merges it underneath yours; `!reference` takes one piece
# and puts it where you say. Use it to keep a template's list and add
# your own lines around it, which `extends` can't do for lists. Here it
# pastes the `make` list from `.snippets` in ci/base.yml.
- !reference [.snippets, make]
- echo "built at $CI_COMMIT_SHORT_SHA" > build.txt
# ARTIFACTS are files GitLab keeps when the job ends and hands to later
# jobs. Without this, build.txt would vanish with the container. Part 5
# shows all the settings.
artifacts:
paths: [build.txt]
# `parallel: matrix` turns one job into several, one per combination of the
# values, each with the variable set. GitLab names them "test: [unit]" and
# "test: [integration]". Use it to run the same job for several versions,
# platforms or suites without copying the job.
test:
stage: test
parallel:
matrix:
- SUITE: [unit, integration]
script:
- echo "running the $SUITE tests"
# A TRIGGER job has no script. It starts another pipeline and reports how that
# went. With `include`, the other pipeline is a CHILD: it is described by
# another file in this repository and shown nested inside this pipeline.
# Use a child pipeline to keep a big pipeline in several files, or to
# generate a pipeline from a script.
child:
stage: test
trigger:
include:
- local: ci/child.yml
# The child file has a `spec` with inputs of its own, filled in here.
inputs:
who: "the parent pipeline"
# `mirror` makes this job show the child's status: it waits for the child
# and fails if the child fails. Without it, the job passes the moment the
# child is created, whatever happens afterwards, so a broken child would
# show green. Older files say `strategy: depend`, which GitLab no longer
# recommends.
strategy: mirror
# Pass this pipeline's variables down to the child.
forward:
pipeline_variables: true
# The other kind of trigger: with `project`, GitLab runs ANOTHER PROJECT's
# own pipeline, called a multi-project pipeline. Use it to kick off a
# deploy repository, for example. Its inputs go under `trigger:` itself:
# trigger:
# project: practice/node-service
# inputs: { who: "another project" }The two files this one borrows from:
# A job whose name starts with a dot is HIDDEN: GitLab never runs it. It is a
# template, written so that other jobs can copy it with `extends`. This one
# lives in a separate file, which .gitlab-ci.yml brings in with `include`.
# A YAML anchor could never reach it from there; `extends` can.
.base-build:
image: alpine:3.22
cache:
key: base
paths: [.cache/]
policy: pull
# A hidden job can also hold keys that are not CI/CD keywords at all, like
# `make` here. They are snippets: pieces of YAML kept in one place, for jobs
# to paste with `!reference [.snippets, make]`. Central teams keep their
# shared script lines and rule lists this way.
.snippets:
make:
- echo "a make step, pasted from ci/base.yml with !reference"# A child pipeline is a complete pipeline of its own, described by this file
# and shown nested inside the pipeline that triggered it. It has its own
# `spec`, so the parent can pass it inputs.
spec:
inputs:
who:
default: "nobody" # the parent's trigger fills this in
---
child-job:
image: alpine:3.22
# Inside a child, $CI_PIPELINE_SOURCE is "parent_pipeline". A job with no
# rules runs only for branches and tags, and a merge request's pipeline is
# neither, so without this rule the child would have no jobs there, and
# GitLab would refuse to create it.
rules:
- if: $CI_PIPELINE_SOURCE == "parent_pipeline"
script:
- echo "hello from $[[ inputs.who ]], in a child pipeline"
# The parent's `forward: pipeline_variables` passed its variables down.
- echo "the parent forwarded LOG_LEVEL=$LOG_LEVEL"Deeper: Part IV, and chapter 20 for finding where any job came from.
3. Conditions: which pipelines exist, and which jobs are in them#
GitLab decides twice. First whether this push gets a pipeline at all, with
workflow. Then, for every job, whether it is in that pipeline, with its
rules. Both are lists of questions, asked in order, and the first yes
decides.
# ---- 3. Conditions: which pipelines exist, and which jobs are in them ------
# GitLab decides twice. First, does this push get a pipeline at all? That is
# `workflow`. Then, for every job, is it in this pipeline? That is the job's
# `rules`. Both are lists of conditions that GitLab tries from the top, and
# the first one that matches decides. Nothing else is looked at.
#
# The conditions use predefined variables. The ones you will see most:
# $CI_PIPELINE_SOURCE what caused the pipeline: "push", "schedule",
# "merge_request_event", "web" and others
# $CI_COMMIT_BRANCH the branch name, set only in a branch pipeline
# $CI_COMMIT_TAG the tag name, set only in a tag pipeline
# $CI_OPEN_MERGE_REQUESTS set when the branch has an open merge request
#
# The pattern experts use: put the cases to exclude first, each with
# `when: never`, then the cases to allow, and let everything else fall off
# the end, which means no pipeline. A workflow decides only whether the
# pipeline exists; it can't put a job in or out.
workflow:
# The pipeline's title in the pipeline list. Variables are filled in.
name: "$KIND pipeline for $CI_COMMIT_REF_NAME"
# When a newer commit is pushed, cancel the older pipeline's jobs that are
# marked `interruptible`, since the newer pipeline covers them.
auto_cancel:
on_new_commit: interruptible
# on_job_failure: all # or cancel everything when one job fails
rules:
# A MERGE REQUEST is a request to merge one branch into another, with a
# review. GitLab can run a pipeline for the merge request itself, and
# show the result on it. This rule says: create that pipeline.
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
variables:
# A rule can also set a variable for the whole pipeline.
KIND: "merge request"
# A push to a branch normally gets a branch pipeline too. While a merge
# request is open for that branch, the merge request pipeline already
# covers the push, so don't make a second one. Without this rule, every
# push would run everything twice.
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
# Any other push to a branch: create the pipeline.
- if: $CI_COMMIT_BRANCH
variables:
KIND: "branch" # every job can read KIND
# A push of a tag: create the pipeline. Anything not matched here, such
# as a schedule, gets no pipeline.
- if: $CI_COMMIT_TAG
variables:
KIND: "tag"
# A job's `rules` decide whether the job is part of this pipeline, and how it
# runs. The first rule that matches decides. If no rule matches, the job is
# simply left out, with no message anywhere: that is the most common reason a
# job "didn't run". Order the rules from the most specific case to the most
# general, and put `when: never` rules first to exclude cases. A job can't mix
# `rules` with the older `only`/`except`. To see which jobs a push would get
# before pushing, use Build > Pipeline editor > Validate.
lint:
stage: test
script: echo "lint, in $LINT_MODE mode"
rules:
# `if` tests variables. This one says: always, on the default branch
# (usually main).
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
variables:
LINT_MODE: "full" # a rule can set a variable for this job
# `changes` tests which files the commit changed. Only if a file under
# src/ changed does this rule match. Two surprises: on the first push of
# a new branch it is always true, and in a merge request pipeline it
# compares against the target branch, not the previous commit.
- changes:
paths: [src/**/*]
# regexp: src/.*\.js$ # or a Ruby regular expression instead
compare_to: refs/heads/main
variables:
LINT_MODE: "changed-files" # a different value from a different rule
# Before `rules` existed, jobs used `only:` and `except:` for the same
# purpose. Older templates still have them; they mean "only in these
# cases" and "except in these cases".
# A MANUAL job is a play button: it sits in the pipeline until a person
# starts it. Two things to know. A manual job that a rule adds counts as
# "not finished", so the pipeline shows as blocked until someone runs it,
# unless the rule also says `allow_failure: true`, as here. And a job that
# `needs` a manual job is skipped until the manual job has been run.
approve-docs:
stage: test
script: echo "the docs were approved by hand"
rules:
# `exists` tests whether a file is in the repository at all.
- exists: [docs/notes.md]
when: manual
allow_failure: true
# By default a job waits for the whole previous stage. `needs` lists the jobs
# this one really depends on, and the job starts as soon as those have
# finished, whatever their stage. It also downloads their artifacts. Use it
# to make pipelines faster. A job may need jobs from earlier stages or its
# own; a job that needs a job that isn't in the pipeline fails to be created,
# unless the need is `optional`.
package:
stage: package
needs:
- job: build
# One instance of a matrix job. `artifacts: false` means: wait for it,
# but don't download its files.
- job: test
parallel:
matrix:
- SUITE: unit
artifacts: false
# `optional` means: if this job isn't in the pipeline at all, as on a
# branch where lint's rules didn't match, don't wait for it.
- job: lint
optional: true
# Premium: a job in another project's pipeline.
# - project: group/other
# job: build
# ref: main
# artifacts: true
script:
- cat build.txt
- echo "this is a $KIND pipeline"
# `when` says under what condition a job runs, once it is in the pipeline:
# on_success when no job in an earlier stage failed (the default)
# on_failure only when a job in an earlier stage failed
# always whatever happened earlier
# never not at all
# manual when a person presses play
# delayed after a wait
cleanup:
stage: .post
when: always
# `dependencies` chooses which earlier jobs' artifacts to download. By
# default a job downloads all of them. An empty list downloads none, which
# makes a job like this faster.
dependencies: []
script: echo "cleanup"
# A DELAYED job waits for a set time after its stage is ready, then starts on
# its own, unless someone cancels it. Use it for a pause before a rollout.
warm-up:
stage: deploy
when: delayed
# How long to wait before starting.
start_in: 10 seconds
script: echo "warmed up"Deeper: Part V.
4. What a job is given#
Before a job's first line runs, it receives variables from several places, and the places have an order. Here is the order, and the shape of each kind.
# ---- 4. What a job is given: variables, secrets and identity ---------------
# A job can read values from several places, and when two places set the
# same name, the higher one on this ladder wins:
# 1. a security policy (set by a security team, outside any file)
# 2. values given when the pipeline was started by hand or by a trigger
# 3. the project's settings page, then its groups', then the instance's
# 4. dotenv reports from earlier jobs (part 5)
# 5. the job's own `variables`
# 6. the top-level `variables` below
# 7. GitLab's predefined variables
# The most common surprise: a value set on a settings page (3) silently beats
# the value you wrote in the file (5 or 6). When a value isn't what you wrote,
# check Settings > CI/CD > Variables on the project, then on each of its
# groups, then ask about the instance: the job's page does not show them.
#
# Top-level `variables` reach every job.
variables:
APP: "in-45-minutes" # the plain form: a name and a value
# The long form, with a description and a list of allowed values. GitLab
# then shows the variable as a field on its "Run pipeline" page.
LOG_LEVEL:
value: "info"
description: "How much the jobs print"
options: ["info", "debug"]
# Normally a `$` inside a value refers to another variable. `expand: false`
# keeps the `$` as plain text.
RAW_VALUE:
value: "$APP-literal"
expand: false
show-variables:
stage: build
# The job's own `variables` win over the top-level ones (5 beats 6). A value
# that came in through `extends` counts as the job's own.
variables:
LOG_LEVEL: "debug"
# `inherit` limits what this job takes from `default:` and from the
# top-level `variables`. Here it takes only `image`, and only these three
# variables. The workflow's KIND is not in the list, so this job can't see
# it. Use `inherit` for a job that must be isolated from the rest.
inherit:
default: [image]
variables: [APP, LOG_LEVEL, RAW_VALUE]
# An ID TOKEN is a short-lived, signed statement from GitLab that says which
# project, branch and job this is. A cloud such as AWS or Azure can be set
# up to trust it, and give the job access, so that no long-lived password
# or key has to be stored anywhere. `aud` (audience) is who the token is
# meant for.
id_tokens:
CLOUD_TOKEN:
aud: https://cloud.example.com
script:
- echo "APP=$APP LOG_LEVEL=$LOG_LEVEL RAW_VALUE=$RAW_VALUE KIND=$KIND"
- test -n "$CLOUD_TOKEN" && echo "an ID token was issued for this job"
# DEPLOY_TOKEN is not in this file at all. It was typed into the project's
# settings page (rung 3 of the ladder), which is where secrets belong.
# There it was marked MASKED, so a log shows [MASKED] instead of the
# value, and PROTECTED, so only pipelines on protected branches and tags,
# such as main, receive it. On any other branch it is simply empty, which
# is why a job can work on main and fail on a feature branch. GitLab can
# only mask a value that is one line, without spaces, and at least eight
# characters long; anything else it refuses to mask.
- echo "DEPLOY_TOKEN=$DEPLOY_TOKEN"
# Premium: `secrets` fetches a value at run time from an outside store, and
# by default writes it to a file whose path is in the variable.
# secrets:
# DB_PASSWORD:
# vault: kv/db/password@secret
# file: false # the value in the variable, not a file
# token: $CLOUD_TOKEN # which ID token to present to the store
# API_KEY:
# aws_secrets_manager: { secret_id: api-key }
# CERT:
# azure_key_vault: { name: cert }
# DSN:
# gcp_secret_manager: { name: dsn, version: latest }
# PEPPER:
# gitlab_secrets_manager: { name: pepper }
# Sign the job in to Google Cloud with its ID token (GitLab.com only, beta):
# identity: google_cloudThe job above, as it ran on a protected branch and then in a merge request. The settings variable is masked on one and empty on the other:
$ echo "APP=$APP LOG_LEVEL=$LOG_LEVEL RAW_VALUE=$RAW_VALUE KIND=$KIND"
APP=in-45-minutes LOG_LEVEL=debug RAW_VALUE=$APP-literal KIND=
$ test -n "$CLOUD_TOKEN" && echo "an ID token was issued for this job"
an ID token was issued for this job
$ echo "DEPLOY_TOKEN=$DEPLOY_TOKEN"
DEPLOY_TOKEN=[MASKED]Job succeeded$ echo "APP=$APP LOG_LEVEL=$LOG_LEVEL RAW_VALUE=$RAW_VALUE KIND=$KIND"
APP=in-45-minutes LOG_LEVEL=debug RAW_VALUE=$APP-literal KIND=
$ test -n "$CLOUD_TOKEN" && echo "an ID token was issued for this job"
an ID token was issued for this job
$ echo "DEPLOY_TOKEN=$DEPLOY_TOKEN"
DEPLOY_TOKEN=Job succeededDeeper: Part VI.
5. Execution and data#
Where a job runs, what runs beside it, what it keeps, and what it hands to the jobs after it.
# ---- 5. Execution and data: where a job runs, and what it passes on --------
with-service:
stage: test
# `image` in its long form. `name` is the image; the other keys adjust how
# it is used.
image:
name: alpine:3.22
# An image can have a default command, its ENTRYPOINT. A tool's own image
# often makes that the tool itself, and then your script never runs.
# An empty entrypoint switches that off.
entrypoint: [""]
# Reuse an already downloaded copy of the image instead of pulling it
# again every time.
pull_policy: if-not-present
# Docker-only settings, such as the platform:
# docker: { platform: linux/amd64 }
# A SERVICE is a second container started beside the job, for something the
# job needs to talk to, such as a database. The job reaches it by the
# `alias` as a host name. Here it is a small registry, reached as "reg".
# A service must stay running on its own; when a job can't reach one, read
# the service's own log, which the job page shows separately.
services:
- name: registry:2
alias: reg
variables:
REGISTRY_LOG_LEVEL: "warn" # set inside the service container only
# `tags` sends the job only to runners that carry the same tag, for example
# a runner with a GPU or one inside a private network. If no online runner
# has the tag, the job waits forever.
# tags: [linux]
# `hooks` runs commands at fixed points of the job's life. This one runs
# before the repository is even fetched, for example to set up Git access.
hooks:
pre_get_sources_script:
- echo "about to clone"
variables:
# GIT_DEPTH controls how much history the job fetches. Jobs get a shallow
# copy of their own branch only, which is fast, but a script that reads
# older commits or tags finds nothing there.
GIT_DEPTH: "5"
script:
- wget -qO- http://reg:5000/v2/ && echo "the service answered"
- echo "coverage lines 87.5%"
# `coverage` is a pattern GitLab searches the log with. The number it finds
# becomes the job's test coverage, shown in badges and merge requests.
coverage: '/coverage lines (\d+\.\d+)%/'
keep-files:
stage: package
# An empty `needs` means: depend on nothing, start at once.
needs: []
script:
- mkdir -p out && echo "a report" > out/report.txt
# A DOTENV file is a text file of NAME=value lines. Reported as an
# artifact (below), its lines become variables in every later job.
- echo "VERSION=1.4.0" > build.env
# ARTIFACTS, in full. Files GitLab uploads when the job ends, keeps for a
# while, and gives to later jobs. They are guaranteed to arrive, which is
# what makes them different from the cache further down.
artifacts:
# The name of the download, as shown on the job page.
name: "$CI_JOB_NAME-$CI_COMMIT_SHORT_SHA"
# Which files and folders to keep.
paths: [out/]
# Which of those to leave out.
exclude: [out/**/*.tmp]
# How long GitLab keeps them before deleting them. Without it, the
# instance's default applies.
expire_in: 1 week
# Keep the files even when the job failed: that is when you need them.
when: always
# Show a link to them on the merge request.
expose_as: "report"
# Who may download them from GitLab: all, developer, maintainer or none.
# The older `public: false` means the same as `developer`.
access: developer
# REPORTS are artifacts in a format GitLab understands and shows: test
# results (junit), coverage, security findings, and dotenv for variables.
reports:
dotenv: build.env
# The CACHE is for speed, and nothing else. It keeps files such as
# downloaded packages between pipelines, so the next job doesn't download
# them again.
# Unlike artifacts, a cache is not guaranteed: it may be missing on another
# runner or after a clean-up, so a job must still work without it. The
# first pipeline after a key changes always finds nothing, and caches of
# protected branches are kept apart from the rest (the key gets a
# "-protected" suffix) unless `unprotect` says otherwise.
cache:
# The key names the cache. `files` makes the key change when these files
# change, so a changed dependency list never restores a stale cache.
key:
files: [ci/base.yml]
# files_commits: [ci/base.yml] # or the files' last commits instead
prefix: "$CI_JOB_NAME"
# Which folders to keep. They must be inside the project's folder.
paths: [.cache/]
# `pull` restores the cache, `push` saves it, `pull-push` does both.
policy: pull-push
# Save the cache only when the job passed (or `always`, or `on_failure`).
when: on_success
# Other keys to try when this one has no cache yet.
fallback_keys: [default]
# Caches of protected branches are normally kept apart. This shares them.
unprotect: true
# `true` would also cache every file Git doesn't track, such as build
# output.
untracked: false
use-files:
stage: deploy
needs: [keep-files]
# A RESOURCE GROUP lets only one job of the group run at a time, even across
# pipelines. Use it so that two deploys to the same place never overlap.
resource_group: staging
script:
# out/report.txt arrived as an artifact, and VERSION from the dotenv.
- cat out/report.txt
- echo "VERSION=$VERSION, from the dotenv report"The package job from part 3, as it ran: its artifact from
build arrived, and the workflow's variable named the pipeline's kind:
Downloading artifacts for build (695)...$ echo "before_script runs first, in every job"$ cat build.txt
built at a043c660
$ echo "this is a $KIND pipeline"
this is a branch pipeline$ echo "after_script runs last, even when the script failed"Job succeededDeeper: Part VII.
6. Not in any file#
Some of what shapes your pipeline is a setting on a GitLab page, or a policy set by another team. None of it shows in your file, and some of it shows in no file at all.
| You see | It is | Where to look | Deeper |
|---|---|---|---|
| a value no file sets | a CI/CD variable from the project's, a group's or the instance's settings; it beats the YAML | Settings › CI/CD › Variables, on the project and on each group | chapter 18 |
| a secret that is empty on your branch | a protected variable: only protected branches and tags receive it | Settings › CI/CD › Variables, and Settings › Repository › Protected branches | chapter 27 |
| a job that waits with no runner | runners are registered on the instance, a group or the project, and take jobs whose tags they carry | Settings › CI/CD › Runners | chapter 29 |
| a job in no file, often in a stage you never wrote <badge>ultimate</badge> | a pipeline execution policy or a scan execution policy, set by a security team, which no toggle in your file can switch off | Secure › Policies | chapter 36 |
| a merge that needs an approval you can't give <badge>ultimate</badge> | a merge request approval policy | Secure › Policies, and the merge request's approvals | chapter 18 |
sast, secret_detection, container_scanning jobs | GitLab's own scanners, from templates GitLab ships; they skip merge request pipelines unless AST_ENABLE_MR_PIPELINES is "true" | the included template, then Secure › Security configuration | chapter 34 |
a pipeline with no .gitlab-ci.yml in the project | the file lives at another path or in another project, or Auto DevOps is on | Settings › CI/CD › General pipelines, CI/CD configuration file | chapter 18 |
| a template you never included <badge>premium</badge> | the instance template repository, or a required include the instance adds to every pipeline | ask the platform team | chapter 18 |
| another project refuses your job's token | job token permissions: which projects may use this project's job tokens | Settings › CI/CD › Job token permissions | chapter 28 |
secrets: in a job <badge>premium</badge> | a secret fetched at run time from Vault or a cloud's secret manager | the job's YAML, and the store it names | chapter 27 |
needs: project: <badge>premium</badge> | artifacts from a job in another project's pipeline | the job's YAML | chapter 24 |
dast_configuration: <badge>ultimate</badge> | which site and scanner profiles a DAST job uses | Secure › Security configuration | chapter 34 |
run: instead of script: <badge>experimental</badge> | GitLab Functions, formerly CI/CD Steps: reusable steps in place of shell lines | the job's YAML | chapter 19 |
release: | a job in a tag pipeline that records a release for the tag | Deploy › Releases | chapter 40 |
pages: true on a job | a job that publishes its public/ folder as a website; naming the job pages is the old way | Deploy › Pages | chapter 64 |
Deeper: Part VIII, and chapter 18.
7. After the pipeline#
A deploy job names an environment, and GitLab records what went where. In an organisation that uses GitOps, the job's real work is a commit to another repository, and a controller in the cluster does the rest, later.
# ---- 7. After the pipeline: deploys, environments and releases -------------
# An ENVIRONMENT is a place your software runs, such as staging or production.
# A job that names one is a deploy job, and GitLab records every run of it:
# which commit went where, and when. That record is what "Operate >
# Environments" shows, and its Rollback button reruns an earlier deploy job
# with that older commit, which is the quickest way back.
deploy-staging:
stage: deploy
script:
# Deploy the image this pipeline built, tagged with the commit, and deploy
# that same tag everywhere. A pipeline that builds a fresh image for each
# environment ships something nobody tested.
- echo "deploying $APP:$CI_COMMIT_SHORT_SHA to staging"
environment:
name: staging
# A link to the running service, shown beside the environment.
url: https://staging.example.com
# What kind of environment this is: production, staging, testing,
# development or other. GitLab groups and reports them by tier.
deployment_tier: staging
# The job that tears the environment down, and when to run it on its own.
on_stop: stop-staging
auto_stop_in: 1 day
# Which agent for Kubernetes and which namespace the environment's
# dashboard shows. (`kubernetes: namespace:` on its own is the old form.)
# kubernetes:
# agent: platform/agents:staging
# dashboard: { namespace: staging }
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
# The stop job named above. `action: stop` tells GitLab that this job ends the
# environment rather than deploying to it.
stop-staging:
stage: deploy
script: echo "stopping staging"
environment:
name: staging
action: stop
when: manual
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
deploy-production:
stage: deploy
script:
# Many organisations deploy with GITOPS: the job doesn't touch the cluster
# at all. It writes the new image tag into a Git repository that describes
# the environment, and a controller running in the cluster notices the
# change and applies it, minutes later. So a green job here means
# "written to Git", not "running". To know whether the change is really
# running, look at the controller, Argo CD or Flux, and at the pods, not
# at this job.
- echo "writing tag $CI_COMMIT_SHORT_SHA to the environment repository"
environment:
name: production
url: https://example.com
# Production waits for a person: `manual` gives the job a play button, and
# `manual_confirmation` adds a box with this text that must be confirmed
# before the job starts.
when: manual
manual_confirmation: "Deploy to production?"
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
# A RELEASE is GitLab's record of a version: a tag with a title, notes and
# download links, shown under "Deploy > Releases". A job with `release:` in a
# tag pipeline creates it. The job needs GitLab's `glab` command in its image
# (registry.gitlab.com/gitlab-org/cli), so it stays a comment here.
# release:
# stage: deploy
# script: echo "releasing $CI_COMMIT_TAG"
# release:
# tag_name: $CI_COMMIT_TAG
# tag_message: "Version $CI_COMMIT_TAG"
# name: "Release $CI_COMMIT_TAG"
# description: "Built from $CI_COMMIT_SHORT_SHA"
# milestones: ["1.4"]
# released_at: "2026-09-26T09:00:00Z"
# assets:
# links: [{ name: "binary", url: "https://example.com/app.tar.gz" }]
# rules:
# - if: $CI_COMMIT_TAG
# GITLAB PAGES hosts static websites straight from a repository. A job with
# `pages: true` publishes its `public/` folder as the site; naming the job
# `pages` used to be what did it, and is now deprecated. Pages is switched
# off on this GitLab, so it stays a comment too.
# publish-site:
# stage: deploy
# script: mkdir -p public && echo "hello" > public/index.html
# pages: true
# artifacts:
# paths: [public/]8. The traps, and where you just saw them#
Every trap on the card is a comment on a line above. When one of them bites, the matching card takes you from the symptom to the fix.
| Trap | Where the tour shows it | Card |
|---|---|---|
Your own rules, script or before_script replaced the template's whole list | build, in part 2 | your rules replaced the template's |
| A settings variable beat the value in your YAML | DEPLOY_TOKEN, in part 4 | your change is ignored |
| A protected secret is empty on a branch | DEPLOY_TOKEN, in part 4 | fails only on branches |
| An include or image on a moving branch or tag changed under you | include, in part 2 | the pipeline changed overnight |
| One push made two pipelines | workflow, in part 3 | duplicate pipelines |
changes was true on a new branch's first push | lint, in part 3 | runs on a new branch |
| A manual job blocked the pipeline, or a job that needed it was skipped | approve-docs, in part 3 | a play button that waits |
| No rule matched, so the job silently vanished, and a pipeline with no jobs isn't created | lint, in part 3 | no jobs, no pipeline |
| A child pipeline of a merge request had no jobs, because its jobs had no rules | child-job, in part 2 | no jobs, no pipeline |
| The cache missed, and the job had counted on it | keep-files, in part 5 | a job restores stale files |
| A script needed history the shallow clone didn't have | GIT_DEPTH, in part 5 | history missing |
| The image's entrypoint ran instead of your script | with-service, in part 5 | the script never runs |
| No online runner carries the job's tags | tags, in part 5 | no runner matches its tags |
| Green meant written to Git, not running | deploy-production, in part 7 | deploys appear minutes later |
When something breaks and it isn't in this table, start from the four kinds of symptom on the card: didn't run, failed, wrong result, stuck or drifting. Deeper: Part XI.
What is what#
Every word you will meet in a pipeline file or on a GitLab page, one line each. That
is every keyword of .gitlab-ci.yml, every way configuration gets shared,
every term this book uses, and the old names of things GitLab has renamed. Search it with
⌘K, or read one group at a time. Each line links to where the book explains
it.
This table is built from the keyword index, the reuse catalogue and the glossary, so it says exactly what they say.
| You see | Kind | It means | Where |
|---|---|---|---|
| Keyword | |||
after_script | Keyword | commands that run last, in a separate shell | chapter 29 |
allow_failure | Keyword | lets the pipeline continue if the job fails | chapter 24 |
artifacts | Keyword | files kept after the job, for later jobs and people | chapter 31 |
artifacts:access, :public, :expose_as | Keyword | who can download them, and links in merge requests | chapter 31 |
artifacts:paths, :exclude, :name, :untracked | Keyword | which files to keep | chapter 31 |
artifacts:reports | Keyword | files GitLab reads, such as test results and security reports | chapter 31 |
artifacts:when, :expire_in | Keyword | when to keep them, and for how long | chapter 31 |
before_script | Keyword | commands that run before script, in the same shell | chapter 29 |
cache | Keyword | files reused between jobs and pipelines, to save time | chapter 31 |
cache:key:files, :files_commits, :prefix | Keyword | a key that changes when the files' content changes, or when the files get a new commit, with an optional prefix | chapter 31 |
cache:key, :paths, :policy, :when, :fallback_keys, :unprotect, :untracked | Keyword | which cache, what goes in it, and when it is saved | chapter 31 |
coverage | Keyword | a regular expression that finds the coverage figure in the log | chapter 37 |
dast_configuration | Keyword | DAST site and scanner profiles | – |
default | Keyword | settings every job gets unless it sets its own: image, before_script, tags and others | chapter 14 |
dependencies | Keyword | which earlier jobs' artifacts to download | chapter 31 |
environment | Keyword | where the job deploys, which makes it a deployment job | chapter 40 |
environment:name, :url, :on_stop, :action, :auto_stop_in, :deployment_tier, :kubernetes | Keyword | the environment's name, address, stopping and tier | chapter 40 |
extends | Keyword | copies settings from other jobs, usually hidden ones | chapter 14 |
hooks:pre_get_sources_script | Keyword | commands on the runner before the Git fetch | chapter 29 |
identity | Keyword | identity federation with a cloud provider, in beta | – |
id_tokens | Keyword | signed ID tokens for other systems, such as AWS and Azure | chapter 28 |
image | Keyword | the container image the job runs in | chapter 30 |
image:name, :entrypoint, :pull_policy, :docker, :kubernetes | Keyword | the image's name, entrypoint, pull policy and executor options | chapter 30 |
image, services, cache, before_script, after_script at the top level | Keyword | the same keywords inside default: | chapter 14 |
include | Keyword | copies other YAML files into this configuration | chapter 15 |
include:component | Keyword | a versioned CI/CD component | chapter 16 |
include:inputs | Keyword | values for an included file's inputs | chapter 16 |
include:integrity, include:cache | Keyword | check or cache a remote file | – |
include:local | Keyword | a file from the same repository and branch | chapter 15 |
include:project | Keyword | a file from another project, at a ref | chapter 15 |
include:remote | Keyword | a file from any URL | chapter 15 |
include:rules | Keyword | includes the file only when the rules match | chapter 15 |
include:template | Keyword | a template that ships with GitLab | chapter 15 |
inherit | Keyword | which default: settings and top-level variables the job takes | chapter 14 |
inherit:default, inherit:variables | Keyword | take all, none, or a list | chapter 14 |
inputs, :type, :regex and others | Keyword | typed inputs for one job, which can be changed when the job is run by hand or retried | – |
interruptible | Keyword | lets a newer pipeline cancel this job | chapter 24 |
manual_confirmation | Keyword | a message to confirm before running a manual job | chapter 24 |
needs | Keyword | starts the job as soon as named jobs finish, ignoring stages | chapter 24 |
needs:artifacts | Keyword | whether to download the needed job's artifacts | chapter 31 |
needs:optional | Keyword | a need that may be missing from the pipeline | chapter 24 |
needs:parallel:matrix | Keyword | needs one variant of a matrix job | chapter 14 |
needs:project, needs:pipeline, needs:pipeline:job | Keyword | artifacts or status from other pipelines | chapter 17 |
only, except | Keyword | rules | chapter 23 |
pages | Keyword | a GitLab Pages publishing job | – |
parallel | Keyword | runs several copies of the job | chapter 14 |
parallel:matrix | Keyword | one copy for each combination of variable values | chapter 14 |
publish, and a Pages job named pages | Keyword | the pages keyword | – |
release, :tag_name, :tag_message, :name, :description, :ref, :milestones, :released_at, :assets:links | Keyword | creates a release, and describes it | – |
resource_group | Keyword | only one job in the group runs at a time | chapter 24 |
retry | Keyword | retries the job on failure | chapter 24 |
retry:max, :when, :exit_codes | Keyword | how often, and for which failures | chapter 24 |
rules | Keyword | decides whether the job is in the pipeline, and how it runs | chapter 23 |
rules:changes:paths, :compare_to, :regexp, and rules:exists:regexp | Keyword | the files to check, the ref to compare against, or a Ruby regular expression in place of glob patterns | chapter 23 |
rules:changes, rules:exists | Keyword | conditions on changed or existing files | chapter 23 |
rules:if | Keyword | a condition on variables | chapter 23 |
rules:when, :allow_failure, :needs, :variables, :interruptible | Keyword | what a matching rule changes | chapter 23 |
run | Keyword | a sequence of GitLab Functions steps | chapter 19 |
script | Keyword | the commands the job runs | chapter 29 |
secrets | Keyword | secrets fetched from a secrets manager | chapter 27 |
secrets:file, secrets:token | Keyword | file or value, and which ID token to use | chapter 27 |
secrets:vault, :aws_secrets_manager, :gcp_secret_manager, :azure_key_vault, :gitlab_secrets_manager | Keyword | the provider and the secret | chapter 27, and for AWS and Azure chapter 38 |
services | Keyword | extra containers beside the job, such as a database | chapter 30 |
services:name, :alias, :entrypoint, :command, :variables, :pull_policy | Keyword | the service's image, hostname, start command and settings | chapter 30 |
spec | Keyword | the header of a file, before its --- line | chapter 16 |
spec:include, spec:component, spec:description | Keyword | shared input definitions, component context, and a description | chapter 16 |
spec:inputs, :default, :type, :options, :regex, :rules, :description | Keyword | typed parameters for a component, an included file or a pipeline, with defaults, allowed values and checks | chapter 16 |
stage | Keyword | the stage the job belongs to | chapter 4 |
stages | Keyword | the stages, in order | chapter 24 |
start_in | Keyword | the delay for when: delayed | chapter 24 |
tags | Keyword | which runners may take the job | chapter 29 |
timeout | Keyword | the job's own time limit | chapter 29 |
trigger | Keyword | starts a downstream pipeline | chapter 17 |
trigger:forward, trigger:inputs | Keyword | what the downstream pipeline receives | chapter 17 |
trigger:include | Keyword | a child pipeline from a file in this project | chapter 17 |
trigger:project | Keyword | a pipeline in another project | chapter 17 |
trigger:strategy | Keyword | makes the trigger job wait for, and mirror, the downstream pipeline | chapter 17 |
variables | Keyword | variables for every job, or for one job | chapter 26 |
variables:expand | Keyword | whether $ references in the value are expanded | chapter 26 |
variables:value, :description, :options | Keyword | defaults and choices shown when someone runs a pipeline by hand | chapter 26 |
when | Keyword | when the job runs: on_success, manual, delayed, always, never | chapter 24 |
workflow | Keyword | whether a pipeline is created, and its settings | chapter 22 |
workflow:auto_cancel, :on_new_commit, :on_job_failure | Keyword | which jobs to cancel when a newer commit starts a pipeline, or when a job fails | chapter 22 |
workflow:name | Keyword | the pipeline's name | chapter 22 |
workflow:rules | Keyword | decide whether the pipeline exists at all | chapter 22 |
workflow:rules:variables | Keyword | variables set when a workflow rule matches | chapter 22 |
| Mechanism | |||
| a compliance framework label on the project, and jobs from a file the project does not include | Mechanism | Compliance pipelines: Organisation-wide | Compliance pipelines |
| a form of inputs or variables on the Run pipeline page | Mechanism | Pipeline inputs: Parameters | Pipeline inputs |
a .gitlab-ci.yml created from the Web Editor's template list, matching a file in the administrators' templates project | Mechanism | Instance template repository: Organisation-wide | Instance template repository |
a .gitlab-ci.yml that matches a starter template, while the central copy has moved on | Mechanism | Project and file templates: Organisation-wide | Project and file templates |
a job name that starts with a dot, such as .maven-base: | Mechanism | Hidden jobs: Inside one YAML file | Hidden jobs |
| a job sees a variable that no file defines | Mechanism | Instance and group CI/CD variables: Organisation-wide | Instance and group CI/CD variables |
a job with run: and a list of steps using func:, and no script: | Mechanism | GitLab Functions: Outside the YAML | GitLab Functions |
| a merge request needs extra approvals because of security findings or licences | Mechanism | Merge request approval policies: Organisation-wide | Merge request approval policies |
a spec: inputs: header above a --- line | Mechanism | spec: inputs: Parameters | spec: inputs |
a template reads $JAVA_VERSION, and your file sets it under variables: | Mechanism | Variables as parameters: Parameters | Variables as parameters |
a top-level default: block setting image, before_script, tags or retry | Mechanism | default:: Sharing between jobs | default: |
a top-level variables: block | Mechanism | Top-level variables: Sharing between jobs | Top-level variables |
| behaviour that changes with the runner: environment variables, mounted files or cloud permissions that no file sets | Mechanism | Runner configuration: Outside the YAML | Runner configuration |
curl --request POST …/trigger/pipeline with a trigger token, inside a script | Mechanism | Pipeline trigger API: Whole pipelines | Pipeline trigger API |
curl -sSL … | bash, or a git clone of a scripts repository, inside script: | Mechanism | Scripts fetched at run time: Outside the YAML | Scripts fetched at run time |
&defaults after a key, and *defaults later in the same file | Mechanism | YAML anchors and aliases: Inside one YAML file | YAML anchors and aliases |
<<: *defaults inside a job | Mechanism | YAML merge key: Inside one YAML file | YAML merge key |
extends: .maven-base or extends: [.base, .with-cache] | Mechanism | extends: Sharing between jobs | extends |
image: registry.example.com/devops/ci-tools:3.2 and a script that calls a command no repository defines | Mechanism | Toolbox images: Outside the YAML | Toolbox images |
include: - component: $CI_SERVER_FQDN/devops/components/ecr-push@1.4.0 | Mechanism | include: component: Pulling in files | include: component |
include: local: ci/build.yml, a wildcard such as ci/*.yml, or a bare path under include: | Mechanism | include: local: Pulling in files | include: local |
include: - project: devops/ci-templates with ref: and file: | Mechanism | include: project: Pulling in files | include: project |
include: remote: https://…/pipeline.yml | Mechanism | include: remote: Pulling in files | include: remote |
include: template: Security/SAST.gitlab-ci.yml | Mechanism | include: template: Pulling in files | include: template |
no .gitlab-ci.yml, yet jobs appear with names such as build, test, code_quality and container_scanning | Mechanism | Auto DevOps: Organisation-wide | Auto DevOps |
parallel: matrix: in a job, and job names like test: [21] in the pipeline | Mechanism | parallel: matrix: Sharing between jobs | parallel: matrix |
pipelines run but the repository has no .gitlab-ci.yml | Mechanism | Custom CI/CD configuration file: Organisation-wide | Custom CI/CD configuration file |
- !reference [.snippets, aws_login] inside a script or rules list | Mechanism | !reference tags: Sharing between jobs | !reference tags |
scanner jobs named with a hyphen and a number, such as secret-detection-1 | Mechanism | Scan execution policies: Organisation-wide | Scan execution policies |
script: mc deploy --env prod: a company tool, not a public one | Mechanism | Internal command-line tools: Outside the YAML | Internal command-line tools |
script: mvn -P ci verify with a <parent> POM owned by another team | Mechanism | Build-tool reuse: Outside the YAML | Build-tool reuse |
source /opt/ci/lib.sh | Mechanism | Shell function libraries: Outside the YAML | Shell function libraries |
stages named .pipeline-policy-pre or .pipeline-policy-post | Mechanism | Pipeline execution policies: Organisation-wide | Pipeline execution policies |
trigger: include: - artifact: generated.yml with job: generate | Mechanism | Dynamic child pipelines: Whole pipelines | Dynamic child pipelines |
trigger: include: ci/deploy.yml | Mechanism | Parent–child pipelines: Whole pipelines | Parent–child pipelines |
trigger: project: platform/deployer | Mechanism | Multi-project pipelines: Whole pipelines | Multi-project pipelines |
variables such as SAST_DISABLED, SKIP_SONAR or DEPLOY_ENABLED, tested in rules: - if: | Mechanism | Toggle variables: Parameters | Toggle variables |
| Term | |||
| ACR | Term | Azure Container Registry, where MegaCorp keeps the images that aks-prod runs | chapter 38 |
| agent for Kubernetes | Term | GitLab's program inside a cluster, which connects the cluster to GitLab | chapter 45 |
| anchor, alias, merge key | Term | YAML's own copy and paste, with & and *; it works only inside one file | chapter 13 |
Application, ApplicationSet | Term | Argo CD's objects: an Application says what to deploy, from where and to where; an ApplicationSet makes many from one template | chapter 45 |
| Argo CD | Term | a GitOps controller with a web UI, which syncs Applications from Git into a cluster | chapter 41 |
| artifact | Term | files a job keeps when it ends, for later jobs and for people | chapter 31 |
| AssumeRoleWithWebIdentity | Term | the AWS call that swaps an ID token for temporary AWS keys | chapter 38 |
| Auto DevOps | Term | GitLab's built-in pipeline, for projects with no pipeline file when the setting is on | chapter 18 |
| cache | Term | files kept between jobs to save time; they may be missing at any time | chapter 31 |
| chart | Term | a Helm package: templates plus default values, which render Kubernetes objects | chapter 43 |
| child pipeline | Term | a pipeline that a trigger job starts from a file in the same project | chapter 17 |
| CI Lint | Term | a tool that checks a piece of pipeline YAML | chapter 4 |
| component | Term | a versioned piece of configuration with typed inputs, included with include: component | chapter 16 |
| container registry | Term | a store for a project's container images, under Deploy › Container Registry | chapter 8 |
| container scanning | Term | a scanner that looks for known vulnerabilities in an image | chapter 33 |
| controller | Term | a program in the cluster that watches objects, and changes the cluster to match them | chapter 41 |
default: | Term | settings every job gets unless it sets its own | chapter 14 |
| dependency scanning | Term | a scanner that looks for known vulnerabilities in the libraries a project uses | chapter 33 |
| deployment job | Term | a job with environment:, which records what it deployed, and where | chapter 40 |
| Docker-in-Docker | Term | a Docker engine that runs as a service beside a job, so that the job can build images | chapter 8 |
| dotenv report | Term | a file of NAME=value lines that a job leaves as variables for later jobs | chapter 31 |
| downstream pipeline | Term | a pipeline that another pipeline started: a child or a multi-project pipeline | chapter 17 |
| dynamic child pipeline | Term | a child pipeline whose YAML an earlier job wrote | chapter 17 |
| ECR | Term | Amazon Elastic Container Registry, where AWS keeps container images | chapter 38 |
| Entra ID | Term | Microsoft's identity service for Azure, formerly Azure Active Directory | chapter 38 |
| environment | Term | a place a job deploys to, such as staging, with its history of deployments | chapter 40 |
| environment repository | Term | the Git repository that says what runs in each environment: image tags and settings | chapter 45 |
| executor | Term | how a runner runs jobs: in Docker containers, in Kubernetes pods, in a shell, and others | chapter 29 |
extends | Term | copies another job's settings: maps are merged, lists are replaced | chapter 14 |
| External Secrets Operator | Term | a controller that copies secrets from a cloud's vault into Kubernetes Secrets | chapter 50 |
| federated credential | Term | a rule on an Entra ID application or managed identity that says which outside tokens it accepts, matched exactly | chapter 38 |
| Flux | Term | a GitOps tool made of controllers, configured with objects such as GitRepository, Kustomization and HelmRelease | chapter 45 |
| Full configuration | Term | the pipeline editor's view of every file, merged into one | chapter 4 |
| GitOps | Term | deploying by writing the wanted state to a Git repository, which a tool in the cluster applies | chapter 41 |
| hidden job | Term | a job whose name starts with a dot; it never runs, and exists to be copied | chapter 13 |
| IAM role, trust policy | Term | an AWS identity a job can take on; its trust policy says who may | chapter 38 |
| ID token | Term | a note about the job, signed by GitLab, that another system such as AWS can check | chapter 28 |
include | Term | copies other YAML files into the configuration | chapter 15 |
| input | Term | a typed parameter of a component, an included file or a pipeline, filled in when the pipeline is created | chapter 16 |
| instance, group and project runners | Term | runners for every project, for one group's projects, or for chosen projects | chapter 29 |
| job | Term | one piece of work in a pipeline, run by one runner | chapter 4 |
| job token | Term | CI_JOB_TOKEN, which lets a job call GitLab while it runs | chapter 28 |
| Kustomize, overlay | Term | a tool that lays changes over a base set of objects, one overlay for each environment or cluster | chapter 44 |
| masked, hidden, protected | Term | how a variable is guarded: masked in logs, hidden in the settings, passed only to protected branches and tags | chapter 27 |
| merged results pipeline | Term | a merge request pipeline that runs on the source branch merged into the target | chapter 21 |
| merge request approval policy | Term | a security policy that requires approval when scans find problems | chapter 36 |
| merge request pipeline | Term | a pipeline for a merge request, rather than for a branch | chapter 21 |
| merge train | Term | a queue of merge requests, each tested together with those ahead of it | chapter 21 |
| multi-project pipeline | Term | a pipeline in another project, started by a trigger job | chapter 17 |
| namespace | Term | a named group of objects inside a cluster, with its own permissions and quotas | chapter 54 |
needs | Term | starts a job as soon as the named jobs finish, ignoring stages | chapter 24 |
| OIDC | Term | OpenID Connect, the standard that ID tokens follow | chapter 28 |
| OpenTelemetry | Term | the common standard for producing logs, metrics and traces | chapter 52 |
| OpenTofu | Term | an open-source tool that creates and changes infrastructure from code; it began as a copy of Terraform | chapter 10 |
| pipeline | Term | all the jobs GitLab runs for one commit, for one reason such as a push | chapter 4 |
| pipeline execution policy | Term | a security policy that adds jobs to projects' pipelines | chapter 36 |
| pipeline source | Term | what started a pipeline, held in CI_PIPELINE_SOURCE | chapter 21 |
| plan, apply | Term | OpenTofu's two steps: work out what would change, then change it | chapter 10 |
| predefined variable | Term | a variable GitLab sets, such as CI_COMMIT_BRANCH | predefined variables |
| promotion | Term | moving a version from one environment to the next | chapter 45 |
| protected branch, protected tag | Term | branches and tags that only some people may push to; only their pipelines get protected variables | chapter 27 |
| readiness probe | Term | the check a new pod must pass before Kubernetes sends it traffic, or counts it in a rollout | chapter 51 |
!reference | Term | a YAML tag that pastes one section of another job, such as its rules or script | chapter 14 |
resource_group | Term | lets only one job of the group run at a time | chapter 24 |
rules | Term | decide whether a job is in a pipeline, and how it runs | chapter 23 |
| runner | Term | the program that takes jobs from GitLab and runs them | chapter 29 |
| SAST | Term | static application security testing: a scanner that reads source code | chapter 33 |
| SBOM | Term | software bill of materials: a list of every component in a build | chapter 33 |
| scan execution policy | Term | a security policy that adds scanner jobs to pipelines, or runs them on a schedule | chapter 36 |
| secret detection | Term | a scanner that looks for passwords and keys committed to the repository | chapter 33 |
| security policy project | Term | the project that holds a group's security policies | chapter 36 |
| service | Term | an extra container beside the job, such as a database | chapter 30 |
| settings variable | Term | a variable set in the CI/CD settings of a project, a group or the instance | chapter 26 |
| shallow clone | Term | a copy of the repository with only the newest commits | chapter 32 |
| signals | Term | a running service's logs, metrics and traces | chapter 52 |
| stage | Term | a group of jobs; by default, a stage starts only when the stage before it has succeeded | chapter 24 |
| state | Term | OpenTofu's record of what it has made, which GitLab can keep for a project | chapter 10 |
| state, state lock | Term | OpenTofu's record of what it created, and the lock that stops two runs at once | chapter 48 |
| sync | Term | a controller making the cluster match what Git says | chapter 45 |
| tag | Term | a Git tag names a commit; a runner tag is a label that matches jobs to runners | chapter 29 |
| template | Term | YAML that GitLab ships, included with include: template | chapter 15 |
| toggle | Term | a variable a template checks to switch a job off, such as SAST_DISABLED | chapter 16 |
| trigger job | Term | a job that starts a downstream pipeline instead of running a script | chapter 17 |
| truth table | Term | this book's method for predicting which jobs run in which pipeline | chapter 25 |
| values file | Term | a Helm chart's settings for one service or environment | chapter 43 |
workflow:rules | Term | decide whether a pipeline is created at all | chapter 22 |
| workload identity | Term | a pod's own cloud identity, taken from its Kubernetes service account | chapter 49 |
| Former name | |||
| CI/CD Steps | Former name | GitLab Functions, an experiment | – |
CI_JOB_JWT, CI_JOB_JWT_V2 | Former name | ID tokens, requested with id_tokens | removed in 17.0 |
| Code Quality's CodeClimate template | Former name | any tool's results, imported as a codequality report | the template is deprecated |
| compliance pipelines | Former name | pipeline execution policies | deprecated in 17.3; removal planned for 20.0 |
| Gemnasium, the dependency scanning analyzer | Former name | dependency scanning using an SBOM | Gemnasium deprecated in 17.9; removal proposed for 20.0 |
image, services, cache, before_script or after_script at the top of the file | Former name | the same keywords inside default: | deprecated |
inject_ci, in a pipeline execution policy | Former name | inject_policy | 17.9; inject_ci is deprecated |
| kaniko, for building images | Former name | Docker, Buildah, Podman or rootless BuildKit | GitLab's page marks kaniko removed |
| Limit access to this project, then Authorized groups and projects | Former name | CI/CD job token allowlist | 17.2, then 17.3 |
| Merge when pipeline succeeds | Former name | auto-merge | – |
newly_detected, in a policy's vulnerability_states | Former name | new_needs_triage and new_dismissed | 17.0 |
only, except | Former name | rules | deprecated |
scan result policies, scan_result_policy | Former name | merge request approval policies, approval_policy | only approval_policy is accepted since 17.0 |
Security/SAST.gitlab-ci.yml and the other Security/ templates | Former name | wrappers: each only includes its Jobs/ twin, such as Jobs/SAST.gitlab-ci.yml (chapter 34) | – |
| Token Access, in Settings › CI/CD | Former name | Job token permissions | 17.2 |
The card#
The book on one page: what to ask about any job, where to click, which cards to open when something breaks, and the traps. Print it and keep it next to your screen.
Five questions for any job#
Ask them in this order (chapter 55):
| Question | Look at |
|---|---|
| Where is it defined? | Full configuration, then its extends chain and includes; if it's in no file, policies |
| When does it run? | workflow:rules, the job's final rules, and needs |
| Where does it run? | the job page's runner and tags, and the job's image |
| What does it execute? | its scripts, and the tools and functions those call |
| What goes in and out? | variables and secrets; artifacts, caches, reports, images and deploys |
Five questions for a deployment#
The same five, asked after the pipeline (chapter 55):
| Question | Look at |
|---|---|
| Where is it defined? | the environment repository: the chart, its values, overlays and inline values |
| When does it deploy? | the commit or merge request, then the controller's sync |
| Where does it run? | the cluster and namespace, in the Application or Kustomization |
| What does it run? | the rendered image reference, and whether that tag is in the cluster's registry |
| What goes in and out? | settings and secrets in; logs, metrics and traces out |
Four kinds of symptom#
| Symptom | Check first | Cards |
|---|---|---|
| Didn't run, or ran when it shouldn't | the pipeline exists; the rules matched; everything it needs exists | didn't run |
| Failed | the log's first error; the image and runner; what went in | failed |
| Passed, but did the wrong thing | which value won; which image or artifact; what was deployed | wrong result |
| Stuck, slow, or broke with no commit | runners online; tokens expired; central files or images changed | stuck or drifting |
Writing your team's pipeline#
- Start with a workflow: merge request pipelines while a merge request is open, branch pipelines otherwise (chapter 7).
- Pin every image, include and component to a version.
- Keep what later jobs need as artifacts; use the cache only for speed.
- Put secrets in settings, masked and protected, and expect them to be empty on branches.
- Tag each image with its commit, and deploy that one tag everywhere.
- Give every deploy job an environment, and make production manual.
Which variable value wins, highest first#
Security policies, then values given when the pipeline started, then the project's settings, then the groups', closest first, then the instance's. Below those come dotenv reports, then the job's own YAML, then top-level YAML, then deployment and predefined variables (chapter 26).
Where to look in GitLab#
| Place | Shows |
|---|---|
| Build › Pipeline editor, Full configuration | every file and template, merged |
| Build › Pipeline editor, Validate | the jobs a pipeline would get, and problems with them |
| The pipeline graph | which jobs exist, and any downstream pipelines |
| The job page | the runner, the image, the timeout, and the log |
| Settings › CI/CD › Variables, Runners, General pipelines | settings variables, runners, clone depth and the pipeline file's location |
| Settings › CI/CD › Job token permissions | which projects may use your job tokens |
| Secure › Policies | the security policies that apply |
| Operate › Environments | deployments, approvals and rollback |
| Operate › Kubernetes clusters, Terraform states | the agents that connect clusters; OpenTofu's states and their locks |
Where to look after the pipeline#
| Place | Shows |
|---|---|
| The environment repository's history and merge requests | what should run where, and what waits for a merge |
| Argo CD's Application, or Flux's Kustomization | what the controller applied, and whether it's healthy |
kubectl describe pod, and its events | why a pod won't pull, start or become ready |
| The service's dashboards | whether the deploy worked, after its mark |
Commands worth remembering#
aws sts get-caller-identity # which AWS role is really in use
echo "$MC_ID_TOKEN" | cut -d '.' -f2 | base64 -d | jq . # an ID token's claims, not the token
env | cut -d= -f1 | sort # which variables exist: names only
git fetch --unshallow # full history, for scripts that need it
az account show # who the Azure CLI is signed in as
helm template payments-api ./service -f values.yaml # what a chart renders
kustomize build overlays/staging # what an overlay builds
kubectl rollout status deployment/payments-api -n payments-staging # did the rollout finish?
kubectl logs <pod> --previous # a crashed container's last outputTen traps#
- A job's own
rules,before_script,tagsorservicesreplace the inherited list completely. - A settings variable beats anything in the YAML.
- A job's variables, including those from
extends, beat top-level ones. - Toggles never stop a security policy's jobs.
- GitLab's scanners skip merge request pipelines unless
AST_ENABLE_MR_PIPELINESis"true". - Protected values reach only protected branches and tags.
- A cache can miss at any time; only artifacts are guaranteed.
- Jobs get a shallow clone of their own ref only.
- With GitOps, green means written to Git, not running.
- A pipeline that rebuilds an image ships something nobody tested.
Four traps after the pipeline#
- Healthy means the pods pass their probes, not that requests succeed.
- A running pod keeps the settings and secrets it started with.
- Entra ID matches a federated credential's subject exactly, with no wildcards.
- In Argo CD, inline values beat value files.
How GitLab thinks#
A pipeline is one run of your configuration: jobs, grouped in stages, each job a script that a runner executes in a container. In a large organisation a job is rarely written in one place. It is assembled from your file, the central team's files, settings that live in no file, and the image it runs in, at four moments in turn. The first two you can see on one page, Full configuration. Nearly every question about a job is one of five: where is it defined, when does it run, where does it run, what does it execute, and what goes in and out.
You add a job called smoke-test to the end of the file, give it
stage: smoke, and push. Nothing runs. The commit page shows a red pipeline
with no jobs in it. At the top, one line: smoke-test job: chosen stage smoke does
not exist; available stages are .pre, build, test, package, scan, deploy, .post. You never wrote that list of stages. Where did it come from, and why does
one wrong word stop everything? (the card)
The answer is the shape of the whole system, so this chapter gives you the shape. Each piece gets its own chapter later; here you only need to recognise it.
The engine: six words#
When you push, GitLab starts a pipeline: one run of the configuration, for one commit and one event, such as a push or a merge request. A pipeline is made of jobs, each a script with its settings. Jobs are grouped in stages, which run in the order listed, while the jobs inside one stage run at the same time. GitLab schedules jobs but doesn't run them. A runner, a program registered with GitLab, keeps asking for work. It runs each job with its executor, usually a fresh container started from the job's image. On the pipeline's page, stages are the columns and jobs the boxes; a job's own page holds its log, and names the runner that took it.
Where each word shows up#
| Term | What it is | Where you see it |
|---|---|---|
| Pipeline | one run of the configuration, for one commit and one event | the pipeline's page, with its graph |
| Stage | a named group of jobs; stages run in the order they are listed | the columns of the pipeline graph |
| Job | a script to run, with its settings | a box in the graph; the job's own page holds its log |
| Runner | a program that takes jobs from GitLab and runs them | the job's page names it |
| Executor | how the runner runs a job: a container, a Kubernetes pod, or a plain shell | the first lines of the job log |
| Image | the container image a job runs in | the first lines of the job log |
Three things about names catch people. Any top-level key in the configuration is a
job, except GitLab's own keywords such as stages, variables and
include, so you can't have a job called services. A name that
starts with a dot, such as .maven-base, is a hidden job that never
runs: it exists to be copied by others (chapter 14). And two
definitions of the same name, one in your file and one in an included file, merge into
one job. Names are at most 255 characters, and true, false and
nil work as names only in quotes.
One more word means two things. A Git tag, such as v1.4.0, names
a commit and starts tag pipelines. A runner tag, written with the
tags: keyword, chooses which runner takes a job. In the file,
tags: always means runner tags.
Stages, and the two mistakes that stop a pipeline#
If a configuration never lists its stages, GitLab uses five: .pre,
build, test, deploy and .post. A job
that names no stage lands in test. A project can declare its own list with
stages:, and MegaCorp's shared pipeline does: build,
test, package, scan, deploy. That list
is what your Tuesday error quoted, and it lives in a file your project pulls in, not in
your file. When a job fails, the stages after it usually don't start:
As text
- build: compile (passed)
- test: lint (passed), unit-tests (failed)
- deploy: deploy-review (skipped)
Two stage mistakes stop GitLab from creating a pipeline at all, and both produce your
Tuesday message. A job names a stage that isn't in the list, as smoke-test
did. Or a job names no stage, lands in test, and the project's list leaves
test out. The fix for yours: use a listed stage, or add
smoke to the list, in the file that owns the list.
Where a job comes from#
Picture a job as an onion. Your project's file is at the centre, and each layer around it can add to it or change it:
- Your project's
.gitlab-ci.yml. The file your team writes. In a big organisation it is often twenty lines: aninclude, a few settings, and perhaps one job adjusted. - The central team's shared files. YAML files that your file pulls in
with
include. Most of your jobs are really defined there, as hidden jobs that your jobs copy withextends, snippets pasted with!reference, and adefault:block that applies to every job. Your stage list came from here. - Organisation settings. CI/CD variables set on the settings pages of the project, its groups, and the whole installation, called the instance. They beat anything written in YAML. On the top tier, security policies can add jobs from here too, to every pipeline, from outside any file.
- Run time. The runner's own configuration, and the image the job runs in. The real logic of a script often lives inside the image.
Four moments, and what you can see#
GitLab doesn't apply the layers all at once. It reads each file, joins the files, creates the pipeline, and then runs each job. The moment decides where you can look:
| Moment | What happens | Where to see the result |
|---|---|---|
| Parse: each file is read | YAML's own copy-and-paste is applied: anchors, aliases and merge keys | Build › Pipeline editor › Full configuration |
| Compile: the files are joined | included files are copied in; extends and !reference copy settings between jobs | the same view |
| Create: the pipeline is made | rules decide which jobs exist; a matrix job becomes several; settings variables and policies apply | the pipeline graph, and the settings pages |
| Run: a runner runs one job | the image, the runner's configuration and any downloaded scripts do their part | the job log |
So if what you need happens in the first two moments, one page shows it: Full
configuration, with every include, extends and !reference
already resolved. Two details of that view surprise people. A component's jobs appear with
their inputs already filled in. And the default: block appears on its own,
not copied into each job, so a job's final image may be there rather than in the job.
Five questions for any job#
When a job surprises you, the question you are asking is almost always one of these, and each has a usual place to look:
| Question | Where the answer usually is | Covered in |
|---|---|---|
| Where is it defined? | Full configuration, then the extends chain, then the included file; if it is in no file, a policy | Part IV |
| When does it run? | workflow:rules, the job's rules, needs and stages | Part V |
| Where does it run? | the job's runner tags, the runner's executor and the job's image | Part VII |
| What does it execute? | before_script and script, and whatever those call inside the image | reuse outside the YAML |
| What goes in and out? | variables and secrets in; artifacts, caches, reports, images and deployments out | Part VI and Part VII |
Here is the idea as a real pipeline: a merge request pipeline for payments-api, a
service at MegaCorp, the made-up company this book uses throughout. The project's own
file defines none of these jobs. Three are copied from the central team's hidden jobs,
one comes from a component, one from a file GitLab itself ships, and
policy-sbom is in no file at all: a security policy adds it.
As text
- build: maven-build
- test: maven-test: [17], maven-test: [21], sonar-scan (allow failure)
- package: image-build
- scan: container_scanning (allow failure)
- .pipeline-policy-post: policy-sbom (policy)
Needs: image-build waits only for maven-build; container_scanning waits only for image-build.
From a push to a pipeline#
Between your push and the first job, GitLab does a fixed sequence of things, and each step can stop the pipeline in a different way. Your Tuesday stopped at the validity check, which is why you got a red pipeline with an error. A workflow that says no stops silently instead, and so does a pipeline in which no job's rules matched.
As text
- You push a commit
- GitLab finds the configuration: .gitlab-ci.yml, or the file the settings name
- It fetches every include, and merges everything into one configuration
- It resolves extends, !reference and anchors
- Is the configuration valid? No: A failed pipeline, with the error shown at the top. Yes: the next step.
- Does workflow: rules allow this pipeline? No: No pipeline, and after a push no error. Yes: the next step.
- It checks each job's rules, expands matrices, and adds any policy jobs
- Is at least one job left? No: No pipeline. Yes: the next step.
- GitLab creates the pipeline, and runners start taking its jobs
Here is the whole assembly for one job. payments-api's file is short, and the job GitLab builds from it is not:
Every line of the result came from a file payments-api never opens. The variables, the
image, the cache, the rules and the before_script all arrived through the
include and the extends chain, which chapter 14
walks one link at a time.
The pipeline editor, and CI Lint#
You can watch the first four steps without pushing anything. The pipeline editor is at Build › Pipeline editor. It edits the project's file, and its tabs show what GitLab would do with it:
| Tab | What it shows | Use it to |
|---|---|---|
| Edit | the file, checked against GitLab's schema as you type | catch syntax mistakes before you commit |
| Visualize | every stage and job, with needs drawn as lines between jobs | see the shape of the pipeline |
| Validate | a simulated pipeline for a push to the branch you choose | find problems with rules and needs before you push |
| Full configuration | the whole configuration as one file: includes copied in, extends merged, anchors and !reference replaced | find where a line of a job really comes from |
Full configuration shows one scenario: anything conditional is decided as if the
pipeline were a push to the default branch. Content pasted by !reference can
appear as a list inside a list, with lines that start - -; GitLab documents
that as expected. And the view can't show what happens later: rules, matrices, policies,
settings variables and everything the runner does come after it.
CI Lint#
On the same Validate tab, Lint CI/CD sample checks configuration you paste in, so you can test a change without touching the project's file. Select Simulate pipeline creation for the default branch to go past syntax: the simulation evaluates the rules as a push to the default branch, and finds the problems a real push would hit. Every error message quoted in this book came from that check, run on the local GitLab 19.3. Your Tuesday error would have shown here before you pushed.
Reading a job log#
A job's log is written top to bottom in a fixed order, and the job page lets you fold
each part away. It starts with the runner's name and the executor. Then come Preparing
environment, which says where the job runs, and Getting source, with
the fetch and the commit, and Restoring cache. Then the script itself, each
line echoed after a $ followed by its output. Then after_script,
saving the cache, uploading artifacts, and finally Job succeeded or
ERROR: Job failed. On the right, Job details names the
runner, the job's timeout and where that came from, and its tags.
Every job shows a status, and the graph is read through them. A job is created while its stage hasn't started, pending while it waits for a runner, preparing and then running once one has it, and ends success or failed. Around those come five more. Canceled means someone or something stopped it, and skipped that an earlier stage failed. Manual waits for a person, scheduled is a delayed job counting down, and waiting_for_resource means another job holds the resource it needs. In a stage, jobs are sorted alphabetically, not by when they started or where they were defined.
When a job fails, read the log from the end backwards: the last command echoed is the one that failed, because a failing command stops the script. When a job does the wrong thing, read the log from the top, because the first lines tell you where it really ran. A job that sits in pending for ever is almost always a runner problem: no runner has the tags the job asks for (chapter 29).
Which tool answers which question#
| You want to know | Use | It can't tell you |
|---|---|---|
| what all the files add up to | Full configuration | anything decided when the pipeline is created |
| which jobs a push to a branch would create | Validate, choosing the branch | merge request or scheduled pipelines |
| whether a snippet is valid, or what it would do | CI Lint, with simulation | anything about pipelines other than a default-branch push |
| which jobs exist, and which downstream pipelines ran | the pipeline graph | which file a job came from |
| what ran, where, and what it printed | the job log | values set in settings, or what a policy added |
| values no file shows | Settings › CI/CD › Variables, on the project, its groups and the instance | nothing: this is where they live |
- Read your Tuesday error as GitLab meant it: the stage list lives in the central team's file, and you either use one of its stages or add yours there.
- Open Full configuration and see every job as GitLab assembled it, and say which of the four moments produced any line you don't recognise.
- Ask the five questions about any job, and know which page answers each.
Meet MegaCorp#
MegaCorp is a made-up company, and so is everything in its GitLab. It is built the way large organisations build theirs:
- A central DevOps team writes the shared pipeline files, the ready-made building blocks called components, and the container image most jobs run in.
- The security team writes policies that add scans to every pipeline.
- The platform team runs every deployment.
- Each application team keeps a short file that pulls in everything else.
Every example in this book is a real file from MegaCorp's projects, labelled with its project and path. The same files are loaded into a real GitLab 19.3 instance, and every piece of GitLab output in the book comes from there.
A deploy job in payments-api calls the deployer project through the API and gets 404 Not Found. The project exists: you can open it in the browser. A colleague says "ask platform, it's their project". Who owns what at MegaCorp, and which project is allowed to talk to which? (the card)
The map#
Each box is a GitLab project, and the arrows show who uses whom:
The projects#
You don't need to follow every word in this table yet. The last column says where each project is explained.
| Project | Owned by | Holds | You'll meet it in |
|---|---|---|---|
| megacorp/devops/ci-templates | central DevOps | the shared pipeline files: job templates, script snippets, rules, and two complete pipelines | Part IV |
| megacorp/devops/components | central DevOps | components, which are versioned building blocks with settings: sonar-scan, ecr-push and gitops-deploy | components and inputs |
| megacorp/devops/ci-tools | central DevOps | the toolbox image most jobs run in, its shell functions, and MegaCorp's own mc command | reuse outside the YAML |
| megacorp/security/policies | security | the security policies that add scans and gates to other projects' pipelines | policies and gates |
| megacorp/platform/deployer | platform | the deployment pipeline that every service starts | GitOps repositories and promotion |
| megacorp/platform/charts | platform | the Helm chart that every service is deployed with | Helm |
| megacorp/platform/gitops-config | platform | the environment repository: what runs on eks-prod and aks-prod, and each environment's values | GitOps repositories and promotion |
| megacorp/platform/infra-modules, infra-live | platform | the OpenTofu that builds both clusters, their registries and their identities | OpenTofu and Terraform |
| megacorp/payments/payments-api | the payments team | a Java service: one include, one component, two adjustments | throughout |
| megacorp/web/web-portal | the web team | a Node front end with no .gitlab-ci.yml at all | org-wide reuse |
| megacorp/data/mono | the data team | a monorepo, one repository holding many services, that generates its own child pipelines | pipelines as building blocks |
| megacorp/legacy/billing-batch | nobody, any more | an old file from 2019, using older techniques: YAML anchors, a file included from a URL, and a trigger token | YAML-level reuse |
A service's whole pipeline file#
This is all the configuration the payments team maintains, apart from two small adjustments further down the same file:
include:
- project: megacorp/devops/ci-templates
ref: v4.2.0
file: pipelines/java-service.yml
- component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
inputs:
stage: test
project_key: payments-apiinclude pulls another file's contents into this one, as if they had been
typed here. The first include is MegaCorp's "golden pipeline" for Java services: a
complete, ready-made pipeline that any Java project can take whole. This is what it pulls
in:
include:
- project: megacorp/devops/ci-templates
ref: v4.2.0
file:
- templates/base.yml
- templates/snippets.yml
- templates/rules.yml
- templates/workflow.yml
- templates/java-maven.yml
- templates/container.yml
- templates/security.yml
- templates/deploy.ymlThat is eight shared files, all pinned to the tag v4.2.0. Pinned
means GitLab reads them as they were at that tag. A later change by the central team
doesn't reach payments-api until someone moves the tag in this file.
Those files define hidden jobs. Their names start with a dot, and they never run by themselves. The golden pipeline turns three of them into real jobs:
maven-build:
extends: .maven-build
maven-test:
extends: .maven-test
image-build:
extends: .image-build
needs: [maven-build]So there are four files between payments-api and the script its
maven-test job runs. Chapter 14 walks
that chain one link at a time.
What no file in the project shows#
Some of MegaCorp's configuration isn't in any repository the payments team can read:
- Settings. Variables set on the instance and on the megacorp group,
and web-portal's pointer to a pipeline file in another project. The book shows them in
files called
SETTINGS.yml, which GitLab never reads. - Policies. On GitLab Ultimate, the security team's policy project adds a secret-detection scan to every pipeline. It also adds a job that lists everything the build contains, known as an SBOM (software bill of materials).
- The runners. Their configuration sets environment variables and cloud permissions for every job.
Here are the variables set on the megacorp group. Every project in the group receives them:
# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
- key: MAVEN_CLI_OPTS
value: "--batch-mode --errors --show-version -s .m2/settings.xml"
- key: AWS_ACCOUNT_ID
value: "123456789012"
- key: MC_AZURE_TENANT_ID
value: "00000000-0000-0000-0000-000000000000"
- key: MC_ACR_REGISTRY
value: megacorpprod.azurecr.io
- key: MC_ACR_PUSH_CLIENT_ID
value: "00000000-0000-0000-0000-000000000000"
# the gitlab-acr-push application in Entra ID, which mc_acr_push signs in as
- key: SONAR_HOST_URL
value: https://sonar.example.com
- key: SONAR_TOKEN
value: "(set in the UI)"
masked: true
protected: trueKeep MAVEN_CLI_OPTS in mind. It matters in
chapter 26.
Reading the specimen#
- Every file opens with a comment saying what it is and which chapters use it.
- Every quote is labelled with its project and path. When only part of a file is shown, the label also names the part, called a region.
- Files named
SETTINGS.ymldescribe settings made in GitLab's web pages. GitLab never reads them. - Every pipeline file in the specimen is valid. On the local GitLab, each one creates its pipelines. The deployer creates one only when a service passes it an image to deploy, by design. Like any real setup, the specimen also hides a few surprises, and the exercises find them.
The full, annotated listing is in the appendix.
- Name the seven MegaCorp projects a payments-api pipeline touches, and which team owns each.
- Read a 404 from another project as a permissions question, not a missing project, and know whose settings page to ask about.
- Find in the specimen the file behind any job you meet in the rest of the book.
Build and test#
Your team's pipeline starts with two jobs: one builds the service, and one tests it. Each job runs in a fresh container. So the build job keeps what it made as an artifact, which later jobs receive, and both jobs keep downloaded dependencies in a cache, which later pipelines reuse. The test job hands its results to GitLab as a report, so failing tests show on the pipeline and in merge requests.
A one-line change to the README, and the build job downloads every dependency again, from scratch, for four minutes. Yesterday's run had them. Its log says Checking cache for default-protected... and then nothing was found. Yesterday the key had the lockfile's name in it. What is default, and where did yesterday's cache go? (the card)
This part grows one service's pipeline, a chapter at a time. The service is
greeting-service, a small web service that answers GET /greet?name=Ada with
a greeting. It comes in Node and in Java: pick yours with any Node or
Java tab, and every tab in the book follows. Each chapter ends with a lab
that you do on your own GitLab. Nothing here needs the rest of the book; where a chapter
uses something that a later chapter explains fully, it links there.
Two jobs, two stages#
Here is the pipeline this chapter builds. Each column is a stage, and a stage starts only when the stage before it has passed:
As text
- build: build
- test: unit-tests
stages:
- build
- test
default:
image: node:24.21.0-alpine
build:
stage: build
script:
- npm ci --cache .npm --prefer-offline
- npm run build
cache:
key:
files:
- package-lock.json
paths:
- .npm/
artifacts:
paths:
- dist/
unit-tests:
stage: test
script:
- npm ci --cache .npm --prefer-offline
- npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=junit.xml
cache:
key:
files:
- package-lock.json
paths:
- .npm/
policy: pull
artifacts:
when: always
reports:
junit: junit.xmlstages:
- build
- test
default:
image: maven:3.9.16-eclipse-temurin-25
variables:
MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
MAVEN_CLI_OPTS: "--batch-mode --no-transfer-progress"
build:
stage: build
script:
- mvn $MAVEN_CLI_OPTS package -DskipTests
cache:
key:
files:
- pom.xml
paths:
- .m2/repository/
artifacts:
paths:
- target/greeting-service.jar
unit-tests:
stage: test
script:
- mvn $MAVEN_CLI_OPTS test
cache:
key:
files:
- pom.xml
paths:
- .m2/repository/
policy: pull
artifacts:
when: always
reports:
junit: target/surefire-reports/TEST-*.xmlFrom the top:
stageslists the stages in order.default: imagegives every job the same image unless a job names its own: Node 24, on Alpine Linux. The version is pinned, so the job behaves the same next month.buildinstalls the dependencies and builds.npm ciinstalls exactly what the lockfile,package-lock.json, lists.npm run buildthen writes the service todist/.unit-testsinstalls the dependencies again, because it runs in a new container, and runs the tests withnpm test.
stageslists the stages in order.default: imagegives every job the same image unless a job names its own: Maven 3.9 with Java 25. The version is pinned, so the job behaves the same next month.variablessets two values for Maven.MAVEN_CLI_OPTSholds the options that everymvncommand in the scripts is given:--batch-moderuns Maven without prompts, and--no-transfer-progresskeeps download progress out of the log.MAVEN_OPTSis different: no script mentions it, because Maven itself reads it from the environment. It tells Maven to keep its local repository, the folder where downloaded dependencies go, inside the project, which is what the cache below keeps.buildcompiles and packages. Maven'spackagecompiles the code and makes the jar,target/greeting-service.jar, and-DskipTestsleaves the tests to the next job.unit-testsdownloads the dependencies again, because it runs in a new container, and runs the tests withmvn test.
Artifacts: what later jobs receive#
artifacts: paths tells GitLab which files to keep when the job ends.
The build keeps dist/, the folder that npm run
build writes.
The build keeps target/greeting-service.jar, the jar
that Maven packages.
GitLab uploads them, and every job in a later stage downloads them before its script starts. That is how a later job, such as one that builds an image, gets the build's output without building again.
Without artifacts, the files vanish with the container, as the third break
in your first pipeline showed.
Chapter 31 covers artifacts in full.
Cache: what later pipelines reuse#
Every job downloads its dependencies again. A cache keeps them between pipelines, so that the next job starts from what the last one downloaded:
cache: key: filesnames the file whose content decides the cache:package-lock.json. Change the dependencies, and the key changes, so no job restores a stale cache.cache: pathsis what is kept: npm's download folder,.npm/. The jobs point npm at it with--cache .npm, and--prefer-offlinemakes npm use a package it already has there before asking the registry for it.policy: pullonunit-testsmeans it reads the cache but never writes it back. Onlybuildsaves it.
cache: key: filesnames the file whose content decides the cache:pom.xml. Change the dependencies, and the key changes, so no job restores a stale cache.cache: pathsis what is kept: Maven's local repository,.m2/repository/. That is the folderMAVEN_OPTSpoints Maven at. Without that variable, Maven would download into the container's home folder, and a cache can only keep files inside the project.policy: pullonunit-testsmeans it reads the cache but never writes it back. Onlybuildsaves it.
A pipeline's first run finds no cache. The next one restores it, and its log says so. A cache can still miss at any time, for example on a different runner, so a job must work without it: a cache makes jobs faster, never correct. Chapter 31 explains why.
Test results GitLab can read#
artifacts: reports: junit hands GitLab a file of test results in the JUnit
format. GitLab shows them on the pipeline's Tests tab, and in a merge
request's test summary.
The test runner writes the file itself: --test-reporter=junit
with junit.xml as its destination, beside the usual output to the
terminal.
Maven's test plugin, Surefire, writes one file per test class under
target/surefire-reports/, and the pattern collects them all.
when: always keeps the report even when tests fail, which is when you need it
most.
- Do this
- Key the dependency cache on your lockfile, with
cache: key: files, and give every job that only reads the cachepolicy: pull. - Why
- The key changes whenever the dependencies change, so a job never restores an old cache by mistake, and jobs that only read never overwrite what the build saved.
- Watch out
- One fixed key, shared by every branch and every job, hands jobs files that are old or half-written. The card a job restores stale files shows what that looks like.
- Write a two-job pipeline that builds, tests and keeps its results, in your stack.
- Read a build job's log and say whether the cache was restored, and under which key.
- Tell an artifact from a cache, and choose the right one for a file a later job needs.
LabBuild and test greeting-serviceabout 15 min#
An account on your GitLab, and the service's files for your stack. Copy them from the panes below, keeping their paths.
{
"name": "greeting-service",
"version": "1.0.0",
"private": true,
"description": "The Node service whose pipeline the Build your team's pipeline labs grow.",
"type": "module",
"scripts": {
"build": "node build.js",
"start": "node dist/server.js",
"test": "node --test"
},
"dependencies": {
"zod": "4.6.4"
}
}// Builds the greeting. zod checks the name before it is used.
import { z } from "zod";
const Name = z.string().trim().min(1).max(40);
export function greet(name) {
return `Hello, ${Name.parse(name)}!`;
}// A tiny HTTP service: GET /greet?name=Ada answers {"message":"Hello, Ada!"}.
import { createServer } from "node:http";
import { greet } from "./greet.js";
const port = Number(process.env.PORT ?? 8080);
createServer((req, res) => {
const url = new URL(req.url, "http://localhost");
if (url.pathname !== "/greet") {
res.writeHead(404).end();
return;
}
try {
const body = JSON.stringify({ message: greet(url.searchParams.get("name") ?? "world") });
res.writeHead(200, { "content-type": "application/json" }).end(body);
} catch {
res.writeHead(400).end();
}
}).listen(port, () => console.log(`listening on ${port}`));// The build: copies src/ into dist/, the way a real build would bundle it.
import { cpSync, mkdirSync } from "node:fs";
mkdirSync("dist", { recursive: true });
cpSync("src", "dist", { recursive: true });
console.log("built dist/");// The service's tests, run with node --test.
import { test } from "node:test";
import assert from "node:assert/strict";
import { greet } from "../src/greet.js";
test("greet says hello", () => {
assert.equal(greet("Ada"), "Hello, Ada!");
});
test("greet refuses an empty name", () => {
assert.throws(() => greet(" "));
});# The greeting service's image: its built files and its production dependencies.
FROM node:24.21.0-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY dist/ dist/
EXPOSE 8080
CMD ["node", "dist/server.js"]Then run npm install --package-lock-only once, to create
package-lock.json.
<?xml version="1.0" encoding="UTF-8"?>
<!-- The Java service whose pipeline the Build your team's pipeline labs grow. -->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>greeting-service</artifactId>
<version>1.0.0</version>
<packaging>jar</packaging>
<properties>
<maven.compiler.release>25</maven.compiler.release>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>6.1.3</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<finalName>greeting-service</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.16.0</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.6.0</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<archive>
<manifest>
<mainClass>com.example.greeting.Server</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>
</project>package com.example.greeting;
/** Builds the greeting, after checking the name. */
public final class Greeter {
private Greeter() {
}
public static String greet(String name) {
String n = name == null ? "" : name.strip();
if (n.isEmpty() || n.length() > 40) {
throw new IllegalArgumentException("a name is 1 to 40 characters");
}
return "Hello, " + n + "!";
}
}package com.example.greeting;
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.URLDecoder;
import java.nio.charset.StandardCharsets;
/** A tiny HTTP service: GET /greet?name=Ada answers {"message":"Hello, Ada!"}. */
public final class Server {
private Server() {
}
public static void main(String[] args) throws IOException {
int port = Integer.parseInt(System.getenv().getOrDefault("PORT", "8080"));
HttpServer server = HttpServer.create(new InetSocketAddress(port), 0);
server.createContext("/greet", exchange -> {
String query = exchange.getRequestURI().getRawQuery();
String name = "world";
if (query != null && query.startsWith("name=")) {
name = URLDecoder.decode(query.substring(5), StandardCharsets.UTF_8);
}
int status = 200;
String body;
try {
String message = Greeter.greet(name).replace("\\", "\\\\").replace("\"", "\\\"");
body = "{\"message\":\"" + message + "\"}";
} catch (IllegalArgumentException e) {
status = 400;
body = "";
}
byte[] bytes = body.getBytes(StandardCharsets.UTF_8);
exchange.getResponseHeaders().set("content-type", "application/json");
exchange.sendResponseHeaders(status, bytes.length == 0 ? -1 : bytes.length);
try (OutputStream out = exchange.getResponseBody()) {
out.write(bytes);
}
});
server.start();
System.out.println("listening on " + port);
}
}package com.example.greeting;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import org.junit.jupiter.api.Test;
class GreeterTest {
@Test
void greetsByName() {
assertEquals("Hello, Ada!", Greeter.greet("Ada"));
}
@Test
void refusesAnEmptyName() {
assertThrows(IllegalArgumentException.class, () -> Greeter.greet(" "));
}
}# The greeting service's image: the jar the build job made, on a Java runtime.
FROM eclipse-temurin:25.0.4_7-jre-alpine
COPY target/greeting-service.jar /app/greeting-service.jar
EXPOSE 8080
CMD ["java", "-jar", "/app/greeting-service.jar"]Both stacks also have docs/README.md, any short text about the
service.
- Create a blank project named
greeting-service, clone it, and copy the files in. - Add
.gitlab-ci.ymlwith the pipeline above, for your stack. Commit and push. - Open the pipeline, then
build's log, and find where it saved the cache and uploaded the artifacts. - Change a line in
docs/README.md, and push again. Open the new pipeline'sbuildlog and look for the cache. - Open the pipeline's Tests tab.
Both pipelines pass. In the second, build finds the cache that the first
saved, saves it again, and uploads its artifact:
Restoring cache
Checking cache for 0_package-lock-ec7e7bfbf81f3b7ffde77ecb457c6376d0e9c398-protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache$ npm ci --cache .npm --prefer-offline$ npm run buildSaving cache for successful job
Creating cache 0_package-lock-ec7e7bfbf81f3b7ffde77ecb457c6376d0e9c398-protected...
.npm/: found 24 matching artifact files and directories
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally.
Created cache
Uploading artifacts for successful job
Uploading artifacts...
dist/: found 3 matching artifact files and directories
Uploading artifacts as "archive" to coordinator... 201 Created correlation_id=01M2DK95M6VDYK1HBZNF3X30P2 id=266 status=201 token=[REDACTED]Job succeededRestoring cache
Checking cache for 0_pom-2336814c62c17d1db9b38c969d72005f6a7e387d-protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache$ mvn $MAVEN_CLI_OPTS package -DskipTests[INFO] BUILD SUCCESSSaving cache for successful job
Creating cache 0_pom-2336814c62c17d1db9b38c969d72005f6a7e387d-protected...
.m2/repository/: found 724 matching artifact files and directoriesNo URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally.
Created cache
Uploading artifacts for successful job
Uploading artifacts...
target/greeting-service.jar: found 1 matching artifact files and directories
Uploading artifacts as "archive" to coordinator... 201 Created correlation_id=01M2DSHQ77BPCPRXVJ0WVNJV9X id=318 status=201 token=[REDACTED]Job succeeded- The key ends in
-protectedbecausemainis a protected branch. GitLab keeps the caches of protected pipelines apart from the rest. - "No URL provided" means this runner keeps its cache on its own machine. GitLab.com's runners share a cache stored in Google Cloud Storage, so their log reads differently here.
unit-testsrestores the same cache, and at the end says "Not uploading cache … due to policy". That ispolicy: pull.- The Tests tab counts the service's 2 tests, both passed.
In cache: key: files:, change the file's name to one that doesn't
exist, and push. GitLab ignores a missing file, and when none of the named files exists
the key is default. So the job looks for a different cache, and the one it
saved before is no longer found. Put the name back.
Merge request pipelines#
So far, every push to any branch runs the whole pipeline. Once your team works in merge
requests, you want one pipeline for each push to a merge request, with its result shown
on the merge request, and no second pipeline for the same push. A workflow at
the top of the file decides which kind of pipeline a push gets. Each job's
rules then decide whether the job is in it. A rule on changed files lets a
documentation change skip the build.
You open a merge request for the longer-names branch and push a fix to it. Two pipelines appear for the same commit, one on the branch and one on the merge request, and every job runs twice. The team lead asks why the runners are busy. What made the second pipeline, and which one is the right one? (the card)
Two kinds of pipeline#
GitLab can run two kinds of pipeline for the same commit:
- A branch pipeline runs when you push to a branch. Its
$CI_PIPELINE_SOURCEispush, and$CI_COMMIT_BRANCHholds the branch's name. - A merge request pipeline runs for a merge request, when it opens and
on every push to its source branch. Its
$CI_PIPELINE_SOURCEismerge_request_event, and$CI_COMMIT_BRANCHis not set at all. Its result shows on the merge request.
A job with no rules runs only in branch pipelines. As soon as jobs have
rules, they can run in merge request pipelines too, and one push can then create both
kinds. Chapter 21 covers every kind of pipeline.
A workflow: one pipeline per push#
A workflow is checked before any job. GitLab tries its rules in order, and
the first that matches decides. If none matches, there is no pipeline:
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: $CI_COMMIT_BRANCHworkflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: $CI_COMMIT_BRANCH- A merge request pipeline: create it.
- A branch pipeline for a branch that has an open merge request: don't. The merge request pipeline covers that push.
- Any other branch pipeline: create it. That includes
main.
The workflow is the same for both stacks. Chapter 22 explains workflows in full, including why the second rule matters (duplicate pipelines).
Skipping work when only the docs changed#
Each job now has rules of its own:
build:
stage: build
script:
- npm ci --cache .npm --prefer-offline
- npm run build
cache:
key:
files:
- package-lock.json
paths:
- .npm/
artifacts:
paths:
- dist/
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
- changes:
- src/**/*
- test/**/*
- package.json
- package-lock.jsonbuild:
stage: build
script:
- mvn $MAVEN_CLI_OPTS package -DskipTests
cache:
key:
files:
- pom.xml
paths:
- .m2/repository/
artifacts:
paths:
- target/greeting-service.jar
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
- changes:
- src/**/*
- pom.xml- On the default branch,
main, the job always runs, somainalways has a fresh build. - Anywhere else,
changesdecides. In a merge request pipeline, GitLab compares the merge request's changes with its target branch. The job runs only if a file matching one of the patterns changed.
So a merge request whose changes, taken together, touch only
docs/README.md matches no rule in any job. A pipeline with no jobs isn't
created, so that merge request gets no pipeline at all
(no jobs, no pipeline). One surprise: on a branch
pipeline, changes is always true for the first push of a new branch
(runs on a new branch).
unit-tests carries the same rules, repeated word for word. Repeated rules
drift apart, and Chapter 14 shows the usual way to write them
once. Chapter 23 covers rules in full.
As text
- build: build
- test: unit-tests
- Write a workflow that gives each push exactly one pipeline, of the right kind.
- Say which kind of pipeline a push will get before you push it.
- Skip the build when only documentation changed, and explain the new-branch surprise.
LabOne pipeline per pushabout 15 min#
greeting-service from the previous lab.
- Replace
.gitlab-ci.ymlwith this chapter's pipeline for your stack, and push tomain.practice · node-servicesteps/2-merge-requests.yml# greeting-service (Node): the pipeline at the end of "Merge request pipelines". workflow: rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never - if: $CI_COMMIT_BRANCH stages: - build - test default: image: node:24.21.0-alpine build: stage: build script: - npm ci --cache .npm --prefer-offline - npm run build cache: key: files: - package-lock.json paths: - .npm/ artifacts: paths: - dist/ rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - test/**/* - package.json - package-lock.json unit-tests: stage: test script: - npm ci --cache .npm --prefer-offline - npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=junit.xml cache: key: files: - package-lock.json paths: - .npm/ policy: pull artifacts: when: always reports: junit: junit.xml rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - test/**/* - package.json - package-lock.jsonpractice · java-servicesteps/2-merge-requests.yml# greeting-service (Java): the pipeline at the end of "Merge request pipelines". workflow: rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never - if: $CI_COMMIT_BRANCH stages: - build - test default: image: maven:3.9.16-eclipse-temurin-25 variables: MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository" MAVEN_CLI_OPTS: "--batch-mode --no-transfer-progress" build: stage: build script: - mvn $MAVEN_CLI_OPTS package -DskipTests cache: key: files: - pom.xml paths: - .m2/repository/ artifacts: paths: - target/greeting-service.jar rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - pom.xml unit-tests: stage: test script: - mvn $MAVEN_CLI_OPTS test cache: key: files: - pom.xml paths: - .m2/repository/ policy: pull artifacts: when: always reports: junit: target/surefire-reports/TEST-*.xml rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - pom.xml - Create a branch,
longer-names. In the greeting code, allow names of up to 60 characters instead of 40, and push the branch. - Open a merge request from
longer-namesintomain. - Change the limit to 80 and push again. Count the new pipelines, then close the merge request.
- From
main, create another branch,docs-only, that changes onlydocs/README.md, and push it. Then open a merge request from it, and look for its pipeline. - Close that merge request too.
What each step created, when this lab ran on GitLab 19.3:
| You | Pipelines | Jobs |
|---|---|---|
push the new file to main | one branch pipeline | build, unit-tests |
push longer-names, before the merge request | one branch pipeline | build, unit-tests |
| open the merge request | one merge request pipeline | build, unit-tests |
| push again, with the merge request open | one merge request pipeline, and no branch pipeline | build, unit-tests |
push docs-only | one branch pipeline: changes is true for a new branch's first push | build, unit-tests |
| open the docs-only merge request | none | none |
The merge request pipelines are listed on the merge request, under
Pipelines. Their jobs ran because the merge request changed the greeting
code. The docs-only merge request changed no file that a changes rule names,
so it got no pipeline.
Delete the workflow's second rule and push a code change to the branch again. Two pipelines appear for the one push, one of each kind, and each job runs twice.
Images, the registry and variables#
Production runs greeting-service as a container image, so the pipeline builds one and
pushes it to the project's container registry, tagged with the commit it was built from.
The values that jobs need come from variables: ordinary ones in the YAML, and secrets in
the project's settings. A secret is masked, so logs never show it, and
protected, so only protected branches such as main receive it.
build-image passes on main. On your branch, the same job fails at docker login with unauthorized, and the token it prints is empty. Nothing in the file mentions a branch. Where does the token come from, and why does main get it and your branch not? (the card)
The project's container registry#
Every GitLab project can hold container images in its container registry, under Deploy › Container Registry. On a self-managed GitLab, an administrator must turn the registry on first. A job reaches the registry through variables that GitLab sets for it:
| Variable | Holds |
|---|---|
CI_REGISTRY | the registry's address |
CI_REGISTRY_IMAGE | the registry's address followed by the project's path: this project's image name |
CI_REGISTRY_USER, CI_REGISTRY_PASSWORD | a user name and password that can push to this project's registry |
CI_COMMIT_SHORT_SHA | the first eight characters of the commit's SHA |
Building an image in a job#
A new job builds the image from the service's Dockerfile and pushes it:
build-image:
stage: package
image: docker:29.8.0-cli
services:
- docker:29.8.0-dind
variables:
DOCKER_TLS_CERTDIR: "/certs"
script:
- echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
- docker build --provenance=false --tag "$IMAGE" .
- docker push "$IMAGE"
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHbuild-image:
stage: package
image: docker:29.8.0-cli
services:
- docker:29.8.0-dind
variables:
DOCKER_TLS_CERTDIR: "/certs"
script:
- echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
- docker build --provenance=false --tag "$IMAGE" .
- docker push "$IMAGE"
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH- The job runs in
docker:29.8.0-cli, which has the Docker command but no Docker engine. The engine runs beside it as a service, a second container:docker:29.8.0-dind, short for Docker-in-Docker. DOCKER_TLS_CERTDIRmakes the job and the engine talk over TLS.- The job logs in with the registry's variables, builds the image, and pushes it as
$IMAGE.--provenance=falseleaves out the build's provenance record, so the push is a single image rather than an index with an extra attestation manifest. On the local GitLab this book was checked on, the push failed without it. - The rule builds images only on
main.
Docker-in-Docker needs a runner that allows privileged containers. GitLab.com's own Linux runners do. On your organisation's GitLab, ask your platform team: many allow it only on some runners, or offer another way to build images (chapter 30).
The Dockerfile starts from a pinned runtime image, and copies in what the build job kept as an artifact:
# The greeting service's image: its built files and its production dependencies.
FROM node:24.21.0-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY dist/ dist/
EXPOSE 8080
CMD ["node", "dist/server.js"]# The greeting service's image: the jar the build job made, on a Java runtime.
FROM eclipse-temurin:25.0.4_7-jre-alpine
COPY target/greeting-service.jar /app/greeting-service.jar
EXPOSE 8080
CMD ["java", "-jar", "/app/greeting-service.jar"]Variables in the YAML#
variables:
IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHAvariables:
MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
MAVEN_CLI_OPTS: "--batch-mode --no-transfer-progress"
IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHAThe two MAVEN_ variables are the ones from
Build and test. IMAGE is the new line.
A top-level variables: block gives every job a value. Here
IMAGE is built from two variables that GitLab sets: the project's image name,
and the commit's short SHA. Tagging with the commit means that each image says exactly
which code it holds, and a later deploy can name that one image. Values that aren't
secret belong here, in the file, where reviewers see them (chapter 26).
Secrets in settings: masked and protected#
Secrets never go in the file. You add them under Settings › CI/CD › Variables, where two settings guard them:
- Masked replaces the value with
[MASKED]wherever a job prints it. GitLab can only mask a value it can find reliably, so the value must be a single line, with no spaces, at least 8 characters long. Masked is the default for a new variable. - Protect variable gives the value only to pipelines on protected branches and tags.
This chapter's pipeline has a job that exists only to show both, and that you would never keep:
secret-check:
stage: test
script:
- echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'"On main, which is protected, its log shows [MASKED]. On a
feature branch the value is empty. That second fact is behind a common failure: a job
that works on main and fails on every branch
(fails only on branches). Chapter 27
covers secrets in full.
As text
- build: build
- test: secret-check, unit-tests
- package: build-image
- Build an image in a job and push it to the project's registry, tagged with its commit.
- Put a secret where it belongs, masked and protected, and predict which pipelines will see it empty.
- Explain the difference between a variable in the YAML and one in the settings.
LabPush an image, and watch a secretabout 20 min#
greeting-service from the previous lab.
- Under Settings › CI/CD › Variables, add a variable. Name it
DEPLOY_TOKEN, give it any value of 8 or more characters without spaces, such aslab-deploy-token-4711, make it Masked, and tick Protect variable. - Replace
.gitlab-ci.ymlwith this chapter's pipeline, and push tomain.practice · node-servicesteps/3-image.yml# greeting-service (Node): the pipeline at the end of "Images, the registry and variables". workflow: rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never - if: $CI_COMMIT_BRANCH stages: - build - test - package default: image: node:24.21.0-alpine variables: IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA build: stage: build script: - npm ci --cache .npm --prefer-offline - npm run build cache: key: files: - package-lock.json paths: - .npm/ artifacts: paths: - dist/ rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - test/**/* - package.json - package-lock.json unit-tests: stage: test script: - npm ci --cache .npm --prefer-offline - npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=junit.xml cache: key: files: - package-lock.json paths: - .npm/ policy: pull artifacts: when: always reports: junit: junit.xml rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - test/**/* - package.json - package-lock.json secret-check: stage: test script: - echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'" build-image: stage: package image: docker:29.8.0-cli services: - docker:29.8.0-dind variables: DOCKER_TLS_CERTDIR: "/certs" script: - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin - docker build --provenance=false --tag "$IMAGE" . - docker push "$IMAGE" rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHpractice · java-servicesteps/3-image.yml# greeting-service (Java): the pipeline at the end of "Images, the registry and variables". workflow: rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never - if: $CI_COMMIT_BRANCH stages: - build - test - package default: image: maven:3.9.16-eclipse-temurin-25 variables: MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository" MAVEN_CLI_OPTS: "--batch-mode --no-transfer-progress" IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA build: stage: build script: - mvn $MAVEN_CLI_OPTS package -DskipTests cache: key: files: - pom.xml paths: - .m2/repository/ artifacts: paths: - target/greeting-service.jar rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - pom.xml unit-tests: stage: test script: - mvn $MAVEN_CLI_OPTS test cache: key: files: - pom.xml paths: - .m2/repository/ policy: pull artifacts: when: always reports: junit: target/surefire-reports/TEST-*.xml rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - pom.xml secret-check: stage: test script: - echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'" build-image: stage: package image: docker:29.8.0-cli services: - docker:29.8.0-dind variables: DOCKER_TLS_CERTDIR: "/certs" script: - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin - docker build --provenance=false --tag "$IMAGE" . - docker push "$IMAGE" rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - Open
secret-check's log, thenbuild-image's. - Open Deploy › Container Registry.
- Push any small change to a new branch, and open that pipeline's
secret-checklog.
On main, which is protected, secret-check is given the value,
and the log masks it:
$ echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'"
DEPLOY_TOKEN is '[MASKED]'Job succeeded$ echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'"
DEPLOY_TOKEN is '[MASKED]'Job succeededbuild-image logs in, builds the image, names it with the commit's short
SHA, and pushes it:
$ echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
Login Succeeded$ docker build --provenance=false --tag "$IMAGE" .#10 naming to gitlab.example.com:5050/practice/node-service:0648d3dd done$ docker push "$IMAGE"
The push refers to repository [gitlab.example.com:5050/practice/node-service]0648d3dd: digest: sha256:df19c9e8709250340d0fa909605eef0681c9e4448045eadc8cdd74b1ef89e46d size: 1809Job succeededDeploy › Container Registry lists the project's image with one tag:
that same short SHA, 0648d3dd here.
$ echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin
Login Succeeded$ docker build --provenance=false --tag "$IMAGE" .#7 naming to gitlab.example.com:5050/practice/java-service:d74ed00d done$ docker push "$IMAGE"
The push refers to repository [gitlab.example.com:5050/practice/java-service]d74ed00d: digest: sha256:e25c1d658c4a8bd0fcfd3829bc7235929bb93beb20f6aec4ca37aa6f33363c73 size: 1432Job succeededDeploy › Container Registry lists the project's image with one tag:
that same short SHA, d74ed00d here.
On the branch, the rule keeps build-image out of the pipeline, and
secret-check's value is empty:
$ echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'"
DEPLOY_TOKEN is ''Job succeeded$ echo "DEPLOY_TOKEN is '$DEPLOY_TOKEN'"
DEPLOY_TOKEN is ''Job succeededEnvironments and deploys#
A deploy job names an environment, such as staging or production, and GitLab
records each deployment: which commit went where, and when. greeting-service deploys to
staging on every push to main, and to production only when a person starts
the job. Until someone does, the pipeline shows as blocked.
Friday's pipeline on main is still marked blocked. Every job passed except one, which shows a play button nobody pressed. Staging was deployed on Friday evening; production was not. Was something broken over the weekend, or is this what waiting looks like? (the card)
A deploy job names its environment#
deploy-staging:
stage: deploy
image: alpine:3.22
script:
- echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN"
environment:
name: staging
url: https://staging.greeting.example.com
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
deploy-production:
stage: deploy
image: alpine:3.22
script:
- echo "Deploying $IMAGE to production, with the token $DEPLOY_TOKEN"
environment:
name: production
url: https://greeting.example.com
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: manualdeploy-staging:
stage: deploy
image: alpine:3.22
script:
- echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN"
environment:
name: staging
url: https://staging.greeting.example.com
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
deploy-production:
stage: deploy
image: alpine:3.22
script:
- echo "Deploying $IMAGE to production, with the token $DEPLOY_TOKEN"
environment:
name: production
url: https://greeting.example.com
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: manualenvironment: name makes a job a deployment job. GitLab lists
every environment under Operate › Environments, each with its
deployments, and url puts a link to the running service beside it.
These deploy scripts only print what they would do, with the token masked. A real deploy job runs whatever your organisation deploys with: a cloud's command-line tool, or a commit to a configuration repository that a tool in the cluster applies. Chapter 40 covers the rest.
Production waits for a person#
The production job's rule adds when: manual. The job appears in the
pipeline with a play button, and runs only when someone selects it. A manual job added
by a rule is not allowed to fail, so the pipeline waits for it and shows as blocked
(a play button that waits).
Two more guards exist on paid tiers. A protected environment Premium lets only chosen people deploy to it, and deployment approvals Premium make a deploy wait for someone to approve it. Neither is needed for this lab.
As text
- build: build
- test: unit-tests
- package: build-image
- deploy: deploy-production (manual), deploy-staging
- Give a deploy job an environment, so that GitLab records what went where.
- Make production wait for a person, and read a blocked pipeline as a job waiting for its button.
- Find every deployment of an environment under Operate › Environments.
LabDeploy to staging, then to productionabout 10 min#
greeting-service from the previous lab, with its DEPLOY_TOKEN
variable.
- Replace
.gitlab-ci.ymlwith this chapter's pipeline, and push tomain. The temporarysecret-checkjob is gone.practice · node-servicesteps/4-deploy.yml# greeting-service (Node): the pipeline at the end of "Environments and deploys". workflow: rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never - if: $CI_COMMIT_BRANCH stages: - build - test - package - deploy default: image: node:24.21.0-alpine variables: IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA build: stage: build script: - npm ci --cache .npm --prefer-offline - npm run build cache: key: files: - package-lock.json paths: - .npm/ artifacts: paths: - dist/ rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - test/**/* - package.json - package-lock.json unit-tests: stage: test script: - npm ci --cache .npm --prefer-offline - npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=junit.xml cache: key: files: - package-lock.json paths: - .npm/ policy: pull artifacts: when: always reports: junit: junit.xml rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - test/**/* - package.json - package-lock.json build-image: stage: package image: docker:29.8.0-cli services: - docker:29.8.0-dind variables: DOCKER_TLS_CERTDIR: "/certs" script: - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin - docker build --provenance=false --tag "$IMAGE" . - docker push "$IMAGE" rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH deploy-staging: stage: deploy image: alpine:3.22 script: - echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN" environment: name: staging url: https://staging.greeting.example.com rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH deploy-production: stage: deploy image: alpine:3.22 script: - echo "Deploying $IMAGE to production, with the token $DEPLOY_TOKEN" environment: name: production url: https://greeting.example.com rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH when: manualpractice · java-servicesteps/4-deploy.yml# greeting-service (Java): the pipeline at the end of "Environments and deploys". workflow: rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never - if: $CI_COMMIT_BRANCH stages: - build - test - package - deploy default: image: maven:3.9.16-eclipse-temurin-25 variables: MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository" MAVEN_CLI_OPTS: "--batch-mode --no-transfer-progress" IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA build: stage: build script: - mvn $MAVEN_CLI_OPTS package -DskipTests cache: key: files: - pom.xml paths: - .m2/repository/ artifacts: paths: - target/greeting-service.jar rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - pom.xml unit-tests: stage: test script: - mvn $MAVEN_CLI_OPTS test cache: key: files: - pom.xml paths: - .m2/repository/ policy: pull artifacts: when: always reports: junit: target/surefire-reports/TEST-*.xml rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - pom.xml build-image: stage: package image: docker:29.8.0-cli services: - docker:29.8.0-dind variables: DOCKER_TLS_CERTDIR: "/certs" script: - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin - docker build --provenance=false --tag "$IMAGE" . - docker push "$IMAGE" rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH deploy-staging: stage: deploy image: alpine:3.22 script: - echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN" environment: name: staging url: https://staging.greeting.example.com rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH deploy-production: stage: deploy image: alpine:3.22 script: - echo "Deploying $IMAGE to production, with the token $DEPLOY_TOKEN" environment: name: production url: https://greeting.example.com rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH when: manual - Open the pipeline. Note its status, and
deploy-staging's log. - Open Operate › Environments.
- Back in the pipeline, run
deploy-productionwith its play button.
deploy-staging runs on its own:
$ echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN"
Deploying gitlab.example.com:5050/practice/node-service:7d625fae to staging, with the token [MASKED]Job succeeded$ echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN"
Deploying gitlab.example.com:5050/practice/java-service:fbb6b2d6 to staging, with the token [MASKED]Job succeededThen the pipeline stops with the badge Blocked. Every job has passed
except deploy-production, which waits for its play button.
Operate › Environments lists staging, with its URL and the
deployment you just made.
After you run deploy-production, it prints the same line for production,
and the pipeline passes. The environments list then shows both environments, each with
its URL. Production's URL can take a minute to appear.
OpenTofu in your pipeline#
Infrastructure that your team owns, such as a queue or a storage bucket, is often
written as code with OpenTofu and changed through the pipeline. A merge request shows the
plan, which is what would change, and merging to main applies it. GitLab can
keep OpenTofu's state for you, so there is nothing else to set up.
You add a job that runs tofu plan in OpenTofu's own image. It fails at once, before any of your commands, with the tool's usage text in the log. The same script works on your laptop. What ran instead of your script? (the card)
OpenTofu in two minutes#
OpenTofu is an open-source tool that creates and changes infrastructure from files of
code. It began as a copy of Terraform: its command is tofu where Terraform's
is terraform, and it reads the same kind of files, written in a language
called HCL. The words you need:
| Word | Means |
|---|---|
| configuration | the .tf files: what should exist |
| provider | a plugin that talks to one kind of system, such as a cloud |
| plan | the changes OpenTofu would make: what it would add, change and destroy |
| apply | making those changes |
| state | OpenTofu's record of what it has made, which the next plan compares with |
This lab's configuration makes a random name for the team's queue. Its provider,
random, needs no cloud account, so every plan and apply is real:
terraform {
required_version = ">= 1.12"
backend "http" {}
required_providers {
random = {
source = "hashicorp/random"
version = "3.9.1"
}
}
}variable "team" {
type = string
default = "payments"
}
resource "random_pet" "queue" {
prefix = var.team
length = 2
}
output "queue_name" {
value = random_pet.queue.id
}State that GitLab keeps#
The empty backend "http" {} block says the state lives on a web server,
here GitLab. Variables tell OpenTofu where, and how to sign in:
variables:
TF_STATE_NAME: default
TF_HTTP_ADDRESS: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/$TF_STATE_NAME
TF_HTTP_LOCK_ADDRESS: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/$TF_STATE_NAME/lock
TF_HTTP_UNLOCK_ADDRESS: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/$TF_STATE_NAME/lock
TF_HTTP_LOCK_METHOD: POST
TF_HTTP_UNLOCK_METHOD: DELETE
TF_HTTP_USERNAME: gitlab-ci-token
TF_HTTP_PASSWORD: $CI_JOB_TOKEN| Variable | Tells OpenTofu |
|---|---|
TF_STATE_NAME | the state's name, default. A project can keep several states, one for each name |
TF_HTTP_ADDRESS | where to read and write the state: this project's state, under GitLab's API |
TF_HTTP_LOCK_ADDRESS, TF_HTTP_UNLOCK_ADDRESS | where to lock the state before a change and unlock it after, so that two jobs never write it at once |
TF_HTTP_LOCK_METHOD, TF_HTTP_UNLOCK_METHOD | which HTTP method GitLab expects for each: POST to lock, DELETE to unlock |
TF_HTTP_USERNAME, TF_HTTP_PASSWORD | how to sign in: the fixed user name gitlab-ci-token, and the job's own token |
OpenTofu reads every TF_HTTP_ variable from the environment, which is why
the backend block can stay empty and no script mentions them. The state's
address is under the project's API, and the job signs in with its own job
token. GitLab lists the states under Operate › Terraform states. A
Developer can plan against the state, but applying a change needs the Maintainer role, so
merge requests can plan, and only the pipeline on main applies.
Plan in the merge request, apply on merge#
default:
image:
name: ghcr.io/opentofu/opentofu:1.12.6
entrypoint: [""]
before_script:
- tofu init -input=falsevalidate:
stage: validate
script:
- tofu fmt -check
- tofu validate
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHplan:
stage: plan
script:
- apk add --no-cache jq
- tofu plan -input=false -out=plan.cache
- tofu show -json plan.cache | jq -r '([.resource_changes[]?.change.actions?]|flatten)|{"create":(map(select(.=="create"))|length),"update":(map(select(.=="update"))|length),"delete":(map(select(.=="delete"))|length)}' > plan.json
artifacts:
paths:
- plan.cache
reports:
terraform: plan.json
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHapply:
stage: apply
script:
- tofu apply -input=false plan.cache
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHdefaultrunstofu initbefore every job, with the pinnedghcr.io/opentofu/opentofu:1.12.6image. Itsentrypointis emptied, because the image would otherwise runtofuinstead of your script (the script never runs).validatechecks the formatting and the configuration.validateandplancarry rules for merge request pipelines and formain. A job without rules runs only in branch pipelines, and the workflow drops those while a merge request is open, so without the rules the merge request would get no pipeline at all.plansaves the plan inplan.cache, then turns it into a summary withjq, GitLab's documented command. The summary is a report, so the merge request shows how many things the plan adds, changes and destroys.applyruns only onmain, and applies the plan saved by that pipeline'splanjob.
As text
- Push a change to a branch, and open a merge request
- The merge request pipeline validates and plans
- The merge request shows the plan's counts
- Merged into main? No: Nothing changes: the plan was a preview. Yes: the next step.
- The pipeline on main validates and plans again
- apply runs that plan, and GitLab saves the new state
Central teams often wrap these jobs in GitLab's OpenTofu component,
gitlab.com/components/opentofu/validate-plan-apply, included at a version
such as 4.8.1. You will recognise its jobs from these. On a self-managed
GitLab, an administrator mirrors the component's project first.
LegacyGitLab's Terraform templates
As text
- validate: validate
- plan: plan
- apply: apply
- Write a pipeline that plans in a merge request and applies on merge, with the state kept by GitLab.
- Empty a tool image's entrypoint when its default command swallows your script.
- Read a plan's counts on the merge request before anyone merges it.
LabOpenTofu: plan in a merge request, apply on mergeabout 15 min#
An account on your GitLab.
- Create a blank project named
team-infra. Addmain.tfand.gitlab-ci.yml, both shown in full here:practice · team-inframain.tf# team-infra: what the "OpenTofu in your pipeline" lab manages. random_pet needs no cloud # account, so every plan and apply is real. terraform { required_version = ">= 1.12" backend "http" {} required_providers { random = { source = "hashicorp/random" version = "3.9.1" } } } variable "team" { type = string default = "payments" } resource "random_pet" "queue" { prefix = var.team length = 2 } output "queue_name" { value = random_pet.queue.id }practice · team-infra.gitlab-ci.yml# team-infra: the pipeline of the "OpenTofu in your pipeline" lab. GitLab keeps the # state, a merge request shows the plan, and merging to main applies it. workflow: rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never - if: $CI_COMMIT_BRANCH stages: - validate - plan - apply variables: TF_STATE_NAME: default TF_HTTP_ADDRESS: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/$TF_STATE_NAME TF_HTTP_LOCK_ADDRESS: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/$TF_STATE_NAME/lock TF_HTTP_UNLOCK_ADDRESS: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/$TF_STATE_NAME/lock TF_HTTP_LOCK_METHOD: POST TF_HTTP_UNLOCK_METHOD: DELETE TF_HTTP_USERNAME: gitlab-ci-token TF_HTTP_PASSWORD: $CI_JOB_TOKEN default: image: name: ghcr.io/opentofu/opentofu:1.12.6 entrypoint: [""] before_script: - tofu init -input=false validate: stage: validate script: - tofu fmt -check - tofu validate rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH plan: stage: plan script: - apk add --no-cache jq - tofu plan -input=false -out=plan.cache - tofu show -json plan.cache | jq -r '([.resource_changes[]?.change.actions?]|flatten)|{"create":(map(select(.=="create"))|length),"update":(map(select(.=="update"))|length),"delete":(map(select(.=="delete"))|length)}' > plan.json artifacts: paths: - plan.cache reports: terraform: plan.json rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH apply: stage: apply script: - tofu apply -input=false plan.cache rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - Push to
main. When the pipeline finishes, openapply's log, then Operate › Terraform states. - Create a branch. On it, change
length = 2tolength = 3, and push. Then open a merge request from the branch intomain. - Look at the merge request once its pipeline finishes.
- Merge it, and open the
applylog of the pipeline onmain.
The first pipeline on main creates the queue's name, and GitLab saves the
state:
$ tofu apply -input=false plan.cache
Acquiring state lock. This may take a few moments...
random_pet.queue: Creating...
random_pet.queue: Creation complete after 0s [id=payments-helped-sunfish]
Releasing state lock. This may take a few moments...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Outputs:
queue_name = "payments-helped-sunfish"Job succeededOperate › Terraform states lists one state, default. In the
merge request, plan shows what the change would do. The length can't be
changed in place, so OpenTofu would replace the resource:
$ tofu plan -input=false -out=plan.cache
Acquiring state lock. This may take a few moments...
random_pet.queue: Refreshing state... [id=payments-helped-sunfish]
OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
OpenTofu will perform the following actions:
# random_pet.queue must be replaced
-/+ resource "random_pet" "queue" {
~ id = "payments-helped-sunfish" -> (known after apply)
~ length = 2 -> 3 # forces replacement
# (2 unchanged attributes hidden)
}
Plan: 1 to add, 0 to change, 1 to destroy.
Changes to Outputs:
~ queue_name = "payments-helped-sunfish" -> (known after apply)Saved the plan to: plan.cacheReleasing state lock. This may take a few moments...plan.json: found 1 matching artifact files and directories
Uploading artifacts as "terraform" to coordinator... 201 Created correlation_id=01M2DR15CECXP68NQ0SWZNPXZ2 id=303 status=201 token=[REDACTED]Job succeededThe last lines upload plan.json as the terraform report, which
is how the merge request shows the same counts. Merging runs the pipeline on
main, and its apply destroys the old name and creates the new
one:
$ tofu apply -input=false plan.cache
Acquiring state lock. This may take a few moments...
random_pet.queue: Destroying... [id=payments-helped-sunfish]
random_pet.queue: Destruction complete after 0s
random_pet.queue: Creating...
random_pet.queue: Creation complete after 0s [id=payments-likely-careful-dane]
Releasing state lock. This may take a few moments...
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.
Outputs:
queue_name = "payments-likely-careful-dane"Job succeededThe state's serial, which goes up with each change, went from 1 to 2. Your names will
differ, because random_pet picks them at random.
The central team's templates#
Your file doesn't have to hold every job. A central team can publish templates, which are files of hidden jobs that you extend, and components, which are packaged jobs with inputs. Your file includes them at a version you choose. You then change only what your team needs, and the pipeline editor's Full configuration shows the result.
You replace your unit-tests job with two lines: extends: .node-test and a rule of your own. The job now runs in pipelines it never used to, on every tag. You added one rule; the template had three. Where did the other two go? (the card)
Play the central team#
In this lab you are both teams. A second project, central-templates,
holds the central team's files:
templates/node.ymlandtemplates/java.yml, each with one hidden job: a job whose name starts with a dot, which never runs itself and exists to be copied.templates/secret-scan/template.yml, a component: a job with typed inputs at the top, before a---line. Its job has rules of its own, so it runs in merge request pipelines as well as branch pipelines.
.node-test:
stage: test
image: node:24.21.0-alpine
script:
- npm ci --cache .npm --prefer-offline
- npm test
cache:
key:
files:
- package-lock.json
paths:
- .npm/
policy: pull.maven-test:
stage: test
image: maven:3.9.16-eclipse-temurin-25
variables:
MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
script:
- mvn --batch-mode --no-transfer-progress test
cache:
key:
files:
- pom.xml
paths:
- .m2/repository/
policy: pullspec:
inputs:
stage:
default: test
paths:
default: "."
description: The directory to look insecret-scan:
stage: $[[ inputs.stage ]]
image: alpine:3.22
script:
- echo "Looking for private keys in $[[ inputs.paths ]]"
- if grep -rl "BEGIN [A-Z ]*PRIVATE KEY" $[[ inputs.paths ]]; then echo "A private key is committed"; exit 1; fi
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCHThe central team tags the project 1.0.0. That tag is the version your file
asks for.
Include at a version#
include:
- project: practice/central-templates
ref: 1.0.0
file: templates/node.yml
- component: $CI_SERVER_FQDN/practice/central-templates/secret-scan@1.0.0
inputs:
stage: testinclude:
- project: practice/central-templates
ref: 1.0.0
file: templates/java.yml
- component: $CI_SERVER_FQDN/practice/central-templates/secret-scan@1.0.0
inputs:
stage: testprojectandfilename a file in another project, andref: 1.0.0fixes which version of it.componentnames the component by the instance, the project and its own name, then@1.0.0.$CI_SERVER_FQDNis your GitLab's address, so the line works on any instance.inputsfill in the component's inputs.
These files use the group practice. Change it to the group or user that
owns your two projects. Chapter 15 and
chapter 16 cover both kinds of include.
Extend, and change only what you must#
unit-tests:
extends: .node-test
script:
- npm ci --cache .npm --prefer-offline
- npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=junit.xml
artifacts:
when: always
reports:
junit: junit.xml
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
- changes:
- src/**/*
- test/**/*
- package.json
- package-lock.jsonunit-tests:
extends: .maven-test
artifacts:
when: always
reports:
junit: target/surefire-reports/TEST-*.xml
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
- changes:
- src/**/*
- pom.xmlextends copies the hidden job's settings into unit-tests, and
then the job's own settings are laid on top. Lists are replaced whole, and maps are merged
key by key.
So the job's own script replaces the template's, and its
artifacts and rules are added. The stage, the image and the cache
still come from the template.
So the job's own artifacts and rules are
added, and everything else still comes from the template: the stage, the image, the
MAVEN_OPTS variable, the script and the cache.
To see what GitLab made of it, open Build › Pipeline editor, then
Full configuration. Chapter 14
covers extends in full, including why changing a template job's
rules is the classic trap
(your rules replaced the template's).
As text
- build: build
- test: secret-scan, unit-tests
- package: build-image
- deploy: deploy-production (manual), deploy-staging
- Include a central team's template at a pinned version, and a component with its inputs.
- Extend a hidden job and change only what you must, then check the result in Full configuration.
- Explain why your own rules replaced the template's, and keep theirs with a reference.
LabInclude the central team's templatesabout 15 min#
greeting-service from the previous labs.
- Create a blank project named
central-templates, beside greeting-service. Add the files above, plus thisREADME.md:practice · central-templatesREADME.md# central-templates The central team's project in the "central team's templates" lab: a Node and a Java template with hidden jobs, and one CI/CD component. Tag it 1.0.0 before a service includes it. - Under Code › Tags, select New tag. Name it
1.0.0, create it frommain, and select Create tag. - In greeting-service, replace
.gitlab-ci.ymlwith this chapter's pipeline, withpracticechanged to your group or user, and push tomain.practice · node-servicesteps/5-central-templates.yml# greeting-service (Node): the pipeline at the end of "The central team's templates". include: - project: practice/central-templates ref: 1.0.0 file: templates/node.yml - component: $CI_SERVER_FQDN/practice/central-templates/secret-scan@1.0.0 inputs: stage: test workflow: rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never - if: $CI_COMMIT_BRANCH stages: - build - test - package - deploy default: image: node:24.21.0-alpine variables: IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA build: stage: build script: - npm ci --cache .npm --prefer-offline - npm run build cache: key: files: - package-lock.json paths: - .npm/ artifacts: paths: - dist/ rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - test/**/* - package.json - package-lock.json unit-tests: extends: .node-test script: - npm ci --cache .npm --prefer-offline - npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=junit.xml artifacts: when: always reports: junit: junit.xml rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - test/**/* - package.json - package-lock.json build-image: stage: package image: docker:29.8.0-cli services: - docker:29.8.0-dind variables: DOCKER_TLS_CERTDIR: "/certs" script: - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin - docker build --provenance=false --tag "$IMAGE" . - docker push "$IMAGE" rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH deploy-staging: stage: deploy image: alpine:3.22 script: - echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN" environment: name: staging url: https://staging.greeting.example.com rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH deploy-production: stage: deploy image: alpine:3.22 script: - echo "Deploying $IMAGE to production, with the token $DEPLOY_TOKEN" environment: name: production url: https://greeting.example.com rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH when: manualpractice · java-servicesteps/5-central-templates.yml# greeting-service (Java): the pipeline at the end of "The central team's templates". include: - project: practice/central-templates ref: 1.0.0 file: templates/java.yml - component: $CI_SERVER_FQDN/practice/central-templates/secret-scan@1.0.0 inputs: stage: test workflow: rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never - if: $CI_COMMIT_BRANCH stages: - build - test - package - deploy default: image: maven:3.9.16-eclipse-temurin-25 variables: MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository" MAVEN_CLI_OPTS: "--batch-mode --no-transfer-progress" IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA build: stage: build script: - mvn $MAVEN_CLI_OPTS package -DskipTests cache: key: files: - pom.xml paths: - .m2/repository/ artifacts: paths: - target/greeting-service.jar rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - pom.xml unit-tests: extends: .maven-test artifacts: when: always reports: junit: target/surefire-reports/TEST-*.xml rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - changes: - src/**/* - pom.xml build-image: stage: package image: docker:29.8.0-cli services: - docker:29.8.0-dind variables: DOCKER_TLS_CERTDIR: "/certs" script: - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password-stdin - docker build --provenance=false --tag "$IMAGE" . - docker push "$IMAGE" rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH deploy-staging: stage: deploy image: alpine:3.22 script: - echo "Deploying $IMAGE to staging, with the token $DEPLOY_TOKEN" environment: name: staging url: https://staging.greeting.example.com rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH deploy-production: stage: deploy image: alpine:3.22 script: - echo "Deploying $IMAGE to production, with the token $DEPLOY_TOKEN" environment: name: production url: https://greeting.example.com rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH when: manual - Open Build › Pipeline editor, then Full
configuration, and find
unit-tests. - Open the pipeline and
secret-scan's log.
The Full configuration shows unit-tests as GitLab runs it, with the
template's settings copied in:
unit-tests:
stage: test
image: node:24.21.0-alpine
script:
- npm ci --cache .npm --prefer-offline
- npm test -- --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit
--test-reporter-destination=junit.xml
cache:
key:
files:
- package-lock.json
paths:
- ".npm/"
policy: pull
extends: ".node-test"
artifacts:
when: always
reports:
junit:
- junit.xml
rules:
- if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
- changes:
- src/**/*
- test/**/*
- package.json
- package-lock.jsonThe stage, the image and the cache came from .node-test. The script is
the job's own, and replaced the template's.
unit-tests:
stage: test
image: maven:3.9.16-eclipse-temurin-25
variables:
MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
script:
- mvn --batch-mode --no-transfer-progress test
cache:
key:
files:
- pom.xml
paths:
- ".m2/repository/"
policy: pull
extends: ".maven-test"
artifacts:
when: always
reports:
junit:
- target/surefire-reports/TEST-*.xml
rules:
- if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
- changes:
- src/**/*
- pom.xmlThe job's own settings are only its artifacts and rules. Everything else came from
.maven-test.
GitLab writes the merged file its own way: list items sit level with their key, and a
long line wraps onto the next. extends is still listed, but its settings are
already copied in.
secret-scan came from the component, with its inputs filled in, and found
no key:
$ echo "Looking for private keys in ."
Looking for private keys in .
$ if grep -rl "BEGIN [A-Z ]*PRIVATE KEY" .; then echo "A private key is committed"; exit 1; fiJob succeededRead a new check's log once, to see that it really looked. In an Alpine image,
grep is BusyBox's, which lacks some of GNU grep's options, such as
--exclude-dir. Given one, it prints its usage and fails, and inside
if that failure reads as "no key found": the job passes without
looking.
Add a file to greeting-service containing the line
-----BEGIN PRIVATE KEY-----, and push. secret-scan fails and
names the file. Delete the file again.
Reuse at a glance#
Reuse means writing something once and using it in many places. In a large organisation, most of what your job does was written somewhere else. This book counts 37 ways that happens, in seven families. They run from YAML shortcuts inside one file to scripts built into an image.
Four questions sort them:
- How far does it reach?
- Is it pinned to a version?
- Did your project choose it?
- Can the merged view show it?
This chapter is the index to Part IV. The first table lists every mechanism, and the second turns what you see into the mechanism responsible. The chapters that follow give each mechanism its own card.
payments-api's file is nineteen lines. Its pipeline has eleven jobs, in five stages, and a job named policy-sbom that no file in the project mentions. Someone asks you to "just add a step after the build". Eleven jobs came from somewhere; you need to know from where before you touch anything. (the card)
You don't need to remember 37 of anything. You need to recognise which family a thing belongs to when you meet it, and know where that family is explained. This is how to use the chapter when a job surprises you:
As text
- Find what you see in the fingerprint table below
- It names the mechanism, and the chapter that explains it
- Does the merged view show it? Yes: Read it in Full configuration. No: the next step.
- Is it applied when the pipeline is created? Yes: Check the pipeline graph and the settings pages. No: the next step.
- It happens while the job runs: read the job log
Seven families#
The families are ordered by distance. The first lives inside your own file, and the last lives outside YAML altogether. The further away a mechanism is, the harder it is to see from your project:
| Family | Mechanisms | What they have in common | Chapter |
|---|---|---|---|
| Inside one YAML file | 3 | plain YAML shortcuts that GitLab never sees as such | 13 |
| Sharing between jobs | 5 | one job borrowing from another once the files are joined | 14 |
| Pulling in files | 5 | the kinds of include, each fetching configuration from somewhere | 15 |
| Parameters | 4 | values that change what shared configuration does | 16 |
| Whole pipelines | 4 | one pipeline starting another | 17 |
| Organisation-wide | 9 | settings and policies that reach projects without the project asking | 18 |
| Outside the YAML | 7 | logic in images, scripts, build tools and runners | 19 |
Four questions for every mechanism#
For each mechanism, four answers tell you where to look, and whom to ask:
| Question | The answers | Why it matters when something breaks |
|---|---|---|
| Scope: how far does it reach? | one file · one project · cross-project · organisation-wide | tells you which repository or settings page to open |
| Binding: is it pinned? | inline · copied once · pinned or floating · always live | tells you whether a change somewhere else can break you without a commit of yours |
| Control: did you choose it? | opt-in · inherited · enforced | tells you whether you can change it yourself, or have to ask someone |
| Visibility: does the merged view show it? | shown · keyword only · not shown | tells you whether Full configuration answers the question, or you need another tool |
All 37#
Hover over any cell for the long form of its answer.
| Mechanism | Scope | Binding | Control | Phase | Merged view |
|---|---|---|---|---|---|
| Inside one YAML file | |||||
| YAML anchors and aliases | One file | Inline | Opt-in | Parse | Shown |
| YAML merge key | One file | Inline | Opt-in | Parse | Shown |
| Hidden jobs | One project | Inline | Opt-in | Compile | Shown |
| Sharing between jobs | |||||
| extends | One project | Inline | Opt-in | Compile | Shown |
| !reference tags | One project | Inline | Opt-in | Compile | Shown |
| default: | One project | Inline | Opt-in | Compile | Keyword only |
| Top-level variables | One project | Inline | Opt-in | Compile | Keyword only |
| parallel: matrix | One project | Inline | Opt-in | Create | Keyword only |
| Pulling in files | |||||
| include: local | One project | Inline | Opt-in | Compile | Shown |
| include: project | Cross-project | Pinned or floating | Opt-in | Compile | Shown |
| include: remote | Cross-project | Pinned or floating | Opt-in | Compile | Shown |
| include: template | Organisation-wide | Always live | Opt-in | Compile | Shown |
| include: component | Cross-project | Pinned or floating | Opt-in | Compile | Shown |
| Parameters | |||||
| spec: inputs | Cross-project | Inline | Opt-in | Compile | Shown |
| Variables as parameters | One project | Inline | Opt-in | Run | Keyword only |
| Toggle variables | One project | Inline | Opt-in | Create | Keyword only |
| Pipeline inputs | One project | Inline | Opt-in | Create | Keyword only |
| Whole pipelines | |||||
| Parent–child pipelines | One project | Inline | Opt-in | Run | Keyword only |
| Dynamic child pipelines | One project | Inline | Opt-in | Run | Not shown |
| Multi-project pipelines | Cross-project | Always live | Opt-in | Run | Not shown |
| Pipeline trigger API | Cross-project | Always live | Opt-in | Run | Not shown |
| Organisation-wide | |||||
| Custom CI/CD configuration file | Cross-project | Pinned or floating | Enforced | Compile | Not shown |
| Auto DevOps | Organisation-wide | Always live | Inherited | Compile | Not shown |
| Instance and group CI/CD variables | Organisation-wide | Always live | Inherited | Create | Not shown |
| Instance template repository | Organisation-wide | Copied once | Opt-in | Compile | Shown |
| Pipeline execution policies | Organisation-wide | Always live | Enforced | Create | Not shown |
| Scan execution policies | Organisation-wide | Always live | Enforced | Create | Not shown |
| Compliance pipelines | Organisation-wide | Pinned or floating | Enforced | Compile | Not shown |
| Merge request approval policies | Organisation-wide | Always live | Enforced | Run | Not shown |
| Project and file templates | Organisation-wide | Copied once | Opt-in | Compile | Shown |
| Outside the YAML | |||||
| Toolbox images | Organisation-wide | Pinned or floating | Opt-in | Run | Keyword only |
| Scripts fetched at run time | Cross-project | Pinned or floating | Opt-in | Run | Keyword only |
| Shell function libraries | Cross-project | Pinned or floating | Opt-in | Run | Keyword only |
| Build-tool reuse | Cross-project | Pinned or floating | Opt-in | Run | Not shown |
| Internal command-line tools | Organisation-wide | Pinned or floating | Opt-in | Run | Keyword only |
| Runner configuration | Organisation-wide | Always live | Enforced | Run | Not shown |
| GitLab Functions | Cross-project | Pinned or floating | Opt-in | Run | Keyword only |
From what you see to the mechanism#
Search this table for the thing in front of you. Every mechanism is in it at least once.
| You see | Mechanism | Family |
|---|---|---|
&defaults after a key, and *defaults later in the same file | YAML anchors and aliases | Inside one YAML file |
<<: *defaults inside a job | YAML merge key | Inside one YAML file |
a job name that starts with a dot, such as .maven-base: | Hidden jobs | Inside one YAML file |
extends: .maven-base or extends: [.base, .with-cache] | extends | Sharing between jobs |
- !reference [.snippets, aws_login] inside a script or rules list | !reference tags | Sharing between jobs |
a top-level default: block setting image, before_script, tags or retry | default: | Sharing between jobs |
inherit: default: false in a job | default: | Sharing between jobs |
a top-level variables: block | Top-level variables | Sharing between jobs |
inherit: variables: false in a job | Top-level variables | Sharing between jobs |
parallel: matrix: in a job, and job names like test: [21] in the pipeline | parallel: matrix | Sharing between jobs |
include: local: ci/build.yml, a wildcard such as ci/*.yml, or a bare path under include: | include: local | Pulling in files |
include: - project: devops/ci-templates with ref: and file: | include: project | Pulling in files |
include: remote: https://…/pipeline.yml | include: remote | Pulling in files |
include: template: Security/SAST.gitlab-ci.yml | include: template | Pulling in files |
include: - component: $CI_SERVER_FQDN/devops/components/ecr-push@1.4.0 | include: component | Pulling in files |
a spec: inputs: header above a --- line | spec: inputs | Parameters |
$[[ inputs.jdk ]] inside a template | spec: inputs | Parameters |
a template reads $JAVA_VERSION, and your file sets it under variables: | Variables as parameters | Parameters |
variables such as SAST_DISABLED, SKIP_SONAR or DEPLOY_ENABLED, tested in rules: - if: | Toggle variables | Parameters |
| a form of inputs or variables on the Run pipeline page | Pipeline inputs | Parameters |
spec: inputs: at the top of a project's own .gitlab-ci.yml | Pipeline inputs | Parameters |
trigger: include: ci/deploy.yml | Parent–child pipelines | Whole pipelines |
| a downstream box labelled as a child pipeline | Parent–child pipelines | Whole pipelines |
trigger: include: - artifact: generated.yml with job: generate | Dynamic child pipelines | Whole pipelines |
trigger: project: platform/deployer | Multi-project pipelines | Whole pipelines |
| a downstream pipeline that belongs to another project | Multi-project pipelines | Whole pipelines |
curl --request POST …/trigger/pipeline with a trigger token, inside a script | Pipeline trigger API | Whole pipelines |
pipelines run but the repository has no .gitlab-ci.yml | Custom CI/CD configuration file | Organisation-wide |
the project's CI/CD settings name a path such as ci/web.yml@devops/ci-templates | Custom CI/CD configuration file | Organisation-wide |
no .gitlab-ci.yml, yet jobs appear with names such as build, test, code_quality and container_scanning | Auto DevOps | Organisation-wide |
| a job sees a variable that no file defines | Instance and group CI/CD variables | Organisation-wide |
| a variable listed in a group's or the instance's CI/CD settings | Instance and group CI/CD variables | Organisation-wide |
a .gitlab-ci.yml created from the Web Editor's template list, matching a file in the administrators' templates project | Instance template repository | Organisation-wide |
stages named .pipeline-policy-pre or .pipeline-policy-post | Pipeline execution policies | Organisation-wide |
| jobs that appear in the pipeline but in no file you can find | Pipeline execution policies | Organisation-wide |
a job name ending in :policy- followed by two numbers | Pipeline execution policies | Organisation-wide |
scanner jobs named with a hyphen and a number, such as secret-detection-1 | Scan execution policies | Organisation-wide |
| scanner jobs run although no file includes a scanner template | Scan execution policies | Organisation-wide |
| security scans that run on a schedule nobody in the project created | Scan execution policies | Organisation-wide |
| a compliance framework label on the project, and jobs from a file the project does not include | Compliance pipelines | Organisation-wide |
| a merge request needs extra approvals because of security findings or licences | Merge request approval policies | Organisation-wide |
a .gitlab-ci.yml that matches a starter template, while the central copy has moved on | Project and file templates | Organisation-wide |
image: registry.example.com/devops/ci-tools:3.2 and a script that calls a command no repository defines | Toolbox images | Outside the YAML |
curl -sSL … | bash, or a git clone of a scripts repository, inside script: | Scripts fetched at run time | Outside the YAML |
source /opt/ci/lib.sh | Shell function libraries | Outside the YAML |
functions defined in a hidden job and pulled into before_script with !reference | Shell function libraries | Outside the YAML |
script: mvn -P ci verify with a <parent> POM owned by another team | Build-tool reuse | Outside the YAML |
npm run ci driven by a shared configuration package | Build-tool reuse | Outside the YAML |
script: mc deploy --env prod: a company tool, not a public one | Internal command-line tools | Outside the YAML |
| behaviour that changes with the runner: environment variables, mounted files or cloud permissions that no file sets | Runner configuration | Outside the YAML |
a job with run: and a list of steps using func:, and no script: | GitLab Functions | Outside the YAML |
Pinned, floating, copied#
Binding is the question that most often explains "it broke and nobody changed anything". Every shared file is read at some ref: a branch, a tag or a commit:
- A commit never changes.
- A tag normally stays where it was put, although whoever owns it can move it.
- A branch moves every time someone merges into it.
MegaCorp shows all three kinds of binding:
include:
- project: megacorp/devops/ci-templates
ref: v4.2.0
file: pipelines/java-service.yml
- component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
inputs:
stage: test
project_key: payments-api- Pinned. payments-api includes the golden pipeline at the tag
v4.2.0, and the golden pipeline pins everything it includes to the same tag. Nothing upstream can change what payments-api gets until someone edits a version number. - Floating. web-portal's settings point at a pipeline file in ci-templates without naming a ref. It follows whatever that project's default branch holds today.
- Copied. billing-batch's file was created from a starter template in 2019, and has never heard from the template since.
- Sort any reuse mechanism you meet into one of five families, and know which chapter covers it.
- Recognise a mechanism from what it looks like in the file, with the fingerprints table.
- Say which of the four moments each mechanism acts at, and so where to look for its result.
Spot the bugName the mechanism#
Four things a developer found while investigating MegaCorp pipelines:
- A job in billing-batch has a line
<<: *defaults. - A pipeline has a stage called
.pipeline-policy-post, holding a job that no file in the project mentions. - web-portal's pipelines run, but the repository has no
.gitlab-ci.yml. - A
before_scriptcontains- !reference [.snippets, aws_login].
Show the answer
- A YAML merge key. Look for
&defaultsin the same file. - A pipeline execution policy. Look at the group's security policies.
- A custom CI/CD configuration file, set in the project's CI/CD settings. Auto DevOps can also produce pipelines without a file, but its jobs have GitLab's names.
- A !reference tag. Search the included files for
.snippets:.
YAML-level reuse: anchors, merge keys and hidden jobs#
YAML has its own copy and paste, which happens before GitLab reads a single keyword:
- An anchor, written
&name, gives a piece of YAML a name. - An alias, written
*name, pastes that piece somewhere else. - A merge key, written
<<:, pastes the keys of a named map into another map.
All three work only inside the file that defines them. Hidden jobs, whose names start with a dot, are GitLab's own addition: jobs that never run and exist only to be copied from. You'll meet all of this in older files, and in any file whose author never needed to reach across an include.
An older template in billing-batch uses <<: [*base, *java]. You add a variable to *java, and the job never sees it. The same change to *base works. Both anchors are in the same file. Why does the order matter? (There is no symptom card for this one: the answer is in what wins in a merge.)
Anchors, aliases and merge keys#
Here is the whole idea in seven lines:
.base: &base # "&base" gives this map the name base
image: node:22
tags: [linux]
test:
<<: *base # paste the keys of the map named base
script: npm testGitLab never sees &base or *base. The YAML reader has
already replaced them, so GitLab reads test as if it had been written out in
full:
test:
image: node:22
tags: [linux]
script: npm testThe dot in front of .base matters too. Without it, GitLab would try to run
base as a job of its own.
MegaCorp's oldest project still shares configuration this way:
.defaults: &defaults
image: registry.example.com/megacorp/devops/ci-tools:2.9
tags: [megacorp-shared]
only:
- branches
- tags
build:
<<: *defaults
stage: build
script:
- ./gradlew assembleThe same three pieces are at work:
&defaultsnames the map under.defaults.<<: *defaultscopies that map's keys intobuild.- The dot in front of
.defaultsstops GitLab from treating the anchor's home as a job.
By the time GitLab sees the file, build simply has an
image, tags and only of its own. That is why
anchors are the one kind of reuse that leaves no trace in the merged configuration. It
shows the result, with the anchors already replaced. Here is another of billing-batch's
jobs, next to what GitLab made of it:
notify-reports merges *defaults, which says
only: [branches, tags], and then writes only: [tags] itself. A
key written in the job always beats a merged one, so the job runs only for tags.
Anchors have one hard limit: they work only inside the file that defines them.
!reference instead.#What wins in a merge#
Merge keys follow two rules, and both differ from extends, which
chapter 14 covers:
| Situation | Merge key <<: | extends |
|---|---|---|
| the job sets a key the source also sets | the job wins | the job wins |
both sides have a map, such as variables | the job's map replaces the source's whole map: merge keys are shallow | the maps merge key by key |
| two sources set the same key | <<: [*a, *b]: the earlier one, *a, wins | extends: [.a, .b]: the later one, .b, wins |
| the source is in an included file | impossible: anchors stop at the file boundary | works |
The third row catches people who know one mechanism and assume the other behaves the
same way. When you see a list after <<:, the first entry is the one
that sticks.
The local GitLab 19.3 confirmed both rules. A job written
<<: [*a, *b] took its image and its variables from a, the
first in the list. A job that merged *a and wrote its own
variables: kept only its own variable. Every variable from the anchor was
gone.
- Scope
- One file
- Binding
- Inline
- Control
- Opt-in
- Phase
- Parse
- Merged view
- Shown
.ci_image: &ci_image registry.example.com/megacorp/devops/ci-tools:2.9
build:
image: *ci_imageThe YAML reader resolves anchors before GitLab reads the configuration. An alias becomes a copy of whatever its anchor names: a string, a list or a map. GitLab never sees the anchor, only the copy, so the Full configuration view shows the result with every alias replaced.
An alias always has its anchor in the same file. Search that file for
&name, and nowhere else.
- An anchor in an included file is invisible to the file that includes it. Central
teams use
!referenceinstead (chapter 14). - Editing the anchored block changes every alias at once, with no sign at the places that use it.
- Scope
- One file
- Binding
- Inline
- Control
- Opt-in
- Phase
- Parse
- Merged view
- Shown
build:
<<: *defaults
test:
<<: [*defaults, *java21]The merge key copies the keys of the named map into the current map, except for keys
the current map already has. The copy is shallow: a key whose value is a map, such as
variables, is copied or ignored whole, never merged. With a list of maps,
earlier maps win over later ones.
Find the anchor in the same file, then compare keys. Any key the job writes itself wins entirely. Anything else comes from the first map in the list that has it.
- Adding one variable to a job that merges
*defaultsdeletes all of the anchored variables for that job. - The order in a list is the opposite of
extends.
Hidden jobs#
Any top-level key that starts with a dot is ignored as a job. That makes hidden jobs the raw material of nearly every template library:
- They hold anchors, as in billing-batch.
- They are the parents that
extendscopies from. - They hold the snippets that
!referencepastes.
MegaCorp's base template is one:
.megacorp-base:
variables:
MC_TEAM: unknown
artifacts:
expire_in: 7 days
rules:
- !reference [.rules, never-on-schedule]
- !reference [.rules, mr]
- !reference [.rules, default-branch]
- !reference [.rules, release-tag]- Do this
- When two jobs should share settings, write a hidden job and copy it with
extends. Use a YAML anchor only inside one file. - Why
extendsworks across every included file. An anchor stops at the end of the file it is written in, so it cannot reach a job in another file.- Watch out
- If you do use a merge key, remember two rules. A key the job writes itself replaces the merged one completely, and when you merge a list of aliases, the first alias wins.
- Read anchors, aliases and merge keys in an older file, and say which key wins when they collide.
- Explain why an anchor cannot reach a job in another file, and what replaced it.
- Recognise a hidden job, and know that it is the base of every modern template.
Spot the bugThe Gradle job on the wrong JDK#
Someone modernised billing-batch's build job so that it builds on JDK 21. This is
billing-batch/.gitlab-ci.yml, modified:
.defaults: &defaults
image: registry.example.com/megacorp/devops/ci-tools:2.9
variables:
GRADLE_OPTS: "-Dorg.gradle.daemon=false"
.java21: &java21
image: registry.example.com/megacorp/devops/ci-tools/gradle:8-jdk21
build:
<<: [*defaults, *java21]
variables:
BILLING_ENV: ci
script:
- ./gradlew assembleThe job still reports JDK 11, and the Gradle daemon is starting again.
build use, and why has GRADLE_OPTS
gone?Show the answer
It uses ci-tools:2.9. In a merge-key list the earlier map wins, and
*defaults comes first. GRADLE_OPTS is gone because the job
writes its own variables:. Merge keys are shallow, so that map replaces the
anchored one outright.
Put *java21 first and repeat GRADLE_OPTS in the job. Or move
both anchors into hidden jobs and use extends: [.defaults, .java21], which
merges maps key by key and lets the later parent win. See
merge keys.
extends, !reference and default#
Five keywords let one job borrow configuration from somewhere else. Between them they
explain most of the "where did that line come from?" moments in an enterprise pipeline:
extends, !reference, default:, top-level
variables: and parallel: matrix.
One rule explains most of their surprises: maps merge key by key, but lists are
replaced whole. A map is a set of named keys, such as variables. A
list is a sequence of items, such as script
(YAML in five minutes). If you override one line of a template's
rules, script or before_script, you have replaced
all of it.
You add before_script: [echo starting] to maven-test. The job now fails at once: mc_log: command not found. The function comes from the central team's default. Your one line was meant to add, not remove. What did GitLab do with it? (the card)
The five, side by side#
In MegaCorp's template library, almost nothing is a real job. The templates are hidden jobs, whose names start with a dot, and they borrow from each other through the five keywords below. The project's own file then turns a few of them into real jobs.
| Keyword | What it shares | How it combines with the job | Works across included files |
|---|---|---|---|
extends | a whole job, from one or more parents | maps merge key by key, lists are replaced, and the later parent wins | yes |
!reference | one section of another job, such as its script or rules | pastes exactly what it points at, where you put it | yes |
default: | eleven keywords, such as image and before_script, for every job | none: a job's own keyword replaces the default outright | yes, wherever it is defined |
top-level variables: | variables for every job | a job's own variables outrank them | yes |
parallel: matrix | one job definition, run once per combination of values | each generated job gets its own copy of the variables | not applicable |
Before tracing anything, open the one view that resolves all five for you:
Build › Pipeline editor, then the Full configuration tab.
It shows the configuration with included files copied in, extends merged,
YAML anchors replaced and !reference tags replaced.
extends: a job inherits a job#
Take the maven-test job in payments-api's pipeline. The project's own file
only adds one variable to it. The job itself is defined three files away:
pipelines/java-service.yml says maven-test: extends: .maven-test.
.maven-test extends .maven-base, which extends
.megacorp-base. Here is the middle of that chain:
.maven-base:
extends: .megacorp-base
image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
variables:
JAVA_VERSION: "21"
MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
cache:
key:
files: [pom.xml]
paths: [.m2/repository]
before_script:
# a job's own before_script replaces default:before_script, so re-add the functions
- !reference [.snippets, functions]
- !reference [.snippets, maven_settings]Here is what GitLab builds from the whole chain, next to three of the four files that feed it:
Five things to notice, and each is the merge rule at work:
- Variables from three levels become one map.
MC_TEAMcomes from.megacorp-base,JAVA_VERSIONandMAVEN_OPTSfrom.maven-base, andMAVEN_CLI_OPTSfrom the project's file. - So do the artifacts.
expire_in: 7 dayscomes from.megacorp-base, andwhenandreportscome from.maven-test.artifactsis a map, so the two merged. - The image is
.maven-test's, not.maven-base's. When the same key appears at two levels, the later one wins. - The rules are
.maven-test's alone..megacorp-basehad a fourth rule, for release tags, but a list never merges. That is whymaven-testdoes not run in tag pipelines whilemaven-builddoes. - Pasted snippets show as lists inside the list. The
rulesandbefore_scriptentries that start with- -were pasted in by!reference, covered in the next section. GitLab flattens them into one list when the job runs.
Whenever you wonder whether your own key merged with a template's or replaced it, ask these questions in order:
As text
- Is the key's value a map, such as variables, cache or artifacts? Yes: The two maps merge key by key; on a clash, the job's value wins. No: the next step.
- Is it a list, such as script, before_script or rules? Yes: The job's list replaces the template's whole list. No: the next step.
- It is a single value, such as image or stage: the job's value replaces the template's
extends combines a
template with the job that extends it: maps merge key by key, lists are replaced whole.#- Scope
- One project
- Binding
- Inline
- Control
- Opt-in
- Phase
- Compile
- Merged view
- Shown
maven-test:
extends: .maven-test
integration-test:
extends: [.maven-base, .with-database]GitLab resolves the chain of parents first, then merges each level into the next.
Maps such as variables, cache and artifacts merge
key by key. Where the same key appears twice, the later definition wins: the job over
its parent, and the second parent over the first. Lists such as script,
before_script and rules are replaced whole. A job can extend a
hidden job defined in any included file. Chains can be eleven levels deep, though
GitLab's own guidance is to stay within three.
Open Full configuration in the pipeline editor, where the job appears
with everything it inherited already merged in. To see which file contributed a line,
follow the extends names upwards. Search the included files for each
parent's name, such as .maven-test:. Expect every hop to be in a
different file.
- Setting
rules,scriptorbefore_scripton your job throws away the template's version entirely (symptom card). - Setting one variable is safe, because maps merge. People who have been bitten by
lists often copy a whole
variables:block when a single key would do. - A job's variables inherited through
extendsoutrank the project's top-level variables (see top-level variables).
Your rules: replaced the template's entire list. Lists never merge,
so every rule the template had is gone, including the ones that skipped scheduled
pipelines or added tag pipelines.
In Build › Pipeline editor › Full configuration, find the
job. Its rules: shows only your lines.
Rebuild the list: paste the template's rules back with !reference and
put your rule where it belongs. The first matching rule wins, so order matters.
maven-test:
rules:
- if: $SKIP_TESTS == "true"
when: never
- !reference [.maven-test, rules]!reference: paste one section#
extends takes a whole job. !reference takes one piece of
one, and pastes it exactly where you write it. MegaCorp keeps its script fragments in a
hidden job whose keys aren't CI/CD keywords at all. GitLab's own documentation uses the
same pattern.
.snippets:
functions:
- source /opt/megacorp/lib/ci-lib.sh
aws_login:
- mc_aws_login "$AWS_ROLE_ARN"
acr_push:
- mc_acr_push "$IMAGE:$CI_COMMIT_SHA"
maven_settings:
- mkdir -p .m2
- mc_maven_settings > .m2/settings.xmlThe rules library works the same way. Named lists of rules are assembled job by job,
which is how .maven-test got its three rules:
.rules:
mr:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
default-branch:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
release-tag:
- if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
never-on-schedule:
- if: $CI_PIPELINE_SOURCE == "schedule"
when: neverYou will meet the older way of doing this in long-lived projects: a YAML anchor. The two look alike, but they don't reach equally far:
.login: &login
- mc_aws_login "$AWS_ROLE_ARN"
image-build:
before_script: *loginThis works only if .login is in the same file as
image-build.
image-build:
before_script:
- !reference [.snippets, aws_login]This works wherever .snippets is defined, including in an included
file.
A YAML anchor exists only inside the file that defines it. An included file's anchors
are invisible to the file that includes it. The workaround that central teams use is
!reference, which GitLab resolves after it has assembled every included
file.
- Scope
- One project
- Binding
- Inline
- Control
- Opt-in
- Phase
- Compile
- Merged view
- Shown
before_script:
- !reference [.snippets, functions]
rules:
- !reference [.rules, mr]
variables:
NEXUS_URL: !reference [.vars, variables, NEXUS_URL]GitLab replaces the tag with the configuration it names: a job, then a key path inside
it. A list pasted into a list is flattened. The path can name any key in a hidden job,
not only a CI/CD keyword. The tags can be nested up to ten levels deep in
script, before_script and after_script. They work
across included files, but inputs can't be used inside the path.
The tag names its source: the first item is the job to search for, such as
.snippets:. In the Full configuration view the pasted content is already in
place. Nested tags sometimes show extra leading hyphens there, which GitLab documents as
expected and harmless.
- The paste happens when GitLab assembles the configuration, but the pasted lines run inside your job. Functions a snippet defines exist only in the job that pasted it.
- A snippet pasted into
before_scriptdisappears the moment a job redefinesbefore_scriptwithout pasting it again.
default: the layer you never see#
default: sets a keyword for every job that doesn't set it itself. It
lives in a template, so no project file mentions it, yet it is in every job.
default:
image: registry.example.com/megacorp/devops/ci-tools:3.2
tags: [megacorp-shared]
interruptible: true
retry:
max: 2
when: [runner_system_failure, stuck_or_timeout_failure]
before_script:
- !reference [.snippets, functions]
- mc_log "job $CI_JOB_NAME on $CI_COMMIT_REF_NAME"Eleven keywords can have defaults: after_script, artifacts,
before_script, cache, hooks,
id_tokens, image, interruptible,
retry, services and tags. Unlike
extends, a default never merges with anything. If a job has the keyword,
the default for that keyword simply doesn't apply.
MegaCorp's default before_script loads the shell function library that
every other template relies on. The moment a job defines its own
before_script, even a single line, the library is no longer loaded. That
is why .maven-base pastes the functions back in, and says so in a
comment.
MegaCorp uses !reference inside default: as well as inside
jobs. GitLab's documentation shows it only in jobs, but the local GitLab 19.3
accepts a default: before_script built from !reference without
complaint.
Opting out with inherit#
A job can refuse some or all of the defaults with inherit.
inherit: default: takes true, false, or a list of
the keywords to keep. inherit: variables: does the same for top-level
variables.
publish-docs:
stage: deploy
image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
inherit:
default: [tags, retry]
variables: false
script:
- mkdocs build --strict
rules:
- !reference [.rules, default-branch]publish-docs keeps the organisation's runner tags and retry policy. It
drops the default image, which it overrides anyway, and the default
before_script, so none of the shell functions are loaded. It also receives
none of the top-level variables.
Legacyimage, services, cache, before_script and after_script at the top level
Older files set these five keywords at the top level, outside any job, as a way of
applying them to every job. GitLab has deprecated that form. The replacement is the
same keywords under default:. When you meet the old form, read it exactly
as you would a default: block.
- Scope
- One project
- Binding
- Inline
- Control
- Opt-in
- Phase
- Compile
- Merged view
- Keyword only
default:
image: registry.example.com/megacorp/devops/ci-tools:3.2
before_script:
- !reference [.snippets, functions]
publish-docs:
inherit:
default: [tags, retry]For each of the eleven keywords it can hold, default: supplies a value to
every job that doesn't have one. A job's own value replaces the default outright; the
two are never merged. inherit: default: turns the defaults off, entirely
or keyword by keyword. Defaults are not passed to downstream pipelines, which start
again from their own configuration.
Search the included files for default:. Then compare it with the job
in the Full configuration view: a keyword the job doesn't show was never overridden,
so the default applies. Check for inherit: on the job before concluding
anything.
- A single line of
before_scripton a job removes the whole defaultbefore_script, including anything the organisation put there. - A child or multi-project pipeline doesn't get the parent's defaults. The same template can behave differently one pipeline downstream.
Top-level variables#
A variables: block at the top of any file applies to every job. Blocks
from included files and from the project's own file merge into one map:
variables:
GIT_DEPTH: "20"
AWS_REGION: eu-west-2
MC_TEMPLATES_VERSION: "4.2.0"variables:
JAVA_VERSION: "21"
MC_TEAM: paymentsHere, payments-api sets MC_TEAM: payments, but every one of its jobs
still sees unknown. The reason is in the base template:
.megacorp-base:
variables:
MC_TEAM: unknown
artifacts:
expire_in: 7 days
rules:
- !reference [.rules, never-on-schedule]
- !reference [.rules, mr]
- !reference [.rules, default-branch]
- !reference [.rules, release-tag].megacorp-base sets MC_TEAM as a job variable, and
every template extends it. Job variables outrank top-level variables, so the project's
value never reaches a job. The ranking goes further than YAML: a variable set in the
project's, group's or instance's CI/CD settings outranks everything in YAML.
Chapter 26 gives the full ladder.
- Scope
- One project
- Binding
- Inline
- Control
- Opt-in
- Phase
- Compile
- Merged view
- Keyword only
variables:
GIT_DEPTH: "20"
AWS_REGION: eu-west-2
publish-docs:
inherit:
variables: falseTop-level variables become default variables for every job. Blocks from every included
file and the project's own file merge into one map, with the project's file winning on
a clash. A job's own variables, including those it inherits through
extends, outrank them. inherit: variables: can turn them off
for a job, entirely or by name.
Look for the variable in three places, in this order:
- the job's own
variables:in the Full configuration view - the top-level
variables:blocks - the project's, group's and instance's CI/CD settings, which outrank both of the above
- A project cannot override a variable that a template sets at job level by setting it at the top of its own file.
- A top-level variable is not the last word. A CI/CD variable of the same name set in the UI replaces it without a trace in any file.
parallel: matrix: one definition, several jobs#
.maven-test, quoted in the comparison earlier, asks for a matrix of two
JDK versions. The pipeline shows one job per value, named after the job and the value:
As text
- build: maven-build
- test: maven-test: [17], maven-test: [21]
- package: image-build
Needs: image-build waits only for maven-build.
Each generated job gets its own JDK variable, which the image name uses.
A matrix can produce at most 200 jobs. When another job needs only some of them,
needs: parallel: matrix selects them by value.
- Scope
- One project
- Binding
- Inline
- Control
- Opt-in
- Phase
- Create
- Merged view
- Keyword only
.maven-test:
parallel:
matrix:
- JDK: ["17", "21"]GitLab creates one job for every combination of the matrix values. It names each
<job>: [<value>, …], for example maven-test: [17],
and gives each its own copy of the variables. The matrix is expanded when the pipeline
is created, not when the configuration is assembled.
A job name ending in square brackets is a matrix job. Strip the brackets and search
for the name; here, maven-test leads to .maven-test and its
parallel: block. The Full configuration view shows the matrix definition,
not the generated jobs; the pipeline graph shows those.
needs: [maven-test]depends on every job in the matrix. To depend on one, useneeds: parallel: matrix.- Matrix values are plain variables. A value that collides with a variable set elsewhere follows the usual precedence rules.
Which of these jobs run where#
Put the rules from this chapter together and you can predict, without running anything, which of payments-api's jobs exist in each kind of pipeline:
| job | feature branch push | merge request | main | tag v1.4.0 | nightly on main |
|---|---|---|---|---|---|
| maven-build | not in the pipeline | runs | runs | runs | not in the pipeline |
| maven-test: [17] and [21] | not in the pipeline | runs | runs | not in the pipeline | not in the pipeline |
| image-build | not in the pipeline | runs | runs | runs | not in the pipeline |
| publish-docs | not in the pipeline | not in the pipeline | runs | not in the pipeline | runs |
Two cells are worth a second look:
- Tag pipelines have no tests.
maven-test's own rules replaced the base template's list, which was the only one with a tag rule. publish-docsruns every night. Its only rule is "on the default branch", and nothing tells it to skip scheduled pipelines.
How GitLab resolves an extends chaindeep dive
GitLab assembles the configuration first: every included file is copied in, and the
project's file is merged on top. Only then does it resolve extends. For
each job it walks up the chain of parents, .megacorp-base then
.maven-base then .maven-test, and merges downwards, each level
into the next.
At every level the same reverse deep merge applies. Maps are merged key by key, the
later level wins a clash, and a list replaces the list above it whole. With several
parents, extends: [.a, .b], .b is applied after
.a and wins any clash between them.
!reference tags are resolved in the same assembly pass, which is why they
can point into included files when YAML anchors cannot. By the time the pipeline is
created, none of these keywords exists any more. There are only complete jobs, which is
exactly what the Full configuration view shows.
- Do this
- When you extend a template job, change only what you must, and then open the Full configuration to see the result.
- Why
- Settings that are lists, such as
rules,scriptandbefore_script, replace the template's whole list. Settings that are maps, such asvariables, merge key by key. So one line of yours can throw away everything the template had. - Watch out
- To keep a template's rules and add your own, paste its rules into your list with
!reference. The card your rules replaced the template's shows what happens when you forget.
- Trace a job through its extends chain to the file that defines each setting.
- Predict what a job's own key does to the inherited one: merge for maps, replace for lists.
- Keep a template's list and add your own line to it with a reference.
Spot the bugThe team name that never changes#
A dashboard groups jobs by the MC_TEAM variable. It files every
payments-api job under unknown, even though payments-api's own file sets
it:
variables:
JAVA_VERSION: "21"
MC_TEAM: paymentspayments, and what is the least invasive
fix?Show the answer
.megacorp-base sets MC_TEAM: unknown as a job variable, and
every job reaches it through extends. Job variables outrank top-level
variables, so the project's value is overridden in every job.
The least invasive fix is a project CI/CD variable MC_TEAM=payments in
the project's settings, because settings variables outrank anything in YAML. The better
long-term fix is for the template owners to move MC_TEAM to top-level
variables, so that projects can override it. See
top-level variables.
Spot the bugOne line that broke Maven#
The payments team needed a database for maven-test, so they added one
line to their file. This is payments-api/.gitlab-ci.yml, modified:
maven-test:
before_script:
- ./scripts/start-db.sh
variables:
MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"Now both maven-test jobs fail at once. Maven complains that
.m2/settings.xml does not exist.
settings.xml go, and how should
they have added their script?Show the answer
before_script is a list, and lists are replaced whole. The job's one-line
list replaced .maven-base's list. That list was the one that loaded the
function library and wrote .m2/settings.xml.
Keep the template's lines by pasting them back first:
maven-test:
before_script:
- !reference [.maven-base, before_script]
- ./scripts/start-db.shSee the symptom card for the same trap with
rules, and extends.
include: pulling in files#
include copies configuration from other files into yours, before anything
else happens. There are five kinds, and they differ only in where the file comes
from:
local: your own repositoryproject: another projectremote: a URLtemplate: GitLab itselfcomponent: a versioned component
Every included file is merged into one configuration, and your own file wins any clash. The most important thing to read on an include is not the file name but the ref: the branch, tag or commit it reads from. The ref decides whether a change somewhere else can reach you without a commit of yours.
Nobody committed to payments-api since Friday, but this morning's pipeline has a new stage and a job you have never seen. Your file has one include line, and it hasn't changed either. Something you point at moved. What, and how do you stop it moving? (the card)
Five kinds#
| Kind | Fetches from | Binding | You'll see it as |
|---|---|---|---|
| local | the same repository, at the same commit | inline | include: ci/build.yml, or a wildcard such as ci/*.yml |
| project | a file in another project, at a ref | pinned or floating, depending on ref | project: with ref: and file: |
| remote | any public HTTP(S) URL | whatever the URL serves today | remote: https://… |
| template | GitLab's own library of templates | changes when your GitLab is upgraded | template: Security/SAST.gitlab-ci.yml |
| component | a component project, at a version | pinned or floating, depending on the version | component: $CI_SERVER_FQDN/…@1.4.0 |
MegaCorp uses all five. payments-api includes a project file and a component:
include:
- project: megacorp/devops/ci-templates
ref: v4.2.0
file: pipelines/java-service.yml
- component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
inputs:
stage: test
project_key: payments-apiThe security template includes GitLab's own templates:
include:
- template: Security/SAST.gitlab-ci.yml
- template: Security/Secret-Detection.gitlab-ci.yml
- template: Security/Dependency-Scanning.gitlab-ci.yml
- template: Security/Container-Scanning.gitlab-ci.ymlThe monorepo includes a file from its own repository, and billing-batch still pulls one in by URL:
include:
- local: ci/common.ymlinclude:
- remote: https://gitlab.example.com/megacorp/devops/ci-templates/-/raw/main/legacy/notify.ymlHow included files merge#
An included file can include more files, so payments-api's single include becomes ten
files. GitLab then merges everything into one configuration, using the same rule as
extends (chapter 14):
- maps merge at any depth
- lists are replaced whole
- where two files set the same key, the including file beats the included one
The same file may be included twice, and the last inclusion wins.
That is how MegaCorp adjusts GitLab's container-scanning job without copying it. The security template includes GitLab's template, then defines a job with the same name:
Read the merged job from the bottom up. stage, rules and
needs are MegaCorp's. Everything else is GitLab's: the image, the script,
the artifacts, and allow_failure: true. MegaCorp's rules
replaced GitLab's list outright, because lists never merge. From here on, GitLab's own
rules for the job, whatever they were in this version, no longer apply.
Reading a ref#
Every include except local reads its file at some ref. That one word
decides whether a change somewhere else can reach you:
As text
- Is the ref a commit SHA? Yes: Never: a commit can't change. No: the next step.
- Is it a tag, such as v4.2.0? Yes: Only if someone changes the version, or moves the tag. No: the next step.
- Is it a branch, such as main? Yes: Yes, on the very next pipeline. No: the next step.
- There is no ref at all: yes, it follows the other project's default branch
| The include says | You get | Upstream changes reach you |
|---|---|---|
ref: v4.2.0, a tag | that tag's content | only when someone edits the version, or moves the tag |
ref: with a commit SHA | that exact commit | never |
ref: main, a branch | whatever the branch holds when the pipeline is created | at once, on the next pipeline |
no ref at all | the head of the other project's default branch | at once, on the next pipeline |
A missing ref is the easiest time bomb to plant, because it looks tidy
and floats silently. MegaCorp pins everything to v4.2.0, both where
payments-api includes the golden pipeline and inside the golden pipeline itself. A
project that pinned only the first file would still get whatever the nested includes
point at.
What an include can depend on#
An include can carry rules, with if, exists or
changes, and GitLab skips it when they don't match. The variables it can
test are limited, because GitLab evaluates include rules before any job exists:
| Usable in include rules | Not usable |
|---|---|
| project, group and instance CI/CD variables | variables defined in a job |
predefined CI_PROJECT_* variables | top-level variables: in any file |
| variables of triggered, scheduled and manually run pipelines | |
CI_PIPELINE_SOURCE, CI_PIPELINE_TRIGGERED, CI_COMMIT_REF_NAME |
One more trap: in a file included from another project, rules: exists
checks for files in that project, not yours. Use
rules: exists: project to point it back. Included files can also take
parameters, called inputs, which chapter 16
covers.
A nested include: local is read from the project that holds the file
doing the including, not from yours. On the local GitLab 19.3, a file in
ci-templates included local: templates/rules.yml while being included into
payments-api. GitLab found ci-templates' copy, although payments-api has no such
file.
- Scope
- One project
- Binding
- Inline
- Control
- Opt-in
- Phase
- Compile
- Merged view
- Shown
include:
- local: ci/common.yml
- local: ci/jobs/*.ymlGitLab copies the file from the same repository, at the same commit as the pipeline,
into the configuration. A wildcard pulls in every match. A single * doesn't
reach into subfolders; ** does.
Open the path in the same repository, at the commit the pipeline ran for. With a wildcard, list the matching files at that commit, because a file added later changes what the include means.
- A new file that matches a wildcard joins the pipeline without anyone editing the include.
- Inside a file that was itself included from another project, "local" means that project, not yours (see the note above).
- Scope
- Cross-project
- Binding
- Pinned or floating
- Control
- Opt-in
- Phase
- Compile
- Merged view
- Shown
include:
- project: megacorp/devops/ci-templates
ref: v4.2.0
file:
- templates/base.yml
- templates/rules.ymlGitLab fetches each listed file from the other project, at ref, and
merges it into the configuration. Without a ref, it uses the head of that
project's default branch.
Open the other project at the ref shown, not at its default branch, and read the file there. Check whether that file includes more files, and at which refs.
- A branch ref, or no ref at all, floats. The pipeline can change without a commit in your project.
- Pinning the top-level include pins nothing below it, unless the included file pins its own includes too.
- Scope
- Cross-project
- Binding
- Pinned or floating
- Control
- Opt-in
- Phase
- Compile
- Merged view
- Shown
include:
- remote: https://gitlab.example.com/megacorp/devops/ci-templates/-/raw/main/legacy/notify.ymlGitLab downloads the URL when it assembles the configuration, and merges the result. The URL must be publicly reachable. GitLab's guidance is to avoid rate-limited hosts, such as GitLab Pages.
Open the URL. What it serves now is what the next pipeline gets, which is not necessarily what an earlier pipeline got.
- Nothing pins a remote include unless the URL itself names a fixed version. Here it
names the
mainbranch. - The file's owner can change every consumer's pipeline, without any of them knowing the file exists.
- Scope
- Organisation-wide
- Binding
- Always live
- Control
- Opt-in
- Phase
- Compile
- Merged view
- Shown
include:
- template: Security/SAST.gitlab-ci.yml
- template: Security/Secret-Detection.gitlab-ci.ymlGitLab inserts one of the templates it ships with. They cover security scanners, Auto DevOps and common build jobs. The content belongs to the GitLab version your instance runs.
The Full configuration view shows the template's jobs copied in. GitLab's templates
are published in GitLab's own source repository, under
lib/gitlab/ci/templates/. Read the version that matches your instance.
- Upgrading GitLab changes what the template contains, including job names and rules, without any commit in any of your projects.
- Overriding a template job by name merges with it. Overriding its
rulesreplaces them entirely.
- Do this
- Pin every include to a tag or a commit SHA with
ref:. Move to a new version on purpose, in a merge request of your own. - Why
- Your pipeline then changes only when your team decides to change it, and a reviewer sees the change.
- Watch out
- An include of a branch, such as
main, changes your pipeline whenever the central team pushes to that branch. The card the pipeline changed overnight shows how that looks.
- Read every kind of include and say which file it brings in, from where, and at which version.
- Pin an include to a tag or commit, and move it on purpose in a merge request.
- Find the included file behind any job, using the editor's file tree.
Spot the bugThe deploy jobs that never appear#
A team wanted deploy jobs only in projects that opt in, so their file says:
include:
- local: ci/deploy.yml
rules:
- if: $DEPLOY_ENABLED == "true"
build:
variables:
DEPLOY_ENABLED: "true"
script: make buildThe deploy jobs from ci/deploy.yml never appear, on any branch.
Show the answer
Include rules are evaluated while GitLab assembles the configuration, before any job
exists, so they can't see variables defined in a job. $DEPLOY_ENABLED is
empty there, and the rule never matches.
Set DEPLOY_ENABLED as a CI/CD variable in the project's settings, which
include rules can see. Another option is a rule on a predefined variable such as
$CI_COMMIT_REF_NAME. See include: local and
what an include can depend on.
Components, inputs and toggles#
Shared configuration usually needs settings of its own, called parameters, and there are two ways to pass them:
- Inputs are declared by the shared file, with a type. GitLab writes
each value into the file, in place of
$[[ inputs.name ]], when the pipeline is created. After that they never change. - Variables are the older way. The shared file reads
$SOMETHING, and whoever uses it sets the value. Variable precedence decides who really wins.
A component is shared configuration packaged for this: a versioned
file with declared inputs, included with component:. A
toggle is a variable that a rule tests, turning a job on or off.
A colleague sets SONAR_ENABLED: "false" to skip the code-quality scan on a hotfix. The scan still runs. Another colleague sets SAST_DISABLED: 1 and the scan vanishes from every pipeline, including main. Two toggles, two opposite surprises. What decides whether a toggle works? (the card)
Components#
A component lives in a component project, as templates/<name>.yml or
templates/<name>/template.yml. The include names the project, the
component and a version. Here is MegaCorp's Sonar component, which runs a code-quality
scan:
spec:
inputs:
stage:
default: test
project_key:
description: SonarQube project key
quality_gate:
type: boolean
default: true
---sonar-scan:
stage: $[[ inputs.stage ]]
image: registry.example.com/megacorp/devops/ci-tools/sonar-scanner:6
variables:
SONAR_PROJECT_KEY: $[[ inputs.project_key ]]
SONAR_QUALITYGATE_WAIT: "$[[ inputs.quality_gate ]]"
script:
- sonar-scanner -Dsonar.projectKey="$SONAR_PROJECT_KEY" -Dsonar.host.url="$SONAR_HOST_URL"
rules:
- if: $SKIP_SONAR == "true"
when: never
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
allow_failure: truepayments-api includes it, with two inputs:
The inputs are gone from the result. $[[ inputs.stage ]] became
test, and $[[ inputs.project_key ]] became
payments-api. quality_gate wasn't passed, so it took its
default, true.
The version after @ decides what you get:
| Version | Example | Resolves to |
|---|---|---|
| a commit SHA | @e3262fdd… | exactly that commit |
| a tag | @2.1.0 | that tag; a tag beats a branch with the same name |
| a branch | @main | the branch's head when the pipeline is created |
| the latest release | @~latest | the newest version published to the CI/CD Catalog |
| a partial version | @2 or @2.1 | the newest matching version published to the CI/CD Catalog |
Components work from any project, but the last two forms work only for components
published to the CI/CD Catalog, GitLab's searchable list of shared
components. Publishing needs a release, created with the release: keyword,
on a tag that is a version number such as 2.1.0.
Inputs#
Inputs are declared in a header above a --- line. Each has a type:
string (the default), number, boolean or
array. It can also have a default, a description, a fixed list of
options, or a regex the value must match. This is what happens
to them:
As text
- The include passes the inputs it wants, such as project_key: payments-api
- GitLab reads the shared file's spec: header
- Does every required input have a value? No: No pipeline. GitLab names the missing input. Yes: the next step.
- GitLab writes each value in place of $[[ inputs.name ]]
- The result is ordinary YAML, merged like any other include
A value can be transformed on the way in, by adding one or more of four functions
after the name: expand_vars, truncate(offset,length),
posix_escape and split(separator). At most three can be used
at once.
Components aren't the only files that can declare inputs. Any included file can, and
the include passes values with inputs:.
| Inputs | Variables | |
|---|---|---|
| Declared by | the shared file, in its spec: header | nobody: any file or setting can define them |
| Resolved | once, when the pipeline is created | when rules are evaluated, and again when the job runs |
| Can change during the pipeline | no: fixed for the whole run | yes: dotenv reports and scripts can set new values |
| Typed and checked | yes: type, options, regex | no |
| Who wins a clash | the value passed in, else the default | the precedence ladder in chapter 26 |
Two things to know when you trace an input:
- The merged view shows the result, not the input. To see where
payments-apicame from, open the component's own file at the version the include names, and read the include'sinputs:. - A missing required input stops the whole pipeline, before any job
runs. Leave out
project_key, and the local GitLab 19.3 says:
`gitlab.example.com/megacorp/devops/components/sonar-scan@2.1.0`: `project_key` input: required value has not been providedToggles#
A toggle is a variable that a rule checks. MegaCorp's Sonar component has one
(SKIP_SONAR), and so does its container-scanning override:
variables:
CS_IMAGE: $IMAGE_REF
container_scanning:
stage: scan
needs: [image-build]
rules:
- if: $CONTAINER_SCANNING_DISABLED == "true"
when: never
- !reference [.rules, never-on-schedule]
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHSetting CONTAINER_SCANNING_DISABLED to "true" in a project's
variables makes the first rule match, with when: never, so the job never
exists. GitLab's own scanner templates use the same pattern, with variables named after
the scanner.
Rules compare text. A toggle tested with == "true" ignores
1, yes and TRUE. And a toggle only removes a job
that the file itself defines. A scan that a security policy adds doesn't read your
toggles (chapter 36).
GitLab's own templates at GitLab 19.3 name their toggles after the scanner,
and accept either "true" or "1". payments-api's merged
configuration has three: SAST_DISABLED,
SECRET_DETECTION_DISABLED and DEPENDENCY_SCANNING_DISABLED.
MegaCorp's container-scanning override replaced GitLab's rules with its own, so its
CONTAINER_SCANNING_DISABLED accepts only "true".
Pipeline inputs Since 17.11#
A project's own .gitlab-ci.yml can declare inputs for the whole
pipeline. They are filled in when someone runs the pipeline by hand, or passed by the
API, a schedule, a Git push option, or a trigger from another pipeline.
MegaCorp's deployer asks which environment to deploy to:
spec:
inputs:
environment:
type: string
default: staging
options: [staging, production]
---A pipeline can take at most 20 inputs. Give each one a default, because pipelines that start automatically have nobody to fill in the form.
- Scope
- Cross-project
- Binding
- Pinned or floating
- Control
- Opt-in
- Phase
- Compile
- Merged view
- Shown
include:
- component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
inputs:
project_key: payments-apiGitLab fetches templates/sonar-scan.yml, or
templates/sonar-scan/template.yml, from the component project at the version
given. It writes in the inputs and merges the result like any other include.
Everything before the @ is the project path plus the component name. Open
that project at the version given, and read the file under templates/. For
~latest or a partial version, the CI/CD Catalog shows which release it
resolved to.
- A component's job merges with any job of the same name in your pipeline.
@~latestfails with "content not found" if the component has never been published to the catalog.- Global keywords in a component, such as
default:, affect every job in your pipeline.
- Scope
- Cross-project
- Binding
- Inline
- Control
- Opt-in
- Phase
- Compile
- Merged view
- Shown
spec:
inputs:
stage:
default: test
quality_gate:
type: boolean
default: true
---
sonar-scan:
stage: $[[ inputs.stage ]]The header declares what the file accepts. When the pipeline is created, GitLab checks
the values passed in against the declarations, and writes them in place of every
$[[ inputs.… ]]. The result is ordinary configuration, fixed for the rest of
the pipeline.
Read the header at the top of the included file for the defaults. Then read the
inputs: under the include that pulled it in, for the values actually
passed.
- A file with inputs needs the
---line between its header and its jobs. - Inputs can't be used inside a
!referencepath. - An input that holds
$CI_COMMIT_SHAinserts that text. The variable is expanded later, when the job runs.
- Scope
- One project
- Binding
- Inline
- Control
- Opt-in
- Phase
- Run
- Merged view
- Keyword only
# in the template
image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
# in the project
variables:
JAVA_VERSION: "21"The template reads an ordinary CI/CD variable, and the consumer sets it. Nothing declares the variable or checks its value. GitLab resolves it when it evaluates rules and when the job runs, from whichever source ranks highest in variable precedence.
Find where the template reads the variable. Then find every place that sets it: the
job, its extends parents, the top-level blocks, and the CI/CD settings of the
project, group and instance.
- A template that sets the same variable at job level overrides the project's top-level value.
- A settings variable with the same name overrides both, and appears in no file.
- Scope
- One project
- Binding
- Inline
- Control
- Opt-in
- Phase
- Create
- Merged view
- Keyword only
rules:
- if: $SKIP_SONAR == "true"
when: never
- if: $CI_PIPELINE_SOURCE == "merge_request_event"A rule tests the variable when the pipeline is created, and the first matching rule decides whether the job exists. Setting the variable in the project, a schedule or a manual run switches the job off or on without editing any file.
Search the templates' rules for the variable's name. Then check every
place the variable can be set, including CI/CD settings and schedules, which no file
records.
- Rules compare text, so
"true"and1are different values. - A toggle can't remove a job that a security policy adds.
- Scope
- One project
- Binding
- Inline
- Control
- Opt-in
- Phase
- Create
- Merged view
- Keyword only
spec:
inputs:
environment:
default: staging
options: [staging, production]
---
variables:
ENVIRONMENT: $[[ inputs.environment ]]Inputs declared in the project's own configuration become the pipeline's parameters.
They are offered on the New pipeline form, and accepted from the API, schedules, push
options and trigger. As with any input, their values are fixed when the
pipeline is created.
Open the top of the project's .gitlab-ci.yml for the declarations. The
values a particular pipeline used come from whoever started it: a person, a schedule, an
API call or an upstream pipeline.
- An input without a default breaks every pipeline that starts automatically.
- Pipeline variables sent by a trigger outrank YAML variables. In the deployer, an
upstream job's
ENVIRONMENTvariable beats the value built from the input.
- Do this
- Include a component at an exact version, such as
@2.1.0, and pass every input that itsspecrequires. - Why
- An exact version keeps your pipeline the same until you change it. A required input that is missing stops GitLab creating the pipeline at all.
- Watch out
- Never use
~latestor a branch name as the version. And remember that inputs are filled in when GitLab reads the file, so an input cannot hold a value that only exists once a job is running.
- Include a component at a version, pass its inputs, and read what it expands to.
- Tell an input from a variable, and know which one can hold a run-time value.
- Predict whether a toggle will remove a job, and explain why a policy's job ignores it.
Spot the bugThe upgrade to ~latest#
To stop bumping versions by hand, the payments team changed their include to:
include:
- component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@~latest
inputs:
stage: test
project_key: payments-apiNow no pipeline can be created. The error says the component's content was not
found, yet @2.1.0 still works.
~latest need that 2.1.0 doesn't?Show the answer
~latest and partial versions such as @2 resolve only against
versions published to the CI/CD Catalog. MegaCorp's components project has a
2.1.0 tag, but has never been published, so there is no "latest" to find. A
tag works without publishing.
Either keep an explicit tag, or ask the component owners to publish releases to the catalog. See include: component.
Pipelines as building blocks#
A trigger job is a job whose only work is to start another pipeline. The pipeline that starts it is called upstream, and the one it starts downstream. There are four ways it happens:
- A child pipeline runs in the same project, on the same commit.
- A multi-project pipeline runs in another project, with your permissions.
- A dynamic child runs YAML that a job wrote moments earlier.
- A script can do the same thing through the trigger API, with a token.
Two things decide what you see. The first is the trigger job's
strategy: without one, it goes green as soon as the other pipeline
exists, whatever happens next. The second is what crosses the boundary. Variables and
inputs do; default: does not.
The monorepo's trigger job shows green. The child pipeline it started, the docs build, failed ten minutes later, and nobody noticed until the site went stale. In the merge request, the same trigger job fails outright with downstream pipeline can not be created. One trigger job, two different lies. What does a trigger job actually report? (the card)
Four kinds#
| Kind | Written as | Runs in | Its configuration comes from |
|---|---|---|---|
| Parent–child | trigger: include: ci/docs.yml | the same project, ref and commit | a file in the same repository |
| Dynamic child | trigger: include: - artifact: … job: … | the same project, ref and commit | YAML a job wrote during this pipeline |
| Multi-project | trigger: project: … | another project, on the branch named | that project's own configuration |
| Trigger API | curl …/trigger/pipeline in a script | whichever project the call names | that project's own configuration |
Parent and child#
MegaCorp's monorepo builds its documentation in a child pipeline, and only when the docs change:
docs:
stage: build
trigger:
include: ci/docs.yml
rules:
- changes: [docs/**/*]The child runs under the same project, ref and commit as its parent. It inherits the
parent's top-level variables, unless the trigger job says
inherit: variables: false. It does not inherit the parent's
default: block, because defaults never cross into a downstream pipeline.
Children can have children, but only two levels deep.
A child pipeline's jobs follow their own rules. In a merge request pipeline, the child
still sees CI_PIPELINE_SOURCE as parent_pipeline, and jobs with
no rules at all don't join it. On the local GitLab 19.3, mono's
docs trigger failed in every merge request pipeline for exactly that reason
(chapter 22).
Dynamic child pipelines#
The monorepo's services are built by a pipeline that doesn't exist until the pipeline runs. A job writes it, and saves the file as an artifact: a file a job keeps for later jobs to use.
generate-pipeline:
stage: generate
script:
- ci/generate.sh > generated-pipeline.yml
artifacts:
paths: [generated-pipeline.yml]#!/usr/bin/env bash
# ci/generate.sh in megacorp/data/mono
# Writes one child-pipeline job per service that changed. Its output is the YAML
# that the service-pipelines trigger job runs. Used by the chapter "Pipelines as building
# blocks".
set -euo pipefail
base="${CI_MERGE_REQUEST_DIFF_BASE_SHA:-HEAD~1}"
changed=$(git diff --name-only "$base" HEAD -- services/ | cut -d/ -f2 | sort -u)
echo "stages: [build]"
for svc in $changed; do
cat <<EOF
build-${svc}:
stage: build
image: registry.example.com/megacorp/devops/ci-tools:3.2
script:
- make -C services/${svc} build
EOF
done
# a child pipeline needs at least one job, even when nothing changed
if [ -z "$changed" ]; then
printf 'no-changes:\n stage: build\n script: [echo "no service changed"]\n'
fiThen a trigger job runs whatever the generator wrote:
service-pipelines:
stage: build
trigger:
include:
- artifact: generated-pipeline.yml
job: generate-pipeline
strategy: dependThere is no file to read in advance. The configuration that actually ran is the
generated-pipeline.yml artifact of that one job, in that one pipeline. The
generator script is the only thing you can read beforehand.
Multi-project pipelines#
Every MegaCorp service deploys by triggering the platform team's project:
.deploy:
stage: deploy
trigger:
project: megacorp/platform/deployer
branch: main
strategy: depend
variables:
APP: $CI_PROJECT_NAME
IMAGE_REF: $IMAGE_REFAs text
- build: maven-build
- package: image-build
- deploy: deploy-staging
Needs: image-build waits only for maven-build; deploy-staging waits only for image-build.
Downstream: deploy-staging triggers megacorp/platform/deployer (multi-project pipeline).
The downstream pipeline runs the deployer's own configuration, at the head of its
main branch. The person who triggered the upstream pipeline must be allowed
to start pipelines in the deployer project, or the trigger fails.
By default, the downstream pipeline receives the trigger job's variables, including the top-level variables it inherits. They arrive as trigger variables, and take precedence over the downstream project's own variables of the same name.
Some variables stay behind, unless the trigger job sets
forward: pipeline_variables: true:
- values typed into a manual run
- variables set on a schedule
- dotenv variables from the jobs the trigger job needs
Nothing forwarded is passed on again by the next trigger down, unless that trigger job
also sets forward.
That list is about passing values on automatically. Naming a dotenv variable in the
trigger job's own variables is different, and it works. On the local
GitLab 19.3, a trigger job with IMAGE_REF: $IMAGE_REF passed the value
from an earlier job's dotenv report to the downstream pipeline. It did so whether the
trigger job listed that job in needs, or only came in a later stage.
MegaCorp's .deploy relies on this.
What the trigger job's status means#
A green trigger job does not always mean the downstream pipeline passed. It depends on one key:
As text
- Does the trigger job set a strategy? No: It goes green once the downstream pipeline exists, whatever happens next. Yes: the next step.
- Is the strategy mirror? Yes: It copies the downstream pipeline's status exactly. No: the next step.
- The strategy is depend: green when the downstream succeeds, and "running" while it waits for a manual job
| The trigger job has | It turns green when | So a green trigger job means |
|---|---|---|
no strategy | the downstream pipeline has been created | only that the other pipeline started |
strategy: mirror | the downstream pipeline succeeds; it copies its status throughout | the downstream pipeline succeeded |
strategy: depend | the downstream pipeline finishes | an older setting; GitLab's documentation now recommends mirror, because this status doesn't always match |
MegaCorp still uses depend everywhere, as many real template libraries
do. The first row is the trap: a pipeline can be entirely green while the deployment it
started failed.
The trigger API#
Before trigger jobs existed, pipelines started each other with a token and a script, and some still do:
notify-reports:
<<: *defaults
stage: deploy
only: [tags]
script:
- >
curl --fail --request POST
--form "token=$REPORTS_TRIGGER_TOKEN"
--form "ref=main"
--form "variables[BILLING_VERSION]=$CI_COMMIT_TAG"
"https://gitlab.example.com/api/v4/projects/4242/trigger/pipeline"The started pipeline has CI_PIPELINE_SOURCE set to trigger.
Variables sent with the call have the highest precedence of all. The token acts with the
access of the user it belongs to.
- Scope
- One project
- Binding
- Inline
- Control
- Opt-in
- Phase
- Run
- Merged view
- Keyword only
docs:
trigger:
include: ci/docs.ymlGitLab starts a child pipeline from the file, in the same project and on the same ref
and commit. The child inherits the parent's top-level variables but not its
default:. It appears as a card to the right of the parent's graph. Children
nest at most two levels deep.
The trigger job names the child's file. Open the card to the right of the parent's graph to see the child's jobs, then read the file at the same commit.
- Without
strategy: mirror, the trigger job passes as soon as the child is created, even if every job in it then fails. - The parent's
default:, such as its image orbefore_script, doesn't reach the child's jobs. - In a merge request pipeline, a child whose jobs have no rules gets no jobs, and the trigger job fails.
- Scope
- One project
- Binding
- Inline
- Control
- Opt-in
- Phase
- Run
- Merged view
- Not shown
service-pipelines:
trigger:
include:
- artifact: generated-pipeline.yml
job: generate-pipelineOne job writes pipeline configuration as an artifact, and a trigger job starts a child
pipeline from it. The configuration exists only for that pipeline. An
include inside the generated configuration can't use CI/CD variables.
Download the artifact from the generator job of the pipeline you are investigating: that YAML is exactly what ran. Then read the generator script, to understand why it wrote what it wrote.
- No merged view ever shows the child's configuration, because it didn't exist before the pipeline ran.
- The generator must always write at least one job. A generator that sometimes writes nothing breaks the pipeline only sometimes.
- Scope
- Cross-project
- Binding
- Always live
- Control
- Opt-in
- Phase
- Run
- Merged view
- Not shown
deploy-staging:
trigger:
project: megacorp/platform/deployer
branch: main
strategy: dependGitLab starts a pipeline in the other project, on the branch named, using that project's own configuration. The user who triggered the upstream pipeline must be allowed to run pipelines in the downstream project.
Open the card to the right of the graph to reach the downstream pipeline. Its jobs come from the downstream project's configuration at the head of the named branch, not from anything in your project.
- The downstream configuration floats with that branch. The platform team can change your deployment without a commit in your project.
- Someone without access to the downstream project can run your pipeline, but not its deployment.
- Scope
- Cross-project
- Binding
- Always live
- Control
- Opt-in
- Phase
- Run
- Merged view
- Not shown
curl --fail --request POST \
--form "token=$REPORTS_TRIGGER_TOKEN" \
--form "ref=main" \
"https://gitlab.example.com/api/v4/projects/4242/trigger/pipeline"The API creates a pipeline in the project named by its number, on the ref given.
Variables sent with the call outrank every other variable. The pipeline's
CI_PIPELINE_SOURCE is trigger. The token acts with its owner's
project access.
Search scripts for /trigger/pipeline. Look up the project number in the
URL to find which project is being started, and find out who owns the token
variable.
- A number in a URL is the only sign of the relationship. No file names the target project.
- Because the token acts with its owner's access, the call can break when that person's access changes, far from any pipeline file.
- Do this
- Add
strategy: mirrorto a trigger job when your pipeline's result should include the downstream pipeline's result. - Why
- With it, the trigger job shows the downstream pipeline's status: it waits for that
pipeline and fails if it fails. Older files use
strategy: depend, which GitLab no longer recommends because its status doesn't always match. - Watch out
- Without it, the trigger job passes as soon as the downstream pipeline has been created, whatever happens to that pipeline afterwards. Your pipeline can show green while the real work failed.
- Read a parent, child or multi-project pipeline and say which file each part came from.
- Make a trigger job wait for its child and fail with it.
- Explain why a child of a merge request pipeline can end up with no jobs, and fix it.
Spot the bugGreen pipeline, nothing deployed#
A team copied MegaCorp's deploy template into their own file, and trimmed it:
deploy-staging:
stage: deploy
trigger:
project: megacorp/platform/deployer
branch: main
variables:
APP: $CI_PROJECT_NAME
IMAGE_REF: $IMAGE_REFTheir pipelines are green, but staging hasn't changed in a week. The deployer project shows a failed pipeline for every one of their merges.
Show the answer
With no strategy, a trigger job succeeds as soon as the downstream
pipeline is created. It doesn't wait for that pipeline, or reflect how it ends. The
deployer's pipelines failed, but the trigger job had already passed.
Add strategy: mirror, so that the trigger job takes the downstream
pipeline's status. See multi-project pipelines and
what the trigger job's status means.
Organisation-wide reuse: settings and policies#
Nine mechanisms reach a project without any of its files asking for them. Four are available on every tier (GitLab's paid plans are Free, Premium and Ultimate):
- a setting that points the project at a pipeline file somewhere else
- Auto DevOps, GitLab's own ready-made pipeline
- CI/CD variables set in the settings of the instance, group and project
- templates that are copied in when a file or project is created
Four more come with GitLab Ultimate. They are security policies, rules a security team sets once for many projects: pipeline execution policies, scan execution policies and merge request approval policies. The fourth is compliance pipelines, which those policies replace. A ninth, the instance template repository, is a copy-once template list on self-managed Premium.
None of them shows up in the project's files. You find them in settings pages, and in the pipeline itself.
You change MAVEN_CLI_OPTS in payments-api's file, push, and the job still uses the old options. Full configuration shows your new value. The log shows the old one. Nothing in the project explains it. Where else can a variable come from? (the card)
Where each one lives#
Two words first. The instance is the whole GitLab installation. Self-managed means your organisation runs that installation itself, rather than using GitLab.com.
| Mechanism | Who sets it | Where you look | Tier |
|---|---|---|---|
| Custom CI/CD configuration file | a project maintainer | the project's CI/CD settings, under General pipelines | all |
| Auto DevOps | an administrator or group owner, or the project | the Auto DevOps section of the CI/CD settings | all |
| Instance, group and project variables | administrators, group owners, project maintainers | the CI/CD variables settings at each level | all |
| Project and file templates | whoever created the project or file | nowhere afterwards: the copy is the only trace | all; templates kept in a group need Premium |
| Instance template repository | an administrator | Admin › Settings › Templates | Premium, self-managed and Dedicated |
| Pipeline execution policies | the security team | the group's security policies | Ultimate |
| Scan execution policies | the security team | the group's security policies | Ultimate |
| Merge request approval policies | the security team | the group's security policies, and the merge request itself | Ultimate |
| Compliance pipelines | compliance owners | the project's compliance framework label | Ultimate, deprecated |
In the UI, settings variables are under Settings › CI/CD › Variables: in the project, in each group, and in the Admin area for the instance. Security policies are under Secure › Policies, in a project or a group.
The most common puzzle these mechanisms cause is a project that has pipelines but no pipeline file. Check these places, in this order:
As text
- Do the project's CI/CD settings name a configuration file? Yes: A custom configuration file, perhaps in another project. No: the next step.
- Is Auto DevOps on for the project, its group or the instance? Yes: GitLab's own Auto DevOps pipeline. No: the next step.
- Does a scan execution policy apply to the project? Yes: The policy creates a scan pipeline itself. No: the next step.
- Ask the group's owners about compliance frameworks and other policies
A pipeline file that isn't in the repository#
web-portal's repository has no .gitlab-ci.yml, yet it has pipelines. Its
settings point at a file in the templates project:
ci_cd_configuration_file: pipelines/web.yml@megacorp/devops/ci-templatesThe setting accepts three forms:
- a path in the same repository
- a file in another project, written as
path@namespace/project, optionally with:refat the end - a URL ending in
.yml
The project's own pipeline editor can't edit a file that lives somewhere else. That is
often the first sign this setting is in play. On the local GitLab 19.3,
web-portal's pipelines ran the three jobs defined in ci-templates'
pipelines/web.yml.
- Scope
- Cross-project
- Binding
- Pinned or floating
- Control
- Enforced
- Phase
- Compile
- Merged view
- Not shown
pipelines/web.yml@megacorp/devops/ci-templates
pipelines/web.yml@megacorp/devops/ci-templates:v4.2.0GitLab reads the pipeline configuration from the path the setting names, instead of
.gitlab-ci.yml at the root of the repository. The file can be in the same
project, in another project at an optional ref, or at a URL.
Open the project's CI/CD settings and expand General pipelines. If the file is elsewhere, read it in that project at the ref given. The project's own pipeline editor won't show it.
- Without a ref, GitLab reads the file from the other project's default branch. Every change there reaches every consuming project at once.
- A repository with no pipeline file isn't necessarily a project without a pipeline.
Variables from settings#
CI/CD variables can be set on the instance, on each group, and on the project. Every job in every project below them receives them, and they outrank anything in YAML:
# Every project on the instance receives these, unless a group or project
# variable with the same name takes precedence.
instance_variables:
- key: MEGACORP_REGISTRY
value: registry.example.com
- key: HTTPS_PROXY
value: http://proxy.example.com:3128# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
- key: MAVEN_CLI_OPTS
value: "--batch-mode --errors --show-version -s .m2/settings.xml"
- key: AWS_ACCOUNT_ID
value: "123456789012"
- key: MC_AZURE_TENANT_ID
value: "00000000-0000-0000-0000-000000000000"
- key: MC_ACR_REGISTRY
value: megacorpprod.azurecr.io
- key: MC_ACR_PUSH_CLIENT_ID
value: "00000000-0000-0000-0000-000000000000"
# the gitlab-acr-push application in Entra ID, which mc_acr_push signs in as
- key: SONAR_HOST_URL
value: https://sonar.example.com
- key: SONAR_TOKEN
value: "(set in the UI)"
masked: true
protected: trueAmong themselves, the project beats its groups, the closest group beats the ones above it, and groups beat the instance. Only policy variables, variables typed into a manual job, and pipeline variables rank higher. Chapter 26 gives the whole ladder.
- Scope
- Organisation-wide
- Binding
- Always live
- Control
- Inherited
- Phase
- Create
- Merged view
- Not shown
Group megacorp › CI/CD variables
MAVEN_CLI_OPTS = --batch-mode --errors --show-version -s .m2/settings.xmlEvery pipeline in every project under the group, or on the instance, receives the variable. It outranks variables written anywhere in the configuration, including job variables. It can also be protected, masked or hidden.
A job's log never lists its variables. Check the CI/CD variables settings of the project, then each parent group, then the instance. Where you lack access to a level, ask someone who has it.
- Setting the same variable in YAML does nothing: the settings value wins silently.
- A protected variable exists only in pipelines on protected branches and tags.
Auto DevOps#
Auto DevOps is GitLab's own complete pipeline. It runs for a project only if the
project has no .gitlab-ci.yml, and only if it finds a
Dockerfile, or a buildpack that matches the code. A project's own
configuration always wins. On a self-managed instance, the setting for all projects
starts switched on: the local GitLab 19.3 had it on before any setting was
changed. MegaCorp left it on:
# A project with no CI/CD configuration file gets GitLab's Auto DevOps pipeline.
auto_devops:
default_to_auto_devops: true- Scope
- Organisation-wide
- Binding
- Always live
- Control
- Inherited
- Phase
- Compile
- Merged view
- Not shown
Settings › CI/CD › Auto DevOps
[x] Default to Auto DevOps pipelineWhen a project has no CI/CD configuration and Auto DevOps applies to it, GitLab runs its built-in Auto DevOps pipeline, which builds, tests, scans and can deploy. The setting can come from the project, a group or the instance.
If a project without a pipeline file has pipelines, check the project's Auto DevOps setting, then its groups'. Then check the custom configuration file setting, which is the other explanation.
- Adding any
.gitlab-ci.yml, even a small one, switches Auto DevOps off for that project. - Its jobs change with GitLab upgrades, like any GitLab template.
Its jobs are easy to recognise. They have names such as build,
test and code_quality, scanner jobs such as
container_scanning and secret_detection, and deploy jobs such as
review, staging and production.
Templates copied at creation#
Two mechanisms hand a project configuration once, when a file or a project is created, and never again:
- Project templates. A new project is created as a copy of a template
project, including its
.gitlab-ci.yml. GitLab ships some on every tier; keeping your own in a group, as MegaCorp does, needs Premium Premium. - The instance template repository Premium
Self-managed Dedicated. An administrator chooses a project whose
gitlab-cifolder fills the template list in the Web Editor, GitLab's in-browser file editor. Picking one copies it into the new file. Despite the name, these templates can't be used withinclude: template.
# New projects can be created from these. The copy happens once, at creation.
group_project_templates:
source_group: megacorp/templates
templates:
- megacorp/templates/java-service-starter
- megacorp/templates/batch-job-starter# Self-managed only: the .yml files in this project's gitlab-ci/ folder appear in the
# Web Editor's template list when anyone creates a CI/CD file. Choosing one copies it
# into the new file; include: template cannot use them.
instance_template_repository:
project: megacorp/devops/instance-templates
offers:
- gitlab-ci/MegaCorp-Legacy-Java.yml- Scope
- Organisation-wide
- Binding
- Copied once
- Control
- Opt-in
- Phase
- Compile
- Merged view
- Shown
New project › Create from template › Group › megacorp/templates/java-service-starterGitLab copies the template project, including its pipeline file, into the new project when it is created. After that, the copy is an ordinary file in the new project, with no link back to the template.
Compare the project's first commit with the template project's history. A
.gitlab-ci.yml that arrived fully formed in the first commit came from
somewhere.
- Fixes to the template never reach projects created from it. Copies drift apart silently.
- Scope
- Organisation-wide
- Binding
- Copied once
- Control
- Opt-in
- Phase
- Compile
- Merged view
- Shown
Admin › Settings › Templates
Templates project: megacorp/devops/instance-templates
gitlab-ci/MegaCorp-Legacy-Java.ymlThe .yml files in the chosen project's gitlab-ci folder
appear in the template list whenever someone creates a CI/CD file in the Web Editor.
Choosing one copies its content into the new file.
If a project's pipeline file looks like one of the organisation's templates, compare it with the file in the administrators' templates project. Only administrators can see which project that is.
- It is a copy, not an include: the template can change without the project changing.
include: templatereaches only GitLab's own templates, never these.
Security policies Ultimate#
On GitLab Ultimate, a security team keeps policies in a policy project linked to a group, and they apply to every project under it. MegaCorp's policy project defines one of each kind:
pipeline_execution_policy:
- name: MegaCorp guardrails
description: Add an SBOM job to every pipeline.
enabled: true
pipeline_config_strategy: inject_policy
content:
include:
- project: megacorp/security/policies
file: pipeline-policies/megacorp-guardrails.ymlpolicy-sbom:
stage: .pipeline-policy-post
image: registry.example.com/megacorp/devops/ci-tools/syft:1
script:
- syft dir:. -o cyclonedx-json=sbom.cdx.json
artifacts:
reports:
cyclonedx: sbom.cdx.jsonA pipeline execution policy puts its own CI/CD configuration into
every pipeline. With inject_policy, its jobs are added to the project's own
jobs. With override_project_ci, the policy's configuration replaces the
project's entirely. Two reserved stages frame the pipeline:
.pipeline-policy-preruns before everything else. Every other job waits for it, and if it fails, the rest is skipped..pipeline-policy-postruns after everything else.
scan_execution_policy:
- name: Secret detection everywhere
description: Run secret detection in every pipeline on every branch.
enabled: true
rules:
- type: pipeline
branches: ["*"]
actions:
- scan: secret_detection
- name: Nightly dependency scan
description: Scan dependencies on the default branch every night.
enabled: true
rules:
- type: schedule
branches: [main]
cadence: "0 2 * * *"
actions:
- scan: dependency_scanningA scan execution policy adds GitLab's scanners to pipelines, or runs
them on a schedule. Its jobs get a hyphen and a number added to their names, such as
secret-detection-1, so that they can't clash with the project's own scanner
jobs. Scheduled scans run as a bot user, not as a member of your team.
As text
- build: maven-build
- test: maven-test: [17], maven-test: [21], secret-detection-1 (policy)
- .pipeline-policy-post: policy-sbom (policy)
approval_policy:
- name: Block new critical vulnerabilities
description: A merge into main that adds a critical finding needs security's approval.
enabled: true
rules:
- type: scan_finding
branches: [main]
scanners: [sast, secret_detection, dependency_scanning, container_scanning]
vulnerabilities_allowed: 0
severity_levels: [critical]
vulnerability_states: [new_needs_triage]
actions:
- type: require_approval
approvals_required: 1
group_approvers: [megacorp/security]A merge request approval policy adds no jobs. It reads the scanners' results, and when its rule matches, the merge request needs the approvals the policy names. If a required scanner produced no report, the policy by default requires the approval anyway: it fails closed.
LegacyCompliance pipelines
Before pipeline execution policies, a compliance framework could name a pipeline file,
written path@group/project, for every project carrying its label. A
compliance framework is a label an administrator attaches to a project, to mark the
rules it must follow. By default the framework's file ran instead of the
project's own configuration, unless it included the project's
.gitlab-ci.yml explicitly. Compliance pipelines were deprecated in 17.3 and
are planned for removal in 20.0. Use pipeline execution policies instead.
# Legacy: a compliance framework with its own pipeline configuration, the
# predecessor of pipeline execution policies.
compliance_frameworks:
- name: SOX
pipeline_configuration: compliance/sox.yml@megacorp/security/policies
applied_to:
- megacorp/payments/payments-api- Scope
- Organisation-wide
- Binding
- Always live
- Control
- Enforced
- Phase
- Create
- Merged view
- Not shown
pipeline_execution_policy:
- name: MegaCorp guardrails
pipeline_config_strategy: inject_policy
content:
include:
- project: megacorp/security/policies
file: pipeline-policies/megacorp-guardrails.ymlWhen a pipeline is created, GitLab adds the policy's configuration to it, or, with
override_project_ci, uses the policy's configuration instead of the
project's. Policy jobs can run in the reserved stages .pipeline-policy-pre
and .pipeline-policy-post. Policy variables outrank the project's and the
groups'. A policy project holds at most five of these policies. Generally available in
17.3; inject_policy from 17.9.
A job in no project file, or in a .pipeline-policy-* stage, points here.
Open the group's security policies, find the policy's content, and read that
file in the policy project.
- If a policy job has the same name as a project job, the policy job gets
:policy-<project id>-<index>added to its name by default. - A failure in
.pipeline-policy-preskips every other job in the pipeline. - With
override_project_ci, editing your.gitlab-ci.ymlmay change nothing at all.
- Scope
- Organisation-wide
- Binding
- Always live
- Control
- Enforced
- Phase
- Create
- Merged view
- Not shown
scan_execution_policy:
- name: Secret detection everywhere
rules:
- type: pipeline
branches: ["*"]
actions:
- scan: secret_detectionGitLab adds the named scanners to matching pipelines, or runs them on a schedule.
Added jobs are renamed with a hyphen and a number. They run in the test
stage, or in scan-policies when there is no test stage; DAST
runs in dast. For projects without a pipeline file, the policy creates the
configuration itself.
A scanner job with a numbered name, or a scan in a project that includes no scanner template, points here. Open the group's security policies.
- Toggles such as
SAST_DISABLEDskip only the project's own scanner jobs. They can't switch off a scan the policy enforces. - Scheduled scans ignore
[skip ci], and run as thesecurity_policy_botuser.
- Scope
- Organisation-wide
- Binding
- Always live
- Control
- Enforced
- Phase
- Run
- Merged view
- Not shown
approval_policy:
- name: Block new critical vulnerabilities
rules:
- type: scan_finding
severity_levels: [critical]
vulnerabilities_allowed: 0
actions:
- type: require_approval
approvals_required: 1GitLab checks the rule against the scanners' results for the merge request. When it matches, the merge request needs the approvals the policy names. Rules can also look at licence findings, or apply to every merge request. By default, a missing scanner report makes the policy require approval: it fails closed.
A merge request blocked "because of a policy", with every pipeline job green, points here. The merge request page names the policy. The policy file in the policy project shows what it checks.
- Formerly called scan result policies, under the key
scan_result_policy. From 17.0, onlyapproval_policyis accepted. - Removing a scanner job to get past a failing scan can make the merge request need approval, because the report is now missing.
- Scope
- Organisation-wide
- Binding
- Pinned or floating
- Control
- Enforced
- Phase
- Compile
- Merged view
- Not shown
Compliance framework: SOX
Compliance pipeline configuration: compliance/sox.yml@megacorp/security/policiesFor every project with the framework's label, GitLab runs the compliance pipeline configuration instead of the project's own, unless that configuration includes the project's file. Deprecated in 17.3, planned for removal in 20.0, and replaced by pipeline execution policies.
Check the project for a compliance framework label, then read the file its framework names.
- A project whose own
.gitlab-ci.ymlseems to be ignored may have a compliance pipeline that doesn't include it.
- Do this
- Before you set a variable in your YAML, look for a variable with the same name in the settings of the project, its groups and the instance.
- Why
- A settings variable always wins over a value in YAML, whatever you write. The card your change is ignored shows the symptom.
- Watch out
- Settings belong to whoever owns them. If a setting is in your way, ask its owner to change it. Do not work around it in your file, because the next person will not know why.
- Find a variable set in the project's, a group's or the instance's settings, and know that it beats the YAML.
- Recognise a job that a policy added, and know that no toggle in your file removes it.
- Tell the organisation-wide mechanisms apart: settings, policies, templates, the instance's own configuration.
Spot the bugThe scan that won't switch off#
A payments engineer, fed up with a slow secret-detection job, added a toggle to the project's CI/CD variables:
SECRET_DETECTION_DISABLED = trueThe project's own secret-detection job disappeared. A job named
secret-detection-1 still runs in every pipeline.
secret-detection-1 come from, and why does the toggle
not affect it?Show the answer
The numbered name is the mark of a scan execution
policy. MegaCorp's security team enforces secret detection on every branch.
SECRET_DETECTION_DISABLED, which GitLab's template accepts as
"true" or "1", skips only the scanner job the project defines
itself. A scan that a policy enforces ignores it.
Whether the scan should run is the security team's decision, not the project's. Raise the slowness with them; don't look for a way round it.
Reuse outside the YAML#
YAML says what to run. In a large organisation, what actually happens is often decided somewhere else:
- by the job's image
- by a shell library, a company command-line tool, or a downloaded script
- by a build tool's shared configuration
- by the runner the job landed on
None of these shows in the merged configuration. A script line that says
mc deploy tells you nothing about what mc does. The job log is
where you start, and the question to ask of every command is "where did this come
from?"
The image build fails today with a flag that mc no longer accepts. Your YAML has not changed in a month. The job runs in ci-tools:latest, the central team's toolbox image, and last night they pushed a new one. The real logic was never in your file. Where is it? (the card)
Take any command from a job log, and ask these questions in order:
As text
- Did the log download it first, with curl or git clone? Yes: A fetched script: read what that URL served on that day. No: the next step.
- Is it a build tool, such as mvn, npm or gradle? Yes: Shared build configuration, such as a parent POM. No: the next step.
- Is it loaded by a source line in before_script? Yes: A shell library, usually shipped in the image. No: the next step.
- Otherwise it is a program inside the image: read the image's Dockerfile
- Behaves differently on different runners? Compare the runners' configuration
Toolbox images#
A toolbox image is an image a central team builds with every tool the organisation's jobs need. Every MegaCorp job runs in the ci-tools image unless it names another, because the base template sets it as the default:
default:
image: registry.example.com/megacorp/devops/ci-tools:3.2
tags: [megacorp-shared]
interruptible: true
retry:
max: 2
when: [runner_system_failure, stuck_or_timeout_failure]
before_script:
- !reference [.snippets, functions]
- mc_log "job $CI_JOB_NAME on $CI_COMMIT_REF_NAME"The image carries more than tools. It carries MegaCorp's own logic, copied in by its
Dockerfile, the recipe that builds the image:
COPY lib/ci-lib.sh /opt/megacorp/lib/ci-lib.sh
COPY bin/mc /usr/local/bin/mc
RUN chmod +x /usr/local/bin/mc- Scope
- Organisation-wide
- Binding
- Pinned or floating
- Control
- Opt-in
- Phase
- Run
- Merged view
- Keyword only
default:
image: registry.example.com/megacorp/devops/ci-tools:3.2The runner starts the job in a container from the image. Every command the script calls, and every file it reads that isn't in your repository, comes from the image's contents at the moment it was pulled.
The first lines of the job log name the image that was used. Find the project that
builds it, here megacorp/devops/ci-tools, and read its
Dockerfile for what gets copied in. Read the copied files at the version the
image was built from.
- An image tag like
3.2can be pushed again with new contents. Only an image digest, a fixed ID for its exact contents such as@sha256:…, is truly pinned. - Upgrading the image can change what every job in the organisation does, with no commit in any pipeline file.
Shell function libraries#
A shell function is a named group of commands, like a small script inside the shell.
Central teams package functions in a library file rather than repeating shell code in
YAML. A source command loads the file, and MegaCorp's snippet library does
nothing but that:
.snippets:
functions:
- source /opt/megacorp/lib/ci-lib.sh
aws_login:
- mc_aws_login "$AWS_ROLE_ARN"
acr_push:
- mc_acr_push "$IMAGE:$CI_COMMIT_SHA"
maven_settings:
- mkdir -p .m2
- mc_maven_settings > .m2/settings.xmlThe functions themselves live in the image:
# Exchange the job's OIDC ID token for temporary AWS credentials.
mc_aws_login() {
local role="${1:?role ARN required}"
local creds
creds=$(aws sts assume-role-with-web-identity \
--role-arn "$role" \
--role-session-name "gitlab-${CI_PROJECT_ID}-${CI_JOB_ID}" \
--web-identity-token "$MC_ID_TOKEN" \
--duration-seconds 3600 \
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
--output text) || { mc_log "AssumeRoleWithWebIdentity failed for $role"; return 1; }
read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< "$creds"
export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
mc_log "assumed $role"
}- Scope
- Cross-project
- Binding
- Pinned or floating
- Control
- Opt-in
- Phase
- Run
- Merged view
- Keyword only
before_script:
- !reference [.snippets, functions] # source /opt/megacorp/lib/ci-lib.sh
script:
- mc_aws_login "$AWS_ROLE_ARN"A source line loads shell functions into the job's shell. They are then
available to everything that runs later in the same shell: before_script
and script share one. The library usually ships in the image, or arrives
through a snippet or a download.
A command in the log that isn't a real program is a function. Search the snippets for
the source line that loads it, then read the library file in the project
that builds the image.
- A job that redefines
before_scriptloses thesourceline, and fails with "command not found" at the first function call. - Functions are written for one shell. A library written for bash fails in an image
that only has
sh.
Company command-line tools#
Some organisations wrap their deployment steps in a command of their own. MegaCorp's is
called mc:
deploy)
env="${1:?usage: mc deploy <environment> <image-ref>}"
image="${2:?usage: mc deploy <environment> <image-ref>}"
tag="${image##*:}"
mc_log "promoting ${APP} ${tag} to ${env} through the GitOps repository"
git clone --depth 1 \
"https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
yq -i ".image.tag = \"${tag}\"" "gitops/envs/${env}/${APP}.yaml"
git -C gitops commit -am "deploy ${APP} ${tag} to ${env}"
if [ "$env" = production ]; then
# production changes wait in a merge request until someone merges them
git -C gitops push origin "HEAD:deploy/${APP}-${tag}" \
-o merge_request.create -o merge_request.target=main \
-o merge_request.title="Deploy ${APP} ${tag} to production"
else
git -C gitops push origin HEAD:main
fi
;;The deployer's whole job is mc deploy "$ENVIRONMENT" "$IMAGE_REF". That
one line clones a GitOps repository with a token, edits a values file and pushes a
commit. A GitOps repository holds the desired state of the running systems, and a tool in
the cluster applies whatever it says (chapter 45). Nothing in any pipeline file says any of this.
- Scope
- Organisation-wide
- Binding
- Pinned or floating
- Control
- Opt-in
- Phase
- Run
- Merged view
- Keyword only
script:
- mc deploy "$ENVIRONMENT" "$IMAGE_REF"The script calls a company tool, installed in the image or downloaded by an earlier command. All the behaviour is in the tool's own code, at whatever version the image contains.
Find how the tool got into the image, here COPY bin/mc in the Dockerfile,
and read its source. Many such tools print their version. Add that command to a job to
see which version you are running.
- A tool that reads variables such as
APPorGITOPS_TOKENdepends on names that no pipeline file documents.
Scripts fetched at run time#
The oldest trick of all is to download a script and run it, in one line:
test:
<<: *defaults
stage: test
script:
- curl -sSL https://gitlab.example.com/megacorp/devops/scripts/-/raw/main/run-tests.sh | bash- Scope
- Cross-project
- Binding
- Pinned or floating
- Control
- Opt-in
- Phase
- Run
- Merged view
- Keyword only
curl -sSL https://gitlab.example.com/megacorp/devops/scripts/-/raw/main/run-tests.sh | bashThe job downloads a script from a URL or a repository while it runs, and executes it. What runs is whatever the URL served at that moment.
Open the URL from the job log. For a branch URL like …/raw/main/…, check
the file's history, to see what it said when the job ran.
- A URL on a branch is the most floating kind of reuse there is: unpinned, invisible to every GitLab view, and able to change mid-afternoon.
- If the host is down, the job fails, even though nothing about your project changed.
Build-tool reuse#
Build tools share configuration too. A Maven project can inherit its build settings
from a parent POM: a shared pom.xml published by another team. Java
services at MegaCorp do:
<parent>
<groupId>com.example.megacorp</groupId>
<artifactId>megacorp-parent</artifactId>
<version>12.3.0</version>
</parent>- Scope
- Cross-project
- Binding
- Pinned or floating
- Control
- Opt-in
- Phase
- Run
- Merged view
- Not shown
script:
- mvn $MAVEN_CLI_OPTS verifyThe pipeline calls the build tool. The build tool reads configuration that is shared through its own mechanism: a Maven parent POM, a Gradle convention plugin, a shared npm configuration package, or a Makefile included from elsewhere. Plugins, profiles and checks can all arrive that way.
Look in the build file for what it inherits: <parent> in a POM, or
the equivalent in your build tool. Then read that shared configuration at the version
named.
- A new check in the parent POM can fail every build that uses it. The pipeline file won't have changed.
Runner configuration#
The runner itself can change every job it runs, through a file called
config.toml that projects can't see:
[[runners]]
name = "megacorp-eks-shared"
# registered in GitLab as a group runner of megacorp, with the tag megacorp-shared
url = "https://gitlab.example.com"
executor = "kubernetes"
environment = ["MAVEN_OPTS=-Xmx2g", "HTTPS_PROXY=http://proxy.example.com:3128"]
pre_build_script = "echo 'runner: megacorp-eks-shared'"
[runners.kubernetes]
namespace = "gitlab-runners"
image = "registry.example.com/megacorp/devops/ci-tools:3.2"
# jobs run as this service account, which is bound to an AWS IAM role
service_account = "gitlab-runner-jobs"- Scope
- Organisation-wide
- Binding
- Always live
- Control
- Enforced
- Phase
- Run
- Merged view
- Not shown
[[runners]]
environment = ["MAVEN_OPTS=-Xmx2g"]
pre_build_script = "echo 'runner: megacorp-eks-shared'"A runner's config.toml can add or overwrite environment variables for
every job. It can run commands before fetching the source, before the job and after it.
It chooses the executor and shell, can restrict which images jobs may use, and caps the
log size, 4 MB by default. On Kubernetes, the pod's service account can carry cloud
permissions.
The job log names the runner that ran the job. If a job behaves differently depending on which runner took it, compare the runners' configuration. Only the runners' administrators can read it, so ask them.
- A job log longer than the runner's limit is cut off, so the end of a failing job may be missing.
- Moving a job to a different runner, by changing its tags, can change its environment, its permissions and its image policy all at once.
GitLab Functions Experimental#
GitLab's experiment in reusable steps, formerly called CI/CD Steps, replaces a job's script with a list of steps. Each step either calls a published function or runs a script:
lint-docs:
image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
run:
- name: markdown_lint
func: registry.example.com/megacorp/devops/functions/markdownlint:1.2.0
inputs:
path: docs
- name: report
script: echo "markdown lint finished"- Scope
- Cross-project
- Binding
- Pinned or floating
- Control
- Opt-in
- Phase
- Run
- Merged view
- Keyword only
lint-docs:
run:
- name: markdown_lint
func: registry.example.com/megacorp/devops/functions/markdownlint:1.2.0
inputs:
path: docsThe job runs its run: steps in order. A func: step loads a
function and passes it inputs. The function can come from an OCI registry (a server that
stores images and similar packages), a local path or, deprecated, a Git repository. A job
that uses functions can't also have before_script, script or
after_script.
Follow the function's reference to its registry or repository, and read it at the version given.
- It is an experiment. Its keywords have changed before, from
step:tofunc:, and may change again. - Loading functions from OCI registries needs a recent runner. An older runner fails the job, not the pipeline.
- Do this
- Keep the job logic that your team owns in scripts in your own repository, and call
those scripts from
script:. - Why
- You can then run the same scripts on your laptop, and a change to them goes through your team's review.
- Watch out
- When you use the central team's toolbox image, pin its tag. A moving tag, such as
latest, changes your tools without any commit in your project. The card tools changed overnight shows the symptom.
- Follow any command in a job log to where its logic lives: the image, a fetched script, a shell library, or a build tool's own configuration.
- Pin a toolbox image to a tag, and explain what a moving tag does to a pipeline.
- Say which parts of a job Full configuration can never show you.
Spot the bugThe command no repository defines#
The deployer's job log shows it running
mc deploy "staging" "123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:4f2c…".
It then pushes a commit to a repository called gitops-config. A developer searches every
repository they can see for a definition of mc, and finds nothing under that
name.
mc come from, and how would you read exactly what
mc deploy did in that job?Show the answer
The deploy job runs in the ci-tools image, and its log names the image. That image's
Dockerfile, in megacorp/devops/ci-tools, copies bin/mc into
/usr/local/bin/mc. The command is a script in that repository.
Read bin/mc at the version the image was built from, not at the head of
the branch. The deploy case shows the clone, the yq edit and
the push. See toolbox images and
company command-line tools.
Tracing a job to its source#
You now know all 37 mechanisms. This chapter puts them in order, as a procedure. Start from a job in the pipeline graph, and ask one question at a time:
- Is it in the merged configuration? If yes, walk its
extendschain and its!referencetags back to the files. - If it isn't, it was made when the pipeline was created (by a policy, a matrix or a downstream pipeline), or when it ran.
- Either way, finish in the job log. That is where you find the image, the runner, and the commands whose logic lives outside YAML.
maven-test fails with mc_log: command not found, again, in a different project. This time nobody changed before_script. You have the merged view, five included files, and a chain of three extends. You need a method, not a guess. (the card)
Here is the whole procedure on one page. The steps below take each branch in turn:
As text
- Read the job's name and stage in the pipeline graph (step 1)
- Is it in Full configuration, under its name without any [ ] suffix? Yes: Walk its extends chain, !reference tags and includes (step 2). No: the next step.
- Is it in a .pipeline-policy stage, or numbered like secret-detection-1? Yes: A security policy added it (step 4). No: the next step.
- Is it in a card to the right of the graph? Yes: A downstream pipeline: find its trigger job (step 3). No: the next step.
- None of these: check the project's settings for a config file or Auto DevOps (steps 3 and 4)
Step 1: read the name#
The job's name and stage in the pipeline graph often tell you which branch of the procedure you are on, before you open anything:
| The name or stage looks like | It is probably | Go to |
|---|---|---|
| an ordinary name | a job in the configuration | step 2 |
a name ending in […], such as maven-test: [17] | one job of a parallel: matrix | step 2, with the name before the colon |
a stage called .pipeline-policy-pre or .pipeline-policy-post | a pipeline execution policy job | step 4 |
a scanner name with a hyphen and a number, such as secret-detection-1 | a scan execution policy job | step 4 |
a name ending in :policy- and two numbers | a policy job renamed to avoid a clash | step 4 |
| a card to the right of the graph | a downstream pipeline | step 3 |
Step 2: find it in the merged configuration#
Open Build › Pipeline editor, then Full configuration, and search for the job name followed by a colon. If it is there, everything the file-based mechanisms contributed has already been merged in. What remains is to find which file each part came from:
- Follow
extendsupward. Search the included files for each parent's name, such as.maven-test:, then its parent's, until there is none. - Follow every
!reference. Its first item names a hidden job, which you search for in the same way. - Look for
default:and top-levelvariables:in every included file, and check the job forinherit:. - Note the ref of every include you passed through. A branch, or no ref at all, means the answer can change without a commit of yours.
Here is that walk for payments-api's maven-test. It starts from the only
line that mentions the job by name:
maven-build:
extends: .maven-build
maven-test:
extends: .maven-test
image-build:
extends: .image-build
needs: [maven-build]| Hop | File | What it contributes |
|---|---|---|
| 1 | payments-api .gitlab-ci.yml | a MAVEN_CLI_OPTS variable, and the include of the golden pipeline at v4.2.0 |
| 2 | ci-templates pipelines/java-service.yml | the job itself: maven-test: extends: .maven-test |
| 3 | ci-templates templates/java-maven.yml | .maven-test: stage, image, matrix, script, artifacts, rules |
| 4 | ci-templates templates/java-maven.yml | .maven-base: cache, variables, and a before_script built from !reference |
| 5 | ci-templates templates/snippets.yml | the snippets that before_script pastes in |
| 6 | ci-templates templates/base.yml | .megacorp-base: MC_TEAM, artifact expiry, and the rules that .maven-test replaced |
If the project's configuration file lives in another project, its own pipeline editor
can't show it. Open the file in that project, at the ref the setting names. GitLab can
still assemble the whole configuration for you. On the local GitLab 19.3, the CI
Lint API for web-portal, GET /api/v4/projects/:id/ci/lint?dry_run=true,
returned web.yml's jobs with every include from ci-templates already merged in.
Step 3: if it's in no file#
A job that isn't in the merged configuration was added when the pipeline was created, or belongs to another pipeline:
- Matrix jobs don't exist in the configuration under their bracketed names. Search for the name without the brackets.
- Policy jobs are added when the pipeline is created. Go to step 4.
- Jobs in a downstream card belong to another pipeline: a child from a file or a generated artifact, or another project's own pipeline. Open the card. Then find the trigger job in the upstream pipeline, which names the file, the artifact or the project.
- A project without a
.gitlab-ci.ymltakes its whole configuration from a custom configuration file setting, from Auto DevOps, or from a scan execution policy that creates one (chapter 18).
Step 4: settings and policies#
These places hold configuration that no file in the project shows:
- the project's CI/CD settings: its configuration file path, Auto DevOps, and variables
- the CI/CD variables of every parent group, and of the instance
- the group's security policies, on GitLab Ultimate
- the project's compliance framework label, if it has one
If you can't open a level yourself, the question for its owner is specific: "Which variables, policies or settings at your level apply to this project?"
Step 5: finish in the job log#
Configuration tells you what GitLab asked for. The job log tells you what happened. Its first lines name the runner and the image. After that, every command the script ran is echoed. For each command that isn't a standard tool, ask where it came from:
- a function loaded from a library in the image, like MegaCorp's
mc_aws_login - a tool installed in the image, like MegaCorp's
mc - a script downloaded during the job, with
curlorgit clone - behaviour configured in a build tool, like a parent POM or a Makefile
Chapter 19 covers each one. Behaviour that changes with the runner the job landed on, and that nothing in the job explains, belongs to the runner's configuration, which only its administrators can read.
- Take any job back to the file and line that set each of its settings, one hop at a time.
- Find a job that is in no file, and know that a policy put it there.
- See the merged configuration even when the project's file lives in another project.
Spot the bugTrace a production deploy#
web-portal's pipelines on main include a job called
deploy-site that publishes to production. The repository has no
.gitlab-ci.yml. You are asked three things: where is
deploy-site defined, which file decides the command it runs, and which AWS
role does it use?
deploy-site to its source and answer all three.Show the answer
- The project's CI/CD settings name
pipelines/web.yml@megacorp/devops/ci-templates, a custom configuration file. That file definesdeploy-site: extends: .publish-site, with its own rules. .publish-siteis intemplates/node.yml, included atv4.2.0. Itsscriptrunsaws s3 syncand a CloudFront invalidation. Itsbefore_scriptpastes two snippets fromtemplates/snippets.yml, which callmc_aws_loginfrom the function library in the ci-tools image..publish-sitesetsAWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-web-publish, unless a settings variable of the same name overrides it. Check the project's and the groups' variables before you trust the YAML (extends).
Pipeline sources and types#
Every pipeline is created by an event, and GitLab records which one in
CI_PIPELINE_SOURCE: push, merge_request_event,
schedule, web and a dozen more. The kind of pipeline then
decides which variables exist:
CI_COMMIT_BRANCHis set in branch pipelines and schedules, and not in merge request pipelinesCI_COMMIT_TAGis set only for tagsCI_MERGE_REQUEST_*is set only in merge request pipelines
So the first question about any job that ran, or didn't, is "what kind of pipeline was this?" One push to a branch with an open merge request can start two pipelines.
contract-test runs on every push to a branch, and never in a merge request pipeline. Its rule says if: $CI_COMMIT_BRANCH, which looks right: it is a branch. The merge request's pipeline is for the same branch and the same commit. Why is one of them not "a branch"? (the card)
Where pipelines come from#
Seven kinds of event account for almost every pipeline you will meet in an enterprise project:
The full list of values, which is also what the pipelines API reports as a pipeline's
source:
| CI_PIPELINE_SOURCE | What created the pipeline |
|---|---|
push | a push to a branch, or a new tag |
merge_request_event | a merge request being created, a push to its source branch, or Run pipeline on its Pipelines tab |
schedule | a pipeline schedule coming due, or someone selecting Run on it |
web | New pipeline, under Build › Pipelines |
api | the pipelines API |
trigger | the pipeline trigger API, called with a trigger token |
pipeline | a trigger job in another project: a multi-project pipeline |
parent_pipeline | a trigger job in the same project: a child pipeline |
security_orchestration_policy | a scan execution policy's schedule, on GitLab Ultimate |
webide | the Web IDE |
chat | a ChatOps command, typed into a chat tool connected to GitLab |
external | a CI service other than GitLab |
external_pull_request_event | a pull request on GitHub, for a project that mirrors it |
ondemand_dast_scan | an on-demand DAST scan, which tests a running web application for security problems |
ondemand_dast_validation | a check that the site a DAST scan targets belongs to you |
GitLab Duo Agent Platform also runs workload pipelines, on refs named
refs/workloads/…, which pipeline lists show with the source
duo_workflow. They are not part of your CI/CD configuration.
The pipeline list labels each pipeline by type: branch,
tag, merge request, merged results or
merge train. Look at the label before you look at any rule.
What each kind of pipeline sets#
Rules test variables, and the kind of pipeline decides which of those variables exist. GitLab documents these combinations:
| Variable | Branch pipeline | Tag pipeline | Merge request pipeline | Scheduled pipeline |
|---|---|---|---|---|
| CI_PIPELINE_SOURCE | push | push | merge_request_event | schedule |
| CI_COMMIT_BRANCH | the branch | not set | not set | the schedule's branch |
| CI_COMMIT_TAG | not set | the tag | not set | only if the schedule runs on a tag |
| CI_MERGE_REQUEST_* | not set | not set | set while the merge request is open | not set |
| CI_PIPELINE_SCHEDULE_DESCRIPTION | not set | not set | not set | the schedule's description |
Three consequences catch almost everyone:
- A merge request pipeline has no branch. Any rule that tests
CI_COMMIT_BRANCHis false there. TestCI_MERGE_REQUEST_SOURCE_BRANCH_NAMEorCI_MERGE_REQUEST_TARGET_BRANCH_NAMEinstead. - A tag pipeline has no branch either, even when the tagged commit is
on
main. - A schedule on a branch sets
CI_COMMIT_BRANCHexactly as a push does. A rule meant for "merged to main" also matches the nightly schedule onmain, unless it testsCI_PIPELINE_SOURCE. That is why payments-api's docs job runs every night (chapter 14).
A few other variables describe the pipeline rather than the commit:
CI_COMMIT_REF_NAMEis the branch or tag the pipeline is built for.CI_OPEN_MERGE_REQUESTSlists up to four merge requests whose source is the current branch. It is set only in branch and merge request pipelines, and only when such a merge request exists. Chapter 22 uses it to stop duplicate pipelines.CI_COMMIT_BEFORE_SHA, the commit the branch pointed at before the push, is all zeros in merge request pipelines, scheduled pipelines, manual runs, and the first pipeline of a new branch or tag.
MegaCorp names these tests once, in its rules library, and every template picks from it:
.rules:
mr:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
default-branch:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
release-tag:
- if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
never-on-schedule:
- if: $CI_PIPELINE_SOURCE == "schedule"
when: never| Rule | Matches | Watch out for |
|---|---|---|
| mr | every merge request pipeline | nothing else: branch, tag and scheduled pipelines never match it |
| default-branch | pushes to main, and scheduled or manual runs on main | it can't tell a merge to main from the nightly schedule on main |
| release-tag | tag pipelines whose tag looks like v1.4.0 | v1.4.0-rc1 matches nothing, so a release candidate's tag pipeline gets none of these jobs |
| never-on-schedule | removes the job from scheduled pipelines | only if GitLab reaches it before a rule that matches; see chapter 23 |
Merge request pipelines, in three flavours#
A merge request pipeline runs:
- when a merge request is created from a branch with at least one commit
- on every push to the merge request's source branch
- when someone selects Run pipeline on the merge request's Pipelines tab
It tests the source branch alone, not the result of merging it. GitLab has two
stricter variants, and CI_MERGE_REQUEST_EVENT_TYPE tells you which one you
are in:
| Flavour | What it tests | CI_MERGE_REQUEST_EVENT_TYPE | Tier and list label |
|---|---|---|---|
| Merge request pipeline | the source branch only | detached | every tier; merge request |
| Merged results pipeline | a temporary commit that merges the source into the latest target; if the two conflict, GitLab runs a plain merge request pipeline instead | merged_result | Premium; merged results |
| Merge train | the merge request together with every merge request queued ahead of it | merge_train | Premium; merge train |
Merge request pipelines don't exist until the configuration asks for them: some rule
has to match merge_request_event. Merged results pipelines and merge trains
also need a project setting, under Settings › Merge requests.
merge_request_event rule arriving through include: does not
satisfy this requirement, and that the rule must be in the project's own
.gitlab-ci.yml. On the local GitLab, payments-api gets every rule through
includes. It still got a merge request pipeline when its merge request was opened, on
every push after that, and whenever someone selected Run pipeline. If a
project's merge request pipelines go missing, the documented caveat is one more thing to
check.Two situations change what a merge request pipeline can reach:
- Merge requests from forks. The pipeline runs in the fork, with the fork's configuration and CI/CD variables, not the parent's. A member of the parent project can choose to run it in the parent instead. It then uses the fork's configuration with the parent's variables and runners, so GitLab warns first: the fork's code could try to read the parent's secrets.
- Protected variables and runners Since 18.1. A merge request pipeline gets them only when both branches are protected and in the same project, the person running it can push or merge to the target, and a project setting allows it. Otherwise a job that needs a protected secret finds it empty.
One push, two pipelines#
Push to a branch that has an open merge request and GitLab receives two events: the
push, and an update to the merge request. It tries to create a pipeline for each: a
branch pipeline with source push, and a merge request pipeline with source
merge_request_event. Each is created only if its rules add at least one job.
If both qualify, the same commit shows up twice in the merge request's
Pipelines tab, and runs every shared job twice.
The two can disagree, and it matters which one counts:
- Pipelines must succeed checks the merge request pipeline, not the branch pipeline. GitLab documents a race condition in which pipeline's result blocks or passes the merge request, so a green branch pipeline beside a red merge request pipeline can go either way.
- An invalid configuration pushed to a merge request's branch produces two failed pipelines, one of each type.
Which of the two you get is decided by workflow:rules and job rules,
in chapter 22. Since 19.2 a project setting can also decide
it: Settings › CI/CD › General pipelines › Skip branch pipelines for merge
requests Beta. With it on, a push to a merge request's source
branch creates only the merge request pipeline. The push that opens the merge request
still makes a branch pipeline, and jobs with no rules at all start appearing in merge
request pipelines.
Pipelines nobody pushed#
Several kinds of pipeline start without a commit. Each has a trait that confuses people:
| Kind | Source | What to know |
|---|---|---|
| Scheduled | schedule | runs with its owner's permissions; turns Inactive if the owner is blocked or leaves, until a Maintainer takes ownership; its variables exist only in its own pipelines; rules: changes is always true in it |
| New pipeline in the UI | web | values typed into the form override every other variable of the same name, and are not masked |
| API | api | the ref and variables come from the call |
| Trigger token | trigger | CI_PIPELINE_TRIGGERED is true; the token acts with its owner's access (chapter 17) |
| Multi-project | pipeline | every job in it sees pipeline; it runs on the target's default branch unless the trigger job names one |
| Child | parent_pipeline | every job in it sees parent_pipeline, even when the parent is a merge request pipeline |
| Policy scan | security_orchestration_policy | created by a scheduled scan execution policy, not by anything in the project |
A scheduled pipeline execution policy Since 19.2 also creates pipelines on a schedule. They run only the policy's jobs, never the project's (chapter 36).
The child pipeline row is the one that bites. A job in a child pipeline that tests
$CI_PIPELINE_SOURCE == "merge_request_event" never runs, because the source
is always parent_pipeline. The parent's CI_MERGE_REQUEST_*
variables are passed down, so a child job that should run only for merge requests tests
$CI_MERGE_REQUEST_ID instead.
A pipeline can also be created and then skipped. A commit message containing
[ci skip] or [skip ci], in any capitalisation, still creates a
pipeline, with no jobs and the status Skipped. The ci.skip
push option does the same, except for merge request pipelines, which it doesn't skip. A
security policy can forbid skipping.
When a variable exists#
GitLab sets predefined variables in three waves, and a rule can only test variables from the waves before it runs:
| Wave | Set when | Examples | Can be tested by |
|---|---|---|---|
| Pre-pipeline | before GitLab starts building the pipeline | CI_COMMIT_BRANCH, CI_COMMIT_TAG, CI_PIPELINE_SOURCE, CI_OPEN_MERGE_REQUESTS, CI_DEFAULT_BRANCH, CI_MERGE_REQUEST_* | include: rules, workflow: rules, job rules, scripts |
| Pipeline | while GitLab builds the pipeline | CI_PIPELINE_IID, CI_JOB_NAME, CI_JOB_STAGE, CI_NODE_INDEX, CI_ENVIRONMENT_NAME | job rules, scripts |
| Job-only | when a runner picks up the job | CI_JOB_ID, CI_JOB_TOKEN, CI_PIPELINE_ID, CI_PROJECT_DIR, CI_RUNNER_TAGS | scripts only: never workflow, include, rules or trigger jobs |
Three more exclusions follow from the same timing:
- Dotenv variables, which a job writes for later jobs, never reach any rule. Every rule is evaluated before the first job runs.
rules: ifcan't useCI_ENVIRONMENT_SLUG, or the variables GitLab calls persisted, such asCI_PIPELINE_ID,CI_JOB_TOKENandCI_REGISTRY_PASSWORD.includecan't use top-levelvariables:or job variables, because includes are resolved before either exists. It can use settings variables,CI_PROJECT_*,CI_PIPELINE_SOURCE,CI_COMMIT_REF_NAME, and variables from triggers, schedules and manual runs.
When a rule refers to a variable and never matches, look for that variable in these three lists before you look anywhere else.
- Do this
- In rules, test
$CI_PIPELINE_SOURCEto tell a merge request pipeline from a branch pipeline. - Why
- That variable is set in every kind of pipeline, so a rule on it always has something to compare.
- Watch out
- Use
$CI_COMMIT_BRANCHonly when you mean a branch pipeline. In a merge request pipeline it is not set at all, so a rule on it never matches there. The card missing from merge request pipelines shows the symptom.
- Name the seven kinds of pipeline you will meet, and the variable that tells them apart.
- Say which predefined variables exist in each kind, and so which rules can match there.
- Read a rule and predict in which kinds of pipeline the job will appear.
Spot the bugThe check that never runs on merge requests#
payments-api adds a slow contract test. It should run only in merge requests that
target main:
contract-test:
stage: test
script:
- mvn $MAVEN_CLI_OPTS verify -Pcontract
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHIt never appears in any merge request pipeline, including merge requests into
main. It never appears in any other pipeline either.
Show the answer
CI_COMMIT_BRANCH is not set in merge request pipelines, so the second
comparison is false whenever the first is true. The branch a merge request is heading
for is in CI_MERGE_REQUEST_TARGET_BRANCH_NAME:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCHworkflow: whether a pipeline exists#
workflow: rules decides whether a pipeline is created at all. GitLab
checks it after assembling the configuration and before looking at a single job:
- If no rule matches, there is no pipeline. For a push, you get no error anywhere.
- The syntax is the same as job rules, but
whencan only bealwaysornever.
Most organisations use it for one thing: stopping the duplicate branch and merge request pipelines from chapter 21. It can also name pipelines, set a variable for every job, and decide what a newer commit cancels.
You push a tag, v1.5.0-rc1, to cut a release candidate. Nothing happens. No pipeline, no red mark, nothing in the pipeline list. The jobs all have rules that allow tags. Something above the jobs said no before any of them was asked. (the card)
Checked before any job#
Whatever the jobs say, workflow has the first word. A job whose rules
allow tag pipelines never runs if workflow refuses tag pipelines, because
the pipeline it would have run in is never created.
A workflow rule may use if, changes, exists,
variables and auto_cancel. Its when is limited to
always and never. A rule that matches without a
when lets the pipeline be created. GitLab accepts start_in,
allow_failure and needs in a workflow rule without complaint,
and ignores them.
Workflow rules can only test variables that exist before any job does: the pre-pipeline variables of chapter 21, and variables from settings, triggers, schedules and manual runs. A workflow rule that tests a job-only variable can't work.
What you see when no rule matches depends on how the pipeline was asked for. After a push, nothing appears. GitLab gives no error because nothing failed; the pipeline was simply never wanted. When someone asks for a pipeline through the API, GitLab 19.3 refuses it with this error:
The pipeline did not run. Review the workflow:rules configuration for the pipeline.A different error means workflow let the pipeline through, but every job's rules left the job out:
The resulting pipeline would have been empty. Review the rules configuration.MegaCorp's workflow, rule by rule#
Every MegaCorp project that uses a golden pipeline includes this file:
workflow:
name: "$CI_PIPELINE_SOURCE: $CI_COMMIT_REF_NAME"
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: $CI_COMMIT_BRANCH
- if: $CI_COMMIT_TAGRead it top to bottom, stopping at the first rule that matches:
| The pipeline GitLab is asked for | First rule that matches | Created? |
|---|---|---|
| a merge request pipeline | 1: merge_request_event | yes |
| a push to a branch that has an open merge request | 2: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS | no |
a push to any other branch, main included | 3: $CI_COMMIT_BRANCH | yes |
| a tag | 4: $CI_COMMIT_TAG | yes |
a schedule on main | 3 | yes |
| a schedule, API call or trigger on a branch that has an open merge request | 2 | no |
Rules 1 to 3 are GitLab's documented switch pattern: branch pipelines until a merge request is opened, merge request pipelines after that. Rule 4 adds tags.
Rule 2 is broader than it looks. It doesn't ask whether the pipeline came from a push,
so it refuses every pipeline on a branch that has an open merge request, unless
it is a merge request pipeline. That includes pipelines started by an API call, a
schedule, or a trigger from another project. GitLab's documentation gives the guard
for exactly this, && $CI_PIPELINE_SOURCE == "push", and MegaCorp's
rule doesn't have it. The exercise at the end of this chapter is what that costs.
Duplicate pipelines#
One push to a branch with an open merge request makes GitLab try to create two pipelines (chapter 21). You get both whenever some job's rules accept both. The configurations that cause it:
- A final rule with no
if, such as- when: on_successor- when: always, which matches every kind of pipeline. CI Lint and the New pipeline page show the warningJob may allow multiple pipelines to run for a single actionfor it. - Rules that list both
pushandmerge_request_event, with no workflow to choose between them. - Jobs without rules mixed with jobs with rules. A job with no rules at all gets the legacy default: branches and tags, never merge requests. Put it beside a job whose rules match merge request pipelines and each push makes two pipelines, one for each job.
MegaCorp's monorepo has the third problem. It has no workflow section,
its generator jobs have no rules, and its docs trigger has a rule that tests only which
files changed:
docs:
stage: build
trigger:
include: ci/docs.yml
rules:
- changes: [docs/**/*]A rule with only changes matches in any kind of pipeline, merge request
pipelines included. When an open merge request touches docs/, every push to
its branch makes two pipelines: a branch pipeline with the generator jobs, and a merge
request pipeline containing only docs.
The local GitLab showed both pipelines, and a second problem. In every merge request
pipeline, docs failed, with the reason
downstream_pipeline_creation_failed. The child pipeline's own jobs have no
rules, and GitLab documents that such jobs don't join merge request pipelines, child
pipelines included. With none of its jobs added, the child pipeline couldn't be created
(chapter 21).
Teams stop duplicates in one of four ways. Recognise which one a project uses and you know which pipelines to expect:
| Pattern | What the workflow rules say | What you get |
|---|---|---|
| Switch | merge request events; never a branch with an open merge request; any other branch | branch pipelines until a merge request opens, then merge request pipelines only |
| Merge requests, main and tags | merge request events; the default branch; tags; sometimes protected branches | no pipeline at all for a branch without a merge request |
| Branches only | $CI_PIPELINE_SOURCE == "push", and no rule for merge request events | no merge request pipelines, and no merged results or merge trains |
| The project setting | nothing; the setting Skip branch pipelines for merge requests is on | as the switch, for pushes only; beta since 19.2 |
Names and variables for the whole pipeline#
workflow: name gives every pipeline a title in the
pipeline list, and the value is available to jobs as CI_PIPELINE_NAME.
MegaCorp's name shows the source and the ref, such as push: main or
schedule: main. That tells you the kind of pipeline before you open it. If
the name comes out empty, because every variable in it is empty, the pipeline gets no
name.
workflow: rules: variables sets variables when a rule
matches:
- They become default variables for every job, just like top-level
variables:, and they replace top-level variables of the same name. - A variable set in the job itself still wins.
- Trigger jobs pass them downstream, where they replace the downstream project's own
values of the same name. A generic name like
ENVIRONMENTset here can silently change another team's pipeline.
To stop that, give such variables unique names, or have the trigger job list what it
passes with inherit: variables.
What a newer commit cancels#
Push twice in quick succession and GitLab can cancel the older pipeline for you, but
only if the project setting Settings › CI/CD › General pipelines ›
Auto-cancel redundant pipelines is on. It applies only to a newer commit on the
same branch; selecting New pipeline again for the same commit cancels
nothing. With the setting off, interruptible does nothing.
workflow: auto_cancel: on_new_commit takes one of three values.
conservative, the default, cancels the older pipeline unless a job with
interruptible: false has already started; jobs that haven't started yet
count as interruptible. interruptible cancels only the jobs marked
interruptible: true, and the rest carry on. none cancels
nothing.
The companion key on_job_failure: all cancels every running job as
soon as one job fails, instead of letting its stage finish. Both keys can be overridden
per rule with workflow: rules: auto_cancel, for example so that pipelines on
protected branches are never cancelled.
Under the default, a single non-interruptible job decides for the whole pipeline. Once it starts, the pipeline can no longer be auto-cancelled. A downstream pipeline is cancelled along with its parent if none of its own non-interruptible jobs has started yet.
MegaCorp sets interruptible: true in default:, so every job
that inherits it can be cancelled by a newer push. That is fine for builds and tests.
Deployment jobs are different: GitLab advises against making them interruptible, because
cancelling one halfway can leave a partial deployment. When a template puts
interruptible: true in default:, check that deploy jobs set it
back to false.
When no pipeline appears#
Work down this list. It runs from the most common cause to the least:
| What you see | Likely cause | How to confirm |
|---|---|---|
| no pipeline at all after a push | workflow: rules matched nothing | run the rules by hand against the pipeline's variables (chapter 21's table) |
| no pipeline, and workflow would have allowed it | every job's rules left it out; a pipeline with no jobs isn't created | CI Lint, simulating the branch |
| no pipeline, although jobs should match | the only jobs left are in .pre or .post, which can't form a pipeline alone | check which stages the matching jobs are in |
| a pipeline marked Skipped, with no jobs | [ci skip] or [skip ci] in the commit message, or the ci.skip push option | the commit message |
| a failed pipeline with an error and no jobs | the configuration is invalid | the error on the pipeline page, or the pipeline editor |
| jobs or variables missing for no visible reason | a byte-order mark at the start of a YAML file, which the pipeline editor can't show | a hex viewer, or a tool that shows invisible characters |
| a merge request stuck on Checking pipeline status | Pipelines must succeed is on, and nothing lets a pipeline run for its latest commit | the merge request's Pipelines tab |
- Do this
- As soon as any job has a merge request rule, add a
workflowthat allows one kind of pipeline per push. Allow a merge request pipeline while a merge request is open, and a branch pipeline otherwise. - Why
- GitLab checks the workflow before it looks at any job, so it is the one place that decides how many pipelines a push gets.
- Watch out
- Without it, one push can create two pipelines, and every job runs twice. The card duplicate pipelines shows how that looks.
- Write a workflow that names which pushes get a pipeline, and know that it is asked first.
- Explain a push that produced no pipeline, and find the rule that refused it.
- Give each push exactly one pipeline, and set a variable for the whole pipeline from a workflow rule.
Spot the bugThe pipeline nobody can start#
A payments-api developer is working on feature/retry-fix, which has an
open merge request. They want a one-off branch pipeline with an extra variable, so they
open Build › Pipelines › New pipeline, choose their branch, add the
variable and submit. No pipeline is created. The same happens when a colleague uses the
API.
workflow:
name: "$CI_PIPELINE_SOURCE: $CI_COMMIT_REF_NAME"
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: $CI_COMMIT_BRANCH
- if: $CI_COMMIT_TAGShow the answer
Rule 2. A pipeline started on a branch from the UI or the API has
CI_COMMIT_BRANCH set, and CI_OPEN_MERGE_REQUESTS is set
because the branch has an open merge request. The rule doesn't check the source, so it
refuses the pipeline.
- Now: run a merge request pipeline instead. On the merge request's Pipelines tab, Run pipeline with modified values lets you set variables and inputs.
- For good: ask the template owners to narrow the rule to pushes, as
GitLab documents:
if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push".
Job rules: is this job in the pipeline?#
When GitLab creates a pipeline, it reads each job's rules from the top.
The first rule that matches decides the outcome:
- the job is added: normally, as a manual job, or delayed
- or the rule says
when: neverand the job is left out
If no rule matches, the job is left out. A matching rule can also change the job's
allow_failure, needs, variables and
interruptible.
A rule's conditions are if (variables), changes (files
changed) and exists (files present). A job with no rules at all gets an older
default: branches and tags, never merge requests.
docs-build has rules: changes: [docs/**/*]. You push a new branch with a code fix and no docs change, and the docs build runs anyway. On the next push to the same branch it stays away, as it should. What is different about the first push? (the card)
Four gates#
A job passes four gates before it runs, and GitLab checks each at a different moment:
| Gate | Checked | Can test | When it says no |
|---|---|---|---|
include: rules | while GitLab assembles the configuration | pre-pipeline variables, settings variables, and files | the included file's jobs never exist, and nothing in the pipeline mentions them |
workflow: rules | before any job is considered | variables that exist before any job, and files | there is no pipeline; after a push there is no error either |
job rules | once for each job while the pipeline is created, and once for each matrix job | those, plus pipeline variables and the job's own variables | the job is not in the pipeline |
when, needs, allow_failure | while the pipeline runs | the outcome of earlier jobs | the job is skipped, or waits for a person or a timer |
Chapter 22 covered the first two gates, and chapter 24 covers the last. This chapter is about the third.
First match wins#
A job's rules is a list, and GitLab reads it from the top, stopping at the
first rule whose conditions all hold:
- A rule needs at least one of
if,changes,existsorwhen. When a rule has several conditions, all of them must be true. - A rule with only
whenhas no conditions, so it matches whenever GitLab reaches it. It is the "otherwise" at the end of a list. - A matching rule with
when: neverleaves the job out. Any other matching rule adds it. - A matching rule without
whenuses the job's ownwhen, which defaults toon_success. Awhenwritten in the rule overrides the job's. - If GitLab reaches the end of the list without a match, the job is left out.
Order is everything. MegaCorp's two deploy jobs show both common styles:
deploy-staging:
extends: .deploy
needs: [image-build]
variables:
ENVIRONMENT: staging
rules:
- !reference [.rules, never-on-schedule]
- !reference [.rules, default-branch]
deploy-prod:
extends: .deploy
needs: [image-build]
variables:
ENVIRONMENT: production
rules:
- if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
when: manualdeploy-staging pastes two lists from the rules library with
!reference. GitLab flattens them into one list, in the order they are
written (chapter 14). Read it for four pipelines:
| Pipeline | Rule 1: never-on-schedule | Rule 2: default-branch | deploy-staging |
|---|---|---|---|
| the nightly schedule on main | matches, with when: never | not reached | left out |
| a push to main | no | matches | added, on_success |
| a merge request | no | no | left out |
| tag v1.4.0 | no | no | left out |
Swap the two rules and the nightly schedule matches default-branch
first, because a schedule on main sets CI_COMMIT_BRANCH.
Nothing in the list looks wrong; only its order is.
deploy-prod has a single rule, with when: manual. In a tag
pipeline whose tag looks like a release, it is added as a manual job. Everywhere else it
is left out. Because the when: manual is inside a rule, the job blocks its
pipeline until someone runs it; chapter 24 explains
why.
Writing if conditions#
An if is an expression over variables. Some forms use a regular
expression: a pattern for matching text, written between slashes. For example,
/^release/ means "starts with release". These forms cover almost everything
you will read:
| Expression | True when | Watch out for |
|---|---|---|
| $VAR | VAR is set and not empty | any non-empty value counts, including "false" and "0" |
| $VAR == "text" | the value is exactly text | case matters; the variable goes on the left, and only the string is quoted |
| $VAR != "text" | the value is anything else | also true when VAR isn't set at all |
| $VAR == null | VAR is not set | a variable set to "" is not null |
| $VAR == "" | VAR is set, and empty | |
| $A == $B | both have the same value | |
| $VAR =~ /re/ | the regular expression matches part of the value | unanchored: /release/ matches pre-release-notes; write /^release/ |
| $VAR =~ /re/i | the same, ignoring case | |
| $VAR !~ /re/ | the regular expression doesn't match | |
| $VAR =~ $PATTERN | the value matches the regular expression stored in PATTERN, slashes included | variables inside the stored expression are not expanded |
| a && b, a || b | both, or either | && binds tighter than ||; add parentheses to be sure |
| !$VAR, !(a == b) | the opposite; since 18.11 | ! tests for empty or unset, not for "false" |
The surprising rows were checked on the local GitLab 19.3:
"false"counts as true.!=is true for a variable that isn't set.- An empty variable is not
null. &&binds before||.
The same test turned up one more trap. Put a negated expression in quotes:
if: '!($VAR == "x")'. Without the quotes, GitLab rejects it with
invalid expression syntax, even though GitLab's own documentation shows it
unquoted.
Four syntax traps make GitLab reject a configuration, or accept it and mean something else:
- Quote strings, never variables.
$ENV == production(unquoted string),"$ENV" == "production"(quoted variable) and${ENV} == "production"(braces) are all invalid, and GitLab refuses the whole configuration. - A regular expression is one whole
/…/.issue-/.*/doesn't work;/issue-.*/does. A pattern of a single character, such as/./, is rejected asinvalid expression syntax. Matching uses RE2 syntax. - A right-hand side without slashes is not a regular expression.
$A =~ "text"is accepted, but tests whether A's value appears insidetext:"23" =~ "1234"is true. Always write the slashes. - Values are not expanded twice. If a variable's value is
$OTHER, the condition sees that literal text, not OTHER's value. This holds for matrix values too.
A condition can only test a variable that exists when the pipeline is created. If a rule never matches, check the variable against chapter 21's list: job-only variables, persisted variables and dotenv variables are never there.
changes and exists#
Two conditions test files instead of variables:
| Condition | True when | Watch out for |
|---|---|---|
changes: [paths] | a changed file matches one of the paths | "changed" depends on the kind of pipeline; see the next table |
changes: paths with compare_to | a file differs from the branch, tag or commit named | in a merged results pipeline, the comparison also includes the target branch's own changes |
changes: regexp | a changed path matches a Ruby regular expression; since 19.2 | anchor it with \A and \z, not ^ and $ |
exists: [paths] | a file or directory matching one of the paths is in the repository | a directory needs a trailing slash; artifacts are never seen |
exists: paths with project and ref | the file exists in another project, at a ref |
"Changed" is the trap. What a file is compared against depends on the kind of pipeline:
| Pipeline | changes compares with | So it is true |
|---|---|---|
| merge request | the target branch | when the merge request as a whole touches the files |
| push to an existing branch | the branch's previous commit | when this push touches the files |
| the first push of a new branch, or a new tag | nothing | always |
| scheduled, New pipeline, API or trigger: anything without a push | nothing | always |
MegaCorp's monorepo shows the third row. Its docs trigger job has a rule
with changes: [docs/**/*] and nothing else. When the local GitLab received
mono's very first push to main, the pipeline included docs,
although the repository had no docs/ directory at all.
GitLab's advice follows from the table: pair changes with an
if that limits it to merge request or branch pipelines, or give it a
compare_to.
Four smaller rules about paths:
- Don't write
./or//. Paths are compared as plain text, not resolved like a shell path. - A variable ending in a slash turns
$DIR/*intodir//*. - Root-level wildcards need quotes in YAML:
"*.json". - Each list holds at most 50 paths. After 50,000 checks a pattern counts as matched, so
in a huge change or repository every
changesorexistsrule is true.
One more difference: in include: rules, exists looks in the
project and ref of the file that contains the include. In a job, it looks in
the project running the pipeline. A template that works in its own repository can test
the wrong project once someone else includes it.
What a matching rule can change#
| Key in the rule | Effect, only when this rule is the one that matched |
|---|---|
when | on_success, on_failure, always, manual or delayed add the job; never leaves it out |
allow_failure | replaces the job's value; the default is false, even with when: manual |
needs | replaces the job's entire needs list; [] makes it start at once |
variables | adds variables, or overrides the job's, for this case only |
interruptible | replaces the job's value |
when: manual means two different things depending on where it is
written. On the job itself, it makes an optional manual job, with
allow_failure: true. Inside a rule, it makes a blocking one, with
allow_failure: false: the pipeline waits at that stage until someone runs
it. Add allow_failure: true to the rule to get the optional kind.Rules and matrix jobs#
With parallel: matrix, GitLab evaluates the rules separately for each
generated job, using that job's values. In MegaCorp's matrix, a rule
if: $JDK == "17" would keep maven-test: [17] and drop
maven-test: [21]. Matrix values can also appear in changes and
exists paths, which lets a monorepo run each service's job only when that
service's files change.
Legacyonly and except
Older templates control jobs with only and except, which
GitLab has deprecated; use rules instead. They list branch names, regular
expressions and kinds of pipeline, such as branches, tags,
merge_requests, schedules, pushes,
web, api, triggers and
pipelines.
- A job with no
only,exceptorrulesbehaves asonly: [branches, tags]. That is why such a job never appears in merge request pipelines. only: branchesmatches scheduled pipelines too, because a schedule runs on a branch.except: schedulesremoves them.- Mixing
only/exceptjobs withrulesjobs is a classic cause of duplicate pipelines (chapter 22).
To translate a job to rules, write one if rule for each
only entry. For each except entry, write one
when: never rule, then finish the list with
- when: on_success.
Why was this job left out?#
- Name the pipeline. Its label and source tell you which variables exist (chapter 21).
- Get the job's real rules from the Full configuration view.
extendsmay have replaced the rules you expected (the card), and every!referenceis already expanded there. - Walk the list from the top with those variables, and stop at the first match.
- For each variable a rule tests, check that it exists when the pipeline is created, and find where its value is set (chapter 26).
- For each
changes, check which comparison applies in this kind of pipeline. - If the job is in the pipeline but never ran, you have moved on to the fourth gate: chapter 24.
- Do this
- Order a job's rules from the most specific case to the most general, and end with the case you want by default.
- Why
- GitLab stops at the first rule that matches. If a general rule comes first, the specific ones below it are never reached.
- Watch out
- A job with rules, none of which match, is left out of the pipeline without a
message. And a
changesrule is always true on the first push of a new branch. The card runs on a new branch explains why.
- Read a job's rules top to bottom and say whether the job is in a given pipeline, and how it will run.
- Order rules so that the specific cases come first and the default comes last.
- Explain the new-branch surprise of changes, and the silence of a job with no matching rule.
Spot the bugThe smoke test that runs every night#
A team adds a smoke test that should run after each merge to main,
using MegaCorp's rules library:
smoke-test-staging:
stage: deploy
script:
- ./scripts/smoke-test.sh staging
rules:
- !reference [.rules, default-branch]
- !reference [.rules, never-on-schedule]It does run after merges. It also runs at 02:00 every night, in the scheduled
pipeline on main, against a staging environment nobody has changed.
never-on-schedule. Why doesn't it stop the
nightly run?Show the answer
First match wins. A schedule on main sets CI_COMMIT_BRANCH
to main, so default-branch matches first and adds the job.
GitLab never reaches never-on-schedule.
Put the exclusion first, as MegaCorp's deploy-staging does:
rules:
- !reference [.rules, never-on-schedule]
- !reference [.rules, default-branch]See first match wins.
Order and gates: when a job starts, and what stops it#
A job that made it into the pipeline still has to wait its turn:
- Stages run in order. A job starts when every job in the earlier stages has succeeded, been allowed to fail, or been left waiting as an optional manual job.
needsreplaces that barrier with a list of jobs. The job starts when those finish, whatever else is still running.whendecides the rest:on_success(the default),on_failure,always,manualordelayed.
Then come the gates:
allow_failurestops a failure from counting.- A manual job added by a rule holds the pipeline until someone runs it.
resource_groupmakes jobs wait their turn, across pipelines.retry,timeoutandinterruptibledecide what happens when things go wrong.
The pipeline on main says blocked. Every job is green except approve-release, which a rule added with when: manual. Deploy is waiting for it; so is everyone. Yesterday the same job was there and the pipeline passed without anyone pressing anything. What changed the meaning of the button? (the card)
Stages: the default barrier#
Without needs, a stage starts only when the stage before it has
finished. For that barrier, three outcomes count as finished well:
- success
- a failure of a job marked
allow_failure: true - an optional manual job that nobody has started, which counts as skipped
If any other job fails, the pipeline is marked failed and later stages never start. Jobs already running in the failing stage carry on to the end.
The stage list has fixed ends. .pre runs first and .post
last, and neither needs listing. A pipeline execution policy can add
.pipeline-policy-pre before .pre and
.pipeline-policy-post after .post
(chapter 36).
Three smaller rules explain some odd-looking graphs:
- A job with no
stagelands intest. - A listed stage that no job uses is not shown at all.
- A pipeline whose only jobs are in
.preor.postis not created.
needs: a graph instead of a barrier#
A job with needs ignores the stage barrier. It starts as soon as every
job it lists has finished, even while other jobs in earlier stages are still running.
MegaCorp's golden pipeline uses it three times: image-build needs
maven-build, and both container_scanning and
deploy-staging need image-build. Here is what that does to a
pipeline on main:
The pipeline is much shorter, and one bar is in the wrong place.
deploy-staging lists only image-build. So it starts as soon as
the image exists, while maven-test and container_scanning may
still be running. A test that fails afterwards turns the pipeline red, but staging
already has the build. Nobody wrote "deploy without testing"; the missing entries in a
needs list say it for them.
The details that matter when you read a needs list:
needs: []starts the job as soon as the pipeline is created, as.prejobs do. It suits linters and scanners that read only the source.- A job can need jobs in its own stage. A job can list at most 50 jobs.
- Needing a matrix job means needing all of it.
needs: [maven-test]waits for everymaven-test: […]job.needs: parallel: matrixpicks particular ones. - Artifacts follow
needs. A job withneedsdownloads artifacts only from the jobs it lists.artifacts: falseon an entry downloads nothing from that job. Don't combineneedswithdependencies. - Seeing the graph: on the pipeline page, select Job dependencies. The default view groups jobs by stage and hides the lines.
needs and rules collide when a needed job is left out of a
pipeline by its own rules. GitLab then refuses to create the pipeline at all, with an
error naming both jobs. There are two cures:
- Give the needing job rules that keep it out whenever the needed job is out.
- Mark the entry
optional: true. The job then waits for the needed job when it exists, and ignores it when it doesn't. If every entry is optional and none of those jobs exists, the job starts at once, as if it hadneeds: [].
Here is the error from the local GitLab 19.3, for a unit-tests
job that needs a compile job its rules had left out:
'unit-tests' job needs 'compile' job, but 'compile' does not exist in the pipeline. This might be because of the only, except, or rules keywords. To need a job that sometimes does not exist in the pipeline, use needs:optional.Three other keys are spelt needs but do different jobs:
| Key | What it does | Watch out for |
|---|---|---|
needs: project | downloads artifacts from the latest successful run of a job in another project, at a ref; Premium | it doesn't wait: if that project's pipeline is still running, you get the previous run's artifacts |
needs: pipeline with job | lets a child pipeline download artifacts from a job in its parent or a sibling | the job must have succeeded |
needs: pipeline alone | copies the latest status of another project's default-branch pipeline into this job | it is a status mirror, not a dependency |
when: the job's own condition#
| when | The job runs |
|---|---|
on_success (the default) | when every job in earlier stages succeeded, was allowed to fail, or is an unstarted manual job |
on_failure | only when at least one job in an earlier stage failed; for cleanup and notifications |
always | whatever happened earlier |
manual | when a person starts it |
delayed | after the time in start_in |
never | never; allowed only in rules and workflow: rules |
One consequence surprises people: a failed job with
allow_failure: true counts as a success. An on_failure
notification job therefore stays silent when the only failure was allowed.
allow_failure: failures that don't count#
A job with allow_failure: true can fail without failing the pipeline.
The pipeline passes, and the job carries an orange warning. The default depends on how
the job was made manual:
The default for allow_failure depends on where when: manual
was written. On the job itself, the default is true: an optional manual job.
From a rule, the default is false: a blocking manual job. For anything else
it is false.
allow_failure: exit_codes: [137, 255] allows a failure only for those
exit codes. Any other failure still counts.
Scanners are the usual place to find it. In MegaCorp's pipeline on main,
the local GitLab created container_scanning, secret_detection
and sonar-scan all with allow_failure: true. That comes from
GitLab's scanner templates and from the component. A scanner that crashes, or finds
something, doesn't turn the pipeline red unless a policy makes it
(chapter 36).
Manual and delayed jobs#
A manual job is shown as skipped until someone runs it, and there are two kinds:
An optional manual job, with allow_failure: true, lets the pipeline carry
on: it can pass without the job ever running. A blocking one, with
allow_failure: false, stops the pipeline at the job's stage with the status
blocked. Later stages wait, and a merge request set to "Pipelines must succeed"
won't merge a blocked pipeline.
Running a manual job needs permission to merge to the pipeline's branch. On Premium,
a protected environment narrows that to a named list of people. If you are not on it,
the job shows You are not authorized to run this manual job.
manual_confirmation adds an "are you sure?" message. Variables typed in when
starting a manual job are visible to other members and are not masked.
A release of payments-api passes through two manual gates. deploy-prod is
manual by a rule, so the tag pipeline is blocked until someone runs it:
As text
- build: maven-build
- package: image-build
- deploy: deploy-prod (manual)
Needs: image-build waits only for maven-build; deploy-prod waits only for image-build.
Downstream: deploy-prod triggers megacorp/platform/deployer (multi-project pipeline).
The deployer then asks again. Its deploy job is manual whenever
ENVIRONMENT is production:
variables:
ENVIRONMENT: $[[ inputs.environment ]]
deploy:
stage: deploy
image: registry.example.com/megacorp/devops/ci-tools:3.2
resource_group: $APP-$ENVIRONMENT
environment:
name: $ENVIRONMENT/$APP
script:
- mc deploy "$ENVIRONMENT" "$IMAGE_REF"
rules:
- if: $IMAGE_REF == null
when: never
- if: $ENVIRONMENT == "production"
when: manual
- when: on_successSo a production release takes two clicks in two projects. Between them,
deploy-prod uses strategy: depend, and with it the trigger job
shows running for as long as the deployer waits for its manual job.
"The deploy has been running for three hours" usually means "nobody has pressed the
second button". strategy: mirror would copy the deployer's own status
instead.
A delayed job waits start_in, anything from one second
to one week, counted from when its previous stage completes. Its stage doesn't finish
until it has run. A bare number needs quotes ('30' means 30 seconds). To
start one early, select Unschedule, then Run.
resource_group: one at a time#
The deployer's deploy job carries
resource_group: $APP-$ENVIRONMENT. Jobs with the same resource group never
run at the same time, even in different pipelines of the same project. A second
payments-api staging deploy waits for the first, showing
Waiting for resource: payments-api-staging and the status
waiting_for_resource. Deploys of other services, with other names, go
ahead.
The order in which waiting jobs are released is the group's process mode, and it isn't in any YAML file. It is set through the API:
| Process mode | When the resource frees up, GitLab starts |
|---|---|
unordered (the default) | whichever waiting job is ready |
oldest_first | the job from the oldest pipeline |
newest_first | the job from the newest pipeline, so older deploys are skipped over |
newest_ready_first | the newest job that is ready to run |
With oldest_first, a parent pipeline and its child that use the same
resource group can deadlock, each waiting for the other. The documented fix is to put
the resource group on the trigger job instead.
When things go wrong: retry, timeout, interruptible#
retry reruns a failed job up to twice. With
when, it retries only certain kinds of failure; with
exit_codes, only certain exit codes. MegaCorp sets it for every job in
default::
default:
image: registry.example.com/megacorp/devops/ci-tools:3.2
tags: [megacorp-shared]
interruptible: true
retry:
max: 2
when: [runner_system_failure, stuck_or_timeout_failure]
before_script:
- !reference [.snippets, functions]
- mc_log "job $CI_JOB_NAME on $CI_COMMIT_REF_NAME"That configuration has aged twice without anyone touching it:
stuck_or_timeout_failurewas deprecated in 19.1, in favour of more precise reasons. The local GitLab 19.3 accepts it, but warns on every lint:retry uses deprecated `when` value(s): stuck_or_timeout_failure. These match the more specific failure reasons that replaced them; migrate to those reasons.- 19.1 also moved some failures out of
runner_system_failure, into the newrunner_external_dependency_failure(such as a registry that couldn't be reached) andrunner_interrupted. A job that used to be retried after a network blip may now fail at once.
timeout sets how long the job may run, in words:
90 minutes, 3h 30m. It can be longer than the project's
timeout, but never longer than the runner's. It can't be set in
default:. Jobs that stop making progress are dropped, with a failure reason
that says why:
| The job was | Dropped after | Failure reason |
|---|---|---|
| pending, with a runner that could take it | 24 hours | stuck_pending_with_matching_runners |
| pending, with no runner that could take it | 1 hour | stuck_pending_no_matching_runners |
| running, with no updates from the runner | 30 minutes | no_updates_running |
| running past its timeout | the timeout plus 15 minutes | server_timeout_running |
interruptible decides whether a newer commit may cancel
the job; chapter 22 covers it.
- Do this
- Use
needsto start a job as soon as the jobs it depends on have finished, and list every job it takes artifacts from. - Why
- Without
needs, a job waits for the whole previous stage, even for jobs it never uses. - Watch out
- A manual job that a rule adds is not allowed to fail, so the pipeline shows as
blocked until someone runs it. If the job is optional, add
allow_failure: trueto that rule. The card a play button that waits shows the symptom.
- Say when a job starts: after its stage, or as soon as its needs are done.
- Tell a manual job that blocks from one that doesn't, and choose which you want.
- Read a stuck or blocked pipeline and name the gate that holds it.
Spot the bugThe nightly pipeline that never starts#
An earlier version of MegaCorp's security.yml gave the container scan
these rules:
container_scanning:
stage: scan
needs: [image-build]
rules:
- if: $CONTAINER_SCANNING_DISABLED == "true"
when: never
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHWhen the payments team added a nightly schedule on main, every nightly
run failed before a single job started. Pushes to main and merge requests
were fine.
Show the answer
In a schedule on main, CI_COMMIT_BRANCH is
main, so the last rule adds container_scanning. But
image-build inherits .megacorp-base's rules, which start with
never-on-schedule, so it is left out. The pipeline would contain a job that
needs a job that doesn't exist, so GitLab refuses to create it.
- Keep the scan out of schedules too, by putting
- !reference [.rules, never-on-schedule]before the other rules. That is what MegaCorp's currentsecurity.ymldoes. - Or make the need
optional: true, if a scan without a fresh image makes sense.
Truth tables: predicting which jobs run#
A truth table lists every job down the side and every kind of pipeline your project runs across the top. Each cell says what the job does in that pipeline: runs, waits for a person, is allowed to fail, or is absent.
You derive it from the merged configuration by applying the earlier chapters of Part V in order. It answers "why didn't this run?" at a glance. It also finds the jobs that run where nobody meant them to, and the checks that never run where everybody assumes they do.
The nightly schedule is about to be switched on for payments-api, and the release manager asks which jobs it will run. Eleven jobs, three rule lists, a workflow, and no way to try it without running it. You need the answer on paper, before tonight. (the card)
The method#
- Pick the scenarios: the kinds of pipeline the project really gets (chapter 21).
- Apply workflow. Cross out the scenarios it refuses, and note when a scenario stops applying (chapter 22).
- Find each job's final rules in the Full configuration view, after
extends,!reference, templates and components have done their work (chapter 14). - Walk each job's rules for each scenario. The first match fills the cell (chapter 23).
- Mark how the job runs: manual, delayed, or allowed to fail (chapter 24).
- Check every column's
needs. Each needed job must be in the same column, or GitLab won't create that pipeline. - Check the table against real pipelines of each kind.
Here is the whole method, applied to payments-api.
Steps 1 and 2: scenarios and workflow#
| Scenario | Source | What the rules will see | MegaCorp's workflow |
|---|---|---|---|
| feature branch push | push | CI_COMMIT_BRANCH is the feature branch | allowed by rule 3, but only until a merge request is opened; after that, rule 2 refuses it |
| merge request | merge_request_event | CI_MERGE_REQUEST_* is set, and CI_COMMIT_BRANCH is not | allowed by rule 1 |
| main | push | CI_COMMIT_BRANCH equals CI_DEFAULT_BRANCH | allowed by rule 3 |
| tag v1.4.0 | push | CI_COMMIT_TAG is set, and CI_COMMIT_BRANCH is not | allowed by rule 4 |
| nightly on main | schedule | CI_COMMIT_BRANCH is main, and the source is schedule | allowed by rule 3 |
All five scenarios get a pipeline, provided at least one job is added to it.
Step 3: each job's final rules#
Most of payments-api's jobs inherit their rules from the base template:
.megacorp-base:
variables:
MC_TEAM: unknown
artifacts:
expire_in: 7 days
rules:
- !reference [.rules, never-on-schedule]
- !reference [.rules, mr]
- !reference [.rules, default-branch]
- !reference [.rules, release-tag]Not all of them, though. Here is where each job's final list comes from:
| Job | Its rules come from | The list, in order |
|---|---|---|
maven-build | .megacorp-base, through .maven-base and .maven-build | never-on-schedule, mr, default-branch, release-tag |
maven-test | .maven-test, whose own list replaces the base's | never-on-schedule, mr, default-branch |
image-build | .megacorp-base, through .image-build | never-on-schedule, mr, default-branch, release-tag |
container_scanning | MegaCorp's security.yml, whose list replaces the one in GitLab's template | disabled means never; never-on-schedule; merge request event; default branch |
secret_detection | GitLab's Secret-Detection template | disabled means never; two rules that apply only when AST_ENABLE_MR_PIPELINES is "true"; $CI_COMMIT_BRANCH |
sonar-scan | the sonar-scan component | SKIP_SONAR means never; merge request event; default branch |
deploy-staging | java-service.yml | never-on-schedule, default-branch |
deploy-prod | java-service.yml | a tag matching the release pattern, as a manual job |
publish-docs | payments-api's own file | default-branch |
Two of these lists replaced a list the job would otherwise have inherited: the one on
maven-test, and the one on container_scanning. Replaced lists
are where most surprises in a truth table come from. For
secret_detection, the rules are in a template nobody at MegaCorp wrote. The
only way to see them is the Full configuration view.
Steps 4 and 5: the table#
Walk each list for each scenario, stopping at the first match:
| job | feature branch push | merge request | main | tag v1.4.0 | nightly on main |
|---|---|---|---|---|---|
| maven-build | not in the pipeline | runs | runs | runs | not in the pipeline |
| maven-test: [17] and [21] | not in the pipeline | runs | runs | not in the pipeline | not in the pipeline |
| secret_detection | runs, allowed to fail | not in the pipeline | runs, allowed to fail | not in the pipeline | runs, allowed to fail |
| sonar-scan | not in the pipeline | runs, allowed to fail | runs, allowed to fail | not in the pipeline | runs, allowed to fail |
| image-build | not in the pipeline | runs | runs | runs | not in the pipeline |
| container_scanning | not in the pipeline | runs, allowed to fail | runs, allowed to fail | not in the pipeline | not in the pipeline |
| deploy-staging | not in the pipeline | not in the pipeline | runs | not in the pipeline | not in the pipeline |
| deploy-prod | not in the pipeline | not in the pipeline | not in the pipeline | manual: waits for someone to run it | not in the pipeline |
| publish-docs | not in the pipeline | not in the pipeline | runs | not in the pipeline | runs |
Two columns take a moment's thought:
- Merge request,
secret_detection. None of its four rules matches. The two merge request rules needAST_ENABLE_MR_PIPELINES, which nobody set, and the last rule needsCI_COMMIT_BRANCH, which a merge request pipeline doesn't have. - Nightly,
sonar-scan. Its last rule is$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH, a schedule onmainmatches it, and nothing in its list excludes schedules.
Step 6: needs in every column#
| Job | Needs | In the same columns? |
|---|---|---|
image-build | maven-build | yes: merge request, main, tag |
container_scanning | image-build | yes: merge request, main |
deploy-staging | image-build | yes: main |
deploy-prod | image-build | yes: tag |
The nightly column is safe only because container_scanning leaves
schedules out, just as image-build does. Before MegaCorp added that rule, the
nightly column held a scan that needed an image the column didn't have, and GitLab
refused to create the pipeline at all
(chapter 24's exercise).
Step 7: check it against GitLab#
The table above was checked cell by cell against pipelines that a local GitLab 19.3 created for the same five scenarios. You can do the same with less effort:
- Pushes to a branch or tag can be simulated before you push. GitLab's CI Lint runs the rules as if a pipeline were being created (chapter 4).
- Merge request and scheduled pipelines can't be simulated. Find a recent one of each kind in the pipeline list, by its label, and compare its jobs with your column.
- If you use the API, remember that it lists trigger jobs separately
from other jobs. A pipeline's jobs endpoint doesn't show
deploy-stagingordeploy-prod; its bridges endpoint does.
What the table says about payments-api#
Five findings, none of them visible in any single file:
- Merge requests get no secret detection. Once a merge request is
open, MegaCorp's workflow refuses branch pipelines, and GitLab's template skips merge
request pipelines. A secret committed on a merge request branch is first scanned after
the merge. GitLab's documented switch is
AST_ENABLE_MR_PIPELINES: "true", typically set once for the whole group. - A feature branch with no merge request gets only secret detection. No build and no tests run until someone opens the merge request.
- A release tag is built but not tested or scanned. It relies on the
tag sitting on a commit that
main's pipeline already tested. - The nightly pipeline doesn't build or test anything. It republishes the docs and runs two scanners.
- Every "allowed to fail" cell is a scanner. A finding, or a crashed scanner, never turns these pipelines red; only a policy can make it (chapter 36).
- Build a truth table for a pipeline: every job against every kind of pipeline, from the rules alone.
- Check the table against what GitLab actually created, and find the rule that disagrees.
- Answer "what will run if" without pushing anything.
Spot the bugThe release candidate with no pipeline#
The payments team starts tagging release candidates. They push the tag
v1.5.0-rc1 on a commit of main. No pipeline appears, and GitLab
shows no error. The next day, v1.5.0 on the same commit gets a pipeline as
usual.
v1.5.0-rc1 to the table. Why is there no
pipeline?Show the answer
Workflow rule 4 lets the tag pipeline through, but every job's rules then leave it out:
maven-buildandimage-buildreachrelease-tag, whose pattern/^v\d+\.\d+\.\d+$/doesn't match-rc1.maven-test,container_scanning,sonar-scan,deploy-stagingandpublish-docshave no rule for tags at all.secret_detectionneedsCI_COMMIT_BRANCH, which a tag pipeline doesn't have.deploy-prodrepeats the release pattern.
The column is empty, and GitLab doesn't create a pipeline with no jobs. After a push it
says nothing. To build release candidates, widen the pattern, for example to
/^v\d+\.\d+\.\d+(-rc\d+)?$/, in two places: the rules library, and
deploy-prod, which copies the pattern instead of using the library. See
each job's final rules and
when no pipeline appears.
Variables and precedence#
A CI/CD variable is a name and a value that GitLab hands to each job as an environment variable. Variables come from many places:
- your YAML
- the settings pages of the project, its groups and the instance
- whoever started the pipeline, and earlier jobs
- security policies, and GitLab itself
When one name is set in several places, a fixed ladder decides which value wins. Settings beat YAML, and a value given when the pipeline started beats settings. Your YAML is near the bottom. Most "my change had no effect" problems are this ladder at work.
You set LOG_LEVEL: debug at the top of the file to see more output. Three jobs print debug. Two print info, as before. Nothing in those two jobs mentions LOG_LEVEL, but the template they extend does. Which value wins, and by what rule? (the card)
What a variable is#
Here is the smallest example:
variables:
DEPLOY_ENV: staging
show-env:
script:
- echo "Deploying to $DEPLOY_ENV" # prints: Deploying to stagingGitLab passes DEPLOY_ENV to the job as an environment variable: a
named value that every program the script runs can read. In a script you write it as
$DEPLOY_ENV. Variables can also appear in many keywords, such as
image, and in rules, although rules can only see the variables that exist
when the pipeline is created (chapter 21).
Variables in YAML#
A variables: block can sit at the top of a file or inside a job:
- At the top, it sets a default for every job.
- In a job, it sets a value for that job only, and a job's value beats a default of the same name.
Everyone who can read the repository can read these values, so YAML is for ordinary
settings, never for passwords or tokens. variables: {} in a job turns every
default off for that job, and inherit: variables: does the same more
selectively (chapter 14).
MY_VAR: $MY_VAR in a job doesn't pick up the top-level
MY_VAR. The job's own definition hides the default, so the value is the
literal text $MY_VAR. Use a different name.Variables in settings#
Anything secret belongs in settings, under Settings › CI/CD › Variables, on the project, on a group, or for the whole instance. Every job below that level receives the variable. Each one has a few options:
| Option | What it does |
|---|---|
| Key | the name: letters, digits and underscores only |
| Value | up to 10,000 characters |
| Type | Variable, the usual kind, or File: the value is written to a temporary file, and the variable holds the file's path, for tools that need a file |
| Environment scope | limits the variable to jobs deploying to certain environments (Premium, on groups) |
| Protect variable | the variable exists only in pipelines on protected branches and tags |
| Visibility | Visible; Masked, which hides the value in job logs and has been the default since 18.3; or Masked and hidden, which also hides it in the settings page, for ever |
| Expand variable reference | lets the value refer to other variables with $; off by default since 18.6 |
Here are MegaCorp's group variables. Every project under megacorp/ gets
them:
# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
- key: MAVEN_CLI_OPTS
value: "--batch-mode --errors --show-version -s .m2/settings.xml"
- key: AWS_ACCOUNT_ID
value: "123456789012"
- key: MC_AZURE_TENANT_ID
value: "00000000-0000-0000-0000-000000000000"
- key: MC_ACR_REGISTRY
value: megacorpprod.azurecr.io
- key: MC_ACR_PUSH_CLIENT_ID
value: "00000000-0000-0000-0000-000000000000"
# the gitlab-acr-push application in Entra ID, which mc_acr_push signs in as
- key: SONAR_HOST_URL
value: https://sonar.example.com
- key: SONAR_TOKEN
value: "(set in the UI)"
masked: true
protected: trueChanging a project variable needs the Maintainer role, a group variable the Owner role, and an instance variable an administrator. If you can't see a level's settings, you can't see its variables either. That is the first reason variables are hard to trace.
Who wins: the precedence ladder#
When the same name is set in more than one place, the job gets the value from the highest place on this ladder:
| Rung | Where the value comes from | Beats |
|---|---|---|
| 1 | a pipeline execution policy | everything |
| 2 | a scan execution policy | everything below |
| 3 | pipeline variables: a manual run, a schedule, a trigger, the API, an upstream pipeline, or a manual job's form | all settings and all YAML |
| 4 | the project's CI/CD settings | groups, instance, and all YAML |
| 5 | a group's CI/CD settings; the closest subgroup wins | the instance, and all YAML |
| 6 | the instance's CI/CD settings | all YAML |
| 7 | a dotenv report written by an earlier job | your YAML |
| 8 | a job's own variables:, including those it gets through extends | top-level YAML |
| 9 | top-level variables: | GitLab's own values |
| 10–11 | deployment and predefined variables | nothing |
Two of MegaCorp's variables show the ladder at work:
MC_TEAMis set at the top of payments-api's file (rung 9), and in.megacorp-base, a job that every template extends (rung 8). The job value wins, so every job reportsunknown(chapter 14).MAVEN_CLI_OPTSis set in payments-api'smaven-testjob (rung 8), and on the megacorp group (rung 5). The group wins. The option payments-api added is never used, and nothing in any file shows why.
maven-test:
variables:
MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"Values given when the pipeline starts#
Rung 3 holds pipeline variables: values given to one pipeline when it starts. They come from the New pipeline form, a schedule, the API, a trigger token, an upstream trigger job, or the form on a manual job. They beat every settings page, and even GitLab's predefined variables, so they are powerful and easy to misuse.
Because of that, a project can limit who may use them, with Minimum role to
use pipeline variables. On self-managed GitLab the default is Maintainer; new
projects on GitLab.com default to nobody. Someone below the minimum gets
Insufficient permissions to set pipeline variables. The setting also covers
the variables a trigger job sends downstream, including top-level ones it inherits. So a
downstream project with the restriction on can refuse your trigger with:
Failed - (downstream pipeline can not be created, Insufficient permissions to set pipeline variables)GitLab now recommends pipeline inputs instead of pipeline variables (chapter 16), because inputs are typed and checked.
When a $ is expanded#
A value can mention another variable. What happens depends on where the value was set:
- In YAML, references are expanded:
LS_CMD: 'ls "$FLAGS"'uses the value ofFLAGS. Thevariables: expandkeyword can turn that off for one variable. - In settings, a value is taken literally unless its Expand variable reference box is ticked. That box has been off by default since 18.6, and a masked variable can't have it.
- To keep a literal dollar sign in YAML, write
$$.
Inside the job#
Every variable reaches the job as an environment variable. Three details trip people up:
before_scriptandscriptshare one shell;after_scriptdoesn't. A value set withexportin the script is gone byafter_script, and never reaches another job.- To pass a value to a later job, write it to a dotenv report: a small
file of
NAME=valuelines that later jobs receive as variables (rung 7; chapter 31). Rules can't see such values. - Service containers, the helper containers a job starts beside itself, get only the variables written in YAML, not those from settings (chapter 30).
Finding where a value came from#
A job log never lists where each variable came from. Work down the ladder, from the top, and stop at the first place that sets the name:
As text
- Does a security policy set it? Yes: The policy's value wins; ask the security team. No: the next step.
- Was it given when this pipeline started: by hand, a schedule, a trigger or the API? Yes: That value wins over every settings page. No: the next step.
- Is it in the CI/CD settings of the project, a group or the instance? Yes: The closest level wins: project, then the nearest group. No: the next step.
- Did an earlier job write it to a dotenv report? Yes: The dotenv value wins over your YAML. No: the next step.
- Is it in the job's own variables, or in a job it extends? Yes: The job's value wins over the top-level one. No: the next step.
- Otherwise it comes from top-level variables, or from GitLab itself
To look at a value without leaking it, print the variable's name and length,
never a secret's value. Masked values appear in logs as [MASKED]. A command
such as export lists every variable the job has, and GitLab's debug tracing
lists even more (chapter 27).
- Do this
- Put values that are not secret in the YAML, beside the jobs that use them, and put
them in quotes, for example
JAVA_VERSION: "21". Put secrets in the settings, never in the YAML. - Why
- Reviewers see a value in the YAML, and quotes stop YAML from reading a number or a
word such as
yesas something else. - Watch out
- If a change to a YAML value seems to be ignored, look for a settings variable with the same name. The card the old value wins shows what happened.
- Recite the ladder: which source of a variable's value beats which, from a policy down to the predefined ones.
- Find where a job's value really came from, in the YAML or on a settings page.
- Write a variable in the right place for what it is: a plain value, a choice, or a secret.
Spot the bugThe retry option that never retries#
payments-api's tests are flaky, so the team told Maven to rerun failing tests twice, in their own file:
maven-test:
variables:
MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"The job log shows no reruns at all. The merged configuration shows their
MAVEN_CLI_OPTS exactly as they wrote it.
-Dsurefire.rerunFailingTestsCount=2, and
what are the ways to fix it?Show the answer
The megacorp group sets MAVEN_CLI_OPTS in its CI/CD settings (rung 5).
Settings variables beat every value in YAML, including a job's own (rung 8). The merged
configuration shows only YAML, so it can't show the override.
Either give the option another name that the template adds to the Maven command, or
set a project variable MAVEN_CLI_OPTS (rung 4) containing the group's options
plus the new one. The cleanest fix is for the group owners to stop setting a value that
projects are meant to adjust. See who wins.
Secrets#
A secret is a value that must not leak: a password, a token, a private key. There are three ways to get one into a job:
- a CI/CD variable set in settings, which can be protected, masked or hidden
- the
secrets:keyword, which fetches the value from a secrets manager as the job starts (Premium) - the job logging in to a secrets manager or cloud itself, with an ID token (chapter 28)
Each has limits. Most "the secret is empty" problems come from protection rules deciding that this pipeline doesn't get the value. Most leaks come from a script printing the value in a form that masking doesn't recognise.
The deploy job signs in to the registry on main and fails with unauthorized on every feature branch and in every merge request. The password is a settings variable, marked protected, as the security team asked. The job is the same. The branches are not. (the card)
Three ways a secret reaches a job#
| Way | Where the value lives | How the job gets it | Tier |
|---|---|---|---|
| A CI/CD variable | the settings of the project, a group or the instance | as an environment variable, or a file if its type is File | all |
The secrets: keyword | a secrets manager: HashiCorp Vault, AWS, Google Cloud or Azure, or GitLab's own Secrets Manager | GitLab fetches it as the job starts, and hands it over as a file by default | Premium |
| The job logs in itself | anywhere that trusts GitLab's ID tokens | the script exchanges an ID token for access, then reads what it needs | all |
Settings variables are the commonest, and the easiest to get wrong, so most of this
chapter is about them. Never put a secret in .gitlab-ci.yml: everyone who
can read the repository can read it.
Protect, mask, hide: what each one does#
A settings variable has three safety options. They protect against different things, and none of them protects against everything:
| Option | What it does | What it doesn't do |
|---|---|---|
| Protect | the value is delivered only to pipelines running on protected branches and tags | stop code in those pipelines from using or sending the value |
| Mask | the exact value is replaced by [MASKED] in job logs | catch the value if a program prints it changed, for example with a backslash added before a special character |
| Mask and hide | as masked, and nobody can read the value in the settings page again | anything more at run time; it can only be chosen when the variable is created |
| File type | the value is written to a temporary file, and the variable holds its path | stop a script that reads the file from printing it |
GitLab's own documentation is blunt about masking: it "is not a guaranteed way to prevent malicious users from accessing variable values". A masked value must also be a single line with no spaces and at least 8 characters, which rules out most private keys.
Who gets a protected value#
Protection decides which pipelines receive a value at all. This is the rule behind most "works on main, empty on my branch" problems:
- Pipelines on protected branches and tags get protected values.
MegaCorp protects
mainand tags matchingv*. - Pipelines on any other branch don't. The variable is simply not set.
- Merge request pipelines get them only when all four of these are
true:
- the source and target branches are both protected
- both branches are in the same project
- the person running the pipeline can push or merge to the target
- the project setting Allow merge request pipelines to access protected variables and runners is on
- Pipelines from forks run in the fork and get none of the parent project's variables. A parent-project member can choose to run a fork's merge request pipeline in the parent instead. That pipeline gets the parent's variables, though not protected ones, and runs the fork's pipeline file. GitLab shows a warning that must be accepted first.
payments-api protects main and tags matching v*, and keeps
one secret of its own:
protected_branches: [main]
protected_tags: ["v*"]project_variables:
- key: MC_NEXUS_URL
value: https://nexus.example.com/repository/maven-public/
- key: PAYMENTS_DB_PASSWORD
value: "(set in the UI)"
masked: true
protected: trueThe megacorp group adds one more secret, for every project:
# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
- key: MAVEN_CLI_OPTS
value: "--batch-mode --errors --show-version -s .m2/settings.xml"
- key: AWS_ACCOUNT_ID
value: "123456789012"
- key: MC_AZURE_TENANT_ID
value: "00000000-0000-0000-0000-000000000000"
- key: MC_ACR_REGISTRY
value: megacorpprod.azurecr.io
- key: MC_ACR_PUSH_CLIENT_ID
value: "00000000-0000-0000-0000-000000000000"
# the gitlab-acr-push application in Entra ID, which mc_acr_push signs in as
- key: SONAR_HOST_URL
value: https://sonar.example.com
- key: SONAR_TOKEN
value: "(set in the UI)"
masked: true
protected: trueSONAR_TOKEN is protected. sonar-scan runs in every merge
request pipeline, and feature branches aren't protected, so in those pipelines the token
is empty. The scan can't sign in to the Sonar server. The job is allowed to fail, so the
merge request stays green, and nobody notices that the quality gate never ran.
The secrets: keyword Premium#
With secrets:, the value never sits in GitLab's settings at all. GitLab
fetches it from a secrets manager when the job starts:
deploy:
id_tokens:
VAULT_ID_TOKEN:
aud: https://vault.example.com
secrets:
DATABASE_PASSWORD:
vault: production/db/password@ops # engine ops, secret production/db, field password
script:
- ./deploy.sh --password-file "$DATABASE_PASSWORD"Four things to know when you read one:
- It arrives as a file. The variable holds the path to a temporary
file containing the value, which is why the script above passes it as
--password-file. Writefile: falseunder the variable name to get the value itself. - It needs a way to prove who is asking, usually an ID token
(chapter 28). For Vault, a job's only ID token is used
automatically. With several, name one with
token:, or which is used can't be predicted. AWS Secrets Manager looks for a token namedAWS_ID_TOKEN, or uses the runner's own AWS role (chapter 38). - The provider decides who may read. A Vault role, for example, lists
bound claims: values the ID token must carry, such as the project's ID or
ref_protected: "true". GitLab's documentation warns that a role not tied to a project or group accepts a token from any job on the instance. - The providers are HashiCorp Vault, AWS Secrets Manager, Google Cloud Secret Manager, Azure Key Vault, and GitLab's own Secrets Manager.
GitLab Secrets Manager Beta is GitLab's own store,
under Secure › Secrets manager, and has been in public beta since 19.0.
Jobs read it with secrets: … gitlab_secrets_manager:, which needs GitLab
Runner 19.0 or later. A secret can be limited to a branch or an environment. Its value is
masked if a job prints it, and unlike a masked variable it can contain spaces and new
lines. Switching the manager off deletes every secret in it, permanently.
How secrets leak anyway#
None of the options above stops a determined script. These are the usual ways values escape:
- Printing them changed. Base64-encoding a token, or a shell adding escape characters, produces text that masking doesn't recognise.
- Debug tracing. Setting
CI_DEBUG_TRACE: "true"writes every variable the job has into the log. Masked values still show as[MASKED], but everything else appears in full. Only Developers and above can read such logs, unless the variable was set in a runner's own configuration, where the check doesn't apply. - Listing the environment.
export,envorprintenvprint every variable. Masked values show as[MASKED], but unmasked ones don't. File-type variables show only a path. - Artifacts and dotenv files. Anything a job writes into an artifact can be downloaded by people who can see the pipeline.
- Changed pipeline files. A merge request can change
.gitlab-ci.ymlto send a variable anywhere. Protection limits which pipelines have the value, but review of pipeline changes is what actually stops this.
When a secret is empty#
Work through these questions in order. The first yes is almost always the cause:
As text
- Is the variable protected, and this branch or tag not? Yes: Protected values reach only protected branches and tags. No: the next step.
- Is this a merge request pipeline? Yes: Both branches must be protected, and a setting must allow it. No: the next step.
- Is the pipeline running in a fork? Yes: Forks don't receive the parent project's variables. No: the next step.
- Does the variable have an environment scope? Yes: Only jobs deploying to that environment receive it. No: the next step.
- Does the job get it through secrets:? Yes: Check the ID token and the provider's role (chapter 21). No: the next step.
- Check the name matches exactly, and that a higher rung doesn't override it (chapter 19)
- Do this
- Mask every secret, so that logs show
[MASKED]. Protect it as well if only protected branches and tags need it. Prefer a short-lived ID token to a stored credential wherever the other side accepts one. - Why
- A masked value never appears in a log. A protected value never reaches a branch that anyone can push to.
- Watch out
- A protected secret is empty on a feature branch, so a job there that needs it fails. The card fails only on branches shows the symptom.
- Put a secret in the settings, masked and protected, and predict where it will be empty.
- Read [MASKED] in a log and know what GitLab can and cannot mask.
- Choose an ID token over a stored key wherever the other side accepts one.
Spot the bugThe quality gate that never ran#
MegaCorp's platform team notices that Sonar has had no analysis of any payments-api
merge request for months. Analyses of main arrive after every merge. In
merge request pipelines, sonar-scan shows an orange warning, and its log
ends in an authentication error.
# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
- key: MAVEN_CLI_OPTS
value: "--batch-mode --errors --show-version -s .m2/settings.xml"
- key: AWS_ACCOUNT_ID
value: "123456789012"
- key: MC_AZURE_TENANT_ID
value: "00000000-0000-0000-0000-000000000000"
- key: MC_ACR_REGISTRY
value: megacorpprod.azurecr.io
- key: MC_ACR_PUSH_CLIENT_ID
value: "00000000-0000-0000-0000-000000000000"
# the gitlab-acr-push application in Entra ID, which mc_acr_push signs in as
- key: SONAR_HOST_URL
value: https://sonar.example.com
- key: SONAR_TOKEN
value: "(set in the UI)"
masked: true
protected: trueShow the answer
SONAR_TOKEN is a protected group variable. Protected values reach only
pipelines on protected branches and tags. A merge request pipeline from a feature branch
gets them only if both branches are protected, which a feature branch isn't. So the
token is empty, and the scanner can't authenticate.
Nobody noticed because sonar-scan has allow_failure: true: a
failure turns the job orange, not the pipeline red. The owners of the token must decide
whether an unprotected token, a separate token for merge requests, or a different design
is acceptable. See who gets a protected value.
Identity and tokens#
Many jobs must prove who they are before another system lets them in: GitLab's own API and registries, AWS, or a secrets manager. A job can do that in four ways:
- the job token, which GitLab creates for every job, for calls back to GitLab
- an ID token, a signed note about the job that other systems can check
- a stored token, such as a project access token, kept in a CI/CD variable
- the runner's own identity, which every job on that runner shares
The first two are made fresh for each job and stop working when it ends. Stored tokens expire on a date nobody remembers, which is why a pipeline can break with no commit to blame.
A job that has published release notes through the GitLab API every week since last September fails today with 401 Unauthorized. Nobody committed anything. Nobody changed a setting. The token it uses was created by someone who left in May, and its page now says Expired. Four kinds of credential can be in play in a job; which one is this, and why did it die today? (the card)
Four ways a job proves who it is#
| Identity | Made by | Works until | The usual failure |
|---|---|---|---|
The job token, CI_JOB_TOKEN | GitLab, for every job | the job ends | 404 Not Found from a project that hasn't allowed yours |
| An ID token | GitLab, when the job asks with id_tokens: | the job's timeout, or 5 minutes | the other system refuses it, because its details don't match the trust rule |
| A stored token in a variable | a person, once | its expiry date, often a year after it was made | an authentication error, starting the day it expires |
| The runner's own role | the platform team | someone changes the runner | every project on that runner has the same access |
When a call from a job is refused, first work out which of the four it used. Each fails for different reasons, and each is fixed in a different place.
The job token: calling back to GitLab#
Every job gets a variable called CI_JOB_TOKEN. GitLab creates it as the job
starts and cancels it when the job finishes. It lets the job call GitLab as the person who
started the pipeline, but it reaches far less than that person could:
- the container, package and Terraform module registries
- downloading job artifacts, and reading files from a repository
- releases, environments and deployments
- cloning a repository, with
gitlab-ci-tokenas the user name - starting a pipeline in another project through the API
It doesn't work with GitLab's GraphQL API, and GitLab masks it in job logs. Here a job downloads a package that another project published:
fetch-test-data:
script:
- >
curl --fail --header "JOB-TOKEN: $CI_JOB_TOKEN" --output data.zip
"$CI_API_V4_URL/projects/megacorp%2Fdevops%2Fci-tools/packages/generic/test-data/1.0.0/data.zip"Calling another project needs that project's permission. Each project keeps a job token allowlist: the groups and projects whose job tokens it accepts. At first it holds only the project itself. So for the job above to work:
- ci-tools must list payments-api, or the megacorp group, under Settings › CI/CD › Job token permissions.
- The person who started the pipeline must be a member of ci-tools, with a role that allows the action. Being on the allowlist grants nothing by itself.
Public and internal projects are a partial exception. Some of their features, such as artifacts and registries, answer any project's job token unless the project limits them to members.
A refusal rarely says "not allowed". GitLab answers 404 Not Found, as if
the project didn't exist. A refused clone says:
remote: The project you were looking for could not be found or you don't have permission to view it.As text
- Is the call made with CI_JOB_TOKEN? No: The job token isn't involved; see stored tokens below. Yes: the next step.
- Is your project, or its group, missing from the target's allowlist? Yes: A Maintainer of the target adds it under Job token permissions. No: the next step.
- Is the person who started the pipeline not a member of the target? Yes: The job token acts as that person, so they need access. No: the next step.
- Is it a GraphQL call, or an API the job token can't reach? Yes: That call needs a different identity. No: the next step.
- Check the job was still running; the token stops working when the job ends
ID tokens: a signed note about the job#
Systems outside GitLab, such as AWS and Vault, don't accept the job token. For them, a
job asks GitLab for an ID token. This is a short piece of text, called a
JWT, that lists facts about the job and carries GitLab's signature. The receiver checks
the signature against keys that GitLab publishes, so nobody can forge or change the
facts. A job asks for one with id_tokens::
.image-build:
extends: .megacorp-base
stage: package
image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
id_tokens:
MC_ID_TOKEN:
aud: https://gitlab.example.com
variables:
AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
before_script:
- !reference [.snippets, functions]
- !reference [.snippets, aws_login]
script:
- mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
- buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
- buildah push "$IMAGE:$CI_COMMIT_SHA"
- !reference [.snippets, acr_push]
- echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
artifacts:
reports:
dotenv: build.envMC_ID_TOKEN becomes a variable that holds the token. aud, short
for audience, says who the token is for, and a receiver can refuse a token meant for
someone else. MegaCorp's mc_aws_login function then hands the token to AWS.
On main, mc_acr_push hands the same token to Azure
(chapter 38).
The facts in a token are called claims. These matter most. The examples are
for image-build in a payments-api pipeline on main:
| Claim | Example | What it says |
|---|---|---|
sub | project_path:megacorp/payments/payments-api:ref_type:branch:ref:main | the subject: project, kind of ref, and ref, in one string |
aud | https://gitlab.example.com | who the token is for, from aud: in the YAML |
project_path | megacorp/payments/payments-api | the project running the job |
ref, ref_type | main, branch | the branch or tag; in a merge request pipeline, the source branch |
ref_protected | true | whether that branch or tag is protected |
pipeline_source | push | what started the pipeline |
environment | production | the environment, only when the job has one |
exp | a time | when the token expires |
A probe job on the local GitLab 19.3 showed the same shape in four pipelines. On
main, ref_protected was "true", as a string. On a
feature branch it was "false", and the branch's merge request pipeline had the
same sub and ref, with pipeline_source set to
merge_request_event. A tag put ref_type:tag in
sub.
A token expires when the job's timeout runs out, or after 5 minutes if no timeout is
specified. In the probe, each token lasted one hour, the default job timeout. So exchange
it early. MegaCorp logs in to AWS in before_script,
before the long build starts.
How AWS checks an ID token#
Here is the whole exchange for MegaCorp's image push. GitLab signs the token, and the job sends it to AWS's token service, STS. STS makes three checks before it hands back keys that last one hour:
The third check is where most refusals happen. A role's trust policy
says which subjects may use it. MegaCorp's push role accepts any branch of any project
under megacorp/:
# Assumed by image-build, and by the ecr-push component, through mc_aws_login.
ecr_push_role:
arn: arn:aws:iam::123456789012:role/gitlab-ecr-push
trust_policy:
Effect: Allow
Principal:
Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
Action: sts:AssumeRoleWithWebIdentity
Condition:
StringEquals:
gitlab.example.com:aud: https://gitlab.example.com
StringLike:
gitlab.example.com:sub: project_path:megacorp/*:ref_type:branch:ref:*
permissions: push and pull images in every ECR repository in the accountRead the pattern against each pipeline's sub:
- A pipeline on
mainmatches. - A merge request pipeline matches too, because its
refis the source branch, such asfeature/retry. - A pipeline for tag
v1.4.0doesn't match. Its subject saysref_type:tag, and the pattern demandsbranch.
On self-managed GitLab, AWS can check only sub and aud. On
GitLab.com it can also check claims such as project_id and
ref_protected.
Stored tokens, and the day they expire#
Some jobs use a token that a person made once and saved as a masked CI/CD variable. MegaCorp's deploy tool pushes to the GitOps repository with one:
deploy)
env="${1:?usage: mc deploy <environment> <image-ref>}"
image="${2:?usage: mc deploy <environment> <image-ref>}"
tag="${image##*:}"
mc_log "promoting ${APP} ${tag} to ${env} through the GitOps repository"
git clone --depth 1 \
"https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
yq -i ".image.tag = \"${tag}\"" "gitops/envs/${env}/${APP}.yaml"
git -C gitops commit -am "deploy ${APP} ${tag} to ${env}"
if [ "$env" = production ]; then
# production changes wait in a merge request until someone merges them
git -C gitops push origin "HEAD:deploy/${APP}-${tag}" \
-o merge_request.create -o merge_request.target=main \
-o merge_request.title="Deploy ${APP} ${tag} to production"
else
git -C gitops push origin HEAD:main
fi
;;project_variables:
- key: GITOPS_TOKEN
value: "(set in the UI)"
masked: true
protected: true
# A project access token created in megacorp/platform/gitops-config, with the
# Maintainer role and the write_repository scope. Nobody entered an expiry date,
# so GitLab set one 365 days after the token was created.GitLab has several kinds of stored token:
| Token | Acts as | Expires |
|---|---|---|
| Personal access token | one person, with that person's access | on its expiry date, 365 days after creation if none was entered |
| Project or group access token | a bot user that GitLab creates for the project or group | the same as a personal access token |
| Deploy token | a project or group, for repositories and registries | never, unless a date was set |
Access tokens stop working at midnight UTC on their expiry date. The job that uses one runs unchanged and then fails with an authentication error. Nothing in the repository changed, so nothing in its history explains the failure. GitLab emails the Maintainers and Owners of the token's project 60, 30 and 7 days before a project access token expires. They aren't always the people who look after the pipeline that uses it.
gitlab-deploy-token reaches the project's
jobs automatically, as CI_DEPLOY_USER and CI_DEPLOY_PASSWORD. No
variable needs to be set.The runner's own role#
The machine or pod that runs a job can have an identity of its own. MegaCorp's shared runner starts every job as a Kubernetes service account that AWS maps to a role:
[[runners]]
name = "megacorp-eks-shared"
# registered in GitLab as a group runner of megacorp, with the tag megacorp-shared
url = "https://gitlab.example.com"
executor = "kubernetes"
environment = ["MAVEN_OPTS=-Xmx2g", "HTTPS_PROXY=http://proxy.example.com:3128"]
pre_build_script = "echo 'runner: megacorp-eks-shared'"
[runners.kubernetes]
namespace = "gitlab-runners"
image = "registry.example.com/megacorp/devops/ci-tools:3.2"
# jobs run as this service account, which is bound to an AWS IAM role
service_account = "gitlab-runner-jobs"# The role behind the gitlab-runner-jobs service account on the megacorp-eks-shared
# runner. Every job on that runner can use it without asking. The EKS cluster is what
# AWS trusts for this role, not GitLab.
runner_role:
arn: arn:aws:iam::123456789012:role/gitlab-runner-jobs
permissions: pull images from every ECR repository in the accountAny job on that runner can use this role without asking. The access is the same for
every project, it appears in no .gitlab-ci.yml, and the platform team
controls it. When a job's AWS calls act as the wrong identity, run
aws sts get-caller-identity. It prints the role the job is really using, and
needs no permissions (chapter 38).
- Do this
- Give each job that talks to a cloud its own
id_tokensentry, with the audience the cloud role expects. Let the role's trust policy decide which branches and tags may use it. - Why
- An ID token is made for one job and expires minutes later. Nothing has to be stored, and nothing can leak from a settings page.
- Watch out
- Do not store long-lived cloud keys as variables. They expire, or they outlive the people who made them. The card it worked for months shows what that failure looks like.
- Tell the four credentials a job can use apart: job token, ID token, stored key, and the runner's own identity.
- Read an authentication failure and say which credential failed and why.
- Replace a stored cloud key with an ID token and a trust policy.
Spot the bugDeploys that stopped on a Tuesday#
No MegaCorp service has deployed since Tuesday morning. Every deployer pipeline fails
in mc deploy, at the clone of the GitOps repository, with an authentication
error. Nothing was merged into deployer, ci-tools or gitops-config for a week, and the
same pipelines deployed on Monday.
project_variables:
- key: GITOPS_TOKEN
value: "(set in the UI)"
masked: true
protected: true
# A project access token created in megacorp/platform/gitops-config, with the
# Maintainer role and the write_repository scope. Nobody entered an expiry date,
# so GitLab set one 365 days after the token was created.Show the answer
GITOPS_TOKEN is a project access token. Access tokens stop working at
midnight UTC on their expiry date, which is 365 days after creation unless someone chose
another. The clone sends the expired token, and GitLab refuses it. No file changed, so no
commit explains the failure.
Create a new token and update the variable. To stop it recurring, track the expiry date. Also make sure the people GitLab emails, the Maintainers and Owners of gitops-config, know that the deploys depend on this token. See stored tokens.
Runners and executors#
GitLab only schedules jobs; a runner runs them. Which runner takes a job depends on three things:
- which runners the project may use
- whether a runner has every tag the job asks for
- whether a protected runner accepts this branch
Once a runner has the job, the runner's own configuration shapes what happens, and your
project can't see it. A job stuck in pending is almost always a problem with
the first part. A job that behaves differently from run to run is often the second.
load-test has sat in pending for forty minutes. Its page says it is stuck because no runner matches all of its tags: linux and big-memory. Yesterday it ran in three minutes. Nobody touched the job. Somewhere, a runner went away. (the card)
Which runners a project can use#
Every runner belongs to one of three scopes:
| Scope | Available to | Which waiting job it takes next |
|---|---|---|
| Instance runner | every project on the instance, unless a project or group turns instance runners off | a fair-usage queue: jobs from projects with the fewest jobs already running come first |
| Group runner | every project and subgroup in its group | the job that has waited longest |
| Project runner | only the projects it has been turned on for; a fork doesn't get it | the job that has waited longest |
To see the runners your project can use, open Settings › CI/CD and
expand Runners. Each runner shows its tags and its status. A runner is
online if it has contacted GitLab in the last two hours, offline
after that, and stale after seven days.
MegaCorp has one shared runner, megacorp-eks-shared. It is a group runner
of megacorp with the tag megacorp-shared, and every job asks for
that tag through default: in the base template
(chapter 14).
How a job finds a runner#
When a job's turn comes, it waits in pending until a runner that fits asks
for work. Work through these checks in order:
As text
- Is any runner available to the project online? No: Nothing can take the job; check the runners' status. Yes: the next step.
- Does the job list tags that no online runner has all of? Yes: A runner needs every tag the job lists. No: the next step.
- Does the job list no tags, and no runner takes untagged jobs? Yes: Add a tag, or tick Run untagged jobs on a runner. No: the next step.
- Are the only runners that fit protected, and this branch isn't? Yes: Protected runners take jobs only from protected branches and tags. No: the next step.
- Are the runners that fit paused, or already running as many jobs as they allow? Yes: The job waits its turn; it isn't stuck. No: the next step.
- A runner takes the job, and it moves to running
The job's page says which problem GitLab suspects. These are its messages in GitLab 19.3:
The job page says which. "This job is stuck because of one of the following problems. There are no active runners online, no runners for the protected branch, or no runners that match all of the job's tags" means the job has tags, and the page lists them after the message. "This job is stuck because the project doesn't have any runners online assigned to it" means the project's runners exist, but none is online. "This job is stuck because you don't have any active runners that can run this job" means no runner available to the project can take it.
What a runner does with a job#
A runner takes every job through the same steps. Some come from your YAML, some from the runner's configuration, and the runner does the rest by itself:
Two things surprise people:
after_scriptruns in a new shell. Variables exported inscriptare gone. It gets 5 minutes, unlessRUNNER_AFTER_SCRIPT_TIMEOUTgives it another limit.- The runner can add commands of its own.
pre_build_scriptruns just before yourbefore_script, in the same shell, and nothing in your project shows it.
What the runner's configuration decides#
A runner's settings live in a file called config.toml, on the runner's
machine. This is MegaCorp's:
[[runners]]
name = "megacorp-eks-shared"
# registered in GitLab as a group runner of megacorp, with the tag megacorp-shared
url = "https://gitlab.example.com"
executor = "kubernetes"
environment = ["MAVEN_OPTS=-Xmx2g", "HTTPS_PROXY=http://proxy.example.com:3128"]
pre_build_script = "echo 'runner: megacorp-eks-shared'"
[runners.kubernetes]
namespace = "gitlab-runners"
image = "registry.example.com/megacorp/devops/ci-tools:3.2"
# jobs run as this service account, which is bound to an AWS IAM role
service_account = "gitlab-runner-jobs"These settings change jobs most often:
| Setting | What it does to your job |
|---|---|
executor | where the job runs: a Kubernetes pod here; elsewhere a container, or the machine itself |
environment | adds or overwrites environment variables in every job; here, a Maven memory setting and a proxy |
pre_build_script | runs before your before_script, in the same shell |
image | the image for jobs that don't set one |
output_limit | the largest log the runner sends, 4096 KB by default; the rest is cut off |
concurrent and limit | how many jobs run at once; the others wait in pending |
allowed_images, privileged, pull_policy | which images jobs may use, whether containers get extra powers, and when images are downloaded again |
| CPU and memory settings | how much the job's container gets, on Kubernetes |
Executors#
The executor decides where a job's commands run, which changes what a job can rely on:
| Executor | Each job runs | A clean start for every job? |
|---|---|---|
| Kubernetes | in a new pod in a cluster: one container for the job, and one for each service | yes |
| Docker | in a new container, from the job's image | yes |
| Docker Autoscaler | like Docker, on machines created on demand | yes |
| Instance | on a whole machine created on demand, with no container | it depends on the setup |
| Shell | directly on the runner's machine | no: files and tools from earlier jobs may still be there |
The Shell, SSH, VirtualBox, Parallels and Custom executors are in maintenance mode: they
get security fixes, but no new features. The shell and instance executors don't support
image: or services:. A job that works on one runner but not on
another often landed on runners with different executors.
Time limits#
Three limits can stop a job:
- The job timeout. This is the job's own
timeout:, or the project's setting if the job has none. A runner's maximum job timeout cuts it down when the runner's is shorter. RUNNER_SCRIPT_TIMEOUT, a job variable that stopsscriptearly, so artifacts can still upload before the job timeout.RUNNER_AFTER_SCRIPT_TIMEOUT, the same forafter_script. Its default is 5 minutes.
So a job with timeout: 2h, on a runner whose maximum is 30 minutes, stops
after 30 minutes.
- Do this
- Add
tagsto a job only when it needs a particular kind of runner, and use only the tags your platform team publishes. - Why
- A job with no tags can run on any shared runner. Every tag you add narrows the runners that can take it.
- Watch out
- A tag that no online runner has leaves the job pending forever, and GitLab marks it as stuck. The card no runner matches its tags shows the symptom.
- Say which runners can take a job, from the job's tags and the runners' scopes.
- Read the first lines of a log and know which runner and executor ran the job.
- Diagnose a pending job in one look: tags, protection, or a queue.
Spot the bugThe job that waits for ever#
A payments-api developer copies a job from a blog post into the project's
.gitlab-ci.yml:
integration-test:
stage: test
tags: [docker]
image: maven:3.9-eclipse-temurin-21
script:
- mvn verify -PintegrationEvery other job runs. integration-test stays in pending, and
its page says it is stuck, listing the tag docker.
Show the answer
A job's own tags replace the default: tags completely. So this
job asks only for docker, and no longer for megacorp-shared.
MegaCorp's only runner has the tag megacorp-shared, and a runner must have
every tag a job lists. No runner qualifies, so the job waits for ever.
Delete the tags line, so that the job inherits megacorp-shared
again. It still runs in a container, because the runner uses the Kubernetes executor.
See how a job finds a runner.
Images, services and Docker#
Most jobs run inside a container. The job's image: decides which tools its
script can use. services: start extra containers beside it, such as a database
for tests. Building container images inside a job is a problem of its own, because the
classic way, Docker-in-Docker, needs a privileged runner.
The usual surprises are an image whose entrypoint swallows the script, a service that never gets its password, and a registry login that happens in the wrong place.
You change a job's image to amazon/aws-cli:2.27.0. The job now fails before your first command, with the CLI's usage text in the log. Your script is still in the file. The image ran something else instead. (the card)
The job's image#
image: names the container image a job runs in. It can be a name, which
means the latest tag, a name and a tag, or a name and a digest:
unit-test:
image: node:22 # a name and a tag; with no registry named, from Docker Hub
script:
- npm testMegaCorp's base template sets default: image to its toolbox image,
ci-tools (chapter 19). Four things to know about any image:
- It needs a shell. The image must contain
shorbash, andgrep, because the runner sends the script to the container's shell. - Its entrypoint must not get in the way. The Docker executor starts the
container with the image's own entrypoint and hands it a shell. An image whose entrypoint
runs a program, such as
amazon/aws-cli, never reaches your script, andentrypoint: [""]removes it. The Kubernetes executor ignores entrypoints unless the runner turns onFF_KUBERNETES_HONOR_ENTRYPOINT. So the same image can work on one runner and fail on another. - Scripts run in
/builds/followed by the project's path, not in the image's own working directory. - The runner decides when to download it again. Its pull policy is
always,if-not-presentornever. A job can ask for one withimage: pull_policy:, if the runner allows it.
aws-version:
image:
name: amazon/aws-cli
entrypoint: [""] # needed on the Docker executor
script:
- aws --versionImages from private registries#
The runner downloads the image before the job starts, so the job's script can't log in for it. The runner looks for registry credentials in this order:
- a
config.jsonfile in/root/.dockeron the runner - a
DOCKER_AUTH_CONFIGCI/CD variable - a
DOCKER_AUTH_CONFIGset in the runner'sconfig.toml - a
config.jsonin the home directory of the user that runs the runner
Images in the same GitLab's container registry need none of this: the runner uses the job token (chapter 28). The person who started the pipeline needs at least the Reporter role in the image's project. That project must also let your project's job token in, which is off by default.
Every pull from Docker Hub counts against Docker Hub's rate limits. GitLab's dependency proxy is a cache in front of it. Put a prefix on the image name, and the runner signs in to the proxy by itself:
unit-test:
image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/node:22Services: containers beside the job#
services: starts more containers for the length of the job, most often a
database for tests. They start before before_script, and stop when the job
ends, even if it fails:
integration-test:
image: maven:3.9-eclipse-temurin-21
services:
- name: postgres:16
alias: db
variables:
POSTGRES_DB: payments_test
POSTGRES_PASSWORD: test-only-password # the database lives only for this job
script:
- mvn verify -Ddb.url=jdbc:postgresql://db:5432/payments_testWhat trips people up:
- A service is another machine on the network. Connect to its hostname,
such as
db, not to a local socket. On Kubernetes, the services share the job's network, solocalhostworks too, but two services can't use the same port. - A service adds no programs to the job. Listing
node:22as a service doesn't give your script anodecommand. - The hostname comes from the image name, unless you set an
alias.postgres:16is reachable aspostgres. - Services get only YAML variables. GitLab doesn't trust service containers by default, so variables from settings don't reach them. To pass one, re-assign it in the YAML under a different name.
- A job's
servicesreplacedefault: services, like every list (chapter 14). - A service that fails to start leaves a line in the job log:
*** WARNING: Service XYZ probably didn't start properly. SettingCI_DEBUG_SERVICES: "true"adds the services' own logs, but can reveal masked values.
Building images inside a job#
A job that runs docker build needs a Docker daemon, and the job's container
doesn't have one. There are four ways round that, and the runner decides which of them
are possible:
| Method | How it works | Privileged runner? | The catch |
|---|---|---|---|
| Docker-in-Docker | a docker:dind service runs a daemon for the job | yes | privileged containers can break out to the host; no layer cache between jobs |
| Socket binding | the runner mounts the host's Docker socket into the job | no | the job controls the host's daemon, and can remove other jobs' containers |
| Buildah | builds images with no daemon at all | no | a different command line from docker |
| Rootless BuildKit | Docker's own build engine, run without a daemon | no | the runner must still allow the system calls it uses |
As text
- Does the runner run privileged containers? Yes: Docker-in-Docker works: a docker:dind service, with TLS. No: the next step.
- Does the runner mount the host's Docker socket? Yes: docker build works, but the job controls the host's daemon. No: the next step.
- Does the runner allow the system calls that rootless builds need? Yes: Rootless BuildKit, or Buildah. No: the next step.
- Rootless Buildah, which GitLab suggests when the runner can't be changed
kaniko, once the usual answer, is no longer maintained, and GitLab has marked its page as removed. MegaCorp builds with Buildah, so no job needs a Docker daemon:
.image-build:
extends: .megacorp-base
stage: package
image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
id_tokens:
MC_ID_TOKEN:
aud: https://gitlab.example.com
variables:
AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
before_script:
- !reference [.snippets, functions]
- !reference [.snippets, aws_login]
script:
- mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
- buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
- buildah push "$IMAGE:$CI_COMMIT_SHA"
- !reference [.snippets, acr_push]
- echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
artifacts:
reports:
dotenv: build.envThe job runs in a Buildah image rather than the toolbox. It logs in to ECR with the
job's ID token (chapter 28), builds and pushes. On main, it
pushes the same image to Azure's registry too. Then it writes the image's name to a dotenv
report for later jobs (chapter 31).
Docker-in-Docker, set up correctly#
GitLab recommends Docker-in-Docker with TLS. The runner must be privileged, and must share a certificates directory with the service. The job then looks like this:
build-image:
image: docker:24.0.5-cli
services:
- docker:24.0.5-dind
variables:
DOCKER_TLS_CERTDIR: "/certs"
script:
- docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .On the Kubernetes executor, also set DOCKER_HOST: tcp://docker:2376,
DOCKER_TLS_VERIFY: 1 and
DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client". The image's entrypoint would
normally set them, but that executor doesn't run it. Without TLS, the port is 2375 and
DOCKER_TLS_CERTDIR is empty.
Pin the image versions, because docker:latest can change under you. Log in
to registries in before_script: the daemon in the service is new for every
job, and doesn't know the runner's credentials.
- Do this
- Pin every image to a version tag, such as
node:24.21.0-alpine, and never tolatest. - Why
- A pinned image makes the job behave the same next month. The card tools changed overnight shows what a moving tag does.
- Watch out
- When you use a tool's own image, check its entrypoint, the command the image runs
by default. If the entrypoint is the tool itself, your script never runs, and you need
entrypoint: [""]. The card the script never runs shows the symptom.
- Choose a job's image and pin it, and empty an entrypoint that swallows your script.
- Run a service beside a job and reach it by its alias.
- Build an image inside a job, and know what the runner must allow for that.
Spot the bugThe test database that never started#
payments-api adds integration tests with a PostgreSQL service. Following the advice in
chapter 27, the team keeps the database password out of the YAML. They save it in the
project's settings as a masked variable named POSTGRES_PASSWORD:
integration-test:
image: maven:3.9-eclipse-temurin-21
services:
- name: postgres:16
alias: db
script:
- mvn verify -Ddb.url=jdbc:postgresql://db:5432/postgresThe job log warns that the service probably didn't start properly, and the tests can't
connect to db. Someone adds POSTGRES_PASSWORD: $POSTGRES_PASSWORD
to the job's variables, and nothing changes.
Show the answer
Variables from settings don't reach service containers, which GitLab doesn't trust by
default. The postgres image needs POSTGRES_PASSWORD to start, so
it doesn't.
A re-assignment only works under a different name. POSTGRES_PASSWORD:
$POSTGRES_PASSWORD uses the same name, so it isn't expanded. This is the trap from
chapter 26. Rename the setting, for example to
TEST_DB_PASSWORD, and write POSTGRES_PASSWORD: $TEST_DB_PASSWORD
in the YAML. Or, because this database lives only for one job, a fixed test-only password
in the YAML protects nothing worth hiding. See services.
Artifacts, cache and dotenv#
Every job starts in a fresh place, so anything a job makes is lost unless it is saved. GitLab has two ways to keep files, and they are easy to confuse:
- Artifacts are a job's results, such as a built
.jar. GitLab stores them and hands them to later jobs in the same pipeline. They are guaranteed. - A cache is a speed-up, such as downloaded libraries. The runner may reuse it in later jobs and pipelines, but it can be missing at any time.
A third tool, the dotenv report, passes small values, such as an image name, from one job to later ones as variables.
The package job reads version.txt, which the build job wrote. On Monday it finds last Thursday's file, with last Thursday's version. Both jobs passed. The file is in the cache, keyed on the branch, and the build job had nothing to write today. Which of the two ways of passing files should this have been? (the card)
Two ways to keep files#
The rule that follows: never rely on the cache for something a job must have. If a later job needs a file, make it an artifact. If losing the file only makes a job slower, cache it.
Artifacts#
MegaCorp's Maven templates keep both kinds of artifact. maven-build keeps
the built jar, and maven-test keeps a test report:
.maven-build:
extends: .maven-base
stage: build
script:
- mvn $MAVEN_CLI_OPTS -DskipTests package
artifacts:
paths: [target/*.jar].maven-test:
extends: .maven-base
stage: test
image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JDK}
parallel:
matrix:
- JDK: ["17", "21"]
script:
- mvn $MAVEN_CLI_OPTS verify
artifacts:
when: always
reports:
junit: target/surefire-reports/TEST-*.xml
rules:
- !reference [.rules, never-on-schedule]
- !reference [.rules, mr]
- !reference [.rules, default-branch]| Keyword | What it decides | Default |
|---|---|---|
artifacts:paths | which files to keep, relative to the project directory | nothing |
artifacts:when | keep them when the job succeeds, fails, or always | on_success |
artifacts:expire_in | how long GitLab keeps them | the instance's setting, but the latest successful pipeline's are kept anyway |
artifacts:reports | files that GitLab reads and shows, such as test results | uploaded even when the job fails |
artifacts:access | who can download them in the UI and API | all |
dependencies | which earlier jobs' artifacts a job downloads; [] means none | every job in earlier stages |
Three behaviours cause most artifact surprises:
- A failed job keeps nothing, unless
artifacts:whensays otherwise. That is whymaven-testhaswhen: always: the output of a failed test run is the output you want. Reports are uploaded even without it. needsanddependenciesnarrow the download. A job with either gets artifacts only from the jobs it names (chapter 24).- Artifacts expire. Every job that extends MegaCorp's
.megacorp-basekeeps them for 7 days. A job that needs expired artifacts fails before its script starts.
That failure, and the log line of a job that saved nothing, look like this:
This job could not start because it could not retrieve the needed artifacts.
No files to uploadThe first means a missing dependency, expired artifacts or missing permissions. The second, in the earlier job's log, means its path was wrong, or the file was never made.
Cache#
MegaCorp's Maven jobs cache the local Maven repository, so that each job doesn't download every library again:
.maven-base:
extends: .megacorp-base
image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
variables:
JAVA_VERSION: "21"
MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
cache:
key:
files: [pom.xml]
paths: [.m2/repository]
before_script:
# a job's own before_script replaces default:before_script, so re-add the functions
- !reference [.snippets, functions]
- !reference [.snippets, maven_settings]The key decides which saved cache a job gets. Jobs with the same key
share one cache, in this pipeline and in later ones. Here the key is made from the
contents of pom.xml, so a change to the dependencies starts a new cache.
| Keyword | What it decides | Default |
|---|---|---|
cache:key | which cache to use; key: files: makes one from up to two files' contents | default |
cache:paths | which files to save and restore | none, so nothing is cached |
cache:policy | pull only restores, push only saves, pull-push does both | pull-push |
cache:when | save on success, on failure, or always | on_success |
cache:fallback_keys | other keys to try when the key has no cache yet | none |
cache:unprotect | share caches between protected and unprotected branches | false |
MegaCorp's Node templates use the policy well. The base sets pull, so jobs
only read the cache, and the build job alone saves it:
.node-base:
extends: .megacorp-base
image: registry.example.com/megacorp/devops/ci-tools/node:22
cache:
key:
files: [package-lock.json]
paths: [.npm/]
policy: pull
before_script:
- npm ci --cache .npm --prefer-offline.node-build:
extends: .node-base
stage: build
cache:
policy: pull-push
script:
- npm run build
artifacts:
paths: [dist/]A cache that never seems to hit is the most common complaint. Check these in order:
As text
- Do the jobs run on different runners, with no shared cache storage? Yes: Each runner keeps its own cache; shared storage, such as S3, fixes it. No: the next step.
- Is this branch protected when the one that saved the cache wasn't, or the other way round? Yes: Protected and unprotected branches get separate caches, by design. No: the next step.
- Does the key change more often than you think? Yes: A key made from a file changes whenever that file does. No: the next step.
- Do two jobs use the same key for different paths? Yes: Each overwrites the other's cache; give them different keys. No: the next step.
- Did the job that saves the cache fail? Yes: Caches are saved only on success, by default. No: the next step.
- Read the cache lines in the job log; they name the key it tried
Why one branch can have two caches#
GitLab adds a suffix to every cache key. Pipelines for protected branches and tags get
-protected. So, since 18.4.5, does any pipeline started by someone with the
Maintainer or Owner role. Every other pipeline gets -non_protected.
So when a Developer and a Maintainer both push to the same feature branch, their pipelines use two different caches, and each misses the other's files. The project setting Use separate caches for protected branches, under Settings › CI/CD › General pipelines, turns the separation off. GitLab advises that only where everyone with the Developer role is highly trusted.
Also: caches are restored before artifacts. If a job caches and keeps the same path, the artifact overwrites what the cache restored.
dotenv: passing values to later jobs#
Variables set in a script vanish when the job ends. To hand a value to later jobs, write
it to a file of NAME=value lines, and declare the file as a
dotenv report. MegaCorp's image build passes on the image it pushed:
.image-build:
extends: .megacorp-base
stage: package
image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
id_tokens:
MC_ID_TOKEN:
aud: https://gitlab.example.com
variables:
AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
before_script:
- !reference [.snippets, functions]
- !reference [.snippets, aws_login]
script:
- mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
- buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
- buildah push "$IMAGE:$CI_COMMIT_SHA"
- !reference [.snippets, acr_push]
- echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
artifacts:
reports:
dotenv: build.envLater jobs receive IMAGE_REF as an ordinary variable. The rules:
- Who gets it: jobs in later stages, or only the jobs that name this one
in
needsordependencies. - Not in rules. Rules are decided when the pipeline is created, before any job has written anything (chapter 21).
- Its rung on the ladder: a dotenv value beats the job's YAML variables, but not settings or pipeline variables (chapter 26).
- A strict format: UTF-8, no blank lines, no comments, no quotes, and no multi-line values, in at most 5 KB. On self-managed GitLab, at most 20 of these variables are passed on by default.
- Not for secrets: anyone who can see the pipeline can read the file.
- Do this
- Use artifacts for everything a later job needs, and use the cache only to make jobs
faster. Key a dependency cache on the lockfile:
key: files: [package-lock.json]. - Why
- An artifact is guaranteed to arrive. A cache can miss at any time. A key on the lockfile changes whenever the dependencies change, so nobody restores an old cache.
- Watch out
- One fixed key, shared by every branch and every job, hands jobs files that are old or half-written. The card a job restores stale files shows what that looks like.
- Choose artifacts for what a later job needs and the cache for speed, and never confuse the two.
- Key a cache so that it can't serve stale files, and read a log to see whether it hit.
- Pass a variable from one job to the next with a dotenv report.
Spot the bugThe retry that couldn't start#
The v1.5.0 release pipeline failed at image-build, because of the
trust policy in chapter 38. Eight days later the role is fixed, and someone retries
image-build. It fails at once, before its script runs:
This job could not start because it could not retrieve the needed artifacts..megacorp-base:
variables:
MC_TEAM: unknown
artifacts:
expire_in: 7 days
rules:
- !reference [.rules, never-on-schedule]
- !reference [.rules, mr]
- !reference [.rules, default-branch]
- !reference [.rules, release-tag]Show the answer
image-build needs maven-build, so it must download the jar
that job saved. Every MegaCorp job extends .megacorp-base, whose artifacts
expire after 7 days, and eight days have passed. GitLab keeps the artifacts of the latest
successful pipeline on each ref no matter what, but this pipeline failed.
Retry maven-build first, so that a new jar exists, and then
image-build. Or start a new pipeline for the tag. See
artifacts.
Git in CI#
Before a job's script runs, the runner fetches the code. What it fetches is not the full repository you have on your laptop:
- only the pipeline's own branch or tag, and no others
- only the newest commits, 20 by default in new projects
- the pipeline's commit, checked out directly, so there is no current branch
Most scripts never notice. Scripts that compare with main, count commits
or read tags do notice, and they often fail only on big changes.
A job runs git describe --tags to name the build. In CI it prints fatal: No names found. On your laptop it prints the version. The job's clone has the commit, but not the tag before it, nor the fifty commits in between. How much of the repository does a job actually get? (the card)
What a job's copy contains#
GitLab tells the runner exactly which refs to fetch, and they depend on the kind of pipeline:
| Pipeline | What is fetched | So the job has |
|---|---|---|
| Branch pipeline | the pipeline's commit, and its own branch | origin/<branch>, but no other branch; on a feature branch, no origin/main |
| Tag pipeline | the pipeline's commit, and that tag | that one tag, so git describe can't see older ones |
| Merge request pipeline | the pipeline's commit only, as refs/pipelines/<id> | no branch names at all |
The runner then checks out the pipeline's commit, not a branch, so Git has no current branch. Git calls this a detached HEAD. Take names from variables instead:
CI_COMMIT_REF_NAMEis the branch or tag being built.CI_COMMIT_BRANCHis set only in branch pipelines.CI_MERGE_REQUEST_TARGET_BRANCH_NAMEis the target, in merge request pipelines.
Shallow clones#
A shallow clone fetches only the newest commits. New projects fetch 20,
set under Settings › CI/CD › General pipelines › Git shallow clone, and
the GIT_DEPTH variable overrides that. MegaCorp's base template sets
GIT_DEPTH: "20" for every job that includes it.
Three kinds of script need more history than that:
- Comparing with an older commit, such as
git diffagainst a merge request's base, fails when that commit is older than the clone. - Reading tags, with
git describeor a version made from tags, needs the tags and the commits between them. - Running old commits. GitLab's documentation warns that a retried or
queued job may find its commit outside the clone, and the log then says
unresolved reference. A depth of 1 makes this likely.
Fix it in the one job that needs history. Raise GIT_DEPTH for that job, or
fetch what the script needs, for example with git fetch --unshallow.
As text
- Does the script use another branch, such as main? Yes: That branch isn't in the job's copy; fetch it first. No: the next step.
- Does it use an older commit, such as a merge request's base? Yes: The commit may be older than GIT_DEPTH; fetch more history. No: the next step.
- Does it read tags, or run git describe? Yes: A tag pipeline fetches one tag, and other pipelines none. No: the next step.
- Does it ask Git for the current branch? Yes: There is none; use CI_COMMIT_REF_NAME. No: the next step.
- Check GIT_STRATEGY; with none or empty, the job gets no fresh code at all
Variables that shape the clone#
| Variable | What it does | Default |
|---|---|---|
GIT_STRATEGY | clone starts fresh; fetch reuses the last copy where the executor keeps one; none and empty skip Git, for jobs that only use artifacts | the project's setting |
GIT_DEPTH | how many commits to fetch | the project's setting: 20 in new projects |
GIT_CHECKOUT | "false" fetches without checking out the pipeline's commit | "true" |
GIT_CLEAN_FLAGS | how git clean tidies a reused copy; none skips it | -ffdx |
GIT_FETCH_EXTRA_FLAGS | flags added to git fetch | --prune --quiet |
GIT_SUBMODULE_STRATEGY | normal fetches submodules, recursive their submodules too | submodules aren't fetched |
On the Kubernetes executor each job gets a new pod, so fetch usually has no
earlier copy to reuse, and clones anyway.
Submodules#
A submodule is another repository placed inside yours, pinned to one commit. The runner
fetches submodules only when GIT_SUBMODULE_STRATEGY is normal or
recursive. For them to work in CI:
- In
.gitmodules, point at submodules on the same GitLab with a relative URL, such as../../devops/ci-tools.git, or with an HTTPS URL. A relative URL can resolve wrongly in forks. - For SSH URLs, set
GIT_SUBMODULE_FORCE_HTTPS, so the runner fetches them over HTTPS. - The job token fetches them. The person who started the pipeline needs at least the Reporter role in the submodule's project, and that project must allow your project's job token (chapter 28).
GIT_SUBMODULE_DEPTHsets their depth separately fromGIT_DEPTH.
If a job with GIT_STRATEGY: fetch fails with fatal: run_command
returned non-zero status, GitLab's documentation suggests switching to
GIT_STRATEGY: clone.
- Do this
- If a job reads Git history, for example the commits between two points or a version
from tags, set
GIT_DEPTHfor that job. Set it to0for a full clone. - Why
- By default a job gets a shallow copy of the repository: only the latest commits, and no tags from before them.
- Watch out
- A command that needs older history fails or gives a wrong answer in that shallow copy, and the error does not say why. The card history missing shows the symptom.
- Say what a job's clone contains, for each kind of pipeline.
- Give a job the history it needs, and no more.
- Explain why a merge request pipeline may be testing a commit that is on no branch.
Spot the bugThe generator that fails on big merge requests#
In megacorp/data/mono, generate-pipeline works in most merge
request pipelines. It fails in a few, always on long-lived branches with dozens of
commits, and the log shows that git diff can't find a commit. mono's clone
depth is the project default of 20:
#!/usr/bin/env bash
# ci/generate.sh in megacorp/data/mono
# Writes one child-pipeline job per service that changed. Its output is the YAML
# that the service-pipelines trigger job runs. Used by the chapter "Pipelines as building
# blocks".
set -euo pipefail
base="${CI_MERGE_REQUEST_DIFF_BASE_SHA:-HEAD~1}"
changed=$(git diff --name-only "$base" HEAD -- services/ | cut -d/ -f2 | sort -u)
echo "stages: [build]"
for svc in $changed; do
cat <<EOF
build-${svc}:
stage: build
image: registry.example.com/megacorp/devops/ci-tools:3.2
script:
- make -C services/${svc} build
EOF
done
# a child pipeline needs at least one job, even when nothing changed
if [ -z "$changed" ]; then
printf 'no-changes:\n stage: build\n script: [echo "no service changed"]\n'
fiShow the answer
A merge request pipeline fetches only the pipeline's commit, and only the newest 20
commits of its history. CI_MERGE_REQUEST_DIFF_BASE_SHA is the base of the merge
request's diff. On a branch with 20 or more commits, that base is older than the clone, so
git diff can't find it. Give this job more history: raise
GIT_DEPTH for it, or run git fetch --unshallow before the
diff.
The other mistake is the fallback, HEAD~1, used in every other pipeline. It
compares with the previous commit only, so a push of three commits builds just the
services changed in the last one. For pushes, CI_COMMIT_BEFORE_SHA holds the
branch's tip before the push. It is all zeros for new branches, schedules and manual runs,
so the script still needs a fallback. See shallow clones.
The scanning landscape#
A scanner is a job that reads something your pipeline has, looking for known problems. It might read the source code, the libraries the code uses, the image the pipeline built, or the running application. Enterprise pipelines often run five or six scanners. Each one writes a security report, and GitLab shows the findings in merge requests and on security pages, if your tier includes them.
Three questions explain almost any scanner: what does it read, who put it in my pipeline, and what can it block?
A merge request touches the payment code, and its pipeline ran six scanner jobs. The merge request page shows no findings at all, and the reviewer asks whether the scans found nothing or reported nothing. You can't tell which. Where do a scanner's results go, and who reads them? (the card)
What each scanner reads#
| Scanner | Reads | Looks for | Tier |
|---|---|---|---|
| SAST | the source code | risky code, such as a database query built from user input | Free |
| Secret detection | the source code | passwords, keys and tokens committed by mistake | Free |
| IaC scanning | infrastructure files, such as Terraform | insecure cloud and cluster settings | Free |
| Dependency scanning | the dependencies, as an SBOM | libraries with known vulnerabilities | Ultimate |
| Container scanning | the built image | vulnerable packages inside the image | Free |
| DAST and API testing | the running application | weaknesses that show only when the app runs | Ultimate |
An SBOM, a software bill of materials, is a list of every library a project uses, with versions. A free tier can run a scanner and still not show its findings anywhere except the job itself.
Who put it in your pipeline#
A scanner job arrives in one of four ways, and the way decides who can change it:
| Arrives through | How you recognise it | Who can change it |
|---|---|---|
| A GitLab template | an include: template: line, in your file or a central one | whoever owns that file; a project can adjust a job by redefining it under the same name |
| A central template or component | an include: of your platform team's files | the platform team |
| A scan execution policy | jobs that are in no file, with names such as secret-detection-1 | only the security team, through the policy project |
| A pipeline execution policy | jobs in the stages .pipeline-policy-pre and .pipeline-policy-post | only the security team |
MegaCorp uses all four. Its central template includes GitLab's scanners, and its security policy project adds more to every pipeline:
include:
- template: Security/SAST.gitlab-ci.yml
- template: Security/Secret-Detection.gitlab-ci.yml
- template: Security/Dependency-Scanning.gitlab-ci.yml
- template: Security/Container-Scanning.gitlab-ci.ymlscan_execution_policy:
- name: Secret detection everywhere
description: Run secret detection in every pipeline on every branch.
enabled: true
rules:
- type: pipeline
branches: ["*"]
actions:
- scan: secret_detection
- name: Nightly dependency scan
description: Scan dependencies on the default branch every night.
enabled: true
rules:
- type: schedule
branches: [main]
cadence: "0 2 * * *"
actions:
- scan: dependency_scanningWhere the findings appear#
| Place | What it shows | Tier |
|---|---|---|
| The job's log and artifacts | the raw report, as a JSON file | every tier |
| The pipeline's Security tab | every finding from that pipeline | Ultimate |
| The merge request's Reports tab | what the merge would add or fix | Ultimate |
| The vulnerability report | vulnerabilities on the default branch | Ultimate |
GitLab calls a problem on a branch a finding. It becomes a vulnerability when the branch is merged into the default branch. Findings expire with the job's artifacts, or after 90 days by default on self-managed GitLab and 30 days on GitLab.com.
What a scanner can block#
Three different things can stop a change, and they are easy to mix up:
- The scanner job fails. The scanner crashed, or couldn't read what it needed. GitLab's scanner jobs are allowed to fail, so this shows only as a warning, and the pipeline stays green.
- The merge request needs extra approval. A merge request approval policy reads the findings and demands approval, for example from the security team (chapter 36).
- A third-party gate fails the job on purpose, such as a quality gate that isn't met (chapter 35).
- Name what each kind of scanner reads and produces, and which tier shows its findings where.
- Tell a scan that found nothing from one whose report never arrived.
- Read a merge request's security widget and know what it can and cannot show.
Spot the bugWhose job is it?#
On MegaCorp's production GitLab, which has Ultimate, a main pipeline of
payments-api shows these jobs, among others:
secret_detection
container_scanning
secret-detection-1
policy-sbomUse the specimen files above, and chapter 18.
Show the answer
secret_detectioncomes from GitLab's secret detection template, which MegaCorp'ssecurity.ymlincludes. The platform team owns that include. A project could redefine the job by name.container_scanningcomes from GitLab's container scanning template, andsecurity.ymlreplaces its rules andneeds. The platform team owns it.secret-detection-1comes from the scan execution policy "Secret detection everywhere". The numbered suffix gives it away. Only the security team can change it, in the policy project.policy-sbomcomes from the pipeline execution policy "MegaCorp guardrails", which puts it in.pipeline-policy-post. Only the security team can change it.
GitLab's own scanners#
GitLab's scanners arrive as templates. Each template adds jobs whose rules decide when they appear, and four behaviours explain most surprises:
- By default they run in branch pipelines, not merge request pipelines.
- Some add a job only when matching files exist, or only on Ultimate.
- They are all allowed to fail, so a broken scanner shows only a warning.
- Their jobs sit in the
teststage, so a pipeline must have one.
container_scanning runs in every merge request pipeline and always passes. Today you notice it scans alpine:latest, not the image the pipeline built. The template was included months ago and adjusted by the central team. Somewhere between their adjustment and GitLab's template, the image name got lost. (the card)
What the templates add#
Each Security/ template that MegaCorp includes contains a single line: an
include of its Jobs/ twin, which holds the jobs.
| Template | Jobs it adds | When a job appears |
|---|---|---|
Jobs/SAST.gitlab-ci.yml | one per analyzer, such as semgrep-sast | only if files that analyzer reads exist, such as *.java or *.py; gitlab-advanced-sast also needs Ultimate |
Jobs/Secret-Detection.gitlab-ci.yml | secret_detection | in branch pipelines |
Jobs/Dependency-Scanning.gitlab-ci.yml | one per package manager, such as gemnasium-maven-dependency_scanning | only on Ultimate; its Gemnasium analyzer is deprecated |
Jobs/Container-Scanning.gitlab-ci.yml | container_scanning | in branch pipelines; it scans the image named in CS_IMAGE |
Every job's first rule checks its toggle, such as SAST_DISABLED. Every job
has allow_failure: true. The secret detection and container scanning reports
can be downloaded only by people with the Developer role or above.
On the local GitLab, which is the Free edition, these templates add just two jobs to
payments-api's main pipeline: secret_detection and
container_scanning. There is no SAST job, because the specimen repository
holds only a pom.xml, and no .java files. There is no dependency
scanning job either, because those jobs need Ultimate.
The merge request gap#
The scanner jobs' rules follow the same three steps:
- In a merge request pipeline, run only if
AST_ENABLE_MR_PIPELINESis"true". - If that variable is on and the branch has an open merge request, skip the job in the branch pipeline, so that nothing is scanned twice.
- Otherwise, run in the branch pipeline.
So by default the scanners run only in branch pipelines. On the local GitLab, a push to
a payments-api merge request branch produced both pipelines. The branch pipeline had
secret_detection; the merge request pipeline didn't. Anyone reading only the
merge request pipeline sees no secret detection at all.
As text
- Is this a merge request pipeline? Yes: Scanners skip merge request pipelines unless AST_ENABLE_MR_PIPELINES is "true". No: the next step.
- Is the scanner's toggle set, in YAML or in settings? Yes: A toggle such as SAST_DISABLED removes the job. No: the next step.
- Is it SAST, and the repository has no files that the analyzer reads? Yes: SAST adds jobs only for the languages it finds. No: the next step.
- Is it dependency scanning, on Free or Premium? Yes: Those jobs appear only on Ultimate. No: the next step.
- Check whether someone redefined the job and replaced its rules
A pipeline without a test stage fails in a different way. It can't be
created at all, because the scanner jobs name a stage that doesn't exist
(chapter 24).
Changing a template's job#
To change a template's job, define a job with the same name. GitLab merges the two, and your keys win. MegaCorp moves container scanning into its own stage, points it at the image it has just built, and replaces its rules:
variables:
CS_IMAGE: $IMAGE_REF
container_scanning:
stage: scan
needs: [image-build]
rules:
- if: $CONTAINER_SCANNING_DISABLED == "true"
when: never
- !reference [.rules, never-on-schedule]
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHReplacing rules replaces all of them, because lists are never merged
(chapter 14). MegaCorp's rules keep the toggle, and add merge
request pipelines, which the template alone would skip. That's why the local captures show
container_scanning in payments-api's merge request pipeline, but not
secret_detection.
dependencies: [], and MegaCorp's version doesn't replace it.
An empty dependencies blocks every artifact download, dotenv reports included,
even with needs: [image-build]. On the local GitLab 19.3, a probe job
with both keys got an empty IMAGE_REF, and so an empty CS_IMAGE.
So MegaCorp's container scan never receives the image it was meant to scan. Adding
dependencies: [image-build] fixed the probe. See
the symptom card.- Do this
- Include GitLab's scanner templates and leave their jobs alone where you can. To change one, set only the variables that the template documents.
- Why
- The templates change with every GitLab release. A job you copied and edited stops getting those changes.
- Watch out
- The templates' jobs run in branch pipelines. If your workflow uses merge request pipelines, check that your merge requests are scanned too. The card no scans in merge requests shows the gap.
- Include a GitLab scanner template and know which of its jobs will run, in which pipelines.
- Adjust a template's job without losing what the template set.
- Find why a scanner ran against the wrong target, or not in merge requests at all.
Spot the bugThe scan the reviewers never saw#
A reviewer opens a payments-api merge request and checks its pipeline. There is no
secret_detection job. A colleague finds one in a different pipeline, for the
same commit, on the merge request's branch.
include:
- template: Security/SAST.gitlab-ci.yml
- template: Security/Secret-Detection.gitlab-ci.yml
- template: Security/Dependency-Scanning.gitlab-ci.yml
- template: Security/Container-Scanning.gitlab-ci.ymlShow the answer
GitLab's secret detection template runs its job in branch pipelines, and in merge
request pipelines only when AST_ENABLE_MR_PIPELINES is "true".
Nobody set it, so the scan ran in the branch pipeline only.
Set AST_ENABLE_MR_PIPELINES: "true", in the YAML or in settings. The job
then runs in merge request pipelines, and is skipped in the branch pipeline while a merge
request is open. Alternatively, redefine secret_detection with rules of your
own, as MegaCorp did for container scanning. See the merge request
gap.
Third-party scanners#
Many organisations also run tools that GitLab doesn't ship, such as SonarQube, Snyk or
Trivy. They all follow one pattern: a job runs the tool, the tool's exit code decides
whether the job fails, and a report file may carry the findings into GitLab. Find those
three things, and the job's allow_failure and needs, and you
know what the tool can really stop.
The SonarQube quality gate failed on Monday's merge request. It merged anyway, and deployed. The job showed a red cross, and a small "allowed to fail" beside it that nobody read. A scanner that cannot block is a scanner that reports. Which did the team want? (the card)
Tool, exit code, report, gate#
As text
- The job runs the tool's command-line program
- Does the tool exit with an error when it finds problems? No: The job passes whatever it finds; many tools need an option for this. Yes: the next step.
- Is the job allowed to fail? Yes: A failure shows only as a warning, and blocks nothing. No: the next step.
- Do the jobs that matter wait for it? No: With needs, a deploy can start without waiting for the scan. Yes: the next step.
- A failure fails the pipeline, and jobs that wait for the scan don't run
A report file lets GitLab show the tool's findings, instead of leaving them in the job log or on the tool's own website:
| Report | For | Where GitLab shows it |
|---|---|---|
codequality | lint and code-quality findings, in a simple JSON format | merge requests, and the pipeline's Code Quality tab (Premium) |
sarif | security findings from any tool that writes SARIF 2.1.0 | the Security tab and the vulnerability report (Ultimate, since 19.2) |
junit | test results | the pipeline's Tests tab, and the merge request's test summary |
For code quality, GitLab's preferred route is exactly this: run your own linter and import its report. GitLab's built-in CodeClimate template is deprecated.
MegaCorp's SonarQube job#
MegaCorp's platform team ships SonarQube as a component (chapter 16):
sonar-scan:
stage: $[[ inputs.stage ]]
image: registry.example.com/megacorp/devops/ci-tools/sonar-scanner:6
variables:
SONAR_PROJECT_KEY: $[[ inputs.project_key ]]
SONAR_QUALITYGATE_WAIT: "$[[ inputs.quality_gate ]]"
script:
- sonar-scanner -Dsonar.projectKey="$SONAR_PROJECT_KEY" -Dsonar.host.url="$SONAR_HOST_URL"
rules:
- if: $SKIP_SONAR == "true"
when: never
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
allow_failure: trueRead it with the pattern:
- Tool:
sonar-scanner, which sends the code to the Sonar server for analysis. - Exit code: the component's
quality_gateinput setsSONAR_QUALITYGATE_WAIT. It is meant to make the scanner wait for Sonar's quality gate, and fail the job when the gate fails. - Report: none. The findings stay on the Sonar server.
- Gate:
allow_failure: true, so even a failed gate is only a warning.
It also needs SONAR_TOKEN, which is protected, so in merge request
pipelines from feature branches the scan can't sign in (chapter 27).
- Do this
- Make a scanner tool exit with an error on the findings you care about, and keep its report as an artifact.
- Why
- A scanner job can only block a merge if the job fails. The report lets people see why it failed.
- Watch out
- Do not set
allow_failure: trueon a scanner unless its result is only advice. With it, the job can fail and the merge still goes through. The card the gate didn't stop a deploy shows the result.
- Add a third-party scanner as a job whose failure means what the team wants it to mean.
- Publish its report so GitLab shows the findings where reviewers look.
- Say whether a scanner job can block a merge, and make it block when it should.
Spot the bugThe quality gate that couldn't stop a deploy#
A change that fails MegaCorp's Sonar quality gate is merged into payments-api's
main. The pipeline goes green, and deploy-staging ships the
change. sonar-scan shows an orange warning.
deploy-staging:
extends: .deploy
needs: [image-build]
variables:
ENVIRONMENT: staging
rules:
- !reference [.rules, never-on-schedule]
- !reference [.rules, default-branch]
deploy-prod:
extends: .deploy
needs: [image-build]
variables:
ENVIRONMENT: production
rules:
- if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
when: manualShow the answer
sonar-scanhasallow_failure: true, so its failure never fails the pipeline.deploy-staginghasneeds: [image-build]. It starts as soon as the image is built, without waiting for jobs in theteststage, so it doesn't wait forsonar-scanat all (chapter 24).
To make the gate stop deploys, both would have to change: the scan must be allowed to fail the pipeline, and the deploy must wait for it. Whether the gate should block is a decision for the teams that own the pipeline. See the pattern.
Policies and gates#
Security policies are rules the security team writes once for many projects. They need Ultimate, and come in three kinds:
- scan execution policies add scanner jobs
- pipeline execution policies add any jobs, or replace the pipeline
- merge request approval policies demand approval when scans find problems
Policies win over anything a project writes. A toggle such as
SAST_DISABLED skips a project's own scanner jobs, never a policy's. Turning
a mandated scan off is a decision for the security team, not a change to the YAML.
This morning every pipeline in the group has two new jobs, secret-detection-1 and policy-sbom, in a stage called .pipeline-policy-post. No file in any project mentions them. Your toggles don't remove them. Someone with a different kind of power put them there. (the card)
Three kinds of policy#
| Policy | What it does | How you recognise it |
|---|---|---|
| Scan execution | adds scanner jobs to pipelines, or runs scans on a schedule | jobs such as secret-detection-1, in test, or in scan-policies when there is no test stage |
| Pipeline execution | adds any jobs, or replaces the project's pipeline | jobs in .pipeline-policy-pre, which runs first, and .pipeline-policy-post, which runs last |
| Merge request approval | requires approval when scans find problems | an approval rule on the merge request, named after the policy |
Pipeline execution policies have a scheduled variant Since 19.2. It creates pipelines of its own on a schedule, in every project it covers, and runs only the policy's jobs, never the project's.
All three live in one file, .gitlab/security-policies/policy.yml, in a
separate security policy project linked to the group. A project can hold
at most five policies of each kind. Chapter 18 shows MegaCorp's first
two. This is the third:
approval_policy:
- name: Block new critical vulnerabilities
description: A merge into main that adds a critical finding needs security's approval.
enabled: true
rules:
- type: scan_finding
branches: [main]
scanners: [sast, secret_detection, dependency_scanning, container_scanning]
vulnerabilities_allowed: 0
severity_levels: [critical]
vulnerability_states: [new_needs_triage]
actions:
- type: require_approval
approvals_required: 1
group_approvers: [megacorp/security]Toggles against policies#
GitLab's scanner templates switch off with variables: SAST_DISABLED,
SECRET_DETECTION_DISABLED, CONTAINER_SCANNING_DISABLED and
others. GitLab's documentation is exact about their limit: "Skipping jobs does not prevent
any security jobs defined by scan execution policies from running." Two more rules follow:
- A project can't override a policy's scanner jobs, even by using the same job name.
- In a project under a scan execution policy, the scanners' exclude settings, such as
SAST_EXCLUDED_PATHS, are preset, and only a policy can change them.
So when a team wants a mandated scan off, the answer isn't in their YAML. The security team decides, for example by narrowing the policy's scope.
When a merge request needs the security team#
MegaCorp's approval policy looks at merges into main. If a scan finds a new
critical vulnerability, one member of megacorp/security must approve. The
rule also applies when GitLab can't check it:
As text
- Did a scan find a new critical vulnerability? Yes: The rule is working; fix the finding, or ask for approval. No: the next step.
- Did a scanner the rule names produce no report? Yes: The rule can't be checked, and by default that also requires approval. No: the next step.
- Does the policy target this branch? No: Look for another policy; each shows as its own approval rule. Yes: the next step.
- Open Secure › Policies, and read the policy's rule
The second case surprises people. By default a policy fails closed: "Invalid
or unenforceable rules of a policy require approval." So a scanner that was skipped or
crashed doesn't let a merge request slip through. It blocks it until someone approves.
The policy can say fallback_behavior: fail: open instead.
Finding the policies#
Anyone who can see a project can open its policies at Secure › Policies. The list shows each policy and whether it is enforced. Changes go through a merge request in the policy project:
- Recognise a job that a policy added, and find the policy under Secure › Policies.
- Tell the four kinds of policy apart, and say what each can add, require or block.
- Know what your file can and cannot change about a policy's jobs.
Spot the bugThe toggle that only half worked#
payments-api's pipelines run two secret detection jobs, and the team wants to save
time. They add this to their .gitlab-ci.yml:
variables:
SECRET_DETECTION_DISABLED: "true"secret_detection disappears from their pipelines, but
secret-detection-1 still runs.
Show the answer
secret_detection comes from GitLab's template, included by MegaCorp's
security.yml, and the template honours the toggle.
secret-detection-1 comes from the scan execution policy "Secret detection
everywhere". Toggles skip only a project's own jobs, never a policy's.
Removing the second job isn't the team's decision. They can ask the security team to change the policy's scope. Having the job twice is expected: GitLab runs both, because the two can use different settings. See toggles against policies.
Building Java, Node and front ends#
A build job is an ordinary job with the right image. What makes builds confusing in a big organisation is everything around the compiler:
- where the dependencies come from: a company mirror, and a cache
- what the build produces, and where it goes: an artifact, a package registry, an image or a bucket
- which version number the result carries
Most "works on my machine" failures come from the first. Most "which version is running?" confusion comes from the last.
The release job publishes payments-api 1.8.0-SNAPSHOT. The tag was v1.8.0. The Maven build took the version from pom.xml, which nobody bumps, and the pipeline never told it otherwise. Where does a build's version number come from, and where should it? (the card)
A Maven build at MegaCorp#
MegaCorp's Maven jobs start from one hidden job. It picks the JDK image, caches the local repository, and writes a Maven settings file before the script runs:
.maven-base:
extends: .megacorp-base
image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
variables:
JAVA_VERSION: "21"
MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
cache:
key:
files: [pom.xml]
paths: [.m2/repository]
before_script:
# a job's own before_script replaces default:before_script, so re-add the functions
- !reference [.snippets, functions]
- !reference [.snippets, maven_settings]The settings file comes from a function in the ci-tools image. It sends every download through MegaCorp's Nexus server, a company copy of the public repositories:
mc_maven_settings() {
cat <<EOF
<settings><mirrors><mirror><id>nexus</id><mirrorOf>*</mirrorOf><url>${MC_NEXUS_URL:-https://nexus.example.com/repository/maven-public/}</url></mirror></mirrors></settings>
EOF
}Three variables decide how Maven runs, and only one is in the YAML:
| Variable | Set in | Effect |
|---|---|---|
MAVEN_CLI_OPTS | the megacorp group's CI/CD settings | batch mode, and the settings file above; it overrides any value a project writes in YAML (chapter 26) |
MAVEN_OPTS | the template | keeps Maven's local repository inside the project, so that it can be cached |
MC_NEXUS_URL | payments-api's CI/CD settings | which Nexus the mirror points at |
maven-build packages the jar without running the tests, and keeps it as an
artifact. maven-test runs the tests twice, on JDK 17 and 21, with
parallel: matrix (chapter 14).
A Node front end at MegaCorp#
The Node templates install with npm ci, from a cache keyed on
package-lock.json, and build into dist/
(chapter 31). web-portal's whole pipeline is three jobs:
node-build:
extends: .node-build
node-test:
extends: .node-test
deploy-site:
extends: .publish-site
needs: [node-build]
rules:
- !reference [.rules, never-on-schedule]
- !reference [.rules, default-branch]deploy-site copies dist/ to an S3 bucket and clears the
CloudFront cache (chapter 38). web-portal has no
.gitlab-ci.yml of its own: its settings point at this file
(chapter 18).
As text
- Does it download dependencies? Yes: CI downloads through the company mirror and proxy, which may lack or block a package. No: the next step.
- Does it need a file that isn't in Git, such as a local settings file? Yes: The job has only the repository, artifacts and variables. No: the next step.
- Is the tool's version different from yours? Yes: The job uses the image's version; check the image's tag. No: the next step.
- Did it restore an old cache? Yes: Check the cache key, or clear the cache (chapter 24). No: the next step.
- Compare the job's variables with your shell's; settings variables may differ
Which version is it?#
Three numbers can describe one build, and they often disagree:
| Number | Comes from | payments-api's example |
|---|---|---|
| The version in the build file | pom.xml or package.json, as committed | 1.8.0-SNAPSHOT |
| The Git tag | the release tag, in CI_COMMIT_TAG, which is set only in tag pipelines | v1.5.0 |
| The image tag | the commit, in CI_COMMIT_SHA | a 40-character commit hash |
MegaCorp's mc tool can work out a version in any pipeline: the tag if there
is one, otherwise 0.0.0- followed by the short commit hash.
version)
echo "${CI_COMMIT_TAG:-0.0.0-${CI_COMMIT_SHORT_SHA}}"
;;Nothing in the Maven templates uses it, though. The exercise shows the result.
Test results and coverage#
- Test reports in JUnit XML, saved as
artifacts: reports: junit, fill the pipeline's Tests tab and the merge request's test summary. The summary lists newly failed tests. A report never changes the job's status: the test command's exit code does. - A coverage percentage comes from the job log. The
coveragekeyword holds a regular expression, and GitLab shows the first number it matches in the merge request. - Line-by-line coverage comes from a Cobertura or JaCoCo file saved as
artifacts: reports: coverage_report. GitLab marks the lines in the merge request's diff.
Publishing packages to GitLab's registries#
Some teams publish libraries to GitLab's own package registry, using the job token as the password.
Maven. A settings file sends the job token as a header:
<settings>
<servers>
<server>
<id>gitlab-maven</id>
<configuration>
<httpHeaders>
<property>
<name>Job-Token</name>
<value>${CI_JOB_TOKEN}</value>
</property>
</httpHeaders>
</configuration>
</server>
</servers>
</settings>The POM's distribution repository is
${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/maven, and a job runs
mvn deploy. To refuse a version that is already published, a group Owner
turns off Allow duplicates under Settings › Packages and
registries.
npm. The job writes an .npmrc that points the package's
scope at the project's registry, with the job token:
echo "@megacorp:registry=https://${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/" > .npmrc
echo "//${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN}" >> .npmrc
npm publishThe scope is the top-level group of the project that hosts the package, in lower case:
@megacorp. A package name can belong to only one project per top-level
group. Publishing it from a second project fails, even with a new version.
- Do this
- Build once, in one job, and pass the result to later jobs as an artifact. Test and
package that one result. Publish test results with
artifacts: reports: junit. - Why
- Every later job then works on the same build, and test results show in the merge request.
- Watch out
- Use
npm ci, notnpm install, and give Maven--batch-mode. Otherwise a build can install different versions from one run to the next, and Maven fills the log with progress lines.
- Read MegaCorp's build jobs for Java, Node and a front end, and say what each caches, keeps and reports.
- Give a build its version from the pipeline, not from a file nobody bumps.
- Build once and pass the result on, instead of rebuilding in every job.
Spot the bugThe release that called itself a snapshot#
After payments-api's v1.5.0 release, the support team checks the running
service. Its version page, which reads the jar's version, says
1.8.0-SNAPSHOT. The image is the one that the v1.5.0 tag
pipeline built.
<artifactId>payments-api</artifactId>
<version>1.8.0-SNAPSHOT</version>.maven-build:
extends: .maven-base
stage: build
script:
- mvn $MAVEN_CLI_OPTS -DskipTests package
artifacts:
paths: [target/*.jar]1.8.0-SNAPSHOT come from, and what would make the jar
carry the release's version?Show the answer
Maven takes the version from pom.xml, which says
1.8.0-SNAPSHOT. The build job runs mvn package without changing
it, so every pipeline, the tag pipeline included, builds a jar with that version. The tag
names the commit, but never reaches Maven.
Pass the version into the build in tag pipelines, for example from mc
version, which returns CI_COMMIT_TAG there. Or make each release a
commit that sets the POM's version, and tag that commit. See which
version is it.
GitLab, AWS and Azure#
Jobs that build and deploy to AWS or Azure need permissions there. The modern way, and MegaCorp's way, stores no password anywhere. Each job trades its ID token for short-lived access, as chapter 28 showed: temporary keys for an AWS role, or a sign-in as an application in Microsoft Entra ID, Azure's identity service.
This chapter covers:
- what must match on each side, and who owns each part
- which credentials each cloud's command-line tool really uses
- working with several AWS accounts or Azure subscriptions
- ECR and ACR, secrets from each cloud, and the usual deploy targets
- what to check, in order, when AWS or Azure says no
The AWS login works on every branch pipeline and fails on every tag pipeline with Not authorized to perform sts:AssumeRoleWithWebIdentity. Same job, same role, same token keyword. The role's trust policy names branches. A tag is not a branch, and the token says so. (the card)
Cloud words in two minutes#
If a cloud is new to you, these are the only words this chapter needs:
| Word | Meaning |
|---|---|
| Account | a separate AWS space with its own resources and bill, named by a 12-digit number such as 123456789012 |
| Region | a location where resources live, such as eu-west-2 (London); jobs set it with AWS_REGION |
| IAM | AWS's identity service: who may do what |
| Role | an identity with permissions and no password; a person or a system assumes it for a while |
| Trust policy | the rules on a role that say who may assume it |
| Permission policy | the rules on a role that say what it may do once assumed |
| ARN | the full name of anything in AWS, such as arn:aws:iam::123456789012:role/gitlab-ecr-push |
| STS | the Security Token Service, which hands out temporary keys for a role |
| ECR | the Elastic Container Registry, where images are stored |
| S3 and CloudFront | file storage, and the network that serves those files quickly worldwide |
| ECS, EKS and Lambda | three places to run code: AWS's container service, Kubernetes clusters, and single functions |
| Secrets Manager and Parameter Store | two AWS services that store secrets and settings |
| Word | Meaning |
|---|---|
| Tenant | an organisation's directory in Microsoft Entra ID, named by a GUID such as 00000000-0000-0000-0000-000000000000 |
| Subscription | a separate Azure space with its own resources and bill, also named by a GUID |
| Resource group | a named folder for resources inside a subscription, such as rg-delivery-prod |
| Microsoft Entra ID | Azure's identity service: who may sign in, and as what; it used to be called Azure Active Directory |
| App registration | an identity for software, named by its client ID; its service principal is what receives permissions |
| Managed identity | an identity that Azure creates and looks after for a resource, with no secret to store |
| Federated credential | a rule on an app registration or managed identity that says which outside tokens it accepts instead of a secret |
| Role assignment | a role, such as AcrPush, given to an identity at a scope: a subscription, a resource group or one resource |
| ACR | Azure Container Registry, where images are stored |
| AKS and Container Apps | two places to run containers: Kubernetes clusters, and a service that runs containers without a cluster to manage |
| Key Vault | Azure's store for secrets, keys and certificates |
What must match#
A job gets into a cloud only when several pairs line up. Each pair has one half in GitLab and the other half in the cloud, and a different team often owns each half.
On AWS, five pairs must match:
At MegaCorp the GitLab halves live in the templates, which the platform team owns:
.image-build:
extends: .megacorp-base
stage: package
image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
id_tokens:
MC_ID_TOKEN:
aud: https://gitlab.example.com
variables:
AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
before_script:
- !reference [.snippets, functions]
- !reference [.snippets, aws_login]
script:
- mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
- buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
- buildah push "$IMAGE:$CI_COMMIT_SHA"
- !reference [.snippets, acr_push]
- echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
artifacts:
reports:
dotenv: build.envThe AWS halves are set in the AWS account itself. This file describes MegaCorp's:
# AWS accepts ID tokens signed by this GitLab, for this audience.
oidc_provider:
arn: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
url: https://gitlab.example.com
audiences: [https://gitlab.example.com]# Assumed by image-build, and by the ecr-push component, through mc_aws_login.
ecr_push_role:
arn: arn:aws:iam::123456789012:role/gitlab-ecr-push
trust_policy:
Effect: Allow
Principal:
Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
Action: sts:AssumeRoleWithWebIdentity
Condition:
StringEquals:
gitlab.example.com:aud: https://gitlab.example.com
StringLike:
gitlab.example.com:sub: project_path:megacorp/*:ref_type:branch:ref:*
permissions: push and pull images in every ECR repository in the accountA mismatch in the first four pairs stops the login: STS refuses, and no keys come back. A mismatch in the fifth lets the login succeed, and a later command fails with an access-denied error. That difference tells you where to look.
On Azure, the pairs are much the same:
| In GitLab | In Entra ID and Azure | Matched how |
|---|---|---|
the instance's URL, which becomes the token's iss | the federated credential's issuer, https://gitlab.example.com | exactly |
aud in the job's id_tokens: | the credential's single audience | exactly |
the project, the kind of ref and the ref, which make the token's sub | the credential's subject | exactly, with no wildcards |
the client ID and tenant ID that az login names | the app registration | exactly |
| what the job does once signed in | role assignments, such as AcrPush on the registry | by scope |
This file describes MegaCorp's application for image pushes, and what it may do:
# The identity that image jobs sign in as. Each federated credential names one kind of
# pipeline that may use it.
acr_push_app:
display_name: gitlab-acr-push
client_id: 00000000-0000-0000-0000-000000000000
tenant_id: 00000000-0000-0000-0000-000000000000
federated_credentials:
- name: payments-api-main
issuer: https://gitlab.example.com
subject: project_path:megacorp/payments/payments-api:ref_type:branch:ref:main
audiences: [https://gitlab.example.com]# What the app may do once signed in, on one registry only. AcrPush pushes and pulls
# images. az acr login also has to look the registry up in Azure Resource Manager, which
# AcrPush doesn't allow; Microsoft documents the second role for that, and it can also
# change or delete the registry, which is one more reason to trust main only.
role_assignments:
- principal: gitlab-acr-push
role: AcrPush
scope: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-delivery-prod/providers/Microsoft.ContainerRegistry/registries/megacorpprod
- principal: gitlab-acr-push
role: Container Registry Contributor and Data Access Configuration Administrator
scope: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-delivery-prod/providers/Microsoft.ContainerRegistry/registries/megacorpprodThe big difference from AWS is the subject. Entra ID matches it exactly, with no
wildcards. So each project and branch that may sign in needs its own federated credential,
and one application holds at most 20 of them. MegaCorp pushes images to Azure only from
each service's main, which needs one credential per service. A wrong subject
is saved without complaint: the error shows only when a job tries to sign in.
How a MegaCorp job logs in#
The template runs snippets that call functions from the ci-tools image:
.snippets:
functions:
- source /opt/megacorp/lib/ci-lib.sh
aws_login:
- mc_aws_login "$AWS_ROLE_ARN"
acr_push:
- mc_acr_push "$IMAGE:$CI_COMMIT_SHA"
maven_settings:
- mkdir -p .m2
- mc_maven_settings > .m2/settings.xml# Exchange the job's OIDC ID token for temporary AWS credentials.
mc_aws_login() {
local role="${1:?role ARN required}"
local creds
creds=$(aws sts assume-role-with-web-identity \
--role-arn "$role" \
--role-session-name "gitlab-${CI_PROJECT_ID}-${CI_JOB_ID}" \
--web-identity-token "$MC_ID_TOKEN" \
--duration-seconds 3600 \
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
--output text) || { mc_log "AssumeRoleWithWebIdentity failed for $role"; return 1; }
read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< "$creds"
export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
mc_log "assumed $role"
}Step by step:
aws sts assume-role-with-web-identitysends the role's ARN and the ID token to STS.- STS checks the token and the trust policy. It returns three values: an access key, a secret key and a session token.
- The function exports them as
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEYandAWS_SESSION_TOKEN. Every laterawscommand in the job uses them. - They expire after 3600 seconds, which is one hour.
To see which identity a job is really using, add this line. It needs no permissions, and it prints the ARN of the role in use:
aws sts get-caller-identity# Sign in to Azure with the job's OIDC ID token, as the Entra application that trusts it.
mc_azure_login() {
local client_id="${1:?application (client) ID required}"
az login --service-principal -u "$client_id" -t "$MC_AZURE_TENANT_ID" \
--federated-token "$MC_ID_TOKEN" --output none \
|| { mc_log "az login failed for $client_id"; return 1; }
mc_log "signed in to Azure as $client_id"
}
# Push an image that is already built to Azure's registry as well. Only main's images go
# there: Entra matches a federated credential's subject exactly, and aks-prod deploys main.
mc_acr_push() {
local image="${1:?image required}"
if [ "${CI_COMMIT_BRANCH:-}" != "$CI_DEFAULT_BRANCH" ]; then
mc_log "not on $CI_DEFAULT_BRANCH, so not pushing to $MC_ACR_REGISTRY"
return 0
fi
mc_azure_login "$MC_ACR_PUSH_CLIENT_ID"
az acr login --name "${MC_ACR_REGISTRY%%.*}" --expose-token --query accessToken --output tsv \
| buildah login --username 00000000-0000-0000-0000-000000000000 --password-stdin "$MC_ACR_REGISTRY"
buildah push "$image" "docker://$MC_ACR_REGISTRY/${image##*/}"
mc_log "pushed $MC_ACR_REGISTRY/${image##*/}"
}Step by step:
- On any branch but
main,mc_acr_pushstops at once. Merge request and tag pipelines never sign in to Azure. az login --service-principal --federated-tokensends the ID token to Entra ID, with the application's client ID and the tenant ID. Both IDs come from themegacorpgroup's CI/CD variables.- Entra ID compares the token's issuer, subject and audience with the application's federated credentials. If one matches, the CLI is signed in as the application.
az acr login --expose-tokentrades that sign-in for a registry token, valid for 3 hours. Buildah logs in with it, under the all-zero user name that ACR expects, and pushes.
To see who a job is signed in as, add this line. It prints the subscription in use, and the identity that signed in:
az account showLogging in to AWS without a script#
The AWS CLI can do the exchange by itself. Put the ID token in a file, and set two variables that name the role and the file:
publish:
id_tokens:
AWS_ID_TOKEN:
aud: https://gitlab.example.com
variables:
AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-web-publish
AWS_ROLE_SESSION_NAME: gitlab-$CI_JOB_ID
script:
- echo "$AWS_ID_TOKEN" > /tmp/web-identity-token
- export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/web-identity-token
- aws sts get-caller-identityEvery aws command then assumes the role on its own. This works only when no
access keys are set, because keys come first in the order below.
Which credentials each CLI uses#
A job can have credentials from several places at once:
- credentials that a login step created
- keys that someone stored in CI/CD settings long ago
- an identity that belongs to the runner's machine
The two clouds' command-line tools deal with this differently.
The AWS CLI doesn't combine them. It takes the first it finds, in a fixed order:
As text
- Does the command pass --profile, or keys, on its command line? Yes: Those are used. No: the next step.
- Are AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set? Yes: Those keys are used, whether a login step exported them or they came from settings. No: the next step.
- Are AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE both set? Yes: The CLI assumes that role, with the token in the file. No: the next step.
- Does the image contain AWS configuration files? Yes: Their profile is used. No: the next step.
- Otherwise the role of the machine or pod is used, often the runner's own role
Two failures follow from this order:
- Old keys in settings win silently. A leftover
AWS_ACCESS_KEY_IDin a group's CI/CD variables beats any role, until a login step overwrites it. A job that skips the login then acts as whoever owns those keys. - A skipped login falls through to the runner. A job that sets its own
before_scriptreplaces the template's, so the login never runs (chapter 14). The job then acts as the runner's role, which at MegaCorp can pull images but not push them.
The Azure CLI doesn't search in an order. It uses the sign-in that az login
left in its local cache, under its configuration directory: $HOME/.azure,
unless AZURE_CONFIG_DIR names another. Each command runs against the current
subscription, which az account set --subscription changes. A machine's
managed identity is used only after az login --identity.
Two failures follow:
- A skipped sign-in usually fails loudly. In a fresh container the
cache is empty, so the first
azcommand stops and asks for a sign-in. - On a shell runner, the last sign-in lingers. The cache lives in the
runner user's home directory, so it survives from one job to the next. A job that skips
its own sign-in can act as whoever signed in before it. Give each job its own directory
with
AZURE_CONFIG_DIR, such as one inside the build folder.
Several accounts and subscriptions#
Large organisations split each cloud into several spaces, often one for shared tools and one for each environment.
A job can reach another AWS account in two ways:
- Direct trust. Each account has a role whose trust policy accepts the
ID token, and the job logs in to the account it needs. Each account checks the token's
subitself, so production can accept only release tags, for example. - Role chaining. The job logs in to one role, then uses it to assume a role in the other account. AWS limits a chained session to one hour, whatever the role allows, and asking for longer fails.
In Azure, one sign-in reaches every subscription in the tenant where the identity has a
role assignment. az account list shows them, and az account set
--subscription picks one. There's no chaining to worry about: what the identity may
do in each subscription depends only on its role assignments there.
MegaCorp needs none of this. Its pipelines touch only the tooling account, 123456789012, where ECR and the website's bucket live, and one registry in Azure. Deploys go through the GitOps repository, and a controller inside each cluster pulls the new image (chapter 45). The pipeline never holds a production cluster's credentials at all.
Registries: pushing images, and pulling job images#
An ECR registry's address contains the account and the region:
123456789012.dkr.ecr.eu-west-2.amazonaws.com. To push, a job logs in to AWS
first, then to the registry:
aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"The user name is always AWS, and the password this prints is valid for 12
hours.
An ACR registry's address is its name followed by .azurecr.io:
megacorpprod.azurecr.io. To push, a job signs in to Azure first, then asks
for a registry token:
az acr login --name megacorpprod --expose-token --query accessToken --output tsv \
| buildah login --username 00000000-0000-0000-0000-000000000000 --password-stdin megacorpprod.azurecr.ioThe user name is always the all-zero GUID, and the token is valid for 3 hours. Without
--expose-token, az acr login hands the token to a Docker daemon
itself, which a Buildah job doesn't have. The registry's admin user is off by default, and
Microsoft advises leaving it off.
Tagging and promoting images is the subject of chapter 39.
Pulling a job's own image: from either registry is different. The runner
pulls it before the job starts, so nothing in the job's script can help. The runner needs
its own way in.
Job images from ECR or ACR#
For ECR, GitLab's documented way is the ECR credential helper. It is a small program
that gets a registry password from AWS whenever the runner needs one. It must be installed
on the runner, and the runner needs AWS credentials of its own. A
DOCKER_AUTH_CONFIG variable then tells the runner to use it:
{ "credHelpers": { "123456789012.dkr.ecr.eu-west-2.amazonaws.com": "ecr-login" } }GitLab's documentation notes that credential helpers aren't available on instance runners, or on any runner whose machine you can't configure.
For ACR, a common way is a registry token: a name and password that ACR creates, tied to
a scope map that allows pulls from chosen repositories only. Tokens exist in
every ACR tier, and work with an ordinary registry login, so they fit in
DOCKER_AUTH_CONFIG like any other password.
MegaCorp avoids the question: its job images come from
registry.example.com, not from either cloud.
Secrets from AWS and Azure#
A secret in a cloud's vault can reach software in three ways. The best one keeps it out of the pipeline completely:
| Route | Who reads the secret | What the job sees |
|---|---|---|
The secrets: keyword, with aws_secrets_manager or azure_key_vault (Premium) | the runner, as the job starts | a file holding the value; for AWS, the value itself with file: false |
| The job's script calls the cloud's CLI | the job | the value, which GitLab can't mask because it never saw it |
| The running service reads it: through an ECS task definition, or the External Secrets Operator on a cluster | the service, when it starts | nothing at all |
With the keyword, the runner does the fetching. It logs in with an ID token named
AWS_ID_TOKEN and the role in AWS_ROLE_ARN. The region comes from
AWS_REGION, which MegaCorp's base template sets for every job:
db-migrate:
id_tokens:
AWS_ID_TOKEN:
aud: https://gitlab.example.com
variables:
AWS_ROLE_ARN: arn:aws:iam::123456789012:role/payments-db-migrate
secrets:
DB_PASSWORD:
aws_secrets_manager:
secret_id: payments/db
field: password
script:
- ./migrate.sh --password-file "$DB_PASSWORD"Without an ID token, the runner uses its own AWS role instead. On the Kubernetes
executor, that role must be on the runner manager, the part that starts the job pods. If
it is only on the pods, the job fails with no EC2 IMDS role found.
From a script, the commands look like this. Capture the value in a variable, and never print it:
DB_PASSWORD=$(aws secretsmanager get-secret-value --secret-id payments/db --query SecretString --output text)
DB_URL=$(aws ssm get-parameter --name /payments/db-url --with-decryption --query Parameter.Value --output text)The third route is the cleanest. An ECS task definition can name the secret, and ECS reads it when the container starts, using the task's execution role:
"secrets": [
{ "name": "DB_PASSWORD",
"valueFrom": "arn:aws:secretsmanager:eu-west-2:123456789012:secret:payments/db-AbCdEf" }
]The pipeline deploys a task definition that holds only the secret's name. The value never passes through GitLab.
With the keyword, the runner signs in as the application in
AZURE_CLIENT_ID, with the ID token that token: names, and reads
the vault at AZURE_KEY_VAULT_SERVER_URL:
db-migrate:
id_tokens:
AZURE_JWT:
aud: https://gitlab.example.com
variables:
AZURE_KEY_VAULT_SERVER_URL: https://megacorp-prod-kv.vault.azure.net
AZURE_CLIENT_ID: 00000000-0000-0000-0000-000000000000
AZURE_TENANT_ID: 00000000-0000-0000-0000-000000000000
secrets:
DB_PASSWORD:
token: $AZURE_JWT
azure_key_vault:
name: payments-db-password
version: 00000000000000000000000000000000
script:
- ./migrate.sh --password-file "$DB_PASSWORD"The runner writes the value to a temporary file, and DB_PASSWORD holds the
file's path. version names one version of the secret: a GUID written without
dashes.
From a script, after az login:
DB_PASSWORD=$(az keyvault secret show --vault-name megacorp-prod-kv --name payments-db-password --query value --output tsv)The third route is the cleanest here too. On MegaCorp's clusters, the External Secrets Operator copies the secret from Key Vault into the service's namespace, with its own identity (the External Secrets Operator). The value never passes through GitLab.
The usual deploy targets#
Most deploys are one of these. Each is a few CLI commands, run with the short-lived access the job got:
| Target | What the job runs | Where to look when it fails |
|---|---|---|
| Static site on S3 and CloudFront | aws s3 sync, then aws cloudfront create-invalidation | if visitors still see old files, check that the invalidation ran |
| ECS | a new task definition that names the new image, then aws ecs update-service and aws ecs wait services-stable | the wait checks every 15 seconds and gives up after 40 checks, with exit code 255; the service's events show why new tasks fail |
| EKS | aws eks update-kubeconfig, then kubectl or helm, or a GitOps handoff | besides its AWS permissions, the role needs access inside the cluster, which the cluster's owners grant |
| Lambda | aws lambda update-function-code, with a new image or zip file | the role's permission on that function, then the function's own logs |
| Infrastructure as code | OpenTofu or Terraform, with state stored in GitLab; or CloudFormation | the role needs every permission the change needs, so guard it with protected branches and environments |
| Target | What the job runs | Where to look when it fails |
|---|---|---|
| AKS | az aks get-credentials, then kubelogin convert-kubeconfig -l azurecli, then kubectl or helm; or a GitOps handoff | besides its Azure roles, the identity needs access inside the cluster, which the cluster's owners grant |
| Container Apps | az containerapp update --name … --resource-group … --image …, which starts a new revision of the app | the app's revisions, then the new revision's logs |
| Infrastructure as code | OpenTofu or Terraform with the azurerm provider, with state stored in GitLab | the identity needs every permission the change needs, so guard it with protected branches and environments |
MegaCorp's website job is the first AWS kind:
.publish-site:
extends: .megacorp-base
stage: deploy
id_tokens:
MC_ID_TOKEN:
aud: https://gitlab.example.com
variables:
AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-web-publish
before_script:
- !reference [.snippets, functions]
- !reference [.snippets, aws_login]
script:
- aws s3 sync dist/ "s3://$SITE_BUCKET/" --delete
- aws cloudfront create-invalidation --distribution-id "$CF_DISTRIBUTION_ID" --paths '/*'
environment:
name: production
url: https://www.example.comGitLab's ready-made AWS templates#
GitLab ships templates for AWS, and some older pipelines use them:
AWS/Deploy-ECS.gitlab-ci.ymlbuilds an image into GitLab's registry, creates a new task definition revision, and updates an ECS service. You setCI_AWS_ECS_CLUSTER,CI_AWS_ECS_SERVICEand a task definition variable. The deploy waits for the rollout unlessCI_AWS_ECS_WAIT_FOR_ROLLOUT_COMPLETE_DISABLEDis set.AWS/CF-Provision-and-Deploy-EC2.gitlab-ci.ymlcreates a CloudFormation stack, pushes the build to S3, and deploys it to EC2.- Both expect access keys stored as the CI/CD variables
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEYandAWS_DEFAULT_REGION. GitLab's own page says ID tokens are more secure, but don't work with these templates. - GitLab's image
registry.gitlab.com/gitlab-org/cloud-deploy/aws-basecontains the AWS CLI. If its output fails on a non-ASCII character, setLANG: "UTF-8".
Small things that bite#
- The
amazon/aws-cliimage. Its entrypoint is theawscommand itself. On the Docker executor, which keeps an image's entrypoint, it can't run your script unless you setentrypoint: [""]. The Kubernetes executor ignores entrypoints by default, so the same job can work on one runner and fail on another (chapter 30). - The region. AWS commands without
--regionuseAWS_REGION, which MegaCorp sets toeu-west-2. A resource in another region looks as if it doesn't exist. - AWS keys expire after an hour. MegaCorp asks for 3600 seconds. A job that runs longer after logging in gets expired-token errors part-way through.
- The ID token can expire first. Log in at the start of the job, not after a long build.
- The cloud must reach GitLab. AWS's STS and Microsoft Entra ID both
download GitLab's signing keys from the instance. A self-managed GitLab that the internet
can't reach gets
InvalidIdentityTokenfrom AWS. GitLab documents a workaround for AWS that publishes the keys elsewhere; for Azure, its page says the instance must be publicly accessible. - A new federated credential takes a few minutes. Until it has spread
through Entra ID, sign-ins can fail with
AADSTS70021, as if it didn't exist.
When AWS or Azure says no#
Read the error first, because it names the step that failed. Then ask these in order:
As text
- Does the login fail with InvalidIdentityToken? Yes: AWS can't fetch GitLab's signing keys; can the internet reach the instance?. No: the next step.
- Does it say "Not authorized to perform sts:AssumeRoleWithWebIdentity"? Yes: Compare the token's sub and aud with the trust policy. No: the next step.
- Does the trust policy put a * inside StringEquals? Yes: Wildcards only work with StringLike. No: the next step.
- Did the login step run at all? No: Something replaced before_script, so another identity is in use. Yes: the next step.
- Does get-caller-identity show an unexpected role? Yes: Other credentials come first; see which credentials the CLI uses. No: the next step.
- Did the job run for over an hour after logging in? Yes: The keys expired; log in again, or split the job. No: the next step.
- The role lacks permission for that action, and its owners must add it
As text
- Is the pipeline on a branch other than main? Yes: mc_acr_push skips Azure there by design; nothing was refused. No: the next step.
- Does az login fail with AADSTS70021? Yes: No federated credential matched the token's issuer and subject; compare them character by character, or wait a few minutes after adding one. No: the next step.
- Does the error blame the audience, as AADSTS700212 does? Yes: The credential's audience differs from aud in the job's id_tokens. No: the next step.
- Can the internet reach the GitLab instance? No: Entra ID can't fetch GitLab's signing keys. Yes: the next step.
- Does az acr login fail to find the registry? Yes: The identity can't read the registry in Azure Resource Manager; it needs the role Microsoft documents for az acr login. No: the next step.
- The push itself is refused: the identity lacks AcrPush on the registry
To compare sub with the trust policy or the federated credential, print
the token's claims, not the token. A token has three parts separated by dots, and the
middle part holds the claims. GitLab's documentation suggests this command:
echo "$MC_ID_TOKEN" | cut -d '.' -f2 | base64 -d | jq .The claims include the email address of the person who started the pipeline, so remove
the line when you have your answer. base64 may also print base64:
invalid input and return an error, because a token leaves out the padding that
base64 expects. It happened with two tokens in four on the local
GitLab 19.3, and the claims still printed in full.
- Do this
- For each AWS role, write the trust policy's subject condition for exactly the refs
that need it, for example
project_path:megacorp/payments/payments-api:ref_type:branch:ref:main. In Entra ID, add one federated credential for each project and branch that must sign in, spelled exactly. - Why
- The cloud compares the token's subject with that condition, character by character. Anything that does not match is refused.
- Watch out
- Test the role from tag pipelines and merge request pipelines too, because their subjects differ from a branch's. The card refused in some pipelines shows the symptom.
- Sign a job in to AWS or Azure with an ID token and no stored key, and read the token's claims.
- Write a trust policy or federated credential that matches exactly the pipelines that should sign in.
- Read a refused sign-in and say which claim didn't match.
Spot the bugThe release that couldn't push#
payments-api tags v1.5.0. In the tag pipeline, image-build
fails during the login:
An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentityThe same job pushed an image from main an hour earlier, and from a merge
request pipeline this morning. This is the push role:
# Assumed by image-build, and by the ecr-push component, through mc_aws_login.
ecr_push_role:
arn: arn:aws:iam::123456789012:role/gitlab-ecr-push
trust_policy:
Effect: Allow
Principal:
Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
Action: sts:AssumeRoleWithWebIdentity
Condition:
StringEquals:
gitlab.example.com:aud: https://gitlab.example.com
StringLike:
gitlab.example.com:sub: project_path:megacorp/*:ref_type:branch:ref:*
permissions: push and pull images in every ECR repository in the accountShow the answer
A tag pipeline's ID token has a subject that ends in
ref_type:tag:ref:v1.5.0. The pattern accepts only
ref_type:branch, so STS refuses. Pipelines on main, and merge
request pipelines, match because both run on branches.
Add a second pattern for release tags, such as
project_path:megacorp/*:ref_type:tag:ref:v*. IAM accepts a list of values for
one condition, and any match is enough. payments-api protects tags matching
v*, which limits who can create them. On self-managed GitLab, AWS can't check
ref_protected, so the pattern is the only guard.
While you are there, notice that ref:* lets any branch push images,
including unreviewed feature branches. See
how AWS checks an ID token.
Spot the bugThe branch Azure didn't know#
The payments team starts releasing from branches such as release/2026-09.
At its request, the platform team changes mc_acr_push so that
release/* branches push to ACR too. The first pipeline on
release/2026-09 fails in image-build:
AADSTS70021: No matching federated identity record found for presented assertion.Pushes from main still work, and nobody has touched Entra ID.
# The identity that image jobs sign in as. Each federated credential names one kind of
# pipeline that may use it.
acr_push_app:
display_name: gitlab-acr-push
client_id: 00000000-0000-0000-0000-000000000000
tenant_id: 00000000-0000-0000-0000-000000000000
federated_credentials:
- name: payments-api-main
issuer: https://gitlab.example.com
subject: project_path:megacorp/payments/payments-api:ref_type:branch:ref:main
audiences: [https://gitlab.example.com]Show the answer
The job's token has the subject
project_path:megacorp/payments/payments-api:ref_type:branch:ref:release/2026-09.
The application's only federated credential names ref:main. Entra ID compares
subjects exactly, with no wildcards, so nothing matches and the sign-in fails.
A credential for release/2026-09 would work, but only for that branch, and
each release would need another, up to 20 per application. The simpler choice is to keep
pushing to Azure only from main, and release the image that main
built (from build to deploy). See
what must match.
Images to ECR and ACR#
MegaCorp builds each service's image in the pipeline, tags it with the commit's SHA,
and pushes it to Amazon ECR. On main, it pushes the same image to Azure
Container Registry too, for aks-prod. The deploy jobs receive that exact reference. The
goal is build once, promote many: the image that staging tested is the
image that production runs.
The common way to break that goal is to build again later, for example in a tag pipeline. A rebuild is a new image, even from the same commit.
A retried push to ECR fails with ImageTagAlreadyExistsException. The first attempt had pushed the image and then timed out on the last step. The tag is right. The registry is protecting you from something; what? (the card)
How an image is named#
An image reference has three parts: the registry, the repository and a tag. MegaCorp's look like this:
123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:3f9c1e2…
└──────────── registry ─────────────────────┘ └─ repository ─┘ └ tag ┘megacorpprod.azurecr.io/payments-api:3f9c1e2…
└───── registry ──────┘ └─ repository ─┘ └ tag ┘A tag is only a label, and labels can move. A commit SHA as the tag ties the image to
exactly one commit, which is why MegaCorp uses $CI_COMMIT_SHA. A tag such as
latest says nothing about what is inside, and changes with every push.
From build to deploy#
image-build logs in to AWS and ECR, builds with Buildah, and pushes. On
main, it pushes the same image to ACR as well, under the same repository and
tag (chapter 38). Then it writes the ECR reference to a dotenv report as
IMAGE_REF (chapter 31). The deploy jobs pass it on to the
deployer:
.deploy:
stage: deploy
trigger:
project: megacorp/platform/deployer
branch: main
strategy: depend
variables:
APP: $CI_PROJECT_NAME
IMAGE_REF: $IMAGE_REFThe deployer writes only the tag into the environment repository. Each cluster's values name its own registry, so eks-prod pulls the image from ECR, and aks-prod pulls the same tag from ACR (from tag to pod).
payments-api's tag pipeline runs maven-build and image-build
again before deploy-prod. The release tag usually points at a commit that
main already built and staging already tested. The rebuild pushes a second
image under the same SHA tag. Base images and libraries may have changed in the
meantime, so it is not the image that was tested. What happens next depends on the ECR
repository:
- Mutable tags: the new image quietly replaces the tested one.
- Immutable tags: the push fails with
ImageTagAlreadyExistsException.
ACR never sees the rebuild. A tag pipeline isn't on main, so
image-build skips the push to Azure, and aks-prod keeps the image that
main built. With mutable tags in ECR, the two clusters would then run
different images under the same tag.
Promoting without rebuilding#
To promote, deploy the reference that was already built and tested. To give the same image a friendly name as well, add a tag in ECR without pulling or pushing anything:
MANIFEST=$(aws ecr batch-get-image --repository-name payments-api \
--image-ids imageTag="$CI_COMMIT_SHA" --query 'images[].imageManifest' --output text)
aws ecr put-image --repository-name payments-api \
--image-tag "$CI_COMMIT_TAG" --image-manifest "$MANIFEST"Now v1.5.0 and the commit's SHA name the same image. With immutable tags,
ECR refuses any attempt to move either of them.
ACR's tags are mutable by default. Its owners can lock a released tag, so that nobody can overwrite or delete it:
az acr repository update --name megacorpprod --image payments-api:3f9c1e2… --write-enabled falseAs text
- Does the deploy use a tag such as latest? Yes: You can't tell from the tag; look up the image's digest in the registry. No: the next step.
- Was the same tag pushed by more than one pipeline? Yes: With mutable tags, the last push won; compare the pipelines' times. No: the next step.
- Did the deploy receive IMAGE_REF from the build it followed? No: It may be deploying an older reference; check the trigger's variables. Yes: the next step.
- The tag names one image; check the cluster really runs that tag
- Do this
- Tag each image with the commit it was built from, such as
$CI_COMMIT_SHORT_SHA, and deploy that one tag to every environment. - Why
- One image, tested once, is what runs everywhere. Rebuilding for each environment or each cloud ships something nobody tested.
- Watch out
- With immutable tags in ECR, pushing the same tag again fails, so a retried job fails. The card the tag already exists shows it. In ACR, ask for the tags you release to be locked.
- Push an image to ECR or ACR from a job, named and tagged so that a deploy can find it.
- Explain immutable tags, and make a retry safe.
- Deploy one tested tag everywhere instead of rebuilding per environment.
Spot the bugTwo images, one tag#
Commit 3f9c1e2 was built on main, deployed to staging, and
tested for a week. Then it was tagged v1.5.0. The tag pipeline built and
pushed payments-api:3f9c1e2… again, and deploy-prod shipped
it. In production, a bug appears that staging never showed. At the time, payments-api's
ECR repository had mutable tags. The platform team has made them immutable since.
maven-build:
extends: .maven-build
maven-test:
extends: .maven-test
image-build:
extends: .image-build
needs: [maven-build]Show the answer
image-build runs in tag pipelines too, so the release built the commit a
second time. That rebuild picked up whatever base image and libraries were current, made
a different image, and pushed it under the same tag. With mutable tags it replaced the
tested image. Production then deployed the new one.
Build once: skip image-build in tag pipelines, and deploy the image that
main built. Add the release tag with put-image. Turning on tag
immutability in ECR makes a rebuild fail loudly instead of replacing the image. See
from build to deploy.
Environments and deployments#
An environment is GitLab's record of a place you deploy to: a name, a
URL, and a history of deployments. A job with environment: becomes a
deployment job, and that unlocks five safeguards:
- who may deploy there
- approvals before a deploy runs
- one deploy at a time
- refusing outdated deploys
- re-deploying an earlier version
At MegaCorp every environment lives in the deployer project, so that is where to look when a deploy won't start.
deploy-staging sits in waiting for resource. Nothing else in its pipeline is running. Another pipeline, from a colleague's merge, finished deploying to staging ten minutes ago, and its job is green. The resource it waits for is held by a job that has already ended. (the card)
Environments and deployments#
MegaCorp's deployer names each environment after the target and the service:
variables:
ENVIRONMENT: $[[ inputs.environment ]]
deploy:
stage: deploy
image: registry.example.com/megacorp/devops/ci-tools:3.2
resource_group: $APP-$ENVIRONMENT
environment:
name: $ENVIRONMENT/$APP
script:
- mc deploy "$ENVIRONMENT" "$IMAGE_REF"
rules:
- if: $IMAGE_REF == null
when: never
- if: $ENVIRONMENT == "production"
when: manual
- when: on_successEach run of this job records a deployment of
staging/payments-api or production/payments-api. Its history
appears under Operate › Environments. The names come from variables, so
one job serves every service. The resource_group lets only one deploy per
service and environment run at a time (chapter 24).
An environment also scopes variables. A CI/CD variable limited to the
production/* environments reaches only jobs deploying there
(chapter 26).
A deployment here records the deploy job, not the running software. At MegaCorp the job
succeeds once mc deploy has written the new tag into the environment
repository. The cluster rolls it out afterwards, so a green deployment can still end in
pods that never start. Chapter 41 follows the rest of the way,
and chapter 52 shows how to tell whether it worked.
Who may deploy, and approvals Premium#
- Protected environments. A Maintainer can protect an environment, and list who is Allowed to deploy: roles, users or groups. Anyone else's deploy job to it can't run.
- Deployment approvals. A protected environment can require approvals. Its deployments are blocked until all of them are given, under Operate › Environments, on the deployment's status badge. Approving doesn't start the job: someone must still run it.
.deploy, can use deployment-only access
to a protected environment only if it has an environment: of its own.
Without one, GitLab treats it as an ordinary job, with ordinary permissions.One at a time, newest wins#
Two pipelines can race. The newer one deploys first, and then the older one finishes and puts the old version back. The project setting Prevent outdated deployment jobs stops that. An older deployment job then fails with:
The deployment job is older than the latest deployment, and therefore failed.GitLab judges a job's age by when it started, not by its commit. So running an old pipeline's manual deploy blocks a newer pipeline's manual deploy.
Rolling back#
Under Operate › Environments, each earlier successful deployment offers
Rollback environment, and the latest offers Re-deploy to
environment. Both simply run that deployment's job again. That works only if
the job holds everything the deploy needs. At MegaCorp it does: re-running the job writes
the old IMAGE_REF's tag into the environment repository, straight to
main for staging, and as a merge request for production
(promotion by merge request). Reverting the deploy's commit in
that repository does the same, and leaves a clearer history
(drift, and going back).
As text
- Is it waiting for approvals? Yes: Approve it under Operate › Environments, then run it; approval doesn't start it. No: the next step.
- Is the environment protected, and you aren't allowed to deploy? Yes: Ask a Maintainer to add you, or someone allowed to run it. No: the next step.
- Is another deploy in the same resource group running? Yes: It waits its turn. No: the next step.
- Is it older than the latest deployment? Yes: Prevent outdated deployment jobs blocks it; run a newer pipeline. No: the next step.
- Is there a deploy freeze? Yes: GitLab blocks deployments until the freeze ends. No: the next step.
- Check its rules: a manual job waits for someone to run it
- Do this
- Give every deploy job an
environment, make the production jobwhen: manual, and add aresource_group. On Premium, protect the production environment as well. - Why
- GitLab then records what runs where, a person decides when production changes, and two deploys to the same environment never run at the same time.
- Watch out
- With a resource group, a second deploy waits until the first has finished, and the pipeline looks stuck while it does. The card a deploy waits explains it.
- Give a deploy job an environment and a resource group, and read what GitLab records for each deployment.
- Roll back from Operate › Environments, and know what a rollback reruns.
- Explain a deploy that waits, and find the job holding its resource.
Spot the bugApproved, but not deployed#
The production environment in megacorp/platform/deployer needs one approval. A release
manager approves a payments-api deployment at 17:55 and goes home. At 09:00 the next
day, production still runs the old version. The deployer pipeline shows the
deploy job waiting.
variables:
ENVIRONMENT: $[[ inputs.environment ]]
deploy:
stage: deploy
image: registry.example.com/megacorp/devops/ci-tools:3.2
resource_group: $APP-$ENVIRONMENT
environment:
name: $ENVIRONMENT/$APP
script:
- mc deploy "$ENVIRONMENT" "$IMAGE_REF"
rules:
- if: $IMAGE_REF == null
when: never
- if: $ENVIRONMENT == "production"
when: manual
- when: on_successShow the answer
Production deploys are when: manual in the deployer's rules. A deployment
approval unblocks the job, but never starts it. Someone allowed to deploy must still
select Run. The approval was needed, but it wasn't enough.
Make running the job part of the release manager's routine. See who may deploy, and approvals.
The delivery map#
Your pipeline ends when the image is in the registry. Four more steps happen before anyone uses the new version, and GitLab sees only the first. The new tag is written into a GitOps repository. A controller in the cluster notices, renders the chart or overlay, and applies it. Kubernetes replaces the old pods with new ones. Then the running service sends back signals: logs, metrics and traces. This part follows those steps, one chapter each, and this chapter is the map. When a deploy is missing, walk the map from left to right.
The deploy pipeline is green. Staging serves last week's build. The job that "deployed" ran mc deploy, printed done, and exited zero. Somewhere between that job and a running pod is a whole system that GitLab never sees. You need the map of it. (the card)
From commit to cluster, and back#
| Step | What happens | Who owns it |
|---|---|---|
| build and push | your pipeline builds the image and pushes it, tagged, to a registry | your team |
| request the deploy | a job writes the new tag into a values file in the GitOps repository | your team, often through a central job |
| sync | a controller in the cluster sees the change, renders it, and applies the result | the platform team runs the controller; the values are yours |
| roll out | Kubernetes starts pods from the new image, checks them, and stops the old ones | the cluster: the cloud and the platform team |
| run and report | the service runs, and sends logs, metrics and traces to where people watch | your team |
Your team owns the two ends: the service and its pipeline, and what the service reports. The platform team owns the middle: the clusters, the controllers, the add-ons they install, and the shape of the GitOps repository. The cloud provider runs the managed Kubernetes, the registry and the identities. So a failure in the middle is usually found by you and fixed with them. The chapters that follow give you enough to find it.
Push and pull#
A pipeline can deploy in two ways:
- Push: a job changes the target itself, for example by running
helm upgradeagainst a cluster, or updating a cloud service. - Pull, or GitOps: a job writes the new version into a Git repository, and a controller running inside the cluster notices and applies it.
The two patterns ask different things of the pipeline, and tell it different things back. With push, the pipeline needs credentials for the cluster or the cloud service, and when the job finishes you know the change was applied, and ready, if the job waited for it. With pull, the pipeline needs write access to the GitOps repository and nothing else, and when the job finishes you know only that the change was committed.
Pull keeps production credentials out of the pipeline, since only the controller touches the cluster. It also means that Git holds the truth: to see what should be running, read the repository; to go back, revert a commit. The price is that the pipeline can't see the deploy happen. MegaCorp pulls.
MegaCorp's delivery side#
| Repository | Owned by | Holds | Reaches |
|---|---|---|---|
| megacorp/payments/payments-api | the payments team | the service and its pipeline file | ECR and ACR, with images; the deployer, with a trigger |
| megacorp/platform/deployer | the platform team | the deploy pipeline, which runs mc deploy | gitops-config: a commit for staging, a merge request for production |
| megacorp/platform/gitops-config | the platform team; teams change their own values | each cluster's Flux setup, the add-ons, the ApplicationSets, and each service's values | eks-prod and aks-prod, which pull from it |
| megacorp/platform/charts | the platform team | the shared service chart | every ApplicationSet, at a tagged version |
| megacorp/platform/infra-modules | the platform team | OpenTofu modules for clusters, registries and identities | the AWS account and the Azure subscription |
The last step that GitLab shows is the deployer's job. It runs mc deploy,
which writes the new tag into the environment's values file in gitops-config: straight to
main for staging, and in a merge request for production:
deploy)
env="${1:?usage: mc deploy <environment> <image-ref>}"
image="${2:?usage: mc deploy <environment> <image-ref>}"
tag="${image##*:}"
mc_log "promoting ${APP} ${tag} to ${env} through the GitOps repository"
git clone --depth 1 \
"https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
yq -i ".image.tag = \"${tag}\"" "gitops/envs/${env}/${APP}.yaml"
git -C gitops commit -am "deploy ${APP} ${tag} to ${env}"
if [ "$env" = production ]; then
# production changes wait in a merge request until someone merges them
git -C gitops push origin "HEAD:deploy/${APP}-${tag}" \
-o merge_request.create -o merge_request.target=main \
-o merge_request.title="Deploy ${APP} ${tag} to production"
else
git -C gitops push origin HEAD:main
fi
;;Both clusters run the same way. The platform team's Flux installs the add-ons, including Argo CD, from gitops-config. The payments team's services are then deployed by Argo CD, one Application for each environment. The chapters on chapter 43 and chapter 44 take the files apart.
Five questions for a deployment#
The decoding method asks five questions of a job (chapter 55). They work for a deployment too, with different answers:
| Question | For a job | For a deployment |
|---|---|---|
| Where is it defined? | the merged configuration | the chart, the values and the overlays, in the GitOps repository |
| When does it run? | the pipeline's source and the job's rules | after a merge, when the controller next syncs |
| Where does it run? | a runner, with its tags | a cluster, a namespace, some nodes |
| What does it run? | its image and script | an image, named by tag and digest, with its command and config |
| What goes in and out? | variables and artifacts | in: config and secrets; out: logs, metrics and traces |
When a merged change never reaches production, ask the first two in order. Read the GitOps repository, then the controller's sync status, then the pods' events. The rest of this part shows where each of those is.
- Draw the path from an image in the registry to a pod that serves requests, and name who owns each piece.
- Say what a green deploy job actually proves, for push and for pull deployments.
- Know where to look next when the pipeline is green and the service is old.
Spot the bugGreen, then nothing#
payments-api's pipeline on main passed at 10:02, including the trigger to the
deployer. At 10:20, staging still serves the old version.
Show the answer
Any step after the pipeline: the deployer's commit to gitops-config, the controller's sync, or the rollout. Check them from left to right. First, does the staging values file in gitops-config name the new tag? If it does, has the controller synced since that commit? If it has, did the new pods start and stay up? The pipeline can't tell you any of this, because GitLab sees nothing after the commit.
The lab kit: tools, runtimes, networks and tokens#
The labs in this part come in two kinds. Render-and-read labs need only
the helm and kustomize commands, and no cluster. Cluster
labs run a small Kubernetes cluster on your computer with kind, which runs each
cluster node as a container. Every cluster lab also has a render-and-read path.
| Tool | The labs use | For |
|---|---|---|
helm | 4.3 | rendering charts |
kustomize | 5.8, or kubectl 1.36, which has it built in | building overlays |
kubectl | 1.36 | reading the cluster: within one minor version of the cluster |
kind | 0.33, with the Kubernetes 1.36.4 node image | the local cluster |
flux | 2.9 | installing Flux, and reading its objects |
argocd | 3.5 | reading Argo CD's applications |
| a container runtime | see below | running kind's node |
A container runtime. kind uses Docker by default. Docker Desktop is free only for personal use, education, non-commercial open source, and businesses with fewer than 250 employees and less than US$10 million in revenue; everyone else needs a paid subscription. On Linux, Docker Engine is open source and free. Elsewhere, open-source runtimes such as Rancher Desktop and Podman Desktop are alternatives. kind's support for Podman is experimental, and needs extra setup.
Company networks. kind passes HTTP_PROXY,
HTTPS_PROXY and NO_PROXY into its nodes, so a proxy that works for
Docker usually works for the cluster. A proxy that inspects HTTPS re-signs every certificate,
so its certificate authority must be trusted inside the nodes too; ask your IT team for it.
If a registry such as ghcr.io is blocked, pull the image wherever you can and
load it with kind load docker-image <image>. The first run downloads
several hundred megabytes of images. On a slow connection, a chart's install can run out
of time while they download (a HelmRelease that
timed out).
Tokens. The cluster labs use two deploy tokens. A project's deploy
token works only in that project, and only for what its scopes allow. Flux and Argo CD read
the environment repository with one that has the read_repository scope. The
cluster pulls the image with one that has read_registry on the project that
built it. A token on the environment repository can't pull another project's images,
however it's scoped (pods that can't pull their
image). Deploy tokens are free on every GitLab offering. Project access tokens need
Premium or Ultimate on GitLab.com. Avoid personal access tokens: they act as you, in every
project you can reach. A lab that needs more says why.
Containers and Kubernetes, enough to read#
A container runs one image: a program packaged with everything it needs. Kubernetes runs containers across a cluster of machines, from objects that describe what should run. You'll read a handful of them. A Deployment keeps a number of pods running, each with the service's container, and replaces them when their template changes. A Service gives those pods one stable name, and an Ingress brings outside traffic to it. Secrets hold settings that must stay hidden, and a ServiceAccount gives pods an identity. You rarely write these objects: MegaCorp's chart writes them from your values. This chapter shows how to read what it writes.
The new pods show Running, and 0/1 ready, and the rollout waits. The platform team says "check your probe". You have never opened a Kubernetes manifest. You need enough of it to read this one. (the card)
Images and containers#
An image is a program packaged with everything it needs to run. It's stored in a
registry, under a repository and a tag, such as payments-api:3f9c2e1a. A
container is one running copy of an image. Chapter 30
covered how pipelines build images; here, they're what runs.
A tag names an image, but tags can move. A digest, sha256: followed by 64
hexadecimal characters, names exactly one image and never moves. When you need to know
exactly what ran, the digest is the answer.
The objects you'll read#
Kubernetes objects are YAML, like pipelines, and each one names its
apiVersion and kind. A controller in the cluster watches each kind,
and makes the cluster match what the objects ask for. MegaCorp's chart makes these for
each service:
| Object | What it is | What you change, in the service's values |
|---|---|---|
| Deployment | keeps a number of pods running from one pod template, and replaces them when the template changes | replicaCount, image.tag, env, probes and resources |
| ReplicaSet | made by the Deployment, one for each version of the pod template; it keeps that version's pods running | nothing: you read it, to see which versions exist |
| Pod | the smallest thing Kubernetes runs: one or more containers that share a network and storage | nothing directly: pods come and go |
| Service | one stable name and address for a set of pods, which it finds by their labels | usually nothing |
| Ingress | routes outside HTTP and HTTPS traffic to a Service, by host and path; an ingress controller must be running | ingress.host |
| ExternalSecret and Secret | the Secret holds the service's secrets, which the External Secrets Operator copies from the cloud's vault | secrets.keys |
| ServiceAccount | the pods' identity inside the cluster and, through its annotations, in the cloud | serviceAccount.annotations |
| Namespace | the named group that holds one service's objects in one environment, such as payments-staging | nothing: the platform decides |
Reading a Deployment#
This is payments-api's Deployment in staging, as MegaCorp's chart renders it:
---
# Source: service/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
labels:
app.kubernetes.io/name: payments-api
app.kubernetes.io/managed-by: Helm
helm.sh/chart: service-3.4.0
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: payments-api
template:
metadata:
labels:
app.kubernetes.io/name: payments-api
spec:
serviceAccountName: payments-api
containers:
- name: app
image: "123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:3f9c2e1a"
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
env:
- name: PAYMENTS_PROVIDER_URL
value: "https://sandbox.pay.example.com"
- name: SPRING_PROFILES_ACTIVE
value: "staging"
envFrom:
- secretRef:
name: payments-api
readinessProbe:
httpGet:
path: /actuator/health
port: http
livenessProbe:
httpGet:
path: /actuator/health
port: http
initialDelaySeconds: 20
resources:
limits:
memory: 512Mi
requests:
cpu: 100m
memory: 256Mireplicasis how many pods run. Staging runs one; production runs three.selectorand the template'slabelsmust agree. They're how the Deployment, and the Service, know which pods are theirs.templateis the pod template. Change anything under it, such as the image, and the Deployment makes a new ReplicaSet and moves the pods over to it (rolling, the default).imageis the full reference: registry, repository and tag.mc deploychanges only the tag.envholds plain settings.envFromadds every key of thepayments-apiSecret as a variable (the External Secrets Operator).readinessProbedecides when a pod may receive traffic.livenessProberestarts a container that stops answering.resources.requestsis what the pod is given, and what Kubernetes uses to choose a node.limits.memoryis the most it may use; a container that goes over it can be killed.
None of it was typed into the Deployment. The port, the probe's path and the resources are the chart's defaults, which every service starts from:
port: 8080
env: {} # plain settings, as NAME: value
probes:
path: /actuator/health
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 512MiThe image, the settings and the replica count come from staging's values file, which overrides those defaults for payments-api (values).
What you see with kubectl#
You read a running service with a few kubectl commands. Each one needs the
namespace, such as -n payments-staging:
| Command | Shows | Look for |
|---|---|---|
kubectl get pods | each pod, whether it's ready, its status and its restarts | 1/1 ready and Running; restarts that keep growing |
kubectl describe pod <pod> | a pod's settings, with its events at the end | why it can't pull its image, start, or become ready |
kubectl logs <pod> | what the container wrote to standard output | the service's own errors; add --previous for a run that crashed |
kubectl get deployment payments-api | how many pods are ready, up to date and available | a rollout that hasn't finished |
kubectl rollout status deployment/payments-api | waits until a rollout finishes, or fails | successfully rolled out, or a progress-deadline error |
This is payments-api in staging, on the local cluster the book was checked on:
NAME READY STATUS RESTARTS AGE
payments-api-796d6777f9-stwjv 1/1 Running 0 14mNAME READY UP-TO-DATE AVAILABLE AGE
payments-api 1/1 1 1 14mAnd the parts of kubectl describe pod you'll read most: the pod's status, its
image, its readiness check, and the events at the end. The image was already
present because the local cluster can't pull from ECR, so a stand-in was loaded into it
under the same name. On a real cluster, the kubelet pulls the image first, and says
so.
Name: payments-api-796d6777f9-stwjv
Namespace: payments-stagingStatus: Running Image: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:3f9c2e1a Ready: True Readiness: http-get http://:http/actuator/health delay=0s timeout=1s period=10s #success=1 #failure=3 Type StatusEvents:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 14m default-scheduler Successfully assigned payments-staging/payments-api-796d6777f9-stwjv to delivery-lab-control-plane
Normal Pulled 14m kubelet spec.containers{app}: Container image "123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:3f9c2e1a" already present on machine and can be accessed by the pod
Normal Created 14m kubelet spec.containers{app}: Container created
Normal Started 14m kubelet spec.containers{app}: Container startedMost of what you read about a pod comes from other objects. Its image comes from the Deployment, which comes from the chart and your values. Its secrets come from the ExternalSecret, and its traffic from the Service and the Ingress. When something is wrong, the object at fault is often not the pod.
- Do this
- Change a service through its values, never by editing its objects in the cluster.
Point the readiness probe at something that shows the service can do its work. Give each
pod the CPU and memory it really needs in
resources.requests, and a memory limit it will not reach in normal use. - Why
- The controller puts back whatever Git says, so a change made with
kubectlis undone. A good probe keeps traffic away from a pod that is not ready. - Watch out
- A memory limit that is too low makes the cluster kill the pod, and the restart looks like a crash in your code.
- Read a Deployment, a Service and an Ingress, and say what each does for your service.
- Read a pod's status and its events, and tell an image problem from a probe problem from a crash.
- Ask kubectl the four questions that explain most rollouts that stall.
Spot the bugRead production on aks-prod#
This is payments-api's Deployment for production on aks-prod, rendered from the same chart:
---
# Source: service/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
labels:
app.kubernetes.io/name: payments-api
app.kubernetes.io/managed-by: Helm
helm.sh/chart: service-3.4.0
spec:
replicas: 3
selector:
matchLabels:
app.kubernetes.io/name: payments-api
template:
metadata:
labels:
app.kubernetes.io/name: payments-api
azure.workload.identity/use: "true"
spec:
serviceAccountName: payments-api
containers:
- name: app
image: "megacorpprod.azurecr.io/payments-api:1c7d4b09"
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
env:
- name: PAYMENTS_PROVIDER_URL
value: "https://pay.example.com"
- name: SPRING_PROFILES_ACTIVE
value: "production"
envFrom:
- secretRef:
name: payments-api
readinessProbe:
httpGet:
path: /actuator/health
port: http
livenessProbe:
httpGet:
path: /actuator/health
port: http
initialDelaySeconds: 20
resources:
limits:
memory: 512Mi
requests:
cpu: 100m
memory: 256MiDB_PASSWORD come
from, and when does a new pod start receiving traffic?Show the answer
Three pods, from replicas: 3. The image is
megacorpprod.azurecr.io/payments-api:1c7d4b09: Azure's registry, with
production's tag. DB_PASSWORD isn't in the Deployment at all. envFrom
loads every key of the payments-api Secret, which the External Secrets
Operator fills from Key Vault. A new pod receives traffic once its readiness probe on
/actuator/health passes (0/1 ready).
Helm#
Every service needs the same handful of Kubernetes objects, with different names, images
and settings. Helm writes those objects once, as a chart: YAML templates with gaps,
and a default for every gap. A values file fills the gaps for one service in one
environment, and helm template prints the result. That is how a new image tag
travels from a values file into the Deployment that runs the pods. MegaCorp has one chart
for every service, and each service's values live in gitops-config.
You change the memory limit in payments-api's production values file. Two clusters pick it up. The third keeps the old limit. Same chart, same values file, same tag. The third cluster's Application sets the limit inline, and inline wins. Who gets to say which value wins? (the card)
One chart, many services#
A service on Kubernetes needs the same few objects. A Deployment runs its pods, a Service sits in front of them, and an Ingress brings in traffic from outside. A ServiceAccount carries its identity, and something brings in its secrets. Written by hand, that is about a hundred lines of YAML for each service in each environment, and the copies drift apart.
Helm writes the objects once, with gaps where services differ. MegaCorp's platform team
keeps one chart, service, in megacorp/platform/charts. Teams don't
edit it. They write values.
| Word | Means |
|---|---|
| chart | a folder of templates and default values, versioned as one package |
| template | a YAML file with gaps, each written {{ … }} |
| values | what fills the gaps: the chart's defaults, then yours |
| render | fill every gap and print plain YAML, which is what helm template does |
| release | one installation of a chart in a cluster, with a numbered revision for each change |
What's in a chart#
A chart is a folder with three kinds of file. Chart.yaml names the chart and
its version:
# service/Chart.yaml in megacorp/platform/charts: the shared Helm chart that every
# MegaCorp HTTP service deploys with. The charts repository tags each release, such as
# service-3.4.0, and the ApplicationSets name that tag.
# Used by the chapters "Helm" and "Argo CD".
apiVersion: v2
name: service
description: One MegaCorp HTTP service, with its Service, Ingress and secrets
type: application
version: 3.4.0values.yaml holds a default for every gap. The image's repository and tag are
empty, because only a service and an environment can answer them:
image:
repository: "" # set for each service
tag: "" # set for each environment; mc deploy writes it
pullPolicy: IfNotPresentThe templates folder holds YAML with gaps: deployment.yaml,
service.yaml (with the ServiceAccount and the Ingress) and
externalsecret.yaml. A file whose name starts with an underscore, such as
_helpers.tpl, makes no objects: it holds named pieces that the other templates
reuse.
The charts repository tags each release, such as service-3.4.0, and each
ApplicationSet names the tag it uses. A new chart version reaches a service only when someone
changes that line.
Reading a template#
Here is the line that sets the image, in the Deployment template:
image: "{{ .Values.image.repository }}:{{ required "image.tag is set for each environment" .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}Everything between {{ and }} is an action, which Helm
replaces with text. Everything else is copied as it is. This book shades actions, so you can
tell them from the YAML around them. The actions you will meet most:
| You see | It means |
|---|---|
{{ .Values.image.tag }} | the value at image.tag |
{{ required "…" .Values.image.tag }} | the same, but stop with that message if it's empty |
{{ include "service.name" . }} | the text of a named template from _helpers.tpl |
{{- toYaml .Values.resources | nindent 12 }} | a whole block of values, indented 12 spaces. The - trims the spaces before the action |
{{- if .Values.ingress.enabled }} … {{- end }} | the lines between, only when the value is true |
{{- range .Values.secrets.keys }} … {{- end }} | the lines between, once for each item in the list |
{{- with .Values.podLabels }} … {{- end }} | the lines between, only when the value is set, with . meaning that value |
Where the name comes fromdeep dive
{{ include "service.name" . }} is the release name, cut to 63
characters: the first word after helm template or helm install.
Argo CD sets it from releaseName in the ApplicationSet, so every object is
called payments-api, whatever the environment. The namespace, not the name,
keeps staging and production apart.
Values: the defaults, then yours#
Helm starts from the chart's values.yaml, then lays each values file over it,
in order. Maps merge key by key, and anything else is replaced, lists included. So a values
file only says what differs:
image:
repository: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api
tag: 3f9c2e1aThe staging file sets the repository and the tag, and keeps the chart's defaults for everything it doesn't mention, such as the probes and the resources.
Argo CD adds layers of its own. From weakest to strongest: the chart's
values.yaml, value files, inline values, valuesObject,
and single parameters, which beat everything. aks-prod uses that: its
ApplicationSet adds inline values that switch the image to Azure's registry and give the pods
their Azure identity:
valuesObject:
image:
repository: megacorpprod.azurecr.io/payments-api
serviceAccount:
annotations:
azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
podLabels:
azure.workload.identity/use: "true"Render it yourself#
You never need a cluster to see what a chart makes. helm template takes a
release name, the chart and your values, and prints the manifests.
--show-only keeps one template's output:
helm template payments-api ./service -n payments-staging \
-f envs/staging/payments-api.yaml --show-only templates/deployment.yaml---
# Source: service/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
labels:
app.kubernetes.io/name: payments-api
app.kubernetes.io/managed-by: Helm
helm.sh/chart: service-3.4.0
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: payments-api
template:
metadata:
labels:
app.kubernetes.io/name: payments-api
spec:
serviceAccountName: payments-api
containers:
- name: app
image: "123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:3f9c2e1a"
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
env:
- name: PAYMENTS_PROVIDER_URL
value: "https://sandbox.pay.example.com"
- name: SPRING_PROFILES_ACTIVE
value: "staging"
envFrom:
- secretRef:
name: payments-api
readinessProbe:
httpGet:
path: /actuator/health
port: http
livenessProbe:
httpGet:
path: /actuator/health
port: http
initialDelaySeconds: 20
resources:
limits:
memory: 512Mi
requests:
cpu: 100m
memory: 256MiRead it from the top. The name comes from the release name, and the labels from
_helpers.tpl. The replicas, the image and the settings come from the staging
file. envFrom is there because the file lists a secret. Everything the cluster
will run is on the page.
A required value that's missing stops the render, with the chart's own message:
Error: execution error at (service/templates/deployment.yaml:28:52): image.tag is set for each environment
Use --debug flag to render out invalid YAMLhelm lint checks a chart's structure in the same way, also without a
cluster.
helm
template, which fills every gap and prints the manifests. Argo CD applies them; Flux
installs them as a Helm release.#From a tag to a pod#
When a deploy seems not to have happened, follow the tag through three places, in order:
As text
- Does the environment's values file in gitops-config name the new tag? No: The deploy never reached gitops-config. Read the deployer's job log. Yes: the next step.
- Does the rendered Deployment name it? No: Something overrides the file: inline values or parameters. Yes: the next step.
- Do the running pods report it? No: The controller hasn't synced yet, or the new pods can't start. Yes: the next step.
- The new version is running
The first two need no cluster: read the file, and render it. The third asks the cluster which image each pod was started from:
NAME IMAGE
payments-api-6c854bc48b-lzhss 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:1c7d4b09
payments-api-6c854bc48b-nsd64 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:1c7d4b09
payments-api-6c854bc48b-qpdqw 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:1c7d4b09That is production on the local cluster: three pods, as replicaCount: 3 in
production's values file asks, each started from the tag 1c7d4b09 that the same
file names.
Releases, and who installs them#
helm install and helm upgrade render a chart and apply the
result, and keep a release: a record in the cluster of what was applied, with a
revision number for each change. helm list shows releases, helm
history shows a release's revisions, and helm rollback goes back to
one.
With GitOps, people rarely run these commands. What happens to a chart depends on the controller:
Flux, through a HelmRelease, installs and upgrades a real release, so
helm list shows it. Argo CD renders the chart with helm template,
then applies and tracks the output itself, so helm list shows nothing.
So on MegaCorp's clusters, helm list shows the add-ons that Flux installed,
and none of the services that Argo CD deploys, even though their objects carry the label
app.kubernetes.io/managed-by: Helm.
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
argo-cd argocd 2 2026-09-14 06:40:20.289277196 +0000 UTC deployed argo-cd-10.9.0 v3.5.2
external-secrets external-secrets 1 2026-09-14 06:33:33.744648979 +0000 UTC deployed external-secrets-2.10.0 v2.10.0 On the local cluster, Flux installed both of these. payments-api runs in staging and
production, yet it has no release: Argo CD deployed it. The line left out is the lab
cluster's own ingress controller, installed with helm directly.
--atomic is now
--rollback-on-failure, and --force is now
--force-replace. The old names still work, with a warning. New releases are
applied with server-side apply.- Do this
- Put everything that differs by environment in that environment's values file, and nothing environment-specific in the chart.
- Why
- The chart then stays one thing, and reviewers can see every difference between environments in one file each.
- Watch out
- Never type an image tag into a chart or a template. The deploy writes tags into the
values files. Mark the tag as
requiredin the template, so that a render with no tag stops with an error instead of deploying nothing.
- Read a chart and its values files, and say what a template will render.
- Render the chart yourself with helm template and compare it with what runs.
- Recite the order in which Helm's value sources win.
Spot the bugWhich image runs on aks-prod?#
payments-api's production values file names the tag 1c7d4b09 and an ECR
repository. aks-prod's ApplicationSet adds the inline values shown above.
Show the answer
megacorpprod.azurecr.io/payments-api:1c7d4b09. The tag comes from
the production values file. The repository comes from the inline values, which beat value
files. Rendered with the same values, the Deployment says so:
kind: Deployment name: payments-api azure.workload.identity/use: "true" image: "megacorpprod.azurecr.io/payments-api:1c7d4b09"LabRender the shared chart with your valuesabout 10 min#
Helm 4.3 on your computer, and the image you pushed in chapter 8. No cluster.
- In a new folder, save MegaCorp's chart as
service, keeping these paths:megacorp/platform/chartsservice/Chart.yaml# service/Chart.yaml in megacorp/platform/charts: the shared Helm chart that every # MegaCorp HTTP service deploys with. The charts repository tags each release, such as # service-3.4.0, and the ApplicationSets name that tag. # Used by the chapters "Helm" and "Argo CD". apiVersion: v2 name: service description: One MegaCorp HTTP service, with its Service, Ingress and secrets type: application version: 3.4.0megacorp/platform/chartsservice/values.yaml# service/values.yaml in megacorp/platform/charts: the defaults every service starts # from. A service's file in gitops-config sets what differs. # Used by the chapters "Helm" and "GitOps repositories and promotion". image: repository: "" # set for each service tag: "" # set for each environment; mc deploy writes it pullPolicy: IfNotPresent replicaCount: 2 port: 8080 env: {} # plain settings, as NAME: value probes: path: /actuator/health resources: requests: cpu: 100m memory: 256Mi limits: memory: 512Mi serviceAccount: annotations: {} # the pod's cloud identity, where the cloud needs one podLabels: {} ingress: enabled: false className: nginx host: "" secrets: store: "" # the ClusterSecretStore to read from keys: [] # each one: {name: DB_PASSWORD, remote: payments-api/staging/db-password}megacorp/platform/chartsservice/templates/_helpers.tpl{{- /* service/templates/_helpers.tpl in megacorp/platform/charts: named templates that the other templates share, so every object gets the same name and labels. Used by the chapter "Helm". */}} {{- define "service.name" -}} {{- .Release.Name | trunc 63 | trimSuffix "-" -}} {{- end -}} {{- define "service.selectorLabels" -}} app.kubernetes.io/name: {{ include "service.name" . }} {{- end -}} {{- define "service.labels" -}} {{ include "service.selectorLabels" . }} app.kubernetes.io/managed-by: {{ .Release.Service }} helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }} {{- end -}}megacorp/platform/chartsservice/templates/deployment.yaml{{- /* service/templates/deployment.yaml in megacorp/platform/charts: the Deployment that runs a service's pods. Used by the chapters "Containers and Kubernetes, enough to read" and "Helm". */}} apiVersion: apps/v1 kind: Deployment metadata: name: {{ include "service.name" . }} labels: {{- include "service.labels" . | nindent 4 }} spec: replicas: {{ .Values.replicaCount }} selector: matchLabels: {{- include "service.selectorLabels" . | nindent 6 }} template: metadata: labels: {{- include "service.selectorLabels" . | nindent 8 }} {{- with .Values.podLabels }} {{- toYaml . | nindent 8 }} {{- end }} spec: serviceAccountName: {{ include "service.name" . }} containers: - name: app image: "{{ .Values.image.repository }}:{{ required "image.tag is set for each environment" .Values.image.tag }}" imagePullPolicy: {{ .Values.image.pullPolicy }} ports: - name: http containerPort: {{ .Values.port }} {{- with .Values.env }} env: {{- range $name, $value := . }} - name: {{ $name }} value: {{ $value | quote }} {{- end }} {{- end }} {{- if .Values.secrets.keys }} envFrom: - secretRef: name: {{ include "service.name" . }} {{- end }} readinessProbe: httpGet: path: {{ .Values.probes.path }} port: http livenessProbe: httpGet: path: {{ .Values.probes.path }} port: http initialDelaySeconds: 20 resources: {{- toYaml .Values.resources | nindent 12 }}megacorp/platform/chartsservice/templates/service.yaml{{- /* service/templates/service.yaml in megacorp/platform/charts: the service's ServiceAccount, the Service in front of its pods, and the Ingress that brings outside traffic in. Used by the chapters "Containers and Kubernetes, enough to read", "Helm" and "AWS and Azure for delivery". */}} apiVersion: v1 kind: ServiceAccount metadata: name: {{ include "service.name" . }} labels: {{- include "service.labels" . | nindent 4 }} {{- with .Values.serviceAccount.annotations }} annotations: {{- toYaml . | nindent 4 }} {{- end }} --- apiVersion: v1 kind: Service metadata: name: {{ include "service.name" . }} labels: {{- include "service.labels" . | nindent 4 }} spec: selector: {{- include "service.selectorLabels" . | nindent 4 }} ports: - name: http port: 80 targetPort: http {{- if .Values.ingress.enabled }} --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: {{ include "service.name" . }} labels: {{- include "service.labels" . | nindent 4 }} spec: ingressClassName: {{ .Values.ingress.className }} rules: - host: {{ required "ingress.host is needed when ingress.enabled is true" .Values.ingress.host }} http: paths: - path: / pathType: Prefix backend: service: name: {{ include "service.name" . }} port: name: http {{- end }}megacorp/platform/chartsservice/templates/externalsecret.yaml{{- /* service/templates/externalsecret.yaml in megacorp/platform/charts: asks the External Secrets Operator to copy a service's secrets from the cloud's vault into a Kubernetes Secret, which the Deployment reads. Used by the chapters "Helm" and "Secrets and config after the pipeline". */}} {{- if .Values.secrets.keys }} apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: {{ include "service.name" . }} labels: {{- include "service.labels" . | nindent 4 }} spec: refreshInterval: 1h secretStoreRef: kind: ClusterSecretStore name: {{ required "secrets.store names the ClusterSecretStore" .Values.secrets.store }} target: name: {{ include "service.name" . }} data: {{- range .Values.secrets.keys }} - secretKey: {{ .name }} remoteRef: key: {{ .remote }} {{- end }} {{- end }} - Beside the
servicefolder, save your values asgreeting-values.yaml, with your own image's repository and tag:practice · deliverygreeting-values.yaml# practice/delivery/greeting-values.yaml: your values for MegaCorp's shared chart, for the # greeting-service image you pushed in "Images, the registry and variables". Change the # repository to your project's image, and the tag to one of its tags. image: repository: gitlab.example.com:5050/practice/node-service tag: 0648d3dd replicaCount: 1 port: 8080 probes: path: /greet resources: requests: cpu: 50m memory: 64Mi limits: memory: 128Mi - Render the Deployment:
helm template greeting ./service -f greeting-values.yaml --show-only templates/deployment.yaml - Find the image line and the probe path, and say which file each came from.
The Deployment is called greeting, after the release name. Its image and
probe path come from your file, and its labels and the rest from the chart:
---
# Source: service/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: greeting
labels:
app.kubernetes.io/name: greeting
app.kubernetes.io/managed-by: Helm
helm.sh/chart: service-3.4.0
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: greeting
template:
metadata:
labels:
app.kubernetes.io/name: greeting
spec:
serviceAccountName: greeting
containers:
- name: app
image: "gitlab.example.com:5050/practice/node-service:0648d3dd"
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
readinessProbe:
httpGet:
path: /greet
port: http
livenessProbe:
httpGet:
path: /greet
port: http
initialDelaySeconds: 20
resources:
limits:
memory: 128Mi
requests:
cpu: 50m
memory: 64MiDelete the tag line from your values file and render again. Helm
stops with the chart's message:
Error: execution error at (service/templates/deployment.yaml:28:52): image.tag is set for each environment
Use --debug flag to render out invalid YAMLKustomize#
Kustomize changes plain YAML without templates. A base is ordinary YAML that
works on its own. An overlay names the base and lists only what differs for one
environment: a namespace, an image tag, a patch. kustomize build prints the base
with the overlay's changes applied, and kubectl and Flux have Kustomize built in. MegaCorp
uses it to set up each cluster's add-ons, and many teams use it for their services. Always
read the build's output: a change that matches nothing is skipped without a word.
An overlay sets a new image tag for the add-on, and the cluster keeps running the old one. The build succeeds. The tag is spelled right. The overlay's images entry names an image that the base doesn't use by that name, and Kustomize changes nothing without a word. (the card)
Plain YAML, then changes#
Helm fills gaps in templates. Kustomize has no gaps: its input is the same YAML you would
apply by hand. It edits that YAML with changes that you list in a file called
kustomization.yaml.
| Word | Means |
|---|---|
| base | a folder of plain YAML, with a kustomization.yaml listing its files, that works on its own |
| overlay | a folder whose kustomization.yaml names a base and lists changes |
| patch | one change to one object: set a field, add one, or remove one |
| build | apply an overlay's changes to its base and print the result |
A base and an overlay#
The lab at the end of this chapter uses greeting-service, the Node service from
Part III. Its base is a Deployment and a Service, with the image tagged
latest. Staging's overlay changes three things:
# practice/delivery/kustomize/overlays/staging/kustomization.yaml: staging's overlay. It takes
# the base, puts it in its own namespace, pins the image's tag, and runs two replicas.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: greeting-staging
resources:
- ../../base
images:
- name: gitlab.example.com:5050/practice/node-service
newTag: 0648d3dd
patches:
- target:
kind: Deployment
name: greeting
patch: |-
- op: replace
path: /spec/replicas
value: 2resourcesnames the base, as a path to its folder.namespaceputs every object ingreeting-staging.imagesfinds each container whose image has the name given, and sets its tag. The name is the image without its tag, and it must match exactly.patcheschanges one object. This patch is a list of operations, each with a path into the object: replace the value at/spec/replicaswith 2.
kustomize build overlays/staging prints the result:
kind: Service name: greeting
namespace: greeting-stagingkind: Deployment name: greeting
namespace: greeting-staging replicas: 2 - image: gitlab.example.com:5050/practice/node-service:0648d3ddKustomize prints every map with its keys in alphabetical order, so
image now comes before name. The meaning is the same.
A patch as part of an object#
A patch can also be written as a piece of the object, and Kustomize merges it in. aks-prod's add-ons overlay does that. It keeps MegaCorp's base of add-ons, and gives the External Secrets Operator the Azure identity it needs to read Key Vault:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
patches:
- target:
kind: HelmRelease
name: external-secrets
patch: |-
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: external-secrets
spec:
values:
serviceAccount:
annotations:
azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
podLabels:
azure.workload.identity/use: "true"The patch names the object by kind and name, and gives only the fields to add. Maps merge key by key, so the values the base already set, such as the service account's name, stay:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: external-secrets
namespace: external-secrets
spec:
chart:
spec:
chart: external-secrets
sourceRef:
kind: HelmRepository
name: external-secrets
namespace: flux-system
version: 2.10.0
install:
crds: CreateReplace
interval: 30m
upgrade:
crds: CreateReplace
values:
podLabels:
azure.workload.identity/use: "true"
serviceAccount:
annotations:
azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
name: external-secretseks-prod needs none of this, because EKS Pod Identity is set up in AWS, not in the cluster. So eks-prod's Flux builds the base itself.
Where Kustomize runs#
| Where | How |
|---|---|
| your terminal | kustomize build <folder>, or kubectl kustomize <folder>: kubectl 1.36 has Kustomize 5.8.1 built in |
kubectl apply -k <folder> | builds the folder and applies the result |
| Flux | each Kustomization object names a path in a Git source, and Flux builds and applies it |
| Argo CD | an Application whose path holds a kustomization.yaml is built with Kustomize |
So the file you read in a GitOps repository is never what the cluster gets. The build's output is.
A change that matches nothing#
When an images entry names an image that isn't there, Kustomize doesn't
complain. It builds as usual, and the tag stays as it was. Here is the same overlay with
node-service misspelt as node-servce. The namespace and the replicas
still change, which makes the miss easy to overlook:
kind: Service namespace: greeting-stagingkind: Deployment namespace: greeting-staging replicas: 2 - image: gitlab.example.com:5050/practice/node-service:latestBefore merging a change to an overlay, build it and read the line you meant to change.
LegacypatchesStrategicMerge and commonLabels
patchesStrategicMerge and patchesJson6902 became
patches, commonLabels became labels, and a base is
listed under resources, not bases. Old files still build, with a
warning at the top of the output:
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.kustomize edit fix rewrites the file for you.- Do this
- Keep the base valid on its own, and put each environment's differences in its
overlay. Change image tags with
images, not by editing the base. - Why
- Every overlay then starts from the same tested base, and a tag change is one line that reviewers can see.
- Watch out
- Build the overlay and read the output before you merge. A patch or an
imagesentry whose name matches nothing changes nothing, and Kustomize gives no warning.
- Read a base and its overlays, and say what kustomize build will produce.
- Change an image tag in an overlay so that it actually applies.
- Build the overlay and read the output before you merge.
Spot the bugWhat does aks-prod's overlay change?#
eks-prod's Flux builds infrastructure/add-ons/base. aks-prod's builds
infrastructure/add-ons/aks, the overlay shown above.
Show the answer
Only the External Secrets Operator's HelmRelease. On aks-prod its values gain the
azure.workload.identity/client-id annotation for the service account, and the
azure.workload.identity/use: "true" label for the pods. The namespaces, the Helm
repositories and Argo CD's HelmRelease are the base's, unchanged, on both clusters.
Without that patch, the Operator's pods would have no Azure identity (a pod without its Azure identity).
LabBuild an overlayabout 10 min#
Kustomize 5.8, or kubectl 1.36, which has it built in. No cluster.
- Save the base, with your own image in place of
gitlab.example.com:5050/practice/node-service:practice · deliverykustomize/base/kustomization.yaml# practice/delivery/kustomize/base/kustomization.yaml: the base for the Kustomize lab: # greeting-service as plain YAML, the same in every environment. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - greeting.yamlpractice · deliverykustomize/base/greeting.yaml# practice/delivery/kustomize/base/greeting.yaml: greeting-service's Deployment and Service, # with the image your Build labs pushed. Change the image to your project's. The overlays # set its tag. apiVersion: apps/v1 kind: Deployment metadata: name: greeting labels: app.kubernetes.io/name: greeting spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: greeting template: metadata: labels: app.kubernetes.io/name: greeting spec: containers: - name: app image: gitlab.example.com:5050/practice/node-service:latest ports: - name: http containerPort: 8080 readinessProbe: httpGet: path: /greet port: http --- apiVersion: v1 kind: Service metadata: name: greeting labels: app.kubernetes.io/name: greeting spec: selector: app.kubernetes.io/name: greeting ports: - name: http port: 80 targetPort: http - Save staging's overlay, shown above, as
overlays/staging/kustomization.yaml, with your image's name and one of its tags. - Run
kustomize build overlays/staging, orkubectl kustomize overlays/staging. - Find the namespace, the replicas and the image, and say which line of the overlay set each.
Every object is in greeting-staging, the Deployment has 2 replicas, and the
image has your tag:
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: greeting
name: greeting
namespace: greeting-staging
spec:
ports:
- name: http
port: 80
targetPort: http
selector:
app.kubernetes.io/name: greeting
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: greeting
name: greeting
namespace: greeting-staging
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: greeting
template:
metadata:
labels:
app.kubernetes.io/name: greeting
spec:
containers:
- image: gitlab.example.com:5050/practice/node-service:0648d3dd
name: app
ports:
- containerPort: 8080
name: http
readinessProbe:
httpGet:
path: /greet
port: httpIn the overlay's images entry, misspell the image's name, and build
again. The build succeeds, and the image is back to latest.
GitOps repositories and promotion#
With GitOps, a deploy is a commit. A service's application repository holds its code and pipeline, and an environment repository says what runs where. The pipeline's last job writes the new image tag into the environment repository, and a controller in each cluster applies whatever that repository says. Promotion to production is a merge request in the environment repository, so merging it is the approval, and going back is a revert. A change made in the cluster by hand doesn't last: the controller puts back what the repository says.
At 10:02 the pipeline goes green. At 10:05 staging still serves the old version, and you open an incident. At 10:12 the new version is live, and nobody did anything. The pipeline's last act was a commit to gitops-config. Something else did the deploying, on its own clock. (the card)
Two kinds of repository#
The application repository holds the service's code, its Dockerfile and its pipeline file. The team changes it in merge requests, and its pipeline builds and pushes images. The environment repository holds what runs in each cluster and environment: which chart or overlay, which values, which image tag. Deploy jobs change it, and so do people, in merge requests.
Keeping them apart makes the environment repository's history the history of deploys: one
commit for each change to what runs. Its permissions can differ too: many people change
code, and fewer may change production. MegaCorp's environment repository is gitops-config.
Some teams keep their deployment files in the application repository instead, in a folder
such as deploy/, and point the controller at that folder.
Inside an environment repository#
| Folder in gitops-config | Holds | Owned by |
|---|---|---|
clusters/ | each cluster's Flux setup: where it syncs from, and what it runs, in order | the platform team |
infrastructure/ | the add-ons every cluster runs, and each cloud's changes to them | the platform team |
apps/ | each cluster's ApplicationSets: which services it deploys, from which chart | the platform team, with the teams |
envs/ | each service's values in each environment | the teams, and the deploy job |
A service's team mostly touches one file for each environment. This is payments-api's in production:
image:
repository: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api
tag: 1c7d4b09The job that bumps a tag#
MegaCorp's service pipelines don't write to gitops-config themselves. A service's deploy
job triggers the platform team's deployer pipeline, whose job runs mc
deploy:
deploy)
env="${1:?usage: mc deploy <environment> <image-ref>}"
image="${2:?usage: mc deploy <environment> <image-ref>}"
tag="${image##*:}"
mc_log "promoting ${APP} ${tag} to ${env} through the GitOps repository"
git clone --depth 1 \
"https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
yq -i ".image.tag = \"${tag}\"" "gitops/envs/${env}/${APP}.yaml"
git -C gitops commit -am "deploy ${APP} ${tag} to ${env}"
if [ "$env" = production ]; then
# production changes wait in a merge request until someone merges them
git -C gitops push origin "HEAD:deploy/${APP}-${tag}" \
-o merge_request.create -o merge_request.target=main \
-o merge_request.title="Deploy ${APP} ${tag} to production"
else
git -C gitops push origin HEAD:main
fi
;;- It clones gitops-config with the deployer's token, and changes one line: the tag, in the service's file for that environment.
- For staging, it pushes the commit to
main, and the controllers pick it up on their next sync. - For production, it pushes a branch, and asks GitLab to open a merge request. Nothing changes in production until someone merges it.
The trigger jobs use strategy: depend, so a service's pipeline turns red if
the deployer's pipeline fails (chapter 17). But the deployer's pipeline is green as
soon as the push succeeds. Anything that goes wrong after that never reaches GitLab: a
controller that hasn't synced, or pods that crash.
Promotion by merge request#
To promote is to move a version that is already running in one environment into the next. In an environment repository, that is a change to the next environment's file, and a merge request is the safest way to make it:
- it shows exactly what will change: one tag, from one value to another;
- whoever may merge into
mainmay deploy, so the branch's protection is the approval rule; - the merge is the record of who promoted what, and when.
With push options, the deploy job opens the merge request in the same push, and GitLab answers with its link:
remote:
remote: View merge request for deploy/greeting-7d625fae:
remote: https://gitlab.example.com/practice/greeting-gitops/-/merge_requests/3
remote:Without the options, GitLab only offers a link to create one, and nobody is asked to review anything:
remote:
remote: To create a merge request for deploy/greeting-7d625fae-plain, visit:
remote: https://gitlab.example.com/practice/greeting-gitops/-/merge_requests/new?merge_request%5Bsource_branch%5D=deploy%2Fgreeting-7d625fae-plain
remote:Drift, and going back#
Drift is any difference between the cluster and the repository. It usually comes
from a change made by hand, such as kubectl edit or kubectl scale.
Both controllers put the repository's version back:
What happens to a change made by hand depends on the controller. Flux reverts it at
the Kustomization's next reconciliation, every ten minutes at MegaCorp. Argo CD with
selfHeal: true reverts it a few seconds after noticing. Argo CD without
selfHeal shows the Application as OutOfSync until someone syncs.
So the way to change a running service is always a commit, and that includes going back.
kubectl rollout undo and helm rollback change the cluster, and the
controller changes it straight back. Revert the deploy's commit in the environment
repository instead:
git revert <the deploy's commit>The revert is itself a commit, so it deploys like any other change, and the history shows it.
Automatic image updates#
Some setups have no deploy job. A controller watches the registry, picks new tags by a rule, and writes them into the environment repository itself. You can tell by what's in the repository:
You can recognise each from what it leaves behind. A comment such as
{"$imagepolicy": "flux-system:greeting"} at the end of an
image: line, with ImagePolicy and ImageUpdateAutomation objects, is Flux's
image automation, which commits to the branch its ImageUpdateAutomation names. An
ImageUpdater object, or the annotation
argocd-image-updater.argoproj.io/image-list on an Application, is Argo CD
Image Updater, which writes either the Application's parameters or commits to Git.
If a tag changed and no pipeline ran, look for these. MegaCorp doesn't use them: it wants a pipeline, with its tests, to decide what gets deployed.
GitLab's agent for Kubernetes#
The agent for Kubernetes is a program installed in a cluster that keeps a connection open to GitLab. It is available in every tier, and does two jobs:
- Push, through the agent. Projects and groups listed under
ci_accessin the agent's configuration get akubeconfigfile in every job, at$KUBECONFIG, with a context for each agent they may use.kubectlandhelmthen reach the cluster without any cluster credentials stored in GitLab. - Pull, with Flux. When the agent sees a push to a repository that a Flux GitRepository watches, it can tell Flux to sync at once. Otherwise Flux checks on its own interval.
As text
- Did the deployer's job run, or is it a manual job waiting? No: A waiting manual job deploys nothing. Yes: the next step.
- Did the commit reach the environment repository? No: Read the deploy job's log; the push may have been refused. Yes: the next step.
- For production, was the merge request merged? No: Nothing deploys until someone merges it. Yes: the next step.
- Has the controller synced since that commit? No: It syncs on its interval; wait, or ask for a sync. Yes: the next step.
- Did the new pods start and stay up? No: The cluster's events show why; the pipeline never will. Yes: the next step.
- Check which image the environment repository now names
- Do this
- When your pipeline hands off to GitOps, name the job for what it does, such as
request-deploy, and give its environment the URL where the deploy can be watched. - Why
- The job ends when the change is committed to the environment repository, not when the service is running. The name says so, and the URL shows where to look next.
- Watch out
- People read a green job as a finished deploy. The card deploys appear minutes later shows what the gap looks like.
- Read an environment repository and say what runs where, from its files alone.
- Follow a tag bump from the pipeline's commit to the controller's sync.
- Promote a version from staging to production the way the repository expects.
Spot the bugGreen, but still the old version#
At 10:02 a payments-api pipeline on main goes green, including
deploy-staging and the deployer pipeline it triggered. At 10:05, staging still
serves the old version. MegaCorp's clusters run Flux, connected through GitLab's agent,
and gitops-config is a private project. By 10:12 the new version is live, and nobody
changed anything.
Show the answer
The pipelines ended when mc deploy pushed the new tag to gitops-config.
Because that repository is private, the agent couldn't trigger an immediate sync, so Flux
picked the change up on its next scheduled check. The pods then rolled out. None of this
shows in GitLab.
At 10:05, check the latest commit in gitops-config, then the controller's last sync, then the cluster's rollout. See the job that bumps a tag.
LabPromote by merge requestabout 10 min#
An account on your GitLab, git, and two tags of the image you pushed in
chapter 8. No cluster.
- Create a blank project named
greeting-gitops. Addenvs/production/greeting.yaml: your values from the Helm lab (render the shared chart), with the older tag. Push tomain. - Change the tag to the newer one, and commit.
- Push the commit to a new branch, and ask for a merge request:
git push origin HEAD:deploy/greeting-<tag> -o merge_request.create \ -o merge_request.target=main -o merge_request.title="Deploy greeting <tag> to production" - Open the merge request, and read its changes.
GitLab answers on the push with the merge request's link:
remote:
remote: View merge request for deploy/greeting-7d625fae:
remote: https://gitlab.example.com/practice/greeting-gitops/-/merge_requests/3
remote:The merge request changes one line of envs/production/greeting.yaml: the tag.
Merging it is the deploy's approval.
Push the same commit to another new branch, without the -o options.
GitLab only offers a link to create a merge request, and nobody is asked:
remote:
remote: To create a merge request for deploy/greeting-7d625fae-plain, visit:
remote: https://gitlab.example.com/practice/greeting-gitops/-/merge_requests/new?merge_request%5Bsource_branch%5D=deploy%2Fgreeting-7d625fae-plain
remote:Flux#
Flux is a set of controllers that keep a cluster in step with Git. A GitRepository says what to fetch, and how often. A Kustomization says which folder to apply, how often, and what to do about drift. A HelmRelease installs a chart with its values. MegaCorp's platform team uses Flux to install each cluster's add-ons, in order, from gitops-config. GitLab's agent for Kubernetes can tell Flux about each push, so that it syncs at once instead of at its next check, but only for public projects and the agent's own project.
A merged change to the monitoring folder never reaches the cluster. Flux's other Kustomizations reconcile every minute. This one's row says dependency 'flux-system/add-ons' is not ready, and has for an hour. It waits for another Kustomization, which itself waits for a HelmRelease that is failing its health check. (the card)
Flux's objects#
| Kind | apiVersion | What it does |
|---|---|---|
| GitRepository | source.toolkit.fluxcd.io/v1 | fetches a repository at a branch or tag, every interval |
| HelmRepository | source.toolkit.fluxcd.io/v1 | fetches a chart repository's index |
| Kustomization | kustomize.toolkit.fluxcd.io/v1 | builds one folder of a source with Kustomize, and applies the result, every interval |
| HelmRelease | helm.toolkit.fluxcd.io/v2 | installs or upgrades a chart, with values, as a Helm release |
| Receiver | notification.toolkit.fluxcd.io/v1 | a webhook that tells Flux to fetch now |
| Provider and Alert | notification.toolkit.fluxcd.io/v1beta3 | send Flux's events elsewhere, such as to Grafana or a chat |
| ImagePolicy and ImageUpdateAutomation | image.toolkit.fluxcd.io/v1 | pick new image tags, and commit them to Git (automation) |
Two things share a name. Flux's Kustomization is an object in the cluster that applies a
folder. Kustomize's kustomization.yaml is a file in that folder, which says how
to build it (chapter 44). A folder without one still works: Flux generates it,
for all the manifests in the folder.
How eks-prod's Flux starts#
flux bootstrap gitlab installs Flux in a cluster, and commits two objects
that point Flux at the repository it came from:
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: flux-system
namespace: flux-system
spec:
interval: 1m0s
ref:
branch: main
secretRef:
name: flux-system
url: https://gitlab.example.com/megacorp/platform/gitops-config.gitapiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: flux-system
namespace: flux-system
spec:
interval: 10m0s
path: ./clusters/eks-prod
prune: true
sourceRef:
kind: GitRepository
name: flux-systemFlux fetches main every minute. At least every ten minutes, it applies
./clusters/eks-prod, and with prune: true it deletes whatever that
folder no longer holds. Everything else on the cluster follows from that folder.
Add-ons, then settings, then applications#
eks-prod's folder holds three more Kustomizations. The first installs the add-ons:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: add-ons
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
path: ./infrastructure/add-ons/base
prune: true
wait: true
sourceRef:
kind: GitRepository
name: flux-systemWith wait: true, the Kustomization is ready only once every object it applied
is healthy. retryInterval retries a failure after a minute, instead of ten. The
second waits for the first:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: config
namespace: flux-system
spec:
dependsOn:
- name: add-ons
interval: 10m
path: ./infrastructure/config/eks
prune: true
sourceRef:
kind: GitRepository
name: flux-systemdependsOn applies config only after add-ons is
ready. The order matters: the ClusterSecretStore in config is a kind that the
External Secrets Operator's chart adds, so it can't be applied before that chart is
installed. The third, apps, waits for config in the same way, and
applies the ApplicationSets that Argo CD turns into applications (chapter 47).
While a dependency isn't ready, flux get kustomizations says so. This is the
local cluster, where apps waits directly on add-ons, a minute after
Flux started installing the add-ons:
NAME REVISION SUSPENDED READY MESSAGE
add-ons False Unknown Reconciliation in progress
apps False False dependency 'flux-system/add-ons' is not readyCharts through Flux#
An add-on is a HelmRelease, which names a chart, its version, where to fetch it, and its values:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: external-secrets
namespace: external-secrets
spec:
interval: 30m
chart:
spec:
chart: external-secrets
version: 2.10.0
sourceRef:
kind: HelmRepository
name: external-secrets
namespace: flux-system
install:
crds: CreateReplace
upgrade:
crds: CreateReplace
values:
serviceAccount:
name: external-secretsThe HelmRepository it names is in the same file. crds: CreateReplace makes
Flux create the chart's custom resource definitions, and replace them when a new chart
version changes them. By default, Flux creates them on install and skips them on upgrade.
Flux installs a real Helm release, so helm list shows it, unlike an Argo CD
application (no release in helm list).
Each Helm action has five minutes by default, including the chart's hook Jobs. An
install that runs out of time stays failed: Flux tries again only if
install.remediation.retries says so, and the default is 0. On the local
cluster, Argo CD's first install timed out while its image was still downloading:
NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
argocd argo-cd 10.9.0 False False Helm install failed for release argocd/argo-cd with chart argo-cd@10.9.0: failed pre-install: timeout waiting for: [Job/argocd/argo-cd-argocd-redis-secret-init status: 'InProgress']
external-secrets external-secrets 2.10.0 False True Helm install succeeded for release external-secrets/external-secrets.v1 with chart external-secrets@2.10.0With the image in place, flux reconcile helmrelease argo-cd -n argocd --force
installed it (a HelmRelease that timed
out).
Drift and pruning#
- Drift. Every ten minutes, a Kustomization compares the cluster with
Git, and puts back what differs. The annotation
kustomize.toolkit.fluxcd.io/ssachanges that for one object:Overridereverts every change, and is the default;Mergekeeps fields that other tools added;IfNotPresentcreates the object and then leaves it alone. - Pruning. With
prune: true, objects that Flux applied earlier, but that the current revision no longer holds, are deleted from the cluster.
GitLab's agent, and when Flux syncs#
Left alone, Flux sees a new commit only at its next fetch. GitLab's agent for Kubernetes, in every tier, can shorten that. It finds the GitRepository objects that point at GitLab projects, sets up a Receiver for each, and calls it on every push, so Flux fetches at once. The cluster needs Flux's source and notification controllers as well as the agent.
There's a catch. The agent can reach only its own configuration project and public projects. A private environment repository, like gitops-config, is still synced on Flux's own schedule (deploys appear minutes later).
On the local cluster, once everything had synced, Flux showed this:
NAME REVISION SUSPENDED READY MESSAGE
add-ons main@sha1:9f3c2960 False True Applied revision: main@sha1:9f3c2960
apps main@sha1:9f3c2960 False True Applied revision: main@sha1:9f3c2960NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
argocd argo-cd 10.9.0 False True Helm upgrade succeeded for release argocd/argo-cd.v2 with chart argo-cd@10.9.0
external-secrets external-secrets 2.10.0 False True Helm install succeeded for release external-secrets/external-secrets.v1 with chart external-secrets@2.10.0The local cluster applies add-ons and apps only: its secret
store is a stand-in, so it has no config. Argo CD's release is at revision 2,
because its first install timed out and was forced again.
- Read Flux's sources, Kustomizations and HelmReleases, and say what each watches and applies.
- Ask Flux why a change hasn't arrived, with three commands.
- Trace a dependency chain to the object that is really stuck.
LabLet Flux deploy greeting-serviceabout 20 min#
A cluster from the lab kit, with Flux installed by
flux install; the kustomize folder from
the Kustomize lab; and the image you pushed in
chapter 8. No cluster? Do the first step, then run
kustomize build kustomize/overlays/staging: that is exactly what Flux applies,
and the expected output shows what it did with it.
- On your GitLab, create a project called
greeting-gitops, and push thekustomizefolder to it. - Add two deploy tokens, each under Settings › Repository, in
Deploy tokens. In
greeting-gitops, add one with theread_repositoryscope, for Flux. In the project that built your image, add one with theread_registryscope, for the cluster. Neither can reach any other project. - Make the namespace, and let its pods pull your image with the second token:
kubectl create namespace greeting-staging kubectl create secret docker-registry gitlab-registry -n greeting-staging \ --docker-server=<your registry> --docker-username=<pull token's username> --docker-password=<pull token> kubectl patch serviceaccount default -n greeting-staging -p '{"imagePullSecrets": [{"name": "gitlab-registry"}]}' - Give Flux the first token, then point it at the repository and at staging's overlay:
If your GitLab's certificate comes from a private authority, add
flux create secret git greeting --url=https://<your GitLab>/<your group>/greeting-gitops.git \ --username=<token's username> --password=<token> flux create source git greeting --url=https://<your GitLab>/<your group>/greeting-gitops.git \ --branch=main --secret-ref=greeting --interval=1m flux create kustomization greeting --source=GitRepository/greeting \ --path=./kustomize/overlays/staging --prune=true --interval=1m --wait=true--ca-crt-file=<its certificate>to the first command. - List the pods:
kubectl get pods -n greeting-staging. - In the overlay, change
newTagto another tag of your image, then commit and push. Wait a minute or two, runflux get kustomizations greeting, and list the pods with their images.
Flux applies the overlay, and waits until everything it applied is healthy:
✚ generating Kustomization
► applying Kustomization
✔ Kustomization created
◎ waiting for Kustomization reconciliation
✔ Kustomization greeting is ready
✔ applied revision main@sha1:dde381ae59c419f817c765433d13371b651adf72Two pods run, as the overlay's patch says:
NAME READY STATUS RESTARTS AGE
greeting-b7f6b6f75-fhvgq 1/1 Running 0 17s
greeting-b7f6b6f75-x8cgt 1/1 Running 0 17sAfter the push, the Kustomization shows the new commit, and the pods run the new tag:
NAME REVISION SUSPENDED READY MESSAGE
greeting main@sha1:fd092043 False True Applied revision: main@sha1:fd092043NAME IMAGE
greeting-744b5b7bc4-lnq5g gitlab.example.com:5050/practice/node-service:7d625fae
greeting-744b5b7bc4-wwtwm gitlab.example.com:5050/practice/node-service:7d625fae- Do this
- For each Flux Kustomization you add, set
prune: true. AdddependsOnwhen it needs another Kustomization's objects first, andwait: truewhen others depend on it. - Why
- With prune, a file deleted from Git is deleted from the cluster too. Without it, old objects stay behind. The two ordering settings make Flux apply things in the right order.
- Watch out
- Keep the
intervalat one minute or more. Flux accepts nothing shorter, and a very short interval only makes the controller busy.
Spot the bugThe setting that waited#
A merge request adds a second ClusterSecretStore to infrastructure/config/eks.
Twenty minutes after it was merged, the store still isn't on eks-prod. The
add-ons Kustomization shows the External Secrets Operator's HelmRelease failing,
since a chart upgrade that someone merged the day before.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: config
namespace: flux-system
spec:
dependsOn:
- name: add-ons
interval: 10m
path: ./infrastructure/config/eks
prune: true
sourceRef:
kind: GitRepository
name: flux-systemShow the answer
config depends on add-ons, and add-ons waits for
the health of everything it applied. While the HelmRelease fails, add-ons isn't
ready, so Flux never applies config, whatever it holds. Fix the failing
HelmRelease first; config follows by itself
(a Kustomization waiting for another).
Argo CD#
Argo CD keeps applications in a cluster in step with Git, and shows them in a web UI. An Application names what to deploy, from which repositories, to which cluster and namespace. An ApplicationSet makes Applications from a template, such as one for each environment. Every Application has two statuses: sync, which compares the cluster with Git, and health, which says whether what runs works. MegaCorp's teams deploy with Argo CD: the payments team's ApplicationSet makes payments-api's Applications, for staging and production, on each cluster.
Argo CD's sync fails with resource :Namespace is not permitted in project. The Application is fine. The manifests are fine. The AppProject the Application belongs to does not allow it to create namespaces, and the new chart version tries to. (the card)
Argo CD's objects#
| Kind | What it says | At MegaCorp |
|---|---|---|
| AppProject | which repositories a team's Applications may use, where they may deploy, and which cluster-wide kinds they may create | payments: the charts and gitops-config, into payments-* namespaces, which it may create (a namespace the project doesn't allow) |
| Application | one set of objects: its sources, its destination, and how to sync | made by the ApplicationSet, one for each environment |
| ApplicationSet | a template, and a generator that fills it in to make Applications | payments-api, with a list of two environments |
All three are argoproj.io/v1alpha1 objects in the argocd
namespace, where Argo CD runs.
payments-api's ApplicationSet#
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: payments
namespace: argocd
spec:
description: The payments team's services
sourceRepos:
- https://gitlab.example.com/megacorp/platform/charts.git
- https://gitlab.example.com/megacorp/platform/gitops-config.git
destinations:
- server: https://kubernetes.default.svc
namespace: payments-*
# CreateNamespace makes each environment's Namespace, which is cluster-wide, so the
# project must allow that kind, and only with the team's names
clusterResourceWhitelist:
- group: ""
kind: Namespace
name: payments-*apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: payments-api
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- list:
elements:
- env: staging
- env: production
template:
metadata:
name: "payments-api-{{ .env }}"
annotations:
notifications.argoproj.io/subscribe.on-deployed.grafana: "payments-api|{{ .env }}"
spec:
project: payments
sources:
- repoURL: https://gitlab.example.com/megacorp/platform/charts.git
targetRevision: service-3.4.0
path: service
helm:
releaseName: payments-api
valueFiles:
- "$values/envs/{{ .env }}/payments-api.yaml"
- repoURL: https://gitlab.example.com/megacorp/platform/gitops-config.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: "payments-{{ .env }}"
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true- The list generator gives two elements,
stagingandproduction. The template is filled in once for each, with{{ .env }}replaced. - There are two sources. The first is the shared chart, at the tag
service-3.4.0. The second is gitops-config itself, namedvalues, so that the chart'svalueFilescan start with$valuesand read each environment's file from it. - The destination,
https://kubernetes.default.svc, is the cluster Argo CD runs in, and the namespace is made if it's missing. - The sync policy is automated, with pruning and self-healing.
Sync and health#
| Status | Means |
|---|---|
| Synced | the cluster matches Git, at the revision shown |
| OutOfSync | something in the cluster differs from Git: a new commit, or a change by hand |
| Healthy | every object works; a Deployment is healthy once all its pods are updated and ready |
| Progressing | not healthy yet, but still making progress |
| Degraded | something has failed, such as a Deployment whose rollout missed its deadline |
| Suspended | waiting for something outside to resume it |
| Missing | the object isn't in the cluster |
| Unknown | Argo CD can't judge the object's health |
The two are separate. An Application can be Synced and Degraded: Git and the cluster agree, and what they agree on doesn't work. An Application takes the worst health of its objects.
How it syncs#
Argo CD checks Git about every two minutes, with up to a minute's random delay added.
With automated sync, it syncs any Application that is OutOfSync. Two options, both off by
default, make it stricter: prune deletes objects that Git no longer holds, and
selfHeal puts back a change made by hand, about five seconds after it
happens.
Within a sync, objects can go in order. The annotation
argocd.argoproj.io/sync-wave gives an object a wave number, 0 by default;
lower waves go first. argocd.argoproj.io/hook turns an object into a hook that
runs at a phase of the sync, such as a Job that migrates a database before anything else
changes:
metadata:
annotations:
argocd.argoproj.io/hook: PreSyncValues, and what wins#
Argo CD renders a chart itself, with helm template, and applies the result,
so no Helm release exists (no release in helm list).
When the same value is set in several places, the order is: parameters, then inline
valuesObject, then inline values, then value files, then the chart's
own values.yaml. On aks-prod, the ApplicationSet sets the image's repository
inline, so it beats the values file's:
valuesObject:
image:
repository: megacorpprod.azurecr.io/payments-api
serviceAccount:
annotations:
azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
podLabels:
azure.workload.identity/use: "true"In the UI, and from the command line#
The UI shows each Application with its sync and health, and a tree of the objects it made, down to the pods. From there you can sync by hand, see what differs from Git, and run built-in actions such as restarting a Deployment. Argo CD's notifications can also mark each deploy on your dashboards (chapter 52).
argocd app get shows the same from the command line. This is payments-api in
staging on the local cluster, with each object's sync and health:
Name: argocd/payments-api-staging
Project: payments
Server: https://kubernetes.default.svc
Namespace: payments-staging
URL: https://argocd.example.com/applications/payments-api-staging
Sources:
- Repo: https://gitlab.example.com/megacorp/platform/charts.git
Target: service-3.4.0
Path: service
Helm Values: $values/envs/staging/payments-api.yaml
- Repo: https://gitlab.example.com/megacorp/platform/gitops-config.git
Target: main
Ref: values
SyncWindow: Sync Allowed
Sync Policy: Automated (Prune)
Sync Status: Synced to service-3.4.0
Health Status: Healthy
GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
Namespace payments-staging Running Synced namespace/payments-staging created
ServiceAccount payments-staging payments-api Synced serviceaccount/payments-api created
Service payments-staging payments-api Synced Healthy service/payments-api created
apps Deployment payments-staging payments-api Synced Healthy deployment.apps/payments-api created
networking.k8s.io Ingress payments-staging payments-api Synced Healthy ingress.networking.k8s.io/payments-api created
external-secrets.io ExternalSecret payments-staging payments-api Synced Healthy externalsecret.external-secrets.io/payments-api created- Read an Application, an ApplicationSet and an AppProject, and say what each allows and deploys.
- Read sync and health status, and know what OutOfSync, Progressing and Degraded each mean.
- Find why a sync failed from the Application's own status, before asking anyone.
LabThe same service, with Argo CDabout 15 min#
The cluster and the project from the Flux lab, and Argo CD on the cluster. Install it from the chart MegaCorp's platform team uses:
helm install argo-cd argo-cd --repo https://argoproj.github.io/argo-helm --version 10.9.0 \
--namespace argocd --create-namespaceNo cluster? Read the steps and the expected output: Argo CD applies the same overlay
that kustomize build shows.
- Let Flux let go of greeting-service. With pruning on, deleting the Kustomization
deletes what it applied:
flux delete kustomization greeting --silent flux delete source git greeting --silent - Use the
argocdcommand through your kube context, without logging in to a server:kubectl config set-context --current --namespace=argocd argocd login --core - Give Argo CD the repository's deploy token, then create the application:
If your GitLab's certificate comes from a private authority, run
argocd repo add https://<your GitLab>/<your group>/greeting-gitops.git \ --username <token's username> --password <token> argocd app create greeting --repo https://<your GitLab>/<your group>/greeting-gitops.git \ --path kustomize/overlays/staging --dest-server https://kubernetes.default.svc \ --dest-namespace greeting-staging --sync-policy automated --auto-prune --self-healargocd cert add-tls <its host name> --from <its certificate>first. - Wait a minute, then run
argocd app get greeting.
application 'greeting' createdThe application is Synced and Healthy, with the objects it made:
Name: argocd/greeting
Project: default
Server: https://kubernetes.default.svc
Namespace: greeting-staging
URL: https://argocd.example.com/applications/greeting
Source:
- Repo: https://gitlab.example.com/practice/greeting-gitops.git
Target:
Path: kustomize/overlays/staging
SyncWindow: Sync Allowed
Sync Policy: Automated (Prune)
Sync Status: Synced to (fd09204)
Health Status: Healthy
GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
Service greeting-staging greeting Synced Healthy service/greeting created
apps Deployment greeting-staging greeting Synced Healthy deployment.apps/greeting created- Do this
- For an Application you own, turn on automated sync with
pruneandselfHeal. Keep settings in value files. - Why
- Git then stays the truth: a change in Git is applied, a deleted object is removed, and a change made by hand in the cluster is undone.
- Watch out
- Use inline values only for what really differs between clusters. An inline value beats the same setting in a value file, and reviewers looking at the value files will not see it.
Spot the bugSynced, and Degraded#
After a deploy, payments-api-staging shows Synced and Progressing for ten minutes. Then it shows Synced and Degraded. Nobody has changed anything since the deploy's commit.
Show the answer
Synced means the cluster matches the deploy's commit: Argo CD did its part. Degraded means what it applied doesn't work. The Deployment was Progressing while its new pods weren't ready, and after its ten-minute deadline it counts as failed. Look at the new pods: their events and logs say why they never became ready (a rollout stuck half-way).
OpenTofu and Terraform#
The clusters, registries and identities that delivery runs on are made from code, with
OpenTofu or Terraform. The platform team writes modules; a root
configuration calls them with inputs; plan shows what would change, and
apply makes it so. The state records what exists, and a lock lets only
one run write it at a time. You met plan-in-a-merge-request and apply-on-merge in
Part III. This chapter adds what surrounds them: modules, reading a plan, the state's
lock, and who changes what.
Every plan fails at once: Error acquiring the state lock. A colleague's apply was cancelled mid-way an hour ago, and the lock it took is still there. Someone suggests -lock=false. Someone else says that is how you corrupt a state. Who is right? (the card)
Two names, one language#
OpenTofu is a community fork of Terraform. It is open source, under the Mozilla Public License, and has been a CNCF Sandbox project since April 2025. Terraform is under the Business Source License. The two read the same language, HCL, and most configurations work in both. Where they differ, you will notice:
| What | OpenTofu | Terraform |
|---|---|---|
| the command | tofu | terraform |
where hashicorp/aws comes from | registry.opentofu.org | registry.terraform.io |
.tofu files | read; a .tofu file wins over a .tf file of the same name | not read |
| the licence | Mozilla Public License 2.0 | Business Source License 1.1 |
OpenTofu can also encrypt state and plan files. The rest of this chapter says OpenTofu, and means both.
Modules, and the root configuration#
A module is a folder of .tf files with inputs, called variables, and
outputs. MegaCorp's platform team keeps one for each cloud in infra-modules. This is what the
AWS one asks for:
variable "cluster_name" {
type = string
}
variable "kubernetes_version" {
type = string
default = "1.36"
}
variable "subnet_ids" {
type = list(string)
}
variable "repositories" {
description = "ECR repositories to create, such as payments-api"
type = set(string)
}
variable "workloads" {
description = "Pods that get an AWS role, keyed by name, each with its namespace and service account"
type = map(object({
namespace = string
service_account = string
}))
}For each workload, it makes an IAM role that EKS Pod Identity hands to that namespace's service account:
resource "aws_iam_role" "workload" {
for_each = var.workloads
name = "${var.cluster_name}-${each.key}"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "pods.eks.amazonaws.com" }
Action = ["sts:AssumeRole", "sts:TagSession"]
}]
})
}
resource "aws_eks_pod_identity_association" "workload" {
for_each = var.workloads
cluster_name = aws_eks_cluster.this.name
namespace = each.value.namespace
service_account = each.value.service_account
role_arn = aws_iam_role.workload[each.key].arn
}A module does nothing on its own. A root configuration, the folder where you run
tofu plan, calls it with inputs. MegaCorp's is infra-live. It names each
module's Git repository and a tag, so a module's new version reaches production only when
someone changes the tag:
module "eks_prod" {
source = "git::https://gitlab.example.com/megacorp/platform/infra-modules.git//aws?ref=aws-2.3.0"
cluster_name = "eks-prod"
subnet_ids = ["subnet-0a1b2c3d4e5f60718", "subnet-0b2c3d4e5f6071829"]
repositories = ["payments-api", "web-portal"]
workloads = {
payments-api-staging = { namespace = "payments-staging", service_account = "payments-api" }
payments-api-production = { namespace = "payments-production", service_account = "payments-api" }
}
}Reading a plan in a merge request#
A plan starts each change with a symbol. + creates, - destroys,
and ~ changes in place. -/+ destroys and creates again, because
something that can't change in place did. Here is team-infra's plan from
chapter 10:
$ tofu plan -input=false -out=plan.cache
Acquiring state lock. This may take a few moments...
random_pet.queue: Refreshing state... [id=payments-helped-sunfish]
OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
OpenTofu will perform the following actions:
# random_pet.queue must be replaced
-/+ resource "random_pet" "queue" {
~ id = "payments-helped-sunfish" -> (known after apply)
~ length = 2 -> 3 # forces replacement
# (2 unchanged attributes hidden)
}
Plan: 1 to add, 0 to change, 1 to destroy.
Changes to Outputs:
~ queue_name = "payments-helped-sunfish" -> (known after apply)Saved the plan to: plan.cacheReleasing state lock. This may take a few moments...plan.json: found 1 matching artifact files and directories
Uploading artifacts as "terraform" to coordinator... 201 Created correlation_id=01M2DR15CECXP68NQ0SWZNPXZ2 id=303 status=201 token=[REDACTED]Job succeededRead every must be replaced line before you merge. For a random name, a
replacement costs nothing. For a database, a registry or a cluster, it can mean losing what
it held. # forces replacement marks the line that caused it, and the counts at
the end are what the merge request shows.
State, and its lock#
The state is OpenTofu's record of what it has made. A team's state lives remotely: in GitLab, as in Part III, or in cloud storage. While a run writes it, the backend locks it, so a second run can't write at the same time. That second run stops at once:
$ tofu plan -input=false -out=plan.cache
Acquiring state lock. This may take a few moments...
╷
│ Error: Error acquiring the state lock
│
│ Error message: HTTP remote state already locked:
│ ID=5b0e8c3a-7d2f-4c1e-9a64-3f8b2d1c0e97
│ Lock Info:
│ ID: 5b0e8c3a-7d2f-4c1e-9a64-3f8b2d1c0e97
│ Path:
│ Operation: OperationTypePlan
│ Who: root
│ Version: 1.12.6
│ Created: 2026-09-13 20:23:16.966 +0000 UTC
│ Info:
│
│
│ OpenTofu acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵ERROR: Job failed: exit code 1Only three lines describe the lock that's in the way: ID, Who
and Created. With GitLab-managed state, Who is the GitLab user
whose job took the lock. The other lines describe your own run: GitLab fills
Operation and Version in from the request that failed, which is why
this blocked plan says OperationTypePlan.
Usually the other run is still going, and waiting is the fix. A lock can outlive its run, though, when an apply is cancelled or its runner dies. Then:
- Make sure no job is still running against this state.
- Unlock it. In GitLab, open Operate › Terraform states, and use the
state's Actions menu; this needs the Maintainer or Owner role. Or run
tofu force-unlock <ID>, which removes the lock without changing any resources. - Run the pipeline again, and read its plan: the interrupted apply may have done part of its work.
Don't reach for -lock=false. It lets two runs write the same state, and the
second overwrites what the first recorded.
What your team changes, and what the platform team owns#
| You want | Where the change goes | Who applies it |
|---|---|---|
| a registry repository for a new service | an entry in infra-live's repositories | the platform team, after reviewing your merge request |
| a cloud identity for your pods | an entry in infra-live's workloads | the platform team, likewise |
| a queue, bucket or database of your own | your team's root configuration, if your organisation has one, calling the platform's modules | your team's pipeline, as in the Build part |
| the cluster, its nodes, its network | nowhere you can change | the platform team |
So a team's usual change is a small merge request to someone else's repository. Read the plan in it as carefully as your own.
- Do this
- Pin every module to a tag, never to a branch. Before you merge, read every line in
the plan that says
must be replaced. - Why
- A pinned module changes only when you change it. A replaced resource is destroyed and created again, which can lose data or cause downtime.
- Watch out
- Never edit a state file by hand, and never apply with
-lock=false. If a lock is stale, unlock it deliberately with the unlock command, so that two runs never write the state at once.
- Read a plan and say what will be created, changed, destroyed or replaced.
- Unlock a stale state safely, and know why applying without a lock is never the answer.
- Pin modules, and read a plan that wants to change something nobody edited.
Spot the bugLocked for forty minutes#
team-infra's plan job fails with the lock error above. Its
Created time was forty minutes ago. The project's pipelines show an
apply job that was cancelled forty minutes ago, and nothing running now.
Show the answer
The cancelled apply took the lock and never released it, so the lock outlived its run.
With nothing running, unlock the state: Operate › Terraform states, the
state's Actions menu, Unlock, or tofu force-unlock
with the lock's ID. Then run the pipeline again, and read the new plan before
anything applies: the cancelled apply may have made some of its changes. Don't use
-lock=false to get past it.
See state locked.
AWS and Azure for delivery#
EKS and AKS are Kubernetes run for you by AWS and by Azure. For a pipeline owner, what differs between them is identity: how a pod gets the right to read a secret or a bucket, and how a pipeline signs in when it must. Neither needs a stored password. A pod proves who it is with its service account's token, and a job with its GitLab ID token; the cloud checks that token against a trust set up once. MegaCorp runs the same service on both clouds, so this chapter shows each thing side by side.
A pod on AKS can't reach Key Vault. Its log says no AZURE_FEDERATED_TOKEN_FILE. The same service on EKS reaches Secrets Manager fine. The pod's service account carries the annotation; the pod doesn't carry the label. Two clouds, two ways of giving a pod an identity. (the card)
The same service on two clouds#
| What | AWS | Azure |
|---|---|---|
| the cluster | EKS | AKS |
| the registry | ECR | ACR |
| a job's identity | an IAM role that trusts GitLab's ID tokens | an Entra app with a federated credential for GitLab |
| a pod's identity | EKS Pod Identity, or IRSA on older setups | AKS workload identity |
| secrets | Secrets Manager | Key Vault |
| logs and metrics | CloudWatch | Azure Monitor |
How a pod gets a cloud identity#
On both clouds, the identity belongs to a Kubernetes service account, in its namespace. Every pod that runs as that service account gets it.
With EKS Pod Identity, the link is made in AWS, not in the cluster: an association says that one namespace's service account gets one IAM role. MegaCorp's AWS module makes a role and an association for each workload:
resource "aws_iam_role" "workload" {
for_each = var.workloads
name = "${var.cluster_name}-${each.key}"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "pods.eks.amazonaws.com" }
Action = ["sts:AssumeRole", "sts:TagSession"]
}]
})
}
resource "aws_eks_pod_identity_association" "workload" {
for_each = var.workloads
cluster_name = aws_eks_cluster.this.name
namespace = each.value.namespace
service_account = each.value.service_account
role_arn = aws_iam_role.workload[each.key].arn
}The role trusts pods.eks.amazonaws.com, and an agent on each node hands its
credentials to the pods. Nothing in the chart changes. Older setups use IRSA instead. You can
tell by an eks.amazonaws.com/role-arn annotation on the service account, and a
role that trusts the cluster's own OIDC provider, checking
system:serviceaccount:<namespace>:<name>.
AKS workload identity needs three things to line up:
- the service account's
azure.workload.identity/client-idannotation, naming the managed identity; - the pod's label
azure.workload.identity/use: "true", without which AKS doesn't give the pod its token; - a federated credential on that identity, with the cluster's issuer, the subject
system:serviceaccount:<namespace>:<name>, and the audienceapi://AzureADTokenExchange.
resource "azurerm_user_assigned_identity" "workload" {
for_each = var.workloads
name = "${var.cluster_name}-${each.key}"
resource_group_name = var.resource_group_name
location = var.location
}
resource "azurerm_federated_identity_credential" "workload" {
for_each = var.workloads
name = each.key
parent_id = azurerm_user_assigned_identity.workload[each.key].id
issuer = azurerm_kubernetes_cluster.this.oidc_issuer_url
subject = "system:serviceaccount:${each.value.namespace}:${each.value.service_account}"
audience = ["api://AzureADTokenExchange"]
}In MegaCorp's chart, aks-prod's ApplicationSet sets the first two, through its inline values (values: the defaults, then yours).
So payments-api in payments-staging and in payments-production get
different identities, even from the same chart: the namespace is part of the match.
How a pipeline signs in#
A job presents its GitLab ID token, and assumes a role whose trust policy checks the token's audience and subject. Chapter 38 covers it, with the five things that must match (five things that must match).
The job declares an ID token and hands it to the Azure CLI:
id_tokens:
GITLAB_OIDC_TOKEN:
aud: https://gitlab.example.com
script:
- az login --service-principal -u "$AZURE_CLIENT_ID" -t "$AZURE_TENANT_ID" --federated-token "$GITLAB_OIDC_TOKEN"Entra ID accepts the token when one of the app's federated credentials matches its issuer, subject and audience:
# The identity that image jobs sign in as. Each federated credential names one kind of
# pipeline that may use it.
acr_push_app:
display_name: gitlab-acr-push
client_id: 00000000-0000-0000-0000-000000000000
tenant_id: 00000000-0000-0000-0000-000000000000
federated_credentials:
- name: payments-api-main
issuer: https://gitlab.example.com
subject: project_path:megacorp/payments/payments-api:ref_type:branch:ref:main
audiences: [https://gitlab.example.com]The match is exact: Entra allows no wildcards in a credential. So each kind of pipeline that
signs in needs a credential of its own, one for main, another for each tag or
environment, and an app can have at most 20. A token that matches none fails with
AADSTS70021: No matching federated identity record found for presented
assertion. So does one that arrives just after its credential was added, before the
credential has spread through Entra.
When a pipeline reaches a cluster#
With GitOps, a pipeline never touches the cluster. When a job must, for a push deploy or to
run kubectl, it needs three things:
| Step | AWS | Azure |
|---|---|---|
| sign in | assume the job's role | az login with the ID token |
| get a kubeconfig | aws eks update-kubeconfig --name <cluster> | az aks get-credentials, then kubelogin convert-kubeconfig -l azurecli |
| permissions in the cluster | an EKS access entry for the role, with an access policy or a Kubernetes group | Kubernetes roles, or Azure roles if the cluster uses them, for the identity |
GitLab's agent for Kubernetes is the other way in: its ci_access gives jobs a
kubeconfig without any cloud credentials for the cluster
(GitLab's agent for Kubernetes).
- Do this
- Give each environment's service account its own cloud identity, and keep the namespace in every trust. Use an ID token instead of cloud keys in CI/CD variables whenever the cloud accepts one.
- Why
- One identity per environment means staging can never act as production, and a token that expires in minutes cannot leak for long.
- Watch out
- On Azure, each branch, tag or environment that signs in needs its own federated credential, spelled exactly, and an application can have at most 20 of them. Plan the names before you run out.
- Say how a pod gets a cloud identity on EKS and on AKS, and what each needs on the pod, the service account and the cloud.
- Read an access-denied error from a pod and name the missing piece.
- Keep environments apart with one identity per namespace.
Spot the bugKey Vault works on one cluster only#
ExternalSecrets sync on eks-prod, but none sync on aks-prod: the External Secrets Operator there can't sign in to Key Vault. Its HelmRelease comes from aks-prod's add-ons overlay.
Show the answer
Three things, from the Operator's side, and one more in Azure:
- the service account's
azure.workload.identity/client-idannotation, and - the pods'
azure.workload.identity/use: "true"label, both set by the overlay's patch (a patch as part of an object); - a federated credential on the managed identity for
system:serviceaccount:external-secrets:external-secrets, with the audienceapi://AzureADTokenExchange, made with the platform team's OpenTofu; - and a role on the vault, such as Key Vault Secrets User, for that identity.
Check them in that order. A missing label is the quietest: the pod starts, but gets no token.
Secrets and config after the pipeline#
A pipeline's secrets live in CI/CD variables. A running service's secrets can't: they must reach its pods without ever being written into Git in plain text. There are three common ways. An operator copies them from the cloud's vault into the cluster, which is MegaCorp's way. The environment repository holds them encrypted. Or the service reads the vault itself. Settings that aren't secret go in the environment's values file. Either way, a running pod keeps the values it started with: a new value needs new pods.
The database password was rotated at noon. The ExternalSecret updated the Secret within a minute. The service kept failing with the old password until someone restarted the pods at three. A running pod keeps the values it started with. Who should have restarted it? (the card)
How a secret reaches a pod#
| Way | Git holds | Who fetches or decrypts | You'll see |
|---|---|---|---|
| copied from the cloud's vault | an ExternalSecret: which keys, from which store | the External Secrets Operator, with its own cloud identity | ExternalSecret and ClusterSecretStore objects |
| encrypted with SOPS | the Secret, with its values encrypted | Flux, with a key it holds | ENC[AES256_GCM,… values and a sops: block |
| sealed with Sealed Secrets | a SealedSecret | the Sealed Secrets controller in the cluster | kind: SealedSecret, from bitnami.com/v1alpha1 |
| read by the service itself | nothing | the service, with its own cloud identity | the vault's address in the service's settings |
Whichever way it arrives, a Kubernetes Secret is not encrypted by default. It is stored base64-encoded, and anyone who may read Secrets in the namespace can read it. So who may read Secrets there matters as much as how they got there.
The External Secrets Operator#
Each MegaCorp cluster has one store, called cloud-secrets on both, so a
service's values don't change between clouds:
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: cloud-secrets
spec:
provider:
aws:
service: SecretsManager
region: eu-west-2apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: cloud-secrets
spec:
provider:
azurekv:
authType: WorkloadIdentity
vaultUrl: https://megacorp-prod-kv.vault.azure.net
serviceAccountRef:
name: external-secrets
namespace: external-secretsA service's values file names the keys it needs, and the shared chart turns them into an ExternalSecret:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: {{ include "service.name" . }}
labels:
{{- include "service.labels" . | nindent 4 }}
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: {{ required "secrets.store names the ClusterSecretStore" .Values.secrets.store }}
target:
name: {{ include "service.name" . }}
data:
{{- range .Values.secrets.keys }}
- secretKey: {{ .name }}
remoteRef:
key: {{ .remote }}
{{- end }}The Operator reads each key from the vault, and writes a Secret with the service's name in
its namespace. It reads again every refreshInterval, an hour here. The Deployment
reads that Secret as environment variables, through envFrom.
On the local cluster, a Fake store stands in for the vault, under the same name,
cloud-secrets. The ExternalSecret reports that it synced, and the Secret it
made holds one key:
NAME STORETYPE STORE REFRESH INTERVAL STATUS READY LAST SYNC
payments-api ClusterSecretStore cloud-secrets 1h SecretSynced True 14mNAME TYPE DATA AGE
payments-api Opaque 1 14mSecrets encrypted in Git#
Some teams keep secrets in the environment repository, encrypted, so that Git holds everything the cluster needs.
- SOPS encrypts the values in a file and leaves the keys readable, with a
key from age, OpenPGP or a cloud's key service. A
.sops.yamlfile says which files and fields to encrypt, and with which key. Flux decrypts them when a Kustomization hasdecryption: provider: sops. - Sealed Secrets encrypts a whole Secret with the public key of a
controller running in the cluster, using
kubeseal. Only that controller can decrypt it. By default, a SealedSecret works only under the name and namespace it was sealed for, so it can't be copied into another namespace and read there.
Config for each environment#
Settings that aren't secret belong in the environment's values file, where reviewers can read them:
env:
SPRING_PROFILES_ACTIVE: staging
PAYMENTS_PROVIDER_URL: https://sandbox.pay.example.com
secrets:
store: cloud-secrets
keys:
- name: DB_PASSWORD
remote: payments-api/staging/db-password
ingress:
enabled: true
host: payments.staging.example.comMegaCorp's chart writes env into the Deployment's pod template. So a changed
setting changes the template, and Kubernetes rolls out new pods with it. A ConfigMap works
differently. A pod that reads one as environment variables keeps the old values until it
restarts. Mounted as files, a ConfigMap is refreshed after a while. Secrets behave the same
way: a rotated secret reaches a running pod's variables only when the pod restarts.
- Do this
- Name secrets in an ExternalSecret and keep their values out of Git. Give the secret store the same name on every cluster. Keep settings in ConfigMaps and secrets in Secrets.
- Why
- The same ExternalSecret then works on every cluster, and nobody has to read a secret to review a change.
- Watch out
- A running pod keeps the secret it started with. After a secret is rotated, make sure the pods restart, or have the service read the secret from a mounted file that it reads again.
- Read an ExternalSecret and say where its values come from, and where they end up.
- Tell a ConfigMap from a Secret, and know which a running pod will notice changing.
- Rotate a secret so that the pods actually pick it up.
Spot the bugThe old password, an hour later#
At 09:00, payments-api's database password is rotated in Key Vault. At 09:40, and again at 10:15, payments-api's staging pods on aks-prod still use the old password. Nobody has restarted anything.
Show the answer
Two waits add up. The ExternalSecret reads the vault once an hour, so the Secret in the
namespace can change up to an hour after the rotation. Then the pods read that Secret as
environment variables, which never change in a running pod: they keep the old value until
they restart. Once the Secret holds the new password, new pods get it. They start on the next
deploy that changes the pod template, or when someone restarts the Deployment. With Argo CD,
use its built-in restart action on the Deployment, from the UI or with argocd app actions
run (config for each environment).
Releasing safely#
A release is the moment new pods start taking real traffic. Kubernetes' default is a rolling update: it swaps pods a few at a time, and only counts a new pod once its readiness probe passes, so a broken version stalls instead of taking everything down. Blue-green starts a whole copy of the new version and switches traffic at once. Canary sends a small share of traffic to the new version and grows it while checks pass. Both need a tool such as Argo Rollouts or Flagger. Approvals come before a release, and going back is a revert in Git.
The rollout stopped with three new pods and two old ones. After ten minutes the Deployment reports ProgressDeadlineExceeded. The new pods pass their readiness probe. They answer requests with 500s, because the probe checks only that the process is up. (the card)
Rolling, the default#
MegaCorp's shared chart doesn't name a strategy, so every Deployment rolls. Kubernetes starts new pods and stops old ones in steps. By default it runs at most a quarter more pods than it should, and keeps at least three quarters of them available. A new pod counts only when its readiness probe passes:
readinessProbe:
httpGet:
path: {{ .Values.probes.path }}
port: http
livenessProbe:
httpGet:
path: {{ .Values.probes.path }}
port: http
initialDelaySeconds: 20
resources:
{{- toYaml .Values.resources | nindent 12 }}So a version whose pods never become ready doesn't replace the old one: the old pods keep
serving, and the rollout waits. After ten minutes without progress, the Deployment reports
ProgressDeadlineExceeded. It does not roll back by itself. With GitOps, you go back
by reverting the deploy's commit (drift, and going back).
A readiness probe that checks too little, such as one that always answers, takes that safety away. A pod then counts as ready while it can't serve anyone.
Blue-green and canary#
Kubernetes has no blue-green or canary of its own. Two tools add them, one from each side of GitOps:
With Argo Rollouts, from the Argo project, you'll see a Rollout object in
place of the Deployment, and the canary's share is set by setWeight in its
steps. With Flagger, from the Flux project, you'll see a Canary
object beside the Deployment, and the share climbs by stepWeight up to
maxWeight in its analysis.
A canary in Argo Rollouts is a list of steps: send some traffic, wait, check, send more.
strategy:
canary:
steps:
- setWeight: 20
- pause: {duration: 5m}
- analysis:
templates:
- templateName: success-rate
- setWeight: 50
- pause: {}setWeight: 20sends a fifth of the traffic to the new version. With a traffic router, such as a service mesh or an ingress controller, that share is exact. Without one, Argo Rollouts gets close by running a fifth of the pods on the new version.analysisruns the checks anAnalysisTemplatedescribes, such as a success rate from Prometheus. If they fail, the rollout aborts, and all traffic goes back to the stable version.pause: {}waits for a person, who continues withkubectl argo rollouts promote.
Blue-green in Argo Rollouts uses two Services: activeService for real traffic,
and previewService, which points at the new version while it is tested. The switch
happens on promotion. Flagger does the same with a Canary object, and needs a
traffic router for its weights.
Approvals, and going back#
Three gates put a person in the way. Merging production's merge request in the
environment repository, guarded by that repository's branch protection
(promotion by merge request). Promoting a paused canary or
blue-green rollout, with kubectl argo rollouts promote. And approving a GitLab
deployment job, in GitLab environments, where approvals are Premium
(environments and deployments).
Going back depends on how far the release got. A canary that fails its checks goes back by
itself. Anything already released goes back by reverting its commit in the environment
repository, so the controller deploys the old version again, and the history says so.
kubectl rollout undo would be undone by the controller.
- Do this
- Give every service a readiness probe that checks what the service needs in order to serve, not only that the process is running. Keep the default rolling update unless your team runs Argo Rollouts or Flagger.
- Why
- Kubernetes sends traffic to a pod as soon as the probe passes. A probe that only checks the process lets traffic reach a pod that cannot answer.
- Watch out
- When a release goes wrong, revert the commit in the environment repository. Do not fix the cluster by hand: the controller will put the broken version back.
- Read a rolling update's progress and say why it stopped.
- Write a readiness probe that checks what the service needs to serve.
- Roll back a bad release the GitOps way: revert the commit.
Spot the bugHalf a release#
payments-api's new version reaches staging with 4 replicas. Ten minutes later, 3 pods run the old version and 1 runs the new one, which restarts again and again. The service still answers.
Show the answer
A rolling update is stuck. The new pod never passes its readiness probe, so Kubernetes won't
count it or stop more old pods, and the old pods keep serving. After ten minutes the
Deployment reports ProgressDeadlineExceeded, but it doesn't roll back by itself.
Find out why the new pod restarts from its events and logs, then revert the deploy's commit in
the environment repository, or fix forward with a new commit
(rolling, the default).
Observability#
Once a deploy is out, its signals say whether it works. Logs record what happened, one event at a time. Metrics count things over time, such as requests, errors and restarts. Traces follow one request through every service it touches. OpenTelemetry is the common way to produce all three, and a collector sends them to the backends that keep them. In the open, those are Prometheus, Loki and Tempo. On AWS, they're CloudWatch and X-Ray; on Azure, Azure Monitor and Application Insights. You read them on dashboards, where each deploy is marked. A controller that says "synced and healthy" only means the new pods started. The signals say whether they work.
The new pods restart every forty seconds. kubectl logs shows nothing, because the container dies before it logs. The dashboard shows the error rate climbing since the deploy marker at 15:02. The previous container's log has the answer, if you know to ask for it. (the card)
Three signals#
| Signal | What it is | It answers | You'll see |
|---|---|---|---|
| logs | lines a service writes as things happen, one event each | what happened to this request, and why it failed | lines on standard output, often as JSON |
| metrics | numbers counted over time, each with labels | how often, how fast, how many | a /metrics page for Prometheus to read, or numbers sent over OTLP |
| traces | one request's path through the services: a span for each step, all sharing a trace ID | where the time went, and which service failed | a waterfall of spans, one bar per step |
A pod's logs are what its containers write to standard output and standard error.
Kubernetes keeps them on the node for a while, and kubectl logs reads them from
there. To keep them longer and search them, an agent on each node sends them to a backend.
Metrics and traces need the service to produce them, usually through a library.
OpenTelemetry and the collector#
OpenTelemetry is a project of the Cloud Native Computing Foundation. It gives most languages one standard set of APIs, SDKs and tools for producing traces, metrics and logs. It isn't a backend, and stores nothing. Its protocol, OTLP, carries signals to whatever does, on port 4317 over gRPC or 4318 over HTTP.
The OpenTelemetry Collector sits between services and backends. It receives signals, processes them, for example by adding the pod's name, and exports them to one or more backends. So each service sends everything to one address, and the platform team decides where it goes. Each cloud's own agent does a similar job for that cloud's backends.
An OpenTelemetry SDK reads its settings from environment variables, so these are the ones you'll see in a service's values:
Three variables do most of the work. OTEL_SERVICE_NAME sets the service's
name on every signal it sends. OTEL_RESOURCE_ATTRIBUTES adds more names and
values, such as service.version=3f9c2e1a,deployment.environment.name=staging.
OTEL_EXPORTER_OTLP_ENDPOINT says where to send the signals, usually the
collector.
The open example: Prometheus, Grafana, Loki and Tempo#
- Prometheus keeps metrics. It pulls them: every so often, it reads each
service's metrics page over HTTP, which is called scraping. It stores each number with its
time and labels, and its query language, PromQL, reads them back. Its rules raise alerts, and
Alertmanager sends them to people. A cluster that runs the Prometheus Operator has
ServiceMonitorobjects, frommonitoring.coreos.com/v1. Each one tells Prometheus which Services to scrape, and on which port. A service's chart may include one. - Loki keeps logs. It indexes only each stream's labels, such as the namespace and pod, not the text of the lines, which keeps it cheap. Its query language is LogQL.
- Tempo keeps traces.
- Grafana shows all three on dashboards, side by side, and can alert on them.
Where each signal lives on AWS and Azure#
| Signal | Open example | AWS | Azure |
|---|---|---|---|
| metrics | Prometheus | CloudWatch, through Container Insights; or Amazon Managed Service for Prometheus | Azure Monitor managed service for Prometheus |
| logs | Loki | CloudWatch Logs | a Log Analytics workspace, through Container insights |
| traces | Tempo | X-Ray, sent with OpenTelemetry | Application Insights |
| dashboards | Grafana | CloudWatch dashboards, or Amazon Managed Grafana | the Azure portal, or Azure Managed Grafana |
On EKS, the amazon-cloudwatch-observability add-on installs Container
Insights, which collects the cluster's metrics and logs. On AKS, Container insights collects
the containers' logs, and managed Prometheus collects metrics. Both clouds point you to
OpenTelemetry for traces. AWS put X-Ray's own SDKs and daemon into maintenance mode in
February 2026, and recommends OpenTelemetry instead; X-Ray still keeps and shows the traces.
Application Insights accepts OpenTelemetry too.
MegaCorp's platform team runs one Grafana, at grafana.example.com, that reads the signals from both clouds, and teams build their dashboards there. GitLab also has tracing, metrics and logs of its own, as an experiment since GitLab 18.1. This book doesn't use them.
Marking each deploy#
A graph that jumps is easier to read when you can see what changed just before. So MegaCorp marks each deploy on Grafana's graphs, as what Grafana calls an annotation. The pipeline can't do this well. With GitOps, it only commits to the environment repository, and never learns when the new pods are ready. The controller does. So Argo CD sends the mark, through its notifications controller, which the platform team sets up in Argo CD's chart values:
notifications:
secret:
create: false
notifiers:
service.grafana: |
apiUrl: https://grafana.example.com/api
apiKey: $grafana-api-key
triggers:
trigger.on-deployed: |
- description: Application is synced and healthy.
send: [app-deployed]
oncePer: app.status.operationState?.syncResult?.revision
when: >-
app.status.operationState != nil
and app.status.operationState.phase in ['Succeeded']
and app.status.health.status == 'Healthy'
and (!time.Parse(app.status.health.lastTransitionTime).Add(1 * time.Minute).Before(time.Parse(app.status.operationState.finishedAt))
or time.Parse(app.status.health.lastTransitionTime).Before(time.Parse(app.status.operationState.startedAt)))
templates:
template.app-deployed: |
message: "{{ .app.metadata.name }} is synced and healthy"service.grafanasays where to send marks.$grafana-api-keynames a key in the notifications Secret, which an ExternalSecret copies from the cloud's vault. That's whysecret.createis false: the chart mustn't make the Secret too.trigger.on-deployedis copied from Argo CD's catalog of triggers. It fires once a sync has succeeded and the Application is healthy.template.app-deployedis the mark's text.
Nothing is sent until an Application subscribes. MegaCorp's ApplicationSet subscribes each payments-api Application, with two tags for dashboards to pick marks by:
annotations:
notifications.argoproj.io/subscribe.on-deployed.grafana: "payments-api|{{ .env }}"The key comes from the same store as every other secret:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: argocd-notifications-secret
namespace: argocd
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: cloud-secrets
target:
name: argocd-notifications-secret
data:
- secretKey: grafana-api-key
remoteRef:
key: platform-grafana-api-keyA dashboard shows the marks by asking for annotations with those tags. Flux can send the
same marks, through a Provider of type grafana and an Alert that picks which
events to send.
Did my deploy work?#
Go through these in order. Each step needs the one before it, so the first one that fails is where to look.
- The change is in the environment repository. For staging, the deploy job's commit is on main. For production, its merge request is merged (promotion by merge request). No commit means the pipeline didn't deploy: read the deploy job's log.
- The controller applied that commit. Argo CD shows the Application as Synced to that commit, and Healthy. With Flux, the Kustomization is Ready, and its last applied revision is that commit. If not, the controller's message says why.
- The rollout finished.
kubectl rollout status deployment/payments-api -n payments-stagingexits with 0 once every new pod is ready. If the rollout stalls, it ends witherror: deployment "payments-api" exceeded its progress deadline, and exits with 1. Then the pods' events and logs say why (rolling, the default). - The pods run the new version. The Deployment's image tag is the one the deploy wrote.
- The signals agree. After the deploy's mark, compare the error rate and
the response times with before it. Look for restarts, and for new errors in the logs. With
service.versionset, you can split metrics and traces by version.
Step 2 looks different with each controller. This is greeting-service from the cluster labs, once each controller had applied the pushed commit:
NAME REVISION SUSPENDED READY MESSAGE
greeting main@sha1:fd092043 False True Applied revision: main@sha1:fd092043READY is True, and the revision is the commit that was pushed.
While Flux is still applying it, or waiting for what it applied to become healthy,
READY is Unknown, with Reconciliation in progress.
Name: argocd/greeting
Project: default
Server: https://kubernetes.default.svc
Namespace: greeting-staging
URL: https://argocd.example.com/applications/greeting
Source:
- Repo: https://gitlab.example.com/practice/greeting-gitops.git
Target:
Path: kustomize/overlays/staging
SyncWindow: Sync Allowed
Sync Policy: Automated (Prune)
Sync Status: Synced to (fd09204)
Health Status: Healthy
GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
Service greeting-staging greeting Synced Healthy service/greeting created
apps Deployment greeting-staging greeting Synced Healthy deployment.apps/greeting createdSync Status is Synced, to the commit in brackets, and
Health Status is Healthy. Each object has its own row, with its
own sync and health.
If only the last step fails, the deploy didn't work, even though the controller calls it healthy. Healthy means the pods pass their probes, not that users' requests succeed. Revert the deploy's commit in the environment repository (drift, and going back), or fix forward with a new one.
- Read a crashed container's previous log, and a pod's events, in the right order.
- Find your deploy on the dashboards, and read logs, metrics and traces for one request.
- Run the after-deploy checklist to the signals, not just to the controller's green.
LabBreak a deploy three waysabout 20 min#
The application from the Argo CD lab, synced and
healthy. No cluster? Make each change, run kustomize build
kustomize/overlays/staging to see what the cluster would get, and read the signs
below, which are what the cluster showed.
- An image that can't be pulled. Set
newTagtodoes-not-exist, then commit and push. Once Argo CD has synced, runkubectl get pods -n greeting-staging, thenkubectl describe podon the new pod. Revert the commit, and push. - A value that makes the pod crash. Add this to the overlay's
patches, then commit and push:Run- target: kind: Deployment name: greeting patch: |- - op: add path: /spec/template/spec/containers/0/env value: - name: PORT value: eight-thousandkubectl get pods -n greeting-staging, thenkubectl logson the new pod with--previous. Revert the commit, and push. - A change made by hand. Run
kubectl scale deployment greeting -n greeting-staging --replicas=5. Runkubectl get deployment greeting -n greeting-stagingat once, and again 30 seconds later.
An image that can't be pulled. The new pod never starts, and the old ones keep serving: a rolling update removes old pods only once new ones are ready.
NAME READY STATUS RESTARTS AGE
greeting-675f65c675-bzkfg 0/1 ImagePullBackOff 0 15s
greeting-744b5b7bc4-cw8rs 1/1 Running 0 3m35s
greeting-744b5b7bc4-scqmm 1/1 Running 0 3m35sThe pod's events say why. The registry answered not found: it has no such
tag.
Name: greeting-675f65c675-bzkfgStatus: Pending Image: gitlab.example.com:5050/practice/node-service:does-not-exist State: Waiting
Reason: ImagePullBackOffEvents:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 15s default-scheduler Successfully assigned greeting-staging/greeting-675f65c675-bzkfg to delivery-lab-control-plane
Normal BackOff 14s kubelet spec.containers{app}: Back-off pulling image "gitlab.example.com:5050/practice/node-service:does-not-exist"
Warning Failed 14s kubelet spec.containers{app}: Error: ImagePullBackOff
Normal Pulling 3s (x2 over 14s) kubelet spec.containers{app}: Pulling image "gitlab.example.com:5050/practice/node-service:does-not-exist"
Warning Failed 3s (x2 over 14s) kubelet spec.containers{app}: Failed to pull image "gitlab.example.com:5050/practice/node-service:does-not-exist": rpc error: code = NotFound desc = failed to pull and unpack image "gitlab.example.com:5050/practice/node-service:does-not-exist": failed to resolve reference "gitlab.example.com:5050/practice/node-service:does-not-exist": gitlab.example.com:5050/practice/node-service:does-not-exist: not found
Warning Failed 3s (x2 over 14s) kubelet spec.containers{app}: Error: ErrImagePullArgo CD did its part: the application is Synced, and it stays Progressing while the Deployment waits for a pod that will never be ready. After the Deployment's ten-minute deadline, it turns Degraded (pods that can't pull their image).
Name: argocd/greeting
Project: default
Server: https://kubernetes.default.svc
Namespace: greeting-staging
URL: https://argocd.example.com/applications/greeting
Source:
- Repo: https://gitlab.example.com/practice/greeting-gitops.git
Target:
Path: kustomize/overlays/staging
SyncWindow: Sync Allowed
Sync Policy: Automated (Prune)
Sync Status: Synced to (37af6e9)
Health Status: Progressing
GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
Service greeting-staging greeting Synced Healthy service/greeting unchanged
apps Deployment greeting-staging greeting Synced Progressing deployment.apps/greeting configuredA value that makes the pod crash. This time the image pulls, and the
container starts, but it exits at once. Kubernetes starts it again, and waits longer before
each try: CrashLoopBackOff. The old pods keep serving here too.
NAME READY STATUS RESTARTS AGE
greeting-744b5b7bc4-cw8rs 1/1 Running 0 8m56s
greeting-744b5b7bc4-scqmm 1/1 Running 0 8m56s
greeting-8544d877ff-b72bq 0/1 CrashLoopBackOff 1 (3s ago) 5sThe last run's output names the value. The service read PORT as a number,
got NaN, and couldn't listen on it
(pods that restart again and again).
node:net:2551
validatePort(options.port, 'options.port');
^
RangeError [ERR_SOCKET_BAD_PORT]: options.port should be >= 0 and < 65536. Received type number (NaN).
at Server.listen (node:net:2551:5)
at file:///app/dist/server.js:19:4
at ModuleJob.run (node:internal/modules/esm/module_job:561:25)
at async node:internal/modules/esm/loader:647:26
at async asyncRunEntryPointWithESMLoader (node:internal/modules/run_main:101:5) {
code: 'ERR_SOCKET_BAD_PORT'
}
Node.js v24.21.0A change made by hand. Right after the scale, the Deployment wants five pods, and only two are ready:
deployment.apps/greeting scaledNAME READY UP-TO-DATE AVAILABLE AGE
greeting 2/5 5 2 11mHalf a minute later, it's back to two. Argo CD saw that the cluster no longer matched Git and, with self-healing on, put the Deployment back as Git says. To run more pods, change the overlay and push (how it syncs).
NAME READY UP-TO-DATE AVAILABLE AGE
greeting 2/2 2 2 12m- Do this
- Ask for your Applications to be subscribed to
on-deployed, so that each deploy is marked on your dashboards. SetOTEL_SERVICE_NAME, and setservice.versionto the image tag. Write logs to standard output, one event per line. - Why
- Every signal then says which service and which version sent it, and each deploy is visible on the same graph as its effects.
- Watch out
- After each deploy, go through the checklist all the way to the signals. A deploy that is healthy in the controller can still be failing in the dashboards.
Spot the bugHealthy, and failing#
At 14:02, payments-api 1c7d4b09 reaches production. Argo CD shows payments-api-production
as Synced and Healthy, and all its pods are ready. On the service's Grafana dashboard, a mark
tagged payments-api and production appears at 14:03. Just after it,
the share of failed requests rises from 0.1% to 4%, and stays there.
Show the answer
No. Healthy means the new pods pass their readiness probes, not that they serve every
request. The mark shows the failures started with this deploy. Check the logs and traces of
failed requests, split by service.version, to confirm they come from 1c7d4b09.
Then revert the deploy's commit in the environment repository, so Argo CD deploys the old
version again, or fix forward with a new commit (did my deploy
work?). A canary with an analysis step would have stopped this at a small share of the
traffic (blue-green and canary).
Around your service#
Some tools shape how your service runs without being in its code or its pipeline. A service mesh puts a proxy in the path of every request between services. It encrypts that traffic, retries failed requests, and can split traffic between two versions. Feature flags turn code on for some users without a deploy: GitLab has its own, and OpenFeature is a standard way to read any tool's flags. A developer portal such as Backstage lists every service, and its templates create new repositories with a pipeline already in them. Each of these leaves fingerprints you can learn to spot.
Your repository was created from the platform team's portal template last spring. Since then the template gained a security scan on every merge request. Yours never got it, because a portal template is a copy, not a link, and nobody noticed until an audit asked when the scans had last run on your code. Meanwhile a feature flag turned on in production first. Three things around your service, each with its own rules. (the card)
A service mesh, as you see it#
A service mesh adds a proxy between your service and every service it calls. Your code still calls the other service by its usual address. The proxy adds what the platform team configured: mutual TLS, retries, timeouts, traffic splits and metrics. Istio and Linkerd are the common meshes.
| You see | What it means |
|---|---|
a second container in your pod, istio-proxy or linkerd-proxy, so the pod shows 2/2 ready | the mesh runs as a sidecar: a proxy container added to each pod |
the namespace label istio-injection=enabled, or the annotation linkerd.io/inject: enabled | pods created in that namespace get the sidecar |
the label istio.io/dataplane-mode=ambient, and no extra container | Istio's ambient mode: a proxy on each node handles mutual TLS, and a separate waypoint proxy handles HTTP features |
a VirtualService, from networking.istio.io/v1 | Istio's routes, retries and timeouts for a service |
an HTTPRoute, from gateway.networking.k8s.io/v1 | the Kubernetes Gateway API's routes, a standard that many meshes and ingress controllers read |
A sidecar is added when a pod is created, so pods that were already running need a restart
to get one. To read your service's own logs in a pod with a sidecar, name its container:
kubectl logs with -c.
Mutual TLS means both ends of a connection prove who they are with certificates, and the traffic between them is encrypted. The mesh does it for you. Linkerd, for example, turns it on for all TCP traffic between meshed pods. Each certificate is tied to the pod's service account, and is replaced every day. Your service still speaks plain HTTP to its own proxy.
Retries and timeouts are set in the mesh's routes. By default, Istio retries a failed HTTP request twice, and sets no timeout. A retry is only safe for a request that can be repeated. For a service that takes payments, find out how the mesh retries, and make a repeated request harmless, for example with an idempotency key.
Where a canary's share is set#
Splitting traffic between two versions is a job for the mesh or the ingress controller. The share is a pair of weights on a route. In the Gateway API, a route that sends a tenth of payments-api's traffic to a canary looks like this:
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: payments-api
spec:
parentRefs:
- name: shop
rules:
- backendRefs:
- name: payments-api-stable
port: 80
weight: 90
- name: payments-api-canary
port: 80
weight: 10Weights are shares, not percentages: each backend gets its weight divided by the total. Who changes them depends on the setup:
| Setup | Where the share is set | Who changes it |
|---|---|---|
| by hand | weight in the route, in the environment repository | a person, through a merge request |
| Argo Rollouts | setWeight in the Rollout's steps | Argo Rollouts, which edits the route as each step runs (blue-green and canary) |
| Flagger | stepWeight and maxWeight in the Canary's analysis | Flagger, which edits the route while the checks pass |
| a feature flag's percent rollout | the flag's strategy, in GitLab | whoever owns the flag, with no deploy |
The last row is different. It splits users, not requests between two versions: both code paths are in the same deploy, and the flag picks one.
Feature flags#
A feature flag lets a deploy carry code that stays off until someone turns it on, for everyone or for some users. So releasing a feature no longer has to happen at the moment you deploy it.
GitLab has feature flags in every tier, under Deploy › Feature flags. Each flag has one or more strategies, and each strategy applies to some environments:
| Strategy | Turns the flag on for |
|---|---|
| All users | everyone |
| Percent rollout | a share of page views; you choose whether a user or session keeps the same answer |
| Percent of users | a share of signed-in users; the application must pass each user's ID |
| User IDs | the users you list in the strategy |
| User list | a named list of users, kept in GitLab and shared between flags |
Your application reads its flags with an Unleash client library, because GitLab's flag API
is compatible with Unleash, a feature-flag tool. The client needs the connection details that
the Feature flags page shows. It also needs an application name, which must be the
environment's name, such as production, not the service's. A self-managed GitLab
allows 200 flags per project; GitLab.com allows 50, 150 or 200, by tier.
OpenFeature, an incubating project of the Cloud Native Computing Foundation, is a standard API for reading flags from any flag tool. Your code calls OpenFeature, and a provider translates each call for the tool in use. So changing tools doesn't change your code.
Old flags are debt. Once a feature is on for everyone, remove its flag and the code path it turned off.
Developer portals, and where a pipeline came from#
A developer portal is a website that lists every service and who owns it, with links to its repository, pipelines and dashboards. Backstage is the common one. Spotify created it, and it is now an incubating project of the Cloud Native Computing Foundation. MegaCorp's portal knows payments-api from this file at the root of its repository:
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: payments-api
description: Takes card payments for MegaCorp's shop
annotations:
backstage.io/source-template: template:default/java-service
backstage.io/source-location: url:https://gitlab.example.com/megacorp/payments/payments-api/
spec:
type: service
lifecycle: production
owner: group:default/paymentsThe portal's software templates create new repositories. A template copies a skeleton of
files and fills in the names you type. Its publish:gitlab step pushes the new
repository to GitLab, and its catalog:register step adds the new
catalog-info.yaml to the portal. The skeleton usually includes a
.gitlab-ci.yml, so a new service has a pipeline from its first commit.
A template copies files once. It isn't an include: when the platform team improves a
template, repositories made from it earlier don't change. That's why MegaCorp's templates write
a .gitlab-ci.yml that includes the golden pipeline, instead of copying its jobs
(include: pulling in files). A pipeline that came from a template has
these fingerprints:
- a
catalog-info.yamlat the root, with abackstage.io/source-templateannotation that names the template - a first commit called
initial commit, by the authorScaffolder, unless the template changed those defaults - a pipeline file that matches other services' files, comments and all
- Do this
- If a portal template made your repository, change its pipeline in your own repository. If the fix belongs in the template too, tell the platform team. Turn a new feature flag on in staging before production, and remove it once its feature is fully on.
- Why
- The template does not push changes into repositories it has already made, so your copy is yours to fix. A flag that stays behind is one more thing nobody understands later.
- Watch out
- Before you rely on the service mesh's retries, check that your requests can be repeated safely. A retried payment is a double payment.
- Say what a portal template gives you once, and what a shared template keeps giving you.
- Turn a feature flag on in the right order, and remove it when its feature is fully on.
- Know when the mesh's retries are safe for your service, and when they double a payment.
Spot the bugThe fix that never arrived#
billing-api's .gitlab-ci.yml has a sonar job written out in full.
Last month, the platform team fixed a bug in that job in the portal's "Java service"
template. billing-api's pipeline still runs the old job. Its repository has a
catalog-info.yaml with backstage.io/source-template:
template:default/java-service.
Show the answer
The annotation shows that the portal's template created billing-api. A template copies its
files once, so billing-api's .gitlab-ci.yml has been its own copy since then, and
later changes to the template don't reach it. Fix the job in billing-api's own file. Better
still, have the job come from the golden pipeline or a component that the file includes, so
later fixes arrive with a version bump
(developer portals, and where a pipeline came from).
Platform words, decoded#
The words a platform team uses, one row each: what the thing is, why you care as a developer, and where you'll meet it. Where a chapter covers a word, the last column links there. Search finds every row.
Traffic, names and certificates#
| Term | What it is | Why you care | Where you'll see it |
|---|---|---|---|
| ingress controller | a program in the cluster that reads Ingress objects and sends outside traffic to the right Service | your service's public address and TLS go through it | an Ingress in your service's chart (Helm); Ingress NGINX, a popular one, is retired, with no fixes after March 2026 |
| Gateway API | Kubernetes' newer routing API, with Gateway and HTTPRoute objects from gateway.networking.k8s.io | routes, traffic splits and canaries are written with it | HTTPRoute objects (where a canary's share is set) |
| cert-manager | a controller that gets TLS certificates, from Let's Encrypt or a private authority, and renews them before they expire | your HTTPS certificate is renewed without anyone asking | Certificate and ClusterIssuer objects; the annotation cert-manager.io/cluster-issuer on an Ingress |
| ExternalDNS | a controller that creates DNS records for Services and Ingresses, in Route 53, Azure DNS and other providers | your service's hostname appears in DNS when it's deployed | the annotation external-dns.kubernetes.io/hostname, or an Ingress's hosts |
| service mesh | proxies in the path of every request between services, adding mutual TLS, retries and traffic splits | your calls get retries and encryption you didn't write | Istio and Linkerd (a service mesh, as you see it) |
| sidecar | an extra container that runs beside your service's container, in the same pod | the pod shows 2/2, and kubectl logs needs -c | istio-proxy or linkerd-proxy in your pod (a service mesh) |
| mutual TLS, mTLS | both ends of a connection prove who they are with certificates, and the traffic is encrypted | a service can check which service is calling it | the mesh's settings (a service mesh) |
| NetworkPolicy | rules for which pods may talk to which; the cluster's network plugin must support them | a call that times out may be blocked, not broken | NetworkPolicy objects, from networking.k8s.io/v1, in your namespace |
Scaling and capacity#
| Term | What it is | Why you care | Where you'll see it |
|---|---|---|---|
| HorizontalPodAutoscaler, HPA | changes a workload's number of pods to match its use of CPU, memory or another measure | the number of pods changes without a deploy | HorizontalPodAutoscaler objects, from autoscaling/v2; kubectl get hpa |
| VerticalPodAutoscaler, VPA | an add-on, not built into Kubernetes, that changes pods' CPU and memory requests | your pods' requests may differ from the values file | VerticalPodAutoscaler objects, where the add-on is installed |
| KEDA | scales workloads on events, such as a queue's length, down to zero and back, working with the HPA | a service with no pods starts when work arrives | ScaledObject objects |
| requests and limits | the CPU and memory a container asks for, and the most it may use | too little memory and the container is killed; ask for too much and it waits for a node with room | resources in a chart's values (reading a Deployment) |
| Cluster Autoscaler | adds nodes when pods can't be scheduled for lack of room, and removes nodes that stay underused | a pod may wait in Pending for a new node | the cloud's node groups |
| Karpenter | launches nodes that fit the pods the scheduler can't place, and removes nodes that aren't needed | a pod may wait in Pending for a node chosen to fit it | NodePool objects, from karpenter.sh/v1; on AKS, node auto-provisioning runs Karpenter |
| node pool, node group | a set of the cluster's machines of one kind | your pods may be placed on one by labels and taints | AKS node pools; EKS managed node groups |
| PodDisruptionBudget, PDB | limits how many of an app's pods may be down at once during planned disruptions, such as a node upgrade | without one, an upgrade may take down more of your pods at once than you can spare | PodDisruptionBudget objects, from policy/v1 |
| ResourceQuota | limits a namespace's total CPU, memory and number of objects | a deploy can fail because the namespace is full | ResourceQuota objects; errors that say exceeded quota |
Kubernetes, extended#
| Term | What it is | Why you care | Where you'll see it |
|---|---|---|---|
| namespace | a named group of objects in a cluster, with its own permissions and quotas | your service's objects live in one for each environment | payments-staging and payments-production at MegaCorp (the objects you'll read) |
| custom resource, CRD | a new kind of object added to Kubernetes, such as HelmRelease or ExternalSecret; a CRD defines it | most tools in this chapter are custom resources plus a controller | apiVersions you don't recognise; kubectl get crd |
| controller | a loop that watches objects, and changes the cluster to match what they ask for | a change you make by hand can be undone by one | Flux, Argo CD and the External Secrets Operator (push and pull) |
| operator | a controller, with its own custom resources, that runs an application such as a database for you | a database your service uses may be run by one | custom resources named after the application |
| admission webhook | code that checks each object sent to the Kubernetes API, and may change it, before it's stored | a deploy can be refused, or changed on its way in | errors that say admission webhook "…" denied the request |
| reconcile | what a controller does: compare what's asked for with what exists, and change what exists to match | the controllers' messages use the word | drift, and going back |
| drift | the cluster differing from Git, usually after a change by hand | the controller puts it back, or reports it | drift, and going back |
Policy and the supply chain#
| Term | What it is | Why you care | Where you'll see it |
|---|---|---|---|
| policy engine | checks objects against the organisation's rules as they reach the cluster | a deploy can be refused for breaking a rule, not for a bug | Kyverno and OPA Gatekeeper; GitLab's own policies are something else (policies and gates) |
| Kyverno | a policy engine whose policies are Kubernetes objects: they check, change, create or clean up objects, and can verify image signatures | a refused deploy's message names the policy | ClusterPolicy and Policy objects, from kyverno.io/v1; a graduated CNCF project since March 2026 |
| OPA Gatekeeper | an admission webhook that enforces policies written in Rego, Open Policy Agent's language; its audit lists what already breaks a rule | a refused deploy's message names the constraint | ConstraintTemplate objects, and the constraints made from them |
| image signing | a signature that proves who signed an image, and that it hasn't changed since | the cluster may refuse an image that isn't signed | cosign |
| cosign | Sigstore's tool to sign and verify images and other artifacts; keyless signing ties a short-lived certificate to an OIDC identity, such as a CI job's, and logs it in Rekor | a pipeline signs, and the cluster's policy checks the signature | cosign sign and cosign verify in jobs (ID tokens); signatures in the registry, beside the image |
| SBOM | a software bill of materials: every component in an image or build, in a format such as CycloneDX or SPDX | scanners and auditors read it to find vulnerable parts | GitLab's dependency scanning writes CycloneDX files (GitLab's scanners) |
Delivery#
| Term | What it is | Why you care | Where you'll see it |
|---|---|---|---|
| GitOps | Git holds what should run, and a controller in the cluster pulls it in | a deploy is a commit, and going back is a revert | push and pull |
| environment repository | the repository that says what runs in each environment: image tags and settings | your deploy job commits there | megacorp/platform/gitops-config (the repositories) |
| promotion | moving a version from one environment to the next | production's deploy is a merge request | promotion by merge request |
| Helm chart | a package of templates and default values that renders Kubernetes objects | your service's objects come from one | Chart.yaml, values.yaml and templates/ (Helm) |
| Kustomize overlay | changes laid over a base set of objects, one overlay for each environment or cluster | a setting may come from the overlay, not the base | kustomization.yaml files (Kustomize) |
| Flux | a GitOps tool made of several controllers, configured with objects such as GitRepository, Kustomization and HelmRelease | on MegaCorp's clusters, it installs the platform's add-ons | clusters/eks-prod/platform.yaml (Flux) |
| Argo CD | a GitOps controller with a web UI; an Application says what to deploy, from where, and to which namespace | your service's sync and health show on its UI | Application and ApplicationSet objects (Argo CD) |
| ApplicationSet | makes many Argo CD Applications from one template, such as one for each environment | changing it changes every environment's Application | apps/eks-prod/applicationset.yaml at MegaCorp |
| image automation | a controller that finds a new image tag in the registry, and writes it into Git | a commit in the environment repository may come from a bot | Flux's image automation; Argo CD Image Updater (automation) |
| progressive delivery | releasing to a growing share of traffic, while checks pass | a bad version reaches few users | blue-green and canary |
| Argo Rollouts | a controller for canary and blue-green releases, with steps and analysis; a Rollout object takes the Deployment's place | your service's pods are managed by a Rollout, not a Deployment | Rollout and AnalysisTemplate objects (blue-green and canary) |
| Flagger | a controller for canary and blue-green releases, from the Flux project; a Canary object sits beside the Deployment | checks decide whether the new version goes on | Canary objects (blue-green and canary) |
| feature flag | a switch that turns code on for some or all users, without a deploy | releasing can happen after deploying | GitLab's **Deploy › Feature flags**; OpenFeature (feature flags) |
Identity, secrets and infrastructure#
| Term | What it is | Why you care | Where you'll see it |
|---|---|---|---|
| workload identity | a pod's own cloud identity, from its Kubernetes service account, with no stored key | your service reaches the cloud without a password | pods and their cloud identity |
| IRSA, EKS Pod Identity | the two ways a pod on EKS gets an IAM role | they decide your pod's AWS permissions | the annotation eks.amazonaws.com/role-arn, or a Pod Identity association (pods and their cloud identity) |
| External Secrets Operator | copies secrets from the cloud's vault into Kubernetes Secrets | your secrets' values never go into Git | ExternalSecret and ClusterSecretStore objects (the External Secrets Operator) |
| SOPS, Sealed Secrets | two ways to keep secrets in Git, encrypted | you'll see secret values in Git that you can't read | ENCAES256_GCM,… values, or kind: SealedSecret ([secrets encrypted in Git) |
| OpenTofu, Terraform | tools that create cloud resources from code, with a plan to review before each apply | your cluster, registry and roles came from them | .tf files, and plans in merge requests (OpenTofu and Terraform) |
| state, state lock | the tool's record of what it created, and the lock that stops two runs at once | a stale lock blocks every apply | state and its lock |
| Crossplane | a control plane framework: cloud resources become Kubernetes objects, and a platform team's compositions turn one request into many resources | you may ask for a database with a Kubernetes object, not a Terraform module | composite resources that the platform team defines; a graduated CNCF project since October 2025 |
| landing zone | an organisation's standard cloud setup: accounts or subscriptions made to one pattern, with rules applied to all of them | your team's AWS account or Azure subscription came from it | AWS Control Tower and its Account Factory; Azure's platform and application landing zones |
Signals and reliability#
| Term | What it is | Why you care | Where you'll see it |
|---|---|---|---|
| OpenTelemetry Collector | receives signals from services, processes them, and sends them on to backends | your service sends everything to one address | OTEL_EXPORTER_OTLP_ENDPOINT in your settings (OpenTelemetry and the collector) |
| Prometheus | keeps metrics, pulling them from each service's metrics page; PromQL queries them | your service's graphs and alerts come from it | ServiceMonitor objects (the open example) |
| Grafana | dashboards over metrics, logs and traces | where you look after a deploy | marking each deploy |
| SLI | a service level indicator: a measure of the service, such as the share of requests that succeed | it's what your service is judged by | dashboards and alerts |
| SLO | a service level objective: a target for an SLI, such as 99.9% of requests succeeding over 30 days | a service can miss it while every pipeline is green | the team's agreed targets |
| SLA | a service level agreement: a contract with users, with consequences if its SLOs are missed | it's why some SLOs are strict | contracts, not code |
| error budget | how much an SLO allows to fail: at 99.9%, one request in a thousand | a bad deploy uses it up | SLO dashboards |
| runbook | the written steps for handling a known problem | the alert that wakes someone should link to one | alerts, and the on-call team's pages |
Platforms and portals#
| Term | What it is | Why you care | Where you'll see it |
|---|---|---|---|
| platform, internal developer platform | "an integrated collection of capabilities defined and presented according to the needs of the platform's users", in the CNCF's words: the tools, templates and services an organisation's teams build and run on | the platform team builds it for teams like yours | a portal, templates, and self-service APIs |
| golden path | a supported way to build a kind of service: a project template and its docs, with a pipeline that builds, scans, tests, deploys and observes it | the easy way is also the supported one | software templates (developer portals) |
| developer portal | a website that lists every service and its owner, with links to its repository, pipelines and dashboards | it tells you who owns a service | Backstage (developer portals) |
| Backstage | an open-source developer portal, created by Spotify, now an incubating CNCF project | its templates may have made your repository | catalog-info.yaml (developer portals) |
| software template | a portal's recipe for a new repository: a skeleton of files, filled in and published | your pipeline file may be a copy of one | the annotation backstage.io/source-template (developer portals) |
The decoding method#
Everything about a pipeline job comes down to five questions: where it is defined, when it runs, where it runs, what it executes, and what goes in and out. Chapter 4 named them. This chapter turns them into a procedure you can follow on any job in any organisation, and shows it on one of MegaCorp's jobs from start to finish. A deploy gets the same five questions, asked of the cluster instead of the pipeline.
When something is already wrong, start from the symptom instead. The decision tree at the end sends you to the right chapter of symptom cards.
The team next door is on leave, and their nightly schedule ran nothing at 03:00: no pipeline, no error. You have never opened the project. Forty jobs, six included files, a policy stage, and a workflow someone wrote last year. You have twenty minutes before the stand-up. You need an order to ask questions in. (the card)
The five questions, in order#
| Question | Look first at | Then | Chapters |
|---|---|---|---|
| Where is it defined? | the job's name in the pipeline graph, then Full configuration | its extends chain, !reference tags and includes; if it's in no file, policies and settings | 12–20 |
| When does it run? | workflow:rules, then the job's own final rules | needs and stages; compare pipeline types in a truth table | 21–25 |
| Where does it run? | the job's page: the runner's name and tags | the runner's executor, and the job's image | 29–30 |
| What does it execute? | before_script, script and after_script in Full configuration | the scripts, functions and tools those lines call inside the image | 19 and 29 |
| What goes in and out? | the variables and secrets it needs, from YAML and from settings | its artifacts, caches, reports, and what it pushes or deploys | 26–32 |
Answer them in this order, because each answer narrows the next. You can't read a job's rules until you know where they come from. You can't judge its inputs until you know which runner and image it had.
Each question has one trap that catches most people:
- Defined: a policy's job is in no file at all (chapter 20).
- When: a job's own
rulesreplace a template's list completely (chapter 14). - Where: the runner's configuration changes jobs, and no project can see it (chapter 29).
- What: the script calls a tool or function from the image, whose code is somewhere else (chapter 19).
- In and out: a settings variable beats anything in the YAML (chapter 26).
One job, decoded#
Here is payments-api's image-build, decoded using nothing but the
specimen:
| Question | The answer for image-build | Found in |
|---|---|---|
| Where is it defined? | image-build in java-service.yml extends .image-build in container.yml, which extends .megacorp-base in base.yml | the Full configuration view, and chapter 20's steps |
| When does it run? | .megacorp-base's rules: merge requests, main and release tags, never schedules; it needs maven-build | the .rules library, and chapter 25's truth table |
| Where does it run? | on the megacorp-shared runner, in a Kubernetes pod, from the buildah image | default: tags, the runner's configuration, and the job's image |
| What does it execute? | the ci-tools functions and the AWS login from the snippets, then Buildah's login, build and push, and on main a second push, to Azure | the !reference lines in before_script, and ci-lib.sh |
| What goes in and out? | in: an ID token, AWS_ROLE_ARN, ECR_REGISTRY, AWS_REGION and the jar from maven-build; out: an image in ECR, on main in ACR too, and IMAGE_REF in a dotenv report | its variables and needs, and its artifacts |
After five short answers, the job holds no more surprises.
A deployment, decoded#
A deploy has the same five questions, with different places to look
(five questions for a deployment). Here they are for payments-api
1c7d4b09, deployed to production on eks-prod:
| Question | The answer for payments-api in production | Found in |
|---|---|---|
| Where is it defined? | the shared chart service at service-3.4.0, with the values in envs/production/payments-api.yaml, through Argo CD's ApplicationSet | the ApplicationSet's sources, then the values file (chapter 45) |
| When does it deploy? | when the deploy's merge request is merged in the environment repository, and Argo CD next syncs | the merge request, then the Application's sync status |
| Where does it run? | eks-prod, in the payments-production namespace | the ApplicationSet's destination |
| What does it run? | 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:1c7d4b09, in the Deployment that the chart renders | helm template with the same values (chapter 43) |
| What goes in and out? | in: settings from the values file, and secrets that the External Secrets Operator copies from Secrets Manager; out: logs, metrics and traces, and a mark on the dashboards | the ExternalSecret, and the service's dashboards (chapter 52) |
When a deploy is missing or wrong, answer them in the same order. The first answer that surprises you is where to look (chapter 52).
Starting from a symptom#
Each symptom card names a symptom, then its cause, how to confirm it, and how to fix it. The cards come in four chapters, one per group, with the most common causes first: didn't run, failed, wrong result, and stuck or drifting. For your own pipelines, the worksheet is a printable form of the five questions.
- Decode any pipeline in a fixed order: the five questions, each narrowing the next.
- Go from a symptom to a card in one step, using the four kinds of symptom.
- Write down what you found so the next person doesn't start from zero.
Spot the bugDecode publish-docs#
payments-api defines one job of its own. Answer the five questions for it:
publish-docs:
stage: deploy
image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
inherit:
default: [tags, retry]
variables: false
script:
- mkdocs build --strict
rules:
- !reference [.rules, default-branch]Show the answer
- Defined in payments-api's own file. Nothing extends it, and
inherittakes onlytagsandretryfromdefault:. - When: in pipelines for
mainonly, through thedefault-branchrule. - Where: on the megacorp-shared runner, because it inherits
tags, in thedocs:1.4image. - What:
mkdocs build --strict. - In and out: no YAML variables, because
variables: false, but settings variables still arrive. Out: nothing at all.
The last answer is the worry. The job builds the documentation site and then ends.
With no artifacts and no deploy step, the site is thrown away with the
job's pod. Its name promises a publication that never happens. See
the five questions.
Field guide to enterprise setups#
Large organisations build their pipelines in a handful of recognisable ways. Each leaves fingerprints: signs in a repository, a pipeline graph or a job log that tell you which setup you are looking at. Recognise the setup first, and you know where the answers live before you start reading YAML. This chapter lists ten pipeline setups and eight delivery setups, with their fingerprints and where to confirm each one.
Your organisation is not MegaCorp. This morning its pipeline gained a stage nobody on your team added, and your project's history shows no change: the golden pipeline you include moved under you. Every large organisation builds from the same ten shapes, and each shape has its own way of changing overnight. Name yours, and the rest of the book tells you where its traps are. (the card)
Ten setups and their fingerprints#
| Setup | Fingerprints | Confirm it | Chapters |
|---|---|---|---|
| The golden pipeline | a .gitlab-ci.yml of a few lines: one include: project: of a central file at a tag, a few variables | Full configuration shows dozens of jobs that the file never names | 14–15 |
| Components | include: component: lines ending in @1.2.0, with inputs: | the component's project, and its spec: inputs | 16 |
| The generated monorepo | a job that writes YAML, then a trigger that runs it; child pipelines to the right of the graph | the generator script, and the trigger's include: artifact: | 17 |
| The central deployer | deploy jobs that are trigger: project: jobs; environments that live in another project | the deployer project's pipeline and environments | 17, 40 |
| GitOps delivery | deploy jobs that only commit to a configuration repository | that repository's history, and the controller's sync status | 45 |
| Policy-injected jobs | jobs named like secret-detection-1, or in .pipeline-policy-pre and .pipeline-policy-post stages | **Secure › Policies** | 18, 36 |
| The invisible runner layer | behaviour that differs by runner; environment variables nobody set in YAML | the runner's name in the job log, and the runner's configuration | 19, 29 |
| The toolbox image | scripts calling commands such as mc or shell functions that the repository doesn't contain | the job's image, and the image's own repository | 19 |
| Cloud access by identity | id_tokens: in templates, a login snippet, and no cloud keys in settings | the role's trust policy, and aws sts get-caller-identity | 28, 38 |
| The 2019 legacy file | anchors and <<: merge keys, only: and except:, include: remote:, trigger tokens | the parse phase: anchors are resolved in Full configuration | 13, 15, 23 |
MegaCorp shows all ten: payments-api is a golden pipeline with a component, mono is a generated monorepo, the deployer is central and GitOps-based, and billing-batch is the 2019 legacy file.
Eight delivery setups and their fingerprints#
After the pipeline, deploys follow a handful of setups of their own. These are the ones you'll meet, with where to confirm each:
| Setup | Fingerprints | Confirm it | Chapters |
|---|---|---|---|
| Push deploys | a deploy job runs kubectl apply, helm upgrade --install or a cloud's CLI, and holds the cluster's credentials | the job's script, and where its kubeconfig or cloud identity comes from | 41, 38 |
| GitOps with Argo CD | an environment repository with Application or ApplicationSet files; deploy jobs that only commit | the Application's sync status and revision, in Argo CD's UI | 47 |
| GitOps with Flux | a clusters/<name>/flux-system/ folder in a repository, with Kustomization and HelmRelease objects | the Kustomization's Ready condition, and the revision it last applied | 46 |
| GitLab's agent for Kubernetes | a .gitlab/agents/<name>/config.yaml file in some project; jobs whose $KUBECONFIG has contexts for agent connections | **Operate › Kubernetes clusters**, and the agent's configuration file | 46 |
| Image automation | commits in the environment repository by a bot, changing only image tags | ImagePolicy and ImageUpdateAutomation objects, or an Argo CD ImageUpdater | 45 |
| Infrastructure from modules | .tf or .tofu files that call modules at ?ref= tags; plans in merge requests; backend "http" | **Operate › Terraform states**, and the plan job's output | 48 |
| Progressive delivery | Rollout objects where Deployments would be, or Canary objects beside them; AnalysisTemplates | the Rollout's or the Canary's status, step by step | 51 |
| A repository from a portal template | a catalog-info.yaml with a backstage.io/source-template annotation; a first commit called initial commit, by Scaffolder | the portal's template, and whether the pipeline file copies jobs or includes a central file | 53 |
MegaCorp shows five of them. Flux installs its platform add-ons, Argo CD deploys its services, the platform team's OpenTofu builds both clusters from modules, and payments-api's repository came from the portal's template. Its deploy marks come from Argo CD too (chapter 52).
No .gitlab-ci.yml, but pipelines run#
This is the most disorienting fingerprint of all. Five mechanisms can produce it, and each has a place to confirm it:
As text
- Does Settings › CI/CD › General pipelines name a configuration file elsewhere? Yes: The pipeline comes from that file, perhaps in another project. No: the next step.
- Are the only jobs scanners with names like secret-detection-1? Yes: A scan execution policy creates a pipeline file for the project implicitly. No: the next step.
- Is a pipeline execution policy set to override_project_ci? Yes: The policy's configuration replaces the project's entirely. No: the next step.
- Is Auto DevOps turned on for the project, group or instance? Yes: GitLab's Auto DevOps pipeline runs. No: the next step.
- Does the project belong to a compliance framework with its own pipeline? Yes: A compliance pipeline runs; these are deprecated in favour of policies. No: the next step.
- Check the pipeline's source, and its first job's origin (chapter 13)
MegaCorp's web-portal is the first case. Its settings point at
pipelines/web.yml in ci-templates, so its repository needs no pipeline file
(chapter 18):
ci_cd_configuration_file: pipelines/web.yml@megacorp/devops/ci-templates- Recognise the ten shapes of enterprise setup, and name which ones your organisation uses.
- Say where each shape hides its logic, and which chapter decodes it.
- Predict the traps a shape brings before you meet them.
Spot the bugWhich setup is this?#
You join a team and open their newest pipeline. You notice three things:
- the project's
.gitlab-ci.ymlis nine lines long, and includes one file fromplatform/ci-libraryatref: v7.1.0 - one job,
deploy-prod, is a trigger job pointing atplatform/release - two jobs are called
sast-1andsecret-detection-1
Show the answer
- A golden pipeline: read
platform/ci-libraryatv7.1.0, starting from Full configuration. - A central deployer: the real deploy happens in
platform/release's pipeline and environments. - Policy-injected jobs: the numbered scanners come from a scan execution policy, so check Secure › Policies.
Idioms#
Central teams repeat a small number of patterns, called idioms, in every organisation. Each one solves a real problem for the team that writes the templates. Each also hides something from the people who read the pipelines. Learn to spot them, and an unfamiliar pipeline stops looking unfamiliar. For each idiom, this chapter shows what it looks like, why central teams use it, and how it confuses a reader.
A job's rule says if: $DEPLOY_KEY. It matches on main and never on a branch. The variable is protected, so on a branch it is empty, and an empty variable is false. MegaCorp's templates are full of idioms like this: patterns that look like magic until you have seen each one once. (the card)
A rules library, pulled in with !reference#
This is the idiom that confuses most readers, so here it is in full. MegaCorp keeps its conditions in one hidden job:
.rules:
mr:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
default-branch:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
release-tag:
- if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
never-on-schedule:
- if: $CI_PIPELINE_SOURCE == "schedule"
when: neverJobs assemble their rules from it: - !reference [.rules, mr].
- Why central teams use it: "which pipelines does this job run in" is decided once, with one name for each condition. Every template agrees on what a release tag looks like.
- What it hides: the conditions live in another file, so a job's rules
read like a list of names. The names say nothing about the order, and in rules the order
decides the outcome, because the first match wins. If you replace the job's
rules:, every referenced rule disappears with them (chapter 14).
The catalogue#
| Idiom | Looks like | Why central teams use it | How it confuses you | Chapters |
|---|---|---|---|---|
| The hidden base job | extends: .megacorp-base | shared rules, variables and expiry in one place | settings arrive from files you never opened; its variables beat your top-level ones | 14 |
| Script snippets | - !reference [.snippets, aws_login] | the same few script lines in every template | a job's own before_script drops them; the functions they call live in the image | 14, 19 |
| Variables as parameters | image: maven:3.9-jdk${JAVA_VERSION} | one template serves many projects | the default is in the template; a settings variable beats your YAML | 14, 26 |
| Toggles | if: $SKIP_SONAR == "true" then when: never | an escape hatch without editing the template | a toggle set in settings is invisible; policy jobs ignore toggles | 34, 36 |
| The pinned include | include: project: … ref: v4.2.0 | projects upgrade when they choose to | what runs is that tag's content, not the central repo's current version; a tag can be moved | 15 |
| Components with inputs | component: …/sonar-scan@2.1.0 with inputs: | typed, versioned settings | inputs are filled in before anything else, even into job names | 16 |
| The dotenv hand-off | echo "IMAGE_REF=…" >> build.env | passes a build's result to later jobs | no file shows the value; needs and dependencies can block it; rules can't see it | 31 |
| The trigger to a deployer | trigger: project: …/deployer with strategy: depend | one team controls every environment | the deploy happens in another project's pipeline | 17, 40 |
| Never on schedule | the first rule excludes schedule | nightly pipelines run only what they should | jobs are silently absent from scheduled pipelines | 23, 25 |
| Opting out with inherit | inherit: default: [tags, retry] | a job that shouldn't get the shared image or scripts | which defaults still apply is easy to misread | 14 |
| No duplicate pipelines | a workflow rule with $CI_OPEN_MERGE_REQUESTS | one pipeline per push, not two | once a merge request opens, branch pipelines stop appearing | 22 |
| Build once, deploy by reference | the image tagged with $CI_COMMIT_SHA, then passed on | what is tested is what ships | a pipeline that rebuilds breaks the promise quietly | 39 |
Most real pipelines combine several idioms in a single job. Recognising each one tells you which file to open next.
- Read the dozen idioms MegaCorp's templates use, and say what each is for.
- Recognise a rule that tests a protected variable, and predict where it matches.
- Write your own rules and snippets the way the central team will expect to read them.
Spot the bugCount the idioms#
This is the template behind payments-api's image-build:
.image-build:
extends: .megacorp-base
stage: package
image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
id_tokens:
MC_ID_TOKEN:
aud: https://gitlab.example.com
variables:
AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
before_script:
- !reference [.snippets, functions]
- !reference [.snippets, aws_login]
script:
- mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
- buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
- buildah push "$IMAGE:$CI_COMMIT_SHA"
- !reference [.snippets, acr_push]
- echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
artifacts:
reports:
dotenv: build.envShow the answer
- The hidden base job:
extends: .megacorp-base, which also brings in the rules library and never on schedule. - Script snippets: the three
!reference [.snippets, …]lines, two before the script and one in it. - Variables as parameters:
AWS_ROLE_ARNandECR_REGISTRY, which a project could override. - The dotenv hand-off:
IMAGE_REFwritten tobuild.env. - Build once, deploy by reference: the image tagged with
$CI_COMMIT_SHA.
The job reaches payments-api through a pinned include as well:
java-service.yml includes it at v4.2.0. See
the catalogue.
Worksheet: decode your own pipeline#
Print this page, or copy it into a private note, and fill it in for one of your own projects. It is the five questions, for a job and for a deployment, plus the few facts about your organisation that make every answer easier. Keep your answers private, because they describe your organisation's systems.
Once for your organisation#
Most of these answers are the same for every project, so you only find them once:
| Question | Where to find the answer | Your answer |
|---|---|---|
| Where do the central templates and components live? | the include: lines of any project, and the platform team's group | … |
| Which versions do projects pin, and who moves them? | the ref: of each include, and the templates' release notes | … |
| Which runners run your jobs, and which tags do they need? | Settings › CI/CD › Runners, and default: tags | … |
| Which executor do those runners use? | the first lines of any job log | … |
| Which CI/CD variables do your groups and the instance set? | each group's Settings › CI/CD › Variables, if you can see it | … |
| Which security policies apply? | Secure › Policies | … |
| How do jobs reach the cloud? | id_tokens: in the templates, and the login script | … |
| Where do deployments really happen? | trigger jobs, environments, and any GitOps repository | … |
For each job you need to understand#
| Question | Where to look | Your answer |
|---|---|---|
| Where is it defined? | Full configuration; its extends chain; if it's in no file, policies | … |
| When does it run? | workflow:rules, its final rules, needs; compare pipeline types | … |
| Where does it run? | the job page's runner and tags; the job's image | … |
| What does it execute? | its scripts, and the tools and functions they call | … |
| What goes in and out? | variables, secrets, artifacts, caches, reports, images, deployments | … |
For each deployment you need to understand#
The same five questions, asked after the pipeline (a deployment, decoded):
| Question | Where to look | Your answer |
|---|---|---|
| Where is it defined? | the environment repository; the chart and its version; values files, overlays and inline values | … |
| When does it deploy? | the commit or merge request that changed it; the controller and its sync policy | … |
| Where does it run? | the cluster and namespace, in the controller's Application or Kustomization | … |
| What does it run? | the image reference, as rendered; whether that tag is in the cluster's registry | … |
| What goes in and out? | settings and secrets, and where each comes from; its logs, metrics and traces, and where they're kept | … |
When something goes wrong#
Write down, in this order:
- The symptom, in one sentence, and its group: didn't run, failed, wrong result, or stuck or drifting (chapter 55).
- The pipeline and job, with links, and what changed since the last good run: a commit, a setting, a template version, a runner, a date.
- The cards you tried, and what each confirmation showed.
- The fix, and who had to make it: you, the platform team, the security team or the cloud team.
Questions for your platform team#
Some answers aren't visible from a project at all. These are worth asking once:
- What do the runners add to every job: environment variables, a
pre_build_script, limits? - Which of the group's CI/CD variables are protected, and why?
- Which stored tokens do the pipelines depend on, who owns them, and when do they expire?
- How are template versions released, and can a tag ever move?
- Which security policies are enforced, and on which projects?
- Which controller deploys each environment, how often does it sync, and who may merge into the environment repository?
- Where do the services' logs, metrics and traces go, and is each deploy marked on the dashboards?
Cards: didn't run#
Use these cards when a job is missing from a pipeline, when the pipeline wasn't created, or when a job ran somewhere it shouldn't have. The most common causes come first. Each card gives the symptom, the cause, how to confirm it, and the fix. One more card for this group is in chapter 14: your rules replaced the template's. The last cards cover what happens after the pipeline.
Merge request pipelines and branch pipelines are different pipelines, with
different variables. CI_COMMIT_BRANCH, for example, isn't set in merge request
pipelines, so a rule that tests it is false there. The job has no rule for
$CI_PIPELINE_SOURCE == "merge_request_event".
Open both pipelines and compare their job lists. Each pipeline's page shows
its source. Then read the job's final rules in Build › Pipeline
editor › Full configuration.
Add a rule for merge request pipelines, or reuse your organisation's rule from its
rules library with !reference. See merge request
pipelines.
workflow:rules decide whether a pipeline exists. None of them
matched this push, or the one that matched said when: never.
Run pipeline and the API answer with:
The pipeline did not run. Review the workflow:rules configuration for the pipeline.Read workflow: in Full configuration.
Add the missing pipeline source or branch to the workflow rules, or push where the workflow allows. See the first matching rule.
Every job's rules excluded it for this source and branch. MegaCorp's deployer
does this deliberately: all its jobs need IMAGE_REF, which only a trigger
supplies.
GitLab says:
The resulting pipeline would have been empty. Review the rules configuration.The Validate tab, run for the same branch, shows no jobs.
Provide what the rules expect, such as the variable a trigger would send, or change the rules. See when a pipeline is missing.
v1.5.0-rc1 creates no pipeline, and shows no
error anywhere.The workflow allows tag pipelines, but no job's rules match this tag. MegaCorp's
release rule accepts only tags shaped like v1.5.0, exactly
^v\d+\.\d+\.\d+$. A pipeline with no jobs isn't created, and on a push GitLab
doesn't say why.
Run Validate for the tag: it shows no jobs. Starting a pipeline
for the tag by hand gives The resulting pipeline would have been empty. Review the
rules configuration.
Name tags the way the rules expect, or widen the rule if release candidates should build. See what the truth table says.
secret_detection, appear in branch
pipelines but never in merge request pipelines.GitLab's scanner templates run their jobs in merge request pipelines only when
AST_ENABLE_MR_PIPELINES is "true". Otherwise the jobs run in
branch pipelines only.
In Full configuration, the job's rules contain the
AST_ENABLE_MR_PIPELINES conditions. The branch pipeline for the same commit
has the job.
Set AST_ENABLE_MR_PIPELINES: "true", in the YAML or in settings. The
job then moves to the merge request pipeline. Or redefine the job's rules. See
the merge request gap.
Its rules start with a rule that excludes schedules. MegaCorp's
.megacorp-base begins with !reference [.rules,
never-on-schedule], so every job that extends it skips scheduled pipelines.
The job's final rules in Full configuration start with the schedule rule, and
the pipeline's source is schedule.
If the job should run on a schedule, give it rules without that line, or ask the template's owners to change the base. See each job's final rules.
The job is manual, through when: manual on the job or in the rule
that matched. Manual jobs run only when someone starts them. A manual job added by a rule
blocks the later stages, unless it has allow_failure: true.
The job page offers Run, and its rules or when in
Full configuration say manual.
Run it. If nothing should wait for it, add allow_failure: true to that
rule. See manual jobs.
rules: changes runs on the first push of a new branch, or
on a tag, although none of its files changed.For new branches, for tags, and for pipelines without a push event,
rules: changes is always true. There is nothing to compare with. On the local
GitLab, mono's first push ran its docs job although there was no
docs/ folder.
The pipeline is a tag pipeline, or the branch's first pipeline.
Compare against a fixed branch with changes: compare_to:, or add a
rule that excludes tags. See rules about files.
The workflow allows both a branch pipeline and a merge request pipeline for the same push.
The pipelines list shows two pipelines for the same commit, one of them marked as a merge request pipeline.
Add a workflow rule that skips branch pipelines while a merge request is open,
using $CI_OPEN_MERGE_REQUESTS. See duplicate
pipelines.
Someone set its toggle, such as CONTAINER_SCANNING_DISABLED, in the
project's or a group's CI/CD settings. The template's first rule removes the job.
MegaCorp's own container scanning rule accepts only "true".
Look for the variable in the project's and each group's CI/CD settings. Its rule is the job's first.
Remove the setting if the scan should run. A scan execution policy's jobs are never affected by toggles. See toggles against policies.
Each SAST analyzer's job exists only when files that it reads are in the
repository, such as *.java or *.py for
semgrep-sast.
In Full configuration, compare the analyzer's exists patterns with
the repository's files.
None is needed if the repository has no code in those languages. Otherwise check that the source files are where the patterns look. See what the templates add.
Those jobs' rules require the dependency_scanning feature, which
comes with Ultimate. On Free and Premium, the template adds nothing, without an
error.
The jobs' rules in Full configuration test
$GITLAB_FEATURES.
Use Ultimate, or run another dependency scanner as an ordinary job (chapter 35).
The needed job's rules left it out of this pipeline, while the job that needs it is still in.
GitLab's message, captured here with other job names, says:
'unit-tests' job needs 'compile' job, but 'compile' does not exist in the pipeline.
This might be because of the only, except, or rules keywords. To need a job that
sometimes does not exist in the pipeline, use needs:optional.Mark the need as optional, or give both jobs matching rules:
unit-tests:
needs:
- job: compile
optional: trueSee needs.
A job names a stage missing from stages:. Often a project redefines
stages: without test, which GitLab's scanner templates
use.
The message names the job, its stage and the stages that exist:
orphan job: chosen stage nowhere does not exist; available stages are .pre, build, .postAdd the stage to stages:, or move the job to one that exists. See
stages.
Two common forms fail. A negation written as !( … ) is invalid. A
quoted variable, as in "$VAR" == "x", breaks the YAML itself.
CI Lint shows one of these, captured on the local GitLab:
jobs:job:rules:rule if invalid expression syntax
(): did not find expected key while parsing a block mapping at line 5 column 7Use != for negation, and never quote the variable:
if: $CI_COMMIT_BRANCH != "nope". See writing
conditions.
Rules are decided when the pipeline is created, before any job runs. Values from
a dotenv report, or from export in a script, don't exist yet.
Find where the variable is set. If it is set by a job, no rule can see it.
Make the decision inside the job's script instead, or in a child pipeline that a later job generates. See when each thing is decided.
if: $DEPLOY_KEY matches on main, but never
on feature branches.The variable is protected. Protected variables exist only in pipelines on protected branches and tags, so elsewhere the rule sees an unset variable.
In the CI/CD settings where it is defined, the variable has Protect variable ticked.
Don't gate jobs on secrets. Test the branch, or an ordinary variable, instead. See who gets a protected value.
The cards from here on are for what happens after the pipeline: the environment repository, the controllers, the cluster and the clouds.
Production deploys wait for a person. mc deploy pushes production's change to a branch of the environment repository and opens a merge request, which nobody has merged yet. The controller deploys only what is on main.
The environment repository has an open merge request called Deploy payments-api 1c7d4b09 to production, or similar.
Review it and merge it. See promotion by merge request.
MegaCorp's image-build pushes to ACR only in pipelines on main. This image was built in a merge request, tag or feature-branch pipeline, where Entra ID wouldn't accept the job's token anyway.
The job log says:
[megacorp] not on main, so not pushing to megacorpprod.azurecr.ioDeploy images built on main, which reach both registries. See how a MegaCorp job logs in.
Cards: failed#
Use these cards when a job, or the pipeline, turned red. Start with the job log's first error, not its last line. The most common causes come first. Each card gives the symptom, the cause, how to confirm it, and the fix. The last cards cover what happens after the pipeline.
mc_log, can't be found. The same template works in other projects.The job sets its own before_script. That replaces the template's or
the default's list completely, including the line that loaded the functions.
In Full configuration, the job's before_script shows only your
lines.
Put the template's lines back with !reference, then add yours. See
the before_script exercise.
The image is in a private registry, and the runner has no credentials for it. The job's own script can't help, because the runner pulls the image before the job starts.
The error appears in the job log's preparation lines, before any of your commands.
Give the runner credentials with DOCKER_AUTH_CONFIG, or a credential
helper on the runner. For images in the same GitLab, allow your project's job token in
the image's project. See private registries.
Its token is a protected variable. Protected values reach only pipelines on protected branches and tags, so elsewhere the variable is empty.
In the CI/CD settings, the variable is protected. In the job, print the variable's length, never its value.
Decide with the token's owners: an unprotected token with fewer rights, a separate token for merge requests, or no sign-in there. See who gets a protected value.
The role's trust policy doesn't accept this token. Usually the
sub pattern allows branches but not tags, or the audience is wrong. A wildcard
written with StringEquals never matches.
An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentityPrint the token's claims, not the token, and compare sub and
aud with the trust policy.
Ask the account's owners to add a pattern for the missing pipelines, using
StringLike. See when AWS says no.
The downstream project limits who may set pipeline variables. The variables the trigger job sends, including top-level ones it inherits, count as pipeline variables.
The trigger job shows:
Failed - (downstream pipeline can not be created, Insufficient permissions to set pipeline variables)Send inputs instead of variables, or ask the downstream project to allow the role. See values given when the pipeline starts.
404 Not Found, although
the project exists.The job used CI_JOB_TOKEN. The other project hasn't added yours to
its job token allowlist, or the person who started the pipeline isn't a member
there.
A refused clone says:
remote: The project you were looking for could not be found or you don't have permission to view it.A Maintainer of the target adds your project, or its group, under Settings › CI/CD › Job token permissions. See the job token.
unresolved reference.The job has a shallow clone: only the newest commits, 20 by default, and only the pipeline's own ref.
The commit the script needs is older than the clone's depth, or is on a branch the job didn't fetch.
Raise GIT_DEPTH for that job, or fetch what the script needs, for
example with git fetch --unshallow. See shallow
clones.
amazon/aws-cli, the job fails
without running your script. The same job works on another runner.The image's entrypoint runs the tool, not a shell. The Docker executor keeps the entrypoint. The Kubernetes executor ignores it by default.
Compare the runners' executors. The failing one uses Docker.
Set entrypoint: [""] under image:. See the
job's image.
*** WARNING: Service XYZ probably didn't start properly.The service lacked a setting it needs to start, such as its password. That value is a CI/CD variable from settings, and those never reach service containers.
The variable is defined in settings, not in the YAML.
CI_DEBUG_SERVICES: "true" shows the service's own log, but can reveal masked
values.
Re-assign it in the YAML under a different name, such as
POSTGRES_PASSWORD: $TEST_DB_PASSWORD. See services.
timeout says 2
hours.The runner has a shorter maximum job timeout. A runner's maximum wins over the job's own timeout and the project's.
The runner's settings show its maximum job timeout.
Use a runner with a longer maximum, or split the job. See time limits.
The child configuration's jobs have no rules. In a merge request pipeline, a
child whose jobs have no rules gets no jobs at all, so it can't be created. On the local
GitLab, mono's docs trigger failed this way in every merge request
pipeline.
The trigger job's failure reason is
downstream_pipeline_creation_failed.
Give the child's jobs rules that allow merge request pipelines. See child pipelines.
$MY_VAR instead of a
value.The job defines MY_VAR: $MY_VAR. The job's own definition hides the
default of the same name, so there's nothing to expand.
The job's variables in Full configuration re-use the name.
Use a different name for the job's variable. See variables in YAML.
The earlier job's artifacts: paths matched nothing: the path is
wrong, or the file wasn't created.
The earlier job's log says No files to upload.
Fix the path, which is relative to the project directory, or the command that makes the file. See artifacts.
AWS must download GitLab's signing keys from the instance, and can't reach it. This happens with a self-managed GitLab behind a firewall.
An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn't retrieve verification key from your identity providerMake the instance's OIDC keys reachable, or publish them elsewhere, as GitLab's documentation describes. See small things that bite.
Prevent outdated deployment jobs is on, and a newer deployment already ran. GitLab judges a job's age by when it started.
The deployment job is older than the latest deployment, and therefore failed.Deploy from a newer pipeline, or use Rollback environment to go back deliberately. See one at a time, newest wins.
ImageTagAlreadyExistsException.The repository has immutable tags, and this tag was already pushed, often by an earlier pipeline that built the same commit.
ECR already lists the tag, with an earlier push time.
Don't rebuild what was already built: deploy the existing image, and add new tags
with put-image. See promoting without
rebuilding.
A job's image must contain sh or bash, and
grep, because the runner sends the script to a shell. Very small images
leave them out.
Run the image locally and look for sh.
Use a variant of the image that includes a shell, or build one. See the job's image.
The cards from here on are for what happens after the pipeline: the environment repository, the controllers, the cluster and the clouds.
ErrImagePull, then ImagePullBackOff.Kubernetes can't pull the image: its name or tag doesn't exist in the registry that the cluster uses, or the nodes aren't allowed to pull from it. Kubernetes keeps trying, and waits longer each time, up to five minutes.
The pod's events, in kubectl describe pod, name the image and the registry's answer. An answer that ends in not found means the registry has no such tag: check the tag, and the registry. On aks-prod, the usual cause is a tag that exists only in ECR (an image not in Azure's registry). An answer that ends like this means the tag may well exist, but the cluster isn't allowed to pull it:
pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failedDeploy a tag that exists in the cluster's registry. If it exists and the pull is refused, check what the cluster pulls with. On aks-prod, the nodes need AcrPull. A pull secret made from a GitLab deploy token needs read_registry on the project that built the image; a token from any other project is refused. See from tag to pod.
CrashLoopBackOff.The container starts and then exits, over and over. Kubernetes waits longer before each restart: 10 seconds, then 20, then 40, up to five minutes. The cause is in the program, or in what it was given, such as a missing setting or secret.
Read the last run's output with kubectl logs <pod> --previous. The pod's events show the restarts.
Fix what the logs show, in the code or in the environment's values. See config for each environment.
image.tag is set for each environment.MegaCorp's chart refuses to render without an image tag. Its Deployment template uses Helm's required, and the environment's values file has no image.tag.
Error: execution error at (service/templates/deployment.yaml:28:52): image.tag is set for each environmentThe values file for that environment has no tag under image.
Set the tag. mc deploy normally writes it; a values file made by hand needs one too. See values.
AADSTS70021: No matching federated identity record found for presented assertion.No federated credential on the application matches the token's issuer and subject. Entra ID compares them exactly, with no wildcards, so each new branch or tag needs a credential of its own. A credential added in the last few minutes may not have spread yet.
Print the token's claims, and compare sub with each federated credential's subject, character by character.
Add a credential for that exact subject, or sign in only from refs that have one. See what must match.
AADSTS700212, about the token's audience.The federated credential's single audience differs from the token's aud. For a pod on AKS, the audience is api://AzureADTokenExchange. For a GitLab job, it is whatever the job's id_tokens: asks for.
Compare the credential's audience with aud in the token's claims.
Make them the same. See pods and their cloud identity.
az acr login fails, although the identity may push to the registry.az acr login needs to read the registry through Azure Resource Manager. AcrPush gives data-plane access only, to push and pull images, and no control-plane access.
The identity's role assignments on the registry show AcrPush and nothing else.
The registry's owners add the role that Microsoft documents for az acr login: Container Registry Contributor and Data Access Configuration Administrator. See what must match.
AZURE_FEDERATED_TOKEN_FILE variable.AKS gives a pod its Azure identity only when the pod template has the label azure.workload.identity/use: "true", and its service account names the identity in the azure.workload.identity/client-id annotation. Without the label, the webhook leaves the pod alone.
kubectl get pod <pod> -o yaml shows neither the label nor the variable.
Add both through the chart's values, as MegaCorp's aks-prod values do. See pods and their cloud identity.
AccessDenied naming the nodes' role, or finds no AWS credentials at all.The pod has no identity of its own: its service account has neither the eks.amazonaws.com/role-arn annotation nor a Pod Identity association. The AWS SDK then falls back to whatever the node allows, often the node's own role.
The service account's annotations, and the cluster's Pod Identity associations for that namespace and service account.
Annotate the service account, or ask for an association, through the platform team's OpenTofu. See pods and their cloud identity.
Error acquiring the state lock.Another run holds the state's lock, or a cancelled or killed job took it and never released it.
The log shows the lock:
│ Error: Error acquiring the state lock
│
│ Error message: HTTP remote state already locked:
│ ID=5b0e8c3a-7d2f-4c1e-9a64-3f8b2d1c0e97In the Lock Info that follows, ID, Who and Created describe the lock. Operate › Terraform states shows the state as locked.
If a run is still going, wait for it. If nothing is running, a Maintainer unlocks the state under Operate › Terraform states, or runs tofu force-unlock with the lock's ID. Then read a fresh plan before anything applies. See state and its lock.
Helm install failed and timeout
waiting for, and nothing tries again.A Helm action has five minutes by default, and that includes the chart's hook
Jobs. The install ran out of time, often because an image was still downloading. Flux
retries a failed install only if install.remediation.retries allows it, and
the default is 0, so the release stays failed until its settings or its chart
change.
flux get helmreleases -A names what the install was waiting
for:
Helm install failed for release argocd/argo-cd with chart argo-cd@10.9.0: failed pre-install: timeout waiting for: [Job/argocd/argo-cd-argocd-redis-secret-init status: 'InProgress']A Kustomization that applied it with wait: true fails its health check
too.
Once the cause has passed, flux reconcile helmrelease argo-cd -n argocd
--force makes one more attempt. To avoid it next time, give the HelmRelease a longer
timeout, or some install.remediation.retries. See
charts through Flux.
resource :Namespace is not permitted in project,
and nothing is deployed.The Application syncs with CreateNamespace=true, so Argo CD makes the
destination's Namespace. A Namespace is a cluster-wide object, and a project other than
default allows no cluster-wide kinds unless its
clusterResourceWhitelist lists them.
In argocd app get, the Namespace's row shows
SyncFailed, with this message:
resource :Namespace is not permitted in project paymentsThe AppProject has no clusterResourceWhitelist, or one without
Namespace.
Allow the kind in the project, only with the team's names:
clusterResourceWhitelist:
- group: ""
kind: Namespace
name: payments-*Or have the platform team create the namespaces, and drop CreateNamespace.
See Argo CD's objects.
x509: certificate signed by unknown authority.The Git server's certificate comes from a private authority, as a self-managed GitLab's often does, and Argo CD's repo server doesn't trust it.
The Application's condition names the repository it couldn't read:
ComparisonError: Failed to load target state: failed to generate manifest for source 1 of 2: rpc error: code = Unknown desc = failed to list refs: Get "https://gitlab.example.com/megacorp/platform/charts.git/info/refs?service=git-upload-pack": tls: failed to verify certificate: x509: certificate signed by unknown authorityGive Argo CD the authority's certificate: in the argocd-tls-certs-cm
ConfigMap, under the server's name, or with argocd cert add-tls. Restarting the
repo server makes sure it has read it; then refresh the Application. See
how it syncs.
Cards: wrong result#
Use these cards when everything passed, but the result is wrong: the wrong value, the wrong image, the wrong version, or nothing at all. These are the hardest problems, because no log line points at them. The most common causes come first. The last cards cover what happens after the pipeline.
The same name is set in CI/CD settings, on the project, a group or the instance.
A settings variable beats every value in YAML, even a job's own. MegaCorp's group sets
MAVEN_CLI_OPTS this way.
The job log never says where a value came from, and Full configuration shows only YAML. Look for the name in the project's CI/CD variables, then each group's.
Change or remove the settings variable, or use a name the settings don't set. See who wins.
.gitlab-ci.yml is ignored in some
jobs, which use a template's value instead.A job's own variables, including those it gets through extends, beat
top-level ones. MegaCorp's .megacorp-base sets MC_TEAM:
unknown, so every job that extends it reports unknown.
In Full configuration, the job's own variables list the
name.
Set the value on the job, or ask the template's owners to remove the job-level default. See the MC_TEAM exercise.
A later pipeline built the same commit again and pushed the same tag. With mutable tags, the rebuild replaced the image that staging tested.
ECR shows that the tag was pushed after the staging deploy, and the release pipeline contains an image build.
Build once, and promote the existing image. Make tags immutable, so that a rebuild fails loudly. See from build to deploy.
With GitOps, green means only that the new version was written to Git. The controller may not have synced yet, or the rollout failed in the cluster. A manual deploy job that is still waiting deploys nothing, either.
Check the GitOps repository's latest commit, then the controller's last sync, then the cluster's rollout.
Wait for the sync or ask for one, and fix any failed rollout. See MegaCorp's handoff.
The gate's job is allowed to fail. Or the deploy's needs don't
include it, so the deploy never waits for it.
The gate job shows an orange warning, and the deploy job's needs
name other jobs only.
Decide with the owners whether the gate should block. If it should, remove
allow_failure and make the deploy wait for it. See
the quality gate exercise.
The merge request's Reports tab and the pipeline's Security tab need Ultimate. Also, GitLab's scanners run in the branch pipeline by default, not in the merge request pipeline.
Check your tier, and which pipeline has the scanner jobs. The raw report is in the scanner job's artifacts.
Below Ultimate, read the report artifact. To scan in merge request pipelines, set
AST_ENABLE_MR_PIPELINES: "true". See where the
findings appear.
The later job doesn't receive that job's artifacts. Its needs or
dependencies leave the job out, or it uses needs with
artifacts: false. An empty dependencies: [] blocks them even when
needs names the job.
Read the later job's needs and dependencies in Full
configuration, and check that the earlier job saved a dotenv report.
Add the earlier job to needs, and to dependencies too if
the job has that list. See dotenv, and
the container scan that gets no image.
Two jobs share a cache key but cache different paths, so each overwrites the other's cache. Or a job caches and keeps the same path, and the cache is restored before the artifacts.
Compare the jobs' cache keys and paths in Full configuration.
Give each set of paths its own key, and don't cache what you keep as an artifact. See cache.
The script compares with HEAD~1, which is only the previous commit.
Changes in the push's earlier commits are missed.
Read the base the script uses in push pipelines.
For pushes, compare with CI_COMMIT_BEFORE_SHA, with a fallback for when
it is all zeros. See the monorepo exercise.
1.8.0-SNAPSHOT.The build takes its version from the committed build file. Nothing passes the release tag into the build.
Read the version in pom.xml or package.json, and the
build command.
Pass CI_COMMIT_TAG into the build in tag pipelines, or commit the
release version before tagging. See which version is it.
The login step didn't run, often because the job's own
before_script replaced the template's. The AWS CLI then used the next
credentials it found, such as the runner's own role.
aws sts get-caller-identity prints the role really in use.
Restore the login step. See which credentials the AWS CLI uses.
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are still set
in CI/CD settings, from before OIDC. Keys in environment variables come first, unless a
login step overwrites them.
Look for the two variables in the project's and groups' settings, and run
aws sts get-caller-identity.
Remove the old keys, after checking that nothing else still needs them. See which credentials the AWS CLI uses.
The job has no artifacts and no step that publishes its output, so
the output is deleted with the job's environment. MegaCorp's publish-docs
builds its site this way.
The job has no artifacts: and no deploy command.
Keep the output as an artifact, or add a step that publishes it. See the publish-docs exercise.
1.10 turns into 1.1, and a rule
comparing it never matches.YAML reads an unquoted 1.10 as a number, and the number loses its
trailing zero.
The value is unquoted in the YAML. On the local GitLab, VERSION:
1.10 reached a rule as 1.1.
Quote it: VERSION: "1.10". See YAML in five
minutes.
Its token is protected, so merge request pipelines on feature branches get an empty token, and the scan can't sign in. The job is allowed to fail, so each failure is only a warning.
The scan job shows warnings in merge request pipelines, and the token is protected in the settings.
Decide on a token for merge requests, and make the failures visible. See the quality gate that never ran.
Runners differ in their executor and their configuration: extra environment
variables, a pre_build_script, resource limits. No project can see any of
it.
The job log names the runner. Compare the runners' settings with the platform team.
Pin the job to the right runners with tags, or make the runners consistent. See what the runner's configuration decides.
GitLab's container scanning template gives the job dependencies: [].
An empty list blocks every artifact download, dotenv reports included, and adding
needs doesn't lift it. So a value such as IMAGE_REF, from the
build job's dotenv report, is empty in the scan, and so is the CS_IMAGE built
from it.
In Full configuration, the merged container_scanning job has both
dependencies: [] and needs. On the local GitLab 19.3, a
job with both keys printed an empty value.
Add dependencies: [image-build] to your version of the job, naming the
job that writes the report. See changing a template's
job.
The cards from here on are for what happens after the pipeline: the environment repository, the controllers, the cluster and the clouds.
An images: entry changes only images whose name matches its name exactly. A misspelt name matches nothing, and Kustomize says nothing.
kustomize build on the overlay still shows the base's image, such as …/node-service:latest.
Copy the image's name from the base. See what Kustomize does silently.
Something with higher precedence sets the same value there. In Argo CD, parameters beat inline valuesObject, which beats value files. MegaCorp's aks-prod ApplicationSet sets the image's repository inline, for example.
Read the Application's sources, and look for valuesObject, values or parameters that set the same key.
Change the value where it wins, or remove the inline one. See values.
kubectl disappears within seconds.Argo CD's automated sync, with selfHeal: true, puts back what Git says a few seconds after the cluster drifts. Flux corrects drift at its next reconciliation too.
The Application's syncPolicy, and a sync in its history just after your change.
Change the environment repository instead. See drift, and going back.
Deleting a file from Git deletes its objects only when the controller prunes. Argo CD prunes only with prune: true, which is off by default, and a Flux Kustomization only with prune: true.
The Application's or the Kustomization's settings. Argo CD also marks the leftover object as needing to be pruned.
Turn pruning on, or delete the object by hand once. See drift, and going back.
Pods read ConfigMaps and Secrets as environment variables when they start, and never again. A rotated secret also waits for the External Secrets Operator's next refresh, hourly at MegaCorp.
The Secret or ConfigMap holds the new value, and the pods started before it changed.
Restart the pods, with a new deploy or Argo CD's restart action. See config for each environment.
helm list shows no release for a service that runs in the cluster.Argo CD renders charts with helm template and applies the objects itself, so Helm never records a release. Only charts installed by Helm, or by Flux's HelmRelease, appear.
The service has an Application in Argo CD.
Nothing is broken. Read the release in Argo CD instead. See releases.
Cards: stuck or drifting#
Use these cards when jobs wait for ever, run slowly, or break although nobody changed anything. Drift has no commit to blame, so look outside the repository: runners, tokens, included files, images and policies. The most common causes come first. The last cards cover what happens after the pipeline.
No online runner has every tag the job lists. A job's own tags
replace the default tags completely.
The page says:
This job is stuck because of one of the following problems. There are no active runners online, no runners for the protected branch, or no runners that match all of the job's tags:The job's tags are listed after the message.
Fix the job's tags or the runner's. See how a job finds a runner.
The project's runners are offline: none has contacted GitLab for more than two hours.
The job page says:
This job is stuck because the project doesn't have any runners online assigned to it.Settings › CI/CD › Runners shows their status.
The runners' owners restart or replace them. See which runners a project can use.
main, but stay pending on feature branches.The only runners that fit are protected. Protected runners take jobs only from protected branches and tags.
The runners' settings show Protected.
Provide an unprotected runner for other branches, or protect the branch. See how a job finds a runner.
The runners are already running as many jobs as they allow. Instance runners also share a fair-usage queue between projects.
Many jobs are pending at once. The platform team can confirm the runners'
concurrent and limit settings.
Add runner capacity, or run fewer jobs in parallel. See what the runner's configuration decides.
A stored access token expired, at midnight UTC on its expiry date. Unless someone chose a date, that is 365 days after it was created.
The token's page lists it as expired.
Create a new token, update the variable, and track the next expiry date. See stored tokens.
Your configuration includes a file from another project at a branch, or at a tag that someone moved. When that file changes, so does your pipeline.
Check the include's ref in Full configuration, then the other
project's recent commits.
Pin includes to tags, and treat tags as never moving. See refs in includes.
The job's image tag, such as node:22, now points at a newer image.
Anyone who can push to that repository can move a tag.
Compare the image's digest in the registry with the digest it had before.
Name the image by digest, as name@sha256:…, or use tags that their
owners never overwrite. See the job's image.
The temporary keys last one hour, because MegaCorp asks for 3600 seconds. Chained roles are limited to one hour, whatever you ask for.
The failures start more than an hour after the login step.
Log in again before the late steps, or split the job. See small things that bite.
The environment needs approvals, or the job is manual. An approval doesn't start the job.
Operate › Environments shows the deployment waiting, and the job offers Run.
Approve it, then run it. See who may deploy, and approvals.
Another job in the same resource_group is running, in another
pipeline. Only one job per resource group runs at a time.
The job is waiting for its resource. Look for the same deploy in other pipelines.
Let the other job finish, or find what is holding the resource. See one at a time.
secret-detection-1, or in .pipeline-policy stages.The security team added or changed a security policy.
Secure › Policies lists the policies that apply.
Nothing in your YAML can remove them. Talk to the security team. See three kinds of policy.
The cache is never found. The runners may have no shared cache storage, the key may keep changing, or protected and unprotected branches may have separate caches.
The cache lines in the job log name the key it tried, and whether it was found.
Work through the cache flow. See cache.
Docker Hub limits how many pulls it allows, counting each request for an image's manifest.
The failing images all come from Docker Hub.
Pull through GitLab's dependency proxy, or keep copies in your own registry. See private registries.
A deploy freeze is set for the project, and GitLab blocks deployments during it.
The environment's deployments list shows the next freeze.
Wait for the freeze to end, or ask whoever set it. See one at a time, newest wins.
The GitOps controller syncs on a timer. GitLab's agent can trigger an immediate sync only for its own configuration project and for public projects.
Compare the GitOps commit's time with the controller's last sync.
Shorten the controller's sync interval, or accept the delay. See GitLab's agent for Kubernetes.
The cards from here on are for what happens after the pipeline: the environment repository, the controllers, the cluster and the clouds.
The Application has no automated sync, so it waits for someone to sync it. With automated sync, Argo CD checks Git about every two minutes, and syncs whatever is OutOfSync; a sync that fails shows as failed, not as waiting.
The Application's syncPolicy has no automated block, and its last sync is older than the commit.
Sync it, or ask for automated sync. See drift, and going back.
ProgressDeadlineExceeded.The new pods never become ready, so Kubernetes won't replace more old ones. It doesn't roll back by itself.
kubectl rollout status ends with this, and exits with 1:
error: deployment "payments-api" exceeded its progress deadlineFind out why the new pods aren't ready, then revert or fix forward. See rolling, the default.
Running but show 0/1 ready, and the rollout waits.The readiness probe keeps failing. MegaCorp's chart probes /actuator/health unless a service's values say otherwise, and a service that doesn't serve that path is never ready.
The pod's events show failed readiness probes, with the path and the status code.
Set probes.path in the service's values. See rolling, the default.
kustomize build prints warnings that commonLabels or patchesStrategicMerge is deprecated.The overlay uses older fields. Kustomize 5.8 still builds them, and asks you to move to the new ones.
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.Run kustomize edit fix, or change to labels and patches by hand, and check that the build's output hasn't changed. See patches.
Something the configuration relies on has changed: a module's ?ref= tag was moved to another commit, or someone changed the resources by hand, outside OpenTofu.
The plan's changes, the module tag's history in its repository, and the cloud's activity log.
Protect the module repositories' tags so they can't move, and bring a hand-made change into code or undo it. See modules.
The Kustomization depends on another through dependsOn, and that one
isn't ready, often because one of its objects is failing its health check. Flux doesn't
apply a Kustomization until everything it depends on is ready.
flux get kustomizations shows the waiting one's message, such as:
dependency 'flux-system/add-ons' is not readyThe dependency's own row, or flux get helmreleases -A, shows what is failing
there.
Fix the dependency; the waiting Kustomization follows by itself. See add-ons, then settings, then applications.
The final test#
One file, ten mistakes. The file is payments-api's pipeline after a week of well-meant changes. Each mistake is one this book has explained, and each has a symptom card. Read the scenario, find all ten, and for each one say what will go wrong and which card describes it. Then open the answer key. Next, review a change to MegaCorp's delivery configuration, with five more mistakes. Last, write one file of your own from a short brief, and check it the way its answer shows.
The scenario#
It's Friday afternoon. payments-api's team asks you to review their pipeline changes before they merge them. They tell you what they meant to do:
- Integration tests must run on every merge request, against a PostgreSQL service. The
database password is in the project's CI/CD settings, as
POSTGRES_PASSWORD. changed-filesshould list the files a merge request changes, for the reviewers.smoke-testchecks the image that the pipeline has just built.- The API contract version is
1.10. - Everything else should behave exactly as before.
MegaCorp's templates, runners and settings are the ones described in this book.
Spot the bugTen mistakes in one file#
# final-test/payments-api/.gitlab-ci.yml
# The final test: payments-api's pipeline file after a busy week of changes. It contains
# exactly ten mistakes, planted on purpose. Used by the chapter "The final test". GitLab
# never runs it.
include:
- project: megacorp/devops/ci-templates
ref: main
file: pipelines/java-service.yml
- component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
inputs:
stage: test
project_key: payments-api
variables:
JAVA_VERSION: "21"
MC_TEAM: payments
API_VERSION: 1.10
maven-test:
variables:
MC_TEAM: $MC_TEAM
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
image-build:
before_script:
- echo "Building $CI_PROJECT_NAME for $MC_TEAM"
integration-test:
stage: test
tags: [docker]
image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk21
services:
- name: postgres:16
alias: db
script:
- mvn verify -Pintegration -Dapi.version=$API_VERSION
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
changed-files:
stage: test
script:
- git diff --name-only origin/main...HEAD > changed.txt
artifacts:
paths: [changed.txt]
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
smoke-test:
stage: deploy
needs: [maven-build]
script:
- ./smoke.sh "$IMAGE_REF"
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHShow the answer
| # | The mistake | What goes wrong | Card |
|---|---|---|---|
| 1 | ref: main on the central include | the pipeline changes whenever ci-templates' main changes, with no commit here | a moving include |
| 2 | API_VERSION: 1.10, unquoted | YAML reads a number, so the tests get 1.1 | 1.10 becomes 1.1 |
| 3 | MC_TEAM: $MC_TEAM in maven-test | the job gets the literal text $MC_TEAM | a literal variable |
| 4 | maven-test's own rules | they replace the template's list: the job leaves merge request pipelines, and now runs in schedules | your rules replaced the template's |
| 5 | image-build's own before_script | the functions and the AWS login are gone, so the build fails at mc_retry | before_script replaced |
| 6 | tags: [docker] on integration-test | no MegaCorp runner has that tag, so the job waits for ever | stuck on tags |
| 7 | the database password only in settings | settings variables never reach services, so PostgreSQL doesn't start | a service without its variables |
| 8 | integration-test runs only on main | it never runs in merge request pipelines, as it was meant to | no merge request rule |
| 9 | git diff … origin/main in a merge request pipeline | that pipeline fetches only its own commit, so origin/main doesn't exist | shallow history |
| 10 | smoke-test needs maven-build, not image-build | it doesn't receive IMAGE_REF, so it tests nothing | an empty dotenv variable |
Scoring: seven or more, and you're ready for real incidents: you can decode pipelines you have never seen. Five or six: reread the cards you missed, then try again tomorrow. Fewer than five: work through chapter 55 and the four card chapters once more. Every mistake here is in real pipelines somewhere, usually more than one at a time.
After the pipeline: a change that never reached aks-prod#
It's Monday. payments-api 1c7d4b09 went live on eks-prod last week, but
production on aks-prod still runs the old version. A colleague's merge request to
gitops-config says it fixes that. It changes two files. You're asked to review them before
anyone merges. MegaCorp's chart, controllers and clusters are the ones that
Part X describes.
Spot the bugFive mistakes on the way to aks-prod#
# final-test/platform/gitops-config/apps/aks-prod/applicationset.yaml
# The final test's delivery part: aks-prod's ApplicationSet as a colleague's merge request
# would leave it. Together with envs/production/payments-api.yaml, it holds exactly five
# mistakes, planted on purpose. Used by the chapter "The final test". No cluster reads it.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: payments-api
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- list:
elements:
- env: staging
- env: production
template:
metadata:
name: "payments-api-{{ .env }}"
annotations:
notifications.argoproj.io/subscribe.on-deployed.grafana: "payments-api|{{ .env }}"
spec:
project: payments
sources:
- repoURL: https://gitlab.example.com/megacorp/platform/charts.git
targetRevision: service-3.4.0
path: service
helm:
releaseName: payments-api
valueFiles:
- "$values/envs/{{ .env }}/payments-api.yaml"
valuesObject:
image:
repository: megacorpprod.azurecr.io/payment-api
serviceAccount:
annotations:
azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
podLabels: {}
- repoURL: https://gitlab.example.com/megacorp/platform/gitops-config.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: "payments-{{ .env }}"
syncPolicy:
syncOptions:
- CreateNamespace=true# final-test/platform/gitops-config/envs/production/payments-api.yaml
# The final test's delivery part: payments-api's production values as a colleague's merge
# request would leave them. Together with apps/aks-prod/applicationset.yaml, they hold
# exactly five mistakes, planted on purpose. Used by the chapter "The final test".
image:
repository: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api
tags: 1c7d4b09
replicaCount: 3
probes:
path: /healthz
env:
SPRING_PROFILES_ACTIVE: production
PAYMENTS_PROVIDER_URL: https://pay.example.com
secrets:
store: cloud-secrets
keys:
- name: DB_PASSWORD
remote: payments-api/production/db-password
ingress:
enabled: true
host: payments.example.comShow the answer
| # | The mistake | What goes wrong | Card |
|---|---|---|---|
| 1 | no automated: under the ApplicationSet's syncPolicy | aks-prod's Applications never sync by themselves, so every merged change waits for someone to press Sync | OutOfSync, and it stays that way |
| 2 | megacorpprod.azurecr.io/payment-api in the inline values | that repository doesn't exist, so the new pods can't pull their image | an image that can't be pulled |
| 3 | podLabels: {}, without azure.workload.identity/use: "true" | the pods get no Azure identity, although their service account names one | a pod without its Azure identity |
| 4 | tags: instead of tag: under image | the chart finds no image.tag and refuses to render, on eks-prod as well as aks-prod | the chart needs a tag |
| 5 | probes: path: /healthz | payments-api answers health checks on /actuator/health, so the new pods never become ready, and the rollout stalls | 0/1 ready |
The mistakes hide each other. Mistake 1 stops anything from syncing, and mistake 4 stops the chart from rendering, so mistakes 2, 3 and 5 would appear one at a time, each after the one before was fixed. A review catches all five at once. That's why deploy changes go through merge requests (promotion by merge request).
Scoring: four or five, and you can follow a change from Git to a running pod. Fewer: work through chapter 55 and the delivery cards again.
Then write one#
Finding mistakes is half of owning a pipeline. The other half is writing rules that do what your team means. This task uses only what Part III and Part V teach.
Spot the buginvoice-api's rules#
MegaCorp's billing team is starting a new Java service, invoice-api. Its pipeline has
five jobs: build, unit-tests, deploy-staging,
deploy-production and nightly-report. The team asks you for a
.gitlab-ci.yml whose rules do this:
- A push creates one pipeline: a merge request pipeline while a merge request is open, and a branch pipeline otherwise.
buildandunit-testsrun for every push tomainand for every release tag, such asv1.2.3. In a merge request or on another branch, they run only whensrc/orpom.xmlchanged.deploy-stagingruns after every push tomain.deploy-productionruns only in a release tag's pipeline, and waits for a person to start it.nightly-reportruns only in scheduled pipelines, and nothing else runs in them.
Show the answer
Here is one answer that meets the brief. Yours may differ and still be right: what matters is that its table matches.
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: $CI_COMMIT_BRANCH
- if: $CI_COMMIT_TAGbuild:
stage: build
script:
- mvn --batch-mode --no-transfer-progress package -DskipTests
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"
when: never
- if: $CI_COMMIT_TAG
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
- changes:
- src/**/*
- pom.xml
unit-tests:
stage: test
script:
- mvn --batch-mode --no-transfer-progress test
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"
when: never
- if: $CI_COMMIT_TAG
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
- changes:
- src/**/*
- pom.xmldeploy-staging:
stage: deploy
image: alpine:3.22
script:
- echo "Deploying $CI_COMMIT_SHORT_SHA to staging"
environment:
name: staging
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE == "push"
deploy-production:
stage: deploy
image: alpine:3.22
script:
- echo "Deploying $CI_COMMIT_TAG to production"
environment:
name: production
rules:
- if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
when: manualnightly-report:
stage: test
image: alpine:3.22
script:
- echo "The nightly report for $CI_COMMIT_BRANCH"
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"Three details trip people up:
- The workflow's last rule,
$CI_COMMIT_TAG. Without it, a tag creates no pipeline at all (a tag with no pipeline). deploy-stagingalso checks$CI_PIPELINE_SOURCE == "push", because a scheduled pipeline onmainhas$CI_COMMIT_BRANCHset tomaintoo (what each kind of pipeline sets).- A manual job added by a rule holds the tag's pipeline at blocked until someone runs it. That is what "waits for a person" means here (a play button that waits).
| job | feature branch, code | merge request, code | merge request, docs only | main | tag v1.2.3 | schedule |
|---|---|---|---|---|---|---|
| build | runs | runs | not in the pipeline | runs | runs | not in the pipeline |
| unit-tests | runs | runs | not in the pipeline | runs | runs | not in the pipeline |
| deploy-staging | not in the pipeline | not in the pipeline | not in the pipeline | runs | not in the pipeline | not in the pipeline |
| deploy-production | not in the pipeline | not in the pipeline | not in the pipeline | not in the pipeline | manual: waits for someone to run it | not in the pipeline |
| nightly-report | not in the pipeline | not in the pipeline | not in the pipeline | not in the pipeline | not in the pipeline | runs |
In the docs-only merge request no job matches, so GitLab creates no pipeline at all.
Checking your own answer. Paste it into Build › Pipeline
editor in a project of yours, and validate it with main selected: the
jobs shown should match the main column. The Validate tab simulates only a
push, so check the other columns for real. Push a tag, open a merge request with a code
change, open another whose only change is to the docs, and run a schedule once. The
docs-only branch's first push still gets a branch pipeline with build and
unit-tests, because changes is true for a new branch; the column
is about its merge request. If every column matches, you're ready to own a team's
pipeline.
Keyword index#
Every .gitlab-ci.yml keyword in GitLab 19.3, what it does, and where
this book explains it. Keywords written as parent:child are settings inside
the parent keyword. A dash means the book doesn't cover that keyword.
Global keywords#
| Keyword | What it does | Covered in |
|---|---|---|
default | settings every job gets unless it sets its own: image, before_script, tags and others | chapter 14 |
include | copies other YAML files into this configuration | chapter 15 |
include:local | a file from the same repository and branch | chapter 15 |
include:project | a file from another project, at a ref | chapter 15 |
include:remote | a file from any URL | chapter 15 |
include:template | a template that ships with GitLab | chapter 15 |
include:component | a versioned CI/CD component | chapter 16 |
include:inputs | values for an included file's inputs | chapter 16 |
include:rules | includes the file only when the rules match | chapter 15 |
include:integrity, include:cache | check or cache a remote file | – |
stages | the stages, in order | chapter 24 |
workflow | whether a pipeline is created, and its settings | chapter 22 |
workflow:rules | decide whether the pipeline exists at all | chapter 22 |
workflow:name | the pipeline's name | chapter 22 |
workflow:rules:variables | variables set when a workflow rule matches | chapter 22 |
workflow:auto_cancel, :on_new_commit, :on_job_failure | which jobs to cancel when a newer commit starts a pipeline, or when a job fails | chapter 22 |
Header keywords#
| Keyword | What it does | Covered in |
|---|---|---|
spec | the header of a file, before its --- line | chapter 16 |
spec:inputs, :default, :type, :options, :regex, :rules, :description | typed parameters for a component, an included file or a pipeline, with defaults, allowed values and checks | chapter 16 |
spec:include, spec:component, spec:description | shared input definitions, component context, and a description | chapter 16 |
Job keywords#
| Keyword | What it does | Covered in |
|---|---|---|
after_script | commands that run last, in a separate shell | chapter 29 |
allow_failure | lets the pipeline continue if the job fails | chapter 24 |
artifacts | files kept after the job, for later jobs and people | chapter 31 |
artifacts:paths, :exclude, :name, :untracked | which files to keep | chapter 31 |
artifacts:when, :expire_in | when to keep them, and for how long | chapter 31 |
artifacts:reports | files GitLab reads, such as test results and security reports | chapter 31 |
artifacts:access, :public, :expose_as | who can download them, and links in merge requests | chapter 31 |
before_script | commands that run before script, in the same shell | chapter 29 |
cache | files reused between jobs and pipelines, to save time | chapter 31 |
cache:key, :paths, :policy, :when, :fallback_keys, :unprotect, :untracked | which cache, what goes in it, and when it is saved | chapter 31 |
cache:key:files, :files_commits, :prefix | a key that changes when the files' content changes, or when the files get a new commit, with an optional prefix | chapter 31 |
coverage | a regular expression that finds the coverage figure in the log | chapter 37 |
dast_configuration | DAST site and scanner profiles | – |
dependencies | which earlier jobs' artifacts to download | chapter 31 |
environment | where the job deploys, which makes it a deployment job | chapter 40 |
environment:name, :url, :on_stop, :action, :auto_stop_in, :deployment_tier, :kubernetes | the environment's name, address, stopping and tier | chapter 40 |
extends | copies settings from other jobs, usually hidden ones | chapter 14 |
hooks:pre_get_sources_script | commands on the runner before the Git fetch | chapter 29 |
identity | identity federation with a cloud provider, in beta | – |
id_tokens | signed ID tokens for other systems, such as AWS and Azure | chapter 28 |
image | the container image the job runs in | chapter 30 |
image:name, :entrypoint, :pull_policy, :docker, :kubernetes | the image's name, entrypoint, pull policy and executor options | chapter 30 |
inputs, :type, :regex and others | typed inputs for one job, which can be changed when the job is run by hand or retried | – |
inherit | which default: settings and top-level variables the job takes | chapter 14 |
inherit:default, inherit:variables | take all, none, or a list | chapter 14 |
interruptible | lets a newer pipeline cancel this job | chapter 24 |
needs | starts the job as soon as named jobs finish, ignoring stages | chapter 24 |
needs:optional | a need that may be missing from the pipeline | chapter 24 |
needs:artifacts | whether to download the needed job's artifacts | chapter 31 |
needs:project, needs:pipeline, needs:pipeline:job | artifacts or status from other pipelines | chapter 17 |
needs:parallel:matrix | needs one variant of a matrix job | chapter 14 |
pages | a GitLab Pages publishing job | – |
parallel | runs several copies of the job | chapter 14 |
parallel:matrix | one copy for each combination of variable values | chapter 14 |
release, :tag_name, :tag_message, :name, :description, :ref, :milestones, :released_at, :assets:links | creates a release, and describes it | – |
resource_group | only one job in the group runs at a time | chapter 24 |
retry | retries the job on failure | chapter 24 |
retry:max, :when, :exit_codes | how often, and for which failures | chapter 24 |
rules | decides whether the job is in the pipeline, and how it runs | chapter 23 |
rules:if | a condition on variables | chapter 23 |
rules:changes, rules:exists | conditions on changed or existing files | chapter 23 |
rules:changes:paths, :compare_to, :regexp, and rules:exists:regexp | the files to check, the ref to compare against, or a Ruby regular expression in place of glob patterns | chapter 23 |
rules:when, :allow_failure, :needs, :variables, :interruptible | what a matching rule changes | chapter 23 |
run | a sequence of GitLab Functions steps | chapter 19 |
script | the commands the job runs | chapter 29 |
secrets | secrets fetched from a secrets manager | chapter 27 |
secrets:vault, :aws_secrets_manager, :gcp_secret_manager, :azure_key_vault, :gitlab_secrets_manager | the provider and the secret | chapter 27, and for AWS and Azure chapter 38 |
secrets:file, secrets:token | file or value, and which ID token to use | chapter 27 |
services | extra containers beside the job, such as a database | chapter 30 |
services:name, :alias, :entrypoint, :command, :variables, :pull_policy | the service's image, hostname, start command and settings | chapter 30 |
stage | the stage the job belongs to | chapter 4 |
tags | which runners may take the job | chapter 29 |
timeout | the job's own time limit | chapter 29 |
trigger | starts a downstream pipeline | chapter 17 |
trigger:include | a child pipeline from a file in this project | chapter 17 |
trigger:project | a pipeline in another project | chapter 17 |
trigger:strategy | makes the trigger job wait for, and mirror, the downstream pipeline | chapter 17 |
trigger:forward, trigger:inputs | what the downstream pipeline receives | chapter 17 |
when | when the job runs: on_success, manual, delayed, always, never | chapter 24 |
start_in | the delay for when: delayed | chapter 24 |
manual_confirmation | a message to confirm before running a manual job | chapter 24 |
variables | variables for every job, or for one job | chapter 26 |
variables:value, :description, :options | defaults and choices shown when someone runs a pipeline by hand | chapter 26 |
variables:expand | whether $ references in the value are expanded | chapter 26 |
Deprecated keywords#
| Keyword | Use instead | Covered in |
|---|---|---|
only, except | rules | chapter 23 |
image, services, cache, before_script, after_script at the top level | the same keywords inside default: | chapter 14 |
publish, and a Pages job named pages | the pages keyword | – |
Predefined variables#
GitLab sets these variables in every pipeline, and each one exists only from a certain moment. This appendix lists the variables you will meet most, when each exists, and which kinds of pipeline set the ones that differ. GitLab's documentation has the full list.
When a variable exists#
| Phase | Usable in | Examples |
|---|---|---|
| Pre-pipeline | everything, including include:rules | CI_COMMIT_*, CI_PIPELINE_SOURCE, CI_DEFAULT_BRANCH, CI_OPEN_MERGE_REQUESTS, CI_MERGE_REQUEST_*, CI_PROJECT_ID, CI_SERVER_HOST |
| Pipeline | job rules and scripts, but not include:rules | CI_PIPELINE_IID, CI_JOB_NAME, CI_JOB_STAGE, CI_NODE_INDEX, CI_ENVIRONMENT_NAME, GITLAB_USER_LOGIN |
| Job-only | scripts only: not workflow, include, rules or trigger jobs | CI_PIPELINE_ID, CI_PIPELINE_URL, CI_JOB_ID, CI_JOB_TOKEN, CI_PROJECT_DIR, CI_REGISTRY_PASSWORD |
Which pipelines set them#
| Variable | Branch | Tag | Merge request | Scheduled |
|---|---|---|---|---|
CI_COMMIT_BRANCH | yes | – | – | yes |
CI_COMMIT_TAG | – | yes | – | only if the schedule runs on a tag |
CI_PIPELINE_SOURCE is push | yes | yes | – | – |
CI_PIPELINE_SOURCE is merge_request_event | – | – | yes | – |
CI_PIPELINE_SOURCE is schedule | – | – | – | yes |
CI_MERGE_REQUEST_* | – | – | yes | – |
CI_OPEN_MERGE_REQUESTS | if the branch has an open merge request | – | yes | – |
CI_COMMIT_BEFORE_SHA is all zeros in merge request pipelines, scheduled
pipelines, manual runs, and the first pipeline of a branch or tag.
The variables you meet most#
| Variable | Holds | Phase |
|---|---|---|
CI_PIPELINE_SOURCE | what started the pipeline: push, merge_request_event, schedule, web, api, trigger, pipeline, parent_pipeline, and others | pre-pipeline |
CI_COMMIT_BRANCH | the branch, in branch pipelines only | pre-pipeline |
CI_COMMIT_TAG | the tag, in tag pipelines | pre-pipeline |
CI_COMMIT_REF_NAME | the branch or tag being built | pre-pipeline |
CI_COMMIT_REF_SLUG | the ref name in lower case, at most 63 bytes, with anything but 0-9 and a-z replaced by -, for URLs and host names | pre-pipeline |
CI_COMMIT_REF_PROTECTED | true for a protected branch or tag | pre-pipeline |
CI_COMMIT_SHA, CI_COMMIT_SHORT_SHA | the commit, and its first eight characters | pre-pipeline |
CI_COMMIT_BEFORE_SHA | the branch's latest commit before this push | pre-pipeline |
CI_DEFAULT_BRANCH | the project's default branch | pre-pipeline |
CI_OPEN_MERGE_REQUESTS | up to four open merge requests from this branch | pre-pipeline |
CI_MERGE_REQUEST_IID, …_SOURCE_BRANCH_NAME, …_TARGET_BRANCH_NAME, …_DIFF_BASE_SHA, …_EVENT_TYPE | the merge request, in merge request pipelines | pre-pipeline |
CI_PROJECT_ID, CI_PROJECT_PATH, CI_PROJECT_NAME | the project | pre-pipeline |
CI_SERVER_HOST, CI_SERVER_FQDN, CI_API_V4_URL | the GitLab instance, and its API | pre-pipeline |
CI_REGISTRY, CI_REGISTRY_IMAGE | the container registry, and the project's image path in it | pre-pipeline |
CI_PIPELINE_IID | the pipeline's number within the project | pipeline |
CI_JOB_NAME, CI_ENVIRONMENT_NAME | the job's name, and its environment | pipeline |
GITLAB_USER_LOGIN | who started the pipeline, or the manual job | pipeline |
CI_PIPELINE_ID, CI_JOB_ID | instance-wide IDs of the pipeline and the job | job-only |
CI_PROJECT_DIR | where the repository is cloned, and where the job runs | job-only |
CI_JOB_TOKEN | the job's token, valid only while the job runs | job-only |
CI_REGISTRY_USER, CI_REGISTRY_PASSWORD | credentials for the project's registry | job-only |
Where variables can't be used#
rules:ifcan't useCI_ENVIRONMENT_SLUG, or variables GitLab stores with the job:CI_PIPELINE_ID,CI_PIPELINE_URL,CI_JOB_ID,CI_JOB_TOKEN,CI_JOB_URL,CI_REGISTRY_USER,CI_REGISTRY_PASSWORDand others like them.includecan use settings variables,CI_PROJECT_*,CI_PIPELINE_SOURCE,CI_COMMIT_REF_NAME, and variables given when the pipeline starts. It can't use variables defined in the YAML.- No rule can use a value from a dotenv report (chapter 21).
Rules syntax#
A reference for rules: and its conditions in GitLab 19.3: how rules
are evaluated, how to write if: expressions, and what changes:
and exists: really check. Every line comes from GitLab's documentation, or
was captured on a local 19.3 instance.
How rules are evaluated#
- Rules are evaluated when the pipeline is created, before any job runs.
- They are checked in order, and the first rule that matches decides. If none matches, the job isn't added.
- Every rule needs
if,changes,existsorwhen, and every clause in one rule must hold. - A matching rule without
whenuses the job'swhen, which defaults toon_success. Awhenin the rule overrides the job's. - A rule can also set
allow_failure,needs, which replaces the job's list,variablesandinterruptible. - A rule with
whenbut noifbrings the warningJob may allow multiple pipelines to run for a single action, in CI Lint and on manual runs. - GitLab's documentation says jobs with no rules "default to
except: merge_requests": they don't join merge request pipelines.
if expressions#
| Write | Means | Notes |
|---|---|---|
$VAR == "value" | equal | the variable on the left; strings quoted, variables not |
$VAR != "value" | not equal | |
$VAR | set, and not empty | |
$VAR == null | not set | |
$VAR == "" | set, but empty | |
$VAR =~ /pattern/ | matches the regular expression | RE2 syntax, case-sensitive, /i to ignore case; a pattern without anchors matches anywhere in the value |
$VAR !~ /pattern/ | doesn't match | |
a && b, a || b | and, or | && binds before ||; brackets group |
!$VAR | empty, or not set | since 18.11; so !"false" is false |
Traps, each confirmed in the documentation or on a local 19.3 instance:
${VAR}and"$VAR"don't work inif. A quoted variable breaks the YAML itself:(): did not find expected key while parsing a block mapping at line 5 column 7.- An unquoted
!( … )around a comparison is rejected withjobs:job:rules:rule if invalid expression syntax. Use!=. - A one-character regular expression,
/./, is invalid. - A variable on the right of
=~is used as the regular expression, so its value must include the slashes. Variables inside a/…/pattern aren't expanded. - A right side that isn't a
/…/pattern turns into a substring test, the other way round.$CI_COMMIT_BRANCH =~ "xmainx"was true formain. - A variable whose value mentions another variable isn't expanded in
if. - An unquoted YAML number changes:
1.10arrives as1.1.
changes and exists#
| Clause | Checks | Traps |
|---|---|---|
changes | in merge request pipelines, files changed against the target branch; in branch pipelines, against the previous commit | always true for new branches, tags, and pipelines without a push: schedules, manual runs; compare_to sets another base; at most 50 patterns |
exists | files in the repository, relative to the project directory | directories need a trailing slash (since 18.2); artifacts are invisible to it; beyond 50,000 files it always matches |
In include and workflow#
include:rulesaccepts onlyif,existsandchanges, and can use only the variables that exist before the pipeline is created (predefined variables). Itsexistssearches the project that holds theinclude.workflow:rulesdecide whether the pipeline exists at all (chapter 22).
Error messages#
Search this page for the exact text you see. Each message is quoted as GitLab, the runner, a cloud or a delivery tool prints it: captured on a local GitLab 19.3 instance, or copied from the documentation. Each links to the symptom card or section that explains it. Messages that contain job or stage names were captured with the names shown; yours will differ.
When the pipeline is created#
| Message | Means | Go to |
|---|---|---|
The pipeline did not run. Review the workflow:rules configuration for the pipeline. | no workflow:rules matched | workflow refused |
The resulting pipeline would have been empty. Review the rules configuration. | the workflow allowed it, but no job's rules matched | empty pipeline |
'unit-tests' job needs 'compile' job, but 'compile' does not exist in the pipeline. | a needed job was left out by its rules; the message goes on to suggest needs:optional | needs a missing job |
orphan job: chosen stage nowhere does not exist; available stages are .pre, build, .post | a job names a stage missing from stages: | stage not listed |
jobs:job:rules:rule if invalid expression syntax | an expression GitLab can't parse, such as !( … ) | rule syntax |
(): did not find expected key while parsing a block mapping at line 5 column 7 | broken YAML, for example a quoted variable in if: | rule syntax |
Insufficient permissions to set pipeline variables | your role may not set variables when starting a pipeline | values given when the pipeline starts |
Failed - (downstream pipeline can not be created, Insufficient permissions to set pipeline variables) | the downstream project refuses the trigger job's variables | pipeline variables refused |
downstream_pipeline_creation_failed | a trigger job's failure reason: the downstream pipeline couldn't be created | child pipeline in merge requests |
CI Lint also warns. Job may allow multiple pipelines to run for a single
action means a rule has when but no if
(rules syntax). On MegaCorp's templates, it also prints this
deprecation warning, which suggests its own fix:
retry uses deprecated `when` value(s): stuck_or_timeout_failure. These match the more
specific failure reasons that replaced them; migrate to those reasons. See
https://docs.gitlab.com/ci/yaml/#retrywhenOn the job page, or in the job log#
| Message | Means | Go to |
|---|---|---|
This job is stuck because of one of the following problems. There are no active runners online, no runners for the protected branch, or no runners that match all of the job's tags: | no online runner fits the job's tags, or only protected runners fit | stuck on tags |
This job is stuck because the project doesn't have any runners online assigned to it. | the project's runners are offline | runners offline |
This job is stuck because you don't have any active runners that can run this job. | no runner available to the project can take the job | how a job finds a runner |
This job could not start because it could not retrieve the needed artifacts. | needed artifacts expired, or aren't reachable | artifacts unavailable |
No files to upload | artifacts: paths matched nothing | no files to upload |
*** WARNING: Service XYZ probably didn't start properly | a service container didn't open its port in time | a service without its variables |
The deployment job is older than the latest deployment, and therefore failed. | Prevent outdated deployment jobs stopped an old deploy | outdated deploy |
This deployment job does not run automatically and must be started manually, but it's older than the latest deployment, and therefore can't run. | the same, for a manual deploy | outdated deploy |
unresolved reference | the commit isn't in the shallow clone | shallow history |
remote: The project you were looking for could not be found or you don't have permission to view it. | a clone with the job token was refused | job token 404 |
fatal: run_command returned non-zero status | submodules with GIT_STRATEGY: fetch; GitLab suggests clone | submodules |
fatal: unable to access 'https://gitlab.example.com/…/….git/': Could not resolve proxy: proxy.example.com | a proxy variable from settings, or from the runner's configuration, names a proxy the runner can't reach, and Git obeys it; a job's YAML can't unset a settings variable | variables in settings |
ERROR: Job failed (system failure): resolving secrets: operation error Secrets Manager: GetSecretValue, …, no EC2 IMDS role found, … | the runner manager has no AWS role for secrets: | secrets from AWS |
From Azure#
| Message | Means | Go to |
|---|---|---|
AADSTS70021: No matching federated identity record found for presented assertion. | no federated credential matches the token's issuer and subject, or one was added minutes ago | no matching federated credential |
AADSTS700212 | the federated credential's audience differs from the token's aud | audience mismatch |
[megacorp] not on main, so not pushing to megacorpprod.azurecr.io | MegaCorp's image-build skipped the push to ACR outside main, by design | an image not in Azure's registry |
After the pipeline#
| Message | Means | Go to |
|---|---|---|
ErrImagePull, then ImagePullBackOff | Kubernetes can't pull the pod's image | an image that can't be pulled |
CrashLoopBackOff | the container keeps exiting, and Kubernetes waits longer before each restart | a crash loop |
error: deployment "payments-api" exceeded its progress deadline | the new pods didn't become ready within ten minutes | a rollout stuck half-way |
Error: execution error at (service/templates/deployment.yaml:28:52): image.tag is set for each environment | the environment's values file has no image tag | the chart needs a tag |
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. … | an overlay uses an older Kustomize field | deprecated Kustomize fields |
Error acquiring the state lock | another run holds the state's lock, or a cancelled one left it behind | state locked |
Helm install failed for release … failed pre-install: timeout waiting for: [Job/…] | a HelmRelease's install ran out of time, five minutes by default, and Flux won't retry it by itself | a HelmRelease that timed out |
dependency 'flux-system/add-ons' is not ready | a Flux Kustomization waits for another one that isn't ready | a Kustomization waiting for another |
resource :Namespace is not permitted in project payments | Argo CD may not create the namespace, because the project allows no cluster-wide kinds | a namespace the project doesn't allow |
tls: failed to verify certificate: x509: certificate signed by unknown authority | Argo CD, or another client, doesn't trust the Git server's certificate | a certificate Argo CD doesn't trust |
From AWS#
| Message | Means | Go to |
|---|---|---|
An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity | the role's trust policy doesn't accept this token | AssumeRole denied |
An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn't retrieve verification key from your identity provider | AWS can't reach GitLab's signing keys | invalid identity token |
ImageTagAlreadyExistsException | a push to an immutable tag that already exists | tag already exists |
The annotated specimen#
Every file of MegaCorp's specimen, in full. The chapters quote parts of these files; here you can read each one from top to bottom. The files are grouped by the project that owns them. Each group starts with what is worth noticing, linked to the pages that explain it.
A line such as # catalog: include-project marks one of the 37 reuse
mechanisms: the construct just below it is an example. Files named
SETTINGS.yml, and those under _settings, _runners,
_aws and _azure, describe configuration kept in GitLab's
settings, on the runners, or in the clouds. GitLab doesn't read them.
Where each mechanism appears#
Each marker links to its mechanism card. To see a mechanism at work, find it here, then open the file.
| Chapter | Markers, and the file each is in |
|---|---|
| 13 | yaml-anchors and yaml-merge-key in billing-batch; hidden-jobs in base.yml |
| 14 | default-section and global-variables in base.yml; extends, reference-tag and parallel-matrix in java-maven.yml |
| 15 | include-local in mono; include-project in payments-api; include-remote in billing-batch; include-template in security.yml |
| 16 | include-component in payments-api; spec-inputs in sonar-scan.yml; variables-as-parameters in java-maven.yml; toggle-variables in security.yml; pipeline-inputs in the deployer |
| 17 | parent-child and dynamic-child in mono; multi-project in deploy.yml; trigger-api in billing-batch |
| 18 | custom-config-path in web-portal's settings; instance-group-variables in the instance and group settings; auto-devops and instance-template-repo in the instance settings; project-templates and compliance-pipeline in the group settings; scan-execution-policy, pipeline-execution-policy and mr-approval-policy in policy.yml |
| 19 | toolbox-image in base.yml and the Dockerfile; shell-libraries in snippets.yml; internal-cli in mc; fetched-scripts in billing-batch; build-tool-reuse in pom.xml; runner-config in config.toml; ci-steps in mono's docs.yml |
payments-api: one service#
Start here: this is everything the service team wrote. Worth noticing:
- Its jobs come from the golden pipeline, pinned at
v4.2.0, and from a component pinned at2.1.0(chapter 15). maven-test:has noextends. It merges into the golden pipeline's job of the same name, and adds a variable (chapter 15).- That
MAVEN_CLI_OPTSloses to the group variable of the same name, because settings beat YAML (chapter 26). MC_TEAM: paymentsnever reaches a job..megacorp-basesetsMC_TEAMas a job variable, which beats top-level ones (chapter 14).publish-docsruns every night, because its only rule is the default branch. Its script only builds the site: nothing publishes it (chapter 55).- Only
mainandv*tags are protected, so only their pipelines receivePAYMENTS_DB_PASSWORD(chapter 27). - The POM's version stays
1.8.0-SNAPSHOT: nothing sets it from the Git tag (chapter 37).
# .gitlab-ci.yml in megacorp/payments/payments-api
# A Java service's entire pipeline file: include the golden pipeline and a
# component, set a few variables, adjust two jobs.
# Used by the chapters "Meet MegaCorp", "extends, !reference and default", "include:
# pulling in files", "Tracing a job to its source" and "Truth tables: predicting which jobs
# run".
# catalog: include-project
include:
- project: megacorp/devops/ci-templates
ref: v4.2.0
file: pipelines/java-service.yml
# catalog: include-component
- component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
inputs:
stage: test
project_key: payments-api
variables:
JAVA_VERSION: "21"
MC_TEAM: payments
maven-test:
variables:
MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"
publish-docs:
stage: deploy
image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
inherit:
default: [tags, retry]
variables: false
script:
- mkdocs build --strict
rules:
- !reference [.rules, default-branch]# SETTINGS.yml for megacorp/payments/payments-api
# This file DESCRIBES the project's settings in the GitLab UI. GitLab does not read it.
# Used by the chapters "Variables and precedence" and "Secrets".
protected_branches: [main]
protected_tags: ["v*"]
project_variables:
- key: MC_NEXUS_URL
value: https://nexus.example.com/repository/maven-public/
- key: PAYMENTS_DB_PASSWORD
value: "(set in the UI)"
masked: true
protected: true<?xml version="1.0" encoding="UTF-8"?>
<!-- pom.xml in megacorp/payments/payments-api (excerpt). The service inherits its
build, including the "ci" profile the pipeline calls, from MegaCorp's parent POM.
Used by the chapters "Reuse outside the YAML" and "Building Java, Node and front ends".
-->
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<!-- catalog: build-tool-reuse -->
<parent>
<groupId>com.example.megacorp</groupId>
<artifactId>megacorp-parent</artifactId>
<version>12.3.0</version>
</parent>
<artifactId>payments-api</artifactId>
<version>1.8.0-SNAPSHOT</version>
</project># catalog-info.yaml in megacorp/payments/payments-api: how MegaCorp's developer portal, a
# Backstage, knows the service. The portal's "Java service" template wrote this file and the
# first .gitlab-ci.yml when it created the repository. Used by the chapters "Around your
# service" and "Field guide to enterprise setups".
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: payments-api
description: Takes card payments for MegaCorp's shop
annotations:
backstage.io/source-template: template:default/java-service
backstage.io/source-location: url:https://gitlab.example.com/megacorp/payments/payments-api/
spec:
type: service
lifecycle: production
owner: group:default/paymentsdevops/ci-templates: the central templates#
The central team's templates, at the v4.2.0 tag that payments-api pins.
Worth noticing:
- Only the two files in
pipelines/create jobs. The files intemplates/hold defaults, hidden jobs and snippets, exceptsecurity.yml, which pulls in GitLab's scanner jobs (chapter 34). .megacorp-base's rules decide when almost every job runs: merge requests, the default branch and release tags, never schedules (chapter 25)..maven-testbrings its ownrules, which replace that list, so tag pipelines have no tests (chapter 14)..maven-baseand.image-buildset their ownbefore_script, which replaces the default one, so each loads the functions again (chapter 14)..image-buildruns in tag pipelines too, so a release builds a new image instead of promoting the one staging tested (chapter 39). With today's trust policy, it can't even push it (chapter 38)..deployhandsIMAGE_REFto the deployer. The value comes fromimage-build's dotenv report (chapter 31).security.yml'scontainer_scanningaddsneeds, but keeps the template'sdependencies: []. SoIMAGE_REF, and theCS_IMAGEbuilt from it, are empty in the scan (the symptom card).legacy/notify.ymlstill usesonly, and an older toolbox image. Every project that includes it by URL gets each change at once (include: remote).
# pipelines/java-service.yml in megacorp/devops/ci-templates
# The "golden pipeline" for Java services. One include gives a project a build,
# tests, scanners, an image in ECR and deployments.
# Used by the chapters "Meet MegaCorp", "Reuse at a glance", "include: pulling in files",
# "Tracing a job to its source" and "Truth tables: predicting which jobs run".
include:
- project: megacorp/devops/ci-templates
ref: v4.2.0
file:
- templates/base.yml
- templates/snippets.yml
- templates/rules.yml
- templates/workflow.yml
- templates/java-maven.yml
- templates/container.yml
- templates/security.yml
- templates/deploy.yml
stages: [build, test, package, scan, deploy]
maven-build:
extends: .maven-build
maven-test:
extends: .maven-test
image-build:
extends: .image-build
needs: [maven-build]
deploy-staging:
extends: .deploy
needs: [image-build]
variables:
ENVIRONMENT: staging
rules:
- !reference [.rules, never-on-schedule]
- !reference [.rules, default-branch]
deploy-prod:
extends: .deploy
needs: [image-build]
variables:
ENVIRONMENT: production
rules:
- if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
when: manual# pipelines/web.yml in megacorp/devops/ci-templates
# The whole pipeline for static web front ends. Projects never include this
# file: their CI/CD settings point straight at it, so their repositories carry
# no .gitlab-ci.yml at all. Used by the chapters "Organisation-wide reuse: settings and
# policies", "Tracing a job to its source" and "Building Java, Node and front ends".
include:
- project: megacorp/devops/ci-templates
ref: v4.2.0
file:
- templates/base.yml
- templates/snippets.yml
- templates/rules.yml
- templates/workflow.yml
- templates/node.yml
stages: [build, test, deploy]
node-build:
extends: .node-build
node-test:
extends: .node-test
deploy-site:
extends: .publish-site
needs: [node-build]
rules:
- !reference [.rules, never-on-schedule]
- !reference [.rules, default-branch]# templates/base.yml in megacorp/devops/ci-templates
# The foundation: defaults for every job, organisation-wide variables, and the
# hidden base job that almost every template extends.
# Used by the chapters "Meet MegaCorp", "extends, !reference and default", "Job rules: is
# this job in the pipeline?", "Variables and precedence" and "Runners and executors".
# catalog: default-section, toolbox-image
default:
image: registry.example.com/megacorp/devops/ci-tools:3.2
tags: [megacorp-shared]
interruptible: true
retry:
max: 2
when: [runner_system_failure, stuck_or_timeout_failure]
before_script:
- !reference [.snippets, functions]
- mc_log "job $CI_JOB_NAME on $CI_COMMIT_REF_NAME"
# catalog: global-variables
variables:
GIT_DEPTH: "20"
AWS_REGION: eu-west-2
MC_TEMPLATES_VERSION: "4.2.0"
# catalog: hidden-jobs
.megacorp-base:
variables:
MC_TEAM: unknown
artifacts:
expire_in: 7 days
rules:
- !reference [.rules, never-on-schedule]
- !reference [.rules, mr]
- !reference [.rules, default-branch]
- !reference [.rules, release-tag]# templates/rules.yml in megacorp/devops/ci-templates
# A rules library: named lists of rules that jobs assemble with !reference.
# Used by the chapters "extends, !reference and default", "Job rules: is this job in the
# pipeline?" and "Idioms".
.rules:
mr:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
default-branch:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
release-tag:
- if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
never-on-schedule:
- if: $CI_PIPELINE_SOURCE == "schedule"
when: never# templates/workflow.yml in megacorp/devops/ci-templates
# Which pipelines exist at all: merge request pipelines, the default branch and
# tags, with no second branch pipeline while a merge request is open.
# Used by the chapters "Pipeline sources and types" and "workflow: whether a pipeline
# exists".
workflow:
name: "$CI_PIPELINE_SOURCE: $CI_COMMIT_REF_NAME"
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: $CI_COMMIT_BRANCH
- if: $CI_COMMIT_TAG# templates/snippets.yml in megacorp/devops/ci-templates
# Reusable script fragments. Jobs pull them in with !reference; the functions
# they call live in the ci-tools image, in /opt/megacorp/lib/ci-lib.sh.
# Used by the chapters "extends, !reference and default", "Reuse outside the YAML",
# "Identity and tokens" and "GitLab, AWS and Azure".
# catalog: shell-libraries
.snippets:
functions:
- source /opt/megacorp/lib/ci-lib.sh
aws_login:
- mc_aws_login "$AWS_ROLE_ARN"
acr_push:
- mc_acr_push "$IMAGE:$CI_COMMIT_SHA"
maven_settings:
- mkdir -p .m2
- mc_maven_settings > .m2/settings.xml# templates/java-maven.yml in megacorp/devops/ci-templates
# Build and test for Maven services. Everything here is hidden: a pipeline entry
# file turns these templates into real jobs.
# Used by the chapters "extends, !reference and default", "Variables and precedence",
# "Artifacts, cache and dotenv" and "Building Java, Node and front ends".
# catalog: extends, variables-as-parameters
.maven-base:
extends: .megacorp-base
image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
variables:
JAVA_VERSION: "21"
MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
cache:
key:
files: [pom.xml]
paths: [.m2/repository]
before_script:
# a job's own before_script replaces default:before_script, so re-add the functions
- !reference [.snippets, functions]
- !reference [.snippets, maven_settings]
.maven-build:
extends: .maven-base
stage: build
script:
- mvn $MAVEN_CLI_OPTS -DskipTests package
artifacts:
paths: [target/*.jar]
# catalog: reference-tag, parallel-matrix
.maven-test:
extends: .maven-base
stage: test
image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JDK}
parallel:
matrix:
- JDK: ["17", "21"]
script:
- mvn $MAVEN_CLI_OPTS verify
artifacts:
when: always
reports:
junit: target/surefire-reports/TEST-*.xml
rules:
- !reference [.rules, never-on-schedule]
- !reference [.rules, mr]
- !reference [.rules, default-branch]# templates/node.yml in megacorp/devops/ci-templates
# Install, test and build Node front ends, then publish the static files to S3
# behind CloudFront. Used by the chapters "Artifacts, cache and dotenv", "Building Java,
# Node and front ends" and "GitLab, AWS and Azure".
.node-base:
extends: .megacorp-base
image: registry.example.com/megacorp/devops/ci-tools/node:22
cache:
key:
files: [package-lock.json]
paths: [.npm/]
policy: pull
before_script:
- npm ci --cache .npm --prefer-offline
.node-build:
extends: .node-base
stage: build
cache:
policy: pull-push
script:
- npm run build
artifacts:
paths: [dist/]
.node-test:
extends: .node-base
stage: test
script:
- npm test -- --ci
artifacts:
when: always
reports:
junit: junit.xml
.publish-site:
extends: .megacorp-base
stage: deploy
id_tokens:
MC_ID_TOKEN:
aud: https://gitlab.example.com
variables:
AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-web-publish
before_script:
- !reference [.snippets, functions]
- !reference [.snippets, aws_login]
script:
- aws s3 sync dist/ "s3://$SITE_BUCKET/" --delete
- aws cloudfront create-invalidation --distribution-id "$CF_DISTRIBUTION_ID" --paths '/*'
environment:
name: production
url: https://www.example.com# templates/container.yml in megacorp/devops/ci-templates
# Build the service image without Docker-in-Docker and push it to Amazon ECR, tagged by
# commit, using the job's OIDC identity; on main, push the same image to Azure Container
# Registry too. Used by the chapters "Identity and tokens", "Images, services and Docker",
# "GitLab, AWS and Azure" and "Images to ECR and ACR".
.image-build:
extends: .megacorp-base
stage: package
image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
id_tokens:
MC_ID_TOKEN:
aud: https://gitlab.example.com
variables:
AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
before_script:
- !reference [.snippets, functions]
- !reference [.snippets, aws_login]
script:
- mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
- buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
- buildah push "$IMAGE:$CI_COMMIT_SHA"
- !reference [.snippets, acr_push]
- echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
artifacts:
reports:
dotenv: build.env# templates/security.yml in megacorp/devops/ci-templates
# GitLab's own scanners, pulled in as GitLab-shipped templates, plus MegaCorp's
# adjustments to the container scan. Used by the chapters "The scanning landscape",
# "GitLab's own scanners" and "Policies and gates".
# catalog: include-template
include:
- template: Security/SAST.gitlab-ci.yml
- template: Security/Secret-Detection.gitlab-ci.yml
- template: Security/Dependency-Scanning.gitlab-ci.yml
- template: Security/Container-Scanning.gitlab-ci.yml
# catalog: toggle-variables
variables:
CS_IMAGE: $IMAGE_REF
container_scanning:
stage: scan
needs: [image-build]
rules:
- if: $CONTAINER_SCANNING_DISABLED == "true"
when: never
- !reference [.rules, never-on-schedule]
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH# templates/deploy.yml in megacorp/devops/ci-templates
# Hand the built image to the platform team's deployer project, which owns every
# environment. Used by the chapters "Pipelines as building blocks", "Environments and
# deployments" and "GitOps repositories and promotion".
# catalog: multi-project
.deploy:
stage: deploy
trigger:
project: megacorp/platform/deployer
branch: main
strategy: depend
variables:
APP: $CI_PROJECT_NAME
IMAGE_REF: $IMAGE_REF# legacy/notify.yml in megacorp/devops/ci-templates
# An old notification job that billing-batch still pulls in by URL with
# include: remote, straight from this repository's raw file view.
# Used by the chapters "include: pulling in files" and "Reuse outside the YAML".
notify-chat:
stage: deploy
image: registry.example.com/megacorp/devops/ci-tools:2.9
script:
- curl --fail -X POST -d "text=billing-batch $CI_COMMIT_REF_NAME finished" "$CHAT_WEBHOOK_URL"
only:
- tagsdevops/ci-tools: the toolbox image#
None of this is YAML, yet every job runs in this image unless it picks another one. Worth noticing:
- The default
before_scriptsourcesci-lib.sh, so every job can call its functions (chapter 19). mc_aws_loginswaps the job's ID token for AWS keys that last one hour (chapter 38).mc_retryruns a command up to three times, waiting longer each time.mc deploychanges one line in the GitOps repository, usingGITOPS_TOKEN(chapter 45).mc versionprints the Git tag, or0.0.0-and the short commit when there is no tag (chapter 37).
# Dockerfile in megacorp/devops/ci-tools: MegaCorp's toolbox image.
# Every job runs in it unless it picks another image, and the shell functions
# and the "mc" tool that pipelines call live inside it.
# Used by the chapters "Reuse outside the YAML", "Identity and tokens" and "Images,
# services and Docker".
FROM public.ecr.aws/amazonlinux/amazonlinux:2023
RUN dnf install -y --allowerasing git jq make tar gzip unzip curl-minimal && dnf clean all
RUN curl -sSLo /tmp/awscli.zip https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip \
&& unzip -q /tmp/awscli.zip -d /tmp && /tmp/aws/install && rm -rf /tmp/aws /tmp/awscli.zip
RUN curl -sSLo /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.44.3/yq_linux_amd64 \
&& chmod +x /usr/local/bin/yq
# catalog: toolbox-image
COPY lib/ci-lib.sh /opt/megacorp/lib/ci-lib.sh
COPY bin/mc /usr/local/bin/mc
RUN chmod +x /usr/local/bin/mc#!/usr/bin/env bash
# lib/ci-lib.sh in megacorp/devops/ci-tools: MegaCorp's shell function library.
# It is baked into the ci-tools image as /opt/megacorp/lib/ci-lib.sh, and jobs
# source it through the .snippets functions snippet.
# Used by the chapters "extends, !reference and default", "Reuse outside the YAML",
# "Identity and tokens" and "GitLab, AWS and Azure".
mc_log() { printf '[megacorp] %s\n' "$*"; }
mc_retry() {
local n=0
until "$@"; do
n=$((n + 1))
[ "$n" -ge 3 ] && return 1
mc_log "retry $n: $*"
sleep $((n * 5))
done
}
# Exchange the job's OIDC ID token for temporary AWS credentials.
mc_aws_login() {
local role="${1:?role ARN required}"
local creds
creds=$(aws sts assume-role-with-web-identity \
--role-arn "$role" \
--role-session-name "gitlab-${CI_PROJECT_ID}-${CI_JOB_ID}" \
--web-identity-token "$MC_ID_TOKEN" \
--duration-seconds 3600 \
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
--output text) || { mc_log "AssumeRoleWithWebIdentity failed for $role"; return 1; }
read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< "$creds"
export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
mc_log "assumed $role"
}
# Sign in to Azure with the job's OIDC ID token, as the Entra application that trusts it.
mc_azure_login() {
local client_id="${1:?application (client) ID required}"
az login --service-principal -u "$client_id" -t "$MC_AZURE_TENANT_ID" \
--federated-token "$MC_ID_TOKEN" --output none \
|| { mc_log "az login failed for $client_id"; return 1; }
mc_log "signed in to Azure as $client_id"
}
# Push an image that is already built to Azure's registry as well. Only main's images go
# there: Entra matches a federated credential's subject exactly, and aks-prod deploys main.
mc_acr_push() {
local image="${1:?image required}"
if [ "${CI_COMMIT_BRANCH:-}" != "$CI_DEFAULT_BRANCH" ]; then
mc_log "not on $CI_DEFAULT_BRANCH, so not pushing to $MC_ACR_REGISTRY"
return 0
fi
mc_azure_login "$MC_ACR_PUSH_CLIENT_ID"
az acr login --name "${MC_ACR_REGISTRY%%.*}" --expose-token --query accessToken --output tsv \
| buildah login --username 00000000-0000-0000-0000-000000000000 --password-stdin "$MC_ACR_REGISTRY"
buildah push "$image" "docker://$MC_ACR_REGISTRY/${image##*/}"
mc_log "pushed $MC_ACR_REGISTRY/${image##*/}"
}
mc_maven_settings() {
cat <<EOF
<settings><mirrors><mirror><id>nexus</id><mirrorOf>*</mirrorOf><url>${MC_NEXUS_URL:-https://nexus.example.com/repository/maven-public/}</url></mirror></mirrors></settings>
EOF
}#!/usr/bin/env bash
# bin/mc in megacorp/devops/ci-tools: MegaCorp's internal CI command-line tool,
# shipped in the ci-tools image as /usr/local/bin/mc. The deployer pipeline calls
# "mc deploy"; release jobs call "mc version". Used by the chapters "Reuse outside the
# YAML", "Building Java, Node and front ends" and "GitOps repositories and promotion".
# catalog: internal-cli
set -euo pipefail
source /opt/megacorp/lib/ci-lib.sh
cmd="${1:-help}"
shift || true
case "$cmd" in
deploy)
env="${1:?usage: mc deploy <environment> <image-ref>}"
image="${2:?usage: mc deploy <environment> <image-ref>}"
tag="${image##*:}"
mc_log "promoting ${APP} ${tag} to ${env} through the GitOps repository"
git clone --depth 1 \
"https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
yq -i ".image.tag = \"${tag}\"" "gitops/envs/${env}/${APP}.yaml"
git -C gitops commit -am "deploy ${APP} ${tag} to ${env}"
if [ "$env" = production ]; then
# production changes wait in a merge request until someone merges them
git -C gitops push origin "HEAD:deploy/${APP}-${tag}" \
-o merge_request.create -o merge_request.target=main \
-o merge_request.title="Deploy ${APP} ${tag} to production"
else
git -C gitops push origin HEAD:main
fi
;;
version)
echo "${CI_COMMIT_TAG:-0.0.0-${CI_COMMIT_SHORT_SHA}}"
;;
*)
echo "usage: mc deploy <environment> <image-ref> | mc version" >&2
exit 2
;;
esacdevops/components: CI/CD components#
Three components. A project includes each at a version it chooses. Worth noticing:
- The
spec:header ends at---. Each$[[ inputs.… ]]is replaced when the pipeline is created, before any variable is expanded (chapter 16). sonar-scanhasallow_failure: trueand aSKIP_SONARtoggle, so even when the scan fails, the pipeline carries on (chapter 35).ecr-pushputs the repository input in its job's name, so each include with a different repository makes its own job (chapter 16).gitops-deploydeploys only from the default branch, and waits for a person whendeploy_whenismanual(chapter 45).
# templates/sonar-scan.yml in megacorp/devops/components: the sonar-scan CI/CD component.
# Projects include it with
# component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@<version>
# Used by the chapters "Components, inputs and toggles" and "Third-party scanners".
# catalog: spec-inputs
spec:
inputs:
stage:
default: test
project_key:
description: SonarQube project key
quality_gate:
type: boolean
default: true
---
sonar-scan:
stage: $[[ inputs.stage ]]
image: registry.example.com/megacorp/devops/ci-tools/sonar-scanner:6
variables:
SONAR_PROJECT_KEY: $[[ inputs.project_key ]]
SONAR_QUALITYGATE_WAIT: "$[[ inputs.quality_gate ]]"
script:
- sonar-scanner -Dsonar.projectKey="$SONAR_PROJECT_KEY" -Dsonar.host.url="$SONAR_HOST_URL"
rules:
- if: $SKIP_SONAR == "true"
when: never
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
allow_failure: true# templates/ecr-push/template.yml in megacorp/devops/components: the ecr-push component.
# Builds an image with buildah and pushes it to Amazon ECR, authenticating with the
# job's OIDC ID token. Used by the chapters "Components, inputs and toggles" and "Images to
# ECR".
spec:
inputs:
repository:
description: ECR repository name
context:
default: .
role_arn:
default: arn:aws:iam::123456789012:role/gitlab-ecr-push
tag:
default: $CI_COMMIT_SHA
---
ecr-push-$[[ inputs.repository ]]:
stage: package
image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
id_tokens:
MC_ID_TOKEN:
aud: https://gitlab.example.com
variables:
AWS_ROLE_ARN: $[[ inputs.role_arn ]]
ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
script:
- source /opt/megacorp/lib/ci-lib.sh
- mc_aws_login "$AWS_ROLE_ARN"
- aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
- buildah bud -t "$ECR_REGISTRY/$[[ inputs.repository ]]:$[[ inputs.tag ]]" "$[[ inputs.context ]]"
- buildah push "$ECR_REGISTRY/$[[ inputs.repository ]]:$[[ inputs.tag ]]"# templates/gitops-deploy/template.yml in megacorp/devops/components: the gitops-deploy component.
# It wraps the multi-project trigger to the platform team's deployer, so a project can
# deploy with one include instead of extending the .deploy template.
# Used by the chapters "Components, inputs and toggles" and "GitOps repositories and promotion".
spec:
inputs:
environment:
options: [staging, production]
app:
default: $CI_PROJECT_NAME
deploy_when:
default: on_success
options: [on_success, manual]
---
deploy-$[[ inputs.environment ]]:
stage: deploy
trigger:
project: megacorp/platform/deployer
branch: main
strategy: depend
variables:
APP: $[[ inputs.app ]]
ENVIRONMENT: $[[ inputs.environment ]]
IMAGE_REF: $IMAGE_REF
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: $[[ inputs.deploy_when ]]data/mono: a monorepo#
A monorepo that doesn't use the golden pipeline. Worth noticing:
generate-pipelinewrites a pipeline file, andservice-pipelinesruns it as a child pipeline (chapter 17).generate.shfails on merge requests with a long history, and builds too little after a push of several commits (chapter 32).docsstarts its child pipeline only when files underdocs/changed. For a new branch,changesis always true (chapter 23).lint-docsruns GitLab Functions instead of a script, an experimental feature (chapter 19).
# .gitlab-ci.yml in megacorp/data/mono
# A monorepo. A generator job writes one child-pipeline job per changed service,
# and a static child pipeline builds the docs. Used by the chapters "include: pulling in
# files", "Pipelines as building blocks" and "Field guide to enterprise setups".
# catalog: include-local
include:
- local: ci/common.yml
stages: [generate, build]
generate-pipeline:
stage: generate
script:
- ci/generate.sh > generated-pipeline.yml
artifacts:
paths: [generated-pipeline.yml]
# catalog: dynamic-child
service-pipelines:
stage: build
trigger:
include:
- artifact: generated-pipeline.yml
job: generate-pipeline
strategy: depend
# catalog: parent-child
docs:
stage: build
trigger:
include: ci/docs.yml
rules:
- changes: [docs/**/*]# ci/common.yml in megacorp/data/mono
# Defaults shared by the monorepo's own pipeline, pulled in with include: local.
# Used by the chapter "include: pulling in files".
default:
image: registry.example.com/megacorp/devops/ci-tools:3.2
tags: [megacorp-shared]#!/usr/bin/env bash
# ci/generate.sh in megacorp/data/mono
# Writes one child-pipeline job per service that changed. Its output is the YAML
# that the service-pipelines trigger job runs. Used by the chapter "Pipelines as building
# blocks".
set -euo pipefail
base="${CI_MERGE_REQUEST_DIFF_BASE_SHA:-HEAD~1}"
changed=$(git diff --name-only "$base" HEAD -- services/ | cut -d/ -f2 | sort -u)
echo "stages: [build]"
for svc in $changed; do
cat <<EOF
build-${svc}:
stage: build
image: registry.example.com/megacorp/devops/ci-tools:3.2
script:
- make -C services/${svc} build
EOF
done
# a child pipeline needs at least one job, even when nothing changed
if [ -z "$changed" ]; then
printf 'no-changes:\n stage: build\n script: [echo "no service changed"]\n'
fi# ci/docs.yml in megacorp/data/mono
# The child pipeline that builds the monorepo's documentation. It also carries an
# experimental job that uses GitLab Functions (formerly CI/CD Steps) instead of a script.
# Used by the chapters "Pipelines as building blocks" and "Reuse outside the YAML".
build-docs:
image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
script:
- mkdocs build --strict
# catalog: ci-steps
lint-docs:
image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
run:
- name: markdown_lint
func: registry.example.com/megacorp/devops/functions/markdownlint:1.2.0
inputs:
path: docs
- name: report
script: echo "markdown lint finished"web/web-portal: a project with no pipeline file#
web-portal's repository has no .gitlab-ci.yml. Its settings point at
pipelines/web.yml in the templates project, shown above, and that file is its
whole pipeline (chapter 18). For the other ways a project
without the file still runs pipelines, see chapter 56.
# SETTINGS.yml for megacorp/web/web-portal
# This file DESCRIBES the project's CI/CD settings in the GitLab UI. GitLab does
# not read it. The repository itself has no .gitlab-ci.yml: its pipeline comes
# from the central templates project. Used by the chapters "Organisation-wide reuse:
# settings and policies", "Tracing a job to its source" and "Field guide to enterprise
# setups".
# catalog: custom-config-path
ci_cd_configuration_file: pipelines/web.yml@megacorp/devops/ci-templates
project_variables:
- key: SITE_BUCKET
value: www-example-com-site
- key: CF_DISTRIBUTION_ID
value: E2EXAMPLE1234platform/deployer: every deploy goes through here#
The platform team's project. Service pipelines trigger it, and people can run it by hand. Worth noticing:
- Someone running it by hand picks an environment from the input's options. When a
service triggers it, the trigger's
ENVIRONMENTbeats the value built from the input (pipeline inputs). resource_group: $APP-$ENVIRONMENTlets only one deploy of an app to an environment run at a time (chapter 24).- Without an
IMAGE_REFthere is no deploy, and production waits for a person. GITOPS_TOKENexpires a year after it was created. From that day, every deploy fails (chapter 28).
# .gitlab-ci.yml in megacorp/platform/deployer
# The platform team's deployment pipeline. Service pipelines trigger it as a
# multi-project pipeline, and people can also run it by hand from the UI.
# Used by the chapters "Pipelines as building blocks", "Environments and deployments" and
# "GitOps repositories and promotion".
# catalog: pipeline-inputs
spec:
inputs:
environment:
type: string
default: staging
options: [staging, production]
---
variables:
ENVIRONMENT: $[[ inputs.environment ]]
deploy:
stage: deploy
image: registry.example.com/megacorp/devops/ci-tools:3.2
resource_group: $APP-$ENVIRONMENT
environment:
name: $ENVIRONMENT/$APP
script:
- mc deploy "$ENVIRONMENT" "$IMAGE_REF"
rules:
- if: $IMAGE_REF == null
when: never
- if: $ENVIRONMENT == "production"
when: manual
- when: on_success# SETTINGS.yml for megacorp/platform/deployer
# This file DESCRIBES the project's settings in the GitLab UI. GitLab does not read it.
# Used by the chapters "Identity and tokens" and "GitOps repositories and promotion".
project_variables:
- key: GITOPS_TOKEN
value: "(set in the UI)"
masked: true
protected: true
# A project access token created in megacorp/platform/gitops-config, with the
# Maintainer role and the write_repository scope. Nobody entered an expiry date,
# so GitLab set one 365 days after the token was created.platform/charts: the shared chart#
Every MegaCorp service is deployed with this chart; each environment's values file says what differs. Worth noticing:
- The image's tag has no default. Rendering without one stops with the chart's own message (the chart needs a tag).
- The ExternalSecret exists only when a service's values name secret keys, and the
Deployment reads the Secret it makes through
envFrom(chapter 50). - The probes' path defaults to
/actuator/health, which suits the Java services and nothing else (0/1 ready). - Region markers in the templates are Helm comments, so
helm templatenever prints them.
# service/Chart.yaml in megacorp/platform/charts: the shared Helm chart that every
# MegaCorp HTTP service deploys with. The charts repository tags each release, such as
# service-3.4.0, and the ApplicationSets name that tag.
# Used by the chapters "Helm" and "Argo CD".
apiVersion: v2
name: service
description: One MegaCorp HTTP service, with its Service, Ingress and secrets
type: application
version: 3.4.0# service/values.yaml in megacorp/platform/charts: the defaults every service starts
# from. A service's file in gitops-config sets what differs.
# Used by the chapters "Helm" and "GitOps repositories and promotion".
image:
repository: "" # set for each service
tag: "" # set for each environment; mc deploy writes it
pullPolicy: IfNotPresent
replicaCount: 2
port: 8080
env: {} # plain settings, as NAME: value
probes:
path: /actuator/health
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 512Mi
serviceAccount:
annotations: {} # the pod's cloud identity, where the cloud needs one
podLabels: {}
ingress:
enabled: false
className: nginx
host: ""
secrets:
store: "" # the ClusterSecretStore to read from
keys: [] # each one: {name: DB_PASSWORD, remote: payments-api/staging/db-password}{{- /*
service/templates/_helpers.tpl in megacorp/platform/charts: named templates that the
other templates share, so every object gets the same name and labels.
Used by the chapter "Helm".
*/}}
{{- define "service.name" -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- define "service.selectorLabels" -}}
app.kubernetes.io/name: {{ include "service.name" . }}
{{- end -}}
{{- define "service.labels" -}}
{{ include "service.selectorLabels" . }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
{{- end -}}{{- /*
service/templates/deployment.yaml in megacorp/platform/charts: the Deployment that runs a
service's pods. Used by the chapters "Containers and Kubernetes, enough to read" and "Helm".
*/}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "service.name" . }}
labels:
{{- include "service.labels" . | nindent 4 }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
{{- include "service.selectorLabels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "service.selectorLabels" . | nindent 8 }}
{{- with .Values.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
serviceAccountName: {{ include "service.name" . }}
containers:
- name: app
image: "{{ .Values.image.repository }}:{{ required "image.tag is set for each environment" .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: {{ .Values.port }}
{{- with .Values.env }}
env:
{{- range $name, $value := . }}
- name: {{ $name }}
value: {{ $value | quote }}
{{- end }}
{{- end }}
{{- if .Values.secrets.keys }}
envFrom:
- secretRef:
name: {{ include "service.name" . }}
{{- end }}
readinessProbe:
httpGet:
path: {{ .Values.probes.path }}
port: http
livenessProbe:
httpGet:
path: {{ .Values.probes.path }}
port: http
initialDelaySeconds: 20
resources:
{{- toYaml .Values.resources | nindent 12 }}{{- /*
service/templates/service.yaml in megacorp/platform/charts: the service's ServiceAccount,
the Service in front of its pods, and the Ingress that brings outside traffic in.
Used by the chapters "Containers and Kubernetes, enough to read", "Helm" and "AWS and
Azure for delivery".
*/}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "service.name" . }}
labels:
{{- include "service.labels" . | nindent 4 }}
{{- with .Values.serviceAccount.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
---
apiVersion: v1
kind: Service
metadata:
name: {{ include "service.name" . }}
labels:
{{- include "service.labels" . | nindent 4 }}
spec:
selector:
{{- include "service.selectorLabels" . | nindent 4 }}
ports:
- name: http
port: 80
targetPort: http
{{- if .Values.ingress.enabled }}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "service.name" . }}
labels:
{{- include "service.labels" . | nindent 4 }}
spec:
ingressClassName: {{ .Values.ingress.className }}
rules:
- host: {{ required "ingress.host is needed when ingress.enabled is true" .Values.ingress.host }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: {{ include "service.name" . }}
port:
name: http
{{- end }}{{- /*
service/templates/externalsecret.yaml in megacorp/platform/charts: asks the External Secrets
Operator to copy a service's secrets from the cloud's vault into a Kubernetes Secret, which
the Deployment reads. Used by the chapters "Helm" and "Secrets and config after the pipeline".
*/}}
{{- if .Values.secrets.keys }}
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: {{ include "service.name" . }}
labels:
{{- include "service.labels" . | nindent 4 }}
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: {{ required "secrets.store names the ClusterSecretStore" .Values.secrets.store }}
target:
name: {{ include "service.name" . }}
data:
{{- range .Values.secrets.keys }}
- secretKey: {{ .name }}
remoteRef:
key: {{ .remote }}
{{- end }}
{{- end }}platform/gitops-config: the environment repository#
What runs on eks-prod and aks-prod. Flux reads the clusters folders, and
Argo CD reads apps and envs. Worth noticing:
- Each cluster applies three Kustomizations in order: add-ons, then their settings, then the applications (chapter 46).
- aks-prod's add-ons are eks-prod's, with one patch that gives the External Secrets Operator its Azure identity (chapter 44).
- Both clusters name their secret store
cloud-secrets, so a service's values don't change between clouds (chapter 50). - One ApplicationSet makes payments-api's Application for each environment, from the
chart at a tag and the values in
envs(chapter 47). aks-prod's sets the image's repository inline, which beats the values files (chapter 47). - Argo CD marks each deploy on the dashboards, with a key that an ExternalSecret copies from the vault (chapter 52).
# clusters/eks-prod/flux-system/gotk-sync.yaml in megacorp/platform/gitops-config: written by
# "flux bootstrap gitlab". It points eks-prod's Flux at this repository and at this folder.
# The bootstrap also wrote gotk-components.yaml, Flux's own install, which is left out here.
# Used by the chapters "GitOps repositories and promotion" and "Flux".
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: flux-system
namespace: flux-system
spec:
interval: 1m0s
ref:
branch: main
secretRef:
name: flux-system
url: https://gitlab.example.com/megacorp/platform/gitops-config.git
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: flux-system
namespace: flux-system
spec:
interval: 10m0s
path: ./clusters/eks-prod
prune: true
sourceRef:
kind: GitRepository
name: flux-system# clusters/eks-prod/platform.yaml in megacorp/platform/gitops-config: what eks-prod runs, as
# three Flux Kustomizations that go in order: the add-ons, then their settings, then the
# teams' applications. Used by the chapters "GitOps repositories and promotion" and "Flux".
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: add-ons
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
path: ./infrastructure/add-ons/base
prune: true
wait: true
sourceRef:
kind: GitRepository
name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: config
namespace: flux-system
spec:
dependsOn:
- name: add-ons
interval: 10m
path: ./infrastructure/config/eks
prune: true
sourceRef:
kind: GitRepository
name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
dependsOn:
- name: config
interval: 10m
path: ./apps/eks-prod
prune: true
sourceRef:
kind: GitRepository
name: flux-system# clusters/aks-prod/flux-system/gotk-sync.yaml in megacorp/platform/gitops-config: written by
# "flux bootstrap gitlab" for aks-prod. The same as eks-prod's, apart from the folder.
# Used by the chapter "Flux".
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: flux-system
namespace: flux-system
spec:
interval: 1m0s
ref:
branch: main
secretRef:
name: flux-system
url: https://gitlab.example.com/megacorp/platform/gitops-config.git
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: flux-system
namespace: flux-system
spec:
interval: 10m0s
path: ./clusters/aks-prod
prune: true
sourceRef:
kind: GitRepository
name: flux-system# clusters/aks-prod/platform.yaml in megacorp/platform/gitops-config: what aks-prod runs. The
# same three steps as eks-prod, from the Azure folders.
# Used by the chapters "Flux" and "AWS and Azure for delivery".
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: add-ons
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
path: ./infrastructure/add-ons/aks
prune: true
wait: true
sourceRef:
kind: GitRepository
name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: config
namespace: flux-system
spec:
dependsOn:
- name: add-ons
interval: 10m
path: ./infrastructure/config/aks
prune: true
sourceRef:
kind: GitRepository
name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
dependsOn:
- name: config
interval: 10m
path: ./apps/aks-prod
prune: true
sourceRef:
kind: GitRepository
name: flux-system# infrastructure/add-ons/base/kustomization.yaml in megacorp/platform/gitops-config: the
# add-ons every MegaCorp cluster runs, as Kustomize's base. eks-prod uses it as it is, and
# aks-prod's overlay changes it. Used by the chapters "Kustomize" and "Flux".
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- add-ons.yaml# infrastructure/add-ons/base/add-ons.yaml in megacorp/platform/gitops-config: the External
# Secrets Operator and Argo CD, which Flux installs from their Helm charts at pinned versions.
# Used by the chapters "Kustomize", "Flux", "Argo CD", "Secrets and config after the
# pipeline" and "Observability".
apiVersion: v1
kind: Namespace
metadata:
name: external-secrets
---
apiVersion: v1
kind: Namespace
metadata:
name: argocd
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: external-secrets
namespace: flux-system
spec:
interval: 1h
url: https://charts.external-secrets.io
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: argo
namespace: flux-system
spec:
interval: 1h
url: https://argoproj.github.io/argo-helm
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: external-secrets
namespace: external-secrets
spec:
interval: 30m
chart:
spec:
chart: external-secrets
version: 2.10.0
sourceRef:
kind: HelmRepository
name: external-secrets
namespace: flux-system
install:
crds: CreateReplace
upgrade:
crds: CreateReplace
values:
serviceAccount:
name: external-secrets
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: argo-cd
namespace: argocd
spec:
interval: 30m
chart:
spec:
chart: argo-cd
version: 10.9.0
sourceRef:
kind: HelmRepository
name: argo
namespace: flux-system
values:
configs:
params:
server.insecure: true
notifications:
secret:
create: false
notifiers:
service.grafana: |
apiUrl: https://grafana.example.com/api
apiKey: $grafana-api-key
triggers:
trigger.on-deployed: |
- description: Application is synced and healthy.
send: [app-deployed]
oncePer: app.status.operationState?.syncResult?.revision
when: >-
app.status.operationState != nil
and app.status.operationState.phase in ['Succeeded']
and app.status.health.status == 'Healthy'
and (!time.Parse(app.status.health.lastTransitionTime).Add(1 * time.Minute).Before(time.Parse(app.status.operationState.finishedAt))
or time.Parse(app.status.health.lastTransitionTime).Before(time.Parse(app.status.operationState.startedAt)))
templates:
template.app-deployed: |
message: "{{ .app.metadata.name }} is synced and healthy"# infrastructure/add-ons/aks/kustomization.yaml in megacorp/platform/gitops-config: aks-prod's
# overlay. It takes the base, and gives the External Secrets Operator the Azure identity that
# it reads Key Vault with. Used by the chapters "Kustomize" and "AWS and Azure for delivery".
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
patches:
- target:
kind: HelmRelease
name: external-secrets
patch: |-
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: external-secrets
spec:
values:
serviceAccount:
annotations:
azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
podLabels:
azure.workload.identity/use: "true"# infrastructure/config/eks/secret-store.yaml in megacorp/platform/gitops-config: eks-prod's
# secret store. Services name "cloud-secrets"; on this cluster it reads AWS Secrets Manager,
# as the External Secrets Operator's own identity, through EKS Pod Identity. Also the key
# Argo CD's notifications use to mark deploys on Grafana's dashboards.
# Used by the chapters "Secrets and config after the pipeline", "AWS and Azure for
# delivery" and "Observability".
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: cloud-secrets
spec:
provider:
aws:
service: SecretsManager
region: eu-west-2
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: argocd-notifications-secret
namespace: argocd
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: cloud-secrets
target:
name: argocd-notifications-secret
data:
- secretKey: grafana-api-key
remoteRef:
key: platform-grafana-api-key# infrastructure/config/aks/secret-store.yaml in megacorp/platform/gitops-config: aks-prod's
# secret store. The same name, "cloud-secrets", so a service's values don't change; on this
# cluster it reads Azure Key Vault through AKS workload identity. Also the key Argo CD's
# notifications use to mark deploys on Grafana's dashboards.
# Used by the chapters "Secrets and config after the pipeline", "AWS and Azure for
# delivery" and "Observability".
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: cloud-secrets
spec:
provider:
azurekv:
authType: WorkloadIdentity
vaultUrl: https://megacorp-prod-kv.vault.azure.net
serviceAccountRef:
name: external-secrets
namespace: external-secrets
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: argocd-notifications-secret
namespace: argocd
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: cloud-secrets
target:
name: argocd-notifications-secret
data:
- secretKey: grafana-api-key
remoteRef:
key: platform-grafana-api-key# apps/eks-prod/applicationset.yaml in megacorp/platform/gitops-config: the payments team's
# Argo CD project on eks-prod, and the ApplicationSet that makes one payments-api
# Application for each environment. Used by the chapters "GitOps repositories and
# promotion", "Argo CD" and "Observability".
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: payments
namespace: argocd
spec:
description: The payments team's services
sourceRepos:
- https://gitlab.example.com/megacorp/platform/charts.git
- https://gitlab.example.com/megacorp/platform/gitops-config.git
destinations:
- server: https://kubernetes.default.svc
namespace: payments-*
# CreateNamespace makes each environment's Namespace, which is cluster-wide, so the
# project must allow that kind, and only with the team's names
clusterResourceWhitelist:
- group: ""
kind: Namespace
name: payments-*
---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: payments-api
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- list:
elements:
- env: staging
- env: production
template:
metadata:
name: "payments-api-{{ .env }}"
annotations:
notifications.argoproj.io/subscribe.on-deployed.grafana: "payments-api|{{ .env }}"
spec:
project: payments
sources:
- repoURL: https://gitlab.example.com/megacorp/platform/charts.git
targetRevision: service-3.4.0
path: service
helm:
releaseName: payments-api
valueFiles:
- "$values/envs/{{ .env }}/payments-api.yaml"
- repoURL: https://gitlab.example.com/megacorp/platform/gitops-config.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: "payments-{{ .env }}"
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true# apps/aks-prod/applicationset.yaml in megacorp/platform/gitops-config: the same project and
# ApplicationSet on aks-prod. The inline values point payments-api at Azure's registry, and
# give its pods their Azure identity. Used by the chapters "Argo CD" and "AWS and Azure for
# delivery".
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: payments
namespace: argocd
spec:
description: The payments team's services
sourceRepos:
- https://gitlab.example.com/megacorp/platform/charts.git
- https://gitlab.example.com/megacorp/platform/gitops-config.git
destinations:
- server: https://kubernetes.default.svc
namespace: payments-*
# CreateNamespace makes each environment's Namespace, which is cluster-wide, so the
# project must allow that kind, and only with the team's names
clusterResourceWhitelist:
- group: ""
kind: Namespace
name: payments-*
---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: payments-api
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- list:
elements:
- env: staging
- env: production
template:
metadata:
name: "payments-api-{{ .env }}"
annotations:
notifications.argoproj.io/subscribe.on-deployed.grafana: "payments-api|{{ .env }}"
spec:
project: payments
sources:
- repoURL: https://gitlab.example.com/megacorp/platform/charts.git
targetRevision: service-3.4.0
path: service
helm:
releaseName: payments-api
valueFiles:
- "$values/envs/{{ .env }}/payments-api.yaml"
valuesObject:
image:
repository: megacorpprod.azurecr.io/payments-api
serviceAccount:
annotations:
azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
podLabels:
azure.workload.identity/use: "true"
- repoURL: https://gitlab.example.com/megacorp/platform/gitops-config.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: "payments-{{ .env }}"
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true# envs/staging/payments-api.yaml in megacorp/platform/gitops-config: payments-api's values in
# staging. mc deploy writes the tag; people change the rest by merge request.
# Used by the chapters "Helm", "GitOps repositories and promotion" and "Argo CD".
image:
repository: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api
tag: 3f9c2e1a
replicaCount: 1
env:
SPRING_PROFILES_ACTIVE: staging
PAYMENTS_PROVIDER_URL: https://sandbox.pay.example.com
secrets:
store: cloud-secrets
keys:
- name: DB_PASSWORD
remote: payments-api/staging/db-password
ingress:
enabled: true
host: payments.staging.example.com# envs/production/payments-api.yaml in megacorp/platform/gitops-config: payments-api's values
# in production. A new tag arrives in a merge request that mc deploy opens, and goes live
# when someone merges it. Used by the chapters "GitOps repositories and promotion" and
# "Releasing safely".
image:
repository: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api
tag: 1c7d4b09
replicaCount: 3
env:
SPRING_PROFILES_ACTIVE: production
PAYMENTS_PROVIDER_URL: https://pay.example.com
secrets:
store: cloud-secrets
keys:
- name: DB_PASSWORD
remote: payments-api/production/db-password
ingress:
enabled: true
host: payments.example.complatform/infra-modules and infra-live: OpenTofu#
The platform team's modules for each cloud, and the root configuration that calls them. Worth noticing:
- infra-live calls each module at a tag, so a module's change reaches the clusters only when the tag in infra-live moves (chapter 48).
- A team asks for a registry or a pod identity by adding an entry to infra-live, in a merge request the platform team reviews and applies (chapter 48).
- ECR's tags are immutable, and the AKS cluster's nodes may pull from ACR.
# aws/main.tf in megacorp/platform/infra-modules: the AWS side of MegaCorp's delivery
# platform, as one OpenTofu module: an EKS cluster, an ECR repository for each service, and
# an EKS Pod Identity role for each workload. The platform team applies it.
# Used by the chapters "OpenTofu and Terraform" and "AWS and Azure for delivery".
terraform {
required_version = ">= 1.12.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 6.0"
}
}
}
variable "cluster_name" {
type = string
}
variable "kubernetes_version" {
type = string
default = "1.36"
}
variable "subnet_ids" {
type = list(string)
}
variable "repositories" {
description = "ECR repositories to create, such as payments-api"
type = set(string)
}
variable "workloads" {
description = "Pods that get an AWS role, keyed by name, each with its namespace and service account"
type = map(object({
namespace = string
service_account = string
}))
}
resource "aws_iam_role" "cluster" {
name = "${var.cluster_name}-cluster"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "eks.amazonaws.com" }
Action = ["sts:AssumeRole", "sts:TagSession"]
}]
})
}
resource "aws_eks_cluster" "this" {
name = var.cluster_name
version = var.kubernetes_version
role_arn = aws_iam_role.cluster.arn
vpc_config {
subnet_ids = var.subnet_ids
}
access_config {
authentication_mode = "API"
}
}
resource "aws_eks_addon" "pod_identity_agent" {
cluster_name = aws_eks_cluster.this.name
addon_name = "eks-pod-identity-agent"
}
resource "aws_ecr_repository" "service" {
for_each = var.repositories
name = each.value
image_tag_mutability = "IMMUTABLE"
}
resource "aws_iam_role" "workload" {
for_each = var.workloads
name = "${var.cluster_name}-${each.key}"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "pods.eks.amazonaws.com" }
Action = ["sts:AssumeRole", "sts:TagSession"]
}]
})
}
resource "aws_eks_pod_identity_association" "workload" {
for_each = var.workloads
cluster_name = aws_eks_cluster.this.name
namespace = each.value.namespace
service_account = each.value.service_account
role_arn = aws_iam_role.workload[each.key].arn
}
output "repository_urls" {
value = { for name, repo in aws_ecr_repository.service : name => repo.repository_url }
}
output "workload_role_arns" {
value = { for name, role in aws_iam_role.workload : name => role.arn }
}# azure/main.tf in megacorp/platform/infra-modules: the Azure side of MegaCorp's delivery
# platform, as one OpenTofu module: an AKS cluster with workload identity, an Azure
# Container Registry that the cluster pulls from, and a managed identity for each workload.
# Used by the chapters "OpenTofu and Terraform" and "AWS and Azure for delivery".
terraform {
required_version = ">= 1.12.0"
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "~> 4.0"
}
}
}
variable "resource_group_name" {
type = string
}
variable "location" {
type = string
default = "westeurope"
}
variable "cluster_name" {
type = string
}
variable "registry_name" {
description = "Globally unique, letters and digits only, such as megacorpprod"
type = string
}
variable "workloads" {
description = "Pods that get an Azure identity, keyed by name, each with its namespace and service account"
type = map(object({
namespace = string
service_account = string
}))
}
resource "azurerm_kubernetes_cluster" "this" {
name = var.cluster_name
location = var.location
resource_group_name = var.resource_group_name
dns_prefix = var.cluster_name
kubernetes_version = "1.36"
oidc_issuer_enabled = true
workload_identity_enabled = true
default_node_pool {
name = "system"
node_count = 3
vm_size = "Standard_D4s_v5"
}
identity {
type = "SystemAssigned"
}
}
resource "azurerm_container_registry" "this" {
name = var.registry_name
resource_group_name = var.resource_group_name
location = var.location
sku = "Standard"
admin_enabled = false
}
resource "azurerm_role_assignment" "cluster_pulls" {
scope = azurerm_container_registry.this.id
role_definition_name = "AcrPull"
principal_id = azurerm_kubernetes_cluster.this.kubelet_identity[0].object_id
}
resource "azurerm_user_assigned_identity" "workload" {
for_each = var.workloads
name = "${var.cluster_name}-${each.key}"
resource_group_name = var.resource_group_name
location = var.location
}
resource "azurerm_federated_identity_credential" "workload" {
for_each = var.workloads
name = each.key
parent_id = azurerm_user_assigned_identity.workload[each.key].id
issuer = azurerm_kubernetes_cluster.this.oidc_issuer_url
subject = "system:serviceaccount:${each.value.namespace}:${each.value.service_account}"
audience = ["api://AzureADTokenExchange"]
}
output "workload_client_ids" {
value = { for name, id in azurerm_user_assigned_identity.workload : name => id.client_id }
}# main.tf in megacorp/platform/infra-live: the platform team's root configuration for both
# clouds. It calls the modules in infra-modules at pinned tags. A team asks for a registry or
# a pod identity by adding an entry here, in a merge request that the platform team reviews
# and applies. Used by the chapters "OpenTofu and Terraform" and "AWS and Azure for delivery".
terraform {
required_version = ">= 1.12.0"
backend "http" {}
}
provider "aws" {
region = "eu-west-2"
}
provider "azurerm" {
features {}
subscription_id = "00000000-0000-0000-0000-000000000000"
}
module "eks_prod" {
source = "git::https://gitlab.example.com/megacorp/platform/infra-modules.git//aws?ref=aws-2.3.0"
cluster_name = "eks-prod"
subnet_ids = ["subnet-0a1b2c3d4e5f60718", "subnet-0b2c3d4e5f6071829"]
repositories = ["payments-api", "web-portal"]
workloads = {
payments-api-staging = { namespace = "payments-staging", service_account = "payments-api" }
payments-api-production = { namespace = "payments-production", service_account = "payments-api" }
}
}
module "aks_prod" {
source = "git::https://gitlab.example.com/megacorp/platform/infra-modules.git//azure?ref=azure-1.8.0"
resource_group_name = "rg-delivery-prod"
cluster_name = "aks-prod"
registry_name = "megacorpprod"
workloads = {
payments-api-staging = { namespace = "payments-staging", service_account = "payments-api" }
payments-api-production = { namespace = "payments-production", service_account = "payments-api" }
}
}Settings, runners, AWS and Azure#
What MegaCorp keeps outside every repository. Projects can't see most of it, yet all of it reaches their jobs. Worth noticing:
- A group variable such as
MAVEN_CLI_OPTSbeats anything a project writes in YAML (chapter 26). SONAR_TOKENis protected, so a merge request pipeline from an unprotected branch gets it empty (chapter 27).- The instance runs Auto DevOps for any project without a pipeline file (chapter 18).
- The runner adds variables, a default image and an AWS role to every job it runs (chapter 29).
- The ECR push role trusts branches only, so a tag pipeline can't push
(chapter 38). The web role trusts web-portal's
mainand nothing else. - The runner's role lets every job on that runner pull from ECR without logging in (chapter 28).
- Azure trusts one exact pipeline per federated credential: payments-api's
main, with no wildcards (chapter 38).
# _settings/instance.yml
# This file DESCRIBES settings made in the GitLab UI of MegaCorp's self-managed
# instance, gitlab.example.com. GitLab does not read it: it exists so the book can
# show what those settings are and where they bite.
# Used by the chapters "Organisation-wide reuse: settings and policies", "Variables and
# precedence" and "Runners and executors".
# catalog: instance-group-variables
# Every project on the instance receives these, unless a group or project
# variable with the same name takes precedence.
instance_variables:
- key: MEGACORP_REGISTRY
value: registry.example.com
- key: HTTPS_PROXY
value: http://proxy.example.com:3128
# catalog: instance-template-repo
# Self-managed only: the .yml files in this project's gitlab-ci/ folder appear in the
# Web Editor's template list when anyone creates a CI/CD file. Choosing one copies it
# into the new file; include: template cannot use them.
instance_template_repository:
project: megacorp/devops/instance-templates
offers:
- gitlab-ci/MegaCorp-Legacy-Java.yml
# catalog: auto-devops
# A project with no CI/CD configuration file gets GitLab's Auto DevOps pipeline.
auto_devops:
default_to_auto_devops: true# _settings/group-megacorp.yml
# This file DESCRIBES settings made in the GitLab UI on the top-level group
# "megacorp". GitLab does not read it.
# Used by the chapters "Organisation-wide reuse: settings and policies", "Variables and
# precedence", "Policies and gates" and "GitLab, AWS and Azure".
# catalog: instance-group-variables
# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
- key: MAVEN_CLI_OPTS
value: "--batch-mode --errors --show-version -s .m2/settings.xml"
- key: AWS_ACCOUNT_ID
value: "123456789012"
- key: MC_AZURE_TENANT_ID
value: "00000000-0000-0000-0000-000000000000"
- key: MC_ACR_REGISTRY
value: megacorpprod.azurecr.io
- key: MC_ACR_PUSH_CLIENT_ID
value: "00000000-0000-0000-0000-000000000000"
# the gitlab-acr-push application in Entra ID, which mc_acr_push signs in as
- key: SONAR_HOST_URL
value: https://sonar.example.com
- key: SONAR_TOKEN
value: "(set in the UI)"
masked: true
protected: true
# catalog: project-templates
# New projects can be created from these. The copy happens once, at creation.
group_project_templates:
source_group: megacorp/templates
templates:
- megacorp/templates/java-service-starter
- megacorp/templates/batch-job-starter
# catalog: compliance-pipeline
# Legacy: a compliance framework with its own pipeline configuration, the
# predecessor of pipeline execution policies.
compliance_frameworks:
- name: SOX
pipeline_configuration: compliance/sox.yml@megacorp/security/policies
applied_to:
- megacorp/payments/payments-api# _runners/config.toml
# An excerpt of the GitLab Runner configuration on MegaCorp's shared runners.
# Projects cannot see this file, yet everything in it reaches their jobs.
# Used by the chapters "Reuse outside the YAML", "Identity and tokens" and "Runners and
# executors".
concurrent = 20
# catalog: runner-config
[[runners]]
name = "megacorp-eks-shared"
# registered in GitLab as a group runner of megacorp, with the tag megacorp-shared
url = "https://gitlab.example.com"
executor = "kubernetes"
environment = ["MAVEN_OPTS=-Xmx2g", "HTTPS_PROXY=http://proxy.example.com:3128"]
pre_build_script = "echo 'runner: megacorp-eks-shared'"
[runners.kubernetes]
namespace = "gitlab-runners"
image = "registry.example.com/megacorp/devops/ci-tools:3.2"
# jobs run as this service account, which is bound to an AWS IAM role
service_account = "gitlab-runner-jobs"# _aws/iam.yml
# This file DESCRIBES how MegaCorp's AWS account 123456789012 trusts GitLab. AWS holds
# the real configuration, as JSON policies. GitLab does not read this file.
# Used by the chapters "Identity and tokens" and "GitLab, AWS and Azure".
# AWS accepts ID tokens signed by this GitLab, for this audience.
oidc_provider:
arn: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
url: https://gitlab.example.com
audiences: [https://gitlab.example.com]
# Assumed by image-build, and by the ecr-push component, through mc_aws_login.
ecr_push_role:
arn: arn:aws:iam::123456789012:role/gitlab-ecr-push
trust_policy:
Effect: Allow
Principal:
Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
Action: sts:AssumeRoleWithWebIdentity
Condition:
StringEquals:
gitlab.example.com:aud: https://gitlab.example.com
StringLike:
gitlab.example.com:sub: project_path:megacorp/*:ref_type:branch:ref:*
permissions: push and pull images in every ECR repository in the account
# Assumed by web-portal's deploy-site job.
web_publish_role:
arn: arn:aws:iam::123456789012:role/gitlab-web-publish
trust_policy:
Effect: Allow
Principal:
Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
Action: sts:AssumeRoleWithWebIdentity
Condition:
StringEquals:
gitlab.example.com:aud: https://gitlab.example.com
gitlab.example.com:sub: project_path:megacorp/web/web-portal:ref_type:branch:ref:main
permissions: write to the www.example.com bucket, and invalidate its CloudFront cache
# The role behind the gitlab-runner-jobs service account on the megacorp-eks-shared
# runner. Every job on that runner can use it without asking. The EKS cluster is what
# AWS trusts for this role, not GitLab.
runner_role:
arn: arn:aws:iam::123456789012:role/gitlab-runner-jobs
permissions: pull images from every ECR repository in the account# _azure/entra.yml
# This file DESCRIBES how MegaCorp's Azure tenant trusts GitLab, and what MegaCorp's
# pipelines may do in Azure. Entra ID and Azure hold the real configuration. GitLab does
# not read this file. Tenant and subscription 00000000-0000-0000-0000-000000000000 are
# placeholders. Used by the chapters "GitLab, AWS and Azure" and "AWS and Azure for
# delivery".
# The identity that image jobs sign in as. Each federated credential names one kind of
# pipeline that may use it.
acr_push_app:
display_name: gitlab-acr-push
client_id: 00000000-0000-0000-0000-000000000000
tenant_id: 00000000-0000-0000-0000-000000000000
federated_credentials:
- name: payments-api-main
issuer: https://gitlab.example.com
subject: project_path:megacorp/payments/payments-api:ref_type:branch:ref:main
audiences: [https://gitlab.example.com]
# What the app may do once signed in, on one registry only. AcrPush pushes and pulls
# images. az acr login also has to look the registry up in Azure Resource Manager, which
# AcrPush doesn't allow; Microsoft documents the second role for that, and it can also
# change or delete the registry, which is one more reason to trust main only.
role_assignments:
- principal: gitlab-acr-push
role: AcrPush
scope: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-delivery-prod/providers/Microsoft.ContainerRegistry/registries/megacorpprod
- principal: gitlab-acr-push
role: Container Registry Contributor and Data Access Configuration Administrator
scope: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-delivery-prod/providers/Microsoft.ContainerRegistry/registries/megacorpprodsecurity/policies: rules no project shows#
MegaCorp's security policy project. Nothing here appears in any project's configuration, and no toggle stops it (chapter 36). Worth noticing:
- Secret detection runs in every pipeline, on every branch.
- The guardrails policy adds
policy-sbomto every pipeline, in the reserved stage.pipeline-policy-post. - The approval policy doesn't set
fallback_behavior, so it fails closed. A merge request whose required scans produced no report needs the security team's approval too (chapter 36).
# .gitlab/security-policies/policy.yml in megacorp/security/policies
# MegaCorp's security policy project, linked to the top-level group. These rules
# reach every project without appearing in any project's CI/CD configuration.
# Used by the chapters "Organisation-wide reuse: settings and policies", "Tracing a job to
# its source", "GitLab's own scanners" and "Policies and gates".
# catalog: scan-execution-policy
scan_execution_policy:
- name: Secret detection everywhere
description: Run secret detection in every pipeline on every branch.
enabled: true
rules:
- type: pipeline
branches: ["*"]
actions:
- scan: secret_detection
- name: Nightly dependency scan
description: Scan dependencies on the default branch every night.
enabled: true
rules:
- type: schedule
branches: [main]
cadence: "0 2 * * *"
actions:
- scan: dependency_scanning
# catalog: pipeline-execution-policy
pipeline_execution_policy:
- name: MegaCorp guardrails
description: Add an SBOM job to every pipeline.
enabled: true
pipeline_config_strategy: inject_policy
content:
include:
- project: megacorp/security/policies
file: pipeline-policies/megacorp-guardrails.yml
# catalog: mr-approval-policy
approval_policy:
- name: Block new critical vulnerabilities
description: A merge into main that adds a critical finding needs security's approval.
enabled: true
rules:
- type: scan_finding
branches: [main]
scanners: [sast, secret_detection, dependency_scanning, container_scanning]
vulnerabilities_allowed: 0
severity_levels: [critical]
vulnerability_states: [new_needs_triage]
actions:
- type: require_approval
approvals_required: 1
group_approvers: [megacorp/security]# pipeline-policies/megacorp-guardrails.yml in megacorp/security/policies
# The CI/CD configuration that the "MegaCorp guardrails" pipeline execution
# policy injects into every project's pipeline. Used by the chapters "Organisation-wide
# reuse: settings and policies", "Tracing a job to its source" and "Policies and gates".
policy-sbom:
stage: .pipeline-policy-post
image: registry.example.com/megacorp/devops/ci-tools/syft:1
script:
- syft dir:. -o cyclonedx-json=sbom.cdx.json
artifacts:
reports:
cyclonedx: sbom.cdx.jsonlegacy/billing-batch: the older ways#
Created in 2019 from a starter template, and never updated. Worth noticing:
- Anchors and merge keys do the work of
extends, andonlydoes the work ofrules(chapter 13). - Its include follows another project's
mainby URL, so any change there reaches it at once (chapter 12). testdownloads a script and runs it. What it does is in no file of the project (chapter 19).notify-reportsstarts a pipeline in project 4242 with a trigger token (chapter 17).
# .gitlab-ci.yml in megacorp/legacy/billing-batch
# A 2019-era project, created from a starter template and never updated since.
# It shows the older ways configuration was shared.
# Used by the chapters "YAML-level reuse: anchors, merge keys and hidden jobs", "include:
# pulling in files", "Pipelines as building blocks" and "Reuse outside the YAML".
# catalog: project-templates
# catalog: include-remote
include:
- remote: https://gitlab.example.com/megacorp/devops/ci-templates/-/raw/main/legacy/notify.yml
# catalog: yaml-anchors, yaml-merge-key
.defaults: &defaults
image: registry.example.com/megacorp/devops/ci-tools:2.9
tags: [megacorp-shared]
only:
- branches
- tags
build:
<<: *defaults
stage: build
script:
- ./gradlew assemble
# catalog: fetched-scripts
test:
<<: *defaults
stage: test
script:
- curl -sSL https://gitlab.example.com/megacorp/devops/scripts/-/raw/main/run-tests.sh | bash
# catalog: trigger-api
notify-reports:
<<: *defaults
stage: deploy
only: [tags]
script:
- >
curl --fail --request POST
--form "token=$REPORTS_TRIGGER_TOKEN"
--form "ref=main"
--form "variables[BILLING_VERSION]=$CI_COMMIT_TAG"
"https://gitlab.example.com/api/v4/projects/4242/trigger/pipeline"Glossary and former names#
First, the features GitLab has renamed or replaced. When an older template, an older page or a colleague uses a name you can't find, look it up here. Then, every term this book uses, in one line each, with a link to where it's explained.
Former names#
| You may meet | Today, at 19.3 | When it changed |
|---|---|---|
scan result policies, scan_result_policy | merge request approval policies, approval_policy | only approval_policy is accepted since 17.0 |
newly_detected, in a policy's vulnerability_states | new_needs_triage and new_dismissed | 17.0 |
| compliance pipelines | pipeline execution policies | deprecated in 17.3; removal planned for 20.0 |
inject_ci, in a pipeline execution policy | inject_policy | 17.9; inject_ci is deprecated |
CI_JOB_JWT, CI_JOB_JWT_V2 | ID tokens, requested with id_tokens | removed in 17.0 |
| Token Access, in Settings › CI/CD | Job token permissions | 17.2 |
| Limit access to this project, then Authorized groups and projects | CI/CD job token allowlist | 17.2, then 17.3 |
| Merge when pipeline succeeds | auto-merge | – |
| CI/CD Steps | GitLab Functions, an experiment | – |
only, except | rules | deprecated |
image, services, cache, before_script or after_script at the top of the file | the same keywords inside default: | deprecated |
Security/SAST.gitlab-ci.yml and the other Security/ templates | wrappers: each only includes its Jobs/ twin, such as Jobs/SAST.gitlab-ci.yml (chapter 34) | – |
| Gemnasium, the dependency scanning analyzer | dependency scanning using an SBOM | Gemnasium deprecated in 17.9; removal proposed for 20.0 |
| Code Quality's CodeClimate template | any tool's results, imported as a codequality report | the template is deprecated |
| kaniko, for building images | Docker, Buildah, Podman or rootless BuildKit | GitLab's page marks kaniko removed |
Terms#
| Term | Means | Explained in |
|---|---|---|
| ACR | Azure Container Registry, where MegaCorp keeps the images that aks-prod runs | chapter 38 |
Application, ApplicationSet | Argo CD's objects: an Application says what to deploy, from where and to where; an ApplicationSet makes many from one template | chapter 45 |
| Argo CD | a GitOps controller with a web UI, which syncs Applications from Git into a cluster | chapter 41 |
| chart | a Helm package: templates plus default values, which render Kubernetes objects | chapter 43 |
| controller | a program in the cluster that watches objects, and changes the cluster to match them | chapter 41 |
| Entra ID | Microsoft's identity service for Azure, formerly Azure Active Directory | chapter 38 |
| environment repository | the Git repository that says what runs in each environment: image tags and settings | chapter 45 |
| External Secrets Operator | a controller that copies secrets from a cloud's vault into Kubernetes Secrets | chapter 50 |
| federated credential | a rule on an Entra ID application or managed identity that says which outside tokens it accepts, matched exactly | chapter 38 |
| Flux | a GitOps tool made of controllers, configured with objects such as GitRepository, Kustomization and HelmRelease | chapter 45 |
| Kustomize, overlay | a tool that lays changes over a base set of objects, one overlay for each environment or cluster | chapter 44 |
| namespace | a named group of objects inside a cluster, with its own permissions and quotas | chapter 54 |
| OpenTelemetry | the common standard for producing logs, metrics and traces | chapter 52 |
| promotion | moving a version from one environment to the next | chapter 45 |
| readiness probe | the check a new pod must pass before Kubernetes sends it traffic, or counts it in a rollout | chapter 51 |
!reference | a YAML tag that pastes one section of another job, such as its rules or script | chapter 14 |
| agent for Kubernetes | GitLab's program inside a cluster, which connects the cluster to GitLab | chapter 45 |
| anchor, alias, merge key | YAML's own copy and paste, with & and *; it works only inside one file | chapter 13 |
| artifact | files a job keeps when it ends, for later jobs and for people | chapter 31 |
| AssumeRoleWithWebIdentity | the AWS call that swaps an ID token for temporary AWS keys | chapter 38 |
| Auto DevOps | GitLab's built-in pipeline, for projects with no pipeline file when the setting is on | chapter 18 |
| cache | files kept between jobs to save time; they may be missing at any time | chapter 31 |
| child pipeline | a pipeline that a trigger job starts from a file in the same project | chapter 17 |
| CI Lint | a tool that checks a piece of pipeline YAML | chapter 4 |
| component | a versioned piece of configuration with typed inputs, included with include: component | chapter 16 |
| container registry | a store for a project's container images, under Deploy › Container Registry | chapter 8 |
| container scanning | a scanner that looks for known vulnerabilities in an image | chapter 33 |
default: | settings every job gets unless it sets its own | chapter 14 |
| dependency scanning | a scanner that looks for known vulnerabilities in the libraries a project uses | chapter 33 |
| deployment job | a job with environment:, which records what it deployed, and where | chapter 40 |
| Docker-in-Docker | a Docker engine that runs as a service beside a job, so that the job can build images | chapter 8 |
| dotenv report | a file of NAME=value lines that a job leaves as variables for later jobs | chapter 31 |
| downstream pipeline | a pipeline that another pipeline started: a child or a multi-project pipeline | chapter 17 |
| dynamic child pipeline | a child pipeline whose YAML an earlier job wrote | chapter 17 |
| ECR | Amazon Elastic Container Registry, where AWS keeps container images | chapter 38 |
| environment | a place a job deploys to, such as staging, with its history of deployments | chapter 40 |
| executor | how a runner runs jobs: in Docker containers, in Kubernetes pods, in a shell, and others | chapter 29 |
extends | copies another job's settings: maps are merged, lists are replaced | chapter 14 |
| Full configuration | the pipeline editor's view of every file, merged into one | chapter 4 |
| GitOps | deploying by writing the wanted state to a Git repository, which a tool in the cluster applies | chapter 41 |
| hidden job | a job whose name starts with a dot; it never runs, and exists to be copied | chapter 13 |
| IAM role, trust policy | an AWS identity a job can take on; its trust policy says who may | chapter 38 |
| ID token | a note about the job, signed by GitLab, that another system such as AWS can check | chapter 28 |
include | copies other YAML files into the configuration | chapter 15 |
| input | a typed parameter of a component, an included file or a pipeline, filled in when the pipeline is created | chapter 16 |
| instance, group and project runners | runners for every project, for one group's projects, or for chosen projects | chapter 29 |
| job | one piece of work in a pipeline, run by one runner | chapter 4 |
| job token | CI_JOB_TOKEN, which lets a job call GitLab while it runs | chapter 28 |
| masked, hidden, protected | how a variable is guarded: masked in logs, hidden in the settings, passed only to protected branches and tags | chapter 27 |
| merge request approval policy | a security policy that requires approval when scans find problems | chapter 36 |
| merge request pipeline | a pipeline for a merge request, rather than for a branch | chapter 21 |
| merge train | a queue of merge requests, each tested together with those ahead of it | chapter 21 |
| merged results pipeline | a merge request pipeline that runs on the source branch merged into the target | chapter 21 |
| multi-project pipeline | a pipeline in another project, started by a trigger job | chapter 17 |
needs | starts a job as soon as the named jobs finish, ignoring stages | chapter 24 |
| OIDC | OpenID Connect, the standard that ID tokens follow | chapter 28 |
| OpenTofu | an open-source tool that creates and changes infrastructure from code; it began as a copy of Terraform | chapter 10 |
| pipeline | all the jobs GitLab runs for one commit, for one reason such as a push | chapter 4 |
| pipeline execution policy | a security policy that adds jobs to projects' pipelines | chapter 36 |
| pipeline source | what started a pipeline, held in CI_PIPELINE_SOURCE | chapter 21 |
| plan, apply | OpenTofu's two steps: work out what would change, then change it | chapter 10 |
| predefined variable | a variable GitLab sets, such as CI_COMMIT_BRANCH | predefined variables |
| protected branch, protected tag | branches and tags that only some people may push to; only their pipelines get protected variables | chapter 27 |
resource_group | lets only one job of the group run at a time | chapter 24 |
rules | decide whether a job is in a pipeline, and how it runs | chapter 23 |
| runner | the program that takes jobs from GitLab and runs them | chapter 29 |
| SAST | static application security testing: a scanner that reads source code | chapter 33 |
| SBOM | software bill of materials: a list of every component in a build | chapter 33 |
| scan execution policy | a security policy that adds scanner jobs to pipelines, or runs them on a schedule | chapter 36 |
| secret detection | a scanner that looks for passwords and keys committed to the repository | chapter 33 |
| security policy project | the project that holds a group's security policies | chapter 36 |
| service | an extra container beside the job, such as a database | chapter 30 |
| settings variable | a variable set in the CI/CD settings of a project, a group or the instance | chapter 26 |
| shallow clone | a copy of the repository with only the newest commits | chapter 32 |
| signals | a running service's logs, metrics and traces | chapter 52 |
| stage | a group of jobs; by default, a stage starts only when the stage before it has succeeded | chapter 24 |
| state | OpenTofu's record of what it has made, which GitLab can keep for a project | chapter 10 |
| state, state lock | OpenTofu's record of what it created, and the lock that stops two runs at once | chapter 48 |
| sync | a controller making the cluster match what Git says | chapter 45 |
| tag | a Git tag names a commit; a runner tag is a label that matches jobs to runners | chapter 29 |
| template | YAML that GitLab ships, included with include: template | chapter 15 |
| toggle | a variable a template checks to switch a job off, such as SAST_DISABLED | chapter 16 |
| trigger job | a job that starts a downstream pipeline instead of running a script | chapter 17 |
| truth table | this book's method for predicting which jobs run in which pipeline | chapter 25 |
| values file | a Helm chart's settings for one service or environment | chapter 43 |
workflow:rules | decide whether a pipeline is created at all | chapter 22 |
| workload identity | a pod's own cloud identity, taken from its Kubernetes service account | chapter 49 |
End of the book