GitLab CI/CD, Decoded

GitLab CI/CD, Decoded#

Read, trace and fix any enterprise pipeline

Compiled bykodebot

GitLab 19.3 GitLab.com and self-managed Worked examples on AWS

Preface#

Your project's pipeline file is twenty lines long. The pipeline it produces is not.

In a large organisation, a central team writes the CI/CD configuration and everyone else consumes it. Your file includes a template, which includes eight more. A job inherits from a job that inherits from another, in a repository you have never opened. Variables arrive from settings pages nobody on your team can see. Scanners appear that no file mentions, and the script your job runs lives inside an image. When something breaks, the answer is almost never in the file in front of you.

GitLab's documentation explains each feature on its own, which is the right way to learn to write a pipeline. This book is about reading one: taking any job in any pipeline and finding where it came from, why it ran, what it was given and what it did. Nothing here depends on one organisation's setup. Everything is taught on MegaCorp, a fictional company whose GitLab is as tangled as the real ones.

Who it is for#

You are a developer. Your pipelines come from someone else's templates, and you need to understand them well enough to fix them. The book assumes you know what CI/CD is, and nothing more:

You don't need to have written a .gitlab-ci.yml, or any YAML.
You don't need to know GitLab's words for things.
You don't need to have run a container.

Every term is explained where it first appears. The short chapter Before you start gives you the Git and GitLab words, enough YAML to read any pipeline file, and a first pipeline, all in about ten minutes. Wherever an idea has parts, an order or a decision in it, a diagram shows it.

What reading it gets you#

Level	You can	What gets you there
0 · Zero	little: the YAML looks like noise
1 · Vocabulary	say what every keyword does	Parts I and II
2 · Recognition	open any pipeline and name the patterns in it	the rest of the book, read once
3 · Tracing	take any job back to where it was defined, why it ran and what it was given	the exercise at the end of each chapter in Parts II to VII
4 · Diagnosis	go from a symptom to its cause in minutes	the worksheet on your own pipeline, then a few real incidents with the symptom cards open
5 · Authoring	design and refactor a template library	not this book's goal

Level 4 is the goal. Reading gets you to level 2 and the exercises to level 3. The worksheet, which you fill in privately against your own organisation's pipeline, closes the gap. The final test tells you when you're there.

How to read it#

Mode	Shows	Use it for
Fast	the core of every chapter; deep dives, legacy asides and exercises fold into one-line headings	a first pass
Full	everything, with exercise answers still folded until you ask	the detail

Use the toggle in the sidebar. There is a third way in: press ⌘K and type what you see. A symptom ("blocked"), a keyword (!reference), a variable or an error message takes you to the answer.

If you have 45 minutes, read these in Fast mode, in this order:

Before you start, if CI/CD is all you know
the one-page map
reuse at a glance
tracing a job to its source
job rules
variables and precedence
the decoding method

Then skim the symptom cards, so that you know what is there.

Conventions#

PremiumUltimate The lowest GitLab tier that has the feature. No tier badge means every tier has it.
Self-managedGitLab.com The feature exists only on that offering.
ExperimentalBetaDeprecated GitLab's own status for the feature.
Since 17.11 The release that introduced it. If your organisation runs its own GitLab, check which version it is on.

Examples come in a few shapes. A You wrote pane shows configuration as it sits in its file. A GitLab builds pane beside it shows what GitLab made of it, marked captured from GitLab when it is GitLab's own output. A symptom card goes from what you see to the cause, how to confirm it and the fix. A spot the bug exercise ends every chapter in Parts II to VII.

What it targets#

Scope	Baseline	Note
GitLab	GitLab 19.3	features newer than this are left out; recent ones carry a since badge
Offerings	GitLab.com and self-managed	differences are called out where they exist
Cloud	Worked examples on AWS	the ideas transfer; the commands are AWS's

On accuracy#

GitLab releases every month, and search results do not sort by version. Every version-sensitive claim here was checked against GitLab's documentation. Every "GitLab builds" pane marked as captured is real output from a GitLab 19.3 instance, loaded with MegaCorp's projects exactly as this book shows them. Where a behaviour needs a tier that instance doesn't have, the pane says described, not captured.

It will still age. When this book and GitLab's documentation disagree, the documentation is right.

What it does not cover#

Not covered	Why
Designing a template library	you learn why central teams build what they build, not how to build it
Running GitLab	instance settings, upgrades and runner fleets belong to administrators; you see what their choices do to your jobs
Triaging security findings	you learn what each scanner does and why it blocks you, not whether a vulnerability matters
Other CI systems	there are no comparisons with GitHub Actions, Jenkins or others
Other clouds	the worked examples use AWS; the ideas carry over
GitLab beyond pipelines	Pages, wikis, planning and the AI features appear only where a pipeline touches them

NextBefore you start5 min

Before you start#

This book assumes you know what CI/CD is: code gets built, tested and shipped automatically whenever it changes. It assumes nothing else. This short chapter gives you the three things the rest of the book leans on:

the Git and GitLab words
enough YAML to read any pipeline file
a first pipeline, from a push to a result

If you already know all three, skip to chapter 1.

Git and GitLab words#

Git keeps the history of a project's files as a chain of saved changes. GitLab hosts that history and runs pipelines on it. Every word below appears on almost every page of this book:

Figure A project's history. Commits on main, a feature branch with its merge request, the merge back into main, and a tag on a release commit.#

Word	What it is	Why a pipeline cares
commit	one saved change to the files, with an ID like `4f2c9a1`, called its SHA	every pipeline runs for exactly one commit
branch	a named line of commits, such as `feature/login`	pushing to a branch starts a branch pipeline
default branch	the main line of the project, usually called `main`	deployments usually start from it
push	sending your new commits from your computer to GitLab	the most common way a pipeline starts
merge request	a request to merge one branch into another, with review and discussion; other tools call it a pull request	GitLab can run a pipeline for it, called a merge request pipeline
merge	combining a branch's commits into another branch	merging to `main` usually starts the pipeline that deploys
tag	a permanent name for one commit, usually a release number such as `v1.4.0`	pushing a tag starts a tag pipeline, often the one that releases
protected branch or tag	one that only chosen people may push to	secrets and deploy permissions are often limited to protected branches
project	GitLab's home for one repository, with its settings, pipelines and merge requests	a pipeline belongs to a project
group	a folder of projects, which can contain further groups	settings made on a group, such as CI/CD variables, reach every project inside it

YAML in five minutes#

A pipeline is described in a file called .gitlab-ci.yml, written in YAML. YAML is a way of writing data as indented text. Almost everything you will see is one of these few shapes:

# A comment: YAML ignores everything after #

name: payments-api        # a key and its value
retries: 2                # a number
enabled: true             # true or false

owner:                    # a map: the keys indented under "owner" belong to it
  team: payments
  email: payments@example.com

stages:                   # a list: one item per line, each starting with "-"
  - build
  - test

tags: [shared, linux]     # the same kind of list, written on one line

script: |                 # a block of text, kept line by line
  echo "first line"
  echo "second line"

Four rules explain nearly every YAML mistake you will meet:

Indentation is meaning. Keys indented under another key belong to it. Use spaces, never tabs; two spaces per level is the convention.
A dash starts a list item. - build is one item in a list. A key followed by a colon, such as stage:, starts a key in a map.
Numbers and true/false are not text. 21 is read as a number and true as a yes-or-no value. When you mean text, put it in quotes: JAVA_VERSION: "21". GitLab's variables are always text, which is why you will see so many quotes in pipeline files.
Some characters change the meaning. A value that starts with *, &, !, [ or {, or contains a colon followed by a space, means something special to YAML. Put such values in quotes when you mean plain text. Chapter 6 explains what the special ones do.

Numbers can even change as they are read. On the local GitLab 19.3, a variable written VERSION: 1.10 without quotes reached a rule as 1.1, so $VERSION == "1.10" never matched. Written as "1.10", it did.

In a .gitlab-ci.yml, every top-level key is one of two things. It is either a GitLab keyword that sets something for the whole pipeline, such as stages, or the name of a job. Here is the shape of the pipeline file you are about to read:

Figure The shape of a small pipeline file. The file is a map. Each top-level key is either a keyword or a job. A job is a map of its own settings, and its script is a list of commands.#

Your first pipeline file#

This is a complete .gitlab-ci.yml for a small Node.js project. It has two jobs: one builds the code, and one tests it.

stages:
  - build
  - test

build-app:
  stage: build
  image: node:22
  script:
    - npm ci
    - npm run build

unit-tests:
  stage: test
  image: node:22
  script:
    - npm ci
    - npm test

Line	What it means
`stages:`	the stages of this pipeline, in the order they run: first build, then test
`build-app:`	a job. Its name is up to you, as long as it isn't a GitLab keyword
`stage: build`	this job belongs to the build stage
`image: node:22`	the job runs inside a container started from the `node:22` image, which has Node.js installed
`script:`	the commands the job runs, in order. If one fails, the job stops and fails
`unit-tests:`	a second job, in the test stage, with the same image and its own commands

An image is a packaged set of tools and files. A container is a running copy of it, fresh for every job and thrown away afterwards. So nothing a job does on its own disk reaches the next job, unless the job saves it on purpose (chapter 24). That is why both jobs above run npm ci.

When you push a commit, GitLab turns this file into a pipeline. The pipeline graph shows each stage as a column and each job as a box:

Pipeline The pipeline this file produces: build-app in the build stage, then unit-tests in the test stage#

As text

build: build-app
test: unit-tests

Here is what happens between your push and the result:

Figure From a push to a result. Every job runs in a fresh container, and a failed job stops the stages after it.#

As text

You push a commit to GitLab
GitLab reads .gitlab-ci.yml at that commit
GitLab creates a pipeline with its stages and jobs
A runner takes the next job and starts a container from the job's image
The runner runs the job's script, one command at a time
Did every command succeed? No: The job fails, and later stages don't start. Yes: the next step.
The next stage starts. When every job has passed, the pipeline passes

A runner is a program, running somewhere your organisation chose, that asks GitLab for jobs and runs them. Chapter 2 takes each of these words further.

Every pipeline in this book is built from the same pieces: keywords, jobs, stages, images and scripts. The difficulty in a large organisation is not the pieces. It is that a real pipeline is assembled from many files, settings and teams you can't see from your own project. Finding those, and reading them, is what the rest of the book teaches.

Next · Part I01The one-page map4 min

The one-page map#

Part I · Orientation · Chapter 01· 4 min read

In a large organisation, a job is rarely written in one place. It is put together from layers:

your project's pipeline file
shared files written by a central team
settings that live in no file at all
the machine and the image the job runs on

GitLab combines these layers at four moments: when it reads each YAML file, when it joins the files together, when it creates the pipeline, and when a job runs. You can see the result of the first two in one place, Build › Pipeline editor › Full configuration. Everything after that you trace another way.

Nearly every question about a job is one of five. Where is it defined? When does it run? Where does it run? What does it execute? What goes in and out?

This chapter is the map you will keep coming back to. Don't try to learn every term on it now. Each one is explained properly in the chapter the map points to.

Where a job comes from#

Picture a job as an onion. Your project's file is at the centre, and each layer around it can add to it or change it:

Figure The layers that make up one job, from your own file outwards. Each outer layer can add to the ones inside it, or override them.#

From the inside out:

Your project's .gitlab-ci.yml. In a big organisation it is often only twenty lines. It has an include, which pulls in the central team's files, a few settings, and perhaps one job adjusted. Sometimes there is no file at all, because the project's settings point to a file in another project.
The central team's shared files. These are YAML files that your file pulls in:
- hidden jobs: templates for jobs that never run themselves
- real jobs that copy from a hidden job with extends
- snippets pasted into jobs with !reference
- a default: block that applies to every job
Part II explains each of these.
Organisation settings. CI/CD variables, which are named values set on the settings pages of the project, of its groups, and of the whole GitLab installation, called the instance. They beat anything written in YAML. On GitLab's top tier, Ultimate, security policies can also add jobs to every pipeline.
Run time. The runner's own configuration, and the image the job runs in. The real logic of a script often lives inside the image.

Four moments, and what you can see#

GitLab doesn't apply all the layers at once. It works in four phases, and the phase decides where you can see the result:

Figure When GitLab resolves each kind of reuse, and the boundary of what the Full configuration view shows.#

Phase	When	What happens	Can the merged view show it?
Parse	GitLab reads each YAML file on its own	YAML's own copy-and-paste features are applied: anchors, aliases and merge keys	yes
Compile	GitLab joins the files together	included files are copied in; `extends` and `!reference` copy settings from one job into another	yes
Create	GitLab creates the pipeline	rules decide which jobs exist; a job with `parallel: matrix` becomes several jobs; settings variables and policies apply	no: look at the pipeline graph and the settings
Run	a runner runs one job	the image, the runner's configuration and any downloaded scripts do their part	no: read the job log

So if what you need to know happens in the first two phases, one page shows it. If it happens later, the table of tools at the end of this chapter says where to look instead.

Two details of the merged view surprise people:

A component's jobs appear with their inputs already filled in. Chapter 9 explains inputs.
The default: block appears on its own, not copied into each job. A job's final image or retry setting may therefore be in default:, not in the job.

Five questions for any job#

When a job surprises you, the question you are asking is almost always one of these:

Question	Where the answer usually is	Covered in
Where is it defined?	the Full configuration view, then the `extends` chain, then the included file; if it is in no file, a policy	Part II
When does it run?	`workflow:rules`, the job's `rules`, `needs` and stages	Part III
Where does it run?	the job's runner `tags`, the runner's executor and the job's `image`	Part V
What does it execute?	`before_script` and `script`, and whatever those call inside the image	reuse outside the YAML
What goes in and out?	variables and secrets in; artifacts, caches, reports, images and deployments out	Part IV and Part V

The same idea, as a pipeline#

Here is a merge request pipeline for payments-api, a service at MegaCorp. MegaCorp is the made-up company this book uses throughout. Each column is a stage and each box is a job. The project's own file defines none of these jobs:

Pipeline A payments-api merge request pipeline. On GitLab Ultimate, MegaCorp's security policy also adds policy-sbom.#

As text

build: maven-build
test: maven-test: [17], maven-test: [21], sonar-scan (allow failure)
package: image-build
scan: container_scanning (allow failure)
.pipeline-policy-post: policy-sbom (policy)

Needs: image-build waits only for maven-build; container_scanning waits only for image-build.

Each job has a different origin, and each is traced a different way:

maven-build, maven-test and image-build are copied from the central team's hidden jobs, with extends.
sonar-scan comes from a component: a packaged, versioned piece of pipeline that a project includes.
container_scanning comes from a file GitLab itself ships for its scanners, adjusted by MegaCorp.
policy-sbom is in none of the project's files at all. A security policy adds it.

Chapter 4 introduces the projects behind them.

The tools that answer the questions#

Five places in GitLab answer most questions, and you will use each of them in this book:

Tool	Where	What it answers
Full configuration	Build › Pipeline editor, then the Full configuration tab	what GitLab assembled: included files, `extends`, `!reference` and anchors, all resolved
Validate	Build › Pipeline editor, then the Validate tab	which jobs a push would create, and problems with `needs` and `rules`
The pipeline graph	the pipeline's page	which jobs exist, in which stage, and which downstream pipelines were triggered
The job log	the job's page	what ran, in which image, on which runner, and what the script printed
CI/CD variables	the CI/CD settings of the project, its groups and the instance	values that no file shows

The rest of the book takes this map one layer and one phase at a time.

Next02The engine: pipelines, stages, jobs and runners6 min

The engine: pipelines, stages, jobs and runners#

Part I · Orientation · Chapter 02· 6 min read

A pipeline is one run of your CI/CD configuration, started by an event such as a push. It is made of jobs, grouped into stages. Stages run in order, and the jobs inside one stage run at the same time.

A job is a script plus what it needs to run: an image, variables, and files from earlier jobs. A runner takes the job from GitLab and runs it.

That is the whole engine. The rest of the book is about where its parts come from.

If you read Before you start, you have met these words already. This chapter makes them precise, and shows you how to read what GitLab shows you.

Six words#

Term	What it is	Where you see it
Pipeline	one run of the configuration, for one commit and one event	the pipeline's page, with its graph
Stage	a named group of jobs; stages run in the order they are listed	the columns of the pipeline graph
Job	a script to run, with its settings	a box in the graph; the job's own page holds its log
Runner	a program somewhere that takes jobs from GitLab and runs them	the job's page names it
Executor	how the runner runs a job: in a container, in a Kubernetes pod (a group of containers Kubernetes runs together), or directly in a shell	the first lines of the job log
Image	the container image a job runs in, when the executor uses containers	the first lines of the job log

Stages#

If a configuration never lists its stages, GitLab uses five: .pre, build, test, deploy and .post. A job that doesn't name a stage lands in test. A project can declare its own list with stages:. MegaCorp's golden pipeline does: build, test, package, scan, deploy.

Stages run one after another, and the jobs within a stage run in parallel. When a job fails, the stages after it usually don't start. "Usually", because a job can be allowed to fail, and a job can use needs to start before its stage would (chapter 17).

Pipeline A small pipeline where a test failed, so the deploy stage never started#

As text

build: compile (passed)
test: lint (passed), unit-tests (failed)
deploy: deploy-review (skipped)

.pre and .post never need listing. Every stage you list runs after .pre and before .post. A security policy can add two more stages outside even those (chapter 29).

Two stage mistakes stop GitLab from creating the pipeline at all. The first is a job whose stage isn't in the list. The local GitLab 19.3 reports it like this:

orphan job: chosen stage nowhere does not exist; available stages are .pre, build, .post

The second is easier to miss. A job with no stage goes into test. If a project's stages: list leaves test out, that job fails in the same way, with chosen stage test does not exist.

Reading the pipeline graph#

Columns are stages, in the order they run.
Within a stage, jobs are sorted alphabetically by name. They are not sorted by when they started, or by where they were defined. The graph tells you nothing about which file a job came from.
Numbered jobs are grouped. Names that differ only by a counter, such as build ruby 1/3 and build ruby 2/3, collapse into one box with a count.
Every job shows a status. A job usually moves through the same few, in order:

Figure The usual life of a job, with the status GitLab shows at each point. Manual, delayed and queued jobs pause on the way; the table below lists those statuses too.#

As text

created: its stage hasn't started yet
pending: waiting for a runner that can take it
preparing: a runner is getting ready to run it
running: the script is running
Did every command succeed? No: failed. Yes: the next step.
success

Status	What it means for you
created	the job exists, but its turn hasn't come: an earlier stage is still running
pending	its turn has come, and it is waiting for a runner that can take it
preparing	a runner has it and is getting the environment ready
running	the script is running
success	it finished, and every command succeeded
failed	a command failed, or the runner gave up on the job
canceled	someone or something stopped it
skipped	it didn't run, usually because an earlier stage failed
manual	it waits for someone to start it
scheduled	a delayed job, counting down to its start
waiting_for_resource	it is ready, but another job holds the resource it needs

A job that sits in pending for ever is almost always a runner problem: no runner has the tags the job asks for (chapter 22). GitLab also has canceling and waiting_for_callback. You will rarely see either.

What can be a job#

Any top-level key in the configuration is a job, apart from GitLab's own keywords. That rule makes some mistakes quiet:

Keywords are never jobs. image, services, stages, before_script, after_script, variables, cache and include can't be job names. A top-level key called services: is read as the keyword, not as a job named "services".
A leading dot hides a job. .maven-base: never runs. It exists to be borrowed from (chapter 7).
Names are unique. Two definitions of the same name, one in your file and one in an included file, merge into one job. Two genuinely separate jobs can't share a name.
The limit is 255 characters. true, false and nil work as names only in quotes.

Runners, executors and images#

GitLab schedules jobs but doesn't run them. Runners do. A runner is registered with GitLab, keeps asking it for work, and runs each job it accepts with its executor:

With the Docker executor, each job runs in a fresh container started from the job's image.
With the Kubernetes executor, each job runs in a pod.
With the shell executor, jobs run directly on the runner's machine.

A runner has tags: labels such as linux or megacorp-shared. A job lists tags too. A runner takes a job only if it has every tag the job lists. If a job sets its own tags, they replace any default: tags completely rather than adding to them.

Gotcha

GitLab uses the word "tag" for two unrelated things. A Git tag, such as v1.4.0, names a commit and starts tag pipelines. A runner tag, set with the tags: keyword, chooses which runner takes a job. In .gitlab-ci.yml, tags: always means runner tags.

Runners belong to the whole instance, to a group, or to a single project. Their configuration is invisible from your project, which makes them one of the layers on the one-page map.

Reading a job log#

A job's log is written top to bottom, in a fixed order. The runner starts each part with a heading, and the job page lets you fold each part away. These are the headings as GitLab Runner 19.3 writes them:

Heading	What happens
`Running with gitlab-runner 19.3.0`, then the runner's name	which runner took the job
`Preparing the "shell" executor`	the executor starts; the heading names the one in use
`Preparing environment`	where the job runs: `Running on` a host or a pod
`Getting source from Git repository`	the fetch, its depth, and the commit checked out
`Restoring cache`	caches are fetched, if the job has any
`Executing "step_script" stage of the job script`	`before_script`, then `script`: each line echoed after a `$`, then its output
`Running after_script`	`after_script`, in a separate shell
`Saving cache for successful job`	caches are saved
`Uploading artifacts for successful job`	artifacts and reports are uploaded
`Cleaning up project directory and file based variables`	the runner tidies up
`Job succeeded`, or `ERROR: Job failed: …`	the result

To open a job, select it in its pipeline's graph, or find it under Build › Jobs. The log fills the page. On the right, Job details names the runner, the job's timeout and where that came from, and the tags:

Figure A sketch of a job's page, opened from the pipeline graph or from Build › Jobs. The log's sections fold; Job details on the right show where the job ran, its time limit and its tags.#

When a job fails, read the log from the end backwards. The last command echoed is the one that failed, because a failing command stops the script. When a job does the wrong thing, read the log from the top, because the first lines tell you where it really ran.

One job's whole log#

A small job, and the log it produced on the local GitLab 19.3, with GitLab Runner 19.3.0 and the shell executor. It was the job's first run, so there was no cache to restore yet:

You wroteprobes/runner-log · .gitlab-ci.yml

job-log:
  tags: [probe]
  variables:
    GREETING: hello
  before_script:
    - echo "before_script runs first"
  script:
    - echo "$GREETING from $CI_JOB_NAME on $CI_COMMIT_REF_NAME"
    - mkdir -p out && date -u > out/when.txt
  after_script:
    - echo "after_script runs in a separate shell"
  cache:
    key: probe
    paths: [out/]
  artifacts:
    paths: [out/]

GitLab ranjob logcaptured from GitLab

Running with gitlab-runner 19.3.0 (9cbf0074)
  on megacorp-probe-runner ja3Fs1cny, system ID: r_J2OH1b2ZRVSh
Preparing the "shell" executor
Using Shell (bash) executor...
Preparing environment
Running on a8b1839d4033...
Getting source from Git repository
Gitaly correlation ID: 01M28RS9Z3HQHR8KCNZ8EXV5BR
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /home/gitlab-runner/builds/ja3Fs1cny/0/probes/runner-log/.git/
Checking out 8afcdf7d as detached HEAD (ref is main)...

Skipping Git submodules setup
Restoring cache
Checking cache for probe-protected...
Runtime platform                                    arch=amd64 os=linux pid=636 revision=9cbf0074 version=19.3.0
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
WARNING: Cache file does not exist
Failed to extract cache
Executing "step_script" stage of the job script
$ echo "before_script runs first"
before_script runs first
$ echo "$GREETING from $CI_JOB_NAME on $CI_COMMIT_REF_NAME"
hello from job-log on main
$ mkdir -p out && date -u > out/when.txt
Running after_script
Running after script...
$ echo "after_script runs in a separate shell"
after_script runs in a separate shell
Saving cache for successful job
Creating cache probe-protected...
Runtime platform                                    arch=amd64 os=linux pid=679 revision=9cbf0074 version=19.3.0
out/: found 2 matching artifact files and directories
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally.
Created cache
Uploading artifacts for successful job
Uploading artifacts...
Runtime platform                                    arch=amd64 os=linux pid=703 revision=9cbf0074 version=19.3.0
out/: found 2 matching artifact files and directories
Uploading artifacts as "archive" to coordinator... 201 Created  correlation_id=01M28RSBYCCD8F3XZ1HXESKXPN id=146 status=201 token=[REDACTED]
Cleaning up project directory and file based variables

Job succeeded

Next03How GitLab builds a pipeline4 min

How GitLab builds a pipeline#

Part I · Orientation · Chapter 03· 4 min read

When you push, GitLab does a fixed sequence of things before any job runs:

It finds the configuration: .gitlab-ci.yml, or the file the project's settings name.
It fetches every included file and merges them all into one configuration.
It resolves extends, !reference and anchors, and checks the result is valid.
It decides whether a pipeline exists (workflow: rules) and which jobs are in it (each job's rules).
It creates the jobs, and runners start taking them.

Two tools let you watch the first four steps without pushing anything: the pipeline editor and CI Lint.

From a push to a pipeline#

Every pipeline goes through the same checks, and each one can stop it. Knowing which check stopped it tells you where to look:

Figure What GitLab does between a push and the first job. Each question is a point where a pipeline can stop, and each stop looks different.#

As text

You push a commit
GitLab finds the configuration: .gitlab-ci.yml, or the file the settings name
It fetches every include, and merges everything into one configuration
It resolves extends, !reference and anchors
Is the configuration valid? No: A failed pipeline, with the error shown at the top. Yes: the next step.
Does workflow: rules allow this pipeline? No: No pipeline, and after a push no error. Yes: the next step.
It checks each job's rules, expands matrices, and adds any policy jobs
Is at least one job left? No: No pipeline. Yes: the next step.
GitLab creates the pipeline, and runners start taking its jobs

The first four steps are the compile and create phases from chapter 1. Parts II and III of this book take them one at a time: Part II the files and how they merge, Part III the rules.

Here is the whole assembly for one job. payments-api's file is short, and the job GitLab builds from it is not:

You wrotemegacorp/payments/payments-api · .gitlab-ci.yml

include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file: pipelines/java-service.yml
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api

You wrotemegacorp/devops/ci-templates · pipelines/java-service.yml

maven-build:
  extends: .maven-build

maven-test:
  extends: .maven-test

image-build:
  extends: .image-build
  needs: [maven-build]

GitLab buildsmerged configurationcaptured from GitLab

maven-build:
  variables:
    MC_TEAM: unknown
    JAVA_VERSION: '21'
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  artifacts:
    expire_in: 7 days
    paths:
    - target/*.jar
  rules:
  - - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
  - - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  - - if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
  - - if: "$CI_COMMIT_TAG =~ /^v\\d+\\.\\d+\\.\\d+$/"
  extends: ".maven-build"
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
  cache:
    key:
      files:
      - pom.xml
    paths:
    - ".m2/repository"
  before_script:
  - - source /opt/megacorp/lib/ci-lib.sh
  - - mkdir -p .m2
    - mc_maven_settings > .m2/settings.xml
  stage: build
  script:
  - mvn $MAVEN_CLI_OPTS -DskipTests package

Every line of the result came from a file payments-api never opens. The variables, the image, the cache, the rules and the before_script all arrived through the include and the extends chain, which chapter 7 walks one link at a time.

The pipeline editor#

The pipeline editor is at Build › Pipeline editor. It edits the project's .gitlab-ci.yml, and its tabs show what GitLab would do with it:

Figure A sketch of the pipeline editor, at Build › Pipeline editor, with the Full configuration tab selected. It shows payments-api's configuration with every include, extends and !reference resolved.#

Tab	What it shows	Use it to
Edit	the file, checked against GitLab's schema as you type; the result appears at the top of the page	catch syntax mistakes before you commit
Visualize	every stage and job, with `needs` drawn as lines between jobs	see the shape of the pipeline
Validate	a simulated pipeline for a push to the branch you choose, under Pipeline run source	find problems with `rules` and `needs` before you push
Full configuration	the whole configuration as one file: includes copied in, `extends` merged, anchors and `!reference` replaced	find where a line of a job really comes from

A file-tree button in the upper-right corner lists every file the configuration includes, and opens each one. A commit form at the bottom of every tab saves your changes.

Three details matter when you rely on the Full configuration tab:

It shows one scenario. Anything conditional, such as an include with rules, is decided as if the pipeline were a push to the default branch. A file included only for merge requests won't appear.
Extra hyphens are harmless. Content pasted by !reference can appear as a list inside a list, with lines that start - -. GitLab documents this as expected, and it doesn't change what runs.
It can't show what happens later. Rules for each job, matrices, policies, settings variables and anything the runner does all come after this view (chapter 1).

If the project's configuration lives in another project, the editor can't open it. Chapter 13 shows how to see it anyway.

CI Lint: checking a snippet#

CI Lint checks configuration you paste in, so you can test a change without touching the project's file. It lives on the same Validate tab:

Open Build › Pipeline editor, then the Validate tab.
Select Lint CI/CD sample, and paste the configuration.
To go further than syntax, select Simulate pipeline creation for the default branch.
Select Validate.

Figure A sketch of CI Lint on the Validate tab of Build › Pipeline editor. It shows a pasted sample, the option to simulate a push to the default branch, and the error GitLab returned for a stage that isn't listed.#

A simulation evaluates the rules as a push to the default branch, so it finds the problems a real push would hit. Every error message quoted in this book came from the same check, run on the local GitLab 19.3. For example, a job whose stage isn't in stages: gets:

orphan job: chosen stage nowhere does not exist; available stages are .pre, build, .post

The same check is available outside the browser: from the GitLab CLI (glab ci lint), from the GitLab extension for VS Code, and from the API.

Which tool answers which question#

You want to know	Use	It can't tell you
what all the files add up to	Full configuration	anything decided when the pipeline is created
which jobs a push to a branch would create	Validate, choosing the branch	merge request or scheduled pipelines
whether a snippet is valid, or what it would do	CI Lint, with simulation	anything about pipelines other than a default-branch push
how the jobs depend on each other	Visualize	which of them a given pipeline will contain
which jobs a merge request or schedule gets	a real pipeline of that kind, or a truth table (chapter 18)	nothing: this is the ground truth

None of these tools shows variables set in settings, policies, or what the runner and the image do. For those, you need the pipeline graph, the settings pages and the job log, and the rest of Part I introduces them.

Next04Meet MegaCorp3 min

Meet MegaCorp#

Part I · Orientation · Chapter 04· 3 min read

MegaCorp is a made-up company, and so is everything in its GitLab. It is built the way large organisations build theirs:

A central DevOps team writes the shared pipeline files, the ready-made building blocks called components, and the container image most jobs run in.
The security team writes policies that add scans to every pipeline.
The platform team runs every deployment.
Each application team keeps a short file that pulls in everything else.

Every example in this book is a real file from MegaCorp's projects, labelled with its project and path. The same files are loaded into a real GitLab 19.3 instance, and every piece of GitLab output in the book comes from there.

The map#

Each box is a GitLab project, and the arrows show who uses whom:

Figure MegaCorp's projects: who includes, triggers or runs what, and what reaches every project without appearing in any file.#

The projects#

You don't need to follow every word in this table yet. The last column says where each project is explained.

Project	Owned by	Holds	You'll meet it in
megacorp/devops/ci-templates	central DevOps	the shared pipeline files: job templates, script snippets, rules, and two complete pipelines	Part II
megacorp/devops/components	central DevOps	components, which are versioned building blocks with settings: sonar-scan, ecr-push and gitops-deploy	components and inputs
megacorp/devops/ci-tools	central DevOps	the toolbox image most jobs run in, its shell functions, and MegaCorp's own `mc` command	reuse outside the YAML
megacorp/security/policies	security	the security policies that add scans and gates to other projects' pipelines	policies and gates
megacorp/platform/deployer	platform	the deployment pipeline that every service starts	deploy patterns
megacorp/payments/payments-api	the payments team	a Java service: one include, one component, two adjustments	throughout
megacorp/web/web-portal	the web team	a Node front end with no `.gitlab-ci.yml` at all	org-wide reuse
megacorp/data/mono	the data team	a monorepo, one repository holding many services, that generates its own child pipelines	pipelines as building blocks
megacorp/legacy/billing-batch	nobody, any more	an old file from 2019, using older techniques: YAML anchors, a file included from a URL, and a trigger token	YAML-level reuse

A service's whole pipeline file#

This is all the configuration the payments team maintains, apart from two small adjustments further down the same file:

megacorp/payments/payments-api.gitlab-ci.ymlinclude

include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file: pipelines/java-service.yml
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api

include pulls another file's contents into this one, as if they had been typed here. The first include is MegaCorp's "golden pipeline" for Java services: a complete, ready-made pipeline that any Java project can take whole. This is what it pulls in:

megacorp/devops/ci-templatespipelines/java-service.ymlincludes

include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file:
      - templates/base.yml
      - templates/snippets.yml
      - templates/rules.yml
      - templates/workflow.yml
      - templates/java-maven.yml
      - templates/container.yml
      - templates/security.yml
      - templates/deploy.yml

That is eight shared files, all pinned to the tag v4.2.0. Pinned means GitLab reads them as they were at that tag. A later change by the central team doesn't reach payments-api until someone moves the tag in this file.

Those files define hidden jobs. Their names start with a dot, and they never run by themselves. The golden pipeline turns three of them into real jobs:

megacorp/devops/ci-templatespipelines/java-service.ymljobs

maven-build:
  extends: .maven-build

maven-test:
  extends: .maven-test

image-build:
  extends: .image-build
  needs: [maven-build]

So there are four files between payments-api and the script its maven-test job runs. Chapter 7 walks that chain one link at a time.

What no file in the project shows#

Some of MegaCorp's configuration isn't in any repository the payments team can read:

Settings. Variables set on the instance and on the megacorp group, and web-portal's pointer to a pipeline file in another project. The book shows them in files called SETTINGS.yml, which GitLab never reads.
Policies. On GitLab Ultimate, the security team's policy project adds a secret-detection scan to every pipeline. It also adds a job that lists everything the build contains, known as an SBOM (software bill of materials).
The runners. Their configuration sets environment variables and cloud permissions for every job.

Here are the variables set on the megacorp group. Every project in the group receives them:

settingsgroup-megacorp.ymlgroup-variables

# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
  - key: MAVEN_CLI_OPTS
    value: "--batch-mode --errors --show-version -s .m2/settings.xml"
  - key: AWS_ACCOUNT_ID
    value: "123456789012"
  - key: SONAR_HOST_URL
    value: https://sonar.example.com
  - key: SONAR_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true

Keep MAVEN_CLI_OPTS in mind. It matters in chapter 19.

Reading the specimen#

Every file opens with a comment saying what it is and which chapters use it.
Every quote is labelled with its project and path. When only part of a file is shown, the label also names the part, called a region.
Files named SETTINGS.yml describe settings made in GitLab's web pages. GitLab never reads them.
Every pipeline file in the specimen is valid. On the local GitLab, each one creates its pipelines. The deployer creates one only when a service passes it an image to deploy, by design. Like any real setup, the specimen also hides a few surprises, and the exercises find them.

The full, annotated listing is in the appendix.

Next · Part II05Reuse at a glance6 min

Reuse at a glance#

Part II · Reuse · Chapter 05· 6 min read

Reuse means writing something once and using it in many places. In a large organisation, most of what your job does was written somewhere else. This book counts 37 ways that happens, in seven families. They run from YAML shortcuts inside one file to scripts built into an image.

Four questions sort them:

How far does it reach?
Is it pinned to a version?
Did your project choose it?
Can the merged view show it?

This chapter is the index to Part II. The first table lists every mechanism, and the second turns what you see into the mechanism responsible. The chapters that follow give each mechanism its own card.

You don't need to remember 37 of anything. You need to recognise which family a thing belongs to when you meet it, and know where that family is explained. This is how to use the chapter when a job surprises you:

Figure Using this index when a job surprises you: the fingerprint table names the mechanism, and the mechanism's visibility says where to look#

As text

Find what you see in the fingerprint table below
It names the mechanism, and the chapter that explains it
Does the merged view show it? Yes: Read it in Full configuration. No: the next step.
Is it applied when the pipeline is created? Yes: Check the pipeline graph and the settings pages. No: the next step.
It happens while the job runs: read the job log

Seven families#

The families are ordered by distance. The first lives inside your own file, and the last lives outside YAML altogether. The further away a mechanism is, the harder it is to see from your project:

Figure The seven families of reuse, ordered from closest to your file to furthest from it, with the number of mechanisms in each.#

Family	Mechanisms	What they have in common	Chapter
Inside one YAML file	3	plain YAML shortcuts that GitLab never sees as such	6
Sharing between jobs	5	one job borrowing from another once the files are joined	7
Pulling in files	5	the kinds of `include`, each fetching configuration from somewhere	8
Parameters	4	values that change what shared configuration does	9
Whole pipelines	4	one pipeline starting another	10
Organisation-wide	9	settings and policies that reach projects without the project asking	11
Outside the YAML	7	logic in images, scripts, build tools and runners	12

Four questions for every mechanism#

For each mechanism, four answers tell you where to look, and whom to ask:

Question	The answers	Why it matters when something breaks
Scope: how far does it reach?	one file · one project · cross-project · organisation-wide	tells you which repository or settings page to open
Binding: is it pinned?	inline · copied once · pinned or floating · always live	tells you whether a change somewhere else can break you without a commit of yours
Control: did you choose it?	opt-in · inherited · enforced	tells you whether you can change it yourself, or have to ask someone
Visibility: does the merged view show it?	shown · keyword only · not shown	tells you whether Full configuration answers the question, or you need another tool

All 37#

Hover over any cell for the long form of its answer.

Mechanism	Scope	Binding	Control	Phase	Merged view
Inside one YAML file
YAML anchors and aliases	One file	Inline	Opt-in	Parse	Shown
YAML merge key	One file	Inline	Opt-in	Parse	Shown
Hidden jobs	One project	Inline	Opt-in	Compile	Shown
Sharing between jobs
extends	One project	Inline	Opt-in	Compile	Shown
!reference tags	One project	Inline	Opt-in	Compile	Shown
default:	One project	Inline	Opt-in	Compile	Keyword only
Top-level variables	One project	Inline	Opt-in	Compile	Keyword only
parallel: matrix	One project	Inline	Opt-in	Create	Keyword only
Pulling in files
include: local	One project	Inline	Opt-in	Compile	Shown
include: project	Cross-project	Pinned or floating	Opt-in	Compile	Shown
include: remote	Cross-project	Pinned or floating	Opt-in	Compile	Shown
include: template	Organisation-wide	Always live	Opt-in	Compile	Shown
include: component	Cross-project	Pinned or floating	Opt-in	Compile	Shown
Parameters
spec: inputs	Cross-project	Inline	Opt-in	Compile	Shown
Variables as parameters	One project	Inline	Opt-in	Run	Keyword only
Toggle variables	One project	Inline	Opt-in	Create	Keyword only
Pipeline inputs	One project	Inline	Opt-in	Create	Keyword only
Whole pipelines
Parent–child pipelines	One project	Inline	Opt-in	Run	Keyword only
Dynamic child pipelines	One project	Inline	Opt-in	Run	Not shown
Multi-project pipelines	Cross-project	Always live	Opt-in	Run	Not shown
Pipeline trigger API	Cross-project	Always live	Opt-in	Run	Not shown
Organisation-wide
Custom CI/CD configuration file	Cross-project	Pinned or floating	Enforced	Compile	Not shown
Auto DevOps	Organisation-wide	Always live	Inherited	Compile	Not shown
Instance and group CI/CD variables	Organisation-wide	Always live	Inherited	Create	Not shown
Instance template repository	Organisation-wide	Copied once	Opt-in	Compile	Shown
Pipeline execution policies	Organisation-wide	Always live	Enforced	Create	Not shown
Scan execution policies	Organisation-wide	Always live	Enforced	Create	Not shown
Compliance pipelines	Organisation-wide	Pinned or floating	Enforced	Compile	Not shown
Merge request approval policies	Organisation-wide	Always live	Enforced	Run	Not shown
Project and file templates	Organisation-wide	Copied once	Opt-in	Compile	Shown
Outside the YAML
Toolbox images	Organisation-wide	Pinned or floating	Opt-in	Run	Keyword only
Scripts fetched at run time	Cross-project	Pinned or floating	Opt-in	Run	Keyword only
Shell function libraries	Cross-project	Pinned or floating	Opt-in	Run	Keyword only
Build-tool reuse	Cross-project	Pinned or floating	Opt-in	Run	Not shown
Internal command-line tools	Organisation-wide	Pinned or floating	Opt-in	Run	Keyword only
Runner configuration	Organisation-wide	Always live	Enforced	Run	Not shown
GitLab Functions	Cross-project	Pinned or floating	Opt-in	Run	Keyword only

From what you see to the mechanism#

Search this table for the thing in front of you. Every mechanism is in it at least once.

You see	Mechanism	Family
`&defaults` after a key, and `*defaults` later in the same file	YAML anchors and aliases	Inside one YAML file
`<<: *defaults` inside a job	YAML merge key	Inside one YAML file
a job name that starts with a dot, such as `.maven-base:`	Hidden jobs	Inside one YAML file
`extends: .maven-base` or `extends: [.base, .with-cache]`	extends	Sharing between jobs
`- !reference [.snippets, aws_login]` inside a script or rules list	!reference tags	Sharing between jobs
a top-level `default:` block setting `image`, `before_script`, `tags` or `retry`	default:	Sharing between jobs
`inherit: default: false` in a job	default:	Sharing between jobs
a top-level `variables:` block	Top-level variables	Sharing between jobs
`inherit: variables: false` in a job	Top-level variables	Sharing between jobs
`parallel: matrix:` in a job, and job names like `test: [21]` in the pipeline	parallel: matrix	Sharing between jobs
`include: local: ci/build.yml`, a wildcard such as `ci/*.yml`, or a bare path under `include:`	include: local	Pulling in files
`include: - project: devops/ci-templates` with `ref:` and `file:`	include: project	Pulling in files
`include: remote: https://…/pipeline.yml`	include: remote	Pulling in files
`include: template: Security/SAST.gitlab-ci.yml`	include: template	Pulling in files
`include: - component: $CI_SERVER_FQDN/devops/components/ecr-push@1.4.0`	include: component	Pulling in files
a `spec: inputs:` header above a `---` line	spec: inputs	Parameters
`$[[ inputs.jdk ]]` inside a template	spec: inputs	Parameters
a template reads `$JAVA_VERSION`, and your file sets it under `variables:`	Variables as parameters	Parameters
variables such as `SAST_DISABLED`, `SKIP_SONAR` or `DEPLOY_ENABLED`, tested in `rules: - if:`	Toggle variables	Parameters
a form of inputs or variables on the Run pipeline page	Pipeline inputs	Parameters
`spec: inputs:` at the top of a project's own `.gitlab-ci.yml`	Pipeline inputs	Parameters
`trigger: include: ci/deploy.yml`	Parent–child pipelines	Whole pipelines
a downstream box labelled as a child pipeline	Parent–child pipelines	Whole pipelines
`trigger: include: - artifact: generated.yml` with `job: generate`	Dynamic child pipelines	Whole pipelines
`trigger: project: platform/deployer`	Multi-project pipelines	Whole pipelines
a downstream pipeline that belongs to another project	Multi-project pipelines	Whole pipelines
`curl --request POST …/trigger/pipeline` with a trigger token, inside a script	Pipeline trigger API	Whole pipelines
pipelines run but the repository has no `.gitlab-ci.yml`	Custom CI/CD configuration file	Organisation-wide
the project's CI/CD settings name a path such as `ci/web.yml@devops/ci-templates`	Custom CI/CD configuration file	Organisation-wide
no `.gitlab-ci.yml`, yet jobs appear with names such as `build`, `test`, `code_quality` and `container_scanning`	Auto DevOps	Organisation-wide
a job sees a variable that no file defines	Instance and group CI/CD variables	Organisation-wide
a variable listed in a group's or the instance's CI/CD settings	Instance and group CI/CD variables	Organisation-wide
a `.gitlab-ci.yml` created from the Web Editor's template list, matching a file in the administrators' templates project	Instance template repository	Organisation-wide
stages named `.pipeline-policy-pre` or `.pipeline-policy-post`	Pipeline execution policies	Organisation-wide
jobs that appear in the pipeline but in no file you can find	Pipeline execution policies	Organisation-wide
a job name ending in `:policy-` followed by two numbers	Pipeline execution policies	Organisation-wide
scanner jobs named with a hyphen and a number, such as `secret-detection-1`	Scan execution policies	Organisation-wide
scanner jobs run although no file includes a scanner template	Scan execution policies	Organisation-wide
security scans that run on a schedule nobody in the project created	Scan execution policies	Organisation-wide
a compliance framework label on the project, and jobs from a file the project does not include	Compliance pipelines	Organisation-wide
a merge request needs extra approvals because of security findings or licences	Merge request approval policies	Organisation-wide
a `.gitlab-ci.yml` that matches a starter template, while the central copy has moved on	Project and file templates	Organisation-wide
`image: registry.example.com/devops/ci-tools:3.2` and a script that calls a command no repository defines	Toolbox images	Outside the YAML
`curl -sSL … \| bash`, or a `git clone` of a scripts repository, inside `script:`	Scripts fetched at run time	Outside the YAML
`source /opt/ci/lib.sh`	Shell function libraries	Outside the YAML
functions defined in a hidden job and pulled into `before_script` with `!reference`	Shell function libraries	Outside the YAML
`script: mvn -P ci verify` with a `<parent>` POM owned by another team	Build-tool reuse	Outside the YAML
`npm run ci` driven by a shared configuration package	Build-tool reuse	Outside the YAML
`script: mc deploy --env prod`: a company tool, not a public one	Internal command-line tools	Outside the YAML
behaviour that changes with the runner: environment variables, mounted files or cloud permissions that no file sets	Runner configuration	Outside the YAML
a job with `run:` and a list of steps using `func:`, and no `script:`	GitLab Functions	Outside the YAML

Pinned, floating, copied#

Binding is the question that most often explains "it broke and nobody changed anything". Every shared file is read at some ref: a branch, a tag or a commit:

A commit never changes.
A tag normally stays where it was put, although whoever owns it can move it.
A branch moves every time someone merges into it.

MegaCorp shows all three kinds of binding:

megacorp/payments/payments-api.gitlab-ci.ymlinclude

include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file: pipelines/java-service.yml
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api

Pinned. payments-api includes the golden pipeline at the tag v4.2.0, and the golden pipeline pins everything it includes to the same tag. Nothing upstream can change what payments-api gets until someone edits a version number.
Floating. web-portal's settings point at a pipeline file in ci-templates without naming a ref. It follows whatever that project's default branch holds today.
Copied. billing-batch's file was created from a starter template in 2019, and has never heard from the template since.

Spot the bugName the mechanism#

Four things a developer found while investigating MegaCorp pipelines:

A job in billing-batch has a line <<: *defaults.
A pipeline has a stage called .pipeline-policy-post, holding a job that no file in the project mentions.
web-portal's pipelines run, but the repository has no .gitlab-ci.yml.
A before_script contains - !reference [.snippets, aws_login].

QuestionWhich mechanism explains each one, and where would you look next?

Show the answer

A YAML merge key. Look for &defaults in the same file.
A pipeline execution policy. Look at the group's security policies.
A custom CI/CD configuration file, set in the project's CI/CD settings. Auto DevOps can also produce pipelines without a file, but its jobs have GitLab's names.
A !reference tag. Search the included files for .snippets:.

Next06YAML-level reuse: anchors, merge keys and hidden jobs5 min

YAML-level reuse: anchors, merge keys and hidden jobs#

Part II · Reuse · Chapter 06· 5 min read

YAML has its own copy and paste, which happens before GitLab reads a single keyword:

An anchor, written &name, gives a piece of YAML a name.
An alias, written *name, pastes that piece somewhere else.
A merge key, written <<:, pastes the keys of a named map into another map.

All three work only inside the file that defines them. Hidden jobs, whose names start with a dot, are GitLab's own addition: jobs that never run and exist only to be copied from. You'll meet all of this in older files, and in any file whose author never needed to reach across an include.

Anchors, aliases and merge keys#

Here is the whole idea in seven lines:

.base: &base              # "&base" gives this map the name base
  image: node:22
  tags: [linux]

test:
  <<: *base               # paste the keys of the map named base
  script: npm test

GitLab never sees &base or *base. The YAML reader has already replaced them, so GitLab reads test as if it had been written out in full:

test:
  image: node:22
  tags: [linux]
  script: npm test

The dot in front of .base matters too. Without it, GitLab would try to run base as a job of its own.

MegaCorp's oldest project still shares configuration this way:

megacorp/legacy/billing-batch.gitlab-ci.ymlanchors

.defaults: &defaults
  image: registry.example.com/megacorp/devops/ci-tools:2.9
  tags: [megacorp-shared]
  only:
    - branches
    - tags

build:
  <<: *defaults
  stage: build
  script:
    - ./gradlew assemble

The same three pieces are at work:

&defaults names the map under .defaults.
<<: *defaults copies that map's keys into build.
The dot in front of .defaults stops GitLab from treating the anchor's home as a job.

By the time GitLab sees the file, build simply has an image, tags and only of its own. That is why anchors are the one kind of reuse that leaves no trace in the merged configuration. It shows the result, with the anchors already replaced. Here is another of billing-batch's jobs, next to what GitLab made of it:

You wrotemegacorp/legacy/billing-batch · .gitlab-ci.yml

notify-reports:
  <<: *defaults
  stage: deploy
  only: [tags]
  script:
    - >
      curl --fail --request POST
      --form "token=$REPORTS_TRIGGER_TOKEN"
      --form "ref=main"
      --form "variables[BILLING_VERSION]=$CI_COMMIT_TAG"
      "https://gitlab.example.com/api/v4/projects/4242/trigger/pipeline"

GitLab buildsmerged configurationcaptured from GitLab

notify-reports:
  image: registry.example.com/megacorp/devops/ci-tools:2.9
  tags:
  - megacorp-shared
  only:
  - tags
  stage: deploy
  script:
  - 'curl --fail --request POST --form "token=$REPORTS_TRIGGER_TOKEN" --form "ref=main"
    --form "variables[BILLING_VERSION]=$CI_COMMIT_TAG" "https://gitlab.example.com/api/v4/projects/4242/trigger/pipeline"

    '

notify-reports merges *defaults, which says only: [branches, tags], and then writes only: [tags] itself. A key written in the job always beats a merged one, so the job runs only for tags.

Anchors have one hard limit: they work only inside the file that defines them.

Figure An alias finds its anchor only inside the same file. Across an include it can't, which is why central teams use !reference instead (chapter 7).#

What wins in a merge#

Merge keys follow two rules, and both differ from extends, which chapter 7 covers:

Situation	Merge key <<:	extends
the job sets a key the source also sets	the job wins	the job wins
both sides have a map, such as `variables`	the job's map replaces the source's whole map: merge keys are shallow	the maps merge key by key
two sources set the same key	`<<: [a, b]`: the earlier one, `*a`, wins	`extends: [.a, .b]`: the later one, `.b`, wins
the source is in an included file	impossible: anchors stop at the file boundary	works

Gotcha

The third row catches people who know one mechanism and assume the other behaves the same way. When you see a list after <<:, the first entry is the one that sticks.

The local GitLab 19.3 confirmed both rules. A job written <<: [*a, *b] took its image and its variables from a, the first in the list. A job that merged *a and wrote its own variables: kept only its own variable. Every variable from the anchor was gone.

Inside one YAML file#

YAML anchors and aliases

Scope: One file
Binding: Inline
Control: Opt-in
Phase: Parse
Merged view: Shown

Looks like

.ci_image: &ci_image registry.example.com/megacorp/devops/ci-tools:2.9

build:
  image: *ci_image

What GitLab does

The YAML reader resolves anchors before GitLab reads the configuration. An alias becomes a copy of whatever its anchor names: a string, a list or a map. GitLab never sees the anchor, only the copy, so the Full configuration view shows the result with every alias replaced.

How to trace it

An alias always has its anchor in the same file. Search that file for &name, and nowhere else.

Gotchas

An anchor in an included file is invisible to the file that includes it. Central teams use !reference instead (chapter 7).
Editing the anchored block changes every alias at once, with no sign at the places that use it.

Inside one YAML file#

YAML merge key

Scope: One file
Binding: Inline
Control: Opt-in
Phase: Parse
Merged view: Shown

Looks like

build:
  <<: *defaults

test:
  <<: [*defaults, *java21]

What GitLab does

The merge key copies the keys of the named map into the current map, except for keys the current map already has. The copy is shallow: a key whose value is a map, such as variables, is copied or ignored whole, never merged. With a list of maps, earlier maps win over later ones.

How to trace it

Find the anchor in the same file, then compare keys. Any key the job writes itself wins entirely. Anything else comes from the first map in the list that has it.

Gotchas

Adding one variable to a job that merges *defaults deletes all of the anchored variables for that job.
The order in a list is the opposite of extends.

Hidden jobs#

Any top-level key that starts with a dot is ignored as a job. That makes hidden jobs the raw material of nearly every template library:

They hold anchors, as in billing-batch.
They are the parents that extends copies from.
They hold the snippets that !reference pastes.

MegaCorp's base template is one:

megacorp/devops/ci-templatestemplates/base.ymlmegacorp-base

.megacorp-base:
  variables:
    MC_TEAM: unknown
  artifacts:
    expire_in: 7 days
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]
    - !reference [.rules, release-tag]

Inside one YAML file#

Hidden jobs

Scope: One project
Binding: Inline
Control: Opt-in
Phase: Compile
Merged view: Shown

Looks like

.megacorp-base:
  artifacts:
    expire_in: 7 days

.snippets:
  functions:
    - source /opt/megacorp/lib/ci-lib.sh

What GitLab does

GitLab doesn't process a job whose name begins with a dot. It never runs, and it never appears in a pipeline. Its content is still there to borrow, through extends, !reference or YAML anchors. Because it is never treated as a job, it can also hold keys that aren't CI/CD keywords at all, like functions above.

How to trace it

Search every included file for the name, with its dot and colon: .megacorp-base:. With several includes, check each one. The name alone doesn't tell you which file defines it.

Gotchas

A dot is also the quickest way to switch a job off. A job that "disappeared" from the pipeline may simply have been hidden.
If two included files define the same hidden job, they merge like any duplicated key, and the file included last wins where they clash.

Spot the bugThe Gradle job on the wrong JDK#

Someone modernised billing-batch's build job so that it builds on JDK 21. This is billing-batch/.gitlab-ci.yml, modified:

.defaults: &defaults
  image: registry.example.com/megacorp/devops/ci-tools:2.9
  variables:
    GRADLE_OPTS: "-Dorg.gradle.daemon=false"

.java21: &java21
  image: registry.example.com/megacorp/devops/ci-tools/gradle:8-jdk21

build:
  <<: [*defaults, *java21]
  variables:
    BILLING_ENV: ci
  script:
    - ./gradlew assemble

The job still reports JDK 11, and the Gradle daemon is starting again.

QuestionWhich image does build use, and why has GRADLE_OPTS gone?

Show the answer

It uses ci-tools:2.9. In a merge-key list the earlier map wins, and *defaults comes first. GRADLE_OPTS is gone because the job writes its own variables:. Merge keys are shallow, so that map replaces the anchored one outright.

Put *java21 first and repeat GRADLE_OPTS in the job. Or move both anchors into hidden jobs and use extends: [.defaults, .java21], which merges maps key by key and lets the later parent win. See merge keys.

Next07extends, !reference and default13 min

extends, !reference and default#

Part II · Reuse · Chapter 07· 13 min read

Five keywords let one job borrow configuration from somewhere else. Between them they explain most of the "where did that line come from?" moments in an enterprise pipeline: extends, !reference, default:, top-level variables: and parallel: matrix.

One rule explains most of their surprises: maps merge key by key, but lists are replaced whole. A map is a set of named keys, such as variables. A list is a sequence of items, such as script (YAML in five minutes). If you override one line of a template's rules, script or before_script, you have replaced all of it.

The five, side by side#

In MegaCorp's template library, almost nothing is a real job. The templates are hidden jobs, whose names start with a dot, and they borrow from each other through the five keywords below. The project's own file then turns a few of them into real jobs.

Keyword	What it shares	How it combines with the job	Works across included files
`extends`	a whole job, from one or more parents	maps merge key by key, lists are replaced, and the later parent wins	yes
`!reference`	one section of another job, such as its `script` or `rules`	pastes exactly what it points at, where you put it	yes
`default:`	eleven keywords, such as `image` and `before_script`, for every job	none: a job's own keyword replaces the default outright	yes, wherever it is defined
top-level `variables:`	variables for every job	a job's own variables outrank them	yes
`parallel: matrix`	one job definition, run once per combination of values	each generated job gets its own copy of the variables	not applicable

Before tracing anything, open the one view that resolves all five for you: Build › Pipeline editor, then the Full configuration tab. It shows the configuration with included files copied in, extends merged, YAML anchors replaced and !reference tags replaced.

extends: a job inherits a job#

Take the maven-test job in payments-api's pipeline. The project's own file only adds one variable to it. The job itself is defined three files away: pipelines/java-service.yml says maven-test: extends: .maven-test. .maven-test extends .maven-base, which extends .megacorp-base. Here is the middle of that chain:

megacorp/devops/ci-templatestemplates/java-maven.ymlmaven-base

.maven-base:
  extends: .megacorp-base
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
  variables:
    JAVA_VERSION: "21"
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  cache:
    key:
      files: [pom.xml]
    paths: [.m2/repository]
  before_script:
    # a job's own before_script replaces default:before_script, so re-add the functions
    - !reference [.snippets, functions]
    - !reference [.snippets, maven_settings]

Here is what GitLab builds from the whole chain, next to three of the four files that feed it:

You wrotemegacorp/payments/payments-api · .gitlab-ci.yml

maven-test:
  variables:
    MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"

You wrotemegacorp/devops/ci-templates · pipelines/java-service.yml

maven-build:
  extends: .maven-build

maven-test:
  extends: .maven-test

image-build:
  extends: .image-build
  needs: [maven-build]

You wrotemegacorp/devops/ci-templates · templates/java-maven.yml

.maven-test:
  extends: .maven-base
  stage: test
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JDK}
  parallel:
    matrix:
      - JDK: ["17", "21"]
  script:
    - mvn $MAVEN_CLI_OPTS verify
  artifacts:
    when: always
    reports:
      junit: target/surefire-reports/TEST-*.xml
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]

GitLab buildsmerged configurationcaptured from GitLab

maven-test:
  variables:
    MC_TEAM: unknown
    JAVA_VERSION: '21'
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
    MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"
  artifacts:
    expire_in: 7 days
    when: always
    reports:
      junit:
      - target/surefire-reports/TEST-*.xml
  rules:
  - - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
  - - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  - - if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
  extends: ".maven-test"
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JDK}
  cache:
    key:
      files:
      - pom.xml
    paths:
    - ".m2/repository"
  before_script:
  - - source /opt/megacorp/lib/ci-lib.sh
  - - mkdir -p .m2
    - mc_maven_settings > .m2/settings.xml
  stage: test
  parallel:
    matrix:
    - JDK:
      - '17'
      - '21'
  script:
  - mvn $MAVEN_CLI_OPTS verify

Five things to notice, and each is the merge rule at work:

Variables from three levels become one map. MC_TEAM comes from .megacorp-base, JAVA_VERSION and MAVEN_OPTS from .maven-base, and MAVEN_CLI_OPTS from the project's file.
So do the artifacts. expire_in: 7 days comes from .megacorp-base, and when and reports come from .maven-test. artifacts is a map, so the two merged.
The image is .maven-test's, not .maven-base's. When the same key appears at two levels, the later one wins.
The rules are .maven-test's alone. .megacorp-base had a fourth rule, for release tags, but a list never merges. That is why maven-test does not run in tag pipelines while maven-build does.
Pasted snippets show as lists inside the list. The rules and before_script entries that start with - - were pasted in by !reference, covered in the next section. GitLab flattens them into one list when the job runs.

Whenever you wonder whether your own key merged with a template's or replaced it, ask these questions in order:

Figure What extends does with each key a job sets: maps merge, lists are replaced, and anything else is simply overwritten#

As text

Is the key's value a map, such as variables, cache or artifacts? Yes: The two maps merge key by key; on a clash, the job's value wins. No: the next step.
Is it a list, such as script, before_script or rules? Yes: The job's list replaces the template's whole list. No: the next step.
It is a single value, such as image or stage: the job's value replaces the template's

Figure How extends combines a template with the job that extends it: maps merge key by key, lists are replaced whole.#

Sharing between jobs#

extends

Scope: One project
Binding: Inline
Control: Opt-in
Phase: Compile
Merged view: Shown

Looks like

maven-test:
  extends: .maven-test

integration-test:
  extends: [.maven-base, .with-database]

What GitLab does

GitLab resolves the chain of parents first, then merges each level into the next. Maps such as variables, cache and artifacts merge key by key. Where the same key appears twice, the later definition wins: the job over its parent, and the second parent over the first. Lists such as script, before_script and rules are replaced whole. A job can extend a hidden job defined in any included file. Chains can be eleven levels deep, though GitLab's own guidance is to stay within three.

How to trace it

Open Full configuration in the pipeline editor, where the job appears with everything it inherited already merged in. To see which file contributed a line, follow the extends names upwards. Search the included files for each parent's name, such as .maven-test:. Expect every hop to be in a different file.

Gotchas

Setting rules, script or before_script on your job throws away the template's version entirely (symptom card).
Setting one variable is safe, because maps merge. People who have been bitten by lists often copy a whole variables: block when a single key would do.
A job's variables inherited through extends outrank the project's top-level variables (below).

Didn't run, or ran when it shouldn'tMerging and inheritance#

You gave a template job one rule of your own. Now it runs in pipelines it never used to, or has vanished from ones it always ran in.

Cause

Your rules: replaced the template's entire list. Lists never merge, so every rule the template had is gone, including the ones that skipped scheduled pipelines or added tag pipelines.

Confirm

In Build › Pipeline editor › Full configuration, find the job. Its rules: shows only your lines.

Fix

Rebuild the list: paste the template's rules back with !reference and put your rule where it belongs. The first matching rule wins, so order matters.

maven-test:
  rules:
    - if: $SKIP_TESTS == "true"
      when: never
    - !reference [.maven-test, rules]

!reference: paste one section#

extends takes a whole job. !reference takes one piece of one, and pastes it exactly where you write it. MegaCorp keeps its script fragments in a hidden job whose keys aren't CI/CD keywords at all. GitLab's own documentation uses the same pattern.

megacorp/devops/ci-templatestemplates/snippets.ymlsnippets

.snippets:
  functions:
    - source /opt/megacorp/lib/ci-lib.sh
  aws_login:
    - mc_aws_login "$AWS_ROLE_ARN"
  maven_settings:
    - mkdir -p .m2
    - mc_maven_settings > .m2/settings.xml

The rules library works the same way. Named lists of rules are assembled job by job, which is how .maven-test got its three rules:

megacorp/devops/ci-templatestemplates/rules.ymlrules-library

.rules:
  mr:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  default-branch:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  release-tag:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
  never-on-schedule:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never

You will meet the older way of doing this in long-lived projects: a YAML anchor. The two look alike, but they don't reach equally far:

.login: &login
  - mc_aws_login "$AWS_ROLE_ARN"

image-build:
  before_script: *login

This works only if .login is in the same file as image-build.

image-build:
  before_script:
    - !reference [.snippets, aws_login]

This works wherever .snippets is defined, including in an included file.

Anchors stop at the file boundary

A YAML anchor exists only inside the file that defines it. An included file's anchors are invisible to the file that includes it. The workaround that central teams use is !reference, which GitLab resolves after it has assembled every included file.

Sharing between jobs#

!reference tags

Scope: One project
Binding: Inline
Control: Opt-in
Phase: Compile
Merged view: Shown

Looks like

before_script:
  - !reference [.snippets, functions]
rules:
  - !reference [.rules, mr]
variables:
  NEXUS_URL: !reference [.vars, variables, NEXUS_URL]

What GitLab does

GitLab replaces the tag with the configuration it names: a job, then a key path inside it. A list pasted into a list is flattened. The path can name any key in a hidden job, not only a CI/CD keyword. The tags can be nested up to ten levels deep in script, before_script and after_script. They work across included files, but inputs can't be used inside the path.

How to trace it

The tag names its source: the first item is the job to search for, such as .snippets:. In the Full configuration view the pasted content is already in place. Nested tags sometimes show extra leading hyphens there, which GitLab documents as expected and harmless.

Gotchas

The paste happens when GitLab assembles the configuration, but the pasted lines run inside your job. Functions a snippet defines exist only in the job that pasted it.
A snippet pasted into before_script disappears the moment a job redefines before_script without pasting it again.

default: the layer you never see#

default: sets a keyword for every job that doesn't set it itself. It lives in a template, so no project file mentions it, yet it is in every job.

megacorp/devops/ci-templatestemplates/base.ymldefault

default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  tags: [megacorp-shared]
  interruptible: true
  retry:
    max: 2
    when: [runner_system_failure, stuck_or_timeout_failure]
  before_script:
    - !reference [.snippets, functions]
    - mc_log "job $CI_JOB_NAME on $CI_COMMIT_REF_NAME"

Eleven keywords can have defaults: after_script, artifacts, before_script, cache, hooks, id_tokens, image, interruptible, retry, services and tags. Unlike extends, a default never merges with anything. If a job has the keyword, the default for that keyword simply doesn't apply.

Gotcha

MegaCorp's default before_script loads the shell function library that every other template relies on. The moment a job defines its own before_script, even a single line, the library is no longer loaded. That is why .maven-base pastes the functions back in, and says so in a comment.

MegaCorp uses !reference inside default: as well as inside jobs. GitLab's documentation shows it only in jobs, but the local GitLab 19.3 accepts a default: before_script built from !reference without complaint.

Opting out with inherit#

A job can refuse some or all of the defaults with inherit. inherit: default: takes true, false, or a list of the keywords to keep. inherit: variables: does the same for top-level variables.

megacorp/payments/payments-api.gitlab-ci.ymlinherit

publish-docs:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
  inherit:
    default: [tags, retry]
    variables: false
  script:
    - mkdocs build --strict
  rules:
    - !reference [.rules, default-branch]

publish-docs keeps the organisation's runner tags and retry policy. It drops the default image, which it overrides anyway, and the default before_script, so none of the shell functions are loaded. It also receives none of the top-level variables.

Legacyimage, services, cache, before_script and after_script at the top level

Older files set these five keywords at the top level, outside any job, as a way of applying them to every job. GitLab has deprecated that form. The replacement is the same keywords under default:. When you meet the old form, read it exactly as you would a default: block.

Sharing between jobs#

default:

Scope: One project
Binding: Inline
Control: Opt-in
Phase: Compile
Merged view: Keyword only

Looks like

default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  before_script:
    - !reference [.snippets, functions]

publish-docs:
  inherit:
    default: [tags, retry]

What GitLab does

For each of the eleven keywords it can hold, default: supplies a value to every job that doesn't have one. A job's own value replaces the default outright; the two are never merged. inherit: default: turns the defaults off, entirely or keyword by keyword. Defaults are not passed to downstream pipelines, which start again from their own configuration.

How to trace it

Search the included files for default:. Then compare it with the job in the Full configuration view: a keyword the job doesn't show was never overridden, so the default applies. Check for inherit: on the job before concluding anything.

Gotchas

A single line of before_script on a job removes the whole default before_script, including anything the organisation put there.
A child or multi-project pipeline doesn't get the parent's defaults. The same template can behave differently one pipeline downstream.

Top-level variables#

A variables: block at the top of any file applies to every job. Blocks from included files and from the project's own file merge into one map:

megacorp/devops/ci-templatestemplates/base.ymlvariables

variables:
  GIT_DEPTH: "20"
  AWS_REGION: eu-west-2
  MC_TEMPLATES_VERSION: "4.2.0"

megacorp/payments/payments-api.gitlab-ci.ymlvariables

variables:
  JAVA_VERSION: "21"
  MC_TEAM: payments

Here, payments-api sets MC_TEAM: payments, but every one of its jobs still sees unknown. The reason is in the base template:

megacorp/devops/ci-templatestemplates/base.ymlmegacorp-base

.megacorp-base:
  variables:
    MC_TEAM: unknown
  artifacts:
    expire_in: 7 days
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]
    - !reference [.rules, release-tag]

.megacorp-base sets MC_TEAM as a job variable, and every template extends it. Job variables outrank top-level variables, so the project's value never reaches a job. The ranking goes further than YAML: a variable set in the project's, group's or instance's CI/CD settings outranks everything in YAML. Chapter 19 gives the full ladder.

Sharing between jobs#

Top-level variables

Scope: One project
Binding: Inline
Control: Opt-in
Phase: Compile
Merged view: Keyword only

Looks like

variables:
  GIT_DEPTH: "20"
  AWS_REGION: eu-west-2

publish-docs:
  inherit:
    variables: false

What GitLab does

Top-level variables become default variables for every job. Blocks from every included file and the project's own file merge into one map, with the project's file winning on a clash. A job's own variables, including those it inherits through extends, outrank them. inherit: variables: can turn them off for a job, entirely or by name.

How to trace it

Look for the variable in three places, in this order:

the job's own variables: in the Full configuration view
the top-level variables: blocks
the project's, group's and instance's CI/CD settings, which outrank both of the above

Gotchas

A project cannot override a variable that a template sets at job level by setting it at the top of its own file.
A top-level variable is not the last word. A CI/CD variable of the same name set in the UI replaces it without a trace in any file.

parallel: matrix: one definition, several jobs#

.maven-test, quoted in the comparison earlier, asks for a matrix of two JDK versions. The pipeline shows one job per value, named after the job and the value:

Pipeline Jobs built from java-maven.yml and container.yml in a payments-api merge request pipeline#

As text

build: maven-build
test: maven-test: [17], maven-test: [21]
package: image-build

Needs: image-build waits only for maven-build.

Each generated job gets its own JDK variable, which the image name uses. A matrix can produce at most 200 jobs. When another job needs only some of them, needs: parallel: matrix selects them by value.

Sharing between jobs#

parallel: matrix

Scope: One project
Binding: Inline
Control: Opt-in
Phase: Create
Merged view: Keyword only

Looks like

.maven-test:
  parallel:
    matrix:
      - JDK: ["17", "21"]

What GitLab does

GitLab creates one job for every combination of the matrix values. It names each <job>: [<value>, …], for example maven-test: [17], and gives each its own copy of the variables. The matrix is expanded when the pipeline is created, not when the configuration is assembled.

How to trace it

A job name ending in square brackets is a matrix job. Strip the brackets and search for the name; here, maven-test leads to .maven-test and its parallel: block. The Full configuration view shows the matrix definition, not the generated jobs; the pipeline graph shows those.

Gotchas

needs: [maven-test] depends on every job in the matrix. To depend on one, use needs: parallel: matrix.
Matrix values are plain variables. A value that collides with a variable set elsewhere follows the usual precedence rules.

Which of these jobs run where#

Put the rules from this chapter together and you can predict, without running anything, which of payments-api's jobs exist in each kind of pipeline:

job	feature branch push	merge request	main	tag v1.4.0	nightly on main
maven-build	not in the pipeline	runs	runs	runs	not in the pipeline
maven-test: [17] and [21]	not in the pipeline	runs	runs	not in the pipeline	not in the pipeline
image-build	not in the pipeline	runs	runs	runs	not in the pipeline
publish-docs	not in the pipeline	not in the pipeline	runs	not in the pipeline	runs

payments-api: the jobs in this chapter, by pipeline·not in the pipeline✓runs

Two cells are worth a second look:

Tag pipelines have no tests. maven-test's own rules replaced the base template's list, which was the only one with a tag rule.
publish-docs runs every night. Its only rule is "on the default branch", and nothing tells it to skip scheduled pipelines.

How GitLab resolves an extends chaindeep dive

GitLab assembles the configuration first: every included file is copied in, and the project's file is merged on top. Only then does it resolve extends. For each job it walks up the chain of parents, .megacorp-base then .maven-base then .maven-test, and merges downwards, each level into the next.

At every level the same reverse deep merge applies. Maps are merged key by key, the later level wins a clash, and a list replaces the list above it whole. With several parents, extends: [.a, .b], .b is applied after .a and wins any clash between them.

!reference tags are resolved in the same assembly pass, which is why they can point into included files when YAML anchors cannot. By the time the pipeline is created, none of these keywords exists any more. There are only complete jobs, which is exactly what the Full configuration view shows.

Spot the bugThe team name that never changes#

A dashboard groups jobs by the MC_TEAM variable. It files every payments-api job under unknown, even though payments-api's own file sets it:

megacorp/payments/payments-api.gitlab-ci.ymlvariables

variables:
  JAVA_VERSION: "21"
  MC_TEAM: payments

QuestionWhy do the jobs never see payments, and what is the least invasive fix?

Show the answer

.megacorp-base sets MC_TEAM: unknown as a job variable, and every job reaches it through extends. Job variables outrank top-level variables, so the project's value is overridden in every job.

The least invasive fix is a project CI/CD variable MC_TEAM=payments in the project's settings, because settings variables outrank anything in YAML. The better long-term fix is for the template owners to move MC_TEAM to top-level variables, so that projects can override it. See top-level variables.

Spot the bugOne line that broke Maven#

The payments team needed a database for maven-test, so they added one line to their file. This is payments-api/.gitlab-ci.yml, modified:

maven-test:
  before_script:
    - ./scripts/start-db.sh
  variables:
    MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"

Now both maven-test jobs fail at once. Maven complains that .m2/settings.xml does not exist.

QuestionThey only added a line. Where did settings.xml go, and how should they have added their script?

Show the answer

before_script is a list, and lists are replaced whole. The job's one-line list replaced .maven-base's list. That list was the one that loaded the function library and wrote .m2/settings.xml.

Keep the template's lines by pasting them back first:

maven-test:
  before_script:
    - !reference [.maven-base, before_script]
    - ./scripts/start-db.sh

See the symptom card for the same trap with rules, and extends.

Next08include: pulling in files6 min

include: pulling in files#

Part II · Reuse · Chapter 08· 6 min read

include copies configuration from other files into yours, before anything else happens. There are five kinds, and they differ only in where the file comes from:

local: your own repository
project: another project
remote: a URL
template: GitLab itself
component: a versioned component

Every included file is merged into one configuration, and your own file wins any clash. The most important thing to read on an include is not the file name but the ref: the branch, tag or commit it reads from. The ref decides whether a change somewhere else can reach you without a commit of yours.

Five kinds#

Kind	Fetches from	Binding	You'll see it as
local	the same repository, at the same commit	inline	`include: ci/build.yml`, or a wildcard such as `ci/*.yml`
project	a file in another project, at a ref	pinned or floating, depending on `ref`	`project:` with `ref:` and `file:`
remote	any public HTTP(S) URL	whatever the URL serves today	`remote: https://…`
template	GitLab's own library of templates	changes when your GitLab is upgraded	`template: Security/SAST.gitlab-ci.yml`
component	a component project, at a version	pinned or floating, depending on the version	`component: $CI_SERVER_FQDN/…@1.4.0`

Figure Where each kind of include fetches its file. Green always matches your own commit. Blue is pinned or floating, depending on the ref or version. Amber is whatever the URL serves today. Grey changes when GitLab is upgraded. Everything is merged into one configuration.#

MegaCorp uses all five. payments-api includes a project file and a component:

megacorp/payments/payments-api.gitlab-ci.ymlinclude

include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file: pipelines/java-service.yml
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api

The security template includes GitLab's own templates:

megacorp/devops/ci-templatestemplates/security.ymlsecurity-includes

include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml
  - template: Security/Container-Scanning.gitlab-ci.yml

The monorepo includes a file from its own repository, and billing-batch still pulls one in by URL:

megacorp/data/mono.gitlab-ci.ymllocal-include

include:
  - local: ci/common.yml

megacorp/legacy/billing-batch.gitlab-ci.ymlremote-include

include:
  - remote: https://gitlab.example.com/megacorp/devops/ci-templates/-/raw/main/legacy/notify.yml

How included files merge#

An included file can include more files, so payments-api's single include becomes ten files. GitLab then merges everything into one configuration, using the same rule as extends (chapter 7):

maps merge at any depth
lists are replaced whole
where two files set the same key, the including file beats the included one

The same file may be included twice, and the last inclusion wins.

That is how MegaCorp adjusts GitLab's container-scanning job without copying it. The security template includes GitLab's template, then defines a job with the same name:

You wrotemegacorp/devops/ci-templates · templates/security.yml

variables:
  CS_IMAGE: $IMAGE_REF

container_scanning:
  stage: scan
  needs: [image-build]
  rules:
    - if: $CONTAINER_SCANNING_DISABLED == "true"
      when: never
    - !reference [.rules, never-on-schedule]
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

GitLab buildsmerged configurationcaptured from GitLab

container_scanning:
  image: "$CS_ANALYZER_IMAGE$CS_IMAGE_SUFFIX"
  stage: scan
  variables:
    GIT_STRATEGY: none
  allow_failure: true
  artifacts:
    access: developer
    reports:
      container_scanning:
      - gl-container-scanning-report.json
      cyclonedx:
      - "**/gl-sbom-*.cdx.json"
    paths:
    - gl-container-scanning-report.json
    - gl-dependency-scanning-report.json
    - "**/gl-sbom-*.cdx.json"
  dependencies: []
  script:
  - gtcs scan
  rules:
  - if: $CONTAINER_SCANNING_DISABLED == "true"
    when: never
  - - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
  - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  - if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
  needs:
  - image-build

Read the merged job from the bottom up. stage, rules and needs are MegaCorp's. Everything else is GitLab's: the image, the script, the artifacts, and allow_failure: true. MegaCorp's rules replaced GitLab's list outright, because lists never merge. From here on, GitLab's own rules for the job, whatever they were in this version, no longer apply.

Reading a ref#

Every include except local reads its file at some ref. That one word decides whether a change somewhere else can reach you:

Figure Will a change upstream reach you without a commit of yours? Read the ref on the include.#

As text

Is the ref a commit SHA? Yes: Never: a commit can't change. No: the next step.
Is it a tag, such as v4.2.0? Yes: Only if someone changes the version, or moves the tag. No: the next step.
Is it a branch, such as main? Yes: Yes, on the very next pipeline. No: the next step.
There is no ref at all: yes, it follows the other project's default branch

The include says	You get	Upstream changes reach you
`ref: v4.2.0`, a tag	that tag's content	only when someone edits the version, or moves the tag
`ref:` with a commit SHA	that exact commit	never
`ref: main`, a branch	whatever the branch holds when the pipeline is created	at once, on the next pipeline
no `ref` at all	the head of the other project's default branch	at once, on the next pipeline

A missing ref is the easiest time bomb to plant, because it looks tidy and floats silently. MegaCorp pins everything to v4.2.0, both where payments-api includes the golden pipeline and inside the golden pipeline itself. A project that pinned only the first file would still get whatever the nested includes point at.

What an include can depend on#

An include can carry rules, with if, exists or changes, and GitLab skips it when they don't match. The variables it can test are limited, because GitLab evaluates include rules before any job exists:

Usable in include rules	Not usable
project, group and instance CI/CD variables	variables defined in a job
predefined `CI_PROJECT_*` variables	top-level `variables:` in any file
variables of triggered, scheduled and manually run pipelines
`CI_PIPELINE_SOURCE`, `CI_PIPELINE_TRIGGERED`, `CI_COMMIT_REF_NAME`

One more trap: in a file included from another project, rules: exists checks for files in that project, not yours. Use rules: exists: project to point it back. Included files can also take parameters, called inputs, which chapter 9 covers.

A nested include: local is read from the project that holds the file doing the including, not from yours. On the local GitLab 19.3, a file in ci-templates included local: templates/rules.yml while being included into payments-api. GitLab found ci-templates' copy, although payments-api has no such file.

Pulling in files#

include: local

Scope: One project
Binding: Inline
Control: Opt-in
Phase: Compile
Merged view: Shown

Looks like

include:
  - local: ci/common.yml
  - local: ci/jobs/*.yml

What GitLab does

GitLab copies the file from the same repository, at the same commit as the pipeline, into the configuration. A wildcard pulls in every match. A single * doesn't reach into subfolders; ** does.

How to trace it

Open the path in the same repository, at the commit the pipeline ran for. With a wildcard, list the matching files at that commit, because a file added later changes what the include means.

Gotchas

A new file that matches a wildcard joins the pipeline without anyone editing the include.
Inside a file that was itself included from another project, "local" means that project, not yours (see the note above).

Pulling in files#

include: project

Scope: Cross-project
Binding: Pinned or floating
Control: Opt-in
Phase: Compile
Merged view: Shown

Looks like

include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file:
      - templates/base.yml
      - templates/rules.yml

What GitLab does

GitLab fetches each listed file from the other project, at ref, and merges it into the configuration. Without a ref, it uses the head of that project's default branch.

How to trace it

Open the other project at the ref shown, not at its default branch, and read the file there. Check whether that file includes more files, and at which refs.

Gotchas

A branch ref, or no ref at all, floats. The pipeline can change without a commit in your project.
Pinning the top-level include pins nothing below it, unless the included file pins its own includes too.

Pulling in files#

include: remote

Scope: Cross-project
Binding: Pinned or floating
Control: Opt-in
Phase: Compile
Merged view: Shown

Looks like

include:
  - remote: https://gitlab.example.com/megacorp/devops/ci-templates/-/raw/main/legacy/notify.yml

What GitLab does

GitLab downloads the URL when it assembles the configuration, and merges the result. The URL must be publicly reachable. GitLab's guidance is to avoid rate-limited hosts, such as GitLab Pages.

How to trace it

Open the URL. What it serves now is what the next pipeline gets, which is not necessarily what an earlier pipeline got.

Gotchas

Nothing pins a remote include unless the URL itself names a fixed version. Here it names the main branch.
The file's owner can change every consumer's pipeline, without any of them knowing the file exists.

Pulling in files#

include: template

Scope: Organisation-wide
Binding: Always live
Control: Opt-in
Phase: Compile
Merged view: Shown

Looks like

include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml

What GitLab does

GitLab inserts one of the templates it ships with. They cover security scanners, Auto DevOps and common build jobs. The content belongs to the GitLab version your instance runs.

How to trace it

The Full configuration view shows the template's jobs copied in. GitLab's templates are published in GitLab's own source repository, under lib/gitlab/ci/templates/. Read the version that matches your instance.

Gotchas

Upgrading GitLab changes what the template contains, including job names and rules, without any commit in any of your projects.
Overriding a template job by name merges with it. Overriding its rules replaces them entirely.

Spot the bugThe deploy jobs that never appear#

A team wanted deploy jobs only in projects that opt in, so their file says:

include:
  - local: ci/deploy.yml
    rules:
      - if: $DEPLOY_ENABLED == "true"

build:
  variables:
    DEPLOY_ENABLED: "true"
  script: make build

The deploy jobs from ci/deploy.yml never appear, on any branch.

QuestionWhy is the include always skipped, and how can the project opt in?

Show the answer

Include rules are evaluated while GitLab assembles the configuration, before any job exists, so they can't see variables defined in a job. $DEPLOY_ENABLED is empty there, and the rule never matches.

Set DEPLOY_ENABLED as a CI/CD variable in the project's settings, which include rules can see. Another option is a rule on a predefined variable such as $CI_COMMIT_REF_NAME. See include: local and what an include can depend on.

Next09Components, inputs and toggles7 min

Components, inputs and toggles#

Part II · Reuse · Chapter 09· 7 min read

Shared configuration usually needs settings of its own, called parameters, and there are two ways to pass them:

Inputs are declared by the shared file, with a type. GitLab writes each value into the file, in place of $[[ inputs.name ]], when the pipeline is created. After that they never change.
Variables are the older way. The shared file reads $SOMETHING, and whoever uses it sets the value. Variable precedence decides who really wins.

A component is shared configuration packaged for this: a versioned file with declared inputs, included with component:. A toggle is a variable that a rule tests, turning a job on or off.

Components#

A component lives in a component project, as templates/<name>.yml or templates/<name>/template.yml. The include names the project, the component and a version. Here is MegaCorp's Sonar component, which runs a code-quality scan:

megacorp/devops/componentstemplates/sonar-scan.ymlspec

spec:
  inputs:
    stage:
      default: test
    project_key:
      description: SonarQube project key
    quality_gate:
      type: boolean
      default: true
---

megacorp/devops/componentstemplates/sonar-scan.ymljob

sonar-scan:
  stage: $[[ inputs.stage ]]
  image: registry.example.com/megacorp/devops/ci-tools/sonar-scanner:6
  variables:
    SONAR_PROJECT_KEY: $[[ inputs.project_key ]]
    SONAR_QUALITYGATE_WAIT: "$[[ inputs.quality_gate ]]"
  script:
    - sonar-scanner -Dsonar.projectKey="$SONAR_PROJECT_KEY" -Dsonar.host.url="$SONAR_HOST_URL"
  rules:
    - if: $SKIP_SONAR == "true"
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  allow_failure: true

payments-api includes it, with two inputs:

You wrotemegacorp/payments/payments-api · .gitlab-ci.yml

include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file: pipelines/java-service.yml
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api

GitLab buildsmerged configurationcaptured from GitLab

sonar-scan:
  stage: test
  image: registry.example.com/megacorp/devops/ci-tools/sonar-scanner:6
  variables:
    SONAR_PROJECT_KEY: payments-api
    SONAR_QUALITYGATE_WAIT: true
  script:
  - sonar-scanner -Dsonar.projectKey="$SONAR_PROJECT_KEY" -Dsonar.host.url="$SONAR_HOST_URL"
  rules:
  - if: $SKIP_SONAR == "true"
    when: never
  - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  - if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
  allow_failure: true

The inputs are gone from the result. $[[ inputs.stage ]] became test, and $[[ inputs.project_key ]] became payments-api. quality_gate wasn't passed, so it took its default, true.

The version after @ decides what you get:

Version	Example	Resolves to
a commit SHA	`@e3262fdd…`	exactly that commit
a tag	`@2.1.0`	that tag; a tag beats a branch with the same name
a branch	`@main`	the branch's head when the pipeline is created
the latest release	`@~latest`	the newest version published to the CI/CD Catalog
a partial version	`@2` or `@2.1`	the newest matching version published to the CI/CD Catalog

Components work from any project, but the last two forms work only for components published to the CI/CD Catalog, GitLab's searchable list of shared components. Publishing needs a release, created with the release: keyword, on a tag that is a version number such as 2.1.0.

Inputs#

Inputs are declared in a header above a --- line. Each has a type: string (the default), number, boolean or array. It can also have a default, a description, a fixed list of options, or a regex the value must match. This is what happens to them:

Figure How an input reaches a job. GitLab checks every value against the header before it creates the pipeline.#

As text

The include passes the inputs it wants, such as project_key: payments-api
GitLab reads the shared file's spec: header
Does every required input have a value? No: No pipeline. GitLab names the missing input. Yes: the next step.
GitLab writes each value in place of $[[ inputs.name ]]
The result is ordinary YAML, merged like any other include

A value can be transformed on the way in, by adding one or more of four functions after the name: expand_vars, truncate(offset,length), posix_escape and split(separator). At most three can be used at once.

Components aren't the only files that can declare inputs. Any included file can, and the include passes values with inputs:.

	Inputs	Variables
Declared by	the shared file, in its `spec:` header	nobody: any file or setting can define them
Resolved	once, when the pipeline is created	when rules are evaluated, and again when the job runs
Can change during the pipeline	no: fixed for the whole run	yes: dotenv reports and scripts can set new values
Typed and checked	yes: type, options, regex	no
Who wins a clash	the value passed in, else the default	the precedence ladder in chapter 19

Two things to know when you trace an input:

The merged view shows the result, not the input. To see where payments-api came from, open the component's own file at the version the include names, and read the include's inputs:.
A missing required input stops the whole pipeline, before any job runs. Leave out project_key, and the local GitLab 19.3 says:

`gitlab.example.com/megacorp/devops/components/sonar-scan@2.1.0`: `project_key` input: required value has not been provided

Toggles#

A toggle is a variable that a rule checks. MegaCorp's Sonar component has one (SKIP_SONAR), and so does its container-scanning override:

megacorp/devops/ci-templatestemplates/security.ymlcontainer-scan

variables:
  CS_IMAGE: $IMAGE_REF

container_scanning:
  stage: scan
  needs: [image-build]
  rules:
    - if: $CONTAINER_SCANNING_DISABLED == "true"
      when: never
    - !reference [.rules, never-on-schedule]
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Setting CONTAINER_SCANNING_DISABLED to "true" in a project's variables makes the first rule match, with when: never, so the job never exists. GitLab's own scanner templates use the same pattern, with variables named after the scanner.

Gotcha

Rules compare text. A toggle tested with == "true" ignores 1, yes and TRUE. And a toggle only removes a job that the file itself defines. A scan that a security policy adds doesn't read your toggles (chapter 29).

GitLab's own templates at GitLab 19.3 name their toggles after the scanner, and accept either "true" or "1". payments-api's merged configuration has three: SAST_DISABLED, SECRET_DETECTION_DISABLED and DEPENDENCY_SCANNING_DISABLED. MegaCorp's container-scanning override replaced GitLab's rules with its own, so its CONTAINER_SCANNING_DISABLED accepts only "true".

Pipeline inputs Since 17.11#

A project's own .gitlab-ci.yml can declare inputs for the whole pipeline. They are filled in when someone runs the pipeline by hand, or passed by the API, a schedule, a Git push option, or a trigger from another pipeline. MegaCorp's deployer asks which environment to deploy to:

megacorp/platform/deployer.gitlab-ci.ymlinputs

spec:
  inputs:
    environment:
      type: string
      default: staging
      options: [staging, production]
---

A pipeline can take at most 20 inputs. Give each one a default, because pipelines that start automatically have nobody to fill in the form.

Pulling in filesSince 17.0#

include: component

Scope: Cross-project
Binding: Pinned or floating
Control: Opt-in
Phase: Compile
Merged view: Shown

Looks like

include:
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      project_key: payments-api

What GitLab does

GitLab fetches templates/sonar-scan.yml, or templates/sonar-scan/template.yml, from the component project at the version given. It writes in the inputs and merges the result like any other include.

How to trace it

Everything before the @ is the project path plus the component name. Open that project at the version given, and read the file under templates/. For ~latest or a partial version, the CI/CD Catalog shows which release it resolved to.

Gotchas

A component's job merges with any job of the same name in your pipeline.
@~latest fails with "content not found" if the component has never been published to the catalog.
Global keywords in a component, such as default:, affect every job in your pipeline.

ParametersSince 17.0#

spec: inputs

Scope: Cross-project
Binding: Inline
Control: Opt-in
Phase: Compile
Merged view: Shown

Looks like

spec:
  inputs:
    stage:
      default: test
    quality_gate:
      type: boolean
      default: true
---
sonar-scan:
  stage: $[[ inputs.stage ]]

What GitLab does

The header declares what the file accepts. When the pipeline is created, GitLab checks the values passed in against the declarations, and writes them in place of every $[[ inputs.… ]]. The result is ordinary configuration, fixed for the rest of the pipeline.

How to trace it

Read the header at the top of the included file for the defaults. Then read the inputs: under the include that pulled it in, for the values actually passed.

Gotchas

A file with inputs needs the --- line between its header and its jobs.
Inputs can't be used inside a !reference path.
An input that holds $CI_COMMIT_SHA inserts that text. The variable is expanded later, when the job runs.

Parameters#

Variables as parameters

Scope: One project
Binding: Inline
Control: Opt-in
Phase: Run
Merged view: Keyword only

Looks like

# in the template
image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}

# in the project
variables:
  JAVA_VERSION: "21"

What GitLab does

The template reads an ordinary CI/CD variable, and the consumer sets it. Nothing declares the variable or checks its value. GitLab resolves it when it evaluates rules and when the job runs, from whichever source ranks highest in variable precedence.

How to trace it

Find where the template reads the variable. Then find every place that sets it: the job, its extends parents, the top-level blocks, and the CI/CD settings of the project, group and instance.

Gotchas

A template that sets the same variable at job level overrides the project's top-level value.
A settings variable with the same name overrides both, and appears in no file.

Parameters#

Toggle variables

Scope: One project
Binding: Inline
Control: Opt-in
Phase: Create
Merged view: Keyword only

Looks like

rules:
  - if: $SKIP_SONAR == "true"
    when: never
  - if: $CI_PIPELINE_SOURCE == "merge_request_event"

What GitLab does

A rule tests the variable when the pipeline is created, and the first matching rule decides whether the job exists. Setting the variable in the project, a schedule or a manual run switches the job off or on without editing any file.

How to trace it

Search the templates' rules for the variable's name. Then check every place the variable can be set, including CI/CD settings and schedules, which no file records.

Gotchas

Rules compare text, so "true" and 1 are different values.
A toggle can't remove a job that a security policy adds.

ParametersSince 17.11#

Pipeline inputs

Scope: One project
Binding: Inline
Control: Opt-in
Phase: Create
Merged view: Keyword only

Looks like

spec:
  inputs:
    environment:
      default: staging
      options: [staging, production]
---
variables:
  ENVIRONMENT: $[[ inputs.environment ]]

What GitLab does

Inputs declared in the project's own configuration become the pipeline's parameters. They are offered on the New pipeline form, and accepted from the API, schedules, push options and trigger. As with any input, their values are fixed when the pipeline is created.

How to trace it

Open the top of the project's .gitlab-ci.yml for the declarations. The values a particular pipeline used come from whoever started it: a person, a schedule, an API call or an upstream pipeline.

Gotchas

An input without a default breaks every pipeline that starts automatically.
Pipeline variables sent by a trigger outrank YAML variables. In the deployer, an upstream job's ENVIRONMENT variable beats the value built from the input.

Spot the bugThe upgrade to ~latest#

To stop bumping versions by hand, the payments team changed their include to:

include:
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@~latest
    inputs:
      stage: test
      project_key: payments-api

Now no pipeline can be created. The error says the component's content was not found, yet @2.1.0 still works.

QuestionWhat does ~latest need that 2.1.0 doesn't?

Show the answer

~latest and partial versions such as @2 resolve only against versions published to the CI/CD Catalog. MegaCorp's components project has a 2.1.0 tag, but has never been published, so there is no "latest" to find. A tag works without publishing.

Either keep an explicit tag, or ask the component owners to publish releases to the catalog. See include: component.

Next10Pipelines as building blocks7 min

Pipelines as building blocks#

Part II · Reuse · Chapter 10· 7 min read

A trigger job is a job whose only work is to start another pipeline. The pipeline that starts it is called upstream, and the one it starts downstream. There are four ways it happens:

A child pipeline runs in the same project, on the same commit.
A multi-project pipeline runs in another project, with your permissions.
A dynamic child runs YAML that a job wrote moments earlier.
A script can do the same thing through the trigger API, with a token.

Two things decide what you see. The first is the trigger job's strategy: without one, it goes green as soon as the other pipeline exists, whatever happens next. The second is what crosses the boundary. Variables and inputs do; default: does not.

Four kinds#

Figure Four ways one pipeline starts another. Child and dynamic child pipelines run in the same project and on the same commit. A multi-project trigger, or a script calling the trigger API, starts a pipeline in another project, which runs that project's own configuration.#

Kind	Written as	Runs in	Its configuration comes from
Parent–child	`trigger: include: ci/docs.yml`	the same project, ref and commit	a file in the same repository
Dynamic child	`trigger: include: - artifact: … job: …`	the same project, ref and commit	YAML a job wrote during this pipeline
Multi-project	`trigger: project: …`	another project, on the branch named	that project's own configuration
Trigger API	`curl …/trigger/pipeline` in a script	whichever project the call names	that project's own configuration

Parent and child#

MegaCorp's monorepo builds its documentation in a child pipeline, and only when the docs change:

megacorp/data/mono.gitlab-ci.ymlstatic-child

docs:
  stage: build
  trigger:
    include: ci/docs.yml
  rules:
    - changes: [docs/**/*]

The child runs under the same project, ref and commit as its parent. It inherits the parent's top-level variables, unless the trigger job says inherit: variables: false. It does not inherit the parent's default: block, because defaults never cross into a downstream pipeline. Children can have children, but only two levels deep.

A child pipeline's jobs follow their own rules. In a merge request pipeline, the child still sees CI_PIPELINE_SOURCE as parent_pipeline, and jobs with no rules at all don't join it. On the local GitLab 19.3, mono's docs trigger failed in every merge request pipeline for exactly that reason (chapter 15).

Dynamic child pipelines#

The monorepo's services are built by a pipeline that doesn't exist until the pipeline runs. A job writes it, and saves the file as an artifact: a file a job keeps for later jobs to use.

megacorp/data/mono.gitlab-ci.ymlgenerate

generate-pipeline:
  stage: generate
  script:
    - ci/generate.sh > generated-pipeline.yml
  artifacts:
    paths: [generated-pipeline.yml]

megacorp/data/monoci/generate.sh

#!/usr/bin/env bash
# ci/generate.sh in megacorp/data/mono
# Writes one child-pipeline job per service that changed. Its output is the YAML
# that the service-pipelines trigger job runs. Used by chapter 10.
set -euo pipefail

base="${CI_MERGE_REQUEST_DIFF_BASE_SHA:-HEAD~1}"
changed=$(git diff --name-only "$base" HEAD -- services/ | cut -d/ -f2 | sort -u)

echo "stages: [build]"
for svc in $changed; do
  cat <<EOF
build-${svc}:
  stage: build
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  script:
    - make -C services/${svc} build
EOF
done

# a child pipeline needs at least one job, even when nothing changed
if [ -z "$changed" ]; then
  printf 'no-changes:\n  stage: build\n  script: [echo "no service changed"]\n'
fi

Then a trigger job runs whatever the generator wrote:

megacorp/data/mono.gitlab-ci.ymldynamic-child

service-pipelines:
  stage: build
  trigger:
    include:
      - artifact: generated-pipeline.yml
        job: generate-pipeline
    strategy: depend

There is no file to read in advance. The configuration that actually ran is the generated-pipeline.yml artifact of that one job, in that one pipeline. The generator script is the only thing you can read beforehand.

Multi-project pipelines#

Every MegaCorp service deploys by triggering the platform team's project:

megacorp/devops/ci-templatestemplates/deploy.ymldeploy-trigger

.deploy:
  stage: deploy
  trigger:
    project: megacorp/platform/deployer
    branch: main
    strategy: depend
  variables:
    APP: $CI_PROJECT_NAME
    IMAGE_REF: $IMAGE_REF

Pipeline payments-api on main: deploy-staging starts a pipeline in the deployer project (test and scan jobs left out)#

As text

build: maven-build
package: image-build
deploy: deploy-staging

Needs: image-build waits only for maven-build; deploy-staging waits only for image-build.

Downstream: deploy-staging triggers megacorp/platform/deployer (multi-project pipeline).

The downstream pipeline runs the deployer's own configuration, at the head of its main branch. The person who triggered the upstream pipeline must be allowed to start pipelines in the deployer project, or the trigger fails.

By default, the downstream pipeline receives the trigger job's variables, including the top-level variables it inherits. They arrive as trigger variables, and take precedence over the downstream project's own variables of the same name.

Some variables stay behind, unless the trigger job sets forward: pipeline_variables: true:

values typed into a manual run
variables set on a schedule
dotenv variables from the jobs the trigger job needs

Nothing forwarded is passed on again by the next trigger down, unless that trigger job also sets forward.

That list is about passing values on automatically. Naming a dotenv variable in the trigger job's own variables is different, and it works. On the local GitLab 19.3, a trigger job with IMAGE_REF: $IMAGE_REF passed the value from an earlier job's dotenv report to the downstream pipeline. It did so whether the trigger job listed that job in needs, or only came in a later stage. MegaCorp's .deploy relies on this.

What the trigger job's status means#

A green trigger job does not always mean the downstream pipeline passed. It depends on one key:

Figure What a trigger job's colour tells you, depending on its strategy#

As text

Does the trigger job set a strategy? No: It goes green once the downstream pipeline exists, whatever happens next. Yes: the next step.
Is the strategy mirror? Yes: It copies the downstream pipeline's status exactly. No: the next step.
The strategy is depend: green when the downstream succeeds, and "running" while it waits for a manual job

The trigger job has	It turns green when	So a green trigger job means
no `strategy`	the downstream pipeline has been created	only that the other pipeline started
`strategy: mirror`	the downstream pipeline succeeds; it copies its status throughout	the downstream pipeline succeeded
`strategy: depend`	the downstream pipeline finishes	an older setting; GitLab's documentation now recommends `mirror`, because this status doesn't always match

MegaCorp still uses depend everywhere, as many real template libraries do. The first row is the trap: a pipeline can be entirely green while the deployment it started failed.

The trigger API#

Before trigger jobs existed, pipelines started each other with a token and a script, and some still do:

megacorp/legacy/billing-batch.gitlab-ci.ymltrigger-api

notify-reports:
  <<: *defaults
  stage: deploy
  only: [tags]
  script:
    - >
      curl --fail --request POST
      --form "token=$REPORTS_TRIGGER_TOKEN"
      --form "ref=main"
      --form "variables[BILLING_VERSION]=$CI_COMMIT_TAG"
      "https://gitlab.example.com/api/v4/projects/4242/trigger/pipeline"

The started pipeline has CI_PIPELINE_SOURCE set to trigger. Variables sent with the call have the highest precedence of all. The token acts with the access of the user it belongs to.

Whole pipelines#

Parent–child pipelines

Scope: One project
Binding: Inline
Control: Opt-in
Phase: Run
Merged view: Keyword only

Looks like

docs:
  trigger:
    include: ci/docs.yml

What GitLab does

GitLab starts a child pipeline from the file, in the same project and on the same ref and commit. The child inherits the parent's top-level variables but not its default:. It appears as a card to the right of the parent's graph. Children nest at most two levels deep.

How to trace it

The trigger job names the child's file. Open the card to the right of the parent's graph to see the child's jobs, then read the file at the same commit.

Gotchas

Without strategy: mirror, the trigger job passes as soon as the child is created, even if every job in it then fails.
The parent's default:, such as its image or before_script, doesn't reach the child's jobs.
In a merge request pipeline, a child whose jobs have no rules gets no jobs, and the trigger job fails.

Whole pipelines#

Dynamic child pipelines

Scope: One project
Binding: Inline
Control: Opt-in
Phase: Run
Merged view: Not shown

Looks like

service-pipelines:
  trigger:
    include:
      - artifact: generated-pipeline.yml
        job: generate-pipeline

What GitLab does

One job writes pipeline configuration as an artifact, and a trigger job starts a child pipeline from it. The configuration exists only for that pipeline. An include inside the generated configuration can't use CI/CD variables.

How to trace it

Download the artifact from the generator job of the pipeline you are investigating: that YAML is exactly what ran. Then read the generator script, to understand why it wrote what it wrote.

Gotchas

No merged view ever shows the child's configuration, because it didn't exist before the pipeline ran.
The generator must always write at least one job. A generator that sometimes writes nothing breaks the pipeline only sometimes.

Whole pipelines#

Multi-project pipelines

Scope: Cross-project
Binding: Always live
Control: Opt-in
Phase: Run
Merged view: Not shown

Looks like

deploy-staging:
  trigger:
    project: megacorp/platform/deployer
    branch: main
    strategy: depend

What GitLab does

GitLab starts a pipeline in the other project, on the branch named, using that project's own configuration. The user who triggered the upstream pipeline must be allowed to run pipelines in the downstream project.

How to trace it

Open the card to the right of the graph to reach the downstream pipeline. Its jobs come from the downstream project's configuration at the head of the named branch, not from anything in your project.

Gotchas

The downstream configuration floats with that branch. The platform team can change your deployment without a commit in your project.
Someone without access to the downstream project can run your pipeline, but not its deployment.

Whole pipelines#

Pipeline trigger API

Scope: Cross-project
Binding: Always live
Control: Opt-in
Phase: Run
Merged view: Not shown

Looks like

curl --fail --request POST \
  --form "token=$REPORTS_TRIGGER_TOKEN" \
  --form "ref=main" \
  "https://gitlab.example.com/api/v4/projects/4242/trigger/pipeline"

What GitLab does

The API creates a pipeline in the project named by its number, on the ref given. Variables sent with the call outrank every other variable. The pipeline's CI_PIPELINE_SOURCE is trigger. The token acts with its owner's project access.

How to trace it

Search scripts for /trigger/pipeline. Look up the project number in the URL to find which project is being started, and find out who owns the token variable.

Gotchas

A number in a URL is the only sign of the relationship. No file names the target project.
Because the token acts with its owner's access, the call can break when that person's access changes, far from any pipeline file.

Spot the bugGreen pipeline, nothing deployed#

A team copied MegaCorp's deploy template into their own file, and trimmed it:

deploy-staging:
  stage: deploy
  trigger:
    project: megacorp/platform/deployer
    branch: main
  variables:
    APP: $CI_PROJECT_NAME
    IMAGE_REF: $IMAGE_REF

Their pipelines are green, but staging hasn't changed in a week. The deployer project shows a failed pipeline for every one of their merges.

QuestionHow can a pipeline be green when its deployment failed?

Show the answer

With no strategy, a trigger job succeeds as soon as the downstream pipeline is created. It doesn't wait for that pipeline, or reflect how it ends. The deployer's pipelines failed, but the trigger job had already passed.

Add strategy: mirror, so that the trigger job takes the downstream pipeline's status. See multi-project pipelines and what the trigger job's status means.

Next11Organisation-wide reuse: settings and policies11 min

Organisation-wide reuse: settings and policies#

Part II · Reuse · Chapter 11· 11 min read

Nine mechanisms reach a project without any of its files asking for them. Four are available on every tier (GitLab's paid plans are Free, Premium and Ultimate):

a setting that points the project at a pipeline file somewhere else
Auto DevOps, GitLab's own ready-made pipeline
CI/CD variables set in the settings of the instance, group and project
templates that are copied in when a file or project is created

Four more come with GitLab Ultimate. They are security policies, rules a security team sets once for many projects: pipeline execution policies, scan execution policies and merge request approval policies. The fourth is compliance pipelines, which those policies replace. A ninth, the instance template repository, is a copy-once template list on self-managed Premium.

None of them shows up in the project's files. You find them in settings pages, and in the pipeline itself.

Where each one lives#

Two words first. The instance is the whole GitLab installation. Self-managed means your organisation runs that installation itself, rather than using GitLab.com.

Mechanism	Who sets it	Where you look	Tier
Custom CI/CD configuration file	a project maintainer	the project's CI/CD settings, under General pipelines	all
Auto DevOps	an administrator or group owner, or the project	the Auto DevOps section of the CI/CD settings	all
Instance, group and project variables	administrators, group owners, project maintainers	the CI/CD variables settings at each level	all
Project and file templates	whoever created the project or file	nowhere afterwards: the copy is the only trace	all; templates kept in a group need Premium
Instance template repository	an administrator	Admin › Settings › Templates	Premium, self-managed and Dedicated
Pipeline execution policies	the security team	the group's security policies	Ultimate
Scan execution policies	the security team	the group's security policies	Ultimate
Merge request approval policies	the security team	the group's security policies, and the merge request itself	Ultimate
Compliance pipelines	compliance owners	the project's compliance framework label	Ultimate, deprecated

In the UI, settings variables are under Settings › CI/CD › Variables: in the project, in each group, and in the Admin area for the instance. Security policies are under Secure › Policies, in a project or a group.

The most common puzzle these mechanisms cause is a project that has pipelines but no pipeline file. Check these places, in this order:

Figure A project has pipelines but no .gitlab-ci.yml. Each question points to a mechanism in this chapter.#

As text

Do the project's CI/CD settings name a configuration file? Yes: A custom configuration file, perhaps in another project. No: the next step.
Is Auto DevOps on for the project, its group or the instance? Yes: GitLab's own Auto DevOps pipeline. No: the next step.
Does a scan execution policy apply to the project? Yes: The policy creates a scan pipeline itself. No: the next step.
Ask the group's owners about compliance frameworks and other policies

A pipeline file that isn't in the repository#

web-portal's repository has no .gitlab-ci.yml, yet it has pipelines. Its settings point at a file in the templates project:

megacorp/web/web-portalSETTINGS.ymlconfig-path

ci_cd_configuration_file: pipelines/web.yml@megacorp/devops/ci-templates

The setting accepts three forms:

a path in the same repository
a file in another project, written as path@namespace/project, optionally with :ref at the end
a URL ending in .yml

The project's own pipeline editor can't edit a file that lives somewhere else. That is often the first sign this setting is in play. On the local GitLab 19.3, web-portal's pipelines ran the three jobs defined in ci-templates' pipelines/web.yml.

Organisation-wide#

Custom CI/CD configuration file

Scope: Cross-project
Binding: Pinned or floating
Control: Enforced
Phase: Compile
Merged view: Not shown

Looks like

pipelines/web.yml@megacorp/devops/ci-templates
pipelines/web.yml@megacorp/devops/ci-templates:v4.2.0

What GitLab does

GitLab reads the pipeline configuration from the path the setting names, instead of .gitlab-ci.yml at the root of the repository. The file can be in the same project, in another project at an optional ref, or at a URL.

How to trace it

Open the project's CI/CD settings and expand General pipelines. If the file is elsewhere, read it in that project at the ref given. The project's own pipeline editor won't show it.

Gotchas

Without a ref, GitLab reads the file from the other project's default branch. Every change there reaches every consuming project at once.
A repository with no pipeline file isn't necessarily a project without a pipeline.

Variables from settings#

CI/CD variables can be set on the instance, on each group, and on the project. Every job in every project below them receives them, and they outrank anything in YAML:

settingsinstance.ymlinstance-variables

# Every project on the instance receives these, unless a group or project
# variable with the same name takes precedence.
instance_variables:
  - key: MEGACORP_REGISTRY
    value: registry.example.com
  - key: HTTPS_PROXY
    value: http://proxy.example.com:3128

settingsgroup-megacorp.ymlgroup-variables

# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
  - key: MAVEN_CLI_OPTS
    value: "--batch-mode --errors --show-version -s .m2/settings.xml"
  - key: AWS_ACCOUNT_ID
    value: "123456789012"
  - key: SONAR_HOST_URL
    value: https://sonar.example.com
  - key: SONAR_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true

Among themselves, the project beats its groups, the closest group beats the ones above it, and groups beat the instance. Only policy variables, variables typed into a manual job, and pipeline variables rank higher. Chapter 19 gives the whole ladder.

Organisation-wide#

Instance and group CI/CD variables

Scope: Organisation-wide
Binding: Always live
Control: Inherited
Phase: Create
Merged view: Not shown

Looks like

Group megacorp › CI/CD variables
  MAVEN_CLI_OPTS = --batch-mode --errors --show-version -s .m2/settings.xml

What GitLab does

Every pipeline in every project under the group, or on the instance, receives the variable. It outranks variables written anywhere in the configuration, including job variables. It can also be protected, masked or hidden.

How to trace it

A job's log never lists its variables. Check the CI/CD variables settings of the project, then each parent group, then the instance. Where you lack access to a level, ask someone who has it.

Gotchas

Setting the same variable in YAML does nothing: the settings value wins silently.
A protected variable exists only in pipelines on protected branches and tags.

Auto DevOps#

Auto DevOps is GitLab's own complete pipeline. It runs for a project only if the project has no .gitlab-ci.yml, and only if it finds a Dockerfile, or a buildpack that matches the code. A project's own configuration always wins. On a self-managed instance, the setting for all projects starts switched on: the local GitLab 19.3 had it on before any setting was changed. MegaCorp left it on:

settingsinstance.ymlauto-devops

# A project with no CI/CD configuration file gets GitLab's Auto DevOps pipeline.
auto_devops:
  default_to_auto_devops: true

Organisation-wide#

Auto DevOps

Scope: Organisation-wide
Binding: Always live
Control: Inherited
Phase: Compile
Merged view: Not shown

Looks like

Settings › CI/CD › Auto DevOps
  [x] Default to Auto DevOps pipeline

What GitLab does

When a project has no CI/CD configuration and Auto DevOps applies to it, GitLab runs its built-in Auto DevOps pipeline, which builds, tests, scans and can deploy. The setting can come from the project, a group or the instance.

How to trace it

If a project without a pipeline file has pipelines, check the project's Auto DevOps setting, then its groups'. Then check the custom configuration file setting, which is the other explanation.

Gotchas

Adding any .gitlab-ci.yml, even a small one, switches Auto DevOps off for that project.
Its jobs change with GitLab upgrades, like any GitLab template.

Its jobs are easy to recognise. They have names such as build, test and code_quality, scanner jobs such as container_scanning and secret_detection, and deploy jobs such as review, staging and production.

Templates copied at creation#

Two mechanisms hand a project configuration once, when a file or a project is created, and never again:

Project templates. A new project is created as a copy of a template project, including its .gitlab-ci.yml. GitLab ships some on every tier; keeping your own in a group, as MegaCorp does, needs Premium Premium.
The instance template repository Premium Self-managed Dedicated. An administrator chooses a project whose gitlab-ci folder fills the template list in the Web Editor, GitLab's in-browser file editor. Picking one copies it into the new file. Despite the name, these templates can't be used with include: template.

settingsgroup-megacorp.ymlproject-templates

# New projects can be created from these. The copy happens once, at creation.
group_project_templates:
  source_group: megacorp/templates
  templates:
    - megacorp/templates/java-service-starter
    - megacorp/templates/batch-job-starter

settingsinstance.ymltemplate-repo

# Self-managed only: the .yml files in this project's gitlab-ci/ folder appear in the
# Web Editor's template list when anyone creates a CI/CD file. Choosing one copies it
# into the new file; include: template cannot use them.
instance_template_repository:
  project: megacorp/devops/instance-templates
  offers:
    - gitlab-ci/MegaCorp-Legacy-Java.yml

Organisation-wide#

Project and file templates

Scope: Organisation-wide
Binding: Copied once
Control: Opt-in
Phase: Compile
Merged view: Shown

Looks like

New project › Create from template › Group › megacorp/templates/java-service-starter

What GitLab does

GitLab copies the template project, including its pipeline file, into the new project when it is created. After that, the copy is an ordinary file in the new project, with no link back to the template.

How to trace it

Compare the project's first commit with the template project's history. A .gitlab-ci.yml that arrived fully formed in the first commit came from somewhere.

Gotchas

Fixes to the template never reach projects created from it. Copies drift apart silently.

Organisation-widePremiumSelf-managed#

Instance template repository

Scope: Organisation-wide
Binding: Copied once
Control: Opt-in
Phase: Compile
Merged view: Shown

Looks like

Admin › Settings › Templates
  Templates project: megacorp/devops/instance-templates
    gitlab-ci/MegaCorp-Legacy-Java.yml

What GitLab does

The .yml files in the chosen project's gitlab-ci folder appear in the template list whenever someone creates a CI/CD file in the Web Editor. Choosing one copies its content into the new file.

How to trace it

If a project's pipeline file looks like one of the organisation's templates, compare it with the file in the administrators' templates project. Only administrators can see which project that is.

Gotchas

It is a copy, not an include: the template can change without the project changing.
include: template reaches only GitLab's own templates, never these.

Security policies Ultimate#

On GitLab Ultimate, a security team keeps policies in a policy project linked to a group, and they apply to every project under it. MegaCorp's policy project defines one of each kind:

megacorp/security/policies.gitlab/security-policies/policy.ymlpipeline-execution

pipeline_execution_policy:
  - name: MegaCorp guardrails
    description: Add an SBOM job to every pipeline.
    enabled: true
    pipeline_config_strategy: inject_policy
    content:
      include:
        - project: megacorp/security/policies
          file: pipeline-policies/megacorp-guardrails.yml

megacorp/security/policiespipeline-policies/megacorp-guardrails.ymlguardrails

policy-sbom:
  stage: .pipeline-policy-post
  image: registry.example.com/megacorp/devops/ci-tools/syft:1
  script:
    - syft dir:. -o cyclonedx-json=sbom.cdx.json
  artifacts:
    reports:
      cyclonedx: sbom.cdx.json

A pipeline execution policy puts its own CI/CD configuration into every pipeline. With inject_policy, its jobs are added to the project's own jobs. With override_project_ci, the policy's configuration replaces the project's entirely. Two reserved stages frame the pipeline:

.pipeline-policy-pre runs before everything else. Every other job waits for it, and if it fails, the rest is skipped.
.pipeline-policy-post runs after everything else.

megacorp/security/policies.gitlab/security-policies/policy.ymlscan-execution

scan_execution_policy:
  - name: Secret detection everywhere
    description: Run secret detection in every pipeline on every branch.
    enabled: true
    rules:
      - type: pipeline
        branches: ["*"]
    actions:
      - scan: secret_detection
  - name: Nightly dependency scan
    description: Scan dependencies on the default branch every night.
    enabled: true
    rules:
      - type: schedule
        branches: [main]
        cadence: "0 2 * * *"
    actions:
      - scan: dependency_scanning

A scan execution policy adds GitLab's scanners to pipelines, or runs them on a schedule. Its jobs get a hyphen and a number added to their names, such as secret-detection-1, so that they can't clash with the project's own scanner jobs. Scheduled scans run as a bot user, not as a member of your team.

Pipeline payments-api on main, on GitLab Ultimate: jobs from both policies appear beside the project's own (described, not captured; other jobs left out)#

As text

build: maven-build
test: maven-test: [17], maven-test: [21], secret-detection-1 (policy)
.pipeline-policy-post: policy-sbom (policy)

megacorp/security/policies.gitlab/security-policies/policy.ymlapproval

approval_policy:
  - name: Block new critical vulnerabilities
    description: A merge into main that adds a critical finding needs security's approval.
    enabled: true
    rules:
      - type: scan_finding
        branches: [main]
        scanners: [sast, secret_detection, dependency_scanning, container_scanning]
        vulnerabilities_allowed: 0
        severity_levels: [critical]
        vulnerability_states: [new_needs_triage]
    actions:
      - type: require_approval
        approvals_required: 1
        group_approvers: [megacorp/security]

A merge request approval policy adds no jobs. It reads the scanners' results, and when its rule matches, the merge request needs the approvals the policy names. If a required scanner produced no report, the policy by default requires the approval anyway: it fails closed.

LegacyCompliance pipelines

Before pipeline execution policies, a compliance framework could name a pipeline file, written path@group/project, for every project carrying its label. A compliance framework is a label an administrator attaches to a project, to mark the rules it must follow. By default the framework's file ran instead of the project's own configuration, unless it included the project's .gitlab-ci.yml explicitly. Compliance pipelines were deprecated in 17.3 and are planned for removal in 20.0. Use pipeline execution policies instead.

settingsgroup-megacorp.ymlcompliance

# Legacy: a compliance framework with its own pipeline configuration, the
# predecessor of pipeline execution policies.
compliance_frameworks:
  - name: SOX
    pipeline_configuration: compliance/sox.yml@megacorp/security/policies
    applied_to:
      - megacorp/payments/payments-api

Organisation-wideUltimateSince 17.3#

Pipeline execution policies

Scope: Organisation-wide
Binding: Always live
Control: Enforced
Phase: Create
Merged view: Not shown

Looks like

pipeline_execution_policy:
  - name: MegaCorp guardrails
    pipeline_config_strategy: inject_policy
    content:
      include:
        - project: megacorp/security/policies
          file: pipeline-policies/megacorp-guardrails.yml

What GitLab does

When a pipeline is created, GitLab adds the policy's configuration to it, or, with override_project_ci, uses the policy's configuration instead of the project's. Policy jobs can run in the reserved stages .pipeline-policy-pre and .pipeline-policy-post. Policy variables outrank the project's and the groups'. A policy project holds at most five of these policies. Generally available in 17.3; inject_policy from 17.9.

How to trace it

A job in no project file, or in a .pipeline-policy-* stage, points here. Open the group's security policies, find the policy's content, and read that file in the policy project.

Gotchas

If a policy job has the same name as a project job, the policy job gets :policy-<project id>-<index> added to its name by default.
A failure in .pipeline-policy-pre skips every other job in the pipeline.
With override_project_ci, editing your .gitlab-ci.yml may change nothing at all.

Organisation-wideUltimate#

Scan execution policies

Scope: Organisation-wide
Binding: Always live
Control: Enforced
Phase: Create
Merged view: Not shown

Looks like

scan_execution_policy:
  - name: Secret detection everywhere
    rules:
      - type: pipeline
        branches: ["*"]
    actions:
      - scan: secret_detection

What GitLab does

GitLab adds the named scanners to matching pipelines, or runs them on a schedule. Added jobs are renamed with a hyphen and a number. They run in the test stage, or in scan-policies when there is no test stage; DAST runs in dast. For projects without a pipeline file, the policy creates the configuration itself.

How to trace it

A scanner job with a numbered name, or a scan in a project that includes no scanner template, points here. Open the group's security policies.

Gotchas

Toggles such as SAST_DISABLED skip only the project's own scanner jobs. They can't switch off a scan the policy enforces.
Scheduled scans ignore [skip ci], and run as the security_policy_bot user.

Organisation-wideUltimate#

Merge request approval policies

Scope: Organisation-wide
Binding: Always live
Control: Enforced
Phase: Run
Merged view: Not shown

Looks like

approval_policy:
  - name: Block new critical vulnerabilities
    rules:
      - type: scan_finding
        severity_levels: [critical]
        vulnerabilities_allowed: 0
    actions:
      - type: require_approval
        approvals_required: 1

What GitLab does

GitLab checks the rule against the scanners' results for the merge request. When it matches, the merge request needs the approvals the policy names. Rules can also look at licence findings, or apply to every merge request. By default, a missing scanner report makes the policy require approval: it fails closed.

How to trace it

A merge request blocked "because of a policy", with every pipeline job green, points here. The merge request page names the policy. The policy file in the policy project shows what it checks.

Gotchas

Formerly called scan result policies, under the key scan_result_policy. From 17.0, only approval_policy is accepted.
Removing a scanner job to get past a failing scan can make the merge request need approval, because the report is now missing.

Organisation-wideUltimateDeprecated#

Compliance pipelines

Scope: Organisation-wide
Binding: Pinned or floating
Control: Enforced
Phase: Compile
Merged view: Not shown

Looks like

Compliance framework: SOX
  Compliance pipeline configuration: compliance/sox.yml@megacorp/security/policies

What GitLab does

For every project with the framework's label, GitLab runs the compliance pipeline configuration instead of the project's own, unless that configuration includes the project's file. Deprecated in 17.3, planned for removal in 20.0, and replaced by pipeline execution policies.

How to trace it

Check the project for a compliance framework label, then read the file its framework names.

Gotchas

A project whose own .gitlab-ci.yml seems to be ignored may have a compliance pipeline that doesn't include it.

Spot the bugThe scan that won't switch off#

A payments engineer, fed up with a slow secret-detection job, added a toggle to the project's CI/CD variables:

SECRET_DETECTION_DISABLED = true

The project's own secret-detection job disappeared. A job named secret-detection-1 still runs in every pipeline.

QuestionWhere does secret-detection-1 come from, and why does the toggle not affect it?

Show the answer

The numbered name is the mark of a scan execution policy. MegaCorp's security team enforces secret detection on every branch. SECRET_DETECTION_DISABLED, which GitLab's template accepts as "true" or "1", skips only the scanner job the project defines itself. A scan that a policy enforces ignores it.

Whether the scan should run is the security team's decision, not the project's. Raise the slowness with them; don't look for a way round it.

Next12Reuse outside the YAML7 min

Reuse outside the YAML#

Part II · Reuse · Chapter 12· 7 min read

YAML says what to run. In a large organisation, what actually happens is often decided somewhere else:

by the job's image
by a shell library, a company command-line tool, or a downloaded script
by a build tool's shared configuration
by the runner the job landed on

None of these shows in the merged configuration. A script line that says mc deploy tells you nothing about what mc does. The job log is where you start, and the question to ask of every command is "where did this come from?"

Take any command from a job log, and ask these questions in order:

Figure Where a command in a job log comes from. Each answer is one section of this chapter.#

As text

Did the log download it first, with curl or git clone? Yes: A fetched script: read what that URL served on that day. No: the next step.
Is it a build tool, such as mvn, npm or gradle? Yes: Shared build configuration, such as a parent POM. No: the next step.
Is it loaded by a source line in before_script? Yes: A shell library, usually shipped in the image. No: the next step.
Otherwise it is a program inside the image: read the image's Dockerfile
Behaves differently on different runners? Compare the runners' configuration

Toolbox images#

A toolbox image is an image a central team builds with every tool the organisation's jobs need. Every MegaCorp job runs in the ci-tools image unless it names another, because the base template sets it as the default:

megacorp/devops/ci-templatestemplates/base.ymldefault

default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  tags: [megacorp-shared]
  interruptible: true
  retry:
    max: 2
    when: [runner_system_failure, stuck_or_timeout_failure]
  before_script:
    - !reference [.snippets, functions]
    - mc_log "job $CI_JOB_NAME on $CI_COMMIT_REF_NAME"

The image carries more than tools. It carries MegaCorp's own logic, copied in by its Dockerfile, the recipe that builds the image:

megacorp/devops/ci-toolsDockerfilelibrary

COPY lib/ci-lib.sh /opt/megacorp/lib/ci-lib.sh
COPY bin/mc /usr/local/bin/mc
RUN chmod +x /usr/local/bin/mc

Outside the YAML#

Toolbox images

Scope: Organisation-wide
Binding: Pinned or floating
Control: Opt-in
Phase: Run
Merged view: Keyword only

Looks like

default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2

What GitLab does

The runner starts the job in a container from the image. Every command the script calls, and every file it reads that isn't in your repository, comes from the image's contents at the moment it was pulled.

How to trace it

The first lines of the job log name the image that was used. Find the project that builds it, here megacorp/devops/ci-tools, and read its Dockerfile for what gets copied in. Read the copied files at the version the image was built from.

Gotchas

An image tag like 3.2 can be pushed again with new contents. Only an image digest, a fixed ID for its exact contents such as @sha256:…, is truly pinned.
Upgrading the image can change what every job in the organisation does, with no commit in any pipeline file.

Shell function libraries#

A shell function is a named group of commands, like a small script inside the shell. Central teams package functions in a library file rather than repeating shell code in YAML. A source command loads the file, and MegaCorp's snippet library does nothing but that:

megacorp/devops/ci-templatestemplates/snippets.ymlsnippets

.snippets:
  functions:
    - source /opt/megacorp/lib/ci-lib.sh
  aws_login:
    - mc_aws_login "$AWS_ROLE_ARN"
  maven_settings:
    - mkdir -p .m2
    - mc_maven_settings > .m2/settings.xml

The functions themselves live in the image:

megacorp/devops/ci-toolslib/ci-lib.shaws-login

# Exchange the job's OIDC ID token for temporary AWS credentials.
mc_aws_login() {
  local role="${1:?role ARN required}"
  local creds
  creds=$(aws sts assume-role-with-web-identity \
    --role-arn "$role" \
    --role-session-name "gitlab-${CI_PROJECT_ID}-${CI_JOB_ID}" \
    --web-identity-token "$MC_ID_TOKEN" \
    --duration-seconds 3600 \
    --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
    --output text) || { mc_log "AssumeRoleWithWebIdentity failed for $role"; return 1; }
  read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< "$creds"
  export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
  mc_log "assumed $role"
}

Outside the YAML#

Shell function libraries

Scope: Cross-project
Binding: Pinned or floating
Control: Opt-in
Phase: Run
Merged view: Keyword only

Looks like

before_script:
  - !reference [.snippets, functions]   # source /opt/megacorp/lib/ci-lib.sh
script:
  - mc_aws_login "$AWS_ROLE_ARN"

What GitLab does

A source line loads shell functions into the job's shell. They are then available to everything that runs later in the same shell: before_script and script share one. The library usually ships in the image, or arrives through a snippet or a download.

How to trace it

A command in the log that isn't a real program is a function. Search the snippets for the source line that loads it, then read the library file in the project that builds the image.

Gotchas

A job that redefines before_script loses the source line, and fails with "command not found" at the first function call.
Functions are written for one shell. A library written for bash fails in an image that only has sh.

Company command-line tools#

Some organisations wrap their deployment steps in a command of their own. MegaCorp's is called mc:

megacorp/devops/ci-toolsbin/mcdeploy

deploy)
  env="${1:?usage: mc deploy <environment> <image-ref>}"
  image="${2:?usage: mc deploy <environment> <image-ref>}"
  mc_log "promoting $image to $env through the GitOps repository"
  git clone --depth 1 \
    "https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
  yq -i ".apps.\"${APP}\".image = \"${image}\"" "gitops/envs/${env}/values.yaml"
  git -C gitops commit -am "deploy ${APP} ${image} to ${env}"
  git -C gitops push origin HEAD:main
  ;;

The deployer's whole job is mc deploy "$ENVIRONMENT" "$IMAGE_REF". That one line clones a GitOps repository with a token, edits a values file and pushes a commit. A GitOps repository holds the desired state of the running systems, and a tool in the cluster applies whatever it says (chapter 34). Nothing in any pipeline file says any of this.

Outside the YAML#

Internal command-line tools

Scope: Organisation-wide
Binding: Pinned or floating
Control: Opt-in
Phase: Run
Merged view: Keyword only

Looks like

script:
  - mc deploy "$ENVIRONMENT" "$IMAGE_REF"

What GitLab does

The script calls a company tool, installed in the image or downloaded by an earlier command. All the behaviour is in the tool's own code, at whatever version the image contains.

How to trace it

Find how the tool got into the image, here COPY bin/mc in the Dockerfile, and read its source. Many such tools print their version. Add that command to a job to see which version you are running.

Gotchas

A tool that reads variables such as APP or GITOPS_TOKEN depends on names that no pipeline file documents.

Scripts fetched at run time#

The oldest trick of all is to download a script and run it, in one line:

megacorp/legacy/billing-batch.gitlab-ci.ymlfetched

test:
  <<: *defaults
  stage: test
  script:
    - curl -sSL https://gitlab.example.com/megacorp/devops/scripts/-/raw/main/run-tests.sh | bash

Outside the YAML#

Scripts fetched at run time

Scope: Cross-project
Binding: Pinned or floating
Control: Opt-in
Phase: Run
Merged view: Keyword only

Looks like

curl -sSL https://gitlab.example.com/megacorp/devops/scripts/-/raw/main/run-tests.sh | bash

What GitLab does

The job downloads a script from a URL or a repository while it runs, and executes it. What runs is whatever the URL served at that moment.

How to trace it

Open the URL from the job log. For a branch URL like …/raw/main/…, check the file's history, to see what it said when the job ran.

Gotchas

A URL on a branch is the most floating kind of reuse there is: unpinned, invisible to every GitLab view, and able to change mid-afternoon.
If the host is down, the job fails, even though nothing about your project changed.

Build-tool reuse#

Build tools share configuration too. A Maven project can inherit its build settings from a parent POM: a shared pom.xml published by another team. Java services at MegaCorp do:

megacorp/payments/payments-apipom.xmlparent

<parent>
  <groupId>com.example.megacorp</groupId>
  <artifactId>megacorp-parent</artifactId>
  <version>12.3.0</version>
</parent>

Outside the YAML#

Build-tool reuse

Scope: Cross-project
Binding: Pinned or floating
Control: Opt-in
Phase: Run
Merged view: Not shown

Looks like

script:
  - mvn $MAVEN_CLI_OPTS verify

What GitLab does

The pipeline calls the build tool. The build tool reads configuration that is shared through its own mechanism: a Maven parent POM, a Gradle convention plugin, a shared npm configuration package, or a Makefile included from elsewhere. Plugins, profiles and checks can all arrive that way.

How to trace it

Look in the build file for what it inherits: <parent> in a POM, or the equivalent in your build tool. Then read that shared configuration at the version named.

Gotchas

A new check in the parent POM can fail every build that uses it. The pipeline file won't have changed.

Runner configuration#

The runner itself can change every job it runs, through a file called config.toml that projects can't see:

runnersconfig.tomlrunner

[[runners]]
  name = "megacorp-eks-shared"
  # registered in GitLab as a group runner of megacorp, with the tag megacorp-shared
  url = "https://gitlab.example.com"
  executor = "kubernetes"
  environment = ["MAVEN_OPTS=-Xmx2g", "HTTPS_PROXY=http://proxy.example.com:3128"]
  pre_build_script = "echo 'runner: megacorp-eks-shared'"
  [runners.kubernetes]
    namespace = "gitlab-runners"
    image = "registry.example.com/megacorp/devops/ci-tools:3.2"
    # jobs run as this service account, which is bound to an AWS IAM role
    service_account = "gitlab-runner-jobs"

Outside the YAML#

Runner configuration

Scope: Organisation-wide
Binding: Always live
Control: Enforced
Phase: Run
Merged view: Not shown

Looks like

[[runners]]
  environment = ["MAVEN_OPTS=-Xmx2g"]
  pre_build_script = "echo 'runner: megacorp-eks-shared'"

What GitLab does

A runner's config.toml can add or overwrite environment variables for every job. It can run commands before fetching the source, before the job and after it. It chooses the executor and shell, can restrict which images jobs may use, and caps the log size, 4 MB by default. On Kubernetes, the pod's service account can carry cloud permissions.

How to trace it

The job log names the runner that ran the job. If a job behaves differently depending on which runner took it, compare the runners' configuration. Only the runners' administrators can read it, so ask them.

Gotchas

A job log longer than the runner's limit is cut off, so the end of a failing job may be missing.
Moving a job to a different runner, by changing its tags, can change its environment, its permissions and its image policy all at once.

GitLab Functions Experimental#

GitLab's experiment in reusable steps, formerly called CI/CD Steps, replaces a job's script with a list of steps. Each step either calls a published function or runs a script:

megacorp/data/monoci/docs.ymlsteps

lint-docs:
  image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
  run:
    - name: markdown_lint
      func: registry.example.com/megacorp/devops/functions/markdownlint:1.2.0
      inputs:
        path: docs
    - name: report
      script: echo "markdown lint finished"

Outside the YAMLExperimental#

GitLab Functions

Scope: Cross-project
Binding: Pinned or floating
Control: Opt-in
Phase: Run
Merged view: Keyword only

Looks like

lint-docs:
  run:
    - name: markdown_lint
      func: registry.example.com/megacorp/devops/functions/markdownlint:1.2.0
      inputs:
        path: docs

What GitLab does

The job runs its run: steps in order. A func: step loads a function and passes it inputs. The function can come from an OCI registry (a server that stores images and similar packages), a local path or, deprecated, a Git repository. A job that uses functions can't also have before_script, script or after_script.

How to trace it

Follow the function's reference to its registry or repository, and read it at the version given.

Gotchas

It is an experiment. Its keywords have changed before, from step: to func:, and may change again.
Loading functions from OCI registries needs a recent runner. An older runner fails the job, not the pipeline.

Spot the bugThe command no repository defines#

The deployer's job log shows it running mc deploy "staging" "123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:4f2c…". It then pushes a commit to a repository called gitops-config. A developer searches every repository they can see for a definition of mc, and finds nothing under that name.

QuestionWhere does mc come from, and how would you read exactly what mc deploy did in that job?

Show the answer

The deploy job runs in the ci-tools image, and its log names the image. That image's Dockerfile, in megacorp/devops/ci-tools, copies bin/mc into /usr/local/bin/mc. The command is a script in that repository.

Read bin/mc at the version the image was built from, not at the head of the branch. The deploy case shows the clone, the yq edit and the push. See toolbox images and company command-line tools.

Next13Tracing a job to its source4 min

Tracing a job to its source#

Part II · Reuse · Chapter 13· 4 min read

You now know all 37 mechanisms. This chapter puts them in order, as a procedure. Start from a job in the pipeline graph, and ask one question at a time:

Is it in the merged configuration? If yes, walk its extends chain and its !reference tags back to the files.
If it isn't, it was made when the pipeline was created (by a policy, a matrix or a downstream pipeline), or when it ran.
Either way, finish in the job log. That is where you find the image, the runner, and the commands whose logic lives outside YAML.

Here is the whole procedure on one page. The steps below take each branch in turn:

Figure Tracing any job to where it came from. Whichever branch you take, finish in the job log (step 5).#

As text

Read the job's name and stage in the pipeline graph (step 1)
Is it in Full configuration, under its name without any [ ] suffix? Yes: Walk its extends chain, !reference tags and includes (step 2). No: the next step.
Is it in a .pipeline-policy stage, or numbered like secret-detection-1? Yes: A security policy added it (step 4). No: the next step.
Is it in a card to the right of the graph? Yes: A downstream pipeline: find its trigger job (step 3). No: the next step.
None of these: check the project's settings for a config file or Auto DevOps (steps 3 and 4)

Step 1: read the name#

The job's name and stage in the pipeline graph often tell you which branch of the procedure you are on, before you open anything:

The name or stage looks like	It is probably	Go to
an ordinary name	a job in the configuration	step 2
a name ending in `[…]`, such as `maven-test: [17]`	one job of a `parallel: matrix`	step 2, with the name before the colon
a stage called `.pipeline-policy-pre` or `.pipeline-policy-post`	a pipeline execution policy job	step 4
a scanner name with a hyphen and a number, such as `secret-detection-1`	a scan execution policy job	step 4
a name ending in `:policy-` and two numbers	a policy job renamed to avoid a clash	step 4
a card to the right of the graph	a downstream pipeline	step 3

Step 2: find it in the merged configuration#

Open Build › Pipeline editor, then Full configuration, and search for the job name followed by a colon. If it is there, everything the file-based mechanisms contributed has already been merged in. What remains is to find which file each part came from:

Follow extends upward. Search the included files for each parent's name, such as .maven-test:, then its parent's, until there is none.
Follow every !reference. Its first item names a hidden job, which you search for in the same way.
Look for default: and top-level variables: in every included file, and check the job for inherit:.
Note the ref of every include you passed through. A branch, or no ref at all, means the answer can change without a commit of yours.

Here is that walk for payments-api's maven-test. It starts from the only line that mentions the job by name:

megacorp/devops/ci-templatespipelines/java-service.ymljobs

maven-build:
  extends: .maven-build

maven-test:
  extends: .maven-test

image-build:
  extends: .image-build
  needs: [maven-build]

Hop	File	What it contributes
1	payments-api `.gitlab-ci.yml`	a `MAVEN_CLI_OPTS` variable, and the include of the golden pipeline at `v4.2.0`
2	ci-templates `pipelines/java-service.yml`	the job itself: `maven-test: extends: .maven-test`
3	ci-templates `templates/java-maven.yml`	`.maven-test`: stage, image, matrix, script, artifacts, rules
4	ci-templates `templates/java-maven.yml`	`.maven-base`: cache, variables, and a `before_script` built from `!reference`
5	ci-templates `templates/snippets.yml`	the snippets that `before_script` pastes in
6	ci-templates `templates/base.yml`	`.megacorp-base`: `MC_TEAM`, artifact expiry, and the rules that `.maven-test` replaced

If the project's configuration file lives in another project, its own pipeline editor can't show it. Open the file in that project, at the ref the setting names. GitLab can still assemble the whole configuration for you. On the local GitLab 19.3, the CI Lint API for web-portal, GET /api/v4/projects/:id/ci/lint?dry_run=true, returned web.yml's jobs with every include from ci-templates already merged in.

Step 3: if it's in no file#

A job that isn't in the merged configuration was added when the pipeline was created, or belongs to another pipeline:

Matrix jobs don't exist in the configuration under their bracketed names. Search for the name without the brackets.
Policy jobs are added when the pipeline is created. Go to step 4.
Jobs in a downstream card belong to another pipeline: a child from a file or a generated artifact, or another project's own pipeline. Open the card. Then find the trigger job in the upstream pipeline, which names the file, the artifact or the project.
A project without a .gitlab-ci.yml takes its whole configuration from a custom configuration file setting, from Auto DevOps, or from a scan execution policy that creates one (chapter 11).

Step 4: settings and policies#

These places hold configuration that no file in the project shows:

the project's CI/CD settings: its configuration file path, Auto DevOps, and variables
the CI/CD variables of every parent group, and of the instance
the group's security policies, on GitLab Ultimate
the project's compliance framework label, if it has one

If you can't open a level yourself, the question for its owner is specific: "Which variables, policies or settings at your level apply to this project?"

Step 5: finish in the job log#

Configuration tells you what GitLab asked for. The job log tells you what happened. Its first lines name the runner and the image. After that, every command the script ran is echoed. For each command that isn't a standard tool, ask where it came from:

a function loaded from a library in the image, like MegaCorp's mc_aws_login
a tool installed in the image, like MegaCorp's mc
a script downloaded during the job, with curl or git clone
behaviour configured in a build tool, like a parent POM or a Makefile

Chapter 12 covers each one. Behaviour that changes with the runner the job landed on, and that nothing in the job explains, belongs to the runner's configuration, which only its administrators can read.

Spot the bugTrace a production deploy#

web-portal's pipelines on main include a job called deploy-site that publishes to production. The repository has no .gitlab-ci.yml. You are asked three things: where is deploy-site defined, which file decides the command it runs, and which AWS role does it use?

QuestionTrace deploy-site to its source and answer all three.

Show the answer

The project's CI/CD settings name pipelines/web.yml@megacorp/devops/ci-templates, a custom configuration file. That file defines deploy-site: extends: .publish-site, with its own rules.
.publish-site is in templates/node.yml, included at v4.2.0. Its script runs aws s3 sync and a CloudFront invalidation. Its before_script pastes two snippets from templates/snippets.yml, which call mc_aws_login from the function library in the ci-tools image.
.publish-site sets AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-web-publish, unless a settings variable of the same name overrides it. Check the project's and the groups' variables before you trust the YAML (extends).

Next · Part III14Pipeline sources and types7 min

Pipeline sources and types#

Part III · Conditions · Chapter 14· 7 min read

Every pipeline is created by an event, and GitLab records which one in CI_PIPELINE_SOURCE: push, merge_request_event, schedule, web and a dozen more. The kind of pipeline then decides which variables exist:

CI_COMMIT_BRANCH is set in branch pipelines and schedules, and not in merge request pipelines
CI_COMMIT_TAG is set only for tags
CI_MERGE_REQUEST_* is set only in merge request pipelines

So the first question about any job that ran, or didn't, is "what kind of pipeline was this?" One push to a branch with an open merge request can start two pipelines.

Where pipelines come from#

Seven kinds of event account for almost every pipeline you will meet in an enterprise project:

Figure Each kind of event sets CI_PIPELINE_SOURCE, and the kind of pipeline decides which ref variables are set. A push to a branch with an open merge request can start both of the first two.#

The full list of values, which is also what the pipelines API reports as a pipeline's source:

CI_PIPELINE_SOURCE	What created the pipeline
`push`	a push to a branch, or a new tag
`merge_request_event`	a merge request being created, a push to its source branch, or Run pipeline on its Pipelines tab
`schedule`	a pipeline schedule coming due, or someone selecting Run on it
`web`	New pipeline, under Build › Pipelines
`api`	the pipelines API
`trigger`	the pipeline trigger API, called with a trigger token
`pipeline`	a trigger job in another project: a multi-project pipeline
`parent_pipeline`	a trigger job in the same project: a child pipeline
`security_orchestration_policy`	a scan execution policy's schedule, on GitLab Ultimate
`webide`	the Web IDE
`chat`	a ChatOps command, typed into a chat tool connected to GitLab
`external`	a CI service other than GitLab
`external_pull_request_event`	a pull request on GitHub, for a project that mirrors it
`ondemand_dast_scan`	an on-demand DAST scan, which tests a running web application for security problems
`ondemand_dast_validation`	a check that the site a DAST scan targets belongs to you

GitLab Duo Agent Platform also runs workload pipelines, on refs named refs/workloads/…, which pipeline lists show with the source duo_workflow. They are not part of your CI/CD configuration.

The pipeline list labels each pipeline by type: branch, tag, merge request, merged results or merge train. Look at the label before you look at any rule.

What each kind of pipeline sets#

Rules test variables, and the kind of pipeline decides which of those variables exist. GitLab documents these combinations:

Variable	Branch pipeline	Tag pipeline	Merge request pipeline	Scheduled pipeline
CI_PIPELINE_SOURCE	`push`	`push`	`merge_request_event`	`schedule`
CI_COMMIT_BRANCH	the branch	not set	not set	the schedule's branch
CI_COMMIT_TAG	not set	the tag	not set	only if the schedule runs on a tag
CI_MERGE_REQUEST_*	not set	not set	set while the merge request is open	not set
CI_PIPELINE_SCHEDULE_DESCRIPTION	not set	not set	not set	the schedule's description

Three consequences catch almost everyone:

A merge request pipeline has no branch. Any rule that tests CI_COMMIT_BRANCH is false there. Test CI_MERGE_REQUEST_SOURCE_BRANCH_NAME or CI_MERGE_REQUEST_TARGET_BRANCH_NAME instead.
A tag pipeline has no branch either, even when the tagged commit is on main.
A schedule on a branch sets CI_COMMIT_BRANCH exactly as a push does. A rule meant for "merged to main" also matches the nightly schedule on main, unless it tests CI_PIPELINE_SOURCE. That is why payments-api's docs job runs every night (chapter 7).

A few other variables describe the pipeline rather than the commit:

CI_COMMIT_REF_NAME is the branch or tag the pipeline is built for.
CI_OPEN_MERGE_REQUESTS lists up to four merge requests whose source is the current branch. It is set only in branch and merge request pipelines, and only when such a merge request exists. Chapter 15 uses it to stop duplicate pipelines.
CI_COMMIT_BEFORE_SHA, the commit the branch pointed at before the push, is all zeros in merge request pipelines, scheduled pipelines, manual runs, and the first pipeline of a new branch or tag.

MegaCorp names these tests once, in its rules library, and every template picks from it:

megacorp/devops/ci-templatestemplates/rules.ymlrules-library

.rules:
  mr:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  default-branch:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  release-tag:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
  never-on-schedule:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never

Rule	Matches	Watch out for
mr	every merge request pipeline	nothing else: branch, tag and scheduled pipelines never match it
default-branch	pushes to main, and scheduled or manual runs on main	it can't tell a merge to main from the nightly schedule on main
release-tag	tag pipelines whose tag looks like `v1.4.0`	`v1.4.0-rc1` matches nothing, so a release candidate's tag pipeline gets none of these jobs
never-on-schedule	removes the job from scheduled pipelines	only if GitLab reaches it before a rule that matches; see chapter 16

Merge request pipelines, in three flavours#

A merge request pipeline runs:

when a merge request is created from a branch with at least one commit
on every push to the merge request's source branch
when someone selects Run pipeline on the merge request's Pipelines tab

It tests the source branch alone, not the result of merging it. GitLab has two stricter variants, and CI_MERGE_REQUEST_EVENT_TYPE tells you which one you are in:

Flavour	What it tests	CI_MERGE_REQUEST_EVENT_TYPE	Tier and list label
Merge request pipeline	the source branch only	`detached`	every tier; `merge request`
Merged results pipeline	a temporary commit that merges the source into the latest target; if the two conflict, GitLab runs a plain merge request pipeline instead	`merged_result`	Premium; `merged results`
Merge train	the merge request together with every merge request queued ahead of it	`merge_train`	Premium; `merge train`

Merge request pipelines don't exist until the configuration asks for them: some rule has to match merge_request_event. Merged results pipelines and merge trains also need a project setting, under Settings › Merge requests.

Note

GitLab's documentation for GitLab 19.3 warns that a merge_request_event rule arriving through include: does not satisfy this requirement, and that the rule must be in the project's own .gitlab-ci.yml. On the local GitLab, payments-api gets every rule through includes. It still got a merge request pipeline when its merge request was opened, on every push after that, and whenever someone selected Run pipeline. If a project's merge request pipelines go missing, the documented caveat is one more thing to check.

Two situations change what a merge request pipeline can reach:

Merge requests from forks. The pipeline runs in the fork, with the fork's configuration and CI/CD variables, not the parent's. A member of the parent project can choose to run it in the parent instead. It then uses the fork's configuration with the parent's variables and runners, so GitLab warns first: the fork's code could try to read the parent's secrets.
Protected variables and runners Since 18.1. A merge request pipeline gets them only when both branches are protected and in the same project, the person running it can push or merge to the target, and a project setting allows it. Otherwise a job that needs a protected secret finds it empty.

One push, two pipelines#

Push to a branch that has an open merge request and GitLab receives two events: the push, and an update to the merge request. It tries to create a pipeline for each: a branch pipeline with source push, and a merge request pipeline with source merge_request_event. Each is created only if its rules add at least one job. If both qualify, the same commit shows up twice in the merge request's Pipelines tab, and runs every shared job twice.

The two can disagree, and it matters which one counts:

Pipelines must succeed checks the merge request pipeline, not the branch pipeline. GitLab documents a race condition in which pipeline's result blocks or passes the merge request, so a green branch pipeline beside a red merge request pipeline can go either way.
An invalid configuration pushed to a merge request's branch produces two failed pipelines, one of each type.

Which of the two you get is decided by workflow:rules and job rules, in chapter 15. Since 19.2 a project setting can also decide it: Settings › CI/CD › General pipelines › Skip branch pipelines for merge requests Beta. With it on, a push to a merge request's source branch creates only the merge request pipeline. The push that opens the merge request still makes a branch pipeline, and jobs with no rules at all start appearing in merge request pipelines.

Pipelines nobody pushed#

Several kinds of pipeline start without a commit. Each has a trait that confuses people:

Kind	Source	What to know
Scheduled	`schedule`	runs with its owner's permissions; turns Inactive if the owner is blocked or leaves, until a Maintainer takes ownership; its variables exist only in its own pipelines; `rules: changes` is always true in it
New pipeline in the UI	`web`	values typed into the form override every other variable of the same name, and are not masked
API	`api`	the ref and variables come from the call
Trigger token	`trigger`	`CI_PIPELINE_TRIGGERED` is `true`; the token acts with its owner's access (chapter 10)
Multi-project	`pipeline`	every job in it sees `pipeline`; it runs on the target's default branch unless the trigger job names one
Child	`parent_pipeline`	every job in it sees `parent_pipeline`, even when the parent is a merge request pipeline
Policy scan	`security_orchestration_policy`	created by a scheduled scan execution policy, not by anything in the project

A scheduled pipeline execution policy Since 19.2 also creates pipelines on a schedule. They run only the policy's jobs, never the project's (chapter 29).

The child pipeline row is the one that bites. A job in a child pipeline that tests $CI_PIPELINE_SOURCE == "merge_request_event" never runs, because the source is always parent_pipeline. The parent's CI_MERGE_REQUEST_* variables are passed down, so a child job that should run only for merge requests tests $CI_MERGE_REQUEST_ID instead.

A pipeline can also be created and then skipped. A commit message containing [ci skip] or [skip ci], in any capitalisation, still creates a pipeline, with no jobs and the status Skipped. The ci.skip push option does the same, except for merge request pipelines, which it doesn't skip. A security policy can forbid skipping.

When a variable exists#

GitLab sets predefined variables in three waves, and a rule can only test variables from the waves before it runs:

Wave	Set when	Examples	Can be tested by
Pre-pipeline	before GitLab starts building the pipeline	`CI_COMMIT_BRANCH`, `CI_COMMIT_TAG`, `CI_PIPELINE_SOURCE`, `CI_OPEN_MERGE_REQUESTS`, `CI_DEFAULT_BRANCH`, `CI_MERGE_REQUEST_*`	`include: rules`, `workflow: rules`, job `rules`, scripts
Pipeline	while GitLab builds the pipeline	`CI_PIPELINE_IID`, `CI_JOB_NAME`, `CI_JOB_STAGE`, `CI_NODE_INDEX`, `CI_ENVIRONMENT_NAME`	job `rules`, scripts
Job-only	when a runner picks up the job	`CI_JOB_ID`, `CI_JOB_TOKEN`, `CI_PIPELINE_ID`, `CI_PROJECT_DIR`, `CI_RUNNER_TAGS`	scripts only: never `workflow`, `include`, `rules` or trigger jobs

Three more exclusions follow from the same timing:

Dotenv variables, which a job writes for later jobs, never reach any rule. Every rule is evaluated before the first job runs.
rules: if can't use CI_ENVIRONMENT_SLUG, or the variables GitLab calls persisted, such as CI_PIPELINE_ID, CI_JOB_TOKEN and CI_REGISTRY_PASSWORD.
include can't use top-level variables: or job variables, because includes are resolved before either exists. It can use settings variables, CI_PROJECT_*, CI_PIPELINE_SOURCE, CI_COMMIT_REF_NAME, and variables from triggers, schedules and manual runs.

When a rule refers to a variable and never matches, look for that variable in these three lists before you look anywhere else.

Spot the bugThe check that never runs on merge requests#

payments-api adds a slow contract test. It should run only in merge requests that target main:

contract-test:
  stage: test
  script:
    - mvn $MAVEN_CLI_OPTS verify -Pcontract
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

It never appears in any merge request pipeline, including merge requests into main. It never appears in any other pipeline either.

QuestionWhy does the rule never match, and what should it say?

Show the answer

CI_COMMIT_BRANCH is not set in merge request pipelines, so the second comparison is false whenever the first is true. The branch a merge request is heading for is in CI_MERGE_REQUEST_TARGET_BRANCH_NAME:

rules:
  - if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH

See what each kind of pipeline sets.

Next15workflow: whether a pipeline exists6 min

workflow: whether a pipeline exists#

Part III · Conditions · Chapter 15· 6 min read

workflow: rules decides whether a pipeline is created at all. GitLab checks it after assembling the configuration and before looking at a single job:

If no rule matches, there is no pipeline. For a push, you get no error anywhere.
The syntax is the same as job rules, but when can only be always or never.

Most organisations use it for one thing: stopping the duplicate branch and merge request pipelines from chapter 14. It can also name pipelines, set a variable for every job, and decide what a newer commit cancels.

Checked before any job#

Whatever the jobs say, workflow has the first word. A job whose rules allow tag pipelines never runs if workflow refuses tag pipelines, because the pipeline it would have run in is never created.

A workflow rule may use if, changes, exists, variables and auto_cancel. Its when is limited to always and never. A rule that matches without a when lets the pipeline be created. GitLab accepts start_in, allow_failure and needs in a workflow rule without complaint, and ignores them.

Workflow rules can only test variables that exist before any job does: the pre-pipeline variables of chapter 14, and variables from settings, triggers, schedules and manual runs. A workflow rule that tests a job-only variable can't work.

What you see when no rule matches depends on how the pipeline was asked for. After a push, nothing appears. GitLab gives no error because nothing failed; the pipeline was simply never wanted. When someone asks for a pipeline through the API, GitLab 19.3 refuses it with this error:

The pipeline did not run. Review the workflow:rules configuration for the pipeline.

A different error means workflow let the pipeline through, but every job's rules left the job out:

The resulting pipeline would have been empty. Review the rules configuration.

MegaCorp's workflow, rule by rule#

Every MegaCorp project that uses a golden pipeline includes this file:

megacorp/devops/ci-templatestemplates/workflow.ymlworkflow

workflow:
  name: "$CI_PIPELINE_SOURCE: $CI_COMMIT_REF_NAME"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH
    - if: $CI_COMMIT_TAG

Read it top to bottom, stopping at the first rule that matches:

The pipeline GitLab is asked for	First rule that matches	Created?
a merge request pipeline	1: `merge_request_event`	yes
a push to a branch that has an open merge request	2: `$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS`	no
a push to any other branch, `main` included	3: `$CI_COMMIT_BRANCH`	yes
a tag	4: `$CI_COMMIT_TAG`	yes
a schedule on `main`	3	yes
a schedule, API call or trigger on a branch that has an open merge request	2	no

Rules 1 to 3 are GitLab's documented switch pattern: branch pipelines until a merge request is opened, merge request pipelines after that. Rule 4 adds tags.

Rule 2 is broader than it looks. It doesn't ask whether the pipeline came from a push, so it refuses every pipeline on a branch that has an open merge request, unless it is a merge request pipeline. That includes pipelines started by an API call, a schedule, or a trigger from another project. GitLab's documentation gives the guard for exactly this, && $CI_PIPELINE_SOURCE == "push", and MegaCorp's rule doesn't have it. The exercise at the end of this chapter is what that costs.

Duplicate pipelines#

One push to a branch with an open merge request makes GitLab try to create two pipelines (chapter 14). You get both whenever some job's rules accept both. The configurations that cause it:

A final rule with no if, such as - when: on_success or - when: always, which matches every kind of pipeline. CI Lint and the New pipeline page show the warning Job may allow multiple pipelines to run for a single action for it.
Rules that list both push and merge_request_event, with no workflow to choose between them.
Jobs without rules mixed with jobs with rules. A job with no rules at all gets the legacy default: branches and tags, never merge requests. Put it beside a job whose rules match merge request pipelines and each push makes two pipelines, one for each job.

MegaCorp's monorepo has the third problem. It has no workflow section, its generator jobs have no rules, and its docs trigger has a rule that tests only which files changed:

megacorp/data/mono.gitlab-ci.ymlstatic-child

docs:
  stage: build
  trigger:
    include: ci/docs.yml
  rules:
    - changes: [docs/**/*]

A rule with only changes matches in any kind of pipeline, merge request pipelines included. When an open merge request touches docs/, every push to its branch makes two pipelines: a branch pipeline with the generator jobs, and a merge request pipeline containing only docs.

The local GitLab showed both pipelines, and a second problem. In every merge request pipeline, docs failed, with the reason downstream_pipeline_creation_failed. The child pipeline's own jobs have no rules, and GitLab documents that such jobs don't join merge request pipelines, child pipelines included. With none of its jobs added, the child pipeline couldn't be created (chapter 14).

Teams stop duplicates in one of four ways. Recognise which one a project uses and you know which pipelines to expect:

Pattern	What the workflow rules say	What you get
Switch	merge request events; never a branch with an open merge request; any other branch	branch pipelines until a merge request opens, then merge request pipelines only
Merge requests, main and tags	merge request events; the default branch; tags; sometimes protected branches	no pipeline at all for a branch without a merge request
Branches only	`$CI_PIPELINE_SOURCE == "push"`, and no rule for merge request events	no merge request pipelines, and no merged results or merge trains
The project setting	nothing; the setting Skip branch pipelines for merge requests is on	as the switch, for pushes only; beta since 19.2

Names and variables for the whole pipeline#

workflow: name gives every pipeline a title in the pipeline list, and the value is available to jobs as CI_PIPELINE_NAME. MegaCorp's name shows the source and the ref, such as push: main or schedule: main. That tells you the kind of pipeline before you open it. If the name comes out empty, because every variable in it is empty, the pipeline gets no name.

workflow: rules: variables sets variables when a rule matches:

They become default variables for every job, just like top-level variables:, and they replace top-level variables of the same name.
A variable set in the job itself still wins.
Trigger jobs pass them downstream, where they replace the downstream project's own values of the same name. A generic name like ENVIRONMENT set here can silently change another team's pipeline.

To stop that, give such variables unique names, or have the trigger job list what it passes with inherit: variables.

What a newer commit cancels#

Push twice in quick succession and GitLab can cancel the older pipeline for you, but only if the project setting Settings › CI/CD › General pipelines › Auto-cancel redundant pipelines is on. It applies only to a newer commit on the same branch; selecting New pipeline again for the same commit cancels nothing. With the setting off, interruptible does nothing.

workflow: auto_cancel: on_new_commit	What a newer commit cancels
`conservative` (the default)	the older pipeline, unless a job with `interruptible: false` has already started; jobs that haven't started count as interruptible
`interruptible`	only the jobs marked `interruptible: true`; the rest carry on
`none`	nothing

The companion key on_job_failure: all cancels every running job as soon as one job fails, instead of letting its stage finish. Both keys can be overridden per rule with workflow: rules: auto_cancel, for example so that pipelines on protected branches are never cancelled.

Under the default, a single non-interruptible job decides for the whole pipeline. Once it starts, the pipeline can no longer be auto-cancelled. A downstream pipeline is cancelled along with its parent if none of its own non-interruptible jobs has started yet.

MegaCorp sets interruptible: true in default:, so every job that inherits it can be cancelled by a newer push. That is fine for builds and tests. Deployment jobs are different: GitLab advises against making them interruptible, because cancelling one halfway can leave a partial deployment. When a template puts interruptible: true in default:, check that deploy jobs set it back to false.

When no pipeline appears#

Work down this list. It runs from the most common cause to the least:

What you see	Likely cause	How to confirm
no pipeline at all after a push	`workflow: rules` matched nothing	run the rules by hand against the pipeline's variables (chapter 14's table)
no pipeline, and workflow would have allowed it	every job's rules left it out; a pipeline with no jobs isn't created	CI Lint, simulating the branch
no pipeline, although jobs should match	the only jobs left are in `.pre` or `.post`, which can't form a pipeline alone	check which stages the matching jobs are in
a pipeline marked Skipped, with no jobs	`[ci skip]` or `[skip ci]` in the commit message, or the `ci.skip` push option	the commit message
a failed pipeline with an error and no jobs	the configuration is invalid	the error on the pipeline page, or the pipeline editor
jobs or variables missing for no visible reason	a byte-order mark at the start of a YAML file, which the pipeline editor can't show	a hex viewer, or a tool that shows invisible characters
a merge request stuck on Checking pipeline status	Pipelines must succeed is on, and nothing lets a pipeline run for its latest commit	the merge request's Pipelines tab

Spot the bugThe pipeline nobody can start#

A payments-api developer is working on feature/retry-fix, which has an open merge request. They want a one-off branch pipeline with an extra variable, so they open Build › Pipelines › New pipeline, choose their branch, add the variable and submit. No pipeline is created. The same happens when a colleague uses the API.

megacorp/devops/ci-templatestemplates/workflow.ymlworkflow

workflow:
  name: "$CI_PIPELINE_SOURCE: $CI_COMMIT_REF_NAME"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH
    - if: $CI_COMMIT_TAG

QuestionWhich rule refuses the pipeline, and what are the two ways round it?

Show the answer

Rule 2. A pipeline started on a branch from the UI or the API has CI_COMMIT_BRANCH set, and CI_OPEN_MERGE_REQUESTS is set because the branch has an open merge request. The rule doesn't check the source, so it refuses the pipeline.

Now: run a merge request pipeline instead. On the merge request's Pipelines tab, Run pipeline with modified values lets you set variables and inputs.
For good: ask the template owners to narrow the rule to pushes, as GitLab documents: if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push".

See MegaCorp's workflow, rule by rule.

Next16Job rules: is this job in the pipeline?7 min

Job rules: is this job in the pipeline?#

Part III · Conditions · Chapter 16· 7 min read

When GitLab creates a pipeline, it reads each job's rules from the top. The first rule that matches decides the outcome:

the job is added: normally, as a manual job, or delayed
or the rule says when: never and the job is left out

If no rule matches, the job is left out. A matching rule can also change the job's allow_failure, needs, variables and interruptible.

A rule's conditions are if (variables), changes (files changed) and exists (files present). A job with no rules at all gets an older default: branches and tags, never merge requests.

Four gates#

A job passes four gates before it runs, and GitLab checks each at a different moment:

Figure The condition funnel. Four gates, checked while GitLab assembles the configuration, creates the pipeline and runs it. Each gate says no in its own way.#

Gate	Checked	Can test	When it says no
`include: rules`	while GitLab assembles the configuration	pre-pipeline variables, settings variables, and files	the included file's jobs never exist, and nothing in the pipeline mentions them
`workflow: rules`	before any job is considered	variables that exist before any job, and files	there is no pipeline; after a push there is no error either
job `rules`	once for each job while the pipeline is created, and once for each matrix job	those, plus pipeline variables and the job's own variables	the job is not in the pipeline
`when`, `needs`, `allow_failure`	while the pipeline runs	the outcome of earlier jobs	the job is skipped, or waits for a person or a timer

Chapter 15 covered the first two gates, and chapter 17 covers the last. This chapter is about the third.

First match wins#

A job's rules is a list, and GitLab reads it from the top, stopping at the first rule whose conditions all hold:

A rule needs at least one of if, changes, exists or when. When a rule has several conditions, all of them must be true.
A rule with only when has no conditions, so it matches whenever GitLab reaches it. It is the "otherwise" at the end of a list.
A matching rule with when: never leaves the job out. Any other matching rule adds it.
A matching rule without when uses the job's own when, which defaults to on_success. A when written in the rule overrides the job's.
If GitLab reaches the end of the list without a match, the job is left out.

Order is everything. MegaCorp's two deploy jobs show both common styles:

megacorp/devops/ci-templatespipelines/java-service.ymldeploys

deploy-staging:
  extends: .deploy
  needs: [image-build]
  variables:
    ENVIRONMENT: staging
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, default-branch]

deploy-prod:
  extends: .deploy
  needs: [image-build]
  variables:
    ENVIRONMENT: production
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
      when: manual

deploy-staging pastes two lists from the rules library with !reference. GitLab flattens them into one list, in the order they are written (chapter 7). Read it for four pipelines:

Pipeline	Rule 1: never-on-schedule	Rule 2: default-branch	deploy-staging
the nightly schedule on main	matches, with when: never	not reached	left out
a push to main	no	matches	added, on_success
a merge request	no	no	left out
tag v1.4.0	no	no	left out

Swap the two rules and the nightly schedule matches default-branch first, because a schedule on main sets CI_COMMIT_BRANCH. Nothing in the list looks wrong; only its order is.

deploy-prod has a single rule, with when: manual. In a tag pipeline whose tag looks like a release, it is added as a manual job. Everywhere else it is left out. Because the when: manual is inside a rule, the job blocks its pipeline until someone runs it; chapter 17 explains why.

Writing if conditions#

An if is an expression over variables. Some forms use a regular expression: a pattern for matching text, written between slashes. For example, /^release/ means "starts with release". These forms cover almost everything you will read:

Expression	True when	Watch out for
$VAR	VAR is set and not empty	any non-empty value counts, including `"false"` and `"0"`
$VAR == "text"	the value is exactly text	case matters; the variable goes on the left, and only the string is quoted
$VAR != "text"	the value is anything else	also true when VAR isn't set at all
$VAR == null	VAR is not set	a variable set to `""` is not null
$VAR == ""	VAR is set, and empty
$A == $B	both have the same value
$VAR =~ /re/	the regular expression matches part of the value	unanchored: `/release/` matches `pre-release-notes`; write `/^release/`
$VAR =~ /re/i	the same, ignoring case
$VAR !~ /re/	the regular expression doesn't match
$VAR =~ $PATTERN	the value matches the regular expression stored in PATTERN, slashes included	variables inside the stored expression are not expanded
a && b, a \|\| b	both, or either	`&&` binds tighter than `\|\|`; add parentheses to be sure
!$VAR, !(a == b)	the opposite; since 18.11	`!` tests for empty or unset, not for `"false"`

The surprising rows were checked on the local GitLab 19.3:

"false" counts as true.
!= is true for a variable that isn't set.
An empty variable is not null.
&& binds before ||.

The same test turned up one more trap. Put a negated expression in quotes: if: '!($VAR == "x")'. Without the quotes, GitLab rejects it with invalid expression syntax, even though GitLab's own documentation shows it unquoted.

Four syntax traps make GitLab reject a configuration, or accept it and mean something else:

Quote strings, never variables. $ENV == production (unquoted string), "$ENV" == "production" (quoted variable) and ${ENV} == "production" (braces) are all invalid, and GitLab refuses the whole configuration.
A regular expression is one whole /…/. issue-/.*/ doesn't work; /issue-.*/ does. A pattern of a single character, such as /./, is rejected as invalid expression syntax. Matching uses RE2 syntax.
A right-hand side without slashes is not a regular expression. $A =~ "text" is accepted, but tests whether A's value appears inside text: "23" =~ "1234" is true. Always write the slashes.
Values are not expanded twice. If a variable's value is $OTHER, the condition sees that literal text, not OTHER's value. This holds for matrix values too.

A condition can only test a variable that exists when the pipeline is created. If a rule never matches, check the variable against chapter 14's list: job-only variables, persisted variables and dotenv variables are never there.

changes and exists#

Two conditions test files instead of variables:

Condition	True when	Watch out for
`changes: [paths]`	a changed file matches one of the paths	"changed" depends on the kind of pipeline; see the next table
`changes: paths` with `compare_to`	a file differs from the branch, tag or commit named	in a merged results pipeline, the comparison also includes the target branch's own changes
`changes: regexp`	a changed path matches a Ruby regular expression; since 19.2	anchor it with `\A` and `\z`, not `^` and `$`
`exists: [paths]`	a file or directory matching one of the paths is in the repository	a directory needs a trailing slash; artifacts are never seen
`exists: paths` with `project` and `ref`	the file exists in another project, at a ref

"Changed" is the trap. What a file is compared against depends on the kind of pipeline:

Pipeline	changes compares with	So it is true
merge request	the target branch	when the merge request as a whole touches the files
push to an existing branch	the branch's previous commit	when this push touches the files
the first push of a new branch, or a new tag	nothing	always
scheduled, New pipeline, API or trigger: anything without a push	nothing	always

MegaCorp's monorepo shows the third row. Its docs trigger job has a rule with changes: [docs/**/*] and nothing else. When the local GitLab received mono's very first push to main, the pipeline included docs, although the repository had no docs/ directory at all.

GitLab's advice follows from the table: pair changes with an if that limits it to merge request or branch pipelines, or give it a compare_to.

Four smaller rules about paths:

Don't write ./ or //. Paths are compared as plain text, not resolved like a shell path.
A variable ending in a slash turns $DIR/* into dir//*.
Root-level wildcards need quotes in YAML: "*.json".
Each list holds at most 50 paths. After 50,000 checks a pattern counts as matched, so in a huge change or repository every changes or exists rule is true.

One more difference: in include: rules, exists looks in the project and ref of the file that contains the include. In a job, it looks in the project running the pipeline. A template that works in its own repository can test the wrong project once someone else includes it.

What a matching rule can change#

Key in the rule	Effect, only when this rule is the one that matched
`when`	`on_success`, `on_failure`, `always`, `manual` or `delayed` add the job; `never` leaves it out
`allow_failure`	replaces the job's value; the default is `false`, even with `when: manual`
`needs`	replaces the job's entire `needs` list; `[]` makes it start at once
`variables`	adds variables, or overrides the job's, for this case only
`interruptible`	replaces the job's value

Gotcha

when: manual means two different things depending on where it is written. On the job itself, it makes an optional manual job, with allow_failure: true. Inside a rule, it makes a blocking one, with allow_failure: false: the pipeline waits at that stage until someone runs it. Add allow_failure: true to the rule to get the optional kind.

Rules and matrix jobs#

With parallel: matrix, GitLab evaluates the rules separately for each generated job, using that job's values. In MegaCorp's matrix, a rule if: $JDK == "17" would keep maven-test: [17] and drop maven-test: [21]. Matrix values can also appear in changes and exists paths, which lets a monorepo run each service's job only when that service's files change.

Legacyonly and except

Older templates control jobs with only and except, which GitLab has deprecated; use rules instead. They list branch names, regular expressions and kinds of pipeline, such as branches, tags, merge_requests, schedules, pushes, web, api, triggers and pipelines.

A job with no only, except or rules behaves as only: [branches, tags]. That is why such a job never appears in merge request pipelines.
only: branches matches scheduled pipelines too, because a schedule runs on a branch. except: schedules removes them.
Mixing only/except jobs with rules jobs is a classic cause of duplicate pipelines (chapter 15).

To translate a job to rules, write one if rule for each only entry. For each except entry, write one when: never rule, then finish the list with - when: on_success.

Why was this job left out?#

Name the pipeline. Its label and source tell you which variables exist (chapter 14).
Get the job's real rules from the Full configuration view. extends may have replaced the rules you expected (the card), and every !reference is already expanded there.
Walk the list from the top with those variables, and stop at the first match.
For each variable a rule tests, check that it exists when the pipeline is created, and find where its value is set (chapter 19).
For each changes, check which comparison applies in this kind of pipeline.
If the job is in the pipeline but never ran, you have moved on to the fourth gate: chapter 17.

Spot the bugThe smoke test that runs every night#

A team adds a smoke test that should run after each merge to main, using MegaCorp's rules library:

smoke-test-staging:
  stage: deploy
  script:
    - ./scripts/smoke-test.sh staging
  rules:
    - !reference [.rules, default-branch]
    - !reference [.rules, never-on-schedule]

It does run after merges. It also runs at 02:00 every night, in the scheduled pipeline on main, against a staging environment nobody has changed.

QuestionThe list includes never-on-schedule. Why doesn't it stop the nightly run?

Show the answer

First match wins. A schedule on main sets CI_COMMIT_BRANCH to main, so default-branch matches first and adds the job. GitLab never reaches never-on-schedule.

Put the exclusion first, as MegaCorp's deploy-staging does:

rules:
  - !reference [.rules, never-on-schedule]
  - !reference [.rules, default-branch]

See first match wins.

Next17Order and gates: when a job starts, and what stops it8 min

Order and gates: when a job starts, and what stops it#

Part III · Conditions · Chapter 17· 8 min read

A job that made it into the pipeline still has to wait its turn:

Stages run in order. A job starts when every job in the earlier stages has succeeded, been allowed to fail, or been left waiting as an optional manual job.
needs replaces that barrier with a list of jobs. The job starts when those finish, whatever else is still running.
when decides the rest: on_success (the default), on_failure, always, manual or delayed.

Then come the gates:

allow_failure stops a failure from counting.
A manual job added by a rule holds the pipeline until someone runs it.
resource_group makes jobs wait their turn, across pipelines.
retry, timeout and interruptible decide what happens when things go wrong.

Stages: the default barrier#

Without needs, a stage starts only when the stage before it has finished. For that barrier, three outcomes count as finished well:

success
a failure of a job marked allow_failure: true
an optional manual job that nobody has started, which counts as skipped

If any other job fails, the pipeline is marked failed and later stages never start. Jobs already running in the failing stage carry on to the end.

The stage list has fixed ends. .pre runs first and .post last, and neither needs listing. A pipeline execution policy can add .pipeline-policy-pre before .pre and .pipeline-policy-post after .post (chapter 29).

Three smaller rules explain some odd-looking graphs:

A job with no stage lands in test.
A listed stage that no job uses is not shown at all.
A pipeline whose only jobs are in .pre or .post is not created.

needs: a graph instead of a barrier#

A job with needs ignores the stage barrier. It starts as soon as every job it lists has finished, even while other jobs in earlier stages are still running. MegaCorp's golden pipeline uses it three times: image-build needs maven-build, and both container_scanning and deploy-staging need image-build. Here is what that does to a pipeline on main:

Figure The same payments-api jobs on main, ordered by stages alone, then with MegaCorp's needs. Durations are illustrative. With needs, deploy-staging waits only for image-build.#

The pipeline is much shorter, and one bar is in the wrong place. deploy-staging lists only image-build. So it starts as soon as the image exists, while maven-test and container_scanning may still be running. A test that fails afterwards turns the pipeline red, but staging already has the build. Nobody wrote "deploy without testing"; the missing entries in a needs list say it for them.

The details that matter when you read a needs list:

needs: [] starts the job as soon as the pipeline is created, as .pre jobs do. It suits linters and scanners that read only the source.
A job can need jobs in its own stage. A job can list at most 50 jobs.
Needing a matrix job means needing all of it. needs: [maven-test] waits for every maven-test: […] job. needs: parallel: matrix picks particular ones.
Artifacts follow needs. A job with needs downloads artifacts only from the jobs it lists. artifacts: false on an entry downloads nothing from that job. Don't combine needs with dependencies.
Seeing the graph: on the pipeline page, select Job dependencies. The default view groups jobs by stage and hides the lines.

needs and rules collide when a needed job is left out of a pipeline by its own rules. GitLab then refuses to create the pipeline at all, with an error naming both jobs. There are two cures:

Give the needing job rules that keep it out whenever the needed job is out.
Mark the entry optional: true. The job then waits for the needed job when it exists, and ignores it when it doesn't. If every entry is optional and none of those jobs exists, the job starts at once, as if it had needs: [].

Here is the error from the local GitLab 19.3, for a unit-tests job that needs a compile job its rules had left out:

'unit-tests' job needs 'compile' job, but 'compile' does not exist in the pipeline. This might be because of the only, except, or rules keywords. To need a job that sometimes does not exist in the pipeline, use needs:optional.

Three other keys are spelt needs but do different jobs:

Key	What it does	Watch out for
`needs: project`	downloads artifacts from the latest successful run of a job in another project, at a ref; Premium	it doesn't wait: if that project's pipeline is still running, you get the previous run's artifacts
`needs: pipeline` with `job`	lets a child pipeline download artifacts from a job in its parent or a sibling	the job must have succeeded
`needs: pipeline` alone	copies the latest status of another project's default-branch pipeline into this job	it is a status mirror, not a dependency

when: the job's own condition#

when	The job runs
`on_success` (the default)	when every job in earlier stages succeeded, was allowed to fail, or is an unstarted manual job
`on_failure`	only when at least one job in an earlier stage failed; for cleanup and notifications
`always`	whatever happened earlier
`manual`	when a person starts it
`delayed`	after the time in `start_in`
`never`	never; allowed only in `rules` and `workflow: rules`

One consequence surprises people: a failed job with allow_failure: true counts as a success. An on_failure notification job therefore stays silent when the only failure was allowed.

allow_failure: failures that don't count#

A job with allow_failure: true can fail without failing the pipeline. The pipeline passes, and the job carries an orange warning. The default depends on how the job was made manual:

The job	allow_failure defaults to
has `when: manual` on the job itself	`true`: an optional manual job
gets `when: manual` from a rule	`false`: a blocking manual job
anything else	`false`

allow_failure: exit_codes: [137, 255] allows a failure only for those exit codes. Any other failure still counts.

Scanners are the usual place to find it. In MegaCorp's pipeline on main, the local GitLab created container_scanning, secret_detection and sonar-scan all with allow_failure: true. That comes from GitLab's scanner templates and from the component. A scanner that crashes, or finds something, doesn't turn the pipeline red unless a policy makes it (chapter 29).

Manual and delayed jobs#

A manual job is shown as skipped until someone runs it, and there are two kinds:

Kind	allow_failure	What the pipeline does
Optional	`true`	carries on; the pipeline can pass without it ever running
Blocking	`false`	stops at the job's stage with status blocked; later stages wait, and Pipelines must succeed won't merge a blocked pipeline

Running a manual job needs permission to merge to the pipeline's branch. On Premium, a protected environment narrows that to a named list of people. If you are not on it, the job shows You are not authorized to run this manual job. manual_confirmation adds an "are you sure?" message. Variables typed in when starting a manual job are visible to other members and are not masked.

A release of payments-api passes through two manual gates. deploy-prod is manual by a rule, so the tag pipeline is blocked until someone runs it:

Pipeline payments-api tag v1.4.0: the release waits at deploy-prod, which then starts the deployer's own pipeline#

As text

build: maven-build
package: image-build
deploy: deploy-prod (manual)

Needs: image-build waits only for maven-build; deploy-prod waits only for image-build.

Downstream: deploy-prod triggers megacorp/platform/deployer (multi-project pipeline).

The deployer then asks again. Its deploy job is manual whenever ENVIRONMENT is production:

megacorp/platform/deployer.gitlab-ci.ymldeploy

variables:
  ENVIRONMENT: $[[ inputs.environment ]]

deploy:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  resource_group: $APP-$ENVIRONMENT
  environment:
    name: $ENVIRONMENT/$APP
  script:
    - mc deploy "$ENVIRONMENT" "$IMAGE_REF"
  rules:
    - if: $IMAGE_REF == null
      when: never
    - if: $ENVIRONMENT == "production"
      when: manual
    - when: on_success

So a production release takes two clicks in two projects. Between them, deploy-prod uses strategy: depend, and with it the trigger job shows running for as long as the deployer waits for its manual job. "The deploy has been running for three hours" usually means "nobody has pressed the second button". strategy: mirror would copy the deployer's own status instead.

A delayed job waits start_in, anything from one second to one week, counted from when its previous stage completes. Its stage doesn't finish until it has run. A bare number needs quotes ('30' means 30 seconds). To start one early, select Unschedule, then Run.

resource_group: one at a time#

The deployer's deploy job carries resource_group: $APP-$ENVIRONMENT. Jobs with the same resource group never run at the same time, even in different pipelines of the same project. A second payments-api staging deploy waits for the first, showing Waiting for resource: payments-api-staging and the status waiting_for_resource. Deploys of other services, with other names, go ahead.

The order in which waiting jobs are released is the group's process mode, and it isn't in any YAML file. It is set through the API:

Process mode	When the resource frees up, GitLab starts
`unordered` (the default)	whichever waiting job is ready
`oldest_first`	the job from the oldest pipeline
`newest_first`	the job from the newest pipeline, so older deploys are skipped over
`newest_ready_first`	the newest job that is ready to run

With oldest_first, a parent pipeline and its child that use the same resource group can deadlock, each waiting for the other. The documented fix is to put the resource group on the trigger job instead.

When things go wrong: retry, timeout, interruptible#

retry reruns a failed job up to twice. With when, it retries only certain kinds of failure; with exit_codes, only certain exit codes. MegaCorp sets it for every job in default::

megacorp/devops/ci-templatestemplates/base.ymldefault

default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  tags: [megacorp-shared]
  interruptible: true
  retry:
    max: 2
    when: [runner_system_failure, stuck_or_timeout_failure]
  before_script:
    - !reference [.snippets, functions]
    - mc_log "job $CI_JOB_NAME on $CI_COMMIT_REF_NAME"

That configuration has aged twice without anyone touching it:

stuck_or_timeout_failure was deprecated in 19.1, in favour of more precise reasons. The local GitLab 19.3 accepts it, but warns on every lint:

retry uses deprecated `when` value(s): stuck_or_timeout_failure. These match the more specific failure reasons that replaced them; migrate to those reasons.

19.1 also moved some failures out of runner_system_failure, into the new runner_external_dependency_failure (such as a registry that couldn't be reached) and runner_interrupted. A job that used to be retried after a network blip may now fail at once.

timeout sets how long the job may run, in words: 90 minutes, 3h 30m. It can be longer than the project's timeout, but never longer than the runner's. It can't be set in default:. Jobs that stop making progress are dropped, with a failure reason that says why:

The job was	Dropped after	Failure reason
pending, with a runner that could take it	24 hours	`stuck_pending_with_matching_runners`
pending, with no runner that could take it	1 hour	`stuck_pending_no_matching_runners`
running, with no updates from the runner	30 minutes	`no_updates_running`
running past its timeout	the timeout plus 15 minutes	`server_timeout_running`

interruptible decides whether a newer commit may cancel the job; chapter 15 covers it.

Spot the bugThe nightly pipeline that never starts#

An earlier version of MegaCorp's security.yml gave the container scan these rules:

container_scanning:
  stage: scan
  needs: [image-build]
  rules:
    - if: $CONTAINER_SCANNING_DISABLED == "true"
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

When the payments team added a nightly schedule on main, every nightly run failed before a single job started. Pushes to main and merge requests were fine.

QuestionWhat stops the scheduled pipeline from being created, and what are the two ways to fix it?

Show the answer

In a schedule on main, CI_COMMIT_BRANCH is main, so the last rule adds container_scanning. But image-build inherits .megacorp-base's rules, which start with never-on-schedule, so it is left out. The pipeline would contain a job that needs a job that doesn't exist, so GitLab refuses to create it.

Keep the scan out of schedules too, by putting - !reference [.rules, never-on-schedule] before the other rules. That is what MegaCorp's current security.yml does.
Or make the need optional: true, if a scan without a fresh image makes sense.

See needs: a graph instead of a barrier.

Next18Truth tables: predicting which jobs run5 min

Truth tables: predicting which jobs run#

Part III · Conditions · Chapter 18· 5 min read

A truth table lists every job down the side and every kind of pipeline your project runs across the top. Each cell says what the job does in that pipeline: runs, waits for a person, is allowed to fail, or is absent.

You derive it from the merged configuration by applying chapters 14 to 17 in order. It answers "why didn't this run?" at a glance. It also finds the jobs that run where nobody meant them to, and the checks that never run where everybody assumes they do.

The method#

Pick the scenarios: the kinds of pipeline the project really gets (chapter 14).
Apply workflow. Cross out the scenarios it refuses, and note when a scenario stops applying (chapter 15).
Find each job's final rules in the Full configuration view, after extends, !reference, templates and components have done their work (chapter 7).
Walk each job's rules for each scenario. The first match fills the cell (chapter 16).
Mark how the job runs: manual, delayed, or allowed to fail (chapter 17).
Check every column's needs. Each needed job must be in the same column, or GitLab won't create that pipeline.
Check the table against real pipelines of each kind.

Here is the whole method, applied to payments-api.

Steps 1 and 2: scenarios and workflow#

Scenario	Source	What the rules will see	MegaCorp's workflow
feature branch push	`push`	`CI_COMMIT_BRANCH` is the feature branch	allowed by rule 3, but only until a merge request is opened; after that, rule 2 refuses it
merge request	`merge_request_event`	`CI_MERGE_REQUEST_*` is set, and `CI_COMMIT_BRANCH` is not	allowed by rule 1
main	`push`	`CI_COMMIT_BRANCH` equals `CI_DEFAULT_BRANCH`	allowed by rule 3
tag v1.4.0	`push`	`CI_COMMIT_TAG` is set, and `CI_COMMIT_BRANCH` is not	allowed by rule 4
nightly on main	`schedule`	`CI_COMMIT_BRANCH` is `main`, and the source is `schedule`	allowed by rule 3

All five scenarios get a pipeline, provided at least one job is added to it.

Step 3: each job's final rules#

Most of payments-api's jobs inherit their rules from the base template:

megacorp/devops/ci-templatestemplates/base.ymlmegacorp-base

.megacorp-base:
  variables:
    MC_TEAM: unknown
  artifacts:
    expire_in: 7 days
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]
    - !reference [.rules, release-tag]

Not all of them, though. Here is where each job's final list comes from:

Job	Its rules come from	The list, in order
`maven-build`	`.megacorp-base`, through `.maven-base` and `.maven-build`	never-on-schedule, mr, default-branch, release-tag
`maven-test`	`.maven-test`, whose own list replaces the base's	never-on-schedule, mr, default-branch
`image-build`	`.megacorp-base`, through `.image-build`	never-on-schedule, mr, default-branch, release-tag
`container_scanning`	MegaCorp's `security.yml`, whose list replaces the one in GitLab's template	disabled means never; never-on-schedule; merge request event; default branch
`secret_detection`	GitLab's Secret-Detection template	disabled means never; two rules that apply only when `AST_ENABLE_MR_PIPELINES` is `"true"`; `$CI_COMMIT_BRANCH`
`sonar-scan`	the sonar-scan component	`SKIP_SONAR` means never; merge request event; default branch
`deploy-staging`	`java-service.yml`	never-on-schedule, default-branch
`deploy-prod`	`java-service.yml`	a tag matching the release pattern, as a manual job
`publish-docs`	payments-api's own file	default-branch

Two of these lists replaced a list the job would otherwise have inherited: the one on maven-test, and the one on container_scanning. Replaced lists are where most surprises in a truth table come from. For secret_detection, the rules are in a template nobody at MegaCorp wrote. The only way to see them is the Full configuration view.

Steps 4 and 5: the table#

Walk each list for each scenario, stopping at the first match:

job	feature branch push	merge request	main	tag v1.4.0	nightly on main
maven-build	not in the pipeline	runs	runs	runs	not in the pipeline
maven-test: [17] and [21]	not in the pipeline	runs	runs	not in the pipeline	not in the pipeline
secret_detection	runs, allowed to fail	not in the pipeline	runs, allowed to fail	not in the pipeline	runs, allowed to fail
sonar-scan	not in the pipeline	runs, allowed to fail	runs, allowed to fail	not in the pipeline	runs, allowed to fail
image-build	not in the pipeline	runs	runs	runs	not in the pipeline
container_scanning	not in the pipeline	runs, allowed to fail	runs, allowed to fail	not in the pipeline	not in the pipeline
deploy-staging	not in the pipeline	not in the pipeline	runs	not in the pipeline	not in the pipeline
deploy-prod	not in the pipeline	not in the pipeline	not in the pipeline	manual: waits for someone to run it	not in the pipeline
publish-docs	not in the pipeline	not in the pipeline	runs	not in the pipeline	runs

payments-api: every job, by pipeline. Derived from the rules, and checked against real pipelines on a local GitLab 19.3.·not in the pipeline✓runs!runs, allowed to fail▶manual: waits for someone to run it

Two columns take a moment's thought:

Merge request, secret_detection. None of its four rules matches. The two merge request rules need AST_ENABLE_MR_PIPELINES, which nobody set, and the last rule needs CI_COMMIT_BRANCH, which a merge request pipeline doesn't have.
Nightly, sonar-scan. Its last rule is $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH, a schedule on main matches it, and nothing in its list excludes schedules.

Step 6: needs in every column#

Job	Needs	In the same columns?
`image-build`	`maven-build`	yes: merge request, main, tag
`container_scanning`	`image-build`	yes: merge request, main
`deploy-staging`	`image-build`	yes: main
`deploy-prod`	`image-build`	yes: tag

The nightly column is safe only because container_scanning leaves schedules out, just as image-build does. Before MegaCorp added that rule, the nightly column held a scan that needed an image the column didn't have, and GitLab refused to create the pipeline at all (chapter 17's exercise).

Step 7: check it against GitLab#

The table above was checked cell by cell against pipelines that a local GitLab 19.3 created for the same five scenarios. You can do the same with less effort:

Pushes to a branch or tag can be simulated before you push. GitLab's CI Lint runs the rules as if a pipeline were being created (chapter 3).
Merge request and scheduled pipelines can't be simulated. Find a recent one of each kind in the pipeline list, by its label, and compare its jobs with your column.
If you use the API, remember that it lists trigger jobs separately from other jobs. A pipeline's jobs endpoint doesn't show deploy-staging or deploy-prod; its bridges endpoint does.

What the table says about payments-api#

Five findings, none of them visible in any single file:

Merge requests get no secret detection. Once a merge request is open, MegaCorp's workflow refuses branch pipelines, and GitLab's template skips merge request pipelines. A secret committed on a merge request branch is first scanned after the merge. GitLab's documented switch is AST_ENABLE_MR_PIPELINES: "true", typically set once for the whole group.
A feature branch with no merge request gets only secret detection. No build and no tests run until someone opens the merge request.
A release tag is built but not tested or scanned. It relies on the tag sitting on a commit that main's pipeline already tested.
The nightly pipeline doesn't build or test anything. It republishes the docs and runs two scanners.
Every "allowed to fail" cell is a scanner. A finding, or a crashed scanner, never turns these pipelines red; only a policy can make it (chapter 29).

Spot the bugThe release candidate with no pipeline#

The payments team starts tagging release candidates. They push the tag v1.5.0-rc1 on a commit of main. No pipeline appears, and GitLab shows no error. The next day, v1.5.0 on the same commit gets a pipeline as usual.

QuestionAdd a column for v1.5.0-rc1 to the table. Why is there no pipeline?

Show the answer

Workflow rule 4 lets the tag pipeline through, but every job's rules then leave it out:

maven-build and image-build reach release-tag, whose pattern /^v\d+\.\d+\.\d+$/ doesn't match -rc1.
maven-test, container_scanning, sonar-scan, deploy-staging and publish-docs have no rule for tags at all.
secret_detection needs CI_COMMIT_BRANCH, which a tag pipeline doesn't have.
deploy-prod repeats the release pattern.

The column is empty, and GitLab doesn't create a pipeline with no jobs. After a push it says nothing. To build release candidates, widen the pattern, for example to /^v\d+\.\d+\.\d+(-rc\d+)?$/, in two places: the rules library, and deploy-prod, which copies the pattern instead of using the library. See each job's final rules and when no pipeline appears.

Next · Part IV19Variables and precedence6 min

Variables and precedence#

Part IV · What a job is given · Chapter 19· 6 min read

A CI/CD variable is a name and a value that GitLab hands to each job as an environment variable. Variables come from many places:

your YAML
the settings pages of the project, its groups and the instance
whoever started the pipeline, and earlier jobs
security policies, and GitLab itself

When one name is set in several places, a fixed ladder decides which value wins. Settings beat YAML, and a value given when the pipeline started beats settings. Your YAML is near the bottom. Most "my change had no effect" problems are this ladder at work.

What a variable is#

Here is the smallest example:

variables:
  DEPLOY_ENV: staging

show-env:
  script:
    - echo "Deploying to $DEPLOY_ENV"     # prints: Deploying to staging

GitLab passes DEPLOY_ENV to the job as an environment variable: a named value that every program the script runs can read. In a script you write it as $DEPLOY_ENV. Variables can also appear in many keywords, such as image, and in rules, although rules can only see the variables that exist when the pipeline is created (chapter 14).

Variables in YAML#

A variables: block can sit at the top of a file or inside a job:

At the top, it sets a default for every job.
In a job, it sets a value for that job only, and a job's value beats a default of the same name.

Everyone who can read the repository can read these values, so YAML is for ordinary settings, never for passwords or tokens. variables: {} in a job turns every default off for that job, and inherit: variables: does the same more selectively (chapter 7).

Gotcha

A job can't read a default into a variable of the same name. MY_VAR: $MY_VAR in a job doesn't pick up the top-level MY_VAR. The job's own definition hides the default, so the value is the literal text $MY_VAR. Use a different name.

Variables in settings#

Anything secret belongs in settings, under Settings › CI/CD › Variables, on the project, on a group, or for the whole instance. Every job below that level receives the variable. Each one has a few options:

Option	What it does
Key	the name: letters, digits and underscores only
Value	up to 10,000 characters
Type	Variable, the usual kind, or File: the value is written to a temporary file, and the variable holds the file's path, for tools that need a file
Environment scope	limits the variable to jobs deploying to certain environments (Premium, on groups)
Protect variable	the variable exists only in pipelines on protected branches and tags
Visibility	Visible; Masked, which hides the value in job logs and has been the default since 18.3; or Masked and hidden, which also hides it in the settings page, for ever
Expand variable reference	lets the value refer to other variables with `$`; off by default since 18.6

Here are MegaCorp's group variables. Every project under megacorp/ gets them:

settingsgroup-megacorp.ymlgroup-variables

# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
  - key: MAVEN_CLI_OPTS
    value: "--batch-mode --errors --show-version -s .m2/settings.xml"
  - key: AWS_ACCOUNT_ID
    value: "123456789012"
  - key: SONAR_HOST_URL
    value: https://sonar.example.com
  - key: SONAR_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true

Changing a project variable needs the Maintainer role, a group variable the Owner role, and an instance variable an administrator. If you can't see a level's settings, you can't see its variables either. That is the first reason variables are hard to trace.

Who wins: the precedence ladder#

When the same name is set in more than one place, the job gets the value from the highest place on this ladder:

Figure The variable precedence ladder in GitLab 19.3. When a name is set in several places, the highest rung wins. Everything you write in .gitlab-ci.yml sits below every settings page.#

Rung	Where the value comes from	Beats
1	a pipeline execution policy	everything
2	a scan execution policy	everything below
3	pipeline variables: a manual run, a schedule, a trigger, the API, an upstream pipeline, or a manual job's form	all settings and all YAML
4	the project's CI/CD settings	groups, instance, and all YAML
5	a group's CI/CD settings; the closest subgroup wins	the instance, and all YAML
6	the instance's CI/CD settings	all YAML
7	a dotenv report written by an earlier job	your YAML
8	a job's own `variables:`, including those it gets through `extends`	top-level YAML
9	top-level `variables:`	GitLab's own values
10–11	deployment and predefined variables	nothing

Two of MegaCorp's variables show the ladder at work:

MC_TEAM is set at the top of payments-api's file (rung 9), and in .megacorp-base, a job that every template extends (rung 8). The job value wins, so every job reports unknown (chapter 7).
MAVEN_CLI_OPTS is set in payments-api's maven-test job (rung 8), and on the megacorp group (rung 5). The group wins. The option payments-api added is never used, and nothing in any file shows why.

megacorp/payments/payments-api.gitlab-ci.ymloverride

maven-test:
  variables:
    MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"

Values given when the pipeline starts#

Rung 3 holds pipeline variables: values given to one pipeline when it starts. They come from the New pipeline form, a schedule, the API, a trigger token, an upstream trigger job, or the form on a manual job. They beat every settings page, and even GitLab's predefined variables, so they are powerful and easy to misuse.

Because of that, a project can limit who may use them, with Minimum role to use pipeline variables. On self-managed GitLab the default is Maintainer; new projects on GitLab.com default to nobody. Someone below the minimum gets Insufficient permissions to set pipeline variables. The setting also covers the variables a trigger job sends downstream, including top-level ones it inherits. So a downstream project with the restriction on can refuse your trigger with:

Failed - (downstream pipeline can not be created, Insufficient permissions to set pipeline variables)

GitLab now recommends pipeline inputs instead of pipeline variables (chapter 9), because inputs are typed and checked.

When a $ is expanded#

A value can mention another variable. What happens depends on where the value was set:

In YAML, references are expanded: LS_CMD: 'ls "$FLAGS"' uses the value of FLAGS. The variables: expand keyword can turn that off for one variable.
In settings, a value is taken literally unless its Expand variable reference box is ticked. That box has been off by default since 18.6, and a masked variable can't have it.
To keep a literal dollar sign in YAML, write $$.

Inside the job#

Every variable reaches the job as an environment variable. Three details trip people up:

before_script and script share one shell; after_script doesn't. A value set with export in the script is gone by after_script, and never reaches another job.
To pass a value to a later job, write it to a dotenv report: a small file of NAME=value lines that later jobs receive as variables (rung 7; chapter 24). Rules can't see such values.
Service containers, the helper containers a job starts beside itself, get only the variables written in YAML, not those from settings (chapter 23).

Finding where a value came from#

A job log never lists where each variable came from. Work down the ladder, from the top, and stop at the first place that sets the name:

Figure Finding where a variable's value came from. Ask from the top; the first yes is the value your job sees.#

As text

Does a security policy set it? Yes: The policy's value wins; ask the security team. No: the next step.
Was it given when this pipeline started: by hand, a schedule, a trigger or the API? Yes: That value wins over every settings page. No: the next step.
Is it in the CI/CD settings of the project, a group or the instance? Yes: The closest level wins: project, then the nearest group. No: the next step.
Did an earlier job write it to a dotenv report? Yes: The dotenv value wins over your YAML. No: the next step.
Is it in the job's own variables, or in a job it extends? Yes: The job's value wins over the top-level one. No: the next step.
Otherwise it comes from top-level variables, or from GitLab itself

To look at a value without leaking it, print the variable's name and length, never a secret's value. Masked values appear in logs as [MASKED]. A command such as export lists every variable the job has, and GitLab's debug tracing lists even more (chapter 20).

Spot the bugThe retry option that never retries#

payments-api's tests are flaky, so the team told Maven to rerun failing tests twice, in their own file:

megacorp/payments/payments-api.gitlab-ci.ymloverride

maven-test:
  variables:
    MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"

The job log shows no reruns at all. The merged configuration shows their MAVEN_CLI_OPTS exactly as they wrote it.

QuestionWhy does Maven never see -Dsurefire.rerunFailingTestsCount=2, and what are the ways to fix it?

Show the answer

The megacorp group sets MAVEN_CLI_OPTS in its CI/CD settings (rung 5). Settings variables beat every value in YAML, including a job's own (rung 8). The merged configuration shows only YAML, so it can't show the override.

Either give the option another name that the template adds to the Maven command, or set a project variable MAVEN_CLI_OPTS (rung 4) containing the group's options plus the new one. The cleanest fix is for the group owners to stop setting a value that projects are meant to adjust. See who wins.

Next20Secrets6 min

Secrets#

Part IV · What a job is given · Chapter 20· 6 min read

A secret is a value that must not leak: a password, a token, a private key. There are three ways to get one into a job:

a CI/CD variable set in settings, which can be protected, masked or hidden
the secrets: keyword, which fetches the value from a secrets manager as the job starts (Premium)
the job logging in to a secrets manager or cloud itself, with an ID token (chapter 21)

Each has limits. Most "the secret is empty" problems come from protection rules deciding that this pipeline doesn't get the value. Most leaks come from a script printing the value in a form that masking doesn't recognise.

Three ways a secret reaches a job#

Way	Where the value lives	How the job gets it	Tier
A CI/CD variable	the settings of the project, a group or the instance	as an environment variable, or a file if its type is File	all
The `secrets:` keyword	a secrets manager: HashiCorp Vault, AWS, Google Cloud or Azure, or GitLab's own Secrets Manager	GitLab fetches it as the job starts, and hands it over as a file by default	Premium
The job logs in itself	anywhere that trusts GitLab's ID tokens	the script exchanges an ID token for access, then reads what it needs	all

Settings variables are the commonest, and the easiest to get wrong, so most of this chapter is about them. Never put a secret in .gitlab-ci.yml: everyone who can read the repository can read it.

Protect, mask, hide: what each one does#

A settings variable has three safety options. They protect against different things, and none of them protects against everything:

Option	What it does	What it doesn't do
Protect	the value is delivered only to pipelines running on protected branches and tags	stop code in those pipelines from using or sending the value
Mask	the exact value is replaced by `[MASKED]` in job logs	catch the value if a program prints it changed, for example with a backslash added before a special character
Mask and hide	as masked, and nobody can read the value in the settings page again	anything more at run time; it can only be chosen when the variable is created
File type	the value is written to a temporary file, and the variable holds its path	stop a script that reads the file from printing it

GitLab's own documentation is blunt about masking: it "is not a guaranteed way to prevent malicious users from accessing variable values". A masked value must also be a single line with no spaces and at least 8 characters, which rules out most private keys.

Who gets a protected value#

Protection decides which pipelines receive a value at all. This is the rule behind most "works on main, empty on my branch" problems:

Pipelines on protected branches and tags get protected values. MegaCorp protects main and tags matching v*.
Pipelines on any other branch don't. The variable is simply not set.
Merge request pipelines get them only when all four of these are true:
- the source and target branches are both protected
- both branches are in the same project
- the person running the pipeline can push or merge to the target
- the project setting Allow merge request pipelines to access protected variables and runners is on
A merge request from an ordinary feature branch never qualifies.
Pipelines from forks run in the fork and get none of the parent project's variables. A parent-project member can choose to run a fork's merge request pipeline in the parent instead. That pipeline gets the parent's variables, though not protected ones, and runs the fork's pipeline file. GitLab shows a warning that must be accepted first.

payments-api protects main and tags matching v*, and keeps one secret of its own:

megacorp/payments/payments-apiSETTINGS.ymlprotection

protected_branches: [main]
protected_tags: ["v*"]

megacorp/payments/payments-apiSETTINGS.ymlproject-variables

project_variables:
  - key: MC_NEXUS_URL
    value: https://nexus.example.com/repository/maven-public/
  - key: PAYMENTS_DB_PASSWORD
    value: "(set in the UI)"
    masked: true
    protected: true

The megacorp group adds one more secret, for every project:

settingsgroup-megacorp.ymlgroup-variables

# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
  - key: MAVEN_CLI_OPTS
    value: "--batch-mode --errors --show-version -s .m2/settings.xml"
  - key: AWS_ACCOUNT_ID
    value: "123456789012"
  - key: SONAR_HOST_URL
    value: https://sonar.example.com
  - key: SONAR_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true

SONAR_TOKEN is protected. sonar-scan runs in every merge request pipeline, and feature branches aren't protected, so in those pipelines the token is empty. The scan can't sign in to the Sonar server. The job is allowed to fail, so the merge request stays green, and nobody notices that the quality gate never ran.

The secrets: keyword Premium#

With secrets:, the value never sits in GitLab's settings at all. GitLab fetches it from a secrets manager when the job starts:

deploy:
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://vault.example.com
  secrets:
    DATABASE_PASSWORD:
      vault: production/db/password@ops     # engine ops, secret production/db, field password
  script:
    - ./deploy.sh --password-file "$DATABASE_PASSWORD"

Four things to know when you read one:

It arrives as a file. The variable holds the path to a temporary file containing the value, which is why the script above passes it as --password-file. Write file: false under the variable name to get the value itself.
It needs a way to prove who is asking, usually an ID token (chapter 21). For Vault, a job's only ID token is used automatically. With several, name one with token:, or which is used can't be predicted. AWS Secrets Manager looks for a token named AWS_ID_TOKEN, or uses the runner's own AWS role (chapter 31).
The provider decides who may read. A Vault role, for example, lists bound claims: values the ID token must carry, such as the project's ID or ref_protected: "true". GitLab's documentation warns that a role not tied to a project or group accepts a token from any job on the instance.
The providers are HashiCorp Vault, AWS Secrets Manager, Google Cloud Secret Manager, Azure Key Vault, and GitLab's own Secrets Manager.

GitLab Secrets Manager Beta is GitLab's own store, under Secure › Secrets manager, and has been in public beta since 19.0. Jobs read it with secrets: … gitlab_secrets_manager:, which needs GitLab Runner 19.0 or later. A secret can be limited to a branch or an environment. Its value is masked if a job prints it, and unlike a masked variable it can contain spaces and new lines. Switching the manager off deletes every secret in it, permanently.

How secrets leak anyway#

None of the options above stops a determined script. These are the usual ways values escape:

Printing them changed. Base64-encoding a token, or a shell adding escape characters, produces text that masking doesn't recognise.
Debug tracing. Setting CI_DEBUG_TRACE: "true" writes every variable the job has into the log. Masked values still show as [MASKED], but everything else appears in full. Only Developers and above can read such logs, unless the variable was set in a runner's own configuration, where the check doesn't apply.
Listing the environment. export, env or printenv print every variable. Masked values show as [MASKED], but unmasked ones don't. File-type variables show only a path.
Artifacts and dotenv files. Anything a job writes into an artifact can be downloaded by people who can see the pipeline.
Changed pipeline files. A merge request can change .gitlab-ci.yml to send a variable anywhere. Protection limits which pipelines have the value, but review of pipeline changes is what actually stops this.

When a secret is empty#

Work through these questions in order. The first yes is almost always the cause:

Figure A secret is empty, or wrong, in one pipeline but fine in another. Ask these in order.#

As text

Is the variable protected, and this branch or tag not? Yes: Protected values reach only protected branches and tags. No: the next step.
Is this a merge request pipeline? Yes: Both branches must be protected, and a setting must allow it. No: the next step.
Is the pipeline running in a fork? Yes: Forks don't receive the parent project's variables. No: the next step.
Does the variable have an environment scope? Yes: Only jobs deploying to that environment receive it. No: the next step.
Does the job get it through secrets:? Yes: Check the ID token and the provider's role (chapter 21). No: the next step.
Check the name matches exactly, and that a higher rung doesn't override it (chapter 19)

Spot the bugThe quality gate that never ran#

MegaCorp's platform team notices that Sonar has had no analysis of any payments-api merge request for months. Analyses of main arrive after every merge. In merge request pipelines, sonar-scan shows an orange warning, and its log ends in an authentication error.

settingsgroup-megacorp.ymlgroup-variables

# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
  - key: MAVEN_CLI_OPTS
    value: "--batch-mode --errors --show-version -s .m2/settings.xml"
  - key: AWS_ACCOUNT_ID
    value: "123456789012"
  - key: SONAR_HOST_URL
    value: https://sonar.example.com
  - key: SONAR_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true

QuestionWhy does the scan fail only in merge request pipelines, and why has nobody noticed?

Show the answer

SONAR_TOKEN is a protected group variable. Protected values reach only pipelines on protected branches and tags. A merge request pipeline from a feature branch gets them only if both branches are protected, which a feature branch isn't. So the token is empty, and the scanner can't authenticate.

Nobody noticed because sonar-scan has allow_failure: true: a failure turns the job orange, not the pipeline red. The owners of the token must decide whether an unprotected token, a separate token for merge requests, or a different design is acceptable. See who gets a protected value.

Next21Identity and tokens7 min

Identity and tokens#

Part IV · What a job is given · Chapter 21· 7 min read

Many jobs must prove who they are before another system lets them in: GitLab's own API and registries, AWS, or a secrets manager. A job can do that in four ways:

the job token, which GitLab creates for every job, for calls back to GitLab
an ID token, a signed note about the job that other systems can check
a stored token, such as a project access token, kept in a CI/CD variable
the runner's own identity, which every job on that runner shares

The first two are made fresh for each job and stop working when it ends. Stored tokens expire on a date nobody remembers, which is why a pipeline can break with no commit to blame.

Four ways a job proves who it is#

Figure The four identities a job can use, and what each one reaches. The job token and ID tokens are made for each job. Stored tokens and the runner's role last much longer.#

Identity	Made by	Works until	The usual failure
The job token, `CI_JOB_TOKEN`	GitLab, for every job	the job ends	`404 Not Found` from a project that hasn't allowed yours
An ID token	GitLab, when the job asks with `id_tokens:`	the job's timeout, or 5 minutes	the other system refuses it, because its details don't match the trust rule
A stored token in a variable	a person, once	its expiry date, often a year after it was made	an authentication error, starting the day it expires
The runner's own role	the platform team	someone changes the runner	every project on that runner has the same access

When a call from a job is refused, first work out which of the four it used. Each fails for different reasons, and each is fixed in a different place.

The job token: calling back to GitLab#

Every job gets a variable called CI_JOB_TOKEN. GitLab creates it as the job starts and cancels it when the job finishes. It lets the job call GitLab as the person who started the pipeline, but it reaches far less than that person could:

the container, package and Terraform module registries
downloading job artifacts, and reading files from a repository
releases, environments and deployments
cloning a repository, with gitlab-ci-token as the user name
starting a pipeline in another project through the API

It doesn't work with GitLab's GraphQL API, and GitLab masks it in job logs. Here a job downloads a package that another project published:

fetch-test-data:
  script:
    - >
      curl --fail --header "JOB-TOKEN: $CI_JOB_TOKEN" --output data.zip
      "$CI_API_V4_URL/projects/megacorp%2Fdevops%2Fci-tools/packages/generic/test-data/1.0.0/data.zip"

Calling another project needs that project's permission. Each project keeps a job token allowlist: the groups and projects whose job tokens it accepts. At first it holds only the project itself. So for the job above to work:

ci-tools must list payments-api, or the megacorp group, under Settings › CI/CD › Job token permissions.
The person who started the pipeline must be a member of ci-tools, with a role that allows the action. Being on the allowlist grants nothing by itself.

Public and internal projects are a partial exception. Some of their features, such as artifacts and registries, answer any project's job token unless the project limits them to members.

A refusal rarely says "not allowed". GitLab answers 404 Not Found, as if the project didn't exist. A refused clone says:

remote: The project you were looking for could not be found or you don't have permission to view it.

Figure A job's call to another GitLab project fails with 404. Ask these in order.#

As text

Is the call made with CI_JOB_TOKEN? No: The job token isn't involved; see stored tokens below. Yes: the next step.
Is your project, or its group, missing from the target's allowlist? Yes: A Maintainer of the target adds it under Job token permissions. No: the next step.
Is the person who started the pipeline not a member of the target? Yes: The job token acts as that person, so they need access. No: the next step.
Is it a GraphQL call, or an API the job token can't reach? Yes: That call needs a different identity. No: the next step.
Check the job was still running; the token stops working when the job ends

ID tokens: a signed note about the job#

Systems outside GitLab, such as AWS and Vault, don't accept the job token. For them, a job asks GitLab for an ID token. This is a short piece of text, called a JWT, that lists facts about the job and carries GitLab's signature. The receiver checks the signature against keys that GitLab publishes, so nobody can forge or change the facts. A job asks for one with id_tokens::

megacorp/devops/ci-templatestemplates/container.ymlimage-build

.image-build:
  extends: .megacorp-base
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
    IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
    - buildah push "$IMAGE:$CI_COMMIT_SHA"
    - echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
  artifacts:
    reports:
      dotenv: build.env

MC_ID_TOKEN becomes a variable that holds the token. aud, short for audience, says who the token is for, and a receiver can refuse a token meant for someone else. MegaCorp's mc_aws_login function then hands the token to AWS (chapter 31).

The facts in a token are called claims. These matter most. The examples are for image-build in a payments-api pipeline on main:

Claim	Example	What it says
`sub`	`project_path:megacorp/payments/payments-api:ref_type:branch:ref:main`	the subject: project, kind of ref, and ref, in one string
`aud`	`https://gitlab.example.com`	who the token is for, from `aud:` in the YAML
`project_path`	`megacorp/payments/payments-api`	the project running the job
`ref`, `ref_type`	`main`, `branch`	the branch or tag; in a merge request pipeline, the source branch
`ref_protected`	`true`	whether that branch or tag is protected
`pipeline_source`	`push`	what started the pipeline
`environment`	`production`	the environment, only when the job has one
`exp`	a time	when the token expires

A probe job on the local GitLab 19.3 showed the same shape in four pipelines. On main, ref_protected was "true", as a string. On a feature branch it was "false", and the branch's merge request pipeline had the same sub and ref, with pipeline_source set to merge_request_event. A tag put ref_type:tag in sub.

A token expires when the job's timeout runs out, or after 5 minutes if no timeout is specified. In the probe, each token lasted one hour, the default job timeout. So exchange it early. MegaCorp logs in to AWS in before_script, before the long build starts.

How AWS checks an ID token#

Here is the whole exchange for MegaCorp's image push. GitLab signs the token, and the job sends it to AWS's token service, STS. STS makes three checks before it hands back keys that last one hour:

Figure The identity flow. The job trades its ID token for temporary AWS keys, then pushes the image to ECR. Below: how the trust policy's subject pattern matches three of payments-api's pipelines.#

The third check is where most refusals happen. A role's trust policy says which subjects may use it. MegaCorp's push role accepts any branch of any project under megacorp/:

awsiam.ymlecr-push-role

# Assumed by image-build, and by the ecr-push component, through mc_aws_login.
ecr_push_role:
  arn: arn:aws:iam::123456789012:role/gitlab-ecr-push
  trust_policy:
    Effect: Allow
    Principal:
      Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
    Action: sts:AssumeRoleWithWebIdentity
    Condition:
      StringEquals:
        gitlab.example.com:aud: https://gitlab.example.com
      StringLike:
        gitlab.example.com:sub: project_path:megacorp/*:ref_type:branch:ref:*
  permissions: push and pull images in every ECR repository in the account

Read the pattern against each pipeline's sub:

A pipeline on main matches.
A merge request pipeline matches too, because its ref is the source branch, such as feature/retry.
A pipeline for tag v1.4.0 doesn't match. Its subject says ref_type:tag, and the pattern demands branch.

On self-managed GitLab, AWS can check only sub and aud. On GitLab.com it can also check claims such as project_id and ref_protected.

Stored tokens, and the day they expire#

Some jobs use a token that a person made once and saved as a masked CI/CD variable. MegaCorp's deploy tool pushes to the GitOps repository with one:

megacorp/devops/ci-toolsbin/mcdeploy

deploy)
  env="${1:?usage: mc deploy <environment> <image-ref>}"
  image="${2:?usage: mc deploy <environment> <image-ref>}"
  mc_log "promoting $image to $env through the GitOps repository"
  git clone --depth 1 \
    "https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
  yq -i ".apps.\"${APP}\".image = \"${image}\"" "gitops/envs/${env}/values.yaml"
  git -C gitops commit -am "deploy ${APP} ${image} to ${env}"
  git -C gitops push origin HEAD:main
  ;;

megacorp/platform/deployerSETTINGS.ymlproject-variables

project_variables:
  - key: GITOPS_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true
    # A project access token created in megacorp/platform/gitops-config, with the
    # Maintainer role and the write_repository scope. Nobody entered an expiry date,
    # so GitLab set one 365 days after the token was created.

GitLab has several kinds of stored token:

Token	Acts as	Expires
Personal access token	one person, with that person's access	on its expiry date, 365 days after creation if none was entered
Project or group access token	a bot user that GitLab creates for the project or group	the same as a personal access token
Deploy token	a project or group, for repositories and registries	never, unless a date was set

Access tokens stop working at midnight UTC on their expiry date. The job that uses one runs unchanged and then fails with an authentication error. Nothing in the repository changed, so nothing in its history explains the failure. GitLab emails the Maintainers and Owners of the token's project 60, 30 and 7 days before a project access token expires. They aren't always the people who look after the pipeline that uses it.

Note

A deploy token named exactly gitlab-deploy-token reaches the project's jobs automatically, as CI_DEPLOY_USER and CI_DEPLOY_PASSWORD. No variable needs to be set.

The runner's own role#

The machine or pod that runs a job can have an identity of its own. MegaCorp's shared runner starts every job as a Kubernetes service account that AWS maps to a role:

runnersconfig.tomlrunner

[[runners]]
  name = "megacorp-eks-shared"
  # registered in GitLab as a group runner of megacorp, with the tag megacorp-shared
  url = "https://gitlab.example.com"
  executor = "kubernetes"
  environment = ["MAVEN_OPTS=-Xmx2g", "HTTPS_PROXY=http://proxy.example.com:3128"]
  pre_build_script = "echo 'runner: megacorp-eks-shared'"
  [runners.kubernetes]
    namespace = "gitlab-runners"
    image = "registry.example.com/megacorp/devops/ci-tools:3.2"
    # jobs run as this service account, which is bound to an AWS IAM role
    service_account = "gitlab-runner-jobs"

awsiam.ymlrunner-role

# The role behind the gitlab-runner-jobs service account on the megacorp-eks-shared
# runner. Every job on that runner can use it without asking. The EKS cluster is what
# AWS trusts for this role, not GitLab.
runner_role:
  arn: arn:aws:iam::123456789012:role/gitlab-runner-jobs
  permissions: pull images from every ECR repository in the account

Any job on that runner can use this role without asking. The access is the same for every project, it appears in no .gitlab-ci.yml, and the platform team controls it. When a job's AWS calls act as the wrong identity, run aws sts get-caller-identity. It prints the role the job is really using, and needs no permissions (chapter 31).

Spot the bugDeploys that stopped on a Tuesday#

No MegaCorp service has deployed since Tuesday morning. Every deployer pipeline fails in mc deploy, at the clone of the GitOps repository, with an authentication error. Nothing was merged into deployer, ci-tools or gitops-config for a week, and the same pipelines deployed on Monday.

megacorp/platform/deployerSETTINGS.ymlproject-variables

project_variables:
  - key: GITOPS_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true
    # A project access token created in megacorp/platform/gitops-config, with the
    # Maintainer role and the write_repository scope. Nobody entered an expiry date,
    # so GitLab set one 365 days after the token was created.

QuestionWhat most likely changed, why does the history show nothing, and what would stop it happening again?

Show the answer

GITOPS_TOKEN is a project access token. Access tokens stop working at midnight UTC on their expiry date, which is 365 days after creation unless someone chose another. The clone sends the expired token, and GitLab refuses it. No file changed, so no commit explains the failure.

Create a new token and update the variable. To stop it recurring, track the expiry date. Also make sure the people GitLab emails, the Maintainers and Owners of gitops-config, know that the deploys depend on this token. See stored tokens.

Next · Part V22Runners and executors4 min

Runners and executors#

Part V · Execution and data · Chapter 22· 4 min read

GitLab only schedules jobs; a runner runs them. Which runner takes a job depends on three things:

which runners the project may use
whether a runner has every tag the job asks for
whether a protected runner accepts this branch

Once a runner has the job, the runner's own configuration shapes what happens, and your project can't see it. A job stuck in pending is almost always a problem with the first part. A job that behaves differently from run to run is often the second.

Which runners a project can use#

Every runner belongs to one of three scopes:

Scope	Available to	Which waiting job it takes next
Instance runner	every project on the instance, unless a project or group turns instance runners off	a fair-usage queue: jobs from projects with the fewest jobs already running come first
Group runner	every project and subgroup in its group	the job that has waited longest
Project runner	only the projects it has been turned on for; a fork doesn't get it	the job that has waited longest

To see the runners your project can use, open Settings › CI/CD and expand Runners. Each runner shows its tags and its status. A runner is online if it has contacted GitLab in the last two hours, offline after that, and stale after seven days.

MegaCorp has one shared runner, megacorp-eks-shared. It is a group runner of megacorp with the tag megacorp-shared, and every job asks for that tag through default: in the base template (chapter 7).

How a job finds a runner#

When a job's turn comes, it waits in pending until a runner that fits asks for work. Work through these checks in order:

Figure Why is my job stuck in pending? Ask these in order.#

As text

Is any runner available to the project online? No: Nothing can take the job; check the runners' status. Yes: the next step.
Does the job list tags that no online runner has all of? Yes: A runner needs every tag the job lists. No: the next step.
Does the job list no tags, and no runner takes untagged jobs? Yes: Add a tag, or tick Run untagged jobs on a runner. No: the next step.
Are the only runners that fit protected, and this branch isn't? Yes: Protected runners take jobs only from protected branches and tags. No: the next step.
Are the runners that fit paused, or already running as many jobs as they allow? Yes: The job waits its turn; it isn't stuck. No: the next step.
A runner takes the job, and it moves to running

The job's page says which problem GitLab suspects. These are its messages in GitLab 19.3:

Message on the job page	What it usually means
`This job is stuck because of one of the following problems. There are no active runners online, no runners for the protected branch, or no runners that match all of the job's tags:`	the job has tags, which the page lists after the message
`This job is stuck because the project doesn't have any runners online assigned to it.`	the project's runners exist, but none is online
`This job is stuck because you don't have any active runners that can run this job.`	no runner available to the project can take this job

Note

A protected runner can also take jobs from merge request pipelines, under the same four conditions as protected variables (chapter 20).

What a runner does with a job#

A runner takes every job through the same steps. Some come from your YAML, some from the runner's configuration, and the runner does the rest by itself:

Figure The steps a runner takes for every job, from top to bottom. The steps marked config.toml come from the runner's configuration, which your project can't see. The job timeout covers every step.#

Two things surprise people:

after_script runs in a new shell. Variables exported in script are gone. It gets 5 minutes, unless RUNNER_AFTER_SCRIPT_TIMEOUT gives it another limit.
The runner can add commands of its own. pre_build_script runs just before your before_script, in the same shell, and nothing in your project shows it.

What the runner's configuration decides#

A runner's settings live in a file called config.toml, on the runner's machine. This is MegaCorp's:

runnersconfig.tomlrunner

[[runners]]
  name = "megacorp-eks-shared"
  # registered in GitLab as a group runner of megacorp, with the tag megacorp-shared
  url = "https://gitlab.example.com"
  executor = "kubernetes"
  environment = ["MAVEN_OPTS=-Xmx2g", "HTTPS_PROXY=http://proxy.example.com:3128"]
  pre_build_script = "echo 'runner: megacorp-eks-shared'"
  [runners.kubernetes]
    namespace = "gitlab-runners"
    image = "registry.example.com/megacorp/devops/ci-tools:3.2"
    # jobs run as this service account, which is bound to an AWS IAM role
    service_account = "gitlab-runner-jobs"

These settings change jobs most often:

Setting	What it does to your job
`executor`	where the job runs: a Kubernetes pod here; elsewhere a container, or the machine itself
`environment`	adds or overwrites environment variables in every job; here, a Maven memory setting and a proxy
`pre_build_script`	runs before your `before_script`, in the same shell
`image`	the image for jobs that don't set one
`output_limit`	the largest log the runner sends, 4096 KB by default; the rest is cut off
`concurrent` and `limit`	how many jobs run at once; the others wait in `pending`
`allowed_images`, `privileged`, `pull_policy`	which images jobs may use, whether containers get extra powers, and when images are downloaded again
CPU and memory settings	how much the job's container gets, on Kubernetes

Executors#

The executor decides where a job's commands run, which changes what a job can rely on:

Executor	Each job runs	A clean start for every job?
Kubernetes	in a new pod in a cluster: one container for the job, and one for each service	yes
Docker	in a new container, from the job's image	yes
Docker Autoscaler	like Docker, on machines created on demand	yes
Instance	on a whole machine created on demand, with no container	it depends on the setup
Shell	directly on the runner's machine	no: files and tools from earlier jobs may still be there

The Shell, SSH, VirtualBox, Parallels and Custom executors are in maintenance mode: they get security fixes, but no new features. The shell and instance executors don't support image: or services:. A job that works on one runner but not on another often landed on runners with different executors.

Time limits#

Three limits can stop a job:

The job timeout. This is the job's own timeout:, or the project's setting if the job has none. A runner's maximum job timeout cuts it down when the runner's is shorter.
RUNNER_SCRIPT_TIMEOUT, a job variable that stops script early, so artifacts can still upload before the job timeout.
RUNNER_AFTER_SCRIPT_TIMEOUT, the same for after_script. Its default is 5 minutes.

So a job with timeout: 2h, on a runner whose maximum is 30 minutes, stops after 30 minutes.

Spot the bugThe job that waits for ever#

A payments-api developer copies a job from a blog post into the project's .gitlab-ci.yml:

integration-test:
  stage: test
  tags: [docker]
  image: maven:3.9-eclipse-temurin-21
  script:
    - mvn verify -Pintegration

Every other job runs. integration-test stays in pending, and its page says it is stuck, listing the tag docker.

QuestionWhy does no runner take it, and what is the smallest fix?

Show the answer

A job's own tags replace the default: tags completely. So this job asks only for docker, and no longer for megacorp-shared. MegaCorp's only runner has the tag megacorp-shared, and a runner must have every tag a job lists. No runner qualifies, so the job waits for ever.

Delete the tags line, so that the job inherits megacorp-shared again. It still runs in a container, because the runner uses the Kubernetes executor. See how a job finds a runner.

Next23Images, services and Docker5 min

Images, services and Docker#

Part V · Execution and data · Chapter 23· 5 min read

Most jobs run inside a container. The job's image: decides which tools its script can use. services: start extra containers beside it, such as a database for tests. Building container images inside a job is a problem of its own, because the classic way, Docker-in-Docker, needs a privileged runner.

The usual surprises are an image whose entrypoint swallows the script, a service that never gets its password, and a registry login that happens in the wrong place.

The job's image#

image: names the container image a job runs in. It can be a name, which means the latest tag, a name and a tag, or a name and a digest:

unit-test:
  image: node:22          # a name and a tag; with no registry named, from Docker Hub
  script:
    - npm test

MegaCorp's base template sets default: image to its toolbox image, ci-tools (chapter 12). Four things to know about any image:

It needs a shell. The image must contain sh or bash, and grep, because the runner sends the script to the container's shell.
Its entrypoint must not get in the way. The Docker executor starts the container with the image's own entrypoint and hands it a shell. An image whose entrypoint runs a program, such as amazon/aws-cli, never reaches your script, and entrypoint: [""] removes it. The Kubernetes executor ignores entrypoints unless the runner turns on FF_KUBERNETES_HONOR_ENTRYPOINT. So the same image can work on one runner and fail on another.
Scripts run in /builds/ followed by the project's path, not in the image's own working directory.
The runner decides when to download it again. Its pull policy is always, if-not-present or never. A job can ask for one with image: pull_policy:, if the runner allows it.

aws-version:
  image:
    name: amazon/aws-cli
    entrypoint: [""]      # needed on the Docker executor
  script:
    - aws --version

Images from private registries#

The runner downloads the image before the job starts, so the job's script can't log in for it. The runner looks for registry credentials in this order:

a config.json file in /root/.docker on the runner
a DOCKER_AUTH_CONFIG CI/CD variable
a DOCKER_AUTH_CONFIG set in the runner's config.toml
a config.json in the home directory of the user that runs the runner

Images in the same GitLab's container registry need none of this: the runner uses the job token (chapter 21). The person who started the pipeline needs at least the Reporter role in the image's project. That project must also let your project's job token in, which is off by default.

Every pull from Docker Hub counts against Docker Hub's rate limits. GitLab's dependency proxy is a cache in front of it. Put a prefix on the image name, and the runner signs in to the proxy by itself:

unit-test:
  image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/node:22

Services: containers beside the job#

services: starts more containers for the length of the job, most often a database for tests. They start before before_script, and stop when the job ends, even if it fails:

integration-test:
  image: maven:3.9-eclipse-temurin-21
  services:
    - name: postgres:16
      alias: db
  variables:
    POSTGRES_DB: payments_test
    POSTGRES_PASSWORD: test-only-password   # the database lives only for this job
  script:
    - mvn verify -Ddb.url=jdbc:postgresql://db:5432/payments_test

Figure What runs beside a job. The job's container reaches each service by its hostname, and every container shares /builds. Variables from settings reach only the job's container.#

What trips people up:

A service is another machine on the network. Connect to its hostname, such as db, not to a local socket. On Kubernetes, the services share the job's network, so localhost works too, but two services can't use the same port.
A service adds no programs to the job. Listing node:22 as a service doesn't give your script a node command.
The hostname comes from the image name, unless you set an alias. postgres:16 is reachable as postgres.
Services get only YAML variables. GitLab doesn't trust service containers by default, so variables from settings don't reach them. To pass one, re-assign it in the YAML under a different name.
A job's services replace default: services, like every list (chapter 7).
A service that fails to start leaves a line in the job log: *** WARNING: Service XYZ probably didn't start properly. Setting CI_DEBUG_SERVICES: "true" adds the services' own logs, but can reveal masked values.

Building images inside a job#

A job that runs docker build needs a Docker daemon, and the job's container doesn't have one. There are four ways round that, and the runner decides which of them are possible:

Method	How it works	Privileged runner?	The catch
Docker-in-Docker	a `docker:dind` service runs a daemon for the job	yes	privileged containers can break out to the host; no layer cache between jobs
Socket binding	the runner mounts the host's Docker socket into the job	no	the job controls the host's daemon, and can remove other jobs' containers
Buildah	builds images with no daemon at all	no	a different command line from `docker`
Rootless BuildKit	Docker's own build engine, run without a daemon	no	the runner must still allow the system calls it uses

Figure How can this job build a container image? It depends on the runner.#

As text

Does the runner run privileged containers? Yes: Docker-in-Docker works: a docker:dind service, with TLS. No: the next step.
Does the runner mount the host's Docker socket? Yes: docker build works, but the job controls the host's daemon. No: the next step.
Does the runner allow the system calls that rootless builds need? Yes: Rootless BuildKit, or Buildah. No: the next step.
Rootless Buildah, which GitLab suggests when the runner can't be changed

kaniko, once the usual answer, is no longer maintained, and GitLab has marked its page as removed. MegaCorp builds with Buildah, so no job needs a Docker daemon:

megacorp/devops/ci-templatestemplates/container.ymlimage-build

.image-build:
  extends: .megacorp-base
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
    IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
    - buildah push "$IMAGE:$CI_COMMIT_SHA"
    - echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
  artifacts:
    reports:
      dotenv: build.env

The job runs in a Buildah image rather than the toolbox. It logs in to ECR with the job's ID token (chapter 21), builds, pushes, and writes the image's name to a dotenv report for later jobs (chapter 24).

Docker-in-Docker, set up correctly#

GitLab recommends Docker-in-Docker with TLS. The runner must be privileged, and must share a certificates directory with the service. The job then looks like this:

build-image:
  image: docker:24.0.5-cli
  services:
    - docker:24.0.5-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .

On the Kubernetes executor, also set DOCKER_HOST: tcp://docker:2376, DOCKER_TLS_VERIFY: 1 and DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client". The image's entrypoint would normally set them, but that executor doesn't run it. Without TLS, the port is 2375 and DOCKER_TLS_CERTDIR is empty.

Pin the image versions, because docker:latest can change under you. Log in to registries in before_script: the daemon in the service is new for every job, and doesn't know the runner's credentials.

Spot the bugThe test database that never started#

payments-api adds integration tests with a PostgreSQL service. Following the advice in chapter 20, the team keeps the database password out of the YAML. They save it in the project's settings as a masked variable named POSTGRES_PASSWORD:

integration-test:
  image: maven:3.9-eclipse-temurin-21
  services:
    - name: postgres:16
      alias: db
  script:
    - mvn verify -Ddb.url=jdbc:postgresql://db:5432/postgres

The job log warns that the service probably didn't start properly, and the tests can't connect to db. Someone adds POSTGRES_PASSWORD: $POSTGRES_PASSWORD to the job's variables, and nothing changes.

QuestionWhy does the database get no password, and why didn't the added line help?

Show the answer

Variables from settings don't reach service containers, which GitLab doesn't trust by default. The postgres image needs POSTGRES_PASSWORD to start, so it doesn't.

A re-assignment only works under a different name. POSTGRES_PASSWORD: $POSTGRES_PASSWORD uses the same name, so it isn't expanded. This is the trap from chapter 19. Rename the setting, for example to TEST_DB_PASSWORD, and write POSTGRES_PASSWORD: $TEST_DB_PASSWORD in the YAML. Or, because this database lives only for one job, a fixed test-only password in the YAML protects nothing worth hiding. See services.

Next24Artifacts, cache and dotenv5 min

Artifacts, cache and dotenv#

Part V · Execution and data · Chapter 24· 5 min read

Every job starts in a fresh place, so anything a job makes is lost unless it is saved. GitLab has two ways to keep files, and they are easy to confuse:

Artifacts are a job's results, such as a built .jar. GitLab stores them and hands them to later jobs in the same pipeline. They are guaranteed.
A cache is a speed-up, such as downloaded libraries. The runner may reuse it in later jobs and pipelines, but it can be missing at any time.

A third tool, the dotenv report, passes small values, such as an image name, from one job to later ones as variables.

Two ways to keep files#

Figure Artifacts versus cache. Artifacts carry results forward inside one pipeline, and always arrive. A cache carries speed-ups to later runs, and may miss.#

The rule that follows: never rely on the cache for something a job must have. If a later job needs a file, make it an artifact. If losing the file only makes a job slower, cache it.

Artifacts#

MegaCorp's Maven templates keep both kinds of artifact. maven-build keeps the built jar, and maven-test keeps a test report:

megacorp/devops/ci-templatestemplates/java-maven.ymlmaven-build

.maven-build:
  extends: .maven-base
  stage: build
  script:
    - mvn $MAVEN_CLI_OPTS -DskipTests package
  artifacts:
    paths: [target/*.jar]

megacorp/devops/ci-templatestemplates/java-maven.ymlmaven-test

.maven-test:
  extends: .maven-base
  stage: test
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JDK}
  parallel:
    matrix:
      - JDK: ["17", "21"]
  script:
    - mvn $MAVEN_CLI_OPTS verify
  artifacts:
    when: always
    reports:
      junit: target/surefire-reports/TEST-*.xml
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]

Keyword	What it decides	Default
`artifacts:paths`	which files to keep, relative to the project directory	nothing
`artifacts:when`	keep them when the job succeeds, fails, or always	`on_success`
`artifacts:expire_in`	how long GitLab keeps them	the instance's setting, but the latest successful pipeline's are kept anyway
`artifacts:reports`	files that GitLab reads and shows, such as test results	uploaded even when the job fails
`artifacts:access`	who can download them in the UI and API	`all`
`dependencies`	which earlier jobs' artifacts a job downloads; `[]` means none	every job in earlier stages

Three behaviours cause most artifact surprises:

A failed job keeps nothing, unless artifacts:when says otherwise. That is why maven-test has when: always: the output of a failed test run is the output you want. Reports are uploaded even without it.
needs and dependencies narrow the download. A job with either gets artifacts only from the jobs it names (chapter 17).
Artifacts expire. Every job that extends MegaCorp's .megacorp-base keeps them for 7 days. A job that needs expired artifacts fails before its script starts.

That failure, and the log line of a job that saved nothing, look like this:

This job could not start because it could not retrieve the needed artifacts.
No files to upload

The first means a missing dependency, expired artifacts or missing permissions. The second, in the earlier job's log, means its path was wrong, or the file was never made.

Cache#

MegaCorp's Maven jobs cache the local Maven repository, so that each job doesn't download every library again:

megacorp/devops/ci-templatestemplates/java-maven.ymlmaven-base

.maven-base:
  extends: .megacorp-base
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
  variables:
    JAVA_VERSION: "21"
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  cache:
    key:
      files: [pom.xml]
    paths: [.m2/repository]
  before_script:
    # a job's own before_script replaces default:before_script, so re-add the functions
    - !reference [.snippets, functions]
    - !reference [.snippets, maven_settings]

The key decides which saved cache a job gets. Jobs with the same key share one cache, in this pipeline and in later ones. Here the key is made from the contents of pom.xml, so a change to the dependencies starts a new cache.

Keyword	What it decides	Default
`cache:key`	which cache to use; `key: files:` makes one from up to two files' contents	`default`
`cache:paths`	which files to save and restore	none, so nothing is cached
`cache:policy`	`pull` only restores, `push` only saves, `pull-push` does both	`pull-push`
`cache:when`	save on success, on failure, or always	`on_success`
`cache:fallback_keys`	other keys to try when the key has no cache yet	none
`cache:unprotect`	share caches between protected and unprotected branches	`false`

MegaCorp's Node templates use the policy well. The base sets pull, so jobs only read the cache, and the build job alone saves it:

megacorp/devops/ci-templatestemplates/node.ymlnode-base

.node-base:
  extends: .megacorp-base
  image: registry.example.com/megacorp/devops/ci-tools/node:22
  cache:
    key:
      files: [package-lock.json]
    paths: [.npm/]
    policy: pull
  before_script:
    - npm ci --cache .npm --prefer-offline

megacorp/devops/ci-templatestemplates/node.ymlnode-build

.node-build:
  extends: .node-base
  stage: build
  cache:
    policy: pull-push
  script:
    - npm run build
  artifacts:
    paths: [dist/]

A cache that never seems to hit is the most common complaint. Check these in order:

Figure The cache never seems to be used. Check these in order.#

As text

Do the jobs run on different runners, with no shared cache storage? Yes: Each runner keeps its own cache; shared storage, such as S3, fixes it. No: the next step.
Is this branch protected when the one that saved the cache wasn't, or the other way round? Yes: Protected and unprotected branches get separate caches, by design. No: the next step.
Does the key change more often than you think? Yes: A key made from a file changes whenever that file does. No: the next step.
Do two jobs use the same key for different paths? Yes: Each overwrites the other's cache; give them different keys. No: the next step.
Did the job that saves the cache fail? Yes: Caches are saved only on success, by default. No: the next step.
Read the cache lines in the job log; they name the key it tried

Note

On the Kubernetes executor, every job runs in a new pod. A cache survives between jobs only if the runner keeps it somewhere lasting, such as S3, or a volume that outlives the pod. MegaCorp's runner excerpt doesn't show its cache settings, so ask the platform team.

Why one branch can have two caches#

GitLab adds a suffix to every cache key. Pipelines for protected branches and tags get -protected. So, since 18.4.5, does any pipeline started by someone with the Maintainer or Owner role. Every other pipeline gets -non_protected.

So when a Developer and a Maintainer both push to the same feature branch, their pipelines use two different caches, and each misses the other's files. The project setting Use separate caches for protected branches, under Settings › CI/CD › General pipelines, turns the separation off. GitLab advises that only where everyone with the Developer role is highly trusted.

Also: caches are restored before artifacts. If a job caches and keeps the same path, the artifact overwrites what the cache restored.

dotenv: passing values to later jobs#

Variables set in a script vanish when the job ends. To hand a value to later jobs, write it to a file of NAME=value lines, and declare the file as a dotenv report. MegaCorp's image build passes on the image it pushed:

megacorp/devops/ci-templatestemplates/container.ymlimage-build

.image-build:
  extends: .megacorp-base
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
    IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
    - buildah push "$IMAGE:$CI_COMMIT_SHA"
    - echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
  artifacts:
    reports:
      dotenv: build.env

Later jobs receive IMAGE_REF as an ordinary variable. The rules:

Who gets it: jobs in later stages, or only the jobs that name this one in needs or dependencies.
Not in rules. Rules are decided when the pipeline is created, before any job has written anything (chapter 14).
Its rung on the ladder: a dotenv value beats the job's YAML variables, but not settings or pipeline variables (chapter 19).
A strict format: UTF-8, no blank lines, no comments, no quotes, and no multi-line values, in at most 5 KB. On self-managed GitLab, at most 20 of these variables are passed on by default.
Not for secrets: anyone who can see the pipeline can read the file.

Spot the bugThe retry that couldn't start#

The v1.5.0 release pipeline failed at image-build, because of the trust policy in chapter 31. Eight days later the role is fixed, and someone retries image-build. It fails at once, before its script runs:

This job could not start because it could not retrieve the needed artifacts.

megacorp/devops/ci-templatestemplates/base.ymlmegacorp-base

.megacorp-base:
  variables:
    MC_TEAM: unknown
  artifacts:
    expire_in: 7 days
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]
    - !reference [.rules, release-tag]

QuestionWhat is missing, why wasn't it kept, and what should they do instead?

Show the answer

image-build needs maven-build, so it must download the jar that job saved. Every MegaCorp job extends .megacorp-base, whose artifacts expire after 7 days, and eight days have passed. GitLab keeps the artifacts of the latest successful pipeline on each ref no matter what, but this pipeline failed.

Retry maven-build first, so that a new jar exists, and then image-build. Or start a new pipeline for the tag. See artifacts.

Next25Git in CI4 min

Git in CI#

Part V · Execution and data · Chapter 25· 4 min read

Before a job's script runs, the runner fetches the code. What it fetches is not the full repository you have on your laptop:

only the pipeline's own branch or tag, and no others
only the newest commits, 20 by default in new projects
the pipeline's commit, checked out directly, so there is no current branch

Most scripts never notice. Scripts that compare with main, count commits or read tags do notice, and they often fail only on big changes.

What a job's copy contains#

GitLab tells the runner exactly which refs to fetch, and they depend on the kind of pipeline:

Pipeline	What is fetched	So the job has
Branch pipeline	the pipeline's commit, and its own branch	`origin/<branch>`, but no other branch; on a feature branch, no `origin/main`
Tag pipeline	the pipeline's commit, and that tag	that one tag, so `git describe` can't see older ones
Merge request pipeline	the pipeline's commit only, as `refs/pipelines/<id>`	no branch names at all

The runner then checks out the pipeline's commit, not a branch, so Git has no current branch. Git calls this a detached HEAD. Take names from variables instead:

CI_COMMIT_REF_NAME is the branch or tag being built.
CI_COMMIT_BRANCH is set only in branch pipelines.
CI_MERGE_REQUEST_TARGET_BRANCH_NAME is the target, in merge request pipelines.

Shallow clones#

A shallow clone fetches only the newest commits. New projects fetch 20, set under Settings › CI/CD › General pipelines › Git shallow clone, and the GIT_DEPTH variable overrides that. MegaCorp's base template sets GIT_DEPTH: "20" for every job that includes it.

Figure A job's clone with GIT_DEPTH 20. The newest 20 commits are fetched and older ones aren't. So a script that needs an older commit works on small changes and fails on big ones.#

Three kinds of script need more history than that:

Comparing with an older commit, such as git diff against a merge request's base, fails when that commit is older than the clone.
Reading tags, with git describe or a version made from tags, needs the tags and the commits between them.
Running old commits. GitLab's documentation warns that a retried or queued job may find its commit outside the clone, and the log then says unresolved reference. A depth of 1 makes this likely.

Fix it in the one job that needs history. Raise GIT_DEPTH for that job, or fetch what the script needs, for example with git fetch --unshallow.

Figure A script fails on Git history in CI but works on a laptop. Ask these in order.#

As text

Does the script use another branch, such as main? Yes: That branch isn't in the job's copy; fetch it first. No: the next step.
Does it use an older commit, such as a merge request's base? Yes: The commit may be older than GIT_DEPTH; fetch more history. No: the next step.
Does it read tags, or run git describe? Yes: A tag pipeline fetches one tag, and other pipelines none. No: the next step.
Does it ask Git for the current branch? Yes: There is none; use CI_COMMIT_REF_NAME. No: the next step.
Check GIT_STRATEGY; with none or empty, the job gets no fresh code at all

Variables that shape the clone#

Variable	What it does	Default
`GIT_STRATEGY`	`clone` starts fresh; `fetch` reuses the last copy where the executor keeps one; `none` and `empty` skip Git, for jobs that only use artifacts	the project's setting
`GIT_DEPTH`	how many commits to fetch	the project's setting: 20 in new projects
`GIT_CHECKOUT`	`"false"` fetches without checking out the pipeline's commit	`"true"`
`GIT_CLEAN_FLAGS`	how `git clean` tidies a reused copy; `none` skips it	`-ffdx`
`GIT_FETCH_EXTRA_FLAGS`	flags added to `git fetch`	`--prune --quiet`
`GIT_SUBMODULE_STRATEGY`	`normal` fetches submodules, `recursive` their submodules too	submodules aren't fetched

On the Kubernetes executor each job gets a new pod, so fetch usually has no earlier copy to reuse, and clones anyway.

Submodules#

A submodule is another repository placed inside yours, pinned to one commit. The runner fetches submodules only when GIT_SUBMODULE_STRATEGY is normal or recursive. For them to work in CI:

In .gitmodules, point at submodules on the same GitLab with a relative URL, such as ../../devops/ci-tools.git, or with an HTTPS URL. A relative URL can resolve wrongly in forks.
For SSH URLs, set GIT_SUBMODULE_FORCE_HTTPS, so the runner fetches them over HTTPS.
The job token fetches them. The person who started the pipeline needs at least the Reporter role in the submodule's project, and that project must allow your project's job token (chapter 21).
GIT_SUBMODULE_DEPTH sets their depth separately from GIT_DEPTH.

If a job with GIT_STRATEGY: fetch fails with fatal: run_command returned non-zero status, GitLab's documentation suggests switching to GIT_STRATEGY: clone.

Spot the bugThe generator that fails on big merge requests#

In megacorp/data/mono, generate-pipeline works in most merge request pipelines. It fails in a few, always on long-lived branches with dozens of commits, and the log shows that git diff can't find a commit. mono's clone depth is the project default of 20:

megacorp/data/monoci/generate.sh

#!/usr/bin/env bash
# ci/generate.sh in megacorp/data/mono
# Writes one child-pipeline job per service that changed. Its output is the YAML
# that the service-pipelines trigger job runs. Used by chapter 10.
set -euo pipefail

base="${CI_MERGE_REQUEST_DIFF_BASE_SHA:-HEAD~1}"
changed=$(git diff --name-only "$base" HEAD -- services/ | cut -d/ -f2 | sort -u)

echo "stages: [build]"
for svc in $changed; do
  cat <<EOF
build-${svc}:
  stage: build
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  script:
    - make -C services/${svc} build
EOF
done

# a child pipeline needs at least one job, even when nothing changed
if [ -z "$changed" ]; then
  printf 'no-changes:\n  stage: build\n  script: [echo "no service changed"]\n'
fi

QuestionWhy do only the big merge requests fail, and what other mistake hides in the same line?

Show the answer

A merge request pipeline fetches only the pipeline's commit, and only the newest 20 commits of its history. CI_MERGE_REQUEST_DIFF_BASE_SHA is the base of the merge request's diff. On a branch with 20 or more commits, that base is older than the clone, so git diff can't find it. Give this job more history: raise GIT_DEPTH for it, or run git fetch --unshallow before the diff.

The other mistake is the fallback, HEAD~1, used in every other pipeline. It compares with the previous commit only, so a push of three commits builds just the services changed in the last one. For pushes, CI_COMMIT_BEFORE_SHA holds the branch's tip before the push. It is all zeros for new branches, schedules and manual runs, so the script still needs a fallback. See shallow clones.

Next · Part VI26The scanning landscape3 min

The scanning landscape#

Part VI · Scanners and gates · Chapter 26· 3 min read

A scanner is a job that reads something your pipeline has, looking for known problems. It might read the source code, the libraries the code uses, the image the pipeline built, or the running application. Enterprise pipelines often run five or six scanners. Each one writes a security report, and GitLab shows the findings in merge requests and on security pages, if your tier includes them.

Three questions explain almost any scanner: what does it read, who put it in my pipeline, and what can it block?

What each scanner reads#

Figure The scanner map. Each scanner reads one thing along the build path and writes a security report. Where the findings appear depends on the tier.#

Scanner	Reads	Looks for	Tier
SAST	the source code	risky code, such as a database query built from user input	Free
Secret detection	the source code	passwords, keys and tokens committed by mistake	Free
IaC scanning	infrastructure files, such as Terraform	insecure cloud and cluster settings	Free
Dependency scanning	the dependencies, as an SBOM	libraries with known vulnerabilities	Ultimate
Container scanning	the built image	vulnerable packages inside the image	Free
DAST and API testing	the running application	weaknesses that show only when the app runs	Ultimate

An SBOM, a software bill of materials, is a list of every library a project uses, with versions. A free tier can run a scanner and still not show its findings anywhere except the job itself.

Who put it in your pipeline#

A scanner job arrives in one of four ways, and the way decides who can change it:

Arrives through	How you recognise it	Who can change it
A GitLab template	an `include: template:` line, in your file or a central one	whoever owns that file; a project can adjust a job by redefining it under the same name
A central template or component	an `include:` of your platform team's files	the platform team
A scan execution policy	jobs that are in no file, with names such as `secret-detection-1`	only the security team, through the policy project
A pipeline execution policy	jobs in the stages `.pipeline-policy-pre` and `.pipeline-policy-post`	only the security team

MegaCorp uses all four. Its central template includes GitLab's scanners, and its security policy project adds more to every pipeline:

megacorp/devops/ci-templatestemplates/security.ymlsecurity-includes

include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml
  - template: Security/Container-Scanning.gitlab-ci.yml

megacorp/security/policies.gitlab/security-policies/policy.ymlscan-execution

scan_execution_policy:
  - name: Secret detection everywhere
    description: Run secret detection in every pipeline on every branch.
    enabled: true
    rules:
      - type: pipeline
        branches: ["*"]
    actions:
      - scan: secret_detection
  - name: Nightly dependency scan
    description: Scan dependencies on the default branch every night.
    enabled: true
    rules:
      - type: schedule
        branches: [main]
        cadence: "0 2 * * *"
    actions:
      - scan: dependency_scanning

Where the findings appear#

Place	What it shows	Tier
The job's log and artifacts	the raw report, as a JSON file	every tier
The pipeline's Security tab	every finding from that pipeline	Ultimate
The merge request's Reports tab	what the merge would add or fix	Ultimate
The vulnerability report	vulnerabilities on the default branch	Ultimate

GitLab calls a problem on a branch a finding. It becomes a vulnerability when the branch is merged into the default branch. Findings expire with the job's artifacts, or after 90 days by default on self-managed GitLab and 30 days on GitLab.com.

What a scanner can block#

Three different things can stop a change, and they are easy to mix up:

The scanner job fails. The scanner crashed, or couldn't read what it needed. GitLab's scanner jobs are allowed to fail, so this shows only as a warning, and the pipeline stays green.
The merge request needs extra approval. A merge request approval policy reads the findings and demands approval, for example from the security team (chapter 29).
A third-party gate fails the job on purpose, such as a quality gate that isn't met (chapter 28).

Spot the bugWhose job is it?#

On MegaCorp's production GitLab, which has Ultimate, a main pipeline of payments-api shows these jobs, among others:

secret_detection
container_scanning
secret-detection-1
policy-sbom

Use the specimen files above, and chapter 11.

QuestionFor each job, where is it defined, and who could change it?

Show the answer

secret_detection comes from GitLab's secret detection template, which MegaCorp's security.yml includes. The platform team owns that include. A project could redefine the job by name.
container_scanning comes from GitLab's container scanning template, and security.yml replaces its rules and needs. The platform team owns it.
secret-detection-1 comes from the scan execution policy "Secret detection everywhere". The numbered suffix gives it away. Only the security team can change it, in the policy project.
policy-sbom comes from the pipeline execution policy "MegaCorp guardrails", which puts it in .pipeline-policy-post. Only the security team can change it.

See who put it in your pipeline.

Next27GitLab's own scanners3 min

GitLab's own scanners#

Part VI · Scanners and gates · Chapter 27· 3 min read

GitLab's scanners arrive as templates. Each template adds jobs whose rules decide when they appear, and four behaviours explain most surprises:

By default they run in branch pipelines, not merge request pipelines.
Some add a job only when matching files exist, or only on Ultimate.
They are all allowed to fail, so a broken scanner shows only a warning.
Their jobs sit in the test stage, so a pipeline must have one.

What the templates add#

Each Security/ template that MegaCorp includes contains a single line: an include of its Jobs/ twin, which holds the jobs.

Template	Jobs it adds	When a job appears
`Jobs/SAST.gitlab-ci.yml`	one per analyzer, such as `semgrep-sast`	only if files that analyzer reads exist, such as `.java` or `.py`; `gitlab-advanced-sast` also needs Ultimate
`Jobs/Secret-Detection.gitlab-ci.yml`	`secret_detection`	in branch pipelines
`Jobs/Dependency-Scanning.gitlab-ci.yml`	one per package manager, such as `gemnasium-maven-dependency_scanning`	only on Ultimate; its Gemnasium analyzer is deprecated
`Jobs/Container-Scanning.gitlab-ci.yml`	`container_scanning`	in branch pipelines; it scans the image named in `CS_IMAGE`

Every job's first rule checks its toggle, such as SAST_DISABLED. Every job has allow_failure: true. The secret detection and container scanning reports can be downloaded only by people with the Developer role or above.

On the local GitLab, which is the Free edition, these templates add just two jobs to payments-api's main pipeline: secret_detection and container_scanning. There is no SAST job, because the specimen repository holds only a pom.xml, and no .java files. There is no dependency scanning job either, because those jobs need Ultimate.

The merge request gap#

The scanner jobs' rules follow the same three steps:

In a merge request pipeline, run only if AST_ENABLE_MR_PIPELINES is "true".
If that variable is on and the branch has an open merge request, skip the job in the branch pipeline, so that nothing is scanned twice.
Otherwise, run in the branch pipeline.

So by default the scanners run only in branch pipelines. On the local GitLab, a push to a payments-api merge request branch produced both pipelines. The branch pipeline had secret_detection; the merge request pipeline didn't. Anyone reading only the merge request pipeline sees no secret detection at all.

Figure One of GitLab's scanner jobs is missing from a pipeline. Ask these in order.#

As text

Is this a merge request pipeline? Yes: Scanners skip merge request pipelines unless AST_ENABLE_MR_PIPELINES is "true". No: the next step.
Is the scanner's toggle set, in YAML or in settings? Yes: A toggle such as SAST_DISABLED removes the job. No: the next step.
Is it SAST, and the repository has no files that the analyzer reads? Yes: SAST adds jobs only for the languages it finds. No: the next step.
Is it dependency scanning, on Free or Premium? Yes: Those jobs appear only on Ultimate. No: the next step.
Check whether someone redefined the job and replaced its rules

A pipeline without a test stage fails in a different way. It can't be created at all, because the scanner jobs name a stage that doesn't exist (chapter 17).

Changing a template's job#

To change a template's job, define a job with the same name. GitLab merges the two, and your keys win. MegaCorp moves container scanning into its own stage, points it at the image it has just built, and replaces its rules:

megacorp/devops/ci-templatestemplates/security.ymlcontainer-scan

variables:
  CS_IMAGE: $IMAGE_REF

container_scanning:
  stage: scan
  needs: [image-build]
  rules:
    - if: $CONTAINER_SCANNING_DISABLED == "true"
      when: never
    - !reference [.rules, never-on-schedule]
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Replacing rules replaces all of them, because lists are never merged (chapter 7). MegaCorp's rules keep the toggle, and add merge request pipelines, which the template alone would skip. That's why the local captures show container_scanning in payments-api's merge request pipeline, but not secret_detection.

Gotcha

One key from the template survives the merge and undoes MegaCorp's intent. GitLab's job has dependencies: [], and MegaCorp's version doesn't replace it. An empty dependencies blocks every artifact download, dotenv reports included, even with needs: [image-build]. On the local GitLab 19.3, a probe job with both keys got an empty IMAGE_REF, and so an empty CS_IMAGE. So MegaCorp's container scan never receives the image it was meant to scan. Adding dependencies: [image-build] fixed the probe. See the symptom card.

Spot the bugThe scan the reviewers never saw#

A reviewer opens a payments-api merge request and checks its pipeline. There is no secret_detection job. A colleague finds one in a different pipeline, for the same commit, on the merge request's branch.

megacorp/devops/ci-templatestemplates/security.ymlsecurity-includes

include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml
  - template: Security/Container-Scanning.gitlab-ci.yml

QuestionWhy does the scan run in one pipeline and not the other, and how could it run in the merge request pipeline instead?

Show the answer

GitLab's secret detection template runs its job in branch pipelines, and in merge request pipelines only when AST_ENABLE_MR_PIPELINES is "true". Nobody set it, so the scan ran in the branch pipeline only.

Set AST_ENABLE_MR_PIPELINES: "true", in the YAML or in settings. The job then runs in merge request pipelines, and is skipped in the branch pipeline while a merge request is open. Alternatively, redefine secret_detection with rules of your own, as MegaCorp did for container scanning. See the merge request gap.

Next28Third-party scanners2 min

Third-party scanners#

Part VI · Scanners and gates · Chapter 28· 2 min read

Many organisations also run tools that GitLab doesn't ship, such as SonarQube, Snyk or Trivy. They all follow one pattern: a job runs the tool, the tool's exit code decides whether the job fails, and a report file may carry the findings into GitLab. Find those three things, and the job's allow_failure and needs, and you know what the tool can really stop.

Tool, exit code, report, gate#

Figure What can a third-party scanner job actually stop? Follow it from start to finish.#

As text

The job runs the tool's command-line program
Does the tool exit with an error when it finds problems? No: The job passes whatever it finds; many tools need an option for this. Yes: the next step.
Is the job allowed to fail? Yes: A failure shows only as a warning, and blocks nothing. No: the next step.
Do the jobs that matter wait for it? No: With needs, a deploy can start without waiting for the scan. Yes: the next step.
A failure fails the pipeline, and jobs that wait for the scan don't run

A report file lets GitLab show the tool's findings, instead of leaving them in the job log or on the tool's own website:

Report	For	Where GitLab shows it
`codequality`	lint and code-quality findings, in a simple JSON format	merge requests, and the pipeline's Code Quality tab (Premium)
`sarif`	security findings from any tool that writes SARIF 2.1.0	the Security tab and the vulnerability report (Ultimate, since 19.2)
`junit`	test results	the pipeline's Tests tab, and the merge request's test summary

For code quality, GitLab's preferred route is exactly this: run your own linter and import its report. GitLab's built-in CodeClimate template is deprecated.

MegaCorp's SonarQube job#

MegaCorp's platform team ships SonarQube as a component (chapter 9):

megacorp/devops/componentstemplates/sonar-scan.ymljob

sonar-scan:
  stage: $[[ inputs.stage ]]
  image: registry.example.com/megacorp/devops/ci-tools/sonar-scanner:6
  variables:
    SONAR_PROJECT_KEY: $[[ inputs.project_key ]]
    SONAR_QUALITYGATE_WAIT: "$[[ inputs.quality_gate ]]"
  script:
    - sonar-scanner -Dsonar.projectKey="$SONAR_PROJECT_KEY" -Dsonar.host.url="$SONAR_HOST_URL"
  rules:
    - if: $SKIP_SONAR == "true"
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  allow_failure: true

Read it with the pattern:

Tool: sonar-scanner, which sends the code to the Sonar server for analysis.
Exit code: the component's quality_gate input sets SONAR_QUALITYGATE_WAIT. It is meant to make the scanner wait for Sonar's quality gate, and fail the job when the gate fails.
Report: none. The findings stay on the Sonar server.
Gate: allow_failure: true, so even a failed gate is only a warning.

It also needs SONAR_TOKEN, which is protected, so in merge request pipelines from feature branches the scan can't sign in (chapter 20).

Spot the bugThe quality gate that couldn't stop a deploy#

A change that fails MegaCorp's Sonar quality gate is merged into payments-api's main. The pipeline goes green, and deploy-staging ships the change. sonar-scan shows an orange warning.

megacorp/devops/ci-templatespipelines/java-service.ymldeploys

deploy-staging:
  extends: .deploy
  needs: [image-build]
  variables:
    ENVIRONMENT: staging
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, default-branch]

deploy-prod:
  extends: .deploy
  needs: [image-build]
  variables:
    ENVIRONMENT: production
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
      when: manual

QuestionName the two separate reasons the failed gate didn't stop the deploy.

Show the answer

sonar-scan has allow_failure: true, so its failure never fails the pipeline.
deploy-staging has needs: [image-build]. It starts as soon as the image is built, without waiting for jobs in the test stage, so it doesn't wait for sonar-scan at all (chapter 17).

To make the gate stop deploys, both would have to change: the scan must be allowed to fail the pipeline, and the deploy must wait for it. Whether the gate should block is a decision for the teams that own the pipeline. See the pattern.

Next29Policies and gates3 min

Policies and gates#

Part VI · Scanners and gates · Chapter 29· 3 min read

Security policies are rules the security team writes once for many projects. They need Ultimate, and come in three kinds:

scan execution policies add scanner jobs
pipeline execution policies add any jobs, or replace the pipeline
merge request approval policies demand approval when scans find problems

Policies win over anything a project writes. A toggle such as SAST_DISABLED skips a project's own scanner jobs, never a policy's. Turning a mandated scan off is a decision for the security team, not a change to the YAML.

Three kinds of policy#

Policy	What it does	How you recognise it
Scan execution	adds scanner jobs to pipelines, or runs scans on a schedule	jobs such as `secret-detection-1`, in `test`, or in `scan-policies` when there is no `test` stage
Pipeline execution	adds any jobs, or replaces the project's pipeline	jobs in `.pipeline-policy-pre`, which runs first, and `.pipeline-policy-post`, which runs last
Merge request approval	requires approval when scans find problems	an approval rule on the merge request, named after the policy

Pipeline execution policies have a scheduled variant Since 19.2. It creates pipelines of its own on a schedule, in every project it covers, and runs only the policy's jobs, never the project's.

All three live in one file, .gitlab/security-policies/policy.yml, in a separate security policy project linked to the group. A project can hold at most five policies of each kind. Chapter 11 shows MegaCorp's first two (chapter 11). This is the third:

megacorp/security/policies.gitlab/security-policies/policy.ymlapproval

approval_policy:
  - name: Block new critical vulnerabilities
    description: A merge into main that adds a critical finding needs security's approval.
    enabled: true
    rules:
      - type: scan_finding
        branches: [main]
        scanners: [sast, secret_detection, dependency_scanning, container_scanning]
        vulnerabilities_allowed: 0
        severity_levels: [critical]
        vulnerability_states: [new_needs_triage]
    actions:
      - type: require_approval
        approvals_required: 1
        group_approvers: [megacorp/security]

Toggles against policies#

GitLab's scanner templates switch off with variables: SAST_DISABLED, SECRET_DETECTION_DISABLED, CONTAINER_SCANNING_DISABLED and others. GitLab's documentation is exact about their limit: "Skipping jobs does not prevent any security jobs defined by scan execution policies from running." Two more rules follow:

A project can't override a policy's scanner jobs, even by using the same job name.
In a project under a scan execution policy, the scanners' exclude settings, such as SAST_EXCLUDED_PATHS, are preset, and only a policy can change them.

So when a team wants a mandated scan off, the answer isn't in their YAML. The security team decides, for example by narrowing the policy's scope.

When a merge request needs the security team#

MegaCorp's approval policy looks at merges into main. If a scan finds a new critical vulnerability, one member of megacorp/security must approve. The rule also applies when GitLab can't check it:

Figure Why does my merge request need the security team's approval? Ask these in order.#

As text

Did a scan find a new critical vulnerability? Yes: The rule is working; fix the finding, or ask for approval. No: the next step.
Did a scanner the rule names produce no report? Yes: The rule can't be checked, and by default that also requires approval. No: the next step.
Does the policy target this branch? No: Look for another policy; each shows as its own approval rule. Yes: the next step.
Open Secure › Policies, and read the policy's rule

The second case surprises people. By default a policy fails closed: "Invalid or unenforceable rules of a policy require approval." So a scanner that was skipped or crashed doesn't let a merge request slip through. It blocks it until someone approves. The policy can say fallback_behavior: fail: open instead.

Finding the policies#

Anyone who can see a project can open its policies at Secure › Policies. The list shows each policy and whether it is enforced. Changes go through a merge request in the policy project:

Figure A simplified sketch of the Policies page and the policy editor, not a screenshot. A green checkmark marks an enabled policy. The editor saves changes as a merge request in the security policy project.#

Spot the bugThe toggle that only half worked#

payments-api's pipelines run two secret detection jobs, and the team wants to save time. They add this to their .gitlab-ci.yml:

variables:
  SECRET_DETECTION_DISABLED: "true"

secret_detection disappears from their pipelines, but secret-detection-1 still runs.

QuestionWhy did only one job go, and what should the team do if they want the other one gone too?

Show the answer

secret_detection comes from GitLab's template, included by MegaCorp's security.yml, and the template honours the toggle. secret-detection-1 comes from the scan execution policy "Secret detection everywhere". Toggles skip only a project's own jobs, never a policy's.

Removing the second job isn't the team's decision. They can ask the security team to change the policy's scope. Having the job twice is expected: GitLab runs both, because the two can use different settings. See toggles against policies.

Next · Part VII30Building Java, Node and front ends3 min

Building Java, Node and front ends#

Part VII · Delivery · Chapter 30· 3 min read

A build job is an ordinary job with the right image. What makes builds confusing in a big organisation is everything around the compiler:

where the dependencies come from: a company mirror, and a cache
what the build produces, and where it goes: an artifact, a package registry, an image or a bucket
which version number the result carries

Most "works on my machine" failures come from the first. Most "which version is running?" confusion comes from the last.

A Maven build at MegaCorp#

MegaCorp's Maven jobs start from one hidden job. It picks the JDK image, caches the local repository, and writes a Maven settings file before the script runs:

megacorp/devops/ci-templatestemplates/java-maven.ymlmaven-base

.maven-base:
  extends: .megacorp-base
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
  variables:
    JAVA_VERSION: "21"
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  cache:
    key:
      files: [pom.xml]
    paths: [.m2/repository]
  before_script:
    # a job's own before_script replaces default:before_script, so re-add the functions
    - !reference [.snippets, functions]
    - !reference [.snippets, maven_settings]

The settings file comes from a function in the ci-tools image. It sends every download through MegaCorp's Nexus server, a company copy of the public repositories:

megacorp/devops/ci-toolslib/ci-lib.shmaven-settings

mc_maven_settings() {
  cat <<EOF
<settings><mirrors><mirror><id>nexus</id><mirrorOf>*</mirrorOf><url>${MC_NEXUS_URL:-https://nexus.example.com/repository/maven-public/}</url></mirror></mirrors></settings>
EOF
}

Three variables decide how Maven runs, and only one is in the YAML:

Variable	Set in	Effect
`MAVEN_CLI_OPTS`	the megacorp group's CI/CD settings	batch mode, and the settings file above; it overrides any value a project writes in YAML (chapter 19)
`MAVEN_OPTS`	the template	keeps Maven's local repository inside the project, so that it can be cached
`MC_NEXUS_URL`	payments-api's CI/CD settings	which Nexus the mirror points at

maven-build packages the jar without running the tests, and keeps it as an artifact. maven-test runs the tests twice, on JDK 17 and 21, with parallel: matrix (chapter 7).

A Node front end at MegaCorp#

The Node templates install with npm ci, from a cache keyed on package-lock.json, and build into dist/ (chapter 24). web-portal's whole pipeline is three jobs:

megacorp/devops/ci-templatespipelines/web.ymljobs

node-build:
  extends: .node-build

node-test:
  extends: .node-test

deploy-site:
  extends: .publish-site
  needs: [node-build]
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, default-branch]

deploy-site copies dist/ to an S3 bucket and clears the CloudFront cache (chapter 31). web-portal has no .gitlab-ci.yml of its own: its settings point at this file (chapter 11).

Figure A build works on your machine but fails in CI. Ask these in order.#

As text

Does it download dependencies? Yes: CI downloads through the company mirror and proxy, which may lack or block a package. No: the next step.
Does it need a file that isn't in Git, such as a local settings file? Yes: The job has only the repository, artifacts and variables. No: the next step.
Is the tool's version different from yours? Yes: The job uses the image's version; check the image's tag. No: the next step.
Did it restore an old cache? Yes: Check the cache key, or clear the cache (chapter 24). No: the next step.
Compare the job's variables with your shell's; settings variables may differ

Which version is it?#

Three numbers can describe one build, and they often disagree:

Number	Comes from	payments-api's example
The version in the build file	`pom.xml` or `package.json`, as committed	`1.8.0-SNAPSHOT`
The Git tag	the release tag, in `CI_COMMIT_TAG`, which is set only in tag pipelines	`v1.5.0`
The image tag	the commit, in `CI_COMMIT_SHA`	a 40-character commit hash

MegaCorp's mc tool can work out a version in any pipeline: the tag if there is one, otherwise 0.0.0- followed by the short commit hash.

megacorp/devops/ci-toolsbin/mcversion

version)
  echo "${CI_COMMIT_TAG:-0.0.0-${CI_COMMIT_SHORT_SHA}}"
  ;;

Nothing in the Maven templates uses it, though. The exercise shows the result.

Test results and coverage#

Test reports in JUnit XML, saved as artifacts: reports: junit, fill the pipeline's Tests tab and the merge request's test summary. The summary lists newly failed tests. A report never changes the job's status: the test command's exit code does.
A coverage percentage comes from the job log. The coverage keyword holds a regular expression, and GitLab shows the first number it matches in the merge request.
Line-by-line coverage comes from a Cobertura or JaCoCo file saved as artifacts: reports: coverage_report. GitLab marks the lines in the merge request's diff.

Publishing packages to GitLab's registries#

Some teams publish libraries to GitLab's own package registry, using the job token as the password.

Maven. A settings file sends the job token as a header:

<settings>
  <servers>
    <server>
      <id>gitlab-maven</id>
      <configuration>
        <httpHeaders>
          <property>
            <name>Job-Token</name>
            <value>${CI_JOB_TOKEN}</value>
          </property>
        </httpHeaders>
      </configuration>
    </server>
  </servers>
</settings>

The POM's distribution repository is ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/maven, and a job runs mvn deploy. To refuse a version that is already published, a group Owner turns off Allow duplicates under Settings › Packages and registries.

npm. The job writes an .npmrc that points the package's scope at the project's registry, with the job token:

echo "@megacorp:registry=https://${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/" > .npmrc
echo "//${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN}" >> .npmrc
npm publish

The scope is the top-level group of the project that hosts the package, in lower case: @megacorp. A package name can belong to only one project per top-level group. Publishing it from a second project fails, even with a new version.

Spot the bugThe release that called itself a snapshot#

After payments-api's v1.5.0 release, the support team checks the running service. Its version page, which reads the jar's version, says 1.8.0-SNAPSHOT. The image is the one that the v1.5.0 tag pipeline built.

megacorp/payments/payments-apipom.xmlversion

<artifactId>payments-api</artifactId>
<version>1.8.0-SNAPSHOT</version>

megacorp/devops/ci-templatestemplates/java-maven.ymlmaven-build

.maven-build:
  extends: .maven-base
  stage: build
  script:
    - mvn $MAVEN_CLI_OPTS -DskipTests package
  artifacts:
    paths: [target/*.jar]

QuestionWhere does 1.8.0-SNAPSHOT come from, and what would make the jar carry the release's version?

Show the answer

Maven takes the version from pom.xml, which says 1.8.0-SNAPSHOT. The build job runs mvn package without changing it, so every pipeline, the tag pipeline included, builds a jar with that version. The tag names the commit, but never reaches Maven.

Pass the version into the build in tag pipelines, for example from mc version, which returns CI_COMMIT_TAG there. Or make each release a commit that sets the POM's version, and tag that commit. See which version is it.

Next31GitLab and AWS9 min

GitLab and AWS#

Part VII · Delivery · Chapter 31· 9 min read

Jobs that build and deploy to AWS need AWS permissions. The modern way, and MegaCorp's way, stores no AWS password anywhere. Each job trades its ID token for temporary keys to an AWS role, as chapter 21 showed.

This chapter covers:

the five things on each side that must match, and who owns each
which credentials a job really uses when several are present
working with several AWS accounts
ECR, secrets from AWS, and the usual deploy targets
what to check, in order, when AWS says no

AWS words in two minutes#

If AWS is new to you, these are the only words this chapter needs:

Word	Meaning
Account	a separate AWS space with its own resources and bill, named by a 12-digit number such as `123456789012`
Region	a location where resources live, such as `eu-west-2` (London); jobs set it with `AWS_REGION`
IAM	AWS's identity service: who may do what
Role	an identity with permissions and no password; a person or a system assumes it for a while
Trust policy	the rules on a role that say who may assume it
Permission policy	the rules on a role that say what it may do once assumed
ARN	the full name of anything in AWS, such as `arn:aws:iam::123456789012:role/gitlab-ecr-push`
STS	the Security Token Service, which hands out temporary keys for a role
ECR	the Elastic Container Registry, where images are stored
S3 and CloudFront	file storage, and the network that serves those files quickly worldwide
ECS, EKS and Lambda	three places to run code: AWS's container service, Kubernetes clusters, and single functions
Secrets Manager and Parameter Store	two AWS services that store secrets and settings

Five things that must match#

A job gets into AWS only when five pairs line up. Each pair has one half in GitLab and the other half in AWS, and a different team often owns each half:

Figure The five pairs that must match before a job can use an AWS role. The first four decide whether STS hands out keys at all. The fifth decides what the keys can do.#

At MegaCorp the GitLab halves live in the templates, which the platform team owns:

megacorp/devops/ci-templatestemplates/container.ymlimage-build

.image-build:
  extends: .megacorp-base
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
    IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
    - buildah push "$IMAGE:$CI_COMMIT_SHA"
    - echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
  artifacts:
    reports:
      dotenv: build.env

The AWS halves are set in the AWS account itself. This file describes MegaCorp's:

awsiam.ymloidc-provider

# AWS accepts ID tokens signed by this GitLab, for this audience.
oidc_provider:
  arn: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
  url: https://gitlab.example.com
  audiences: [https://gitlab.example.com]

awsiam.ymlecr-push-role

# Assumed by image-build, and by the ecr-push component, through mc_aws_login.
ecr_push_role:
  arn: arn:aws:iam::123456789012:role/gitlab-ecr-push
  trust_policy:
    Effect: Allow
    Principal:
      Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
    Action: sts:AssumeRoleWithWebIdentity
    Condition:
      StringEquals:
        gitlab.example.com:aud: https://gitlab.example.com
      StringLike:
        gitlab.example.com:sub: project_path:megacorp/*:ref_type:branch:ref:*
  permissions: push and pull images in every ECR repository in the account

A mismatch in the first four pairs stops the login: STS refuses, and no keys come back. A mismatch in the fifth lets the login succeed, and a later command fails with an access-denied error. That difference tells you where to look.

The template runs two snippets before the script. The second calls a function from the ci-tools image:

megacorp/devops/ci-templatestemplates/snippets.ymlsnippets

.snippets:
  functions:
    - source /opt/megacorp/lib/ci-lib.sh
  aws_login:
    - mc_aws_login "$AWS_ROLE_ARN"
  maven_settings:
    - mkdir -p .m2
    - mc_maven_settings > .m2/settings.xml

megacorp/devops/ci-toolslib/ci-lib.shaws-login

# Exchange the job's OIDC ID token for temporary AWS credentials.
mc_aws_login() {
  local role="${1:?role ARN required}"
  local creds
  creds=$(aws sts assume-role-with-web-identity \
    --role-arn "$role" \
    --role-session-name "gitlab-${CI_PROJECT_ID}-${CI_JOB_ID}" \
    --web-identity-token "$MC_ID_TOKEN" \
    --duration-seconds 3600 \
    --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
    --output text) || { mc_log "AssumeRoleWithWebIdentity failed for $role"; return 1; }
  read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< "$creds"
  export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
  mc_log "assumed $role"
}

Step by step:

aws sts assume-role-with-web-identity sends the role's ARN and the ID token to STS.
STS checks the token and the trust policy. It returns three values: an access key, a secret key and a session token.
The function exports them as AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. Every later aws command in the job uses them.
They expire after 3600 seconds, which is one hour.

To see which identity a job is really using, add this line. It needs no permissions, and it prints the ARN of the role in use:

aws sts get-caller-identity

The AWS CLI can do the exchange by itself. Put the ID token in a file, and set two variables that name the role and the file:

publish:
  id_tokens:
    AWS_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-web-publish
    AWS_ROLE_SESSION_NAME: gitlab-$CI_JOB_ID
  script:
    - echo "$AWS_ID_TOKEN" > /tmp/web-identity-token
    - export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/web-identity-token
    - aws sts get-caller-identity

Every aws command then assumes the role on its own. This works only when no access keys are set, because keys come first in the order below.

Which credentials the AWS CLI uses#

A job can have AWS credentials from several places at once:

keys that a login step exported
keys that someone stored in CI/CD settings long ago
a role on the runner

The AWS CLI doesn't combine them. It takes the first it finds, in a fixed order:

Figure Which identity will an aws command use? The CLI stops at the first yes.#

As text

Does the command pass --profile, or keys, on its command line? Yes: Those are used. No: the next step.
Are AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set? Yes: Those keys are used, whether a login step exported them or they came from settings. No: the next step.
Are AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE both set? Yes: The CLI assumes that role, with the token in the file. No: the next step.
Does the image contain AWS configuration files? Yes: Their profile is used. No: the next step.
Otherwise the role of the machine or pod is used, often the runner's own role

Two failures follow from this order:

Old keys in settings win silently. A leftover AWS_ACCESS_KEY_ID in a group's CI/CD variables beats any role, until a login step overwrites it. A job that skips the login then acts as whoever owns those keys.
A skipped login falls through to the runner. A job that sets its own before_script replaces the template's, so the login never runs (chapter 7). The job then acts as the runner's role, which at MegaCorp can pull images but not push them.

Several AWS accounts#

Large organisations split AWS into several accounts, often one for shared tools and one for each environment. A job can reach another account in two ways:

Figure Reaching several AWS accounts. Each account's role can trust the ID token directly. Role chaining, from one role to the next, also works, but its keys last at most one hour.#

Direct trust. Each account has a role whose trust policy accepts the ID token, and the job logs in to the account it needs. Each account checks the token's sub itself, so production can accept only release tags, for example.
Role chaining. The job logs in to one role, then uses it to assume a role in the other account. AWS limits a chained session to one hour, whatever the role allows, and asking for longer fails.

MegaCorp needs neither. Its pipelines touch only the tooling account, 123456789012, where ECR and the website's bucket live. Deploys go through the GitOps repository, and a controller inside each cluster pulls the new image (chapter 34). The pipeline never holds production AWS keys at all.

ECR: pushing images, and pulling job images#

An ECR registry's address contains the account and the region: 123456789012.dkr.ecr.eu-west-2.amazonaws.com. To push, a job logs in to AWS first, then to the registry:

aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"

The user name is always AWS, and the password this prints is valid for 12 hours. Tagging and promoting images is the subject of chapter 32.

Pulling a job's own image: from ECR is different. The runner pulls it before the job starts, so nothing in the job's script can help. The runner needs its own way in.

Job images from ECR#

GitLab's documented way is the ECR credential helper. It is a small program that gets a registry password from AWS whenever the runner needs one. It must be installed on the runner, and the runner needs AWS credentials of its own. A DOCKER_AUTH_CONFIG variable then tells the runner to use it:

{ "credHelpers": { "123456789012.dkr.ecr.eu-west-2.amazonaws.com": "ecr-login" } }

GitLab's documentation notes that credential helpers aren't available on instance runners, or on any runner whose machine you can't configure. MegaCorp avoids the question: its job images come from registry.example.com, not from ECR.

Secrets from AWS#

A secret in AWS Secrets Manager or Parameter Store can reach software in three ways. The best one keeps it out of the pipeline completely:

Route	Who reads the secret	What the job sees
The `secrets:` keyword, with `aws_secrets_manager` (Premium)	the runner, as the job starts	a file holding the value, or the value itself with `file: false`
The job's script calls the AWS CLI	the job	the value, which GitLab can't mask because it never saw it
The running service reads it, for example through an ECS task definition	the service, when it starts	nothing at all

With the keyword, the runner does the fetching. It logs in with an ID token named AWS_ID_TOKEN and the role in AWS_ROLE_ARN. The region comes from AWS_REGION, which MegaCorp's base template sets for every job:

db-migrate:
  id_tokens:
    AWS_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/payments-db-migrate
  secrets:
    DB_PASSWORD:
      aws_secrets_manager:
        secret_id: payments/db
        field: password
  script:
    - ./migrate.sh --password-file "$DB_PASSWORD"

Without an ID token, the runner uses its own AWS role instead. On the Kubernetes executor, that role must be on the runner manager, the part that starts the job pods. If it is only on the pods, the job fails with no EC2 IMDS role found.

From a script, the commands look like this. Capture the value in a variable, and never print it:

DB_PASSWORD=$(aws secretsmanager get-secret-value --secret-id payments/db --query SecretString --output text)
DB_URL=$(aws ssm get-parameter --name /payments/db-url --with-decryption --query Parameter.Value --output text)

The third route is the cleanest. An ECS task definition can name the secret, and ECS reads it when the container starts, using the task's execution role:

"secrets": [
  { "name": "DB_PASSWORD",
    "valueFrom": "arn:aws:secretsmanager:eu-west-2:123456789012:secret:payments/db-AbCdEf" }
]

The pipeline deploys a task definition that holds only the secret's name. The value never passes through GitLab.

The usual deploy targets#

Most AWS deploys are one of these. Each is a few CLI commands, run with the role's temporary keys:

Target	What the job runs	Where to look when it fails
Static site on S3 and CloudFront	`aws s3 sync`, then `aws cloudfront create-invalidation`	if visitors still see old files, check that the invalidation ran
ECS	a new task definition that names the new image, then `aws ecs update-service` and `aws ecs wait services-stable`	the wait checks every 15 seconds and gives up after 40 checks, with exit code 255; the service's events show why new tasks fail
EKS	`aws eks update-kubeconfig`, then `kubectl` or `helm`, or a GitOps handoff	besides its AWS permissions, the role needs access inside the cluster, which the cluster's owners grant
Lambda	`aws lambda update-function-code`, with a new image or zip file	the role's permission on that function, then the function's own logs
Infrastructure as code	OpenTofu or Terraform, with state stored in GitLab; or CloudFormation	the role needs every permission the change needs, so guard it with protected branches and environments

MegaCorp's website job is the first kind:

megacorp/devops/ci-templatestemplates/node.ymlpublish-site

.publish-site:
  extends: .megacorp-base
  stage: deploy
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-web-publish
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - aws s3 sync dist/ "s3://$SITE_BUCKET/" --delete
    - aws cloudfront create-invalidation --distribution-id "$CF_DISTRIBUTION_ID" --paths '/*'
  environment:
    name: production
    url: https://www.example.com

GitLab's ready-made AWS templates#

GitLab ships templates for AWS, and some older pipelines use them:

AWS/Deploy-ECS.gitlab-ci.yml builds an image into GitLab's registry, creates a new task definition revision, and updates an ECS service. You set CI_AWS_ECS_CLUSTER, CI_AWS_ECS_SERVICE and a task definition variable. The deploy waits for the rollout unless CI_AWS_ECS_WAIT_FOR_ROLLOUT_COMPLETE_DISABLED is set.
AWS/CF-Provision-and-Deploy-EC2.gitlab-ci.yml creates a CloudFormation stack, pushes the build to S3, and deploys it to EC2.
Both expect access keys stored as the CI/CD variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION. GitLab's own page says ID tokens are more secure, but don't work with these templates.
GitLab's image registry.gitlab.com/gitlab-org/cloud-deploy/aws-base contains the AWS CLI. If its output fails on a non-ASCII character, set LANG: "UTF-8".

Small things that bite#

The amazon/aws-cli image. Its entrypoint is the aws command itself. On the Docker executor, which keeps an image's entrypoint, it can't run your script unless you set entrypoint: [""]. The Kubernetes executor ignores entrypoints by default, so the same job can work on one runner and fail on another (chapter 23).
The region. Commands without --region use AWS_REGION, which MegaCorp sets to eu-west-2. A resource in another region looks as if it doesn't exist.
Keys expire after an hour. MegaCorp asks for 3600 seconds. A job that runs longer after logging in gets expired-token errors part-way through.
The ID token can expire first. Log in at the start of the job, not after a long build.
AWS must reach GitLab. STS downloads GitLab's signing keys from the instance. A self-managed GitLab that the internet can't reach gets InvalidIdentityToken. GitLab documents a workaround that publishes the keys elsewhere.

When AWS says no#

Read the error first, because it names the step that failed. Then ask these in order:

Figure A job's AWS call is refused. Ask these in order; the first yes is usually the cause.#

As text

Does the login fail with InvalidIdentityToken? Yes: AWS can't fetch GitLab's signing keys; can the internet reach the instance?. No: the next step.
Does it say "Not authorized to perform sts:AssumeRoleWithWebIdentity"? Yes: Compare the token's sub and aud with the trust policy. No: the next step.
Does the trust policy put a * inside StringEquals? Yes: Wildcards only work with StringLike. No: the next step.
Did the login step run at all? No: Something replaced before_script, so another identity is in use. Yes: the next step.
Does get-caller-identity show an unexpected role? Yes: Other credentials come first; see which credentials the CLI uses. No: the next step.
Did the job run for over an hour after logging in? Yes: The keys expired; log in again, or split the job. No: the next step.
The role lacks permission for that action, and its owners must add it

To compare sub with the trust policy, print the token's claims, not the token. A token has three parts separated by dots, and the middle part holds the claims. GitLab's documentation suggests this command:

echo "$MC_ID_TOKEN" | cut -d '.' -f2 | base64 -d | jq .

The claims include the email address of the person who started the pipeline, so remove the line when you have your answer. base64 may also print base64: invalid input and return an error, because a token leaves out the padding that base64 expects. It happened with two tokens in four on the local GitLab 19.3, and the claims still printed in full.

Spot the bugThe release that couldn't push#

payments-api tags v1.5.0. In the tag pipeline, image-build fails during the login:

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

The same job pushed an image from main an hour earlier, and from a merge request pipeline this morning. This is the push role:

awsiam.ymlecr-push-role

# Assumed by image-build, and by the ecr-push component, through mc_aws_login.
ecr_push_role:
  arn: arn:aws:iam::123456789012:role/gitlab-ecr-push
  trust_policy:
    Effect: Allow
    Principal:
      Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
    Action: sts:AssumeRoleWithWebIdentity
    Condition:
      StringEquals:
        gitlab.example.com:aud: https://gitlab.example.com
      StringLike:
        gitlab.example.com:sub: project_path:megacorp/*:ref_type:branch:ref:*
  permissions: push and pull images in every ECR repository in the account

QuestionWhy is only the tag pipeline refused, and how would you change the role without opening it to every tag?

Show the answer

A tag pipeline's ID token has a subject that ends in ref_type:tag:ref:v1.5.0. The pattern accepts only ref_type:branch, so STS refuses. Pipelines on main, and merge request pipelines, match because both run on branches.

Add a second pattern for release tags, such as project_path:megacorp/*:ref_type:tag:ref:v*. IAM accepts a list of values for one condition, and any match is enough. payments-api protects tags matching v*, which limits who can create them. On self-managed GitLab, AWS can't check ref_protected, so the pattern is the only guard.

While you are there, notice that ref:* lets any branch push images, including unreviewed feature branches. See how AWS checks an ID token.

Next32Images to ECR3 min

Images to ECR#

Part VII · Delivery · Chapter 32· 3 min read

MegaCorp builds each service's image in the pipeline, tags it with the commit's SHA, and pushes it to Amazon ECR. The deploy jobs receive that exact reference. The goal is build once, promote many: the image that staging tested is the image that production runs.

The common way to break that goal is to build again later, for example in a tag pipeline. A rebuild is a new image, even from the same commit.

How an image is named#

An image reference has three parts: the registry, the repository and a tag. MegaCorp's looks like this:

123456789012.dkr.ecr.eu-west-2.amazonaws.com/payments-api:3f9c1e2…
└──────────── registry ─────────────────────┘ └─ repository ─┘ └ tag ┘

A tag is only a label, and labels can move. A commit SHA as the tag ties the image to exactly one commit, which is why MegaCorp uses $CI_COMMIT_SHA. A tag such as latest says nothing about what is inside, and changes with every push.

From build to deploy#

image-build logs in to AWS and ECR, builds with Buildah, pushes, and writes the reference to a dotenv report as IMAGE_REF (chapter 24). The deploy jobs pass it on to the deployer:

megacorp/devops/ci-templatestemplates/deploy.ymldeploy-trigger

.deploy:
  stage: deploy
  trigger:
    project: megacorp/platform/deployer
    branch: main
    strategy: depend
  variables:
    APP: $CI_PROJECT_NAME
    IMAGE_REF: $IMAGE_REF

Figure Build once, promote many. Top: one image, tagged by commit, moves from staging to production. Bottom: MegaCorp's tag pipeline builds the same commit again, so production can get an image that staging never tested.#

payments-api's tag pipeline runs maven-build and image-build again before deploy-prod. The release tag usually points at a commit that main already built and staging already tested. The rebuild pushes a second image under the same SHA tag. Base images and libraries may have changed in the meantime, so it is not the image that was tested. What happens next depends on the ECR repository:

Mutable tags: the new image quietly replaces the tested one.
Immutable tags: the push fails with ImageTagAlreadyExistsException.

Promoting without rebuilding#

To promote, deploy the reference that was already built and tested. To give the same image a friendly name as well, add a tag in ECR without pulling or pushing anything:

MANIFEST=$(aws ecr batch-get-image --repository-name payments-api \
  --image-ids imageTag="$CI_COMMIT_SHA" --query 'images[].imageManifest' --output text)
aws ecr put-image --repository-name payments-api \
  --image-tag "$CI_COMMIT_TAG" --image-manifest "$MANIFEST"

Now v1.5.0 and the commit's SHA name the same image. With immutable tags, ECR refuses any attempt to move either of them.

Figure Which image is production really running? Ask these in order.#

As text

Does the deploy use a tag such as latest? Yes: You can't tell from the tag; look up the image's digest in ECR. No: the next step.
Was the same tag pushed by more than one pipeline? Yes: With mutable tags, the last push won; compare the pipelines' times. No: the next step.
Did the deploy receive IMAGE_REF from the build it followed? No: It may be deploying an older reference; check the trigger's variables. Yes: the next step.
The tag names one image; check the cluster really runs that tag

Spot the bugTwo images, one tag#

Commit 3f9c1e2 was built on main, deployed to staging, and tested for a week. Then it was tagged v1.5.0. The tag pipeline built and pushed payments-api:3f9c1e2… again, and deploy-prod shipped it. In production, a bug appears that staging never showed. The ECR repository has mutable tags.

megacorp/devops/ci-templatespipelines/java-service.ymljobs

maven-build:
  extends: .maven-build

maven-test:
  extends: .maven-test

image-build:
  extends: .image-build
  needs: [maven-build]

QuestionHow can production run code that staging never tested, when both used the same tag?

Show the answer

image-build runs in tag pipelines too, so the release built the commit a second time. That rebuild picked up whatever base image and libraries were current, made a different image, and pushed it under the same tag. With mutable tags it replaced the tested image. Production then deployed the new one.

Build once: skip image-build in tag pipelines, and deploy the image that main built. Add the release tag with put-image. Turning on tag immutability in ECR makes a rebuild fail loudly instead of replacing the image. See from build to deploy.

Next33Environments and deployments2 min

Environments and deployments#

Part VII · Delivery · Chapter 33· 2 min read

An environment is GitLab's record of a place you deploy to: a name, a URL, and a history of deployments. A job with environment: becomes a deployment job, and that unlocks five safeguards:

who may deploy there
approvals before a deploy runs
one deploy at a time
refusing outdated deploys
re-deploying an earlier version

At MegaCorp every environment lives in the deployer project, so that is where to look when a deploy won't start.

Environments and deployments#

MegaCorp's deployer names each environment after the target and the service:

megacorp/platform/deployer.gitlab-ci.ymldeploy

variables:
  ENVIRONMENT: $[[ inputs.environment ]]

deploy:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  resource_group: $APP-$ENVIRONMENT
  environment:
    name: $ENVIRONMENT/$APP
  script:
    - mc deploy "$ENVIRONMENT" "$IMAGE_REF"
  rules:
    - if: $IMAGE_REF == null
      when: never
    - if: $ENVIRONMENT == "production"
      when: manual
    - when: on_success

Each run of this job records a deployment of staging/payments-api or production/payments-api. Its history appears under Operate › Environments. The names come from variables, so one job serves every service. The resource_group lets only one deploy per service and environment run at a time (chapter 17).

An environment also scopes variables. A CI/CD variable limited to the production/* environments reaches only jobs deploying there (chapter 19).

Who may deploy, and approvals Premium#

Protected environments. A Maintainer can protect an environment, and list who is Allowed to deploy: roles, users or groups. Anyone else's deploy job to it can't run.
Deployment approvals. A protected environment can require approvals. Its deployments are blocked until all of them are given, under Operate › Environments, on the deployment's status badge. Approving doesn't start the job: someone must still run it.

Note

A trigger job, such as MegaCorp's .deploy, can use deployment-only access to a protected environment only if it has an environment: of its own. Without one, GitLab treats it as an ordinary job, with ordinary permissions.

One at a time, newest wins#

Two pipelines can race. The newer one deploys first, and then the older one finishes and puts the old version back. The project setting Prevent outdated deployment jobs stops that. An older deployment job then fails with:

The deployment job is older than the latest deployment, and therefore failed.

GitLab judges a job's age by when it started, not by its commit. So running an old pipeline's manual deploy blocks a newer pipeline's manual deploy.

Rolling back#

Under Operate › Environments, each earlier successful deployment offers Rollback environment, and the latest offers Re-deploy to environment. Both simply run that deployment's job again. That works only if the job holds everything the deploy needs. At MegaCorp it does: re-running the job writes the old IMAGE_REF into the GitOps repository.

Figure A deploy job won't run. Ask these in order.#

As text

Is it waiting for approvals? Yes: Approve it under Operate › Environments, then run it; approval doesn't start it. No: the next step.
Is the environment protected, and you aren't allowed to deploy? Yes: Ask a Maintainer to add you, or someone allowed to run it. No: the next step.
Is another deploy in the same resource group running? Yes: It waits its turn. No: the next step.
Is it older than the latest deployment? Yes: Prevent outdated deployment jobs blocks it; run a newer pipeline. No: the next step.
Is there a deploy freeze? Yes: GitLab blocks deployments until the freeze ends. No: the next step.
Check its rules: a manual job waits for someone to run it

Spot the bugApproved, but not deployed#

The production environment in megacorp/platform/deployer needs one approval. A release manager approves a payments-api deployment at 17:55 and goes home. At 09:00 the next day, production still runs the old version. The deployer pipeline shows the deploy job waiting.

megacorp/platform/deployer.gitlab-ci.ymldeploy

variables:
  ENVIRONMENT: $[[ inputs.environment ]]

deploy:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  resource_group: $APP-$ENVIRONMENT
  environment:
    name: $ENVIRONMENT/$APP
  script:
    - mc deploy "$ENVIRONMENT" "$IMAGE_REF"
  rules:
    - if: $IMAGE_REF == null
      when: never
    - if: $ENVIRONMENT == "production"
      when: manual
    - when: on_success

QuestionWhy didn't the deploy happen after the approval?

Show the answer

Production deploys are when: manual in the deployer's rules. A deployment approval unblocks the job, but never starts it. Someone allowed to deploy must still select Run. The approval was needed, but it wasn't enough.

Make running the job part of the release manager's routine. See who may deploy, and approvals.

Next34Deploy patterns and GitOps3 min

Deploy patterns and GitOps#

Part VII · Delivery · Chapter 34· 3 min read

A pipeline can deploy in two ways:

Push: a job changes the target itself, for example by updating an ECS service or running helm against a cluster.
Pull, or GitOps: a job writes the new version into a Git repository, and a controller running inside the cluster notices and applies it.

MegaCorp pulls. With pull, a green pipeline means only that the request was written. Whether the new version is running is something only the controller and the cluster can tell you.

Push and pull#

Pattern	The pipeline needs	When the job finishes, you know
Push to ECS	AWS keys for the service's account	the service is stable, if the job waits for it (chapter 31)
Push to EKS	cluster access, through GitLab's agent or AWS	the release applied, and ready, if the job waits
Pull (GitOps)	write access to the GitOps repository only	only that the change was committed

Pull keeps production credentials out of the pipeline, since only the controller touches the cluster. The price is that the pipeline can't see the deploy happen.

MegaCorp's handoff#

A service's deploy job triggers the deployer's pipeline. The deployer's job runs mc deploy, which commits the new image into the GitOps repository:

megacorp/devops/ci-toolsbin/mcdeploy

deploy)
  env="${1:?usage: mc deploy <environment> <image-ref>}"
  image="${2:?usage: mc deploy <environment> <image-ref>}"
  mc_log "promoting $image to $env through the GitOps repository"
  git clone --depth 1 \
    "https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
  yq -i ".apps.\"${APP}\".image = \"${image}\"" "gitops/envs/${env}/values.yaml"
  git -C gitops commit -am "deploy ${APP} ${image} to ${env}"
  git -C gitops push origin HEAD:main
  ;;

Figure The GitOps handoff. The pipelines end when the new image is committed to gitops-config. The controller's sync and the cluster's rollout happen later, and GitLab doesn't see them.#

The trigger jobs use strategy: depend, so a service's pipeline turns red if the deployer's pipeline fails (chapter 10). But the deployer's pipeline is green as soon as the push succeeds. Anything that goes wrong after that never reaches GitLab: a controller that hasn't synced, or pods that crash.

GitLab's agent for Kubernetes#

The agent for Kubernetes is a program installed in a cluster that keeps a connection open to GitLab. It is available in every tier, and does two jobs:

Push, through the agent. Projects and groups listed under ci_access in the agent's configuration get a kubeconfig file in every job, at $KUBECONFIG, with a context for each agent they may use. kubectl and helm then reach the cluster without any cluster credentials stored in GitLab.
Pull, with Flux. GitLab integrates the Flux controller. When the agent sees a push to a repository Flux watches, it can tell Flux to sync at once. Otherwise Flux checks on its own timer.

Note

The agent can trigger an immediate sync only for its own configuration project and for public projects. For a private GitOps repository, such as MegaCorp's, Flux syncs on its timer, so a deploy can appear minutes after the pipeline goes green.

Figure The pipeline is green, but the new version isn't running. Ask these in order.#

As text

Did the deployer's job run, or is it a manual job waiting? No: A waiting manual job deploys nothing (chapter 33). Yes: the next step.
Did the commit reach the GitOps repository? No: Read the deploy job's log; the push may have been refused. Yes: the next step.
Has the controller synced since that commit? No: It syncs on a timer; wait, or ask for a sync. Yes: the next step.
Did the new pods start and stay up? No: The cluster's events show why; the pipeline never will. Yes: the next step.
Check which image the GitOps repository now names

Spot the bugGreen, but still the old version#

At 10:02 a payments-api pipeline on main goes green, including deploy-staging and the deployer pipeline it triggered. At 10:05, staging still serves the old version. MegaCorp's clusters run Flux, connected through GitLab's agent, and gitops-config is a private project. By 10:12 the new version is live, and nobody changed anything.

QuestionWhat happened between 10:02 and 10:12, and where would you have looked at 10:05?

Show the answer

The pipelines ended when mc deploy pushed the new image to gitops-config. Because that repository is private, the agent couldn't trigger an immediate sync, so Flux picked the change up on its next scheduled check. The pods then rolled out. None of this shows in GitLab.

At 10:05, check the latest commit in gitops-config, then the controller's last sync, then the cluster's rollout. See MegaCorp's handoff.

Next · Part VIII35The decoding method3 min

The decoding method#

Part VIII · Decode and debug · Chapter 35· 3 min read

Everything about a pipeline job comes down to five questions: where it is defined, when it runs, where it runs, what it executes, and what goes in and out. Chapter 1 named them. This chapter turns them into a procedure you can follow on any job in any organisation, and shows it on one of MegaCorp's jobs from start to finish.

When something is already wrong, start from the symptom instead. The decision tree at the end sends you to the right chapter of symptom cards.

The five questions, in order#

Question	Look first at	Then	Chapters
Where is it defined?	the job's name in the pipeline graph, then Full configuration	its `extends` chain, `!reference` tags and includes; if it's in no file, policies and settings	5–13
When does it run?	`workflow:rules`, then the job's own final `rules`	`needs` and stages; compare pipeline types in a truth table	14–18
Where does it run?	the job's page: the runner's name and tags	the runner's executor, and the job's image	22–23
What does it execute?	`before_script`, `script` and `after_script` in Full configuration	the scripts, functions and tools those lines call inside the image	12 and 22
What goes in and out?	the variables and secrets it needs, from YAML and from settings	its artifacts, caches, reports, and what it pushes or deploys	19–25

Answer them in this order, because each answer narrows the next. You can't read a job's rules until you know where they come from. You can't judge its inputs until you know which runner and image it had.

Each question has one trap that catches most people:

Defined: a policy's job is in no file at all (chapter 13).
When: a job's own rules replace a template's list completely (chapter 7).
Where: the runner's configuration changes jobs, and no project can see it (chapter 22).
What: the script calls a tool or function from the image, whose code is somewhere else (chapter 12).
In and out: a settings variable beats anything in the YAML (chapter 19).

One job, decoded#

Here is payments-api's image-build, decoded using nothing but the specimen:

Question	The answer for image-build	Found in
Where is it defined?	`image-build` in java-service.yml extends `.image-build` in container.yml, which extends `.megacorp-base` in base.yml	the Full configuration view, and chapter 13's steps
When does it run?	`.megacorp-base`'s rules: merge requests, `main` and release tags, never schedules; it needs `maven-build`	the `.rules` library, and chapter 18's truth table
Where does it run?	on the megacorp-shared runner, in a Kubernetes pod, from the buildah image	`default: tags`, the runner's configuration, and the job's `image`
What does it execute?	the ci-tools functions and the AWS login from the snippets, then Buildah's login, build and push	the `!reference` lines in `before_script`, and ci-lib.sh
What goes in and out?	in: an ID token, `AWS_ROLE_ARN`, `ECR_REGISTRY`, `AWS_REGION` and the jar from `maven-build`; out: an image in ECR, and `IMAGE_REF` in a dotenv report	its variables and `needs`, and its artifacts

After five short answers, the job holds no more surprises.

Starting from a symptom#

Figure The debugging decision tree. Start from what you can see, pick one of four groups, make its first three checks, then open that group's chapter of symptom cards.#

Each symptom card names a symptom, then its cause, how to confirm it, and how to fix it. The cards come in four chapters, one per group, with the most common causes first: didn't run, failed, wrong result, and stuck or drifting. For your own pipelines, the worksheet is a printable form of the five questions.

Spot the bugDecode publish-docs#

payments-api defines one job of its own. Answer the five questions for it:

megacorp/payments/payments-api.gitlab-ci.ymlinherit

publish-docs:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
  inherit:
    default: [tags, retry]
    variables: false
  script:
    - mkdocs build --strict
  rules:
    - !reference [.rules, default-branch]

QuestionWhere is it defined, when does it run, where does it run, what does it execute, and what goes in and out? Which answer should worry its owners?

Show the answer

Defined in payments-api's own file. Nothing extends it, and inherit takes only tags and retry from default:.
When: in pipelines for main only, through the default-branch rule.
Where: on the megacorp-shared runner, because it inherits tags, in the docs:1.4 image.
What: mkdocs build --strict.
In and out: no YAML variables, because variables: false, but settings variables still arrive. Out: nothing at all.

The last answer is the worry. The job builds the documentation site and then ends. With no artifacts and no deploy step, the site is thrown away with the job's pod. Its name promises a publication that never happens. See the five questions.

Next36Field guide to enterprise setups2 min

Field guide to enterprise setups#

Part VIII · Decode and debug · Chapter 36· 2 min read

Large organisations build their pipelines in a handful of recognisable ways. Each leaves fingerprints: signs in a repository, a pipeline graph or a job log that tell you which setup you are looking at. Recognise the setup first, and you know where the answers live before you start reading YAML. This chapter lists ten setups, with their fingerprints and where to confirm each one.

Ten setups and their fingerprints#

Setup	Fingerprints	Confirm it	Chapters
The golden pipeline	a `.gitlab-ci.yml` of a few lines: one `include: project:` of a central file at a tag, a few variables	Full configuration shows dozens of jobs that the file never names	7–8
Components	`include: component:` lines ending in `@1.2.0`, with `inputs:`	the component's project, and its `spec: inputs`	9
The generated monorepo	a job that writes YAML, then a trigger that runs it; child pipelines to the right of the graph	the generator script, and the trigger's `include: artifact:`	10
The central deployer	deploy jobs that are `trigger: project:` jobs; environments that live in another project	the deployer project's pipeline and environments	10, 33
GitOps delivery	deploy jobs that only commit to a configuration repository	that repository's history, and the controller's sync status	34
Policy-injected jobs	jobs named like `secret-detection-1`, or in `.pipeline-policy-pre` and `.pipeline-policy-post` stages	Secure › Policies	11, 29
The invisible runner layer	behaviour that differs by runner; environment variables nobody set in YAML	the runner's name in the job log, and the runner's configuration	12, 22
The toolbox image	scripts calling commands such as `mc` or shell functions that the repository doesn't contain	the job's image, and the image's own repository	12
Cloud access by identity	`id_tokens:` in templates, a login snippet, and no cloud keys in settings	the role's trust policy, and `aws sts get-caller-identity`	21, 31
The 2019 legacy file	anchors and `<<:` merge keys, `only:` and `except:`, `include: remote:`, trigger tokens	the parse phase: anchors are resolved in Full configuration	6, 8, 16

MegaCorp shows all ten: payments-api is a golden pipeline with a component, mono is a generated monorepo, the deployer is central and GitOps-based, and billing-batch is the 2019 legacy file.

No .gitlab-ci.yml, but pipelines run#

This is the most disorienting fingerprint of all. Five mechanisms can produce it, and each has a place to confirm it:

Figure A project has no .gitlab-ci.yml, yet pipelines run. Check these in order.#

As text

Does Settings › CI/CD › General pipelines name a configuration file elsewhere? Yes: The pipeline comes from that file, perhaps in another project. No: the next step.
Are the only jobs scanners with names like secret-detection-1? Yes: A scan execution policy creates a pipeline file for the project implicitly. No: the next step.
Is a pipeline execution policy set to override_project_ci? Yes: The policy's configuration replaces the project's entirely. No: the next step.
Is Auto DevOps turned on for the project, group or instance? Yes: GitLab's Auto DevOps pipeline runs. No: the next step.
Does the project belong to a compliance framework with its own pipeline? Yes: A compliance pipeline runs; these are deprecated in favour of policies. No: the next step.
Check the pipeline's source, and its first job's origin (chapter 13)

MegaCorp's web-portal is the first case. Its settings point at pipelines/web.yml in ci-templates, so its repository needs no pipeline file (chapter 11):

megacorp/web/web-portalSETTINGS.ymlconfig-path

ci_cd_configuration_file: pipelines/web.yml@megacorp/devops/ci-templates

Spot the bugWhich setup is this?#

You join a team and open their newest pipeline. You notice three things:

the project's .gitlab-ci.yml is nine lines long, and includes one file from platform/ci-library at ref: v7.1.0
one job, deploy-prod, is a trigger job pointing at platform/release
two jobs are called sast-1 and secret-detection-1

QuestionWhich three setups are in play, and where would you look first for each?

Show the answer

A golden pipeline: read platform/ci-library at v7.1.0, starting from Full configuration.
A central deployer: the real deploy happens in platform/release's pipeline and environments.
Policy-injected jobs: the numbered scanners come from a scan execution policy, so check Secure › Policies.

See ten setups and their fingerprints.

Next37Idioms2 min

Idioms#

Part VIII · Decode and debug · Chapter 37· 2 min read

Central teams repeat a small number of patterns, called idioms, in every organisation. Each one solves a real problem for the team that writes the templates. Each also hides something from the people who read the pipelines. Learn to spot them, and an unfamiliar pipeline stops looking unfamiliar. For each idiom, this chapter shows what it looks like, why central teams use it, and how it confuses a reader.

A rules library, pulled in with !reference#

This is the idiom that confuses most readers, so here it is in full. MegaCorp keeps its conditions in one hidden job:

megacorp/devops/ci-templatestemplates/rules.ymlrules-library

.rules:
  mr:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  default-branch:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  release-tag:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
  never-on-schedule:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never

Jobs assemble their rules from it: - !reference [.rules, mr].

Why central teams use it: "which pipelines does this job run in" is decided once, with one name for each condition. Every template agrees on what a release tag looks like.
What it hides: the conditions live in another file, so a job's rules read like a list of names. The names say nothing about the order, and in rules the order decides the outcome, because the first match wins. If you replace the job's rules:, every referenced rule disappears with them (chapter 7).

The catalogue#

Idiom	Looks like	Why central teams use it	How it confuses you	Chapters
The hidden base job	`extends: .megacorp-base`	shared rules, variables and expiry in one place	settings arrive from files you never opened; its variables beat your top-level ones	7
Script snippets	`- !reference [.snippets, aws_login]`	the same few script lines in every template	a job's own `before_script` drops them; the functions they call live in the image	7, 12
Variables as parameters	`image: maven:3.9-jdk${JAVA_VERSION}`	one template serves many projects	the default is in the template; a settings variable beats your YAML	7, 19
Toggles	`if: $SKIP_SONAR == "true"` then `when: never`	an escape hatch without editing the template	a toggle set in settings is invisible; policy jobs ignore toggles	27, 29
The pinned include	`include: project: … ref: v4.2.0`	projects upgrade when they choose to	what runs is that tag's content, not the central repo's current version; a tag can be moved	8
Components with inputs	`component: …/sonar-scan@2.1.0` with `inputs:`	typed, versioned settings	inputs are filled in before anything else, even into job names	9
The dotenv hand-off	`echo "IMAGE_REF=…" >> build.env`	passes a build's result to later jobs	no file shows the value; `needs` and `dependencies` can block it; rules can't see it	24
The trigger to a deployer	`trigger: project: …/deployer` with `strategy: depend`	one team controls every environment	the deploy happens in another project's pipeline	10, 33
Never on schedule	the first rule excludes `schedule`	nightly pipelines run only what they should	jobs are silently absent from scheduled pipelines	16, 18
Opting out with inherit	`inherit: default: [tags, retry]`	a job that shouldn't get the shared image or scripts	which defaults still apply is easy to misread	7
No duplicate pipelines	a workflow rule with `$CI_OPEN_MERGE_REQUESTS`	one pipeline per push, not two	once a merge request opens, branch pipelines stop appearing	15
Build once, deploy by reference	the image tagged with `$CI_COMMIT_SHA`, then passed on	what is tested is what ships	a pipeline that rebuilds breaks the promise quietly	32

Most real pipelines combine several idioms in a single job. Recognising each one tells you which file to open next.

Spot the bugCount the idioms#

This is the template behind payments-api's image-build:

megacorp/devops/ci-templatestemplates/container.ymlimage-build

.image-build:
  extends: .megacorp-base
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
    IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
    - buildah push "$IMAGE:$CI_COMMIT_SHA"
    - echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
  artifacts:
    reports:
      dotenv: build.env

QuestionWhich idioms from this chapter can you find, directly or through what the job extends?

Show the answer

The hidden base job: extends: .megacorp-base, which also brings in the rules library and never on schedule.
Script snippets: the two !reference [.snippets, …] lines.
Variables as parameters: AWS_ROLE_ARN and ECR_REGISTRY, which a project could override.
The dotenv hand-off: IMAGE_REF written to build.env.
Build once, deploy by reference: the image tagged with $CI_COMMIT_SHA.

The job reaches payments-api through a pinned include as well: java-service.yml includes it at v4.2.0. See the catalogue.

Next38Worksheet: decode your own pipeline1 min

Worksheet: decode your own pipeline#

Part VIII · Decode and debug · Chapter 38· 1 min read

Print this page, or copy it into a private note, and fill it in for one of your own projects. It is the five questions, plus the few facts about your organisation that make every answer easier. Keep your answers private, because they describe your organisation's systems.

Once for your organisation#

Most of these answers are the same for every project, so you only find them once:

Question	Where to find the answer	Your answer
Where do the central templates and components live?	the `include:` lines of any project, and the platform team's group	…
Which versions do projects pin, and who moves them?	the `ref:` of each include, and the templates' release notes	…
Which runners run your jobs, and which tags do they need?	Settings › CI/CD › Runners, and `default: tags`	…
Which executor do those runners use?	the first lines of any job log	…
Which CI/CD variables do your groups and the instance set?	each group's Settings › CI/CD › Variables, if you can see it	…
Which security policies apply?	Secure › Policies	…
How do jobs reach the cloud?	`id_tokens:` in the templates, and the login script	…
Where do deployments really happen?	trigger jobs, environments, and any GitOps repository	…

For each job you need to understand#

Question	Where to look	Your answer
Where is it defined?	Full configuration; its `extends` chain; if it's in no file, policies	…
When does it run?	`workflow:rules`, its final `rules`, `needs`; compare pipeline types	…
Where does it run?	the job page's runner and tags; the job's image	…
What does it execute?	its scripts, and the tools and functions they call	…
What goes in and out?	variables, secrets, artifacts, caches, reports, images, deployments	…

When something goes wrong#

Write down, in this order:

The symptom, in one sentence, and its group: didn't run, failed, wrong result, or stuck or drifting (chapter 35).
The pipeline and job, with links, and what changed since the last good run: a commit, a setting, a template version, a runner, a date.
The cards you tried, and what each confirmation showed.
The fix, and who had to make it: you, the platform team, the security team or the cloud team.

Questions for your platform team#

Some answers aren't visible from a project at all. These are worth asking once:

What do the runners add to every job: environment variables, a pre_build_script, limits?
Which of the group's CI/CD variables are protected, and why?
Which stored tokens do the pipelines depend on, who owns them, and when do they expire?
How are template versions released, and can a tag ever move?
Which security policies are enforced, and on which projects?

Next39Cards: didn't run7 min

Cards: didn't run#

Part VIII · Decode and debug · Chapter 39· 7 min read

Use these cards when a job is missing from a pipeline, when the pipeline wasn't created, or when a job ran somewhere it shouldn't have. The most common causes come first. Each card gives the symptom, the cause, how to confirm it, and the fix. One more card for this group is in chapter 7: your rules replaced the template's.

Didn't run, or ran when it shouldn'tRules and conditions#

A job runs in branch pipelines but is missing from merge request pipelines, or the other way round.

Cause

Merge request pipelines and branch pipelines are different pipelines, with different variables. CI_COMMIT_BRANCH, for example, isn't set in merge request pipelines, so a rule that tests it is false there. The job has no rule for $CI_PIPELINE_SOURCE == "merge_request_event".

Confirm

Open both pipelines and compare their job lists. Each pipeline's page shows its source. Then read the job's final rules in Build › Pipeline editor › Full configuration.

Fix

Add a rule for merge request pipelines, or reuse your organisation's rule from its rules library with !reference. See merge request pipelines.

Didn't run, or ran when it shouldn'tRules and conditions#

A push creates no pipeline at all, or Run pipeline refuses to start one.

Cause

workflow:rules decide whether a pipeline exists. None of them matched this push, or the one that matched said when: never.

Confirm

Run pipeline and the API answer with:

The pipeline did not run. Review the workflow:rules configuration for the pipeline.

Read workflow: in Full configuration.

Fix

Add the missing pipeline source or branch to the workflow rules, or push where the workflow allows. See the first matching rule.

Didn't run, or ran when it shouldn'tRules and conditions#

The workflow allows the pipeline, yet it isn't created, because no job would have been in it.

Cause

Every job's rules excluded it for this source and branch. MegaCorp's deployer does this deliberately: all its jobs need IMAGE_REF, which only a trigger supplies.

Confirm

GitLab says:

The resulting pipeline would have been empty. Review the rules configuration.

The Validate tab, run for the same branch, shows no jobs.

Fix

Provide what the rules expect, such as the variable a trigger would send, or change the rules. See when a pipeline is missing.

Didn't run, or ran when it shouldn'tRules and conditions#

Pushing a tag such as v1.5.0-rc1 creates no pipeline, and shows no error anywhere.

Cause

The workflow allows tag pipelines, but no job's rules match this tag. MegaCorp's release rule accepts only tags shaped like v1.5.0, exactly ^v\d+\.\d+\.\d+$. A pipeline with no jobs isn't created, and on a push GitLab doesn't say why.

Confirm

Run Validate for the tag: it shows no jobs. Starting a pipeline for the tag by hand gives The resulting pipeline would have been empty. Review the rules configuration.

Fix

Name tags the way the rules expect, or widen the rule if release candidates should build. See what the truth table says.

Didn't run, or ran when it shouldn'tScanners and policies#

GitLab's scanner jobs, such as secret_detection, appear in branch pipelines but never in merge request pipelines.

Cause

GitLab's scanner templates run their jobs in merge request pipelines only when AST_ENABLE_MR_PIPELINES is "true". Otherwise the jobs run in branch pipelines only.

Confirm

In Full configuration, the job's rules contain the AST_ENABLE_MR_PIPELINES conditions. The branch pipeline for the same commit has the job.

Fix

Set AST_ENABLE_MR_PIPELINES: "true", in the YAML or in settings. The job then moves to the merge request pipeline. Or redefine the job's rules. See the merge request gap.

Didn't run, or ran when it shouldn'tRules and conditions#

A job runs on every push, but never in scheduled pipelines.

Cause

Its rules start with a rule that excludes schedules. MegaCorp's .megacorp-base begins with !reference [.rules, never-on-schedule], so every job that extends it skips scheduled pipelines.

Confirm

The job's final rules in Full configuration start with the schedule rule, and the pipeline's source is schedule.

Fix

If the job should run on a schedule, give it rules without that line, or ask the template's owners to change the base. See each job's final rules.

Didn't run, or ran when it shouldn'tRules and conditions#

A job shows a play button and never starts. The stages after it may wait too.

Cause

The job is manual, through when: manual on the job or in the rule that matched. Manual jobs run only when someone starts them. A manual job added by a rule blocks the later stages, unless it has allow_failure: true.

Confirm

The job page offers Run, and its rules or when in Full configuration say manual.

Fix

Run it. If nothing should wait for it, add allow_failure: true to that rule. See manual jobs.

Didn't run, or ran when it shouldn'tRules and conditions#

A job with rules: changes runs on the first push of a new branch, or on a tag, although none of its files changed.

Cause

For new branches, for tags, and for pipelines without a push event, rules: changes is always true. There is nothing to compare with. On the local GitLab, mono's first push ran its docs job although there was no docs/ folder.

Confirm

The pipeline is a tag pipeline, or the branch's first pipeline.

Fix

Compare against a fixed branch with changes: compare_to:, or add a rule that excludes tags. See rules about files.

Didn't run, or ran when it shouldn'tRules and conditions#

Every push to a merge request's branch creates two pipelines, and every job runs twice.

Cause

The workflow allows both a branch pipeline and a merge request pipeline for the same push.

Confirm

The pipelines list shows two pipelines for the same commit, one of them marked as a merge request pipeline.

Fix

Add a workflow rule that skips branch pipelines while a merge request is open, using $CI_OPEN_MERGE_REQUESTS. See duplicate pipelines.

Didn't run, or ran when it shouldn'tScanners and policies#

A scanner job has vanished from every pipeline, and nobody changed the YAML.

Cause

Someone set its toggle, such as CONTAINER_SCANNING_DISABLED, in the project's or a group's CI/CD settings. The template's first rule removes the job. MegaCorp's own container scanning rule accepts only "true".

Confirm

Look for the variable in the project's and each group's CI/CD settings. Its rule is the job's first.

Fix

Remove the setting if the scan should run. A scan execution policy's jobs are never affected by toggles. See toggles against policies.

Didn't run, or ran when it shouldn'tScanners and policies#

The SAST template is included, but no SAST job appears.

Cause

Each SAST analyzer's job exists only when files that it reads are in the repository, such as *.java or *.py for semgrep-sast.

Confirm

In Full configuration, compare the analyzer's exists patterns with the repository's files.

Fix

None is needed if the repository has no code in those languages. Otherwise check that the source files are where the patterns look. See what the templates add.

Didn't run, or ran when it shouldn'tScanners and policies#

The dependency scanning template is included, but its jobs never appear.

Cause

Those jobs' rules require the dependency_scanning feature, which comes with Ultimate. On Free and Premium, the template adds nothing, without an error.

Confirm

The jobs' rules in Full configuration test $GITLAB_FEATURES.

Fix

Use Ultimate, or run another dependency scanner as an ordinary job (chapter 28).

Didn't run, or ran when it shouldn'tRules and conditions#

The pipeline isn't created, and the error names a job that another job needs.

Cause

The needed job's rules left it out of this pipeline, while the job that needs it is still in.

Confirm

GitLab's message, captured here with other job names, says:

'unit-tests' job needs 'compile' job, but 'compile' does not exist in the pipeline.
This might be because of the only, except, or rules keywords. To need a job that
sometimes does not exist in the pipeline, use needs:optional.

Fix

Mark the need as optional, or give both jobs matching rules:

unit-tests:
  needs:
    - job: compile
      optional: true

See needs.

Didn't run, or ran when it shouldn'tYAML itself#

The pipeline isn't created, and the error says a stage doesn't exist.

Cause

A job names a stage missing from stages:. Often a project redefines stages: without test, which GitLab's scanner templates use.

Confirm

The message names the job, its stage and the stages that exist:

orphan job: chosen stage nowhere does not exist; available stages are .pre, build, .post

Fix

Add the stage to stages:, or move the job to one that exists. See stages.

Didn't run, or ran when it shouldn'tYAML itself#

The pipeline isn't created, because of a rule written in a way that looks reasonable.

Cause

Two common forms fail. A negation written as !( … ) is invalid. A quoted variable, as in "$VAR" == "x", breaks the YAML itself.

Confirm

CI Lint shows one of these, captured on the local GitLab:

jobs:job:rules:rule if invalid expression syntax
(): did not find expected key while parsing a block mapping at line 5 column 7

Fix

Use != for negation, and never quote the variable: if: $CI_COMMIT_BRANCH != "nope". See writing conditions.

Didn't run, or ran when it shouldn'tVariables#

A rule tests a variable that an earlier job sets, and never matches.

Cause

Rules are decided when the pipeline is created, before any job runs. Values from a dotenv report, or from export in a script, don't exist yet.

Confirm

Find where the variable is set. If it is set by a job, no rule can see it.

Fix

Make the decision inside the job's script instead, or in a child pipeline that a later job generates. See when each thing is decided.

Didn't run, or ran when it shouldn'tVariables#

A rule such as if: $DEPLOY_KEY matches on main, but never on feature branches.

Cause

The variable is protected. Protected variables exist only in pipelines on protected branches and tags, so elsewhere the rule sees an unset variable.

Confirm

In the CI/CD settings where it is defined, the variable has Protect variable ticked.

Fix

Don't gate jobs on secrets. Test the branch, or an ordinary variable, instead. See who gets a protected value.

Next40Cards: failed6 min

Cards: failed#

Part VIII · Decode and debug · Chapter 40· 6 min read

Use these cards when a job, or the pipeline, turned red. Start with the job log's first error, not its last line. The most common causes come first. Each card gives the symptom, the cause, how to confirm it, and the fix.

FailedMerging and inheritance#

A job fails at once, saying a company function or tool, such as mc_log, can't be found. The same template works in other projects.

Cause

The job sets its own before_script. That replaces the template's or the default's list completely, including the line that loaded the functions.

Confirm

In Full configuration, the job's before_script shows only your lines.

Fix

Put the template's lines back with !reference, then add yours. See the before_script exercise.

FailedImages and Docker#

The job fails before its script, while the runner is pulling the job's image.

Cause

The image is in a private registry, and the runner has no credentials for it. The job's own script can't help, because the runner pulls the image before the job starts.

Confirm

The error appears in the job log's preparation lines, before any of your commands.

Fix

Give the runner credentials with DOCKER_AUTH_CONFIG, or a credential helper on the runner. For images in the same GitLab, allow your project's job token in the image's project. See private registries.

FailedArtifacts, cache and data#

A job fails before its script starts.

Cause

It needs artifacts from an earlier job, and can't get them. They may have expired, the job may not be a dependency, or the user may lack permission.

Confirm

The job says:

This job could not start because it could not retrieve the needed artifacts.

Fix

Retry the earlier job first, so that new artifacts exist, or run a new pipeline. Lengthen expire_in if jobs routinely run later. See artifacts.

FailedSecrets, tokens and identity#

A job that signs in to something fails with an authentication error, but only on feature branches or in merge request pipelines.

Cause

Its token is a protected variable. Protected values reach only pipelines on protected branches and tags, so elsewhere the variable is empty.

Confirm

In the CI/CD settings, the variable is protected. In the job, print the variable's length, never its value.

Fix

Decide with the token's owners: an unprotected token with fewer rights, a separate token for merge requests, or no sign-in there. See who gets a protected value.

FailedSecrets, tokens and identity#

The AWS login step fails in some pipelines, for example only tag pipelines, with this error:

Cause

The role's trust policy doesn't accept this token. Usually the sub pattern allows branches but not tags, or the audience is wrong. A wildcard written with StringEquals never matches.

Confirm

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

Print the token's claims, not the token, and compare sub and aud with the trust policy.

Fix

Ask the account's owners to add a pattern for the missing pipelines, using StringLike. See when AWS says no.

FailedVariables#

A trigger job fails, and the downstream pipeline is never created.

Cause

The downstream project limits who may set pipeline variables. The variables the trigger job sends, including top-level ones it inherits, count as pipeline variables.

Confirm

The trigger job shows:

Failed - (downstream pipeline can not be created, Insufficient permissions to set pipeline variables)

Fix

Send inputs instead of variables, or ask the downstream project to allow the role. See values given when the pipeline starts.

FailedSecrets, tokens and identity#

A call to another GitLab project fails with 404 Not Found, although the project exists.

Cause

The job used CI_JOB_TOKEN. The other project hasn't added yours to its job token allowlist, or the person who started the pipeline isn't a member there.

Confirm

A refused clone says:

remote: The project you were looking for could not be found or you don't have permission to view it.

Fix

A Maintainer of the target adds your project, or its group, under Settings › CI/CD › Job token permissions. See the job token.

FailedGit in CI#

A script that uses Git history fails in CI, often only on big merge requests. Or a retried old job reports unresolved reference.

Cause

The job has a shallow clone: only the newest commits, 20 by default, and only the pipeline's own ref.

Confirm

The commit the script needs is older than the clone's depth, or is on a branch the job didn't fetch.

Fix

Raise GIT_DEPTH for that job, or fetch what the script needs, for example with git fetch --unshallow. See shallow clones.

FailedImages and Docker#

With a tool's own image, such as amazon/aws-cli, the job fails without running your script. The same job works on another runner.

Cause

The image's entrypoint runs the tool, not a shell. The Docker executor keeps the entrypoint. The Kubernetes executor ignores it by default.

Confirm

Compare the runners' executors. The failing one uses Docker.

Fix

Set entrypoint: [""] under image:. See the job's image.

FailedImages and Docker#

Tests can't reach a service, such as a database, and the log warns: *** WARNING: Service XYZ probably didn't start properly.

Cause

The service lacked a setting it needs to start, such as its password. That value is a CI/CD variable from settings, and those never reach service containers.

Confirm

The variable is defined in settings, not in the YAML. CI_DEBUG_SERVICES: "true" shows the service's own log, but can reveal masked values.

Fix

Re-assign it in the YAML under a different name, such as POSTGRES_PASSWORD: $TEST_DB_PASSWORD. See services.

FailedRunners and execution#

A job stops at, say, 30 minutes, although its timeout says 2 hours.

Cause

The runner has a shorter maximum job timeout. A runner's maximum wins over the job's own timeout and the project's.

Confirm

The runner's settings show its maximum job timeout.

Fix

Use a runner with a longer maximum, or split the job. See time limits.

FailedRules and conditions#

A trigger job for a child pipeline fails in merge request pipelines, but works in branch pipelines.

Cause

The child configuration's jobs have no rules. In a merge request pipeline, a child whose jobs have no rules gets no jobs at all, so it can't be created. On the local GitLab, mono's docs trigger failed this way in every merge request pipeline.

Confirm

The trigger job's failure reason is downstream_pipeline_creation_failed.

Fix

Give the child's jobs rules that allow merge request pipelines. See child pipelines.

FailedVariables#

A command receives the literal text $MY_VAR instead of a value.

Cause

The job defines MY_VAR: $MY_VAR. The job's own definition hides the default of the same name, so there's nothing to expand.

Confirm

The job's variables in Full configuration re-use the name.

Fix

Use a different name for the job's variable. See variables in YAML.

FailedArtifacts, cache and data#

A later job can't find a file that an earlier job should have made.

Cause

The earlier job's artifacts: paths matched nothing: the path is wrong, or the file wasn't created.

Confirm

The earlier job's log says No files to upload.

Fix

Fix the path, which is relative to the project directory, or the command that makes the file. See artifacts.

FailedSecrets, tokens and identity#

Every AWS login fails, in every project, with an error about a verification key.

Cause

AWS must download GitLab's signing keys from the instance, and can't reach it. This happens with a self-managed GitLab behind a firewall.

Confirm

An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn't retrieve verification key from your identity provider

Fix

Make the instance's OIDC keys reachable, or publish them elsewhere, as GitLab's documentation describes. See small things that bite.

FailedDeployments and delivery#

A deploy job fails straight away, in an older pipeline.

Cause

Prevent outdated deployment jobs is on, and a newer deployment already ran. GitLab judges a job's age by when it started.

Confirm

The deployment job is older than the latest deployment, and therefore failed.

Fix

Deploy from a newer pipeline, or use Rollback environment to go back deliberately. See one at a time, newest wins.

FailedDeployments and delivery#

An image push to ECR fails with ImageTagAlreadyExistsException.

Cause

The repository has immutable tags, and this tag was already pushed, often by an earlier pipeline that built the same commit.

Confirm

ECR already lists the tag, with an earlier push time.

Fix

Don't rebuild what was already built: deploy the existing image, and add new tags with put-image. See promoting without rebuilding.

FailedImages and Docker#

A job fails at once with a tool's small image, before any of your commands.

Cause

A job's image must contain sh or bash, and grep, because the runner sends the script to a shell. Very small images leave them out.

Confirm

Run the image locally and look for sh.

Fix

Use a variant of the image that includes a shell, or build one. See the job's image.

Next41Cards: wrong result6 min

Cards: wrong result#

Part VIII · Decode and debug · Chapter 41· 6 min read

Use these cards when everything passed, but the result is wrong: the wrong value, the wrong image, the wrong version, or nothing at all. These are the hardest problems, because no log line points at them. The most common causes come first.

Passed, but did the wrong thingVariables#

You change a variable in the YAML, and the job still uses the old value.

Cause

The same name is set in CI/CD settings, on the project, a group or the instance. A settings variable beats every value in YAML, even a job's own. MegaCorp's group sets MAVEN_CLI_OPTS this way.

Confirm

The job log never says where a value came from, and Full configuration shows only YAML. Look for the name in the project's CI/CD variables, then each group's.

Fix

Change or remove the settings variable, or use a name the settings don't set. See who wins.

Passed, but did the wrong thingVariables#

A variable set at the top of your .gitlab-ci.yml is ignored in some jobs, which use a template's value instead.

Cause

A job's own variables, including those it gets through extends, beat top-level ones. MegaCorp's .megacorp-base sets MC_TEAM: unknown, so every job that extends it reports unknown.

Confirm

In Full configuration, the job's own variables list the name.

Fix

Set the value on the job, or ask the template's owners to remove the job-level default. See the MC_TEAM exercise.

Passed, but did the wrong thingDeployments and delivery#

Production shows a bug that staging never had, although both deployed the same image tag.

Cause

A later pipeline built the same commit again and pushed the same tag. With mutable tags, the rebuild replaced the image that staging tested.

Confirm

ECR shows that the tag was pushed after the staging deploy, and the release pipeline contains an image build.

Fix

Build once, and promote the existing image. Make tags immutable, so that a rebuild fails loudly. See from build to deploy.

Passed, but did the wrong thingDeployments and delivery#

The deploy pipeline is green, but the old version is still running.

Cause

With GitOps, green means only that the new version was written to Git. The controller may not have synced yet, or the rollout failed in the cluster. A manual deploy job that is still waiting deploys nothing, either.

Confirm

Check the GitOps repository's latest commit, then the controller's last sync, then the cluster's rollout.

Fix

Wait for the sync or ask for one, and fix any failed rollout. See MegaCorp's handoff.

Passed, but did the wrong thingScanners and policies#

A failed quality or security gate didn't stop a deploy.

Cause

The gate's job is allowed to fail. Or the deploy's needs don't include it, so the deploy never waits for it.

Confirm

The gate job shows an orange warning, and the deploy job's needs name other jobs only.

Fix

Decide with the owners whether the gate should block. If it should, remove allow_failure and make the deploy wait for it. See the quality gate exercise.

Passed, but did the wrong thingScanners and policies#

Security scans ran, but the merge request shows no findings.

Cause

The merge request's Reports tab and the pipeline's Security tab need Ultimate. Also, GitLab's scanners run in the branch pipeline by default, not in the merge request pipeline.

Confirm

Check your tier, and which pipeline has the scanner jobs. The raw report is in the scanner job's artifacts.

Fix

Below Ultimate, read the report artifact. To scan in merge request pipelines, set AST_ENABLE_MR_PIPELINES: "true". See where the findings appear.

Passed, but did the wrong thingArtifacts, cache and data#

A variable that an earlier job wrote to a dotenv report is empty in a later job.

Cause

The later job doesn't receive that job's artifacts. Its needs or dependencies leave the job out, or it uses needs with artifacts: false. An empty dependencies: [] blocks them even when needs names the job.

Confirm

Read the later job's needs and dependencies in Full configuration, and check that the earlier job saved a dotenv report.

Fix

Add the earlier job to needs, and to dependencies too if the job has that list. See dotenv, and the container scan that gets no image.

Passed, but did the wrong thingArtifacts, cache and data#

A job finds files that shouldn't be there: stale ones, or another job's.

Cause

Two jobs share a cache key but cache different paths, so each overwrites the other's cache. Or a job caches and keeps the same path, and the cache is restored before the artifacts.

Confirm

Compare the jobs' cache keys and paths in Full configuration.

Fix

Give each set of paths its own key, and don't cache what you keep as an artifact. See cache.

Passed, but did the wrong thingGit in CI#

After a push of several commits, some changed services weren't built.

Cause

The script compares with HEAD~1, which is only the previous commit. Changes in the push's earlier commits are missed.

Confirm

Read the base the script uses in push pipelines.

Fix

For pushes, compare with CI_COMMIT_BEFORE_SHA, with a fallback for when it is all zeros. See the monorepo exercise.

Passed, but did the wrong thingDeployments and delivery#

A release reports a development version, such as 1.8.0-SNAPSHOT.

Cause

The build takes its version from the committed build file. Nothing passes the release tag into the build.

Confirm

Read the version in pom.xml or package.json, and the build command.

Fix

Pass CI_COMMIT_TAG into the build in tag pipelines, or commit the release version before tagging. See which version is it.

Passed, but did the wrong thingSecrets, tokens and identity#

AWS commands act as the wrong identity. For example, pushes are denied, although the job's role is allowed to push.

Cause

The login step didn't run, often because the job's own before_script replaced the template's. The AWS CLI then used the next credentials it found, such as the runner's own role.

Confirm

aws sts get-caller-identity prints the role really in use.

Fix

Restore the login step. See which credentials the AWS CLI uses.

Passed, but did the wrong thingSecrets, tokens and identity#

A job acts as an old user or account in AWS, although it has an ID token and a role.

Cause

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are still set in CI/CD settings, from before OIDC. Keys in environment variables come first, unless a login step overwrites them.

Confirm

Look for the two variables in the project's and groups' settings, and run aws sts get-caller-identity.

Fix

Remove the old keys, after checking that nothing else still needs them. See which credentials the AWS CLI uses.

Passed, but did the wrong thingArtifacts, cache and data#

A job passes, but whatever it built is nowhere to be found.

Cause

The job has no artifacts and no step that publishes its output, so the output is deleted with the job's environment. MegaCorp's publish-docs builds its site this way.

Confirm

The job has no artifacts: and no deploy command.

Fix

Keep the output as an artifact, or add a step that publishes it. See the publish-docs exercise.

Passed, but did the wrong thingYAML itself#

A version such as 1.10 turns into 1.1, and a rule comparing it never matches.

Cause

YAML reads an unquoted 1.10 as a number, and the number loses its trailing zero.

Confirm

The value is unquoted in the YAML. On the local GitLab, VERSION: 1.10 reached a rule as 1.1.

Fix

Quote it: VERSION: "1.10". See YAML in five minutes.

Passed, but did the wrong thingSecrets, tokens and identity#

A scanner has checked only the main branch for months, and nobody noticed.

Cause

Its token is protected, so merge request pipelines on feature branches get an empty token, and the scan can't sign in. The job is allowed to fail, so each failure is only a warning.

Confirm

The scan job shows warnings in merge request pipelines, and the token is protected in the settings.

Fix

Decide on a token for merge requests, and make the failures visible. See the quality gate that never ran.

Passed, but did the wrong thingRunners and execution#

The same job behaves differently depending on which runner takes it.

Cause

Runners differ in their executor and their configuration: extra environment variables, a pre_build_script, resource limits. No project can see any of it.

Confirm

The job log names the runner. Compare the runners' settings with the platform team.

Fix

Pin the job to the right runners with tags, or make the runners consistent. See what the runner's configuration decides.

Passed, but did the wrong thingScanners and policies#

The container scan runs in every merge request pipeline, yet it never scans the image that the pipeline built.

Cause

GitLab's container scanning template gives the job dependencies: []. An empty list blocks every artifact download, dotenv reports included, and adding needs doesn't lift it. So a value such as IMAGE_REF, from the build job's dotenv report, is empty in the scan, and so is the CS_IMAGE built from it.

Confirm

In Full configuration, the merged container_scanning job has both dependencies: [] and needs. On the local GitLab 19.3, a job with both keys printed an empty value.

Fix

Add dependencies: [image-build] to your version of the job, naming the job that writes the report. See changing a template's job.

Next42Cards: stuck or drifting5 min

Cards: stuck or drifting#

Part VIII · Decode and debug · Chapter 42· 5 min read

Use these cards when jobs wait for ever, run slowly, or break although nobody changed anything. Drift has no commit to blame, so look outside the repository: runners, tokens, included files, images and policies. The most common causes come first.

Stuck, slow, or broke without a commitRunners and execution#

A job stays pending, and its page says it is stuck because no runner matches all of its tags.

Cause

No online runner has every tag the job lists. A job's own tags replace the default tags completely.

Confirm

The page says:

This job is stuck because of one of the following problems. There are no active runners online, no runners for the protected branch, or no runners that match all of the job's tags:

The job's tags are listed after the message.

Fix

Fix the job's tags or the runner's. See how a job finds a runner.

Stuck, slow, or broke without a commitRunners and execution#

Every job in a project waits in pending.

Cause

The project's runners are offline: none has contacted GitLab for more than two hours.

Confirm

The job page says:

This job is stuck because the project doesn't have any runners online assigned to it.

Settings › CI/CD › Runners shows their status.

Fix

The runners' owners restart or replace them. See which runners a project can use.

Stuck, slow, or broke without a commitRunners and execution#

Jobs run on main, but stay pending on feature branches.

Cause

The only runners that fit are protected. Protected runners take jobs only from protected branches and tags.

Confirm

The runners' settings show Protected.

Fix

Provide an unprotected runner for other branches, or protect the branch. See how a job finds a runner.

Stuck, slow, or broke without a commitRunners and execution#

Jobs wait a long time in pending at busy times, then run normally.

Cause

The runners are already running as many jobs as they allow. Instance runners also share a fair-usage queue between projects.

Confirm

Many jobs are pending at once. The platform team can confirm the runners' concurrent and limit settings.

Fix

Add runner capacity, or run fewer jobs in parallel. See what the runner's configuration decides.

Stuck, slow, or broke without a commitTime bombs and drift#

A job that has worked for months suddenly fails with an authentication error, and nobody committed anything.

Cause

A stored access token expired, at midnight UTC on its expiry date. Unless someone chose a date, that is 365 days after it was created.

Confirm

The token's page lists it as expired.

Fix

Create a new token, update the variable, and track the next expiry date. See stored tokens.

Stuck, slow, or broke without a commitTime bombs and drift#

The pipeline changed overnight, but your project's history shows no change.

Cause

Your configuration includes a file from another project at a branch, or at a tag that someone moved. When that file changes, so does your pipeline.

Confirm

Check the include's ref in Full configuration, then the other project's recent commits.

Fix

Pin includes to tags, and treat tags as never moving. See refs in includes.

Stuck, slow, or broke without a commitTime bombs and drift#

A job's tools behave differently from one day to the next, with no change in the project.

Cause

The job's image tag, such as node:22, now points at a newer image. Anyone who can push to that repository can move a tag.

Confirm

Compare the image's digest in the registry with the digest it had before.

Fix

Name the image by digest, as name@sha256:…, or use tags that their owners never overwrite. See the job's image.

Stuck, slow, or broke without a commitTime bombs and drift#

A long job's AWS calls start failing part-way through.

Cause

The temporary keys last one hour, because MegaCorp asks for 3600 seconds. Chained roles are limited to one hour, whatever you ask for.

Confirm

The failures start more than an hour after the login step.

Fix

Stuck, slow, or broke without a commitDeployments and delivery#

A deploy waits and never starts.

Cause

The environment needs approvals, or the job is manual. An approval doesn't start the job.

Confirm

Operate › Environments shows the deployment waiting, and the job offers Run.

Fix

Approve it, then run it. See who may deploy, and approvals.

Stuck, slow, or broke without a commitDeployments and delivery#

A deploy job waits, although nothing else seems to be running in its pipeline.

Cause

Another job in the same resource_group is running, in another pipeline. Only one job per resource group runs at a time.

Confirm

The job is waiting for its resource. Look for the same deploy in other pipelines.

Fix

Let the other job finish, or find what is holding the resource. See one at a time.

Stuck, slow, or broke without a commitTime bombs and drift#

New jobs appear in every pipeline overnight, with names like secret-detection-1, or in .pipeline-policy stages.

Cause

The security team added or changed a security policy.

Confirm

Secure › Policies lists the policies that apply.

Fix

Nothing in your YAML can remove them. Talk to the security team. See three kinds of policy.

Stuck, slow, or broke without a commitArtifacts, cache and data#

Jobs are slow, because they download everything every time.

Cause

The cache is never found. The runners may have no shared cache storage, the key may keep changing, or protected and unprotected branches may have separate caches.

Confirm

The cache lines in the job log name the key it tried, and whether it was found.

Fix

Work through the cache flow. See cache.

Stuck, slow, or broke without a commitTime bombs and drift#

Pulls from Docker Hub fail at busy times, and work when the job is retried.

Cause

Docker Hub limits how many pulls it allows, counting each request for an image's manifest.

Confirm

The failing images all come from Docker Hub.

Fix

Pull through GitLab's dependency proxy, or keep copies in your own registry. See private registries.

Stuck, slow, or broke without a commitDeployments and delivery#

Deployments are blocked for a period, and the pipeline hasn't changed.

Cause

A deploy freeze is set for the project, and GitLab blocks deployments during it.

Confirm

The environment's deployments list shows the next freeze.

Fix

Wait for the freeze to end, or ask whoever set it. See one at a time, newest wins.

Stuck, slow, or broke without a commitDeployments and delivery#

Deploys appear several minutes after the pipeline goes green.

Cause

The GitOps controller syncs on a timer. GitLab's agent can trigger an immediate sync only for its own configuration project and for public projects.

Confirm

Compare the GitOps commit's time with the controller's last sync.

Fix

Shorten the controller's sync interval, or accept the delay. See GitLab's agent for Kubernetes.

Next43The final test2 min

The final test#

Part VIII · Decode and debug · Chapter 43· 2 min read

One file, ten mistakes. The file is payments-api's pipeline after a week of well-meant changes. Each mistake is one this book has explained, and each has a symptom card. Read the scenario, find all ten, and for each one say what will go wrong and which card describes it. Then open the answer key.

The scenario#

It's Friday afternoon. payments-api's team asks you to review their pipeline changes before they merge them. They tell you what they meant to do:

Integration tests must run on every merge request, against a PostgreSQL service. The database password is in the project's CI/CD settings, as POSTGRES_PASSWORD.
changed-files should list the files a merge request changes, for the reviewers.
smoke-test checks the image that the pipeline has just built.
The API contract version is 1.10.
Everything else should behave exactly as before.

MegaCorp's templates, runners and settings are the ones described in this book.

Spot the bugTen mistakes in one file#

final test · broken on purposepayments-api/.gitlab-ci.yml

# final-test/payments-api/.gitlab-ci.yml
# The final test: payments-api's pipeline file after a busy week of changes. It contains
# exactly ten mistakes, planted on purpose. Used by chapter 43. GitLab never runs it.

include:
  - project: megacorp/devops/ci-templates
    ref: main
    file: pipelines/java-service.yml
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api

variables:
  JAVA_VERSION: "21"
  MC_TEAM: payments
  API_VERSION: 1.10

maven-test:
  variables:
    MC_TEAM: $MC_TEAM
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

image-build:
  before_script:
    - echo "Building $CI_PROJECT_NAME for $MC_TEAM"

integration-test:
  stage: test
  tags: [docker]
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk21
  services:
    - name: postgres:16
      alias: db
  script:
    - mvn verify -Pintegration -Dapi.version=$API_VERSION
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

changed-files:
  stage: test
  script:
    - git diff --name-only origin/main...HEAD > changed.txt
  artifacts:
    paths: [changed.txt]
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

smoke-test:
  stage: deploy
  needs: [maven-build]
  script:
    - ./smoke.sh "$IMAGE_REF"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

QuestionFind the ten mistakes. For each, say what will go wrong, and which symptom card describes it.

Show the answer

#	The mistake	What goes wrong	Card
1	`ref: main` on the central include	the pipeline changes whenever ci-templates' `main` changes, with no commit here	a moving include
2	`API_VERSION: 1.10`, unquoted	YAML reads a number, so the tests get `1.1`	1.10 becomes 1.1
3	`MC_TEAM: $MC_TEAM` in `maven-test`	the job gets the literal text `$MC_TEAM`	a literal variable
4	`maven-test`'s own `rules`	they replace the template's list: the job leaves merge request pipelines, and now runs in schedules	your rules replaced the template's
5	`image-build`'s own `before_script`	the functions and the AWS login are gone, so the build fails at `mc_retry`	before_script replaced
6	`tags: [docker]` on `integration-test`	no MegaCorp runner has that tag, so the job waits for ever	stuck on tags
7	the database password only in settings	settings variables never reach services, so PostgreSQL doesn't start	a service without its variables
8	`integration-test` runs only on `main`	it never runs in merge request pipelines, as it was meant to	no merge request rule
9	`git diff … origin/main` in a merge request pipeline	that pipeline fetches only its own commit, so `origin/main` doesn't exist	shallow history
10	`smoke-test` needs `maven-build`, not `image-build`	it doesn't receive `IMAGE_REF`, so it tests nothing	an empty dotenv variable

Scoring: eight or more, and you can decode pipelines you have never seen. Five to seven: reread the cards you missed, then try again tomorrow. Fewer than five: work through chapters 35 and 39 to 42 once more. Every mistake here is in real pipelines somewhere, usually more than one at a time.

Next · Appendix44Keyword index3 min

Keyword index#

Appendix · Appendices · Chapter 44· 3 min read

Every .gitlab-ci.yml keyword in GitLab 19.3, what it does, and where this book explains it. Keywords written as parent:child are settings inside the parent keyword. A dash means the book doesn't cover that keyword.

Global keywords#

Keyword	What it does	Covered in
`default`	settings every job gets unless it sets its own: `image`, `before_script`, `tags` and others	chapter 7
`include`	copies other YAML files into this configuration	chapter 8
`include:local`	a file from the same repository and branch	chapter 8
`include:project`	a file from another project, at a `ref`	chapter 8
`include:remote`	a file from any URL	chapter 8
`include:template`	a template that ships with GitLab	chapter 8
`include:component`	a versioned CI/CD component	chapter 9
`include:inputs`	values for an included file's inputs	chapter 9
`include:rules`	includes the file only when the rules match	chapter 8
`include:integrity`, `include:cache`	check or cache a remote file	–
`stages`	the stages, in order	chapter 17
`workflow`	whether a pipeline is created, and its settings	chapter 15
`workflow:rules`	decide whether the pipeline exists at all	chapter 15
`workflow:name`	the pipeline's name	chapter 15
`workflow:rules:variables`	variables set when a workflow rule matches	chapter 15
`workflow:auto_cancel`, `:on_new_commit`, `:on_job_failure`	which jobs to cancel when a newer commit starts a pipeline, or when a job fails	chapter 15

Header keywords#

Keyword	What it does	Covered in
`spec`	the header of a file, before its `---` line	chapter 9
`spec:inputs`, `:default`, `:type`, `:options`, `:regex`, `:rules`, `:description`	typed parameters for a component, an included file or a pipeline, with defaults, allowed values and checks	chapter 9
`spec:include`, `spec:component`, `spec:description`	shared input definitions, component context, and a description	chapter 9

Job keywords#

Keyword	What it does	Covered in
`after_script`	commands that run last, in a separate shell	chapter 22
`allow_failure`	lets the pipeline continue if the job fails	chapter 17
`artifacts`	files kept after the job, for later jobs and people	chapter 24
`artifacts:paths`, `:exclude`, `:name`, `:untracked`	which files to keep	chapter 24
`artifacts:when`, `:expire_in`	when to keep them, and for how long	chapter 24
`artifacts:reports`	files GitLab reads, such as test results and security reports	chapter 24
`artifacts:access`, `:public`, `:expose_as`	who can download them, and links in merge requests	chapter 24
`before_script`	commands that run before `script`, in the same shell	chapter 22
`cache`	files reused between jobs and pipelines, to save time	chapter 24
`cache:key`, `:paths`, `:policy`, `:when`, `:fallback_keys`, `:unprotect`, `:untracked`	which cache, what goes in it, and when it is saved	chapter 24
`cache:key:files`, `:files_commits`, `:prefix`	a key that changes when the files' content changes, or when the files get a new commit, with an optional prefix	chapter 24
`coverage`	a regular expression that finds the coverage figure in the log	chapter 30
`dast_configuration`	DAST site and scanner profiles	–
`dependencies`	which earlier jobs' artifacts to download	chapter 24
`environment`	where the job deploys, which makes it a deployment job	chapter 33
`environment:name`, `:url`, `:on_stop`, `:action`, `:auto_stop_in`, `:deployment_tier`, `:kubernetes`	the environment's name, address, stopping and tier	chapter 33
`extends`	copies settings from other jobs, usually hidden ones	chapter 7
`hooks:pre_get_sources_script`	commands on the runner before the Git fetch	chapter 22
`identity`	identity federation with a cloud provider, in beta	–
`id_tokens`	signed ID tokens for other systems, such as AWS	chapter 21
`image`	the container image the job runs in	chapter 23
`image:name`, `:entrypoint`, `:pull_policy`, `:docker`, `:kubernetes`	the image's name, entrypoint, pull policy and executor options	chapter 23
`inputs`, `:type`, `:regex` and others	typed inputs for one job, which can be changed when the job is run by hand or retried	–
`inherit`	which `default:` settings and top-level variables the job takes	chapter 7
`inherit:default`, `inherit:variables`	take all, none, or a list	chapter 7
`interruptible`	lets a newer pipeline cancel this job	chapter 17
`needs`	starts the job as soon as named jobs finish, ignoring stages	chapter 17
`needs:optional`	a need that may be missing from the pipeline	chapter 17
`needs:artifacts`	whether to download the needed job's artifacts	chapter 24
`needs:project`, `needs:pipeline`, `needs:pipeline:job`	artifacts or status from other pipelines	chapter 10
`needs:parallel:matrix`	needs one variant of a matrix job	chapter 7
`pages`	a GitLab Pages publishing job	–
`parallel`	runs several copies of the job	chapter 7
`parallel:matrix`	one copy for each combination of variable values	chapter 7
`release`, `:tag_name`, `:tag_message`, `:name`, `:description`, `:ref`, `:milestones`, `:released_at`, `:assets:links`	creates a release, and describes it	–
`resource_group`	only one job in the group runs at a time	chapter 17
`retry`	retries the job on failure	chapter 17
`retry:max`, `:when`, `:exit_codes`	how often, and for which failures	chapter 17
`rules`	decides whether the job is in the pipeline, and how it runs	chapter 16
`rules:if`	a condition on variables	chapter 16
`rules:changes`, `rules:exists`	conditions on changed or existing files	chapter 16
`rules:changes:paths`, `:compare_to`, `:regexp`, and `rules:exists:regexp`	the files to check, the ref to compare against, or a Ruby regular expression in place of glob patterns	chapter 16
`rules:when`, `:allow_failure`, `:needs`, `:variables`, `:interruptible`	what a matching rule changes	chapter 16
`run`	a sequence of GitLab Functions steps	chapter 12
`script`	the commands the job runs	chapter 22
`secrets`	secrets fetched from a secrets manager	chapter 20
`secrets:vault`, `:aws_secrets_manager`, `:gcp_secret_manager`, `:azure_key_vault`, `:gitlab_secrets_manager`	the provider and the secret	chapter 20
`secrets:file`, `secrets:token`	file or value, and which ID token to use	chapter 20
`services`	extra containers beside the job, such as a database	chapter 23
`services:name`, `:alias`, `:entrypoint`, `:command`, `:variables`, `:pull_policy`	the service's image, hostname, start command and settings	chapter 23
`stage`	the stage the job belongs to	chapter 2
`tags`	which runners may take the job	chapter 22
`timeout`	the job's own time limit	chapter 22
`trigger`	starts a downstream pipeline	chapter 10
`trigger:include`	a child pipeline from a file in this project	chapter 10
`trigger:project`	a pipeline in another project	chapter 10
`trigger:strategy`	makes the trigger job wait for, and mirror, the downstream pipeline	chapter 10
`trigger:forward`, `trigger:inputs`	what the downstream pipeline receives	chapter 10
`when`	when the job runs: `on_success`, `manual`, `delayed`, `always`, `never`	chapter 17
`start_in`	the delay for `when: delayed`	chapter 17
`manual_confirmation`	a message to confirm before running a manual job	chapter 17
`variables`	variables for every job, or for one job	chapter 19
`variables:value`, `:description`, `:options`	defaults and choices shown when someone runs a pipeline by hand	chapter 19
`variables:expand`	whether `$` references in the value are expanded	chapter 19

Deprecated keywords#

Keyword	Use instead	Covered in
`only`, `except`	`rules`	chapter 16
`image`, `services`, `cache`, `before_script`, `after_script` at the top level	the same keywords inside `default:`	chapter 7
`publish`, and a Pages job named `pages`	the `pages` keyword	–

Next45Predefined variables1 min

Predefined variables#

Appendix · Appendices · Chapter 45· 1 min read

GitLab sets these variables in every pipeline, and each one exists only from a certain moment. This appendix lists the variables you will meet most, when each exists, and which kinds of pipeline set the ones that differ. GitLab's documentation has the full list.

When a variable exists#

Phase	Usable in	Examples
Pre-pipeline	everything, including `include:rules`	`CI_COMMIT_`, `CI_PIPELINE_SOURCE`, `CI_DEFAULT_BRANCH`, `CI_OPEN_MERGE_REQUESTS`, `CI_MERGE_REQUEST_`, `CI_PROJECT_ID`, `CI_SERVER_HOST`
Pipeline	job `rules` and scripts, but not `include:rules`	`CI_PIPELINE_IID`, `CI_JOB_NAME`, `CI_JOB_STAGE`, `CI_NODE_INDEX`, `CI_ENVIRONMENT_NAME`, `GITLAB_USER_LOGIN`
Job-only	scripts only: not `workflow`, `include`, `rules` or trigger jobs	`CI_PIPELINE_ID`, `CI_PIPELINE_URL`, `CI_JOB_ID`, `CI_JOB_TOKEN`, `CI_PROJECT_DIR`, `CI_REGISTRY_PASSWORD`

Which pipelines set them#

Variable	Branch	Tag	Merge request	Scheduled
`CI_COMMIT_BRANCH`	yes	–	–	yes
`CI_COMMIT_TAG`	–	yes	–	only if the schedule runs on a tag
`CI_PIPELINE_SOURCE` is `push`	yes	yes	–	–
`CI_PIPELINE_SOURCE` is `merge_request_event`	–	–	yes	–
`CI_PIPELINE_SOURCE` is `schedule`	–	–	–	yes
`CI_MERGE_REQUEST_*`	–	–	yes	–
`CI_OPEN_MERGE_REQUESTS`	if the branch has an open merge request	–	yes	–

CI_COMMIT_BEFORE_SHA is all zeros in merge request pipelines, scheduled pipelines, manual runs, and the first pipeline of a branch or tag.

The variables you meet most#

Variable	Holds	Phase
`CI_PIPELINE_SOURCE`	what started the pipeline: `push`, `merge_request_event`, `schedule`, `web`, `api`, `trigger`, `pipeline`, `parent_pipeline`, and others	pre-pipeline
`CI_COMMIT_BRANCH`	the branch, in branch pipelines only	pre-pipeline
`CI_COMMIT_TAG`	the tag, in tag pipelines	pre-pipeline
`CI_COMMIT_REF_NAME`	the branch or tag being built	pre-pipeline
`CI_COMMIT_REF_SLUG`	the ref name in lower case, at most 63 bytes, with anything but `0-9` and `a-z` replaced by `-`, for URLs and host names	pre-pipeline
`CI_COMMIT_REF_PROTECTED`	`true` for a protected branch or tag	pre-pipeline
`CI_COMMIT_SHA`, `CI_COMMIT_SHORT_SHA`	the commit, and its first eight characters	pre-pipeline
`CI_COMMIT_BEFORE_SHA`	the branch's latest commit before this push	pre-pipeline
`CI_DEFAULT_BRANCH`	the project's default branch	pre-pipeline
`CI_OPEN_MERGE_REQUESTS`	up to four open merge requests from this branch	pre-pipeline
`CI_MERGE_REQUEST_IID`, `…_SOURCE_BRANCH_NAME`, `…_TARGET_BRANCH_NAME`, `…_DIFF_BASE_SHA`, `…_EVENT_TYPE`	the merge request, in merge request pipelines	pre-pipeline
`CI_PROJECT_ID`, `CI_PROJECT_PATH`, `CI_PROJECT_NAME`	the project	pre-pipeline
`CI_SERVER_HOST`, `CI_SERVER_FQDN`, `CI_API_V4_URL`	the GitLab instance, and its API	pre-pipeline
`CI_REGISTRY`, `CI_REGISTRY_IMAGE`	the container registry, and the project's image path in it	pre-pipeline
`CI_PIPELINE_IID`	the pipeline's number within the project	pipeline
`CI_JOB_NAME`, `CI_ENVIRONMENT_NAME`	the job's name, and its environment	pipeline
`GITLAB_USER_LOGIN`	who started the pipeline, or the manual job	pipeline
`CI_PIPELINE_ID`, `CI_JOB_ID`	instance-wide IDs of the pipeline and the job	job-only
`CI_PROJECT_DIR`	where the repository is cloned, and where the job runs	job-only
`CI_JOB_TOKEN`	the job's token, valid only while the job runs	job-only
`CI_REGISTRY_USER`, `CI_REGISTRY_PASSWORD`	credentials for the project's registry	job-only

Where variables can't be used#

rules:if can't use CI_ENVIRONMENT_SLUG, or variables GitLab stores with the job: CI_PIPELINE_ID, CI_PIPELINE_URL, CI_JOB_ID, CI_JOB_TOKEN, CI_JOB_URL, CI_REGISTRY_USER, CI_REGISTRY_PASSWORD and others like them.
include can use settings variables, CI_PROJECT_*, CI_PIPELINE_SOURCE, CI_COMMIT_REF_NAME, and variables given when the pipeline starts. It can't use variables defined in the YAML.
No rule can use a value from a dotenv report (chapter 14).

Next46Rules syntax2 min

Rules syntax#

Appendix · Appendices · Chapter 46· 2 min read

A reference for rules: and its conditions in GitLab 19.3: how rules are evaluated, how to write if: expressions, and what changes: and exists: really check. Every line comes from GitLab's documentation, or was captured on a local 19.3 instance.

How rules are evaluated#

Rules are evaluated when the pipeline is created, before any job runs.
They are checked in order, and the first rule that matches decides. If none matches, the job isn't added.
Every rule needs if, changes, exists or when, and every clause in one rule must hold.
A matching rule without when uses the job's when, which defaults to on_success. A when in the rule overrides the job's.
A rule can also set allow_failure, needs, which replaces the job's list, variables and interruptible.
A rule with when but no if brings the warning Job may allow multiple pipelines to run for a single action, in CI Lint and on manual runs.
GitLab's documentation says jobs with no rules "default to except: merge_requests": they don't join merge request pipelines.

if expressions#

Write	Means	Notes
`$VAR == "value"`	equal	the variable on the left; strings quoted, variables not
`$VAR != "value"`	not equal
`$VAR`	set, and not empty
`$VAR == null`	not set
`$VAR == ""`	set, but empty
`$VAR =~ /pattern/`	matches the regular expression	RE2 syntax, case-sensitive, `/i` to ignore case; a pattern without anchors matches anywhere in the value
`$VAR !~ /pattern/`	doesn't match
`a && b`, `a \|\| b`	and, or	`&&` binds before `\|\|`; brackets group
`!$VAR`	empty, or not set	since 18.11; so `!"false"` is false

Traps, each confirmed in the documentation or on a local 19.3 instance:

${VAR} and "$VAR" don't work in if. A quoted variable breaks the YAML itself: (): did not find expected key while parsing a block mapping at line 5 column 7.
An unquoted !( … ) around a comparison is rejected with jobs:job:rules:rule if invalid expression syntax. Use !=.
A one-character regular expression, /./, is invalid.
A variable on the right of =~ is used as the regular expression, so its value must include the slashes. Variables inside a /…/ pattern aren't expanded.
A right side that isn't a /…/ pattern turns into a substring test, the other way round. $CI_COMMIT_BRANCH =~ "xmainx" was true for main.
A variable whose value mentions another variable isn't expanded in if.
An unquoted YAML number changes: 1.10 arrives as 1.1.

changes and exists#

Clause	Checks	Traps
`changes`	in merge request pipelines, files changed against the target branch; in branch pipelines, against the previous commit	always true for new branches, tags, and pipelines without a push: schedules, manual runs; `compare_to` sets another base; at most 50 patterns
`exists`	files in the repository, relative to the project directory	directories need a trailing slash (since 18.2); artifacts are invisible to it; beyond 50,000 files it always matches

In include and workflow#

include:rules accepts only if, exists and changes, and can use only the variables that exist before the pipeline is created (predefined variables). Its exists searches the project that holds the include.
workflow:rules decide whether the pipeline exists at all (chapter 15).

Next47Error messages2 min

Error messages#

Appendix · Appendices · Chapter 47· 2 min read

Search this page for the exact text you see. Each message is quoted as GitLab, the runner or AWS prints it: captured on a local GitLab 19.3 instance, or copied from the documentation. Each links to the symptom card or section that explains it. Messages that contain job or stage names were captured with the names shown; yours will differ.

When the pipeline is created#

Message	Means	Go to
`The pipeline did not run. Review the workflow:rules configuration for the pipeline.`	no `workflow:rules` matched	workflow refused
`The resulting pipeline would have been empty. Review the rules configuration.`	the workflow allowed it, but no job's rules matched	empty pipeline
`'unit-tests' job needs 'compile' job, but 'compile' does not exist in the pipeline.`	a needed job was left out by its rules; the message goes on to suggest `needs:optional`	needs a missing job
`orphan job: chosen stage nowhere does not exist; available stages are .pre, build, .post`	a job names a stage missing from `stages:`	stage not listed
`jobs:job:rules:rule if invalid expression syntax`	an expression GitLab can't parse, such as `!( … )`	rule syntax
`(): did not find expected key while parsing a block mapping at line 5 column 7`	broken YAML, for example a quoted variable in `if:`	rule syntax
`Insufficient permissions to set pipeline variables`	your role may not set variables when starting a pipeline	values given when the pipeline starts
`Failed - (downstream pipeline can not be created, Insufficient permissions to set pipeline variables)`	the downstream project refuses the trigger job's variables	pipeline variables refused
`downstream_pipeline_creation_failed`	a trigger job's failure reason: the downstream pipeline couldn't be created	child pipeline in merge requests

CI Lint also warns. Job may allow multiple pipelines to run for a single action means a rule has when but no if (rules syntax). On MegaCorp's templates, it also prints this deprecation warning, which suggests its own fix:

retry uses deprecated `when` value(s): stuck_or_timeout_failure. These match the more
specific failure reasons that replaced them; migrate to those reasons. See
https://docs.gitlab.com/ci/yaml/#retrywhen

On the job page, or in the job log#

Message	Means	Go to
`This job is stuck because of one of the following problems. There are no active runners online, no runners for the protected branch, or no runners that match all of the job's tags:`	no online runner fits the job's tags, or only protected runners fit	stuck on tags
`This job is stuck because the project doesn't have any runners online assigned to it.`	the project's runners are offline	runners offline
`This job is stuck because you don't have any active runners that can run this job.`	no runner available to the project can take the job	how a job finds a runner
`This job could not start because it could not retrieve the needed artifacts.`	needed artifacts expired, or aren't reachable	artifacts unavailable
`No files to upload`	`artifacts: paths` matched nothing	no files to upload
`*** WARNING: Service XYZ probably didn't start properly`	a service container didn't open its port in time	a service without its variables
`The deployment job is older than the latest deployment, and therefore failed.`	Prevent outdated deployment jobs stopped an old deploy	outdated deploy
`This deployment job does not run automatically and must be started manually, but it's older than the latest deployment, and therefore can't run.`	the same, for a manual deploy	outdated deploy
`unresolved reference`	the commit isn't in the shallow clone	shallow history
`remote: The project you were looking for could not be found or you don't have permission to view it.`	a clone with the job token was refused	job token 404
`fatal: run_command returned non-zero status`	submodules with `GIT_STRATEGY: fetch`; GitLab suggests `clone`	submodules
`fatal: unable to access 'https://gitlab.example.com/…/….git/': Could not resolve proxy: proxy.example.com`	a proxy variable from settings, or from the runner's configuration, names a proxy the runner can't reach, and Git obeys it; a job's YAML can't unset a settings variable	variables in settings
`ERROR: Job failed (system failure): resolving secrets: operation error Secrets Manager: GetSecretValue, …, no EC2 IMDS role found, …`	the runner manager has no AWS role for `secrets:`	secrets from AWS

From AWS#

Message	Means	Go to
`An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity`	the role's trust policy doesn't accept this token	AssumeRole denied
`An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn't retrieve verification key from your identity provider`	AWS can't reach GitLab's signing keys	invalid identity token
`ImageTagAlreadyExistsException`	a push to an immutable tag that already exists	tag already exists

Next48The annotated specimen10 min

The annotated specimen#

Appendix · Appendices · Chapter 48· 10 min read

Every file of MegaCorp's specimen, in full. The chapters quote parts of these files; here you can read each one from top to bottom. The files are grouped by the project that owns them. Each group starts with what is worth noticing, linked to the pages that explain it.

A line such as # catalog: include-project marks one of the 37 reuse mechanisms: the construct just below it is an example. Files named SETTINGS.yml, and those under _settings, _runners and _aws, describe configuration kept in GitLab's settings, on the runners, or in AWS. GitLab doesn't read them.

Where each mechanism appears#

Each marker links to its mechanism card. To see a mechanism at work, find it here, then open the file.

Chapter	Markers, and the file each is in
6	`yaml-anchors` and `yaml-merge-key` in billing-batch; `hidden-jobs` in base.yml
7	`default-section` and `global-variables` in base.yml; `extends`, `reference-tag` and `parallel-matrix` in java-maven.yml
8	`include-local` in mono; `include-project` in payments-api; `include-remote` in billing-batch; `include-template` in security.yml
9	`include-component` in payments-api; `spec-inputs` in sonar-scan.yml; `variables-as-parameters` in java-maven.yml; `toggle-variables` in security.yml; `pipeline-inputs` in the deployer
10	`parent-child` and `dynamic-child` in mono; `multi-project` in deploy.yml; `trigger-api` in billing-batch
11	`custom-config-path` in web-portal's settings; `instance-group-variables` in the instance and group settings; `auto-devops` and `instance-template-repo` in the instance settings; `project-templates` and `compliance-pipeline` in the group settings; `scan-execution-policy`, `pipeline-execution-policy` and `mr-approval-policy` in policy.yml
12	`toolbox-image` in base.yml and the Dockerfile; `shell-libraries` in snippets.yml; `internal-cli` in `mc`; `fetched-scripts` in billing-batch; `build-tool-reuse` in pom.xml; `runner-config` in config.toml; `ci-steps` in mono's docs.yml

payments-api: one service#

Start here: this is everything the service team wrote. Worth noticing:

Its jobs come from the golden pipeline, pinned at v4.2.0, and from a component pinned at 2.1.0 (chapter 8).
maven-test: has no extends. It merges into the golden pipeline's job of the same name, and adds a variable (chapter 8).
That MAVEN_CLI_OPTS loses to the group variable of the same name, because settings beat YAML (chapter 19).
MC_TEAM: payments never reaches a job. .megacorp-base sets MC_TEAM as a job variable, which beats top-level ones (chapter 7).
publish-docs runs every night, because its only rule is the default branch. Its script only builds the site: nothing publishes it (chapter 35).
Only main and v* tags are protected, so only their pipelines receive PAYMENTS_DB_PASSWORD (chapter 20).
The POM's version stays 1.8.0-SNAPSHOT: nothing sets it from the Git tag (chapter 30).

megacorp/payments/payments-api.gitlab-ci.yml

# .gitlab-ci.yml in megacorp/payments/payments-api
# A Java service's entire pipeline file: include the golden pipeline and a
# component, set a few variables, adjust two jobs.
# Used by chapters 04, 07, 08, 13 and 18.

# catalog: include-project
include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file: pipelines/java-service.yml
  # catalog: include-component
  - component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@2.1.0
    inputs:
      stage: test
      project_key: payments-api

variables:
  JAVA_VERSION: "21"
  MC_TEAM: payments

maven-test:
  variables:
    MAVEN_CLI_OPTS: "--batch-mode -Dsurefire.rerunFailingTestsCount=2"

publish-docs:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
  inherit:
    default: [tags, retry]
    variables: false
  script:
    - mkdocs build --strict
  rules:
    - !reference [.rules, default-branch]

megacorp/payments/payments-apiSETTINGS.yml

# SETTINGS.yml for megacorp/payments/payments-api
# This file DESCRIBES the project's settings in the GitLab UI. GitLab does not read it.
# Used by chapters 19 and 20.

protected_branches: [main]
protected_tags: ["v*"]

project_variables:
  - key: MC_NEXUS_URL
    value: https://nexus.example.com/repository/maven-public/
  - key: PAYMENTS_DB_PASSWORD
    value: "(set in the UI)"
    masked: true
    protected: true

megacorp/payments/payments-apipom.xml

<?xml version="1.0" encoding="UTF-8"?>
<!-- pom.xml in megacorp/payments/payments-api (excerpt). The service inherits its
     build, including the "ci" profile the pipeline calls, from MegaCorp's parent POM.
     Used by chapters 12 and 30. -->
<project xmlns="http://maven.apache.org/POM/4.0.0">
  <modelVersion>4.0.0</modelVersion>
  <!-- catalog: build-tool-reuse -->
  <parent>
    <groupId>com.example.megacorp</groupId>
    <artifactId>megacorp-parent</artifactId>
    <version>12.3.0</version>
  </parent>
  <artifactId>payments-api</artifactId>
  <version>1.8.0-SNAPSHOT</version>
</project>

devops/ci-templates: the central templates#

The central team's templates, at the v4.2.0 tag that payments-api pins. Worth noticing:

Only the two files in pipelines/ create jobs. The files in templates/ hold defaults, hidden jobs and snippets, except security.yml, which pulls in GitLab's scanner jobs (chapter 27).
.megacorp-base's rules decide when almost every job runs: merge requests, the default branch and release tags, never schedules (chapter 18).
.maven-test brings its own rules, which replace that list, so tag pipelines have no tests (chapter 7).
.maven-base and .image-build set their own before_script, which replaces the default one, so each loads the functions again (chapter 7).
.image-build runs in tag pipelines too, so a release builds a new image instead of promoting the one staging tested (chapter 32). With today's trust policy, it can't even push it (chapter 31).
.deploy hands IMAGE_REF to the deployer. The value comes from image-build's dotenv report (chapter 24).
security.yml's container_scanning adds needs, but keeps the template's dependencies: []. So IMAGE_REF, and the CS_IMAGE built from it, are empty in the scan (the symptom card).
legacy/notify.yml still uses only, and an older toolbox image. Every project that includes it by URL gets each change at once (include: remote).

megacorp/devops/ci-templatespipelines/java-service.yml

# pipelines/java-service.yml in megacorp/devops/ci-templates
# The "golden pipeline" for Java services. One include gives a project a build,
# tests, scanners, an image in ECR and deployments.
# Used by chapters 04, 05, 08, 13 and 18.

include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file:
      - templates/base.yml
      - templates/snippets.yml
      - templates/rules.yml
      - templates/workflow.yml
      - templates/java-maven.yml
      - templates/container.yml
      - templates/security.yml
      - templates/deploy.yml

stages: [build, test, package, scan, deploy]

maven-build:
  extends: .maven-build

maven-test:
  extends: .maven-test

image-build:
  extends: .image-build
  needs: [maven-build]

deploy-staging:
  extends: .deploy
  needs: [image-build]
  variables:
    ENVIRONMENT: staging
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, default-branch]

deploy-prod:
  extends: .deploy
  needs: [image-build]
  variables:
    ENVIRONMENT: production
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
      when: manual

megacorp/devops/ci-templatespipelines/web.yml

# pipelines/web.yml in megacorp/devops/ci-templates
# The whole pipeline for static web front ends. Projects never include this
# file: their CI/CD settings point straight at it, so their repositories carry
# no .gitlab-ci.yml at all. Used by chapters 11, 13 and 30.

include:
  - project: megacorp/devops/ci-templates
    ref: v4.2.0
    file:
      - templates/base.yml
      - templates/snippets.yml
      - templates/rules.yml
      - templates/workflow.yml
      - templates/node.yml

stages: [build, test, deploy]

node-build:
  extends: .node-build

node-test:
  extends: .node-test

deploy-site:
  extends: .publish-site
  needs: [node-build]
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, default-branch]

megacorp/devops/ci-templatestemplates/base.yml

# templates/base.yml in megacorp/devops/ci-templates
# The foundation: defaults for every job, organisation-wide variables, and the
# hidden base job that almost every template extends.
# Used by chapters 04, 07, 16, 19 and 22.

# catalog: default-section, toolbox-image
default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  tags: [megacorp-shared]
  interruptible: true
  retry:
    max: 2
    when: [runner_system_failure, stuck_or_timeout_failure]
  before_script:
    - !reference [.snippets, functions]
    - mc_log "job $CI_JOB_NAME on $CI_COMMIT_REF_NAME"

# catalog: global-variables
variables:
  GIT_DEPTH: "20"
  AWS_REGION: eu-west-2
  MC_TEMPLATES_VERSION: "4.2.0"

# catalog: hidden-jobs
.megacorp-base:
  variables:
    MC_TEAM: unknown
  artifacts:
    expire_in: 7 days
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]
    - !reference [.rules, release-tag]

megacorp/devops/ci-templatestemplates/rules.yml

# templates/rules.yml in megacorp/devops/ci-templates
# A rules library: named lists of rules that jobs assemble with !reference.
# Used by chapters 07, 16 and 37.

.rules:
  mr:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  default-branch:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  release-tag:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
  never-on-schedule:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never

megacorp/devops/ci-templatestemplates/workflow.yml

# templates/workflow.yml in megacorp/devops/ci-templates
# Which pipelines exist at all: merge request pipelines, the default branch and
# tags, with no second branch pipeline while a merge request is open.
# Used by chapters 14 and 15.

workflow:
  name: "$CI_PIPELINE_SOURCE: $CI_COMMIT_REF_NAME"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH
    - if: $CI_COMMIT_TAG

megacorp/devops/ci-templatestemplates/snippets.yml

# templates/snippets.yml in megacorp/devops/ci-templates
# Reusable script fragments. Jobs pull them in with !reference; the functions
# they call live in the ci-tools image, in /opt/megacorp/lib/ci-lib.sh.
# Used by chapters 07, 12, 21 and 31.

# catalog: shell-libraries
.snippets:
  functions:
    - source /opt/megacorp/lib/ci-lib.sh
  aws_login:
    - mc_aws_login "$AWS_ROLE_ARN"
  maven_settings:
    - mkdir -p .m2
    - mc_maven_settings > .m2/settings.xml

megacorp/devops/ci-templatestemplates/java-maven.yml

# templates/java-maven.yml in megacorp/devops/ci-templates
# Build and test for Maven services. Everything here is hidden: a pipeline entry
# file turns these templates into real jobs.
# Used by chapters 07, 19, 24 and 30.

# catalog: extends, variables-as-parameters
.maven-base:
  extends: .megacorp-base
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JAVA_VERSION}
  variables:
    JAVA_VERSION: "21"
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  cache:
    key:
      files: [pom.xml]
    paths: [.m2/repository]
  before_script:
    # a job's own before_script replaces default:before_script, so re-add the functions
    - !reference [.snippets, functions]
    - !reference [.snippets, maven_settings]

.maven-build:
  extends: .maven-base
  stage: build
  script:
    - mvn $MAVEN_CLI_OPTS -DskipTests package
  artifacts:
    paths: [target/*.jar]

# catalog: reference-tag, parallel-matrix
.maven-test:
  extends: .maven-base
  stage: test
  image: registry.example.com/megacorp/devops/ci-tools/maven:3.9-jdk${JDK}
  parallel:
    matrix:
      - JDK: ["17", "21"]
  script:
    - mvn $MAVEN_CLI_OPTS verify
  artifacts:
    when: always
    reports:
      junit: target/surefire-reports/TEST-*.xml
  rules:
    - !reference [.rules, never-on-schedule]
    - !reference [.rules, mr]
    - !reference [.rules, default-branch]

megacorp/devops/ci-templatestemplates/node.yml

# templates/node.yml in megacorp/devops/ci-templates
# Install, test and build Node front ends, then publish the static files to S3
# behind CloudFront. Used by chapters 24, 30 and 31.

.node-base:
  extends: .megacorp-base
  image: registry.example.com/megacorp/devops/ci-tools/node:22
  cache:
    key:
      files: [package-lock.json]
    paths: [.npm/]
    policy: pull
  before_script:
    - npm ci --cache .npm --prefer-offline

.node-build:
  extends: .node-base
  stage: build
  cache:
    policy: pull-push
  script:
    - npm run build
  artifacts:
    paths: [dist/]

.node-test:
  extends: .node-base
  stage: test
  script:
    - npm test -- --ci
  artifacts:
    when: always
    reports:
      junit: junit.xml

.publish-site:
  extends: .megacorp-base
  stage: deploy
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-web-publish
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - aws s3 sync dist/ "s3://$SITE_BUCKET/" --delete
    - aws cloudfront create-invalidation --distribution-id "$CF_DISTRIBUTION_ID" --paths '/*'
  environment:
    name: production
    url: https://www.example.com

megacorp/devops/ci-templatestemplates/container.yml

# templates/container.yml in megacorp/devops/ci-templates
# Build the service image without Docker-in-Docker and push it to Amazon ECR,
# tagged by commit, using the job's OIDC identity. Used by chapters 21, 23, 31 and 32.

.image-build:
  extends: .megacorp-base
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-ecr-push
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
    IMAGE: $ECR_REGISTRY/$CI_PROJECT_NAME
  before_script:
    - !reference [.snippets, functions]
    - !reference [.snippets, aws_login]
  script:
    - mc_retry aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$IMAGE:$CI_COMMIT_SHA" .
    - buildah push "$IMAGE:$CI_COMMIT_SHA"
    - echo "IMAGE_REF=$IMAGE:$CI_COMMIT_SHA" >> build.env
  artifacts:
    reports:
      dotenv: build.env

megacorp/devops/ci-templatestemplates/security.yml

# templates/security.yml in megacorp/devops/ci-templates
# GitLab's own scanners, pulled in as GitLab-shipped templates, plus MegaCorp's
# adjustments to the container scan. Used by chapters 26, 27 and 29.

# catalog: include-template
include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml
  - template: Security/Container-Scanning.gitlab-ci.yml

# catalog: toggle-variables
variables:
  CS_IMAGE: $IMAGE_REF

container_scanning:
  stage: scan
  needs: [image-build]
  rules:
    - if: $CONTAINER_SCANNING_DISABLED == "true"
      when: never
    - !reference [.rules, never-on-schedule]
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

megacorp/devops/ci-templatestemplates/deploy.yml

# templates/deploy.yml in megacorp/devops/ci-templates
# Hand the built image to the platform team's deployer project, which owns every
# environment. Used by chapters 10, 33 and 34.

# catalog: multi-project
.deploy:
  stage: deploy
  trigger:
    project: megacorp/platform/deployer
    branch: main
    strategy: depend
  variables:
    APP: $CI_PROJECT_NAME
    IMAGE_REF: $IMAGE_REF

megacorp/devops/ci-templateslegacy/notify.yml

# legacy/notify.yml in megacorp/devops/ci-templates
# An old notification job that billing-batch still pulls in by URL with
# include: remote, straight from this repository's raw file view.
# Used by chapters 08 and 12.

notify-chat:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools:2.9
  script:
    - curl --fail -X POST -d "text=billing-batch $CI_COMMIT_REF_NAME finished" "$CHAT_WEBHOOK_URL"
  only:
    - tags

devops/ci-tools: the toolbox image#

None of this is YAML, yet every job runs in this image unless it picks another one. Worth noticing:

The default before_script sources ci-lib.sh, so every job can call its functions (chapter 12).
mc_aws_login swaps the job's ID token for AWS keys that last one hour (chapter 31).
mc_retry runs a command up to three times, waiting longer each time.
mc deploy changes one line in the GitOps repository, using GITOPS_TOKEN (chapter 34).
mc version prints the Git tag, or 0.0.0- and the short commit when there is no tag (chapter 30).

megacorp/devops/ci-toolsDockerfile

# Dockerfile in megacorp/devops/ci-tools: MegaCorp's toolbox image.
# Every job runs in it unless it picks another image, and the shell functions
# and the "mc" tool that pipelines call live inside it.
# Used by chapters 12, 21 and 23.

FROM public.ecr.aws/amazonlinux/amazonlinux:2023
RUN dnf install -y --allowerasing git jq make tar gzip unzip curl-minimal && dnf clean all
RUN curl -sSLo /tmp/awscli.zip https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip \
 && unzip -q /tmp/awscli.zip -d /tmp && /tmp/aws/install && rm -rf /tmp/aws /tmp/awscli.zip
RUN curl -sSLo /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.44.3/yq_linux_amd64 \
 && chmod +x /usr/local/bin/yq

# catalog: toolbox-image
COPY lib/ci-lib.sh /opt/megacorp/lib/ci-lib.sh
COPY bin/mc /usr/local/bin/mc
RUN chmod +x /usr/local/bin/mc

megacorp/devops/ci-toolslib/ci-lib.sh

#!/usr/bin/env bash
# lib/ci-lib.sh in megacorp/devops/ci-tools: MegaCorp's shell function library.
# It is baked into the ci-tools image as /opt/megacorp/lib/ci-lib.sh, and jobs
# source it through the .snippets functions snippet.
# Used by chapters 07, 12, 21 and 31.

mc_log() { printf '[megacorp] %s\n' "$*"; }

mc_retry() {
  local n=0
  until "$@"; do
    n=$((n + 1))
    [ "$n" -ge 3 ] && return 1
    mc_log "retry $n: $*"
    sleep $((n * 5))
  done
}

# Exchange the job's OIDC ID token for temporary AWS credentials.
mc_aws_login() {
  local role="${1:?role ARN required}"
  local creds
  creds=$(aws sts assume-role-with-web-identity \
    --role-arn "$role" \
    --role-session-name "gitlab-${CI_PROJECT_ID}-${CI_JOB_ID}" \
    --web-identity-token "$MC_ID_TOKEN" \
    --duration-seconds 3600 \
    --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
    --output text) || { mc_log "AssumeRoleWithWebIdentity failed for $role"; return 1; }
  read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< "$creds"
  export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
  mc_log "assumed $role"
}

mc_maven_settings() {
  cat <<EOF
<settings><mirrors><mirror><id>nexus</id><mirrorOf>*</mirrorOf><url>${MC_NEXUS_URL:-https://nexus.example.com/repository/maven-public/}</url></mirror></mirrors></settings>
EOF
}

megacorp/devops/ci-toolsbin/mc

#!/usr/bin/env bash
# bin/mc in megacorp/devops/ci-tools: MegaCorp's internal CI command-line tool,
# shipped in the ci-tools image as /usr/local/bin/mc. The deployer pipeline calls
# "mc deploy"; release jobs call "mc version". Used by chapters 12, 30 and 34.
# catalog: internal-cli
set -euo pipefail
source /opt/megacorp/lib/ci-lib.sh

cmd="${1:-help}"
shift || true
case "$cmd" in
  deploy)
    env="${1:?usage: mc deploy <environment> <image-ref>}"
    image="${2:?usage: mc deploy <environment> <image-ref>}"
    mc_log "promoting $image to $env through the GitOps repository"
    git clone --depth 1 \
      "https://gitops-bot:${GITOPS_TOKEN}@gitlab.example.com/megacorp/platform/gitops-config.git" gitops
    yq -i ".apps.\"${APP}\".image = \"${image}\"" "gitops/envs/${env}/values.yaml"
    git -C gitops commit -am "deploy ${APP} ${image} to ${env}"
    git -C gitops push origin HEAD:main
    ;;
  version)
    echo "${CI_COMMIT_TAG:-0.0.0-${CI_COMMIT_SHORT_SHA}}"
    ;;
  *)
    echo "usage: mc deploy <environment> <image-ref> | mc version" >&2
    exit 2
    ;;
esac

devops/components: CI/CD components#

Three components. A project includes each at a version it chooses. Worth noticing:

The spec: header ends at ---. Each $[[ inputs.… ]] is replaced when the pipeline is created, before any variable is expanded (chapter 9).
sonar-scan has allow_failure: true and a SKIP_SONAR toggle, so even when the scan fails, the pipeline carries on (chapter 28).
ecr-push puts the repository input in its job's name, so each include with a different repository makes its own job (chapter 9).
gitops-deploy deploys only from the default branch, and waits for a person when deploy_when is manual (chapter 34).

megacorp/devops/componentstemplates/sonar-scan.yml

# templates/sonar-scan.yml in megacorp/devops/components: the sonar-scan CI/CD component.
# Projects include it with
#   component: $CI_SERVER_FQDN/megacorp/devops/components/sonar-scan@<version>
# Used by chapters 09 and 28.

# catalog: spec-inputs
spec:
  inputs:
    stage:
      default: test
    project_key:
      description: SonarQube project key
    quality_gate:
      type: boolean
      default: true
---

sonar-scan:
  stage: $[[ inputs.stage ]]
  image: registry.example.com/megacorp/devops/ci-tools/sonar-scanner:6
  variables:
    SONAR_PROJECT_KEY: $[[ inputs.project_key ]]
    SONAR_QUALITYGATE_WAIT: "$[[ inputs.quality_gate ]]"
  script:
    - sonar-scanner -Dsonar.projectKey="$SONAR_PROJECT_KEY" -Dsonar.host.url="$SONAR_HOST_URL"
  rules:
    - if: $SKIP_SONAR == "true"
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  allow_failure: true

megacorp/devops/componentstemplates/ecr-push/template.yml

# templates/ecr-push/template.yml in megacorp/devops/components: the ecr-push component.
# Builds an image with buildah and pushes it to Amazon ECR, authenticating with the
# job's OIDC ID token. Used by chapters 09 and 32.

spec:
  inputs:
    repository:
      description: ECR repository name
    context:
      default: .
    role_arn:
      default: arn:aws:iam::123456789012:role/gitlab-ecr-push
    tag:
      default: $CI_COMMIT_SHA
---

ecr-push-$[[ inputs.repository ]]:
  stage: package
  image: registry.example.com/megacorp/devops/ci-tools/buildah:1.37
  id_tokens:
    MC_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: $[[ inputs.role_arn ]]
    ECR_REGISTRY: 123456789012.dkr.ecr.eu-west-2.amazonaws.com
  script:
    - source /opt/megacorp/lib/ci-lib.sh
    - mc_aws_login "$AWS_ROLE_ARN"
    - aws ecr get-login-password | buildah login --username AWS --password-stdin "$ECR_REGISTRY"
    - buildah bud -t "$ECR_REGISTRY/$[[ inputs.repository ]]:$[[ inputs.tag ]]" "$[[ inputs.context ]]"
    - buildah push "$ECR_REGISTRY/$[[ inputs.repository ]]:$[[ inputs.tag ]]"

megacorp/devops/componentstemplates/gitops-deploy/template.yml

# templates/gitops-deploy/template.yml in megacorp/devops/components: the gitops-deploy component.
# It wraps the multi-project trigger to the platform team's deployer, so a project can
# deploy with one include instead of extending the .deploy template.
# Used by chapters 09 and 34.

spec:
  inputs:
    environment:
      options: [staging, production]
    app:
      default: $CI_PROJECT_NAME
    deploy_when:
      default: on_success
      options: [on_success, manual]
---

deploy-$[[ inputs.environment ]]:
  stage: deploy
  trigger:
    project: megacorp/platform/deployer
    branch: main
    strategy: depend
  variables:
    APP: $[[ inputs.app ]]
    ENVIRONMENT: $[[ inputs.environment ]]
    IMAGE_REF: $IMAGE_REF
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: $[[ inputs.deploy_when ]]

data/mono: a monorepo#

A monorepo that doesn't use the golden pipeline. Worth noticing:

generate-pipeline writes a pipeline file, and service-pipelines runs it as a child pipeline (chapter 10).
generate.sh fails on merge requests with a long history, and builds too little after a push of several commits (chapter 25).
docs starts its child pipeline only when files under docs/ changed. For a new branch, changes is always true (chapter 16).
lint-docs runs GitLab Functions instead of a script, an experimental feature (chapter 12).

megacorp/data/mono.gitlab-ci.yml

# .gitlab-ci.yml in megacorp/data/mono
# A monorepo. A generator job writes one child-pipeline job per changed service,
# and a static child pipeline builds the docs. Used by chapters 08, 10 and 36.

# catalog: include-local
include:
  - local: ci/common.yml

stages: [generate, build]

generate-pipeline:
  stage: generate
  script:
    - ci/generate.sh > generated-pipeline.yml
  artifacts:
    paths: [generated-pipeline.yml]

# catalog: dynamic-child
service-pipelines:
  stage: build
  trigger:
    include:
      - artifact: generated-pipeline.yml
        job: generate-pipeline
    strategy: depend

# catalog: parent-child
docs:
  stage: build
  trigger:
    include: ci/docs.yml
  rules:
    - changes: [docs/**/*]

megacorp/data/monoci/common.yml

# ci/common.yml in megacorp/data/mono
# Defaults shared by the monorepo's own pipeline, pulled in with include: local.
# Used by chapter 08.

default:
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  tags: [megacorp-shared]

megacorp/data/monoci/generate.sh

#!/usr/bin/env bash
# ci/generate.sh in megacorp/data/mono
# Writes one child-pipeline job per service that changed. Its output is the YAML
# that the service-pipelines trigger job runs. Used by chapter 10.
set -euo pipefail

base="${CI_MERGE_REQUEST_DIFF_BASE_SHA:-HEAD~1}"
changed=$(git diff --name-only "$base" HEAD -- services/ | cut -d/ -f2 | sort -u)

echo "stages: [build]"
for svc in $changed; do
  cat <<EOF
build-${svc}:
  stage: build
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  script:
    - make -C services/${svc} build
EOF
done

# a child pipeline needs at least one job, even when nothing changed
if [ -z "$changed" ]; then
  printf 'no-changes:\n  stage: build\n  script: [echo "no service changed"]\n'
fi

megacorp/data/monoci/docs.yml

# ci/docs.yml in megacorp/data/mono
# The child pipeline that builds the monorepo's documentation. It also carries an
# experimental job that uses GitLab Functions (formerly CI/CD Steps) instead of a script.
# Used by chapters 10 and 12.

build-docs:
  image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
  script:
    - mkdocs build --strict

# catalog: ci-steps
lint-docs:
  image: registry.example.com/megacorp/devops/ci-tools/docs:1.4
  run:
    - name: markdown_lint
      func: registry.example.com/megacorp/devops/functions/markdownlint:1.2.0
      inputs:
        path: docs
    - name: report
      script: echo "markdown lint finished"

web/web-portal: a project with no pipeline file#

web-portal's repository has no .gitlab-ci.yml. Its settings point at pipelines/web.yml in the templates project, shown above, and that file is its whole pipeline (chapter 11). For the other ways a project without the file still runs pipelines, see chapter 36.

megacorp/web/web-portalSETTINGS.yml

# SETTINGS.yml for megacorp/web/web-portal
# This file DESCRIBES the project's CI/CD settings in the GitLab UI. GitLab does
# not read it. The repository itself has no .gitlab-ci.yml: its pipeline comes
# from the central templates project. Used by chapters 11, 13 and 36.

# catalog: custom-config-path
ci_cd_configuration_file: pipelines/web.yml@megacorp/devops/ci-templates

project_variables:
  - key: SITE_BUCKET
    value: www-example-com-site
  - key: CF_DISTRIBUTION_ID
    value: E2EXAMPLE1234

platform/deployer: every deploy goes through here#

The platform team's project. Service pipelines trigger it, and people can run it by hand. Worth noticing:

Someone running it by hand picks an environment from the input's options. When a service triggers it, the trigger's ENVIRONMENT beats the value built from the input (pipeline inputs).
resource_group: $APP-$ENVIRONMENT lets only one deploy of an app to an environment run at a time (chapter 17).
Without an IMAGE_REF there is no deploy, and production waits for a person.
GITOPS_TOKEN expires a year after it was created. From that day, every deploy fails (chapter 21).

megacorp/platform/deployer.gitlab-ci.yml

# .gitlab-ci.yml in megacorp/platform/deployer
# The platform team's deployment pipeline. Service pipelines trigger it as a
# multi-project pipeline, and people can also run it by hand from the UI.
# Used by chapters 10, 33 and 34.

# catalog: pipeline-inputs
spec:
  inputs:
    environment:
      type: string
      default: staging
      options: [staging, production]
---

variables:
  ENVIRONMENT: $[[ inputs.environment ]]

deploy:
  stage: deploy
  image: registry.example.com/megacorp/devops/ci-tools:3.2
  resource_group: $APP-$ENVIRONMENT
  environment:
    name: $ENVIRONMENT/$APP
  script:
    - mc deploy "$ENVIRONMENT" "$IMAGE_REF"
  rules:
    - if: $IMAGE_REF == null
      when: never
    - if: $ENVIRONMENT == "production"
      when: manual
    - when: on_success

megacorp/platform/deployerSETTINGS.yml

# SETTINGS.yml for megacorp/platform/deployer
# This file DESCRIBES the project's settings in the GitLab UI. GitLab does not read it.
# Used by chapters 21 and 34.

project_variables:
  - key: GITOPS_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true
    # A project access token created in megacorp/platform/gitops-config, with the
    # Maintainer role and the write_repository scope. Nobody entered an expiry date,
    # so GitLab set one 365 days after the token was created.

Settings, runners and AWS#

What MegaCorp keeps outside every repository. Projects can't see most of it, yet all of it reaches their jobs. Worth noticing:

A group variable such as MAVEN_CLI_OPTS beats anything a project writes in YAML (chapter 19).
SONAR_TOKEN is protected, so a merge request pipeline from an unprotected branch gets it empty (chapter 20).
The instance runs Auto DevOps for any project without a pipeline file (chapter 11).
The runner adds variables, a default image and an AWS role to every job it runs (chapter 22).
The ECR push role trusts branches only, so a tag pipeline can't push (chapter 31). The web role trusts web-portal's main and nothing else.
The runner's role lets every job on that runner pull from ECR without logging in (chapter 21).

settingsinstance.yml

# _settings/instance.yml
# This file DESCRIBES settings made in the GitLab UI of MegaCorp's self-managed
# instance, gitlab.example.com. GitLab does not read it: it exists so the book can
# show what those settings are and where they bite.
# Used by chapters 11 (org-wide reuse), 19 (variables) and 22 (runners).

# catalog: instance-group-variables
# Every project on the instance receives these, unless a group or project
# variable with the same name takes precedence.
instance_variables:
  - key: MEGACORP_REGISTRY
    value: registry.example.com
  - key: HTTPS_PROXY
    value: http://proxy.example.com:3128

# catalog: instance-template-repo
# Self-managed only: the .yml files in this project's gitlab-ci/ folder appear in the
# Web Editor's template list when anyone creates a CI/CD file. Choosing one copies it
# into the new file; include: template cannot use them.
instance_template_repository:
  project: megacorp/devops/instance-templates
  offers:
    - gitlab-ci/MegaCorp-Legacy-Java.yml

# catalog: auto-devops
# A project with no CI/CD configuration file gets GitLab's Auto DevOps pipeline.
auto_devops:
  default_to_auto_devops: true

settingsgroup-megacorp.yml

# _settings/group-megacorp.yml
# This file DESCRIBES settings made in the GitLab UI on the top-level group
# "megacorp". GitLab does not read it.
# Used by chapters 11 (org-wide reuse), 19 (variables) and 29 (policies and gates).

# catalog: instance-group-variables
# Every project under megacorp/ receives these. A group variable takes precedence
# over any variable of the same name in a project's .gitlab-ci.yml.
group_variables:
  - key: MAVEN_CLI_OPTS
    value: "--batch-mode --errors --show-version -s .m2/settings.xml"
  - key: AWS_ACCOUNT_ID
    value: "123456789012"
  - key: SONAR_HOST_URL
    value: https://sonar.example.com
  - key: SONAR_TOKEN
    value: "(set in the UI)"
    masked: true
    protected: true

# catalog: project-templates
# New projects can be created from these. The copy happens once, at creation.
group_project_templates:
  source_group: megacorp/templates
  templates:
    - megacorp/templates/java-service-starter
    - megacorp/templates/batch-job-starter

# catalog: compliance-pipeline
# Legacy: a compliance framework with its own pipeline configuration, the
# predecessor of pipeline execution policies.
compliance_frameworks:
  - name: SOX
    pipeline_configuration: compliance/sox.yml@megacorp/security/policies
    applied_to:
      - megacorp/payments/payments-api

runnersconfig.toml

# _runners/config.toml
# An excerpt of the GitLab Runner configuration on MegaCorp's shared runners.
# Projects cannot see this file, yet everything in it reaches their jobs.
# Used by chapters 12 (reuse outside the YAML), 21 (identity and tokens) and 22 (runners
# and executors).

concurrent = 20

# catalog: runner-config
[[runners]]
  name = "megacorp-eks-shared"
  # registered in GitLab as a group runner of megacorp, with the tag megacorp-shared
  url = "https://gitlab.example.com"
  executor = "kubernetes"
  environment = ["MAVEN_OPTS=-Xmx2g", "HTTPS_PROXY=http://proxy.example.com:3128"]
  pre_build_script = "echo 'runner: megacorp-eks-shared'"
  [runners.kubernetes]
    namespace = "gitlab-runners"
    image = "registry.example.com/megacorp/devops/ci-tools:3.2"
    # jobs run as this service account, which is bound to an AWS IAM role
    service_account = "gitlab-runner-jobs"

awsiam.yml

# _aws/iam.yml
# This file DESCRIBES how MegaCorp's AWS account 123456789012 trusts GitLab. AWS holds
# the real configuration, as JSON policies. GitLab does not read this file.
# Used by chapters 21 (identity and tokens) and 31 (GitLab and AWS).

# AWS accepts ID tokens signed by this GitLab, for this audience.
oidc_provider:
  arn: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
  url: https://gitlab.example.com
  audiences: [https://gitlab.example.com]

# Assumed by image-build, and by the ecr-push component, through mc_aws_login.
ecr_push_role:
  arn: arn:aws:iam::123456789012:role/gitlab-ecr-push
  trust_policy:
    Effect: Allow
    Principal:
      Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
    Action: sts:AssumeRoleWithWebIdentity
    Condition:
      StringEquals:
        gitlab.example.com:aud: https://gitlab.example.com
      StringLike:
        gitlab.example.com:sub: project_path:megacorp/*:ref_type:branch:ref:*
  permissions: push and pull images in every ECR repository in the account

# Assumed by web-portal's deploy-site job.
web_publish_role:
  arn: arn:aws:iam::123456789012:role/gitlab-web-publish
  trust_policy:
    Effect: Allow
    Principal:
      Federated: arn:aws:iam::123456789012:oidc-provider/gitlab.example.com
    Action: sts:AssumeRoleWithWebIdentity
    Condition:
      StringEquals:
        gitlab.example.com:aud: https://gitlab.example.com
        gitlab.example.com:sub: project_path:megacorp/web/web-portal:ref_type:branch:ref:main
  permissions: write to the www.example.com bucket, and invalidate its CloudFront cache

# The role behind the gitlab-runner-jobs service account on the megacorp-eks-shared
# runner. Every job on that runner can use it without asking. The EKS cluster is what
# AWS trusts for this role, not GitLab.
runner_role:
  arn: arn:aws:iam::123456789012:role/gitlab-runner-jobs
  permissions: pull images from every ECR repository in the account

security/policies: rules no project shows#

MegaCorp's security policy project. Nothing here appears in any project's configuration, and no toggle stops it (chapter 29). Worth noticing:

Secret detection runs in every pipeline, on every branch.
The guardrails policy adds policy-sbom to every pipeline, in the reserved stage .pipeline-policy-post.
The approval policy doesn't set fallback_behavior, so it fails closed. A merge request whose required scans produced no report needs the security team's approval too (chapter 29).

megacorp/security/policies.gitlab/security-policies/policy.yml

# .gitlab/security-policies/policy.yml in megacorp/security/policies
# MegaCorp's security policy project, linked to the top-level group. These rules
# reach every project without appearing in any project's CI/CD configuration.
# Used by chapters 11, 13, 27 and 29.

# catalog: scan-execution-policy
scan_execution_policy:
  - name: Secret detection everywhere
    description: Run secret detection in every pipeline on every branch.
    enabled: true
    rules:
      - type: pipeline
        branches: ["*"]
    actions:
      - scan: secret_detection
  - name: Nightly dependency scan
    description: Scan dependencies on the default branch every night.
    enabled: true
    rules:
      - type: schedule
        branches: [main]
        cadence: "0 2 * * *"
    actions:
      - scan: dependency_scanning

# catalog: pipeline-execution-policy
pipeline_execution_policy:
  - name: MegaCorp guardrails
    description: Add an SBOM job to every pipeline.
    enabled: true
    pipeline_config_strategy: inject_policy
    content:
      include:
        - project: megacorp/security/policies
          file: pipeline-policies/megacorp-guardrails.yml

# catalog: mr-approval-policy
approval_policy:
  - name: Block new critical vulnerabilities
    description: A merge into main that adds a critical finding needs security's approval.
    enabled: true
    rules:
      - type: scan_finding
        branches: [main]
        scanners: [sast, secret_detection, dependency_scanning, container_scanning]
        vulnerabilities_allowed: 0
        severity_levels: [critical]
        vulnerability_states: [new_needs_triage]
    actions:
      - type: require_approval
        approvals_required: 1
        group_approvers: [megacorp/security]

megacorp/security/policiespipeline-policies/megacorp-guardrails.yml

# pipeline-policies/megacorp-guardrails.yml in megacorp/security/policies
# The CI/CD configuration that the "MegaCorp guardrails" pipeline execution
# policy injects into every project's pipeline. Used by chapters 11, 13 and 29.

policy-sbom:
  stage: .pipeline-policy-post
  image: registry.example.com/megacorp/devops/ci-tools/syft:1
  script:
    - syft dir:. -o cyclonedx-json=sbom.cdx.json
  artifacts:
    reports:
      cyclonedx: sbom.cdx.json

legacy/billing-batch: the older ways#

Created in 2019 from a starter template, and never updated. Worth noticing:

Anchors and merge keys do the work of extends, and only does the work of rules (chapter 6).
Its include follows another project's main by URL, so any change there reaches it at once (chapter 5).
test downloads a script and runs it. What it does is in no file of the project (chapter 12).
notify-reports starts a pipeline in project 4242 with a trigger token (chapter 10).

megacorp/legacy/billing-batch.gitlab-ci.yml

# .gitlab-ci.yml in megacorp/legacy/billing-batch
# A 2019-era project, created from a starter template and never updated since.
# It shows the older ways configuration was shared.
# Used by chapters 06, 08, 10 and 12.
# catalog: project-templates

# catalog: include-remote
include:
  - remote: https://gitlab.example.com/megacorp/devops/ci-templates/-/raw/main/legacy/notify.yml

# catalog: yaml-anchors, yaml-merge-key
.defaults: &defaults
  image: registry.example.com/megacorp/devops/ci-tools:2.9
  tags: [megacorp-shared]
  only:
    - branches
    - tags

build:
  <<: *defaults
  stage: build
  script:
    - ./gradlew assemble

# catalog: fetched-scripts
test:
  <<: *defaults
  stage: test
  script:
    - curl -sSL https://gitlab.example.com/megacorp/devops/scripts/-/raw/main/run-tests.sh | bash

# catalog: trigger-api
notify-reports:
  <<: *defaults
  stage: deploy
  only: [tags]
  script:
    - >
      curl --fail --request POST
      --form "token=$REPORTS_TRIGGER_TOKEN"
      --form "ref=main"
      --form "variables[BILLING_VERSION]=$CI_COMMIT_TAG"
      "https://gitlab.example.com/api/v4/projects/4242/trigger/pipeline"

Next49Glossary and former names3 min

Glossary and former names#

Appendix · Appendices · Chapter 49· 3 min read

First, the features GitLab has renamed or replaced. When an older template, an older page or a colleague uses a name you can't find, look it up here. Then, every term this book uses, in one line each, with a link to where it's explained.

Former names#

You may meet	Today, at 19.3	When it changed
scan result policies, `scan_result_policy`	merge request approval policies, `approval_policy`	only `approval_policy` is accepted since 17.0
`newly_detected`, in a policy's `vulnerability_states`	`new_needs_triage` and `new_dismissed`	17.0
compliance pipelines	pipeline execution policies	deprecated in 17.3; removal planned for 20.0
`inject_ci`, in a pipeline execution policy	`inject_policy`	17.9; `inject_ci` is deprecated
`CI_JOB_JWT`, `CI_JOB_JWT_V2`	ID tokens, requested with `id_tokens`	removed in 17.0
Token Access, in Settings › CI/CD	Job token permissions	17.2
Limit access to this project, then Authorized groups and projects	CI/CD job token allowlist	17.2, then 17.3
Merge when pipeline succeeds	auto-merge	–
CI/CD Steps	GitLab Functions, an experiment	–
`only`, `except`	`rules`	deprecated
`image`, `services`, `cache`, `before_script` or `after_script` at the top of the file	the same keywords inside `default:`	deprecated
`Security/SAST.gitlab-ci.yml` and the other `Security/` templates	wrappers: each only includes its `Jobs/` twin, such as `Jobs/SAST.gitlab-ci.yml` (chapter 27)	–
Gemnasium, the dependency scanning analyzer	dependency scanning using an SBOM	Gemnasium deprecated in 17.9; removal proposed for 20.0
Code Quality's CodeClimate template	any tool's results, imported as a `codequality` report	the template is deprecated
kaniko, for building images	Docker, Buildah, Podman or rootless BuildKit	GitLab's page marks kaniko removed

Terms#

Term	Means	Explained in
`!reference`	a YAML tag that pastes one section of another job, such as its rules or script	chapter 7
agent for Kubernetes	GitLab's program inside a cluster, which connects the cluster to GitLab	chapter 34
anchor, alias, merge key	YAML's own copy and paste, with `&` and `*`; it works only inside one file	chapter 6
artifact	files a job keeps when it ends, for later jobs and for people	chapter 24
AssumeRoleWithWebIdentity	the AWS call that swaps an ID token for temporary AWS keys	chapter 31
Auto DevOps	GitLab's built-in pipeline, for projects with no pipeline file when the setting is on	chapter 11
cache	files kept between jobs to save time; they may be missing at any time	chapter 24
child pipeline	a pipeline that a trigger job starts from a file in the same project	chapter 10
CI Lint	a tool that checks a piece of pipeline YAML	chapter 3
component	a versioned piece of configuration with typed inputs, included with `include: component`	chapter 9
container scanning	a scanner that looks for known vulnerabilities in an image	chapter 26
`default:`	settings every job gets unless it sets its own	chapter 7
dependency scanning	a scanner that looks for known vulnerabilities in the libraries a project uses	chapter 26
deployment job	a job with `environment:`, which records what it deployed, and where	chapter 33
dotenv report	a file of `NAME=value` lines that a job leaves as variables for later jobs	chapter 24
downstream pipeline	a pipeline that another pipeline started: a child or a multi-project pipeline	chapter 10
dynamic child pipeline	a child pipeline whose YAML an earlier job wrote	chapter 10
ECR	Amazon Elastic Container Registry, where AWS keeps container images	chapter 31
environment	a place a job deploys to, such as staging, with its history of deployments	chapter 33
executor	how a runner runs jobs: in Docker containers, in Kubernetes pods, in a shell, and others	chapter 22
`extends`	copies another job's settings: maps are merged, lists are replaced	chapter 7
Full configuration	the pipeline editor's view of every file, merged into one	chapter 3
GitOps	deploying by writing the wanted state to a Git repository, which a tool in the cluster applies	chapter 34
hidden job	a job whose name starts with a dot; it never runs, and exists to be copied	chapter 6
IAM role, trust policy	an AWS identity a job can take on; its trust policy says who may	chapter 31
ID token	a note about the job, signed by GitLab, that another system such as AWS can check	chapter 21
`include`	copies other YAML files into the configuration	chapter 8
input	a typed parameter of a component, an included file or a pipeline, filled in when the pipeline is created	chapter 9
instance, group and project runners	runners for every project, for one group's projects, or for chosen projects	chapter 22
job	one piece of work in a pipeline, run by one runner	chapter 2
job token	`CI_JOB_TOKEN`, which lets a job call GitLab while it runs	chapter 21
masked, hidden, protected	how a variable is guarded: masked in logs, hidden in the settings, passed only to protected branches and tags	chapter 20
merge request approval policy	a security policy that requires approval when scans find problems	chapter 29
merge request pipeline	a pipeline for a merge request, rather than for a branch	chapter 14
merge train	a queue of merge requests, each tested together with those ahead of it	chapter 14
merged results pipeline	a merge request pipeline that runs on the source branch merged into the target	chapter 14
multi-project pipeline	a pipeline in another project, started by a trigger job	chapter 10
`needs`	starts a job as soon as the named jobs finish, ignoring stages	chapter 17
OIDC	OpenID Connect, the standard that ID tokens follow	chapter 21
pipeline	all the jobs GitLab runs for one commit, for one reason such as a push	chapter 2
pipeline execution policy	a security policy that adds jobs to projects' pipelines	chapter 29
pipeline source	what started a pipeline, held in `CI_PIPELINE_SOURCE`	chapter 14
predefined variable	a variable GitLab sets, such as `CI_COMMIT_BRANCH`	predefined variables
protected branch, protected tag	branches and tags that only some people may push to; only their pipelines get protected variables	chapter 20
`resource_group`	lets only one job of the group run at a time	chapter 17
`rules`	decide whether a job is in a pipeline, and how it runs	chapter 16
runner	the program that takes jobs from GitLab and runs them	chapter 22
SAST	static application security testing: a scanner that reads source code	chapter 26
SBOM	software bill of materials: a list of every component in a build	chapter 26
scan execution policy	a security policy that adds scanner jobs to pipelines, or runs them on a schedule	chapter 29
secret detection	a scanner that looks for passwords and keys committed to the repository	chapter 26
security policy project	the project that holds a group's security policies	chapter 29
service	an extra container beside the job, such as a database	chapter 23
settings variable	a variable set in the CI/CD settings of a project, a group or the instance	chapter 19
shallow clone	a copy of the repository with only the newest commits	chapter 25
stage	a group of jobs; by default, a stage starts only when the stage before it has succeeded	chapter 17
tag	a Git tag names a commit; a runner tag is a label that matches jobs to runners	chapter 22
template	YAML that GitLab ships, included with `include: template`	chapter 8
toggle	a variable a template checks to switch a job off, such as `SAST_DISABLED`	chapter 9
trigger job	a job that starts a downstream pipeline instead of running a script	chapter 10
truth table	this book's method for predicting which jobs run in which pipeline	chapter 18
`workflow:rules`	decide whether a pipeline is created at all	chapter 15

Next50One-page card1 min

One-page card#

Appendix · Appendices · Chapter 50· 1 min read

The book on one page, to print and keep next to your screen.

Five questions for any job#

Ask them in this order (chapter 35):

Question	Look at
Where is it defined?	Full configuration, then its `extends` chain and includes; if it's in no file, policies
When does it run?	`workflow:rules`, the job's final `rules`, and `needs`
Where does it run?	the job page's runner and tags, and the job's image
What does it execute?	its scripts, and the tools and functions those call
What goes in and out?	variables and secrets; artifacts, caches, reports, images and deploys

Four kinds of symptom#

Symptom	Check first	Cards
Didn't run, or ran when it shouldn't	the pipeline exists; the rules matched; everything it needs exists	didn't run
Failed	the log's first error; the image and runner; what went in	failed
Passed, but did the wrong thing	which value won; which image or artifact; what was deployed	wrong result
Stuck, slow, or broke with no commit	runners online; tokens expired; central files or images changed	stuck or drifting

Which variable value wins, highest first#

Security policies, then values given when the pipeline started, then the project's settings, then the groups', closest first, then the instance's. Below those come dotenv reports, then the job's own YAML, then top-level YAML, then deployment and predefined variables (chapter 19).

Where to look in GitLab#

Place	Shows
Build › Pipeline editor, Full configuration	every file and template, merged
Build › Pipeline editor, Validate	the jobs a pipeline would get, and problems with them
The pipeline graph	which jobs exist, and any downstream pipelines
The job page	the runner, the image, the timeout, and the log
Settings › CI/CD › Variables, Runners, General pipelines	settings variables, runners, clone depth and the pipeline file's location
Settings › CI/CD › Job token permissions	which projects may use your job tokens
Secure › Policies	the security policies that apply
Operate › Environments	deployments, approvals and rollback

Commands worth remembering#

aws sts get-caller-identity                              # which AWS role is really in use
echo "$MC_ID_TOKEN" | cut -d '.' -f2 | base64 -d | jq .  # an ID token's claims, not the token
env | cut -d= -f1 | sort                                 # which variables exist: names only
git fetch --unshallow                                    # full history, for scripts that need it

Ten traps#

A job's own rules, before_script, tags or services replace the inherited list completely.
A settings variable beats anything in the YAML.
A job's variables, including those from extends, beat top-level ones.
Toggles never stop a security policy's jobs.
GitLab's scanners skip merge request pipelines unless AST_ENABLE_MR_PIPELINES is "true".
Protected values reach only protected branches and tags.
A cache can miss at any time; only artifacts are guaranteed.
Jobs get a shallow clone of their own ref only.
With GitOps, green means written to Git, not running.
A pipeline that rebuilds an image ships something nobody tested.

End of the book