Project TBD Pipeline (CI/CD)¶

Overview¶

This document describes the CI/CD pipeline architecture for service repositories operating under the Trunk-Based Development (TBD) model.

Branching model:

Single deployable branch: main
Short-lived feature branches merged via Merge Requests
Short-lived hotfix/{version} branches for emergency fixes
Automated semantic versioning via Git Tags
Deployments driven by ArgoCD (GitOps)

Key principles:

Every merge to main produces a versioned, deployable artifact
Version numbers are calculated automatically from commit messages (Conventional Commits)
No CHANGELOG.md in the repository — changelogs exist only in GitLab Releases
Deployments are tag-based: build once, deploy many times

DRAFT vs DFT¶

Two names for the draft-quality environment appear in different layers — they refer to the same environment but serve different purposes:

Layer	Value	Where
Pipeline name prefix / deployment target	`DRAFT`	GitLab pipeline workflow (`PIPELINE_ENV`, `CI_DEPLOY_TARGET`)
Overlay directory	`dft`	`workloads/<group>/<service>/overlays/dft/`
`env.ENV` value in overlay	`DFT`	Passed to the application as an environment variable

The main pipeline auto-deploys with CI_DEPLOY_TARGET=DRAFT; the GitOps update writes to the dft/ overlay; the container sees ENV=DFT.

Branch Setup¶

Both merge methods are supported:

Merge commit — merge commits (Merge branch ...) are always allowed and ignored by version analysis
Fast-forward merge — preferred for a cleaner linear history
Squash merge — when used, the MR title becomes the commit message and must follow Conventional Commits format (validated by the validate MR job)

Pipeline Types¶

The system uses six distinct pipeline types, triggered automatically by different Git events.

Main Pipeline (push/merge to main)¶

Trigger: Push or MR merge to main Pipeline name: [DRAFT] <commit title>

Stage	Job	Description
pre	validate repo, trivy, sonarqube, clamav	Security and quality scans (each individually toggleable)
prepare	pipeline gate	Serializes concurrent main pipelines (waits for older to finish)
version	get version	Calculates next semantic version from commit history
build	build	Docker build + push to Harbor registry
postbuild	apply version	Creates annotated Git tag (`vX.Y.Z`) and pushes it → triggers Tag pipeline
deploy	deploy (DRAFT)	Auto-deploys to the DRAFT environment via ArgoCD GitOps
info	info	Slack notification with build summary

Flow:

MR merge to main
  │
  ├─ scans (trivy, sonarqube, clamav — individually toggleable)
  ├─ pipeline gate (wait for older pipelines)
  ├─ get version (semantic analysis → e.g. "1.5.0", bump=minor)
  ├─ build (docker build+push)
  ├─ apply version (create+push tag v1.5.0)
  │     └─ triggers Tag Pipeline
  └─ deploy DRAFT (auto, via ArgoCD)

Tag Pipeline¶

Trigger: Tag push from apply version job on main Pipeline name: [TAG] v1.5.0

Stage	Job	Description
version	get version	Parses version from tag name (`v1.5.0` → `1.5.0`)
deploy	Deploy DEV	Manual button — triggers API Deploy pipeline

No build occurs

The Docker image was already built and pushed in the Main pipeline.

Deploy Pipeline (API trigger)¶

Trigger: GitLab API pipeline trigger (ref=main), initiated by a manual Deploy button Pipeline name: [DEV] deploy v1.5.0 (or [TEST], [STAGE], [PROD] for hotfix deploys)

Stage	Job	Description
version	get version	Reads version from `CI_COMMIT_TAG` (passed via API variable)
postbuild	create release	Skipped here if the release was already created in a hotfix tag pipeline
deploy	deploy	Deploys to the target environment via ArgoCD GitOps
finish	Create Hotfix	Manual button — creates `hotfix/{version}` branch from the deployed tag
info	info	Slack notification with version and changelog

Why ref=main?

Vault JWT authentication requires the pipeline to run on a protected branch. The API trigger runs on main with CI_ENV and CI_COMMIT_TAG passed as variables.

Hotfix Branch Pipeline¶

Trigger: Push to hotfix/* branch (created via "Create Hotfix" button on a deploy pipeline) Pipeline name: [HOTFIX] <commit title>

Stage	Job	Description
pre	validate repo, trivy, sonarqube, clamav	Scans (same as main, individually toggleable)
version	get version	Calculates next free patch in the branch's minor version (`v1.5.0.1` → `v1.5.1`)
build	build	Docker build + push (interruptible: new commits cancel older builds)
deploy	Apply Hotfix	Manual button — creates and pushes tag `v{CI_VERSION}` with annotated message `hotfix {branch}` → triggers Hotfix Tag pipeline

auto_cancel on hotfix branches

Unlike main, hotfix branch pipelines use auto_cancel.on_new_commit: interruptible. Pushing a new commit to the hotfix branch cancels the interruptible jobs (e.g. build) of the prior pipeline.

Hotfix Tag Pipeline¶

Trigger: Tag push from the Apply Hotfix job (tag message starts with hotfix) Pipeline name: [HOTFIX][TAG] v1.5.1

Stage	Job	Description
version	get version	Parses version from the hotfix tag
postbuild	create release	Publishes a GitLab Release with changelog since the previous release
deploy	Deploy TEST	Manual button — triggers API Deploy pipeline
deploy	Deploy STAGE	Manual button — triggers API Deploy pipeline
deploy	Deploy PROD	Manual button — triggers API Deploy pipeline

The hotfix tag pipeline is distinguished from the regular tag pipeline by the tag's annotation message (prefix hotfix), which is set by TBD-ApplyHotfixVersion. A regular tag (from apply version on main) has no such message and triggers the plain Tag pipeline with a single Deploy DEV button.

Deploy buttons

Buttons are visible only for environments listed in the project's CI_DEPLOY_ENVS variable.

MR Pipeline¶

Trigger: Merge Request event (open/update MR) Pipeline name: [MR] <commit title>

Stage	Job	Description
pre	validate MR	Validates MR title against Conventional Commits format
version	get version	Calculates candidate version (validation only, not applied)
build	build	Docker build (validation only, image not used for deployment)

Semantic Versioning¶

Versions are calculated automatically from commit messages following the Conventional Commits specification.

Commit Format¶

<type>[(<scope>)]: <description>

Type	Version Bump	Changelog Section	Example
`feat`	Minor (0.Y+1.0)	Added	`feat: add user dashboard`
`fix`	Patch (0.0.Z+1)	Fixed	`fix(auth): resolve token expiry`
`chore`	Patch	Changed	`chore: update dependencies`
`remove`	Patch	Removed	`remove: drop legacy API endpoint`
`BREAKING CHANGE`	Major (X+1.0.0)	Changed	`BREAKING CHANGE: remove v1 API`

BREAKING CHANGE variants

Both BREAKING CHANGE: (with space) and BREAKING-CHANGE: (with hyphen) are accepted. They can appear in the commit subject or body.

Merge commits (Merge branch ...) are always allowed and ignored by version analysis.

Non-semantic commits default to patch bump but are blocked from being pushed to main by the server-side hook.

Version Calculation Logic¶

On main:

Find the nearest ancestor tag matching v* from the current commit
If no tags exist, start from 0.0.0
Analyze all commits since that tag (full commit messages including body)
Determine the highest bump level (major > minor > patch)
Apply bump to produce the new version

Idempotency: If HEAD already has a tag, the version is parsed from that tag (no recalculation). CI_SEMANTIC_BUMP is set to "none", and apply version skips tag creation. Pipeline reruns produce identical results.

On hotfix/X.Y.Z branches:

Parse the major/minor from the branch name (hotfix/1.5.0 → major=1, minor=5)
Scan all tags matching v1.5.* and find the maximum patch number
Next version is v1.5.{max+1}
If HEAD already points to a tag in this minor, reuse it (idempotent rerun)

On MR pipelines: candidate version = nearest ancestor tag + pipeline IID as a 4^th component (1.5.0.42). Used for Docker image tag; no Git tag is created.

Tag Format¶

Git tags use the v prefix: v1.2.3
CI_VERSION strips the prefix: 1.2.3
CI_VERSION_FULL = CI_VERSION
CI_VERSION_TAG = general (backward compatibility only)
CI_SEMANTIC_BUMP = major | minor | patch | none
Initial version: 0.0.0 → first feat: commit → 0.1.0

Pipeline Serialization¶

The pipeline gate job prevents concurrent Main pipelines from racing. It polls the GitLab API for older active pipelines on the same ref and waits (15s intervals) until they complete.

Combined with workflow.auto_cancel.on_new_commit: none (main ref), this ensures:

No main pipeline is cancelled by a newer push
Main pipelines execute in order of creation
Each commit receives its own version tag

Info

Requires GIT_ACCESS_TOKEN (Project/Group Access Token with api scope).

MR Title Validation¶

MR titles are validated against Conventional Commits format in the validate MR job. This ensures the squash-merge commit message on main will be parseable by the versioning system.

Valid formats:

feat: add user login
fix(auth): resolve token expiry
chore: update dependencies
BREAKING CHANGE: remove v1 API

Controlled by CI_VALIDATE_MR variable (default: enabled). Set to "false" to skip.

GitLab Release & Changelog¶

Releases are created by the create release job in two contexts:

Deploy pipeline (CI_ENV set) — normal flow after Deploy DEV
Hotfix tag pipeline (tag message starts with hotfix) — release is created directly, before any deploy

The job:

Fetches all existing GitLab Releases via API
Walks ancestor tags backwards to find the previous released tag (not just any tag — it must have an actual GitLab Release)
Generates the changelog from commits between the previous release and the current tag
Publishes the changelog as a GitLab Release
Adds a link to the current pipeline as a release asset

Warning

No CHANGELOG.md file is committed to the repository. Changelogs exist exclusively in GitLab Releases.

Idempotency

If a release for the tag already exists, the job fetches the existing release notes (for Slack notification) and exits without creating a duplicate.

Changelog sections follow the Keep a Changelog format:

Added — feat: commits
Changed — chore: and BREAKING CHANGE: commits
Fixed — fix: commits
Removed — remove: commits

Service Repository Configuration¶

.gitlab-ci.yml Template¶

include:
  - project: "ICVR/General/Templates"
    file: "/docker-builds/docker.gitlab-ci.yml"
  - project: "ICVR/General/Templates"
    file: "/kubernetes/gitops-deploy.gitlab-ci.yml"
  - project: "ICVR/General/Templates"
    file: "/common/TBD-version.gitlab-ci.yml"
    ref: feature/tbd-v2
variables:
  CI_DEPLOY_ENVS: DEV,STAGE,PROD,TEST
  CI_SONARQUBE_SCAN: "false"
  CI_CLAMAV_SCAN: "false"
  CI_TRIVY_SCAN: "false"
  CI_VALIDATE_REPO: "false"
build:
  extends: .build
  variables:
    CI_VAULT_CLIENT: ICVR
    CI_VAULT_PROJECT: Harbor
    DOCKER_FILE: Dockerfile
    DOCKER_DEFAULT_CONTEXT: "."
    DOCKER_IMAGE_NAME: devops/my-service
    DOCKER_ARGS: >-
      --build-arg VERSION=$CI_VERSION_FULL
      --build-arg COMMIT=$CI_COMMIT_SHA
      --build-arg DATE=$CI_PIPELINE_CREATED_AT
deploy:
  extends: .deploy
  variables:
    CI_VAULT_CLIENT: ICVR
    CI_VAULT_PROJECT: ArgoCD/AppSyncSecrets
    CI_VAULT_ROLE: icvr-argocd-appsyncsecrets
    ARGOCD_APP_BASE_PATH: "workloads/<group>/<service>"
    # Optional — passes $CI_ENV into the overlay's env.ENV field
    # CI_VARIABLE_MAPPINGS: '{"ENV": "$CI_ENV"}'

Include Order¶

The include order matters — TBD-version.gitlab-ci.yml must be last because it overrides workflow rules, job definitions, and rule sets from the base templates:

docker.gitlab-ci.yml — base .build job (Docker build + push)
gitops-deploy.gitlab-ci.yml — base .deploy job (ArgoCD GitOps update)
TBD-version.gitlab-ci.yml — TBD overrides (workflow, versioning, deploy menu, pipeline gate)

Required CI/CD Variables¶

Variable	Scope	Description
`GIT_ACCESS_TOKEN`	Project/Group	GitLab API token (`api` scope) for pipeline gate, deploy trigger, release creation, deployment registration, branch operations
`CI_DEPLOY_ENVS`	Project	Comma-separated list of enabled environments (controls deploy button visibility)
`BUILD_SCRIPTS`	Group (inherited)	Path to buildscripts directory in CI runner
`BUILD_SCRIPTS_REF_NAME`	Template	Branch of buildscripts repo (set in TBD template)

Optional CI/CD Variables¶

Variable	Default	Description
`CI_SONARQUBE_SCAN`	`"true"`	Enable/disable SonarQube scan
`CI_TRIVY_SCAN`	`"true"`	Enable/disable Trivy container scan
`CI_CLAMAV_SCAN`	`"true"`	Enable/disable ClamAV scan
`CI_VALIDATE_REPO`	`"true"`	Enable/disable repository structure validation
`CI_VALIDATE_MR`	`"true"`	Enable/disable MR title validation
`CI_VARIABLE_MAPPINGS`	—	JSON object mapping overlay values.yaml fields to CI variables (e.g. `{"ENV": "$CI_ENV"}`)

Docker Build: OCI Annotations¶

Service Dockerfiles should accept build arguments for version metadata, exposed as environment variables and OCI labels:

ARG VERSION=unknown
ARG COMMIT=unknown
ARG DATE=unknown
ENV APP_VERSION=${VERSION}
ENV APP_COMMIT=${COMMIT}
ENV APP_BUILD_DATE=${DATE}
LABEL org.opencontainers.image.version="${VERSION}" \
      org.opencontainers.image.revision="${COMMIT}" \
      org.opencontainers.image.created="${DATE}"

These are passed automatically via DOCKER_ARGS in the build job.

Environment Mapping¶

Environment	Pipeline Trigger	Behavior
DRAFT (overlay: `dft`)	Main pipeline (auto)	Auto-deploy after every merge to main
DEV	Tag pipeline → Deploy DEV → API pipeline	Manual trigger
TST / STG / PRD	Environment Promotion (separate ArgoCD repo)	Promotion copies `image.tag` between overlays
TEST / STAGE / PROD (hotfix)	Hotfix Tag pipeline → Deploy TEST/STAGE/PROD → API pipeline	Direct deploy of a hotfix version

Blocked Triggers¶

Source	Status	Reason
`web` (Run Pipeline UI)	Blocked	Prevents manual runs that bypass versioning
`schedule`	Allowed by base templates	Not overridden by TBD
`push` to non-`main`/non-`hotfix/*` branches without MR	Blocked	Only MR pipelines run on feature branches

Deployment Registration¶

The deploy job registers deployments with the GitLab Deployments API in after_script:

Uses environment.action: prepare to prevent GitLab auto-registration with incorrect SHA
For deploy pipelines (ref=main), resolves the actual tag SHA via Tags API
For hotfix versions with a 4-digit suffix (e.g. v1.5.0.42), strips to the base tag (v1.5.0) for SHA resolution
Ensures the target environment exists (idempotent create) before registering the deployment

Architecture Diagram¶

┌─────────────────────────────────────────────────────────────────────┐
│                        SERVICE REPOSITORY                           │
│                                                                     │
│  feature/* ──MR──► main ──push──► Main Pipeline [DRAFT]             │
│                                    │                                │
│                          ┌─────────┼──────────┐                     │
│                          │         │          │                     │
│                     get version   build   deploy DRAFT              │
│                          │                    │                     │
│                     apply version             │                     │
│                          │                    ▼                     │
│                     push tag v1.5.0     ArgoCD (dft/)               │
│                          │                                          │
│                          ▼                                          │
│                    Tag Pipeline [TAG]                               │
│                          │                                          │
│                    Deploy DEV (manual button)                       │
│                          │                                          │
│                          ▼                                          │
│                    API Pipeline [DEV] (ref=main)                    │
│                          │                                          │
│                 ┌────────┼────────┐                                 │
│                 │        │        │                                 │
│           get version  create   deploy DEV                          │
│                        release    │                                 │
│                          │        ▼                                 │
│                    GitLab Release  ArgoCD (dev/)                    │
│                          │                                          │
│                    Create Hotfix (manual)                           │
│                          │                                          │
│                          ▼                                          │
│                    hotfix/1.5.0 branch                              │
│                          │                                          │
│                    Hotfix Branch Pipeline [HOTFIX]                  │
│                          │                                          │
│                 ┌────────┼────────┐                                 │
│                 │        │        │                                 │
│           get version  build    Apply Hotfix (manual)               │
│                                   │                                 │
│                                   ▼                                 │
│                           push tag v1.5.1                           │
│                           (message: "hotfix hotfix/1.5.0")          │
│                                   │                                 │
│                                   ▼                                 │
│                     Hotfix Tag Pipeline [HOTFIX][TAG]               │
│                                   │                                 │
│                          ┌────────┼──────────────┐                  │
│                          │                       │                  │
│                    create release    Deploy TEST/STAGE/PROD         │
│                                               │                     │
│                                               ▼                     │
│                                      API Pipeline (ref=main)        │
│                                      deploy → ArgoCD                │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

End-to-End Flow: Development¶

From Feature Branch to Production

1. Push to feature/* branch   → MR pipeline validates (title + build)
2. Merge MR to main           → Main pipeline: build + tag v1.5.0 + deploy DRAFT
3. Tag pipeline               → Click "Deploy DEV" button
4. API pipeline               → Create release + deploy DEV
5. ArgoCD repo                → Run promotion: DEV → TST (release created)
6. ArgoCD repo                → Run promotion: TST → STG
7. ArgoCD repo                → Run promotion: STG → PRD
Hotfix:
  a. Click "Create Hotfix" on a deploy pipeline → hotfix/X.Y.Z branch
  b. Push fix commits to hotfix branch → hotfix branch pipeline builds
  c. Click "Apply Hotfix" → tag vX.Y.(Z+1) pushed → hotfix tag pipeline
  d. On hotfix tag pipeline: click Deploy TEST / STAGE / PROD
Rollback:
  - Promotion rollback: click rollback button on the promotion pipeline
  - Version rollback:   run an older tag's "Deploy DEV" in service repo