Skip to content

Environment Promotion

Overview

This document describes the environment promotion pipeline for the ArgoCD GitOps repository. Promotion copies image tags from a source environment overlay to a target environment overlay across all services in a workload group.

Key properties:

  • Strictly linear promotion: DEV → TST → STG → PRD (forward only, never backwards)
  • Atomic: all services in a workload group are promoted in a single commit
  • ArgoCD auto-syncs after push — no manual ArgoCD interaction required
  • Only image.tag is copied; environment-specific values (env, namespace, hostname, fullnameOverride) are never touched
  • Rollback via a one-click manual button (restores pre-promotion tags)

ArgoCD Repository Structure

argocd/
├── .gitlab-ci.yml                    # Promotion pipeline definition
└── workloads/
    ├── promote-state.yaml            # Promotion state (all envs, committed)
    └── <group>/
        ├── <service-a>/
        │   ├── base/
        │   │   └── values.yaml
        │   └── overlays/
        │       ├── dft/values.yaml
        │       ├── dev/values.yaml
        │       ├── tst/values.yaml
        │       ├── stg/values.yaml
        │       └── prd/values.yaml
        └── <service-b>/
            └── ...

Overlay values.yaml Structure

Each environment overlay defines applications with their image tags and environment-specific configuration:

fullnameOverride: "my-service-dev"          # Environment-specific, NOT promoted
namespace: "my-service-dev"                 # Environment-specific, NOT promoted
applications:
  - name: "my-app"
    image:
      repository: "harbor.icvr.xyz/icvr/devops/my-service"
      tag: "1.5.0"                          # ← THIS is promoted (copied from source)
    gitlabProjectId: 1331                   # Used for changelog and deployment tracking
    env:
      ENV: "DEV"                            # Environment-specific, NOT promoted
    service:
      http:
        port: "8080"
        ingress:
          hosts:
            - host: "my-service-dev.app.icvr.xyz"   # NOT promoted

What Is Promoted

Field Promoted? Notes
applications[].image.tag Yes Copied from source to target
applications[].image.repository No Same across environments
applications[].env.* No Environment-specific
namespace No Environment-specific
fullnameOverride No Environment-specific
hostname / ingress hosts No Environment-specific
envFeatureFlags No Managed separately

Promotion Pipeline

Trigger

The promotion pipeline is triggered exclusively via the GitLab Web UI ("Run Pipeline" button in the ArgoCD repository).

All other trigger sources are blocked:

Source Status
web (Run Pipeline) Allowed
push Blocked
merge_request_event Blocked
api Blocked
trigger Blocked
schedule Blocked
pipeline Blocked

Pipeline Variables

When running the pipeline from the Web UI, the following variables are presented as dropdowns:

Variable Default Options Description
WORKLOAD_GROUP tbd-test Dropdown (e.g., tbd-test) Workload group to promote
SOURCE_ENV DEV DEV, TST, STG Source environment to promote FROM
TARGET_ENV TST TST, STG, PRD Target environment to promote TO
VERSION_SERVICE frontend Dropdown (e.g., frontend) Lead service for release version extraction
DRY_RUN true false, true Preview changes without committing
FORCE false false, true Bypass the linear promotion matrix. Allows any SOURCE→TARGET except same-env. Use with care.

FORCE

FORCE=true appends (FORCED) to the pipeline name and allows out-of-order promotions (e.g. STG → DEV for reverting, or DEV → PRD for emergency direct delivery). The only constraint it cannot bypass is SOURCE_ENV == TARGET_ENV.

VERSION_SERVICE limitation

Release version is determined by a single lead service (VERSION_SERVICE, e.g. frontend). If only other services changed (e.g. backend 0.4.1 → 0.5.0) but frontend stayed the same, the release tag v{frontend_version} already exists from a previous promotion. Result: promotion commits and pushes correctly, but no new tag, no release, no changelog is created — the backend changes are silently untracked.

Pipeline Stages

Stage Job Trigger Description
promote promote Automatic Copy image tags, commit, push, create release (DEV→TST only)
rollback rollback Manual button Restore pre-promotion tags from state artifact

Pipeline name: Promote [DEV] v1.5.0 -> [TST] v1.3.0 (or Preview [...] for dry runs, suffixed with (FORCED) when FORCE=true).

Note

The workflow-level name — shown before the promote script rewrites it — is Run [$SOURCE_ENV] -> [$TARGET_ENV]$FORCE_SUFFIX. The promote script upgrades it to the format above as soon as it resolves the lead-service versions.

Promotion Flow

GitLab Web UI → Run Pipeline
  │
  ├─ Validate: WORKLOAD_GROUP exists (has services with source overlays)
  ├─ Validate: SOURCE_ENV → TARGET_ENV is allowed (DEV→TST, TST→STG, STG→PRD)
  │
  ├─ For each service in workload group:
  │   ├─ Read source overlay (SOURCE_ENV/values.yaml)
  │   ├─ Read target overlay (TARGET_ENV/values.yaml)
  │   ├─ For each application:
  │   │   ├─ Capture pre-state (current target tag)
  │   │   ├─ Compare source tag vs target tag
  │   │   └─ Collect yq update expression if different
  │   └─ Apply all tag updates in a single yq call per service
  │
  ├─ Write state file (workloads/promote-state.yaml)
  ├─ Commit + push (workloads/ directory — overlays + state file)
  │
  ├─ [DEV→TST only] Create Git tag + GitLab Release with changelog
  │
  ├─ Register deployment on ArgoCD project environment
  └─ Register deployments on source service projects (if gitlabProjectId set)

Allowed Promotion Matrix

Source Target Allowed Creates Release?
DEV TST Yes Yes (tag + changelog)
TST STG Yes No
STG PRD Yes No
Any other combination Only with FORCE=true Yes if source is DEV

A release (tag + changelog) is created whenever the source is DEV, regardless of target. For non-DEV sources, the target overlay is updated in place without creating a new tag.

Environment Directory Mapping

Environment Name Directory
DFT dft
DEV dev
TST tst
STG stg
PRD prd

State File

Each promotion writes a state file at workloads/promote-state.yaml, committed alongside the tag changes. The state file is partitioned by target environment and workload group — multiple promotions to different environments coexist without overwriting each other.

environments:
  "TST":
    "tbd-test":                               # SCOPE_KEY = WORKLOAD_GROUP
      promotion:
        version: "1.5.0"
        pipeline_id: "12345"
        source_env: "DEV"
        target_env: "TST"
        created_at: "2026-04-01T12:00:00Z"
        created_by: "icvr"
        tag: "v1.5.0"
      pre_state:
        tbd-test:                             # group name
          frontend:                           # service name
            tbd-test: "1.3.0"                 # app name: old tag
          backend:
            tbd-test-backend: "1.2.0"
      post_state:
        tbd-test:
          frontend:
            tbd-test: "1.5.0"
          backend:
            tbd-test-backend: "1.4.0"
  "STG":
    "tbd-test":
      promotion: ...
      pre_state: ...
      post_state: ...

Purpose:

  • pre_state — tag values before promotion (used by rollback)
  • post_state — tag values after promotion (audit trail)
  • promotion — metadata (who, when, what version, pipeline link)

Rollback

The rollback job is a manual button available after promotion completes. It restores image.tag values from the pre_state section of the state file artifact.

Rollback Flow

Manual click → rollback job
  │
  ├─ Read state file from artifact (workloads/promote-state.yaml)
  ├─ Validate: state file exists (was promote job successful?)
  ├─ Detect state format (legacy flat vs new nested per-env)
  ├─ Checkout latest branch
  │
  ├─ For each service in pre_state:
  │   ├─ Read current target overlay
  │   ├─ Compare current tag vs pre_state tag
  │   └─ Restore pre_state tag if different (batched yq call)
  │
  ├─ Commit + push
  ├─ Register rollback deployment on ArgoCD project
  └─ Register rollback on service projects

Rollback Properties

  • Independent of release — rollback reads the state artifact, not GitLab Releases
  • Idempotent — if tags already match pre_state, no commit is made
  • Scoped — only the promoted workload group is affected
  • Time-limited — state artifact expires after 7 days (GitLab artifact retention)

GitLab Release (when SOURCE is DEV)

When SOURCE_ENV == DEV, the pipeline creates a Git tag and GitLab Release in the ArgoCD repository. The common case is DEV → TST; FORCE-driven DEV → STG or DEV → PRD promotions also produce a release.

Release Version

The version is extracted from the VERSION_SERVICE lead service's image.tag in the source overlay. For example, if VERSION_SERVICE=frontend and the frontend's DEV tag is 1.5.0, the release tag is v1.5.0.

VERSION_SERVICE is auto-prefixed with WORKLOAD_GROUP when set (e.g., frontend → tbd-test/frontend).

Changelog Generation

For each promoted service that has a gitlabProjectId in its overlay:

  1. Call GitLab Compare API on the source project to get commits between old and new tags
  2. Classify commits using Conventional Commits rules
  3. Format into Keep a Changelog sections (Added, Changed, Fixed, Removed)

Info

Each service may have multiple releases between promotions (e.g. v0.3.1, v0.4.0, v0.5.0). Compare API covers the entire range v{old}..v{new} in one call.

The aggregated changelog is posted as the GitLab Release description:

Promotion: tbd-test [DEV] v1.5.0 → [TST] v1.3.0

# Services
| Service | App | Previous | New |
|---|---|---|---|
| frontend | tbd-test | 1.3.0 | 1.5.0 |
| backend | tbd-test-backend | 1.2.0 | 1.4.0 |

# Changelog

## tbd-test (1.3.0 → 1.5.0)
**Added**
- add user dashboard
- add health check endpoint
**Fixed**
- resolve login timeout

## tbd-test-backend (1.2.0 → 1.4.0)
**Added**
- add /api/info endpoint

Why Tied to DEV?

  • DEV is the first step out of development — promoting from DEV marks a version as "release candidate"
  • Subsequent promotions (TST → STG, STG → PRD) carry the same tag forward without re-tagging
  • Avoids duplicate releases for the same version

Deployment Tracking

The promotion pipeline registers deployments in GitLab for visibility.

ArgoCD Repository

Registers an environment deployment on the ArgoCD project itself, using the release tag commit SHA as the deployment reference.

Source Service Projects

For each promoted application with a gitlabProjectId, registers a deployment on the source project:

  • Resolves the commit SHA from the tag (v<version>) via GitLab Tags API
  • For hotfix versions (e.g. v0.23.0.185), strips to base tag (v0.23.0) for SHA resolution
  • Creates a deployment with the environment name and tier mapping
Target Environment GitLab Tier
DFT development
DEV development
TST testing
STG staging
PRD production

Dry Run Mode

Setting DRY_RUN=true (the default) in the pipeline variables runs the full promotion logic but:

  • Does not write the state file
  • Does not commit or push changes
  • Does not create tags or releases
  • Does not register deployments
  • Shows a git diff of what would change in workloads/
  • Pipeline name is prefixed with Preview instead of Promote
  • Useful for validating promotion before executing

Adding a New Workload Group

To onboard a new workload group for promotion:

  1. Create the directory structure: workloads/<group>/<service>/overlays/{dft,dev,tst,stg,prd}/values.yaml
  2. Each values.yaml must have applications[] with image.tag and gitlabProjectId

  3. Add the group name to the WORKLOAD_GROUP dropdown in argocd.test/.gitlab-ci.yml:

    WORKLOAD_GROUP:
  value: "tbd-test"
  options:
    - "tbd-test"
    - "my-new-group"    # ← add here

  4. If the group has a lead service for versioning, add it to VERSION_SERVICE options:

    VERSION_SERVICE:
  value: "frontend"
  options:
    - "frontend"
    - "my-lead-service"   # ← add here

Architecture Diagram

--- config: layout: elk --- flowchart TB A["GitLab Web UI"] -->|"Run Pipeline"| B["Pipeline variables<br/>WORKLOAD_GROUP = tbd-test<br/>SOURCE_ENV = DEV<br/>TARGET_ENV = TST"] B --> C["promote"] C --> D["Copy tags<br/>DEV → TST"] C --> E["Write state file<br/>workloads/promote-state.yaml"] C --> F["[DEV → TST] Create<br/>tag + release"] D --> G["Atomic commit + push"] E --> G F --> G G --> H["ArgoCD auto-sync<br/>(all services in group)"] H --> I["rollback<br/>(manual button, 7-day window)"] I --> J["Restore pre_state tags"] J --> K["Atomic commit + push"] K --> L["ArgoCD auto-sync<br/>(rollback)"]

End-to-End Flow: From Code to Production

Developer Workflow

1. Push to feature/* branch   → MR pipeline validates
2. Merge MR to main           → Main pipeline: build + tag v1.5.0 + deploy DFT
3. Tag pipeline               → Click "Deploy DEV" button
4. API pipeline               → Create release + deploy DEV
5. ArgoCD repo                → Run promotion: DEV → TST (release created)
6. ArgoCD repo                → Run promotion: TST → STG
7. ArgoCD repo                → Run promotion: STG → PRD
Rollback at any stage: click rollback button on the promotion pipeline.