CI/CD Pipelines

Docker solves the "works on my machine" problem by packaging your application into a portable container. Kubernetes solves the "how do I run it reliably at scale" problem. But there is still a gap: how does code actually travel from a developer's laptop into that cluster — safely, automatically, and repeatably?

That gap is what CI/CD pipelines fill. CI/CD (Continuous Integration / Continuous Delivery — and in its most automated form, Continuous Deployment) is the automated system that takes every code change and carries it through building, testing, security scanning, and deployment without human intervention. It is the difference between "we'll deploy next Tuesday" and "every merged pull request is live within 15 minutes."

This matters especially when working with AI coding tools. When an AI agent generates dozens of small changes per session, a manual deployment process becomes a bottleneck. CI/CD pipelines are the infrastructure that makes AI-assisted development scalable — and safe.

The Three Practices#

CI/CD is a single acronym but it covers three distinct practices that build on each other:

Most teams practice Continuous Delivery, not full Continuous Deployment. Having a human approval gate before production is pragmatic, not a failure — it lets teams coordinate deployment timing, communicate upcoming changes to stakeholders, and align with business schedules without slowing the pipeline itself.

A common misconception about CI: Long-lived feature branches where developers work in isolation for days are not Continuous Integration — they are delayed integration. The longer branches diverge from each other, the larger and more painful the eventual merge conflict. Real CI means integrating to the main branch at least daily.

The Pipeline Architecture#

A production CI/CD pipeline is a chain of automated stages. A failure at any stage stops the pipeline and notifies the team — bad code cannot advance to the next stage.

Pipeline Stages in Detail#

GitHub Actions: Building a Pipeline#

GitHub Actions is the most widely adopted CI/CD platform for teams on GitHub. Pipelines are defined as YAML files inside .github/workflows/ and version-controlled alongside your code — which means pipeline changes go through the same pull request and code review process as your application code. Here is an annotated, production-realistic workflow:

name: CI/CD Pipeline

# ── Triggers ──────────────────────────────────────────────────
on:
  push:
    branches: [main]       # Full pipeline (CI + deploy) on merges to main
  pull_request:
    branches: [main]       # CI only (no deploy) on pull requests

# Prevent duplicate runs: cancel older runs when a newer commit is pushed
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  # ── Job 1: Lint and Test (runs on every push and PR) ──────────
  ci:
    name: Lint & Test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'           # Cache node_modules between runs

      - name: Install dependencies
        run: npm ci              # Reproducible install from lockfile (not npm install)

      - name: Lint
        run: npm run lint

      - name: Run tests
        run: npm test

  # ── Job 2: Build and Push Docker Image ───────────────────────
  build:
    name: Build & Push Image
    needs: ci                    # Only runs if the ci job passes
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'   # Skip this job on pull requests
    permissions:
      contents: read
      packages: write            # Required to push to GitHub Container Registry
    outputs:
      image-tag: ${{ steps.tag.outputs.value }}   # Pass the image tag to deploy jobs

    steps:
      - uses: actions/checkout@v4

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}   # Auto-provisioned — no manual setup

      - name: Compute image tag
        id: tag
        # Tag with commit SHA for traceability: e.g., ghcr.io/myorg/myapp:sha-a3f82c1
        run: echo "value=ghcr.io/${{ github.repository }}:sha-${{ github.sha }}" >> $GITHUB_OUTPUT

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.tag.outputs.value }}
          cache-from: type=gha          # Reuse Docker layer cache from previous runs
          cache-to: type=gha,mode=max

  # ── Job 3: Deploy to Staging (automatic) ─────────────────────
  deploy-staging:
    name: Deploy to Staging
    needs: build
    runs-on: ubuntu-latest
    environment: staging           # Links to the "staging" GitHub environment
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to staging cluster
        run: ./scripts/deploy.sh staging
        env:
          IMAGE_TAG: ${{ needs.build.outputs.image-tag }}
          KUBE_CONFIG: ${{ secrets.STAGING_KUBE_CONFIG }}

  # ── Job 4: Deploy to Production (requires manual approval) ───
  deploy-production:
    name: Deploy to Production
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production        # Pauses here until a required reviewer approves
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to production cluster
        run: ./scripts/deploy.sh production
        env:
          IMAGE_TAG: ${{ needs.build.outputs.image-tag }}
          KUBE_CONFIG: ${{ secrets.PROD_KUBE_CONFIG }}

Key Concepts Explained#

needs — Creates a sequential dependency. build only runs if ci passes. deploy-staging only runs if build succeeds. Without needs, all jobs run in parallel simultaneously.

environment — Links a job to a named environment configured in your GitHub repository's Settings → Environments page. You configure protection rules there (required reviewers, wait timers, branch restrictions). When the job runs, it pauses until those rules are satisfied before proceeding. Environment-scoped secrets (STAGING_KUBE_CONFIG, PROD_KUBE_CONFIG) are only injected after protection rules pass — a feature-branch pipeline cannot access production credentials.

concurrency with cancel-in-progress: true — Without this, pushing three commits in quick succession launches three pipelines that all try to deploy simultaneously. This setting cancels any older in-progress run on the same branch when a newer one starts. Only the latest commit matters.

${{ secrets.GITHUB_TOKEN }} — A short-lived token automatically provisioned by GitHub for each workflow run. It has the right permissions to push to the repository's container registry without any manual secret configuration. This is why pushing to GHCR from CI requires no separate credential setup.

cache-from: type=gha — CI runners start from a completely clean environment on every run. Without layer caching, every build re-downloads the base image and reinstalls all dependencies from scratch. GitHub Actions cache stores the Docker layer cache between runs, cutting a typical warm build from several minutes to under 30 seconds.

npm ci vs. npm install — npm ci installs from the lockfile exactly, fails if the lockfile is missing or inconsistent with package.json, and never modifies package.json. It is the correct command for CI because it guarantees reproducible installs.

CI/CD Platforms at a Glance#

Practical guidance: For most teams starting today, GitHub Actions is the right choice if your code is on GitHub. It requires zero infrastructure to operate and covers the full CI/CD workflow out of the box. GitLab CI is the right choice if you want security scanning, container registry, and CD all in the same platform without stitching together tools. Jenkins is worth considering only if you have requirements cloud platforms cannot meet — most teams that inherit a Jenkins setup eventually migrate away.

Environment Promotion: Build Once, Promote Everywhere#

One of the most important CI/CD principles: build the Docker image once and promote the same immutable artifact through every environment — never rebuild per environment.

Why "same image" matters: If you rebuild the Docker image separately for staging and production — even from the same Git commit — minor differences can cause the two images to diverge. A base image tag like node:20 may have received a new patch version between the two builds; an unpinned package may have resolved to a different version; the build cache may have been warm for one and cold for the other. You end up testing one artifact and shipping a slightly different one. This is a common source of "it worked in staging" production incidents.

How environment-specific config works: The image tag sha-a3f82c1 is identical in every environment. What differs is the configuration injected at runtime: database URLs, API keys, resource limits, and feature flags. These are injected as environment variables at deploy time — never baked into the image. This aligns directly with the Docker best practice of never embedding secrets in image layers.

Environment Protection Rules#

Modern CI/CD platforms let you attach protection rules to environments that control when a deployment job can proceed:

Push-Based vs. Pull-Based CD (GitOps)#

Traditional CD pipelines use a push-based model: when a build finishes, the CI system runs kubectl apply to update the cluster. This works, but it means the CI system must hold production cluster credentials — a security surface that can be exploited if the CI system is compromised.

GitOps is an alternative where the CI pipeline does not talk to the cluster at all. Instead, it updates a Git repository containing the desired cluster state (Kubernetes manifests or Helm values). An operator running inside the cluster continuously polls the Git repository and applies changes it detects. Git becomes the single source of truth.

Deployment Strategies#

How you ship a new version matters as much as what you ship. A naive deployment — stop the old version, start the new one — creates downtime and gives you no way to catch problems before all users are affected. The three standard strategies each offer a different balance of complexity, cost, and risk.

CI/CD Anti-Patterns#

Common mistakes that undermine pipeline reliability and trust:

What AI Agents Get Wrong with CI/CD#

Connecting Docker, Kubernetes, and CI/CD#

The three sections of this chapter are designed to work as a complete delivery system. Each layer answers a different question.

Docker — How do I package my application and its dependencies into a portable, reproducible unit?

CI/CD — How do I take that unit from a commit through automated testing, security scanning, and into a registry — and then safely into production?

Kubernetes — How do I run that unit reliably at scale, with self-healing, autoscaling, and zero-downtime updates?

Each layer depends on the previous one. Containers without CI/CD require manual deploys — you have solved the packaging problem but not the delivery problem. CI/CD without containers produces environment-inconsistent artifacts. And Kubernetes without CI/CD means manually building images and running kubectl apply by hand for every change.

Together, they form the foundation of modern production engineering — and the infrastructure that makes AI-assisted development at scale both safe and sustainable.

Summary#

Sources:

• Continuous Integration — Martin Fowler
• Blue Green Deployments — Martin Fowler
• What is CI/CD? — Atlassian
• GitHub Actions Workflow Syntax — GitHub Docs
• Using Environments for Deployment — GitHub Docs
• Kubernetes Deployments — kubernetes.io
• Introduction to GitOps — gitops.tech
• ArgoCD Getting Started — argo-cd.readthedocs.io
• Argo Rollouts — argoproj.github.io
• CD4ML: Continuous Delivery for Machine Learning — Martin Fowler