Shift-Left Security: Container Scanning in CI

The appeal of shift-left security is obvious: finding vulnerabilities early in development costs less to fix than finding them in production. Container scanning belongs in your CI pipeline, not as an afterthought in staging.

The practical challenge is equally obvious. Teams enable container scanning and immediately face 47 critical vulnerabilities. Half are in base image packages the application never uses. A quarter have no available fix. The rest require major version bumps that would take weeks to properly test. The team disables scanning “temporarily” and never re-enables it.

This pattern repeats constantly. The scanner becomes noise that developers ignore or bypass. Security metrics look terrible because nobody’s actually scanning. The tool that was supposed to improve security has made it worse by creating learned helplessness.

This article covers how to integrate container scanning that developers will actually use: scanner selection, policy configuration that balances security and velocity, exception workflows for the inevitable unfixable CVEs, and metrics that tell you whether your security posture is actually improving.

Understanding Container Vulnerabilities

Where Vulnerabilities Hide

Container images are layered, and vulnerabilities can exist at every layer. Understanding where they come from helps you prioritize what to fix and what to accept.

Here’s the uncomfortable reality for a typical Node.js application: about 60% of vulnerabilities are in layers developers don’t directly control.

Your application code — the part you write and fully control — accounts for roughly 5% of CVEs. The npm packages you directly depend on add another 35%, but many of those are transitive dependencies you’ve never heard of. The remaining 60% comes from the OS packages and base image, which most developers treat as a black box.

This distribution explains why “just fix all the CVEs” is impractical. You can’t fix a vulnerability in Debian’s libssl by editing your code. You need to update your base image, hope the distro has patched it, or accept the risk.

CVE Scoring and Severity

CVSS scores tell you how bad a vulnerability could be in the worst case. They don’t tell you how bad it is for you.

The problem is context. A CVSS 9.8 critical in an XML parser doesn’t matter if your application only parses JSON. A CVSS 6.5 medium in your authentication library matters a lot if you’re running a public-facing API.

Some CVEs have been rated critical for years with no known exploits. Others get weaponized within days of disclosure. The CVSS score alone doesn’t capture this — you need to check whether there’s an active exploit in the wild (CISA’s Known Exploited Vulnerabilities catalog is useful here) and whether the vulnerability applies to your deployment context.

Types of Findings

Scanners detect different categories of issues, and each requires a different response.

Scanner Selection

Popular Scanner Comparison

The container scanning landscape has consolidated around a few tools. Here’s how they compare:

On the commercial side, Snyk offers excellent developer experience with fix PRs, prioritization, and IDE integration. Aqua (the company behind Trivy) offers runtime protection, policy engines, and enterprise features. Both come with enterprise pricing.

Running Trivy in CI

Trivy’s GitHub Action makes integration straightforward. Here’s a pattern that works well: run the scan twice. The first pass reports all findings to GitHub’s Security tab without failing the build. The second pass is the actual gate, configured to fail only on fixable critical vulnerabilities. For GitLab CI, the pattern is similar.

name: Security Scan

on:
  push:
    branches: [main]
  pull_request:

jobs:
  trivy-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'myapp:${{ github.sha }}'
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'
          exit-code: '0'  # Report only, don't fail

      - name: Upload Trivy scan results
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: 'trivy-results.sarif'

      - name: Trivy gate check
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'myapp:${{ github.sha }}'
          severity: 'CRITICAL'
          ignore-unfixed: true
          exit-code: '1'  # Fail on fixable criticals

stages:
  - build
  - security
  - deploy

build:
  stage: build
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

container_scan:
  stage: security
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    # Report all findings
    - trivy image
        --exit-code 0
        --severity CRITICAL,HIGH
        --format json
        --output trivy-report.json
        $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    # Gate on fixable criticals only
    - trivy image
        --exit-code 1
        --severity CRITICAL
        --ignore-unfixed
        $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  artifacts:
    reports:
      container_scanning: trivy-report.json
    paths:
      - trivy-report.json

The ignore-unfixed: true (GH Action) option and --ignore-unfixed flag (GitLab CI) are crucial. Without it, you’ll fail builds on CVEs that have no available patch — frustrating developers with problems they can’t solve.

Policy Configuration

Defining Security Gates

The scanner itself is just a tool. The policy — what you do with the findings — determines whether scanning improves security or becomes noise.

Trivy supports a configuration file that centralizes policy decisions:

severity:
  - CRITICAL
  - HIGH

# Don't fail on unfixable vulnerabilities
ignore-unfixed: true

# Skip directories that don't ship to production
skip-dirs:
  - test/
  - docs/

# Custom ignore file for acknowledged risks
ignorefile: .trivyignore

timeout: 10m
format: table
exit-code: 1

The ignore file is where you document exceptions. Every entry should explain why it’s ignored:

  # CVE in test dependency, not shipped to production
  CVE-2023-12345

  # Disputed CVE, vendor confirms not exploitable in our context
  CVE-2023-67890

  # No fix available, risk accepted until Q2 2024
  # Ticket: SEC-123
  # Expires: 2024-06-30
  CVE-2023-11111

  # False positive - we don't use the affected function
  CVE-2023-22222

Tiered Severity Gates

Not every pipeline stage needs the same policy. Development branches should be permissive — you don’t want to block a developer from iterating on a feature because of a CVE in the base image. Production deployments should be strict.

Here’s a tiered approach that balances security and velocity:

The key insight is that you’re not lowering security by being permissive on dev branches — you’re increasing adoption. A scanner that developers trust and use beats a strict scanner that gets disabled.

Exception Workflows

Some vulnerabilities can’t be fixed immediately. Maybe there’s no patch. Maybe the fix requires a breaking change that needs a full release cycle. You need a process for handling these legitimately rather than pretending they don’t exist.

The workflow should have four steps:

The exception reasons I see most often:

SBOM Generation

What Is an SBOM

A Software Bill of Materials is an ingredients list for your software. It answers: what packages are included, what versions, what licenses, and where did they come from?

Three scenarios make SBOMs worth the effort:

Generating SBOMs with Trivy

Trivy can generate SBOMs in multiple formats:

# SPDX format (common for license compliance)
trivy image --format spdx-json --output sbom.spdx.json myapp:latest

# CycloneDX format (common for security)
trivy image --format cyclonedx --output sbom.cdx.json myapp:latest

# Generate from filesystem without building image
trivy fs --format cyclonedx --output sbom.cdx.json .

# Scan an existing SBOM for vulnerabilities
trivy sbom sbom.cdx.json

The last command is particularly useful. Instead of scanning images repeatedly, you can scan the SBOM — which is much faster since it’s just a JSON file. When a new CVE drops, scan all your SBOMs to find affected services in seconds.

For CI integration, generate the SBOM alongside your image and attach it as an attestation using cosign:

sbom:
  stage: security
  script:
    - trivy image
        --format cyclonedx
        --output sbom-${CI_COMMIT_SHA}.json
        $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

    # Attach SBOM to container image as attestation
    - cosign attest
        --predicate sbom-${CI_COMMIT_SHA}.json
        --type cyclonedx
        $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

  artifacts:
    paths:
      - sbom-${CI_COMMIT_SHA}.json

For most teams, CycloneDX is the right choice. It’s designed with security in mind and supports VEX (Vulnerability Exploitability eXchange) documents for communicating which vulnerabilities actually affect your deployment. VEX lets you publish statements like “CVE-2023-12345 does not affect us because we don’t use the vulnerable XML parsing feature”—turning your .trivyignore decisions into a machine-readable format that downstream consumers can use.

Base Image Strategy

Choosing Secure Base Images

The single biggest factor in your CVE count is your base image choice. A full Ubuntu image ships with hundreds of packages you’ll never use — and each one is a potential vulnerability.

# ❌ BAD: Full OS image - hundreds of packages
FROM ubuntu:22.04
# CVEs: ~50-100 typically
# Size: ~77MB compressed

# ⚠️ BETTER: Slim variant - fewer packages
FROM python:3.11-slim
# CVEs: ~20-50 typically
# Size: ~45MB compressed

# ✅ GOOD: Alpine - minimal packages
FROM python:3.11-alpine
# CVEs: ~5-15 typically
# Size: ~17MB compressed
# Caveat: musl libc compatibility issues

# ✅ BEST: Distroless - no shell, minimal
FROM gcr.io/distroless/python3
# CVEs: ~0-5 typically
# Size: ~15MB compressed
# Caveat: No shell for debugging

# ✅ ALTERNATIVE: Chainguard images
FROM cgr.dev/chainguard/python:latest
# CVEs: Usually 0
# Size: ~25MB compressed
# Caveat: Free for :latest only; pinned versions require subscription

Multi-Stage Builds for Security

Multi-stage builds let you use a full toolchain for building while shipping a minimal image to production. The build stage has compilers, package managers, and dev dependencies. The production stage has only what’s needed to run.

# Stage 1: Build with full toolchain
FROM node:20 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Stage 2: Production with minimal image
FROM gcr.io/distroless/nodejs20-debian12
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./

CMD ["dist/index.js"]

The vulnerability reduction is dramatic. A single-stage build using node:20 might have 60 CVEs: 45 in OS packages, 3 in Node.js, and 12 in dev dependencies. The multi-stage build using distroless drops to 5 CVEs: 2 in the minimal OS layer, 3 in Node.js, and zero in dev dependencies (because they’re not included). That’s a 92% reduction from a Dockerfile change.

The tradeoff is debuggability. With no shell in the production image, you can’t docker exec into a running container to poke around. For most production workloads, that’s actually a feature — if you need to debug, you should be looking at logs and metrics, not SSH’ing into containers.

Runtime vs Build-Time Scanning

Scanning Strategy

Container scanning can happen at multiple points in the lifecycle. Each stage catches different problems and has different tradeoffs.

Continuous Registry Scanning

New CVEs are published daily. An image that was clean yesterday might have critical vulnerabilities today. Continuous registry scanning catches these without requiring rebuilds.

Most container registries support this natively. Here’s how to configure ECR enhanced scanning with Terraform:

# Enable enhanced scanning with Amazon Inspector
resource "aws_ecr_registry_scanning_configuration" "main" {
  scan_type = "ENHANCED"

  rule {
    scan_frequency = "CONTINUOUS_SCAN"
    repository_filter {
      filter      = "*"
      filter_type = "WILDCARD"
    }
  }
}

# Alert on new findings via EventBridge
resource "aws_cloudwatch_event_rule" "inspector_findings" {
  name        = "inspector-vulnerability-findings"
  description = "Capture Inspector2 vulnerability findings"

  event_pattern = jsonencode({
    source      = ["aws.inspector2"]
    detail-type = ["Inspector2 Finding"]
  })
}

resource "aws_cloudwatch_event_target" "sns" {
  rule      = aws_cloudwatch_event_rule.inspector_findings.name
  target_id = "security-alerts"
  arn       = aws_sns_topic.security_alerts.arn
}

For Harbor, you can configure Trivy scanning and pull prevention policies through the Harbor Helm chart or API. The key capability is blocking pulls of vulnerable images — if someone tries to deploy an image with critical CVEs, the registry refuses to serve it. This hard stop prevents vulnerable code from reaching production even if CI gates were bypassed.

Handling Common Scenarios

The “Can’t Fix” Situation

Not every CVE can be fixed immediately. Here are the scenarios you’ll encounter and how to handle them:

For transitive dependencies, package managers offer override mechanisms:

{
  "name": "myapp",
  "overrides": {
    "nth-check": "2.1.1",
    "semver": "7.5.4"
  }
}

module myapp

go 1.21

require (
    github.com/example/lib v1.2.3
)

// Force specific version of transitive dependency
replace golang.org/x/crypto => golang.org/x/crypto v0.17.0
replace golang.org/x/net => golang.org/x/net v0.19.0

# For pip: pin vulnerable transitive dependencies explicitly
# These take precedence over what other packages request
cryptography==41.0.7
certifi==2023.11.17
urllib3==2.1.0

# Or use pip-compile with constraints file:
# pip-compile --constraint constraints.txt requirements.in

source "https://rubygems.org"

gem "rails", "~> 7.0"

# Force specific versions of transitive dependencies
gem "nokogiri", ">= 1.15.4"  # CVE-2023-XXXXX
gem "puma", ">= 6.4.0"       # Security fix

Each ecosystem handles this differently. npm uses overrides (yarn uses resolutions), Go uses replace directives, pip relies on explicit pinning or constraints files, and Ruby lets you declare gems directly even if they’re transitive. Use with caution — you’re overriding what the package author tested against.

With common scenarios handled, the remaining challenge is proving your security posture is actually improving.

Metrics and Reporting

Security Posture Metrics

Tracking the right metrics tells you whether your security posture is actually improving or just generating noise. Here are the categories that matter:

Reporting to Stakeholders

Different audiences need different views of the same data:

Integration Patterns

GitHub Security Tab Integration

GitHub’s Security tab provides a centralized view of vulnerabilities across your repositories. Trivy can upload findings in SARIF format, which GitHub parses and displays alongside Dependabot alerts and CodeQL results.

# Upload findings to GitHub Security tab
- name: Upload to GitHub Security
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: 'trivy-results.sarif'

Once uploaded, findings appear in the Security tab under Code scanning alerts. Developers see them in PR checks with inline annotations. The interface is similar to Dependabot — familiar to anyone who’s dealt with dependency updates.

Slack Notifications

For teams that live in Slack, immediate notification on scan failures keeps security visible without requiring developers to check dashboards:

- name: Notify Slack on Critical CVEs
  if: failure()
  uses: slackapi/slack-github-action@v1
  with:
    payload: |
      {
        "blocks": [
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "🚨 *Security Scan Failed*\n*Repository:* ${{ github.repository }}\n*Branch:* ${{ github.ref_name }}"
            }
          },
          {
            "type": "actions",
            "elements": [
              {
                "type": "button",
                "text": {"type": "plain_text", "text": "View Results"},
                "url": "${{ github.server_url }}/${{ github.repository }}/security/code-scanning"
              }
            ]
          }
        ]
      }
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_SECURITY_WEBHOOK }}

The if: failure() condition means this only fires when the security gate fails — not on every scan. Nobody wants notification fatigue for successful builds.

Conclusion

Shift-left security catches vulnerabilities early when they’re cheapest to fix — in the developer’s IDE or PR, not in production after a breach. But the goal isn’t zero findings; it’s actionable findings that developers can actually resolve.

The patterns that make container scanning successful:

Security gates must balance protection with velocity. Gates that block everything get disabled. Gates that developers trust and use — even if they’re more permissive — deliver better security outcomes than strict gates that nobody runs.