Kubernetes Secrets: ESO vs CSI vs Init Containers

Introduction

It’s 3 AM and Vault is down. Your on-call engineer gets paged because deployments are failing — pods stuck in ContainerCreating, blocking a critical hotfix. Meanwhile, another team’s services keep humming along despite the same outage. The difference isn’t luck. It’s how secrets get into pods.

Kubernetes secrets have a fundamental problem: they’re base64-encoded, not encrypted. They sit in etcd alongside your cluster state, readable by anyone with RBAC access to the namespace. External secret managers like Vault, AWS Secrets Manager, and Azure Key Vault solve the security problem by keeping secrets outside the cluster — but they create a new problem. Your pods now depend on an external service, and that dependency has failure modes you need to understand before the 3 AM page.

Three patterns dominate secret injection:

Each pattern fails differently when the secret manager goes away. Consider a 30-minute Vault outage during a leadership election:

ESO‘s pods keep running with their last-synced secrets, and new pods start successfully because the Kubernetes Secret already exists — it’s just stale. CSI Driver pods that are already running continue fine, but any new pods can’t fetch their secrets and get stuck. Init containers? That depends entirely on what retry and fallback logic you built.

Native Kubernetes Secrets

Before diving into external secret managers, let’s establish what native Kubernetes secrets actually provide. A Secret is a Kubernetes object that stores sensitive data — passwords, tokens, keys — separate from pod specs and container images. Pods consume secrets either as environment variables or as files mounted from a volume:

apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
  namespace: production
type: Opaque
data:
  username: YWRtaW4=           # base64("admin")
  password: c3VwZXJzZWNyZXQ=   # base64("supersecret")
---
apiVersion: v1
kind: Pod
metadata:
  name: app-with-secrets
spec:
  containers:
    - name: app
      image: myapp:latest
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: database-credentials
              key: username
      volumeMounts:
        - name: secrets
          mountPath: /etc/secrets
          readOnly: true
  volumes:
    - name: secrets
      secret:
        secretName: database-credentials

When mounted as a volume, each key in the Secret becomes a separate file. In this example, the pod gets /etc/secrets/username and /etc/secrets/password—plain text files containing the decoded secret values. Your application reads these files directly.

The critical thing to understand: base64 is encoding, not encryption. Anyone with RBAC access to read secrets in a namespace can decode them trivially. By default, secrets are stored unencrypted in etcd — your cluster’s backing store. An etcd backup contains every secret in plaintext.

You can enable encryption at rest using an EncryptionConfiguration that tells the API server to encrypt secrets before writing to etcd:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          name: aws-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
          cachesize: 1000
          timeout: 3s
      - identity: {}  # Fallback for reading old unencrypted secrets

Even with encryption at rest, native secrets have operational limitations. There’s no rotation mechanism — you update secrets manually and redeploy pods. There’s no versioning — if you overwrite a secret, the old value is gone. And while Kubernetes audit logs can show who accessed the Secret object, they can’t tell you who accessed the actual secret value inside a running container. These gaps are why external secret managers exist.

External Secrets Operator (ESO)

External Secrets Operator is a Kubernetes operator that syncs secrets from external managers — Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager — into native Kubernetes secrets. The operator runs in your cluster, periodically fetches secrets from the external source, and creates or updates Kubernetes Secret objects that your pods consume normally.

ESO uses three custom resources. A ClusterSecretStore (or namespace-scoped SecretStore) defines how to connect to the external secret manager — endpoint, authentication method, default paths. An ExternalSecret declares which secrets to sync from that store and how to map them into a Kubernetes Secret. The operator watches ExternalSecret resources and reconciles them on a configurable interval.

CSI Architecture and Setup

A ClusterSecretStore for AWS Secrets Manager using IAM Roles for Service Accounts:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

For Vault, the store configures Kubernetes authentication:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

Azure Key Vault follows the same pattern, using workload identity for authentication:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: azure-keyvault
spec:
  provider:
    azurekv:
      tenantId: "your-tenant-id"
      vaultUrl: "https://your-vault.vault.azure.net"
      authType: WorkloadIdentity
      serviceAccountRef:
        name: external-secrets-sa
        namespace: external-secrets

ExternalSecret Configuration

With a ClusterSecretStore configured, you create ExternalSecret resources that specify which secrets to sync. The simplest form maps individual secret properties to Kubernetes Secret keys:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: production/database
        property: username
    - secretKey: password
      remoteRef:
        key: production/database
        property: password

The refreshInterval controls how often ESO checks the external source for changes. Shorter intervals mean fresher secrets but more API calls to your secret manager. For most workloads, 15-30 minutes balances freshness against API costs.

For secrets where you need to construct values from multiple sources — connection strings, for example — ESO supports templating:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-url
  namespace: production
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: database-url
    template:
      data:
        DATABASE_URL: |
          postgresql://{{ .username }}:{{ .password }}@{{ .host }}:5432/{{ .database }}
  data:
    - secretKey: username
      remoteRef:
        key: production/database
        property: username
    - secretKey: password
      remoteRef:
        key: production/database
        property: password
    - secretKey: host
      remoteRef:
        key: production/database
        property: host
    - secretKey: database
      remoteRef:
        key: production/database
        property: database

The key insight with ESO is the decoupling between secret fetching and pod lifecycle. The controller runs its reconciliation loop independently — watching ExternalSecrets, fetching from the external manager, updating Kubernetes Secrets. Pods consume those Secrets normally, unaware that they came from Vault or AWS. This decoupling is what gives ESO its graceful degradation: the Kubernetes Secret exists even when the external manager doesn’t respond.

ESO Failure Modes

ESO‘s failure modes are relatively benign because it decouples secret fetching from pod lifecycle. When the secret manager goes down, the ESO controller logs errors and keeps retrying — but the Kubernetes Secret it already created remains unchanged. Existing pods keep running with their last-synced values. Crucially, new pods can also start: they mount the Kubernetes Secret normally, unaware that ESO is failing to sync. The Kubernetes Secret itself is the cache — it persists in etcd independent of the external manager’s availability.

The downside: silent staleness. If you rotate a database password in Vault but ESO can’t sync for two hours, your pods run with the old password. They work fine until something restarts them with the now-invalid cached secret. This is why monitoring sync status matters:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: eso-alerts
spec:
  groups:
    - name: eso.rules
      rules:
        - alert: ExternalSecretSyncFailed
          expr: |
            externalsecret_status_condition{condition="Ready", status="False"} == 1
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "ExternalSecret {{ $labels.name }} sync failing"
        - alert: ExternalSecretStale
          expr: |
            time() - externalsecret_reconcile_timestamp_seconds > 7200
          labels:
            severity: warning
          annotations:
            summary: "ExternalSecret {{ $labels.name }} not synced in 2+ hours"

You can check sync status directly with kubectl:

kubectl get externalsecret -A
# Look for Ready condition in STATUS column

Secrets Store CSI Driver

The Secrets Store CSI Driver takes a fundamentally different approach than ESO. Instead of syncing secrets to Kubernetes Secret objects, it mounts secrets directly into pods as volumes. When a pod starts, the CSI driver fetches secrets from the external manager and presents them as files in the container’s filesystem — no Kubernetes Secret involved (unless you explicitly configure one).

The driver runs as a DaemonSet on every node, paired with a provider plugin for your secret manager (Vault, AWS, Azure, GCP). A SecretProviderClass resource defines which secrets to fetch and how to present them.

Architecture and Setup

A SecretProviderClass for Vault specifies the Vault address, authentication role, and which secrets to mount:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-secrets
  namespace: production
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.example.com"
    roleName: "production-app"
    objects: |
      - objectName: "database-username"
        secretPath: "secret/data/production/database"
        secretKey: "username"
      - objectName: "database-password"
        secretPath: "secret/data/production/database"
        secretKey: "password"

For AWS Secrets Manager, the configuration uses IRSA for authentication:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
  namespace: production
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "production/database"
        objectType: "secretsmanager"
        jmesPath:
          - path: username
            objectAlias: db-username
          - path: password
            objectAlias: db-password

Pod Configuration

Pods reference the SecretProviderClass through a CSI volume. The driver authenticates using the pod’s service account, fetches the secrets, and mounts them as files:

apiVersion: v1
kind: Pod
metadata:
  name: app-with-csi-secrets
  namespace: production
spec:
  serviceAccountName: production-app
  containers:
    - name: app
      image: myapp:latest
      volumeMounts:
        - name: secrets
          mountPath: /mnt/secrets
          readOnly: true
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: vault-secrets

The pod cannot start until the volume mounts successfully — which means until secrets are fetched from Vault. This is the key difference from ESO: the dependency is synchronous.

The CSI driver optionally supports rotation. Enable it by setting --enable-secret-rotation=true on the driver DaemonSet and adding rotationPollInterval to your SecretProviderClass. When enabled, the driver periodically re-fetches secrets and updates the mounted files. Your application must detect file changes (using inotify or periodic re-reads) to pick up rotated values — the driver updates files, but it can’t restart your process.

The flow differs fundamentally from ESO: secrets are fetched synchronously during pod startup, not asynchronously by a controller.

CSI Failure Modes

The CSI driver’s failure mode is the opposite of ESO‘s: loud and blocking. If the secret manager is unavailable when a pod starts, the volume mount fails. The pod stays in ContainerCreating with events showing MountVolume.SetUp failed. No graceful degradation — no start.

This has cascading implications. A brief Vault outage during a deployment blocks the entire rollout. A node reboot during an incident restarts pods that can’t fetch their secrets. Horizontal pod autoscaler scales up pods that immediately get stuck.

Diagnosing CSI failures:

# Check pod events for mount errors
kubectl describe pod <pod-name>

# Check CSI driver pods are healthy
kubectl get pods -n kube-system -l app=secrets-store-csi-driver

# Check provider logs for auth or connectivity issues
kubectl logs -n kube-system -l app=secrets-store-csi-driver-provider-vault

You can optionally configure the SecretProviderClass to also sync secrets to a Kubernetes Secret (via secretObjects), giving you a fallback if needed — but at that point you’re combining the complexity of CSI with the staleness risk of cached secrets.

Init Container Pattern

The init container pattern is the DIY approach: you write shell scripts that fetch secrets before your main application starts. No operators, no CRDs — just an init container that runs the Vault CLI or AWS CLI, writes secrets to a shared volume, then exits. The main container mounts that volume and reads the files.

This gives you complete control over failure handling. You decide the retry count, backoff strategy, fallback sources, and logging. The cost is maintaining that logic yourself.

Basic Implementation

The pattern uses an emptyDir volume with medium: Memory (tmpfs) shared between the init container and main container. The init container authenticates to Vault using the pod’s service account, fetches secrets, and writes them as files:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-with-init-secrets
  namespace: production
spec:
  template:
    spec:
      serviceAccountName: production-app
      volumes:
        - name: secrets
          emptyDir:
            medium: Memory
      initContainers:
        - name: fetch-secrets
          image: vault:latest
          env:
            - name: VAULT_ADDR
              value: "https://vault.example.com"
          command:
            - /bin/sh
            - -c
            - |
              set -e
              VAULT_TOKEN=$(vault write -field=token auth/kubernetes/login \
                role=production-app \
                jwt=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token))
              export VAULT_TOKEN

              MAX_RETRIES=5
              RETRY_DELAY=5
              for i in $(seq 1 $MAX_RETRIES); do
                if vault kv get -format=json secret/production/database > /secrets/database.json; then
                  jq -r '.data.data.username' /secrets/database.json > /secrets/db-username
                  jq -r '.data.data.password' /secrets/database.json > /secrets/db-password
                  rm /secrets/database.json
                  exit 0
                fi
                echo "Attempt $i failed, retrying in ${RETRY_DELAY}s..."
                sleep $RETRY_DELAY
              done
              echo "Failed to fetch secrets after $MAX_RETRIES attempts"
              exit 1
          volumeMounts:
            - name: secrets
              mountPath: /secrets
      containers:
        - name: app
          image: myapp:latest
          volumeMounts:
            - name: secrets
              mountPath: /etc/secrets
              readOnly: true

The main container reads /etc/secrets/db-username and /etc/secrets/db-password as plain text files. Using medium: Memory keeps secrets in tmpfs rather than writing them to disk — important for compliance.

Fallback Pattern

For resilience during secret manager outages, you can combine the init container with a fallback Kubernetes Secret. Try the external source first; fall back to cached credentials if unavailable:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-with-fallback
spec:
  template:
    spec:
      volumes:
        - name: secrets
          emptyDir:
            medium: Memory
        - name: fallback-secrets
          secret:
            secretName: database-credentials-fallback
            optional: true
      initContainers:
        - name: fetch-secrets
          image: amazon/aws-cli:latest
          command:
            - /bin/sh
            - -c
            - |
              set -e
              if aws secretsmanager get-secret-value \
                --secret-id production/database \
                --query SecretString \
                --output text > /secrets/credentials.json 2>/dev/null; then
                jq -r '.username' /secrets/credentials.json > /secrets/db-username
                jq -r '.password' /secrets/credentials.json > /secrets/db-password
                rm /secrets/credentials.json
              elif [ -f /fallback/username ]; then
                echo "Using fallback secrets"
                cp /fallback/username /secrets/db-username
                cp /fallback/password /secrets/db-password
              else
                echo "No secrets available!"
                exit 1
              fi
          volumeMounts:
            - name: secrets
              mountPath: /secrets
            - name: fallback-secrets
              mountPath: /fallback
              readOnly: true
      containers:
        - name: app
          image: myapp:latest
          volumeMounts:
            - name: secrets
              mountPath: /etc/secrets
              readOnly: true

The fallback Secret could be synced by ESO or maintained manually. This gives you CSI-like freshness with ESO-like resilience — at the cost of maintaining the logic yourself.

Sidecar Refresh Pattern

Init containers only run at pod startup. For long-running pods that need rotated secrets, add a sidecar container that periodically refreshes the files:

containers:
  - name: app
    image: myapp:latest
    volumeMounts:
      - name: secrets
        mountPath: /etc/secrets
        readOnly: true
  - name: secret-refresher
    image: vault:latest
    command:
      - /bin/sh
      - -c
      - |
        while true; do
          sleep 300
          VAULT_TOKEN=$(vault write -field=token auth/kubernetes/login \
            role=production-app \
            jwt=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token))
          export VAULT_TOKEN
          vault kv get -field=username secret/production/database > /secrets/db-username.new
          vault kv get -field=password secret/production/database > /secrets/db-password.new
          mv /secrets/db-username.new /secrets/db-username
          mv /secrets/db-password.new /secrets/db-password
        done
    volumeMounts:
      - name: secrets
        mountPath: /secrets

The atomic mv ensures your application sees either the old or new file, never a partial write. Your application still needs to detect file changes (inotify or periodic re-read) to pick up rotated values.

Comparison and Selection

With three patterns to choose from, the decision comes down to two questions: how fresh do your secrets need to be, and what failure mode can your application tolerate?

The table captures raw capabilities, but choosing a pattern requires weighting those capabilities against your operational reality. A team with strong GitOps practices and tolerance for minutes of staleness will land in a different place than a team running payment infrastructure where a stale database credential means failed transactions. The deciding factor is usually the failure mode: ESO fails silently with stale data, CSI fails loudly by blocking pod startup, and init containers fail however you code them to. Each is the right answer for different risk profiles.

Recommendations by Use Case

Production Considerations

Whichever pattern you choose, getting secrets into pods is only half the problem. Keeping them secure, monitoring for failures, and responding to incidents requires additional configuration.

Security Hardening

Whichever pattern you choose, the security fundamentals remain the same: authenticate with short-lived credentials, scope access narrowly, and restrict network paths.

A hardened ESO setup uses internal-only endpoints, mTLS, and scoped Vault roles:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-hardened
spec:
  provider:
    vault:
      server: "https://vault.internal.example.com"
      path: "secret"
      version: "v2"
      namespace: "production"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets-production"
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets
      caProvider:
        type: Secret
        name: vault-ca
        namespace: external-secrets
        key: ca.crt

Network policies restrict egress from the ESO namespace to only the Vault endpoint:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-vault-access
  namespace: external-secrets
spec:
  podSelector:
    matchLabels:
      app: external-secrets
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 10.0.0.0/8
      ports:
        - protocol: TCP
          port: 8200

Monitoring and Alerting

Monitor three things: the injection mechanism (ESO sync status, CSI mount success), the secret manager (Vault health, latency), and the end-to-end path (for example, whether a test secret can actually be fetched).

For ESO, the key metric is sync status. An ExternalSecret that hasn’t synced in multiple refresh intervals indicates a problem:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: secret-injection-alerts
spec:
  groups:
    - name: secrets.rules
      rules:
        - alert: ExternalSecretSyncFailed
          expr: externalsecret_status_condition{condition="Ready",status="False"} == 1
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "ExternalSecret {{ $labels.name }} not syncing"
        - alert: SecretCSIMountFailed
          expr: |
            increase(kubelet_csi_operations_seconds_count{
              driver_name="secrets-store.csi.k8s.io",
              method_name="NodePublishVolume",
              grpc_status_code!="OK"
            }[5m]) > 0
          labels:
            severity: critical
          annotations:
            summary: "CSI secret mount failures on {{ $labels.node }}"
        - alert: VaultUnreachable
          expr: vault_up == 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Vault unreachable from cluster"

Conclusion

Secret injection is ultimately about failure modes. ESO fails silently — your pods keep running with stale secrets until you notice sync errors in monitoring. CSI driver fails loudly — pods don’t start, deployments block, and you know immediately something is wrong. Init containers fail however you code them.

For most organizations, ESO is the right default. It’s operationally simple, GitOps-friendly, and its failure mode (staleness) is tolerable for the vast majority of workloads. Reserve CSI driver for applications with strict compliance requirements or real-time credential needs. Use init containers only when you need behavior that neither operator provides.

The 3 AM Vault outage that opened this article? The team running ESO stayed asleep — their pods kept serving traffic with cached secrets. The team running CSI driver got paged when deployments failed. Both outcomes are defensible, depending on what you’re building. The mistake is not understanding which failure mode you’ve chosen.