Kubernetes Secrets Management: Best Practices and Tools Compared

Kubernetes Secrets are base64-encoded, not encrypted. Anyone with RBAC access to read Secrets in a namespace can decode every credential stored there in seconds. And if etcd is not encrypted at rest, your secrets are stored in plaintext on disk. This is the starting point for every Kubernetes cluster, and it is fundamentally insecure for production workloads.

This guide compares the four most popular approaches to solving Kubernetes secrets management: Sealed Secrets, External Secrets Operator (ESO), HashiCorp Vault, and SOPS. We cover the architecture, installation, tradeoffs, and practical examples for each.

The Problem with Native Kubernetes Secrets

Before diving into solutions, let us understand exactly what is wrong with the default approach.

Base64 Is Not Encryption

# Create a secret
kubectl create secret generic db-creds \
  --from-literal=password='s3cur3-p@ssw0rd'

# Read it back -- base64 decode reveals the value
kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d
# Output: s3cur3-p@ssw0rd

Any user, service account, or CI/CD pipeline with get secrets RBAC permission in the namespace can read every secret. There is no fine-grained access control at the individual secret level.

Secrets in Git

The most common anti-pattern is committing Secret manifests to Git:

# DO NOT DO THIS
apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
data:
  password: czNjdXIzLXBAc3N3MHJk  # base64 of s3cur3-p@ssw0rd

Even in private repositories, this exposes secrets to every developer, every CI runner, every backup system, and every tool that has read access to the repo. Once a secret is committed, it lives in Git history forever — even if you delete the file in a subsequent commit.

etcd Encryption at Rest

By default, etcd stores all Kubernetes data (including Secrets) in plaintext. You should always enable encryption at rest:

# Check if encryption is configured
kubectl get apiserver -o jsonpath='{.items[0].spec.encryption}'

# Verify encryption is active
kubectl describe apiservice v1 | grep -i encrypt

Even with etcd encryption enabled, the encryption key is typically stored on the control plane node. Managed Kubernetes services (EKS, GKE, AKS) handle this for you, but self-managed clusters need explicit configuration.

What Good Secrets Management Looks Like

A proper secrets management solution should provide:

1. Encryption at rest — Secrets encrypted with keys you control
2. Fine-grained access control — Per-secret or per-path permissions
3. Audit trail — Who accessed which secret and when
4. Automatic rotation — Secrets rotated without application downtime
5. Git-safe storage — Encrypted secrets that can be safely committed to version control
6. Separation of concerns — Developers define what secrets they need; security teams control the values

Option 1: Bitnami Sealed Secrets

Sealed Secrets is the simplest approach. It lets you encrypt secrets client-side so that only the cluster’s controller can decrypt them. The encrypted form (a SealedSecret resource) is safe to commit to Git.

Architecture

1. A controller runs in the cluster with an asymmetric key pair (RSA)
2. You encrypt secrets locally using the kubeseal CLI with the controller’s public key
3. The encrypted SealedSecret is committed to Git
4. The controller decrypts it and creates a standard Kubernetes Secret

Installation

# Install the controller
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system

# Install the CLI
brew install kubeseal  # macOS
# or download from GitHub releases

Usage

# Create a regular secret manifest (do NOT apply it)
kubectl create secret generic db-creds \
  --from-literal=password='s3cur3-p@ssw0rd' \
  --dry-run=client -o yaml > secret.yaml

# Encrypt it with kubeseal
kubeseal --format yaml < secret.yaml > sealed-secret.yaml

# The sealed-secret.yaml is safe to commit to Git
cat sealed-secret.yaml

The resulting SealedSecret looks like:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-creds
  namespace: production
spec:
  encryptedData:
    password: AgBy8hXM9kLt...  # RSA-encrypted, safe for Git

Tradeoffs

Pros:

• Simplest to set up and understand
• Encrypted secrets can be stored in Git (GitOps-friendly)
• No external dependencies beyond the cluster controller

Cons:

• No automatic rotation
• No audit trail of secret access
• Secrets are still stored as standard K8s Secrets after decryption
• If the controller’s private key is lost, all secrets are unrecoverable
• Encryption is tied to a specific cluster (cannot share across clusters)

Best for: Small teams using GitOps who want encrypted secrets in Git without external infrastructure.

Option 2: External Secrets Operator (ESO)

External Secrets Operator synchronizes secrets from external secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault) into Kubernetes Secrets. Instead of storing secret values anywhere in your Kubernetes manifests, you store a reference to the external provider.

Architecture

1. ESO runs as a controller in your cluster
2. You create a SecretStore (or ClusterSecretStore) pointing to your provider
3. You create ExternalSecret resources that reference specific secrets by path
4. ESO fetches the values and creates native K8s Secrets
5. Pods consume the K8s Secrets normally (env vars, volume mounts)
6. ESO periodically re-fetches to pick up rotated values

Installation

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace

AWS Secrets Manager Example

# Step 1: SecretStore -- how to connect to AWS
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa  # IRSA-enabled SA
---
# Step 2: ExternalSecret -- what to fetch
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-database-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: app-db-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: /production/app/database
        property: username
    - secretKey: password
      remoteRef:
        key: /production/app/database
        property: password

The full implementation with IRSA authentication, ClusterSecretStore, templated secrets, and bulk sync is available in Template 18: External Secrets Operator.

Tradeoffs

Pros:

• Leverages enterprise-grade secret managers (AWS, GCP, Azure, Vault)
• Automatic rotation via refreshInterval
• Audit trail provided by the external provider (CloudTrail, Vault audit log)
• Supports 20+ providers
• GitOps-friendly (only references are in Git, not values)

Cons:

• Requires an external secret manager (additional infrastructure cost)
• Network dependency — if the provider is unreachable, new secrets cannot be synced
• Secrets are still stored as standard K8s Secrets after sync
• More complex setup than Sealed Secrets

Best for: Organizations already using cloud secret managers who want seamless Kubernetes integration.

Option 3: HashiCorp Vault

Vault is the most full-featured secrets management platform. It provides encryption, dynamic secrets, automatic rotation, fine-grained policies, and a complete audit trail. It integrates with Kubernetes via the Vault Agent Injector or the Vault CSI Provider.

Architecture

Vault can run as an external service or inside the cluster. The two main integration patterns with Kubernetes are:

Vault Agent Injector (sidecar pattern):

1. A mutating webhook intercepts pod creation
2. An init container fetches secrets from Vault at startup
3. A sidecar container keeps secrets refreshed during the pod’s lifetime
4. Secrets are written to a shared tmpfs volume (never touch disk)

Vault CSI Provider:

1. Uses the Kubernetes Secrets Store CSI Driver
2. Mounts secrets as files into pod volumes
3. No sidecar needed (lower resource overhead)
4. Secrets are populated when the volume is mounted

Installation

# Add the HashiCorp Helm repo
helm repo add hashicorp https://helm.releases.hashicorp.com

# Install Vault in dev mode (for testing only)
helm install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  --set "server.dev.enabled=true"

# Install the injector for sidecar-based secret injection
helm install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  --set "injector.enabled=true" \
  --set "server.ha.enabled=true" \
  --set "server.ha.replicas=3"

Vault Agent Injector Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    metadata:
      annotations:
        # These annotations tell the Vault Agent Injector what to do
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "app-role"
        vault.hashicorp.com/agent-inject-secret-db-creds: "secret/data/production/database"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "secret/data/production/database" -}}
          export DB_HOST="{{ .Data.data.host }}"
          export DB_USER="{{ .Data.data.username }}"
          export DB_PASS="{{ .Data.data.password }}"
          {{- end -}}
    spec:
      serviceAccountName: app-sa
      containers:
        - name: app
          image: myapp:v1.0
          command: ["/bin/sh", "-c", "source /vault/secrets/db-creds && ./start.sh"]

Dynamic Secrets

Vault’s killer feature is dynamic secrets. Instead of storing static database passwords, Vault generates short-lived credentials on demand:

# Enable the database secrets engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/production \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@db.example.com:5432/app" \
  allowed_roles="app-role" \
  username="vault_admin" \
  password="vault_admin_password"

# Create a role that generates credentials with a 1-hour TTL
vault write database/roles/app-role \
  db_name=production \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Each pod gets its own unique database credentials that automatically expire. If a credential is compromised, the blast radius is limited to one pod and one hour.

Tradeoffs

Pros:

• Most comprehensive secrets management solution
• Dynamic secrets (short-lived, per-pod credentials)
• Fine-grained access policies (per-path, per-identity)
• Complete audit trail of every secret access
• Encryption as a service (transit secrets engine)
• Supports secrets rotation, lease renewal, and revocation

Cons:

• Significant operational complexity (HA, unsealing, backup, upgrades)
• Resource overhead (Vault servers + agent sidecars in every pod)
• Steep learning curve
• Can be a single point of failure if not properly configured for HA
• Enterprise features (namespaces, Sentinel policies) require a paid license

Best for: Large organizations with dedicated platform teams that need dynamic secrets, audit compliance, and multi-cluster management.

Option 4: Mozilla SOPS

SOPS (Secrets OPerationS) encrypts individual values within YAML/JSON files while leaving keys and structure visible. This means you can commit encrypted secrets to Git and review diffs meaningfully.

Architecture

1. SOPS encrypts secret values using AWS KMS, GCP KMS, Azure Key Vault, or PGP keys
2. Encrypted files are committed to Git
3. A CI/CD pipeline or operator (such as Flux SOPS integration) decrypts them at deploy time
4. Decrypted values become standard Kubernetes Secrets

Usage

# Encrypt a secret file with AWS KMS
sops --encrypt --kms arn:aws:kms:us-east-1:123456789012:key/abc-123 \
  secret.yaml > secret.enc.yaml

# The encrypted file looks like this:
cat secret.enc.yaml

apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
data:
  password: ENC[AES256_GCM,data:abc123...,iv:def456...,tag:ghi789...]
sops:
  kms:
    - arn: arn:aws:kms:us-east-1:123456789012:key/abc-123
      created_at: "2026-02-09T00:00:00Z"
  version: 3.8.1

Notice that the structure is visible (you can see password is the key) but the value is encrypted. This makes Git diffs meaningful — you can see which keys changed without seeing the values.

Flux Integration

If you use Flux for GitOps, SOPS decryption is built in:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: app-secrets
  namespace: flux-system
spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-kms  # Contains KMS credentials
  path: ./secrets
  sourceRef:
    kind: GitRepository
    name: app-repo

Tradeoffs

Pros:

• Simple, file-based approach that fits naturally into GitOps workflows
• Encrypted diffs are reviewable in pull requests
• No additional infrastructure beyond a KMS key
• Works with any CI/CD system
• Supports multiple encryption backends (KMS, PGP, age)

Cons:

• No automatic rotation
• No audit trail of secret access (only Git history of changes)
• Manual key management if using PGP
• Secrets are still standard K8s Secrets after decryption
• Encryption key rotation requires re-encrypting all files

Best for: Teams using GitOps (Flux, ArgoCD) who want encrypted secrets in Git without managing additional infrastructure.

Comparison Matrix

Feature	Sealed Secrets	ESO	Vault	SOPS
Complexity	Low	Medium	High	Low
Encryption at rest	Yes (RSA)	Depends on provider	Yes (AES-256-GCM)	Yes (KMS/PGP)
Automatic rotation	No	Yes (refreshInterval)	Yes (dynamic secrets)	No
Audit trail	No	Via provider	Yes (built-in)	Git history only
Dynamic secrets	No	No	Yes	No
GitOps friendly	Yes	Yes	Partial	Yes
External dependency	None	Secret manager	Vault cluster	KMS key
Cost	Free	Provider costs	Vault infrastructure	KMS costs (~$1/key/mo)
Multi-cluster	No	Yes	Yes	Yes

Recommendations by Organization Size

Startups and Small Teams (1-20 engineers)

Start with Sealed Secrets or SOPS. Both are simple, require no external infrastructure, and integrate well with GitOps. Use SOPS if you are already on AWS/GCP and can leverage KMS.

Mid-Size Organizations (20-100 engineers)

Use External Secrets Operator with your cloud provider’s secret manager (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault). You get automatic rotation, audit trails, and fine-grained access control without managing Vault infrastructure.

Large Enterprises (100+ engineers)

Deploy HashiCorp Vault for dynamic secrets, comprehensive audit trails, and centralized policy management. The operational overhead is justified by the security benefits at scale. Consider Vault with ESO as the Kubernetes integration layer.

Essential Practices Regardless of Tool

Whichever tool you choose, follow these practices:

1. Enable etcd encryption at rest — This is the baseline. Without it, secrets are plaintext on disk.
2. Restrict RBAC for secrets — Limit who can get, list, and watch secrets. Most developers should not have direct secret access.
3. Mount secrets as files, not environment variables — Environment variables appear in /proc/<pid>/environ, container inspect output, and crash dumps. File mounts are more secure.
4. Rotate secrets regularly — Even with the best tooling, secrets should be rotated on a schedule (90 days maximum for static secrets).
5. Audit secret access — Enable Kubernetes API audit logging for all secret operations (get, list, watch, create, update, delete).
6. Never commit plaintext secrets to Git — Use pre-commit hooks to scan for accidentally committed credentials.

Putting It Into Practice

Kubernetes secrets management is not a one-tool problem. The right solution depends on your team size, infrastructure, compliance requirements, and operational maturity. Start with the simplest approach that meets your security requirements and evolve as your needs grow.

The complete External Secrets Operator setup — with IRSA authentication, SecretStore, ClusterSecretStore, templated secrets, and bulk sync — is available in Template 18: External Secrets Operator as part of the K8s Security Pro bundle.

---

Related Templates

Implement what you’ve learned with these production-ready YAML templates:

• Template 18: External Secrets Operator — AWS Secrets Manager integration with IRSA, SecretStore, ExternalSecret, and bulk sync.
• Template 05: Secure Service Account — Lock down service accounts that access secrets with automountServiceAccountToken controls.
• Template 04: Least Privilege RBAC — Restrict who can read, list, and watch secrets at the namespace level.
• Template 10: Comprehensive Audit Policy — Log all secret access operations for compliance and incident response.

Related Articles

• Kubernetes RBAC Best Practices: Least Privilege Done Right — Control who can access secrets through proper RBAC configuration.
• Kubernetes Runtime Security with Falco: A Complete Guide — Detect unauthorized secret access at runtime with Falco rules.
• Kubernetes CIS Benchmark and SOC2 Compliance Guide — Meet CIS 5.4.1 and 5.4.2 requirements for secrets management.