Kubernetes Audit Logging: Complete Setup and Analysis Guide

Q: Can I customize the audit policy on managed Kubernetes (EKS, GKE, AKS)?

EKS uses a fixed audit policy but all events are available in CloudWatch Logs. GKE enables audit logging by default with configurable detail levels. AKS supports custom diagnostic settings. For full policy customization, you need a self-managed cluster.

Q: How do I search audit logs for a specific incident?

Filter by user.username (who), verb (what action), objectRef.resource (what resource), objectRef.namespace (where), and stageTimestamp (when). Centralized logging with Elasticsearch or Loki makes this search fast and straightforward.

Every API call to your Kubernetes cluster — every kubectl exec, every secret read, every RBAC change — passes through the API server. Audit logging captures these events and gives you a complete record of who did what, when, and from where. Without it, you’re flying blind. A compromised service account could be reading every secret in your cluster right now and you would have no way to know.

Audit logging is not optional for production clusters. It’s required by CIS Benchmark (control 1.2.18), expected by SOC2 auditors (CC7.1), and essential for incident response. This guide covers everything from configuring the audit policy to detecting real security incidents in your logs.

Why Audit Logging Matters

Consider a typical attack scenario: an attacker gains code execution inside a pod, reads the auto-mounted service account token, and starts querying the Kubernetes API. Without audit logging, you’ll never know:

Which secrets were read
Whether RBAC roles were modified
If new pods were created for persistence
Whether the attacker accessed other namespaces
When the compromise began

With audit logging enabled, every one of these actions generates an audit event with the source IP, user identity, resource accessed, and the full request and response bodies (depending on your policy level). This is your forensic trail.

Audit logs are also your primary evidence for compliance audits. When a SOC2 auditor asks “how do you monitor access to sensitive resources?”, your audit policy and log retention configuration are the answer.

Audit Policy Levels

Kubernetes audit logging uses a policy file that defines what to log and at what detail level. Each rule in the policy specifies one of four levels:

Level	What’s Captured	Use Case
`None`	Nothing is logged	Health checks, discovery endpoints, noise reduction
`Metadata`	Request metadata only (user, verb, resource, timestamp)	General activity monitoring, RBAC auditing
`Request`	Metadata + request body	Detecting what was submitted (new pods, role changes)
`RequestResponse`	Metadata + request body + response body	Full forensics on sensitive resources (secrets, exec)

The key insight is that you don’t want the same level for everything. Logging RequestResponse for all resources would generate enormous volumes of data — most of it useless. A well-designed policy applies maximum detail where it matters (secrets, exec, RBAC) and minimal logging for noise (health checks, status updates).

How the Policy Evaluator Works

The API server evaluates audit policy rules top to bottom and uses the first matching rule. This means:

Put specific rules first (e.g., “log nothing for health checks”)
Put general rules last (e.g., “log metadata for everything else”)
Order matters — a broad rule early in the file can mask specific rules later

If no rule matches a request, the default behavior is to not log it. Always include a catch-all rule at the bottom.

Building an Effective Audit Policy

Here’s a production-ready audit policy with 16 rules, organized by priority. This is the same policy included in Template 10: Audit Policy.

Rule 1: Skip Noise

The single most important optimization. These endpoints are hit constantly by kubelets, controllers, and health checks:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Rule 1: Skip health checks and discovery (high volume, low value)
  - level: None
    nonResourceURLs:
      - "/healthz*"
      - "/readyz*"
      - "/livez*"
      - "/openapi/*"
      - "/api"
      - "/api/*"
      - "/apis"
      - "/apis/*"
    verbs: ["get"]

Without this rule, your audit log will be dominated by thousands of health check entries per minute.

Rule 2: Skip System Noise

System components generate continuous watch requests that flood logs:

  # Rule 2: Skip system component noise
  - level: None
    users:
      - "system:kube-scheduler"
      - "system:kube-proxy"
      - "system:apiserver"
    verbs: ["get", "watch", "list"]

Rule 3: Full Capture on Secrets

Secrets are the crown jewels. Log everything:

  # Rule 3: Full request+response for Secrets
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]

This captures who created, read, updated, or deleted any secret, including the request and response bodies. Yes, this means secret values appear in audit logs — make sure your log storage is encrypted and access-controlled.

Rule 4: Full Capture on Exec and Attach

kubectl exec and kubectl attach give direct shell access to containers. This is the number one post-exploitation technique:

  # Rule 4: Full capture on exec, attach, portforward
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward"]

When investigating a breach, the first thing you’ll look for is exec events. This rule ensures you have the full command that was executed.

Rule 5: Log RBAC Changes

RBAC modifications are a high-severity event — an attacker with sufficient permissions will create new ClusterRoleBindings to escalate privileges:

  # Rule 5: Log all RBAC mutations
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
    verbs: ["create", "update", "patch", "delete"]

Rule 6: Authentication Events

Track who’s authenticating and from where:

  # Rule 6: Authentication events
  - level: RequestResponse
    resources:
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]
      - group: ""
        resources: ["serviceaccounts/token"]

Rule 7: Admission Controller Activity

If you’re using Kyverno or OPA/Gatekeeper, log their decisions:

  # Rule 7: Admission controller mutations
  - level: Request
    resources:
      - group: "admissionregistration.k8s.io"
        resources: ["validatingwebhookconfigurations", "mutatingwebhookconfigurations"]

Rules 8-12: Workload and Configuration Changes

  # Rule 8: Workload changes (deployments, daemonsets, statefulsets)
  - level: Request
    resources:
      - group: "apps"
        resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]
    verbs: ["create", "update", "patch", "delete"]

  # Rule 9: ConfigMap changes
  - level: Request
    resources:
      - group: ""
        resources: ["configmaps"]
    verbs: ["create", "update", "patch", "delete"]

  # Rule 10: Namespace lifecycle
  - level: Request
    resources:
      - group: ""
        resources: ["namespaces"]
    verbs: ["create", "update", "patch", "delete"]

  # Rule 11: Service and endpoint changes
  - level: Request
    resources:
      - group: ""
        resources: ["services", "endpoints"]
    verbs: ["create", "update", "patch", "delete"]

  # Rule 12: Network policy changes
  - level: Request
    resources:
      - group: "networking.k8s.io"
        resources: ["networkpolicies"]
    verbs: ["create", "update", "patch", "delete"]

Rules 13-16: Node, Storage, and Catch-All

  # Rule 13: Node changes
  - level: Request
    resources:
      - group: ""
        resources: ["nodes", "nodes/status"]
    verbs: ["create", "update", "patch", "delete"]

  # Rule 14: PV and PVC changes
  - level: Request
    resources:
      - group: ""
        resources: ["persistentvolumes", "persistentvolumeclaims"]
    verbs: ["create", "update", "patch", "delete"]

  # Rule 15: Skip read-only operations on non-sensitive resources
  - level: None
    verbs: ["get", "watch", "list"]

  # Rule 16: Catch-all -- log metadata for everything else
  - level: Metadata
    omitStages:
      - RequestReceived

Rule 15 drops read-only operations on non-sensitive resources (secrets and exec are already captured by earlier rules). Rule 16 catches anything that didn’t match a previous rule.

Configuring the API Server

The audit policy file must be accessible to the API server. The configuration depends on your cluster type.

Self-Managed Clusters (kubeadm)

Place the policy file on the control plane node and configure the API server:

# Copy policy to the control plane
sudo cp audit-policy.yaml /etc/kubernetes/audit-policy.yaml

# Edit the API server manifest
sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

Add these flags:

spec:
  containers:
    - command:
        - kube-apiserver
        - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
        - --audit-log-path=/var/log/kubernetes/audit/audit.log
        - --audit-log-maxage=30
        - --audit-log-maxbackup=10
        - --audit-log-maxsize=100

Mount the policy file and log directory:

volumeMounts:
  - name: audit-policy
    mountPath: /etc/kubernetes/audit-policy.yaml
    readOnly: true
  - name: audit-log
    mountPath: /var/log/kubernetes/audit/
volumes:
  - name: audit-policy
    hostPath:
      path: /etc/kubernetes/audit-policy.yaml
      type: File
  - name: audit-log
    hostPath:
      path: /var/log/kubernetes/audit/
      type: DirectoryOrCreate

EKS

EKS manages the control plane, so you enable audit logging through the cluster configuration:

aws eks update-cluster-config \
  --name my-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

Logs are sent to CloudWatch Logs under /aws/eks/<cluster-name>/cluster. You cannot customize the audit policy on EKS — AWS uses a fixed policy.

GKE

GKE enables audit logging by default. Logs appear in Cloud Logging under k8s_cluster resource type. You can configure the detail level:

gcloud container clusters update my-cluster \
  --enable-master-global-access \
  --logging=SYSTEM,WORKLOAD

AKS

AKS supports diagnostic settings for audit logging:

az monitor diagnostic-settings create \
  --name audit-logs \
  --resource <aks-resource-id> \
  --logs '[{"category":"kube-audit","enabled":true,"retentionPolicy":{"enabled":true,"days":90}}]' \
  --workspace <log-analytics-workspace-id>

Log Backends: File vs Webhook

Kubernetes supports two audit log backends:

File Backend

Writes audit events to a local file on the control plane node. Simple but limited:

Logs are only on the control plane node
No real-time streaming
Must be rotated and shipped separately
Good for small clusters or as a fallback

Webhook Backend

Sends audit events to an external HTTP endpoint in real time:

apiVersion: audit.k8s.io/v1
kind: Policy
# ... policy rules ...
---
apiVersion: v1
kind: Config
clusters:
  - name: audit-webhook
    cluster:
      server: https://audit-collector.monitoring.svc:443/audit
      certificate-authority: /etc/kubernetes/pki/audit-ca.crt
contexts:
  - name: audit-webhook
    context:
      cluster: audit-webhook
current-context: audit-webhook

Configure the API server:

--audit-webhook-config-file=/etc/kubernetes/audit-webhook-config.yaml
--audit-webhook-batch-max-size=100
--audit-webhook-batch-max-wait=5s

The webhook backend enables real-time streaming to your SIEM or log aggregation system. Use both file and webhook for redundancy — the file backend serves as a fallback if the webhook destination is unavailable.

Log Aggregation: EFK, Loki, and SIEM Integration

Raw audit log files on control plane nodes are not useful for investigation. You need centralized aggregation, indexing, and search.

EFK Stack (Elasticsearch + Fluentd + Kibana)

The traditional approach for Kubernetes log aggregation:

# Fluentd DaemonSet configuration for audit logs
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/kubernetes/audit/audit.log
      pos_file /var/log/fluentd/audit.log.pos
      tag kubernetes.audit
      <parse>
        @type json
        time_key stageTimestamp
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>

    <match kubernetes.audit>
      @type elasticsearch
      host elasticsearch.logging.svc
      port 9200
      index_name k8s-audit
      type_name _doc
    </match>

Grafana Loki

A lighter-weight alternative that uses labels instead of full-text indexing:

# Promtail configuration for audit logs
scrape_configs:
  - job_name: kubernetes-audit
    static_configs:
      - targets: [localhost]
        labels:
          job: k8s-audit
          __path__: /var/log/kubernetes/audit/audit.log
    pipeline_stages:
      - json:
          expressions:
            verb: verb
            user: user.username
            resource: objectRef.resource
            namespace: objectRef.namespace
            response_code: responseStatus.code
      - labels:
          verb:
          user:
          resource:
          namespace:

Loki uses significantly less storage than Elasticsearch and pairs naturally with Grafana for dashboards.

SIEM Integration

For enterprise environments, forward audit logs to your SIEM (Splunk, Datadog, Sumo Logic):

# Splunk HEC (HTTP Event Collector) via Fluentd
<match kubernetes.audit>
  @type splunk_hec
  hec_host splunk.example.com
  hec_port 8088
  hec_token YOUR_HEC_TOKEN
  index kubernetes
  sourcetype kube:audit
</match>

Detecting Security Incidents from Audit Logs

Raw logs are only useful if you know what to look for. Here are the most important patterns to detect:

Pattern 1: Unauthorized Secret Access

An attacker with a compromised service account will immediately try to read secrets:

verb: "get" OR "list"
objectRef.resource: "secrets"
user.username: NOT IN [known-controllers]

Alert on any service account reading secrets it doesn’t normally access. Build a baseline of normal secret access patterns first.

Pattern 2: Exec into Running Pods

kubectl exec is the most common post-exploitation technique:

verb: "create"
objectRef.resource: "pods/exec"
objectRef.subresource: "exec"

Alert on exec events outside of business hours, from unknown source IPs, or targeting production namespaces.

Pattern 3: RBAC Privilege Escalation

Watch for creation of new ClusterRoleBindings or modifications to existing roles:

verb: "create" OR "update" OR "patch"
objectRef.resource: "clusterrolebindings" OR "clusterroles"

Any unexpected RBAC change should trigger an immediate alert. Legitimate RBAC changes should go through GitOps and be predictable.

Pattern 4: Service Account Token Creation

An attacker may create new tokens for lateral movement:

verb: "create"
objectRef.resource: "serviceaccounts/token"

Pattern 5: New Pod in System Namespaces

Pods appearing in kube-system or kube-public outside of cluster upgrades are suspicious:

verb: "create"
objectRef.resource: "pods"
objectRef.namespace: "kube-system" OR "kube-public"

Pattern 6: Repeated 403 Forbidden Responses

An attacker probing API permissions generates 403 responses:

responseStatus.code: 403
COUNT > 10 within 60 seconds
GROUP BY user.username

This indicates someone is testing what their compromised credentials can access.

Building Alert Rules

Combine these patterns into alert rules in your monitoring system. For example, in Prometheus with Loki:

groups:
  - name: kubernetes-audit-alerts
    rules:
      - alert: UnauthorizedSecretAccess
        expr: |
          count_over_time({job="k8s-audit"} |= "secrets" | json | verb="get" | user_username!~"system:.*" [5m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Non-system user accessed secrets"

Compliance Requirements for Audit Logging

CIS Benchmark

Control	Description	Requirement
1.2.18	Ensure audit logs are enabled	`--audit-policy-file` must be set
1.2.19	Ensure audit log maxage is set	`--audit-log-maxage=30` (minimum 30 days)
1.2.20	Ensure audit log maxbackup is set	`--audit-log-maxbackup=10` (minimum 10)
1.2.21	Ensure audit log maxsize is set	`--audit-log-maxsize=100` (minimum 100MB)

SOC2 (CC7.1 - Monitoring)

SOC2 requires that you:

Monitor system components for anomalies
Have defined detection procedures for security events
Retain logs for a defined period (typically 90 days to 1 year)
Have alerting on security-relevant events

Audit logging with the patterns above satisfies all four requirements. Document your audit policy, retention period, and alerting rules as evidence for your SOC2 auditor.

PCI-DSS

PCI-DSS Requirement 10 mandates tracking all access to cardholder data environments. If your Kubernetes cluster processes payment data, your audit policy must capture all access to relevant namespaces and secrets.

Log Retention and Storage

Plan your storage requirements carefully. A busy cluster can generate gigabytes of audit logs per day.

Sizing estimates:

Cluster Size	Audit Level	Daily Log Volume
Small (20 nodes)	Metadata only	500MB - 1GB
Medium (50 nodes)	Mixed (as above)	2GB - 5GB
Large (200+ nodes)	Mixed (as above)	10GB - 50GB

Retention recommendations:

Hot storage (searchable): 30 days minimum
Warm storage (compressed, queryable): 90 days
Cold storage (archived): 1 year (for compliance)

Use log lifecycle policies to automatically move data between tiers:

# Elasticsearch ILM policy example
PUT _ilm/policy/k8s-audit-policy
{
  "policy": {
    "phases": {
      "hot": { "actions": { "rollover": { "max_size": "50gb", "max_age": "7d" } } },
      "warm": { "min_age": "30d", "actions": { "shrink": { "number_of_shards": 1 } } },
      "cold": { "min_age": "90d", "actions": { "freeze": {} } },
      "delete": { "min_age": "365d", "actions": { "delete": {} } }
    }
  }
}

Putting It Into Practice

Start with the audit policy in this guide, deploy it to your cluster, and set up log aggregation. Then build detection rules for the six patterns above. The entire setup can be done in a day and will transform your ability to detect and investigate security incidents.

The complete 16-rule audit policy, ready to deploy, is available in Template 10: Audit Policy as part of the K8s Security Pro template pack. It’s already optimized for noise reduction and covers all the CIS Benchmark requirements.

Frequently Asked Questions

What happens if audit logging is not enabled on my cluster?

Without audit logging, you have no record of API server activity. You cannot detect unauthorized secret access, exec sessions, RBAC changes, or lateral movement. During incident response, you’ll be unable to determine what the attacker accessed or how long the breach lasted. CIS Benchmark control 1.2.18 explicitly requires audit logging to be enabled.

Does audit logging impact API server performance?

Yes, but the impact is manageable with a well-designed policy. The None level rules at the top of your policy eliminate the highest-volume, lowest-value events. File-based logging adds minimal latency. Webhook-based logging can add a few milliseconds per request but can be configured with batching (--audit-webhook-batch-max-wait=5s) to reduce the impact. Most production clusters see less than 2% overhead.

Can I customize the audit policy on managed Kubernetes (EKS, GKE, AKS)?

EKS uses a fixed audit policy that you cannot customize, but all audit events are available in CloudWatch Logs. GKE enables audit logging by default with configurable detail levels. AKS supports custom diagnostic settings for log routing. For full policy customization, you need a self-managed cluster (kubeadm, k3s) or a managed service that supports custom API server flags.

How long should I retain audit logs?

CIS Benchmark requires a minimum of 30 days. SOC2 auditors typically expect 90 days of searchable logs and up to 1 year of archived logs. PCI-DSS Requirement 10.7 requires 1 year of retention with 3 months immediately available. Start with 90 days of hot storage and archive to cold storage for the compliance-required duration.

How do I search audit logs for a specific incident?

Filter by the relevant fields: user.username (who), verb (what action), objectRef.resource (what resource), objectRef.namespace (where), and stageTimestamp (when). For example, to find all exec events in the production namespace during a specific time window, filter for verb=create, objectRef.resource=pods/exec, objectRef.namespace=production, and the time range. Centralized logging with Elasticsearch or Loki makes this search fast and straightforward.

Implement what you’ve learned with these production-ready YAML templates:

Template 10: Audit Policy — 16-rule API server audit policy covering secrets, RBAC, exec, and authentication events.
Template 12: Falco Runtime Security Rules — 7 runtime detection rules that complement audit logging with process-level visibility.
Template 19: Kyverno Policy Bundle — Admission-time enforcement that works alongside audit logging for defense in depth.

Kubernetes CIS Benchmark and SOC2 Compliance: A Practical Guide — Map your audit logging controls to CIS and SOC2 compliance frameworks.
Kubernetes Security Tools Compared: Trivy, Falco, OPA, Kyverno, and Kubescape — See how audit logging fits into the broader Kubernetes security toolchain.