Kubernetes In-Cluster Scanner¶

The Kubernetes In-Cluster Scanner (kubernetes-inspector) is a first-class VirtualSutra agent that audits live Kubernetes clusters for security misconfigurations using the Kubernetes API - no cloud credentials required. It is dispatched directly by the orchestrator, one task per K8SCloudConfig in the inspection scope, running in parallel with cloud and codebase scanner tasks.

Image: ghcr.io/virtual-sutra/kubernetes-inspector Redis stream: vsutra:tasks:kubernetes_inspector AgentType: KUBERNETES_INSPECTOR

Capabilities¶

Compliance mappings are emitted for cis_kubernetes and nist_csf (NSA Kubernetes Hardening Guide).

Prerequisites¶

The scanner connects to the Kubernetes API server. No cloud provider credentials are needed.

• In-cluster: the agent pod's ServiceAccount must be able to list what a scan reads. The Helm chart grants that (see In-cluster).
• External: a kubeconfig with read access to the target cluster(s). The scan only lists: every read, and what grants it, are in Access a scan needs.

Configuration¶

Kubernetes clusters to scan are declared in InspectionConfig.scope.kubernetes_clusters as a list of K8SCloudConfig objects.

{
  "scope": {
    "kubernetes_clusters": [
      {
        "cluster_name": "prod-us-east-1",
        "kubeconfig_ref": "env://KUBECONFIG_BASE64",
        "context": "arn:aws:eks:us-east-1:123456789012:cluster/prod",
        "namespaces": [],
        "skip_namespaces": ["kube-system", "kube-public"],
        "enable_pod_checks": true,
        "enable_rbac_checks": true,
        "enable_network_checks": true,
        "enable_pss_checks": true
      }
    ]
  }
}

K8SCloudConfig fields¶

Authentication¶

In-cluster (default)¶

Leave kubeconfig_ref unset. The agent uses the pod's mounted ServiceAccount token automatically.

The Helm chart creates a ClusterRole and ClusterRoleBinding named after the release (<release>-kubernetes-inspector) and binds it to the agent's ServiceAccount, unless rbac.create is false. It is broader than a scan needs: get and list (no watch) on pods, namespaces, serviceaccounts, services, configmaps, clusterroles, clusterrolebindings, roles, rolebindings, networkpolicies, podsecuritypolicies, jobs, cronjobs, deployments, replicasets, statefulsets and daemonsets. A scan only uses the list reads in Access a scan needs. On the SaaS profile the chart creates no ClusterRole and unmounts the token, because a hosted inspector scans tenants' clusters through their kubeconfigs and refuses the in-cluster fallback.

Where the chart does not create the role, bind the read-only vsutra-scanner-read ClusterRole shown under Access a scan needs, with a ServiceAccount subject for the agent's ServiceAccount, which grants exactly those reads.

External clusters¶

Set kubeconfig_ref to a secret reference that resolves to the kubeconfig. The value may be the kubeconfig itself (YAML) or base64 of it:

# The kubeconfig as it is ...
export KUBECONFIG_RAW="$(kubectl config view --minify --flatten)"

# ... or base64 of it, for a secret store (SECRETS_BACKEND=env)
base64 -w 0 ~/.kube/config > kubeconfig.b64
export KUBECONFIG_BASE64="$(cat kubeconfig.b64)"

Then reference it in inspection.json:

{
  "kubeconfig_ref": "env://KUBECONFIG_BASE64"
}

The exact rule, and what is refused, is in Kubernetes scanning. In short: a value that is a kubeconfig is used as given; otherwise it is base64-decoded once and must then be one; anything else is refused, naming kubeconfig_ref and never the value, with no fall back to in-cluster credentials.

A saved connection's credential is sent to the scanner raw; a secret you keep under env:// or vault:// conventionally holds base64. Both work.

The client is built from a new kubeconfig that holds only the allowed fields of the selected context. A kubeconfig that would run a command, read a file in the scanner's pod, use the pod's own cloud identity or route through a proxy is refused, naming the field. Allowed: token and/or client-certificate-data with client-key-data for the user; server (an https:// URL), certificate-authority-data, insecure-skip-tls-verify and tls-server-name for the cluster. The one exec block accepted is gke-gcloud-auth-plugin, which is never run (see below).

GKE clusters¶

A kubeconfig whose user is the gke-gcloud-auth-plugin exec block needs gcp_sa_key_ref: the scanner never runs the plugin, and exchanges the service-account key for an access token itself. The key's token_uri must be Google's. A GKE cluster saved from a GCP discovery is reached the same way with its cloud account's key, whose service account needs roles/container.viewer on the cluster's project.

Findings Schema¶

resource_type values¶

resource_id format¶

{cluster_name}/{namespace}/{kind}/{name}    a workload (kind in lower case: deployment, cronjob, ...)
{cluster_name}/{namespace}/{pod_name}       a pod nothing controls, a static pod
{cluster_name}/{namespace}                  a namespace-level finding
{cluster_name}/{binding_name}               a cluster-scoped finding

Examples:

• prod-us-east-1/default/deployment/web (one finding for all of the Deployment's pods)
• prod-us-east-1/default/my-pod (a bare pod)
• prod-us-east-1/default (namespace-level finding)
• prod-us-east-1/cluster-admin-binding (cluster-scoped)

Every segment is escaped (% as %25, then / as %2F), so a cluster name with a / (an EKS ARN) stays one segment. Tell the forms apart by the number of segments, never by a segment's value.

Workloads with no running pod¶

A workload with no running pod is assessed from its pod template, under the same resource id, with the tag assessed_from: template. See Workloads with no running pod.

Compliance controls¶

Each finding states its controls in compliance_controls, per check, for the frameworks where a control exists for what the check reads:

• cis_kubernetes - CIS Kubernetes Benchmark recommendation numbers (the resource-limits and image-tag checks have no CIS recommendation and state none)
• nist_csf - NIST Cybersecurity Framework 2.0 subcategory IDs
• soc2, nist_800_53, pci_dss and iso27001

Environment Variables¶

Example inspection.json¶

{
  "run_name": "prod-k8s-audit",
  "scope": {
    "kubernetes_clusters": [
      {
        "cluster_name": "prod-us-east-1",
        "kubeconfig_ref": "env://KUBECONFIG_BASE64",
        "skip_namespaces": ["kube-system", "kube-public", "monitoring"],
        "enable_pod_checks": true,
        "enable_rbac_checks": true,
        "enable_network_checks": true,
        "enable_pss_checks": true
      },
      {
        "cluster_name": "staging-eu-west-1",
        "kubeconfig_ref": "vault://secret/data/k8s/staging#kubeconfig",
        "namespaces": ["default", "app", "data"],
        "enable_pod_checks": true,
        "enable_rbac_checks": true,
        "enable_network_checks": false,
        "enable_pss_checks": true
      }
    ]
  },
  "compliance_frameworks": ["cis_kubernetes", "nist_csf"]
}