Kubernetes In-Cluster Scanning¶

VirtualSutra AI can audit live Kubernetes clusters for security misconfigurations using the Kubernetes API - no cloud provider credentials required. Kubernetes scanning runs in parallel with cloud and codebase scanning as part of a regular inspection.

What gets checked¶

Each check group can be enabled or disabled independently per cluster.

Adding a cluster to an inspection¶

In the inspection wizard, expand Kubernetes Clusters under Step 1 - Targets and click Add Cluster.

Enable or disable each check group with the toggle switches.

You can add as many clusters as you need. Each cluster runs as an independent scanner task in parallel.

Authentication¶

Running inside the cluster¶

Leave the Kubeconfig field blank. The scanner uses the pod's mounted ServiceAccount token automatically. The ServiceAccount must be able to list what a scan reads (Access a scan needs). The Helm chart does this for you: it binds a ClusterRole to the agent's ServiceAccount that is broader than a scan needs (read access to workloads, RBAC objects, services and configmaps), unless your administrator turned rbac.create off. On a hosted deployment there is no in-cluster scan: give a kubeconfig.

External clusters¶

Give the scanner a kubeconfig for the cluster, in either form:

• The kubeconfig itself: the YAML file as it is, for example the output of kubectl config view --minify --flatten. Choose the file or paste it in the console, or reference it as a secret.
• Base64 of the kubeconfig: for a secret you keep in an environment variable or a vault entry.

# Optional: encode it for a secret store (either form is accepted)
base64 -w 0 ~/.kube/config

Then paste the kubeconfig (or the base64 string) into the Kubeconfig field in the wizard, or store it as a secret and reference it as env://KUBECONFIG_BASE64. A reference with no scheme (no env://, vault:// and so on) is not looked up: its text is taken as the kubeconfig itself.

The rule the scanner applies. The value the reference resolves to is read like this:

1. If it parses as a YAML mapping that has a clusters, contexts or users key, or says kind: Config, it is the kubeconfig and is used as it is.
2. Otherwise it is base64-decoded once, and the result must be a kubeconfig. The decoding is strict: a character outside the base64 alphabet refuses the value. Line breaks and spaces (base64 tools wrap their output at 76 columns) and missing = padding are accepted.
3. Anything else is refused.

Refused: text that is neither form, a mapping without those keys, base64 of non-text, base64 of base64 (decoded once it is a string, not a mapping), and an empty value. The task fails with a message that names the field (kubeconfig_ref) and the reason, never the value.

A refused value is never replaced by the scanner's own in-cluster credentials. The console's Check applies the same rule to a kubeconfig you paste or upload, so the check and the scan agree on whether the value is a kubeconfig. It does not apply the scan's other refusals (a command, a file path, a proxy, a non-HTTPS server) except on a hosted deployment, where it also applies the rule to a secret reference; elsewhere a reference to a secret that may not exist yet is left unread until the scan.

The kubeconfig is not used as given. The scanner builds a new one from its token or client certificate, server address and certificate authority; a kubeconfig that would run a command, read a file or use a proxy is refused. See Kubernetes In-Cluster Scanner.

GKE clusters (gke-gcloud-auth-plugin)¶

If your kubeconfig uses gke-gcloud-auth-plugin exec-auth (the default for GKE), provide a GCP service account JSON key via the gcp_sa_key_ref field in inspection.json to bypass the gcloud SDK requirement:

{
  "cluster_name": "my-gke-cluster",
  "kubeconfig_ref": "env://KUBECONFIG_BASE64",
  "gcp_sa_key_ref": "env://GCP_SA_KEY"
}

For a saved cluster connection in the console, no second key is uploaded: choose the kubeconfig, then pick the saved GCP cloud account under Cloud account for gcloud sign-in. See Signing in with a cloud account.

A GKE cluster saved from a GCP discovery needs no kubeconfig: it is reached with its cloud account's key, whose service account needs roles/container.viewer on the cluster's project. See Access a scan needs.

Access a scan needs¶

A scan only lists. It never changes anything in the cluster. Every read it makes:

ReplicaSets and Jobs are read only to name the workload a pod belongs to. If they are refused, the check still counts as run, but its findings are filed under the ReplicaSet or Job instead of under the Deployment or CronJob that owns it, so their resource ID changes at every rollout, and Deployments (for ReplicaSets) or CronJobs and Jobs (for Jobs) with no running pod are not assessed from their template either. Deployments, StatefulSets, DaemonSets, CronJobs and Jobs are read for the pod template of a workload with no running pod (see Workloads with no running pod); if one of them is refused, workloads of that kind with no pod are not assessed. The scan records either in its coverage record.

What grants it¶

A GKE cluster found by discovery and scanned with a cloud account's key. The key's service account needs roles/container.viewer on the cluster's project. It carries every read above and nothing that writes. roles/container.clusterViewer, which discovery asks for, lists clusters and reads nothing inside one, so it does not make a scan work.

gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member=serviceAccount:<SERVICE_ACCOUNT_EMAIL> --role=roles/container.viewer

Alternatively bind the read-only ClusterRole below to the service account inside the cluster. GKE presents a Google identity to Kubernetes RBAC as a User named by its e-mail address.

A saved kubeconfig. Bind the ClusterRole below to the identity the kubeconfig signs in as. For a ServiceAccount token use a ServiceAccount subject with its name and namespace in place of the User.

The in-cluster ServiceAccount. The ClusterRole the Helm chart binds to the agent's ServiceAccount (broader than a scan needs, and a scan uses only the reads above), or the ClusterRole below bound to that ServiceAccount.

The ClusterRole grants exactly the reads above. Save it as vsutra-scanner-read.yaml, fill in the subject, and apply it with kubectl apply -f vsutra-scanner-read.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vsutra-scanner-read
rules:
  - apiGroups: [""]
    resources: ["namespaces", "pods"]
    verbs: ["list"]
  - apiGroups: ["apps"]
    resources: ["daemonsets", "deployments", "replicasets", "statefulsets"]
    verbs: ["list"]
  - apiGroups: ["batch"]
    resources: ["cronjobs", "jobs"]
    verbs: ["list"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["networkpolicies"]
    verbs: ["list"]
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["clusterrolebindings"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vsutra-scanner-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vsutra-scanner-read
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: <SERVICE_ACCOUNT_EMAIL>

For a GKE key the subject is the service account's e-mail, as shown. For a saved kubeconfig use the user the kubeconfig signs in as (<KUBECONFIG_USER_NAME>); for the in-cluster scanner use kind: ServiceAccount with <SCANNER_SERVICE_ACCOUNT_NAME> and <SCANNER_NAMESPACE>.

When a scan could not read the cluster¶

A check that could not read part of its scope is recorded as incomplete in the scan's coverage record, with a reason, what it needs and what to do. Every sentence is fixed wording; nothing from your cluster is put into it. The reason is decided from the HTTP status and the kind of error, never from its text.

A check group you switched off is recorded as switched_off. That is not a gap: you asked for it.

Resume or start a new scan¶

A resume scans every Kubernetes cluster of the run again, with every check, using the configuration the run was submitted with: the cluster's address, certificate authority and credential reference. It does not look the saved connection up again, and it cannot be narrowed to the one cluster or check that had the gap. So resume when the cure is outside the run: a grant, a firewall or egress rule, a longer timeout.

Start a new scan when the cure is inside the run's configuration: a new kubeconfig, a new discovery of the cluster, a new service-account key.

How a scan ends¶

Earlier releases failed the task for a scan that read something but had gaps; the run was partial either way. Update the scanner and orchestrator together, or a run with gaps may end completed instead of partial.

A client that could not be built at all (a refused kubeconfig, a failed Google token exchange) fails the task with the refusal and its remediation, and writes no coverage record.

Findings¶

Kubernetes findings appear in the Findings tab alongside all other findings. Filter by Agent: kubernetes-inspector to view them in isolation.

Resource types¶

Resource ID format¶

{cluster_name}/{namespace}/{kind}/{name}    a workload (kind in lower case: deployment, cronjob, ...)
{cluster_name}/{namespace}/{pod_name}       a pod nothing controls, a static pod
{cluster_name}/{namespace}                  a namespace-level finding
{cluster_name}/{binding_name}               a cluster-scoped finding

Examples:

• prod-us-east-1/default/deployment/web (one finding for all of the Deployment's pods)
• prod-us-east-1/default/my-pod (a bare pod)
• prod-us-east-1/default (namespace-level finding)
• prod-us-east-1/cluster-admin-binding (cluster-scoped)

Every segment is escaped (% as %25, then / as %2F), so a cluster name with a / (an EKS ARN) stays one segment. Tell the forms apart by the number of segments, never by a segment's value.

Earlier releases gave every pod-based finding resource_type k8s:pod. A saved filter on k8s:pod in the Resources tab now matches bare and static pods only: filter on the workload's type instead (k8s:deployment, ...). The resource id, and so the finding's identity, mute rules and remediation status, did not change.

Workloads with no running pod¶

A pod-based check reports one finding per workload, read from its pods. A Deployment or StatefulSet scaled to zero, a DaemonSet no node runs, a CronJob whose history holds no pod and a Job nothing owns whose pods are gone are assessed from their pod template instead (spec.template; spec.jobTemplate.spec.template for a CronJob), by the same checks and under the same resource id their pods would give them. A Deployment scaled 3 → 0 → 3 therefore keeps one finding, with its mute rule and remediation status, throughout.

The tag assessed_from says which: pods or template. A template-only finding has pod_count 0 and a closing sentence saying it was read from the template: what the cluster adds when it creates a pod (a LimitRange's default limits, a mutating webhook's sidecar) is not in it, so a template-only finding can differ from the one its running pods would give. A workload that has a running pod is assessed from its pods only, never twice.

The scan reads the templates with list on deployments, statefulsets, daemonsets (apps) and cronjobs, jobs (batch); the chart's ClusterRole grants them. Where one is refused, workloads of that kind with no pod are not assessed and the scan's coverage record says so.

Which containers a pod finding names¶

A finding about privileged containers, missing CPU/memory limits or :latest image tags covers every offending container of the workload, and names them by kind in its tags: containers (regular containers only), init_containers and ephemeral_containers, each a comma-separated list that is empty when there is none of that kind. container is still the first offending name of any kind.

If you wrote a mute rule on a containers value from an earlier scan that listed init or ephemeral containers too, it no longer matches after a rescan - mute rules match tags exactly - and the finding is shown again. Re-create the rule on the new value.

Compliance mappings¶

Kubernetes findings are mapped to:

• CIS Kubernetes Benchmark v1.9 - specific control IDs (e.g. 5.2.1, 5.1.1)
• NIST CSF / NSA Kubernetes Hardening Guide - subcategory IDs

Select cis_kubernetes and/or nist_csf as compliance frameworks when configuring the inspection scope.

Example inspection.json snippet¶

{
  "scope": {
    "kubernetes_clusters": [
      {
        "cluster_name": "prod",
        "kubeconfig_ref": "env://KUBECONFIG_BASE64",
        "skip_namespaces": ["kube-system", "kube-public", "monitoring"],
        "enable_pod_checks": true,
        "enable_rbac_checks": true,
        "enable_network_checks": true,
        "enable_pss_checks": true
      },
      {
        "cluster_name": "staging",
        "kubeconfig_ref": "vault://secret/data/k8s/staging#kubeconfig",
        "namespaces": ["default", "app"],
        "enable_pod_checks": true,
        "enable_rbac_checks": true,
        "enable_network_checks": false,
        "enable_pss_checks": true
      }
    ]
  },
  "compliance_frameworks": ["cis_kubernetes", "nist_csf"]
}

For the full K8SCloudConfig field reference, see the Kubernetes In-Cluster Scanner agent reference.