Kubernetes · Security · Platform engineering · 8 min read ·
Isolating tenants on a shared Kubernetes cluster
A namespace is not a security boundary, but a namespace plus five other controls is a usable one. What separates tenants on a shared cluster, what still needs a cluster of its own, and how to decide which customer gets which.
Every platform team reaches the same fork: one cluster per customer, which is simple and expensive, or many customers on a shared cluster, which is efficient and frightening. The usual answer is both, with a clear rule for who gets which, and a shared cluster hardened enough that the rule is about commercial tier rather than fear.
What a namespace gives you, and what it does not
A namespace scopes names and gives role bindings and quotas something to attach to. On its own it separates nothing: a pod in one namespace can reach a pod in any other, read the same node's secrets if it escapes its container, and consume the whole node's CPU. Treat the namespace as the unit everything else attaches to, not as the isolation.
The six controls that make a shared cluster tenable
1. Network policy, default deny. Every tenant namespace starts with a policy that denies all ingress and egress, then allows exactly: the ingress controller's namespace in, DNS out, the tenant's own database out, and the platform's telemetry collector out. Cross-tenant traffic is impossible by construction, and the policy is part of the tenant's rendered manifest so it cannot be forgotten.
2. Resource quotas and limit ranges. A quota caps what a namespace may request in total; a limit range gives every container a default request and limit so a pod without one cannot be scheduled unbounded. Without both, one tenant's runaway job is every tenant's outage.
3. Pod security. The restricted profile, enforced not audited: no privileged containers, no host namespaces, no root, a read-only root filesystem, every capability dropped. The baseline profile is for workloads you control; tenant workloads get restricted or they get their own cluster.
4. No shared service accounts. Each tenant workload runs as its own service account with no API permissions, and automountServiceAccountToken is off unless a workload demonstrably needs the API. A tenant pod that can talk to the Kubernetes API is a tenant pod that can enumerate the cluster.
5. Separate data, separate keys. Each tenant's data lives in its own database or schema with its own credential, and its secrets are encrypted with a tenant-specific key. The platform's own identity can reach the key store; the tenant's workload cannot. This is the control that turns a container escape from a catastrophe into an incident.
6. Node isolation where the risk warrants it. Taints and tolerations put sensitive tenants on their own node pool, so a kernel-level escape still lands on a node that holds only that tenant's workloads. It costs some bin-packing efficiency and is cheaper than a cluster.
What still needs a cluster of its own
Some requirements cannot be met by controls inside a shared cluster, however good:
- a contractual or regulatory demand for physical separation, or for a specific region or sovereignty boundary the shared cluster does not sit in;
- a customer who needs their own identity provider wired into the cluster's own authentication, not only into the application;
- a workload that needs cluster-level privileges (an operator, a custom scheduler, a device plugin);
- a customer large enough that their peak would starve everyone else, quota or not.
These are commercial facts, so the rule can be commercial too: the shared cluster is the default; a dedicated cluster is a tier with a price, and the same automation builds both.
One manifest, two deployment models
The mistake to avoid is two code paths. A tenant is a record: name, tier, region, deployment model. The provisioning pipeline renders that record into the tenant's manifest, and the manifest decides whether it lands in a namespace on a shared cluster or in a dedicated cluster the same pipeline creates first. The application does not know which it is in. Every control above is in the rendered manifest, so a tenant moving from shared to dedicated is a change of one field and a migration of data, not a re-platforming.
Proving it, continuously
Isolation that is not tested is a belief. A probe that runs on every promotion, signed in as one tenant, tries to reach another tenant's hostname, API, database and metrics, and fails the promotion if anything answers. The same probe, run from inside a tenant pod, tries the cluster API and the neighbouring namespaces. Making the probe part of the delivery pipeline means a regression blocks a release instead of waiting for a penetration test.
Cells, for when the shared cluster grows
A single shared cluster is a single blast radius and a single upgrade window. Past a few dozen tenants, split the shared estate into cells: identical clusters, each holding a slice of tenants, promoted through one at a time so an upgrade reaches one cell, is verified, and only then reaches the next. The tenant record gains a cell field; the pipeline gains a placement rule; nothing about a tenant changes.
The platform we run uses this shape, with the cross-tenant probe as a promotion gate and a dedicated model for enterprise tenants. A Kubernetes platform engagement usually begins by scoring an existing cluster against the six controls and deciding which tenants, if any, need to move.
Keep reading
Platform engineering · GitOps · CI/CD · 7 min
Golden paths that engineers actually take
A platform team's output is not the platform; it is the paths other teams take through it. How to build paved roads that are faster than the alternative, keep them honest with the people who use them, and avoid turning them into mandates.
Observability · Platform engineering · 7 min
SLOs before dashboards
Observability tooling is easy to buy and hard to make useful. Start from what the service promises, measure it from the user's side, and let the alerts, the dashboards and the on-call rota follow from the promise rather than from what the tools export.
CI/CD · Security · GitOps · 8 min
A pipeline that proves what it ships
Scanning, software bills of materials, signatures and admission control are four separate controls that only mean something together. How to build a delivery path where the cluster refuses anything the pipeline did not vouch for.