Multi-Tenancy #

When an organization grows and starts adopting Kubernetes widely, they face one crucial architectural decision: should we provide a separate small physical cluster for every developer team (single-tenant clusters), or combine all teams into one large shared physical cluster (multi-tenant clusters)? The single cluster per team approach is very expensive because it triggers unused node capacity waste and multiplies the infrastructure maintenance burden on platform teams. Conversely, combining many teams into one big cluster (called Multi-Tenancy) offers very high cost efficiency and simplifies policy governance. However, the challenge is security: how do we ensure team A can’t peek at team B’s sensitive data, bugs in team C’s applications don’t paralyze team D’s performance, and one team’s administrators can’t modify global cluster configurations? This article discusses logical vs physical isolation concepts, access restriction architecture, zero-trust networking isolation, and virtual cluster implementation using vCluster.

Soft Multi-Tenancy vs Hard Multi-Tenancy #

The multi-tenancy approach in Kubernetes is divided into two main categories based on isolation levels and trust boundaries:

+-------------------------------------------------------------------------+
|                      SOFT MULTI-TENANCY (LOGICAL)                       |
|   - Uses Namespaces, RBAC, NetworkPolicies, and ResourceQuotas          |
|   - All Tenants share the same API Server and Host Kernel               |
|   - Vulnerability: Host kernel exploit security holes                   |
|   - Suitable for: Internal teams under one trusted organization         |
+-------------------------------------------------------------------------+
|                      HARD MULTI-TENANCY (PHYSICAL/VIRTUAL)              |
|   - Uses Separate Physical Clusters or Virtual Clusters (vCluster)      |
|   - Physically/logically separate Control Plane & etcd database         |
|   - Advantage: High security, minimal privilege escalation risks        |
|   - Suitable for: SaaS companies serving external customers             |
+-------------------------------------------------------------------------+

In the Soft Multi-Tenancy scenario, if a hacker gains root access rights inside a container in one namespace, that hacker can potentially exploit host Linux kernel security holes (like container escapes) to take over the entire Node and read Secrets belonging to other tenants running on the same Node.

For Hard Multi-Tenancy scenarios, physical isolation or high-level virtual encapsulation must be applied to tightly close all cross-tenant exploit holes.

Namespaces as the Basic Isolation Unit #

In Kubernetes, Namespaces are the fundamental logical isolation unit. Namespaces divide a single cluster into several logical regions. However, creating namespaces alone doesn’t immediately secure the cluster. We must equip Namespaces with Pod Security Standards (PSS), ResourceQuota, and LimitRange policies.

Protected Namespace Manifest File #

Here’s an example of a hardened namespace configuration manifest to restrict tenant container privileges:

# File: k8s/namespaces/team-backend-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: team-backend
  labels:
    team: backend-engineers
    environment: production
    # Strictly enable the 'restricted' PSS rule (enforce)
    # This policy automatically rejects Pods running as root, 
    # using host namespaces, or requesting privileged capabilities
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest

---
# Limit cumulative resource quotas to prevent Node capacity monopolies
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-backend-quota
  namespace: team-backend
spec:
  hard:
    requests.cpu: "12"
    requests.memory: "24Gi"
    limits.cpu: "24"
    limits.memory: "48Gi"
    pods: "40"
    services: "15"
    persistentvolumeclaims: "10"

---
# Guarantee default resource specs to prevent pods entering BestEffort QoS
apiVersion: v1
kind: LimitRange
metadata:
  name: team-backend-limitrange
  namespace: team-backend
spec:
  limits:
  - type: Container
    default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "200m"
      memory: "256Mi"

Access Authorization via Tenant-Level RBAC #

The Kubernetes authentication system must not allow developers from one team to modify or even read resources belonging to other teams. We must restrict developer access rights using Role-Based Access Control (RBAC) rules explicitly directed at the Namespace level (Namespace-scoped RBAC).

By default, if we don’t create RoleBindings in a namespace for specific users, Kubernetes applies the Default-Deny security principle (access totally denied).

# File: k8s/rbac/team-backend-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: team-backend-developers-binding
  namespace: team-backend # Restrict the role binding ONLY to this namespace!
subjects:
- kind: Group
  name: "backend-developers-group" # The OIDC group name from Google / Azure AD integration
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  # Uses the built-in default 'admin' ClusterRole 
  # But because it's bound using a RoleBinding (not a ClusterRoleBinding), 
  # KSA users only act as admins inside the 'team-backend' namespace.
  name: admin 
  apiGroup: rbac.authorization.k8s.io

With the configuration above, if a developer in backend-developers-group tries running the following commands:

# Success: The developer has admin rights in their own namespace
kubectl get pods -n team-backend

# Denied: The API Server rejects access because of the missing binding in other namespaces
kubectl get pods -n team-frontend
# Error from server (Forbidden): User "dev-user" cannot list resource "pods" in API group "" in the namespace "team-frontend"

NetworkPolicies: Zero-Trust Network Isolation #

By default, Kubernetes adheres to the flat network model principle. That means every Pod in the cluster can freely send data packets to each other across namespaces without any restrictions. This is a very fatal security hole in production environments. An insecure test microservice pod in the dev namespace can easily directly call the sensitive production database pod in the prod-apps namespace.

We must apply NetworkPolicy policies of the Default-Deny All Ingress & Egress type in every tenant namespace, then explicitly open connection paths only for legitimate services.

# File: k8s/networks/team-backend-netpol.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: team-backend-secure-policy
  namespace: team-backend
spec:
  podSelector: {} # The empty {} character means this rule applies to ALL Pods in this namespace
  policyTypes:
  - Ingress
  - Egress
  
  ingress:
  # Rule 1: Allow traffic from Pods in the same Namespace
  - from:
    - podSelector: {}
  
  # Rule 2: Allow HTTP/gRPC traffic only from the Ingress Controller Namespace
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx # Only from the Nginx Ingress namespace
    ports:
    - port: 8080
      protocol: TCP
      
  # Rule 3: Allow metric scraping from the cluster Monitoring system
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring-system
          
  egress:
  # Rule 1: Allow domain name resolution (DNS) access to the cluster CoreDNS
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
      
  # Rule 2: Allow outbound connections to the public internet
  # But BLOCK access to other internal cluster CIDR IP ranges
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 10.0.0.0/8       # Block access to internal VPC / Cluster A IPs
        - 172.16.0.0/12    # Block access to internal VPC / Cluster B IPs
        - 192.168.0.0/16

vCluster: Virtual Cluster Solutions for Hard Multi-Tenancy #

Although Namespaces and RBAC are very reliable for standard isolation, they have a critical limitation: all tenants share one etcd database and one global API Server.

This limitation triggers several problems in large organizations:

• No CRD Access: Tenants can’t register their own Custom Resource Definitions (CRDs) because CRDs are global cluster-level objects (cluster-wide). If Team A registers a database CRD v1, Team B can’t register v2.
• API Server Overload Risks: One tenant continuously doing wrong API queries can burden the main API Server, causing operational slowness for all other tenants in the cluster.

vCluster (Virtual Clusters) revolutionarily solves this problem. vCluster runs isolated virtual Kubernetes control planes inside one namespace of the host cluster. vClusters have their own API Servers, etcd, and controller-managers running as regular Pods.

flowchart TD
    subgraph HostCluster["Main Physical Cluster (Host Cluster)"]
        direction TB
        HostAPI["Host API Server"]
        HostNodes["Physical Nodes (Node 1, Node 2)"]
        
        subgraph NamespaceTenant["Namespace: vcluster-team-frontend"]
            direction LR
            vAPI["Virtual API Server (vCluster)"]
            vetcd["Virtual etcd (sqlite / etcd Pod)"]
            vSyncer["vCluster Syncer Controller"]
        end
    end
    
    TenantDev["Frontend Developer (kubectl)"] -->|"Send manifests to Port 8443"| vAPI
    vAPI <-->|"Store State"| vetcd
    
    vSyncer -->|"Read Virtual Pods & Connect"| HostAPI
    HostAPI -->|"Schedule Real Physical Pods on"| HostNodes

The vCluster Synchronization Mechanism (Syncer) #

vClusters don’t have their own physical Nodes. When vCluster users send Pod manifests to the Virtual API Server:

1. The Virtual API Server stores the Pod definition in its own virtual etcd database.
2. The vCluster Syncer component monitors the Virtual API Server, detects new Pod creations, then rewrites the Pod names with a unique prefix (e.g. pod-frontend-vcluster-xyz), and sends them to the physical host cluster’s API Server.
3. The host cluster’s API Server schedules those physical Pods on main physical Nodes. From the host cluster’s perspective, vClusters are just regular pods running inside one isolated namespace.

Multi-Tenancy Solution Comparison Table #

Here’s a comparison matrix to help you choose the right isolation method:

Decision Guides: When Should Physical Clusters Be Separated? #

Although multi-tenancy is very efficient, there are certain conditions where we must physically separate clusters:

1. Regulatory Compliance: Legal frameworks like PCI-DSS (for credit card transaction processing) or HIPAA (for health medical records) often require strict physical isolation at the network and compute levels. Credit card transaction workloads must not be mixed with marketing promotion blog applications in the same cluster.
2. Zero Trust Boundaries: If we provide container platforms for external/hostile or untrusted customer organizations (like public SaaS services), namespace isolation alone isn’t secure enough. Host kernel security hole exploits can leak data between customers.
3. Cluster Scale and Capacity Limits: Kubernetes API Servers and etcd have optimal processing capacity limits (usually optimal up to a maximum of 5,000 Nodes or 150,000 Pods). If the combined total workload of all company teams exceeds this scale limit, we must divide the load into several separate physical clusters.

Multi-Tenancy Implementation Anti-Patterns #

Avoid the following two architectural mistakes when configuring multi-tenancy at the production level:

1. Allowing Tenants to Run with Cluster-Wide RBAC Permissions #

Giving administrative access rights or too-loose roles outside the tenant namespace.

# ANTI-PATTERN: Binding a tenant RoleBinding to a global ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding # DON'T: This gives access to the entire cluster!
metadata:
  name: team-frontend-cluster-admin
subjects:
- kind: Group
  name: "frontend-developers"
roleRef:
  kind: ClusterRole
  name: admin # Admin access for all cluster namespaces!

Operational Risks:
- Frontend team developers can accidentally read sensitive Secrets in the backend namespace or delete production database pods belonging to other teams.
✓ SOLUTION: Always use 'RoleBindings' (not 'ClusterRoleBindings') when attaching built-in ClusterRoles (like admin or edit) to tenant subjects so their access scope is limited to one namespace only.

2. Ignoring Cross-Tenant Network Isolation (Flat Namespace Isolation) #

Dividing teams into several separate namespaces and installing strict RBAC, but not installing a single NetworkPolicy rule in the cluster.

Operational Risks:
- Even though frontend developers don't have 'kubectl' access to the backend namespace.
- Their frontend web applications running in the cluster can still send HTTP/TCP network queries directly to internal backend database ports (e.g. calling 'http://postgres-service.backend-namespace.svc.cluster.local:5432').
- If the frontend web application container is compromised by hackers, they can scan and steal all internal cluster database data without firewall obstacles.
✓ SOLUTION: Apply default-deny NetworkPolicy policies in every tenant namespace and explicitly open connection routes only to registered pod IPs.

Multi-Tenancy Audit Checklist #

Use this checklist to audit the multi-tenancy isolation level in your production cluster:

LOGICAL ISOLATION & NAMESPACE POLICIES:
  □ Every team or application is isolated in a dedicated Namespace.
  □ Pod Security Standards (PSS) are configured at the 'restricted' level in every tenant namespace.
  □ ResourceQuota objects are installed to limit cumulative CPU/RAM consumption per namespace.
  □ LimitRanges are configured to guarantee default container requests/limits allocations.

ACCESS AUTHORIZATION (RBAC):
  □ No regular tenants have 'ClusterRoleBinding' access permissions to the global cluster level.
  □ Tenant access permissions are bound using 'RoleBindings' to limit scope to one namespace.
  □ Privilege usage (like creating CRDs, ClusterRoles, or ServiceAccounts) is restricted to platform teams only.
  □ Engineer access is integrated with official company OIDC Groups (not static serviceaccount tokens).

NETWORK ISOLATION (NETWORKPOLICIES):
  □ Default-deny ingress & egress NetworkPolicy policies are active in every tenant namespace.
  □ Cross-namespace communication is totally denied unless explicitly opened.
  □ Pod outbound internet access is restricted to only legitimate destination ports and domains.
  □ Prometheus metric scraping traffic is restricted to only valid monitoring system namespaces.

Summary #

• Apply Layered Isolation — Guarantee multi-tenancy security by combining logical (Namespaces), access (RBAC), network (NetworkPolicies), and quota (ResourceQuotas) isolation.
• NetworkPolicies Are Mandatory — Don’t let your cluster run with a flat network; must enable default-deny NetworkPolicies between namespaces from the start of cluster construction.
• Restrict RBAC Scope via RoleBindings — Always use RoleBindings instead of ClusterRoleBindings to restrict developer access permissions so they’re isolated to one namespace only.
• vCluster for Independent CRDs — Leverage vClusters if your tenants need their own virtual control planes to install custom CRDs without buying additional physical clusters.
• Separate Clusters for Compliance — Immediately create separate physical clusters if your applications are bound by strict legal compliance rules (like PCI-DSS) or serve untrusted external customers.
• Enable Pod Security Standards Label namespaces with pod-security.kubernetes.io/enforce: restricted to prevent tenants from running malicious containers with privileged host access rights.

← Previous: Disaster Recovery   Next: Production Anti-Patterns →