Operator Pattern #
Kubernetes is natively designed to manage stateless applications very reliably. If a web application pod fails, Kubernetes through built-in controllers like Deployment can easily destroy that pod, recreate a new pod, connect it to a Service, and enable liveness probe detection automatically. However, the story becomes very different when we try running stateful applications like relational databases (PostgreSQL, MySQL), message brokers (Apache Kafka, RabbitMQ), or search engines (Elasticsearch). These objects can’t be treated as anonymous entities replaceable at any time. They have strict startup order dependencies, need active-passive data replication synchronization management, require sensitive failover recovery mechanisms, plus complicated backup and data recovery schedules. The Operator Pattern comes to abstract and automate that human operational knowledge into binary code running natively inside Kubernetes.
Stateful Operational Problems and the Operator Solution #
Before the Operator Pattern existed, system administrators managed databases in Kubernetes using a combination of StatefulSets, PersistentVolumeClaims, and a set of external CronJob scripts. This manual approach has limitations when failures happen in the middle of the night:
PostgreSQL Operations Without an Operator (Manual):
1. The Primary Pod suffers a hardware failure at the Node level.
2. Kubernetes detects the dead Node, but doesn't dare do a failover
because of split-brain risks (both databases claim to be Primary).
3. The administrator must wake up and manually scan the replica data.
4. The administrator manually promotes the Replica Pod to the new Primary.
5. The administrator changes the application traffic route to the new Primary pod.
6. The risk of data loss is very high due to human negligence.
PostgreSQL Operations with the CloudNativePG Operator (Automatic):
1. The Primary Pod suffers a hardware failure at the Node level.
2. The Operator Controller detects the failure through the Watch API.
3. The Operator reactively promotes the most up-to-date Replica Pod (lowest replication lag) to the new Primary.
4. The Operator reconfigures other replica pods to point to the new Primary.
5. The Operator automatically updates the Service Selector to redirect traffic.
6. The failover process finishes in seconds without human intervention.
Operator Anatomy: CRDs and Custom Controllers #
Operators are built leveraging two core Kubernetes concepts: Custom Resource Definitions (CRDs) and Custom Controllers.
flowchart TD
User["Developer / Operator (kubectl apply)"] -->|"Send CR Manifest"| K8sAPI["Kubernetes API Server"]
K8sAPI -->|"Store State in"| Etcd["etcd Database"]
subgraph OperatorPod["Operator Pod (Deployment)"]
direction TB
WatchThread["Watch Loop (Observes CR & Pod Changes)"]
ReconcileThread["Reconcile Loop (Matches State)"]
WatchThread --> ReconcileThread
end
K8sAPI <-->|"API Watch Connection"| WatchThread
ReconcileThread -->|"Manage Built-in Resources"| ActiveResources["Pods, Services, PVCs, Secrets"]1. Custom Resource Definition (CRD) #
CRDs extend the Kubernetes API Server schema by registering new object kinds (Kinds). Once a CRD is registered in the cluster, the Kubernetes API Server treats those new objects as if they were built-in Kubernetes objects like Pods or Services. The API Server provides OpenAPI v3 validation to make sure user-entered parameters meet the requirements before being stored in the etcd database.
Here’s an example of a minimal CRD definition file for the PostgresCluster object:
# File: k8s-operator/postgres-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: postgresclusters.database.company.com
spec:
group: database.company.com
versions:
- name: v1alpha1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: ["instances", "storageSize"]
properties:
instances:
type: integer
minimum: 1
maximum: 5
storageSize:
type: string
pattern: '^[0-9]+(Gi|Mi|Ti)$'
version:
type: string
scope: Namespaced
names:
plural: postgresclusters
singular: postgrescluster
kind: PostgresCluster
shortNames:
- pgcluster
2. Custom Resource (CR) #
After the CRD is registered in the cluster, application developers can create instances of that object (called Custom Resources) with a simple YAML file:
# File: app/my-db-cluster.yaml
apiVersion: database.company.com/v1alpha1
kind: PostgresCluster
metadata:
name: billing-database
namespace: production
spec:
instances: 3
storageSize: "100Gi"
version: "15.4"
3. Custom Controller #
The CRD above is just a static data representation in the etcd database. Without a Custom Controller, no actions happen in the cluster. A Custom Controller is a standalone program usually written in Go (or Python/Ansible) running as a Deployment pod inside the cluster. Its main task is watching those PostgresCluster objects, then creating real Pods, Services, and Persistent Volume Claims (PVCs) matching the requested specs.
The Reconciliation Loop #
The core of an Operator’s intelligence lies in the Reconciliation Loop. This is an endless algorithm (infinite loop) continuously running to match the actual state in the cluster with the desired state requested by users.
The strict reconciliation process is described in the following flow diagram:
flowchart TD
WatchEvent["1. Detect Changes (Watch Event/CRD Trigger)"] --> ObserveState["2. Observe (Read Desired State & Actual State)"]
ObserveState --> AnalyzeState["3. Analyze (Calculate the Diff)"]
AnalyzeState --> ActState{"4. Is there a Diff?"}
ActState -- "Yes (Actual != Desired)" --> ReconcileAction["5. Reconcile (Execute Idempotent Actions)"]
ReconcileAction --> UpdateStatus["6. Update the Resource Status (CRD status)"]
ActState -- "No (Actual == Desired)" --> IdleState["7. Return to the Standby State (Idle/Watch)"]
UpdateStatus --> IdleState
IdleState -. "Wait for the Next Event" .-> WatchEventThe Absolute Law: Reconciliation Idempotency #
The reconciliation function (usually defined as the Reconcile(req ctrl.Request) function) must be Idempotent. That means, if the reconciliation function is called a hundred times in a row with the same input data, it must produce exactly the same final cluster state without triggering unwanted side effects (like recreating duplicate new objects).
For example, here’s writing wrong and correct reconciliation logic inside a Controller:
// ANTI-PATTERN: Non-idempotent logic (Triggers endless recreations)
func Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
// DON'T: Always create a new ServiceAccount without checking its existence first!
sa := &corev1.ServiceAccount{
ObjectMeta: metav1.ObjectMeta{Name: "db-service-account", Namespace: req.Namespace},
}
err := r.Create(ctx, sa) // Errors on the 2nd reconciliation because the resource already exists
return ctrl.Result{}, err
}
// ==============================================================================
// CORRECT: Idempotent Logic (Check existence before acting)
func Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
existingSA := &corev1.ServiceAccount{}
err := r.Get(ctx, types.NamespacedName{Name: "db-service-account", Namespace: req.Namespace}, existingSA)
if errors.IsNotFound(err) {
// Create a new one ONLY IF the object isn't found in the cluster
sa := &corev1.ServiceAccount{
ObjectMeta: metav1.ObjectMeta{Name: "db-service-account", Namespace: req.Namespace},
}
err = r.Create(ctx, sa)
return ctrl.Result{}, err
}
// If it already exists, skip the creation process and continue to the next resource
return ctrl.Result{}, nil
}
Popular Production-Level Operators #
The Kubernetes community has developed hundreds of ready-to-use Operators to simplify popular tool operations. Here are three essential Operators that must be installed in production clusters:
1. Prometheus Operator (Monitoring Category) #
The Prometheus Operator converts complex Prometheus configurations into declarative Kubernetes objects. Instead of manually editing prometheus.yml configuration files and restarting pods every time there’s a new application to monitor, we just create ServiceMonitor objects.
# File: monitoring/app-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: payment-api-monitor
namespace: production
labels:
release: prometheus-stack # This label is matched by the Prometheus Controller
spec:
selector:
matchLabels:
app: payment-api # Monitors Services with this label
endpoints:
- port: metrics # The Service port providing Prometheus metrics
interval: 15s
path: /metrics
2. Cert-Manager (TLS Security Category) #
Cert-Manager automates the SSL/TLS certificate lifecycle. It watches Certificate objects, communicates with Let’s Encrypt using the ACME protocol to validate domain ownership, downloads certificates, stores them as Kubernetes Secrets, and does automatic renewals before expiration dates.
# File: security/letsencrypt-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: api-ssl-certificate
namespace: production
spec:
secretName: api-tls-secret # The certificate is stored in this Secret to be mounted to the Ingress
issuerRef:
name: letsencrypt-production-issuer # Reference to the ACME Let's Encrypt ClusterIssuer
kind: ClusterIssuer
commonName: api.company.com
dnsNames:
- api.company.com
3. Strimzi (Kafka Message Broker Category) #
Apache Kafka is famously very hard to manage in Kubernetes because of the organizational complexity of Zookeeper, KRaft, Kafka brokers, topics, and access authorization. Strimzi wraps all that complexity into a declarative operator.
# File: kafka/kafka-cluster.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: prod-kafka-cluster
namespace: kafka-system
spec:
kafka:
version: 3.6.0
replicas: 3 # Create a 3-broker cluster for fault tolerance (HA)
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
storage:
type: persistent-claim
size: 500Gi
class: premium-rwo # Dynamic storage class
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 50Gi
OLM (Operator Lifecycle Manager) #
To manage the installation, version upgrades, and RBAC permissions of various Operators in one cluster, we’re recommended to use the Operator Lifecycle Manager (OLM). OLM acts as an internal App Store inside the Kubernetes cluster.
We can install OLM and monitor the OperatorHub.io catalog through the command line:
# 1. Install OLM into the Kubernetes cluster
kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.27.0/crds.yaml
kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.27.0/olm.yaml
# 2. Install Operators (e.g. cert-manager) via OLM using a Subscription
cat <<EOF | kubectl apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: cert-manager
namespace: operators
spec:
channel: stable
name: cert-manager
source: operatorhubio-catalog
sourceNamespace: olm
EOF
Decision Guide: When to Write Your Own Operator? #
Writing custom Operators requires no small amount of time investment and code maintenance. We must rationally evaluate our needs before starting to build our own Operator.
WE SHOULD WRITE our own Operator if:
✓ The internal application is a complex stateful system with complicated disaster recovery runbooks.
✓ There's a need for reactive integration automation with company external systems (e.g. DB provisioning in K8s must register IPs to external hardware firewalls).
✓ We want to provide a self-contained internal platform-as-a-service (PaaS) for developers in the company.
WE SHOULD NOT WRITE an Operator if:
✗ The application is stateless (just use Deployments, HPAs, and Kustomize/Helm).
✗ There's already a mature, community-supported open-source Operator for our chosen database.
✗ The developer team doesn't have the capacity or deep understanding of the Kubernetes API Client internal cycles.
Tools for Writing Operators #
If we decide to write a custom Operator, don’t build it from scratch using raw HTTP clients. Use the following industry-standard SDKs:
- Kubebuilder (Main Recommendation): The official Kubernetes SIGs framework using the Go language and the
controller-runtimelibrary. Provides the most complete code generator and scaffolding for CRD validation and unit testing using kubebuilder envtest. - Operator SDK: Part of the CNCF project managed by Red Hat. Supports creating Operators using the Go language, Helm manifests (without writing Go code), or Ansible (great for sysadmin teams).
- Kopf (Kubernetes Operator Framework): A Python-based framework for teams that don’t want to use Go but need quickly built custom controller logic.
Operator Pattern Implementation Anti-Patterns #
Avoid the following architectural design mistakes when operating or building Operators:
1. Blocking Reconciliation Status Failures #
Writing reconciliation logic doing synchronous HTTP API calls to third parties or running external backup processes taking hours directly in the controller’s main thread.
// ANTI-PATTERN: Doing slow synchronous external operations inside the Reconcile loop
func (r *PostgresReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
// DON'T: The database backup call blocks the reconciliation cycle for other pgcluster objects!
err := r.runDatabaseBackupToS3Blocking()
return ctrl.Result{}, err
}
Network Blocking Risks:
- All reconciliation cycles for other PostgresCluster objects in the cluster are delayed (*frozen*).
- If another database pod crashes, the Operator can't detect it quickly because the thread is busy processing S3 backups.
✓ SOLUTION: Run heavy processes in separate Kubernetes Jobs.
The Controller just creates a built-in Kubernetes 'Job' object, then ends the reconciliation
cycle immediately. When that Job finishes, the Controller receives new watch event
notifications to update its CRD status.
2. Giving Wildcard RBAC Access Rights (Over-Privileged Operators) #
Writing Operator security manifests with wildcard RBAC permissions (*) on apiGroups, resources, and verbs to ease initial installation.
# ANTI-PATTERN: A too-loose and dangerous Operator Role configuration
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: over-privileged-operator-role
rules:
- apiGroups: ["*"]
resources: ["*"] # DON'T: Offers fatal privilege escalation holes!
verbs: ["*"]
✓ SOLUTION: Apply the Least Privilege Principle.
Restrict Operator access rights to only the specific apiGroup and resources it manages.
For example, if the Operator only manages PostgresClusters, it only needs read/write permissions
on pgclusters, pods, services, secrets, and pvcs. Never give permissions
to cluster-wide resources (like nodes or namespaces) unless truly needed.
Operator Pattern Implementation Audit Checklist #
Use this checklist to evaluate the operational health of Operators in your cluster:
OPERATOR RBAC & SECURITY HARDENING:
□ Operator access rights (ClusterRoles) are restricted to only the specific managed resources (least privilege).
□ Operator containers run using a non-root SecurityContext ('runAsNonRoot: true').
□ Operators are isolated into dedicated namespaces and not mixed with business application workloads.
□ Sensitive credentials for managed databases are stored encrypted using K8s Secrets.
CONTROLLER & RECONCILIATION DESIGN:
□ All logic inside Reconcile functions is idempotent and safe to execute repeatedly.
□ No synchronous blocking external API calls exist in the main loop.
□ Heavy processes (like backup/restore) are delegated to separate Job objects.
□ Subresource status mechanisms on CRDs are used to separately record reconciliation progress.
CRD VALIDATION & LIFECYCLE:
□ CRD spec data structures have strict OpenAPI v3 validation (data types, regex patterns, minimum limits).
□ Official Kubebuilder / Operator SDK binaries are used to guarantee code structure compliance.
□ Operator version upgrades are tested gradually in staging environments using OLM or Helm.
□ Cleanly deleting Custom Resources (CRs) also cleans up related built-in resources via ownerReferences.
Summary #
- Automate Operational Knowledge — The Operator Pattern translates human operational runbooks (failover, backup, cluster scaling) into automated software.
- CRDs and Controllers Are One Unit — CRDs define new declarative objects in etcd, while Controllers materialize those object specs in the real cluster.
- Idempotency Is the Main Law — Make sure your reconciliation logic is safe to execute repeatedly without triggering cluster resource duplication errors.
- Use Jobs for Heavy Tasks — Avoid blocking reconciliation threads; delegate data recovery and backup tasks to separate Kubernetes
Jobobjects.- Use Community Solutions First — Explore OperatorHub.io before writing custom Operators; the community has provided mature Operators for most databases.
- Restrict Operator RBAC Permissions — Protect your cluster from privilege escalation risks by limiting Operator binary access permissions as minimally as possible.
← Previous: Local Development Tools Next: Managed Kubernetes →