Service #

As we know from the Kubernetes network model discussion, every Pod gets its own unique IP address. However, we must also be aware of one important reality: Pods in Kubernetes are ephemeral. Pods can be destroyed and recreated at any time due to node failures, autoscaling, or application updates (rolling updates). Every time a new Pod is born, it carries a new, unpredictable IP address.

This situation creates a big challenge. If we have a group of backend Pods receiving transactions (e.g. 3 replicas) and a group of frontend Pods that need to send requests to those backends, how does the frontend know which IP to contact? We can’t update the frontend config file every time one backend Pod dies or changes IP.

This is where Service comes in as the problem-solving solution. A Service is a Kubernetes abstraction object defining how to access a group of Pods stably. A Service provides one fixed virtual IP address (ClusterIP) and one internal DNS name that never changes as long as the Service object exists.

This article deeply dissects how Services work, the label matching mechanism, detailed differences between the four main Service types, the Headless Service concept, and Service network debugging methods.

How Services Find Pods: Label Selectors #

A Service isn’t physically connected to a specific Pod name. Instead, Services use a dynamic lookup mechanism called a Label Selector.

When we create a Service, we specify the list of labels the target Pods must have. The Service then monitors all Pods in the same namespace, automatically filtering which Pods have matching labels to include in its traffic distribution list.

Let’s visualize how a Service object bridges traffic to several Pods through the following diagram:

flowchart TD
    Client["Frontend Pod (Request)"] --> Svc["Service: backend-svc (10.96.0.10:80)"]
    Svc -. Selector: app=backend .-> Pod1["Pod backend-0 (10.244.1.4:8080)"]
    Svc -. Selector: app=backend .-> Pod2["Pod backend-1 (10.244.2.7:8080)"]
    Svc -. Selector: app=backend .-> Pod3["Pod backend-2 (10.244.1.9:8080)"]

Endpoints and EndpointSlices #

Behind the scenes, when we create a Service with a selector, the Kubernetes API Server automatically creates a companion object called Endpoints. This Endpoints object stores the raw list of IP addresses and ports of all active Pods matching our Service’s selector.

# Command to check a Service's Endpoints
kubectl get endpoints backend-service

The resulting output:

NAME              ENDPOINTS
backend-service   10.244.1.4:8080,10.244.1.9:8080,10.244.2.7:8080

If one backend Pod crashes or fails its readiness probe, the Kubernetes Controller Manager immediately detects the failure and removes the broken Pod’s IP from the Endpoints list. That way, the Service won’t send traffic to unhealthy Pods.

[!NOTE] On large clusters with thousands of Pods, a single Endpoints object can bloat enormously, triggering etcd performance issues from large manifest transfers on every IP change. To solve this, modern Kubernetes (version 1.21+) introduced EndpointSlice. This object splits the Pod IP list into several small segments (slices) of at most 100 endpoints per object to maintain cluster scalability.

The Four Main Service Types #

Kubernetes provides four Service types tailored to our application’s network traffic exposure needs.

flowchart TD
    Types["KUBERNETES SERVICE TYPES"]
    Types --> ClusterIP["ClusterIP (Internal cluster access - Default)"]
    Types --> NodePort["NodePort (Opens a physical port on every worker node)"]
    Types --> LoadBalancer["LoadBalancer (Creates an external cloud load balancer)"]

1. ClusterIP (Default) #

This is the built-in Service type if we don’t explicitly set the type property. ClusterIP allocates an internal cluster virtual IP address only reachable from inside the cluster itself (by other Pods or from inside worker nodes).

Example ClusterIP Manifest #

apiVersion: v1
kind: Service
metadata:
  name: auth-service
  namespace: production
spec:
  type: ClusterIP
  selector:
    app: auth
  ports:
    - name: http
      protocol: TCP
      port: 80 # Virtual Service port
      targetPort: 3000 # Our application container port

Internal cluster applications can reach this service using the instant DNS name: http://auth-service.production.svc.cluster.local or simply http://auth-service if in the same namespace.

2. NodePort #

NodePort is used when we want to open cluster service access from outside networks without a cloud load balancer. Kubernetes opens one special static port from the default range 30000-32767 on every physical worker node in our cluster.

Example NodePort Manifest #

apiVersion: v1
kind: Service
metadata:
  name: web-nodeport-service
  namespace: production
spec:
  type: NodePort
  selector:
    app: frontend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30080 # Physical node port (must be in the 30000-32767 range)

The NodePort Network Workflow #

When an external client contacts any worker node’s physical IP at port 30080 (e.g. http://192.168.1.10:30080), that worker node’s kernel captures the packet, routes it to the virtual ClusterIP Service web-nodeport-service, and forwards it to the target frontend Pod.

3. LoadBalancer #

The LoadBalancer type is the industry-standard method for exposing internal cluster applications directly to the public internet in cloud provider environments (like AWS, GCP, or Azure).

Example LoadBalancer Manifest #

apiVersion: v1
kind: Service
metadata:
  name: public-ingress-service
  namespace: production
spec:
  type: LoadBalancer
  selector:
    app: ingress-nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

The LoadBalancer Network Workflow #

When we apply this manifest on AWS EKS, the cloud provider driver in EKS automatically creates a new physical Load Balancer object in our AWS account (e.g. an AWS Network Load Balancer). AWS provides a stable public DNS address or external IP. Traffic from outside enters the physical cloud Load Balancer, gets distributed to the cluster’s automatic NodePort, and finally lands on our application Pods.

4. ExternalName #

This type doesn’t do virtual load balancing, node port creation, or virtual IP allocation. ExternalName acts purely as an internal cluster DNS alias (CNAME record) routing traffic to external services outside the cluster.

Example ExternalName Manifest #

apiVersion: v1
kind: Service
metadata:
  name: cloud-database-alias
  namespace: production
spec:
  type: ExternalName
  externalName: my-prod-db.c123456789.ap-southeast-1.rds.amazonaws.com

When an application inside the cluster tries to reach cloud-database-alias.production.svc.cluster.local, the internal Kubernetes DNS resolver (CoreDNS) immediately responds by returning that AWS RDS CNAME value, so our application can directly handshake with AWS RDS without going through a cluster proxy intermediary.

Headless Services (clusterIP: None) #

Sometimes we don’t want a single virtual ClusterIP acting as a load balancer. We might be running a distributed database cluster (like Cassandra, MongoDB, or Redis) where each database replica must be reachable directly by other replicas for internal data replication and state synchronization.

To facilitate this need, we use a Headless Service by setting clusterIP: None.

Example Headless Service Manifest #

apiVersion: v1
kind: Service
metadata:
  name: database-headless
  namespace: production
spec:
  clusterIP: None # Mandatory: Makes this Service Headless
  selector:
    app: postgres
  ports:
    - port: 5432
      targetPort: 5432

How Does Headless DNS Work? #

If we do a normal DNS lookup against a ClusterIP Service, CoreDNS returns one virtual ClusterIP. However, if we do a DNS lookup against the Headless Service database-headless, CoreDNS directly returns the entire list of real Pod IP addresses behind that Service.

# Doing a DNS lookup against the Headless Service
nslookup database-headless.production.svc.cluster.local

The DNS response result:

Name:   database-headless.production.svc.cluster.local
Address: 10.244.1.4  # IP of Pod postgres-0
Address: 10.244.1.9  # IP of Pod postgres-1
Address: 10.244.2.7  # IP of Pod postgres-2

When combined with StatefulSets, Headless Services let us have separate stable DNS subdomains for each ordinal Pod index:

  • Pod 0: postgres-0.database-headless.production.svc.cluster.local
  • Pod 1: postgres-1.database-headless.production.svc.cluster.local

Services Without Selectors #

There’s a special scenario where we want to use the Kubernetes Service abstraction, but the target service isn’t inside the cluster (e.g. a legacy database server on an on-premise VM). We can create a Service without defining the .spec.selector parameter, then manually create an Endpoints object with the exact same name.

# 1. Service manifest without a selector
apiVersion: v1
kind: Service
metadata:
  name: legacy-mysql-service
  namespace: production
spec:
  ports:
    - port: 3306
      targetPort: 3306
---
# 2. Manual Endpoints manifest (Name must match the Service)
apiVersion: v1
kind: Endpoints
metadata:
  name: legacy-mysql-service
  namespace: production
subsets:
  - addresses:
      - ip: 192.168.10.150 # Physical database IP outside Kubernetes
    ports:
      - port: 3306

This way, applications inside the cluster just contact legacy-mysql-service as if it were a regular internal cluster database. If that database is later migrated into the cluster, we just add a selector to the Service without changing our application code configuration.

Sticky Sessions: Session Affinity #

By default, a Kubernetes Service distributes traffic to target Pods randomly using an internal load balancing algorithm (usually iptables random mode). However, if our application needs users to always connect to the same container to maintain local session state, we can enable the sessionAffinity property.

spec:
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800 # Keep the session for 3 hours (10800 seconds)

[!WARNING] The sessionAffinity: ClientIP feature works at network layer 4 (based on the client’s source IP address). If our external clients sit behind the same office NAT proxy gateway, then all users from that office get routed to one same container Pod, triggering load imbalance on our cluster.

Service Networking Anti-Patterns vs Solutions #

Let’s study the most common mistakes when designing Services, along with how to fix them.

Anti-Pattern 1: Creating a LoadBalancer Service for Every Internal Microservice #

We have a microservices architecture with 15 internal backend services. We configure all 15 backend Services with the LoadBalancer type so developer teams can easily access each service instantly from outside.

The Bad Consequences #

Each LoadBalancer Service instructs the cloud provider to create one new physical Load Balancer unit in the cloud (e.g. AWS NLB). If we create 15 LoadBalancer Services, we get billed for 15 very expensive physical load balancers every month. Additionally, exposing internal databases or APIs directly to the outside internet opens very dangerous security holes.

Solution Code (ClusterIP + a Single Ingress Controller) #

We must configure all internal backend Services using the default ClusterIP type. To expose public APIs outside the cluster, we deploy one Ingress Controller unit (which only needs one LoadBalancer Service), then create domain/path-based routing rules forwarding to our internal ClusterIPs.

# SOLUTION: Example single Ingress routing to many internal ClusterIP backends
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-gateway-ingress
  namespace: production
spec:
  ingressClassName: nginx
  rules:
    - host: api.mycompany.com
      http:
        paths:
          - path: /users
            pathType: Prefix
            backend:
              service:
                name: user-service # Internal ClusterIP service
                port:
                  number: 80
          - path: /payments
            pathType: Prefix
            backend:
              service:
                name: payment-service # Internal ClusterIP service
                port:
                  number: 80

Anti-Pattern 2: Losing the Original Sender IP on LoadBalancer Services #

We deploy a LoadBalancer Service to receive web traffic. When analyzing our NGINX web access logs, we find all recorded sender IP addresses are internal worker node host IPs, not the real client IPs from the outside internet.

The Bad Consequences #

This happens because by default Kubernetes uses the externalTrafficPolicy: Cluster traffic policy. If a data packet lands on Node A, but our application Pod is on Node B, Node A performs Source NAT (SNAT) to forward the packet to Node B. This SNAT process overwrites the client’s original IP with Node A’s internal IP. We lose the ability to do security auditing, geo-blocking, or IP spam prevention.

Solution Code (Using externalTrafficPolicy: Local) #

We must force the Service to only forward traffic directly to Pods on the same node where the packet landed from the Load Balancer, by setting externalTrafficPolicy: Local.

# SOLUTION: Preserving the Original Client IP
apiVersion: v1
kind: Service
metadata:
  name: web-ingress-service
  namespace: production
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local # Preserves the original external client Source IP
  selector:
    app: frontend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  • By setting Local, the physical cloud Load Balancer only sends traffic to worker nodes actively running our target Pods. No cross-node SNAT happens, so the original client IP is fully readable by our application containers.

Practical Service Debugging Guide #

If our application fails to reach a Service, we can diagnose the issue in a targeted way using the following CLI commands:

1. Verify the Virtual IP Address (ClusterIP) #

Check whether the Service object successfully got an internal cluster virtual IP allocation:

kubectl get svc -n production

If the CLUSTER-IP column is None (and it’s not an intentional Headless Service), or empty, the cluster has run out of internal IP allocation range (Service CIDR pool).

2. Verify Endpoints Availability #

Check whether healthy Pods are registered behind that Service:

kubectl describe svc <service-name> -n <namespace>

Review the Endpoints: output line. If that line is empty or <none>, check the following:

  • Is the Service’s label selector written correctly and matching the labels on the Pods?
  • Is our target Pod in Running status?
  • Is our target Pod failing the readinessProbe check? (Pods failing readiness probes are automatically removed from the Endpoints list).

3. Test DNS Resolution from Inside the Cluster #

Launch a temporary debugger pod and run an nslookup test to make sure CoreDNS works normally:

kubectl run dns-test --rm -i --tty --image=tutum/dnsutils -- nslookup auth-service

If nslookup fails to return a ClusterIP, check the CoreDNS pod logs in the kube-system namespace.

Summary #

  • Services guarantee access point stability: They provide a fixed virtual IP (ClusterIP) and internal DNS name connecting dynamic, ephemeral Pods.
  • Selectors dynamically connect Services and Pods: Services use label selectors to automatically filter target Pods and register their IPs in Endpoints/EndpointSlice objects.
  • Pick the Service type per need: Use ClusterIP for internal backends, NodePort for physical local port exposure, LoadBalancer for automatic cloud integration, and ExternalName for external CNAME aliases.
  • Headless Services for database clusters: Disable the virtual ClusterIP (clusterIP: None) to let DNS queries directly return all Pods’ real IPs; crucial for stateful database clusters.
  • Preserve the original client IP with externalTrafficPolicy: Local: Change the default policy setting to Local on LoadBalancer Services to prevent external client IPs from being overwritten by worker node host Source NAT (SNAT) processes.
  • Use describe and dnsutils for debugging: The first step when Service connections fail is checking Endpoints table matching and ensuring internal DNS query resolution works smoothly.

← Previous: Pod-to-Pod Communication   Next: DNS & Service Discovery →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact