DNS & Service Discovery #

In a dense, dynamic Kubernetes cluster, hundreds of containers can be started, stopped, and rescheduled within seconds. We already know the Service object acts as a virtual load balancer providing a stable IP address (ClusterIP) for a group of dynamic Pods. However, the question remains: how does our application find that Service’s virtual IP address in the first place?

Contacting a virtual IP address directly isn’t how modern systems work. We need an automatic, centralized, dynamic Service Discovery mechanism. In Kubernetes, this mechanism is fully handled by the cluster’s internal DNS (Domain Name System) system.

Cluster DNS dynamically maps human-friendly service names (like auth-service) to their active virtual IP addresses. When we create, delete, or move Services, the cluster DNS server automatically updates its records in real-time. This article covers the cluster DNS architecture, how internal DNS queries work, container resolver configuration, DNS performance optimization techniques, and practical guides for detecting its disruptions.

The CoreDNS Architecture: The Cluster’s Service Discovery Brain #

Since Kubernetes version 1.13, CoreDNS has been the default internal DNS server standard for Kubernetes clusters, replacing the old Kube-DNS. CoreDNS is a modular high-speed DNS server written in Go and is a graduated project under the CNCF umbrella.

CoreDNS is deployed as a standard Deployment (usually 2 or more replicas for high availability) in the kube-system namespace. To connect application Pods to CoreDNS, Kubernetes exposes that CoreDNS Deployment through a special Service named kube-dns.

# Checking the kube-dns Service in the kube-system namespace
kubectl get service -n kube-system kube-dns

The output of the command above:

NAME       TYPE        CLUSTER-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   53/UDP,53/TCP,9153/TCP   365d

The stable ClusterIP of this kube-dns Service (e.g. 10.96.0.10) acts as the DNS resolution anchor for all Pods in the cluster.

How Do Pods Reach CoreDNS? #

Every time the Kubelet creates a new Pod container on a worker node, the Kubelet automatically injects the cluster DNS resolver configuration into the container’s internal /etc/resolv.conf file.

Let’s peek at the standard /etc/resolv.conf contents inside a Pod:

nameserver 10.96.0.10
search production.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

nameserver 10.96.0.10: Directs all DNS queries from inside the container to CoreDNS’s virtual IP address.
search ...: Defines the local domain suffix search list. If we query a short name (e.g. database), the resolver tries completing it sequentially with these suffixes.
options ndots:5: A critical rule telling the resolver that if a queried domain name contains fewer than 5 dots, it must be treated as a local cluster domain first, then searched on external DNS only if that fails.

Standard DNS Record Formats in Kubernetes #

CoreDNS automatically watches the Kubernetes API for every Service object created, changed, or deleted, then creates the corresponding DNS records. Here are the standardized DNS record formats in Kubernetes:

1. A/AAAA Records for Services (ClusterIP) #

Every ClusterIP Service gets an A record (for IPv4) or AAAA record (for IPv6) with this Fully Qualified Domain Name (FQDN) format:

FQDN Format:
  <service-name>.<service-namespace>.svc.<cluster-domain>

Real example:
  payment-service.finance.svc.cluster.local

If our application queries payment-service.finance.svc.cluster.local, CoreDNS directly returns that Service’s virtual IP 10.96.120.45.

2. A Records for Headless Services #

If we use a Headless Service (with the clusterIP: None parameter), DNS queries against that Service’s domain don’t return one virtual IP, but return the entire list of real Pod IP addresses bound to that Service.

database-service.production.svc.cluster.local  ──>  10.244.1.15
                                               ──>  10.244.2.22
                                               ──>  10.244.1.30

3. A Records for Individual Pods #

Besides Services, Kubernetes also provides special DNS records for individual Pods. The DNS address format replaces every dot character in the Pod’s IP with a dash:

Pod DNS Format:
  <pod-ip-with-dashes>.<pod-namespace>.pod.<cluster-domain>

Real example (Pod IP 10.244.1.15 in the production namespace):
  10-244-1-15.production.pod.cluster.local

4. SRV Records for Named Ports #

If our Service explicitly defines port names in its manifest, CoreDNS creates SRV records for discovering the port numbers and protocols in use:

SRV Format:
  _<port-name>._<protocol>.<service-name>.<namespace>.svc.cluster.local

Real example (Port named "http-api" with TCP protocol):
  _http-api._tcp.payment-service.finance.svc.cluster.local

Short Name Lookups vs FQDN #

Thanks to the search parameter in /etc/resolv.conf, developers don’t have to write the full FQDN like payment-service.finance.svc.cluster.local in their application code. We can use short names appropriate to their location context.

Let’s look at a simulation of how the resolver processes short names through the following diagram:

flowchart TD
    Start["Pod in the 'production' namespace queries the short name: 'auth'"] --> Step1{"Try auth.production.svc.cluster.local"}
    Step1 -- "Found (A Record)" --> Success["Return IP: 10.96.0.22 (Success)"]
    Step1 -- "Failed (NXDOMAIN)" --> Step2{"Try auth.svc.cluster.local"}
    Step2 -- "Found" --> Success
    Step2 -- "Failed" --> Step3{"Try auth.cluster.local"}
    Step3 -- "Found" --> Success
    Step3 -- "Failed" --> Step4{"Query the upstream external DNS"}
    Step4 -- "Success" --> SuccessExternal["Return the Public IP"]
    Step4 -- "Failed" --> Failure["Return the NXDOMAIN Error"]

Same Namespace: If the backend Pod and frontend Pod are in the same namespace (e.g. both in the production namespace), the frontend can just call the backend with the short name:
```
http://auth-service/api/login
```
Cross Namespace: If the frontend Pod is in the frontend namespace while the backend is in the backend-api namespace, the short name auth-service won’t resolve because the DNS resolver only searches the local frontend namespace first. We must include at least the target namespace name:
```
http://auth-service.backend-api/api/login
```
Or, more safely, use the full FQDN to prevent ambiguity:
```
http://auth-service.backend-api.svc.cluster.local/api/login
```

Customizing DNS Configuration at the Pod Level #

Kubernetes lets us override or extend the default DNS configuration for specific Pods using the dnsPolicy and dnsConfig properties on the Pod spec.

apiVersion: v1
kind: Pod
metadata:
  name: custom-dns-pod
  namespace: production
spec:
  dnsPolicy: ClusterFirst # Uses the internal cluster CoreDNS (Default)
  dnsConfig:
    nameservers:
      - 1.1.1.1 # Adds an extra external DNS fallback
    searches:
      - custom-company-domain.internal # Adds a special search suffix
    options:
      - name: ndots
        value: "2" # Lowers the ndots value to optimize external DNS query performance
      - name: timeout
        value: "2"
  containers:
    - name: app-container
      image: nginx:alpine

Available dnsPolicy Options #

ClusterFirst (Default): All DNS queries that don’t match the local cluster domain get forwarded to the external upstream DNS configured on the worker node host.
ClusterFirstWithHostNet: Must be explicitly enabled if our Pod uses the hostNetwork: true configuration so the Pod can still use cluster CoreDNS for name resolution.
Default: The Pod inherits its DNS configuration directly from the worker node host OS without going through cluster CoreDNS (not recommended because the Pod can’t discover cluster Services).
None: Ignores all built-in cluster DNS configuration. We must manually define the dnsConfig property completely.

DNS Network Performance Optimization in Production #

On large-scale production clusters with very busy microservice activity, CoreDNS often becomes the cluster’s main performance bottleneck. This problem is most often triggered by the built-in options ndots:5 parameter.

The ndots:5 Bottleneck #

When our application tries to query an external domain (e.g. api.stripe.com), because that domain has fewer than 5 dots, the container’s internal resolver assumes it’s a local cluster domain and runs a chain of queries:

Query: api.stripe.com.production.svc.cluster.local -> Failed (NXDOMAIN)
Query: api.stripe.com.svc.cluster.local -> Failed (NXDOMAIN)
Query: api.stripe.com.cluster.local -> Failed (NXDOMAIN)
Query: api.stripe.com. -> Success (Query sent to the outside internet)

This means for every single external internet domain resolution, CoreDNS is forced to process 3 additional useless garbage queries first. This triggers DNS latency spikes and very high CoreDNS CPU load.

Optimization Solutions #

Use a Trailing Dot for External Domains: In application code, when calling external domains, end the domain with a dot (e.g. api.stripe.com.). The trailing dot tells the DNS resolver this is an absolute external FQDN, so the resolver immediately queries the internet without doing local cluster lookup searches first.
Use NodeLocal DNSCache: We’re advised to deploy the NodeLocal DNSCache addon (a companion DaemonSet). This addon runs a small DNS cache agent on every physical worker node using the node’s local loopback IP. DNS queries from Pods get captured by this local node agent first. If the DNS record is already cached, the response returns within microseconds without calling the central CoreDNS.

DNS & Service Discovery Anti-Patterns vs Solutions #

Let’s study the most fatal DNS management mistakes in Kubernetes along with their manifest code comparisons.

Anti-Pattern 1: Reaching Cross-Namespace Services Using Short Names #

We deploy a frontend Pod in the frontend-zone namespace and configure it to reach the payment service in the backend-zone namespace using only the short name payment-service.

Wrong Manifest Code (Relying on Local Namespace DNS Resolution) #

# DON'T DO THIS: Cross-namespace services will fail to resolve
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-deployment
  namespace: frontend-zone
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-front
  template:
    metadata:
      labels:
        app: web-front
    spec:
      containers:
        - name: app
          image: my-front-app:v1.0.0
          env:
            - name: PAYMENT_API_URL
              value: "http://payment-service:8080/charge" # FAILS: Searches frontend-zone.svc.cluster.local

The Bad Consequences #

When running, our frontend application triggers a Host not found or NXDOMAIN error. The DNS resolver in the frontend-zone namespace only tries to find payment-service.frontend-zone.svc.cluster.local, which of course doesn’t exist.

Solution Code (Using the Explicit Target Namespace) #

# SOLUTION: Explicitly specify the destination namespace location
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-deployment
  namespace: frontend-zone
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-front
  template:
    metadata:
      labels:
        app: web-front
    spec:
      containers:
        - name: app
          image: my-front-app:v1.0.0
          env:
            - name: PAYMENT_API_URL
              value: "http://payment-service.backend-zone.svc.cluster.local:8080/charge" # SUCCESS: Full, safe FQDN

Anti-Pattern 2: Misusing `dnsPolicy: Default` on Application Pods #

We configure the dnsPolicy: Default property on our microservice application pod, assuming “Default” means the best built-in Kubernetes setting.

Wrong Manifest Code (Disconnecting from CoreDNS) #

# DON'T DO THIS: The Pod can't find internal cluster Services
apiVersion: v1
kind: Pod
metadata:
  name: backend-app-isolated
  namespace: production
spec:
  dnsPolicy: Default # DON'T: This ignores cluster CoreDNS and refers to the host VM DNS
  containers:
    - name: app
      image: node:alpine

The Bad Consequences #

By using dnsPolicy: Default, the /etc/resolv.conf file inside the container points directly to the host VM node’s external nameservers (e.g. office resolver IPs or cloud provider DNS IPs). This backend-app-isolated Pod can never resolve DNS for any internal Service (like auth-service or database-service) running inside the cluster.

Solution Code (Use ClusterFirst or Leave It Empty) #

Leave the dnsPolicy property at its default ClusterFirst value (just leave it empty in the manifest so Kubernetes automatically picks ClusterFirst).

# SOLUTION: Using standard cluster DNS resolution
apiVersion: v1
kind: Pod
metadata:
  name: backend-app-connected
  namespace: production
spec:
  # dnsPolicy: ClusterFirst (Defaults to ClusterFirst if left empty)
  containers:
    - name: app
      image: node:alpine

Practical Guide to Diagnosing Cluster DNS Problems #

If our application fails to recognize Service names, we can run interactive DNS health checks.

1. Create a Dedicated DNS Test Pod #

Run a DNS utility container in the same namespace as our problematic application:

kubectl run dns-tester --rm -i --tty --image=tutum/dnsutils --namespace=production -- /bin/sh

2. Test Local DNS Resolution #

From inside the dns-tester container terminal, do an nslookup query against our local Service name:

# Test local service resolution
nslookup auth-service

# Test the default Kubernetes domain resolution
nslookup kubernetes.default

If nslookup returns server can't find auth-service: NXDOMAIN, check whether that Service actually exists in that namespace:

kubectl get svc -n production

3. Check CoreDNS Pod Health #

If all DNS queries fail (including kubernetes.default), check the central CoreDNS pod health:

kubectl get pods -n kube-system -l k8s-app=kube-dns

Review the CoreDNS pod logs to detect integration errors or upstream network issues:

kubectl logs -n kube-system -l k8s-app=kube-dns

Summary #

CoreDNS manages Service Discovery: The internal cluster DNS server runs in the kube-system namespace and automatically updates DNS records as Service objects change in the cluster.
Kubelet injects resolv.conf: Every container gets an /etc/resolv.conf file directing nameserver queries to the kube-dns IP and providing local domain search domains.
Cross-namespace queries must write the target namespace name: Use at least the <service-name>.<namespace> format for cross-namespace communication, or the full FQDN for safety.
Understand the impact of the ndots:5 parameter: External internet domain queries can overload CoreDNS CPU because the resolver tries matching local cluster suffixes first.
Optimize DNS performance: End external address queries with a dot (e.g. api.stripe.com.) or deploy the NodeLocal DNSCache addon to cut DNS query latency.
Use dnsutils for troubleshooting: Always leverage the tutum/dnsutils utility container to run nslookup or dig commands directly inside the cluster.

← Previous: Service Next: Ingress →

DNS & Service Discovery #

The CoreDNS Architecture: The Cluster’s Service Discovery Brain #

How Do Pods Reach CoreDNS? #

Standard DNS Record Formats in Kubernetes #

1. A/AAAA Records for Services (ClusterIP) #

2. A Records for Headless Services #

3. A Records for Individual Pods #

4. SRV Records for Named Ports #

Short Name Lookups vs FQDN #

Cross-Namespace Navigation Rules #

Customizing DNS Configuration at the Pod Level #

Available dnsPolicy Options #

DNS Network Performance Optimization in Production #

The ndots:5 Bottleneck #

Optimization Solutions #

DNS & Service Discovery Anti-Patterns vs Solutions #

Anti-Pattern 1: Reaching Cross-Namespace Services Using Short Names #

Wrong Manifest Code (Relying on Local Namespace DNS Resolution) #

The Bad Consequences #

Solution Code (Using the Explicit Target Namespace) #

Anti-Pattern 2: Misusing dnsPolicy: Default on Application Pods #

Wrong Manifest Code (Disconnecting from CoreDNS) #

The Bad Consequences #

Solution Code (Use ClusterFirst or Leave It Empty) #

Practical Guide to Diagnosing Cluster DNS Problems #

1. Create a Dedicated DNS Test Pod #

2. Test Local DNS Resolution #

3. Check CoreDNS Pod Health #

Summary #

Anti-Pattern 2: Misusing `dnsPolicy: Default` on Application Pods #