Post-Mortem: CoreDNS Connection Pool Exhaustion under AI Agent Load Spike

Incident Summary

On July 8, 2026, at 14:15 UTC, a cluster-wide name resolution outage occurred in our primary production Kubernetes cluster (prod-core-svcs), causing lookup failures across several microservices. Internal services attempting to resolve database DNS endpoints, external payment APIs, and sibling microservices timed out, leading to a cascading failure of user-facing web gateways.

The incident was triggered by a sudden surge in autonomous API queries from our new AI Agentic Orchestration service (aops-agent-scheduler). The load spike exhausted the connection pool limits of our core DNS forwarders.

The service was fully restored by 14:52 UTC (Duration: 37 minutes) after tuning CoreDNS caching, scaling replica counts, and implementing connection reuse policies inside the AI agent’s HTTP client.

The Outage Timeline

Technical Root Cause Analysis (RCA)

To understand why a 5,000-task audit spike could collapse name resolution across the entire cluster, SRE traced the networking path.

The AI Agent scheduler was configured to audit 5,000 simulated microservices. Due to a coding pattern in the agent’s Go HTTP client, it instantiated a new HTTP Transport for every single task without sharing the transport object. This completely bypassed HTTP connection reuse (Keep-Alive), causing 5,000 concurrent sockets to open.

sequenceDiagram
    participant AI as AI Agent Scheduler
    participant KNS as Kube-DNS (CoreDNS)
    participant UP as Upstream DNS (10.x.x.x)

    Note over AI: Instantiates new Transport per audit
    AI->>KNS: 5,000 concurrent lookup queries
    Note over KNS: Cache misses (new subdomains)
    KNS->>UP: Forward 5,000 queries to Upstream Forwarder
    Note over UP: Connection pool limits exhausted
    UP-->>KNS: Dropped packets / i/o timeout
    KNS-->>AI: i/o timeout (Failure)

At the DNS level:

1. Since name resolution for the microservices was not using a cached wild-card route, CoreDNS forwarded these queries upstream.
2. Because the DNS client within Go’s network resolver makes concurrent A and AAAA requests, 5,000 API audits translated to 10,000 concurrent UDP packets hit the DNS system within a 3-second window.
3. CoreDNS upstream forwarder pool exhausted its descriptor limits, causing it to drop UDP packets silently.
4. This silent packet dropping triggered resolving delays, causing other critical workloads (like our database connectors) to fail DNS lookup and crash.

Action Items & Hardening Policies

To prevent this type of failure in the future, we have implemented the following structural changes:

1. HTTP Client Connection Reuse (Code fix)

The AI Agent scheduler code was modified to use a shared, singleton HTTP client.

// BEFORE (Buggy)
func AuditService(url string) {
    client := &http.Client{Transport: &http.Transport{}} // Exhausted sockets
    client.Get(url)
}

// AFTER (Fixed)
var SharedClient = &http.Client{
    Transport: &http.Transport{
        MaxIdleConns:        100,
        MaxIdleConnsPerHost: 20,
        IdleConnTimeout:     90 * time.Second,
    },
}

2. CoreDNS Configuration Hardening

We updated the CoreDNS Corefile config map to enable prefetch and increase caching. This allows CoreDNS to resolve popular queries from cache while asynchronously prefetching expiring records, keeping upstream lookups minimal.

.:53 {
    errors
    health {
        lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    }
    prometheus :9153
    forward . 10.x.x.x {
        max_concurrent 1000
    }
    cache 30 {
        success 9984 30
        denial 9984 5
        prefetch 10 10m 10%
    }
    loop
    reload
    loadbalance
}

3. Autoscale DNS Pods (KEDA)

We integrated CoreDNS with the Kubernetes Event-driven Autoscaler (KEDA) to scale dynamically based on CoreDNS CPU usage and upstream packet drop alerts, ensuring DNS pods automatically scale out before the upstream connection pool is saturated.

Lessons Learned

This incident highlights that SRE is not just about keeping servers online; it is about managing the symbiosis between application code patterns and infrastructure limits. When building autonomous AI agents that operate at high concurrency, developers must design HTTP/network connectors with the same care as traditional high-throughput SRE tooling.