Why Kubernetes Silently Fails: Patterns from Production AI Workloads

Kubernetes has a reputation for being self-healing. And it is — up to a point. The problem is that "self-healing" means the cluster tries to maintain desired state. It doesn't mean the cluster tells you when it's quietly failing to do so in ways that matter to your application.

This post is about the category of failures I call silent degradation — where everything looks green on your dashboard, your pods are running, your deployments show Available: true, and yet your AI inference workloads are slower, less reliable, or subtly wrong.

The Three Categories of Silent Failure

After running post-mortems on a dozen production incidents, I've found silent Kubernetes failures cluster into three categories:

1. Resource contention that doesn't OOM-kill
2. Scheduling decisions that appear correct but aren't
3. Control plane lag that your application never sees coming

Let me walk through each with concrete examples from what I've observed.

1. Resource Contention That Doesn't OOM-Kill

The OOMKiller is visible — you see the pod restart, the event appears in kubectl describe pod, and your alerting fires. What's far harder to catch is the period before the OOM-kill — when your container is memory-pressured but still running.

In AI inference workloads specifically, memory pressure causes the kernel to start swapping and the GPU driver to throttle memory bandwidth. Your model keeps serving responses, but p99 latency quietly doubles. No error. No restart. Just a degraded user experience.

The metric you need is not container memory usage. It's container_memory_working_set_bytes as a percentage of requests.memory. When that ratio exceeds ~80% consistently, you're in the danger zone — even if you're nowhere near your limit.

The Grafana query that saved us:

100 * (
  container_memory_working_set_bytes{container!=""}
  / on(pod, container)
  kube_pod_container_resource_requests{resource="memory"}
)

Set a warning alert at 75%, critical at 90%. You will catch degradation that kubectl top completely misses.

2. Scheduling Decisions That Appear Correct But Aren't

Kubernetes scheduling is eventually consistent. The scheduler makes placement decisions based on the state it sees at scheduling time — not the state that will exist when the pod actually starts consuming resources.

This matters enormously for AI workloads with large model weights. A scheduler can place three inference pods on a node that looks like it has enough CPU and memory — because the other pods on that node haven't finished loading their model weights yet. By the time all three are fully initialised, the node is oversubscribed.

The pattern I look for in kubectl get events:

kubectl get events --field-selector reason=BackOff \
  --sort-by='.lastTimestamp' -A | tail -20

Repeated BackOff events on freshly scheduled pods — without OOM-kills — is almost always a scheduling-time vs. runtime resource mismatch. The fix is enforcing proper resource requests that reflect peak consumption, not average consumption.

The Burstable QoS trap

Most teams set requests lower than limits to give workloads "room to burst." For stateless web services, this is fine. For AI inference, where cold-start memory usage can be 3–4x steady-state, Burstable QoS means your pods are first in line for eviction during node pressure events — and they're the pods you can least afford to evict.

For your model serving pods: set requests == limits. Take the Guaranteed QoS hit on resource efficiency; you'll recover it in reliability.

3. Control Plane Lag

The etcd-backed control plane has a propagation delay between when a state change happens and when all components have processed it. Under normal load, this is milliseconds. Under heavy load — or when etcd is undersized — it can stretch to seconds or minutes.

The symptom: kubectl get pods shows Running, but traffic is still being routed to a pod that has already started failing its readiness checks internally. The endpoints controller hasn't caught up yet.

The metric to watch is etcd_disk_wal_fsync_duration_seconds. If p99 exceeds 10ms, your control plane is under pressure and propagation lag will follow:

histogram_quantile(0.99,
  rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
) * 1000  # convert to ms

Detection Checklist

Here's the runbook I now apply to any cluster supporting AI workloads:

• Alert on container_memory_working_set_bytes / requests.memory > 80%
• Check QoS class for all model-serving pods — should be Guaranteed
• Monitor etcd WAL fsync p99 — alert above 10ms
• Review scheduling events for repeated BackOff without OOM on fresh pods
• Set minReadySeconds on Deployments to give pods time to actually initialise before receiving traffic

The Bigger Pattern

What unifies all three of these is that they're failures of visibility, not failures of the system itself. Kubernetes is doing what it's designed to do — the problem is that what it's designed to do doesn't map cleanly onto the operational requirements of AI inference workloads, which have very different resource profiles from the stateless microservices K8s was originally built around.

The infrastructure engineering work that matters most right now isn't deploying models — it's building the observability layer that makes AI system behaviour legible under production conditions.

That's the gap I'm most interested in closing.