Every organisation claims to do root cause analysis. Very few do it in a way that actually prevents the same class of incident from recurring. After leading RCA for dozens of S1/S2 outages across TCS, Capgemini, and now Acqueon, here's the methodology I've converged on.
The core insight: most RCA processes stop at the trigger, not the root cause. The trigger is the thing that happened right before the outage. The root cause is the system condition that made the trigger possible. Fix only the trigger, and a different trigger will cause the same class of failure six months later.
Phase 1: Immediate Response (0–30 min)
The goal of the first thirty minutes is not diagnosis — it's stabilisation. I've seen many outages extended by engineers trying to understand what happened while the system is still on fire.
The only questions that matter during this phase:
- Is the failure still propagating?
- What is the fastest path to restore service (rollback, failover, scale)?
- Who needs to be notified right now?
Crucially: preserve evidence. Before any remediation action, capture logs, metrics, and system state. I keep a running timestamped incident log from the moment I'm paged — this becomes the raw material for the RCA document.
# Capture state before remediation
kubectl get events --all-namespaces --sort-by='.lastTimestamp' > incident_events_$(date +%s).txt
kubectl describe nodes > incident_nodes_$(date +%s).txt
kubectl top pods --all-namespaces > incident_resource_usage_$(date +%s).txtPhase 2: The Five-Why Trace (post-stabilisation)
Once the system is stable, the RCA begins. I use a modified five-why methodology with one critical addition: at each why, I explicitly ask whether there's a detecting signal we missed.
The goal isn't just to find why it failed. It's to find every point where we could have known it was going to fail — and didn't.
A concrete example from a hypervisor incident at Capgemini:
- Symptom: 47 VMs went offline on an ESXi host
- Why 1: Host entered a failed state due to storage disconnection
- Why 2: Storage disconnection caused by iSCSI path failure
- Why 3: iSCSI path failure because NIC firmware update changed MTU default
- Why 4: Firmware update was applied without validating iSCSI path health post-change
- Why 5: No runbook existed for iSCSI path validation in the firmware update procedure
The trigger was the NIC firmware update. The root cause was an absent validation step in our change management procedure. The fix that actually mattered was the runbook addition — not the firmware rollback.
Phase 3: Cascading Dependency Mapping
In distributed systems, the initial failure is rarely the worst part. The worst part is what it triggers downstream. I draw a dependency graph for every S1 incident.
The format I use:
Primary failure: [component]
↓ caused: [downstream effect 1]
↓ caused: [downstream effect 2]
↓ caused: [customer impact]
↓ also caused: [parallel effect]
↓ caused: [secondary customer impact]This map does two things. First, it reveals blast radius — often much larger than the immediate symptom suggested. Second, it identifies intervention points: places in the dependency chain where a circuit breaker, timeout, or fallback could have contained the failure.
Phase 4: Systemic vs. Symptomatic Fixes
Every RCA should produce two lists of action items: symptomatic fixes (patch the immediate thing that broke) and systemic fixes (change the system so this class of failure becomes less likely or less severe).
Systemic fixes are harder. They often require cross-team coordination, process change, or architectural modification. They're also the only fixes that matter for long-term reliability.
The template I use for each action item:
Action: [specific thing to do]
Owner: [person/team]
Deadline: [date]
Type: symptomatic | systemic
Prevents: [which failure mode this addresses]
Detects: [what signal this creates for earlier detection]
Verified: [how we'll confirm it's done and effective]Phase 5: The Knowledge Base Entry
The final step — and the most commonly skipped — is adding the incident to a searchable knowledge base with enough context that a future engineer can find it when they see a similar pattern.
At Acqueon, I'm building this knowledge base with entries tagged by:
- Failure class (resource exhaustion, network partition, configuration drift, etc.)
- Affected components
- Detection signals that were present (even if we missed them)
- Time-to-detect and time-to-resolve
The goal is that when the next on-call engineer sees a specific pattern of Grafana alerts, they can search the knowledge base, find a related incident, and cut their investigation time in half.
The Metric That Tells You If Your RCA Process Works
Track the rate of recurrence of the same failure class. Not the same incident — the same class. If you're doing RCA well, the second occurrence of any failure class should look noticeably different from the first (shorter MTTD, faster recovery, smaller blast radius) because you've improved detection and response.
If the second occurrence looks identical to the first, your RCA process produced documents — not improvements.