It’s Saturday morning. PagerDuty goes off. The checkout service is returning 500s. Three engineers are paged. In the first 15 minutes: one is rolling back the last deployment, another is SSHing into production to check database queries, a third is changing environment variables live trying to fix it. A fourth is in Slack asking if anyone knows what’s happening. Nobody is talking to each other. Nobody knows who’s in charge. Nobody has confirmed what “down” actually means or what the blast radius is.

The technical problem - a misconfigured environment variable - took 8 minutes to fix once someone actually looked at it. The incident took 90 minutes because the response was uncoordinated.

Most of the time in a production incident is not spent fixing the technical problem. It’s spent on coordination failures.

What an Incident Actually Is

An incident is any unplanned disruption to a service that requires coordinated response. “Coordinated” is the key word - a single engineer fixing a bug alone is not an incident. An incident is when multiple people are involved, when the blast radius is unclear, when the cause is unknown.

The first step is declaring the incident. This sounds bureaucratic but it matters: it establishes that we’re in incident mode, identifies who’s responsible for coordination, and starts the clock for escalation.

Without a declaration, you get the worst of both worlds: engineers partially attending to a problem while still doing other things, duplicate investigation efforts, no clear owner.

The Four Roles

A small incident (two or three engineers) doesn’t need formal roles. A large one where the cause is unclear and multiple systems might be affected does.

Incident commander. Runs the response. Their job is coordination, not technical investigation. They assign tasks, prevent duplicate effort, give status updates every 5-10 minutes, and make the call on escalation. They do not touch production systems during the incident - their attention is managing the response, not fixing the problem.

Technical lead. Does the actual investigation. Owns the hypothesis: “I think the problem is X, and here’s why.” Reports findings to the incident commander.

Comms. Updates the status page, notifies internal stakeholders, handles customer-facing communication. Separates these concerns from the investigation so the technical people don’t have to interrupt their work to write updates.

Scribe. Documents everything in real time: what was checked, what was found, what was changed and when. Invaluable for the postmortem and for anyone joining the incident late.

For a two-person incident, one person acts as incident commander and comms, the other investigates. The split still matters.

The Timeline of a Real Incident

Detection: something alerts, or a user reports it. Before anything else: is this actually a problem? What is the current state? “Checkout is returning 500s” is not specific enough. “All POST /checkout requests are returning 503 starting at 14:23 UTC, affecting all regions, 100% error rate” is a starting point.

Assessment: what’s the blast radius? What is and isn’t affected? Is it degraded or fully down? Are there workarounds? This takes 5 minutes and saves everyone from working on the wrong thing.

Hypothesis formation: the technical lead develops a hypothesis based on recent changes, error messages, and metrics. “We deployed 2 hours ago, the change touched the payment service, errors are only on payment requests” is a hypothesis.

Mitigation: the fastest path to user recovery, even without a root cause. Rolling back a deployment. Disabling a feature flag. Redirecting traffic to a healthy region. Mitigation is not root cause fix - it’s stopping the bleeding.

Resolution: root cause identified, permanent fix deployed, metrics confirmed back to normal.

The Cardinal Rule

Do not change production while investigating.

Every change you make obscures the state you were trying to understand. If the system was in state A when the incident started, and you’ve changed three things trying to fix it, you now don’t know if you’re looking at the effects of state A or the effects of your changes. Root cause analysis becomes guesswork.

Establish what’s happening first. Form a hypothesis. Make one change. Observe the effect. This is the scientific method applied to incidents, and it works.

The temptation to change things immediately is strong - production is down, every second counts, just try something. This instinct is almost always wrong and frequently extends incidents by introducing new variables.

The Mistake Pattern

The most common mistake: everyone investigates, nobody coordinates. You get five engineers simultaneously SSHing into different servers, running overlapping commands, modifying things without telling each other, working from different hypotheses. The collective intelligence is less than any individual’s because there’s no shared model of what’s being tried.

The second most common mistake: changing production before understanding the state. Someone redeploys, someone else restarts a service, someone else disables a feature. Now you have three changes whose effects overlap and nobody knows what fixed it - or if it’s actually fixed or just temporarily stable.

The third: fixing the symptom without the root cause. The service is restarted, it starts working. The restart fixed the immediate problem but the underlying cause remains. Three hours later, it’s back.

After the Incident

A postmortem should be written within 48 hours, while memory is fresh. The purpose is not blame - it’s understanding how to make this category of incident less likely or less severe.

A good postmortem contains:

• A timeline of what happened and when
• Root cause analysis (the actual cause, and the contributing factors that made it possible - good postmortems always find multiple contributing factors, never a single point of blame)
• What worked well in the response
• Concrete action items with owners and deadlines

The most common postmortem failure: vague action items. “Improve monitoring” is not an action item. “Add an alert on POST /checkout error rate exceeding 5% for 2 minutes by September 15, owned by the platform team” is.

Runbooks

An engineer paged at 3am for a service they’ve never touched is set up to fail. Runbooks are the mitigation.

A runbook is not documentation - it’s a checklist for 3am. What to look at first, what “normal” looks like, what the five most common failure modes are for this service and how to address each, and who to escalate to if none of those apply.

A runbook that describes what the service does is not useful at 3am. A runbook that says “if error rate is elevated, check query duration in Datadog (link), if queries are slow check for missing index on (specific column)” is useful.

The test for a runbook: can an engineer who’s never touched this service use it to diagnose the three most common failure modes without asking anyone? If not, it’s documentation, not a runbook.