The first 15 minutes of a production incident

When production goes down, the first fifteen minutes decide most of what follows. Not the technical outcome, but the cost of the outcome. Teams that waste the opening minutes lose time twice: once in chaos, again in a longer than necessary outage. Teams that run the first quarter hour deliberately shorten both.

This is not about being heroic. It is about having one person think clearly while everyone else reacts, and about collecting the facts you will need later while they are still on the screen.

Minute 0–5: stabilize the humans first

Before touching anything, name the problem and the response. Five minutes of silence while people log in independently is the most expensive thing in an incident.

1. Acknowledge out loud. A single thread, a single channel, a single source of truth. “Incident open. P1. Checkout is down. I am triaging.” Someone owns the thread for the whole incident.
2. Declare one incident commander. Everyone else reports findings there. No side channels for the first ten minutes.
3. Ask the two questions that matter: “What changed?” and “What breaks if we revert?”
4. Write down the start time. Almost every incident summary I read gets this wrong thirty minutes later, because nobody recorded it at the start.

The instinct to jump straight into debugging is strong. Resist it for sixty seconds. You will get the debugging time back.

Minute 5–10: gather evidence while it still exists

Incident evidence is perishable. Logs rotate, queues drain, error rates normalize, and the person who saw the first error goes back to their other meeting.

• Capture the error text, screenshot or trace before changing anything.
• Note the current deploy/commit and the previous one that was healthy.
• Record the metric you are using as the north star (error rate, 5xx count, checkout completion, queue depth) and its current value.
• Check recent config changes, migrations, third-party API status pages, and environment-specific differences.

The goal of this window is not to fix anything. It is to make sure that when diagnosis happens, the data exists.

Minute 10–15: pick the stabilization move, not the root-cause fix

Here is the decision that separates disciplined incidents from chaotic ones. In the first fifteen minutes you are almost never ready to fix the root cause. You are ready to reduce impact.

The stabilization ladder, cheapest first:

1. Roll back the last deploy if it can be reverted safely and quickly.
2. Disable the feature flag, queue consumer, cron job or integration that changed.
3. Restart the degraded process. It is a surprisingly effective fix for memory leaks and wedged connections, and it takes thirty seconds.
4. Scale or fail over if capacity or a region is the problem.
5. Bypass the broken path (circular dependency, dead external call, caching layer).

Only after impact is contained is it correct to hunt the root cause with calm. An uncontained incident makes everything harder: every log you read is screaming, every teammate is asking for updates, and the pressure to guess grows with every minute.

What the first quarter hour buys you

• A single source of truth that survives the incident.
• Evidence collected while it was available.
• A decision made from a short ladder instead of a guess.
• A team that knows what “resolved” looks like, because someone said it at minute three.

The teams that do this well are not smarter. They decided in advance that the first fifteen minutes belong to structure, not adrenaline. That one practice, repeated across incidents, is what makes an outage cost minutes instead of hours.

If your team is too small to staff its own on-call rotation, senior escalation exists precisely for this first quarter hour. That is the gap on-call coverage is designed to close: system context and a response path agreed before the incident happens. (Read: On-call for small SaaS teams without burning out and Rollback or hotfix: decide in the middle of an incident.)