Rollback or hotfix: how to decide in the middle of an incident
A failed deployment is the most common production incident there is, and it always ends at the same fork: roll back the change, or patch it forward. Teams burn hours arguing about it because they argue about the wrong question. The right question is not “which fix is correct”. It is “which action restores the critical path fastest with the least risk.”
The two moves, honestly compared
Rollback: revert to the last known-good version.
- Fast when the pipeline supports it. One button, two minutes, done.
- Returns you to a state the system has already survived. That is an enormous advantage: production has proven it can run this version.
- Costs: you lose the new feature or fix temporarily. Schema/data migrations may not revert cleanly. If the deploy changed data format, the old code may not understand the new data.
Hotfix: apply a small patch on top of the broken change.
- Attractive because you keep the new behavior.
- Costs: someone has to write, test and ship code during an active incident, under pressure, against a moving target. The fix may introduce a second bug. You are now debugging two unknowns instead of one.
- Legitimately correct when the data is already incompatible with the old version, or when the old version has its own known critical flaw.
Decision rules that survive pressure
Use these in order, not as a debate:
- Is the broken change isolated? If the bad behavior is clearly tied to one commit, rollback is cheap and obvious.
- Does the old version still understand the data? If a migration already ran or writes changed shape, rollback may break worse than the current state. Check this before promising a revert.
- Can you get the old version out in minutes? If your pipeline makes rollback a long manual procedure, a hotfix can genuinely be faster. If rollback is a button, it is almost always faster.
- Is the team calm enough to write code? Honest answer matters. Code written in the middle of an active outage is the highest-risk code you will ship all year.
- Is the current state getting worse? If error rates are climbing or data is corrupting, stop deliberating and take the lowest-risk action now, even if it is imperfect.
The anti-pattern: fixing the root cause in the outage window
The most expensive mistake is using the incident window to understand the bug. Understanding belongs after stabilization. In the window, your job is to remove the impact.
A clean sequence that ends the debate:
- Stabilize with the fastest safe move (usually rollback).
- Confirm the critical path is healthy again.
- Keep the evidence (the broken version, logs, error traces) before anyone moves on.
- Diagnose and fix properly, afterwards, in normal working conditions, with tests.
When rollback is not the answer
Three cases where rolling forward is right:
- The old version cannot run against current data (post-migration).
- The failure is in infrastructure or third parties, not your code, so there is nothing to roll back.
- The “new” code fixed a security or data-integrity issue, and reverting reintroduces it.
In those cases a hotfix, or even just disabling the failing component, is the stabilization move, and the same rule applies: do the smallest thing that restores the critical path, then fix properly later.
The meta-rule
Rollback and hotfix are not competing philosophies. They are two tools, and the correct choice depends on how fast each one restores the critical path for this system, this data, this team, right now. Systems that decide this in advance (a documented rollback path, a known-good deploy, a named decision-maker) spend minutes at the fork instead of hours.
The teams that handle failed deployments gracefully are not the ones with better luck. They are the ones who already knew, before the incident, which button they would press first.
Deployment failures are the classic after-hours incident: they happen exactly when the team is smallest. If you run a small team without a night rotation, senior escalation on a defined response target covers the window between “deploy went wrong” and “someone competent is looking at it”. (Read: The first 15 minutes of a production incident and On-call for small SaaS teams without burning out.)