How I Lead Through Production Incidents Without Taking Over
The alert reached me while I was cooking dinner: checkout errors were climbing, and the on-call engineer was already in the logs. I opened my laptop, pulled up the same traces, and started debugging. Twenty minutes later a senior engineer asked the question that changed how I run incidents: “Are you commanding this, or debugging it?” I was doing neither well.
I spent years in DevOps before management, and the pull to take over is still the strongest reflex I have. In this guide I cover the discipline I built instead: declaring the incident before the root cause, naming roles out loud, keeping communication on a cadence, escalating early, writing stakeholder updates that reduce anxiety, deciding at the speed of the incident, and running blameless postmortems that change the system.
The manager in the terminal is an absent manager
I was about eighteen months into management when a payments service started failing at 18:40 on a Friday. I knew that system better than anyone on call. I had built parts of it, I could read its traces faster than the engineer who owned it, and I was already opening a terminal before I had consciously decided to. For the next forty minutes I did what I had done for years: narrowed the failure, formed a hypothesis, and chased it into the logs.
The technical work was fine. The response around it was not. Nobody had told support what to say. Two engineers were preparing different rollbacks because no one had compared them. A director had asked for an update three times and received three different answers. We restored service in just over two hours. In the postmortem, the timeline showed that roughly forty minutes had gone into coordination debt: decisions waiting on me, people working in parallel on incompatible fixes, and stakeholders filling the silence with their own theories.
That incident taught me the central lesson of this guide. My value in a production incident is not how fast I can find the bug. It is how quickly the team gets clear ownership, an honest picture of the impact, and a decision it can act on. A manager with a DevOps background feels the pull to fix things because the patterns are familiar. Familiarity is exactly what makes the pull dangerous: I mistake recognition for usefulness.
There is one exception, and it is narrower than it feels in the moment: when I am the only person who can safely perform a specific action, and the team asks me to do it. Even then, I hand command to someone else first and say so out loud. If that exception happens more than once a quarter, it is not an exception. It is a staffing and documentation problem to fix after the incident.
Declare the incident before you understand it
The first minutes of a significant incident are chaotic by nature: conflicting symptoms, several teams investigating in parallel, and a dozen confident theories. The temptation is to find the root cause before declaring anything, because declaring an incident feels like admitting we do not have it under control. That instinct delays the one thing that actually creates control.
I declare first and investigate second. The declaration does not need a cause. It needs scope, severity, an owner, and a place where the response lives. I want five answers before I care about a sixth:
Root cause is deliberately not on the list. Impact and ownership come first because they determine what everyone else should do, including the people who are not technical. If the incident turns out to be smaller than feared, we downgrade it in public. A downgraded incident costs far less than an undeclared one. The severity ladder I use matches the same logic I apply to competing urgent requests: start from what actually breaks, not from how loudly it arrived.
Once the incident exists, I name the roles out loud in the channel. Not as a formality, but because unnamed responsibility is where parallel work turns into conflicting work. These are the five I use; on a small team the same person can hold two, with one exception.
The exception is the Incident Commander and the Technical Lead. They can be the same person on a small, contained incident, but not on a significant one. One role is deciding what the response should do next; the other is deciding what the system should do next. Trying to hold both is how a commander ends up debugging while the response drifts. I am comfortable taking either seat. I am not comfortable pretending I can take both.
The hand-off matters as much as the assignment. If I start as commander and then need to get technical, I say the transfer explicitly: “Priya is Incident Commander now. I am joining the database investigation.” A hand-off announced is a decision. A hand-off assumed is a gap. The same rule applies when someone else takes over from me. The role belongs to the incident, not to the most senior person in the room.
Cadence is the control system
Most incident communication fails in the same way: engineers go quiet while they work, stakeholders fill the silence with anxiety, and the gap between what is known and what is being said becomes its own problem. I do not try to solve that with more meetings. I solve it with a cadence that everyone can predict.
Inside the response, the Incident Commander posts a short update every fifteen to thirty minutes, even when the update is “no change.” The forced rhythm does two useful things. It surfaces blockers early, because saying “still blocked” three times in a row is uncomfortable enough that someone eventually asks for help. And it keeps remote and async participants in the loop without interrupting the people whose hands are on the keyboard.
I also enforce one channel for decisions. Debugging can happen in a war room, a shared doc, or a call, but every decision that matters gets repeated in the main incident channel. If a decision was made in a direct message, it did not happen; the next shift will not find it, and the postmortem will not be able to reconstruct it. The scribe records the decision, the owner, and the timestamp, not a transcript of the conversation.
Language discipline is part of the system. I ban “should be fine” and “I think it is fixed” as status updates. Not because confidence is bad, but because those phrases hide which part is verified. Say either “health checks are passing on eight of ten pods and two are still failing” or “not yet known; next check at 14:45.” Stakeholders remember the moment a confident sentence turned out to be wrong, and they stop trusting every sentence after it.
The last communication rule is protection. Every question from outside the response goes through me or the Communications Lead. The technical team is not available to answer “quick questions” from sales, support, or leadership. That is not secrecy; it is the only way the people doing the work get to stay in the work. I would rather be the person who absorbs ten interruptions than the manager who lets ten interruptions reach the engineer holding the rollback.
Escalate early enough to be useful
Engineers learn to treat escalation as a loss: I should have solved this myself, asking for help is an admission that I could not. That conditioning is expensive during production incidents. A delayed escalation does not protect anyone’s reputation. It converts a technical problem into a longer customer outage, and then everyone gets to share the consequences.
I reframe escalation as buying expertise. The cost of a false alarm is a few minutes of an expert’s time. The cost of a late escalation is measured in customer minutes, and sometimes in the permanent loss of data or trust. The asymmetry is so large that I would rather over-escalate early than under-escalate with confidence. I say that out loud in peacetime, repeatedly, because the habit has to be built before the pressure arrives.
The escalation itself should be a message someone can act on, not a feeling transferred upward. I use four lines: what is happening, what we have tried, what specific help we need, and who should join. Compare “the database is bad, we need help” with “replication lag is forty minutes and growing; we need someone who owns the replica topology to decide between failover and waiting.” The second one can be answered. The first one starts another investigation.
Most of the work happens before the incident. I keep an authority matrix that says who can approve which actions, a contact path for every critical vendor, and a pre-drafted escalation template with the account numbers and severity language already filled in. A junior engineer should be able to summon the right help at 2 a.m. without composing a business case. The organizations that escalate well are not the ones with the bravest engineers. They are the ones that removed the friction.
Escalation also flows down. When a team’s part of the incident is over, I release them explicitly: “Payments, you are clear. Thank you. We will call you if the database work changes that.” Keeping fifteen people in a war room for a contained incident because nobody wants to be the first to leave is not diligence. It is exhaustion with extra steps.
Impact, action, and a time you will speak again
Executives, support, sales, and customer success do not need the query plan. They need to know what is broken, who is affected, what we are doing about it, and when they will hear from us next. The technical detail is not a sign of rigor to them. It is noise that makes them ask more questions, which then reach the engineers.
I write stakeholder updates in four parts, every time, in plain language. The consistency matters as much as the content: after the second update, people stop bracing for bad news and start reading for the decision they need to make.
The commitment to a next update is the line people skip, and it is the one that buys the most trust. It tells stakeholders they can wait instead of asking. It also gives the response team a deadline for producing a readable status, which is a useful forcing function in its own right. A “no change” update at the promised time is still an update. Silence is what people interpret as loss of control.
I do not speculate about cause in stakeholder updates. “We believe it is the vendor” is not a status; it is a theory that may take a week to unwind after the vendor turns out to be fine. I say “not yet confirmed” and leave it there. In a regulated environment I also loop in the people who own legal, compliance, and customer notification before an informal update becomes the official record. The communication plan is part of the response, not an afterthought.
The translation work is the same skill I use when explaining engineering trade-offs to executives: turn a technical measurement into a business consequence, state the uncertainty honestly, and name the decision that is needed. An incident is just the compressed version of that conversation, with less time and more adrenaline.
Reversible decisions fast, irreversible decisions deliberately
Incidents compress decisions that would normally take a week into minutes. The team does not need perfect information. It needs a decision that is proportionate to the risk and early enough to change the outcome. Two questions keep me honest under pressure: can we undo this, and what happens if we wait another thirty minutes?
Reversibility is the first filter, not the severity of the symptom. A rollback, a feature flag turned off, a traffic shift, or a rate limit is usually reversible, so I want those decided quickly even on incomplete evidence. A schema migration, a data deletion, a key rotation, or a failover with consistency implications is not, so I want those slowed down, checked twice, and owned by someone with the authority to accept the risk.
Timeboxing is the second habit. “We will decide between rollback and hotfix by 15:20” is a better instruction than “keep investigating.” A deadline forces the group to name the fact that would settle it, and if that fact is not available by the deadline, we default to the most reversible option. Most incident decisions do not improve with another twenty minutes of discussion. They improve with another twenty minutes of mitigation.
I also separate mitigation from diagnosis. Restoring service and understanding the failure are two different problems with two different clocks. The team can mitigate first and keep diagnosing in parallel, as long as everyone knows which one is currently in charge. When a mitigation changes the system, the decision log records what changed, who changed it, and what would make us reverse it. That log is what lets the next shift inherit a state instead of a mystery.
Consensus is not the goal. The Incident Commander decides after hearing the Technical Lead, and the Technical Lead decides the implementation details after hearing the responders. If the decision is genuinely above the room, we escalate with options, a recommendation, and a deadline rather than escalating the decision itself. And when someone says “I can do this, but I have never done it from this region,” that is valuable information. I pair them with someone who has, then let them keep the keyboard. Fear stated early is cheaper than confidence discovered late.
Recovery is a verification step, not a feeling
The error rate dropping is the most dangerous moment of an incident. Everyone is tired, the graphs are moving in the right direction, and the instinct is to declare victory and close the channel. I have seen that instinct produce a second incident twenty minutes later, usually because the first fix addressed a symptom or because a temporary change was never written down.
Recovery needs evidence, not relief. Before I let the incident move to monitoring, I want the critical user journeys checked end to end, the data checked for consistency, the downstream integrations checked, and the error and latency trends watched long enough to distinguish a real recovery from a traffic lull. The exact checks depend on the system. The discipline does not: somebody other than the person who made the change verifies it.
I use three states in the record: investigating, mitigated, and resolved. Mitigated means the customer impact has stopped but the underlying condition may still be present. Resolved means recovery is verified and the formal closure criteria are met. The distinction matters because a mitigated incident with a temporary change is still an open risk. If the channel closes before that risk has an owner, the cleanup becomes nobody’s job, and the temporary change becomes permanent architecture by accident.
The hand-back to normal operations is part of the response. I confirm the remaining actions, name who owns each one, tell stakeholders what happens next, and close the incident in the system of record. Then I make sure the people who carried the pager get relief. An incident that ends with a heroic all-nighter and no recovery time is not a success story. It is a staffing failure that will show up in the next incident.
The incident ends when the system changes
Service restoration solves the immediate problem. The postmortem is where the organization gets a return on the pain. It is also the step most likely to be skipped, because everyone is tired and the incident is no longer on fire. I schedule it within a business week, while memory is fresh and before the story hardens into a convenient version.
Blameless does not mean nameless, and it does not mean consequence-free. Names belong in the timeline because “Priya ran the migration at 13:58” is a fact, not an accusation. What we do not do is stop at the name. We ask what made that action reasonable, available, or hard to catch: the missing dry run, the alert that fired late, the runbook with two conflicting steps, the on-call rotation that put one person in front of a system they had never seen.
“Human error” is a starting point, not a root cause. People do not wake up intending to cause an outage. They act on the information, tools, and incentives the system gave them. If a deployment took down checkout, the useful finding is not that an engineer pushed the button. It is that the pipeline allowed an untested change to reach production without a canary, and that nobody could see the error rate until customers reported it.
The action items are where most postmortems quietly fail. “Improve monitoring” is not an action item; it is a wish. “Add an alert for checkout error rate above five percent for five minutes, owner Priya, due December 5, verified by a game day” is an action item. I cap the list at three to five because a twenty-item list is a list nobody finishes. Then I review the open items in staff meeting like any other committed work, because a postmortem without follow-through teaches the team that the ritual is performative.
I also share a sanitized summary beyond the immediate team. Cross-team patterns, such as a dependency nobody knew about or a monitoring blind spot shared by three services, only become visible when more than one team reads what happened. The metrics conversation belongs here too: detection time, recovery time, recurrence, and action completion are system measures, not individual performance targets. I treat them the same way I treat any other engineering team performance metric: as a trend to improve, never as a scoreboard for a person.
The mistakes I still watch for
I have made every mistake on this list, most of them more than once. They are worth naming because each one feels productive in the moment, and none of them announces itself as a mistake until the postmortem.
The test I use for my own leadership is simple: if I were unavailable during the next significant incident, would the team know how to organize the response? Not whether they would make every decision the way I would, but whether they would know how to assign roles, communicate, escalate, decide, and close. If the answer is no, the fix is not to become more involved. It is to build the bench, document the authority, and let someone else command while the stakes are low enough to learn.
That is also the cure for the instinct this article is about. The more the team owns the response, the less reason I have to reach for the keyboard. My job is to make myself unnecessary in the technical path and unmistakable in the coordination layer. The best incident I lead is one where the engineers barely notice I am there, and everyone still knows exactly what we are doing and why.
Common questions about leading production incidents
Q1Should the engineering manager be the incident commander?
Not automatically. The right commander is the person who can keep the response organized, make or facilitate decisions, and remove blockers. A manager can hold the role well, especially on a small team, but should not also become the primary technical responder. If the manager is the best commander and the best debugger in the room, the team has a staffing problem to solve after the incident.
Q2Should I join every production incident call?
No. I join when my involvement adds something the response does not already have: cross-team coordination, an authority decision, a stakeholder relationship, or a genuine technical gap. A contained incident owned by an experienced engineer does not need a manager on the bridge. Joining everything trains the team to expect it and makes my presence feel like supervision instead of support.
Q3How often should stakeholders receive updates?
For a significant incident, every fifteen to thirty minutes is a reasonable starting point; lower-severity incidents can move to thirty or sixty minutes. The exact number matters less than committing to it early and keeping it. An update that says nothing has changed is still valuable, because it tells people when the next real update will arrive.
Q4Should we fix the issue or investigate the root cause first?
Restore service first when customers are affected. Mitigation and diagnosis can run in parallel as long as the team is clear about which one is leading. The exception is an irreversible action: a data restore, a destructive migration, or a failover with consistency risk deserves a deliberate pause even while the incident is active. Reversible mitigations should not wait for a complete explanation.
Q5What if an engineer makes a mistake during an incident?
Deal with the immediate impact first, then understand what information and safeguards the person had at the time. A mistake made under pressure with incomplete information is a system problem before it is a performance problem. If there is a genuine conduct or repeated capability issue, handle it separately and privately. Blaming someone in the incident channel guarantees that the next person hides the next mistake.
Q6How do I stop the same incident from happening again?
Turn the postmortem into a small number of owned, dated actions and track them like product work. Look for the conditions that made the failure possible, not just the change that triggered it: the missing canary, the alert that fired late, the runbook nobody had tested. Then verify the fix with a game day or a drill instead of assuming the ticket closed the gap.
Q7What if the team escalates every difficult problem?
Treat it as a signal about ownership, documentation, or authority rather than a discipline problem. Do engineers know which decisions they can make alone? Are the runbooks current? Does escalation get rewarded or quietly punished? The goal is calibrated judgment, not zero escalations. A team that never escalates is not confident; it is either lucky or hiding something.
Q8How do I stay technical without taking over?
Stay technical enough to ask good questions, challenge assumptions, and recognize a bad plan. Read the timeline, ask what has been ruled out, and review the mitigation options. Do not become the person executing the fix unless the team explicitly asks and you have handed command to someone else. Technical fluency is an asset in the commander seat. Technical ownership is what turns the commander seat empty.
The team does the fixing. The process holds the shape. My job is to make sure nobody has to hold it in their head alone.
That means naming the roles before the root cause, communicating on a cadence instead of when asked, escalating early enough to matter, and deciding at the speed the incident requires rather than the speed that feels comfortable.
The incident is over when the system is verified, the temporary changes have owners, and the postmortem has produced work someone will actually finish. Everything before that is coordination, and coordination is the job.