An incident response runbook turns scattered operational knowledge into a repeatable process. This guide provides a reusable framework for defining severity levels, assigning on-call roles, triaging alerts, communicating clearly, restoring service, and reviewing the quality of the response over time.
Overview
An incident response runbook is a practical operating document for managing service disruption, security events, data issues, and other conditions that require coordinated action. It should help responders answer five questions quickly: What is happening? How serious is it? Who owns the next decision? Which actions are safe to take? When should the situation be escalated?
The best runbooks are designed for use under pressure. They do not attempt to document every possible failure mode in one place. Instead, they define a common incident-management workflow and link to more specific procedures for services, infrastructure, databases, deployment systems, and customer communications.
Treat the runbook as an operational tool rather than a static policy document. Its value depends on whether responders can find it, understand it, and trust it during an incident. A useful runbook should include ownership, links to dashboards and logs, escalation contacts, decision points, and clear completion criteria.
For teams improving the wider reliability system, the runbook should connect with on-call practices, observability tools, deployment controls, and software delivery metrics. The on-call rotation best practices guide can help with staffing and handoffs, while the CI/CD pipeline checklist is useful when a release or rollback is part of the incident.
What to track
1. Severity and impact
Use a small number of severity levels with observable criteria. A four-level model is usually easier to apply than a long scale:
- Critical: A major customer-facing service is unavailable, a serious security concern is suspected, or the business impact is broad and immediate.
- High: A significant function is degraded, a workaround is limited, or the issue is affecting an important customer group.
- Moderate: The impact is contained, intermittent, or limited to a smaller part of the system or user base.
- Low: There is little immediate customer impact, but the issue requires investigation or planned remediation.
Define severity using impact, not the number of alerts or the perceived complexity of the technical cause. Include examples for common services so that responders do not spend valuable time debating labels.
2. Roles and ownership
Separate responsibilities even when one person temporarily performs several roles. At minimum, identify:
- Incident commander: Sets priorities, assigns work, makes escalation decisions, and declares the incident resolved.
- Technical lead: Directs diagnosis, mitigation, rollback, failover, or recovery work.
- Communications lead: Maintains the internal timeline and prepares stakeholder or customer updates.
- Scribe: Records timestamps, hypotheses, actions, decisions, and outcomes in the incident channel or tracking system.
- Subject-matter experts: Investigate specific systems without taking overall coordination away from the incident commander.
Record the current on-call contact, backup contact, escalation path, and the location of the incident channel. Avoid relying on personal memory or an undocumented chat group.
3. Triage data
Track the facts needed to establish scope and direction:
- Detection time and source of the alert or report
- Affected service, environment, region, tenant, or user journey
- First known symptom and the last known healthy state
- Recent deployments, configuration changes, infrastructure events, or dependency failures
- Relevant error rates, latency, saturation, availability, queue depth, and resource signals
- Customer impact, workarounds, and any data integrity or security concerns
Link directly to the dashboards, logs, traces, deployment history, feature-flag controls, and rollback instructions that responders are expected to use. If a tool requires special access, document the access owner and a tested alternative.
4. Communication and decision records
Specify when updates are required and where they belong. A practical pattern is to publish an initial status after impact is confirmed, provide updates at a predictable interval for higher-severity incidents, and communicate immediately when severity changes, mitigation begins, service is restored, or customer action is required.
Each update should state the current impact, what is known, what is being investigated, what users should do, and when the next update is expected. Avoid presenting an unverified hypothesis as a root cause.
5. Restoration and follow-up
Define what “restored” means for each service. Recovery may require more than clearing an alert: error rates may need to remain within an acceptable range, queued work may need to drain, data may need validation, and dependent services may need to recover.
Track follow-up actions with an owner, due date, priority, and verification method. Actions should address both the immediate technical weakness and the process conditions that allowed the incident to be detected late, escalated slowly, or repeated.
Cadence and checkpoints
A runbook should be reviewed on a regular schedule and after meaningful operational changes. A monthly check is appropriate for teams with frequent releases or changing infrastructure; a quarterly review may be sufficient for more stable systems. The schedule should be a minimum, not a substitute for event-driven updates.
Use the following checkpoints:
- Monthly operational check: Verify on-call names, escalation contacts, incident-channel locations, dashboard links, access permissions, and links to rollback or failover procedures.
- Quarterly workflow review: Reassess severity definitions, communication expectations, handoff practices, and the criteria for declaring an incident resolved.
- After every significant incident: Compare the documented process with what responders actually did. Update confusing steps, missing links, outdated commands, and unclear ownership.
- Before major changes: Review the runbook before a migration, architecture change, new deployment strategy, regional expansion, or substantial dependency change.
- During an exercise: Test whether a new responder can locate the right dashboard, open the incident channel, identify the escalation path, and execute a safe first action.
Keep a small change log in the runbook. Note the date, editor, reason for the change, and affected service. This makes it easier to identify which procedures need revalidation when systems evolve.
How to interpret changes
Review incident data for patterns rather than treating every event as an isolated technical failure. Useful measures include time to acknowledge, time to assign an incident commander, time to mitigate, time to restore, number of escalations, repeated incidents, alert-to-incident conversion, and the percentage of follow-up actions completed on time. These measures complement broader DORA metrics and reliability reviews; they should not be used to rank individuals.
Longer recovery time can indicate a difficult technical problem, but it can also reveal missing permissions, unclear ownership, noisy alerts, inadequate dashboards, or an unsafe rollback process. A higher incident count may reflect better detection rather than worse reliability. Interpret each change alongside deployment volume, system changes, alert coverage, and the type of impact involved.
When an incident repeats, compare the timeline and contributing conditions rather than simply adding another warning to the runbook. If responders repeatedly use an undocumented workaround, formalize it only after validating its safety. If a step is consistently skipped, determine whether it is unnecessary, too difficult to perform, or poorly placed in the workflow.
Postmortems should be blameless and specific. Describe the timeline, customer or system impact, detection, response, contributing factors, successful actions, delayed actions, and prevention work. Avoid reducing the explanation to a single person’s mistake. The goal is to improve the system, tooling, process, and decision support available to future responders.
When to revisit
Revisit the runbook whenever an incident exposes a gap, a service changes ownership, an alert is replaced, an access path changes, or a deployment and recovery procedure is redesigned. Also schedule a recurring review on a monthly or quarterly cadence so that a period without incidents does not create false confidence.
Use this compact review checklist:
- Can a responder identify severity and assign ownership without asking for undocumented context?
- Do the alert, dashboard, log, trace, deployment, and rollback links still work?
- Are the on-call and escalation details current and reachable?
- Are communication intervals and stakeholder responsibilities unambiguous?
- Does the restoration definition include data validation and dependent services where necessary?
- Have recent incidents, near misses, and exercises been incorporated?
- Does every open improvement action have an owner and a verification step?
Store the runbook where engineers already work, make it searchable, and link to it from alert notifications and service documentation. Pair the review with related reliability work, such as the MTTR guide, the change failure rate guide, and the lead time for changes guide. A runbook is effective when it stays aligned with the systems and teams it is meant to support.
