IT Incident Response SOP Template

Purpose

Restore normal service quickly, keep everyone who is affected informed, and learn from each incident so the same failure is less likely to happen again.

Scope

Covers any unplanned interruption or degradation of a service used by staff or customers, from first alert until the post-incident review. Planned maintenance and routine service requests are out of scope.

Roles and responsibilities

Prerequisites

• An on-call rota with names and contact numbers that is kept current
• A dedicated incident channel or bridge that can be created in seconds
• A status page or other way of telling customers what is happening
• A severity table that defines each level in terms of users affected and impact
• Access to monitoring, logs, deployment history and the runbooks for each service

Procedure

1. Detect and record the alert. Whoever sees an alert or report records the time, source, what is affected and who reported it, and pages the on-call engineer. Do not wait to see if it fixes itself.
2. Declare the incident and open a channel. The on-call engineer declares an incident, opens a dedicated channel named with the date and a short description, and takes the incident commander role until someone else accepts it.
3. Set the severity. The incident commander picks a severity from your table using what users experience, not what the cause might be. Raise it if the impact grows and lower it only after impact has stopped.
4. Assign roles. Name the technical lead, communications lead and scribe in the channel. For a minor incident one person may hold several roles, but the incident commander is always separate from the person fixing.
5. Send the first update. Within the time agreed for the severity, the communications lead posts what is known, who is affected, that the team is working on it and when the next update will come. Do not guess at the cause.
6. Check recent changes. List deployments, configuration changes and provider notices from the last 24 hours. A recent change is the most common cause, so roll it back first if it plausibly explains the symptoms and rollback is safe.
7. Contain the impact. Apply the quickest safe measure: roll back, disable the faulty feature, fail over, scale up or block the traffic source. If a security incident is suspected, isolate affected systems, preserve logs and involve your security contact before changing anything else.
8. Keep the timeline and update on schedule. The scribe records each action with its time. The communications lead sends updates on the promised schedule even when there is nothing new to say.
9. Confirm resolution. Verify against user-facing checks, not only internal graphs. Watch for at least 30 minutes, or the period set for the service, before declaring the incident resolved.
10. Send the closing update. Tell affected people that service is restored, what was affected, the time window and that a review will follow. Close the incident channel to new tasks but keep it for the record.
11. Hold the post-incident review. Within five working days, review the timeline without blame, list contributing causes, and write actions with owners and due dates. Publish the summary to the people affected.

Revision history