Escalation policies for effective incident management

Hierarchical escalation

Hierarchical escalation is when an incident is passed to a team or person based on their experience level or seniority within the organization.

For example, the first responder on-call might be a junior developer new to the team. If they can’t resolve an issue, in a hierarchical organization, they pass that issue to a more senior developer. If the more senior developer also can’t resolve the issue, they again pass to a more senior developer—and on up the line until the issue is resolved.

Functional escalation

Functional escalation is when an incident is passed to a team or person best equipped to resolve it based on their skills or systems knowledge, not their seniority.

For example, the first responder on-call may be a junior developer from a team that focuses on the back end of product X. If they discover that the core problem appears to be coming from an integration with product Y, they may escalate the incident to another junior developer on the product Y team.

Automatic escalation

For teams working with a platform like Opsgenie, you can also set up rules that tell the system to automatically escalate an incident if the primary on-call person doesn’t acknowledge or doesn’t close an alert.

Some teams may favor one escalation method over another, but they aren’t mutually exclusive, and many teams use a mix of hierarchical, functional, and automatic escalation.

The escalation matrix

An escalation matrix is a document or system that defines when escalation should happen and who should handle incidents at each escalation level.

The term is used across a number of industries. Human resources may have an escalation matrix for internal issues. Call centers may have an escalation matrix for customer service issues. And IT and DevOps teams may have one or more matrices that help engineers know how and when to escalate an incident.

The level of detail in a matrix varies greatly from company to company. Some organizations might have a simple hierarchical chart, with each developer escalating to one with a higher skill level as needed. Other organizations might have situation-specific matrices that tell developers which teams to contact for different types of incidents or different severity levels. As with most things in incident management, there is no one-size-fits-all answer for how to develop your organization’s matrix

Treat your escalation policy as guidelines—not a hard and fast set of rules

Technology isn’t static and neither are your teams. Google suggests that if your SRE thinks a specific case calls for a different escalation strategy, give them the freedom to make that judgement call. The point here isn’t to create inflexible rules, but to create guidelines that apply in most situations.

Audit your on-call schedule regularly

Are there any gaps in the schedule? Do you have the right people on call? Do you have the right people on your second and third tiers of on-call? Your on-call schedules and escalation policy should work together for faster incident management.

Set smart thresholds for escalation

Not every incident is created equal, which means not every incident can or should follow the same escalation policy.

For minor incidents, you may not want to alert the on-call engineer until working hours. For major incidents, you probably need that engineer no matter what time of day it is. In the case of multiple incidents, your engineer will need to know what to tackle first and/or if they should escalate one incident to a second engineer immediately.

There’s a balancing act here between ensuring your systems have maximum uptime and meet their SLA promises and SLO goals and making sure engineers aren’t burnt out, overworked, sleep-deprived, and subject to alert fatigue.

Set clear processes for escalation

Should the escalating developer contact the appropriate team or person directly or do they need to go through the help desk? Is there a system in place that the developer should use? How will you track escalation? What responsibilities does the first responder have to make sure the incident is picked up by the next person?

These questions should be clearly addressed by your policy and clearly communicated to all on-call developers in order to keep escalations running smoothly and resolve incidents faster.