A manager’s guide to improving on-call

Being transparent with your teams

Transparency is the key to successful communication. Clarifying expectations around availability is a must when rolling out an on-call system or a change to an existing on-call system. Make sure you think through and clearly answer common employee questions such as:

• Will engineers be on-call overnight?
• If on-call overnight, is there flexibility to work from home the next day? Can an on-call employee start later the next day if they need to catch up on sleep?
• Are developers responsible for doing development work during on-call time?
• How many times per month will a developer be on call? What’s the maximum number of times a single person would be on call?
• How will you compensate on-call employees?

Providing proper training

Best practices for training on-call teams include:

• Developing a training program that addresses both process and common issues
• Providing up-to-date runbooks
• Having new employees shadow experienced on-call engineers
• Giving employees access to past incident reports so they can see how past incidents similar to the one they’re dealing with were successfully resolved

It’s also a good idea to have multiple escalation channels. The typical best practice is to have junior engineers on the primary on-call rotation and schedule senior engineers as backup or secondary rotation. This helps junior engineers develop the required on-call skills while avoiding panic when there’s an issue beyond their expertise.

Keep on-call and development duties separate

Having development duties during on-call usually means lots of context switching and interruptions, especially for companies with frequent incidents and on-call requirements.

This all usually means less development efficiency and more stress for the on-call engineers and can lead to burnout, alert fatigue, and job dissatisfaction. It can also have a negative effect on development sprints, since it’s difficult to estimate how much an on-call person can and will contribute to any given sprint.

This is why, as a best practice, we recommend keeping on-call duties and development duties separate. When on-call employees have free time, they can work on improving on-call-related documentation and automation to eventually improve the sustainability of systems and services.

Fine-tune your on-call process

A healthy on-call system can only exist if it’s constantly improved by fine-tuning processes and systems. Toward this goal, we recommend:

• Evaluating alert priority and urgency and setting systems based on that. Low urgency alerts can wait until morning, letting on-call employees get some much-needed sleep.
• Reducing false positives by classifying alerts based on factors such as root cause, originating system, message, thresholds, etc. This helps differentiate actionable alerts from the rest.
• De-duplicating related alerts to avoid alert fatigue.
• Designing rich alerts that clearly describe an issue and empower the on-call engineers to make effective decisions and apply the knowledge recorded in runbooks.
• Providing alert reports and metrics to on-call teams so that weak areas in systems can be identified and improved. (In other words: don’t let on-call teams get bogged down by the same issues over and over again.)

Review on-call reports—and adjust as needed

To keep things fair and avoid employee burnout, managers should review on-call-related reporting to see:

• How often each team member is paged or woken up
• How long each team member has been on call
• The hourly and daily distributions of on-call duty for each person
• Adjust schedules as needed to distribute work fairly.

Listen to your employees

Management should organize regular all-hands meetings with the on-call engineers to discuss problems, complaints, and areas of weakness—and then take action to resolve the issues.

On-call systems, tools, processes, people, documentation, and training aren’t static things you set and forget. As the company grows, teams learn and change, and incidents shift over time, management should be constantly reevaluating and improving their on-call programs.

The people best equipped to tell you what is and isn’t working are the on-call engineers. Listen to them. Implement changes. And, most importantly, make sure management isn’t the sole decision-maker when it comes to on-call organization and protocol. The more you empower teams to improve their own processes and practices, the more they’ll embrace on-call.