Engineering Note
If the same engineer gets paged every time production breaks, you don't have an on-call rotation.
You have a dependency.
We saw this happen in a Rails production system.
One engineer knew most of the critical areas:
* Deployment issues
* Background jobs
* Database problems
* AWS infrastructure
* Production incidents
So even when someone else was officially on-call, the same person was still being pulled in.
That doesn't scale.
We changed the process:
* Rotated on-call responsibility
* Added runbooks for common incidents
* Improved alert context
* Removed alerts that didn't need immediate action
* Shared incident learnings with the team
The goal wasn't to make everyone responsible for everything.
It was to make sure production didn't depend on one engineer.
We also changed how we looked at alerts.
A page should answer:
What happened?
How serious is it?
What should I check first?
If an alert doesn't need human action, it probably shouldn't wake someone up.
Important learning:
A healthy on-call system isn't about spreading the pain equally.
It's about spreading knowledge, responsibility and decision-making across the team.
Otherwise, your most experienced engineer becomes part of your production infrastructure.
We saw this happen in a Rails production system.
One engineer knew most of the critical areas:
* Deployment issues
* Background jobs
* Database problems
* AWS infrastructure
* Production incidents
So even when someone else was officially on-call, the same person was still being pulled in.
That doesn't scale.
We changed the process:
* Rotated on-call responsibility
* Added runbooks for common incidents
* Improved alert context
* Removed alerts that didn't need immediate action
* Shared incident learnings with the team
The goal wasn't to make everyone responsible for everything.
It was to make sure production didn't depend on one engineer.
We also changed how we looked at alerts.
A page should answer:
What happened?
How serious is it?
What should I check first?
If an alert doesn't need human action, it probably shouldn't wake someone up.
Important learning:
A healthy on-call system isn't about spreading the pain equally.
It's about spreading knowledge, responsibility and decision-making across the team.
Otherwise, your most experienced engineer becomes part of your production infrastructure.