Engineering Note
The 3am Incident: How We Debugged a 30-Minute Outage in Production
3:07 AM.
Phone rings.
"Production is down."
No engineer likes that message.
A few years back, we faced an outage that lasted around 30 minutes on a Rails SaaS platform.
The first assumption was obvious:
Database issue.
CPU looked normal.
Memory looked normal.
But users were getting timeouts across the application.
For the first 10 minutes, everything pointed in different directions.
Then one metric stood out.
Sidekiq queue size had jumped from a few hundred jobs to more than 5,000 pending jobs.
A newly deployed integration had started generating jobs in a loop.
The workers were consuming all available resources, and user-facing requests were getting delayed.
The fix itself took only a few minutes:
* Disable the faulty integration
* Clear the runaway jobs
* Restart affected workers
Production recovered.
The interesting part came afterwards.
The outage was not caused by Rails.
Not by AWS.
Not by the database.
It was caused by a missing safeguard in one background workflow.
What changed after that incident?
* Queue growth alerts were added
* Circuit breakers were introduced for external integrations
* Deployment monitoring became stricter
* Runbooks were documented for faster response
Important learning:
The first explanation during an incident is often wrong.
Good incident response is not about reacting faster.
It is about finding the actual bottleneck before making changes.
That 30-minute outage improved our monitoring more than months of normal operations.
Phone rings.
"Production is down."
No engineer likes that message.
A few years back, we faced an outage that lasted around 30 minutes on a Rails SaaS platform.
The first assumption was obvious:
Database issue.
CPU looked normal.
Memory looked normal.
But users were getting timeouts across the application.
For the first 10 minutes, everything pointed in different directions.
Then one metric stood out.
Sidekiq queue size had jumped from a few hundred jobs to more than 5,000 pending jobs.
A newly deployed integration had started generating jobs in a loop.
The workers were consuming all available resources, and user-facing requests were getting delayed.
The fix itself took only a few minutes:
* Disable the faulty integration
* Clear the runaway jobs
* Restart affected workers
Production recovered.
The interesting part came afterwards.
The outage was not caused by Rails.
Not by AWS.
Not by the database.
It was caused by a missing safeguard in one background workflow.
What changed after that incident?
* Queue growth alerts were added
* Circuit breakers were introduced for external integrations
* Deployment monitoring became stricter
* Runbooks were documented for faster response
Important learning:
The first explanation during an incident is often wrong.
Good incident response is not about reacting faster.
It is about finding the actual bottleneck before making changes.
That 30-minute outage improved our monitoring more than months of normal operations.