Engineering Note
Most monitoring dashboards are full of data but short on answers.
One production incident taught us that having more dashboards doesn't mean better monitoring.
At one point, our team was tracking 120+ metrics across the Rails application and AWS infrastructure.
CPU. Memory. Disk. Network. Queue sizes.
Everything looked healthy.
Yet customers were reporting slow checkouts.
The problem?
We were watching infrastructure.
Not user experience.
After that incident, we simplified our monitoring strategy.
Instead of asking, "What can we measure?"
We asked, "What tells us the system is unhealthy?"
We started focusing on a handful of metrics:
* API response time (P95)
* Error rate
* Sidekiq queue latency
* Database connection usage
* Failed deployment rate
The result was surprising.
We reduced alert noise by nearly 70%, and engineers started identifying production issues much earlier because the alerts actually reflected customer impact.
The biggest lesson?
You don't need hundreds of metrics.
You need the right ones.
Because in production, the best monitoring isn't the dashboard with the most graphs.
It's the one that tells you what needs attention before your customers do.
At one point, our team was tracking 120+ metrics across the Rails application and AWS infrastructure.
CPU. Memory. Disk. Network. Queue sizes.
Everything looked healthy.
Yet customers were reporting slow checkouts.
The problem?
We were watching infrastructure.
Not user experience.
After that incident, we simplified our monitoring strategy.
Instead of asking, "What can we measure?"
We asked, "What tells us the system is unhealthy?"
We started focusing on a handful of metrics:
* API response time (P95)
* Error rate
* Sidekiq queue latency
* Database connection usage
* Failed deployment rate
The result was surprising.
We reduced alert noise by nearly 70%, and engineers started identifying production issues much earlier because the alerts actually reflected customer impact.
The biggest lesson?
You don't need hundreds of metrics.
You need the right ones.
Because in production, the best monitoring isn't the dashboard with the most graphs.
It's the one that tells you what needs attention before your customers do.