Engineering Note
A third-party service went down. Our Rails application stayed available.
That incident showed us why graceful degradation matters.
A SaaS application depends on more than the Rails app itself:
* Payments
* Email
* Search
* File storage
* External APIs
* Background jobs
The risk is that one dependency failure can affect the entire application.
We changed the design so non-critical services could fail without taking down the main user experience:
* Payment service unavailable? Keep the rest of the account accessible.
* Search unavailable? Fall back to a simpler database search.
* Email provider down? Queue messages and retry later.
* Background jobs delayed? Keep customer-facing requests available.
The goal isn't to hide that something is broken.
It's to keep the critical user journey working while the failed dependency recovers.
Important learning:
Reliability isn't only about preventing failures.
It's also about deciding what should continue working when a dependency fails.
That decision is much easier to make before production forces you to make it.
A SaaS application depends on more than the Rails app itself:
* Payments
* Search
* File storage
* External APIs
* Background jobs
The risk is that one dependency failure can affect the entire application.
We changed the design so non-critical services could fail without taking down the main user experience:
* Payment service unavailable? Keep the rest of the account accessible.
* Search unavailable? Fall back to a simpler database search.
* Email provider down? Queue messages and retry later.
* Background jobs delayed? Keep customer-facing requests available.
The goal isn't to hide that something is broken.
It's to keep the critical user journey working while the failed dependency recovers.
Important learning:
Reliability isn't only about preventing failures.
It's also about deciding what should continue working when a dependency fails.
That decision is much easier to make before production forces you to make it.