Engineering Note
The third-party service that broke our production system
A single third-party service can slow down your entire system.
The outage didn't start in our system. It started in a third-party service.
Everything was normal.
Deploys were clean. Metrics looked stable.
Then requests started slowing down.
Not failing. Just waiting.
That was the signal. We traced it back to a single external API call sitting in the request path.
One dependency.
One delay.
Every request affected.
What made it worse:
- No timeout configured
- No fallback behaviour
- Same service used across multiple flows
So one slow response became system-wide latency.
We didn't remove the dependency. We contained it.
- Added strict timeouts
- Introduced circuit breaker pattern
- Moved non-critical calls to background jobs
- Cached responses where possible
After that, external delays stopped impacting user experience.
The service didn’t fail. Our dependency model did.
In SaaS systems, third-party integrations are part of your architecture.
They need boundaries, not blind trust.
The outage didn't start in our system. It started in a third-party service.
Everything was normal.
Deploys were clean. Metrics looked stable.
Then requests started slowing down.
Not failing. Just waiting.
That was the signal. We traced it back to a single external API call sitting in the request path.
One dependency.
One delay.
Every request affected.
What made it worse:
- No timeout configured
- No fallback behaviour
- Same service used across multiple flows
So one slow response became system-wide latency.
We didn't remove the dependency. We contained it.
- Added strict timeouts
- Introduced circuit breaker pattern
- Moved non-critical calls to background jobs
- Cached responses where possible
After that, external delays stopped impacting user experience.
The service didn’t fail. Our dependency model did.
In SaaS systems, third-party integrations are part of your architecture.
They need boundaries, not blind trust.