Engineering Note

The third-party service that broke our production system

A single third-party service can slow down your entire system.

The outage didn't start in our system. It started in a third-party service.

Everything was normal.
Deploys were clean. Metrics looked stable.

Then requests started slowing down.
Not failing. Just waiting.

That was the signal. We traced it back to a single external API call sitting in the request path.

One dependency.
One delay.
Every request affected.

What made it worse:
- No timeout configured
- No fallback behaviour
- Same service used across multiple flows

So one slow response became system-wide latency.

We didn't remove the dependency. We contained it.
- Added strict timeouts
- Introduced circuit breaker pattern
- Moved non-critical calls to background jobs
- Cached responses where possible

After that, external delays stopped impacting user experience.

The service didn’t fail. Our dependency model did.

In SaaS systems, third-party integrations are part of your architecture.
They need boundaries, not blind trust.