Building Reliable Systems on Top of Unreliable APIs

Modern software is built on APIs: Payments, Messaging, Auth, Analytics, CRMs, and Cloud services.

Very few products are truly standalone.

And yet, one reality becomes obvious the moment you ship to production:

Building reliable systems on top of them is one of the hardest backend problems - and one of the least talked about.

The Myth of the “Reliable API”

Most API documentation implies stability, but production tells a different story.

Even good APIs fail - just not always in predictable ways.

The mistake isn’t trusting APIs. The mistake is designing systems that assume they won’t fail.

Reliability Starts With Expecting Failure

Most developers build for the happy path. But building for reliability requires moving past the binary of success and failure. You have to stop asking what happens when things work, and start planning for the grey area of production.

If those questions don’t have architectural answers, the system isn’t reliable - it’s lucky.

Idempotency Is Not Optional

One of the first painful lessons in integrations is duplication. In a distributed system, you cannot guarantee that a request will only be sent once. Networks glitch, and when they do, the default behavior of almost every system is to try again.

Retries happen. Webhooks repeat. Clients resend requests. Without a strict idempotency strategy, your system is vulnerable to critical failures:

Practical Lessons for Implementation

• Idempotency must be enforced at the business logic level, not just the HTTP transport level. Checking for a duplicate ID in your database is more reliable than relying on a gateway.
• Keys should represent intent, not just request IDs. Use a unique identifier for the specific action (like order_id) to ensure that even if a new request is generated, the underlying intent is recognized as a duplicate.
• Assume “at least once” delivery everywhere. Design your consumers to be naturally resilient to duplicates by checking the state of the resource before modifying it.

Webhooks: The Illusion of Order

Webhooks feel like a silver bullet for real-time updates. The mental model is simple: an external service tells you when something happens. Most developers start with a dangerous assumption:

“We’ll just listen for events and update our database.”

In production, that simplicity evaporates. Webhooks arrive out of order, late, or multiple times. Sometimes, they never arrive at all. If your system treats an event as an absolute command, one dropped or delayed packet can corrupt your entire state.

To survive the chaos, decouple ingestion from processing. Acknowledge the webhook with a 200 OK immediately, push the event to a queue, and let a background worker fetch the current 'source of truth' via the API. This ensures you never act on stale or out-of-order data.

Timeouts Are a Design Decision

One of the most underrated reliability tools is the timeout. In an integrated system, waiting is contagious. If an external API hangs for 30 seconds, your application threads hang for 30 seconds. Do that enough times, and your entire service grinds to a halt.

But a timeout alone isn't enough. It must be paired with a Circuit Breaker to stop the bleeding and a clear fallback path to keep the user experience alive while the dependency recovers.

Retries Can Be More Dangerous Than Failures

Retries sound like a safety net, but in production, they are often a self-inflicted DDoS attack. When an external service struggles, thousands of your application instances retrying simultaneously will crush whatever remaining capacity that service has left. This is the Retry Storm.

“Fail and surface the issue.”

If an API returns a 400 (Bad Request) or a 401 (Unauthorized), no amount of retrying will fix the problem. Recognizing the difference between a transient network glitch and a permanent logic error is the difference between a resilient system and a broken one.

Observability Is Part of the Integration

You don’t control external APIs, but you do control how well you understand their failures. Without deep visibility, debugging a failed payment or a dropped webhook becomes a game of finger-pointing between your team and the provider.

Observability is not a luxury - it’s the only way to debug systems you don’t own. By investing in request tracing and clear error classification, you stop treating external APIs as black boxes and start treating them as measurable components of your infrastructure.

Degrading Gracefully Beats Being Correct

In a laboratory environment, systems should always be correct. But in production, reliability is often a measure of usability under pressure. When a dependency fails, your system shouldn't just roll over and die; it should shrink its footprint to protect the core experience.

This approach requires you to categorize your features into 'Critical' and 'Non-critical'. If your auth service is up, but your avatar-generation API is down, the user should still be able to log in. Reliability is the art of failing in a way that the user barely notices.

Reliability Is a Product Decision

One of the hardest truths in backend engineering is that technical excellence has a price tag. Reliability is not an accidental byproduct of good code; it is a deliberate product choice.

Engineering can build a system that never fails, but Product might not be able to afford the complexity or the bill.

Backend engineers are the ones who implement these safeguards, but the best outcomes happen when reliability is discussed explicitly at the product level. When everyone understands the cost of 'perfection', you can build a system that is robust enough for reality without being over-engineered for theory.

Final Thoughts

Building reliable systems on top of unreliable APIs isn’t about perfection.

It’s about:

• Expecting failure
• Containing damage
• Preserving user trust
• Making failure understandable, not mysterious

If you are integrating with external APIs, assume they will fail. Once you move past the 'Happy Path' and start designing for the gray areas - the timeouts, the duplicates, and the partial outages - you stop building lucky systems and start building resilient ones.

Build something that survives the chaos.

Thanks for reading.