The integration that connected two systems that hated each other

The alert came at 2:17 AM. It wasn't for a system crash, but for a consequence: a balance sheet that wouldn't reconcile for the quarter. A transaction batch from the old monolith had failed to post to the new SaaS platform. Because the connection was a brittle, hand-rolled script, the failure was silent. In my experience, this is the inevitable outcome of treating integration as a tactical afterthought instead of a first-class architectural problem.

I've seen this story play out multiple times. It’s the story of two systems, built in different decades with different philosophies, forced into a conversation by business needs. One system speaks in rigid, verbose XML over SOAP with WS-Security headers; the other whispers lean JSON to a JWT-authenticated REST API. They don’t just have different data formats; they have fundamentally different worldviews.

The Architectural Chasm

On one side was the legacy system, a fortress of stability. Its data structures were fixed, its API contract was carved in stone, and it operated on a stately, predictable batch schedule. Change was measured in quarters, not sprints. On the other was a modern SaaS platform: agile, flexible, and constantly evolving. It was designed for a world of webhooks and ephemeral compute, not nightly file drops.

The business mandate was simple: make them work together. The technical reality was a deep mismatch in timing, state management, and schema. A direct connection was impossible.

The Trap of the Quick Script

The first temptation is always the quick fix. Someone, usually under pressure, suggests a simple script. "We'll just write a Python script that pulls the XML, converts it to JSON, and POSTs it to the new API. Run it on a cron job every night."

This path creates a tightly-coupled, invisible point of failure. When the SaaS vendor changes a field name, the script breaks. When the monolith’s batch run is delayed, the script fails with incomplete data. There's no retry logic, no dead-letter queue, and no observability. It’s the architectural equivalent of using duct tape to fix a leaking dam. You only find out it failed when the business impact is undeniable.

Building the Diplomat Service

The durable solution is not a simple connector; it’s a dedicated middleware component. The concept is well-described in Domain-Driven Design as an Anti-Corruption Layer (ACL). I think of it as a diplomat—a service that understands the language and customs of both systems and can translate, buffer, and negotiate between them.

While an off-the-shelf iPaaS (Integration Platform as a Service) can handle simple cases, the deep semantic mismatches and need for fine-grained control over failure modes often demand a custom build. The diplomat's job is to own three responsibilities:

• Translate Semantics, Not Just Syntax. Converting XML to JSON is easy. The hard work is translating meaning. A legacy integer primary key becomes a modern UUID. A fixed-decimal financial value from the monolith must be carefully handled as an integer of cents for the JSON API, never a native float. The diplomat owns this domain logic.
• Decouple Lifecycles. The diplomat insulates each system from the other's changes. When the SaaS API deprecates v2 for v3, only the diplomat's adapter needs an update. The monolith remains untouched, unaware anything changed.
• Absorb the Impedance Mismatch. The monolith sent one massive batch file daily. The SaaS API preferred thousands of small, individual calls and would rate-limit a client that sent too much at once. Our diplomat used a persistent queue to buffer the load, breaking the large batch into thousands of individual messages that workers could process at a controlled pace.

The Boring Patterns That Work at 3am

Building the happy path is maybe 30% of the work. The rest is what makes the system resilient. These are not new ideas; they are durable, well-documented concepts you'll find in classic texts like Gregor Hohpe's Enterprise Integration Patterns and modern guides like the AWS Well-Architected Framework. Craftsmanship here pays dividends.

Our diplomat was a fortress of these reliability patterns. We implemented:

• Idempotency Keys. The SaaS API supported an Idempotency-Key header. We generated a unique, deterministic key for every sub-transaction. If a worker crashed and restarted, the API would see the same key on a resent request and simply return the original success response, preventing duplicate data.
• Exponential Backoff and Retries. Network blips happen. Instead of failing immediately, our workers would retry a failed API call, waiting progressively longer between each attempt. Only after several failures would it give up.
• A Dead-Letter Queue (DLQ). When a message was truly malformed, the final retry didn't discard it. The worker moved it to a separate DLQ with error metadata. This created a worklist for a human to investigate the next day, ensuring no data was ever lost.

An Integration Is a Product, Not a Project

The core lesson is a shift in mindset. An integration layer isn't just glue code. It is a critical, first-class system that deserves the same design rigor, testing, and observability as any other user-facing service.

When you must connect two systems that were never meant to meet, resist the quick script. Design a diplomat. Model the domains, translate the semantics, and buffer the mismatched speeds. Most importantly, build for failure. The robust, boring patterns of queuing and error handling are what transform a brittle connection into a resilient piece of infrastructure—and let you sleep through the night.