The first time I had to think about architecture, not just code

An LLM agent feels like magic until it breaks the downstream pipeline for the fifth time in an hour. The server logs show a familiar, frustrating error: a malformed JSON object, a missing key, a text summary that violates a length constraint. The deterministic, predictable world of the data processing job has just been torpedoed by its clever, non-deterministic partner. My gut reaction is the same dread I felt in the late 90s, staring at a dozen broken Perl scripts.

The problem then, as now, wasn't about the logic inside a single component. It was about the connections between them. It was the first time I had to think about architecture.

The Lesson from a Fragile Script

My first multi-page web app was a collection of isolated Perl CGI scripts. Each one was a self-contained universe that connected to the database, handled its own state, and generated HTML. It worked, until I had to change the database password. I had to edit a dozen files, praying I didn't miss one. I wasn't building an application; I was curating a set of liabilities.

The pain forced a refactor. I pulled all the database code into a single DB.pm module with a clear function: connect, run a query, return results. That was it. That was its contract. Suddenly, the other ten scripts didn't need to know how the database worked. They just needed to call the module. This simple act of separation—of defining a seam and a contract—was my first real architectural decision. It wasn't about a grand design; it was a response to pain.

A Contract Is More Than a Function

This principle of separating components based on stable interfaces is one of the most durable ideas in software. It's the core of what David Parnas described in his foundational 1972 paper on modularization, a concept he called information hiding. The goal is to hide a component's internal volatility behind a contract that changes much more slowly, if at all.

In the 90s, my "contract" was just a function signature in a Perl module. Today, that's not enough. For distributed systems, our contracts need to be explicit and machine-readable, like an OpenAPI schema for a REST API or a Protobuf definition for a gRPC service. These aren't just documentation; they are enforceable agreements that tooling can use to generate clients, servers, and validation rules. They provide far stronger guarantees than my simple function call ever could.

Where Agentic Systems Break the Contract

This brings us back to the rogue LLM. An LLM agent is the ultimate volatile component. It is non-deterministic by nature. You can give it the same prompt twice and get two different answers. Asking it to produce structured data, like JSON, is a constant battle against its creative tendencies. It will hallucinate fields, misunderstand constraints, or simply fail in un-graceful ways.

Connecting an LLM agent directly to a deterministic data pipeline is like plugging a fire hose into a garden sprinkler. The interface is wrong and something is going to break. The challenge is that the agent's contract can't be defined by a simple, static schema. Its behavior is probabilistic. As practitioner Eugene Yan documents, building reliable systems with LLMs requires a whole new set of patterns specifically for handling this volatility.

The architectural question isn't "how do I make the agent smarter?" It's "how do I build a harness around the agent that enforces a contract with the rest of the system?"

An Architecture for Durable Hybrid Systems

The solution is to treat the agent as an untrusted, external system, even when it's part of your own application. We need to build an explicit "anti-corruption layer" around it. This isn't just one component; it's a series of defenses.

First, a **validation shim** receives the raw output from the LLM. It's a ruthless bouncer at the club door. It uses a schema validator (like Pydantic or JSON Schema) to check the structure, types, and constraints of the output. If the output is clean, it's passed through. If it's malformed but salvageable—a string that should be an integer—the shim can try to coerce it into compliance.

Second, if the output is unsalvageable, it doesn't trigger a system-wide failure. Instead, the orchestrator shunts the bad output and its original prompt into a **dead-letter queue**. This protects the downstream pipelines and creates a valuable dataset for later analysis and fine-tuning. The system remains stable.

This is the modern equivalent of my old DB.pm module. It's a component whose job is to hide the chaos of the LLM behind a clean, predictable, and stable interface. The rest of the system doesn't talk to the agent; it talks to the agent's well-behaved guardian.

What to Remember

• Architecture begins at the seams. It is the art of defining and defending the contracts between components, especially between predictable and unpredictable ones.
• Treat LLM agents as volatile, third-party services. Never trust their output directly. Build a harness—a validation layer, a data coercer, a retry handler—that forces their output to conform to a strict contract.
• A dead-letter queue is your best friend. When a non-deterministic component fails, don't crash the system. Isolate the failure, log it, and move on. This makes your system resilient and gives you the data needed to improve the agent.
• The goal is a changeable system. The value of this architecture isn't just that it works today. It's that next year, when a new model family comes out, you can swap out the LLM inside your harness without the rest of your system ever knowing or caring. You've isolated the volatility.