Building systems in a language I couldn't read

The cursor blinked on a green-screen terminal. I was responsible for a system that priced millions in financial instruments every night, and it was written in a proprietary language I couldn't read. The authors were gone, the documentation was folklore. This is a classic engineering problem: managing an opaque, critical system. It’s a problem we’re all facing again, not with legacy code, but with the non-deterministic LLM agents we’re now integrating into our products.

The Lure of the Wrong Problem

My first impulse was to learn the language. This is a common, and I think mistaken, reaction. The real leverage isn't in deciphering syntax; it's in controlling the architecture. Other seemingly obvious paths are just as treacherous. A "big bang" rewrite of a critical, poorly understood system is a famous recipe for failure. Hiring the world's last remaining expert in the language creates a single point of failure and does nothing to fix the underlying architectural debt.

The core issue isn't the code. The issue is its opacity. I realized I needed to stop trying to look inside the box and instead focus on building a better box around it.

From Code to Contracts

My breakthrough came when I shifted focus from the system's implementation to its contract. I treated the core logic as a black box. I couldn't see the gears, but I could meticulously map what went in and what came out. I traced the data, not the code. Where did inputs originate? What was their exact schema? Which downstream reports and databases depended on the output?

By diagramming the flow of data around the system, the black box's connections to the rest of the world became clear. This architectural view grants you control. You don't need to know how it works, only that for a given input, it must produce an output that conforms to a predictable, verifiable structure. The system's behavior becomes your object of control, not its internal state.

Building a Deterministic Harness

With the contracts defined, I started building protective scaffolding in a modern language I knew well. This approach is a classic one, a pattern Martin Fowler famously documented as the StranglerFigApplication. My harness had two main jobs:

• Pre-flight Checks: Before the legacy process ran, the harness performed rigorous validation on all inputs. It caught the schema violations and data inconsistencies that used to cause cryptic late-night failures. A clean error from the harness is infinitely better than a core dump from the black box.
• Post-flight Audits: After the process finished, the harness inspected the output. It verified file integrity, checked for data anomalies, and ensured the output conformed to its contract before it could affect any downstream systems.

I also wrapped the execution in modern observability. I couldn't add logs inside the legacy code, but I could log every input, output, execution time, and memory profile from the outside. The opaque core was still a mystery, but its operational health was suddenly illuminated on a dashboard.

The Modern Black Box: Agentic Systems

That experience feels more relevant today than ever. We're all being asked to integrate a new kind of black box into our systems: LLM agents. Their internal reasoning is often inscrutable and their behavior non-deterministic. An agent might produce perfectly structured JSON nine times and then inexplicably invent a new field on the tenth.

While prompt engineering is a vital tool, relying on it alone for production reliability is like hoping the legacy system just won't fail tonight. The durable architectural pattern is the same: wrap the non-deterministic agent in a deterministic harness. An agent shouldn't execute actions directly. It should propose a plan—a structured API call, a database query—which is then passed to your harness. The harness validates it. Is the syntax correct? Does this action obey business rules? Does it exceed cost or safety limits? Only a validated plan is executed.

This approach mirrors the core idea in influential research like the ReAct paper, where an agent first reasons to generate a plan ("Act") which is then carried out by a deterministic tool. The harness is the tool.

An Architecture for Taming Opacity

Whether you're facing a thirty-year-old mainframe or a state-of-the-art AI, the principles for managing opacity are the same. When you can't trust the internals, you must build trust at the boundaries. The most durable skills in our field are not language-specific, but are the architectural patterns that manage risk and stand the test of time.

Here's what I've learned to do:

• Map the Boundaries First: Ignore the internals. Obsess over the inputs and outputs. Document the system's contract with the outside world.
• Build the Harness: Before changing a single line of opaque code or deploying a new agent, build external validation, verification, and observability. This is your safety net.
• Evolve with the Scaffolding: The harness isn't just for stability; it's your platform for incremental modernization. It allows you to replace, upgrade, or tune the opaque core piece by piece, with confidence.