The legacy system I inherited and slowly came to respect

My first plan for the old Batch Processor was textbook architectural hubris. I saw a collection of monolithic Perl scripts, cron jobs, and SFTP folders, and my mind immediately started designing its replacement. The goal was to methodically strangle the old beast, a process Martin Fowler famously named the Strangler Fig Application.

The new design was elegant on paper. I envisioned a fleet of stateless microservices, a proper message queue, and dashboards that would light up with telemetry. I was thinking about team autonomy and independent scaling. I was wrong about what the system actually needed.

The Arrogance of the Modernizer

The system had performed its critical financial task without significant error for fifteen years. But I saw only its sins. The code was procedural. Deployment was a manual file copy. To me, it was a relic, built without the patterns I favored. For the first six months, my primary job was to keep the old system running while I architected its successor. That’s when my real education began.

While my team and I wrestled with container orchestration and the subtle failure modes of our new distributed system, the old one just kept running. It was the quietest thing in our stack. It never paged me. It never consumed unexpected resources. It just picked up files, processed them, and put down new ones.

A First Lesson in Quiet Competence

Its resilience came from its staggering simplicity. The entire process was idempotent. If a run failed, the cron job would simply trigger it again an hour later. It would re-process the same input file and overwrite the partial output. The blast radius of any failure was a single file.

Our new, elegant system, by contrast, had a dozen potential points of cascading failure. A poison pill message could stall the queue for everyone. A database deadlock could halt the entire fleet of processors. The old system had none of this fragility.

Uncovering the Architectural Scars

My contempt turned into curiosity. I started digging through the ancient version control history and found an engineer who had touched the system in its early days. He laughed when I described my replacement plan. Every "bad" design choice I had mocked was a scar from a production outage—a perfect illustration of the principle now known as Chesterton's Fence. Before removing a fence, you must first understand why it was put there.

• Why flat files over SFTP? Because the network between data centers was unreliable, but SFTP’s retry mechanisms were battle-tested.
• Why a single monolithic script? To eliminate distributed state, making the entire process easy to reason about from a single file.
• Why no database? To remove a massive single point of failure. The filesystem was the database. As Martin Kleppmann details in Designing Data-Intensive Applications, this is a valid—if slow—approach that prioritizes durability above all else.

The system wasn't built from ignorance. It was forged in fire. Its architects had favored brutal pragmatism and optimized for one thing: not breaking at 3 AM.

Durability as a Deterministic Bedrock

This was a humbling lesson. The architecture I had designed was modern, but it was also fragile. It had more moving parts, more dependencies, and a far more complex recovery plan than "rerun the script." My approach completely missed the philosophy that Dan McKinley later articulated so well in his essay Choose Boring Technology.

This respect for a deterministic core is more critical than ever. Agentic systems are probabilistic; they produce a range of possible outcomes. To use them safely in production, you must bound them with auditable systems that are utterly predictable. A boring, reliable workhorse like this legacy processor is the perfect intake and out-take valve for an unpredictable LLM agent, ensuring the inputs are clean and the outputs are sane.

A Modern Architecture of Respect

We never did fully replace that old system. We improved it. We added better monitoring and put the script into a container for predictable execution, but we kept its core architecture—the simple loop of read, process, write. We respected its scars. That respect now informs how I think about composing modern systems, where deterministic and agentic components must cooperate.

What I Carry Forward

I still love building with modern tools. But now, I start with a deep respect for the boring, the simple, and the systems built to last. The lessons are now core to my philosophy:

• Question the "why" before you criticize the "what." An ugly design often hides a hard-won lesson about a subtle failure mode.
• Model your recovery before you model your data. A system that you can’t easily restore is a liability, no matter how elegant.
• Favor simplicity, especially in the critical path. The fewer moving parts, the fewer ways it can break. Idempotency is a prerequisite for sanity.
• The most valuable system is the one that lets you sleep. Durability is the ultimate feature.