FTP, by hand, at 2am — the original deploy pipeline

The only light came from a 17-inch CRT monitor, its hum a low companion to the server fans. On screen, two panes: my local files on the left, the live server on the right. To deploy, I’d drag a file across. That progress bar was the original deployment pipeline, and it was terrifying. Today, we're building systems with non-deterministic LLM agents that can feel just as unpredictable. To manage them, we need to remember the brutal lessons learned from that FTP client at 2am.

The Ritual of Manual Overwrites

There was no `git push`. There was no YAML file. The process was a fragile ceremony. It began with a manual `diff` to identify changed files, followed by a prayer that you hadn't forgotten a dependency. You’d fire up an FTP client like CuteFTP, connect to the production server, and stare at the two panes representing local truth and remote reality.

The tension was real. You’d carefully drag the files, watching the transfer queue. Every file was a point of failure. Did the connection drop mid-transfer, leaving a half-written, zero-byte PHP include on the server? Did you accidentally upload a local configuration file with database credentials for `localhost`? The "rollback plan" was to frantically re-upload an old backup zip and hope you got everything right in your panic.

The Architectural Sin of State Drift

The most insidious problem wasn't just human error; it was the architectural guarantee of divergence. The production server was a mutable pet. It was never a perfect mirror of a known state. Someone might have logged in to "quickly" edit a file to fix a typo, or run a `chmod 777` to get an upload form working and then forgotten about it. Every manual deployment added another layer of undocumented change.

The server’s state constantly drifted from any testable baseline. It was the antithesis of the deterministic, idempotent infrastructure we now take for granted. Each server was a unique artifact, its history a mystery. This chaos is precisely what we risk re-introducing if we let AI agents run unchecked in production environments without a deterministic harness to control them.

From Scar Tissue to Deterministic Pipelines

It's easy to look back and laugh, but the pain of those nights directly motivated the tools we have today. The core principles of modern DevOps were forged in these fires. This wasn't a single leap; it was a gradual evolution. First came scripted deploys with tools like Capistrano and Fabric, which replaced the manual drag-and-drop with automated SSH commands. Then came configuration management systems like Puppet and Chef, introducing the revolutionary idea of declaring a server's desired state.

These steps paved the way for the practices codified by thinkers like Martin Fowler in his foundational work on Continuous Integration. The concepts reached their logical conclusion in the principles of the deployment pipeline, laid out in the book Continuous Delivery by Jez Humble and David Farley. Every feature of a modern pipeline is built on the scar tissue of a 2am failure:

• Partial uploads gave us atomic deployments, where a symlink flip makes the new version appear instantaneously and completely.
• Forgotten files gave us versioned, self-contained artifacts like Docker images. We deploy a single, tested unit, not a spray of loose files.
• The manual checklist gave us automation, making deployment a boring, repeatable non-event triggered by a `git push`.

Determinism is the Bedrock for Agency

This history isn't just nostalgia. It is the playbook for building the next generation of software. We are now tasked with integrating powerful but unpredictable LLM agents into our systems. An agent that can generate its own code or take its own actions is, in effect, a return to the mutable, drifting server state of the FTP era, but on a terrifyingly more complex scale.

The only way to build these hybrid systems safely is to treat the agentic component as an untrusted, non-deterministic core, surrounded by a rigid, deterministic shell. The deployment, monitoring, and rollback of the agent's container and its supporting infrastructure must be brutally reliable and fully automated. We use the hard-won lessons of CI/CD to build a safe sandbox for the chaos of AI.

What the 2am Deploy Teaches Us About AI

The core tension of modern architecture is managing this blend of deterministic automation and agentic work. The principles we learned from fixing broken FTP uploads are the same ones we need to apply today.

• Atomicity is your safety net. Whether deploying new code or a new model version, the system must transition from one known-good state to another instantly. There can be no "in-between" state where an agent has half its tools.
• Isolate the source of variance. The non-deterministic agent is the variable. The infrastructure around it—the pipeline, the observability, the network policies—must be static, versioned, and auditable.
• A rollback plan is not optional. Your plan to disable or revert a misbehaving agent must be more reliable and faster than your plan to deploy it. This is a first-class architectural concern.
• Appreciate your pipeline. That YAML file represents decades of learning how to make systems survivable. It's the critical tool that lets us experiment with agentic intelligence without risking the entire system at 3am.