jcardena.com › Blog › Remote Capacity Is Not Resilience: Governing Shared Compute Dependencies
184 posts
EN ES

Remote Capacity Is Not Resilience: Governing Shared Compute Dependencies

Software

Spare compute is not resilience. How our team added capacity to a production tier through a job contract, and why remote memory must never become the safety net.

Remote Capacity Is Not Resilience: Governing Shared Compute Dependencies

Published November 2026, dated to the month the work was done.

TL;DR: Extra compute is not extra resilience. When our team expanded a production compute tier, failure testing showed that the real question was not how much work the new capacity could absorb, but what the platform was allowed to depend on. Work reaches the capacity through a job contract. Remote memory is opt-in per job, and nothing critical depends on the fabric to stay alive.

The most dangerous dependency in an infrastructure estate is often the one that first looks like spare capacity. Our team added a high-core compute node to an existing production tier. The plan was simple: connect it, share storage and code with it, and let agent workloads use it.

The connection was the easy part. The architecture question was different: what is the platform permitted to depend on? Failure testing answered it. Remote memory adds capacity. It does not add safety, and nothing critical may depend on it.

Add capacity tothe tierConnect over aprivate fabricRun the node froma managed imageExpose workthrough a jobcontractInject failureWrite theguarantees
From added capacity to governed compute

Why capacity and resilience are different

Capacity is how much work a tier can absorb. Resilience is how the tier behaves when part of it fails. The two are easy to confuse, because the same component can deliver the first and quietly undermine the second.

A fast path between nodes adds capacity. If the operating system starts to rely on that path for its own memory or storage, the path also becomes a single point of failure that did not exist before. Nothing in the dashboards changes. The risk only shows up on the day the path blinks.

That is why every new path in our estate gets a written statement of what it may and may not guarantee before anything is allowed to lean on it. In Governance-Driven Contract Design terms, the contract comes before the dependency.

A private fabric with a narrow purpose

The nodes are joined by a dedicated private fabric that carries only tier traffic. General traffic and internet access stay on the regular network. The fabric has no route to anywhere else, by design. With a route, the operating system could try to send general traffic over a path that leads only to another node.

The added node runs from a managed image rather than a hand-built install. If its configuration drifts or breaks, the team rebuilds the image, not the machine. Acceptance testing confirmed compute availability, fabric configuration and end-to-end consistency before the node took any work.

What the tests showed

The team ran an unattended test suite covering network, compute, stability and storage. In our tests, compute-bound workloads such as builds and batch jobs improved materially on the new node, and sustained load on every core passed without hardware warnings. Storage over the fabric was limited by the disks behind it, not by the fabric.

Memory-bound work did not improve. The initial memory configuration constrained bandwidth-sensitive workloads. Local language model inference, which depends on memory bandwidth more than on core count, ran slower on the new node than on the existing one. The remediation is to align the memory architecture with the workload profile. Until then, the node takes compute-bound work only: builds, batches, embeddings and tests.

One lesson came from the measurement itself. An early memory test reported a speed that was physically impossible, because its buffers fit inside the CPU cache. We replaced it with a standard STREAM triad test and report only those results. A benchmark that looks too good usually is. Every number has to add up before it informs a decision.

WORKLOADSAgent sessionsBuild and testjobsBatch jobsJOB CONTRACTJob runnerJobs APISkip unavailablenodesCOMPUTEExisting nodeAdded compute nodeFABRICPrivate fabricNo general routingShared workspaceSAFETYOpt-in scratchspaceNo system memoryover the fabric
The governed compute tier

Work reaches the node through one contract

A fast path is only useful if work can reach it without ceremony and without guessing. Workloads do not log in to the node. They submit jobs through one contract: run the command in the matching shared workspace, return the exit code faithfully, run jobs in parallel, and skip a node that is unavailable instead of waiting on it.

The same contract serves agent sessions and a small jobs API, so a person, a script and an agent all see the same behavior. Load testing validated admission, parallel execution and completion behavior before the tier was opened to production work.

Failure testing set the rule

We evaluated remote memory as an explicitly governed capacity mechanism: memory on the existing node, offered to the added node across the fabric as swap. Under load it worked as designed.

Failure injection then showed the problem. When system memory is paged across the same fabric the system needs in order to recover, a routine network event creates a circular dependency. The node can no longer read its own memory, and core services stall until a timeout releases them.

The conclusion is a rule, not a tuning note. Network swap is capacity, not safety. I will not let the init system or core services depend on that path to stay alive. The team turned system swap over the fabric off and replaced it with something explicit: a per-job scratch space backed by remote memory, which a job must opt into. Within the outage window we tested, a job using it paused during a fabric interruption and then resumed successfully.

What the design guarantees

The outcome comes first: spare compute that workloads can reach quickly. The guarantees protect it:

  • The fabric is private and cannot capture general traffic.
  • The node is rebuilt from an image, so drift is contained.
  • Jobs reach the node through one contract that reports every exit faithfully.
  • Work is placed by workload profile, so memory-bound jobs stay where memory bandwidth is.
  • Remote memory is used only when a job asks for it. The system itself never depends on the fabric to stay alive.

A fast path that quietly becomes a safety net will fail the first time that path blinks. Connecting the capacity was the easy part. Deciding what the connection is allowed to guarantee was the architecture.

Every line holds.

JC
Juan Cardena
Enterprise AI & Agentic Systems Architect

Enterprise architect with 25 years across web, software, data, and AI. MIT CDAO ’25. Writing on agentic AI in production.