Remote Capacity Is Not Resilience: Governing Shared Compute Dependencies
Software
Spare compute is not resilience. How our team added capacity to a production tier through a job contract, and why remote memory must never become the safety net.
Published November 2026, dated to the month the work was done.
TL;DR: Extra compute is not extra resilience. When our team expanded a production compute tier, failure testing showed that the real question was not how much work the new capacity could absorb, but what the platform was allowed to depend on. Work reaches the capacity through a job contract. Remote memory is opt-in per job, and nothing critical depends on the fabric to stay alive.
The most dangerous dependency in an infrastructure estate is often the one that first looks like spare capacity. Our team added a high-core compute node to an existing production tier. The plan was simple: connect it, share storage and code with it, and let agent workloads use it.
The connection was the easy part. The architecture question was different: what is the platform permitted to depend on? Failure testing answered it. Remote memory adds capacity. It does not add safety, and nothing critical may depend on it.
Why capacity and resilience are different
Capacity is how much work a tier can absorb. Resilience is how the tier behaves when part of it fails. The two are easy to confuse, because the same component can deliver the first and quietly undermine the second.
A fast path between nodes adds capacity. If the operating system starts to rely on that path for its own memory or storage, the path also becomes a single point of failure that did not exist before. Nothing in the dashboards changes. The risk only shows up on the day the path blinks.
That is why every new path in our estate gets a written statement of what it may and may not guarantee before anything is allowed to lean on it. In Governance-Driven Contract Design terms, the contract comes before the dependency.
A private fabric with a narrow purpose
The nodes are joined by a dedicated private fabric that carries only tier traffic. General traffic and internet access stay on the regular network. The fabric has no route to anywhere else, by design. With a route, the operating system could try to send general traffic over a path that leads only to another node.
The added node runs from a managed image rather than a hand-built install. If its configuration drifts or breaks, the team rebuilds the image, not the machine. Acceptance testing confirmed compute availability, fabric configuration and end-to-end consistency before the node took any work.
What the tests showed
The team ran an unattended test suite covering network, compute, stability and storage. In our tests, compute-bound workloads such as builds and batch jobs improved materially on the new node, and sustained load on every core passed without hardware warnings. Storage over the fabric was limited by the disks behind it, not by the fabric.
Memory-bound work did not improve. The initial memory configuration constrained bandwidth-sensitive workloads. Local language model inference, which depends on memory bandwidth more than on core count, ran slower on the new node than on the existing one. The remediation is to align the memory architecture with the workload profile. Until then, the node takes compute-bound work only: builds, batches, embeddings and tests.
One lesson came from the measurement itself. An early memory test reported a speed that was physically impossible, because its buffers fit inside the CPU cache. We replaced it with a standard STREAM triad test and report only those results. A benchmark that looks too good usually is. Every number has to add up before it informs a decision.
Work reaches the node through one contract
A fast path is only useful if work can reach it without ceremony and without guessing. Workloads do not log in to the node. They submit jobs through one contract: run the command in the matching shared workspace, return the exit code faithfully, run jobs in parallel, and skip a node that is unavailable instead of waiting on it.
The same contract serves agent sessions and a small jobs API, so a person, a script and an agent all see the same behavior. Load testing validated admission, parallel execution and completion behavior before the tier was opened to production work.
Failure testing set the rule
We evaluated remote memory as an explicitly governed capacity mechanism: memory on the existing node, offered to the added node across the fabric as swap. Under load it worked as designed.
Failure injection then showed the problem. When system memory is paged across the same fabric the system needs in order to recover, a routine network event creates a circular dependency. The node can no longer read its own memory, and core services stall until a timeout releases them.
The conclusion is a rule, not a tuning note. Network swap is capacity, not safety. I will not let the init system or core services depend on that path to stay alive. The team turned system swap over the fabric off and replaced it with something explicit: a per-job scratch space backed by remote memory, which a job must opt into. Within the outage window we tested, a job using it paused during a fabric interruption and then resumed successfully.
What the design guarantees
The outcome comes first: spare compute that workloads can reach quickly. The guarantees protect it:
- The fabric is private and cannot capture general traffic.
- The node is rebuilt from an image, so drift is contained.
- Jobs reach the node through one contract that reports every exit faithfully.
- Work is placed by workload profile, so memory-bound jobs stay where memory bandwidth is.
- Remote memory is used only when a job asks for it. The system itself never depends on the fabric to stay alive.
A fast path that quietly becomes a safety net will fail the first time that path blinks. Connecting the capacity was the easy part. Deciding what the connection is allowed to guarantee was the architecture.
Every line holds.