jcardena.com › Blog › Silence Is Not Health: The Run-Record Contract for Scheduled Work
184 posts
EN ES

Silence Is Not Health: The Run-Record Contract for Scheduled Work

Software

Scheduled jobs can miss cycle after cycle while the status view stays calm, if silence is modeled as success. Every run must leave a record; a missing record is an alarm.

Silence Is Not Health: The Run-Record Contract for Scheduled Work

TL;DR: For an extended period, scheduled workloads on our production platform started on time and delivered almost nothing, while the status view showed no problem. Silence had been modeled as health. The fix is one contract that every scheduled job now carries: every run leaves a record, success or failure, and a missing record is an alarm.

Scheduled work fails most expensively when the day looks quiet. Over successive cycles, workloads on our production platform started on time, ran into their time limit and were stopped. The status view stayed calm throughout.

No one disabled the path, and each piece behaved as designed. The failure was the composition: several normal behaviors lined up so that a recurring failure produced no signal. The lesson is the same one we wrote about in Security Is a Signal, Not a Checklist: you cannot defend what you cannot see, and a system you cannot see failing is not under control.

Scheduler startsthe job on timeA slow step runslongTime limit stopsthe runFailed run iscleaned upOnly success wasloggedStatus view showsno problem
How a scheduled job failed without a signal

What the workloads do

The platform coordinates scheduled content-production and distribution workloads on our compute tier: generation, enrichment, media processing and delivery to downstream channels. They write to a shared store that downstream services read.

The operating model is deliberate. Routine work runs without a person, and human attention goes to the decisions that need judgment. That model worked until several workloads began reaching their execution limits without producing an actionable signal.

Why silence looked like health

Each job has a deadline. When a run passes it, the scheduler stops the run and marks it failed. That behavior is correct. The problem was everything around it:

  • The evidence expired. Failed runs are cleaned up automatically. By the time anyone looked, the logs were gone.
  • Telemetry recorded only success. The pipeline wrote a log line when a cycle completed. A run that was stopped never reached that line, so the log showed one healthy job and silence for the rest.
  • The health check treated missing data as neutral. Failed jobs appeared as "none" instead of red. An absent signal looked the same as a good one.
  • An unchecked dependency. One delivery step depended on a grant that was no longer valid. No preflight tested it, so the job still ran its expensive path and never delivered.

None of these is unusual on its own. Together they produce a system that fails without making any noise. That is a silent betrayal: a failure operators feel as missing deliverables, with no signal in the tools in front of them.

The contract every job now carries

A job that can fail quietly is not automated. It is unattended. In Governance-Driven Contract Design (GDCD) terms, the pipeline had a clear outcome, delivered content, but no written guarantee at the telemetry door. So we wrote one:

Every run must leave a record, whether it succeeds or fails. A missing record is an alarm, not a blank.

Three controls make that guarantee real:

  • Failure is recorded before the deadline, not only success after it. Each run writes heartbeat lines while it works, and a marker that survives when the scheduler stops it.
  • Cheap checks run first. Before any expensive step, the job confirms that its dependencies, including every grant it needs to deliver, are valid. A failed preflight is a fast, loud failure instead of a slow, silent one.
  • The status check treats "nothing heard" as a problem. A job that should have reported and did not is shown as red, with the time of its last record.

The tooling here is incidental. Schedulers, container platforms and telemetry formats will change. The durable principle is that every run must produce verifiable evidence: no run without a record.

Applying the contract across the estate

The team corrected the execution limits and put the run-record contract on every scheduled workload, not only the ones that had failed. It is now part of what a scheduled job must have before it is allowed to run unattended. The review is five questions:

  • What record does a successful run leave, and where does it live?
  • What record does a run leave when something outside it stops it?
  • Which dependencies does the job need in order to deliver, and does a cheap check test each one before the expensive work starts?
  • How long does the evidence of a failed run survive, and is that longer than the time it takes someone to look?
  • What does the status view show when a job that should have reported has not?

A job that cannot answer all five is not ready to run without a person. The questions are deliberately boring. They do not depend on the scheduler, the cloud or the language, which is why they keep working when those change.

SCHEDULECron triggerTime limitRUNCheap checks firstExpensive workHeartbeat linesRECORDSuccess lineFailure lineKilled-run markerWATCHStatus checkMissing record =alarm
A scheduled job that cannot fail silently

For any automation that runs without someone watching it, the test is simple: if it stopped tomorrow, what durable evidence would prove the missed run, and what control would escalate it? If the answer is "nothing would," the system is trusting silence. Ours no longer does. Every line holds.

JC
Juan Cardena
Enterprise AI & Agentic Systems Architect

Enterprise architect with 25 years across web, software, data, and AI. MIT CDAO ’25. Writing on agentic AI in production.