The message arrived at 04:52 on a Tuesday: Reminder job complete. Recipients: 0. Status: OK. Zero recipients, marked as success, the way a broken engine can still show a green rpm needle. Anyone who has run a flight school’s back office — the booking reminders, the maintenance-log nudges, the student progress emails, the fuel-and-booking reconciliations nobody wrote down — knows this species of morning, and knows the particular silence that follows it. This piece walks through three failures we diagnosed at a training operation like ours, and the fix list that came out of the AutomationOps reference page at the Ops Atlas, applied in the order a mechanic would apply it: symptom, cause, guard, test.
Three failures, one shared root
Failure one was the silent reminder. A filter picked out students due for a stage check; that week the filter matched nobody, and the job reported success anyway. Fourteen students went unreminded, and two instructors discovered the gap from the calendar, not the system. Failure two was a tidy-up job that trimmed “stale” rows from a scheduling sheet — and one night its definition of stale caught a block of future bookings, because a column format had changed upstream. Failure three was subtler: a restart script that bounced a flaky kiosk in the dispatch room every ten minutes. It looked heroic on the monthly report. In truth it was hiding a failing cable for three weeks while staff blamed the software, ordered a tablet they did not need, and lost a weekend to a mystery that one honest alarm would have solved. None of these were exotic. All three were unowned jobs trusted by default, which is precisely the condition this discipline exists to treat.
Diagnosis: what AutomationOps calls missing guards
Reading the Atlas entry after the fact was uncomfortable in the useful way. Every one of our failures had a named countermeasure already on the page. The silent reminder needed target validation — a run handed an empty list should refuse to start, because empty never quietly means “nobody” or “everybody”. The sheet trimmer needed a blast-radius cap and a rehearsal habit. The restart script was an anti-pattern in its purest form: auto-remediation hiding a defect instead of surfacing it. What none of them had was an owner, a stated worst case, or a stop switch. In aviation terms, the jobs had been flying without checklists, and we had been signing them off from memory.
Fix one: rehearse the change, then read the rehearsal
The first habit we adopted costs ten minutes per change: run every modification against a pretend set of targets and inspect what it would have done. The one caveat we learned the hard way is that some rehearsal modes have blind spots — in a popular open-source configuration tool, check mode does not execute command-level probes unless they are explicitly included, so a spotless report can conceal the very step that misbehaves. A rehearsal with a blind spot is worse than none, because it manufactures confidence. So we added a rule: anything the rehearsal cannot simulate gets pointed at one dummy record first, and a human watches it land.
Fix two: make nothing mean stop
The reminder fix was twelve words of logic: if the recipient list is empty, halt and ask. We extended it — if the list is ten times larger than last week, halt and ask too. The industry has scar tissue here. A disk-cleaning service at Google wiped production files after an unrepaired empty entry reached its queue, and a 2025 cloud database incident saw a region go dark over an empty DNS record that lingered untouched; absence, unattended, fails systems just as surely as wrong action does. Our jobs now treat an empty or ballooning list the way a pilot treats a fuel gauge reading zero on a full tank: not as a number, but as a question.
Fix three: smaller bites, louder alarms
The sheet trimmer was rebuilt around two numbers. First, a per-run ceiling: it may process twenty rows per pass, never the whole sheet, and the first batch acts as the canary for the rest — if the first twenty look wrong, the remaining hundreds stay untouched. Second, a failure budget: after two errors in one run it stops itself, because retrying through a corrupted column at scale converts a bug into an archaeology project. Both numbers are written beside the job’s description, next to the name of the person answerable for them. Wide-reach automation without a ceiling is not efficiency; it is a bet placed with other people’s data.
Fix four: a switch, and a rehearsal of the switch
Every job now has two documented ways to die: its own pause control, and a master halt that stops the whole collection at once. The authority question is settled in advance — whoever holds the pager pulls first and explains at the post-mortem, because stopping things at 4 a.m. is not a decision you put to a vote. And once a quarter we actually pull the master switch on purpose, during a quiet hour, to prove it works and to time how long full stop takes. The Atlas recommends exactly this drill cadence for the riskier tiers, twice a year on its page, and we settled into the same rhythm; we found the first drill revealed that “stopping” the reminder job also needed a manual note to students, which is the kind of thing you want to learn during a drill rather than an incident.
The month after: the AutomationOps scoreboard
Diagnosis without numbers is just opinion, so we kept a small scoreboard. Hours genuinely returned came first: runs per month, times minutes saved per run, minus minutes spent babysitting, divided by sixty. Then the fallback rate — runs that ended with a human finishing the job by hand — and the induced-incident rate, our adaptation of the change-failure measure the DORA research made standard. Finally, the guard trip count, which surprised us: the empty-list guard fired twice in the first month, both times correctly. Every trip was a failure that did not happen. The restart script, for the record, was retired; the cable was replaced; the kiosk now stays down until a human looks at it, which is the correct amount of ambition for a script.
What a school in our position should carry from this
The lesson generalises beyond reminders and spreadsheets. An academy’s operation is full of small unattended machinery — notification flows, booking exports, log rotations, monitoring jobs — and each one is either managed equipment or a latent surprise. The Atlas’s tier idea gives you the sorting rule: read-only work can stay simple, reversible single changes stay light, wide-but-reversible jobs earn rehearsals and batches, and anything destructive or irreversible requires a second person’s approval, soft deletes, and staged rollout. The fix list that carried us through, compressed to one line each:
- Name an owner. Every job gets a person, in writing, not “the office”.
- Rehearse changes — and read the rehearsal, blind spots and all.
- Refuse nothing. An empty or ballooning target list halts the run.
- Cap the bite. Twenty rows per pass; two failures and the run stops itself.
- Wire the switches, then pull the master halt on purpose every quarter.
- Keep a scoreboard. Hours returned, fallbacks, induced incidents, guard trips.
None of this requires an engineering team; it requires the willingness to put a name, a number and a switch beside every job that runs while you sleep.
Training operations live and die by the reliability of their unglamorous systems, and the same is true of the technology around aviation itself. The monitoring approaches described in predictive maintenance in aviation treat an aircraft the way this discipline treats a job schedule — with trend lines, limits and alarms — while computer vision in aircraft inspection shows machines earning trust gradually, under human review. And for readers building careers on the human side of reliable systems, this simple guide to practical SRE skills is the natural next stop after this page.