The Parallel Test: rebuild everything, believe nothing
Disaster-recovery people have a vocabulary the rest of engineering should steal. A backup plan you’ve only read is a checklist review. Talking through it in a conference room is a tabletop exercise. But the test that counts is the parallel test: stand up the entire company’s infrastructure at a recovery site, from backups, while production keeps running — and see if the rebuilt company actually works. (Its braver sibling, the full-interruption test, actually fails production over.)
The principle under all of it: a backup you haven’t restored is a rumor.
I think architectures work the same way. Every system you’ve been running for a while is a pile of decisions made under old constraints — and you have no idea which of them are still load-bearing versus which are fossils. You can’t find out by reading the code, any more than you can validate a backup by admiring the tape.
When better tools arrive, run the parallel test
The trigger for a DR drill is the calendar. The trigger for an architectural one is the arrival of a materially better tool — because that’s the moment the old constraints visibly stop applying, and every decision they forced comes up for question.
This year handed me exactly that trigger. When I moved the system’s intelligence onto hardware I own, the constraint set changed completely: token costs collapsed toward electricity, latency became mine to engineer, and rate limits stopped existing. So instead of grafting local models onto the old shape, I did the parallel test — re-derived the system beside the running one, and interrogated every piece: would I build it this way today, or did I only build it that way because the API was there?
The answers were humbling. Some decisions survived on merit. Others turned out to be pure fossil — batching designed around per-call pricing that no longer existed, escalations to expensive models for work a cheap owned model now did better, “temporary” glue that had quietly become a foundation. Reading the code would never have found them; each looked reasonable in isolation. Only re-derivation from scratch exposed them, because from scratch, you would never write them.
Make it standing machinery, not a heroic event
A one-time rebuild is a good story. The discipline is making re-derivation routine:
- Every incumbent defends its seat. In my system, no model owns a role by tenure. New candidates run head-to-head against the incumbent in a proving ground — the crucible — on the real tasks of the job, judged blind. The incumbent that can’t re-win its seat loses it. That’s a parallel test, miniaturized and permanent.
- Re-evaluate the whole system on a schedule, adversarially. The white paper’s mid-2026 re-audit exists precisely because “it’s been running fine” is a narrative, not a check. Question everything from scratch; publish the unflattering findings; convert each one into a permanent, machine-checked invariant so the same weakness can’t creep back.
- Treat untested assumptions like untested backups: presumed corrupt. Not “probably fine.” Corrupt, until a restore proves otherwise.
The part nobody budgets for
A parallel test is expensive, and that’s the point — the expense is the honest price of the knowledge. DR teams pay it because the alternative is discovering the backups were bad on the day the building is on fire. Engineering teams skip it because the system “works,” which is exactly what the untested backup looks like right up until the restore fails.
Better tools are arriving on a cadence now — each one an invitation to re-derive, not retrofit. The teams that treat every new capability as a bolt-on will accumulate fossils forever. The teams that periodically rebuild the company in the test facility will actually know what their architecture is made of.
A restore is the only test of a backup. A re-derivation is the only test of an architecture.
The standing version of this discipline — the crucible, the re-audits, the invariant ratchet — is documented in the HMAS white paper.