The Anti-Lookahead Table Had Lookahead: five silent bugs in an AI-built system
For months I’ve been building an automated trading research system almost entirely with AI coding agents — thousands of lines of Python, dozens of services, a validation gauntlet, live paper trading. Every test passed. Every dashboard was green. The system even had a table whose entire reason for existing was preventing look-ahead bias: every value stamped with the date it became knowable, so no backtest could ever peek at the future.
Last week, a systematic audit found five silent correctness bugs. One of them was in that table.
The bugs casual reading never finds
A “diversified” portfolio that was secretly two assets. One strategy held six
instruments across crypto, FX, and metals — on paper. Crypto trades seven days a week;
ETFs trade five. On the portfolio’s shared calendar, the ETF legs had NaN weekend rows, and
pandas rolling windows treat any NaN in the window as disqualifying. Result: the trend
indicator for every five-day-a-week leg was NaN on every bar, forever, and price > NaN
is False. Four of six legs were mathematically incapable of ever being held — through an
entire bull run in one of them. Backtest and live execution shared the code, so both were
wrong in perfect agreement, which is exactly why nothing ever looked broken.
The anti-lookahead table had lookahead. The point-in-time store stamped one feature with the publication date of the underlying article. But 28 rows had been classified by the ML pipeline more than a day after publication. The table asserted knowledge before the knowledge existed. The guard rail was bent in the direction it was built to block.
Backtests that read the future politely. A macro composite lagged its inputs by one day — but one input is published by its source weeks after the date it describes. Every backtest of the strategies using it saw stress readings long before any live trader could have. Live trading was fine; the grades that decided which strategies deserved to live were not.
Event studies that started before the event arrived. Reaction windows were anchored on event timestamps rather than on when the system actually ingested the event — crediting weekend reactions no live system could have traded.
And the meanest one: an earlier code variant that would have accidentally fixed the calendar bug was tried — and rejected, because reviving the dead legs made the backtest grade worse. The artifact out-graded the truth, so the artifact was kept. No individual decision was wrong. The system optimized its way into preserving a defect.
What actually caught them
Not rereading the code. I had reread that code. Every one of these fell to process:
- Adversarial review by an independent model. Every substantive change goes to a second AI whose standing instructions are to refute — severity-ranked findings, required pre-deploy confirmations, REJECT by default on anything high-severity. It has rejected changesets I believed in. It was right. (Why the referee must be code and process, not vibes, and why review has to look one level up from where you’re standing.)
- Mutation-style fixtures. For each bug class, a permanent test asserts the failure signature. One test proves a rising five-day leg inside a seven-day basket must be holdable. Its companion “mutation guard” proves the old math still reproduces the bug — so if the fixture ever stops guarding anything real, the suite says so.
- An invariant ratchet. Every finding becomes a machine-checked invariant a daily healthcheck asserts forever — among them, now: no point-in-time stamp may claim a value was knowable before its data date. Findings don’t close; they ratchet.
- Controls in every re-run. When the contaminated results were regraded under fixed code, clean strategies ran alongside as controls — and matched their original numbers to three decimals. The fix demonstrably touched only what it claimed to touch.
The lesson that generalizes
AI agents write plausible code at superhuman speed — including plausibly wrong code, at the same speed. The failure mode isn’t syntax errors; those get caught. It’s semantic debt that agrees with itself: backtest and production sharing the same wrong assumption, tests that pass because they encode the bug, metrics that improve because of the defect.
The fix is not better prompting. It’s an engineering harness: independent adversarial review, falsification-first tests, invariants that accumulate, and controls that prove your fixes did what you think they did. The harness is boring. Building it feels slower than shipping. And it pays for itself the first time it catches a bug your dashboards were happily reporting as green.
Everything above happened on a paper-trading research system — no live money, and nothing here is investment advice. The methodology is documented in the HMAS white paper. If your team ships AI-written systems and this failure class worries you — it should — that’s the work I do, and for enterprise engagements, micp.com is the front door.