Building an AI Research System That Can Prove Its Work

The design, failed assumptions, audit, and reconstruction of an orchestrated hierarchical multi-agent system for systematic investing.

Executive finding

The most valuable result of this project was not a profitable strategy or a novel agent hierarchy. It was discovering that a technically sophisticated research system could still fail in ordinary operational ways: it rarely placed real orders, important signals never changed state, a retirement check could not evaluate the condition it was meant to catch, and several missing-data paths allowed work to continue when they should have stopped it.

The architecture was impressive. The feedback loops were not closed. That distinction is the paper.

A system has not proved its work because its components are sophisticated. It has proved its work when the evidence needed to reject the result is present, current, and allowed to stop the process.

HMAS began as a way to scale investment research across local and commercial language models while keeping statistical acceptance outside the models. The 2026 audit showed that this separation was necessary but insufficient. A statistical check can be rigorous on paper and still provide no protection if the wrong data reaches it, if an error lets execution continue, or if nobody measures whether its decision had an effect.

The reconstruction therefore begins below the agents: broker truth, per-trade accounting, component heartbeats, stuck-signal alarms, one error stream, checks that stop on missing evidence, and acceptance tests that must demonstrate a real state change. Models remain useful workers and reviewers. They are not the source of truth and they do not approve irreversible actions.

I · The engineering question

This system is a controlled investigation of a general question: what must surround a probabilistic language model before its work can participate in a consequential decision?

Markets are a useful laboratory because the scoreboard eventually becomes external. A strategy that sounds persuasive still loses money if its premise is wrong. A research system must therefore do more than produce ideas. It must preserve the data and decisions that allow an idea to be rejected, and it must continue measuring after the backtest ends.

1.1 · Why process matters more than prediction

The three-body problem supplies the first useful metaphor. An archival NASA explanation describes general three-body motion as chaotic: the governing laws are known, yet no explicit formula predicts the motion indefinitely, even though particular stable solutions exist. A market is not a gravitational system. The narrower parallel is what matters: interacting participants can make a system highly sensitive to its starting conditions, so exact long-range prediction is a fragile objective. The engineering response is not a more confident oracle. It is a process that searches broadly, tests on unseen data, measures deterioration, and limits exposure when the future departs from the model.

Adaptive Markets supplies the economic framework. Andrew Lo begins with the dispute between market efficiency and behavioral finance—the observation that fear, greed, loss aversion, overconfidence, and imitation affect decisions. His Adaptive Markets Hypothesis does not simply choose rationality or emotion. It treats competition, adaptation, and natural selection as forces in financial interaction. Participants learn, enter, leave, copy, and change tactics. The market represented by yesterday's data is therefore not guaranteed to be the market a strategy meets tomorrow.

Liu Cixin's Dark Forest supplies a final operating metaphor. An edge is a scarce opportunity inside a competitive environment. Revealing or exploiting it can change that environment through imitation, crowding, or counter-positioning. That supports two rules: methods can be shown while live parameters remain private, and continued edge must be measured rather than assumed. It does not prove that every strategy must decay, and it cannot excuse overfitting, contaminated data, or feedback loops that never closed.

HMAS is designed to search an environment that cannot be solved, test evidence produced by participants who adapt, and withdraw when the conditions supporting an apparent edge no longer exist.

The original design followed four sound principles:

• Separate proposal from acceptance. Models may generate and critique candidates; fixed software checks decide whether a candidate advances.
• Spend in proportion to consequence. Local models handle broad, routine exploration; more expensive models are reserved for decisions where additional reasoning may justify the cost.
• Test the actual job. Models and strategies are evaluated on the work they will perform, not on reputation or a public benchmark alone.
• Keep irreversible authority outside the model path. Real-capital release requires an explicit human decision.

The audit did not invalidate those principles. It showed that they had been described more completely than they had been implemented.

II · What was built

HMAS is an orchestrated hierarchical multi-agent system. A coordinator breaks research questions into bounded assignments. Workers explore strategy ideas, inspect data, run experiments, and prepare evidence. Reviewers challenge important claims. A software evaluation layer sits outside that conversation and applies statistical checks before a candidate may advance.

2.1 · Three operating layers

2.2 · Built, audited, and planned are different states

2.3 · Why more than one model

The useful argument for multiple models is not that a crowd creates truth. Models can share blind spots and repeat the same source error. The benefit is operational: research can be divided, routine work can run locally, expensive review can be reserved for a small number of decisions, and disagreement can reveal which claims need direct checking.

Agreement between reviewers is supporting evidence, not proof. A material claim still has to resolve to source, a repeatable calculation, a test, or an observed result outside the models’ prose.

III · What the audit found

In July 2026 I re-audited the system end to end: source, live database, execution history, and the documents that described its design. The question was intentionally blunt: profitable systematic processes exist, so why was this system not profitable?

3.1 · Why the paper itself had to change

Earlier editions called the statistical layer an incorruptible referee and described an evaluation process that could not be gamed. The audit did not support those words. Software checks can reduce risk; they remain software, dependent on their inputs, error handling, and operating context. This edition removes claims of infallibility and records the contradiction instead of quietly editing around it.

IV · Decisions that followed

The audit mattered only if it changed what the project was allowed to do. Five decisions now govern the reconstruction.

1. Keep live capital disabled. Architecture work and a promising backtest do not earn the right to affect money. The replacement must first prove real fills, reconciliation, and usable per-trade records in a bounded environment.
2. Build forward instead of patching the legacy loops. The defects were structural and no live capital depended on the old machinery. New components begin with heartbeats, centralized errors, dependency checks, and absence alarms rather than receiving those controls later.
3. Treat missing evidence as a stop condition. A check that cannot run does not pass. A data import that returns only part of the expected population does not report ordinary success. Unknown is a result and must remain visible.
4. Require observed state change. A degradation check must actually remove a test candidate. A regime signal must demonstrate more than one state. An alert must be triggered under a controlled fault. Presence in source is not acceptance evidence.
5. Make temporary safeguards accountable. Every dry-run switch, manual hold, and compensating control needs an owner, a current verification, a deadline for permanent resolution, and a deliberate expiry decision.

V · Verification architecture

Verification is arranged as a ladder. Each step has a different promise. Calling the entire ladder deterministic would hide the distinction the architecture is meant to preserve.

Models can verify downward against evidence: “Does the file exist? Does the number reproduce? Did the deployed condition change?” They are less reliable when asked to settle competing impressions of prose. The ladder is designed to push important claims toward direct, observable evidence.

5.1 · Completion means the world changed

The audit also showed that a model can produce a convincing specification, remediation plan, or summary without the requested change existing in the running system. My working explanation is not that the model intentionally avoided the work. Producing the document is a plausible completion of the request, while the deployed state remains outside its view unless the environment brings that state back as evidence.

The reconstruction therefore defines completion through a check against the intended result. “Added an alert” means a controlled fault produced the alert. “Closed the feedback loop” means the loop changed a decision in a test designed to exercise it. The narrative is useful documentation; it is not the completion signal.

VI · Statistical evidence—and its boundary

The evaluation layer uses several methods because no single score captures the common ways a backtest can mislead. The methods address different failure modes. They should not be assumed to be statistically independent, and passing all of them does not prove that a strategy is real.

• Permutation testing asks whether the result could plausibly appear after the timing relationship believed to contain the signal is destroyed.
• Deflated Sharpe analysis adjusts for short samples, non-normal returns, and the fact that many alternatives were tried before a winner was selected.
• Block-bootstrap analysis resamples sequences of returns to estimate how the strategy behaves across less favorable paths while preserving some time dependence.
• Walk-forward testing repeatedly fits on earlier data and evaluates on later data that was not used for that fit.
• Forward comparison measures what happened after promotion and removes candidates whose live evidence no longer supports the backtest.

Each method can be implemented incorrectly. More importantly, all of them can be undermined by contaminated data, a biased universe, a search history that was not recorded, or an error path that supplies a favorable default. Statistical technique does not excuse source and operational discipline.

6.1 · Position sizing

The audited legacy design discussed fractional Kelly sizing while using model confidence as an approximation for win probability. That is not a defensible production probability. Fractional sizing limits exposure but does not repair a badly calibrated input. The replacement will size only from empirical evidence that has been calibrated against realized outcomes and capped by independent risk limits. Until that evidence exists, the correct size is zero.

6.2 · Event intelligence

The event pipeline can acquire, normalize, and catalogue information. In the audited system, event predictions were not consistently scored against later market reactions. A predictive layer that does not grade its predictions is a narrative layer. It may support research, but it may not influence capital as though its forecasting ability had been established.

VII · The current reconstruction

The replacement is sequenced from external truth upward. Increasing model capacity comes after the system can measure what the existing capacity produces.

1. Broker and account truth: a single record of orders, fills, positions, fees, and reconciliation differences.
2. Per-trade economics: enough detail to reconstruct realized performance and distinguish inactivity from safety.
3. Operating visibility: every component heartbeats, reports dependency health, and writes failures to one searchable stream.
4. Absence alarms: funded-but-no-fills, unchanging signals, missing expected data, and feedback checks that never change a decision.
5. Failing tests for the audited defects: each old weakness is reproduced before the replacement is accepted.
6. Research evaluation: point-in-time universes, stress-spanning validation, recorded search breadth, and forward retirement rules that work for negative results.
7. Agent orchestration: workers and reviewers rejoin once the evidence layer can reject incomplete work without relying on their narration.

This order is a judgment about leverage. More capable models placed on top of incomplete measurement would produce more activity without making the result easier to trust.

VIII · What transfers beyond finance

Finance makes the consequences visible, but none of the central lessons belongs only to trading.

• A software import needs a separate inventory of what should have arrived.
• A migration is incomplete until restoration has been exercised.
• An alert does not exist operationally until a controlled fault has triggered it.
• A model-generated change is not complete until the intended system state is observed.
• A temporary human hold is a real control only while its owner, status, and expiry remain visible.
• A second reviewer is most useful when disagreement routes the claim toward direct evidence.

This is the broader value of HMAS. It is not evidence that a swarm of language models can be trusted with a portfolio. It is a working record of what has to be designed around fallible, persuasive components before their output can participate in any consequential system.

Limits and references

This is a practitioner field report from one evolving system, not a controlled comparison of agent architectures. The audit was conducted by the same owner who designed and directed much of the work, creating selection and interpretation risk. Several causal explanations— including why earlier model-led remediation stopped at documents—remain working hypotheses.

Strategy parameters and the complete private evidence record are withheld. As a result, public readers can evaluate the method and the consistency of the stated lessons, but cannot independently reproduce the investment results. The paper does not claim that the system has established a profitable edge. It has not.

The framing draws on NASA's explanation of chaotic general three-body motion and Andrew Lo's Adaptive Markets Hypothesis. Statistical methods discussed include Bailey and López de Prado on the Probabilistic and Deflated Sharpe Ratios; Harvey, Liu, and Zhu on multiple testing in finance; White on data snooping; Politis and Romano on the stationary bootstrap; Pardo on walk-forward analysis; Kritzman, Li, Page, and Rigobon on systemic fragility; and Kelly and Thorp on capital growth and sizing. These sources define ideas and tools, not guarantees. Their value depends on correct data, implementation, and operating context.