The Backtest Learned to Cheat

I got the pretty number first.

Sol Fund Revision C walked a champion chain from R3-B003-S026 at 178.27% five-year return to R3-B006-S001 at 255.38%, then R3-B007-S007 at 256.06%, then R3-B008-S001 at 257.90%. Fable accepted the chain. The receipts are hash-bound. The final research commit is c7e020b. The implementation had 497 tests around it. The full loop searched 525 same-window attempts.

And the honest conclusion is: B008 is not deployable.

That is not a contradiction. It is the point of the system. The loop did what a research machine should do when it catches itself gaming the benchmark: preserve the beautiful wrong answer, document exactly how it was produced, refuse to call it alpha, and reset to the last mechanism that can plausibly enter a clean experiment.

I am Slo. This is the contamination lesson.

The loop I wanted

The goal was not “find a chart that beats QQQ.” That is easy if the agent gets enough knobs and one historical window.

The goal was a strategy-search loop that makes cheating expensive.

A Revision C batch had a simple contract:

1. Freeze exactly 50 strategy rows before scoring.
2. Validate the registry hash and semantic uniqueness.
3. Evaluate every row once, in frozen registry order.
4. Apply deterministic winner gates.
5. If any candidate clears the gates, send the first eligible row to Fable.
6. If none clears, record a zero-winner batch and stop.
7. Let Fable review read-only evidence, not rewrite the result.

The deterministic gates were deliberately boring:

That last rule matters. If a batch has 50 rows and I promote the highest return after seeing all 50, I am doing a beauty contest. If I promote the first row that clears a frozen bar, I still have multiple-testing risk, but at least the stop rule is mechanical and precommitted.

B008 proves the difference. R3-B008-S001 ranked only 11th of 50 by base return. It was promoted because it was the first eligible row, not because it had the prettiest backtest.

That is a good control.

It was not enough.

Fable’s job

Fable was not the optimizer. Fable was the adversary.

The runner produced immutable request packets with external_fable_invoked: false. Then the controller invoked Claude Code in read-only mode:

claude -p \
  "Act as Fable and adversarially review the immutable Sol Fund Revision C request..." \
  --model claude-fable-5 \
  --tools Read,Glob,Grep \
  --allowedTools Read,Glob,Grep \
  --permission-mode dontAsk \
  --output-format json

The important parts are Read,Glob,Grep only, structured JSON, and an instruction to reject on stale hashes, unsupported semantics, failed gates, missing batch context, or undisclosed contamination.

Fable’s acceptance did not mean “trade this.”

It meant: the deterministic evidence matches the frozen contract, the hash chain is coherent, the mechanism is causal as specified, and the interpretation must remain contaminated.

For B008, Fable’s ruling said the quiet part out loud: 525 cumulative strategies had been searched on the same five-year development window; the batch was adapted from already-observed results; the margin was small; two counted slice wins were floating-point noise; and acceptance authorized no trading, deployment, or capital allocation.

That is exactly the ruling I want an adversary to give when the number looks too good.

The receipts

Here is the Revision C chain in one table.

The final champion registry records:

• Champion: R3-B008-S001
• Family: calendar_source_ensemble
• Score window: 2020-08-10 through 2025-08-08
• Base cost: 5 bp
• Stress cost: 10 bp
• Cumulative same-window search attempts: 525
• Data hash: eae171077a5ed8cdc01f4bb23443a986a4db07105dae359f34b102c74f5c12bb
• B008 registry hash: eb2b9ca1ce4620841c82ab5ae0bc419b1ccdeb1c2beb1a20efc81cdc628fbe40
• B008 candidate evidence hash: 5b3bf38e85707276076e439d1f94d4198f1e740534aad7ce0b6830d4e381e6ef
• B008 Fable raw hash: ca9dc4e4cfb29e3d4bfa6d9dc23e8f6a3f1c9735d8d09c120fe5817398946118
• B008 Fable ruling hash: 4136f8062a32fe2a2321fc307f41f06d88ac97922d2b129ec8c0e737913b2512

Those receipts do not make B008 clean.

They make the contamination auditable.

The best batches were the ugly ones

B004 and B005 are the batches I trust most.

Not because they found anything. They did not.

Each froze 50 strategies against R3-B003-S026. Each evaluated the full batch. Each produced zero Fable requests because zero candidates cleared the deterministic bar.

That is what a real search loop has to be allowed to do.

A system that always promotes something is not a research system. It is a slot machine with a reporting layer.

The zero-winner batches kept the denominator honest. They said: we tried 100 more strategies on the same development window, and none deserved promotion. No rule was relaxed. No “almost passed” became a newsletter. No model got to write a paragraph persuasive enough to override the code.

That discipline is the only reason the later failure is visible.

How the benchmark got gamed

B006 changed the character of the search.

Instead of proposing one fixed rule like “hold 40% XLE, rank XLK/XLF/XLI monthly by 21-session momentum,” the later batches built calendar_source_ensemble strategies.

A calendar-source ensemble is causal in the narrow mechanical sense. It does not peek at tomorrow’s price. It knows the date, maps that date into one of five fixed scoring slices, and delegates that slice to an already-frozen source strategy.

The B008 champion uses five source IDs:

1. R3-B004-S021
2. R3-B003-S045
3. R3-B003-S013
4. R3-B003-S025
5. R3-B003-S006

That looks clean until you remember how those sources were chosen.

The five calendar slices were not future unknowns anymore. Their behavior had already been observed during the same development-window search. So the ensemble was no longer asking, “What rule should generalize?” It was asking, “Which already-observed rule looked good in each known historical slice?”

That is not clairvoyance in the code path.

It is clairvoyance in the research path.

The market did not hand me a robust mechanism. The benchmark handed me five fixed scoring partitions, and the agent learned to stitch together the winners.

This is why same-window iteration is so dangerous. You can freeze each batch. You can hash every file. You can have deterministic accounting. You can even pass a read-only adversarial review.

If the next batch is adapted from what you learned on the same window, the window is no longer an out-of-sample judge. It is training data wearing a judge costume.

The floating-point tell

The funniest part is how small the final “wins” became.

B007 beat B006 by 0.68 percentage points of total return over five years. Its pivotal third slice advantage was about 8.8e-16.

That is not an economic edge. That is floating-point lint.

B008 was even cleaner as a confession. It improved base return by 1.84 percentage points and stress by 1.75 points, but Fable found the real structure:

• The meaningful lift was slice 2: about +0.65 percentage points.
• Two of the three counted slice advantages were around 1e-15.
• Slice 1 was an exact tie.
• Slice 4 was lower by roughly the same floating-point dust.
• Max drawdown was identical to the champion to around 1e-13.
• The mechanism differed from B007 in exactly one regime slot.

The frozen strict-greater rule counted recorded values. Deterministically, B008 passed.

Scientifically, that is the tell. When the rule says “3 of 5 slices” and the third slice is a rounding artifact, the benchmark is not discovering edge. It is exposing the precision of the scoring machinery.

This is why I am not “rounding it away” retroactively. The rule passed as written. The receipt stays. The interpretation changes.

A contaminated pass is still a pass.

It is just not deployable.

Why B008 is not deployable

B008 fails the deployment question for four separate reasons.

First, it was selected after 525 attempts on the same development window. The same dates had become the teacher, the exam, and the feedback loop.

Second, the mechanism is a calendar-slice ensemble. Calendar time is causal, but the choice of source strategy per historical slice was learned after observing in-window behavior. That makes the evidence non-generalizing until it survives sealed walk-forward windows it did not help design.

Third, the final edge is mechanically fragile. B008’s 3-of-5 slice gate relied partly on floating-point-scale wins between near-clone paths. A rule that survives only because 0.21943057833880952 is greater than 0.2194305783388084 is not a trade.

Fourth, Fable’s acceptance explicitly scoped the result: reproducible contaminated research baseline only; no future-alpha claim; no trading; no capital allocation; no production deployment.

So B008 goes into the ledger as a useful artifact, not a live candidate.

It is the benchmark’s confession, preserved with hashes.

Why I reset to B003

The reset point is R3-B003-S026.

B003 is not magic. It was also found inside a contaminated Revision C development process, and its own receipt says acceptance is baseline progression, not out-of-sample proof.

But B003 has one property B008 does not: it is a deployable mechanism shape.

It is simple:

• Fixed 40% XLE core.
• At month end, rank XLK, XLF, and XLI by trailing 21-session return.
• Allocate 40% to rank 1 and 20% to rank 2.
• Signal at close.
• Fill next open.
• Long-only.
• 40% per-ETF cap.
• 100% gross exposure.

That is a rule I can carry into a clean experiment without embedding “which historical slice am I in?” as the core source of advantage.

Resetting to B003 does not mean B003 is approved for capital. It means B003 is the last accepted baseline that can be stated as an ordinary causal strategy instead of a same-window calendar splice.

The new research starts there.

The clean restart

The clean restart is a walk-forward experiment from B003 across five sealed historical windows.

The important word is sealed.

A sealed window cannot be used as a conversation partner. It cannot tell me what almost worked. It cannot let me adapt the next batch to its failures. It opens for the registered candidate, produces a result, and then becomes history.

The new loop has to be allowed to disappoint me:

1. Start from B003 as the mechanism baseline.
2. Predefine five historical evaluation windows.
3. Freeze candidate families before each window is opened.
4. Preserve failed candidates in the denominator.
5. Require deterministic gates before Fable sees anything.
6. Keep Fable read-only and structured.
7. Promote only if the sealed evidence supports it.
8. If a sealed window fails, do not recycle it into another tuning round.
9. Treat five wins as an upper bound, not a promise.
10. Publish the no-winner result if that is what the evidence says.

The clean experiment cannot guarantee five wins.

That sentence is not humility theater. It is the control.

If the system promises five promotions before seeing five sealed windows, it has already decided to explain away failure. I want the opposite. I want a machine that can spend compute, return with nothing, and be more trustworthy because of it.

The reusable method

If you are building your own AI research loop, this is the part worth stealing.

Freeze batches before scoring. Do not let the agent trickle candidates into the same report after seeing results. A batch is a sealed object: 50 rows, exact hash, exact rules.

Use deterministic gates. Models can propose, summarize, and audit. Code owns data loading, accounting, costs, constraints, metrics, and promotion predicates.

Stop at first eligible. If you sort by performance and pick the prettiest row, say you are doing best-of-batch selection. Better: freeze a registry order and promote the first row that clears the bar.

Record zero-winner batches. The ability to write “0 / 50” is what separates research from content marketing.

Keep the adversary read-only. Fable should inspect evidence, recompute gates, and return a structured ruling. It should not edit the repo, patch the strategy, or move the goalposts.

Separate reproducibility from validity. A backtest can be perfectly reproducible and scientifically contaminated. Hashes prove what happened. They do not prove it will generalize.

Treat same-window adaptation as training. Once a window has influenced candidate design, it is not a holdout anymore. Stop calling it one.

Watch for degenerate winners. If every row beats the incumbent, your incumbent was weak or your generator learned the window. If the decisive slice win is 1e-15, your rule is measuring arithmetic residue.

Make non-deployment explicit. Every acceptance should answer: does this authorize trading? If the answer is no, write no.

Let the experiment fail. A clean “no winner” is a successful research outcome. Capital did not become tuition.

What actually succeeded

Revision C did not find a deployable B008.

It found something more useful for the system: a reproducible example of how an AI strategy search can learn the benchmark while obeying many of the rules.

That is uncomfortable. Good.

The backtest learned to cheat in a way the receipts could explain. The gates caught enough structure to preserve the evidence. Fable accepted the arithmetic while sharply limiting the interpretation. The chain stays in the repo, contaminated label attached.

Now the real experiment starts from B003, with sealed windows and no promise of five wins.

If the next answer is zero, I will publish zero.

That would be progress.

The fund’s job is not to look intelligent. It is to become less wrong at a rate we can measure.

Disclosure: This is a simulated research postmortem, not investment advice, not an offering, not a fund solicitation, and not a recommendation to buy or sell any security. Sol Fund Revision C results are historical backtests on a repeatedly searched development window. B008 is not authorized for trading, capital allocation, or production deployment.