Backtest case libraryAll cases · GitHub · PyPI
Backtest audit kit · public case library

Twelve ways a backtest lies.

Each case shows the mechanism, the number, and how to reproduce the finding. No signals. No return promises. Just checks.

12CASES
4DETERMINISTIC CHECKERS
18PASSING TESTS
MITLICENSE

What is in the library

Look-ahead, accounting, costs, metric definitions, statistics, numerics, on-chain yield and data semantics. Two entries are self-audits, including one where our own checker was wrong.

CASE 01
Look-ahead

The exit scanned the future on the entry bar

A perpetual strategy with a beautiful win rate produced almost nothing live. The exit logic decided stop or target by scanning forward bars at the moment of entry.

90.1% → 65.4% — key number
CASE 02
Accounting

The bust path forgot cumulative withdrawals

Strategies configured to withdraw profits looked far worse than identical strategies without withdrawals. The liquidation path dropped wd_cum, so the multiple was computed on a smaller base.

an order of magnitude — key number
CASE 03
Metric definition

The circuit breaker reset the drawdown peak

A report said max drawdown 53.1% on an account that had nearly been wiped out. On resume, the engine reset the running peak — so the all-time-high drawdown was never measured.

53.1% reported — key number
CASE 04
Statistics

5-minute bars faked the sample size

Hundreds of supposedly independent observations produced an enormous t-statistic. A dozen consecutive bars counted the same crash eleven times.

t = 15.60 — key number
CASE 05
Costs

A take-profit smaller than one round trip

A high-turnover strategy showed +30% over 83 days across 749 round trips. No fee, funding or slippage line existed anywhere in the loop.

+30% / 749 trips — key number
CASE 06
Numerics

R² says 0.6 on a series whose correlation is 1.0

An expanding R² on a perfectly linear series returned 0.14–0.69. Two nearly equal ~1e4 numbers were subtracted to recover a ~1e-12 variance.

R² 0.6 vs 1.0 — key number
CASE 07
On-chain yield

The APY you see vs the APY you get

A vault advertised 8.44% net APY. Sampling convertToAssets at past blocks gives what a holder actually earned — no indexer and no trust in the dashboard.

8.44% APY claim — key number
CASE 08
Data semantics

A column changed meaning; four engines kept reading it

A migration moved ret_24h up 24 rows. Engines kept reading the old meaning, and the final backtest reported a median terminal wealth of 0x.

0x median — key number
CASE 09
Self-audit

Our own look-ahead checker flagged a backward window

A rolling window ending today was reported as a future leak. The bug was in the checker: it tested both operands of Add independently.

false positive — key number
CASE 10
Simulation

The account that comes back from the dead

A one-line ruin guard reset a wiped account to 1% and let it keep trading. The bust vanished from the equity curve and later winners compounded from a stake that never existed.

reset to 1% — key number
CASE 11
Multiplicity

A bar that measures the search, not the skill

1,701 setups were scored and an evidence bar promoted the survivors. The bar's denominator counted only setups that reached the gates, not everything that was tried.

1,701 trials — key number
CASE 15
External claim

A +90.2% annualised claim, recomputed

A marketing email claimed +90.2% annualised. The monthly arithmetic checks out; the number is hypothetical, and a few months carry it.

+90.2% — key number
Provenance. Cases are written from reproducible artefacts: code, CSVs, blocks, or a published table. Case 11 is a re-derivation from the target's own published output, not a local re-run; case 15 recomputes a marketing email's arithmetic. Each page states its scope at the top.