Learn · Field guide

7 ways to lie with a backtest (and how to catch each one)

TrueEdge Research · August 2026 · 9 min read

Nobody sets out to publish a fraudulent backtest. Most backtest lies are self-inflicted: reasonable-looking shortcuts that quietly guarantee a beautiful curve. We know because we manufactured one ourselves, a $100k-to-$227k equity curve that dissolved under three honest checks. Here are the seven classic lies, in roughly the order traders commit them, each with the fastest way to catch it.

1. The cherry-picked window

Start the test in March 2020 and every dip-buying strategy is a genius. End it in December 2021 and so is every crypto strategy. Choosing (or quietly nudging) the test window until the result looks best is the simplest lie, and the most common, because it rarely feels like cheating. It feels like “using relevant data.”

Catch it: demand the whole available history plus results per year. A real edge is allowed to have bad years; a windowed mirage usually earns everything inside one regime. If moving the start date by six months kills it, it was never alive.

2. Survivorship bias

Test a strategy on today’s S&P 500 constituents over the last 15 years and you have quietly excluded every company that went bankrupt, was delisted, or shrank away. Your “universe” is a list of winners chosen after the fact, so almost anything you run on it looks smart. The same bias hides in crypto (testing on coins that still exist) and in fund databases (dead funds vanish).

Catch it: ask one question of any stock-picking backtest: “is Enron in your dataset?” Point-in-time universes (the index membership as it was on each historical day) are the fix; if the data source can’t provide one, treat cross-sectional results as upper bounds, not estimates.

3. The look-ahead leak

Using information at bar T that only became available at bar T+1: computing an indicator on the close and entering at that same close, normalizing with the whole dataset’s mean, labeling with tomorrow’s high. Leaks are the hardest lie to spot because the code looks innocent and the results look phenomenal. Suspiciously phenomenal is exactly the tell.

Catch it: audit every input for its timestamp of availability, and re-run the strategy with all entries delayed by one bar. A real edge degrades a little. A leak collapses completely, because its profit lived inside the peek.

4. Over-optimization

Six indicators, each with three tuned parameters, plus a tuned stop, target, and session filter: that is a machine for memorizing history. The equity curve gets smoother with every knob, and each knob makes the strategy more specific to the exact past it was fitted on. The smoothest backtest curves are usually the most fragile live.

Catch it: perturb every parameter by 10-20% and re-run. A robust edge sits on a plateau where neighbors perform similarly; an overfit one sits on a needle where any nudge destroys it. Count your free parameters honestly, and prefer the version of the strategy with fewer.

5. Ignored costs

Commissions, spread, and slippage are small per trade and fatal in aggregate for high-frequency-of-trading systems. A pattern earning 0.05R per trade gross can be a money printer in a costless backtest and a guaranteed loser at 2 ticks round-trip. In our own futures study we modeled 2 ticks round-trip plus 1 tick of slippage, and cost drag alone disqualified entire families of high-turnover patterns.

Catch it: re-run with costs doubled. If the edge inverts, it was a costs-rounding-error, not an edge. Any backtest quoted without its cost assumptions should be treated as gross fiction.

6. P-hacking: the best of a thousand tries

Test enough variations and one will look brilliant by pure arithmetic. The expected best Sharpe of a search grows with its size even on random noise, so the winner of a big sweep is impressive by construction. This is the deepest lie because the search itself is invisible in the final chart: nobody screenshots the 21,887 losers.

Catch it: ask “how many configurations did you try before this one?” and demand the result be scored against that number. That is exactly what the Deflated Sharpe Ratio does; our own best-of-21,888 winner scored a DSR of 0.21 against a 0.95 bar and died on the spot.

7. No out-of-sample test (or a burned one)

If every bar of history was available to the search, the search has already seen the exam. And the subtler version: you held out data, tested against it, tweaked, and tested again. By the third peek your “out-of-sample” set is in-sample with extra steps, and it will flatter you like everything else.

Catch it: seal a slice of history before the first test and open it once, at the end (how to structure this). Track the number of looks. One look is evidence; two is a rumor; three is marketing.

The uncomfortable summary

Every one of these lies makes your backtest look better. None of them make your trading better. That asymmetry is why the default output of honest testing is “no edge found,” and why a tool, or a process, that never says no is not measuring anything at all.

Run the seven catches against any strategy you are about to fund, including, especially, your own favorite. What survives all seven has earned real consideration. What fails was about to fail live, with your money attached.

TrueEdge automates the catches that can be automated: full-history testing with per-run cost models, honest trial counting with the Deflated Sharpe gate, a sealed out-of-sample vault that marks itself burned if reopened, and time-bucketed persistence checks on every candidate.

Try the free tier

Simulated / hypothetical results referenced on this page have inherent limitations and do not represent actual trading (CFTC Rule 4.41). TrueEdge is a research tool, not investment advice. See the full risk disclaimer.