Backtest overfitting: why the best-looking backtest is often the luckiest

What is backtest overfitting?

A backtest replays a set of trading rules on past prices to see how they would have done, and overfitting is what happens when those rules learn that past too well.

Picture two students getting ready for a math exam: one memorizes last year's answer key, while the other learns how to solve the problems. On last year's paper both get every answer right, but on this year's paper only one of them is still fine.

Past prices hold two things: patterns that may repeat (signal) and one-off accidents (noise). A computer that tunes settings cannot tell them apart, so with enough knobs it fits both, though only the signal may come back.

The curve got better at the past by getting worse at the future. That trade-off is the overfitting problem, shown in one chart.

Why does the best backtest look better than it is?

Most backtests are the winner of many tests. Five settings with ten possible values each make 10 × 10 × 10 × 10 × 10 = 100,000 combinations, and when all of them are run and the best is kept, some of that "best" is plain luck.

The Sharpe ratio helps measure how much. It compares average return above a benchmark, usually a cash rate, with the size of its swings. Here it is stated in yearly terms and the benchmark is zero. A Sharpe ratio of 1 means average return above that benchmark equals one typical yearly swing.

Worked example: how good can luck look?

The simulation below was computed for this article. Every strategy in it has zero true edge: each day's return is a random draw that averages zero. Each strategy is scored on five years of daily data (1,260 trading days), the best one is picked, and that same pick is then scored on the next two years. Each row is the average of 1,000 repeats.

Step by step, for the 45-try row:

1. One try. With no edge, five years of results land near a Sharpe ratio of 0, give or take about 0.45, which is 1 divided by the square root of 5 years.
2. The best of 45. The luckiest of 45 independent draws sits on average about 2.24 times that give-or-take above zero, using the published approximation: 2.24 × 0.45 ≈ 1.0. The simulation gives 0.99.
3. New data. The pick never had an edge, so on the next two years it falls back to about zero (−0.02).

This matches published math. Bailey, Borwein, López de Prado and Zhu (2014) show that with five years of data, about 45 independent tries are enough to expect a past Sharpe ratio near 1 from noise alone, while the expected Sharpe ratio on new data is zero. With two years of data, about 7 tries do it. Their formula gives 1.00 for the 45-try row, against 0.99 here.

One caution on the last row: a grid of 100,000 combinations of one rule is not 100,000 independent tries, because neighboring settings behave alike. The true luck bonus is smaller than 1.97, but it points the same way.

The same authors write that a researcher who does not report the number of tries "makes it impossible to assess the risk of overfitting."

How many settings is too many?

There is no universal number, and flat rules found online ("no more than 4 to 6 parameters") come without a traceable basis. The math does show a trade-off: the more versions tried, the more years of data a result needs before luck alone stops reaching a Sharpe ratio of 1. By the Bailey et al. formula, 10 tries need about 2.5 years, 45 tries about 5 years, and 1,000 tries about 10.6 years.

A second simulation shows how this plays out when rules are added one at a time to a market with one real effect. The first setting is a core idea fixed in advance, before any testing, and each extra setting is a filter picked as the best of 20 candidates on past data. One of those candidates is genuinely useful, while the rest have no link to returns.

Past results improve with almost every added setting, even though at most one of the 20 candidates in each round is real. And the one filter that really helped was picked at the first chance in only 16% of the simulated markets, because on past data a lucky filter usually looked better than the real one.

Extra filters also change the trade count. In this simulation they cut time in the market, but they can also chop one position into several entries. Either way, a result is only as precise as the independent information behind it, which is the sample-size problem from Vol 1: Expectancy.

What the research says

• Bank-built strategies lost most of their edge live. Suhonen, Lennkh and Perez (2017) studied 215 "alternative beta" strategies from 17 investment banks. They report "a median 73% deterioration in Sharpe ratios between backtested and live performance periods." The most complex strategies lost over 30 percentage points more than the simplest. A research summary of the paper adds that only 18 of the 215 (8.4%) matched or beat their backtested Sharpe ratio live.
• Backtest Sharpe ratios barely predicted results on unseen data. Wiecki, Campbell, Lent and Stauth (2016) looked at 888 trading algorithms on the Quantopian platform, each with at least six months of out-of-sample results. Those unseen-data results were simulated after the algorithms were fixed. Backtest Sharpe ratios offered "little value in predicting out of sample performance (R² < 0.025)". Put simply, they explained less than 2.5% of the differences. And "the more backtesting a quant has done for a strategy, the larger the discrepancy between backtest and out-of-sample performance."
• Even published academic findings shrink. McLean and Pontiff (2016) re-tested 97 published stock-return predictors. Returns were 26% lower out of sample and 58% lower after publication, and the authors call the 26% "an upper bound estimate of data mining effects."
• A nonsense rule can pass a hold-out test. Arnott, Harvey and Markowitz (2019) show a strategy that sorts stocks by the letters of their ticker symbols. Found on data from 1963 to 1988, it was "validated" on 1989 to 2015. Thousands of letter combinations had been searched, and a false rule still passed the held-back test.

How do systematic quants check for it?

No single test removes overfitting, but published research and practitioner write-ups describe a handful of habits that make it easier to spot.

• Count every try. The backtesting protocol of Arnott, Harvey and Markowitz asks whether every model and variable tried, successful or unsuccessful, was recorded and whether the researchers were aware of the multiple-testing issue. Without that count, a good-looking result cannot be judged.
• Mark the result down. Harvey and Liu (2015) note that "a common practice in evaluating backtests of trading strategies is to discount the reported Sharpe ratios by 50%." They then show a flat 50% is too lenient for small Sharpe ratios (below 0.4) and too harsh for large ones (above 1.0). The deflated Sharpe ratio of Bailey and López de Prado (2014) goes a step further and raises the bar with the number of tries.
• Look for a plateau, not a needle. Practitioner David Bergstrom (Build Alpha) describes the warning sign: "a moving average of 12 works but parameters of 11 and 13 result in substantial trading losses." A real effect tends to survive small changes to its settings, while a lucky one sits on a single spike.
• Start from a reason. The same protocol asks whether "the economic foundation or hypothesis" existed "before the research was conducted." Robot Wealth, a practitioner blog, favors trades "that can be rationalised in a sentence" over blind searching.
• Keep data the rule never saw. Robot Wealth puts it this way: "the purpose of the in-sample data set is not to measure the performance of a strategy." A held-back stretch of data gives a fair score, but only if it is used once and only for a rule chosen without it. That test, and its limits, is the subject of Vol 3.

Common mistakes

• Judging a backtest without the try count. On five years of data, luck alone gives the best of 3 tries a Sharpe ratio of about 0.4, and the best of 3,000 about 1.6 (Bailey et al. formula). The same 1.5 means very different things in each case.
• Re-running the hold-out until it passes. Each look at held-back data, followed by a tweak, turns it into tuning data in all but name. Arnott, Harvey and Markowitz list this as a separate checklist item.
• Reading a smooth curve as proof. In Figure 1, the steadier past curve belongs to the strategy with no edge, and smoothness can also come from missing costs or from rules that peek at data that was not yet known.
• Adding a filter for every bad stretch. Searching for another filter adds trials. In this simulation, tuning more filters widens the average gap between past and new data.
• Keeping the single best setting from a sharp peak. When the neighbors of the chosen setting do much worse, the peak is more likely noise than edge.