Research Paper · Systematic Investing

Out-of-sample testing: why most backtests look better than they are

How walk-forward splits, embargo periods and a multiple-testing haircut separate a durable trading signal from one that only fits its own history.

Key takeaways

  • A backtest is an estimate made on data the researcher has already seen. The more variants tested, the more the best result reflects luck rather than skill.
  • Walk-forward splits with an embargo between training and test windows remove the most common forms of leakage, but they do not correct for the number of trials.
  • We record every variant a researcher runs and apply a multiple-testing haircut to the Sharpe ratio before a signal can be considered for capital.

The problem with a good-looking backtest

Every systematic strategy begins as a backtest: a simulation of how a rule would have traded over history. The simulation is useful, but it is not neutral. By the time a researcher presents a result, they have usually chosen the lookback window, the universe, the rebalancing frequency, the cost model and the start date. Each choice was made with some knowledge of the data, and each one nudges the result upward.

The effect is easy to underestimate. Suppose a researcher tests 100 variants of a signal that has no real predictive power. With ten years of monthly data, the best of those 100 variants will often show an annualized Sharpe ratio above 1.0 purely by chance. Presented on its own, without the other 99, it looks like a discovery.

This is why our research process treats a backtest as a claim to be tested rather than as evidence in itself. The question we ask is not “how good is this result?” but “how good would we expect the best result to be if nothing were there?”

Best in-sample Sharpe ratio versus its out-of-sample Sharpe ratio, by number of variants tested As the number of strategy variants tested rises from 1 to 1,000, the best in-sample Sharpe ratio climbs from about 0.6 to about 1.9, while the out-of-sample Sharpe ratio of that same selected variant falls from about 0.5 to about 0.2. 0.00.51.01.52.0 110501005001,000 Number of strategy variants tested Annualized Sharpe ratio (hypothetical) Best in-sample resultSame variant, out of sample
Values are approximate and simplified for presentation.

Walk-forward splits and the embargo

The first defense is to keep some data out of reach. A single train-and-test split is fragile: the result depends heavily on where the line is drawn, and researchers who see a disappointing test period are tempted to move it. We use walk-forward validation instead. The model is fitted on an expanding window of history, evaluated on the period that follows, and then refitted with that period added. Each test window is used once, in order, the way the strategy would have been run in practice.

Two kinds of leakage survive a naive walk-forward design, and both need explicit treatment.

Purging overlapping labels

Many signals predict returns over a horizon, for example the next 20 trading days. A training observation dated two weeks before the test window has a label that overlaps the test window. Keeping it in the training set lets the model learn from returns it is about to be graded on. We remove, or purge, any training observation whose label horizon reaches into the test period.

Embargo after the test window

Market data is serially correlated. Volatility regimes, credit cycles and liquidity conditions persist for weeks. When the training window resumes immediately after a test period, information from that period seeps back through these persistent features. We leave a gap, typically one to three months depending on the signal’s horizon, between each test window and the next training observation.

Walk-forward design with four validation folds Four folds between 2012 and 2026. In each fold the training window starts in 2012 and expands by two years, a short embargo gap follows, and a two-year test window comes after the gap. The final test window ends in 2026. Fold 1Fold 2Fold 3Fold 4 20122016202020242026 Sample period (year) Validation fold TrainingPurge and embargoTest
Values are approximate and simplified for presentation.

Counting the trials

Walk-forward testing removes leakage from a single model. It does nothing about the researcher who runs the walk-forward test 200 times with different settings and reports the best one. That is the larger source of overstatement in practice, and it can only be addressed by recording what was tried.

Our research platform logs every backtest run against the shared data store, including the parameters, the code version and the result. When a researcher proposes a signal, the review includes the full count of variants run on that idea, not only the one presented. We then apply a multiple-testing haircut: the observed Sharpe ratio is compared with the maximum Sharpe ratio we would expect from that many independent trials of a signal with no skill. The adjustment, often called a deflated Sharpe ratio, also accounts for the length of the sample and for skew and fat tails in the returns.

The practical consequence is a higher bar for ideas that took many attempts. A signal with a hypothetical in-sample Sharpe ratio of 1.2 found on the first try may pass. The same figure found after 300 variants usually does not.

Effect of the multiple-testing haircut on one hypothetical signal
Variants testedObserved SharpeExpected best Sharpe if no skillPasses review
11.20.0Yes
201.20.6Yes, with smaller initial risk
3001.21.1No

What we require before capital

A signal that clears the statistical tests still faces three practical checks before it reaches a portfolio managed by our systematic strategies team:

  • An economic reason. The researcher writes down why the effect should exist and who is on the other side of the trade. A result without a plausible mechanism gets a larger haircut.
  • Independent reproduction. A colleague who did not build the signal re-runs it from the logged code and data versions and confirms the result.
  • A paper-trading period. The signal runs live without capital for a set period, and its live results are compared with the backtest’s prediction for the same dates.

None of these steps guarantees that a strategy will work. Markets change, and some decay is expected even for genuine effects. The aim is narrower: to make sure the number we act on is an honest estimate of what the strategy can do, rather than the best number the data could be made to produce.

The most useful number in a research review is often not the Sharpe ratio. It is how many times the researcher tried before finding it.

This material reflects the views of the author as of the date of publication, is subject to change, and is provided for informational purposes only. It is not investment advice or a recommendation to buy or sell any security. All Sharpe ratios and backtest results shown are hypothetical. Hypothetical results have inherent limitations: they are prepared with the benefit of hindsight, do not reflect actual trading, and may not account for the impact of market liquidity, costs or other factors on decision-making. Quantitative models may not perform as expected. Investing involves risk, including possible loss of principal.

Keep reading

More from Insights.

All insights