Research Paper · Systematic Investing
Out-of-sample testing: why most backtests look better than they are
How walk-forward splits, embargo periods and a multiple-testing haircut separate a durable trading signal from one that only fits its own history.

Key takeaways
- A backtest is an estimate made on data the researcher has already seen. The more variants tested, the more the best result reflects luck rather than skill.
- Walk-forward splits with an embargo between training and test windows remove the most common forms of leakage, but they do not correct for the number of trials.
- We record every variant a researcher runs and apply a multiple-testing haircut to the Sharpe ratio before a signal can be considered for capital.
The problem with a good-looking backtest
Every systematic strategy begins as a backtest: a simulation of how a rule would have traded over history. The simulation is useful, but it is not neutral. By the time a researcher presents a result, they have usually chosen the lookback window, the universe, the rebalancing frequency, the cost model and the start date. Each choice was made with some knowledge of the data, and each one nudges the result upward.
The effect is easy to underestimate. Suppose a researcher tests 100 variants of a signal that has no real predictive power. With ten years of monthly data, the best of those 100 variants will often show an annualized Sharpe ratio above 1.0 purely by chance. Presented on its own, without the other 99, it looks like a discovery.
This is why our research process treats a backtest as a claim to be tested rather than as evidence in itself. The question we ask is not “how good is this result?” but “how good would we expect the best result to be if nothing were there?”
Walk-forward splits and the embargo
The first defense is to keep some data out of reach. A single train-and-test split is fragile: the result depends heavily on where the line is drawn, and researchers who see a disappointing test period are tempted to move it. We use walk-forward validation instead. The model is fitted on an expanding window of history, evaluated on the period that follows, and then refitted with that period added. Each test window is used once, in order, the way the strategy would have been run in practice.
Two kinds of leakage survive a naive walk-forward design, and both need explicit treatment.
Purging overlapping labels
Many signals predict returns over a horizon, for example the next 20 trading days. A training observation dated two weeks before the test window has a label that overlaps the test window. Keeping it in the training set lets the model learn from returns it is about to be graded on. We remove, or purge, any training observation whose label horizon reaches into the test period.
Embargo after the test window
Market data is serially correlated. Volatility regimes, credit cycles and liquidity conditions persist for weeks. When the training window resumes immediately after a test period, information from that period seeps back through these persistent features. We leave a gap, typically one to three months depending on the signal’s horizon, between each test window and the next training observation.
Counting the trials
Walk-forward testing removes leakage from a single model. It does nothing about the researcher who runs the walk-forward test 200 times with different settings and reports the best one. That is the larger source of overstatement in practice, and it can only be addressed by recording what was tried.
Our research platform logs every backtest run against the shared data store, including the parameters, the code version and the result. When a researcher proposes a signal, the review includes the full count of variants run on that idea, not only the one presented. We then apply a multiple-testing haircut: the observed Sharpe ratio is compared with the maximum Sharpe ratio we would expect from that many independent trials of a signal with no skill. The adjustment, often called a deflated Sharpe ratio, also accounts for the length of the sample and for skew and fat tails in the returns.
The practical consequence is a higher bar for ideas that took many attempts. A signal with a hypothetical in-sample Sharpe ratio of 1.2 found on the first try may pass. The same figure found after 300 variants usually does not.
| Variants tested | Observed Sharpe | Expected best Sharpe if no skill | Passes review |
|---|---|---|---|
| 1 | 1.2 | 0.0 | Yes |
| 20 | 1.2 | 0.6 | Yes, with smaller initial risk |
| 300 | 1.2 | 1.1 | No |
What we require before capital
A signal that clears the statistical tests still faces three practical checks before it reaches a portfolio managed by our systematic strategies team:
- An economic reason. The researcher writes down why the effect should exist and who is on the other side of the trade. A result without a plausible mechanism gets a larger haircut.
- Independent reproduction. A colleague who did not build the signal re-runs it from the logged code and data versions and confirms the result.
- A paper-trading period. The signal runs live without capital for a set period, and its live results are compared with the backtest’s prediction for the same dates.
None of these steps guarantees that a strategy will work. Markets change, and some decay is expected even for genuine effects. The aim is narrower: to make sure the number we act on is an honest estimate of what the strategy can do, rather than the best number the data could be made to produce.
The most useful number in a research review is often not the Sharpe ratio. It is how many times the researcher tried before finding it.
This material reflects the views of the author as of the date of publication, is subject to change, and is provided for informational purposes only. It is not investment advice or a recommendation to buy or sell any security. All Sharpe ratios and backtest results shown are hypothetical. Hypothetical results have inherent limitations: they are prepared with the benefit of hindsight, do not reflect actual trading, and may not account for the impact of market liquidity, costs or other factors on decision-making. Quantitative models may not perform as expected. Investing involves risk, including possible loss of principal.


