In-Sample vs Out-of-Sample: The Core Backtesting Split

Key Takeaways

• In-sample data is used to fit and tune a strategy; out-of-sample data is a completely separate period that must never be viewed until the final evaluation step.
• A mean-reversion rule that produced Sharpe 1.4 in-sample delivered Sharpe 0.6 out-of-sample, a gap that is normal and healthy; a near-zero gap would suggest the OOS period was identical to the training regime.
• Every time you adjust the model after looking at out-of-sample results, that data silently becomes in-sample, eliminating its value as a genuine test.
• Out-of-sample windows must be long enough to hold at least several dozen independent trades; a 90-day OOS on a monthly strategy provides only three data points.

What It Is

When you backtest a strategy, you almost always have choices to make: lookback length, signal threshold, stop-loss distance, position size. In-sample (IS) is the historical slice on which those choices are optimised. Out-of-sample (OOS) is a held-out slice the model has never seen during design.

Performance on IS tells you how well the model can fit past data. Performance on OOS tells you whether the pattern generalises. If the two numbers are similar, that is a good sign. If IS looks stellar and OOS looks weak, the model has memorised noise.

The Intuition

Any sufficiently flexible rule can be tweaked to look profitable on any fixed history. The "proof" of edge is that the rule keeps working on data it was not tuned on. Without an OOS check, you have measured the modeller's creativity, not the strategy's predictive power.

There is a subtler point. Each time you look at OOS results and then adjust the model, OOS silently becomes IS. Every "peek" is a hypothesis test against the same data, and with enough peeks, something will look significant by luck. Honest OOS is untouched OOS.

How It Works

A standard single-split workflow looks like this.

total history
|------------- in-sample 70% -------------|-- out-of-sample 30% --|
|   fit parameters, pick thresholds      |   evaluate once, stop  |

Typical splits are 70/30 or 80/20. For machine-learning workflows, a 3-way split is common: train (fit the model), validation (tune hyperparameters), test (final, untouched evaluation). The test set is looked at exactly once, at the end.

Two rules are non-negotiable for time-series data:

1. Time ordering must be preserved. Training data must come strictly before validation, which must come before test. Random shuffles, which are fine in cross-section machine learning, destroy the causal direction that makes a backtest meaningful.
2. Transformations must be fit inside the training window. Scalers, normalisers, PCA, and encoders learn statistics from the data. If you fit them on the full dataset before splitting, future information leaks into the training block. Every learned transformation must be re-fit per split.

Once OOS has been used for a final evaluation, it is burned. If you adjust the model after seeing OOS, you need a genuinely new held-out window.

Worked Example

Suppose you are testing a mean-reversion rule on daily SPY data from 2010 to 2024. That is 15 years. You hold out the last 3 years (2022 to 2024) as OOS and never look at them. The other 12 years are IS.

On IS, you sweep lookback from 5 to 50 days and entry threshold from 1 to 3 standard deviations. Your best IS parameters (lookback 14, threshold 1.8) give Sharpe 1.4 with 11 percent max drawdown. You freeze those numbers.

Now you run the frozen rule once on 2022 to 2024. OOS Sharpe is 0.6, drawdown 18 percent. The edge is real but roughly half the IS number. That gap is normal and healthy. If OOS Sharpe had been 1.3, you would suspect either luck or leakage. If OOS Sharpe had been negative, the IS result was curve-fit.

Common Mistakes

1. Treating a single IS/OOS split as definitive. One split is one realisation of history. A strategy can look strong on 2022 to 2024 and weak on 2019 to 2021 for reasons that have nothing to do with real edge. Walk-forward analysis, which rolls the split across many windows, is more reliable whenever you have enough data.

2. Touching OOS during tuning. Any adjustment made after observing OOS turns that data into IS. Bailey and colleagues describe this as multiple-testing inflation: each look is a hypothesis test, and enough tests guarantee a false positive. Write the final decision rule before loading OOS.

3. Random shuffling in time series. Standard k-fold cross-validation shuffles rows, which is fine for predicting images and disastrous for predicting prices. Use time-based splits, walk-forward, or purged k-fold so that future data never trains on past labels.

4. Fitting transformations on all data. Z-scoring on the full history, computing a rolling mean, or labelling with future returns all import information the model should not have had. These are quiet forms of look-ahead bias that survive even a correct IS/OOS split.

5. OOS windows that are too short. A 90-day OOS on a strategy that trades monthly gives you three data points. With that few trades, the OOS Sharpe is dominated by noise. Prefer OOS windows long enough to hold at least several dozen independent trades.

Frequently Asked Questions

Q: What is in-sample vs out-of-sample testing in simple terms? In-sample is the historical data you use to build and tune your strategy, you can look at it as many times as needed. Out-of-sample is a separate, later period you keep completely untouched until the strategy is finalized, then evaluate it exactly once as a final, honest check.

Q: How does in-sample vs out-of-sample testing affect investment decisions? It determines whether the strategy you trust is real or an artifact of fitting to past noise. If in-sample Sharpe is 1.4 and out-of-sample Sharpe is 0.6, you have a genuine edge that is partially overstated. If in-sample is 1.4 and out-of-sample is negative, the model memorized the past and the rule should not be traded.

Q: What is a real-world example of using this split? A mean-reversion strategy on SPY was tuned on 2010–2021 with parameters locked in, then evaluated once on 2022–2024. The 2022–2024 OOS Sharpe came in at 0.6, compared with 1.4 in-sample. That gap is expected; the OOS result is the honest estimate of live edge.

Q: What is the most damaging mistake investors make with this split? Looking at OOS results and then making "small" adjustments before trading live. Every adjustment converts the OOS data into in-sample data. Once you change anything after seeing OOS, you need a genuinely new, untouched holdout period to have a valid test.

Q: How is in-sample vs out-of-sample different from walk-forward analysis? A single split answers one train-test question. Walk-forward analysis chains many in-sample and out-of-sample pairs sequentially, covering different market regimes and providing a more robust average estimate of real-world performance.

Sources

1. scikit-learn. "Cross-validation: evaluating estimator performance." https://scikit-learn.org/stable/modules/cross_validation.html
2. Hyndman, R.J., Athanasopoulos, G. Forecasting: Principles and Practice (3rd ed.). "Time Series Cross-Validation." https://otexts.com/fpp3/tscv.html
3. AnalystPrep. "Problems in Backtesting and Biases in Data." https://analystprep.com/study-notes/cfa-level-2/problems-in-backtesting/
4. ML4Trading. "Walk-Forward Validation for Time Series." https://ml4trading.io/primer/walk-forward-validation-for-time-series/

Disclaimer

This article is educational content only and is not financial advice. Nothing here is a recommendation to buy, sell, or hold any security. Consult a licensed advisor before making investment decisions.