Definition
Out-of-Sample Testing
Out-of-sample testing evaluates a strategy on data that was deliberately withheld during model development, providing an unbiased check of whether the discovered edge generalises beyond the fitting period.
A common Indian practice is to develop and tune a model on an in-sample period, then run it untouched on a reserved out-of-sample period, never peeking at the held-out data during research. A large gap between in-sample and out-of-sample performance is a red flag for overfitting.
Out-of-sample discipline is hard because data is finite and the temptation to keep tweaking is strong. Techniques like walk-forward analysis and cross-validation systematise the idea, but the cardinal rule remains: the more times you look at the same data, the less reliable any apparent edge becomes.
Related terms
- BacktestingBacktesting is the process of simulating a trading strategy on historical data to estimate how it would have performed, including returns, drawdowns and risk, before committing real capital.
- Walk-Forward AnalysisWalk-forward analysis is a backtesting technique that repeatedly optimises a strategy on one window of historical data and tests it on the immediately following out-of-sample window, rolling forward through time.
- OverfittingOverfitting, or curve-fitting, occurs when a strategy is tuned so closely to historical data that it captures random noise rather than a genuine pattern, and consequently fails on new data.
Plain-English explainer from The Dispatch Investors Encyclopedia. General information, not financial advice.