Validate an idea
Use robustness, out-of-sample and Monte Carlo to pressure-test an edge.
A good backtest is a hypothesis, not a conclusion. This guide is the workflow that separates a real edge from a lucky fit. Run it on any strategy before you believe the headline number.
1. Does it hold out of sample?#
When a hold out split exists, the header shows an always on OOS verdict: performance on data the strategy was never fit on. If it collapses out of sample, stop here: the edge was in the fitting.
Walk forward refits parameters on rolling in sample windows and grades them only on the forward window that follows: the strictest test, and the closest to how you'd retune a live strategy.
2. Is it robust, or a lucky spike?#
- Parameter sensitivity sweeps: does the result survive nearby parameter values, or does it live on an isolated spike that a slightly different period would miss?
- Multi asset robustness: does the same logic work on other markets, or only the one you picked?
- Regime context: does it work across bull, bear and chop, or only one regime?
See the Robustness reference for what each check means.
3. How much was luck?#
Run Monte Carlo to resample the trade order thousands of times. It shows the range of outcomes you could have had and how deep a drawdown to expect. If the 5th percentile path would have wiped you out, the strategy is riskier than the single equity curve suggests.
4. Discount for the search#
If you tried many configurations, read the overfitting aware statistics. A Deflated Sharpe that drops far below the raw Sharpe, or an Overfit Probability (PBO) at or above 50%, means the result is mostly selection luck.
Passing every check makes an edge plausible, not certain. Backtests exclude live latency and venue costs, and past performance never guarantees future results. Size accordingly.
Background reading: why overfitting matters.