Robustness
The overfitting verdict and the checks behind it.
A single good backtest proves very little. The robustness panel gives one honest verdict, is this edge likely to survive?, backed by several independent checks that each attack the result from a different angle.
The checks#
| Check | What it asks |
|---|---|
| OOS split | Does it hold on data it was never fit on? (in sample vs out of sample) |
| Walk forward | Is performance consistent across rolling windows? |
| Regime | Does it work in bull, bear and chop, or only one? |
| Cross market | Does the edge transfer to other assets? |
| Sample size | Are there enough trades to trust the numbers at all? |
Instant legs read the current backtest and appear immediately; heavier legs (cross market, walk forward optimization) run on demand.
Cross market tests four peer markets by default, picked from your strategy's own asset class. In the
terminal's Robustness section you can name them instead: switch Cross market to Choose markets
and add up to four symbols. The run is billed the same flat 5 backtests, the timeframe leg is
unchanged, and your own market is refused as a peer (it would only agree with itself). Agents can do
the same over MCP with start_analysis kind multi_asset and params.assets.
Read a chosen run more carefully than an automatic one: a market you picked that breaks the strategy is still hard evidence of fragility, but markets that hold up are a set you selected, so they cannot show that the edge generalizes.
Parameter-search checks#
These run real backtests across parameter variations to test whether the edge is a stable plateau or a lone spike. They're billed (see plans & credits) and run on demand.
| Check | What it does |
|---|---|
| Parameter sweep | Re-runs across values of one knob, is the result robust across the range, or a single peak? |
| Parameter grid | A 5×5 sweep of two knobs (25 runs); the CSCV split of a grid is what produces the PBO score |
| Joint sweep | Varies up to three parameters together to test region stability, not just one axis at a time |
| Genetic search | Evolves parameter sets against real backtests to find, and stress, the best region |
| Strategy variants | Auto-runs long-only / short-only / risk-scaled versions and ranks them |
| Walk-forward efficiency | Out-of-sample return ÷ in-sample optimum; a low WFE flags a fragile, over-tuned fit |
Overfitting aware statistics#
A raw Sharpe is inflated by how many configurations you tried. Once a parameter search records the trials, these statistics appear on the metric cards, an honest read no other natural language tool ships. PSR, DSR, Haircut and Min. length come from any parameter sweep; PBO needs the CSCV split, so it appears only after a grid or joint sweep.
| Metric | Corrects for | How to read it |
|---|---|---|
| Probabilistic Sharpe (PSR) | Short samples & fat tails | Higher = more confident the Sharpe beats 0 |
| Deflated Sharpe (DSR) | How many configs you tried | ≥95% = survives the search; a big drop vs PSR = search luck |
| Overfit Probability (PBO) | In sample best failing out of sample (grid / joint sweep only) | Low is good; ≥50% = likely overfit |
| Haircut Sharpe | Bonferroni correction for T trials | The Sharpe you can still claim after the search |
| Min. backtest length | Sample too short for the search | Warns when history can't support that many trials |
Over the MCP server, get_overfit_verdict (or GET /v1/backtests/:id/overfit) returns this
same read: Deflated Sharpe, PBO and the holds_up / likely_overfit / insufficient_evidence call. A single
backtest with no parameter search honestly returns insufficient_evidence: run a sweep or grid first.
Your query compiles deterministically to a fixed strategy spec, then a fixed engine scores it. The AI never sees, ranks, or tunes the numbers. Same query ⇒ same spec fingerprint ⇒ same test. Signals use only closed bar data (no look ahead). The results header shows a Reproducible chip and, when a hold out split exists, an always on out of sample verdict.
See how these feed the letter grade in the Metrics reference.