# Robustness

> One "is this overfit?" verdict, backed by several independent checks.

Source: https://docs.texttoquant.com/reference/robustness

---

A single good backtest proves very little. The robustness panel gives one honest verdict, *is this
edge likely to survive?*, backed by several independent checks that each attack the result from a
different angle.

## The checks

| Check | What it asks |
| --- | --- |
| OOS split | Does it hold on data it was never fit on? (in sample vs out of sample) |
| Walk forward | Is performance consistent across rolling windows? |
| Regime | Does it work in bull, bear and chop, or only one? |
| Cross market | Does the edge transfer to other assets? |
| Sample size | Are there enough trades to trust the numbers at all? |

<Callout variant="info" title="Instant + on demand">
  Instant legs read the current backtest and appear immediately; heavier legs (cross market,
  walk forward optimization) run on demand.
</Callout>

<Callout variant="info" title="Choosing the cross-market peers">
  Cross market tests four peer markets by default, picked from your strategy's own asset class. In the
  terminal's Robustness section you can name them instead: switch **Cross market** to *Choose markets*
  and add up to four symbols. The run is billed the same flat 5 backtests, the timeframe leg is
  unchanged, and your own market is refused as a peer (it would only agree with itself). Agents can do
  the same over MCP with `start_analysis` kind `multi_asset` and `params.assets`.

  Read a chosen run more carefully than an automatic one: a market you picked that *breaks* the
  strategy is still hard evidence of fragility, but markets that hold up are a set you selected, so
  they cannot show that the edge generalizes.
</Callout>

## Parameter-search checks

These run real backtests across parameter variations to test whether the edge is a stable plateau or
a lone spike. They're billed (see [plans & credits](/docs/concepts/plans-and-credits)) and run on
demand.

| Check | What it does |
| --- | --- |
| Parameter sweep | Re-runs across values of one knob, is the result robust across the range, or a single peak? |
| Parameter grid | A 5×5 sweep of two knobs (25 runs); the CSCV split of a grid is what produces the PBO score |
| Joint sweep | Varies up to three parameters together to test region stability, not just one axis at a time |
| Genetic search | Evolves parameter sets against real backtests to find, and stress, the best region |
| Strategy variants | Auto-runs long-only / short-only / risk-scaled versions and ranks them |
| Walk-forward efficiency | Out-of-sample return ÷ in-sample optimum; a low WFE flags a fragile, over-tuned fit |

## Overfitting aware statistics

A raw Sharpe is inflated by how many configurations you tried. Once a parameter search records the
trials, these statistics appear on the metric cards, an honest read no other natural language tool
ships. PSR, DSR, Haircut and Min. length come from any parameter sweep; **PBO needs the CSCV split,
so it appears only after a grid or joint sweep**.

| Metric | Corrects for | How to read it |
| --- | --- | --- |
| Probabilistic Sharpe (PSR) | Short samples & fat tails | Higher = more confident the Sharpe beats 0 |
| Deflated Sharpe (DSR) | How many configs you tried | ≥95% = survives the search; a big drop vs PSR = search luck |
| Overfit Probability (PBO) | In sample best failing out of sample (grid / joint sweep only) | Low is good; ≥50% = likely overfit |
| Haircut Sharpe | Bonferroni correction for T trials | The Sharpe you can still claim after the search |
| Min. backtest length | Sample too short for the search | Warns when history can't support that many trials |

<Callout variant="info" title="Get the verdict from an agent">
  Over the [MCP server](/docs/api/mcp), `get_overfit_verdict` (or `GET /v1/backtests/:id/overfit`) returns this
  same read: Deflated Sharpe, PBO and the `holds_up` / `likely_overfit` / `insufficient_evidence` call. A single
  backtest with no parameter search honestly returns `insufficient_evidence`: run a sweep or grid first.
</Callout>

<Callout variant="success" title="Reproducible by construction">
  Your query compiles deterministically to a fixed strategy spec, then a fixed engine scores it. The
  AI never sees, ranks, or tunes the numbers. Same query ⇒ same spec fingerprint ⇒ same test. Signals
  use only closed bar data (no look ahead). The results header shows a Reproducible chip and, when a
  hold out split exists, an always on out of sample verdict.
</Callout>

See how these feed the letter grade in the [Metrics reference](/docs/reference/metrics).