sim_platform.research.overfitting¶
Guards against backtest overfitting: deflated Sharpe, PBO and leak-free splits.
A researcher who tries many Agent prompts, rule parameters or model files and keeps the best one will see a high Sharpe ratio by chance alone. These pure functions measure how much of a result survives that selection:
probabilistic_sharpe_ratio(PSR): the probability that the true Sharpe ratio exceeds a benchmark, allowing for sample length, skewness and fat tails.deflated_sharpe_ratio(DSR): PSR against the Sharpe ratio the best ofnworthless trials would be expected to reach.probability_of_backtest_overfitting(PBO): combinatorially symmetric cross-validation over a matrix of candidate returns.walk_forward_splitsandpurged_k_fold_splits: train/test index ranges that never let a test period inform its own training data.
References: Bailey and López de Prado, “The Sharpe Ratio Efficient Frontier” (2012) and “The Deflated Sharpe Ratio” (2014); Bailey, Borwein, López de Prado and Zhu, “The Probability of Backtest Overfitting” (2017); López de Prado, “Advances in Financial Machine Learning” (2018), chapter 7. Sharpe ratios here are per period (not annualised), because the formulas are stated per observation.
- sim_platform.research.overfitting.per_period_sharpe(returns)[source]¶
Mean over standard deviation of one return series; None when undefined.
- Parameters:
returns (Sequence[float])
- Return type:
float | None
- sim_platform.research.overfitting.probabilistic_sharpe_ratio(returns, benchmark_sharpe=0.0)[source]¶
Probability that the true per-period Sharpe ratio exceeds
benchmark_sharpe.- Parameters:
returns (Sequence[float]) – Per-period returns (for example daily), at least three values.
benchmark_sharpe (float) – The per-period Sharpe ratio to beat; 0 tests for any skill.
- Returns:
A probability in [0, 1], or None for fewer than three returns or zero variance.
- Return type:
float | None
- sim_platform.research.overfitting.expected_maximum_sharpe(trials, sharpe_variance)[source]¶
Sharpe ratio the best of
trialsindependent zero-skill strategies would reach.- Parameters:
trials (int) – Number of configurations tried (prompts, parameters, model files).
sharpe_variance (float) – Variance of the per-period Sharpe ratios across those trials.
- Returns:
The expected maximum per-period Sharpe ratio; 0 for a single trial.
- Return type:
float
- sim_platform.research.overfitting.deflated_sharpe_ratio(returns, *, trial_sharpes=None, trials=None, sharpe_variance=None)[source]¶
Probability that the selected strategy’s Sharpe ratio is not a selection artefact.
Give either the per-period Sharpe ratios of every trial (
trial_sharpes), or the number of trials with the variance of their Sharpe ratios. A value above 0.95 is the usual threshold for a result that survives the search that produced it.- Returns:
A probability in [0, 1], or None when
returnscannot define a Sharpe ratio.- Raises:
ValueError – If neither trial description is given.
- Parameters:
returns (Sequence[float])
trial_sharpes (Sequence[float] | None)
trials (int | None)
sharpe_variance (float | None)
- Return type:
float | None
- class sim_platform.research.overfitting.OverfittingEstimate(probability, splits, logits)[source]¶
Bases:
objectResult of combinatorially symmetric cross-validation.
probabilityis the share of splits where the in-sample winner ranked in the bottom half out of sample.logitsholds each split’s relative out-of-sample rank as log(w / (1 - w)); negative values are the overfit splits.- Parameters:
probability (float)
splits (int)
logits (Tuple[float, ...])
- probability: float¶
- splits: int¶
- logits: Tuple[float, ...]¶
- sim_platform.research.overfitting.probability_of_backtest_overfitting(returns_by_candidate, blocks=8)[source]¶
Estimate how often choosing the best backtest picks a below-median strategy.
- Parameters:
returns_by_candidate (Sequence[Sequence[float]]) – One equally long per-period return series for each candidate configuration (at least two candidates).
blocks (int) – Even number of contiguous blocks the time axis is cut into; every choice of half the blocks is one in-sample/out-of-sample split.
- Returns:
The estimate over all C(blocks, blocks / 2) splits.
- Raises:
ValueError – For fewer than two candidates, unequal lengths, an odd or too large
blocks.- Return type:
- sim_platform.research.overfitting.walk_forward_splits(length, train, test, *, step=None, anchored=False, gap=0)[source]¶
Successive train/test windows that always test after they train.
- Parameters:
length (int) – Number of observations.
train (int) – Training window length (the first window’s length when
anchored).test (int) – Test window length.
step (int | None) – How far each split moves forward; defaults to
test.anchored (bool) – Keep the training start at 0 so the window grows.
gap (int) – Observations skipped between training and test, for labels that look ahead.
- Returns:
((train_start, train_stop), (test_start, test_stop)) half-open ranges in order.
- Return type:
List[Tuple[Tuple[int, int], Tuple[int, int]]]
- sim_platform.research.overfitting.purged_k_fold_splits(length, folds, *, label_horizon=0, embargo=0)[source]¶
K contiguous test folds with leaking training observations removed.
A label computed from the next
label_horizonobservations overlaps the test fold when it starts within that distance before the fold, so those observations are purged.embargoobservations after the fold are dropped as well, because serial correlation lets them carry test information backwards.- Returns:
For each fold, the sorted training indices and the (start, stop) test range.
- Parameters:
length (int)
folds (int)
label_horizon (int)
embargo (int)
- Return type:
List[Tuple[List[int], Tuple[int, int]]]