sim_platform.research.sweeps¶
Experiment sweeps: run a grid of parameters or prompts and tabulate the results.
A sweep starts from one suite entry (a rule, model-file or Agent entry, see
ResearchBatch.evaluate_suite) and a grid that names entry fields by dotted path,
for example {'quantity': [1, 5], 'hold_ns': [...]} for a rule or
{'model.temperature': [0, 0.7], 'settings.budget.max_tokens': [...]} for an Agent.
Every combination becomes its own entry; all of them run on every recording with the
same seeds, so the table compares like with like.
Choosing the best cell of a grid is itself a search, so every entry is evaluated with
trials equal to the number of combinations: its deflated Sharpe ratio accounts for
the whole sweep, and the probability of backtest overfitting is reported across cells.
- sim_platform.research.sweeps.expand_grid(base, grid)[source]¶
Every combination of
gridapplied tobase: (parameters, suite entry) pairs.Entry names list the parameters (
quantity=5, hold_ns=60); a grid field namednameorrepetitionsis refused, since those describe the sweep itself.- Parameters:
base (Mapping[str, Any])
grid (Mapping[str, Sequence[Any]])
- Return type:
List[Tuple[Dict[str, Any], Dict[str, Any]]]
- sim_platform.research.sweeps.sweep_table(combinations, suites, metric)[source]¶
One row per combination: its parameters and, per recording, the metric’s figures.
- Parameters:
combinations (Sequence[Tuple[Dict[str, Any], Dict[str, Any]]])
suites (Sequence[Mapping[str, Any]])
metric (str)
- Return type:
List[Dict[str, Any]]
- sim_platform.research.sweeps.heatmap(rows, x, y, *, value='mean')[source]¶
A matrix of
valueover two swept fields; other fields are averaged over.Returns
{'x', 'y', 'x_values', 'y_values', 'matrix'}withmatrix[i][j]fory_values[i]andx_values[j](None where no combination has a value).- Parameters:
rows (Sequence[Mapping[str, Any]])
x (str)
y (str)
value (str)
- Return type:
Dict[str, Any]
- sim_platform.research.sweeps.render_heatmap_svg(data, *, title=None)[source]¶
An SVG heat map of
heatmap()output, symmetric around zero, values in cells.- Parameters:
data (Mapping[str, Any])
title (str | None)
- Return type:
str
- sim_platform.research.sweeps.run_sweep(batch, paths, base, grid, *, metric='total_return', **arguments)[source]¶
Run every combination on every recording through
batch.evaluate_suite.Returns
{'format', 'metric', 'grid', 'combinations', 'rows', 'best', 'suites'}:rowsfromsweep_table;bestthe combination with the best mean (by the metric’s direction; None for a metric without one), whose deflated Sharpe ratio counts every combination as a trial.argumentsgo toevaluate_suite.- Parameters:
batch (Any)
paths (List[Any])
base (Mapping[str, Any])
grid (Mapping[str, Sequence[Any]])
metric (str)
arguments (Any)
- Return type:
Dict[str, Any]