sim_platform.research.sweeps

Experiment sweeps: run a grid of parameters or prompts and tabulate the results.

A sweep starts from one suite entry (a rule, model-file or Agent entry, see ResearchBatch.evaluate_suite) and a grid that names entry fields by dotted path, for example {'quantity': [1, 5], 'hold_ns': [...]} for a rule or {'model.temperature': [0, 0.7], 'settings.budget.max_tokens': [...]} for an Agent. Every combination becomes its own entry; all of them run on every recording with the same seeds, so the table compares like with like.

Choosing the best cell of a grid is itself a search, so every entry is evaluated with trials equal to the number of combinations: its deflated Sharpe ratio accounts for the whole sweep, and the probability of backtest overfitting is reported across cells.

sim_platform.research.sweeps.expand_grid(base, grid)[source]

Every combination of grid applied to base: (parameters, suite entry) pairs.

Entry names list the parameters (quantity=5, hold_ns=60); a grid field named name or repetitions is refused, since those describe the sweep itself.

Parameters:
  • base (Mapping[str, Any])

  • grid (Mapping[str, Sequence[Any]])

Return type:

List[Tuple[Dict[str, Any], Dict[str, Any]]]

sim_platform.research.sweeps.sweep_table(combinations, suites, metric)[source]

One row per combination: its parameters and, per recording, the metric’s figures.

Parameters:
  • combinations (Sequence[Tuple[Dict[str, Any], Dict[str, Any]]])

  • suites (Sequence[Mapping[str, Any]])

  • metric (str)

Return type:

List[Dict[str, Any]]

sim_platform.research.sweeps.heatmap(rows, x, y, *, value='mean')[source]

A matrix of value over two swept fields; other fields are averaged over.

Returns {'x', 'y', 'x_values', 'y_values', 'matrix'} with matrix[i][j] for y_values[i] and x_values[j] (None where no combination has a value).

Parameters:
  • rows (Sequence[Mapping[str, Any]])

  • x (str)

  • y (str)

  • value (str)

Return type:

Dict[str, Any]

sim_platform.research.sweeps.render_heatmap_svg(data, *, title=None)[source]

An SVG heat map of heatmap() output, symmetric around zero, values in cells.

Parameters:
  • data (Mapping[str, Any])

  • title (str | None)

Return type:

str

sim_platform.research.sweeps.run_sweep(batch, paths, base, grid, *, metric='total_return', **arguments)[source]

Run every combination on every recording through batch.evaluate_suite.

Returns {'format', 'metric', 'grid', 'combinations', 'rows', 'best', 'suites'}: rows from sweep_table; best the combination with the best mean (by the metric’s direction; None for a metric without one), whose deflated Sharpe ratio counts every combination as a trial. arguments go to evaluate_suite.

Parameters:
  • batch (Any)

  • paths (List[Any])

  • base (Mapping[str, Any])

  • grid (Mapping[str, Sequence[Any]])

  • metric (str)

  • arguments (Any)

Return type:

Dict[str, Any]