m/market labResearch, with a record.
Source on GitHub ↗

Design proposal: evaluating Agents against baselines

Status: accepted on 2026-11-15. The project owner delegated the decisions below; each one follows common quant-research practice and can be revisited. All phases are implemented (user guide), including turn-based Agents (decision 7).

Goal

A quantitative researcher asks one question: does this Agent trade better than a person, a rule or a model on the same market, and is the difference real? Market Lab already runs all four on the same execution path. What it lacks is a way to put their results side by side, to repeat a stochastic Agent enough times to measure it, and to rule out the most common objection to LLM trading results: the model may remember the prices.

Where the code stands

  • One run produces a saved report: run configuration, fills, archived bars and daily closes. The Desktop result page computes Sharpe ratio, maximum drawdown, win rate, turnover and holding time from it, in desktop/presentation.py. Python and batch callers cannot reach those numbers.
  • ResearchBatch.compare() runs up to 20 Rule or model-file configurations once each on independent accounts and lists their summaries. It has no repetitions, statistics or Agent entries.
  • An Agent run calls a model whose replies differ between runs. One run of an Agent is one sample, not its performance.
  • A price-bar file keeps its real ticker, dates and prices. A model trained after those dates may have seen them.

1. Metrics move out of the Desktop

research/metrics.py holds the per-run metrics as pure functions of a saved run's view. The Desktop result page, the batch API and the evaluation report all call it, so each figure has one definition. New figures:

Metric Definition
Total return final equity / starting cash − 1, equity marked at the last archived close
Maximum drawdown already computed; now also returned to Python
Sortino ratio mean daily return / downside deviation (returns below 0), × √252
Exposure share of archived bars that end with a position open

Each stays None when it cannot be computed, as today. Risk-free rate is zero.

2. An evaluation is built from saved runs

Decision 1: an evaluation report reads saved runs; it does not own how they were run. Platform.evaluation(profile, entries, baseline=...) takes named groups of run ids, for example {"Agent v3": [...], "Buy and hold": [...], "LSTM": [...]}. Runs made in the Desktop, from the batch API or by a script all count the same. The batch runner in section 3 is one way to produce those runs, not the only one.

Decision 2: runs must be comparable or the report refuses them. All runs in one evaluation must share the market identity (recording file size and modification time, news, synthetic-book options, symbols and prepared start), starting cash, the execution settings and the short-selling rule. A mismatch raises an error naming the differing field. Equal footing is the point of the comparison. Live or Controlled pacing may differ. An Agent stops at its chosen end while other methods run to the end of the recording; the report warns when the windows differ rather than refusing.

3. Repetitions and statistics

Decision 3: repeat stochastic entries; run deterministic ones once. ResearchBatch.evaluate_suite(paths, entries) runs each entry repetitions times on each recording (default 10 for an Agent, once for a rule or model file), each run on its own new account, optionally in the researcher's profile so the runs can be evaluated with saved Agent runs. A baseline whose values are all identical is treated as a constant.

Decision 3a: Agents keep real time; concurrency only shortens the wait. An Agent's reaction time is part of what Live measures, so a suite never accelerates the market for it. Rule and model-file runs in the same suite start at the Agents' start_ns and stop at the first event at or after their end_ns, so every run covers the same market time. concurrency (1–16, default 1) plays several Agent runs at once, each on its own account and clock; markets are prepared one at a time first, since a workspace prepares one market at once. The researcher chooses the level, because a model provider that slows down under concurrent requests would change every run's results. Turn-based Agents (decision 7) answer the other question, decision quality without speed.

Repetitions on one recording measure the Agent's own randomness on that market, not market uncertainty. A suite therefore accepts several recordings and reports each one separately; robustness across periods is read across recordings, not inferred from repetitions.

Decision 4: statistics without assumptions about the distribution. For each entry and metric the report gives n, mean, sample standard deviation, min, max and a 95% percentile bootstrap confidence interval of the mean (10,000 resamples). Against the baseline entry it gives the difference in means, its bootstrap interval and a two-sided permutation p-value (exact when there are few enough arrangements, else 10,000 permutations). When the baseline is deterministic, the test is a sign-flip test of the entry's values minus the baseline's single value.

  • p-values across entries are adjusted with Holm's method, because comparing several Agents with one baseline is several tests.
  • The random generator is seeded (default 2026) and the seed is written into the report, so the same runs give the same report.
  • With fewer than 5 values in an entry, its interval and p-value are reported but marked insufficient; the Desktop shows them greyed.
  • No overall score is invented. The researcher chooses the metric; total return is the default.

4. Disguised market data

Decision 5: disguise the data, not the Agent's view of it. Platform.disguise_recording(path) writes a disguised copy of a price-bar CSV under a neutral file name, and every method then uses that copy as an ordinary recording:

  • Symbol: a CSV carries no ticker; the researcher gives the experiment a neutral symbol such as ASSET1.
  • Dates: shifted by a whole number of weeks so weekdays are kept. Default first date 2100-01-04, a Monday, which no model has seen.
  • Prices: multiplied by one factor and rounded to the cent; volumes unchanged. Default factor drawn from [0.5, 2] with a seed. Returns are kept up to rounding.
  • Other columns such as Adj Close are dropped.

Because the copy is the recording, every participant sees the same disguised market, Replay needs nothing new, and the comparison stays fair. The mapping is returned to the researcher only. News and ITCH recordings cannot be disguised in this phase.

Decision 6: record what the model might know, and warn; never block. The evaluation takes a training-data cutoff date per entry, since the researcher knows which model each entry used. A run whose data dates lie in the future is recognised as disguised without any extra record. The evaluation report flags each Agent run:

Flag Meaning
disguised data was disguised
after-cutoff every data date is after the declared cutoff
possible-memorisation real dates on or before the cutoff
cutoff-unknown real dates and no declared cutoff
dates-unknown an ITCH recording, which carries no date

5. Model cost and latency

The Agent keeps only the token counts of each provider reply (input_tokens, output_tokens, total_tokens, cached input tokens) and sends them with the model receipt; the Runtime checks they are non-negative integers and writes them into the attempt_completed record of the operation log. Each attempt_dispatched record now carries its dispatch time. The evaluation reads the sealed log of each Agent run and adds model calls, failed calls, input and output tokens and mean reply latency. Money cost is tokens × a price the researcher enters; Market Lab ships no price list, because provider prices change.

6. Desktop Compare runs page

Compare runs lists finished runs grouped by what made their decisions: an Agent revision, a rule with its settings, a model file or manual trading. Once one run is chosen, only runs with the same market and settings stay selectable, so the comparability check of section 2 rarely has to refuse. The researcher picks the metric, the baseline and, for Agent groups, the model's training cutoff; the page shows the table from section 3 with the flags from section 4 and exports the report as JSON. When neither the baseline nor an entry varies between runs (rules repeat exactly), the difference is exact and the page says so instead of showing a test that could never reach significance.

7. Turn-based Agents

Decision 7: a turn-based Agent is the screening mode; Live is the validation mode. Agent settings take speed='turn' beside '1x'. The experiment clock is then turn-based: it never moves with real time, and a TurnPacer thread in the owning process advances it. The pacer moves the clock only when the experiment is settled at the current time: every market event up to it processed, every market notification received by the Runtime, the Runtime fully stepped at that time (it reports stepped_ns and next_ns only in this mode), no Agent work running and no model reply outstanding. It then jumps to the earliest of the Runtime's next own time, the next market event and the end. The owner executes events and commands stamped before its clock, so an event or a command at the current instant gets one nanosecond. A model reply outstanding longer than the attempt timeout in real time is given its deadline, so the attempt times out as it would in Live.

Evaluations refuse to mix Live and turn-based Agent runs, report no reply time for turn-based runs, and latency_cost() reports the Live mean minus the turn-based mean of the same Agent on the same market with a bootstrap interval.

Phases

Phase Scope Status
1 Metrics module; Desktop uses it Implemented
2 Evaluation report from saved runs, statistics, comparability check Implemented
3 evaluate_suite for Rule and model-file entries, several recordings Implemented
4 Disguised price-bar recordings; memorisation flags Implemented
5a Agent entries in evaluate_suite, sharing the Agents' window; concurrency Implemented
5b Model token usage, reply latency and cost Implemented
6 Desktop Compare runs page Implemented
7 Turn-based Agents, latency cost Implemented

Section index · All documentation

Maintained source: docs/architecture/agent-evaluation-design.md