m/market labResearch, with a record.
Source on GitHub ↗

Experiment lifetime and shutdown

An experiment has several independent confirmations. A financial terminal state does not by itself mean that its output is archived or all children have exited.

Owner State/resource it controls Confirmation
Platform Registered controllers and shared stores Roots close before database/workspace
ResearchController Runner and borrowed market lifetime Runner closes before market is released
Method-specific DecisionRunner Agent/Rule host, Tools, Runtime and recorder closed becomes true after resource release succeeds
ExperimentControls Play/Pause/Stop coordination Runtime/execution/participant acknowledgements, then resource-close confirmation
ExperimentSession Financial event/command timeline ready → running → pausing → paused; terminal via ending
Financial ExecutionSession Writer lease, RAM account execution and financial commits Financial finish or explicit abandonment of untrusted execution
ResultDelivery Ordered JSON/results and delivery gate End Future completes after ordered output has drained, or reports output failure
Process hosts Actual child/IPC lifetime Close confirms exit/drain; a failed close may be retried

Controls

Play starts/resumes the clock and background execution, then participant capability gates. Pause freezes command admission/time and waits for execution, Runtime and delivery gates. It is not an OS suspension of arbitrary Python code. Stop closes admission, invalidates unstarted commands, finishes the financial timeline and drains output. Accepted fills are retained; unfilled quantities are cancelled by financial termination. Agent failure alone does not stop the market or cancel accepted orders.

The control barrier closes Runtime during normal Stop. Runners also check Runtime's close confirmation when releasing resources, because setup failure or a concurrent terminal event can bypass that normal path. A confirmed Runtime close is not repeated. Hosts/recorders are released in their established sequence; unsuccessful release retains runner/root ownership. Do not close the borrowed market while an executor may still use it.

Failures and deadlines

ExperimentSession keeps the initiating failure in error; later failures while stopping participants or finishing finance appear separately in cleanup_errors. This preserves the cause without pretending that cleanup succeeded. Unknown financial-storage errors do not initiate a second speculative transaction to cancel the account.

CONTROL_ACK_TIMEOUT_SECONDS and EXECUTION_DRAIN_TIMEOUT_SECONDS name internal coordination bounds in controls.py. They are not the user-configured model timeout or a promise about total shutdown time across all phases. Runtime's work/model attempt policy and provider network timeout retain separate owners.

Read ExperimentSession, Agent runner and Live Policy runner for the corresponding close/error paths. Tests cover delayed output, process failure, failed close/retry and initial failure followed by financial-cleanup failure. See verification.

Section index · All documentation

Maintained source: docs/architecture/lifecycle.md