m/market labResearch, with a record.
Source on GitHub ↗

Standalone native replay

build/itch_replay reconstructs the public market book from ITCH BinaryFILE and writes snapshots. It does not own a simulated account or place a user's trading orders. For the Python application, start with the user guide.

Build and replay

make all
build/itch_replay --input /absolute/path/feed.bin --symbol AAPL \
  --depth 5 --with-offset --output /absolute/path/aapl.csv

The input is uncompressed. Decompress gzip separately and use data you have permission to process. The executable does not implement a successful --help command; use this reference. A C++17 compiler is required. The build also produces the Python native market library.

Flag Value / effect
--input FILE Required uncompressed ITCH BinaryFILE
--symbol SYMBOL Required, 1–8 bytes; case is not normalized
--output CSV Publish a staged file on success; otherwise CSV goes directly to stdout
--depth N Display levels per side; default 5, range 1–1000; internal orders are not depth-limited
--with-offset Include offsets in the uncompressed source
--allow-implicit-eof Accept only immediate EOF after a complete locate-zero End-of-Messages event
--check-book Recompute price-level aggregation for extra book checks; potentially expensive
--verify Repeat replay from the same starting point and compare results; not an independent correctness proof
--checkpoint-out FILE Save a checkpoint at successful termination, plus requested periodic saves
--checkpoint-every-frames N Save at positive absolute frame multiples; requires a save destination
--stop-after-frame N Save and stop after absolute frame N; 0 means before processing any event
--resume FILE Load saved state and resume at the next frame; does not by itself request saving
--resume-csv Continue the verified saved CSV prefix; requires resume, output and a CSV-bound checkpoint/journal

Frame counts refer to BinaryFILE records, not network sequence numbers. Parent output directories must exist. Input, output and checkpoint paths must not alias protected files; loading and saving the same checkpoint is allowed. Resume must retain compatible symbol, depth, offset and ending options. Unsupported flags or missing/invalid values fail. See the input contract for accepted messages and truncation behavior.

Saving replay state and continuing CSV output

Three choices are separate: where computation begins, which CSV segment is published, and whether a checkpoint is written. Plain --resume emits only the new suffix. --resume-csv verifies and copies the saved journal prefix, then appends new rows into a new journal. It does not itself enable checkpoint writes.

Save choice Destination
Explicit --checkpoint-out That path
Resume plus periodic save or explicit stop, without checkpoint-out Update the loaded checkpoint
Resume alone, including resume-csv alone No checkpoint update
First run with periodic save/stop but no checkpoint-out Rejected

Examples below are alternatives; keep the original checkpoint/journal if testing multiple recovery modes.

# Replay to frame 5, retaining recoverable state and CSV prefix.
build/itch_replay --input feed.bin --symbol AAPL --output aapl.csv \
  --checkpoint-out aapl.cp --checkpoint-every-frames 2 --stop-after-frame 5

# Resume computation, publishing only the suffix. The original checkpoint is unchanged.
build/itch_replay --input feed.bin --symbol AAPL --resume aapl.cp --output suffix.csv

# Continue complete CSV and keep updating the checkpoint/journal binding.
build/itch_replay --input feed.bin --symbol AAPL --resume aapl.cp --resume-csv \
  --output aapl.csv --checkpoint-every-frames 2

A checkpoint is a saved replay state used to resume processing without starting again. It records source identity/boundary, book state, compatible options and accumulated statistics. A CSV-bound checkpoint additionally records a trusted journal prefix. Preserve the checkpoint and its referenced journal together; journal references use absolute paths. A state-only checkpoint supports computation/suffix recovery, not complete CSV continuation. Plain resume followed by another save produces state-only recovery, so keep using --resume-csv if you need a checkpoint bound to complete CSV output.

If the journal contains rows beyond the saved boundary, recovery ignores that tail and reprocesses source from the checkpoint. The copied prefix must pass size/header/row-count and integrity checks. A missing, truncated or inconsistent prefix is rejected. Extra rows are not treated as committed progress. Do not indiscriminately delete recovery .partial.* files: an existing checkpoint may reference one.

Publication, errors and statistics

With --output, replay and optional verification complete before staged CSV publication. An ordinary failure preserves an existing output and the most recent successful checkpoint. A stopped prefix has not validated the unread remainder or ending. stdout may already contain partial output on failure; always inspect exit status. Checkpoint/CSV publication is not a multi-file database transaction or a general power-loss durability guarantee.

stderr statistic Meaning
frames, selected, hash Cumulative frames, data rows and row hash, including loaded progress
emitted Newly computed rows in this invocation
scope=full/prefix/suffix Published coverage, separate from whether input ending was reached
complete=yes/no Valid ending reached, or stopped before it
verified=yes Requested repeatability pass completed
elapsed_ms, frames_per_s Replay work, including scans/checkpoints; not total process/publication time
mean_selected_event_ns, max_selected_event_ns Selected native book/snapshot/formatting work; not end-to-end network latency
peak_rss_bytes Whole-process peak memory

--verify does not run a second replay at every periodic checkpoint. A checkpoint may be recoverable before final repeatability verification has completed.

Implementation: options, checkpoint serialization, staged publication. Tests: replay regression and opt-in full-day recovery check. For loopback UDP, see network input; for component ownership, see native architecture. No real exchange connection is claimed.

Section index · All documentation

Maintained source: docs/reference/native-engine.md