See the result.
Keep its limits.

Start with an answer you can inspect

These small, deliberately chosen fixtures demonstrate command behavior. They are not studies of detector accuracy or agent productivity.

What else uses this function?

roam --json impact calculate_total returned three direct callers in a four-function Python checkout fixture. The captured response reports partial_success: false, truncated: false and cap_applied: false. That establishes the returned traversal's reported completion, not complete runtime coverage.

• Recorded: 13 September 2026, Roam 14.1.0; one synthetic fixture, no model involved. This is not a timing comparison.
• Inspect: full captured JSON and the explained result.
• Reproduce: the public homepage contract test creates the fixture, runs the real CLI and compares selected fields. Timestamps and index metadata can differ in a new run.

Does a different lookup preserve the behavior?

The algorithm example includes JavaScript source, captured detector output and behavior tests. One control deliberately shows why replacing every indexOf check with a Set is unsafe: handling NaN can change the answer.

Recorded: 13 September 2026, Roam 14.1.0; a selected synthetic example, not a speed study. Its public regression tests candidate behavior. No workload speedup is established.

A recorded miss: pre-executed answer coverage

Historical result, not rerun for this page. On 27 July 2026, a fixed set of 60 repository-agnostic prompts was replayed through Roam's task compiler on FastAPI, Gin and Svelte, plus Roam as the home-repository control. The question was how often an envelope carried a pre-executed answer, called an l1_probe. This measured routing, not answer accuracy. The historical 45% target came from a different corpus; even the home-repository control fell below it on this one.

The control changes the interpretation. The target was based on an earlier, unavailable prompt set. Roam itself did not reach that target on this corpus. The miss is retained; it does not by itself establish a cross-repository transfer failure. Nor does an answer-carrying envelope establish that its answer is correct.

Engine and environment

Roam 13.10.0, commit b6a8e87f, fresh environment from an archived checkout. Hardware and full dependency lock are not given in the public record. The record notes a missing NumPy dependency during indexing.

Population and model

60 authored prompts × four repositories, one recorded sweep per repository. Mechanical compiler replay, not a comparison of coding-agent models or accepted code changes.

Metric and uncertainty

L1 envelope count over reported successful compiles. The record reports no compile errors or empty-probe findings. No repeated-run distribution or independently graded answer accuracy is available. A routing fraction is not a token- or cost-saving estimate.

Public artifacts

Protocol and target · 60-prompt corpus · Target revisions · Full reported result and caveats.

Reproduction status

Protocol, corpus and target revisions are public. Raw per-run envelopes are not in that published directory; this page does not claim an independently reproduced result. Use the recorded engine and target revisions for a historical rerun; a run on today's source is a new observation.

Older agent comparisons: useful history, not a current promise

The May–July 2026 measurement archive preserves compiler/no-compiler comparisons, turns, input tokens, dollar cost and wall time, including losses and later remeasurements. It names the original model and engine where known. Raw cells for those agent comparisons are retained privately, so the public tables alone cannot reproduce the aggregates.

Some treatments cost more. A larger early run reversed the apparent winner of a smaller one after an instruction confounder was removed. Small repeated cells and changed task sets limit what can be generalized. Passing the same ten planted bugs in both arms did not establish quality parity; lower dollars could coincide with more input tokens.

Repair-sibling retrieval has public saved cases

The archive's 576 multi-site fixes from 12 Python repositories are available in the frozen corpus, with per-case results and arm definitions. This is a narrower question than general code search: given one repair, rank other places that need the same repair.

The saved results report average ranking quality at ten results (nDCG@10) of about 0.6045 for repair-intent reranking versus 0.5406 for the lexical baseline. The graph candidate pool without that reranking scores about 0.2579—worse than lexical search. The archive's recall interval includes no improvement. A ranking gain is not a measured reduction in bugs, agent cost or elapsed time.

Reproduction boundary: the public regression pins the frozen corpus, executes the current scorer on its saved repair sites, and checks the saved treatment-versus-lexical metric. It does not rerun candidate collection and all four rankings from the original repositories. The frozen protocol describes reranking the same lexical pool; the results describe reranking a graph pool. Keep that protocol discrepancy visible rather than treating a passing regression as full reproduction or confirmed preregistration adherence.

The saved repair artifacts do not identify the full original engine and environment or a clear measurement timestamp. A present-day regression run does not fill in that missing provenance.

These measurements do not imply general accuracy, token, cost, or latency improvements across arbitrary repositories and agents.

Measure the question that matters to your workflow

Roam makes repository relationships and repeatable checks available without asking a model to derive each result anew. That mechanism is useful to test—not a universal savings claim. Count index creation, refreshes, tool calls and the agent's work consuming the results.

1. Choose a recurring task and a capable baseline using your current tools. Fix the source revision, inputs, model and acceptance tests.
2. Compare with Roam on the same task. Keep failed attempts, missing answers, false findings and cases where Roam adds no useful result.
3. Verify outcomes independently. Report time, tokens and cost separately, with sample counts and missing measurements—not just the best run.

See the comparison method and benchmark accounting rules. The available saved-cell analyzer accounts for observed artifacts; it does not establish assignments or task success.

Run the public example checks

In a trusted Roam source checkout with its locked development environment installed, run these regression tests. The algorithm behavior control also requires Node.js. These are example checks, not an agent benchmark:

uv run --no-sync pytest tests/test_homepage_contract.py::test_homepage_walkthrough_and_connection_example_execute tests/test_site_algorithm_example.py -n 0

Try one result on your own code · Compare the kinds of tools · Read the trust boundaries