The model held fixed, the agent left to differ

Agent benchmarks usually compare one number by changing three things at once: the model, the prompt scaffolding, and the tool's own overhead. tomo-labs holds the model fixed. A trace proxy forwards every request from every agent to the same free model with the same deterministic decoding, whatever wire dialect the agent's SDK speaks, and every result is graded from the files it left on disk, not from what it claims to have done.

Get started View on GitHub

Eight agents run through the same harness today: tomo, codex, opencode, claude-code, openclaw, hermes, gemini-cli, and pi. Every one runs in its own throwaway container, every request and response it sends is captured verbatim, and adding one more agent is a Dockerfile and a small adapter script, not a fork of the harness.

go run ./cmd/lab build   # base, proxy, and every wired tool image
go run ./cmd/lab run tomo   # run tomo through every scenario
go run ./cmd/lab report   # summarize every captured run as a table

What it measures

Correctness, from a checker that grades the work left on disk, never the model's own account of what it did.
Tokens, memory, wall time, and disk, the same measurement for every agent because the proxy and the harness take it, not the tool.
Install footprint, the tool's own bytes on top of the shared base image, a real cost most benchmarks never show.
Time to first byte and latency, timed by the proxy on the same upstream call for every tool.

Where to go next

New here? Start with the overview for the whole feature set in one place, then the installation guide and the quick start.
Studying an agent? The tools pages are a deep dive on each one: command surface, architecture, captured system prompt, and a Hi! run traced end to end.
Want the numbers? See results for the full comparison table and the 00-hello baseline.
Curious what each agent actually sends? The prompts pages carry every wired agent's real system prompt, recovered from the trace and versioned so drift shows in the diff.
Adding an agent to the comparison? Adding a tool covers the two files a new agent needs.
Need the exact command surface? The CLI reference lists every lab command and flag.

Getting started Install tomo-labs and run your first agent sweep: build the images, run a tool through every scenario, and read the report. Guides Task-oriented walkthroughs for tomo-labs: how a run works end to end, reading the results table, the scenario suite, adding a new tool, and keeping the wired agents current. Evals The eval tiers beside the core scenarios: whole public benchmarks rendered into the same task shape, selected with --suite, materialized and validated by lab gen, and graded on the host. Experiments Write-ups of single runs worth reading in full: one tool on one task, what it did turn by turn, why it passed or failed, and what the run says about the tool or the task. Each report pins the exact tool version, model, and commit so you can reproduce it. Tools One research page per wired agent: what it is, its command surface, how the lab drives it, how it works inside, the system prompt it actually sent, and a Hi! run traced end to end. Predictions The prediction ledger: every measurable claim about a future run, filed before the run, with a floor, a ceiling, and the mechanism that would explain the number. Verdicts are written against the filed band, hit or miss, and an entry is never edited after its run starts. Prompts The system prompt every wired agent actually sent, recovered verbatim from the trace proxy with lab prompts, one page per tool and versioned so drift shows up in the diff. Reference The exhaustive lab CLI surface: every command and flag.