The model held fixed, the agent left to differ
Agent benchmarks usually compare one number by changing three things at once: the model, the prompt scaffolding, and the tool's own overhead. tomo-labs holds the model fixed. A trace proxy forwards every request from every agent to the same free model with the same deterministic decoding, whatever wire dialect the agent's SDK speaks, and every result is graded from the files it left on disk, not from what it claims to have done.
Seven agents run through the same harness today: tomo, codex, opencode, claude-code, openclaw, hermes, and gemini-cli.
Every one runs in its own throwaway container, every request and response it sends is captured verbatim, and adding one more agent is a Dockerfile and a small adapter script, not a fork of the harness.
go run ./cmd/lab build # base, proxy, and every wired tool image
go run ./cmd/lab run tomo # run tomo through every scenario
go run ./cmd/lab report # summarize every captured run as a table
What it measures
- Correctness, from a checker that grades the work left on disk, never the model's own account of what it did.
- Tokens, memory, wall time, and disk, the same measurement for every agent because the proxy and the harness take it, not the tool.
- Install footprint, the tool's own bytes on top of the shared base image, a real cost most benchmarks never show.
- Time to first byte and latency, timed by the proxy on the same upstream call for every tool.
Where to go next
- New here? Start with the installation guide, then the quick start.
- Want the numbers?
See results for the full comparison table and the
00-hellobaseline. - Adding an agent to the comparison? Adding a tool covers the two files a new agent needs.
- Need the exact command surface?
The CLI reference lists every
labcommand and flag.