lab

thesis

The same model does not do the same work in every harness. Give a cheap open model a plan-build-check recipe inside a real coding harness and it can close much of the gap to a raw call from a pricier setup — or not. The lab runs that experiment under controlled conditions and publishes the receipts, so the claim is checkable rather than asserted.

the claim

One prompt. One model. Three ways of running it: a raw one-shot, a vanilla harness, and the same harness driving a house recipe. Everything else held fixed — same provider, same routing, same fixture. What changes is the execution recipe, and what we measure is the cost of that change against the quality of what it builds. This is the founding study; the standings it feeds decay as models and prices move, and get re-run.

how the lab is hash-stamped

Every run executes in a throwaway container that cannot reach the internet except through a single allow-listed path to the model provider — no other host, no side channels. The model's API key is passed in memory only and never touches disk, a log, or the receipt; the receipt carries a short fingerprint of the key, nothing more. Cost is measured across the whole run, including the harness's own calls — the number other comparisons quietly leave out. Nothing is published until the artifact passes a secrets-and-dependencies scan. The preregistration below was committed, with an independent timestamp, before the first paid run — so the arms, repetitions, run order, and scoring rule could not be chosen after seeing the results.

take the recipe home

The recipes compared here are not locked in a product. Each is a small bundle of skills and seat configuration you can download, drop into your own harness with your own key, and run unchanged. The artifact is the ad: what wins here, you can take with you.

method

These are small, honest studies, not a universal benchmark. A result of N=3 on one task, judged by one eye, is directional and illustrative — not claim-grade. We publish the dispersion, the failures, and the decay date alongside every verdict, and we never collapse several different dimensions into one invented score. Costs are labeled by how they were obtained: metered (billed and measured), estimated (token-derived), or unmetered (subscription or a separate quota pool — never shown as a dollar price). Receipts are hash-stamped: content-addressed manifests with digests, not signed immutability. Where a human blind rank has not yet been recorded, the quality column shows an AI screening read, labeled as exactly that, and the human verdict lands later as its own snapshot.