An agent only runs unattended on the parts it has proven it gets right. Evals are the proof — a set of known cases the agent must pass, re-run every time anything changes. Passing them is what lets the human gate move up; failing one blocks the change. This is the foundation under everything on See it run.
Cases defined from your engagement · automated gate being stood up
What an eval is
Not a vibe check. Each case is a concrete situation with an expected behavior and a hard pass/fail line — the same shape every time.
1Situation→2Expected behavior→3Pass / fail line→4Re-run on every change
A change to the recipe, a tool, or a connector triggers the full set. Green means the agent still behaves correctly and the change can ship; red blocks it. That is the mechanism that makes improvement safe — and the mechanism that earns each step of autonomy below.
The golden cases
The cases drawn from your engagement — the judgment calls an expert must get right on an offline-conversion build. These are the hardest, most consequential ones; the set grows as the work does.
GC-1Reconcile to your definition — never our own★ Keystonecase · live
SituationYour production job defines "converted" with filters a naive extraction won't replicate — window, status/stage, internal & test exclusions, source scope, identifier set, consent. A self-derived pull returns a materially larger count.The agent mustFind and read your authoritative definition, reconcile its extraction to it, and on a material gap stop and align the definitions — never present a self-derived number as fact.
Pass
Final numbers reconcile to your query within drift; the gap is explained by named definitional axes.
Fail — the trap it prevents
Presenting a self-derived count (e.g. ~11,034) as "validated" instead of your ~3,630 — a ~3× overstatement.
GC-2Decompose the send by quality & compliance tiercase · live
SituationThe current send applies no consent filter, so it mixes non-consented hashed PII (a customer-data-terms breach) in with clean rows.The agent mustSplit the send into tiers — deterministic-id / consented-PII / non-consented / non-platform-id / unsendable — flag the non-consented tier as a compliance risk, and recommend the consent-filtered clean subset.
Pass
Non-consented PII is surfaced as a finding with a count; the recommended send is consent-filtered.
Fail
Reports a single "uploadable" figure, or silently adopts the client's no-consent behavior.
GC-3Say exactly what each number means — no overclaimcase · live
SituationNumbers live at different semantic levels: reachable (could match), realized (did match), attributed (platform-credited), uploaded, converted (settled).The agent mustNever blur reachable ≠ realized, attributed ≠ uploaded, sent ≠ matched; state the distinction wherever a number could be misread; mark go-live-only values as pending, not asserted.
Pass
Each headline number carries its correct semantic and certainty; nothing measurable only at go-live is asserted as realized.
Fail
Presents reachable as realized, or attributed as uploaded.
How a case becomes autonomy
Autonomy is granted per phase, by evidence — never wholesale. A phase moves up the ladder only once the cases that cover it stay green. The gate climbs; it never just disappears.
todayAssistedNothing irreversible runs on its own. Prepare + validate → you approve → go-live. The cases run on every change; you see every step.
nextApprove-runOnce the read-and-prepare cases stay green, those phases run unattended; a human approves one batch.
thenBy-exceptionOn proven phases the full cycle runs; a human reviews anomalies and new cases only — each new case joins the set.
payoffBuilderThe agent proposes and stands up new work itself — gated by the same proof discipline.
This is the layer the capability stack calls Eval — the proof, and it's what makes the autonomy ladder credible rather than a promise.
Where this stands
Honest status — the cases are real and defined from your build; the automated gate that runs them on every change is being stood up.
The golden caseslive
Defined from instance #1 (your engagement) — GC-1/2/3 above, with the situation, expected behavior, and pass/fail line for each. Checked by us on every change today.
The automated regression gateillustrative
Formalizing each case into a scored input → expected-judgment pair with fixtures, so the set runs automatically and blocks any change that regresses — the gate that unlocks the next autonomy rung.
Why we show this honestly. GC-3 is the case about not overclaiming — so this page holds itself to it. The cases are real; the automated gate is in progress. We show the foundation as it is, not a green badge that isn't wired yet.