The scoreboard
Share of 12 questions answered correctly, mean of three runs, each checked against a value computed live from the database. On multiple choice these ten models cluster in the 90s. On real investigative work they spread from 92% to 25%.
Every model here can write SQL — the gap isn't knowledge. It's rigor over messy data: noticing that
succeeded and Succeeded are both real charges, that amounts are in cents, that some
rows are duplicated. Miss one trap and the number is wrong. Claude Opus 4.8 is the only model that
consistently catches all of them; everyone else bleeds on the questions where the traps compound.
The five traps
This isn't a quiz with a clever distractor. It's an ordinary, uncleaned production database — the kind a real analyst inherits. We don't tell the model what's wrong with it. The only instruction is: "this is raw production data, investigate before you trust any number." Spell the traps out and even a small model scores 100%; leave them to be discovered and the field falls apart.
Amounts are in cents
Every payment is stored in amount_cents. Answer in dollars without dividing by 100 and you're off by 100×.
Two spellings of success
Successful charges are stored as both succeeded and Succeeded. A naive
status='succeeded' silently drops $361K of revenue.
Double-recorded charges
Some payments were written twice (an idempotency slip). Sum without de-duplicating and totals inflate.
Deleted accounts still transact
Some customers are flagged is_deleted but their rows remain. Revenue figures should exclude them — you have to notice the flag.
MRR in mixed currencies
Subscription MRR is in USD, EUR and GBP. Any comparison needs an FX join first — the raw maximum and the USD maximum are different customers.
Discovery, not a checklist
Every trap is discoverable from the schema and the DISTINCT values. A careful analyst finds
them; a rushing model doesn't. That gap is the benchmark.
Where they break
All twelve questions, ranked by how often the field got them right (out of 30 model-runs). The categorical questions — "which region," "which customer" — survive a missed trap, because relative order doesn't change. The exact-dollar questions, where a single missed trap corrupts the number, are where the board is decided.
| Q | What it asks | Traps it depends on | Solved |
|---|---|---|---|
| 11 | Distinct successful charges (after de-dup) | dedup · casing · deletes | 3/30 |
| 09 | Highest-revenue month, in dollars | cents · casing · dedup · deletes | 6/30 |
| 12 | Revenue by 'wire' method, in dollars | all five + method filter | 6/30 |
| 01 | Total net revenue for 2024, in dollars | cents · casing · dedup · deletes | 8/30 |
| 05 | Average Enterprise MRR, in USD | currency FX | 9/30 |
| 06 | Overall churn rate | — | 16/30 |
| 10 | Churned customers with a high-severity ticket | two-table join | 16/30 |
| 04 | "At-risk" active customers | three-table join | 20/30 |
| 07 | Highest active MRR, in USD | currency FX | 20/30 |
| 08 | Deleted customers who still paid | deletes · casing | 20/30 |
| 02 | Top region by Q3 revenue | ranking (trap-robust) | 21/30 |
| 03 | Largest-refund customer, in dollars | cents | 21/30 |
The single hardest question — count the distinct successful charges after removing duplicate rows — was answered correctly in just a handful of the thirty model-runs. It's not conceptually hard. It just requires catching three quiet defects at once, which is exactly what separates a model that can do the job from one that can only describe it.
The harness that nearly lied
The most important thing we learned building this isn't a score. It's that a fair cross-lab agentic benchmark is a harness-engineering problem before it's a model-ranking one. Our first leaderboards were wrong, twice, and both times it was our wiring — not the model.
Artifact 1 — silent non-answers
A model can fail a run for a reason that has nothing to do with capability — emitting prose that describes a query instead of making a structured tool call, then answering nothing. Scored naively that's a false 0. Our harness flags any run that makes zero tool calls and excludes it rather than recording a zero it didn't earn.
Artifact 2 — running out the clock
Thorough models kept querying until the step budget ran out, never stopping to answer — and scored a false 0. We added a forced final turn with the tool removed, compelling an answer. That single fix moved Gemini 2.5 Pro from 8% to 42% and Flash from 19% to 56%.
Native tool-calling per provider, three runs, ground-truth grading, and every surprising result traced back to a transcript before it's trusted. If a strong model lands at the bottom of your board, suspect your rig first. Getting the harness right is unglamorous — and it's exactly where most public benchmarks quietly go wrong.
Method & honest limits
A benchmark that overstates itself isn't worth publishing. Here's exactly what these runs do and don't establish.
No LLM judge
Every answer is an exact integer checked against a value computed live from the seeded database — objective right/wrong, no model grading another model.
Discovery is the difficulty
Enumerate the traps in the prompt and even Haiku scores 100%. The difficulty is that the model must find them — which is fair, because the task tells it the data is dirty.
Single config, three runs
Scores are the mean of three runs at one setting. The bottom cluster (Grok, Qwen, DeepSeek) is high variance; read order below ~40% as approximate, not final.
A snapshot, with an asterisk
Model versions move weekly. One provider's API hit a usage cap mid-run, so the Claude scores are from clean runs on the identical harness rather than a fresh re-run.
The takeaway
The test isn't what a model knows. It's whether it notices what's wrong with the data.
Multiple choice is solved — a quiz can't tell frontier models apart anymore. The moment you hand them something that looks like an actual job — a database nobody cleaned, a question with no answer key — the field opens from a ten-point huddle to an eighty-point spread. Not because the questions are obscure, but because doing the work means catching what everyone else glossed over.
That's the capability worth measuring, and measuring it takes a harness you've stress-tested against its own failures — native tools, real ground truth, multiple runs, every surprise traced to a transcript before you publish a number. We built this one in-house to prove the harness.
The ceiling is what our experts author. Pasiflora turns a vetted expert network into evaluations rigorous enough to find the real edge of what these models can do. This board is the floor.