The scoreboard
Overall pass@1 across all 200 tasks. Whiskers show the Wilson 95% confidence interval. At this sample size the gap between first and last is statistically real; neighboring models' intervals still overlap, so read adjacent ranks as close calls.
At 80 questions the top of the board was a near-tie. At 200 it isn't. Gemini 2.5 Flash answered 199 of 200 correctly — its confidence interval (97.2–99.9%) sits clear of GPT-5.2's (78.8–88.9%), more than eight points above its upper bound, so the order is now statistically real, not single-run noise. Gemini's lone miss is a counting puzzle it flubs on every run — the one durable crack we've found in the board's leader.
Where the separation lives
Four tiers, tuned to bite in sequence. The disciplines separate too: every model aces Biology and Computer Science, so the frontier is decided in Physics, Chemistry, and Math — and, above all, on the expert and brutal tiers.
By discipline
The same runs, cut by field. Biology and Computer Science are effectively solved by everyone; the real spread — and GPT-5.2's soft Physics score — lives in the physical sciences and math.
| Model | Physics | Chemistry | Mathematics | Biology | Comp Sci |
|---|---|---|---|---|---|
| Gemini 2.5 Flash | 100 | 100 | 98 | 100 | 100 |
| Claude Opus 4.8 | 100 | 90 | 96 | 100 | 100 |
| Claude Sonnet 4.6 | 91 | 95 | 91 | 100 | 100 |
| Claude Haiku 4.5 | 89 | 86 | 83 | 100 | 100 |
| GPT-5.2 | 78 | 81 | 79 | 97 | 97 |
| # | Model | Provider | Overall | Core | Hard | Expert | Brutal | 95% CI |
|---|---|---|---|---|---|---|---|---|
| 1 | Gemini 2.5 Flash | 99.5% | 100% | 100% | 100% | 90% | 97.2 – 99.9 | |
| 2 | Claude Opus 4.8 | Anthropic | 97.0% | 100% | 97% | 97% | 90% | 93.6 – 98.6 |
| 3 | Claude Sonnet 4.6 | Anthropic | 94.5% | 94% | 98% | 94% | 70% | 90.4 – 96.9 |
| 4 | Claude Haiku 4.5 | Anthropic | 90.0% | 94% | 96% | 84% | 70% | 85.1 – 93.4 |
| 5 | GPT-5.2 | OpenAI | 84.5% | 94% | 85% | 79% | 80% | 78.8 – 88.9 |
Single pass@1 run per model; Gemini's near-perfect score holds across three separate runs (50, 80, and 200 tasks). Biology and Computer Science are now n=30 each — the earlier n=2 caveat is gone. A parallel free-text, rubric-judged tier saw all five models score a perfect 60/60: given room to show their work, frontier models don't make the skipped-step errors that multiple-choice distractors catch.
What it measures
Four things a benchmark has to get right to separate a reasoner from a memorizer.
Not recall
Every task requires a multi-step calculation. There is no lookup that shortcuts the arithmetic.
Plausible wrong answers
Each wrong choice is the output of a specific, common mistake — dropped the ½, forgot friction, miscounted. Guessing pays poorly.
Four tiers
Core, hard, expert, brutal — each tuned to bite harder than the last, so the board separates instead of saturating.
Re-derived keys
verify_keys.py recomputes all 200
answers independently and fails loudly on any mismatch — the key is trusted because a second process agrees.
The dataset
200 closed-form tasks — 33 core, 89 hard, 68 expert, 10 brutal — one correct letter each, across five disciplines. The hardest tiers reach into amphoteric pH, the parallel-axis theorem, Bayesian inversion, the Chinese Remainder Theorem, and bounce-series distance. Every task carries its own machine-checked derivation, so the whole bank is auditable end to end.
The one question Gemini keeps missing
Q80, brutal tier — a counting problem Gemini 2.5 Flash gets wrong on every run, guessing the naive 1/4 instead of the correct 1/5. Divisibility by 4 hinges on the last two digits, and the available digits change the count.
A 5-digit number is formed using each of the digits 1, 2, 3, 4, 5 exactly once. What fraction of such numbers are divisible by 4?
The harness
The whole loop is a handful of small scripts. One adapter per lab (Anthropic, OpenAI, Google) sends the identical prompt to each model; grading is exact-letter match — no partial credit, no judge to argue with.
Load the quiz — tasks.json
200 tasks, each tagged discipline / subdomain / difficulty: question, four choices, one correct letter, and a Python derivation. This is the only file you swap to point the benchmark at a new domain.
Re-derive the keys — verify_keys.py
Integrity gate.
Evaluates each task's derivation independently and asserts it matches the stored key before any model is called. A wrong key can't reach the scoreboard — the run aborts.
Run every model — run_eval.py
Sends the identical prompt to Claude, GPT, and Gemini, collects a single letter, and scores by tier and discipline. pass@1 = first attempt, no tools, no retries.
Publish — results.json → this page
Raw counts, split by difficulty and discipline, become the scoreboard you're reading. Every number here traces back to a line in results.json.
Method & honest limits
A benchmark that overstates itself isn't worth publishing. Here is exactly what this run does and doesn't establish.
One narrow skill
This measures closed-form quantitative reasoning, not general capability. A model that trails here may lead on long-horizon, open-ended, or agentic work.
Single run, but stable
One pass@1 run per model, no voting. At n=200 the top and bottom of the board separate cleanly, though adjacent models' intervals overlap; Gemini's lead repeats across three runs.
Format beats difficulty
Free-text saturated at 60/60 for everyone. Multiple choice with error-shaped distractors is what actually separates these models.
Two easy disciplines
Quantitative Biology and CS are near-solved by all five. The next bank should make them as discriminating as the physical sciences.
The takeaway
The most important number on this page is the floor: the weakest model still scored 84.5%.
For the frontier, closed-form STEM is a solved problem — and the score that's hard to move is the one that proves it.
That isn't for lack of trying. Four difficulty tiers, 200 independently-verified items, five disciplines — plus a separate battery of 30 classic reasoning traps that the leading models solved all thirty of. Standard questions, tricky questions, long multi-step chains: the frontier handles them at 84–99%.
The handful of questions that genuinely separate these models hinge on a single non-obvious step — and authoring those at scale isn't a prompt-engineering problem, it's a domain-expertise problem. It's why GPQA needed PhDs and FrontierMath needed research mathematicians. The frontier of evaluation isn't better tooling; it's harder questions, and harder questions come from experts.
That's the work Pasiflora does — turning a vetted expert network into evaluations that can actually find the edge of what these models know. This benchmark is the floor of that capability, built in-house to test the harness. The ceiling is what our experts author.