Pasiflora Benchmark · Quantitative Reasoning

Any model can recall a formula. Can it run the numbers?

STEM-Bench is a closed-form reasoning test: every answer must be computed, not remembered. Distractors are the results of plausible mistakes, so recall alone fails. Two hundred tasks across four difficulty tiers and five disciplines, five frontier models from three labs — and every answer key is independently re-derived before scoring.

200 tasks · 4 tiers 5 models · 3 labs pass@1 · first attempt ✓ keys re-derived by script
RESULTS

The scoreboard

Overall pass@1 across all 200 tasks. Whiskers show the Wilson 95% confidence interval. At this sample size the intervals are tight and the top-to-bottom order no longer overlaps — the ranking is a result, not a coin flip.

01
Gemini 2.5 Flash
GOOGLE
99.5%199 / 200
02
Claude Opus 4.8
ANTHROPIC
97.0%194 / 200
03
Claude Sonnet 4.6
ANTHROPIC
94.5%189 / 200
04
Claude Haiku 4.5
ANTHROPIC
90.0%180 / 200
05
GPT-5.2
OPENAI
84.5%169 / 200
0%255075100%
199 out of 200

At 80 questions the top of the board was a near-tie. At 200 it isn't. Gemini 2.5 Flash answered 199 of 200 correctly — its confidence interval (97.2–99.9%) sits a clear ten points above GPT-5.2's (78.8–88.9%), so the order is now statistically real, not single-run noise. Gemini's lone miss is a counting puzzle it flubs on every run — the one durable crack we've found in the cheapest model on the board.

Where the separation lives

Four tiers, tuned to bite in sequence. The disciplines separate too: every model aces Biology and Computer Science, so the frontier is decided in Physics, Chemistry, and Math — and, above all, on the expert and brutal tiers.

Gemini 2.5 FlashGOOGLE
core
100%
hard
100%
expert
100%
brutal
90%
Claude Opus 4.8ANTHROPIC
core
100%
hard
97%
expert
97%
brutal
90%
Claude Sonnet 4.6ANTHROPIC
core
94%
hard
98%
expert
94%
brutal
70%
Claude Haiku 4.5ANTHROPIC
core
94%
hard
96%
expert
84%
brutal
70%
GPT-5.2OPENAI
core
94%
hard
85%
expert
79%
brutal
80%

By discipline

The same runs, cut by field. Biology and Computer Science are effectively solved by everyone; the real spread — and GPT-5.2's soft Physics score — lives in the physical sciences and math.

ModelPhysicsChemistryMathematicsBiologyComp Sci
Gemini 2.5 Flash10010098100100
Claude Opus 4.81009096100100
Claude Sonnet 4.6919591100100
Claude Haiku 4.5898683100100
GPT-5.27881799797
≥95% 88–94% 80–87% <80% Physics 45 · Math 53 · Chemistry 42 · Biology 30 · CS 30
#ModelProviderOverallCoreHardExpertBrutal95% CI
1Gemini 2.5 FlashGoogle99.5%100%100%100%90%97.2 – 99.9
2Claude Opus 4.8Anthropic97.0%100%97%97%90%93.6 – 98.6
3Claude Sonnet 4.6Anthropic94.5%94%98%94%70%90.4 – 96.9
4Claude Haiku 4.5Anthropic90.0%94%96%84%70%85.1 – 93.4
5GPT-5.2OpenAI84.5%94%85%79%80%78.8 – 88.9

Single pass@1 run per model; Gemini's near-perfect score holds across three separate runs (50, 80, and 200 tasks). Biology and Computer Science are now n=30 each — the earlier n=2 caveat is gone. A parallel free-text, rubric-judged tier saw all five models score a perfect 60/60: given room to show their work, frontier models don't make the skipped-step errors that multiple-choice distractors catch.

§1

What it measures

Four things a benchmark has to get right to separate a reasoner from a memorizer.

Compute

Not recall

Every task requires a multi-step calculation. There is no lookup that shortcuts the arithmetic.

Distractors

Plausible wrong answers

Each wrong choice is the output of a specific, common mistake — dropped the ½, forgot friction, miscounted. Guessing pays poorly.

Difficulty

Four tiers

Core, hard, expert, brutal — each tuned to bite harder than the last, so the board separates instead of saturating.

Integrity

Re-derived keys

verify_keys.py recomputes all 200 answers independently and fails loudly on any mismatch — the key is trusted because a second process agrees.

§2

The dataset

200 closed-form tasks — 33 core, 89 hard, 68 expert, 10 brutal — one correct letter each, across five disciplines. The hardest tiers reach into amphoteric pH, the parallel-axis theorem, Bayesian inversion, the Chinese Remainder Theorem, and bounce-series distance. Every task carries its own machine-checked derivation, so the whole bank is auditable end to end.

Mathematics 53 Physics 45 Chemistry 42 Biology 30 Computer science 30
§3

The one question Gemini keeps missing

Q80, brutal tier — a counting problem Gemini 2.5 Flash gets wrong on every run, guessing the naive 1/4 instead of the correct 1/5. Divisibility by 4 hinges on the last two digits, and the available digits change the count.

TASK 80 · BRUTALDOMAIN · counting

A 5-digit number is formed using each of the digits 1, 2, 3, 4, 5 exactly once. What fraction of such numbers are divisible by 4?

A 1/4
B 1/5
C 1/6
D 1/3
A number is divisible by 4 iff its last two digits are. valid endings from {1,2,3,4,5}, distinct, divisible by 4: 12, 24, 32, 52 → 4 endings each fixes 2 digits; the other 3 arrange in 3! = 6 ways → 4 × 6 = 24 numbers total arrangements = 5! = 120 fraction = 24 / 120 = 1/5 → B why the trap bites: A 1/4 — assumes the naive "1 in 4 integers" density ← Gemini lands here every run but with no 0 available, only 24 of 120 qualify, not 30
§4

The harness

The whole loop is a handful of small scripts. One adapter per lab (Anthropic, OpenAI, Google) sends the identical prompt to each model; grading is exact-letter match — no partial credit, no judge to argue with.

01

Load the quiz — tasks.json

200 tasks, each tagged discipline / subdomain / difficulty: question, four choices, one correct letter, and a Python derivation. This is the only file you swap to point the benchmark at a new domain.

02

Re-derive the keys — verify_keys.py

Integrity gate.

Evaluates each task's derivation independently and asserts it matches the stored key before any model is called. A wrong key can't reach the scoreboard — the run aborts.

03

Run every model — run_eval.py

Sends the identical prompt to Claude, GPT, and Gemini, collects a single letter, and scores by tier and discipline. pass@1 = first attempt, no tools, no retries.

04

Publish — results.json → this page

Raw counts, split by difficulty and discipline, become the scoreboard you're reading. Every number here traces back to a line in results.json.

§5

Method & honest limits

A benchmark that overstates itself isn't worth publishing. Here is exactly what this run does and doesn't establish.

One narrow skill

This measures closed-form quantitative reasoning, not general capability. A model that trails here may lead on long-horizon, open-ended, or agentic work.

Single run, but stable

One pass@1 run per model, no voting. At n=200 the intervals are tight and non-overlapping top to bottom; Gemini's dominance repeats across three runs.

Format beats difficulty

Free-text saturated at 60/60 for everyone. Multiple choice with error-shaped distractors is what actually separates these models.

Two easy disciplines

Quantitative Biology and CS are near-solved by all five. The next bank should make them as discriminating as the physical sciences.

§6

The takeaway

The most important number on this page is the one that didn't happen: nobody scored below 85%.

For the frontier, closed-form STEM is a solved problem — and the score that's hard to move is the one that proves it.

That isn't for lack of trying. Four difficulty tiers, 200 independently-verified items, five disciplines — plus a separate battery of 30 classic reasoning traps that the leading models solved all thirty of. Standard questions, tricky questions, long multi-step chains: the frontier handles them at 85–99%.

The handful of questions that genuinely separate these models hinge on a single non-obvious step — and authoring those at scale isn't a prompt-engineering problem, it's a domain-expertise problem. It's why GPQA needed PhDs and FrontierMath needed research mathematicians. The frontier of evaluation isn't better tooling; it's harder questions, and harder questions come from experts.

That's the work Pasiflora does — turning a vetted expert network into evaluations that can actually find the edge of what these models know. This benchmark is the floor of that capability, built in-house to test the harness. The ceiling is what our experts author.