Three First Places on SWE-bench Pro
Nobody lied, nobody told the whole story
Anthropic's system card says Fable 5 = 80.3; aggregators say Fable 5 = 80.0 / Mythos 5 = 80.3; Scale's official board reads a notch lower. The difference comes from data splits and scaffolds — not score-gaming.
The three circulating 'first places'
Vendor self-report · Fable 5
Anthropic system-card terms: their own harness, their own runs. Mythos 5 scores the same.
Aggregators · Fable 5
Aggregator sites like llm-stats record Fable 5 as 80.0 and Mythos 5 as 80.3 — a 0.3 gap from the system card.
Scale official · GPT-5.4 xHigh
Scale's official public split (the standardized SEAL framework), 2026-04. Lower numbers, but the strictest terms.
Public / private task counts
SWE-bench Pro has 1865 tasks in total: 731 public, 276 private commercial, 858 held out. Vendors aren't all running the same set.
What SWE-bench Pro is
An enterprise-grade, long-horizon coding benchmark released by Scale AI in 2025-09: 1865 tasks drawn from 41 professional repositories. The public split uses only strong-copyleft GPL-style licenses to suppress training contamination; the private split is closed-source code from contracted startups. Grading is strict: a patch must make the new fail-to-pass tests pass without breaking any existing test. It's an order of magnitude harder than Verified — models scoring 70%+ on Verified started at about 23% on Pro.
Why the numbers don't match
The three sources aren't measuring the same kind of 'score' at all: vendor system cards self-report on their own harnesses; aggregators mix vendor self-reports with third-party reruns; Scale's official board reruns everything under the unified SEAL framework and lists public/private splits separately. Layer on reasoning-effort tiers (high/xhigh) and snapshot dates, and the same model can legitimately own three or four different scores.
Timeline
Scale releases SWE-bench Pro; initial baselines: GPT-5 at 23.3% / Opus 4.1 at 23.1% (public split).
Fable 5 / Mythos 5 launch, system cards self-report 80.3; aggregators gradually record them as 80.0 / 80.3.
Aggregated boards settle: Mythos 5 at 80.3, Fable 5 at 80.0, Opus 5 at 79.2 lead; Scale's official board reads markedly lower under its terms.
Confirmed vs easily misread
Confirmed
All three sets of numbers genuinely exist with their own provenance; the differences are fully explained by data split (public/private), scaffold (vendor harness vs the unified SEAL framework) and effort tier.
Easily misread
'X tops SWE-bench Pro at 80+' and 'SWE-bench Pro peaks in the 50s' can both be true — the former is vendor self-report/aggregator terms, the latter is Scale's official unified-framework terms. Always cite the terms when quoting a number.
Which terms to read
Cross-model selection
Read Scale's official board or unified-framework reruns like vals.ai: lower numbers, but comparable — one ruler for every model.
Tracking the frontier
Read vendor system cards and aggregated boards: fresher and higher numbers, but only comparable vertically against historical numbers under the same terms.
How not to be fooled by leaderboards
Three steps: ① Ask about the data split — the 731 public tasks or the 276 private ones; ② ask about the scaffold — a vendor's own harness or the unified SEAL framework; ③ ask about the effort tier and the date. Don't compare numbers horizontally unless all three answers line up.
On QCode
The models at the front of the boards — Fable 5, Opus 5, the full GPT-5.6 line, GLM-5.3, Kimi K3, DeepSeek V4 Pro — are all callable with one QCode key. Rather than arguing over terms, run your own repo once: one endpoint, one scaffold, numbers you produce yourself.
FAQ
So who is actually first on SWE-bench Pro?
By aggregator terms: Mythos 5 (80.3); by vendor self-report: Fable 5 and Mythos 5 tied at 80.3; Scale's official unified-framework numbers sit a notch lower overall. Each of the three 'first places' comes with its own terms.
Why is the official board so much lower than vendor self-reports?
Scale's official board reruns every model under the unified SEAL framework and reports public/private splits separately; vendor self-reports use their own tuned harnesses. The framework gap alone can cost 20 points on the hard tasks.
What's the difference between the public and private splits?
The 731 public tasks come from strong-copyleft GPL-style repos; the 276 private tasks are closed-source commercial code from contracted startups — lowest contamination risk and generally lower scores. A further 858 held-out tasks are not public.
How does SWE-bench Pro relate to Verified?
Pro is the successor benchmark released in 2025-09: longer-horizon, closer to enterprise tasks, contamination-resistant by design. Models at 70%+ on Verified started at about 23% on Pro.
Does the effort tier affect scores?
Yes. The high and xhigh tiers differ visibly on hard tasks, and vendor self-reports usually take the top tier. Confirm both sides' tiers match before comparing.
What's the most reliable way to rerun for selection?
Fix one scaffold (e.g. the official SWE-Agent or SEAL), use a unified API endpoint, and run real issues from your own repos — closer to your workload than any third-party board.
Sources
Scale's official SWE-bench Pro page (labs.scale.com, fetched 2026-08-22), Anthropic's Fable 5 system card (via vellum), llm-stats, the benchlm aggregated board (08-21), techjacksolutions' reporting on the terms dispute.
Rather than trusting boards, run your own
The front-running models are callable on the same QCode endpoint — rerun your own repos under one scaffold.
Related reading
SWE-bench Pro 2026 ranking
The latest standings snapshot.
Terminal-Bench 2.1 explained
Another benchmark with three leaderboards.
SWE-bench production reality
The gap between leaderboard scores and real engineering.
Leaderboard numbers are a 2026-08-22 snapshot, with sources cited in the text. Not affiliated with Scale / Anthropic.