Deep dive · disputed benchmark data · checked 2026-10-20

Claude Fable 5 on SWE-Bench Pro: Where the 80.3% Score Comes From, and Whether to Trust It

Claude Fable 5 scored 80.3% on SWE-Bench Pro -- one of the highest published numbers out there. But how was it actually measured, and why do independent evaluators report different results? Here's both sides.

#80.3% Score#SWE-Bench Pro#Methodology Dispute#Claude Fable 5

Checked 2026-10-20: still Anthropic's own measurement

Checked 2026-10-20: the 80.3% is still Anthropic's official figure, produced with its own evaluation scaffolding; Scale's official leaderboard page (labs.scale.com) lists no rows for Fable 5, Opus 5 or Sonnet 5 that day. The aggregated boards (llm-stats / benchlm, snapshots of 2026-08-18/21) record Fable 5 at 80.0% with Mythos 5 leading at 80.3% - both numbers exist, the difference is the methodology. Note also that Scale's official unified-framework board is lower across the board. The full comparison of the three methodologies is on the site under "SWE-bench Pro three first places".

Last checked 2026-10-20 (Scale's official leaderboard page has no Fable 5 row; the two readings, official self-report 80.3 / aggregated board 80.0, are unchanged). Earlier check: 2026-08-22 (both methodologies side by side: official self-report 80.3 / aggregated board 80.0).

SWE-Bench Pro Leaderboard Comparison

The SWE-Bench Pro scores Anthropic published at Fable 5's launch, compared against other flagship models from the same period. All figures come from official/evaluator-published materials -- pay attention to the "Evaluator/Methodology" column, since these weren't all produced by the same neutral test run.

Model	SWE-Bench Pro Score	Evaluator / Methodology	—	Note
Claude Fable 5	80.3%	Anthropic's own evaluation scaffolding	—	Tied with Mythos 5
Claude Opus 4.8	69.2%	Anthropic's own evaluation scaffolding	—	Previous-generation Claude flagship, included as a baseline
GPT-5.5	58.6%	Comparison data published by Anthropic	—	OpenAI's flagship model; not independently verified by OpenAI
Gemini 3.1 Pro	54.2%	Comparison data published by Anthropic	—	Google's flagship model; not independently verified by Google

A Closer Look at the Score

How the 80.3% was measured

SWE-Bench Pro is an evaluation suite built on real-world software engineering tasks: models must complete fixes, refactors and similar work inside codebases that resemble real development scenarios — closer to production complexity than earlier SWE-Bench versions. Claude Fable 5 scored 80.3% in Anthropic's official materials (tied with Mythos 5), while the aggregated boards (llm-stats / benchlm) record 80.0% for Fable 5 and 80.3% for Mythos 5 — both numbers genuinely exist; the question is not which one is fabricated, but under which methodology each was measured.

Key clarification: official evaluation methodology, not a neutral reproduction

The 80.3% figure was produced using Anthropic's own evaluation scaffolding -- in other words, this is an official-methodology score, not one independently reproduced by a neutral third-party test harness. That distinction matters: the scaffolding determines how the model reads each task, how it calls tools, whether it can retry after a failure, and how much context budget it gets -- details that materially affect the final pass rate. And when the vendor being tested builds and tunes that scaffolding itself, there's inherent room for it to be more favorable to their own model. techjacksolutions.com published a skeptical report specifically noting that when independent evaluators reproduced SWE-Bench Pro using their own methodology, the resulting scores differed from Anthropic's official number. This doesn't mean 80.3% is fake or meaningless -- it means it's a score under official evaluation methodology, which is a different thing from an independent reproduction, and readers deserve to know both sides rather than just the number from the launch announcement.

What this kind of score is -- and isn't -- good for

Benchmark scores are a useful data point, but not the whole story for model selection. SWE-Bench Pro reflects performance on a specific distribution of tasks -- bug-fix-style work in real-world codebases -- and your own use case might look very different: greenfield builds, complex system design, large-scale refactors under long context windows. The score doesn't necessarily transfer. On top of that, inconsistent methodology (official scaffolding vs. independent reproduction) means numbers on the same leaderboard weren't necessarily produced under identical, fair conditions. When actually choosing a model, look beyond the benchmark score at your specific task type, context-length needs, and the cost differences between models -- and ideally run a small-scale test with your own real tasks before deciding.

How This Page Relates to the Other Two

These three pages are complementary, not redundant -- each one covers a different question.

Want the full leaderboard?

For the complete SWE-Bench Pro rankings across all major models, see the "SWE-Bench Pro 2026 Leaderboard" page. This page goes deep on one model, Fable 5, and the story behind its score -- it doesn't re-list the full rankings here.

Want the broader methodology critique?

The general methodological limitations of benchmarks like SWE-Bench -- task-selection bias, scaffolding dependence, how representative they are of production work -- are covered in depth in "SWE-Bench and the Reality Gap in Production." This page relies on that page's conclusions rather than re-arguing them here.

Using Fable 5 on QCode

QCode already offers Claude Fable 5, accessible with a single key. For pricing details and comparisons against other models, see the pricing guide -- that's not covered here.

Timeline

The key dated points that are publicly documented (checked 2026-10-20).

2026-06-09

Claude Fable 5 launches, 80.3% score published

Anthropic launches Claude Fable 5; official evaluation materials show an 80.3% score on SWE-Bench Pro, tied with Mythos 5, launched the same day.

Primary Sources

the-decoder.com on the Claude Fable 5 launch and its benchmark data; the sceptical techjacksolutions.com report on the gap between the official SWE-Bench Pro score and independently reproduced results; the vellum.ai breakdown of the SWE-Bench Pro benchmark; Anthropic's official release page. Added on 2026-10-20: Anthropic's official model overview (platform.claude.com/docs/en/about-claude/models/overview, fetched 2026-10-20) and Scale's official leaderboard page (labs.scale.com/leaderboard/swe_bench_pro_public, fetched 2026-10-20).

FAQ

Is Claude Fable 5's 80.3% on SWE-Bench Pro the highest score out there?

It depends on the methodology: in Anthropic's official materials Fable 5 and Mythos 5 tie at 80.3%; on the aggregated boards (llm-stats / benchlm, 08-18 snapshot) Mythos 5 leads at 80.3%, Fable 5 is second at 80.0%, and Opus 5 third at 79.2%. Please cite the methodology when quoting.

Is this score trustworthy? Why is it disputed?

The score itself is a genuinely published number, not a fabrication; but 80.3% is the official-methodology result from Anthropic's own evaluation scaffolding, while the aggregated boards (llm-stats / benchlm) record 80.0%. Scale's official board on the unified SEAL framework is lower still across the board. The essence of the dispute is 'who built the test conditions and which data split was used', not whether the score exists. For the three-methodology comparison, see 'SWE-bench Pro's Three First Places'.

What's the actual difference between official methodology and independent reproduction?

The evaluation scaffolding determines how a model reads task descriptions, calls tools, retries after failures, and how much context/step budget it's given -- implementation details that materially affect the final pass rate. When the vendor whose model is being tested builds and tunes that scaffolding themselves, results can diverge from what a neutral third party running all models under one consistent standard would get. This is a general issue with vendor-published benchmark numbers, not something unique to Fable 5.

Has Fable 5.2 shipped? Which Fable is current now?

As of 2026-10-20 the official model overview still lists claude-fable-5-1 as the current Fable (retirement not sooner than 2027-09-01); there is no claude-fable-5-2 id. What Anthropic has added since 2026-09 is Opus 5.5 (2026-09-22), Sonnet 5.5 (2026-09-28) and Haiku 5.5 (2026-10-07).

Create a QCode account

One key across Claude, GPT and Chinese models, billed per token — live rates on /models.

Create account

Models and live rates View plans