SWE-Bench scores look good, real tasks often fail
Don't be fooled by leaderboards
Contamination, overfitting, old repos, missing integration tests… Understand these gaps to build credible evaluation on your own codebase.
Why high-scoring models still fail in real projects
SWE-Bench tasks mostly come from public old repos the model may have seen. Real codebases have private domain knowledge, complex builds, cross-service integration — none of which are in the benchmark.
Specific causes of the gap
Benchmark contamination, old repos, failures on long-horizon tasks, missing integration tests and production-specific constraints. Recognizing these is the first step.
How to build evaluation for your own repo
Use real issues or PR descriptions as tasks, record success rate, average iterations, human interventions, and introduced regression bugs. Create internally comparable data.
Multi-model A/B testing on QCode
Run the same task across models and compare results. A unified key enables fast switching; the dashboard makes usage and results traceable.
Build your own real benchmark on QCode
Run real issues or PR descriptions from your repo on QCode, record success rate, average tokens, human intervention count, and form internal comparable data.
SWE-Bench reality FAQ
Are leaderboard scores useless?
They have reference value as a "relative capability" signal. You must run small-scale A/B on your own repo to know real effect.
What metrics should we measure?
Task completion rate, average iterations, human review time, introduced regression bugs, final code maintainability score.
Is Terminal-Bench more reliable than SWE-Bench?
It focuses more on terminal agent capabilities and is a useful complement. It still has overfitting risks; real value still needs your own validation.
How does QCode help us measure?
A unified key makes it easy to quickly switch multiple models to run the same task. The dashboard provides traceable usage and results for continuous tracking and improvement.
Build your production benchmark
Run real tasks with QCode. Let data speak.