A new model shipped — how do you know if it is worth switching
Public benchmarks answer "how does it do on standard problems". You want to know "how does it do on my codebase". Those two answers frequently differ.
Four principles
Use your own tasks
Twenty real tasks you handled recently predict a model’s value to you better than any public leaderboard.
One task set for every candidate
Comparing models on different problems yields a meaningless conclusion. The set must be fixed.
Hide the names when scoring
Knowing which output came from which model measurably shifts your judgement. Cover the names first.
One run proves nothing
Run each task three times. If the three differ widely, the conclusion "it is better" is itself unstable.
Why public benchmarks do not help here
Public benchmarks solve a comparability problem: a fixed problem set and a fixed scoring method let different models sit in one coordinate system. That has research value but limited selection value, for three reasons. The problems usually have nothing to do with your codebase or domain; the scoring may not correspond to the quality dimensions you care about; and once a model is widely used, the chance of benchmark problems leaking into training data rises over time. A high score does not imply "better for you".
About newly released models
In the first days after a release, public opinion swings hard: early adopters have small samples and skewed task distributions, and launch-period congestion leads people to mistake latency problems for quality problems. If you are not in a hurry, evaluating a week or two later gives you far more stable information.
The steps
Select and freeze twenty real tasks. The quality of this step determines the value of the whole evaluation.
Every candidate runs the same set, three times per task, recording cost and latency.
Score blind on the dimensions you care about, then bring cost back into the picture.
Leaderboards narrow the field; your own runs make the decision
What this page can assert
The methodology of a self-built evaluation is general and independent of which model is involved: fix the task set, compare on the same problems, score blind, sample repeatedly, and record cost alongside. These hold for any model selection.
What this page does not state
We have not run a systematic evaluation of Ornith-1.5, so this page gives no benchmark figures, no leaderboard positions and no "beats model X" conclusions. Such numbers distort easily when retold, and model selection costs money and time — it should not rest on unverified figures.
Two bases for choosing
Read the leaderboards
Free and quick for narrowing the field. But it cannot answer "is it good for my work", and it is noisy during a launch window.
Build a small evaluation
Roughly two hours, the conclusion maps directly to your situation, and the task set is reusable next time. The cost is assembling the set first.
A self-evaluation you can finish in two hours
Step one: take twenty tasks you actually handled in the last fortnight, covering your normal distribution — do not cherry-pick hard ones, since routine work is the bigger share. Step two: freeze that set and run every candidate on the same twenty. Step three: strip the model names before scoring, using the dimensions you genuinely care about — did it need rework, did it break something elsewhere, did it understand the context. Step four: run each task three times and look at stability. Step five: record cost alongside. When quality is close, cost and speed are what decide.
Does QCode offer Ornith-1.5
To be explicit: QCode does not currently offer Ornith-1.5. What we provide is seven families — Claude, GPT/Codex, Gemini, GLM, Kimi, DeepSeek and Qwen — under one key. If Ornith-1.5 is specifically what you want to evaluate, you will need its own official channel; the method on this page applies there just as well. If your evaluation leads you to stay within the families above, what we offer is access billed by a fixed daily allowance with predictable cost.
FAQ
Can I use Ornith-1.5 on QCode?
Not at present. QCode provides the Claude, GPT/Codex, Gemini, GLM, Kimi, DeepSeek and Qwen families; Ornith-1.5 is not among them.
Why does this page give no benchmark numbers?
Because we have not run them ourselves. Publishing unverified figures harms the person making a selection decision — and that decision costs money and time.
Are twenty tasks enough?
Usually enough for an individual decision, provided the twenty genuinely reflect your normal distribution. If candidates come out very close, extend to fifty and look again.
Why hide the model names?
Because knowing the source measurably shifts scoring. It is especially pronounced for newly released models, where anticipation alone lifts the scores people give.
Is running each task three times wasteful?
That step is the one that saves the most. If three runs differ widely, "it is better" is an unstable conclusion, and a migration decision built on it costs far more.
What if the evaluation shows my current model is fine?
That is a valuable result. The common output of a selection evaluation is not "switch" but "no need to switch" — which saves you the migration cost.
Sources
Model capability and availability follow the publisher’s official statements. This page provides only an evaluation method and does not repeat third-party benchmark numbers.
Evaluation done — now you want predictable access
One key across seven model families, billed by a fixed daily allowance.
Related reading
Models supported by QCode
The families currently available.
Multi-model routing
Layering different models across task types.
Model pricing comparison
Bringing cost into the selection decision.
Third-party model capability and availability follow their publishers’ official statements. This page provides no benchmark figures, only a method you can run. The families QCode currently offers are listed on our models page.