Pick a model with your own workload, not a leaderboard.

Benchmarks are static and generic by design. MMLU tests general knowledge recall. HumanEval tests short, self-contained coding problems. Neither one touches your actual workload: a 40-page vendor contract with inconsistent formatting, a support ticket with three back-and-forth messages and a screenshot, a database migration script with your team's naming conventions baked in.

A model can rank first on a public leaderboard and still get your workload wrong 30% of the time, because the leaderboard never saw anything like your workload. Worse, providers optimize models against these public benchmarks, which means a high score can reflect benchmark-specific tuning rather than general capability.

We've watched teams spend six weeks debating GPT-5 versus Claude versus Gemini based on leaderboard deltas of two or three points, then ship a workflow that fails on the first messy real-world input because nobody tested with real data. The debate itself was the waste. Production readiness starts with your own eval set, not someone else's scoreboard.

The current consensus, repeated in most 2026 vendor blog posts, is "just use the frontier model, cost has come down enough that it doesn't matter." That's true for low-volume, high-value tasks like a single legal review. It's false for anything running at scale.

At 50,000 calls a day, a $3-per-million-token difference between two models is a five-figure monthly line item. And frontier models are not uniformly better at every task; they're often tuned for reasoning-heavy benchmarks while a smaller, cheaper model matches them on structured extraction or classification, which is what most production workflows actually need. The "just use the frontier model" take skips the step where you measure your own task, and it skips cost entirely. Cost per task, not cost per token, is the number that should drive the decision, because it accounts for retries, context length, and output length together.

1. Pull 50 real tasks from your own traffic. Not synthetic examples. Actual support tickets, actual documents, actual queries, sampled to cover common cases and known edge cases in roughly a 70/30 split.
2. Write a rubric before you see any model output. Define what "correct" means for each task type: exact match, contains required fields, passes a schema check, or a human judgment scale of 1 to 5. Writing the rubric first prevents you from moving the goalposts once you like a model's answer.
3. Run all candidate models against the same 50 tasks, blind. Strip model names from the outputs before scoring. Reviewers should not know which model produced which answer; this alone removes a surprising amount of bias toward whichever brand a reviewer prefers.
4. Score accuracy, then price it. Calculate cost per task including input tokens, output tokens, and any retries needed to get a valid structured output. A model that's 3% more accurate but 4x the cost per task is rarely the right default.
5. Check latency against your UX requirement. A model that's accurate and cheap but takes 12 seconds per call fails a live chat use case even if it passes every other test.
6. Pick a primary and a fallback. Your primary model should clear the bar on accuracy, cost, and latency. Your fallback should be a different provider entirely, so a provider outage doesn't take down your workflow.
7. Write down the swap plan. Document exactly what changes if you switch models: prompt adjustments, output parsing, cost dashboards. This turns a future model swap into a half-day task instead of a rewrite.

This whole process takes two focused days for one engineer, and it produces a written recommendation with numbers a CFO can read in five minutes.

The eval set is the deliverable that outlasts the model choice. Once built, it becomes your regression suite for every future model version and every prompt change. Treat it as a permanent asset, not a one-time exercise.

• Sample from real production logs or support queues, anonymized as needed. Synthetic examples miss the messiness of real inputs.
• Cover the distribution: common cases (roughly 70%), edge cases (20%), and adversarial or malformed inputs (10%).
• Include at least five examples where the correct answer is "I don't know" or "escalate to a human." Models that never say they're unsure are a liability, not a feature.
• Version the eval set in your repo alongside your prompts, so you can see exactly what changed between runs.

If you're building an MCP-based tool layer, extend the eval set to cover tool call accuracy too: did the model choose the right tool, with the right arguments, in the right order. Model selection and tool selection are two different evaluation problems, and conflating them hides which one is actually broken.

Vendors price models per million tokens, which makes cross-model comparison feel simple and is precisely why it misleads. A model with a lower per-token price can still cost more per task if it needs longer prompts, produces more verbose output, or requires retries to get valid structured JSON.

Calculate cost per task as: (average input tokens x input price) + (average output tokens x output price) + (retry rate x average retry cost). Run this across your 50-task eval set and you'll often find the "cheap" model isn't cheap once retries are counted, because it fails structured output validation more often and needs a second pass.

We've seen retry rates alone flip a model recommendation. A model priced 40% lower per token ended up costing more per task because it failed schema validation on 1 in 6 calls, each retry doubling the cost of that task.

The biggest hidden cost of model selection isn't picking the wrong model, it's building a system that can't switch. Vendor SDKs each have their own quirks: function calling formats, streaming protocols, system prompt conventions. Code that calls these SDKs directly, scattered across your codebase, turns a model swap into a multi-week refactor.

• Build a thin internal abstraction layer with one interface for "generate," "call tool," and "stream," regardless of provider.
• Keep prompts in version-controlled files, not hardcoded strings, so they transfer between models with minimal rewriting.
• Store your 50-task eval set independent of any model, so re-running it against a new candidate is a config change.
• Review your primary/fallback pairing quarterly, since pricing and capability both shift fast.

This discipline is the same one we apply during Claude implementation engagements: the architecture should never assume permanence of a single vendor's API shape.

Model providers ship new versions silently, sometimes rerouting the same model name to different weights behind the scenes. This means your carefully chosen model can degrade without any code changing on your side. We recommend two triggers for re-running the full evaluation:

• Every quarter, on a calendar reminder, regardless of whether you've heard about a new release. This catches silent drift.
• Immediately after any provider announces a new model version in the family you're using, since pricing and capability often shift together.

Keep the eval set and the scoring rubric stored exactly as they were the first time, so results are comparable across quarters. This is the same eval discipline that should already sit underneath any pilot moving toward production.

On a recent engagement, a mid-market insurance client was locked in a six-week internal debate about whether to use a frontier model or a smaller, cheaper one for claims document extraction. We built a 50-task eval set from their actual claims documents in two days, ran three candidate models blind, and found the mid-tier model matched the frontier model's accuracy on structured field extraction while costing 60% less per task. The frontier model only pulled ahead on a subset of documents with unusual formatting, which we routed to a fallback path instead of upgrading the whole workflow.

The debate that had consumed six weeks was settled with two days of measurement. That pattern repeats across nearly every engagement: the model choice was never the hard part, defining the task and building the eval set was.