OpenAI vs Anthropic vs Gemini vs open-weight: how we choose per task
How should a business go about choosing an LLM for a production system?
Choose per task by benchmarking on your data for accuracy, latency and cost; most production systems route between two or three providers. No single model wins every task, leaderboards do not predict your workload, and a routing layer keeps the choice reversible. Here is the method we use and what it costs.
Choosing an LLM is not a one-time vendor decision; it is a per-task engineering decision that you revisit every few months. The honest answer to "which model is best?" is "best at what, on whose data, at what price?" Every LLM application we ship runs on a routing layer, and the model behind each task is chosen by running your own evaluation set against the candidates and reading the numbers.
This article explains why leaderboards mislead, how we benchmark, what each family of providers tends to be good at in our experience, when open-weight models earn their place, and how a routing layer keeps the whole decision reversible.
Why choosing an LLM is a per-task decision
A production system does many different things. It classifies incoming requests, extracts fields from documents, drafts replies, summarises long threads, writes queries, calls tools and occasionally reasons through a multi-step problem. These tasks have different accuracy requirements, different latency budgets and wildly different volumes. A classifier that runs on every request needs to be fast and cheap; a reasoning step that runs once per hour can afford the largest model available.
Picking one model for all of them means overpaying on the cheap tasks and underperforming on the hard ones. Picking per task means each step gets the model that scores best on your data at a cost you can justify. That is the whole method.
Why public leaderboards do not answer the question
Public benchmarks measure general capability on academic tasks: maths, code, exam questions, multi-turn chat preferences. They are useful for spotting a generational jump and useless for predicting whether a model will extract the right fields from your invoices in Kannada or summarise your support tickets in your house style. The gap between leaderboard rank and performance on a specific workload is large and unpredictable in both directions.
There is a second problem. Models change. Providers release new versions, deprecate old ones and quietly adjust behaviour. A choice made in January on a leaderboard may be wrong by June. The only durable answer is a benchmark you own and can rerun.
How we benchmark: accuracy, latency, cost
The benchmark is the eval suite described in evals: the practice that separates demos from products: one to three hundred real inputs per task with expected outputs and scorers. We run every candidate model against the same suite and record three things per case.
- Accuracy, scored per task with the appropriate method: exact match for extraction and classification, a calibrated rubric for drafts and summaries, recall and groundedness for retrieval
- Latency, measured as time to first token and time to complete, at realistic concurrency rather than a single warm request
- Cost, computed from actual input and output tokens per case multiplied by the provider's published price, then projected to expected monthly volume
The result is a table per task, not a winner. A model that scores a few points lower on accuracy at a third of the cost is usually the right choice for a high-volume classifier and the wrong choice for a legal summary that a person will rely on. The decision is made by reading the table against the task's requirements, and it is written down so it can be revisited.
What the provider families are typically good at
What follows is pattern, not ranking, and it shifts with every release. It reflects what we see across our own benchmarks; your workload may differ, which is the point of running your own.
| Family | Where it tends to fit | Watch for | Typical role in a routed system |
|---|---|---|---|
| OpenAI | Broad general capability, mature tool calling, wide ecosystem | Pricing tiers change; behaviour shifts between versions | Default for tool-heavy agent steps and general drafting |
| Anthropic | Long-context reading, careful instruction following, low tendency to invent | Rate limits at scale need planning | Summaries, analysis, anything where groundedness is judged |
| Google (Gemini) | Very long context, multimodal input, strong price-performance at the mid tier | Regional availability and quota vary | Document and image-heavy pipelines, high-volume mid-tier tasks |
| Open-weight (Llama, Mistral, Qwen and others) | Data never leaves your VPC; cost floor at volume; fine-tunable | You own serving, scaling and upgrades | Classification, extraction and routing at volume; regulated workloads |
Each provider documents its models, pricing and limits: OpenAI, Anthropic, Google AI and, for open weights, the model cards on Hugging Face. Read the pricing page before the benchmark, not after; cost per million tokens is only meaningful once you know the tokens per case.
OpenAI vs Anthropic: the comparison clients ask for most
Clients often frame the decision as OpenAI versus Anthropic because those are the names they know. In practice both are capable across most tasks and the difference on any specific workload is measured in a benchmark, not a blog post. Where we do see consistent patterns: tasks that reward careful reading of long, messy inputs and refusing to invent tend to favour Anthropic in our runs; tasks that lean on a deep ecosystem of tool integrations and fine-tuning options often go to OpenAI. Both change with each release. Run both; keep both in the routing table; switch when the numbers say so.
When open-weight models earn their place
Open-weight models running on your own infrastructure make sense in three situations. First, when data residency or regulation means prompts cannot leave your environment; the self-hosted LLM checklist for BFSI covers that case. Second, when volume is high enough that per-token API pricing exceeds the cost of running a GPU, which for narrow tasks like classification happens sooner than most teams expect. Third, when a task benefits from fine-tuning on your data and you want to own the result.
They do not make sense when the team cannot operate GPU infrastructure, when the task needs frontier-level reasoning, or when volume is low. Our open-weight vs GPT-class benchmark approach walks through the decision with the same suite.
The routing layer: making the choice reversible
Every model call in our systems goes through a routing layer that maps a task name to a provider, a model and a fallback. Application code asks for "classify-intent" or "draft-reply", never for a specific model. Changing the model behind a task is a configuration change, tested by running the suite, not a code change.
The routing layer also handles fallback when a provider is degraded, enforces per-tenant cost budgets, logs tokens and latency per call, and lets you run two models side by side on live traffic to compare. This is what "model-agnostic" means in practice, and it is described further in multi-model routing.
A worked example
A B2B SaaS company building an in-app copilot started with a single frontier model for everything and a monthly bill that alarmed the CFO. We split the workload into five tasks: intent classification, record lookup formatting, reply drafting, thread summarisation and a rare multi-step planning case. We benchmarked four candidates per task on their own data.
The classifier moved to a small open-weight model served in their cloud, at negligible cost and equal accuracy. Lookup formatting went to a mid-tier hosted model. Drafting and summarisation split between two providers based on rubric scores. The planning step stayed on the frontier model, because it ran rarely and accuracy mattered most. The bill fell substantially, accuracy per task held or rose, and every choice is documented in the routing table with its benchmark run. The in-app copilot case study describes the product.
Team and timeline
A model benchmark for an existing system takes one AI engineer about two weeks: one to assemble the per-task suite if none exists, one to run candidates and write up the routing table. For a new build it is part of the LLM application scope, starting at $21,000 / ₹13.6L, and of any SaaS copilot. If the question is strategic, which providers to standardise on across several products, a ten-day Sprint Zero at $3,250 / ₹2,00,000 produces the benchmark and the recommendation, credited to the build that follows. Ongoing re-benchmarking on each major release sits in a Care Plan. All prices are on the pricing page.
Before you start: a checklist
- List the distinct tasks your system performs, with volume and latency budget for each
- Build or reuse an eval suite of real inputs per task
- Shortlist candidates across at least two providers plus one open-weight option
- Record accuracy, latency and tokens per case; project cost to monthly volume
- Decide per task by reading the table against the task's requirements, and write the reason down
- Put a routing layer between application code and providers from day one
- Confirm data-handling terms with each provider against your compliance needs
- Schedule a re-run of the benchmark on every major model release
Questions clients ask
- Should we standardise on one provider for simplicity? No. Two or three in the routing table costs little and removes a single point of failure.
- Do open-weight models match hosted ones? On narrow tasks, often yes; on frontier reasoning, usually not. The benchmark tells you which is which.
- How often should we re-benchmark? On every major release from a provider you use, and quarterly regardless.
- Is the cheapest model good enough? Sometimes. Run it through the suite; if the slice scores hold, use it.
- Can we switch providers later? Yes, if application code never names a model. That is what the routing layer is for.
Related reading
Model and vendor selection: a benchmark-first approach, LLM inference costs: forecasting your monthly bill and what makes an LLM application production-ready extend the points here. Provider pricing and model documentation are on OpenAI's platform docs and Anthropic's docs.
Choose per task, on your data, with a table you can rerun; the name on the invoice matters far less than the number in the benchmark.
Frequently asked questions
Which LLM is best for business use in 2026?
▾
None wins every task. Benchmark candidates on your own inputs for accuracy, latency and cost per task, then route each task to the best fit. Most production systems end up using two or three providers.
Is OpenAI or Anthropic better?
▾
Both are capable across most workloads and the difference on yours is measured, not assumed. Run both against your eval suite, keep both in the routing table, and switch when the numbers change.
When should we use an open-weight model?
▾
When data cannot leave your environment, when volume makes API pricing expensive for a narrow task, or when fine-tuning on your data pays off. You take on serving and upgrades in return.