Model-agnostic by design: why we route across providers
What should you know about model agnostic AI, and why does Eazyware route across providers rather than pick one?
Routing across OpenAI, Anthropic, Google and open-weight models controls cost and risk when any provider changes. Every system we build calls models through one routing layer, with per-task model choices set by evaluation results, so a price change, a deprecation or an outage is a configuration change, not a rewrite.
Model agnostic AI is a design decision we make on every build, not an option we offer. No application code we write calls a model provider directly. It calls a routing layer that knows which model handles which task, what the fallback is, and what each call costs, and that layer speaks to OpenAI, Anthropic, Google and open-weight models running on the client's own hardware. The reason is not ideological. Providers change prices, retire models, have outages and leapfrog each other in quality every few months, and a system wired to one of them inherits every one of those changes. This article explains how the layer is built, how model choices are made, and what it costs to keep the option open.
Why one provider is a risk, not a simplification
Choosing a single provider feels like reducing complexity. In practice it concentrates four risks. Price risk: a per-token change on your busiest task changes your margin overnight. Deprecation risk: providers retire models on their own schedule, and a retired model means a re-evaluation you did not plan for. Availability risk: an outage at your only provider is an outage of your product. Quality risk: the best model for extraction this quarter may not be the best next quarter, and if switching is a rewrite you will not switch. We wrote about the deprecation case specifically in handling model deprecations; the routing layer is the general answer.
What the routing layer does
| Concern | How the layer handles it | Who configures it |
|---|---|---|
| Per-task model choice | Each task (classify, extract, draft, summarise, converse) maps to a primary model and a fallback | Set from evaluation results per slice, reviewed when models change |
| Provider differences | One internal interface for messages, tools, structured output and streaming; adapters per provider | Built once; adapters updated as provider APIs change |
| Fallback and retries | On error, rate limit or timeout, the call moves to the fallback with the same prompt and schema | Thresholds set per task by the engineer |
| Cost and latency tracking | Every call records tokens, cost, latency, model and task | Reported per task and per tenant in observability |
| Prompt portability | Prompts are written against the internal interface with provider-specific tweaks isolated | Engineers, with the eval suite as the check |
| Data residency | Tasks tagged as sensitive route only to approved providers or to self-hosted models | Set by the client's policy, enforced in configuration |
| Caching | Identical or near-identical requests served from cache where the task allows it | Per task, with a freshness rule |
How we choose a model for a task
Evaluation decides, not preference
Every task in a system has a slice in the evaluation suite. When the routing configuration is set, candidate models are run against that slice and compared on accuracy, cost per case and latency. The winner is the primary, the runner-up is the fallback, and the report is kept so the decision can be revisited. Engineers have preferences, and they are not the input. Our approach to comparing candidates is in model and vendor selection: a benchmark-first approach.
Not every task needs the strongest model
A typical system has a handful of tasks with very different demands. Classification and routing of incoming requests are usually served well by a small, fast, cheap model. Drafting a customer-facing reply or reasoning across several documents may need a frontier model. Extraction with a strict schema often sits in between. Routing each task to the cheapest model that passes its threshold is the single largest lever on running cost, and it is only available if the architecture allows it. The mechanics and the trade-offs are in multi-model routing: cutting LLM costs without cutting quality.
Open-weight models are part of the mix
For tasks where data may not leave the client's infrastructure, for high-volume tasks where per-token pricing dominates, and for anyone who wants a fallback that no provider can withdraw, open-weight models on the client's own GPUs are a first-class route. They are evaluated on the same slices as hosted models, and they win some of them. Our private agentic AI service is the self-hosted end of the same architecture.
Prompts that survive a model change
The hard part of multi-provider AI is not the API adapter; it is that prompts tuned for one model degrade on another. We manage this three ways. Prompts are written against our internal interface with structured outputs and tool schemas rather than provider-specific formatting tricks. Provider-specific adjustments live in a small, isolated layer that is easy to see and easy to delete. And every candidate model is run through the full suite before it becomes a primary, so a prompt that only works on one model is caught before it matters. When a provider changes a model version underneath a fixed name, the suite is rerun the same day.
The model is a dependency, and we treat it like one: pinned, tested, and replaceable.
What model agnostic AI costs
There is an honest cost. The routing layer is a component to build and maintain, adapters need updating when provider APIs change, and holding a fallback means keeping a second model's evaluation current. Some provider-specific features arrive first on one platform and are unavailable through a common interface until adapters catch up. We think the cost is modest against the alternative, which is a rewrite the first time a provider makes a decision that does not suit you, and the running-cost savings from per-task routing usually pay for the layer within the first quarter of production use. Where a client genuinely wants a single provider, we still build behind the layer; it costs little and keeps the door open.
A worked example
A D2C brand runs a personalisation engine and a WhatsApp agent that between them make several distinct kinds of model call: classifying incoming messages, drafting replies in the brand's voice in more than one language, and generating product recommendations from a customer's history. During the build, the cheapest model that passed the classification slice was a small hosted one; drafting went to a frontier model; recommendations were largely conventional ranking with a model used only to write the explanation. Some months after launch, a provider changed pricing on the drafting model. The routing configuration was changed to the fallback, which had been kept evaluated, and the suite confirmed no slice regressed. Marketing did not notice, which was the point. The system is outlined in the personalisation and WhatsApp case study.
Team and timeline
The routing layer is part of every LLM application build, from $21,000 or ₹13.6L, and every agent build; it is not a separate line item. In a Launch 6 program it is in place by the end of week one, using our pre-existing component licensed into the deliverables with source, and the per-task model choices are set during the evaluation work in weeks two to five. After handover, re-evaluating candidates when a provider ships a new model is a routine task under a Care Plan, typically a few hours a quarter from the included allowance. Program and Care Plan prices are on the pricing page.
Before you start: a checklist
- List every distinct model task in the system, not just the visible one
- Decide which tasks carry data that may not leave your infrastructure or region
- Confirm the application code calls a routing layer and never a provider directly
- Require a named fallback model per task, evaluated on the same slice as the primary
- Set up cost and latency reporting per task from the first week
- Ask which prompts have provider-specific adjustments and where they live
- Plan a quarterly re-evaluation of candidates as part of support
- Check contracts with providers for training exclusions and data handling
Questions clients ask
- We already have a contract with one provider; does this still apply? Yes. Build behind the layer, use your contracted provider as primary, and keep a fallback evaluated. The commitment is honoured and the exit stays cheap.
- Does routing add latency? A negligible amount for the lookup; the fallback path is slower only when it is used. Per-task routing usually reduces average latency because small tasks go to fast models.
- Who updates the adapters when a provider changes its API? We do under a Care Plan, or your team does from the source we hand over; the adapters are small and documented.
Glossary
- Routing layer: the component that maps each task to a model and provider, with fallback, cost tracking and policy
- Adapter: the code that translates the internal interface to one provider's API
- Primary and fallback: the model a task uses by default and the one it moves to on error or by configuration
- Open-weight model: a model whose weights are published and can be run on your own infrastructure
- Model pinning: fixing a task to a specific model version so an upstream change cannot alter behaviour unnoticed
- Slice: the subset of the evaluation suite that scores one task
Related reading
Our full set of engineering principles is on the about page; the case for self-hosting is in self-hosted LLMs: when running your own model beats an API. For provider-side detail on model lifecycles, the deprecation pages in OpenAI's platform documentation and Anthropic's documentation are the primary sources to watch.
Pick models by measurement, keep the choice in configuration, and no provider's decision can become your emergency.
Frequently asked questions
Does model agnostic AI mean using several providers at once?
▾
Usually, yes, because different tasks are best served by different models. But the architecture matters more than the count: even a system that uses one provider today should call it through a layer that makes switching a configuration change.
How do you avoid vendor lock-in with AI providers?
▾
Route every call through one internal interface, keep prompts free of provider-specific tricks, keep a fallback model evaluated per task, store prompts and evaluation data in your own repository, and keep an open-weight option viable for sensitive or high-volume tasks.
Is a routing layer overkill for a small product?
▾
The layer itself is small and we bring it as a licensed component, so the cost is low. The moment it earns its keep is the first price change, deprecation or outage, and every product with users reaches that moment.