Adding AI to an existing product without a rewrite
How do you add AI to an existing product without rewriting it?
Add AI as a thin service beside your backend that calls your APIs with the user's token; no data model changes required. The AI layer owns prompts, retrieval, routing and evals; your product keeps owning data, permissions and business rules. Here is the architecture, what it touches, what it leaves alone, and the cost.
To add AI to an existing product you do not need a new data model, a new backend or a migration. You need a thin AI service that sits beside what you already run, calls your existing APIs with the signed-in user's token, and returns answers and proposed actions to a few new components in your front end. Nothing in your database changes. This is how every SaaS copilot we build attaches to a product that already has customers, and it is the architecture this article describes.
We cover why the rewrite instinct is wrong, the AI layer's responsibilities, the integration points, what to do about older or thinner APIs, and how the build fits into a team that also has to keep shipping.
Why teams assume they must rewrite to add AI to an existing product
The assumption usually comes from a vendor pitch or a conference talk: AI-native products are built differently, so your product must be rebuilt. It is not true for the overwhelming majority of products. AI features consume your data through the same interfaces your UI uses, and they change data through the same interfaces too. If your product has an API, even an internal one, it already has everything the AI layer needs.
The rewrite instinct is expensive in two ways. It delays the AI feature by the length of the rewrite, and it couples the AI work to the highest-risk project a company can run. Our modernisation versus rewrite comparison sets out the general case; for AI the answer is even clearer, because the AI layer is small and separable.
The AI layer: what it owns and what it does not
| Concern | Owned by the AI layer | Owned by your existing product |
|---|---|---|
| Data storage | None; a small store for conversations, traces and eval data | All business data, unchanged |
| Permissions | None; passes the user's token through | Existing role checks on every API |
| Business rules | None; relies on API validation | Validation and rules in the API |
| Prompts and models | Prompt versions, routing across providers, fallbacks | Nothing |
| Retrieval | Indexes built from your data via API or read replica, with permission filters | Source of truth |
| Actions | Action registry mapping to your endpoints; preview and policy gates | The endpoints themselves |
| Evaluation and tracing | Eval suites, traces, cost metering per tenant | Nothing |
| Front end | A few new components: copilot panel, buttons, preview, citations | The rest of the UI |
The split is deliberate. Everything probabilistic and fast-changing lives in the AI layer, where it can be versioned, evaluated and swapped without touching the product. Everything deterministic and trusted stays where it is. When the AI layer is wrong, the blast radius is an answer or a proposed action, never a corrupted record.
Integration points, in order of effort
Read: calling your APIs for context
The copilot needs to know what the user is looking at and the related records. The AI service fetches them through your API with the user's token, exactly as the front end would. If the API is chatty, a small aggregation endpoint that returns "everything about this ticket" makes the copilot faster and cheaper; that endpoint is a normal feature, not an AI feature.
Retrieve: indexing documents and history
For search and question-answering over documents, the AI layer builds an index from your data. Feed it through the API, an event stream or a read replica; never let it write back. Every indexed chunk carries the source's access control, and retrieval filters by the user's entitlements before ranking. The design is in permission-aware retrieval.
Act: proposing changes through your endpoints
Actions map one-to-one onto existing write endpoints. The copilot proposes, the user confirms, the AI service calls the endpoint with the user's token, and your API's permission checks and validation run as they always have. The model gets no authority the user lacks; the full model is in copilot actions through your existing API.
Show: the front-end components
A copilot panel or inline buttons, a streaming answer component, a citation component that opens the source record, and a preview-and-confirm component for writes. These are added to your existing front end, whether it is React, Angular, Vue or server-rendered, and they call the AI service over a streaming endpoint. Design guidance is in designing copilot UX.
When the API is thin, old or internal-only
Many products have a front end that talks to a backend through endpoints that were never meant for anyone else. That is fine. The AI service is another internal client. Where an endpoint is missing, add it with the same permission checks; where an endpoint returns HTML rather than JSON, add a JSON variant. This work is ordinary API development, and it usually improves the product for its own front end too.
Where there is no API at all, only a monolith with server-rendered pages and direct database access, the AI service can start from a read replica for context and retrieval, with actions deferred until endpoints exist. That is a phased approach rather than a rewrite, and the ReCore programme is designed for it. The university ERP modernisation followed exactly this path: an AI-ready API layer added in front of a system that was not rewritten.
Multi-tenancy, data handling and the questions security will ask
Your product is probably multi-tenant, and the AI layer must be too. Tenant identity travels with every request; indexes are partitioned or filtered by tenant; traces and cost metering are per tenant; prompts never contain another tenant's data. The multi-tenant LLM architecture article covers the design.
Security teams will ask which data leaves your environment and to which providers. The AI layer is where that answer is enforced: a routing layer that sends each task to an approved provider, with an option to route sensitive tenants or tasks to a self-hosted model. Provider data-handling terms from OpenAI and Anthropic should be read before the first call is made, and the answer written up for the security page your enterprise buyers will ask for.
Keeping the AI layer honest: evals and tracing from day one
Because the AI layer is separate, it can carry its own test discipline without slowing your product's release cycle. An eval suite built from real inputs runs on every prompt or model change; traces record every call with tokens, latency and cost per tenant; the routing table lets you swap a provider without touching product code. This is the practice from evals: the practice that separates demos from products, and it is what lets the layer evolve weekly while the product ships on its own schedule.
A worked example
A B2B SaaS company for field-service teams had a mature product, a busy roadmap and a board asking for AI. The engineering lead's first plan was a new services architecture that would take most of a year. Instead, we built a thin AI service beside the existing backend. It read tickets, customers and visits through the existing API with the dispatcher's token, indexed job notes and manuals with permission filters, and exposed two write actions through existing endpoints with preview and confirm.
The product's own codebase changed in four places: a small aggregation endpoint, two front-end components, a JSON variant of one endpoint, and a feature flag. No table changed. The copilot shipped while the roadmap continued, and the model behind each task has been swapped twice since without a product release. The in-app copilot case study describes the outcome.
Team and timeline
Adding an AI layer to an existing product with a reasonable API takes an AI engineer, a backend engineer familiar with your codebase and a front-end engineer roughly six to eight weeks for the first three copilot jobs, including evals, tracing and tenant-aware metering. That is the SaaS copilot scope from $19,500 / ₹12.8L, or a Launch 6 where a fixed six-week date matters. Where API work is substantial, budget for API development and integrations from $7,000 / ₹4.4L. After launch, a Care Plan keeps prompts, models and indexes maintained. Prices are on the pricing page.
Before you start: a checklist
- Inventory the APIs the copilot will need for context, retrieval and actions
- Confirm every write endpoint enforces permission and validation server-side
- Decide how the AI layer receives tenant identity and the user's token
- Choose the retrieval feed: API, events or read replica, never write access
- List missing or HTML-only endpoints and schedule them as ordinary API work
- Pick the routing policy per task and the providers security will approve
- Set up tracing and per-tenant cost metering before the first user
- Put the copilot behind a feature flag with a beta cohort
Glossary
- AI layer: a separate service owning prompts, retrieval, routing, actions and evals
- Aggregation endpoint: one API call that returns all context for a record
- Read replica: a copy of the database used for indexing without touching production writes
- Action registry: the explicit list of operations the copilot may perform, each mapped to an endpoint
- Routing layer: configuration mapping each AI task to a provider, model and fallback
- Feature flag: a switch that exposes the copilot to a chosen cohort
Related reading
Why AI copilots inside SaaS beat standalone chatbots, the ten copilot jobs users actually ask for and build vs buy vs integrate are the natural companions. Martin Fowler's writing on the strangler fig pattern is the primary reference for adding capability beside a system rather than replacing it.
Your product already has the data, the permissions and the rules; the AI layer borrows all three through the API and adds nothing you would have to migrate away from later.
Frequently asked questions
Do we need to change our database to add AI features?
▾
No. The AI layer reads and writes through your existing APIs with the user's token and keeps only its own conversations, traces and eval data. Your data model stays as it is.
What if our product has no proper API?
▾
Add the endpoints the copilot needs with the same permission checks, or start from a read replica for context and retrieval and add actions as endpoints appear. Both are phased work, not a rewrite.
How long does it take to add a copilot to an existing SaaS product?
▾
Six to eight weeks for the first three copilot jobs when a reasonable API exists, including evals, tracing and tenant-aware metering. Substantial API gaps add time but not a rewrite.