Private AI for banks: keeping customer data inside the perimeter

Private AI banking means the model runs where the data already is. An open-weight model is served on GPUs the bank owns or rents inside its own cloud account, retrieval runs against an index the bank built, and nothing about a customer leaves the network to be processed by a third party. SSO decides who can ask, role-based access decides what they can see, and an audit log records every request. That is the whole idea, and it is less exotic than it sounds.

This article explains what a private AI deployment for a bank actually contains, where the real costs and risks sit, how it compares with calling a hosted API, and what a first project looks like. It is written for CIOs, CISOs and heads of digital at banks and NBFCs who have been told "we cannot use AI because of data" and suspect that is only half true.

Why private AI matters for a bank

A bank's constraint is not model quality. Hosted models are excellent. The constraint is that customer records, KYC documents, transaction histories and internal credit notes are regulated data, and sending them to an external inference endpoint creates outsourcing, localisation and audit questions that most risk teams will not sign off quickly. A private deployment removes the egress question entirely and turns the remaining questions into ordinary IT controls the bank already knows how to run.

There is a second, quieter reason. A bank that owns its model, its prompts and its retrieval index can change vendors, upgrade models and rerun evaluations on its own schedule. Nothing is locked to a single provider's roadmap or pricing. Our stance on this is described on the fintech industry page: the client owns the code, the models and the infrastructure, always.

Hosted API vs private deployment: what changes

What a private AI stack for a bank contains

The model layer

Open-weight models in the 8B to 70B range, served with vLLM or a similar inference server on one or more GPUs. A smaller model handles classification, extraction and routing; a larger one handles reasoning and drafting. We keep the routing model-agnostic so a model swap is a configuration change followed by an evaluation run, not a rebuild. Sizing guidance is in GPU sizing for private AI.

Retrieval inside the perimeter

Policies, product documents, circulars, procedures and, where permitted, customer records are chunked, embedded and indexed in a vector store that runs next to the model. Embeddings are produced by a local embedding model, not an external API, because embeddings of customer text are still customer data. Retrieval is permission-aware: the index carries the same access labels as the source systems, so a relationship manager cannot retrieve what they could not open directly.

Identity, access and logging

Every request arrives with an SSO identity. The gateway checks role, applies data scope, records the prompt, the retrieved documents, the model version and the response, and stores that record in the bank's own log platform with retention set by the compliance team. This is the artefact an internal auditor or the regulator will ask for, and it is far easier to produce from one owned log than from a mix of provider dashboards. We cover what auditors look for in AI audit trails.

Guardrails and policy gates

Read-only use cases (summarise, search, draft) are gated by access rights. Any use case that writes to a system, such as raising a service request or updating a customer record, goes through a policy gate: the agent proposes, a rule set checks, and either the action executes within limits or a person approves. No model output touches a core system directly.

Where the risk actually sits

Teams worry about the model. The real risks are elsewhere. First, the retrieval index becomes a shadow copy of sensitive data and must be governed like one: encryption at rest, access labels, a refresh pipeline and a deletion path. Second, logs contain prompts, and prompts contain customer data, so log retention and access need the same treatment as the source records. Third, GPU capacity is a fixed commitment; under-provisioning produces slow answers that staff stop using, and over-provisioning wastes money. Fourth, someone has to own model upgrades and evaluations after go-live, or the deployment quietly freezes at its launch version.

Choosing the first use case

The best first use case for private AI in a bank is internal, read-only and high-volume: a knowledge assistant over circulars, product manuals and process documents for branch and contact-centre staff; a summariser for complaint files; or extraction from KYC and loan documents into a review queue. These prove the stack, the access model and the audit trail without exposing a customer to model output. Customer-facing agents come later, on the same foundation, with policy gates in front of every action.

Avoid starting with anything that requires a credit decision or a regulatory filing to be produced by a model. Those are places where the model can assist a person with evidence, but the decision and the record must remain human and traceable.

Evaluation before and after go-live

A private deployment is evaluated exactly as a hosted one: a golden set of real questions with reviewed answers, run against every model version and prompt change. For a bank we add three checks. Groundedness: does every factual sentence trace to a retrieved document? Leakage: does any answer include data outside the requester's scope? Refusal: does the model decline questions it should not answer, such as requests for another customer's details? These run in shadow mode for a few weeks before anyone relies on the output, following the approach in shadow mode for AI agents.

A worked example

A mid-sized lender wanted to speed up KYC document review without sending scanned identity documents to an external service. The build ran an open-weight vision-language model and an extraction pipeline inside the lender's own cloud account, with the vector index and logs in the same VPC. Reviewers saw extracted fields alongside the source image, with confidence flags, and corrected exceptions in a queue. Every extraction was logged against the reviewer's identity and the document version. The audit team got a single report they could reconcile against the case system. The qualitative outcome was that review moved from re-keying to checking, exceptions became visible instead of buried, and the security review passed because nothing left the perimeter. The KYC document intelligence case study describes the shape of the work.

Team and timeline

A private AI deployment is delivered under our private and self-hosted agentic AI service, which starts at $31,500 / ₹20.8L plus infrastructure. The bank supplies a cloud account or GPU host, an identity provider, access to the document sources and a security contact. We supply an AI engineer, a platform engineer and a lead who owns the evaluation suite and the security documentation. A typical first deployment runs eight to twelve weeks: two weeks of environment and identity set-up, four to six weeks of build against a golden set, and a shadow period before go-live.

Where the use case is not yet chosen, a Sprint Zero discovery sprint at $3,250 / ₹2,00,000 over ten working days produces a ranked list, an architecture and a fixed price, and is credited to the build. Ongoing model upgrades, evaluation reruns and monitoring are covered by a Care Plan; the Enterprise tier at $5,250 a month includes 24×7 cover and a named engineer. Full figures are on the pricing page.

Before you start: a checklist

• Confirm which data classes the first use case touches and who owns each
• Decide the hosting boundary: on-prem, private cloud account, or hybrid
• Name the identity provider and the roles that map to data scope
• Agree log retention, access and deletion rules with compliance before build
• Reserve GPU capacity and a fallback for peak periods
• Assemble a golden set of real questions with reviewed answers
• Assign an owner for model upgrades and evaluations after go-live
• Involve the outsourcing and information security teams in week one, not at sign-off

Glossary

• Open-weight model: a model whose weights are published and can be run on your own hardware
• Data egress: any movement of data outside the controlled environment
• Permission-aware retrieval: search that returns only documents the requester is allowed to see
• Policy gate: a rule layer that checks a proposed action before it executes
• Golden set: reviewed question-and-answer pairs used to evaluate the system
• Shadow mode: running the system on live traffic without acting on its output

Related reading

See self-hosted LLMs for BFSI: a practical checklist, zero data egress: designing AI that never leaves your VPC and the fintech industry page. The RBI's guidelines on outsourcing of IT services are the primary reference for what the regulator expects of third-party arrangements, and the vLLM documentation describes the inference server we use most often.

Keep the model where the data is, log everything against a real identity, and the question changes from "can we use AI" to "which use case first".