Context
The client is a non-banking financial company lending to small businesses across three Indian states. Every loan application arrives as a bundle of documents: identity proofs, address proofs, six to twelve months of bank statements from a dozen different banks, GST returns and a handful of business documents that vary by borrower. An onboarding team read every page, keyed the fields into the loan origination system, cross-checked them against each other and flagged anything that looked inconsistent. The team was good at it. It was also the slowest step in the funnel, the largest source of data-entry errors, and impossible to scale for the festival-season surge without hiring temporary staff who made more mistakes.
The lender had already trialled two vendor OCR products. Both performed well on clean PDFs and badly on the photographed, skewed, low-light documents that most borrowers actually upload from their phones. More importantly, both required uploading customer documents to the vendor's cloud, which the compliance team would not sign off on under their data residency policy.
The problem to solve
Three constraints shaped the brief. The system had to read messy real-world documents, not scanned templates. It had to run inside the lender's own infrastructure, with no customer data leaving their VPC. And it had to produce output the onboarding team could trust, which meant confidence scores, a clear exception queue and a full audit trail, not a black box that was right most of the time.
The commercial question was equally clear: could straight-through processing be raised enough that the same onboarding team could handle peak volume without temporary hires, while reducing the keying errors that were causing downstream rework in credit assessment?
Approach
We started with a three-week ProofRun rather than a proposal. The client gave us an anonymised sample of two hundred real applications, deliberately including the ugliest ones. We benchmarked three open-weight vision-language models and one commercial API against the same sample, scored on field-level accuracy for the twenty-two fields that mattered, and reported the results in a two-page memo with the failures listed by document type.
The memo said two things the client needed to hear. First, an open-weight model running on a single GPU node in their VPC could meet the accuracy threshold on identity and address documents and on statements from the eight largest banks. Second, statements from smaller cooperative banks and hand-annotated GST returns would not reach the threshold without a human in the loop, so the design had to route them to an exception queue rather than pretend.
- Layout-aware parsing with Docling for PDFs and a vision-language model for photographed pages, chosen per page by a lightweight classifier.
- A validation layer that cross-checks extracted fields against each other: name and date of birth across documents, account numbers across statement pages, totals against transaction sums.
- Confidence scoring per field, with thresholds tuned on the golden set, so the queue only receives what a human genuinely needs to look at.
- Retrieval over the lender's own policy documents so the validation rules could be maintained by the compliance team in plain language rather than in code.
- Every model call logged with inputs, outputs, model version and the reviewing user, written to an append-only store the audit team can query.
What we built
The production system runs as a set of services in the lender's AWS account. Documents land in a private bucket, are classified and parsed, and the extracted fields are written to a staging table alongside their confidence scores. A rules engine, driven by the compliance team's policy documents, runs the cross-checks and either releases the application to the loan origination system or places it in the exception queue with the specific fields and pages that need attention highlighted.
The onboarding team works from a review screen we built in React, which shows the original page beside the extracted values, lets them correct a field with one click, and records every correction. Those corrections feed a weekly retraining and re-evaluation loop, so the golden set grows with real edge cases rather than staying frozen at go-live.
The whole stack, including model serving with vLLM, the vector store and the review application, is defined in infrastructure-as-code. The client's platform team can rebuild it from scratch, which was one of their explicit conditions.
Delivery
After the ProofRun, the build ran for fourteen weeks in two-week sprints. The first six weeks produced the pipeline and review screen in shadow mode: the system extracted every application, but the team continued keying manually and we compared the two. That period surfaced a category of failure nobody had predicted, a particular bank's statement format that flipped debit and credit columns depending on the month, and gave the team confidence in the confidence scores before anything relied on them.
Weeks seven to twelve moved to assisted mode, where the team reviewed pre-filled forms rather than blank ones, and the exception thresholds were tuned against real throughput. The last two weeks were hardening: load testing against the previous year's peak-week volume, a security review by the client's information security partner, and handover documentation and training for the platform and onboarding teams.
Outcomes
The lender now processes the majority of applications without a person reading the documents end to end; the onboarding team spends its time on the exception queue and on the judgement calls that the system routes to it. Keying errors reaching credit assessment fell sharply once the cross-checks were live, because the most common mistakes, transposed digits and mismatched names, are exactly what the validation layer catches. The festival-season peak after go-live was handled by the existing team.
Just as important to the client, no customer document has left their infrastructure, and the audit team has a complete, queryable record of every automated decision, which they had never had for the manual process.
What we measured
- Field-level accuracy against the golden set, reported per document type
- Straight-through processing rate
- Exception queue volume and time-to-clear
- Keying errors reaching credit assessment
- Model latency and cost per application
- Data egress: zero, verified
What we learned
Telling the client on day fifteen which documents the system could not read reliably was the most valuable thing we did. It turned the exception queue from an admission of failure into a designed feature, and it meant the onboarding team trusted the system on the documents it did handle. The second lesson was that the review screen mattered as much as the model: a good correction interface is what makes the golden set grow and the system improve after launch.
The engagement continues under a Standard Care Plan with the AI add-on, covering model updates, re-evaluation and the quarterly review of thresholds.
"The first thing they did was tell us which of our documents the model would struggle with. That honesty set the tone for the whole engagement, and the pipeline has been quietly doing the job since."
Client details are anonymised. Figures are described qualitatively and are available under NDA on a call.