azyware
Business

Questions to ask a machine learning development services vendor before you sign

EZ
Eazyware
· 7 min read
Quick answer

What should you ask a machine learning development services vendor?

Ask how success will be measured before any code is written, who owns the model and the training data, what the system costs to run monthly, what happens when accuracy drifts, and how you exit. Vendors who have shipped answer in numbers; vendors who have not answer in adjectives.

Ask a machine learning development services vendor how success will be measured before a line of code is written, who owns the trained model and the training data, what the system will cost to run each month, what happens when accuracy degrades in production, and how you exit. Teams that have shipped production models answer these in numbers.

What follows is the question bank we would use if we were on your side of the table: what each question is really testing, what a strong answer sounds like, and the sentence in a proposal that should make you slow down. It is written for the person who has to sign the statement of work and defend it later.

Why most vendor questionnaires miss

Standard software due diligence asks about team size, methodology, references and security certifications. All of that matters, and none of it predicts whether a machine learning project will work, because machine learning fails in places ordinary software does not. It fails because the training data was labelled by three people who disagreed. It fails because the model was tuned on a random split when the real problem is a time split. It fails because nobody agreed what a false positive costs the business.

Machine learning development services due diligence therefore has to probe evidence and operations rather than process. The useful questions are about measurement, data provenance, running cost and what happens six months after launch. We have written separately about the five ways these projects fail, and almost every one is visible in the answers to a pre-signature conversation if you know what you are listening for.

One more framing point. You are not buying a model. You are buying a decision system that someone in your organisation has to keep running, so judge the vendor on what they leave behind, not on the demo.

Five questions that separate shipped from slideware

Put these five in writing and ask for written answers. The written form matters; it is much harder to be vague in a document that will be attached to a contract.

QuestionWhat a strong answer sounds likeWarning sign
How will we know this model is good enough to ship?A named metric, a threshold, a held-out test set built from your data, and a time-based split"We will fine-tune until accuracy is high"
Who owns the model weights, the training data and the feature pipeline?You do, in writing, including derived artefacts and documentationOwnership "of the deliverable" with the model hosted on the vendor's account
What will this cost to run per month at our volume?A figure with the arithmetic shown: inference, storage, retraining, monitoring"Infrastructure costs are minimal"
What happens when performance drops?Drift monitoring, an alert threshold, a named owner, a retraining trigger and a rollback path"We will retrain periodically"
How do we leave?A handover pack, a runbook, source in your repository, two weeks of paired transitionNo exit clause at all

Ask the fifth question early rather than last. A vendor's answer to "how do we leave" tells you how they think about the relationship, and it is the question most likely to be met with silence in a first call. If the answer is a promise rather than a list of artefacts, treat the engagement as a rental, not a build, and price it accordingly.

Notice that none of these asks about frameworks. Whether the vendor uses PyTorch, XGBoost or scikit-learn tells you very little; whether they can state a decision threshold and the cost of being wrong tells you a great deal.

What should you ask about data before you sign?

Ask about data first, because data decides the ceiling of the project and a vendor who has shipped will interrogate yours before quoting. If a proposal arrives with a fixed price and no data assessment, the price is a guess. These are the questions worth putting in the room.

  • Have you looked at our actual data, or only at our description of it? A serious vendor asks for a sample under NDA before quoting a build.
  • How will labels be produced, and who adjudicates disagreement? Label quality sets the ceiling on model quality, and inter-annotator disagreement is the single most common hidden cost.
  • What is the time split you will evaluate on? Random splits flatter models on any problem where behaviour changes over time, which is most business problems.
  • What is the base rate, and what is the cost of a false positive versus a false negative? A fraud model at 99% accuracy is useless if fraud is 0.5% of transactions.
  • Where will our data sit during training, and in which region? Ask about data residency in the same breath as the model, not after the contract.
  • What happens to our data if we stop working with you? Deletion certificates, not assurances.
  • Which parts of the pipeline will still need a human? Honest vendors name them early.

A vendor who welcomes these questions has been burned by them before, which is exactly the experience you are paying for.

What should a machine learning contract cost and contain?

A scoped machine learning build at Eazyware starts at $17,500 or ₹11.2 lakh and runs to $70,000 or ₹46.4 lakh depending on the number of models, the state of the data and the integration surface. Details of what sits in each band are on the AI/ML development service page and every starting figure is published on the pricing page. If nobody has proved the hardest part yet, a three-week ProofRun at $6,250 or ₹4,00,000 answers the feasibility question before you commit to a build, and a ten-day Sprint Zero at $3,250 or ₹2,00,000 is credited to the work that follows.

The ownership clause

Write it so that it names artefacts: source code, trained weights, training and evaluation datasets, feature definitions, prompts where applicable, infrastructure-as-code and documentation. "Client owns the deliverables" is not enough, because a vendor can hand you an inference endpoint and call it the deliverable. Our position is that you own the code, the infrastructure, the model choices and the documentation from day one.

The degradation clause

Models decay because the world moves. Your contract should say what metric is monitored, at what cadence, what threshold triggers investigation, and who pays for retraining. Care Plans handle this after launch: Essential at $1,000 or ₹68,000 a month, Standard at $2,500 or ₹1,60,000 and Enterprise at $5,250 or ₹3,40,000, with a $750 or ₹40,000 AI add-on that covers evals, cost monitoring and re-indexing. The equivalent scope is described on the maintenance and support page.

The exit clause

Agree the handover artefacts at signature, not at separation: repository access, environment reproduction steps, a model card, the eval suite, and a fixed number of paired transition days. The hidden costs that quotes leave out are mostly exit costs in disguise.

When this level of diligence is the wrong move

There is such a thing as too much procurement. If the engagement is a two-week feasibility test on a public dataset with no personal data and a four-figure budget, running a fifty-question security review and a three-round legal negotiation will cost more than the work. Buy the small thing, watch how the team behaves, and save the heavy diligence for the build that follows.

Diligence is also the wrong tool when the real problem is that you have not decided what you want. No vendor questionnaire will tell you whether churn prediction is worth doing. That is a strategy question, and asking twelve suppliers to answer it produces twelve different proposals that cannot be compared. Decide the use case first, then evaluate vendors against it. A short paid discovery with one team beats a long unpaid comparison across six.

What good answers look like in practice

An NBFC came to us with a document-heavy onboarding process and a clear question: could an ML system read KYC documents accurately enough to cut manual review without raising rejection risk. The pre-build conversation was about the false-negative cost, the labelling standard, and where the documents would sit. The resulting system is described in the KYC document intelligence case study, and the part worth copying is the sequence: agreed metric, agreed data handling, then build.

That sequence is also how you compare two vendors fairly. Give both the same measurable goal and the same data description, and the proposals become comparable documents rather than competing brochures.

A pre-signature checklist

  • Written success metric with a threshold and the test set it is measured on
  • A named time-based evaluation split, agreed before work starts
  • Data residency, retention and deletion terms in the contract body
  • Monthly running cost estimate with the arithmetic attached
  • Drift monitoring, alert thresholds and a named owner after launch
  • Ownership of weights, data, pipelines and documentation stated by artefact
  • A rollback path and a defined incident escalation route
  • Handover pack and paired transition days priced in the original scope

What to put in a machine learning development services RFP covers the document you send before these conversations, a security questionnaire for AI vendors gives you the controls list, and evals over demos explains why we refuse to be judged on a demonstration. The NIST AI Risk Management Framework sets out the govern, map, measure and manage functions that a serious vendor's answers should map onto.

The best question is still the simplest one: show me a system you built that is still running, and tell me who maintains it.

Frequently asked questions

What should you ask a machine learning development services vendor?

▾

Ask how success will be measured and on what held-out data, who owns the model weights and training data, what monthly running cost the system carries at your volume, what triggers retraining when accuracy drifts, and what the exit and handover look like. Written answers to those five questions predict delivery better than any credentials list.

How do you tell if an ML vendor has actually shipped models before?

▾

They ask about your base rate, your label quality and the cost of a false positive before quoting. They propose a time-based evaluation split rather than a random one. They volunteer the parts of the pipeline that will still need a human, and they price monitoring and retraining into the original scope rather than treating both as later work.

What should a machine learning development contract include?

▾

Ownership stated artefact by artefact, covering source, weights, datasets, feature definitions and documentation. A measurable acceptance threshold. Data residency, retention and deletion terms. A drift-monitoring and retraining clause with a named owner. An exit clause with a handover pack and paired transition days priced in at signature rather than negotiated at separation.