Data leakage

Data leakage takes two main forms. Target leakage is when a feature encodes the outcome: a churn model that uses "account closure reason" or a demand model that uses "units shipped" as a feature will look perfect and be useless, because those values do not exist until after the thing being predicted has happened. Train-test contamination is when the evaluation data overlaps with the training data, for example by random splitting of time-series rows or by duplicating customers across sets, so the model is graded on questions it has already seen.

The defence is discipline about time. Every feature must be computable from data available at the moment the prediction would be made, and evaluation must use a split that mirrors deployment: train on the past, test on the future. Pipelines that rebuild features as of a snapshot date, rather than from current tables, make this enforceable.

It is not a rare beginner's mistake; it is the most common reason a machine learning model that scored well in a notebook disappoints in production. Suspiciously high accuracy is the usual symptom, and it should be investigated before it is celebrated.

• CTO / Head of Engineering: it is the first thing to check when a vendor or team reports near-perfect accuracy; ask how the split was done and when each feature is known.
• Data lead: snapshot-based feature construction and time-based splits are non-negotiable engineering standards, not optional rigour.
• CFO: a leaky model passes its acceptance test and then fails in production, after you have paid; insisting on an out-of-time evaluation protects the budget.
• Product manager: leakage often comes from data that is back-filled by operations after the fact; know which fields those are.

The concept of data leakage exists because it is easy to build a model that cheats without anyone intending it to. Databases store the final state of records, not what was known when, and a model trained on final states learns from the future. The result is a validation score that promises far more than production delivers, months of lost time and eroded trust in ML. Preventing it costs effort: reconstructing point-in-time views of data and evaluating strictly out of time. That effort is the difference between an accuracy figure you can plan around and one you cannot.

• Building a churn model for a SaaS product from account snapshots taken ninety days before each label, excluding any field updated at cancellation.
• Training a credit-risk model for an NBFC only on application-time data, not on collection notes added after default.
• Backtesting a retail demand forecast with a rolling time split so each week is predicted from prior weeks only.
• Evaluating a student dropout model on a later academic year than it was trained on.
• Auditing a fraud model for features such as "chargeback filed" that only exist after fraud is confirmed.

ConceptA concept every ML practitioner must recognise, enforced through pipeline design. Eazyware's AI and ML development work uses point-in-time feature construction and out-of-time evaluation as standard, and reports accuracy only on that basis.

Eazyware service that covers it: AI/ML Development. Starting prices are on the pricing page.