Data readiness for AI: the checklist to run before you fund a project

Why AI projects fail on data, not on models

An AI proposal almost always arrives framed as a modelling question. Which algorithm, which vendor, which foundation model. That framing is comfortable because it is the part of the work with the most interesting reading material, and the part least likely to determine whether the project works.

Google’s Machine Learning Crash Course is blunt: “Data trumps all. The quality and size of the dataset matters much more than which shiny algorithm you use to build your model.” The same course estimates practitioners spend around 80% of a project’s time constructing datasets and doing feature engineering. If four-fifths of the effort sits before the model, four-fifths of the risk does too, and a proposal spending one paragraph on data has mispriced itself.

The longer-run version appeared in the 2015 NeurIPS paper Hidden Technical Debt in Machine Learning Systems, which catalogued the maintenance costs accumulating around a working model: entanglement between features, hidden feedback loops, undeclared data dependencies, eroded boundaries between the model and everything feeding it. None are modelling problems. All are why a demo that worked in March is quietly switched off in November.

So the readiness question is not “can a model do this” — for most business problems it probably can. It is whether your organisation can supply that model with correct, explainable, lawfully held data on a repeating schedule, and whether anyone will notice when it stops.

Volume, coverage and label quality: the three hard limits

These three constrain what is possible. Tooling, budget and enthusiasm do not move them.

How much data you actually need

The question is asked as an absolute and only has a relative answer. Google’s crash course publishes the rule of thumb: train on at least an order of magnitude — preferably two — more examples than the model has trainable parameters, and expect models trained on large datasets with few features to outperform models trained on small datasets with many.

Read that second clause carefully, because it inverts the instinct most teams have. Faced with a thin dataset, the temptation is to add columns. Adding columns to a small dataset usually makes it worse. The productive move is more rows and fewer, better-chosen features.

One genuine exception. If you are adapting a pre-trained foundation model rather than training from scratch, the volume needed for a first useful result collapses. What does not collapse is the data needed to evaluate it: you still need a labelled set representing the decisions the system will make, or you have no way of knowing whether it is right, and no way of noticing when a model update changes its behaviour.

Coverage matters more than row count

Total rows is a vanity measure. What limits a model is how many examples it has seen of the outcome you care about, and that outcome is usually rare.

Google’s documentation describes fraudulent card transactions as often making up less than 0.1% of examples, and diagnoses of a rare virus less than 0.01%. At those proportions a randomly drawn training batch may contain no positive examples at all, which is not enough to learn from. The standard remedy is downsampling the majority class and upweighting it to compensate — in their worked example, a 99%/1% split downsampled by a factor of 25 to reach 80%/20%.

So before anyone talks about volume, count the rare class. Fifty thousand orders with eleven confirmed fraud cases is an eleven-example dataset in a costume.

Labels are the part nobody budgets for

A supervised model learns from labels, and label quality sets the ceiling. Two failure modes are common.

The first is the proxy label. Google’s documentation distinguishes a direct label — “identical to the prediction your model is trying to make” — from a proxy, “similar but not identical”: someone subscribing to a bicycle magazine probably but not definitely owns a bicycle. Businesses do this without noticing, predicting “customer will churn” from “cancelled the direct debit”. Models using proxy labels “are only as useful as the connection between the proxy label and the prediction”, and that connection is rarely measured.

The second is rater noise. Where humans produced the labels, the guidance is unambiguous: raters make mistakes, several may need to evaluate the same item, and you should “always double-check your human raters” — including by labelling a sample yourself and comparing. If two experienced colleagues disagree on a third of historic cases, no model resolves that. It learns it.

Finding your data: systems, exports and the shadow spreadsheets

The inventory is unglamorous and takes a couple of days. Do it before scoping anything.

For every system touching the process you want to model — CRM, ERP, billing, support desk, warehouse management, product telemetry, website analytics — record six things: who owns it internally, how data gets out (API, scheduled export, replica, someone clicking Download), the retention period, the grain of each table, the primary key, and whether it holds personally identifiable information.

Two things reliably surface. The first is that the join keys do not line up: the customer identifier in the CRM is not the account number in billing, and the reconciliation lives in one person’s head. That mapping is a deliverable in its own right, and it is where a data analytics engagement usually starts, alongside a data dictionary recording what each field means and who owns the definition.

The second is the shadow spreadsheet. Somewhere in operations sits a tracker holding the only record of the outcome you want to predict — which quotes converted, which installations needed a second visit, which tickets were genuinely resolved. It is nobody’s system of record and it is your only labelled data. Find it early, and treat replacing it with a captured, validated field as part of the project rather than a later tidy-up. Where the capture point is a form or an internal tool, that is a small web application build rather than a data task, and cheaper than modelling around a spreadsheet forever.

Website and marketing data deserves particular caution, because consent state, tag firing order and modelling shape what lands in your analytics tables before you ever query them. The mechanics of consent mode and server-side tagging are worth understanding before you treat behavioural data as ground truth.

Data quality tests you can run this week

You do not need a platform to start. These run as plain SQL against BigQuery, Snowflake, Redshift or Postgres.

1. Uniqueness on every key. Duplicated primary keys silently multiply rows through joins and inflate every downstream count.
2. Null rates per column, tracked over time. A field that was almost never blank last year and is routinely blank since a system upgrade tells you something changed. A one-off null check does not.
3. Accepted values on every categorical field. Status fields accumulate variants: “Complete”, “complete”, “COMPLETED”, “Done”.
4. Referential integrity across systems. Orders whose customer no longer exists; tickets against deleted accounts.
5. Range and distribution checks. Negative quantities, timestamps in the future, prices that moved by a factor of a hundred the day someone changed currency units.
6. Freshness. How old is the newest row, and does that match the schedule you believe exists? Pipelines fail silently more often than loudly.
7. Duplicate entity detection. Not duplicate rows — duplicate customers. One organisation entered four ways fragments every history you build.
8. Definition agreement. Ask two people in different departments to produce “active customers last month” independently. If the numbers differ, that is the finding.

The first four map onto the generic tests dbt ships with out of the box — unique, not_null, accepted_values and relationships — which is a reasonable argument for adopting it once you are running these regularly rather than once.

Automated tests catch mechanical faults, not interpretive ones. Google’s guide to data quality traps covers the rest: the data-reality gap between what a dataset shows and what happened, survivorship bias from only observing cases that made it through, self-selection bias in anything voluntary, and regression to the mean masquerading as the effect of an intervention. Ask who collected each dataset, under what conditions, and what it measures rather than what it is named.

Lineage, ownership and access control

Lineage is, in Google Cloud’s phrasing, “a visual map that tracks the entire lifecycle of your data” — where it came from, where it travelled, what transformed it on the way. It earns its cost four ways: verifying a report is built from what you think it is, tracing root cause when a number goes wrong, seeing downstream dependencies before you change a field, and showing where sensitive data has flowed when someone asks.

Ownership is the cheaper half and gets skipped more often. Every dataset feeding a decision needs a named human accountable for its definition and told when it breaks. “The data team” is not an owner. Neither is a shared inbox.

Access control tends to sit at an extreme. Either every warehouse login can read everything, which makes any conversation about personal data awkward, or access is so restricted that analysts work from stale exports on laptops, which is worse. The workable middle is role-based access on schemas, personal data isolated into tables most roles cannot see, and a documented route to request more.

None of this needs a governance committee. It needs a register — dataset, owner, definition, refresh schedule, sensitivity, downstream consumers — kept current. The NIST AI Risk Management Framework is a useful external scaffold: govern, map, measure and manage, explicitly voluntary rather than a compliance regime.

Privacy and lawful basis before any modelling

This is not legal advice, and specifics depend on your jurisdiction. But there is a sequence that keeps projects out of trouble, and it starts before the data is copied anywhere.

Establish the purpose, separately. Data collected to fulfil orders was collected for that purpose. Training a predictive model on it is a new processing activity, not a continuation of the old one, and it needs its own basis and its own record. Assuming continuity is the most common shortcut.

Assess the risk formally where required. Under the GDPR, Article 35 requires a data protection impact assessment prior to processing likely to result in a high risk to individuals, and names systematic and extensive evaluation based on automated processing among the cases where one is expected. That fits many scoring and profiling models.

Read what regulators have said about AI specifically. The European Data Protection Board’s Opinion 28/2024 covers when a model itself can be considered anonymous, when legitimate interest is available as a basis, and what follows when a model was trained on unlawfully processed data. That last point has teeth: the lawfulness of the training set does not stop mattering once training is finished.

Know whether your use case counts as high-risk. The EU AI Act sorts systems into tiers, and high-risk systems carry obligations covering the quality of the datasets feeding them — specifically to minimise discriminatory outcomes — alongside documentation, traceability and human oversight. Employment screening and access to essential services are in scope, which catches more ordinary applications than teams expect.

The engineering response is to minimise: pseudonymise on the way into the warehouse, keep personally identifiable information out of feature tables where a surrogate key does the job, and isolate the identity join in a controlled table few roles can read.

Building the smallest useful pipeline

The pipeline that gets an AI project into production is smaller than most architecture diagrams suggest, and drawing the diagram first is a reliable way to spend a quarter without producing a prediction.

The modern default is ELT rather than classical ETL: land raw data in the warehouse unmodified, then transform it there in version-controlled SQL. Raw data is preserved, so a transformation bug is fixed by re-running rather than by re-extracting from a source system that may no longer hold the history. Transformations and their tests then live in one repository on one schedule, which stops quality checks drifting away from the logic they protect.

Resist the feature store until you have a reason. It solves specific problems — one feature definition shared by several models, low-latency online serving, and training-serving skew where the value computed at training time differs from the one computed at prediction time. With one model and one batch job, it is infrastructure with no job.

Two of the Rules of Machine Learning apply directly. Rule 4: keep the first model simple and get the infrastructure right, because its value is in proving the plumbing rather than in sophistication. Rule 5: test the infrastructure independently from the machine learning, so a bad prediction can be blamed on the model rather than on a join that silently dropped a third of the rows.

Then monitor the data, not just the model. Google’s guidance is concrete: write a schema the data must satisfy and alert on anomalies, unexpected categorical values and distribution changes; watch for skew between training and serving; compare metrics on slices against the whole; track time since retraining against a threshold. Instrumentation on the systems generating that data belongs in the same budget, which often makes the team building your applications part of the AI project whether or not anyone put them on the plan.

Baselines: the boring model your AI has to beat

Before the model, build the thing the model has to outperform. This is the discipline that most reliably prevents a wasted year.

Rule 1 of the Rules of Machine Learning is “don’t be afraid to launch a product without machine learning”, on the grounds that basic heuristics deliver much of the available value. Rule 2 is to design and implement metrics first, tracking as much as possible in the current system before formalising what an ML system will do. Rule 7 is to turn existing heuristics into features rather than discarding them, since the rules your operations team already applies encode real domain knowledge.

A baseline is deliberately unimpressive. Predict the majority class. Predict last month’s value. Apply the three-line rule the sales team already uses to decide which leads to call first. Then measure it on held-out data, using a metric that reflects the business decision rather than raw accuracy — because with a 1% positive rate, a model predicting “no” every time is 99% accurate and worth nothing.

Two further questions belong here, both commercial rather than technical. What does a false positive cost, and what does a false negative cost? Those numbers set the decision threshold, and until someone answers them the model has no defensible operating point. If your candidate beats the heuristic by a margin that will not survive the cost of maintaining, retraining and explaining a pipeline, ship the heuristic. Choosing which problems deserve a model at all is the subject of where AI actually pays for itself.

Scoring your readiness across five levels

Place yourself honestly. The value is entirely in refusing to skip a level.

Level	What it looks like	Realistically achievable	Next step
0. Ad hoc	Data sits in operational systems and spreadsheets. Reports are rebuilt by hand and rarely agree.	Off-the-shelf tools used by individuals. Nothing depending on your own data.	Inventory the systems; agree five core definitions.
1. Reported	Reporting is roughly trusted, but manual, undocumented and owned by one person.	A one-off analysis on an export that will not survive that person leaving.	Automate the extract; put definitions in version control.
2. Warehoused	A warehouse holds modelled tables. Transformations are code. Basic tests run on a schedule.	A first model trained on a stable dataset, scored in batch.	Add freshness and distribution tests; document each grain.
3. Governed	Lineage visible, owners named, access role-based, personal data isolated, lawful basis recorded.	Models feeding real decisions, with an audit trail when someone asks how one was made.	Set monitoring thresholds and who answers an alert.
4. Operational	Pipelines monitored for skew and freshness. Baselines tracked. Retraining scheduled and reversible.	Several models in production, maintained rather than rebuilt.	Retire those no longer beating their baseline.

What to do at each level

At levels 0 and 1, do not start an AI project. Start a definitions project and remove the single points of failure in reporting: a fraction of the cost, and a prerequisite regardless of what you build later.

At level 2 you can genuinely run a first model, and the discipline that matters is scope — one prediction, one clearly-owned decision, a measured baseline, and a written definition of success agreed before anyone builds.

By level 3 the constraint stops being data and becomes organisational: whether the people receiving a prediction will act on it, and what happens when they disagree. That is change management, and it is where applied machine learning work either lands or quietly does not. At level 4 the work is mostly pruning, and how rarely models that no longer beat their baseline get switched off says more about how organisations treat AI than any capability gap does.

When the honest answer is ‘fix reporting first’

One diagnostic question is worth asking before any of this. Can two people in your organisation independently produce the same number for a basic measure — active customers, gross margin by product, tickets resolved first time?

If not, the AI project will not fix it. It will inherit it, and encode it into a system whose outputs nobody can argue with because they came out of a model. The disagreement does not disappear. It becomes harder to see.

The unpopular recommendation is to spend the first tranche of budget on definitions, instrumentation and a warehouse everyone trusts, and postpone the model. It is unpopular because it is less interesting, and because whoever approved the budget was promised AI. It also leaves something durable: a business that can answer questions about itself is a precondition for every model you might build later, and useful on its own if you never build one.

A reasonable test of anyone proposing an AI project — including us — is whether they asked to see your data before describing the solution. If the proposal arrived before the inventory, it was written about a generic company rather than yours.