The model comes to the data

Why training on personal data worries regulators

Training does not store records the way a database does. It adjusts a model's parameters until they capture the statistical patterns in the training data. Models can also memorise, however, and researchers have extracted names, telephone numbers and email addresses verbatim from a language model simply by querying it. Distinctive and repeated records are the most exposed, and a medical file is full of distinctive records.

Anonymisation is the usual safeguard, but a record stripped of names can often be re-identified by linkage against other datasets, and synthetic data generated to stand in for real records has been shown to offer no better trade-off between privacy and usefulness than traditional anonymisation. Training one model on another's output carries the problem forward, since whatever the first model memorised can reappear in what it generates. None of it can be withdrawn: once a record has shaped a model's parameters, there is no general way to remove it short of training the model again without it.

Each organisation that wants to train also acquires its own copy of the data. This is the duplication problem described in the health data case study, multiplied by every model.

What is the training for?

General-purpose language models such as ChatGPT, Gemini or Claude have no business being trained on personal data. Smaller-scope models, such as a credit score or an epidemiological forecast, are in effect algorithms whose parameters are fitted to personal data.

Personal data can reach a model in three ways: embedded during pre-training, introduced during fine-tuning, or via retrieval while the model is used. Which of these is appropriate depends on the purpose, and there are only a few purposes for which anyone trains a model on personal data.

General capability, the command of language and of the world that large models acquire in pre-training, does not depend on any private individual's records. Personal data reaches web-scale corpora only incidentally, collected along with everything else, and in a Databanking regime it cannot lawfully be held outside a Databank, in a model's parameters or anywhere else. Existing models would have to be cleaned of it or retrained without it, and models built to the Databanking standard exclude it by law.

Population knowledge is the purpose behind most medical and social research models: which patients respond to a treatment, which combination of factors predicts a disease, where demand for transport or energy will fall, whether a policy achieved what it set out to do. The value lies in the pattern, to which any one person's record contributes almost nothing. Each account is also a lifelong record, so a model can follow people over time, through the progression of a disease or the outcome of a treatment years later. Today that requires linking records held by different institutions, and linkage is what makes re-identification easy.

Decisions about a person combine the two. A credit or fraud model is trained on a population and then applied to one applicant, and a recommender is trained on many people's choices and then applied to one account. Under Databanking that application is a query like any other: the model runs against the applicant's account inside the sandbox and returns the decision, which is the bit-scarce result. The score behind it is personal information, and if it is kept at all, it is kept in the Owner's account rather than in the lender's files.

Reaching people is the purpose behind lookalike audiences and targeted advertising, and equally behind job vacancies and invitations to clinical trials. Under Databanking the selection runs inside the Databank over the accounts of Owners who opted in, the offer is placed in each matching Owner's account, and the Requestor learns only how many matched, as the job recruitment case study shows. Who the matches are stays inside the boundary until an Owner responds.

Service to one person, such as an AI assistant that knows its user's appointments, prescriptions and travel plans, needs that person's current facts, and needs them to be correct. These are better retrieved than trained in. Where a model is adapted to one Owner, the adaptation is itself personal information about that Owner, and belongs in that Owner's account.

Training inside the boundary

For population knowledge, the Requestor submits the training run as a registered computation. It runs across the accounts of Owners who have opted in to that class of use, in the same way that the health data case study makes research participation a choice made per study rather than a blanket waiver.

The Requestor never holds the training data, and nor does anyone else outside the boundary. The Databank executes the run as it executes any approved computation, and analyses neither the data nor the model. Records are never released, so what must be shown safe is the trained model rather than an anonymised dataset.

In a market with several Databanks, the Owners who opted in are spread across them, so a training run spans custodians, much as federated learning already spans hospitals without pooling their records. How such a run is best composed across custodians is part of the where computation runs research strand.

The model is the result

What a training run produces is a model, and a model can carry personal information. The whitepaper defines personal information to include any data, however derived, that relates to an identifiable individual, and a model that has memorised a record is such data. A trained model is therefore governed as a result.

It can stay in the Databank, held in the Requestor's own account and registered like any other algorithm, from where it can be applied to Owners' accounts as a query but not withdrawn. Its answers are then results like any other, as small as the use allows and counted against the same leakage budget.

It can also leave, once it is shown not to carry identifiable information. Differential privacy in training gives that demonstration a precise form: it bounds how much any one Owner's records can change the trained model, which makes it the training-time counterpart of the leakage budget that governs ordinary queries. The European Data Protection Board has taken a similar line, that a model trained on personal data is anonymous only if the likelihood of extracting that data from it, or of obtaining it through queries, is insignificant, and that this has to be assessed case by case. The standard a model must meet to leave belongs to the technical architecture, within the custody boundary research strand.

A synthetic dataset generated by a model still inside the boundary is simply a very large result, far beyond any leakage budget, and a model distilled from its outputs inherits what it learned. Either is the model leaving by another route, and is held to the same test.

Ask, rather than remember

Models are increasingly built to look facts up rather than store them. Retrieval can allow a smaller model to match much larger ones, and keeps answers current: a fact retrieved at the moment of use is as fresh as its source, while a fact stored in a model's parameters is as old as its training. What remains in the parameters is general knowledge of language and the world, which changes slowly.

Personal data is the clearest case of knowledge that goes out of date: addresses, medication, balances, employers. Under Databanking, an assistant that needs a fact about its user asks the user's Databank, as any Requestor does. The query is registered and audited, the answer is as small as the task allows and is held only as long as the task needs it, and the Owner receives a share of the fee. Nothing about the Owner has to be in the model's weights at all.

Whatever the model concludes about its user from the answers is itself personal information, and if it is kept, it is kept in the Owner's account.

When the Owner revokes a permission, the next question is refused, and the record is left out of every later training run. A model already trained keeps what it learned, as a published study keeps the contribution of a participant who later withdraws.

Small enough to train in a Databank

Pre-training on public data is where the very large computing budgets go. Training on personal data adds specialisation, which is increasingly cheap. An adapter trained alongside a frozen model can have thousands of times fewer trainable parameters than the model itself. Mixture-of-experts designs allow a single expert to be trained on closed data, independently of the rest, and included or excluded each time the model is used. And the trend is towards smaller models for the same capability.

Databanks therefore do not need frontier-scale computing to host training, which keeps the custodial industry open to more than the few firms that own the largest data centres.

Who pays

The Requestor pays for the training run, as for any other computation, and a share of the fee goes to the Owners whose records were used. Each Owner's audit statement names the run. Retrieval is paid per query in the ordinary way. Data that today is collected once, without payment, and used indefinitely becomes a priced and recorded input to every model that uses it.

← Back to the case studies