
Direct answer: Privacy by design in a personalization system means placing privacy controls at the model, feature and output layers, because that is where re-identification and purpose creep actually happen. A locked-down database does not cover those layers. The workable pattern is to keep the most sensitive signals on-device or session-scoped, attach a declared purpose to every feature, and treat training data, model artifacts and recommendations as personal data until you can demonstrate otherwise.
Most privacy reviews of recommendation and personalization stacks audit the wrong object. They check encryption at rest, access roles, and retention schedules on the event store. Those are necessary, and the system can still leak through them.
A recommender is a function that compresses many users’ behavior into weights, then answers queries about one user’s behavior. Every query is a measurement. The weights memorize some training records. The outputs can reveal what the system knows. None of that shows up in a database audit.
This article places the leaks on a pipeline, maps them to what EU law requires as of November 2026, and ends with decisions you can act on and tests you can run.
Four places a personalization system leaks
A. Signals. Collection is where “just in case” logging turns into liability. Under GDPR, data minimisation and purpose limitation sit in Article 5. A signal with no named use is a compliance defect and an attack surface at the same time.
B. Inferred attributes. A feature store full of derived fields such as “likely parent”, “price-sensitive” or “health-adjacent” is the part of a stack users are least aware of. These inferences are personal data. In some cases, they edge toward special-category data under Article 9. The user never supplied them, so no consent screen ever covered them.
C. The model. Research has shown for years that trained models can reveal whether a specific record was in the training set. Shokri and colleagues demonstrated membership inference against machine-learning-as-a-service models in 2017. Carlini and colleagues later showed training-data extraction from large language models. The risk varies with model size, overfitting, and data duplication. It is rarely zero.
D. The output. This one is the least discussed. Calandrino and co-authors showed in 2011 that the public output of collaborative filtering, such as “customers also bought” lists, can leak individual transactions when an attacker knows part of a target’s history. A recommendation is a disclosure. If your system shows “because you watched X” or lets anyone observe changes in related-item lists over time, you are publishing a signal about specific people.
The dashed arrow in the diagram matters most. Output feeds the next round of signals, so a leak at D compounds into A.
My position: The storage layer is the best-defended part of most stacks and the least likely place for a privacy failure. Put review effort where the defenses are thinnest: inferred features, trained artifacts and visible outputs.
What the law requires today, and what it doesn’t yet
Article 25 GDPR requires data protection by design and by default. The text is old, but the enforcement lens on AI systems is recent. The most useful current document is EDPB Opinion 28/2024, issued 17 December 2024. It covers AI models, and its conclusions bear directly on personalization.
Three points from it matter for builders:
- The EDPB does not accept that every model trained on personal data is anonymous. The threshold is high, and each case is assessed individually. clydeco
- Anonymity requires that both direct extraction of personal data and unintentional disclosure through queries be negligible under reasonable circumstances. That sentence describes leak points C and D almost exactly. legal500
- Legitimate interest can serve as a legal basis for developing and deploying AI, but only after the three-step test of interest, necessity, and balancing. hunton
The regulatory ground has also shifted during 2026, and the two halves of the Digital Omnibus package have moved at very different speeds.
The AI side is settled. The AI Omnibus entered into force as Regulation (EU) 2026/1744 on 27 July 2026. It moves Annex III high-risk obligations from 2 August 2026 to 2 December 2027. A standard product recommender is not an Annex III use, which covers areas such as employment, education access and credit scoring. If your personalization touches those domains, the delay gives you runway. Everywhere else, the AI Act’s timing was never your binding constraint. mondaqorrick
The GDPR side is not settled. The Commission proposed in November 2025 to recognise AI training and operation as a legitimate interest, and to narrow the definition of personal data. On 27 October 2026, the Council took a position that rejects the change to the definition of personal data and drops the explicit AI legitimate-interest amendment, adding a recital instead that says developing AI is not in itself a legitimate interest. One trade outlet reports that early 2027 is considered the earliest realistic date for conclusion. Parliament has not finished its work. digital omnibus eu commission proposes to streamline gdpr and eu ai act +3
My assessment: do not architect around a GDPR loosening that has not happened. The Council’s current text moves toward the EDPB’s case-by-case position, not away from it. A system built to pass the three-step test today will survive either outcome. A system built on the assumption of a presumed legitimate interest will need rework if the final text keeps the Council’s recital.
Source note: the Omnibus status above comes from law-firm and trade summaries, not the Official Journal or the Council’s own documents. Treat the dates as secondary-source figures and check the primary text before relying on them in a compliance memo. Separately, ePrivacy Article 5(3) consent rules for reading or writing device storage continue to apply to client-side signal collection regardless of the GDPR outcome.
Decide where personalization state lives
The most consequential design choice is the location of the state, not the algorithm. Here is a decision matrix I use for scoping. It is editorial analysis, not a benchmark.
| State tier | Where it lives | Privacy strength | What it costs you | Fits |
|---|---|---|---|---|
| On-device | User’s device only | Highest: raw signals never leave | No cross-user learning from that signal; harder to debug; device limits | Sensitive or high-frequency signals, reranking, local history |
| Session-scoped | Server memory, discarded at session end | High: nothing persists to attack or subpoena | Weak cold-start recovery; each session starts nearly blank | Anonymous or logged-out users, sensitive verticals |
| Pseudonymous profile | Server, keyed by ID, retention-limited | Moderate: re-identification risk grows with feature richness | Needs deletion that reaches models, not just tables | Core personalization for consenting users |
| Aggregate/cohort | Server, thresholded groups | Moderate to high if group floors hold | Coarse results; small groups get poor treatment | Trending, segment-level ranking |
Two points on using it.
First, tier by signal, not by product. One product can read purchase history from a pseudonymous profile and rerank on-device using recent browsing. The sensitivity of browsing intent is usually higher than that of purchases, and teams often store them the other way round.
Second, pseudonymous is not anonymous. Narayanan and Shmatikov showed in 2008 that sparse behavioral records can be re-identified by linking a few known data points to a “de-identified” dataset. The Netflix Prize data was the case study. Rich behavioral vectors are exactly that kind of data.
Differential privacy: the budget is where teams get surprised
Differential privacy (DP) bounds how much any single person’s data can change a released result. The bound is a parameter, epsilon (ε). Lower is stricter. DP is the strongest known defense against leak points C and D. It also has a cost that vendor descriptions rarely state: you spend a finite budget every time you release something.
Here is a calculation that shows why this matters. It uses the standard composition results from the DP literature. It is not a test of any specific system.
- Basic composition: releasing k results, each ε-DP, gives total privacy loss of k × ε.
- Advanced composition (Dwork, Rothblum, Vadhan, 2010): total ε′ = ε·√(2k·ln(1/δ′)) + k·ε·(e^ε − 1), with a small added failure probability δ′.
Take δ′ = 10⁻⁶, so ln(1/δ′) ≈ 13.82.
| Scenario | Basic composition | Advanced composition |
|---|---|---|
| k = 200 releases, ε = 0.1 each | 20.0 | ≈ 9.5 (7.43 + 2.10) |
| k = 20 releases, ε = 0.5 each | 10.0 | ≈ 18.2 (11.75 + 6.49) |
Two lessons come out of the table.
- Advanced composition only helps when per-release ε is small. At ε = 0.5, the formula is worse than the naive sum. Pick the tighter of the two bounds. Don’t assume the “advanced” one wins.
- A total ε near 10 is a weak guarantee. The exact meaning depends on context, but you should not describe a system as “private” simply because it uses DP. State the total budget, the unit of protection (event or user), and the time window.
Choosing between event-level and user-level guarantees is where many production claims quietly weaken. Event-level DP protects one click, not a person with fifty clicks. If your marketing copy says “protects users”, check which guarantee you have.
Failure mode and who loses. DP noise has a fixed scale, so signal-to-noise falls as groups shrink. The users with rare tastes, or members of small demographic groups, receive the worst personalization. That is a fairness cost as well as an accuracy cost. Cold-start users are hit twice, because they have little history and the noise is relatively larger. Measure quality by cohort size before launch. An average lift number hides this completely.
Alternative. Where full DP is too costly, the cheaper protections are regularization against overfitting, deduplicating training data, capping per-user contribution, and limiting query access. They provide no formal guarantee. They do reduce the memorization that membership-inference attacks exploit. Label them as hardening, not as proof of privacy.
Purpose limitation is the control that actually bites
Consent banners get the attention. They are the weakest control in the stack, because consent captured at signup says nothing about what a feature does with the data six model versions later.
The control that holds up under audit is a purpose tag on every feature. For each field in the feature store, record the following:
- the declared purpose, for example “on-site ranking” versus “ad targeting”;
- the legal basis for that purpose;
- the retention limit;
- which models consume it.
Then enforce it in the pipeline: training jobs fail if they request a feature whose purpose tag does not include the job’s purpose. This is cheap to build, and it turns a policy document into a build-time check. It also makes the EDPB’s necessity step answerable with evidence. You can show exactly which features each model uses and why.
Deletion is the unsolved problem. A deletion request that clears the profile table but leaves the model trained on that profile is incomplete, and no efficient, general method exists to remove one person’s influence from a trained model. Approaches such as sharded training (SISA, Bourtoule et al., 2019) limit the retraining needed. They cost accuracy or compute. The practical rule: decide a retraining cadence that matches your deletion promise. If you promise deletion within 30 days, models must be retrained or re-sharded on a schedule consistent with that. Otherwise, the promise is aspirational.
An open question: does privacy cost you conversion?
Everyone in this field asks it. I have not found independent, primary-source evidence that gives a general answer, and the vendor figures I have seen are first-party claims with unstated methodology. I am not going to supply a number.
The answer for your system is measurable. Run a holdout test comparing your current pipeline against a variant that moves your most sensitive signals on-device or to session scope. Define the metric in advance (incremental conversion or retention, not click-through), run it long enough to cover your purchase cycle, and segment the results by cohort size. That experiment resolves the question for your traffic. Without it, any claim in either direction is a guess.
What to do first
- Inventory features, not tables. List every derived attribute in the feature store with its purpose, legal basis, and consuming models. Delete the ones that fail to name a purpose. Expect the first pass to remove a meaningful share.
- Attack your own model. Run a membership-inference test against your production ranking model using a held-out set. If the attack’s AUC is meaningfully above 0.5, you have measurable memorization. Fix it with regularization, deduplication, and per-user caps before reaching for DP. (This is a recommended test, not one I have run on any system.)
- Audit outputs for differential visibility. Check whether an observer can infer one person’s activity from changes in public lists or explanations. Remove “because you did X” explanations from any surface that others can see.
- Write your deletion SLA against your retraining cadence. If they conflict, change one of them.
- Move one sensitive signal to the device. Pick the highest-sensitivity, highest-frequency signal and prototype on-device reranking. Compare it with the server baseline using the holdout design above.
- Document the legitimate interest assessment for each processing purpose now, using the three-step structure. It is the one piece of work that pays off under every plausible outcome of the Omnibus negotiations.
Primary sources
Law and regulators
- EDPB Opinion 28/2024 on personal data processing in AI models (PDF)
- General Data Protection Regulation (EU) 2016/679, official text (EUR-Lex)
- AI Act, Regulation (EU) 2024/1689, original text (EUR-Lex)
Research papers
- Shokri et al., “Membership Inference Attacks Against Machine Learning Models” (arXiv)
- Carlini et al., “Extracting Training Data from Large Language Models” (arXiv)
- Narayanan & Shmatikov, “Robust De-anonymization of Large Sparse Datasets” (arXiv)
- Bourtoule et al., “Machine Unlearning” (arXiv)
- Dwork & Roth, “The Algorithmic Foundations of Differential Privacy” (free PDF)



