AI Personalization Tests: How We Test and Score Tools
In short: Every tool on this site is judged on the same five criteria, using the same test protocol, and every score links back to the evidence behind it. If a test wasn’t run, we say so instead of guessing.
Vendor demos show personalization at its best. This page shows how we check what happens on real data, with real constraints. Before reading a result, read the protocol below, because a score is only as trustworthy as the method behind it. Our full approach is on the Methodology page.
The five criteria we score
| Criterion | What we test | Evidence we keep |
|---|---|---|
| Relevance | Do recommendations match user intent, including for new users with no history? | Side-by-side outputs for the same user profiles |
| Measured lift | Does personalization beat a non-personalized control? | A/B test design, sample size, confidence interval |
| Privacy & consent | What data is collected, and can users opt out or delete it? | Data-flow notes, consent settings, vendor documentation |
| Speed & reliability | How fast does it respond, and what happens when it fails? | Latency measurements, fallback behavior |
| Fairness & diversity | Does it trap users in a narrow filter bubble or skew toward certain groups? | Diversity of results across test profiles |
The weights are yours to set. We suggest starting with Relevance 30%, Measured lift 25%, Privacy 20%, Speed 15%, Fairness 10%, and changing them if your business has different priorities (a regulated industry might put Privacy first).
Our test protocol, step by step
- Define the goal. One primary metric per test (click-through, conversion, or time to find an item), chosen before we look at any data.
- Build test profiles. [Number] synthetic profiles covering new visitors, returning users and edge cases. No real personal data is used.
- Run a control. Same content, same traffic, no personalization.
- Run the variant. Same conditions, personalization on.
- Check significance. We report the size of the difference and the uncertainty around it, not just “it won.”
- Repeat and record. Tools update often, so each result carries the test date and the version tested.
Test results
Only publish rows you have actually run.
| Tool | Version | Test date | Relevance | Lift | Privacy | Speed | Fairness | Total |
|---|---|---|---|---|---|---|---|---|
| [Tool name] | [v.] | [date] | [/10] | [% ± CI] | [/10] | [ms] | [/10] | [/100] |
What our tests cannot tell you
Synthetic profiles are not your customers. A tool that wins on our protocol can still underperform on your catalog, your traffic and your data quality. Treat our scores as a shortlist filter, then run a small pilot on your own stack before committing. Where a vendor sponsored access or a free license, we disclose it next to the result.
How to run this test yourself
- Start with one page or one email, not the whole site.
- Keep a control group of at least [your traffic-based number] users.
- Let the test run through a full business cycle, typically at least one to two weeks, so weekday and weekend behavior both show up.
- Decide in advance what result would make you stop.
FAQ
How often do you retest? [Your cadence, e.g. when a tool ships a major update or every six months.]
Do vendors see results before publication? [Your policy. Say plainly whether they can correct factual errors only, or influence scores.]
Can I suggest a tool to test? Yes, use the Contact page and tell us the use case you care about.
Are scores comparable across categories? Only within the same category. Scores for a recommendation engine and an email tool measure different things.
Le prochain pas utile serait de m’envoyer les tests réellement faits (outils, dates, chiffres). Je peux alors remplir le tableau et adapter le titre à vos vrais résultats. Je peux aussi tout convertir en fragment HTML WordPress avec JSON-LD (Article + FAQPage) et des liens internes vers les pages About, Contact et Methodology, selon votre workflow habituel.