Tags: experimentation commerce concept
Testing and Compliance
Date: 2026-08-17
The legal floor under experimentation in the UK: what you may vary, what you must keep true, and where a test that’s statistically clean is still not allowed to ship. It sits below the ethical question rather than answering it — legal and defensible are different standards, and this note only covers the first.
Testing and compliance covers the legal limits on what an experiment may vary — price, claims, consent, accessibility — regardless of what the statistics say.
This is the shape of the constraints, not legal advice. Regimes are named where the mechanism depends on them and marked [CHECK] where current detail needs confirming — UK consumer law was restructured recently and the secondary detail moves.
The four regimes that bite
| Regime | Who enforces | What it constrains |
|---|---|---|
| Consumer protection — misleading actions and omissions | CMA (Competition and Markets Authority) | Price presentation, availability claims, urgency, “was” prices |
| Advertising codes — CAP and BCAP | ASA (Advertising Standards Authority) | Claims in marketing, including on your own site |
| Data protection — UK GDPR and PECR | ICO (Information Commissioner’s Office) | Assignment records, tracking, consent for non-essential cookies |
| Accessibility — Equality Act duties | Individuals, via claims | Variants that exclude disabled users |
[CHECK: the consumer protection framework was restructured by recent legislation which moved enforcement powers and altered the regime for misleading practices and pricing. Confirm which instrument currently applies, and what the enforcement position is, before relying on any specific provision.]
Where tests most often go wrong
1. Price testing between comparable customers. Showing different prices to different users for the same product at the same moment is where commercial instinct meets consumer protection hardest.
The distinctions that matter:
generally acceptable problematic
different prices over TIME different prices to different
(everyone sees today's price) people at the same moment for
the same product
different prices by SEGMENT prices varied by inferred ability
that's transparent and available or willingness to pay
to all — student, trade, volume
"was £80, now £40" where £80 was
promotional structure tests never a genuine selling price
(bundle framing, threshold ← this is the specific one that
presentation) gets enforced against
Reference pricing is the enforcement hotspot. Any “was” price, RRP comparison or “save £X” claim must reflect a genuine prior selling price for a meaningful period — testing the framing of a discount is fine; testing a fictional anchor is not — Price Anchoring, Discounting Strategy, Price Testing.
2. Urgency and scarcity that isn’t true. “Only 3 left” must mean three are left. “Offer ends midnight” must end at midnight. Testing whether a fabricated countdown converts better is testing the profitability of a misleading practice, and the result is unusable regardless of what it says — Scarcity and Urgency.
Testing the presentation of accurate scarcity — where it appears, how it’s worded — is entirely legitimate and is usually where the gain was anyway.
3. Consent mechanics as a test surface. Cookie banners and consent flows are governed by PECR — the Privacy and Electronic Communications Regulations — alongside UK GDPR. Optimising a consent banner for acceptance rate collides directly with the requirement that consent be freely given, specific and informed, and that refusing be as easy as accepting.
A consent banner test that improves opt-in by making refusal harder has produced a compliance problem and a metrics improvement simultaneously, and the second will be cited as justification for the first — Consent Management, UK GDPR and PECR for Analytics.
4. Accessibility regressions. A variant that removes focus indicators, drops semantic structure or fails contrast excludes disabled users. It won’t show in the primary metric — the affected population is small and often doesn’t convert either way — so it must be caught before launch rather than measured — WCAG, Colour Contrast, Focus Management, Accessibility Law in the UK.
5. Regulated claims. Health, financial promotions and environmental claims carry their own regimes on top of general advertising rules. Testing copy variants in these areas needs the claims themselves cleared before the test, not after it wins.
The practical controls
- A pre-launch checklist item for compliance, not a separate process. Three questions on the QA sheet: does this vary price, does it make a claim, does it touch consent or accessibility — Experiment QA
- Route the flagged ones to a named person. Legal, compliance, or whoever holds it. The point is a defined path, not a committee
- Accessibility checks on both arms, automated where possible. Automated tooling catches a minority of issues but catches the regressions cheaply
- Record the decision in the test’s record. “Reviewed by X on date, approved” — this is what makes a challenge answerable a year later — Experiment Archive
- Keep assignment data in your ROPA — the record of processing activities — with a retention period, like any other personal data
- Don’t test your way around a rule. If a change would need approval to ship, it needs approval to test; a test is the change, applied to a subset
Regulated sectors change the picture
In financial services, health, gambling and utilities, sector regulators impose duties that go well beyond the general regimes — including obligations around fair treatment of customers and demonstrating good outcomes, which can require evidence about how experiences were designed and chosen. Experimentation is compatible with all of that and sometimes helps demonstrate it; what it can’t be is undocumented. [CHECK: the specific duties and evidential expectations in your sector, with whoever owns regulatory relationships.]
Where it interacts
- Ethics of Experimentation — the standard above this one; legal is the floor, not the target
- Deceptive Design — the catalogue of patterns that attract enforcement, and the reason several of them now do
- Price Testing — the commerce-side treatment, where the commercial mechanics live
- Consent Management — the mechanics of the consent layer that constrains both testing and measurement