01
Pick one question.
Decide what the change should improve, such as reading to the end or opening the next article, before you build it.
Guide · Media
A/B testing a media website, on real readers.
New modules, navigation or games can make reading more rewarding, or just add noise. The safest way to know is to show them to part of your real audience and measure. This page gives general advice first, then a client example we have documented.
General advice: how to run a test.
02
Change one thing at a time.
If a variant changes the layout, the navigation and the games at once, you will not know which part made the difference.
03
Split the audience.
Serve the variant to a share of readers, for example half, while the others keep the current version on the same pages and over the same period.
04
Measure behaviour, not opinions.
Track events for each variant, such as scroll depth, clicks and next article opened, and use heatmaps and recordings to understand why.
05
Fix the rules in advance.
Small differences over a few days are often noise. Set the duration and the decision rule before starting, and do not stop as soon as a number looks good.
06
Keep, change or remove.
Industrialise what works, drop what does not, and feed the next iteration.
A test plan on one page.
- Hypothesis.One sentence: which change, for which readers, should improve what. Fictional example: a “read next” block with three suggestions makes more readers open a second article.
- Main metric.A single measure decides, chosen before the test: here, the share of sessions that open a second article.
- Guardrail metrics.What must not get worse: page speed, ad views, bounce. A winner that breaks a guardrail is not a winner.
- Split.Half of the traffic gives an answer fastest; a smaller share limits the risk of a bold change. Split by reader, not by page view, so each person keeps the same version.
- Planned duration.At least one or two full weeks, to cover weekday and weekend reading, and long enough to reach the sample size estimated beforehand.
- Decision rule.Written before starting: keep the variant if the main metric improves beyond the agreed margin and no guardrail degrades; otherwise drop or change it.
Limits of interpretation.
A result holds for the pages, audience and period tested. A news peak, a campaign or a seasonal effect can shift it, so compare both versions over the same days.
Looking at the numbers every day and stopping at the first good result inflates false positives. A small difference may be noise: report it as such rather than as a win.
Ideas worth testing on a media site.
Case studies.
Media · Prisma MediaFemme ActuelleAn A/B test lab inside the live site: new modules shown to half of the readers, kept only when they worked.Read the case study →
Business aviationJetfly AviationMarketing, sales and charter teams working in one tool, with proposals connected to the CRM.Read the case study →
Media · Prisma MediaGEOMobile app journeys prototyped in Figma and tested with 200+ beta readers before development.Read the case study →What we can build for you.
Noctua builds prototypes and test labs to validate a product with real users before investing: the variants, the connection to your live site and the measurement. You keep the data and the decision.
Questions.
Do we need to rebuild the site to run tests?
Not necessarily. For Femme Actuelle, variants were built as separate Framer pages and served to real readers of the official site without a full release.
Which tools did the Femme Actuelle lab use?
Framer for the variants, PostHog for the 50/50 split and the events, and Clarity for heatmaps and recordings.
Is half of the audience always the right split?
No. Half gives results fastest, but a smaller share limits the risk when a change is bold. The split is a choice to make before the test starts.
Want to test it on real readers?
Describe it in a few lines. Our assistant helps you frame it, then you choose: a 20-minute call, or a reply from the team by email.