A/B Testing 101: How to Run Experiments That Actually Tell You Something

A/B testing is simple in theory - show half your users version A, half version B, keep the winner - and surprisingly easy to get wrong in practice. Most failed experiments don't fail because the change didn't work; they fail because the measurement was broken. Here's how to run tests you can trust.

Pick one primary metric before you start#

Decide up front which single number decides the test: activation rate, checkout conversion, upgrade clicks. Not five numbers - one. If you measure twenty metrics, one of them will look like a winner by pure chance, and you'll talk yourself into shipping noise. Secondary metrics are fine as guardrails (make sure the new checkout doesn't raise error rates), but the decision rides on one metric.

Randomize by user, not by request#

Assignment has to be per user and sticky: a given person sees the same variant every time, for the whole test. Request-level randomization puts the same user in both variants, which contaminates everything downstream - you can no longer attribute their behavior to either experience. The standard approach is hashing the user id into a bucket, then logging an exposure event (user id, variant, timestamp) the first time each user enters the experiment.

Don't peek, and run full weeks#

The two classic traps:

• Peeking. Checking results daily and stopping when the number looks good inflates false positives dramatically. Decide the sample size or duration in advance and hold to it.
• Short tests. Weekday and weekend users behave differently. A test that runs Tuesday to Thursday measures Tuesday-to-Thursday users. Run at least one full week - two is better - and be skeptical of tiny samples: if a 1% conversion change matters to you, you need far more traffic than most early products have. When in doubt, test bigger swings instead of subtle tweaks.

Watch for novelty effects#

Existing users often click a new thing because it's new, not because it's better. If a winning variant's lift fades over two or three weeks, you measured curiosity. For changes aimed at existing users, segment results by new vs returning before declaring victory.

Reading results with funnels#

The cleanest way to see an experiment is as two funnels side by side: same steps, split by variant, conversion compared step by step. A variant that lifts overall conversion but tanks a middle step tells you something a single number can't. GraphJSON's A/B funnel tool does exactly this - pick the steps, split by variant, and compare drop-off visually; the walkthrough is in the A/B funnels guide. If you're new to funnels generally, start with Funnel Analysis 101, and remember that what happens after conversion matters too - retention curves will tell you if your "winning" variant attracts users who stick around.

When not to A/B test#

Low traffic is the honest constraint: with a few hundred weekly signups, reaching significance on small lifts takes months. In that regime, ship opinionated improvements, watch the metrics move week over week, and save formal experiments for changes big enough to move numbers you can actually measure.