#506

A/B Testing

Run experiments: sticky variant assignment, exposures, conversions and a significance test.

Technique: Hash bucketing (FNV-1a), two-proportion z-test, confidence intervals, sample size

Read the code: src/main/java/com/study/advlab/abtest/AbTestService.javasrc/main/java/com/study/advlab/abtest/Stats.java

Experiments of Northwind Electronics. Each visitor always sees the same variant.

Home page hero image RUNNING

A photo of people using the product beats a product-only photo.

A Product photo 12.0% 35/292

B Lifestyle photo 9.2% 29/316 -23.4%

C Short video 11.6% 34/293 -3.2%

Results Live demo page hero-image

Pricing page headline STOPPED

Showing the monthly price first makes the plans look cheaper.

A Yearly price first 3.7% 94/2528

B Monthly price first winner 5.1% 126/2472 +37.1%

Results Live demo page pricing-headline

Checkout button wording RUNNING

A clear promise (free delivery) on the button makes more people start checkout.

A Control: Buy now 6.5% 80/1222

B Free delivery promise 8.4% 99/1179 +28.3%

Results Live demo page checkout-button