AB Testing Explained

AB Testing: Data-Driven Design

  • AB testing is a data-driven design method, using data to inform design decisions.
  • Definition (Nielsen Norman Group): Testing two or more design variations with a live audience to determine which performs best based on predetermined business success metrics.

Breaking Down AB Testing

  • Example: Designing a call-to-action (CTA) button for a new subscription service.
  • CTA: A prompt or button that encourages users to take a specific action.
  • Question: Should the CTA button label be "Sign up now" or "Become a member"?
How AB Testing Works
  • Test two or more design variations with a live audience.
  • Implement both button versions (A: "Sign up now," B: "Become a member") live on the website.
  • Split the audience 50/50, with each half seeing one version.
  • Real website visitors are included in the test.
Determining the Winner
  • Define a business success metric (e.g., number of subscription sign-ups).
  • Track the number of sign-ups through each button.
  • The version with more sign-ups is the winner, based on real user data.
  • AB testing is a quantitative method using numerical data and large sample sizes (thousands of visitors).
Tools for AB Testing
  • Tools like Optimizely, Google Optimize, and VWO simplify the process.
  • VWO example: Use a built-in visual editor to make design changes (e.g., button text).
  • Complex design changes may require a software developer to code the different versions.
  • AB testing tools manage data and calculations, identifying the winning design.

Real-World Example: Barack Obama's 2008 Election Campaign

  • Objective: Gather email addresses for volunteering and donations.
  • Experiment: Tested different images and button labels on the website.
  • Images: Obama with supporters, more "regal" Obama, Obama with family, video backgrounds of speeches.
  • Button Labels: "Sign up," "Join us now," "Learn more," "Sign up now."
  • Type of Test: Multivariate test (testing different combinations of elements).
The Surprising Results
  • Initial Assumption: Video backgrounds and the "Join us now" button would perform best.
  • Actual Winner: Image of Obama with his family and the "Learn more" button.
Impact of the Winning Combination
  • Sign-Up Rate:
    • Original: 8% (8 sign-ups per 100 visitors)
    • Winning: 11.5% (40% improvement)
    • Improvement=11.5−88≈0.4375=43.75%\text{Improvement} = \frac{11.5 - 8}{8} \approx 0.4375 = 43.75\%
  • Email Addresses Gathered:
    • Original: 7,000,000
    • Winning: 10,000,000 (3,000,000 address improvement)
  • Volunteers:
    • 10% of email addresses became volunteers.
    • Original: 7,000,000×0.10=700,0007,000,000 \times 0.10 = 700,000 volunteers
    • Winning: 10,000,000×0.10=1,000,00010,000,000 \times 0.10 = 1,000,000 volunteers (300,000 improvement)
  • Donations:
    • Average of $21 per email address.
    • Original: 7,000,000 \times 21 = $150,000,000
    • Winning: 10,000,000 \times 21 = $210,000,000 ($60,000,000 improvement)
Conclusion
  • The AB test resulted in approximately 300,000 more volunteers and $60,000,000 in extra donations.
  • Demonstrates the power of AB testing in optimizing design for significant real-world impact.