Analyzing Experiment Results

Analyzing Experiment Results

Key Question

• How to extract the most learnings out of every experiment.

Success Rate

• Less than 25% of experiments are typically successful.
• An experiment is only a failure if you fail to learn.
• Failed experiments help navigate to success.

Three Levels of Analysis

1. Success and Failure
   • Was the variation a success or a failure?
   • Did it improve our target metric?
   • What happened to our trade off metric?
2. Accuracy
   • How accurate was my upside prediction?
   • If prediction is far off, there might be some additional things to dig into.
3. Why
   • Potential reasons for success or failure.
   • Break the experiment into pieces to look at the different inputs into the success metric.
   • Segment to figure out if the experiment influenced different audiences at different levels.
   • Look at qualitative data.
   • Run additional experiments to validate new hypotheses.