Experimentation Works: The Surprising Power of Business Experiments

Physical Copy:

Audiobook:

By: Stefan Thomke

Rating: C+

I have read several books over innovation and what that looks like in practice. How to encourage creativity and out of the box thinking, while also running a healthy business. A big part of innovation is testing what works and what doesn’t. This book was a detailed guide on how to do that. What was apparent was how bad the business community is at guessing what people actually want. 

“Without experiments, many breakthroughs might not happen. And many bad ideas would be implemented, only to fail, wasting resources.”

I.Why experimentation works

“Oftentimes, too, managers rely on their intuition - but ideas that are truly innovative go against experience. In fact, most ideas don’t work.”

“Experience is often context-dependent and a success may result in hubris. Just because something worked for another company in another market doesn’t mean that it works here.”

Experimentation offers fast feedback on what does and doesn’t work. A company culture of systemizing experimentation allows you to build your experimentation capacity. A system for tracking the results of experiments allows you to run concurrent experiments. Finally, using a control group allows you to better trust the results of your experiment. 

The end product is a system for determining what works and what your customers actually want with confidence and at a lower price.

II.What makes a good business experiment?

“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day...We aim to do the unnatural thing of trying to disconfirm our beliefs.” - Jeff Bezos

The 7 Important Questions to Ask When Designing a Good Experiment:

1. Does the experiment have a testable hypothesis?

   1. Quantifiable metrics.

   2. Identifies potential cause and effect.

   3. Can be shown to be false.

   4. Has a clear impact on business outcomes.

   5. Have stakeholders made a commitment to abide by the results? Results must inform decisions in an unbiased manner. Do not cherry-pick data to support your intuition.

   6. Is the experiment doable?

   7. Can we ensure reliable results? Controlled, blinded, double-blinded, multi-centered, multi-operators, etc.

   8. Do we understand cause and effect? Correlation does not necessarily equal causation.

   9. Have we gotten the most value out of the experiment?

   10. Are experiments really driving our decisions? If not, you are wasting time and money.

III. How to experiment online

Microsoft, Facebook, Booking.com, Amazon, Google all conduct more than 10,000 online controlled experiments every year.

1. Test all testable decisions: User interface (color changes), new features, business models, etc.

2. Appreciate the value of small changes

3. Invest in a large-scale experimentation system: It is vital to have proper instrumentation, data pipelines, analysts, and data scientists to ensure fast and inexpensive feedback.

4. Build trust in the system:

   1. Run control vs. control tests.

   2. Provide employees with proper statistical and software training.

   3. Check data for outliers.

   4. Shuffle samples in each experiment to avoid carry-over effects.

   5. Keep it simple: Limit the number of variables in the tests to make better conclusions on causality.

IV. Can your culture handle large-scale experimentation?

The 7 attributes that make up a successful experimentation organization:

1. A learning mindset. Failed experiments are necessary for learning. Try, try, try again.

2. Rewards are consistent with values and objectives (i.e. incentivize your employees to experiment with long-term goals in mind)

3. Humility trumps hubris. Keep ego out of your business. Ego can cause seeing relationships where there are none, confirmation bias, and rejecting conclusive evidence.

4. Experiments have integrity.

5. The tools are trusted. Trust experimentation!

6. Exploration and exploitation are balanced.

   1. Exploration: capture new value through new operations. 

   2. Exploitation: maximizing bottom line by standardizing operations.

   3. Ability to embrace a new leadership model. If decisions are made through testing, what is the role of managers? The roles are:

      1. Set the vision/long-term goals and break them down into testable hypotheses.

      2. Put in place systems and resources to allow large-scale, trustworthy experimentation.

      3. Lead by example and test their own ideas before making decisions.

V. Becoming an experimentation organization

Use these metrics and framework to transition into an experimentation organization.

Experimentation System Metrics:

1. Scale: Number of experiments per week.

2. Scope: How involved in experiments are the employees.

3. Speed: Time from hypothesis formulation to experiment completion.

4. Shared values: Behavior and judgment that facilitate experiments.

5. Skills: Competencies needed to design, run, and analyze experiments.

6. Standards: What are the quality check criteria that facilitates trust.

7. Support: How much technical help, training, and managerial support is available.

ABCDE framework:

1. Awareness: Understand that experimentation matters.

2. Belief: Adopt rigorous framework and tools.

3. Commitment: Allocate resources and change organization.

4. Diffusion: Widen scope and access to tools.

5. Embeddedness: Democratize experimentation.

VI. Seven myths of business experimentation

Myth 1: “Experimentation-driven innovation will kill intuition and judgment.”

Truth: They are complementary rather than substitutes. Testing is a faster and cheaper way to test your intuition.

Myth 2: “Online experiments will lead to incremental innovation but not breakthrough performance changes.”

Truth: Many breakthrough performance changes are motivated by findings from online experiments.

Myth 3: “We don’t have enough hypotheses for large-scale experimentation.” 

Truth: The more you experiment, the more hypotheses you generate from your findings.

Myth 4: “Brick-and-mortar companies don’t have enough transactions to run experiments.” 

Truth: While sample size is definitely a valid concern, the author advises such companies to focus on running bigger and riskier experiments.

Myth 5: “We tried A/B testing, but it had a modest impact on our business performance.” 

Truth: Experimentation is a highly long-term strategy and many failures are to be expected. 

Myth 6: “Understanding causality is no longer needed in the age of big data and business analytics. Why waste time on experiments?” 

Truth: Correlation is not causation. A deeper understanding of why things happen is a competitive advantage.

Myth 7: “Running experiments on customers without advanced consent is always unethical.”

Truth: Awareness may alter results. “People seem unconcerned with the current practice of being emotionally manipulated through advertising and other means, although the harmful effects of these media may have never been rigorously tested.”

Next Steps:

What software or consultants can help set up experimentation in our operations?