Q: Should I always run an experiment when I make a change to my product? If not, when should I not?
If you work at a large company, every change you make is likely run as an experiment – partly because you need to demonstrate your impact, and partly because even small changes can have large unintended consequences.
If you work at a small startup, nothing is run as an experiment because there just isn’t enough data.
So this question is really for people who work at companies in that in-between phase. Which happens to be most companies.
Let’s first talk about why experiments are valuable, the downside of running experiments, and finally when to NOT run an experiment.
Why run an experiment at all?
It’s easy to take this for granted, but the fact that you can know the precise impact of your changes is…incredible. Who outside of the software space is able to tell if a change they made in language, color, or user experience is definitively helping or hurting their business? It’s amazing.
More broadly, the benefits of running your changes as an experiment are many:
- Learning: Experiments help you learn about your users
- Deciding: Experiments tell you if your change had the intended consequence
- Avoiding: Experiments catch unintended consequences
- Quantifying: Experiments precisely quantify the impact of your changes (and your team)
- Aligning: Experiment settle subjective debates
What are the downsides of running an experiment?
Experiments also certainly have downsides:
- Time: Experiments take time to set up and run
- False confidence: Experiments can create false confidence based on misinterpreted results
- Short-term thinking: Experiments can push you to think short-term
- Narrow-thinking: Experiments disincentivize taking bets that are hard to measure
- Bad product: Experiments can introduce awkward user-experiences or legal risk
When should you NOT run an experiment?
In spite of the downsides, I’m a big fan of experiments and generally default to running every change through an experiment. But, there are three cases where you should probably skip the experiment and just ship the change:
1. It’ll take too long to get actionable results
This is by far the main reason to skip running an experiment, and in my experience still far too underappreciated. Until you are at significant scale, experiments probably aren’t even worth thinking about. Particularly on features buried within your product.
Let’s take an example: Say you wanted to measure the conversion impact of a copy change on the last step of your sign-up flow, which is currently converting at 10%.
Question: How many users would need to go through this step in order for you to notice a 5% change in conversion?
It turns out you’d need over 60,000 unique users (per variation!) before you could draw a confident conclusion. That’s 120,000 users going through the flow before you can move on. For most startups, that ends up being far too long.
You can run this math yourself using this excellent sample size calculator from Optimizely, or this one from Evan Williams.
Furthermore, if you are looking to see the impact on your brand, network effects, long-term retention, or anything else that takes months/years to notice, you also shouldn’t wait for an experiment to run its course before making a call. Consider just launching it and keeping a small holdout to measure the long-term impact.
One lever you can play with to reduce the experiment time is the level of confidence required in the final result. When you’re moving fast and are feel OK about shipping slightly negative experiments occasionally, lowering your confidence interval to something like 85% (h/t Yair Livne for the advice) isn’t a terrible idea. Then, as you scale up and the stakes increase, you can increase the confidence interval.
Question to ask yourself: How long do you expect your experiment to take before you have conclusive results? Is waiting that long for this change worth it?
2. The downside risk of the change is low (and the effort is high)
The second most common reason to skip running an experiment is a low risk/reward ratio. Before running an experiment, answer these questions:
- Is your change a best practice and already worked well for others? (e.g. moving buttons above the fold, fewer steps in a flow, making your pitch clearer, etc.)
- Will you be able to detect significant negative impact in other ways? (e.g. before/after data, CX)
- What’s the most negative impact you’ve ever seen from an experiment like this?
- What will you concretely do with experiment results once you have them?
- What’s been the typical time for your team to set-up and analyze experiments?
Now you need to make a decision:
What’s the bigger risk: Making this change without an experiment, or taking your team’s time from higher-impact work?
If you can run experiments quickly and easily (e.g. a few hours), this decision is generally easy: run the experiment. If running experiments is a pain in the butt, and the changes are relatively benign, you can probably skip the experiment. If it’s somewhere in between, it’s a judgment call, but I personally default to running an experiment.
You may notice that the key variable in this formula is the time it takes to run and analyze experiments. The easier that process is, the more often you’ll end up running experiments, and the more you’ll learn. So if anything, spend time making that process easier.
Question to ask yourself: What if you skipped running an experiment or two and instead invested that time into a better experimentation framework?
3. You’re launching something completely new
This bucket almost goes without saying, but to run an experiment you need a control to compare your change against. If you’re launching a brand new independent product, or pivoting the product, you likely have nothing to compare this new product against (other than it not existing). In this case, you’re better off just setting independent success criteria (e.g. a specific retention rate, or a new user signup threshold), vs. coming up with an awkward experiment. This is especially true if going back to the previous product is out of the question and your own path forward is through.
As your company grows, and your individual impact is judged based on quantifiable results from experiments, skipping experiments becomes tougher. That’s normal. Still, no matter what stage your company is at, I hope these examples give your team some direction the next time you face this question.
If you’ve found other good reasons to skip running an experiment, share your thoughts in the comments 👇
🧠 Further study
- Please, please don’t A/B test that (written by our very own community member Tal Raviv!)
- The hypothesis prioritization canvas
- Optimizely's sample size calculator
- Evan’s sample side calulator
- Fostering a culture of experimentation
- A/B Testing — The Advanced Guide
See you next week!
🧠 Inspiration for the week ahead
- Read: How to think for yourself, by Paul Graham
- Watch: Super Practical Guide to Color Theory, Color Models and Perfect Color Palettes
- Behold: 🤯
🔥 Job opportunities
- Product: Descript, KUDO, Hipcamp, Prenda
- Growth: BasisOne, Coda, Instrumentl, Levels, Prenda
- Design: Ashby, Cascade, Office Hours, Runway, Stytch, Watershed
- Engineering manager: Cerebral
- Frontend engineer: Levels, Practice
- Backend engineer: Coda, Sourcetable, Transform
- Fullstack engineer: Cascade, Centered, Cerebral, Icebreaker, Iggy, Primer, Runway, Snackpass
- iOS engineer: Pairplay, Primer, Stytch, Vori
How would you rate this week's newsletter? 🤔
Have someone in your life who would benefit from this newsletter? Refer them and you both get a free month of the paid newsletter.
Give a free month, get a free month
Sincerely,
Lenny 👋