A/A Testing: How I increased conversions 300% by doing absolutely nothing

I loved this piece.

The belief seems to be that if they just keep testing, they will find the answer, and build the business of their dreams.

Most of them are wrong. Many of their businesses would be better off if they didn’t run any A/B tests at all.

The author ran an A/B test on identical emails and found “statistically significant” differences. An increase in opens by 10%! But wait:

to a trained statistician, there is nothing remarkable about these “results.” Given the baseline conversion rate on opens, the sample size simply isn’t large enough to get a reliable result. What’s happening here is just the silly tricks our feeble human minds play on us when we try to measure things.

It’s very possible we are making wrong decisions based on false interpretations of information. Just look at these results from an A/A test:

A 9% increase in opens!
A 20% increase in clicks!
A 51% lower unsubscribe rate!
Finally, an incredible 300% increase in clicks, all by simply doing absolutely nothing!
…to an experienced eye, it’s clear that none of these “tests” have a large enough sample size (when taking to account the baseline conversion rate) to be significant.

The fact is, in so many cases where data is tracked, interpreted, and used to drive decisions, statistics isn’t the core competency of those involved.

To run a test that asks an important question, that uses a large enough sample size to come to a reliable conclusion, and that can do so amidst a minefield of different ways to be lead astray, takes a lot of resources.

You have to design the test, implement the technology, and come up with the various options. If you’re running a lean organization, there are few cases where this is worth the effort.

Running experiments and creating a vision are two different kinds of tasks. It’s possible you lessen your ability to make intuitive insights when you’re in the statistical weeds. Don’t give up on your vision so easily based on “results”.

Our world needs…vision, and if [we’re] busy second-guessing and testing everything (and often making the incorrect decisions based upon these tests), that’s a sad thing

And the author quotes Eric Ries from The Lean Startup:

Science came to stand for the victory of routine work over creative work, mechanization over humanity, and plans over agility.