You are Not Smarter than A/B Testing

The problem with humans is that we make lots of decisions without having data to back us up.

And to compound the problem, unlike the movie Sliding Doors, where the main character gets to see how a different decision plays out in her life, we never know if we made the right decision, because we can't know how things might have turned out otherwise.

Enter A/B testing.  Through the magic of large numbers and averages, A/B testing gives us a way to see how two different decisions play out.  Really, it's magic, because it gives you a way to have two competing options fight each other, UFC style.

The problem is that A/B testing can be hard to set up on your website.  But a newish startup named Optimizely takes the pain out of the process.  I'm seriously amazed at how good it is.  (You can get 25% off the Optimizely monthly fee if you have a FoundersCard, by the way).

Here are examples of how we changed Socialize's messaging based on a series of A/B test results:

Our original messaging centered around "app intelligence".  Optimizely helped us hone that into a message about "boosting downloads,"  which resonated much more with developers.

Now we're focused on the sub-text:  Here's a current Optimizely campaign we have running:

Here's how the results are playing out -- you can see that the message on the right (called "best marketers") is beating the pants off of the original message on the left, with a 67% gain.  (There's another, similar version called "best champions" with an even greater gain of 84% over the original) :

Here are just a few examples of the 1,050 possible permutations our front page can take (we're concurrently running four A/B experiments w/ 3 to 7 options each):

Out of these 1,050 possible combinations, one will deliver the best possible conversion result, and with A/B testing we can find it.  Once we do, we'll focus on another area of the site to test (maybe the blue button, for example) and optimize that.   If you get really hard-core about this (which we're not yet), you can work on optimizing all aspects of the page at once, with true multi-variate testing.  (Optimizely charges substantially more for this, so for now we're just running a series of concurrent A/B tests, which approximates the result you get from true multi-variate testing).

I highly encourage you to start A/B testing your website, if you're not already.  You really don't have any excuse not to, since Optimizely makes it painless to set up.