list segmentation | Patrick Conley

Statistical Significance in Split Testing

In Marketing, we LOVE split testing — and for good reason — it’s a fantastic method of optimizing a campaign for the best conversion rates and ultimately biggest profit margins.

However, one very important thing to keep in mind when performing these split testing experiments is the “statistical significance” of the results from a split test. For those of you who’s eyes just glazed over when I dropped the word statistics… I’ll try to keep this topic high-touch and easy to digest.

If you look up the wiki on statistical significance, you’ll find that we’re essentially attempting to predict if the results we get from a split test are meaningful and repeatable.

Why is this important?

Let’s imagine that we are attempting to test the effectiveness of 2 headline options on the sales page for our next launch. We want to figure out if headline “A” or headline “B” is producing more sales.

Here are the theoretical results after sending 1100 visitors to both sales pages:

[hr]

Example #1

[twocol_one]Headline “A”

# of visitors: 1100

conversion 5%

[/twocol_one]

[twocol_one_last]Headline “B”

# of visitors: 1100

conversion 9%

[/twocol_one_last]

[hr]

Using a Statistical Difference calculator, we can determine that the difference is 4%, with a comparative error of 2.1. Don’t worry if you don’t understand the math behind it, just be aware that a higher comparative error is bad!

The part you should care about is that the answer is “YES“, there is a statistical difference. We have statistically proven that Headline “B” is converting better, so we’ll go ahead and run all of our advertising dollars towards Headline “B”.

Now what happens if we reduce the traffic to the site during the test? Can we still be sure that Headline “B” is better? Let’s suppose that we were only able to drive 200 visitors to each test. Are our results still valid?

[hr]

Example #2

[twocol_one]Headline “A”

# of visitors: 200

conversion 5%

[/twocol_one]

[twocol_one_last]Headline “B”

# of visitors: 200

conversion 9%

[/twocol_one_last]

[hr]

Checking the calculator again, we now see that the comparative error has jumped up from 2.1 to 5 (remember, higher is worse!). And the result is that there is NOT a statistical difference between our results.

What does this mean in practical terms?

Say we have $10,000 to spend on advertising for this campaign. From our quick calculations, we have proven that the results from Example 2 could be a total fluke. We didn’t have enough visitors to the site to validate that the results are meaningful! So now we really DON’T know what headline should yield higher results without sending more traffic to verify. Better do some more testing!

Have you ever run a split test with interesting results? Post your story below in the comments.

One of the most commonly under-used parts of Infusionsoft (or marketing automation in general) is collecting psychographic data and acting upon it. It’s hard because it forces us to use our heads.

It’s much easier to collect demographic data (location, age, etc.) but that’s not nearly as helpful when it comes to developing a solution to a deep seated goal or pain.

For example, let’s pretend that we are creating a few campaigns to market new products for a health and fitness website. If visitors opt-in on a page related to losing weight, we can assume that losing weight is likely a goal of theirs, and we can capture that information by tagging anyone who opts-in on that web page as [Goals -> Lose Weight] in Infusionsoft.

Now that we understand a primary goal of the visitor, we can add them to a sequence that specifically offers advice on the best ways to lose weight. Doing so, we’ll also be able to reduce the chance of losing their attention by sending information that he or she is NOT interested in.

Sending ultra-targeted information will reduce unsubscribe rates and improve conversions!

Now let’s look at another example. An older gentleman visits our theoretical site, fills in his info to download a free report on nutrition to slow down the effects of aging on our body. We notice (by paying attention to Infusionsoft analytics) that he has viewed nearly every page related to nutrition for wellness, but hasn’t clicked a single link related to losing weight.

For this guy, it makes way more sense for us to send him a tailored campaign surrounding the content that he’s shown interest in. By sending him relevant information, he’s more likely to stick around and hopefully become a customer!

Can you think of any good examples of psychographic data to be collecting for your CRM? Share them below in the comments!