- Melbourne, Australia
- FAQ
- About AML
- The List Report
- Contact Us
What is it? Testing.
Although marketers who test clearly achieve better results than their counterparts who don’t, only about 40 percent undertake this surprisingly easy and simple tactic, according to a JupiterResearch survey of more than 600 email marketers. In its January 2005 report, “Effective Email Marketing”, JupiterResearch found that marketers using testing were almost twice as likely to attain conversion rates of 3 percent or better. They also achieved a 68 percent improvement in return over non-testers.
So why aren’t more marketers doing it? Lack of awareness and lack of resources are most often cited. But if you knew that experimenting with your email landing page—the page recipients go to when they click a link in your email—could boost your return by 40 percent, wouldn’t that be worth looking into? According to MarketingSherpa’s “Landing Page Handbook”, marketers who test and tweak their landing pages consistently achieve such marked improvements.
What can you test?
The types of testing email marketers find most worthwhile, according to MarketingSherpa’s 2003 “Email Metrics Survey” of more than 2,000 email marketers include:
You can also test your offer, design, time of day, message layout, day of week and duration before follow-up. Do recipients prefer a 30 percent discount, or a $20 discount? Do some recipients who don’t open HTML emails open their text versions? Do they respond to a catchy subject line, or a straightforward promotion? No other medium makes it as easy to test and act on those tests as email marketing.
So, if this testing thing sounds interesting, how do you do it?
Split your list. Divide your list into two or more groups (sometimes called an A/B split) and changes one characteristic (e.g. subject line) for each group. Assuming that your groups randomly so that each represents an accurate cross-section of your overall recipient base, and everything else about your message remains the same, your results should clearly reveal the best-performing characteristic.
Or, you can test to a subset of your list. Think of it as an email dress rehearsal. Using a technique called nth-testing, you pull every nth record from your list to create a near-random set of subgroups. For example, if you have 1,000 names, and you nth on every 10, each of those 1,000 names is assigned a value of 0, 1, 2, 3, 4, 5, 6, 7, 8, or 9. If you wanted to create two random “test” segments, you would send one offer to segment 0 and another to segment 1. You would then send the best performing offer to segments 2 through 9.
Conduct tests at the same time. Time is a variable, and sending test email A in the morning and test email B in the afternoon can yield very different responses. So send tests out as near to the same time, same day as possible.
Make sure the results are statistically relevant. One or two responses are not enough to tell you whether one test succeeded over another. While purists may argue the exact number, you should try to get at least 50 to 100 responses for each test before you can make broad judgments about which performed best. So, if you are noting, be sure to segment off a large enough chunk of your overall list. You can arrive at number of recipients you’ll need by using your average click through rate. If you have a list of 10,000 recipients and your average click through rate is 5 percent, then you could expect to receive 500 responses. So to get 50 responses, one tenth of that total, you’ll need to test one-tenth of your list, or 1,000 recipients. To get 100 responses you’ll need to test one-fifth of your list, or 2,000 recipients.
To figure out how well your test performed, compare click-through rate of your test group to your average click-through rate, and to the other version to see which did best, or which you will send to the rest of your list. A rule of thumb recommended by some (ClickZExperts “A/B testing for the Mathematically Disinclined") is that you should have at least a three times larger result in order to be able to declare a clear winner.
Maintain a control group. A control group is a random sample of your list that is excluded from the change you are testing. This enables you to compare the behaviour of the test group vs. the control group to determine the precise effect of your change. This might be the group that gets your usual subject line A, against which you are testing subject line B. Or, a great example is frequency testing. If you want to find out whether sending more frequently would be more profitable over the long-term, or whether it would wear down recipients with too many messages, you can compare the response of your test vs. control group over a period of months to get your answer.
The options outlined here barely scratch the surface of all the email testing possibilities available. Understanding campaign responses and their implications for future customer behaviour can get into incredibly complex statistics, analytics and modelling. Fortunately, marketers without a Ph.D. can still use the basic testing methods to significantly enrich their ongoing customer relationships. The real power of email isn’t how easily and quickly you can send it, but how easily and quickly you can figure out what appeals to your recipients, tailor your offers accordingly and keep them coming back for more.
Posted on 27th June 2006