- Melbourne, Australia
- FAQ
- About AML
- The List Report
- Contact Us
Testing haphazardly or running sloppy tests
Testing is a discipline. You have to test continually and carefully. Otherwise, the numbers just won’t mean anything. If you don’t have the skill or patience for ongoing tests, number crunching, and analysis, get someone else involved.
Assuming that your tests are error free
You should actively seek out mistakes on every level. Whether a given test comes out good or bad (but especially if it’s bad), think through the whole process to track down errors. Were the addresses good? Were the return address and bar-code correct? Are phone operators logging every response properly? Did your ad actually run? Are e-mails being dumped as spam? What else might be going wrong?
Drawing the wrong conclusions
Too often, people look at test data and make a snap judgement: “That self-mailer bombed. Self-mailers don’t work” or “We tested a Christmas appeal, and we lost money. Christmas is a bad time for promotions”. This is usually the result of impatience combined with a poorly designed test. Ideally you should test with the express purpose of measuring one variable. And you must test against a proven control. Otherwise, you may conclude that a particular variable affects results when it doesn’t.
Making decisions based on insignificant results
Every test must be statistically valid. This means you must reach enough of your audience to assure that you have accurately sampled that audience and have received enough responses to accurately calculate your results. When you fall below certain minimums, your results are worthless.
Overlooking important results
Numbers are just data. You only have “information” once you analyse your numbers and draw conclusions. That’s why it is important to do more than just list your results; you should play around with them. See what might be hiding in all those digits. Is there a trend? Are your results seasonal? How do your results compare with the industry averages?
Refusing to repeat tests to confirm results
An accurate test is repeatable. In other words, if your results are true, you’ll be able to test again and get similar results. If you don’t, there’s something wrong somewhere. You should always confirm a dramatic change in results, good or bad, with another test.
Filing away results instead of using them
Why test if you just calculate a response rate and throw your report into a filing cabinet? Those numbers are expensive to get, so use them. Analyse every test quantitatively and qualitatively. Show the numbers and write your thoughts and conclusions. Then share the test data and everyone involved. After every test you should know something useful: “This two page letter works just as well as this four page letter” or “This offer increased enquiries from our TV spot by 35%”.
Failing to keep a running record of conclusions
Over time, as you see the results of test after test, you will start to see patterns emerge from the numbers. This is the gold you’re planning for. Organise and list this information as a guideline for future efforts. Building on your hard earned knowledge will dramatically increase your success rate.
Posted on 29th September 2009