Don’t Poke the Bear: The Value of A/B Testing
I was once working on a campaign to introduce a groundbreaking new subscription service, and thought, "Hey, offering a free 6-month subscription will surely drive adoption and loyalty!" But guess what? I have a real-life example where this hypothesis fell flat on its face. This is where the importance of A/B testing comes in.
We were eager to introduce our next-level product that featured best-in-class technology. We decided to target the high-propensity-to-churn segment with the free 6-month offer. Everyone loves free right? Not so much.
The treatment group that received the direct mail and email campaign actually showed a higher rate of churn compared to the control group that was left untouched. We immediately canceled the test and went back to the drawing board. We came up with three hypotheses as to why this happened:
1. Tiptoeing Around Trouble: The Bear Factor The group we were targeting was already at a high propensity to churn. How did we identify them? Our models took into account data such as history of trouble calls, payment status, demographics, level of service, tenure, and renewal window. By reaching out, we were basically giving them a shortcut to the "unsubscribe" button. We poked the bear and reminded them why they weren't fans anymore.
2. When Fancy Doesn't Cut It We assumed that flashy new features would drive adoption. Instead of wowing them, some of our customers felt like we were showing off with bells and whistles they didn't even need. Plus, with the feeling that we were already overcharging them, they saw this investment as a waste of their money.
3. The Sour Aftertaste of Pricing Our offer for the free service was only for 6 months. After that, the full price kicked in, and that price was higher than what they were paying today. That probably left a bad taste in their mouth when they likely already felt they were paying too much.
A/B Testing Rescues the Day We shifted our strategy, and tested the same offer and creative with a new segment: our loyal customers. You know, the ones who've been with us through thick and thin, who don't mind spending a little extra for quality. And guess what? We achieved a 44% lift over control in the adoption rate. We also kept an eye on their long-term retention which was not impacted. Success!
The lesson here? Always let A/B testing be your guiding star. If you're interested in diving deeper into the world of testing campaigns and mastering customer loyalty, reach out to us.