A Beginner's Guide to A/B Testing for Mobile Commerce

The mobile commerce (m-commerce) space has grown 42 percent annually for the past four years—more people are becoming comfortable spending money on their phones, and more retailers are investing in mobile to woo these small screen shoppers. But as the market grows, so does the competition for thumb space and user retention.

To compete, you need to win over users quickly, and A/B testing can help. We suggest focusing on these three areas of the user experience when you’re getting started with m-commerce experiments:

1. Onboarding

2. Activation

3. Retention

Onboarding

A great first-time user experience is critical for users to adopt your app. It’s so important that at any given time more than 50 percent of Taplytics’ customer experiments are focused on improving onboarding. These experiments might test the content and imagery in the first few screens of an app, or the process of getting a user to submit information. Whatever you choose to experiment on, the goal is the same—help your users experience the magical moment of making a purchase as quickly as possible.

Let’s take Frank & Oak for example. The men’s fashion company sets themselves apart by creating a custom shopping experience tailored to each individual’s interests and behaviors. But to get the personal experience, users need to sign up on Frank & Oak’s app first.

As a result, the Frank & Oak team started their A/B testing initiatives with tweaking the signup flow. They first tried changing the field forms and adding the ability to sign up via Facebook. After the first experiment, they tested to see whether an additional option to login with Google would increase signups. You can see the final variation below.

It turns out, that adding a “connect with Google” button increased mobile signups by 150 percent.

While Frank & Oak saw impressive results from a few simple tests, you may need to go through a few more iterations before you find a winning strategy. Here are a some onboarding tests you can run to help get your users through your sign in process to start browsing items.

1. Login Funnels

2. Information Collection

3. Text and Imagery

4. Sign up Options

Activation

Once your users get through the onboarding process, it’s time to activate them. In commerce, this means getting them to purchase something as quickly as possible. To have a shot, our research shows you’ll need to get them to buy within the first two sessions, or it’s unlikely they will ever come back.

For example, when Karmaloop analyzed their customer activation data they noticed that if a first time user placed an item in their Wish List they were much less likely to complete the purchase compared to if they added an item to their cart. Once Karmaloop identified this trend, they set up an experiment to deemphasize the Wish List button in the UI.

Discouraging users to interact with their Wish List allowed Karmaloop to better capture purchase intent. This simple test drove more activations and increased sales by 35 percent.

What can you learn from their experiment? When you’re looking to improve your customer activation, challenge the status quo. Don’t assume that a current feature is achieving your goals. Instead, test your assumptions and use funnel analytics to help brainstorm ideas.

Retention

You pushed customers through onboarding and activation—they bought something! Woot! But you’re not done. The last piece of the puzzle in mobile commerce is retention. How do you turn customers into repeat buyers?

There are many strategies m-commerce companies use to increase retention. One effective strategy is hosting targeted sales on a regular basis. For example, Rue La La keeps a “What’s Hot” section in their app specifically to develop scarcity and drive repeat purchases. Other apps achieve it through targeted notification campaigns, whether via email or push.

In the Rue La La example, they could easily test the effectiveness of the “What’s Hot” section on user retention, by creating an experiment where “What’s Hot” is replaced with another section of the app, or taken out altogether. The team would then analyze if cohorts who saw the section come back and purchase more often. Other ideas for testing retention include:

1. Last chance section

2. Trending list

3. Adding discounts or promos for in-app purchases

4. Changing photography to make items more appealing

Getting Started

The tests discussed here offer ideas to get you started, but to be successful with A/B testing, you’ll have to continually challenge the norm and make sure your team is invested in listening to the results of A/B tests. Some of the biggest improvements can come from the smallest tweaks or even testing a feature you thought was performing well. Once you find a way to optimize your app, you’ll need help from your team to rally around making the change permanent.

If you are new to A/B testing and want to learn more about A/B testing best practices, and how to create a culture that drives experimentation, check out the Mobile Growth Academy at Taplytics for some helpful tips and tricks.