A/B Testing Metrics: How to Choose Your Metrics

Section

Metrics are the guard rails of any A/B test. They keep your team focused on the impact of each experiment, and how it fits into your business’s “big picture.” Without metrics, your testing plan would be haphazard (at best) or a waste of time and resources (at worst). 

But how do you know which metrics to track? In a world where everyone is motivated to become more data-driven, it’s easy to fall into the mindset that any data will do. But that’s not the case. 

To glean valuable insights from your results, and you’ll have well-founded data to inform your marketing efforts.   

How to choose which metrics to use

The simplest way to pick a primary metric is to focus on what you want to improve.

Begin by comparing the results of your marketing campaigns to your quarterly or yearly goals. Which initiatives are performing below standard? What overarching goals is your team lagging on? 

Let’s say your team is reviewing the Q3 performance of key marketing initiatives. When you see the results for your daily newsletter, you realize it’s falling below  the goal of a 25% open rate, instead averaging at 12%. This presents an opportunity to run A/B tests to help optimize the subject lines or email copy – anything that could impact customer engagement.

Most common A/B testing metrics

These key metrics are some of the most popular ones marketers choose to A/B test. 

Conversion rate

A conversion refers to a customer or prospect taking a desired action with your business, like signing up for an email, making a purchase, or downloading an asset. Conversion rates directly measure how much users are interacting with your homepage, newsletter, or social media channels. 

Click-through rate

 Click-through rates represent how many people clicked on a digital asset, among those who received or came across it. It’s most commonly used to measure the effectiveness of digital ads, but it also applies to newsletters, SMS marketing, and social media posts. 

Bounce rate

Bounce rates tell you how many people visited your website and left without taking an action. It can indicate how engaging or useful your site is for visitors. 

Optimizing your bounce rate often entails making the design and layout of your website seamless to navigate, while ensuring the content is relevant to your persona(s). The goal is to keep people on your site, and guide them one step closer to a desired action. 

A/B testing metrics examples

Check out the metrics used for each of these A/B tests and see if a similar setup would work for your test. 

Free trial conversion rate

In a blog post for Segment, Demand Curve proposed an experiment to test if chatbots could increase the number of customers who sign up for businesses’ free trials. In this hypothetical experiment, they decided the north star metric should be the free trial conversion rate because research from Intercom found that website visitors are “82% more likely to convert to customers if they’ve chatted with you first.”

The experts at Demand Curve believed an A/B test before full implementation of the chatbot was necessary because investing in a chat service is expensive. The benefits of the bot need to outweigh the personnel and technical resources invested into getting it up and running. 

An A/B test in this scenario can easily demonstrate if adding the bot does lead to an increased free trial conversion rate. Then, marketing leaders can see if the increase is significant enough to justify the costs of implementing the bot. 

Mobile signups

Online clothing retailer Frank & Oak wanted to test whether adding a “Connect with Google” button to their in-app signup page would increase the rate of mobile signups. Signups are key to Frank & Oak because once customers log in, the retailer can provide personalized fashion recommendations that get users closer to making a purchase. 

The Google button allows customers to quickly and easily link their Google account to their Frank & Oak log-in, rather than manually typing in all of their information to start a new account. 

Once the retailer ran its A/B test, marketers on the team found that adding the Google button increased mobile signups by 150%. It was an easy decision from there to implement the button for all users. 

Sales

Streetwear retailer Karmaloop was interested in testing if decreasing the size of their “Add to Wish List” button could lead to more sales. The company noticed that customers who put items on their wish lists were less likely to buy them. Though the wish list is a nice feature for customers, it’s not useful for the business if it’s hindering potential sales. 

Instead of ditching it completely, Karmaloop proposed deemphasizing the feature by making the button smaller. Their team ran an A/B test to see how a smaller button might affect sales compared to the existing one. The results showed sales increased by 35%, and that persuaded Karmaloop to keep a smaller button. 

Segment’s CDP can help build better A/B tests

Good data is the heart of a well-executed A/B test. Without accurate and comprehensive information, you can’t gain the insights you need to see if the experimental version of your test was truly better. And while most marketers use an A/B testing tool like Optimizely or Apptimize to conduct their tests, they still need a stream of good data to provide up-to-date information for the testing tool to analyze. 

That’s where Segment’s customer data platform comes in. User and e-commerce data usually come from a series of different sources, like HubSpot, Salesforce, and Google Analytics. Segment brings it all together in one platform and an easy-to-navigate dashboard. 

The CDP also automatically validates the data, so you know the information you’re collecting is accurate and reliable. In the CDP, it’s easy to narrow down data for specific audiences and track customer behavior that’s relevant to your test. With Segment’s CDP, your A/B test consists of data that are accurate and specific to your needs.