A/B Testing Best Practices | Segment Growth Center

We go over the A/B testing best practices and how to use this method to optimize your website and marketing campaigns.

Section

An A/B test is when you change one element of a webpage, app, or digital asset, to see if this impacts user behavior. The point of an A/B test is to better understand how content, campaigns, or even UX design is influencing customer decision-making and the quality of your user experience.  

While all experiments begin with a hypothesis, the most successful experimentation campaigns have clear metrics and KPIs attached (i.e. how are you measuring success?). 

We’ve provided a few best practices for A/B testing below, to ensure that your team is leveraging experimentation to drive optimization (not just testing for testing’s sake). 

7 best practices for A/B testing

When starting out with experimentation, it’s important to situate tests (and test ideas) within a larger, big picture framework. That is, which areas of your marketing funnel or product experience should take top priority? What’s the intended impact? 

Knowing the answers to these questions is crucial for a multitude of reasons. For one, you’re going to have to get internal buy-in for your testing program and show how it’s moving the needle for your business. Without a clear understanding of the problem you’re trying to solve, you’ll be hobbling together a testing plan at random – which is a waste of both time and resources. 

Now, let’s go a little deeper into creating an A/B testing framework.

1. Determine the goals

First things first: what's the goal of the A/B test?

When starting out, consider the overarching goals of your business. Is there enough brand awareness among your prospective customers? Is your website traffic too low? Are you trying to incorporate more cost-effective growth strategies?

Often, “trouble spots” are a great place to start A/B testing. For example, if a marketing team isn’t meeting their MQL goals, they can A/B test their landing pages by swapping out different pieces of gated content, to see which topics resonate most with their audience.

2. Understand what you want to test

When whittling down what you should test, keep your ultimate goal in mind.

Let’s say you want to A/B test an email campaign. If the goal is to improve open rates, it makes sense to experiment with the subject line. But if your team is more focused on driving event registrations, it's likely better to test out different CTAs in the body of the email. 

Here are some examples what you can test across your website, email campaigns, social media posts, or design:

  • Headlines

  • CTA text

  • CTA button placement

  • Email subject lines

  • Images

  • Product descriptions

  • Social media copy

  • Google ad copy

  • Free trial lengths

3. Define what KPIs you’ll use

To make heads or tails of your A/B testing campaign, you’ll want to define what success looks like, which means understanding your baseline performance. What are you current metrics when it comes to paid ad campaigns, MQLs, website traffic, and so on?

Knowing where you're starting from is essential to understand how you've improved. It's also worthwhile to compare your current performance with industry benchmarks. These industry-wide averages can serve as a measuring stick of what's working and what isn't (especially compared to some of your competitors). Take website popups: according to Campaign Monitor, the average conversion rate is around 3%, with top performers reaching 10%.

A few common examples of KPIs used to measure success include:

  • Conversion rates

  • Website traffic

  • Revenue

  • Customer lifetime value

  • Engagement (e.g. clicks, open rates, purchase recency, etc.)

  • Product adoption

  • NPS

4. Target the right audience

There will be times when an A/B test seeks to understand the behaviors of a specific audience segment. Perhaps you want to target a specific persona that would be interested in a new product feature, or you want to exclude current customers from a paid ad campaign.

This is where audience segmentation comes into play. You can segment audiences based on behavioral data, demographics (like job title, industry, location), or by visitor source (e.g. paid ad, organic search, and so on.). We're big proponents of always considering users' behavioral and historical data when creating an audience. These two components show how people interact with your brand specifically, and provide important context.

good-audience

Learn more about customer segmentation here.

5. Test at the right time

The timing of your A/B test can significantly impact the results. 

Figuring out the right time to test can feel like the Goldilocks’ conundrum. You don’t want to test in a slower period for your business, as you need a large enough sample size to ensure accuracy. But on the other hand, running a test during an exceptionally busy time (e.g. in the lead up to a holiday) may not reflect user behavior on the day-to-day. 

To determine the right time to test, think back to the goals of the experiment and your baseline performance. If the goal is to optimize a Black Friday campaign, then testing around the holidays is the logical choice. But if the goal is to see how a UX change will impact user behavior on the day-to-day, it's best to pick a time that isn’t influenced by seasonality. 

6. Ensure statistical significance

An important part of A/B testing is statistical significance, or the confirmation that the differences between group A and group B aren’t a fluke. As a quick note, a 95% confidence level is the usual benchmark for ensuring test results are accurate, which would be a p-value of 5%. (You can go deeper into the difference between these values here.)

There are free calculators available that can help you determine the sample size needed to reach statistical significance. These calculators often take into account your website traffic, conversion rate, and sometimes the Minimum Detectable Effect (MDE), which is the smallest lift/improvement that you want to detect from the experiment. 

VWO-calculator

VWO’s plug-and-go significance calculator for A/B tests

7. Understand the data collected

You’ve run the A/B test, you’ve collected the data – now, you need to analyze the results. Experimentation is a pathway to understanding; even a failed experiment can provide important customer insights.

But understanding A/B test results can be complex. You have to navigate statistical significance, false positives, and ensure there hasn’t been a glitch in data collection. Luckily, there are tools that can help you set up A/B testing campaigns and safeguard against misinterpreting results.

Here’s a list of the experimentation tools that integrate with Twilio Segment.

SendGrid-AB-Testing

Interested in hearing more about how Segment can help you?

Connect with a Segment expert who can share more about what Segment can do for you.

Please provide your company email address.
Please enter a valid email address.
Please provide an individual corporate email address.
Please provide a valid full name.
Please provide your phone number.
That phone number is too short.
That phone number is too long.
Please provide a valid phone number.

For information about how Segment handles your personal data, please see our privacy policy.

Thank you, you’re all set!

We'll get back to you shortly. For now, you can create your workspace by clicking below.

Thank you for submitting your request for a demo! Answer 4 more questions to help us pinpoint exactly what your team needs to get started with Segment.

Please provide a valid job title.
Please provide a company size.
Please provide the estimated web traffic.
Please provide a timeline.

For information about how Segment handles your personal data, please see our privacy policy.


Frequently asked questions

This question comes with more than one answer. There may be times when a different test is better suited to your goals. For example, you may want to test multiple different variables on your web page or digital asset, which means a multivariate test would be the better option (since an A/B test only changes one variable).

Or it could be a matter of timing. If you know a certain season or holiday represents a slow time for your business, historically, it may not be worth running an A/B test because traffic will be too low.

The general rule of thumb is that an A/B test should run for seven days at least, but many recommend two weeks. The truth is, the exact timeline will vary depending on your sample size. This calculator can help you figure out the appropriate amount of time to run your experiment, to ensure you reach statistical significance.

Marketers can run A/B tests on their email campaigns with the Twilio SendGrid API. SendGrid can help you tweak subject lines and body copy to optimize open and/or click rates. Chat with an expert about to learn more!