Build better A/B tests with a customer data platform

Segment can help power your entire tech stack with data you can trust. Sign up for a demo for a walkthrough of the platform.

In 2020, you’d think that running A/B tests would be easy. If only.

There may be hundreds of A/B testing tools on the market, but in reality, running these tests requires begging engineering to pull the necessary data, trawling through that data to divide it into unique audiences, and finally, send those audiences over to an A/B testing platform.

That’s where a customer data platform comes in. 

A CDP will help you collect, validate, segment, and send all the data you need to the A/B testing platform of your choice in just a few clicks. It means you can spend more time actually testing out new strategies and less time wrangling data.

With a customer data platform like Segment and an A/B testing tool of your choice, you can start removing the guesswork from your business decisions.

Remember, A/B testing is one of the only scientific ways to measure whether the changes you’re making are having a positive or a negative effect on your business. You wouldn’t want incomplete or dirty data to muddy your tests now, would you?

In this article, you’ll learn how to power your A/B tests with clean, quality, and hyper-specific audiences, and our recommendations on the best A/B testing tools for 2020.

Get the most from your A/B tests with a customer data platform

The first step to better A/B tests with a CDP is by following these three steps.

1. Design your A/B test

Designing a good A/B test takes practice and effort. If you need a primer on the intricacies of how A/B tests work, we strongly recommend reading our friend Julian Shapiro’s guide. 

But the four basic tenets are as follows:

Your baseline (A) and your variation (B)

These are the two versions of the part of the customer experience you want to optimize. The baseline is the original version, while the variation is the updated version. The purpose of the A/B test is to see if the update is actually better than the original.

Your audience segments

Audience segments are the groups you’ll run your test on — groups you want to understand better. Generally, you’ll decide what group to test based on a larger strategy, like whether you want to shift your target audience.

Your sample size

This is how many people you need to run the A/B test on in order to get statistically significant results. Figuring out the right sample size gets into heavy math, but there are plenty of calculators you can use to figure out your sample size. Here are three we recommend:

• Optimizely’s calculator will work for most people.

• A/B Testguide’s calculator gets more technical.

• Evan Miller’s calculator gets into the underlying algebra.

Your timeline

This is how long you need to run the test to get accurate results. If your test is run in too short of a timespan, you risk outside factors impacting your test. Peep Laja, the founder of CXL, recommends at least three to four weeks. He also says that one warning sign of not running your test long enough is getting less than 250–350 conversions per variation in each segment.

2. Collect and segment your data with Segment

As your central hub for collecting and managing customer data, Segment can help with A/B testing in two specific ways: tracking customer behavior and audience segmentation.

Firstly, set up event tracking with Segment to capture customer behavior. You can do this through analytics.page and analytics.track calls, which are JavaScript code snippets you embed on your website. Segment collects this data and organizes it into unique customer profiles, ensuring it’s accurate and that there are no duplicates. 

In the end, you have a clean library of customer profiles containing everything you know about each customer and their behavior.

Then, you can use Personas, Segment’s audience management product, to build audiences based on any data that you’ve collected via Segment. For example, you might want to run an A/B test on users who’ve viewed one of your products above a certain price threshold. 

With Segment, you can do this in seconds. 

You can also use the Personas Computed Traits feature to get account-level calculations that can quickly sort through all your customer data. With these calculations, you can figure out things like who among your customers are the biggest spenders or vice versa.

For instance, if you’re launching a new feature, you can create a segment of customers who spend over $500 a year in your product. You send this data to LaunchDarkly, which then ensures half of this segment gets the new feature (B) while the other half doesn’t (A).

And voila! At the end of the study, you’ll know what your most valuable customers think about your new feature and if it impacted their in-product behavior.

3. Execute, analyze, and iterate

At this point, what’s left is to use your A/B testing tool to set up and execute the A/B test. These tools are so varied that all we can give is a rough overview of the steps involved.

1. With an A/B testing tool, create your baseline and your variation. With tools like Optimizely, you might create two versions of the same landing page.

2. Plug in your audience segments from Segment to inform who you’ll run the A/B test on.

3. Analyze the results, and use those insights to inform future A/B tests.

Choosing the right A/B testing tool

You can A/B test just about every aspect of the customer experience, from top-of-funnel awareness tactics to churn prevention. 

This has given rise to a wide variety of A/B testing tools, each with their own specialties. By analyzing the data of how our user base enables these tools on the Segment customer data platform, we can see that there are some clear favorites. 

But there are many more to explore in our catalog. Here’s are a quick rundown of some of the most popular.

Optimizely

Optimizely is a great all-rounder that enables product managers, engineers, data analysts, and growth marketers to easily leverage real-time, statistically-valid test data to confidently optimize experiences across the customer journey. You can choose what you need or leverage the full platform, including best-in-class front-end and server-side experimentation capabilities, as well as AI-powered personalization and data services. Optimizely’s platform has run over 2 million experiments.   

With Segment and Optimizely, you can do deep funnel analysis to see an experiment’s full impact.

LaunchDarkly

LaunchDarkly is great for in-product tests and optimizations for both front- and back-end features. Their focus is on engineering and product teams looking for a solid rollout plan that tests features and proves value. LaunchDarkly is designed with enterprise businesses in mind.

With Segment and LaunchDarkly, you’ll be able to use a real-time stream of raw event analytics for your experiments.

VWO

VWO specializes in web and mobile optimizations but has in-product testing as well. They have a variety of offerings similar to Optimizely but have been in the product space a little longer. Their feature set also includes heatmaps, visitor recording, and surveys. VWO is designed for medium to enterprise businesses.

With Segment and VWO, you can use test data downstream to measure marketing and revenue impact. 

Google Optimize

Google Optimize has a free version if you’re looking for something quick, but it’s very limited compared to other options. The paid version, Optimize 360, is good if you’re invested in the Google ecosystem.

By using Segment and Google Optimize, you can quickly run your tests using custom dimensions, custom metrics, and content groupings.

Split

By combining feature flags and customer data, Split helps guide your product strategy. You can safely roll out new functionality using sophisticated user targeting, measure the impact of change on engineering, product, and business metrics, and rapidly iterate to refine functionality anywhere in the application stack.

With Segment and Split, you can visualize the impact of each individual feature.

Apptimize

Apptimize, owned by the customer engagement platform Airship, specializes in helping product teams roll out and manage new features, and deliver personalized user experiences.

With Segment and Apptimize, you can tap into your existing Segment user traits and events.

Put A/B testing into action

A/B tests help you perfect the customer experience in increments. They reveal, with certainty, what’s effective and what’s not. Answers like these are invaluable in a time of volatility and change.

Great A/B testing happens when you combine a robust source of customer data, like Segment, with an A/B testing tool suited to your needs, and follow A/B testing best practices. If you’re ready to get going now, we have a few resources to get you started with your own A/B testing experiments.

• Personalize landing pages to a visitor’s interests

• Maximize revenue with personalized pricing plans

• A/B test your tech stack

Create a free Segment account today and get started.