The value of best-in-breed to track and enhance product performance

By implementing and using best-in-class tools you’ll be poised to ensure that (1) your foundational data is accurate and trustworthy and (2) you can act quickly when attempting to answer business questions. 

If we reference the insights funnel above, product performance is fully empowered by your organization's reporting capabilities and infrastructure. With a product analytics tool like Mixpanel and the proper track and identify taxonomy implemented, it’s easy to pinpoint where best to leverage an A/B testing tool like AB Tasty outside of customer testimonials, outside research, and intuition. 

Within analytics teams and boardrooms, conversations occur around how to create a culture of testing. This includes prioritizing automated feedback loops tying statistical evidence to product and marketing changes. 

This idea is broad, and would have a company-wide impact, so it’s important to know the misconceptions about the available tools. 

While an all-in-one solution might seem quick and efficient, testing really isn’t one all-in-one solution. When you stretch one solution over the entirety of the goal, it can limit the potential and accuracy of the testing program.

Instead, we’ve found that you can distill the entire testing process down to three high-level tooling solutions you need to solve for:

1. Data collection through a CDP to provide consistent and accurate data collection

2. A/B testing to make ideas actionable through tests, and then analyze quantitative results

3. Product analytics to first generate hypotheses on what to test, then to tie the tests to your user cohorts and other in-product behavior

With those three areas in place, the following is an example of how you could use Segment, Mixpanel, and AB Tasty.

Steps to Implement

Step 1: Set the business question for what you are trying to improve through testing and analytics reporting

Business questions should be leading this conversation! Data is a seemingly endless flow of information that needs constant attention. Stakeholders need to understand the business value, not feel as if the data projects are out of hand and going nowhere. Once stakeholders are on board, they can implement frameworks that ensure a successful outcome is accessible to an organization for analysis and replication. To help introduce the business question reporting exercise, it's helpful to begin scouring Mixpanel for potential areas that you would like to test over. 

An example of a business question that we can use AB Tasty and Mixpanel to report on is: What is the conversion rate between each step in the funnel? What steps have low conversion rates?

Step 2: Deploy the tooling infrastructure

After spinning up Segment and connecting a client-side source you will then set up connections to AB Tasty and Mixpanel Destinations. 

The AB Tasty destination is currently in Beta as of December 2022, only available on client-side currently and accepts group & identify calls but not track calls. With AB Tasty you can leverage both Segment and Mixpanel data to build Audiences for experimentation and content personalization campaigns. AB Tasty reads in identify calls from Segment, allowing you to build tests within AB Tasty based on the user traits. 

Step 3: Ensure proper taxonomy is being passed into Mixpanel and AB Tasty

On the identify side:

• To properly use this recipe we’ll want to ensure identify calls are made at key activation points like sign-up, log-in, and conversion and user traits contain key user details. This is a requirement for AB Tasty and a necessity for all downstream destinations. 

On the track side:

• In addition to the identify call you’ll need to turn the push integration on so experimentation data is available in Mixpanel. That will allow you to (1) build cohorts for users who are currently in an experiment and (2) build funnels to compare the step-by-step conversion rates of the two experiences. 

Step 4: Deploy a test inside of AB Tasty

To answer the specified question, we’ll want to take your lead or eCommerce funnel and begin to add test ideas to the backlog based on hypotheses. The ideas that are deemed low in effort to create while delivering high value will transition from the idea backlog into a testing roadmap.

At this point, variants for each test will be created. A variant could be anything from adding a step within the funnel to collapsing multiple steps into a single flow. The most important thing is to ensure that the variants created align with key objectives. For example, in testing an add-to-cart button, it wouldn’t make sense to test different texts and shapes if such a design would never be approved for live production in the first place.

Likewise, another important factor to consider for the deployment phase is the amount of traffic to allocate to a test. This is dependent on how much of a risk to the user experience the addition of a test can create. The higher the risk, the lower the traffic that should be allocated while the lower the risk, the more traffic that can potentially be included. With greater traffic allocation, a test can reach statistical significance much faster.

Step 5: Report on statistical significance within AB Tasty and behavioral analytics reporting within Mixpanel for users who have viewed tests

The final piece of the puzzle is to use reporting in both AB Tasty and Mixpanel. Insights around the test performance and their respective statistical significance will live inside of AB Tasty as it will allow you to answer your original business question: What is the conversion rate between each step in the funnel? What steps have low conversion rates?The tool will show you if the test was statistically significant, what the amount of conversion was between the two checkout variations, and if one was more successful than the other. All of this is based on the use of Bayesian inference where the results of each variant are determined based on both the current conditions of a test as well as historical data. 

In this scenario AB Tasty is not your only reporting tool, you have Mixpanel to report and comment on insights in dashboard form. Finally, you have the ability to explore the behavior of your users in Mixpanel to continue to formulate future hypotheses to be tested in AB Tasty.

AB Tasty is a best-in-class testing tool, and Mixpanel provides you with an additional option to communicate your findings and explore your data. By sending an Experiment viewed event to Mixpanel it brings many different ways to segment the impact of the test:

• Funnel report showing the abandonment and conversion rate variance between the two tests

• Segment the funnel based on UTM details 

• Segment the funnel by product or service

• Cohort by users in the test vs those who are not to determine revenue ranges

• Cohort by users in the test vs those who are not to see if it has an impact on sitewide retention

• Create a monthly KPI for the user’s who have taken part in a test

Once you have created all reports necessary you can compile them into a dashboard to provide a model for core reports that all tests should follow. This will help set a model and precedent for future tests. Within a dashboard, you can add text to help communicate your findings.

Wrapping up

In this recipe, we've:

1. Reviewed a foundational learn-first framework 

2. Walked through the advantages of best-in-breed tools vs all-in-one options

3. Implemented Segment to allow for a connective environment between Mixpanel and AB Tasty

4. Added additional reporting capabilities for testing through Mixpanel

If you have any questions about how to use this recipe, please reach out to partner@mammothgrowth.com and we’ll be happy to help. If you'd like help designing and implementing a data strategy to fit the needs of your business, get in touch with Mammoth Growth today!