COURSE 4 • Lesson 1
Launch day only comes around so often. So when it does, you want to make sure your using data to get the most out of this opportunity. Over the years, we've launched countless products and features, and have worked with some of the biggest companies advising them on data strategies for new launches. This chapter outlines some of the most common data pitfalls we've seen heading into launch day, and some easy strategies to avoid them.
You're barreling towards a product launch. There's a new launch energy at team stand-up. Early customer feedback is solid. Your team is cranking on a work-back plan. You think this Launch has the opportunity to be a big breakthrough for the company.
Just as the press goes live and your marketing team hits send on the launch email to announce the new product to your entire customer base, you realize something's wrong.
Your dashboards aren't working. The numbers seem off. The charts aren't moving.
Behind the scenes footage from the Launch Room
Is there a bug somewhere? Are your customers at Burning Man? Did Apple just announce something at the same time?
There's nothing more thrilling than launch day, but there's also nothing more disheartening than a botched launch — or more frequently, not knowing how your launch is being received by the market.
At Segment, we've had our share of new product launches. Over the last few years, we've launched 5 products (analytics.js, Warehouses, Sources, Personas, Protocols) and countless other major and minor features.
Segment product pages! Each represents a major launch day and company milestone
We've also teamed up with some awesome customers — small startups out of YCombinator and large Enterprises like Intuit and Nike — to help build out data strategies for new products and launches.
From these experiences, we've found 7 common launch day data mistakes that end up sidelining a new product launch. These mistakes are easy to avoid and quick to fix if caught early - but they require some investment before launch.
As a product owner, it's easy to look for vanity metrics that confirm our biases. There are so many things we want to be true:
"Our customers love this product."
"We've achieved product-market fit!"
"Our product is changing the way our customers do business."
The problem with vanity metrics is that they allow us to avoid the harsh realities that, if faced, will ultimately lead us to success.
Instead, identify lunch day metrics that will make it impossible to hide from failure and align with end-user value.
For example, when we launched Sources — our out-of-the-box ETL tool to get Salesforce, Zendesk, and Stripe into a data warehouse with a few clicks — we set our key launch metric as the number of accounts with sources data successfully flowing to a warehouse 30 days after launch.
The loooong, complicated funnel to get Sources enabled
To enable Segment Sources, customers had to complete a lot of complex steps. While the early funnel steps indicated intent, completing them did not create value for the customer.
Instead, we pushed ourselves to measure success based on first point in the funnel when a customer could get value from our product. We pinned our success on the first point in the funnel when a customer could get value from our product. Our success == customer success.
This doesn't mean that we didn't monitor launch email opens, landing page views, new signups, or number of new sources enabled on launch day. But it did mean that the key metric we would judge our success or failure was aligned with the value the product was designed to deliver to our customers.
It also enabled us to catch down-funnel issues early, like permissions issues and warehouse setup bugs that wouldn't be immediately obvious if we looked at more superficial metrics.
TL;DR: When you're releasing a new product, write out your launch metrics that matter and confirm whether they align with customer value.
Having metrics that matter — even monitored on a launch dashboard — is not enough to making a launch successful.
To have a successful launch, you need to define what success looks like and be honest about your probability of achieving it.
Our product marketing team has a framework we use for every product launch. We call it the meh-solid-OMG! framework.
The meh-solid-OMG! framework is much as it sounds. For every launch, you describe the post-launch scenario and metrics in which someone at the company would say "meh", "solid", and "OMG!".
This framework allows you to align your team around on the outcome of a launch, and creates a shared barometer for the success of a launch.
You'll need to adjust this framework to your business and product launch, but you'll likely find the core elements should be the consistent…
Metrics: List the launch metrics that matter, from top-of-funnel lead generation, to meaningful product adoption.
Outcomes: Define successful and downside scenarios
Trending: Keep the team up to date with relevant metrics and simple stoplight (green/yellow/red) trends
TL;DR:Ahead of launch day, define a few success scenarios to align the team around different outcomes and their probabilities.
So you've defined the metrics that matter most (link to launch metrics), and you've defined what a successful launch looks like. Now you need to make sure you have the tools in place to measure your success.
First, you're going to want to set up dashboards to measure your launch. Tools like Amplitude and Mixpanel can provide realtime dashboards. For advanced analytics that can tie together multiple channels, we recommend setting up a data warehouse with BI tools like Mode and Looker.
Next, you're going to want to understand where customers are coming from. We've written about attribution tools in the past, and a launch is a great time to get one in place so you understand how different channels are driving product growth.
If you use Salesforce, you'll want to use the launch momentum to feed your sales development team the new leads that are signing up. Segment's SFDC destination makes that easy, but you can also use workflow tools like Zapier and Tray.io.
Lastly, early on you'll want to iterate quickly on the design and user experience, so having a heat-mapping and recording tool like Fullstory in place can help your design team understand where customers are running into issues.
Livechat helps you connect and convert new users, and provides an easy way for confused users to ask for help. On launch day (and often beyond), we recommend the product and success team chat with customers to collect feedback and pave over any rough edges your early product may expose. Drift and Intercom are two popular options here.
If you're integrating these tools natively, you'll need to have them set up before customers start using your product. If you are using Segment, you should be able to enable all these within a few minutes by flipping switches.
TL;DR: Having the right stack in place on launch day can make or break your launch._
Next, you need to define a comprehensive Tracking Plan to make sure you understand your customer journeys with your new product.
We won't spend a ton of time rehashing how to do this, as we've covered the fundamentals of a tracking plan elsewhere, and have a ton of great Tracking Plan templates to get you started here.
There is nothing more frustrating as a product owner than seeing customers not make it through the funnel, but not understanding where or why they fell off. So make sure you've thought through the core steps of your funnel, and are tracking accordingly.
TL;DR: Before launch day, make sure that you have tracking implemented for all of the key steps in your funnel.
As you've read in previous chapters, you can track customer data on the client or the server, or both.
The big trade off is context (or the amount of data you can easily access and gather) vs. reliability (or how much control you have over the actual sending of the data).
The client, being that it's the end-user browser or device, has easy access to user specific attributes, such as cookies, IP address, user agent, referrer, and UTM parameters. However, client-side tracking is exposed to the mayhem of the internet — Ad Blockers and other browser plugins, page unloading, bots, and network issues can all create blindspots in your tracking.
Tracking via the server gives you more control and accuracy, but you lose some of the richness of the device and browser context.
TL;DR: It's best to track mission-critical funnel events on the server.
This one sounds obvious, but you'd be surprised by how often teams get bit by this.
When a new flow goes live, you should test tracking with the same rigor you test the rest of your code and user experience. After all, this is how you'll know that your product is meeting customer needs and whether it's helping your business grow.
We've built Protocols to help automate this step, but if you're just getting started here's a quick checklist you can use…
events and properties follow naming convention standards (e.g. Title Case with object action like Button Clicked)
events are firing across web/mobile, and across popular browsers (via the Segment debugger or showing up in end-tools)
event names are consistent (Order Completed v. Completed Order can wreak havoc on marketing automation and dashboards)
events are not using dynamic names/properties
We recommend doing blackbox testing on your tracking.
This means that you send a user through the product (be it a beta tester, internal team members, or yourself), and check that that experience is reflected on your dashboard. We also recommend you build some exploratory reports off the blackbox test to ensure you'll be able to answer additional questions (like how long it took a user to complete a task, or the path a user took to get to a certain step in the funnel). This ensures you'll be able to reliably answer future questions as they come up.
_TL;DR: QA your tracking before launch day. A little QA upfront can save many hours investigating missing events that weren't fired._
Lastly, no Launch happens in a vacuum.
Your product and company is an ecosystem. Over time, it will grow into a species-rich rainforest, impervious to environmental variability.
But early on, your rainforest may look like more like a few fragile saplings in an empty field. You need to make sure anything you introduce to this ecosystem accelerates growth, and doesn't harm what's already taken root.
So on launch day, remember that your launch is likely going to affect other parts of your product and funnel.
In the short term, how has it impacted top-of-the-funnel metrics, like total visitors and conversion to signup? How many of these customers are becoming trialers, or checking out?
And over the longer term, how has this new product or feature impacted basket sizes, LTV, churn? Are you closing more deals against a competitor? Has this new offering shifted the demographics of your customer base?
While there's bound to be a ton of excitement and investment in the new product, it's important to zoom out to examine how this launch has impacted the business holistically. The last thing you want is to learn is that launch-day growth came at the expense of some other, more important business metric.
_TL;DR: Watch your other dashboards on launch day. Run exploratory analysis to look at how the Launch has impacted slower-moving metrics._
There's nothing better than launch day. It's the great reveal of all your team's hard work, when customers can finally get their hands on your product.
And while avoiding these 7 mistakes won't guarantee the #1 spot on HackerNews and ProductHunt, we hope it'll set you on a path to understanding your customers and growing your business.
Best of luck! The Segment team is rooting for you.
Define the Metrics that Matter Most for your new product
Align on launch day success criteria
Define the important steps in the funnel and name the events
Define what should be tracked client-side tracking
Test your tracking before launch
Monitor impact of new product on other parts of the business
Building a data driven company
Enter your email below and we’ll send lessons directly to you so you can learn at your own pace.