Go back to Blog

Growth & Marketing

Jes Kirkwood on November 15th 2021

Shopify's VP, Growth Morgan Brown reveals how the company's growth team drives results in an exclusive interview.

All Growth & Marketing articles

Diana Smith on December 14th 2015

If you don’t trust your data, it’s useless. This is just one of many helpful nuggets our customer Fareed Mosavat, Senior Growth PM at Instacart, shared with us this week.

If you’re not familiar, Instacart is an awesome service for delivering groceries and is leading the on demand economy with crazy growth.

We took some time with Fareed to discuss challenges that face most data-driven product and growth teams. To Fareed, understanding user behavior is essential for devising, running, and interpreting experiments. And, making data easily accessible in a raw format is the only way the company can measure and achieve their specific goals.

In the Q&A Fareed Covers:

  • How the growth team operates at Instacart

  • When out-of-the-box analytics tools work, and when they fall short

  • Why row-level event data is vital for data confidence and advanced analysis

  • How the team discovered that building a data pipeline to Redshift is harder than it looks

  • Why combining data sources into a single database is the “Holy Grail”

Dive in to the interview below, or get the PDF.

Growth Team at Instacart

Diana: Fareed! Thanks for chatting with us today. Why don’t we start with you telling us a little bit about your role and responsibilities at Instacart.

Fareed: Sure, Diana! I’m the growth product manager. Our team is responsible for consumer growth, so growing our user base, retaining them and activating them through the whole funnel.

Diana: Not a small task! Are you a part of the product team?

Fareed: Yes, It’s a multidisciplinary team, but it’s a product team. We’ve got a designer, a couple engineers and myself. And then we work closely with a bunch of other teams, like analytics and marketing.

Segment loves Instacart

Diana: What’s your main focus right now on the growth team?

Fareed: Number one is just making sure that we have everything that our users are doing recorded, measured, and in a good place. Having our data in order is the only way we can make good product decisions, and Segment helps a lot with that.

Then the second thing we’re really focused on is first-time user activation. So, figuring out what are people doing in their first session, why are they dropping off, when are they dropping off, how can we help them get to their first order.

We know there’s a lot of real-world stuff that happens after their first experience, like quality of service and fulfillment, that are sort of outside of our team’s control. So we’re focusing on getting people that first wonderful experience as quickly as possible, working across the mobile apps and the website.

Switching to SQL

Diana: So I know you’re using Segment Warehouses that loads your user data into Amazon Redshift. Why did you want to do this type of analysis in SQL compared to something like Google Analytics or Mixpanel?

Fareed: Yeah, so we’ve been using Amplitude through Segment for a bunch of stuff, and it’s super helpful. Amplitude helped us look at aggregate numbers, counts, funnels, analyze user segments, and understand how many people are taking certain actions in our product.

But I think there are a couple other reasons why the row-level individual event data, whether it be SQL or somewhere else, is super important.

One is that SQL makes tracking easier to debug. I can watch events fly by in the Segment debugger, but if you have like a lot of data, it’s hard to catch everything. We have very specific taxonomy, and rules, and event names, and everything that need to be correct. With SQL, we can easily diagnose issues like when we forget to pass important traits like userID.

The second is we have a couple of self-defined metrics that are important and specific to the company. And those things tend to be buried in a database somewhere, usually in SQL and sometimes outside of this event data. Being able to merge those metrics and that analysis with our event analytics is really important.

The Holy Grail: One Database. All of the Data.

Diana: What are some of those metrics?

Fareed: Things like quality of service, refunds, consumer support, how much people spend.

We have a lot of steps in our funnel for each order, some of which are online and some of which are not. They are all recorded somewhere, but they happen in different places in the process. The Holy Grail we’re working towards is getting all of the data into one place.

Combining this fulfillment and shipping data with our Segment user behavior data is key for us to connect the dots across the entire customer experience.

With Segment Warehouses, we can put all of the data about how customers are using our apps and websites right into our own Redshift alongside this other data to query as we want.

Diana: What are some of the questions that you’re querying across your Segment data and the internal data to answer?

Fareed: The big one is AB testing and understanding the behavior of users in one test group versus another. We’re currently working towards removing signup from the onboarding flow and making it part of checkout. There isn’t a clear event there, so we need to be able to watch an anonymous user from beginning to end and see their conversion.

We’re measuring what percentage of users that have zero orders, whether they check out on the same day or within seven days, and we want to be able to use any window we want. Those kinds of things are a little bit easier to just define in a set of rules in SQL than it is to try and manipulate a UI to give us exactly what we want.

The second is defining metrics specific to Instacart. Let’s take visitors for example. We have different definitions for visitors: landing page visitors, storefront visitors, and visitors per region. You might be able to figure this out in out-of-the-box tools, but to really trust it you have to know exactly how it was defined. While these out-of-the-box options are great for quick analysis, they tend to be a little bit opaque for understanding session measurement, new vs. returning users, and stuff like that.

Third is making sense of multi-touch attribution. We find that users will visit a couple times before they actually place an order. Previously, we were only attributing that to the last click. But now, because we have all the data, we’re actually able to do a longer attribution cycle and understand how many touches a user had before they actually convert. It’s already been really helpful, but it will become even more important over time as we understand what our marketing looks ROI like and can maximize results from our spends.

On Building a Data Pipeline

Diana: I heard you were building your own Redshift pipeline for a bit before you chose to go with Warehouses. What made you make the switch?

Fareed: We’ve tried a lot of different things here. I think the biggest reason we use Segment is because it gives us the most portability, so we can use whatever services we want and still be able to like keep our instrumentation clean and in a single place. So, we have used a plethora of things.

Our team has played around with S3. The shopper team is using Outbound, we’re using Amplitude, we’ve tried Mixpanel, Google Analytics, tag manager, etc. We try all this stuff, and it’s just flipping switches.

So with Redshift, it was another thing like that where we said to ourselves, “We can bake it off against our own internal system or just turn it on with Segment.” Our main goal was finding something sustainable long term without sacrificing the costs of getting up and running quickly.

Going from schema-less data to like a schema style SQL database turns out to be harder than it looks. With a team of engineers and number of months, we could build something, but it would take a long time and a lot of work to be as fully featured or scalable as Segment Warehouses.

Building data pipelines is not our primary job. But we do need the data in SQL. Luckily, other people have already solved this problem for us.

Plus, it was really important for this data to be in our own Redshift, which Segment could do for us. At the end of the day this is our data, right? Our users took these actions, they did their thing, and it’s important that no matter what we choose from a vendor standpoint that we own this data and that it exists with us.


Thanks so much to Fareed for chatting with us about the growth team at Instacart, getting your data squeaky clean and all into one place to analyze.

If you’re curious about how Segment Warehouses can help you load your web and mobile data into Redshift or Postgres without writing a line of ingestion code, you can learn more here!

Stephen Levin on November 23rd 2015

When your analytics questions run into the edges of out-of-the-box tools, it’s probably time for you to choose a database for analytics. It’s not a good idea to write scripts to query your production database, because you could reorder the data and likely slow down your app. You might also accidentally delete important info if you have data analysts or engineers poking around in there.

You need a separate kind of database for analysis. But which one is right?

In this post, we’ll go over suggestions and best practices for the average company that’s just getting started. Whichever set up you choose, you can make tradeoffs along the way to improve the performance from what we discuss here.

Working with lots of customers to get their DB up and running, we’ve found that the most important criteria to consider are:

  • the type of data you’re analyzing

  • how much of that data you have

  • your engineering team focus

  • how quickly you need it

What is an analytics database?

An analytics database, also called an analytical database, is a data management platform that stores and organizes data for the purpose of business intelligence and analytics. Analytics databases are read-only systems that specialize in quickly returning queries and are more easily scalable. They are typically part of a broader data warehouse.

What types of data are you analyzing?

Think about the data you want to analyze. Does it fit nicely into rows and columns, like a ginormous Excel spreadsheet? Or would it make more sense if you dumped it into a Word Doc?

If you answered Excel, a relational database like Postgres, MySQL, Amazon Redshift or BigQuery will fit your needs. These structured, relational databases are great when you know exactly what kind of data you’re going to receive and how it links together — basically how rows and columns relate. For most types of analytics for customer engagement, relational databases work well. User traits like names, emails, and billing plans fit nicely into a table as do user events and their properties.

On the other hand, if your data fits better on a sheet of paper, you should look into a non-relational (NoSQL) database like Hadoop or Mongo.

Non-relational databases excel with extremely large amounts of data points (think millions) of semi-structured data. Classic examples of semi-structured data are texts like email, books, and social media, audio/visual data, and geographical data. If you’re doing a large amount of text mining, language processing, or image processing, you will likely need to use non-relational data stores.

How much data are you dealing with?

The next question to ask yourself is how much data you’re dealing with. If you're dealing with large volumes of data, then it's more helpful to have a non-relational database because it won’t impose restraints on incoming data, allowing you to write faster and with scalability in mind.

Here’s a handy chart to help you figure out which option is right for you.

These aren’t strict limitations and each can handle more or less data depending on various factors — but we’ve found each to excel within these bounds.

If you’re under 1 TB of data, Postgres will give you a good price to performance ratio. But, it slows down around 6 TB. If you like MySQL but need a little more scale, Aurora (Amazon’s proprietary version) can go up to 64 TB. For petabyte scale, Amazon Redshift is usually a good bet since it’s optimized for running analytics up to 2PB. For parallel processing or even MOAR data, it’s likely time to look into Hadoop.

That said, AWS has told us they run Amazon.com on Redshift, so if you’ve got a top-notch team of DBAs you may be able to scale beyond the 2PB “limit.”

What is your engineering team focused on?

This is another important question to ask yourself in the database discussion. The smaller your overall team, the more likely it is that you’ll need your engineers focusing mostly on building product rather than database pipelines and management. The number of folks you can devote to these projects will greatly affect your options.

With some engineering resources you have more choices — you can go either to a relational or non-relational database. Relational DBs take less time to manage than NoSQL.

If you have some engineers to work on the setup, but can’t put anyone on maintenance, choosing something like PostgresGoogle SQL (a hosted MySQL option) or Segment Warehouses (a hosted Redshift) is likely a better option than Redshift, Aurora or BigQuery, since those require occasional data pipeline fixes. With more time for maintenance, choosing Redshift or BigQuery will give you faster queries at scale.

Side bar: You can use Segment to collect customer data from anywhere and send it to your data warehouse of choice.

Relational databases come with another advantage: you can use SQL to query them. SQL is well-known among analysts and engineers alike, and it’s easier to learn than most programming languages.

On the other hand, running analytics on semi-structured data generally requires, at a minimum, an object-oriented programming background, or better, a code-heavy data science background. Even with the very recent emergence of analytics tools like Hunk for Hadoop, or Slamdata for MongoDB, analyzing these types of data sets will require an advanced analyst or data scientist.

How quickly do you need that data?

While “real-time analytics” is all the rage for use cases like fraud detection and system monitoring, most analyses don’t require real-time data or immediate insights.

When you’re answering questions like what is causing users to churn or how people are moving from your app to your website, accessing your data sources with a slight lag (hourly or daily intervals) is fine. Your data doesn’t change THAT much minute-by-minute.

Therefore, if you’re mostly working on after-the-fact analysis, you should go for a database that is optimized for analytics like Redshift or BigQuery. These kind of databases are designed under the hood to accommodate a large amount of data and to quickly read and join data, making queries fast. They can also load data reasonably fast (hourly) as long as you have someone vacuuming, resizing, and monitoring the cluster.

If you absolutely need real-time data, you should look at an unstructured database like Hadoop. You can design your Hadoop database to load very quickly, though queries may take longer at scale depending on RAM usage, available disk space, and how you structure the data.

Postgres vs. Amazon Redshift vs. Google BigQuery

You’ve probably figured out by now that for most types of user behavior analysis, a relational database is going to be your best bet. Information about how your users interact with your site and apps can easily fit into a structured format.

analytics.track('Completed Order') — select * from ios.completed_order

So now the question is, which SQL database to use? There are four criteria to consider.

Scale vs. Speed

When you need speed, consider Postgres: Under 1TB, Postgres is quite fast for loading and querying. Plus, it’s affordable. As you get closer to their limit of 6TB (inherited by Amazon RDS), your queries will slow down.

That’s why when you need scale, we usually recommend you check out Redshift. In our experience we’ve found Redshift to have the best cost to value ratio.

Flavor of SQL

Redshift is built on a variation of Postgres, and both support good ol’ SQL. Redshift doesn’t support every single data type and function that postgres does, but it’s much closer to industry standard than BigQuery, which has its own flavor of SQL.

Unlike many other SQL-based systems, BigQuery uses the comma syntax to indicate table unions, not joins according to their docs. This means that without being careful regular SQL queries might error out or produce unexpected results. Therefore, many teams we’ve met have trouble convincing their analysts to learn BigQuery’s SQL.

Third-party Ecosystem

Rarely does your data warehouse live on its own. You need to get the data into the database, and you need to use some sort of software on top for data analysis. (Unless you’re a-run-SQL-from-the-command-line kind of gal.)

That’s why folks often like that Redshift has a very large ecosystem of third-party tools. AWS has options like Segment Data Warehouses to load data into Redshift from an analytics API, and they also work with nearly every data visualization tool on the market. Fewer third-party services connect with Google, so pushing the same data into BigQuery may require more engineering time, and you won’t have as many options for BI software.

You can see Amazon’s partners here, and Google’s here.

That said, if you already use Google Cloud Storage instead of Amazon S3, you may benefit from staying in the Google ecosystem. Both services make loading data easiest if if already exists in their respective cloud storage repository, so while it won’t be a deal breaker either way, it’s definitely easier if you already use one to stay with that provider.

Getting Set Up

Now that you have a better idea of what database to use, the next step is figuring out how you’re going to get your data into the database in the first place.

Many people that are new to database design underestimate just how hard it is to build a scalable data pipeline. You have to write your own extraction layer, data collection API, queuing and transformation layers. Each has to scale. Plus, you need to figure out the right schema down to the size and type of each column. The MVP is replicating your production database in a new instance, but that usually means going with a database that’s not optimized for analytics.

Luckily, there are a few options on the market that can help bypass some of these hurdles and automatically do the ETL for you.

But whether you build or buy, getting data into SQL is worth it.

Only with your raw user data in a flexible, SQL format can you answer granular questions about what your customers are doing, accurately measure attribution, understand cross-platform behavior, build company-specific dashboards, and more.


Segment can help!

You can use Segment to collect user data and send it to data warehouses like Redshift, Snowflake, Big Query and more — all in real time and with our simple, powerful analytics API. Get started here 👉

Andy Jiang on November 3rd 2015

Today, we’re excited to share Analytics Academy—a helpful guide to becoming an analytics expert. You’ll start with the basics of how to think about analytics and level up into when is the right time to consider SQL, and more!

The Academy is chock full of best practices we’ve learned by working with thousands of companies on their analytics setup, tool stack, and internal infrastructure.

You can sign up (via email or Slack) for the intro course today! Each week we’ll send a lesson to you on topics including proper analytics implementation, maintaining clean and consistent data, which tools are right for you, and how to leverage your raw data.

Why now?

You may have read Analytics Academy from Segment a few years back. But since then, a few important things have changed.

  • It’s easier than ever to start collecting data and using analytics tools. What’s hard is narrowing your focus and making the data in your tools actually useful.

  • The types of things you can do with your data and the landscape of tools to manage those projects has exploded.

  • On the Segment side, we’ve developed a stronger set of best practices by working with thousands of customers on their analytics setup, preferred stack, and evolving set of data challenges.

And now, we’re bringing these learnings to you!

The new content focuses on a single narrative of analytics—how to use it to improve your product, best practices in data collection, frameworks for selecting a set of tools, and ways to cultivate a data-forward organization.

We hope you find the Academy educational and helpful. If you have any topics you’d like us to cover or ideas on the content, we’d love to hear them! Tweet us your thoughts @Segment!

Diana Smith on October 29th 2015

As mobile user acquisition costs skyrocket, push notifications are becoming a key strategy to keep your users engaged past the install. However, choosing a push notification tool can be a job of its own.

In this blog post, we’ll provide a framework for evaluating push notification tools, cover the top players in the market, and discuss which tools are best for you based on your role, company size, and objective.

Developing your criteria

Before you embark on your journey to find an awesome push tool, you have to ask yourself two questions.

1. What types of messages are you sending?

The first question to answer is what types of messages you want to send. There are two types of push notifications your messages can fall under. Similar to the email world, the first (and admittedly less sophisticated) type of message is “batch and blast” or what is called Mechanical push. You’ll send these messages when users opt-in to notifications for specific types of topics — think Breaking News or San Francisco 49ers score updates.

Also included in the mechanical category are immediate, event-triggered notifications. For example, you might receive a message from Venmo that “Emily just paid you $23 for Ramen,” or from Visa letting you know that “Your credit card was just charged”. The key with understanding Mechanical push is that they are immediate, list-based, or event-triggered notifications.

If Mechanical messages are the only types of notifications you want to send and you’re looking for other developer services like crash reporting, going with Parseis a good bet for a team of developers. It’s the Mailchimp equivalent for the mobile world — lightweight and easy to get started with.

The second type of notification is Behavioral, when you send a personalized message based on past activity someone has done in your app or real-time information like location. These messages are often super personalized and deep link into a particular, contextual page of your app. For example, Steve Madden sends you a message that those boots you were browsing last week are now available in your size. When you tap in, the message takes you right to the product page for the shoe with your size checked.

Behavioral messages, in both the email and the app world, perform much better, and it’s not rocket science to figure out why! The messages are highly targeted, timely, and personalized. And, basically every provider that does behavioral messages, also supports mechanical. They usually compete on scale, reliability, and functionality of behavioral triggers.

There are only few reasons you might not want to use Behavioral notifications: Your app is very transactional, or you’re resource constrained. Perhaps you don’t have technical help to track behaviors your customers are doing in your app or time to set up these campaigns.

2. Do you want an all-in-one or focused solution?

The next decision you’ll have to make is if you want a tool that focuses completely on messaging, or if you want something that also offers product analytics and in-app a/b testing.

There are pros and cons to each.

Going with a tool that’s focused on messaging means that the team you’re working with is fully invested in making that experience amazing. Most tools in the messaging category have full-featured options for sending personalized messages, reporting on campaigns, a/b testing copy, and delivering notifications at the right time for each user. Many of the players in the space who started in push are starting to offer email, so look out for that if you’re interested. Kahuna, and Appboy are the top players here.

That said, Outbound and Iterable are also good options to check out if you’re looking for a cross-platform tool covering push, email, and SMS. These platforms subscribe to the belief that you should first think about where your customers are getting stuck in your product and then use the right channel to send them a message if they don’t take the next step.

Another type of tool is what we call the “all-in-one” mobile marketing and product suites. They offer push and email services, but also do in-app a/b testing, and funnel analytics. If SDK bloat is a big issue for you, these types of tools including MixpanelUrban AirshipLocalytics, and Leanplum might be a better choice.

In terms of pricing, most tools will let you send up to 1,000,000 messages or communicate with 10,000 users for free or on a trial. Then the cost levels up or you have to call to get enterprise pricing. For many of the all-in-one solutions, the push notification and other marketing features are priced as an add-on to the core analytics offering.

Now that you have an idea of your options, let’s get into the nitty gritty of what each of the popular services offer.

The pull of push tools

Kahuna

Kahuna focuses largely on behavioral push notifications. If you are a marketer in the ecommerce space and mobile drives most of your business, Kahuna is a great tool for you. Most of their features help you deliver hyper-personalized communication, from pre-crunching user segments with machine learning to offering delivery at the right time based on a user’s past engagement. Because this segmentation is useful elsewhere, they have also recently added email and Facebook audience targetting capabilities. Kahuna is a great option if you are a part of a larger company (no self-service plan) with significant downloads and are working on engagement and retention. Their features include:

  • Dynamic deep linking that takes users to unique place in the app based on their past behavior.

  • A “ghost push” feature to track which push notifications are causing people to uninstall or opt out of any message in any campaign that helps you find the line between helpful and spammy.

  • The option to use their pre-built segments of new, dormant, active, and inactive users or create custom segments based on an unlimited number of events and attributes.

  • A-E testing and message send time optimization to make sure the highest performing message hits your audience at the right time for each individual.

Kahuna is great for

  • Role: Marketers and Product Managers

  • Customer Company Size: 200-10,000

  • Monthly Active Users: 50,000-25,000,000

  • Objective: Drive customer lifetime value through repeat purchases.

  • Industry: Commerce, Media, Travel

Urban Airship

Urban Airship has over 30,000 apps using its service, from startups to large organizations like Walgreens, ABC News, Alaska Airlines and Airbnb. Helpful for both developer and marketing teams, they offer well-documented APIs for mechanical messaging as well as behavioral targeting capabilities that allow marketers to define segments and deliver highly relevant, real-time messages. Urban Airship offers numerous messaging types such as interactive push notifications, in-app messages an in-app inbox (message center), and mobile wallet. It also offers audience intelligence, funnel app analytics, a-b testing, and a user-level data streaming service to power behavioral messaging in other channels like e-mail, ad platforms and more.

Urban Airship offers:

  • A full suite of mobile messaging and content publishing tools including interactive push notifications, in-app messages, rich landing pages, mobile wallet, and sport for rich messaging (images and video).

  • Easy to use campaign tools and templates that allow you to add a deep-link, develop a landing page, social share, and define segment attributes for any message.

  • Segmentation tools that allow you to personalize your messaging based off in-app behaviors, location (triggers and history), preference center, app events as well as cross-channel data from a CRM or other system.

  • Audience intelligence to identify user-level trends for future campaigns. App analytics are also provided with funnel analysis, conversion reporting, cohort analysis and RFM reporting.

  • A mobile data streaming service that allows you to send user-level information to business systems for omni-channel behavioral targeting and analysis.

Urban Airship is great for

  • Role: Developers, Product Owners and Marketers

  • Customer Company Size: 100-10,000

  • Monthly Active Users: 1,000-50,000,000

  • Objective: Grow and retain your mobile audience.

  • Industry: Retail, Travel, Media, Financial Services

Parse

Parse, recently acquired by Facebook, is a developer friendly platform for push notifications and analytics. If you’re concerned about the health of your app, want to send basic notifications, and price is a factor, Parse is a good option. Beyond basic push features, they will also help you monitor and investigate bugs and crash issues. With Parse, you can

  • Segment audiences based on age, location, and language, and schedule messages in advance.

  • Preview notifications exactly as they will appear across the devices you target.

  • Send notifications via the web portal, REST API, or client SDKs.

  • Monitor the effectiveness of your push campaigns open rate analytics.

  • Confidently evaluate your push messaging ideas with A/B tests to create the most effective, engaging notifications for your app.

Parse is great for

  • Role: Developers

  • Customer Company Size: 25-2,000

  • Monthly Active Users: 500,000-20,000,000

  • Objective: Send basic notifications to customers and understand what is making your app crash.

  • Industry: Gaming, Education, Entertainment

Appboy

Appboy is a communication platform built for mobile-first marketers. If you have mobile product, but email and in-app notifications sometimes make sense, Appboy has a pretty comprehensive set of features for tackling these channels. Beyond the basics of event-triggered notifications and batch “product update” style in-app messages, Appboy will help you target messages based on a variety of customer attributes. They do very well in the messaging and media industries. Using Appboy you can,

  • Target content based on detailed user profiles and past in-app behavior, such as the frequency of app use and number of in-app purchases.

  • Set up user action goals for each message and measure conversions.

  • Create groups of users based on their location to send geo-located messages.

  • Send messages based on when each user will most likely engage with your app.

  • Automatically insert details like nearby movie times, customized recommendations and proprietary content directly into messages.

Appboy is great for

  • Role: Mobile Marketers

  • Customer Company Size: 100-1,000

  • Monthly Active Users: 50,000+

  • Objective: Drive mobile-centric customer engagement, retention and advocacy.

  • Industry: Commerce, Media, Messaging

Outbound

Outbound is a newer tool that’s great for smaller sized companies and startups in the on-demand economy. If you’re laser-focused on getting folks through your funnel, Outbound makes it easy to send messages based on what a user has and hasn’t done in your app. If you have a web-first product with complementary mobile apps, Outbound will help you communicate across email, SMS, and push. You can easily customize notifications based on the device each user prefers. Outbound helps you

  • Set up automated, trigger-based campaigns based on what users do in your app for email, push, and SMS.

  • Send broadcast campaigns with a one-time message to a segment of your users based on their history.

  • Track any action your users take in your website or app and immediately trigger an event, without creating a segment in advance.

  • Automatically update user info — like new email address — with the Outbound API

  • A/B test if email, push, or SMS performs better for a particular message

Outbound is great for

  • Role: Growth Marketers

  • Customer Company Size: 5–200

  • Monthly Active Users: 50,000-1,000,000

  • Objective: Activate users for a cross-platform product.

  • Industry: On-demand, Financial Tech, Marketplaces

Iterable

Iterable started as a marketing automation platform and now also offers push notifications and SMS. It’s best if you’re working with a consumer audience, want to send messages across web and mobile channels, and use both mechanical and behavioral messages in your camapigns. They have a visual workflow for designing lifecycle and engagement campaigns, which makes it easier to map out your messages.

Iterable is great for

  • Role: B2C Marketers

  • Customer Company Size: 50 - 5,000

  • Monthly Active Users: 20,000 - 20,000,000

  • Objective: Send multi-channel messages for both blast and behavioral campaigns

  • Industry: Commerce, Education, Consumer

Leanplum

Leanplum started in mobile a/b testing but has moved into other parts of mobile optimization including analytics and push notifications. They are best for mobile-first companies that have over 100,000 MAUs and are looking to measure how push notifications affect key performance indicators down funnel. They offer a bunch of helpful features for behavioral push notifications and enable you to a/b test mobile designs without resubmitting to the App Store. With Leanplum you can,

  • Trigger personalized messages based on user attributes and specific in-app behaviors.

  • Programmatically tailor the user interface to individual users, for example, switch the default share option to a user’s favorite social platform.

  • Insert custom values within variables and messages to make them feel very personalized.

  • Send messages at the most optimal time, based on each individual user’s past usage pattern.

Leanplum is great for

  • Role: Marketers

  • Customer Company Size: 200-1,000

  • Monthly Active Users: 100,000-10,000,000

  • Objective: Drive retention and engagement with event-driven push notifications.

  • Industry: Travel, Media, Retail

Localytics

Localytics is tailored for companies that plan their communications based on how people are interacting in their app. It’s best for commerce, entertainment, and media apps. In Localytics, the push and marketing features are very connected to their analytics insights, meaning that they encourage you to message users who are dropping off along your funnel. They have attribution, cohort, and funnel reporting to help you understand those drop offs, as well as email and push message options. Localytics helps you:

  • Target campaigns to an existing segment of users, or create an ad hoc filter for each campaign.

  • Schedule campaigns to go out immediately, at a single point in the future, or set up an automated campaign that sends on a recurring, scheduled basis to newly qualified users.

  • Test up to 5 message variants for each campaign and see what wins.

  • Track impressions, clicks, and conversions over time for each campaign, and set up funnels to see how users from each campaign engage overtime.

Localytics is great for

  • Role: Growth and Enterprise Marketers

  • Customer Company Size: 250-500

  • Monthly Active Users: 50,000-50,000,000

  • Objective: Drive engagement across the entire user lifecycle.

  • Industry: Commerce, Entertainment, Media

Mixpanel

Mixpanel is best known for delivering super helpful in-app event analytics. Their funnel analysis, engagement and retention reports are polished and easy to use. Over the past few years, they’ve expanded their scope to include email and push messaging as well as mobile a/b testing. If you’re a product manager who already really enjoys Mixpanel’s analytics offering, using them for basic engagement and retention projects could save you some time. Here’s a look at their push features:

  • Send emails, push notifications, in-app notifications, or SMS text messages based on which platforms people prefer using your product.

  • Set your messages to go out at optimal time, and Mixpanel will tailor that to each user’s timezone.

  • Experiment with your notifications by a/b testing the subject in an email or the message in a push notification.

  • See how your messages perform against conversion events you’re already sending to Mixpanel.

Mixpanel is great for

  • Role: Product Managers

  • Customer Company Size: 5–500

  • Monthly Active Users: 1,000-1,500,000

  • Objective: Improve funnel conversions with personalized notifications.

  • Industry: Consumer Apps, Social, Media

Bundling and Unbundling

We hope this deep dive gave you a better idea of which push tool is best for you!

As you can see from this list, many of the push platforms are trending toward bundling analytics and marketing “jobs to be done” into a single solution. However, it will be interesting to see if this continues.

In the history of web analytics and email tools, we’ve seen single point solutionseat away market share from monolithic suites, and predict this trend to continue in mobile. The biggest factor to watch is how technology to reduce SDK bloat evolves.


P.S.

If you’re interested in exploring these push tools, you might want to look into Segment. We make it easier for you to try push notification, analytics, and optimization services. Instead of integrating each SDK one by one, you can collect customer interaction data with our API, integrate one SDK, and then flip a switch to integrate new tools. (No submitting to the app store!)

We currently offer most of these push tools on our platform. You check out our full list of integrations here, and request support for new tools here.

Andy Jiang on October 13th 2015

As a data infrastructure company, we collect a lot of data for the purposes of -airquote- “making better decisions.” But we’ve found, that just having this data doesn’t help. If our team is going to use it on a daily basis, we need to put it right in front of their faces.

For our sales and success teams, this means embedding useful customer data directly into their workflows in Salesforce and Zendesk. This gives them the context of each user, so they can be more helpful in 1:1 conversations.

This will be a two part post! The first part will go over how we embed custom dashboards in our Salesforce to help our team better anticipate the needs of our customers. The second post will be how we modified our Zendesk to minimize resolution time and to win at customer service.

Anticipate Users’ Needs

At Segment, we give our customers the ability to “raise a hand” to kick off the sales process. Some of these “hand-raises” are explicit, like sending an email asking for a sales demo or chatting with our sales team on Olark.

But some of the signals are more subtle: turning on a business-level integration or a sudden spike of API calls beyond their plan. As a result, our Salesforce needs to look a lot more like a mission control center than a CRM.

To minimize any friction for the sales team to use the data, we pull all relevant account information right into their workflow. Using iFrames and Periscope, we’ve added custom dashboards to our Salesforce Account views. This way our sales team can be proactive about reaching out to our customers and have the right context to be helpful.

Embed Custom Dashboards in Salesforce

If you use Salesforce and Periscope, then here are the steps to set this up.

Other SQL visualization and query tools that allow embeddable reports include Mode, Chartio, and Looker. Please shoot us a tweet if we’re missing any!

1. Setup Periscope queries with product usage charts that can be filtered by account

We say “product usage” charts, but any chart that makes it obvious when one account has a higher value than most others would work. The goal is to help sales team members easily determine the value of the account from a glance. A basic approach would be to look at product usage, but you could also look at # of livechats, marketing content engaged with, or something like that.

For us, our charts measure product usage via API calls, number of projects, and number of integrations. Here is the query we use to generate API calls per week. (Note: group_filter is the filtering parameter that will take the Salesforce account info to return the corresponding chart):

..which generates the top chart in this screenshot. (I added a few random group_filters so not to reveal any sensitive usage information):

Great. Now we have to make these charts available to external domains. We achieve this by clicking “Edit Chart”, clicking the “Settings” tab, looking for “URL for CSV” on the right side, and checking off the “Enable API Access” box.

Note that the gif above is a report generated with a randomized set of data.

This will allow other domains to access the chart via this URL, which is important since we want to embed it in an iFrame in Salesforce.

2. Create a VisualForce Page with the embedded Periscope chart in Salesforce

In order to add custom visual pieces in Salesforce, you’d need to create a Visualforce Page. You can create a new Visualforce Page by first clicking “Setup” on the top right, then “Develop” and “Pages” on the left panel.

After clicking “Pages”, you’ll be sent to Visualforce Pages site, where you can create new ones or edit existing ones. Click “New” to create a new Visualforce Page.

On this page, you can write Visualforce Markup in the textbox that’ll become the HTML on the Salesforce page. Visualforce Markup consists of Visualforce tags, HTML, JavaScript, or any other web-enabled code that defines the user interface components and their appearances.

It’s in here where we add the following code:

Two things to note about this code.

Retrieve the Embeddable Periscope Dashboard

The function periscopeUrl creates a SHA signature of the apiKey with sha.js and a properly formatted Periscope URL that’ll retrieve the right dashboard.

Additionally, the data object that is passed to the periscopeUrl function follows Periscope’s embed options API requirements. The key here is to pass the appropriate filters that ensures the returned dashboard is only for this specific account. We pass {!Account.Name}, which grabs the name of the account from Salesforce.

More information on Periscope’s embed API.

Set up the iFrame

At the top, we have a div with id as container,which will be the container for the iFrame.

We assign the .src property of a new iFrame element to the URL generated by the function periscopeUrl (the .src property specifies the URL of the document to embed in the iFrame).

Then, we append the iFrame element to the #container div. The embeddable dashboard from Periscope, which is represented by a URL, now shows up in this Visualforce page.

3. Embed the Visualforce Page into the Salesforce Account view

Now that you have defined and saved a new Visualforce page, you can add this wherever you’d like in Salesforce.

In the Account page in Salesforce, click on “Edit Layout.” In the top most box, you’ll find a “Visualforce Pages” option on the left. Clicking it will show you the Visualforce Pages you have defined. Find the one you made and drag it onto the page. Finally, hit save!

Boom! Now your sales team is armed with the data they need in real-time!

Leverage Contextual Data

Aside from account usage data, we use a few other tools that help sales reps prioritize accounts and identify the right game plan.

  • ClearBit saves our sales team from having to visit a number of different websites (LinkedIn, Crunchbase, Twitter, etc.) to figure out the specifics of the company to whom they’re talking. Clearbit pulls in information such as and not limited to title, employee count, and even the technologies already installed on the user’s website. Ultimately, the information provided by ClearBit helps us prioritize and optimize our sales resources.

  • Infer helps our team immediately prioritize signups from the companies that are most likely to become our customers. Infer analyzes our historical closed transactions along with qualitative information they gather on our prospects via web-scraping to create a predictive model. Our team now gets an alert as soon as someone from a company that is likely to convert to a customer signs up for Segment, so that they can respond in less than 30 minutes.

Sure, keeping your data in a database for your product team to analyze is good, but that probably doesn’t help the other folks on your team. We’ve learned that collecting and tracking data isn’t enough. We’re continually looking for new ways to make that data actionable by surfacing it where our team makes daily decisions, like Salesforce, Zendesk and more.

How do you have your Salesforce lead or account view setup? Any custom widgets or dashboards you are using? Let us know on Twitter!

Part two of this post about customizing Zendesk to minimize resolution time can be found here.

Andy Jiang on August 27th 2015

Here at Segment, we love frameworks and data. Frameworks help us organize and understand the world, and data helps us stay focused and monitor progress. So, it’s no surprise we use them both to help us project future growth and figure out how to hit our lofty goals.

This is the framework to understanding what vibes the hardest at Segment HQ.

In this post, we’ll share our approach to understanding our growth dynamics, which involves building a mathematical model for marketplace dynamics and aligning internal resources to focus on growth goals. You can’t really get more data-frameworky than this!

Growth Dynamics of Two-Sided Marketplaces

To start from the beginning, we’re a two-sided marketplace with customers on one side and partners on the other. This business model strongly affects how we grow.

For marketplaces like us, growth is driven by “network effects” where the value of the marketplace increases as usage increases. If you look around Silicon Valley for “unicorns,” or some of the fastest growing companies, you’ll no doubt find a few marketplaces. Uber and Airbnb have each grown to over $20B valuations in less than 7 years.

Marketplaces are awesome because, without them, buyers and sellers face complex, risky, and time-consuming transactions. For example, without Lyft and Uber, you hope a random taxi drives by at just the right time. Meanwhile, the cabbies roam around, hoping to spot someone frantically waving their arms in need of a ride. This is a crazy process for connecting riders and drivers, and it’s clearly an inefficient marketplace, especially in cities like SF where the volume of taxi drivers does not meet the demand from riders:

Uber and Lyft significantly reduce that complexity. Riders press a button to hail a driver to their GPS location, and the pair rate each other after each ride to improve trust and safety. Drivers don’t look for people waving their arms, they just wait for their phone to tell them exactly where their next ride begins. In our case, we bring together businesses that care about data and SaaS products that run on data with one API to use them all.

So, thanks to the marketplace, transacting is dramatically more efficient:

The value of the marketplace comes from the ease of transaction, marked by the availability of buyers and sellers on either side of the exchange. In other words: the value increases with the number of participants, attracting even more buyers and sellers.

Marketplace Dynamics

Knowing we’re a marketplace, we borrowed heavily from this HBR article that outlines a rigorous way to model marketplace growth with six separate dynamics:

Buyer-to-Seller Cross-side — prospective buyers tell prospective sellers that they prefer to do business on the platform. “It was hard to find your place. How come you don’t list on Airbnb?”

Seller-to-Buyer Cross-side — prospective sellers expose prospective buyers to the platform. “Buy my adorable elephant mittens on Etsy.”

Buyer Same-side — buyers love the new transaction experience, and tell other prospective buyers to use the platform. “Why would you use a taxi? You should check out Lyft.”

Seller Same-side — sellers love the new transaction experience, and tell other prospective sellers to use the platform. “I made a good chunk of change while I was on vacation, renting my place on Airbnb. You should try it out!”

Direct to Buyers — the marketplace tells buyers about itself directly. “Wow, Uber has a lot of billboards here.”

Direct to Sellers — the marketplace tells sellers about itself directly. “I searched for jobs in Cincinnati and found Lyft.”

Build a Numerical Model

These internal growth dynamics let us put together a simple numerical model. The model uses the current size and growth rate of customers and partners as inputs, and then applies each of the six dynamics to project future performance.

Here is a quick glance at an example growth model using fake data. (If you want to see our actual numbers, click here). Below you can see the representations of the six growth dynamics:

Using the model we’re able to see a range of forward projections based on conservative and aggressive goals. The chart below details two sample scenarios, the Base Case and Growth Case (again, totally made up here for the sake of example!).

The Base case is a conservative, keep-everything-status-quo projection. Based on historic figures, there is already some cross-marketplace network effects going to work, which is great. As a result, this case predicts ~1.26 million customers by the end of the period with a steady ~%4 m/m growth and a total of %58 growth.

Not bad.

The Growth case, alternatively, assumes that we will figure out ways to get more customer signups from our partners (0.4 gradually growing to 10 new customers per partner) which represents the Seller-to-Buyer cross side effect.

This case predicts ~1.44 million customers by the end with total of 80% growth. Better! And this is only adjusting one assumption: the new customers per partner cross-side dynamic.

Define Marketplace Participants

Let’s walk through how this model was put together with our fake data, so you can do the same if you have a platform business.

To start playing around with the model, you can copy this handy sheet we built for this example. 😃

The model contains three tabs that are split up into these groups:

  • Aggregate top-level numbers (total customers and partners) for the entire platform

  • Growth dynamics for customers

  • Growth dynamics for partners

First we defined the participants of each marketplace, as this helps us measure their sizes to put into the model.

For Segment, the definitions are:

  • Customer: somebody who registered an account with us

  • Partner: an integration on our platform

We added these historic numbers to sheets 2 and 3 at the top section (reminder: fake data!):

Then, we looked at historical attribution numbers in order to determine reasonable rates for each growth dynamic.

Examine Historical Data

We’ve been fortunate enough to have been attributing our sources for customers and partners (e.g. recording where they came from) for a meaningful time period. This data is necessary to backfill historic cross-side and same-side growth dynamics.

  • new customers per partner: total partner referral signups

  • new customers per existing customer: signups that mentioned “friends” as how they heard about us

  • customers direct growth: signups attributed to content, paid marketing

We filled them here in the green highlighted cells (note that the blue font indicates the cell is calculated and the black font is hard coded):

Define Growth Assumptions

Once we have historical data, we filled in the future assumptions.

We determined the assumptions by looking at the historic range for that metric, thinking about the factors in our business or situation that impacted that metric, and estimating what that metric will be in the next few months.

Then we continuously adjusted them, so the overall top level growth numbers (aka sheet 1) looks reasonably achievable for us. Sanity checking the assumptions is critical, so take time to revisit them to make sure they make sense.

Let’s take use the Partner Direct growth rate as an example. Historically in this example, it has been 0%. Let’s assume in these past few months, we did nothing with regards to attracting new partners to the platform. However, we plan to put way more focus in this channel: developer evangelism, outbound business development, etc. From that information, a growth case assumption for this rate would be 30% or more.

If there is no historical data to help set the assumption, then ballpark guesses are acceptable, so long as you continue to tweak it so that the overall outputs of the model passes a sanity check later on. Ballpark guesses based on anecdotal stories can be helpful: “the majority of buyers who I talk to say they heard about us from their friends” would make the case that same-side buyers are a strong driving force.

Project Growth and Set Goals

Once we settled on reasonable assumptions, we projected future growth by applying the dynamic rates and used these to set goals. The dynamic nature of the model helps us to easily visualize how growth is impacted by different scenarios. We chose our goals after seeing where our efforts would yield the biggest change; for example, if spending more time on direct customer marketing or figuring out how to drive signups from our partners would have a greater effect.

To identifying these key opportunities, or the dynamics that would make the biggest impact with the smallest improvements, we tested different projections in the model.

Approaching this systematically, we came up with six different scenarios. Each scenario uses conservative estimates for all six assumptions except for one, which is aggressive. Going through this process, we could see which dynamic had the highest impact change in number of customers.

As a thought exercise, here are some intuitive examples of growth opportunities based on your company’s stage.

For earlier companies, it is likely that the cross-side value hasn’t reached a critical tipping point yet; i.e. the size of one marketplace is too small for it to be valuable to the other side. You likely have a classic the chicken or the egg problem. In these situations, it is wise to devote resources on propping up or subsidizing one side of the marketplace via its direct channel, of which the company has complete control (“more marketing spend will yield more users”).

In Segment’s early days, the team subsidized the partners side by building and maintaining the first 100+ integrations. The selection of integrations made Segment appealing to customers. However, now we’re fielding inbound requests from newer partners to be added to our platform—these companies see Segment’s value as a distribution mechanism to acquire users cheaply.

For marketplaces that are already experiencing positive cross-side effects, strengthening these effects can often lead to much higher growth (vs. focusing on a direct channel), due to their compounding impact.

Ultimately, once we identified which channels needed the most love, we could set more granular growth goals, such as “get X customer sign ups from partners”. With clear, quantifiable objectives that we can measure over time, it’s easier for us to stay on track!

Align Company Growth Objectives

As a result, we evaluate every effort in our Product, Marketing, and Partners teams within the platform model. We use the model as a map, helping sequence and compare alternative tactics.

Because our customers are our source of revenue, we’re focused on channels for “customer direct”, “customer same-side”, “partner-to-customer cross-side.” At our current state, partner-side growth dynamics aren’t as important until we have a fully functioning “customer-partner cross-side” effect.

We’ve used these growth channels to organize our resources: our marketing team owns the “customer direct” channel, product team owns “customer same-side”, and our partners team owns the “partner-to-customer cross-side” channel:

Customer Same-Side

  • Team: Product

  • Goal: enthuse our customer base to refer more

  • Metric: # Monthly signups attributed to “friends” or “colleagues”

Customer Direct

  • Team: Marketing

  • Goal: find and convert more fresh customers

  • Metric: # Monthly signups attributed to our content, paid acquisition

Partner-to-Customer Cross-Side

  • Team: Partners

  • Goal: experiment to figure out how to make this dynamic work

  • Metric: # Monthly signups attributed to our partners

We’re able to attribute the signups by asking users where they heard about Segment when they’re signing up.

Now, evaluating or comparing new projects is much clearer when your team’s goal is tied to one metric. We can just ask ourselves if the project in question will positively impact the metric, by how much, by how soon, etc.

Moreover, assigning ownership of metrics to different teams is an excellent way to modularize their efforts and keep them concentrated on a specific target.

Look Beyond the Metrics

Though this was a simplified walk through of how we currently think about our business and its growth challenges, we believe this step-by-step approach helps separate signal from noise. Instead of being overwhelmed by tracking and tackling everything under the sun, we can focus on the growth metrics and initiatives that matter.

But it’s worth noting that metrics and data aren’t everything. We can manically chase after our targets, but we risk doing so at the expense of overall quality. If we’re only concerned with page views, then our content may erode to listicles of corgi gifs. Similarly, if we focus only on sign ups, those leads may be of lower quality and never end up activating.

As such, there are mechanisms outside of this growth model that help inform our decision making. We consider quantifiable parameters, such as activation and churn rates (deliberately omitted in this example and blog post for simplicity). We also set qualitative goals, such as “Build a brand people love” (a top-level marketing team objective to complement its other goal to get more signups) and “Karma” (one of Segment’s values, against which all decisions are made, even on an individual level). These safeguards help us make sure that we don’t just amp up our numbers, but that we build a sustainable, fundamentally useful business.

David Shackelford on July 22nd 2015

At PagerDuty, reliability is our bread and butter, and that starts with the first time users interact with our platform: our trial onboarding.

Improving onboarding

(For a little background on us, PagerDuty is an operations performance platform that helps companies get visibility, reliable alerts, and faster incident resolution times.)

We’d heard from customers that our service was simple to set up and get started, but we also knew there was room for improvement. Users would sometimes finish their trials without seeing all of our features or wouldn’t finish setting up their accounts, which caused them to miss a few alerts. We wanted to help all of our customers reach success, and we knew if we did this well, we’d also boost our trial-to-paid conversion rate.

We talked to our customers, support team, and UX team before diving into the onboarding redesign, but we also wanted to complement their qualitative feedback with quantitative data. We wanted to use telemetry (automatic measurement and transmission of data) to understand exactly what users were doing in our product, and where they were getting blocked, so we could deliver an even better first experience.

Measuring the baseline

Before we started making changes, we needed to establish a baseline: how were users moving through the app? Did their paths match our expectations? Did the paths match the onboarding flow we were trying to push them through?

After researching the user analytics space, we found Mixpanel and Kissmetricshad the best on-paper fit to answering these types of questions. However, the investment (in both money and implementation time) to adopt these kinds of tool was significant — so much that we wanted to test both to make sure we picked the right tool. But, the only way to comprehensively test tools like this is to run them against live data.

That’s where Segment came in. We were excited to find a tool that let us make a single tracking call and view the data, simultaneously, in multiple downstream tools. Segment made it easy to test both platforms at the same time, and with more departments looking at additional analytics tools, the prospect of fast integrations in the future also excited us.

Analytics matchup

We used our questions about user flow and conversion funnels to test if Kissmetrics and Mixpanel could help us understand the current state of affairs. Using Segment, we tracked every page in our web app and as many events as we could think of (excluding a few super-high-volume events such as incident triggering). Then our UX, Product, and Product Marketing teams dove into the tools to evaluate how well they could answer our usage questions.

After spending a few weeks with both tools, we went with Kissmetrics. To be honest, they’re both great, but we liked the funnel calculations in Kissmetrics a bit more. They also offered multiple methods for side-loading and backloading out-of-band data like sales touches, so Kiss was the winner.

Throughout this process, we also learned that we should have tracked far fewer events. If we were to do it again, we’d only collect events that signaled a very important user behavior and that contributed to our metrics for this project. It gets noisy when you track too many events.

Improving the experience

After combing through the data, our user experience team had a lot of ideas to develop a new onboarding flow. We mocked up several approaches and vetted them against both customers and people inside the company. Then, we tried out the design we thought would best communicate our value proposition and help customers get set up quickly. Our approach included both a full-page takeover right after signup, as well as a “nurture bar” afterwards that showed customers the next steps to complete their setup.

After implementation, we tracked customer fall-off at each stage of the takeover wizard to see how we were doing. We also measured “Trial Engagement,” an internal calculation for when an account is likely to convert from their trial (what others call “the aha moment”). Using Kissmetrics, it was very easy to measure how the new design was working against this metric.

Old:

New:

The results

After shipping the new experience, we saw a 25% boost in the “engagement” metric mentioned earlier, measured using Kissmetrics’ weekly cohort reports. Kissmetrics showed us that with the new wizard, new users were actively using the product earlier in their trial and using more of the features in the product. In addition, far fewer new users were ending up in “fail states,” such as being on-call without notification rules set up.

Since then we’ve run various experiments on the trial funnel, and the weekly cohort report has really helpful for looking at the effects of those experiments and determining whether our changes are actually helping users.

Qualitative feedback has also important to get a full picture of how the changes to the product affect the user experience. We’re pretty low-tech in this regard — I dumped a list of users I pull from Kissmetrics into to CSV, then sent them a quick micro-survey to see if anything was unclear with the new onboarding. We also gathered internal feedback from our sales team and support teams to confirm that customers were finding our new experience easy to use and understand.

To get more context on how people are using PagerDuty, we also route Segment data to Google Analytics, which is a great way to look at meta behavior trends. Kiss can tell you who is doing something, but Google Analytics is a little better for asking questions like “Which features get used the most?” or “What Android versions are most of our users on?” Google also manages top-of-funnel analytics (the marketing website) a little better, while Kissmetrics is more powerful once the user is actually in trial.

Next steps

Since our initial work on the wizard, we’ve expanded our use of analytics to look at behavior of both trial accounts and active customers. Whenever I’m about to start work in a particular area of the product, I’ll use Kissmetrics to pull a list of highly active users in that area, and then reach out to them to understand how they’re using our features and what their pain points and goals are. We also implemented mobile tracking with Segment because some of our customers mainly use our service through our mobile app, and installed the ruby gem for code-level tracking.

There are plenty of improvements we’d like to make to our onboarding, but since it’s doing pretty well right now, our next project is going to be investigating simultaneous A/B testing. We move fast, and if we’re running a sales or marketing initiative alongside product changes, sometimes it’s tough to sort out what impacted what. Split-testing trial experiences should let us get cleaner data about how our onboarding redesign is improving our trial users’ experience, and ultimately help us make better decisions about the ideal experience.

What we learned

Like any new initiative, we learned a lot when implementing analytics for the first time. Here are some of our takeaways — hopefully they’re helpful to you as well.

  • Choosing what to track is an art — too few events and you may miss a key action; too many and it gets really noisy. Segment hadn’t yet shipped their “Tracking Plan” feature, so we had to manage our list of tracked events in Google Docs. It wasn’t pretty — in fact, had we done it again, I would have started a fresh new account after we finished our evaluation, tracking far fewer events.

  • Using separate production and development environments is absolutely key, in both Segment and the downstream analytics tools. We have it set up so events on all our local, staging, and load test environments go to “pagerduty-dev”, and only events in the production web stack go to the main account. In addition, we add filters to make sure that we’re not rolling activity on internal employees’ accounts into our metrics.

  • Nobody’s solved the problem of simultaneously looking at user-level and account-level funnel data. We’re currently looking at a Kissmetrics workaround using two different Kiss instances, but it was surprising to us that nobody natively handles pivoting on both levels.

  • User privacy and trust is incredibly important to us. PagerDuty takes its privacy policy seriously, and in choosing our analytics stack, we ensured that any vendor that dealt with user and customer data has a strong privacy policyincluding Safe Harbor compliance.

I hope you found our story helpful! If you have any questions or feedback, hit me up @dshack on Twitter.


A big thanks goes out to Dave from PagerDuty for this piece! If you’re interested in sharing your Segment story, we’d love to have you. Email friends@segment.com to get started!

Luke Gotszling on July 17th 2015

Like many startups in the early stages, at Peruse.io we’ve asked ourselves the question – how do you find product market fit? In this post, we’ll share our process for finding it and the tools we use for user testing, customer development, and data analysis.

For a little background on us, Peruse.io lets you easily search through information in your documents on Box and Dropbox. You can ask questions like, “Where is that sales projection spreadsheet I edited in March?” and we’ll show you.

Our inspiration for starting this company was that finding information in our files is still essentially the same tedious process that existed twenty years ago. We launched at 2015’s Techcrunch Disrupt in New York and now have a few hundred users in our public beta.

What is product market fit?

There are a lot of different definitions for product market fit. These two resonate with us:

“Product/market fit means being in a good market with a product that can satisfy that market.” - Marc Andreessen

“When, in a survey, at least 40% of users say they would be “very disappointed” without your product or service.” - Sean Ellis

More concisely, to us it means 1) our product works as intended, and 2) our users love our product.

We started with enabling users to search for information inside of spreadsheets, which the beta group likes. However, the story doesn’t end here as we’ve received plenty of feature and integration requests. We’re currently on a mission to see which features will bring us closer to product market fit and drive adoption.

Defining product market fit

To see how people are using the product and what’s bringing them back, we track the following events in Segment. We decided to use Segment to reduce development overhead and bloat in our site code, as well as ensure consistent customer data across all of the end tools that we use. For example, a spike in event A in Google Analytics can be easily investigated by searching for the users by event A in our communication tool, Intercom.

Does it work?

We want to track a few things about the new search feature. First is the accuracy of the results, to see if the product works as intended.

To measure accuracy, we’re tracking the following events with Segment: Document SearchedSearch Rated:

The Document Searched event is fired when a user submits a new query. When the query is submitted, results will appear on the interface. The user, assessing the quality of these results, can then rate them by clicking a button:

By including a button for people to rate the accuracy of the search and adding meaningful event tracking, we can look at the ratio of inaccurate Search Ratedevents to either accurate or general Document Searched events to determine if we’ve hit our first goal: to make the product work as expected. In case people only rate bad answers, we can also evaluate the Document Searched property result_clicked, since people will likely only click through answers that look good.

Do they love us?

To measure if our users love us, a more qualitative indicator, we segment our customers to find power users and potential churners, and reach out to them for direct product feedback.

The events necessary to create segments of the power and churn-risk users are Signed Up and Account Authenticated. In addition, we can look at if they have searched any documents from the events we’re already tracking above.

We measure users at risk of churn as people who show few signs of engagement. For Peruse, we believe that there are two types of churn risk users, short-term and long-term. We define these groups respectively as people who haven’t authenticated a Box or Dropbox account within 24 hours and people who haven’t used the service (searched a document) in 14 days. Note that the time intervals were not scientifically determined, as we plan to adjust these in the future, but represent a subset of users large enough to yield an adequate number of meaningful conversations.

Conversely, we identify power users by looking at strong engagement. Based on our existing user base from two months of being in public beta, we’re defining a power user as someone who has authenticated their Dropbox or Box account and searches documents at least once every few days. Our definition of a power user is partially a convenience measure for us: we adjust the “frequency” parameter until the power user sample is large enough to generate sufficient meaningful conversations.

Further segmentation is not being done at this stage of the product (other than ad-hoc segments like users who request a specific feature). However, this is an avenue we may explore in the future to tweak engagement or kick off nurture campaigns.

Tools to help you find product market fit

We’re using the following tools to help get to product market fit:

In order to assess product accuracy and repeat usage (a sign that users love the product), we decided to keep an eye on event data in Google Analytics. For product feedback, we use IntercomFullStory, and Inspectlet for broad site usage trends, alongside communicating with users and event tracking. We don’t need to use each service’s event API, since we can just use analytics.track once. The Segment tracking plan page can be used to select which services receive events.

Google Analytics

We’re using Google Analytics (free!) to track aggregate usage data, visitor origins, trends, and search result feedback counts

Specifically, in our search for product market fit, we are tracking aggregate trends of upvotes compared to downvotes in search result feedback counts. The benchmark for query accuracy is (number of upvotes - number of downvotes) / number of searches performed.

For the next few weeks, we’ll be keeping a close eye on this report, which measures the event Document Searched:

Document Searched is a feature introduced on June 23rd.

Intercom

Intercom is a customer communication tool: you can segment your users into smaller groups (based on their actions, location, and traits) and message them via email, in-app notifications, and live chat. They’re affordable at $50/m (the plan that we’re paying).

As stated, our definition of a “power user” is someone who has authenticated with Box or Dropbox and actively uses the service. Since “Account Authenticated” is an event that is sent when Box or Dropbox is added, it is available as a filter in Intercom. Additionally, “Active” is a user who is “last seen” within a week. With both filters, we create the segment in Intercom:

All names and sensitive information are removed.

Then we use email and in-app messaging for reaching out, with the objective to understand the user’s use case and get feature feedback, to guide future development.

For users with churn risk, we look at opposite triggers: did not authenticate and also low usage. We’ll setup a similar filter in Intercom. Similarly, we’ll email this segment of users, asking them what types of searches they would like to perform or if there are other features that are missing. This helps us prioritize product development.

Another benefit of using Intercom is seeing threaded conversations along with user events on a per user basis. This is very valuable to us in terms of seeing the user’s level of usage at a glance (and their experience in terms of answer quality by looking at how often they upvote and downvote results). Here’s how it looks like for two unique users:

User testing

To better understand user behavior we use FullStory and Inspectlet.

FullStory is a session recording tool that allows you to easily record, replay, search, and analyze each user’s actual experience with your website. These sessions give us a good sense of how users navigate our site, especially on mobile devices:

Inspectlet, a similar service, also provides “eye tracking heatmaps”, which basically is a map of where the your visitors are looking and what parts of the site they’re reading by visualizing their mouse movements (here is a study they reference that examines this correlation).

Eye tracking with Inspectlet shows that people tend to fixate more on the right side than the left. We’ll investigate whether this is something that needs to be addressed or a symptom of reading left-to-right.

Scroll tracking shows that most of the relevant information is above the fold (this page becomes longer when search results are displayed).

From the click tracking heatmap we can see that the results we show at the top tend to be more relevant for our users (for user privacy the search results are not shown on the heatmap).

Both of these tools are great for learning how users interact with our site, if our navigation is useful, and if we’re surfacing helpful results.

Next Steps

Finding product market fit is certainly not easy. However, given the proliferation of tools out there to assist in every part of the product lifecycle (13 Marketing Automation Tools$9 Marketing Stack, and How to Launch a Startup with $99), there are now many ways to become smarter about building features and minimizing the iteration time.

Segment has helped us experiment with new tools without incurring development overhead, and we’ve been very happy with Google Analytics, Intercom, FullStory, and Inspectlet so far. We know we’re not there yet, but we look forward to continuing to monitor our metrics and talk to customers as we make our way toward product market fit.

Morgan Brown on July 7th 2015

As growth marketers building a community for other growth marketers, data is a key component to our success. We’re meticulous about defining our metrics and tracking the necessary events to measure improvement.

But because the product we offer — a community of posts and comments — differs from a traditional SaaS or ecommerce business, the metrics we care about also differ. In this post we’ll explain what these metrics are, the dynamics specific to communities that affect our growth, and how we use this data to improve the user experience.

Cracking the Community

Communities like GrowthHackers, Hacker News, and Product Hunt have specific dynamics that attract, engage, and repel members. Before building a growth model, tracking data, or analyzing our progress, it’s important for us to understand these community dynamics.

The lurker effect

The large majority of visitors get value from community sites without ever logging in, creating an account, or even participating. They read and watch but never vote, comment, or share.

In 2006, Jakob Nielsen outlined this effect as the 90-9-1 rule for communities. He explained that on average 1 percent of community members actively contribute, 9 percent contribute a little, and 90 percent “lurk”. Each segment gets value from the product differently and has different behaviors and needs.

We’ve found this dynamic to be relatively true at GrowthHackers.com. Other communities have confirmed it as well — 80-90 percent of Reddit visitors never create an account, and 500 million people, 76 percent of visitors, read Twitter without ever logging in or tweeting. It’s likely 90 percent of the people reading this right now are lurking. (Hi!)

Marketplace dynamics

In addition to the “lurker” dynamic, communities like GrowthHackers are also affected by marketplace dynamics, where supply and demand are freely exchanged and strongly influence each other.

For background, our goal at GrowthHackers is to help people get better at growth. This means our supply is the content that educates and inspires our users, who comprise the demand-side of the market. The content comes from self-directed contributors, commenters, and curators who vote on submitted articles.

On the demand side, we know there’s a big audience of marketers, product managers, startup founders, and engineers interested in growth. Our goal is to attract and retain more of them, which means that traditional user acquisition and activation funnels are important to us. How do we turn a “lurker” into a participating member? We spend a lot of time on this.

While our user base is pretty easy to measure quantitatively, it’s much more important to measure the quality of the supply-side content. Low quality supply doesn’t help our community members be successful, which in turn lowers demand and hurts the marketplace. We don’t want listicles that lead to face palms here.

Growth goals

Now that we understand how both sides of the market work, we can easily measure the underlying dynamics. This helps us break down our growth challenge into achievable, improvable metrics.

Tracking the Demand-Side

When we look at how we’re helping people achieve their growth goals, we care about a few key data points. These metrics help us understand how our members go from being a first-time visitor, to a consistently engaged community member. To classify our metrics, we use Charlene Li’s Engagement Pyramid.

Watching

New visitors — When measuring traffic, we want to know how we’re growing the top of the funnel. Are new people discovering GrowthHackers (and how)? Because we’re still in the hustle stage, reaching new people who haven’t heard of us before is important.

Engaged time — While GrowthHackers acts as an entry point to resources around the web, engaged time per visitor is an important metric for us to understand whether people are getting the true value out of the site itself.

Someone who reads a few comment streams, watches a video, and reads a related post or growth study, really gets a chance to discover the core product value. This is compared to a visitor that reads one article and never comes back. More time spent likely predicts higher engagement in the long term.*

*Caveat: Sometimes people that submit questionable content also have high engagement times. Particularly, there are some users who attempt to game and manipulate the community by sharing of low quality content and participating in voting rings to get that content to rise. These users have high engaged times, but they actually are net negative on the community. Everycommunity deals with this, so we have to make sure we account for this behavior in our metrics.

Commenting and Curating

Subscribed to newsletter or Created an account — This is a big step for us! It’s the point when a user goes from a lurker to an identified member of the community. At this point, we can tie previously anonymous user actions into a single holistic view of a user’s behavior. Creating an account also enables users to vote, submit, and participate — they are one step closer to emerging from lurker-dom and have demonstrated significant interest in the community.

Voted or Commented — The next step after creating an account is to take some action besides consuming information. On GrowthHackers.com, the key actions are to vote and comment on posts. By voting, users help improve the quality of the content on the homepage. By commenting, they help create unique content that doesn’t exist elsewhere.

Specifically, we measure the percentage of total visitors who voted or commented on at least one post over a given time frame, usually by week or month.

Returning

Retention — Ultimately, our community grows from people who decide that GrowthHackers is a home for them on the web: a go-to spot for growth content, community, and inspiration. Retention is the best proxy we have for actually helping people get better at growth, so we follow it very carefully. Plainly, if people learn things, they will come back.

We measure retention by the percentage of visitors who check out the site and then return within one week. We then slice retention by user behavior, traffic source, and more to understand what levers lead to retention, and how we can improve the product experience.

Because “helping people get better” isn’t a numeric goal, we supplement retention metrics with qualitative surveys and Net Promoter Score with Qualaroo to better understand how much value we’re delivering to people.

You’ll note that we didn’t address submitting content or “producing” as a key metric here. To us, voting and commenting signal engagement better than submitting an article–which for many users is little more than trying to get some eyeballs on their own content.

Tracking the Supply-Side

A marketplace is only as good as its supply. If Airbnb didn’t have any nice homes in a city, or Uber was short on cars when you needed one, you’d be less likely to consider them as options in the future. Similarly, if people who visit GrowthHackers.com don’t find something of value, they’ll be less likely to return.

What’s more, if self-interested people see lame, promotional articles or negative comments, they will pile on more low quality content. Paul Graham calls this the broken window theory as applied to communities.

As a result, our supply-side metrics focus on measuring content quality. We don’t worry as much about content quantity because spam and self-promotional submissions skew those numbers; but we do worry greatly about providing signal in all the noise.

To that end we look at a variety of metrics that together give us a sense of the content quality.

Votes per post — Often, more votes signal a higher quality resource, so this is an important metric for us. However, sometimes voting rings pop up to promote bad content against the natural behavior of the community, so we take this with a grain of salt. While we track votes as one metric, it’s most useful when combined with others.

Time a post spends on the homepage — This is actually a more interesting number to us. Our algorithm favors sustained engagement on a post, boosting high quality content over low quality content temporarily boosted by voting rings. So the longer a post stays on the homepage, the higher quality it tends to be.

Number of comments on a post — Comments are usually a better proxy to quality than votes, because people only interested in getting a post up a page rarely take the time to leave a comment. Of course this isn’t perfect, and some comments are about how terrible an article is. But in aggregate, across a large sample, this number helps identify quality content.

In addition to on-site supply metrics, our interaction from our weekly top posts email and our Twitter act as reasonable proxies for content quality. If more people click through, retweet, and share, that’s good news. We analyze user behavior in those channels for insight into noisier supply-side metrics.

Tracking Before Testing

The GrowthHackers growth team runs via a framework called High Tempo Testing (HTT). HTT is focused around setting and maintaining a testing tempo goal (at least three tests a week!) across our product to accelerate learning. It’s been a big part of our breakout growth in the past few months.

Without tracking the above metrics and below events, we’d have no guideline for generating test ideas or ability to see if our tests ultimately improve our core metrics. And, because we’re looking at effects in behavior over time rather than immediate conversion goals, ensuring we have the right tracking down is critical to finding what changes really worked.

The Nitty Gritty

Here’s a look at the events we track to calculate our metrics.

  • Account Created — Recorded when a user creates an account

  • Account Verified — Recorded when a user’s account is successfully verified

  • Collection Viewed — Recorded when a user views a grouping of posts on a single page, like trending or new

  • Post Viewed — Recorded when a user visits the discussion page of a post

  • Post Upvoted — Recorded when a user upvotes a post

  • Comment Added — Recorded when a user adds a comment to a post

  • Comment Upvoted — Recorded when a user upvotes a comment

If you’d like to learn more about what to track for community sites, check out Segment’s best practice guide. The guide outlines the properties we’re collecting with these events and a few additional events that make sense for communities but don’t relate to our specific KPIs.

Growing GrowthHackers

Before you can identify your key metrics, you first have to understand the dynamics of your business model. Following this plan, tracking the right data, and executing on our High Tempo Testing process, we’ve hit 59 percent user growth in the past few months!

Hopefully, this look into the community model and our data can help you on your way to growth, too. It’s not magic, just data and discipline.

Become a data expert.

Get the latest articles on all things data, product, and growth delivered straight to your inbox.