How to think about analytics

Analytics can be overwhelming. You’re reading thousands of business articles on hundreds of business metrics. Your team is pushing you to collect mounds of analytics data. You’re trying to integrate the data to dozens of analytics and growth tools and their complex APIs. What is the point of it all?

You need to think about analytics differently. That's what we're here for.

Analytics is about learning.

To succeed, your team needs to learn as quickly as possible. It's the only way to make sure that every new release is better than your last one. Unfortunately, learning is almost never the priority.

Most web and mobile products get made like this:

• Build the product.
• Throw in analytics.
• Pull up your analytics. Get overwhelmed. Go back to building product.

Where did the learning happen? What was the point of looking at the analytics? Why'd you even set up any analytics in the first place?!

That leaves us with a disappointing process for making web products:

• Build the product
  

   

That's a recipe for failure.

There is a better way

The Lean Startup, which you've probably heard about but may have never actually read, advocates a much more successful process:

• Build an experimental product.
• Measure how people respond to your experiment.
• Learn whether your experiment worked or not.

The trick though, is that you actually start with step 3.

First, you have to decide what you want to learn. Then, you figure out how you're going to measure it. And only after that do you build your experiment.

That is the most important thing about analytics! If you take nothing else away from this series, take this: Before you build anything — product, marketing campaigns, whatever, decide what you're going to learn from building it.

This process is enormously successful at companies of all sizes. We dug up some of their stories to help you understand how to do it yourself:

Case Study: Airbnb

Airbnb lets people rent out their houses to travelers. One day, one of the co-founders hypothesized that professional photography would help Airbnb hosts attract more renters.

So they ran an experiment.

Airbnb hired 20 professional photographers on a trial basis. The photographers met with a small subset of hosts and took photos of their homes and spare bedrooms.

Then they measured how the new photos performed. And true to their guess, listings with professional photos got 2-3x more bookings!

Airbnb's hypothesis was validated, so they expanded the program. Now they do 5,000 photo shoots a month because they know they increase their bottom line.

PS. This is an example from the highly recommended Lean Analytics book.

Case Study: Analytics.js

Segment provides an analytics API. But before we built anything, we wanted to be sure it was something people wanted. What we wanted to learn:

• Do other people care about simplifying their analytics setup?
• If so, do they want an open-source library or do they want a hosted solution?

Each question would validate a core assumption in our business model.

Before this we'd built a library called analytics.js that simplified our own analytics. So to answer our questions, we open-sourced the analytics.js repo on Github and built a quick signup form for a hosted version. We released the open source version on HackerNews and waited to see what people would do.

Here's what happened:

• 1,800 people starred the repo on Github, which convinced us that people really care about simplifying their analytics setup.
• About a thousand people signed up as interested in the hosted solution. Not everyone was satisfied with just an open source library.

Based on this experiment, we decided to go ahead with building a hosted solution. Validating our core business model assumptions kept us from wasting time building something that nobody wants (which we did for a long time before that). You can make sure you're learning way earlier than us!

Case Study: Twitter

Like most social sites, Twitter has a huge number of registered but inactive users. Twitter used to try to push people to activate by emailing them about new friends on Twitter and popular handles they might want to follow.

But recently a more interesting feature appeared. If an inactive user clicks through one of these email campaigns, Twitter immediately nudges their most active friends to tweet at them! This is real social engineering.

It's pretty clear how the feature came about: someone at Twitter guessed that having friends tweet at you right after you've returned to the service would increase retention. Then, with their hypothesis in mind, they built the feature and launched it.

You can bet that the growth team at Twitter is keeping close tabs on this experiment. Does it increase reactivation rates?

It's fun to see huge, established companies like Twitter using the same process as tiny startups. If this small bit of social engineering boosts reactivation by 1%, how many more millions of people will suddenly start using Twitter?

Let’s put to practice what we just discussed. Think about your own business and figure out what do you need to learn right now?

A good way to start is to think of your One Key Business Metric. Anything you learn that improves that metric is gold.

Your experiment will depend on what stage your business is in:

Just a landing page? Find an experiment that'll help you learn if people actually want your product and what parts of it are most interesting. A/B test landing pages focusing on different features and see what performs best!

Early version released? Find an experiment that will test which features increase engagement. You might want to split test different versions of your product highlighting different features and see which cohorts of users keep returning.

Serious revenue? Find an experiment that'll help you learn how to activate more signups. The Airbnb photography experiment was brilliant because it made listings much more appealing, driving activation and revenue.

If you'd like some feedback on your experiment, send us an email about it, and we'll share our thoughts!