A New Way to Think About Analytics

Analytics is overwhelming. Lots of people think that tracking 73 different metrics means they've solved their analytics problem. Don't do it! Unless you've got employees twiddling their thumbs, you'll never be able to handle that many metrics.

You need to think about analytics differently. That's what we're here for.

Analytics is about learning.

To succeed, your team needs to learn as quickly as possible. It's the only way to make sure that every new release is better than your last one. Unfortunately, learning is almost never the priority.

Most web products get made like this:

  1. Build the product.
  2. Throw in analytics.
  3. Pull up your analytics. Get overwhelmed. Go back to building product.

Where did the learning happen? What was the point of looking at the analytics? Why'd you even set up any analytics in the first place?!

That leaves us with a disappointing process for making web products:

  1. Build the product

That's a recipe for failure.

We know a better way.

The Lean Startup, which you've probably heard about but never actually read, advocates a much more successful process:

  1. Build an experimental product.
  2. Measure how people respond to your experiment.
  3. Learn whether your experiment worked or not.

The trick though, is that you actually start with step 3.

First, you have to decide what you want to learn. Then, you figure out how you're going to measure it. And only after that do you build your experiment.

That is the most important thing about analytics! If you take nothing else away from this series, take this: Before you build anything, decide what you're going to learn from building it.

This process is enormously successful at companies of all sizes. We dug up some of their stories to help you understand how to do it yourself:

Airbnb lets people rent out their houses to travelers. One day, one of the co-founders hypothesized that professional photography would help Airbnb hosts attract more renters. So they ran an experiment. Airbnb hired 20 professional photographers on a trial basis. The photographers met with a small subset of hosts and took photos of their homes and spare bedrooms. Then they measured how the new photos performed. And true to their guess, listings with professional photos got 2-3x more bookings! Airbnb's hypothesis was validated, so they expanded the program. Now they do 5,000 photo shoots a month because they know they increase their bottom line. PS. This is an example from the highly recommended Lean Analytics book.
Segment provides an analytics API. But before we built anything, we wanted to be sure it was something people wanted. What wanted to learn: 1. Do other people care about simplifying their analytics setup? 2. If so, do they want an open-source library or do they want a hosted solution? Each question would validate a core assumption in our business model. Before this we'd built a library called analytics.js that simplified our own analytics. So to answer our questions, we open-sourced the analytics.js repo on Github and built a quick signup form for a hosted version. We released the open source version on HackerNews and waited to see what people would do. Here's what happened: 1. 1,800 people starred the repo on Github, which convinced us that people really care about simplifying their analytics setup. 2. About a thousand people signed up as interested in the hosted solution. Not everyone was satisfied with just an open source library. Based on this experiment, we decided to go ahead with building a hosted solution. Validating our core business model assumptions kept us from wasting time building something that nobody wants. That's why you should do it too.
Like everyone else, Twitter has a huge number of registered but inactive users. The classic way to get these users back is to send them email about New friends on Twitter! But recently a more interesting feature appeared. If an inactive user clicks through one of these email campaigns, Twitter immediately nudges their most active friends to tweet at them! This is real social engineering. It's pretty clear how the feature came about: someone at Twitter guessed that having friends tweet at you right after you've returned to the service would increase retention. Then, with their hypothesis in mind, they built the feature and launched it. You can bet that the growth team at Twitter is keeping close tabs on this experiment. Does it increase reactivation rates? It's fun to see huge, established companies like Twitter using the same process as tiny startups. If this small bit of social engineering boosts reactivation by 1%, how many more millions of people will suddenly start using Twitter?

All right, it's homework time! Think about your own business and figure out what do you need to learn right now?

A good way to start is to think of your One Key Business Metric.

Anything you learn that improves that metric is gold. Your experiment will depend on what stage your business is in:

Just a landing page? Find an experiment that'll help you learn what people really need from your product. The analytics.js open-source launch above might give you some ideas.

Early version released? Find an experiment that'll help you learn how to get more active users. Take a second look at Twitter's social engineering experiment if you need inspiration.

Serious revenue? Find an experiment that'll help you learn how to make more money per user. The Airbnb photography experiment was brilliant because it made them more money as a side-effect of making their customers happier.

Once you've thought of your experiment, send us an email about it and one of our co-founders will reply with feedback on your plan.

Three random experimenters will get a free copy of The Lean Startup!

Each week in this course, we'll be giving you a new, bite-sized lesson on how to run exactly these kinds of experiments with different analytics tools. You won't want to miss it!

Vote on Hacker News