GrowthHackers on Metrics that Matter to Communities

As growth marketers building a community for other growth marketers, data is a key component to our success. We’re meticulous about defining our metrics and tracking the necessary events to measure improvement.

But because the product we offer — a community of posts and comments — differs from a traditional SaaS or e-commerce business, the metrics we care about also differ. In this post we’ll explain what these metrics are, the dynamics specific to communities that affect our growth, and how we use this data to improve the user experience.

Cracking the Community

Communities like GrowthHackers, Hacker News, and Product Hunt have specific dynamics that attract, engage, and repel members. Before building a growth model, tracking data, or analyzing our progress, it’s important for us to understand these community dynamics.

The lurker effect

The large majority of visitors get value from community sites without ever logging in, creating an account, or even participating. They read and watch but never vote, comment, or share.

In 2006, Jakob Nielsen outlined this effect as the 90-9-1 rule for communities. He explained that on average 1 percent of community members actively contribute, 9 percent contribute a little, and 90 percent “lurk”. Each segment gets value from the product differently and has different behaviors and needs.

We’ve found this dynamic to be relatively true at GrowthHackers.com. Other communities have confirmed it as well — 80-90 percent of Reddit visitors never create an account, and 500 million people, 76 percent of visitors, read Twitter without ever logging in or tweeting. It’s likely 90 percent of the people reading this right now are lurking. (Hi!)

Marketplace dynamics

In addition to the “lurker” dynamic, communities like GrowthHackers are also affected by marketplace dynamics, where supply and demand are freely exchanged and strongly influence each other.

For background, our goal at GrowthHackers is to help people get better at growth. This means our supply is the content that educates and inspires our users, who comprise the demand-side of the market. The content comes from self-directed contributors, commenters, and curators who vote on submitted articles.

On the demand side, we know there’s a big audience of marketers, product managers, startup founders, and engineers interested in growth. Our goal is to attract and retain more of them, which means that traditional user acquisition and activation funnels are important to us. How do we turn a “lurker” into a participating member? We spend a lot of time on this.

While our user base is pretty easy to measure quantitatively, it’s much more important to measure the quality of the supply-side content. Low quality supply doesn’t help our community members be successful, which in turn lowers demand and hurts the marketplace. We don’t want listicles that lead to face palms here.

Growth goals

Now that we understand how both sides of the market work, we can easily measure the underlying dynamics. This helps us break down our growth challenge into achievable, improvable metrics.

Tracking the Demand-Side

When we look at how we’re helping people achieve their growth goals, we care about a few key data points. These metrics help us understand how our members go from being a first-time visitor, to a consistently engaged community member. To classify our metrics, we use Charlene Li’s Engagement Pyramid.

Watching

New visitors — When measuring traffic, we want to know how we’re growing the top of the funnel. Are new people discovering GrowthHackers (and how)? Because we’re still in the hustle stage, reaching new people who haven’t heard of us before is important.

Engaged time — While GrowthHackers acts as an entry point to resources around the web, engaged time per visitor is an important metric for us to understand whether people are getting the true value out of the site itself.

Someone who reads a few comment streams, watches a video, and reads a related post or growth study, really gets a chance to discover the core product value. This is compared to a visitor that reads one article and never comes back. More time spent likely predicts higher engagement in the long term.*

*Caveat: Sometimes people that submit questionable content also have high engagement times. Particularly, there are some users who attempt to game and manipulate the community by sharing of low quality content and participating in voting rings to get that content to rise. These users have high engaged times, but they actually are net negative on the community. Everycommunity deals with this, so we have to make sure we account for this behavior in our metrics.

Commenting and Curating

Subscribed to newsletter or Created an account — This is a big step for us! It’s the point when a user goes from a lurker to an identified member of the community. At this point, we can tie previously anonymous user actions into a single holistic view of a user’s behavior. Creating an account also enables users to vote, submit, and participate — they are one step closer to emerging from lurker-dom and have demonstrated significant interest in the community.

Voted or Commented — The next step after creating an account is to take some action besides consuming information. On GrowthHackers.com, the key actions are to vote and comment on posts. By voting, users help improve the quality of the content on the homepage. By commenting, they help create unique content that doesn’t exist elsewhere.

Specifically, we measure the percentage of total visitors who voted or commented on at least one post over a given time frame, usually by week or month.

Returning

Retention — Ultimately, our community grows from people who decide that GrowthHackers is a home for them on the web: a go-to spot for growth content, community, and inspiration. Retention is the best proxy we have for actually helping people get better at growth, so we follow it very carefully. Plainly, if people learn things, they will come back.

We measure retention by the percentage of visitors who check out the site and then return within one week. We then slice retention by user behavior, traffic source, and more to understand what levers lead to retention, and how we can improve the product experience.

Because “helping people get better” isn’t a numeric goal, we supplement retention metrics with qualitative surveys and Net Promoter Score with Qualaroo to better understand how much value we’re delivering to people.

You’ll note that we didn’t address submitting content or “producing” as a key metric here. To us, voting and commenting signal engagement better than submitting an article–which for many users is little more than trying to get some eyeballs on their own content.

Tracking the Supply-Side

A marketplace is only as good as its supply. If Airbnb didn’t have any nice homes in a city, or Uber was short on cars when you needed one, you’d be less likely to consider them as options in the future. Similarly, if people who visit GrowthHackers.com don’t find something of value, they’ll be less likely to return.

What’s more, if self-interested people see lame, promotional articles or negative comments, they will pile on more low quality content. Paul Graham calls this the broken window theory as applied to communities.

As a result, our supply-side metrics focus on measuring content quality. We don’t worry as much about content quantity because spam and self-promotional submissions skew those numbers; but we do worry greatly about providing signal in all the noise.

To that end we look at a variety of metrics that together give us a sense of the content quality.

Votes per post — Often, more votes signal a higher quality resource, so this is an important metric for us. However, sometimes voting rings pop up to promote bad content against the natural behavior of the community, so we take this with a grain of salt. While we track votes as one metric, it’s most useful when combined with others.

Time a post spends on the homepage — This is actually a more interesting number to us. Our algorithm favors sustained engagement on a post, boosting high quality content over low quality content temporarily boosted by voting rings. So the longer a post stays on the homepage, the higher quality it tends to be.

Number of comments on a post — Comments are usually a better proxy to quality than votes, because people only interested in getting a post up a page rarely take the time to leave a comment. Of course this isn’t perfect, and some comments are about how terrible an article is. But in aggregate, across a large sample, this number helps identify quality content.

In addition to on-site supply metrics, our interaction from our weekly top posts email and our Twitter act as reasonable proxies for content quality. If more people click through, retweet, and share, that’s good news. We analyze user behavior in those channels for insight into noisier supply-side metrics.

Tracking Before Testing

The GrowthHackers growth team runs via a framework called High Tempo Testing (HTT). HTT is focused around setting and maintaining a testing tempo goal (at least three tests a week!) across our product to accelerate learning. It’s been a big part of our breakout growth in the past few months.

Without tracking the above metrics and below events, we’d have no guideline for generating test ideas or ability to see if our tests ultimately improve our core metrics. And, because we’re looking at effects in behavior over time rather than immediate conversion goals, ensuring we have the right tracking down is critical to finding what changes really worked.

The Nitty Gritty

Here’s a look at the events we track to calculate our metrics.

• Account Created — Recorded when a user creates an account

• Account Verified — Recorded when a user’s account is successfully verified

• Collection Viewed — Recorded when a user views a grouping of posts on a single page, like trending or new

• Post Viewed — Recorded when a user visits the discussion page of a post

• Post Upvoted — Recorded when a user upvotes a post

• Comment Added — Recorded when a user adds a comment to a post

• Comment Upvoted — Recorded when a user upvotes a comment

If you’d like to learn more about what to track for community sites, check out Segment’s best practice guide. The guide outlines the properties we’re collecting with these events and a few additional events that make sense for communities but don’t relate to our specific KPIs.

Growing GrowthHackers

Before you can identify your key metrics, you first have to understand the dynamics of your business model. Following this plan, tracking the right data, and executing on our High Tempo Testing process, we’ve hit 59 percent user growth in the past few months!

Hopefully, this look into the community model and our data can help you on your way to growth, too. It’s not magic, just data and discipline.