Wiping the slate clean: How Hotjar used Twilio Segment to build a culture of clean data

When Hotjar was founded in 2014, it did so with a bold vision – to give teams of all sizes the insights they need to create experiences their users love.

Fast forward to 2019, and they’re well on the way to achieving this. Having grown into a company of 90+ people, spread across 19 countries, 26,800 customers including Shopify, Adobe, and HubSpot, are using Hotjar to understand their user’s behavior, and optimize their experience.

So it was somewhat of a surprise to Andrew Michael, Head of Business Intelligence who joined Hotjar in 2017, that the team didn’t have the right infrastructure in place to practice what they preach. 

Despite being a behavior analytics company, they lacked the right analytics stack to deliver the best experience possible to their own users. Teammates would try to find data to help inform their work but ended up with more questions than answers.

Incorrectly created data records, loose naming conventions, and a lack of a clear data governance strategy were just a few of the problems they struggled with day-to-day.

It got to a point where people had so little trust in the data that they would bypass it entirely, knowing that bad data could have an adverse effect on their work. 

Twelve months ago, Andrew decided it was time for a change, and put together a proof of concept for what a world-class data infrastructure for the company might look like. His proof of concept was made up of four different tools:

• Segment - for data governance, and to send consistent, accurate data to their downstream tools. 

• Redshift - as a data lake and warehouse, built on top of data collected from Segment.

• Mixpanel - for product analytics and data visualization.

• Mode - to query the data warehouse.

Having used Segment and these tools at several startups he founded previously, Andrew was confident this stack would solve a number of the data infrastructure problems they were struggling with.

Andrew Michael (center) and the churn team at Hotjar. Source.

From intuition to data

At Hotjar, customer feedback is at the core of what they do. All team members are encouraged to obsess over the wants, needs, and opinions of their customers.

They had a robust set up in terms of collecting and managing feedback, but they lacked the data to quantify it. How much feedback was coming from their ideal customer profile? How much of it is coming from small, mom and pop shops who were just kicking the tires? 

Without access to this data, they were still making some decisions based on hunches and assumptions. 

Thanks to Segment, Hotjar were able to combine their qualitative research with quantitative data. The new data infrastructure allowed the team to conduct better user research by giving insights into who their ideal customer profile is. It also validated the user feedback with quantitative data sparking a new era of experimentation and data-formed decision making for the company.

This was just one part of broader efforts within Hotjar to become more data-driven as an organization. Having experimented with a lot with different organizational frameworks like V2MOM, OKRs, etc. they kept running into a familiar problem. 

All of these frameworks demanded the ability to measure the impact of what was being implemented. No matter what framework they used, when it came time to analyze results, they would find themselves lacking the right numbers, or seeing different numbers in different tools. 

For Hotjar to become as strong in quantitative research than they were in qualitative, it was clear they needed a clean foundation of data to work with.

Wiping the slate clean

What Hotjar recognized was that the entire company would benefit from a clean break from their old, historical data that few in the company trusted. 

Andrew says this was a significant blocker for simple, day to day tasks, which then had to be escalated to engineering to fix.

“It just got to the point where people said, "Okay, we can't trust the data in Mixpanel. Let's not look at Mixpanel," and then we need to spend time from our dev resources and say, "Can you put together this report for us?" And we literally need to query our production database to try and understand that.”

So Hotjar took a bold decision – to abandon their historical data and start from scratch. The team cut their losses with their existing data infrastructure setup and built everything off the back of Segment.

This gave the team two things: a pool of clean, accurate data to work from, and, crucially, a renewed internal credibility and confidence from the team in the data they had.

It was tough to start from square one, but almost a year after adopting Segment, they can now do mass historical reporting year-on-year views that they couldn’t before.

Fostering a culture of experimentation

Today, Segment has extended throughout the organization – from product and marketing to sales, support, and operations – to make teams more data-driven than ever before.

For the marketing team, they can now track how visitors are landing on and interacting with their marketing site, and how many of those are converting into paying customers. This helps them optimize their paid media spend and uncover which channels will help drive their next stage of growth.

The marketing team also now has access to vital customer acquisition cost data. Previously, this data had been locked into Google and Facebook’s ad platforms, making it difficult to query. By using Segment, in just a couple of clicks, they were able to capture the data they needed to calculate customer acquisition costs, and then send those conversion events to all of their tools.

For the product team, they can now understand what's driving engagement and retention. As the host of Churn.fm, retention is a topic Andrew is particularly passionate about. Since bringing in their new analytics stack, they have all the key metrics and KPIs they need to build business cases. Thanks to Segment, they can be certain all of these metrics are clean and accurate.

As Andrew said: 

“We’re now confident it's the same data that we see in all our different tools. Segment is that single source of truth.”

Additionally, Segment’s catalog of integrations has given Hotjar a low risk, low maintenance way to experiment with new tools. Previously, to add a new tool to their tech stack, Andrew’s team had to submit a request to engineering to build a custom integration. Once engineering resources had been devoted, the sunk cost fallacy kicked in. They were reluctant to rip and replace tools as needed, even if they weren’t working out.

“Switching costs played a big part in our decision making. The pressure was on us to stay with certain tools, and it was a little more difficult to move around. Segment negates that. If we decide to try a new tool, we don't have that noose around our neck where we're going to need to spend months trying to reintegrate.”

For Hotjar, much of their usage of Segment comes down to a cost/benefit analysis. If someone adds two rogue events that pollute the system, how much is that going to cost them? If they want to adopt a new tool, how much will it cost them to integrate it? 

By saving the time of their existing staff, or having to hire an extra person to manage all this, Hotjar can focus on what’s most important to their business – helping their thousands of customers understand their customers better. 

A cultural shift in the organization

Twelve months after adopting Segment, there has been a 180-degree switch in the culture of data and analytics at Hotjar. Now when they’re kicking off a new project or program, they start with data, which is something that didn’t happen before.

Andrew summed it up nicely:

“Day-to-day, the appetite for data and insights has been overwhelming. When I first started working on the proof of concept, I had a Trello board of 10-15 cards on it. I’d added a number things it'd be nice to track. Since we launched the POC, and people saw how valuable it was, the Trello board exploded. It went from 15 cards to 150 cards in a month, as people were just so excited to get their hands on new data.”