Building a Better Robot: Tracking and Analytics for Slack Bots

In December 2015, Slack announced it would invest $80 million into Slack bot startups. This comes as no surprise for Slack’s 2 million daily active users, and bolsters its strategy to create an ecosystem of productivity and collaboration services. There are already over 4,000 Slack integrations (in addition to 150 official apps) which have over 2.2 million installs. The community is finding Slack useful for nearly everything. And we believe this is just the beginning.

In fact, many companies are forgoing traditional user interface. Siri may be the most popular Natural Language Interface on the planet, but Amazon has Alexa, and now every startup team has Slack bots. Moreover, Slack’s rapidly growing user base platform may prove a powerful distribution channel for startups.

That said, measuring performance and achieving predictable growth for Slack bots is a new type of analytics challenge. What is the best way to measure activation and retention? What metrics are most important to track? How do we attribute new bot installs?

This post is a first look at implementing tracking and analytics for Slack bots. We’ve borrowed and applied many philosophies around traditional web and mobile analytics, as well as user behavior tracking to the context of Slack bots. Keep in mind that things may change as this platform matures with more bots and products!

Tracking and Analytics Guidelines

Analytics is about learning and understanding growth. The general guidelines for deriving a set of events are:

• Understand that everything is a funnel: How do people discover, start using, and pay for your product? What are the most important steps along the way?

• Identify moments where the user derives value: Which key events, when interacting with your product, sends a strong signal of engagement?

• Maintain a consistent naming convention: Will all event names be “Object Action” or “Verb Subject”? What will the casing be? This will minimize headaches and save time for future you or a new team mate when conducting analysis or implementing new tracking code.

If you’d like to learn more about this approach to tracking your product for growth, here is a high-level guide to creating a tracking plan.

We won’t dive into #3 so much here (we have a post coming out soon about various naming conventions—stay tuned!), but we will look at #1 and #2.

Here are the common events (and their properties as sub bullets) that are tracked as a reflection of acquisition, activation, retention, revenue, and referral of the funnel. The properties are important as those allow us to slice and dice the data in our analytics or marketing automation tools. Note that each event is attached to the user with an additional parameter of userId to help downstream analytics tools tie events to users and teams:

• Bot Installed: This event is fired when the bot is initially installed via the “Add to Slack” button.

  • team_id (since this team id from Slack is immutable)

  • source (for attribution, but only if you are visitors are installing your bot from a site that isn’t the Slack app store)

• Bot Activated (optional): This event is fired when a necessary mid-step, such as authenticating a calendar, is completed. Many times bots aren’t able to provide full value until certain accounts are connected.

  • service_authenticated (e.g. “Google Calendar”)

• Message Received: This event is fired when the bot receives a message. It’s important to note that the “topic” or “intent” of this event is sent as a propertyof this event, since the bot receiving a “hi” signals a different interaction than “what is the weather today?”

  • message_type (the “intent” of the message, e.g. “new meeting”)

• Subscription Started (optional): This event is fired when the user begins a subscription and starts paying. This event assumes that the bot follows a SaaS model; if the bot is selling one-off items à la ecommerce, then Order Completedcould be the event. The idea here is that this is the bottom of the funnel and revenue is tracked.

• Bot Deactivated: This event is fired when Slack closes the Real Time connection’s websocket without an explanation, which is what happens when a user disables a bot (here’s a fantastic little post about handling this event).

Note that you can name your events and properties however you’d like! We used “Object Action” for event names and snake cased property names in the example above.

To help illustrate why some of these properties are included, let’s go into some growth objectives.

Measuring Attribution and Acquisition

If you want to get new users for your Slack bot, then install attribution is critical to measure performance of various campaigns and learn where your highest quality “leads” are coming from. To dig into user install attributions in Slack, let’s take a look at the two main ways to install Slack apps:

• From Slack’s app “store”

• From an “Add to Slack” button

Unfortunately, installs directly from Slack’s app store doesn’t provide you with user attribution data—you won’t be able to know how that user got to the Slack’s app store. Of course, it’s still immensely helpful to know that Slack’s own marketplace is allowing users to discover your product (or not).

Here’s a trick to calculate new users from Slack directly using the “Add to Slack” button. The “Add to Slack” button is basically a link to Slack’s Oauth endpoint, which accepts an optional state parameter whose value is passed back to your server upon auth completion. Though the state parameter is typically used in Oauth flows to prevent cross-site forgery requests, we can also append campaign or attribution information so we can attribute installs in our analytics tools.

Here’s an example of a URL that’ll kick off the add to Slack process (replace client_id with yours and state with your randomly generated id and some text that tells you where the user clicked the “Add to Slack” button, e.g. “homepage”):

After the user selects which Slack account to authenticate, Slack will redirect the user to your redirect URI with the state as a query string. Your client-side code can parse the query string to get the attribution information, and (either on the client or server) finally send an .identify() and a .track() call:

Note that the above code is written with Segment’s analytics-node library. Though the keys and values in the traits and properties objects are completely up to you, sending source is what allows you to attribute installs in your analysis.

Now, you can attribute the source of your users, assuming a non-trivial source of your signups is not from Slack’s “store”. Moreover, capturing key activation events (more on that in the next section) can tell you which “Add to Slack” button provides higher quality installs.

Measuring Activation and Retention

An activation event is an interaction when the user receives value. In the case of bots, this typically means when the user asks the bot to do what the bot is meant to do.

Retention, on the other hand, is a bit more tricky to track. Though users have the option of intentionally disabling the bot, it’s much more common for users to slowly forget about it over time.

In the two case studies below, we’ll explore measuring growth as # of requests a bot receives, teasing out “A-ha” moment experiments from the growth of activated users within a team, and finally calculating growth and retention as the ratio of daily active users to monthly active users.

Case study: Birdly

Birdly, a tool for expense management that began as a mobile app, is now focused on building a completely chat-based interface in Slack. Its core value prop is to process expenses from pictures of receipts. However, the new distribution and usage opportunities from Slack’s platform have encouraged the team to expand its vision to provide a suite of back office services.

Image taken from Quang’s post on “Birdly launches a Slack bot for Expense Management”.

Birdly’s current strategy is to experiment with multiple different bots—expense management, turning business cards into Salesforce leads, translation services, managing grocery lists, etc—and see what sticks. With this quick build-measure-learn iteration cycle, analytics around usage is deeply important.

The main event that they track for each bot is new request (triggered when the bot receives a message from a user) with properties request_type that indicates the intent of the message (i.e. uploading a receipt or something else). Keep in mind that the request_type that signals a core usage event varies amongst Birdly’s bots. For Birdly’s main bot, this is expense, whereas it is translate for the translation bot:

Note the above snippet is a fictitious representation of their actual server-side call, written with Segment’s analytics-node library.

To measure the success of the bot, Birdly closely watches the # of new requests that have its corresponding request_type per week.

Growth is also important. The Birdly team aims to maintain its growth rate of 30% to 40% m/m growth on the number of teams who install one of its bots.

Birdly uses a variety of tools (enabled via Segment) to help them measure and monitor these metrics—Google Analytics for measuring web analytics to their site, Mixpanel for measuring usage and trends, Facebook Pixel and Twitter Adsfor tying their custom bot install event to their ad campaigns.

Case Study: Meekan

Meekan, a flexible scheduling assistant, provides both a Slack and Hipchat bot for teams. Meekan can schedule a calendar event for various team mates by looking through everyone’s calendar, identifying availabilities (within parameters that you specify), and adding the event to the respective calendars.

This is me scheduling lunch with my friend, Alex:

And this is me confirming my attendance to SQL SQOOL:

Since Meekan’s value increases as more users within a team authenticates their calendars and schedule meetings via Meekan, the team cares about growth and distribution within a team.

As such, the team tracks installation (including the total number of users in the team), activation (authenticating a calendar, which creates a user profile in Mixpanel), and finally engagement (when a user initiates a request via Meekan).

With this data, Meekan uses Mixpanel to map out the adoption speed within a team:

Chart taken from a custom query with the data in Mixpanel.

Matty, Meekan’s product manager, understands there are many subtle factors in play here. “Some people don’t use calendars, don’t have meetings (or they just accept whatever meeting time is suggested), or are signed into Slack/HipChat, but don’t actually ever use it.”

However, with these conversion events and some other contextual data (size of team, observed scheduling behaviors, profile of the “champion”), the Meekan team plans to explore opportunities for the bot to demonstrate its value via public channels as a passive way to attract more team member adoption. “Chat bots have a lot of power and access to the entire team, even those who’ve never heard about them,” says Matty. “We make sure we don’t abuse this power.”

Since the Meekan team is focused on retention, one metric borrowed from mobile and Facebook apps is the ratio of DAU / MAU (a popular retention metric), where an active user is one who initiates at least one request with Meekan—what they have named meekan sentence, similar to new_request. This event is triggered after Meekan receives and parses a message from a user into four parts: intent, title, date, and time of day. Note that the below is just one example. There are other message intents, such as calendar queries, flight searches, etc.

Image taken from Matty’s post on “Cheating on the Turing Test”.

The team’s focus is on retention—”making sure our current users are happy, and providing them with value, so that they’ll want to keep us around and use the robot daily. It’s very easy to disconnect the robot or forget you’ve ever installed him in the first place.”

The Future of Messaging Products

It’s still early for bot makers and Slack, so keep in mind that best practices may change as more bots join the platform and we learn what works and what doesn’t.

One thing about analytics that won’t change is that you should always be intentional about your tracking. Start with a goal, derive appropriate metrics (best to focus on one or two), and figure out which events are needed to uncover those metrics.

Messaging products, whose primary interface is through natural language, are starting to become more and more popular. On the enterprise side, Hubot and now Slack’s new platform of bots may become ubiquitous in how we initiate requests such as pulling customer account info or ordering a Philz coffee.

On the consumer side, Magic, WeChat (some great analysis from a16z about WeChat’s integral role as more than just a platform), Facebook’s M, and Google’s own Smarter Messaging product, have powerful and grandiose visions of simplifying our lives. Soon, it may be common to interact with a product solely based on natural language verbal or text communication.

In any case, we’re excited to see what bots developers will build!

Have any ideas for tracking a Slack bot? Tweet at us!