How AppDynamics used Twilio Segment to achieve 100% product usage telemetry

The big win

Company wide, there was a concern AppDynamics didn’t have product telemetry data to make the right product decisions. Additionally, the product and customer success teams had little visibility on user engagement.

AppDynamics implemented a customer data platform pilot with Twilio Segment, which then drove a full roll out to each product line. AppDynamics could now instrument new product features and capture user behavior events with a standard schema using Twilio Segment, then send this data to Amplitude for visualization. With Twilio Segment, AppDynamics has been able to achieve nearly 100% product telemetry, with a highly decentralized data system for more streamlined use.

AppDynamics, a Cisco company, is the #1 fastest-growing APM solution in the world. With nearly 2,200 employees, AppDynamics works with companies such as Aflac, Vodafone, United Airlines, Siemens, and Nasdaq, to power their performance activities across multi-cloud and distributed environments, while enabling them to drive innovation. 

In 2019, AppDynamics product leadership realized that lack of depth and coverage of product telemetry data was seriously preventing the product teams from making data-informed decisions. This served as the catalyst for a deeper analysis of how the company managed its product telemetry. After evaluation, the team determined that their existing system was preventing them from getting more granular information about their end users. Specifically, the AppDynamics team discovered that they were unable to understand the health of their accounts, make user-driven product decisions, and invest in features that users actually found valuable. 

The Appdynamics team ran a thorough discovery exercise on industry standard telemetry tools. After evaluating several options, ultimately, the team at AppDynamics decided to implement Twilio Segment as their primary solution for instrumenting telemetry within the product. Appdynamics engineering teams have integrated Twilio Segment within their product. Whenever a user or state activity happens,  Twilio Segment generates events, which are then sent to tools like Amplitude and Snowflake/Tableau for further visualization. Today, the company has achieved significant progress in telemetry for its core product using a decentralized model.

A company-wide lack of visibility into user engagement

For AppDynamics, the end goal was to gain visibility into how people used their product. To achieve this, they needed a way to automatically collect raw data from real-world use, and standardize it in one place to create valuable, actionable analytics.

Before using Twilio Segment, the AppDynamics product team would gain a better understanding of usage by interviewing customers for feedback . The problem was, most interviewees were administrative and accounts personas, not actual end users of the AppDynamics product. This led to a lack of feature-specific feedback, which didn’t provide visibility into user engagement, and often led to the investment into features without understanding how or why they might be adopted.  

Despite the fact that they were able to see frequency data, such as page views, the team couldn’t get a clear picture of how users flowed through the product. And most of the product telemetry data AppDynamics did collect had no business context.

Animesh explains:

“We lacked depth because of not fully understanding the behaviours of the customers we were targeting. We didn’t know who was using the product. And we couldn’t measure the level of engagement accurately enough  to identify which accounts were healthy, and which weren’t.” 

The lack of visibility into customer behavior began to affect the customer success team as well. Not only was it blocking them from identifying and proactively reaching out to unhealthy accounts, but it hindered them from finding upsell opportunities, too. With limited-to-no ability to perform health checks, derive customer insights, or track the performance of onboarding, it created an endless cycle in which the success team could not get enough information for the product team to build better experiences for end users.

Building the case for more product telemetry with Twilio Segment

Animesh led the new product telemetry initiative and was responsible for gathering dedicated resources across the organization, from engineering to program management. Part of this process was showcasing the value of Twilio Segment to key executives, and highlighting the impact it could have on key business outcomes.  The centralized team that Animesh put together began identifying use cases and objectives the company had when trying to deliver telemetry using in-house tools. 

The biggest thing Animesh was looking for was standardization across all the different AppDynamics product areas.

“Standardization was the key characteristic of our new product telemetry initiative. We needed everyone to speak the same language and understand the same set of tools.”

Twilio Segment allowed him to do just that. AppDynamics could instrument new product features and capture user behavior events with a standard schema using Twilio Segment, then send this data to Amplitude for visualization. The whole process took minutes to instrument, at the end of which the data was instantly visualized and tracked against specific success metrics. The team was thrilled with their initial pilot program and began to roll Twilio Segment out to all the different AppDynamics product lines.

“With Twilio Segment, you only had to instrument once and then you could deploy it anywhere without wasting your developers’ time.” 

Telemetry for AppDynamics core product line

With Twilio Segment, AppDynamics has been able to achieve significant progress in telemetry with its core products. Each line has its own decentralized way to collect and monitor data, and every feature that is released has its own telemetry. 

Not long after the team implemented Twilio Segment, they started to see positive changes:

• Product managers used customer behavior data to improve their discovery process and conduct more productive customer interviews

• Product teams could  also identify the best users for specific feedback.

• Customer success teams could identify and act on the most and least engaged accounts.

• Management teams could now identify success and failure of new product features, and better understand how features were being adopted.

The team can now collect user events from their apps and provide a complete data toolkit to every team in the company. Data flows into different destinations and no longer has to be maintained—driving operational, engineering, and developer costs down. 

Because of Twilio Segment’s simple APIs, the team was also able to quickly integrate new tools, maintain an ETL pipeline, and train new developers on product telemetry in Twilio Segment. 

This new data foundation created by Animesh and his team has done so much more than just achieve telemetry. In his own words: 

“B2B tools take a lot of time to prove their utility. With Twilio Segment, we are able to see adoption ramping up in a very short timeframe. It’s a powerful way to keep everyone from developers to product management teams motivated.”