Lessons from SIGNAL Conference: Reduce bottlenecks and manage data processes with code using Segment’s API

The sessions of Signal 2022 featured diverse stories of how customers improved the way they use data. We saw how Veronica Beard reduced their Customer Acquisition Cost with first-party data, and how IBM used Twilio Segment to improve product adoption.

One other customer we highlighted is not a stranger to us here on the Segment blog. Staff Data Engineer at Endeavor, Ilya Galperin, previously wrote about how Segment helped his team create role-based access control.

In his Signal session, Ilya added on to that idea and explained how Endeavor scaled data infrastructure across an enterprise portfolio across their data platform using Segment. The result was ensuring everybody in the organization had access to the same high-quality data.

If you want to watch Ilya’s session, click here! Or if reading is more your speed, keep going.

Endeavor and the tech stack that powers their data

Endeavor is the global leader in entertainment, sports, and fashion. They are a holding company for talent and media agencies, and house many popular brands, including Ultimate Fighting Championship (UFC), Professional Bull Riding (PBR), Euroleague Basketball, and more. They also manage many events and even their own streaming service.

Perhaps needless to say, all of these brands and events generate their own unique set of data.

Ilya sits on the Digital Growth team. Their goal is to enable business units within Endeavor (mostly Marketers) to access their data and create actionable insight off of it.

Endeavor has a large tech stack they work with. Tools used across functions include:

• Business intelligence -  Snowflake, Amazon Quicksight, Streamlit, and more

• Marketing - MailChimp, Facebook, Google, and more

• Integrations - Endeavor API, Salesforce, Cloud Storage, External FTPs

• Machine Learning - Endeavor Machine Learning Platform (developed in house), Vidora, and more

All of these tools had their own administrative and user access, which resulted in a lot of overhead for the Endeavor engineers.

With over 7,000 employees, Endeavor found it difficult to scale their efforts as new tools were being added and new users joined the team.

Endeavor faced concerns in three main areas:

The first issue they had was with scope. Endeavor integrated with a lot of different endpoints to keep organized. Every subsidiary had its own website (sometimes multiple sites), mobile and web applications, Facebook ads, Salesforce instances. It became difficult to manage all these tools.

Endeavor also had an issue with volume. They have over 70 sources, and would get weekly/daily requests for access into Segment, to on/off board Segment sources and destinations. They also had high traffic systems with a lot of customers using their websites/apps.

Finally, Endeavor faced an issue with governance. It was important to have separation of data between different subsidiaries - marketers for Euroleague shouldn’t be looking at UCF user data, which would lead to a disorganized mess of data.

It became clear that the team at Endeavor could not manage all this data and keep it organized through human users. So they turned to Segment to automate the process.

Integration of websites and apps led to organized data and provided access control

To keep the data throughout their tech stack organized, they came across Segment. It seemed a natural fit since a lot of Endeavor’s top-priority tools were already in the Segment integration catalog.

With the use of APIs to connect their tools, Segment enabled Ilya’s team to integrate with sources like javascript websites, mobile applications, web applications that are developed both in-house and out-of-house.

The Segment API allowed Endeavor teams to:

• Add permissions to their users

• Create new (or remove old) sources and destinations

• Add connections from the source to the warehouse

• List invites

With more control over who had access to what tools in their stack, Endeavor could start to change these permissions using code, which removed the overhead of manually adding users.

Below is a setup of their tech stack, and where they included Segment:

Segment Public API enabled self-service and reliability at scale

The self-service methods provided by Segment allowed the engineering teams more agility and less time waiting for one user to grant permissions. Ilya broke up the benefits into three main areas:

1. Ilya shared his idea of the “nuclear sub principle,” in which two pilots each have keys and in order to launch a missile, both keys need to be inserted for the missile to successfully go off. Endeavor’s take on this is ensuring peer review. Engineers review fellow engineers' code, and once approved, the code gets added to Segment. This removes administrator bottlenecks which would act as a single person to add or remove a user, with no single point of failure.

2. You can understand what actions a user did take in Segment since its code base is auditable. The Endeavor team could determine who had access to what resource within Segment at any point in time by reviewing the code base.

3. The Segment infrastructure is defined in configuration files in code, allowing the team at Endeavor to reproduce code from one environment to the other. Endeavor has two Segment workspaces- one for production and a more “experimental” workspace. The experimental workspace acts as a sandbox for engineers, data scientists, and marketers to poke around without causing damage in production. Ilya noted that with copying code, it is very easy for the sandbox to mirror production.

Below are examples of these configuration files. This is a configuration file for a user group.

On the right, we see how this code looks in the Segment user interface. 

When we dive into the code, note that Property[X] can be replaced by any Endeavor subsidiary. User status for each group can be toggled between active and inactive, and all the permissions defined (ex. Read only access) to any source with a label that carries their property/business unit’s name. They have access to a Persona [now Engage] space made for their unique business unit.

Each user to this group is made visible. Below that, you can see the Endeavor-created Segment user. Ilya’s team creates a configuration file for each new marketer, and their permissions and access is defined right in this code.

The image below shows how Endeavor configured connections.

The right shows the sources created from this configuration file.

The code on the left Created sources from configuration files. Property slug translates to one Endeavor subsidiary. The connections created for it are seen below- for the main website, sponsor website mobile apps, etc. Note that the sources can be enabled/disabled. This is an easy way to manage all 70+ sources directly in the code.

With user and connection access able to be modified right in the code, Ilya’s team was able to on/off board at scale as their teams and sources grew.

Self-service for marketers leads to data-driven campaigns

Now that we have a background in the code and tech setup, what was the benefit to markers?

Endeavor created a virtual cycle where marketers were prioritized and able to work in a self-serve manner without a lot of intervention from the engineering team.

By creating sources, they measured behavioral data from websites and applications. Marketers could combine this data with other sources in Segment and the Snowflake warehouse or bring this back to a Persona [Engage] space to create audiences and computed traits.

With destinations, marketers could export these audiences to an email service provider, facebook, instagram, or other marketing channels. The data was then captured directly in Segment, so it was sent back to Snowflake, then their Personas [Engage] audience to build marketing audiences.

Conclusion

Segment helped Endeavor organize the data from its subsidiary entertainment brands and funnel that data to the right teams. It allowed engineers access to edit permissions through code, saving hundreds of engineering hours a week. And it provided marketers with a self-service way of working, so they could focus on building data-driven campaigns.