How Twilio Segment proactively protects customer’s API tokens

Nothing good ever comes from committing an API token to a git repository.

Malicious actors leverage search tools provided by GitHub and GitLab to find API tokens, private keys, usernames and passwords in public repos — juicy secrets that can be used to steal data or run up a big bill.  In a recent example, a hard-coded secret went unnoticed for 5 years, potentially giving attackers access to data of over 200,000 customers.

This isn’t anything new. Secret leakage is a pervasive problem that can have costly consequences for organizations — over 6M secrets were detected in public GitHub repos in 2021.

A study by the North Carolina State University research team found that the median time for secrets to get indexed by GitHub search "was 20 seconds, with times ranging from half a second to over 4 minutes” and “consequences of even rapidly detected secret disclosure is severe and difficult to mitigate short of deleting a repository or reissuing credentials”. 

Once a secret hits a public repo, that API token or password is compromised. There’s no other choice but to rotate it.

Secrets are also put at risk by ineffective off-boarding procedures at many organizations. In some applications, API tokens created by users no longer part of the company may remain. At Segment, we call these "orphaned" tokens, and they can be a cause for concern since they are known by former users.

In this post, I'll explain how our team, the Security Features Team, built solutions to protect our customers from committed and orphaned secrets.

Secret scanning

Luckily, we didn't have to build a secret scanning solution from scratch. GitHub and GitLab offer secret detection/scanning features which alert users when a secret is committed. They both offer partner programs for SaaS providers to take advantage of these features. All we had to do was provide an API token regex pattern, set up a public endpoint, and GitHub/GitLab started sending any matching tokens.

The sequence diagram above is a simplified view of the planned architecture and what happens upon secret detection. Our team's path was clear: we needed to set up a simple REST API server with one publicly accessible endpoint.

Warn or revoke?

The ball is in our court once GitHub/GitLab sends over exposed tokens and we came up with 3 potential options:

• Warn the customer, then leave it up to them to take action.

• Revoke the token, then notify.

• Warn, then revoke the token after 24 hours.

The first option would be the least disruptive to our customers' workflow, but the least secure, since we couldn’t guarantee the notification would be seen in time, or at all. The second option would be the most disruptive, but the most secure, since there would be no chance for bad actors to take advantage of the exposed token. The third option would be a middle ground between both, but, as previously mentioned, there would be no guarantee the warning would be seen in time, and it still leaves time for exploitation. It would also introduce more complexity into our architecture, since we'd need a way to keep track of these exposed tokens and revoke them after a certain amount of time.

We wanted to strike a balance between user experience and security. Would we rather our customer have a security incident where the end user’s (our customers’ customer) data is potentially at risk? Or an incident where there’s some potential lapse of event delivery (that can easily be recovered) and the solution is to simply generate a new key?

We also took a look at GitHub’s secret scanning partners and what they do in the event of an exposed token:

Partner - Method

• Meta - Auto-revoke

• Readme - Auto-revoke

• Zuplo - Warn or auto-revoke (customer preference)

• Sendgrid - Auto-revoke or manual review (depending on token scope)

• DigitalOcean - Auto-revoke

• Supabase - Auto-revoke

• Octopus Deploy - Warn

This small sample gave weight to the decision to proceed with automatically revoking exposed tokens and notifying workspace owners. 

API token pattern

Knowing we would eventually tackle this feature, earlier in the year we changed our API token format so it could easily be pattern matched with a regular expression. Our original pattern was a generic alphanumeric 64 character string ([a-zA-Z0-9]{64}). Using this pattern would have resulted in an untold number of false positives so instead, inspired by GitHub, we added an identifiable prefix to our API token strings. 

`sgp_` standing for SeGmentPublic API token was added and the new pattern would be `sgp_[a-zA-Z0-9]{64}`. Implementation was straightforward:

1. Add a column to the token table to track the token version. All current tokens would be version 1 and the new `sgp_` prefixed tokens would be version 2. If we need to make another fundamental change in the future, those tokens would be version 3 and so forth.

2. Update relevant portions of our backend to account for the new version field.

3. When tokens are created, the `sgp_` prefix is added and a hashed token is stored.

With a new secret format at hand, I submitted a PR to the gitleak project (a great tool for detecting and preventing hardcoded secrets in git repos) and shortly after, the secret scanning team at GitHub reached out inviting us to join their secret scanning partner program. Perfect timing.

Exposed token service

After letting GitHub and GitLab know about our API token format and planned launch date, we proceeded with planning our exposed token service. The end goal would be to set up a public HTTP endpoint both providers can send found tokens to.

Both providers send about the same request body (GitLab leaves out source key) so re-using the core revocation logic would be easy: 

The GitHub documentation on setting up a service like this was great and provided us with all the information we needed to get started. We chose to go with a simple NodeJS express server since it's easy to get up and running with all the existing Segment boilerplate and automated tooling available. However, it’s worth noting this type of workflow lends itself very nicely to a serverless architecture if you don’t expect to receive a lot of exposed tokens.

Once the plan was finalized and approved, it took one engineer about a month to get this out the door. Now, if you ever accidentally commit a Segment token, you’ll get something like this in your inbox:

This email notification is sent to workspace owners each time a token is revoked. It starts by providing metadata about the token and includes a link to where the token was found, thanks to the data supplied by our secret scanning partners.

The following section outlines the steps a customer should take. We recommend checking for any suspicious activity (as an extra precaution) and creating a new token to replace the revoked one. The last section explains that, with the help of our secret scanning partner, we were able to detect and revoke a token that may have been committed accidentally — ensuring the customer it wasn’t Segment that leaked it. 

Metrics

Using our data warehouse, Snowflake, we can run queries and create dashboards to gain insight into how well this new feature is performing. Specifically, we want to know:

• The origin of exposed tokens

• The ratio of revoked tokens to false positives

• Which workspaces had the most revoked tokens

We drink our own champagne and use Segment to collect these metrics. We fire off a track call every time we receive a token from our secret scanning partners and that gets funneled into Snowflake.

Our track call contains the following properties:

• userId: what user triggered this event (we use ` __system__` to show the service triggered the track call)

• event: name of the action that has been performed.

• origin: what secret scanning partner reported the token

• prefix: the token id (the first 5 characters after `sgp_`)

• source: where the token was found

• status: if the token was revoked or a false positive (invalid)

• url: address of where the token was found

Tracking meaningful security product metrics is important to our team. We don't have significant data yet, but we hope to use it to improve the service and aid in future planning efforts. Our security incident response team can also use this data to spot any trends and take necessary actions to protect our customers.

Orphaned API tokens

What about tokens that aren’t necessarily leaked but known by users who are no longer part of your organization? In some applications, API tokens are tied to a user and can’t be used once a user’s permission from an organization is removed. But, at Segment, they are instead tied to a workspace.

When you first log into Segment, you end up in a workspace. Workspaces help you manage access for multiple users and data sources — it’s where all of Segment’s functionality lives.

From here, you can generate an API token that can be used to programmatically manage the workspace. Since this API token is tied to the workspace, not a user, it gives customers the ability to assign different roles and permissions to tokens.  But, this could be a problem since a user can generate a token, write it down, leave the workspace, and still have the ability to perform actions in the workspace.

You end up with what we refer to as an “orphaned” token and this may raise concerns from workspace owners, who expect that users who are no longer part of a workspace are no longer able to perform workspace-specific actions.

Deleting a token as soon as its creator departs would minimize the likelihood of a disgruntled ex-employee causing chaos. However, it’s often the case that folks leave a company on good terms and immediately rotating tokens created by one individual might not be too important to our customers. This is vastly different then a public exposure of a token where someone malicious is guaranteed to eventually find and exploit the token.

Instead, we chose to alert workspace owners and they could decide whether a token is important enough to rotate, delete, or leave as is. Workspace owners will receive emails and see in-app warnings indicating which tokens need attention.

Architecture

This was made possible by a new service we called PAPI alert service. It’s a simple cron job written in TypeScript that runs once a day and is responsible for finding orphaned tokens and alerting workspace owners. At a high level, the service fetches all token creators grouped by workspace ID and fires off a notification for each workspace who has a token creator that is no longer part of that respective workspace. 

Below is a simplified version of how orphaned tokens are found.

We also added two new columns to our API token database table: `firstAlertedAt` and `lastAlertedAt`. `firstAlertedAt` is the date the workspace owners were first notified about the orphaned token while `lastAlertedAt` is the most recent date workspace owners were notified.

Together, these columns are used to determine whether or not to send a notification when an orphaned token is detected: send an initial email if `firstAlertedAt` is null, or send a follow up notification if it’s been 6 months since the `lastAlertedAt` date — a follow up notification will continually send until the token has been deleted. We want to remind customers their workspace contains orphaned tokens and nudge them to rotate them.

Metrics

Previously, I mentioned deciding not to auto-delete orphaned tokens, which was a decision taken during the planning of this project. This was further reinforced when the metrics started to roll in: 25% of orphaned tokens were deleted after the first notification and 13% after the follow-up notification.

At first glance, it seemed like users were okay leaving these orphaned tokens around. However, a closer inspection revealed that most of these tokens were used to support our SCIM feature. Typically, someone from the customer's IT team would join the workspace, generate a token, use the token to set up SCIM with their identity provider, and then leave the workspace. In our case, having an orphaned token was perfectly acceptable as long as our customer was aware.

Conclusion

We hope these two features keep our user’s API tokens safe and help prevent security incidents. By implementing these measures, we’re able to strike a balance between user experience and security, ensuring that the end user data of our customers is protected while also minimizing disruptions to customer workflows.

GitHub and GitLab are making big strides to help prevent secrets from making their way into repos and there are growing efforts in the community to combat this widespread issue (take a look at RFC 8959). In the meantime:

• Leverage existing token/secret scanning partner programs to easily notify customers about leaked credentials and help prevent security incidents.

• Having an easily identifiable prefix for your API tokens/keys make it easy for users to leverage existing tools that detect hard coded secrets and prevent them from being committed.

• Consider notifying customers when a token’s creator is no longer part of the organization/company.