Use Twilio Segment to Predict Fraudulent Behavior

On this page

Enter Twilio Engage, a powerful tool provided by Twilio Segment that harnesses the potential of AI to detect patterns and predict behaviors. By leveraging Predictive Traits (currently in Beta), insurance companies can swiftly identify customers who display potential signs of fraudulent behavior. This cutting-edge solution equips fraud risk teams with the necessary information to take proactive measures, ensuring the integrity of their business and safeguarding the interests of genuine policyholders.

In this recipe, we will explore how to set up Predictive Traits, a new feature of Twilio Segment currently in Beta, to detect customers with patterns of behavior that are likely fraudulent. By leveraging AI through Segment, insurance companies can catch customer fraud early, providing them with a competitive advantage in the ever-evolving landscape of insurance fraud prevention.

Requirements to Follow Along

If you plan to follow along with this recipe by working out the steps on your own machine, be sure to have the following:

• An account for Twilio Segment (free to sign up), which is Twilio’s customer data platform.

• Within Segment, you’ll need access to Twilio Engage, which is a data-first multichannel marketing solution. Engage is a product add-on for Segment users on the Business Plan. Contact your account representative to activate access to Engage.

   

• Access to Predictive Traits. Predictive Traits are a type of Computed Trait used for segmenting audiences in Engage. Because the feature is still in beta (at the time of this writing), access to Predictive Traits is still on an invite-only basis. You can request an invite by going to Engage -> Audiences -> Computed Traits. Then, select Predictive Traits and click the Request Demo button.

Data Setup

For the purposes of this demo, we’ll work with a dataset that contains customers and claims made by customers with some claims marked as fraudulent. For each customer, we have generated a set of events. Events are one of two types:

1. Most events are marked with the event name “Claim Made”.

2. For a subset of those claims, an additional event has been reported, named “Fraudulent Claim Made”.

To load our customer and claim data into Segment, we created an HTTP Tracking API source. We used this source to make Identify calls to Segment. Segment uses Identify information in order to associate future actions with customers based on their ID or email. Next, we made Track calls to associate “Claim Made” and “Fraudulent Claim Made” events (one event per HTTP request) with our customers.

For the purposes of this recipe, we want to predict which customers are likely to have a fraudulent claim reported. To do this, we’ll build a Predictive Trait and use machine learning.

Build a Predictive Trait

If this is your first time working with Twilio Engage, you’ll first need to set up Unify. You can find detailed setup instructions for Unify here.

To begin, click on the Engage link in the Segment navbar.

In Engage, on the Audiences page, we click Computed Traits and then Create Computed Trait. Finally, we select Predictive Traits.

For this recipe, we use the Custom predictive goal option.

When you’ve finished configuring your Predictive Trait, click Create Trait. From there, Segment will take you to the page for your newly created predictive trait.

The initial Status will show as In Progress. It will take some time for Segment to complete the initial calculations for this trait. For larger datasets, this could be up to 48 hours.

Data Requirements for Successful Predictions

Segment’s documentation for Predictive Traits makes clear the following:

Segment doesn’t enforce data requirements for predictions. In machine learning, however, data quality and quantity are critical. Segment recommends that you make predictions for at least 50,000 users and choose a target event that at least 5,000 users have performed in the last 30 days.

For this reason, it’s important that you use a large enough dataset in order to train Segment’s AI engine for predictive computations. Otherwise, you might encounter warnings, unexpected outcomes, or even failed trait computation.

Understanding Predictive Traits

Once you have the prediction set up and the trait computation has been completed, navigate to the Prediction tab. In this tab, you will see the Explore your prediction section. This section lets you visualize and browse your prediction data. From here, you can create custom Audiences from the group for further use, as well as send this data to Destinations. For our example, we’ve set up an AWS S3 bucket as a Destination. As Segment predicts which customers might exhibit fraudulent behavior, that information makes its way as files in S3.

In the Understand your prediction section, you will see information about the quality of the prediction and the statistics which underlie the predictive model.

Area Under the ROC Curve (AUC)

AUC is a measure of the model's ability to distinguish between positive and negative instances. A higher AUC value (closer to 1) indicates better performance. You can use AUC to compare different models or variations of the same model. A higher AUC suggests that the model has a better overall ability to classify instances correctly, regardless of the classification threshold.

Lift quality

Lift quality helps evaluate the effectiveness of a model or a campaign in generating positive outcomes compared to a random approach. Lift measures how much better the model performs compared to random chance. Higher lift values indicate better performance. By analyzing lift values across different segments or groups, you can identify areas where the model is particularly effective and allocate resources accordingly. Lift quality is especially useful in marketing applications.

Log loss

Log loss measures the accuracy of a model's predicted probabilities. It assesses how well the model's predicted probabilities match the true outcomes. A lower log loss indicates better performance. Log loss penalizes the model more if it's confident and wrong, and rewards it for accurate predictions. You can use log loss to compare different models or tune the parameters of a model. It provides a more nuanced assessment of the model's calibration and probability predictions.

By considering these metrics together, you can gain a comprehensive understanding of your predictive model and evaluate the model’s performance at a detailed level.

Conclusion

In this recipe, we’ve walked through how Twilio Engage, a part of Twilio Segment, can be used to enhance fraud detection. By following the outlined recipe requirements and setting up the necessary data infrastructure, insurers can harness the power of AI to build Predictive Traits that identify potential signs of customer fraud. By using the insights from Twilio Segment, insurers can level up their fraud detection and prevention efforts.

This recipe serves as an introduction to Predictive Traits, but the capabilities do not end here. For more information on Predictive Traits and ideas on how to integrate them into your workflow, see the documentation or schedule a demo today.