Enhancing personalized AI experiences with Warehouse Data

Thanks to OpenAI's ChatGPT feature, Artificial Intelligence is capturing the public’s imagination like never before. While OpenAI didn't invent AI, they have played a pivotal role in making generative AI widely accessible and engaging for the masses.

As individuals explore the creative side of AI, businesses are getting in on the fun, leveraging AI to enhance customer relationships. By focusing on tangible examples of AI solving real customer challenges, we can appreciate its true impact beyond the hype.

In this blog, we will introduce an example of how Twilio Segment uses an AI powered customer support workflow that is able to understand a full 360 view of the customer, including their previous purchases, documentation of those productions, and prior support interactions.  This rich context, sourced from a data warehouse, greatly increases the chance of an appropriate tailored response to the support request and reduces the need to escalate to expert agents. 

Background Context

Companies have vast amounts of data stored in their data warehouses, but don’t always have the capability to turn these data stores into actionable value.  This is where a semantic layer such as Segment’s Data Graph comes into play.  

By enabling businesses to map and understand the relationships between different datasets about their customers with tremendous flexibility, data becomes democratized by empowering marketers and business teams with all the data they need to create personalized experiences. 

Foundation model LLMs can produce impressive results all on their own, but an increasingly common approach is the use of Retrieval Augmented Generation (RAG) in order to provide valuable context to reduce hallucinations and ensure personalized results.  This approach is well suited for many use cases: legal document analysis, medical diagnostics, e-commerce product recommendations, and academic research. A core commonality of these use cases is the need for highly specialized and relevant contextual information.  Without enriching the base generative model with this context, the answers will be limited in the specificity and salience. 

In today’s example, we will walk through an in-depth example to combine these concepts by having a knowledge graph powered by RAG architecture to provide excellent automated customer support.  In-context learning is extremely powerful and can often be sufficient to avoid the need for more complicated solutions like fine tuning.  Leveraging increasingly large context windows is an excellent way to improve LLM results.  However, this approach comes with distinct operational tradeoffs including increased latency, higher costs, and the need to manage RAG infrastructure.  For any use case, finding the right balance between these tradeoffs will likely require experimentation.  Being able to optionally include additional context or altering the amount of that context that is provided is one path to such experimentation. 

Problem

You want to provide an automated customer support experience that can handle as many of the easy cases as possible to preserve the expertise of your customer support specialists for more difficult cases. This is because:

• Highly skilled customer support specialists are expensive

• AI is now advanced enough to handle routine or easier customer support

However, AI can only handle routine customer support if we use an appropriate AI architecture. Otherwise, the results could be quite wild, providing hallucinations with incorrect replies.

Here’s a concrete example that we can use to understand the use case and have a clear reference point for walking through the details.

 We have the following requirements:

• Batch Latency 

  • potentially through an email interface

• Low Volume 

  • might increase to medium at some point if successful enough

• Entire Corpus doesn’t fit in the context window

• Quality needs to be high

  • if its low, then a frustrated customer will still need to be helped by a customer support specialist

Solution

A graph based RAG architecture is a good fit for this use case because we want to use a lot of additional context for the prompt to ensure good results and that there isn’t an operational limitation such as cost or latency that prevents us from doing so.  Additionally, the entire corpus of data doesn’t fit in the context window, so we need to limit the augmentation data to only the most relevant as well as fetching personalized information.  Despite its increasing popularity, it’s essential to make sure that the use case is a good fit for RAG or you’ll risk over-engineering and wasted effort. 

As we can see in this example, the RAG context can substantially improve the quality of the results.  Here’s a high level overview of what the entire system might look like:

Let’s trace our example through the system:

1. It starts when a customer submits a support request.

2. The Context Retrieval System (CRS) parses the email, does spam filtering to ensure it's legitimate, and looks up the customer ID from the email address. It uses a knowledge graph, such as Segment’s Data Graph, to construct a query of known customer purchases and performs entity extraction on the customer email to compare this to the returned purchases.

(A knowledge graph is a powerful framework for encoding an interconnected web of data in a way that is both human and machine usable.  By emphasizing entities, relationships, and semantic meaning, knowledge graphs can provide deep insight into underlying data and power many use cases.)

The system successfully identifies the “exercise bike” the customer refers to as Owl Pro Bike Model 4.

The CRS uses this knowledge graph to query the product documentation related to the Owl Pro Bike Model 4, and also customer support interactions for both this particular customer and for other customers related specifically to this product.

Then the system uses an embedding model to translate the customer request into an embedding for semantic search.  This semantic search can refine and find the most relevant results from the product documentation and customer support interactions. An example of how a data warehouse can natively provide this functionality is: SNOWFLAKE.CORTEX.EMBED_TEXT_768('snowflake-arctic-embed-m', customer_query);

It can be used in conjunction with VECTOR_COSINE_DISTANCE() to find the semantically closest results in our already filtered data. This uses this combination of traditional data lookups and semantic search to return a rich context that can be focused specifically on the question the customer is asking while remaining historically informed.

Now, we now have in our context:

• The product documentation for Owl Pro Bike Model 4 filtered by chunks most likely related to problems with operation and noise generation

• The customer’s prior customer support interactions

• Other customers customer support interactions specifically for the Owl Pro Bike Model 4 and the underlying customer problem of high speeds causing noise

3. This rich context is passed to the Answer Generation System. Although its core responsibility is to call an LLM with this context, it also performs some context validation and LLM security checks, including checking if there is a product match for the customer.

The AGS provides visibility and metrics and can be queried ad hoc by support specialists as needed. It can perform A/B tests of different prompt engineering or context retrieval workflows.

4. The answer is then passed through a quality control system visible to expert support agents. This intermediate layer allows quality control by surfacing a random sample of responses for human evaluation. A further iteration here can be to pass the model results to an evaluator model to ask for a confidence in resolving the issue, with lower confidence results being sampled more highly here. It also provides easy interaction when cases are insufficiently resolved by the AI system and need to be escalated.

5. The response is then finally returned to the customer either via email or in customer support portal.

In our happy path scenario here, the issue is resolved with the technician link, but if it hadn’t, they could have seamlessly gotten support from expert agents on follow up questions. 

Conclusion

Today we’ve walked through a concrete use case of providing a highly personalized automated customer support system that is powered by a semantic knowledge graph that provides specific, relevant customer and product information and context to an LLM.   Consider checking out Segment’s Data Graph to see if it might help you solve this or similar use cases you have.  Also keep an eye out for ever evolving Data warehouse native AI and ML capabilities like Snowflake’s Cortex that allow more of this process to be done directly in the warehouse rather than building complex supporting infrastructure.