How Segment Ensures Data Validation at Scale

Section

Imagine you find a packet of chips in your cupboard. The expiration date reads: 11/05/23. But does that mean November 5th or May 11th? The answer will vary depending on where you are in the world, and is just one example of how something that can seem so obvious can actually cause confusion. 

It’s exactly why data validation is so important. 

Data validation is the process of checking that data is accurate and valid before using it, preventing misunderstandings and faulty decision-making. 

What are data validation tools and why are they important?

Data validation tools are software (or capabilities built into software) that assess data against your tracking plan and flag non-compliant entries. For example, if you require a date format of MM/DD/YYYY, a data validation tool would flag, reject, or transform an entry that’s formatted as DD/MM/YY. Another form of data validation is to limit the type of data you can input within a range – say, for dates in November, you can only input 1 to 30.

By using these tools, you enforce quality standards, privacy compliance, and data hygiene. These are a necessary foundation for gleaning accurate information and reliable, evidence-based insights.

Challenges in data validation at scale

Though simple in principle, data validation gets challenging pretty fast. With the mass amounts of data being collected, and the variety of that data, organizations can struggle to ensure validation at scale. Here are a few common challenges to keep an eye out for. 

Diverse data collection points

Take a look at this sample production line. (Yes, we’re still fixed on the chips.) From mixing and cutting the dough to cooling and packaging the chips, every step in the process generates a ton of data, especially if the machines have IoT sensors or relay activity inputs to digital platforms. Different sources and processes would generate data like machine speed, oil temperature, and product volume. They’d also provide input on “cooked by,” “packaged by,” and “consume by” dates.

Having diverse data collection points means you need to be sure that all necessary sources are connected to your data pipeline. This presents the challenge of interoperability – ensuring your tool integrates with your existing systems and doesn’t break your workflows. You’ll also have to make sure each data type is labeled distinctly enough for analysts to avoid confusing one for the other (e.g., “date manufactured” is vague compared to “date baked” or “date packaged”).

Volume and velocity of incoming data

The Frito-Lay factory can make 14,000 pounds of potato chips per hour. When Dollar Shave Club launched, they sold 12,000 subscriptions in 48 hours. Every year, quality graders at Blue Bottle taste more than 1,000 types of coffee to select the best-tasting beans and roasts. 

Each of these use cases requires a different data validation system, depending on how much data it collects, how often, and how fast.

In a giant manufacturing setting, automated QA would be an essential feature of a data validation system. Teams anticipating a surge in orders for a limited period need to set up formatting rules, automated transformation, and temporarily larger bandwidth. On the flip side, manual review is important when data comes in more slowly but also involves lots of nuanced properties, qualitative observations, and opinionated ratings (like the coffee-grading example).

Evolving data standards and formats

As your business evolves, your data standards and formats will, too. You might start a direct-to-consumer business specializing in one thing: making the crispest, tastiest, healthiest potato chips. You sell one product in one flavor and one package size. That means your e-commerce tracking plan would include specs on product SKUs, but you wouldn’t need product IDs and product categories.

As you expand your catalog – say, by selling different package sizes and new flavors – you’d add relevant data properties to your plan. You’d add new events to capture, like when someone views a product category on your site. 

Key features to look for in data validation tools

Improve the efficiency, accuracy, and reliability of your data operations by looking for the following capabilities in a data validation tool.

Automation

Automated data validation tools remove the need for you to check, accept, reject, or reformat every single entry into your system. They let you set and update naming conventions, standards, and privacy rules and automatically perform QA and transformations based on whether or not entries conform. These tools may also give you the option of reviewing non-conforming items instead of automatically rejecting them.

Segment Protocols allows teams to correct bad data or customize data for specific destinations

Real-time validation

If you need to act on events as soon they happen, you’ll need real-time validation. Say you have a one-hour promo on your e-commerce site. You’ll need to instantly check the validity of inputs like payment info, email ads, and delivery addresses. 

Real-time validation can also be mission-critical, particularly when it comes to big data systems. Consider semi-automated vehicles that decide whether a traffic signal means stop or go, health-monitoring devices that issue emergency alerts based on vital sign activity, or factories that run cooling systems whenever the temperature exceeds a certain level.

Cross-checking

Rigorous data standards often require triangulation – or cross-checking input against sources to boost accuracy and consistency and to guarantee a product’s quality. For example, to ensure that the “consume by” date on a bag of chips is correct, you’d check it against information like when and where the raw material was sourced and when the snack was baked and packaged. 

Segment's approach to data validation: Ensuring quality data flow at scale

At Segment, we’ve long lived by the mantra: what good is bat data? This is why we advocate that organizations prioritize data governance and use their tracking plan as the foundation for data collection and validation. (Of course, tracking plans aren’t one-and-done documents – it’s good to review and update your plan periodically.)

Using Protocols, you can then automate QA checks and proactively block bad data before it enters downstream destinations. 

And with Transformations, businesses can change data that doesn’t conform to their predefined plan, or customize data for specific destinations.

Take a look at Segment’s Public API to learn more about Transformations and its different use cases.