This is the first of a 2-part series giving a primer on Modern Data Infrastructure.
Driven by demand from businesses seeking greater value and more control over their customer data, a new approach to data analysis and activation has emerged and become the “go-to” approach for building a data architecture.
Modern Data Infrastructure (MDI) – sometimes called the Modern Data Stack – is centered around an ecosystem of tools businesses use to collect, move, store, transform, analyze, and operationalize their data.
The MDI market has seen explosive growth in the last 5+ years. In fact, between 2015 and 2020 alone, the top 30 data infrastructure startups raised over $8 billion of venture capital. Compared to traditional data architectures and approaches, MDI tools are easier and faster to implement, easier to deploy and maintain, easier to build on top of, usually less expensive, and offer speed and scale as features.
So, with the category gaining so much traction, we thought it would be helpful to put together a primer on the what and why of MDI.
What is a Modern Data Infrastructure?
MDI is a combination of easy-to-integrate tools that can be tailored to solve very specific business use cases. It’s difficult to pinpoint when exactly MDI took off, but it’s likely the first tools that spawned the creation of MDI tools were cloud-based CDPs like Twilio Segment.
CDPs are often a foundational piece of an MDI. CDPs make integrations, one of the most challenging parts of building a traditional data infrastructure, easy. This easy integration has allowed tools that run on customer data to flourish and an ecosystem of data tools to form.
MDI tools share some common features that make them easy to use and economical compared to old-school data architectures.
Common characteristics are:
Cloud-based. MDI tools are cloud-based and managed. They commonly have a free-tier to their managed offering and are often open-source projects that can be hosted on cloud infrastructure.
Warehouse-centric. MDI tools operate directly on the data in your own warehouse or lakehouse. Our research showed that the warehouse is the main trusted data source because it maximizes access, control, governance, and flexibility. Note: We use “warehouse” to mean both data warehouse and data lakehouse architectures in our writing. Their use cases are converging and the functionality they offer is overlapping and intermingling (for example, Snowflake’s external tables support data lakes by acting as the analytics layer on top of them).
Modular and Best-of-Breed. MDI tools can be swapped out for other tools that provide the same or similar functionality. This lets customers of MDI tools choose the best tool for their specific needs and circumstances. This also helps customers of MDI tools avoid vendor lock-in.
Easy to configure. You don’t have to know how to code or manage VMs or k8s on your cloud platform to use MDI tools. They are built to operate with existing cloud platforms and other cloud services. It’s generally as simple to integrate two systems as configuring access via their SaaS APIs or UIs.
What business problems does a Modern Data Infrastructure help solve?
All-in-one marketing platforms – such as Salesforce Marketing Cloud and HubSpot Marketing Hub – have become popular over the last 10-15 years, and with good reason.
They give businesses a single platform for all their digital marketing activities and help create unified customer profiles. They are far from perfect though.
They are rigid in their functionality, the customer profiles they create are often limited and rigid in use, and they make it difficult to work with your data outside of their tooling. Businesses built with MDI avoid this rigidity and expand the possibilities of what they can do with their data.
This helps them solve problems such as:
MarTech and Marketing Analytics. The pandemic made it clear that businesses lacked the agility they needed to operate in a digital-first market. The expectation is you deliver a personalized experience to your customers. To do that, you need the ability to target the right audience with the right message across your digital properties, respond to customer behavior quickly across channels, and measure the impact of everything you execute so you know what works and what doesn’t. All-in-one marketing platforms make it difficult to be as agile as your business needs. Integrating a set of best-of-breed tools is often the best way to help your Marketing team be more agile and deliver better, personalized experiences. MDI was born from this use case and is often the best way to solve it. This is one of the primary use cases for Twilio Segment as well.
ML and Data Science. Durable data storage is inexpensive with MDI. This encourages collecting every bit of customer data you can and storing it in a warehouse. Smart businesses then unleash data scientists and analysts to extract the most critical insights from their data and then apply those insights to their products, processes, and businesses.
Operational Analytics. Businesses want fast, reliable activation of their data analytics to help solve an array of use cases from Sales to Marketing to Product and beyond. The manual, error-prone processes that exist today to get this data into the tools they use are too fragile and slow. MDI tools – specifically reverse ETL – allow you to automate the integration of your data team’s work into any business application. These data pipelines are consistent, resilient, and automated.
What benefits do businesses get from a Modern Data Infrastructure?
Beyond avoiding the pitfalls of all-in-one marketing platforms, a well-built MDI offers businesses several other benefits. These benefits stem largely from the fact that MDI tools are almost exclusively managed SaaS, are frequently built to be cloud native, and are intentionally designed to integrate with other tools in the MDI ecosystem.
The primary benefits a well-built MDI offers businesses are:
Solve bleeding edge use cases. MDI tools give your digital business a competitive advantage by giving you faster and greater value out of your customer data. These tools along with new, long-term data storage strategies enabled by the low cost of storage in the warehouse, let you layer MDI tools on top of large, historic data sets to enable specific, tailored solutions that you couldn’t build before such as conversion attribution, advanced lead scoring, and ML-driven personalization.
Speed and scale of data. Since MDI tools are almost always managed SaaS, they can scale almost infinitely. Expensive computations that took hours or longer with on-prem systems can be scaled in the cloud – often transparently to the user – driving down the time they take to execute.
Easier and faster implementation. It doesn’t take weeks or months to add a new data system anymore. You don’t have to provision infrastructure or deploy software. It’s easy to swap out or add MDI tools to suit your specific use cases. Now you can maximize flexibility and control while still keeping your costs low. You sign up for a new service and configure the integrations you need via the UI. It can take as little as a few hours to set up and integrate a new tool now.
Reduced cost and maintenance. MDI tools generally have consumption-based pricing. You only pay for what you use. Since MDI tools are almost always managed SaaS, you don’t have to invest in maintaining the systems. Maintaining five nines and scaling for peak are no longer your concern. That’s what you pay your vendor for.
Easier to build on top of. MDI tools are built to plugin with other tools. They frequently offer out-of-the-box integrations with popular cloud services and platforms, so stitching together low-or-no-code systems is easier. Product-oriented MDI tools also regularly offer APIs or SDKs, so plugging them into custom-built applications or websites is easy.
Now that we’ve given you the tl;dr of MDI, join us for part two where we'll cover functional features of an MDI (i.e. what an MDI enables you to do), non-functional features of an MDI (i.e. important, sometimes subjective aspects of your data infrastructure that are influenced by an MDI), and connect those features to some of the new data tools on the market today.