What’s a Warehouse?
A warehouse is a central repository of data collected from one or more sources. This is what commonly comes to mind when you think about a relational database: structured data that fits neatly into rows and columns.
In Segment, a Warehouse is a special type of destination. Instead of streaming data to the destination all the time, we load data to them in bulk at regular intervals. When we load data, we insert and update events and objects, and automatically adjust their schema to fit the data you’ve sent to Segment.
When selecting and building a data warehouse, consider three questions:
- What type of data will be collected?
- How many data sources will there be?
- How will the data be used?
Relational databases are great when you know and predefine the information collected and how it will be linked. This is usually the type of database used in the world of user analytics. For instance, a users table might be populated with the columns
Examples of data warehouses include Amazon Redshift, Google BigQuery, and Postgres.
When Segment loads data into your warehouse, each sync goes through two steps:
- Ping: Segment servers connect to your warehouse. For Redshift warehouses, Segment also runs a query to determine how many slices a cluster has. Common reasons a sync might fail at this step include a blocked VPN or IP, a warehouse that isn’t set to be publicly accessible, or an issue with user permissions or credentials.
- Load: Segment de-duplicates the transformed data and loads it into your warehouse. If you have queries set up in your warehouse, they run after the data is loaded into your warehouse.
Looking for the Warehouse Schemas docs?
They’ve moved! Check them out here.
Setting up a warehouse
Managing a warehouse
Analyzing with SQL
This page was last modified: 25 Feb 2022
Questions? Problems? Need more info? Contact Segment Support for assistance!