Amazon Kinesis Firehose Destination

Segment makes it easy to send your data to Amazon Kinesis Firehose (and lots of other destinations). Once you've tracked your data through our open source libraries we'll translate and route your data to Amazon Kinesis Firehose in the format they understand. Learn more about how to use Amazon Kinesis Firehose with Segment.

Getting Started

When you toggle on AWS Kinesis Firehose in Segment, we’ll start sending your specified Segment events to Firehose delivery streams of your choice. Our Kinesis Firehose destination supports all of the Segment methods, and will send data from any one of our libraries.

Event Mapping

To begin using the Kinesis Firehose destination you must first decide on which Segment events you would like to route to which Firehose delivery streams. This mapping then needs to be defined in your destination settings.

Segment track events can map based on their event name. For example, if you have an event called User Registered, and you wanted these events to be published to a Firehose delivery stream called new_users, you would create a row in your destination settings that looks like this:

track event mapping screenshot

Any Segment event type (ie. page, track, identify, screen, etc.) can also be mapped. This allows you to publish all instances of a given Segment event type to a given stream. To do this, you simply to create a row with the event type and it’s corrensponding delivery stream:

page event mapping screenshot

Events can be defined insensitive to case so Page will be equivalent to page. The delivery stream name however needs to be formatted exactly as it is on AWS.

If you would like to route all events to a stream, use an * as the event name.

Data Model

Let’s say you’ve decided to publish your Segment track events named User Registered to your Kinesis Firehose delivery stream named online_registrations. If you send Segment the following track call:

{
  "userId": "user_1",
  "event": "User Registered",
  "properties": {
    "plan": "Pro Annual",
    "account_type" : "Facebook"
  }
}

The Segment Kinesis destination will issue a PutRecord request with the following parameters:

firehose.putRecord({
  Record: {
    Data: JSON.stringify(msg)) + '/n'
  },
  DeliveryStreamName: 'online_registrations'
});

Segment will append a newline character to each record to allow for easy downstream parsing.

Quickstart

In order to get started, you’ll need to perform the following steps:

  1. Create at least one Kinesis Firehose delivery stream.
  2. Create an IAM user to allow Segment permission to write to your Kinesis Firehose stream.
  3. Create an IAM policy to allow Segment to issue PutRecord on behalf of the user.
  4. Attach the IAM policy to the IAM user.
  5. Create a new Kinesis destination in the Segment UI.
  6. Specify at least one event that you would like to publish to a given stream.

1. Create a Kinesis Firehose delivery stream

Follow these instructions to create a new AWS Kinesis Firehose delivery stream.

2. Create an IAM user

Follow these instructions to Create an IAM user to allow Segment permission to write to your Firehose stream.

You’ll need to save the credential file with the Access Key ID, Secret Access Key to enter this information in the Segment destination settings. We will encrypt the access keys that you give us for this user. We firmly recommend creating a dedicated user with only the putRecord policy (below) attached, so that the keys you give us have effectively write-only access to your streams.

3. Create an IAM policy

Sign in to the Identity and Access Management (IAM) console and follow these instructions to Create an IAM policy to allow Segment permission to write to your Firehose stream.

Select the Create Policy from JSON option and use the following template policy in the Policy Document field. Be sure to change the {region}, {account-id} and {stream-name} with the applicable values.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "firehose:PutRecord"
           ],
           "Resource": [
               "arn:aws:firehose:{region}:{account-id}:deliverystream/{stream-name}"
           ]
       }
   ]
}

4. Attach the IAM policy

In the Identity and Access Management (IAM) console, follow these instructions to Attach the IAM policy to the IAM user.

5. Create a new Kinesis Firehose Destination

In the Segment source that you want to connect to your Kinesis Firehose destination, click the “Add Destination” button. Search and select the Kinesis Firehose destination and enter the options: Secret Access Key, Access Key Id, Region, Mapped Streams.


Supported Sources and Connection Modes

WebMobileServer
📱 Device-based
☁️ Cloud-based

To learn more about about Connection Modes and what dictates which we support, see here.

Settings

Segment lets you change these destination settings via your Segment dashboard without having to touch any code.

IAM User Access Key ID

This is the AWS Access Key ID for the Segment Kinesis Firehose AWS IAM user

Map Segment Events to Firehose Delivery Streams

Please input the Segment event names or event types on the left and the desired Firehose delivery stream destinations on the right. This mapping is required for all events you would like in Firehose

AWS Kinesis Firehose Region

The Kinesis Firehose AWS region key

IAM User AWS Secret Access Key

AWS Secret Access Key for the Segment Kinesis Firehose AWS IAM user


If you have any questions or see anywhere we can improve our documentation, please let us know or kick off a conversation in the Segment Community!