Amazon Kinesis Firehose Destination

Segment makes it easy to send your data to Amazon Kinesis Firehose (and lots of other destinations). Once you've tracked your data through our open source libraries we'll translate and route your data to Amazon Kinesis Firehose in the format they understand. Learn more about how to use Amazon Kinesis Firehose with Segment.

Getting Started

When you toggle on AWS Kinesis Firehose in Segment, we’ll start sending your specified Segment events to Firehose delivery streams of your choice. Our Kinesis Firehose destination supports all of the Segment methods, and will send data from any one of our libraries.

Event Mapping

To begin using the Kinesis Firehose destination you must first decide on which Segment events you would like to route to which Firehose delivery streams. This mapping then needs to be defined in your destination settings.

Segment track events can map based on their event name. For example, if you have an event called User Registered, and you wanted these events to be published to a Firehose delivery stream called new_users, you would create a row in your destination settings that looks like this:

track event mapping screenshot

Any Segment event type (ie. page, track, identify, screen, etc.) can also be mapped. This allows you to publish all instances of a given Segment event type to a given stream. To do this, you simply to create a row with the event type and it’s corrensponding delivery stream:

page event mapping screenshot

Events can be defined insensitive to case so Page will be equivalent to page. The delivery stream name however needs to be formatted exactly as it is on AWS.

If you would like to route all events to a stream, use an * as the event name.

Data Model

Let’s say you’ve decided to publish your Segment track events named User Registered to your Kinesis Firehose delivery stream named online_registrations. If you send Segment the following track call:

{
  "userId": "user_1",
  "event": "User Registered",
  "properties": {
    "plan": "Pro Annual",
    "account_type" : "Facebook"
  }
}

The Segment Kinesis destination will issue a PutRecord request with the following parameters:

firehose.putRecord({
  Record: {
    Data: JSON.stringify(msg)) + '/n'
  },
  DeliveryStreamName: 'online_registrations'
});

Segment will append a newline character to each record to allow for easy downstream parsing.

Quickstart

In order to get started, you’ll need to perform the following steps:

  1. Create at least one Kinesis Firehose delivery stream.
  2. Create an IAM policy to allow Segment to issue PutRecord on behalf of the user.
  3. Create an IAM role to allow Segment permission to write to your Kinesis Firehose stream.
  4. Create a new Kinesis destination in the Segment UI.
  5. Specify at least one event that you would like to publish to a given stream.

1. Create a Kinesis Firehose delivery stream

Follow these instructions to create a new AWS Kinesis Firehose delivery stream.

2. Create an IAM policy

Sign in to the Identity and Access Management (IAM) console and follow these instructions to Create an IAM policy to allow Segment permission to write to your Kinesis Firehose Stream.

Select the Create Policy from JSON option and use the following template policy in the Policy Document field. Be sure to change the {region}, {account-id} and {stream-name} with the applicable values.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "firehose:PutRecord"
           ],
           "Resource": [
               "arn:aws:firehose:{region}:{account-id}:deliverystream/{stream-name}"
           ]
       }
   ]
}

3. Create an IAM role

Follow these instructions to Create an IAM role to allow Segment permission to write to your Kinesis Firehose Stream. When prompted to enter an Account ID, enter 595280932656. Make sure to enable ‘Require External ID’ and enter your Segment Source ID as the External ID*. This can be found by navigating to Settings > API Keys from your Segment source homepage. When adding permissions to your new role, find the policy you created above and attach it.

*If you have multiple sources using Kinesis, enter one of their source IDs here for now and then follow the procedure outlined in the Multiple Sources section at the bottom of this doc once you’ve completed this step and saved your IAM role.

4. Create a new Kinesis Firehose Destination

In the Segment source that you want to connect to your Kinesis Firehose destination, click the “Add Destination” button. Search and select the Kinesis Firehose destination and enter the options: Role Address, Region, and Mapped Streams.

Multiple Sources

If you have multiple sources using Kinesis/Firehose, you have two options:

Attach multiple sources to your IAM role

Find the IAM role you created for this destination in the AWS Console in Services > IAM > Roles. Click on the role, and navigate to the ‘Trust Relationships’ tab. Click ‘Edit trust relationship’. You should see a snippet that looks something that looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::595280932656:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "YOUR_SEGMENT_SOURCE_ID"
        }
      }
    }
  ]
}

Replace that snippet with the following, and replace the contents of the array with all of your source IDs.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::595280932656:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": ["YOUR_SEGMENT_SOURCE_ID", "ANOTHER_SOURCE_ID", "A_THIRD_SOURCE_ID"]
        }
      }
    }
  ]
}

Use a single secret ID

If you have so many sources using Kinesis that it is impractical to attach all of their IDs to your IAM role, you can instead opt to set a single ID to use instead. This approach should be avoided in favor of the above approach if possible since it will result in you having to keep track of a secret value. To set this value, go to the Kinesis Firehose destination settings from each of your Segment sources and set the ‘Secret ID’ to a value of your choosing. This value is a secret and should be treated as sensitively as a password. Once all of your sources have been updated to use this value, find the IAM role you created for this destination in the AWS Console in Services > IAM > Roles. Click on the role, and navigate to the ‘Trust Relationships’ tab. Click ‘Edit trust relationship’. You should see a snippet that looks something that looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::595280932656:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "YOUR_SEGMENT_SOURCE_ID"
        }
      }
    }
  ]
}

Replace your source ID (found at “YOUR_SEGMENT_SOURCE_ID”) with your secret ID.


Supported Sources and Connection Modes

WebMobileServer
📱 Device-based
☁️ Cloud-based

To learn more about about Connection Modes and what dictates which we support, see here.

Settings

Segment lets you change these destination settings via your Segment dashboard without having to touch any code.

[DEPRECATED] IAM User Access Key ID

This is the AWS Access Key ID for the Segment Kinesis Firehose AWS IAM user. NOTE: This has been deprecated in favor of the ‘Role Address’ setting, see the documentation for more details.

Map Segment Events to Firehose Delivery Streams

Please input the Segment event names or event types on the left and the desired Firehose delivery stream destinations on the right. This mapping is required for all events you would like in Firehose

AWS Kinesis Firehose Region

The Kinesis Firehose AWS region key

Role Address

The address of the AWS role that will be writing to Kinesis Firehose (ex: arn:aws:iam::874699288871:role/example-role)

[DEPRECATED] IAM User AWS Secret Access Key

AWS Secret Access Key for the Segment Kinesis Firehose AWS IAM user. NOTE: This has been deprecated in favor of the ‘Role Address’ setting, see the documentation for more details.

Secret ID

If you have so many sources that it’s impractical to attach all of their source IDs as external IDs to your IAM role, you can specify a single external ID here instead and attach that as an external ID to your IAM role. This value is a secret and should be treated as a password.


If you have any questions or see anywhere we can improve our documentation, please let us know or kick off a conversation in the Segment Community!