Amazon Kinesis Destination

Segment makes it easy to send your data to Amazon Kinesis (and lots of other destinations). Once you've tracked your data through our open source libraries we'll translate and route your data to Amazon Kinesis in the format they understand. Learn more about how to use Amazon Kinesis with Segment.

Getting Started

When you toggle on AWS Kinesis in Segment, we’ll start sending your customer data to an AWS Kinesis stream. Our AWS Kinesis destination supports all of the Segment methods, and will send data from any one of our libraries.

Data Model

Let’s say you’re connecting your Segment customer data stream to Kinesis Stream arn:aws:kinesis:{region}:{account-id}:stream/{stream-name}. If you send Segment the following track call:

{
  "userId": "user_1",
  "event": "User Registered",
  "properties": {
    "plan": "Pro Annual",
    "account_type" : "Facebook"
  }
}

The Segment Kinesis destination will issue a PutRecord request with the following parameters:

kinesis.putRecord({
  Data: new Buffer(JSON.stringify(msg)).toString('base64')
  PartitionKey: msg.userId() || msg.anonymousId(),
  StreamName: 'stream-name'
});

Segment will use the the userId || anonymousId as the PartitionKey. The partition key is used by Amazon Kinesis to distribute data across shards. Amazon Kinesis segregates the data records that belong to a stream into multiple shards, using the partition key associated with each data record to determine which shard a given data record belongs to.

Note: The json payload is base64 stringified. Read more about our track spec.

Quickstart

In order to get started, you’ll need to perform the following steps:

  1. Create a Kinesis stream
  2. Create an IAM user to allow Segment permission to write to your Kinesis Stream
  3. Create an IAM policy to allow Segment to issue PutRecord on behalf of the user
  4. Attach the IAM policy to the IAM user
  5. Create a new Kinesis destination in the Segment UI.

1. Create a Kinesis Stream

A stream is composed of multiple shards, each of which provides a fixed unit of capacity. The total capacity of the stream is the sum of the capacities of its shards. Each shard corresponds to 1 MB/s of write capacity and 2 MB/s of read capacity. See the Amazon Kinesis Developer Guide for more information on estimating number of shards needed for your stream.

Follow these instructions to create a new AWS Kinesis Stream.

2. Create an IAM user

Follow these instructions to Create an IAM user to allow Segment permission to write to your Kinesis Stream.

You’ll need to save the credential file with the Access Key ID, Secret Access Key to enter this information in the Segment destination setting. We will encrypt the access keys that you give us for this user. We firmly recommend creating a dedicated user with only the putRecord policy (below) attached, so that the keys you give us have effectively write-only access to your streams.

3. Create an IAM policy

Sign in to the Identity and Access Management (IAM) console and follow these instructions to Create an IAM policy to allow Segment permission to write to your Kinesis Stream.

Select the Create Policy from JSON option and use the following template policy in the Policy Document field. Be sure to change the {region}, {account-id} and {stream-name} with the applicable values.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "kinesis:PutRecord"
           ],
           "Resource": [
               "arn:aws:kinesis:{region}:{account-id}:stream/{stream-name}"
           ]
       }
   ]
}

4. Attach the IAM policy

In the Identity and Access Management (IAM) console, follow these instructions to Attach the IAM policy to the IAM user.

5. Create a new Kinesis destination

In the Segment source that you want to connect to your Kinesis destination, click the “Add Destination” button. Search and select the Kinesis destination and enter the options: secretAccessKey, accessKeyId, region, stream.

Debugging

When you get started, we recommend using any of the open source Kinesis tailing utility to validate that data is flowing correctly!


Supported Sources and Connection Modes

WebMobileServer
📱 Device-based
☁️ Cloud-based

To learn more about about Connection Modes and what dictates which we support, see here.

Settings

Segment lets you change these destination settings via your Segment dashboard without having to touch any code.

IAM User Access Key ID

This is the AWS Access Key ID for the Segment Kinesis AWS IAM user

AWS Kinesis Stream Region

The Kinesis Stream’s AWS region key

IAM User AWS Secret Access Key

AWS Secret Access Key for the Segment Kinesis AWS IAM user

AWS Kinesis Stream Name

The Kinesis Stream Name


If you have any questions or see anywhere we can improve our documentation, please let us know or kick off a conversation in the Segment Community!