Amazon Kinesis

Amazon Kinesis enables you to ingest, buffer and process streaming data in real-time. It can handle any amount of streaming data and process data from hundreds of thousands of sources with very low latencies along with the flexibility to choose the tools that best suit the requirements of your application.

RudderStack supports Amazon Kinesis as a destination where you can seamlessly send your event data.

Find the open source code for this destination in the GitHub repository.

Getting started

RudderStack supports sending event data to Kinesis via the following connection modes:

Once you have confirmed that the source platform supports sending events to Kinesis, follow these steps:

1. From your RudderStack dashboard, add the source. Then, from the list of destinations, select AWS Kinesis.
2. Assign a name to your destination and click Continue.

Connection settings

To successfully set up Kinesis as a destination, you need to configure the following settings:

• Region: Enter the AWS Region in which you have created the Kinesis stream.
• Stream name: Specify the name of your Kinesis stream.
• Role-based Authentication: Enable this setting to use the RudderStack IAM role for authentication. For more information on creating an AWS IAM role for RudderStack, refer to this guide.
  • IAM Role ARN: Enter the ARN of the IAM role.

It is highly recommended to enable this setting as the access keys-based authentication method is now deprecated.

• If Role-based Authentication is disabled, you need to enter the AWS Access Key ID and AWS Secret Access Key to authorize RudderStack to write to your Kinesis stream.

In both the role-based and access key-based authentication methods, you need to set a policy specifying the required permissions for RudderStack to write to your Kinesis stream. Refer to the Policy permissions section below for more information.

• Use MessageId as Partition Key: By default, RudderStack uses userId ( or anonymousId, if userId is not present in the payload) as the partition key. You can enable this setting to set your event’s messageId as the partition key for your Kinesis stream. This enables your data to be more evenly distributed across the shards in the stream.

Policy permissions

To authorize RudderStack to write to your stream, you must create an IAM policy that provides the necessary permissions to write to your data stream. Refer to Controlling Access to Amazon Kinesis Data Streams Resources Using IAM for the applicable policies.

A sample permissions policy is shown below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["kinesis:PutRecord"],
      "Resource": ["arn:aws:kinesis:{region}:{account-id}:stream/{stream-name}"]
    }
  ]
}

FAQ

How do I verify if the events are published to my Kinesis stream?

To verify that RudderStack has successfully sent the events to your configured Kinesis stream, you can check the Live Events tab of your source.

To verify if the events are successfully delivered or check for any delivery failures, go to the Live Events tab of your configured destination.

Questions? Contact us by email or on Slack