Select spec version:

Amazon Kinesis Destination Config Reference

Beta

Complete Rudder CLI reference for the Amazon Kinesis destination config keys, source types, and secrets.

Available Plans

free
growth
enterprise

6 minute read

Amazon Kinesis is a streaming destination. RudderStack writes each event as a record to a Kinesis data stream you own.

In an Amazon Kinesis destination spec:

type: kinesis
definition_version: 1

Sample configuration

yaml

version: rudder/v1
kind: destination
metadata:
  name: kinesis-prod
spec:
  id: kinesis-prod
  display_name: Kinesis Production
  type: kinesis
  definition_version: 1
  enabled: true
  config:
    region: us-east-1
    stream: rudder-events-prod
    role_based_auth: true
    iam_role_arn: "arn:aws:iam::123456789012:role/RudderStackKinesisAccess"
    use_message_id: false

    connection_mode:
      web: cloud
      cloud: cloud
    consent_management:
      web:
        - provider: oneTrust
          consents:
            - analytics

The above example uses role-based authentication, so it carries no access keys. Which of iam_role_arn, access_key_id, and access_key you set depends on role_based_auth — see Authentication.

Config keys

config accepts only the keys listed below. The shared config key rules cover unknown keys, defaults, and immutability.

Stream

region

Required

Type: string

Description: AWS region the Kinesis stream was created in, for example us-east-1.

Notes:

At most 100 characters, and must not contain line breaks.
A {{ path || fallback }} template is accepted in place of a literal, and isn’t measured against the length limit.

stream

Required

Type: string

Description: Name of the Kinesis data stream RudderStack writes records to. The stream must already exist.

Notes:

At most 100 characters, and must not contain line breaks.
Templates are accepted on the same terms as region.

use_message_id

Type: boolean

Default value: false

Description: Use the event’s messageId as the record’s partition key. By default RudderStack partitions by userId, or anonymousId when userId is absent. messageId spreads records more evenly across the stream’s shards.

Authentication

role_based_auth selects the authentication method, and decides which of the remaining three keys are required.

Rudder CLI checks only that the keys the selected method needs are present. It doesn’t reject the keys belonging to the other method, so a spec carrying both an iam_role_arn and an access key pair passes validate and applies.
Leave the unused method’s keys out — otherwise you store credentials the destination never reads.

role_based_auth

Required

Type: boolean

Description: Whether to authenticate with an IAM role. Set it to true to use iam_role_arn, or false to use the access key pair.

Notes:

The dashboard defaults this field to true. Rudder CLI requires it explicitly.
A spec that omits this key fails validation with 'role_based_auth' is required.

iam_role_arn

Required

Type: string

Description: ARN of the IAM role RudderStack assumes to write to the stream.

Notes:

Required when role_based_auth is true. Leave it unset otherwise.
At most 100 characters, and must not contain line breaks.
Templates are accepted.

See AWS IAM Role for RudderStack for how to create the role.

access_key_id

RequiredSecret

Type: string

Description: AWS access key ID authorizing RudderStack to write to the stream.

Notes:

Required when role_based_auth is false. Leave it unset otherwise.
At most 100 characters, and must not contain line breaks.

Supply it as a {{ .VAR }} reference rather than a literal — see Secrets.

access_key

RequiredSecret

Type: string

Description: AWS secret access key matching access_key_id.

Notes:

Required when role_based_auth is false. Leave it unset otherwise.
At most 100 characters, and must not contain line breaks.

RudderStack recommends role-based authentication. The access key method is deprecated.

Either method needs an IAM policy granting RudderStack kinesis:PutRecord on the stream. Role-based authentication on its own doesn’t grant it. See Policy permissions for a sample policy.

Per-source keys

Both keys are objects keyed by the local source type — the tokens listed under Source types. A key naming a source type this destination doesn’t support fails validation.

connection_mode

Type: object

Description: Maps each source type you connect to the mode its events reach Kinesis in, using the modes in Source types.

Notes:

An entry is required for each source type you connect — see Connect a source.

yaml

connection_mode:
  web: cloud
  cloud: cloud

consent_management

Type: object

Description: Consent provider configuration per source type. The entry shape, accepted providers, and the rules on resolution_strategy and consents are shared across all destination types — see Consent management.

Source types

Amazon Kinesis accepts events from these source types in the mentioned connection modes:

Source type	Connection mode
`android`	`cloud`
`android_kotlin`	`cloud`
`ios`	`cloud`
`ios_swift`	`cloud`
`web`	`cloud`
`unity`	`cloud`
`react_native`	`cloud`
`flutter`	`cloud`
`cordova`	`cloud`
`cloud`	`cloud`
`warehouse`	`cloud`

Every source type is cloud only — events reach the stream from RudderStack’s servers, never in device mode.

warehouse is the token a Reverse ETL source resolves to. Rudder CLI manages the destination’s warehouse entry — if the destination is connected to a Reverse ETL source, set connection_mode.warehouse: cloud in its spec, or apply removes it.

Rudder CLI can also declare the Reverse ETL connection itself, for the types listed under Supported destinations — see Manage Reverse ETL Connections using Rudder CLI.

The dashboard additionally offers Amazon Kinesis to AMP and Shopify sources. Rudder CLI doesn’t manage those connections, so amp and shopify are invalid here.

Connect a source

An event stream connection to this destination is checked against two rules at validate time.

The source’s type must be supported. A source’s type resolves to one of the tokens above before the check — a JavaScript source resolves to web, and webhook and server-side SDK sources resolve to cloud. Every type an event stream source can resolve to is supported here, so this check always passes.

The destination config must carry a connection_mode entry for that source type. This lives on the destination spec, not on the connection spec. Without it:

destination 'kinesis-prod' config has no 'connection_mode' entry for source type 'web'

Amazon Kinesis needs no additional config keys to connect a source of any type.

Secrets

access_key_id and access_key are the secret keys, and apply only when role_based_auth is false. Write each as a {{ .VAR }} reference and supply the value at apply time:

yaml

config:
  region: us-east-1
  stream: rudder-events-prod
  role_based_auth: false
  access_key_id: "{{ .AWS_ACCESS_KEY_ID }}"
  access_key: "{{ .AWS_SECRET_ACCESS_KEY }}"

bash

export RUDDER_AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export RUDDER_AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
rudder-cli apply

# or
rudder-cli apply --var-file secrets.vars.yaml

Note that:

rudder-cli import writes only the secrets the API returns, as {{ .VAR }} references, and leaves the others out. Add any secret key it leaves out, then supply the variables before you apply — see Secrets.
iam_role_arn isn’t a secret — an ARN identifies a role but grants nothing on its own.

See How to Use Variable Substitution in Rudder CLI.

Changes by release

No changes since this type became available.

For changes across every destination type, see Destination Type Changes by Release.

See more

Amazon Kinesis Destination for the connection settings and the IAM policy RudderStack needs
Destination Type Reference for Rudder CLI for the rules shared across destination types
Destination YAML Reference for the spec envelope

Amazon Kinesis Destination Config Reference

Sample configuration

Config keys

Stream

Authentication

Per-source keys

Source types

Connect a source

Secrets

Changes by release

See more

Questions? Let's figure it out together.