Apache Kafka Destination Config Reference

Apache Kafka is a streaming destination. RudderStack publishes each event as a message to a topic on your Kafka cluster — as JSON, or serialized with Avro — keyed by the event’s userId, or anonymousId when userId is absent, so one user’s events stay in the same partition.

In an Apache Kafka destination spec:

• type: kafka
• definition_version: 1

Sample configuration

version: rudder/v1
kind: destination
metadata:
  name: kafka-prod
spec:
  id: kafka-prod
  display_name: Kafka Production
  type: kafka
  definition_version: 1
  enabled: true
  config:
    host_name: "{{ .KAFKA_HOST_NAME }}"
    port: "9093"
    topic: rudder-events

    ssl_enabled: true
    use_sasl: true
    sasl_type: sha512
    username: "{{ .KAFKA_USERNAME }}"
    password: "{{ .KAFKA_PASSWORD }}"

    enable_multi_topic: true
    event_type_to_topic_map:
      - from: identify
        to: rudder-identifies
      - from: page
        to: rudder-pages
    event_to_topic_map:
      - from: Order Completed
        to: rudder-orders

    connection_mode:
      web: cloud
      cloud: cloud
    consent_management:
      web:
        - provider: oneTrust
          consents:
            - analytics

The above example connects over SSL with SCRAM-SHA-512 authentication — see SSL and SASL. It sends identify and page events and Order Completed to their own topics, and every other event to topic — see Topic routing. It omits the Avro and SSH tunnel keys.

Config keys

config accepts only the keys listed below. The shared config key rules cover unknown keys, defaults, and immutability.

Connection

port: "9093"

SSL and SASL

ssl_enabled and use_sasl decide which of the other keys here apply. SASL works only over SSL — RudderStack doesn’t support SASL on a plaintext connection.

ca_certificate: |
  -----BEGIN CERTIFICATE-----
  MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ
  ...
  -----END CERTIFICATE-----  

Topic routing

RudderStack picks each event’s topic in this order:

1. The topic the event names itself, in integrations.Kafka.topic — see Sending events to Kafka topics.
2. A matching entry in event_type_to_topic_map or event_to_topic_map, when enable_multi_topic is true.
3. topic.

event_type_to_topic_map:
  - from: identify
    to: rudder-identifies

event_to_topic_map:
  - from: Order Completed
    to: rudder-orders

Avro serialization

convert_to_avro: true
avro_schemas:
  - schema_id: "1"
    schema: '{"type": "record", "name": "RudderEvent", "fields": [...]}'

SSH tunnel

Both keys below are internal: the dashboard offers no SSH settings for Kafka. Rudder CLI accepts them so that a destination already set up to connect through an SSH tunnel keeps that setup on update — see Some keys aren’t in the dashboard.

use_ssh: true
ssh:
  host: bastion.example.com
  port: "22"
  user: "{{ .KAFKA_SSH_USER }}"
  public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl rudderstack@bastion"

Per-source keys

Both keys are objects keyed by the local source type — the tokens listed under Source types. A key naming a source type this destination doesn’t support fails validation.

connection_mode:
  web: cloud
  cloud: cloud

Source types

Apache Kafka accepts events from these source types in the mentioned connection modes:

Every source type is cloud only — events reach Kafka from RudderStack’s servers, never in device mode.

warehouse is the token a Reverse ETL source resolves to. Rudder CLI manages the destination’s warehouse entry — if the destination is connected to a Reverse ETL source, set connection_mode.warehouse: cloud in its spec, or apply removes it.

Rudder CLI can also declare the Reverse ETL connection itself, for the types listed under Supported destinations — see Manage Reverse ETL Connections using Rudder CLI.

The dashboard additionally offers Apache Kafka to AMP and Shopify sources. Rudder CLI doesn’t manage those connections, so amp and shopify are invalid here.

Connect a source

An event stream connection to this destination is checked against two rules at validate time.

The source’s type must be supported. A source’s type resolves to one of the tokens above before the check — a JavaScript source resolves to web, and webhook and server-side SDK sources resolve to cloud. Every type an event stream source can resolve to is supported here, so this check always passes.

The destination config must carry a connection_mode entry for that source type. This lives on the destination spec, not on the connection spec. Without it:

destination 'kafka-prod' config has no 'connection_mode' entry for source type 'web'

Apache Kafka needs no additional config keys to connect a source of any type.

Secrets

host_name, username, password, and ssh.user are the secret keys. username and password apply only when ssl_enabled and use_sasl are both true, and ssh.user only when use_ssh is true. Write each one you use as a {{ .VAR }} reference and supply the value at apply time:

config:
  host_name: "{{ .KAFKA_HOST_NAME }}"
  username: "{{ .KAFKA_USERNAME }}"
  password: "{{ .KAFKA_PASSWORD }}"

export RUDDER_KAFKA_HOST_NAME="..."
export RUDDER_KAFKA_USERNAME="..."
export RUDDER_KAFKA_PASSWORD="..."
rudder-cli apply

# or
rudder-cli apply --var-file secrets.vars.yaml

Note that:

• rudder-cli import writes only the secrets the API returns, as {{ .VAR }} references, and leaves the others out. Add any secret key it leaves out, then supply the variables before you apply — see Secrets.
• ca_certificate and ssh.public_key aren’t secrets — a CA certificate and a public key are public by design.

See How to Use Variable Substitution in Rudder CLI.

Changes by release

No changes since this type became available.

For changes across every destination type, see Destination Type Changes by Release.

See more

• Apache Kafka Destination for topic routing through the integrations object, Avro serialization, the partition key, and two-way SSL setup
• Confluent Cloud Destination Config Reference for a managed Kafka cluster on Confluent Cloud
• Destination Type Reference for Rudder CLI for the rules shared across destination types
• Destination YAML Reference for the spec envelope