# Apache Kafka Destination Config Reference

Apache Kafka is a streaming destination. RudderStack publishes each event as a message to a topic on your Kafka cluster — as JSON, or serialized with Avro — keyed by the event’s userId, or anonymousId when userId is absent, so one user’s events stay in the same partition.

In an Apache Kafka destination spec:

Sample configuration

yaml 
version: rudder/v1
kind: destination
metadata:
  name: kafka-prod
spec:
  id: kafka-prod
  display_name: Kafka Production
  type: kafka
  definition_version: 1
  enabled: true
  config:
    host_name: "{{ .KAFKA_HOST_NAME }}"
    port: "9093"
    topic: rudder-events

    ssl_enabled: true
    use_sasl: true
    sasl_type: sha512
    username: "{{ .KAFKA_USERNAME }}"
    password: "{{ .KAFKA_PASSWORD }}"

    enable_multi_topic: true
    event_type_to_topic_map:
      - from: identify
        to: rudder-identifies
      - from: page
        to: rudder-pages
    event_to_topic_map:
      - from: Order Completed
        to: rudder-orders

    connection_mode:
      web: cloud
      cloud: cloud
    consent_management:
      web:
        - provider: oneTrust
          consents:
            - analytics

The above example connects over SSL with SCRAM-SHA-512 authentication — see SSL and SASL. It sends identify and page events and Order Completed to their own topics, and every other event to topic — see Topic routing. It omits the Avro and SSH tunnel keys.

Config keys

config accepts only the keys listed below. The shared config key rules cover unknown keys, defaults, and immutability.

Connection

host_name

RequiredSecret

Type: string

Description: Host name of your Kafka broker. To list several brokers, separate them with commas.

Notes:

  • Each name consists of dot-separated labels of letters, digits, and hyphens, with no port and no scheme — port supplies the port.
  • Validation accepts a space after a comma, but RudderStack appends :<port> to each name exactly as written, so leave the spaces out.
  • Templates aren’t accepted.
Supply it as a {{ .VAR }} reference rather than a literal — see Secrets.

port

Required

Type: string

Description: Port the brokers listen on. Every name in host_name uses it.

Notes:

  • A whole number from 1 to 65535, written as a string. An unquoted number fails validation.
  • Templates aren’t accepted.
yaml 
port: "9093"

topic

Required

Type: string

Description: Topic RudderStack publishes events to, unless an event is routed elsewhere — see Topic routing.

Notes:

  • 1 to 249 characters: letters, digits, ., _, and -.
  • Templates aren’t accepted.

SSL and SASL

ssl_enabled and use_sasl decide which of the other keys here apply. SASL works only over SSL — RudderStack doesn’t support SASL on a plaintext connection.

ssl_enabled

Type: boolean

Description: Connect to the brokers over SSL.

Notes:

  • The dashboard defaults this field to true. Rudder CLI doesn’t fill it in, and RudderStack treats an omitted value as false, so write ssl_enabled: true to connect over SSL.

ca_certificate

Type: string

Description: PEM-encoded CA certificate RudderStack verifies the brokers’ certificates against. When it’s omitted, RudderStack trusts the system’s CAs, which is enough for brokers whose certificates chain to a public CA.

Notes:

  • Applies when ssl_enabled is true. Leave it unset otherwise.
  • Not validated locally.

Write it as a YAML block scalar to keep its line breaks:

yaml 
ca_certificate: |
  -----BEGIN CERTIFICATE-----
  MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ
  ...
  -----END CERTIFICATE-----

See Does my Kafka server require client authentication? for two-way SSL.

use_sasl

Type: boolean

Description: Authenticate to the brokers with SASL, using sasl_type, username, and password. The dashboard calls this Enable SASL with SSL.

Notes:

  • Takes effect only when ssl_enabled is true. With SSL off, RudderStack ignores it and connects without authenticating, and Rudder CLI doesn’t require username.
See Does RudderStack support SASL_PLAINTEXT authentication?.

sasl_type

Type: string

Default value: plain

Description: SASL mechanism RudderStack authenticates with.

Notes:

  • One of plain (PLAIN), sha256 (SCRAM-SHA-256), or sha512 (SCRAM-SHA-512).
  • Applies when ssl_enabled and use_sasl are both true. Leave it unset otherwise.
  • Templates aren’t accepted.

username

RequiredSecret

Type: string

Description: SASL username.

Notes:

  • Required when ssl_enabled and use_sasl are both true. Leave it unset otherwise.
  • 1 to 32 characters: letters, digits, _, and -.
  • Templates aren’t accepted.
Supply it as a {{ .VAR }} reference rather than a literal — see Secrets.

password

Secret

Type: string

Description: SASL password for username.

Notes:

  • Applies when ssl_enabled and use_sasl are both true. Rudder CLI doesn’t require it.
  • At most 100 characters, and must not contain line breaks.
Supply it as a {{ .VAR }} reference rather than a literal — see Secrets.

Topic routing

RudderStack picks each event’s topic in this order:

  1. The topic the event names itself, in integrations.Kafka.topic — see Sending events to Kafka topics.
  2. A matching entry in event_type_to_topic_map or event_to_topic_map, when enable_multi_topic is true.
  3. topic.

enable_multi_topic

Type: boolean

Default value: false

Description: Route events to topics by event type and track event name, using event_type_to_topic_map and event_to_topic_map. When false, RudderStack ignores both mappings.

event_type_to_topic_map

Type: array of objects

Description: Maps event types to topics. track events are routed by name instead, through event_to_topic_map.

Notes:

  • Applies when enable_multi_topic is true. Leave it unset otherwise.
  • from — one of identify, page, screen, group, or alias. Templates aren’t accepted.
  • to — topic name. At most 100 characters, and must not contain line breaks. Unlike topic, its characters aren’t checked. A {{ path || fallback }} template is accepted in place of a literal.
yaml 
event_type_to_topic_map:
  - from: identify
    to: rudder-identifies

event_to_topic_map

Type: array of objects

Description: Maps track event names to topics.

Notes:

  • Applies when enable_multi_topic is true. Leave it unset otherwise.
  • from — track event name, matched exactly, including case. At most 100 characters, and must not contain line breaks.
  • to — topic name, on the same terms as in event_type_to_topic_map.
  • Both fields accept a {{ path || fallback }} template in place of a literal.
yaml 
event_to_topic_map:
  - from: Order Completed
    to: rudder-orders

Avro serialization

convert_to_avro

Type: boolean

Description: Serialize each event with Avro instead of publishing it as JSON. Each event selects its schema by ID, in integrations.Kafka.schemaId.

Notes:

  • With this on, an event fails if it names no schema, names one that isn’t in avro_schemas, or doesn’t fit its schema. The schema has to cover every field the event carries.
See Serializing data using Avro for a sample event and its schema.

avro_schemas

Required

Type: array of objects

Description: Avro schemas events can be serialized with.

Notes:

  • Required when convert_to_avro is true. Leave it unset otherwise.
  • schema_id — ID an event names to select the schema.
  • schema — the Avro schema, as a JSON string.
  • Both fields are required in every entry, even when convert_to_avro is false, and neither is validated further locally.
  • A schema that doesn’t parse stops delivery to this destination entirely, not just for the events that use it.
yaml 
convert_to_avro: true
avro_schemas:
  - schema_id: "1"
    schema: '{"type": "record", "name": "RudderEvent", "fields": [...]}'

embed_avro_schema_id

Type: boolean

Description: Prefix each Avro message with its schema ID, in the format Confluent Schema Registry consumers expect: a zero byte, then the ID as a 4-byte integer.

Notes:

  • Applies when convert_to_avro is true. Leave it unset otherwise.
  • Every schema_id must then be a whole number — an event using a non-numeric ID fails.

SSH tunnel

Both keys below are internal: the dashboard offers no SSH settings for Kafka. Rudder CLI accepts them so that a destination already set up to connect through an SSH tunnel keeps that setup on update — see Some keys aren’t in the dashboard.

use_ssh

Internal

Type: boolean

Description: Connect to the brokers through an SSH tunnel via a bastion host.

ssh

RequiredInternal

Type: object

Description: Bastion host connection details. RudderStack logs in with a private key it holds for this destination. public_key is the matching public key, which belongs in the bastion host’s authorized_keys.

Notes:

  • Required when use_ssh is true, with all four fields. Leave it unset otherwise.
  • A spec that sets use_ssh: true and omits any of the four fields fails validation.
  • host — host name or IP address of the bastion host. Validation accepts a host:port form, but RudderStack appends port to this value, so leave the port out.
  • port — SSH port of the bastion host, from 1 to 65535, written as a string.
  • user — user RudderStack logs in as. 1 to 32 characters: letters, digits, _, and -. Secret — see Secrets.
  • public_key — an ssh-rsa, ssh-ed25519, or ssh-dss public key, optionally followed by a user@host comment.
  • None of the four fields accepts a template.
yaml 
use_ssh: true
ssh:
  host: bastion.example.com
  port: "22"
  user: "{{ .KAFKA_SSH_USER }}"
  public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl rudderstack@bastion"

Per-source keys

Both keys are objects keyed by the local source type — the tokens listed under Source types. A key naming a source type this destination doesn’t support fails validation.

connection_mode

Type: object

Description: Maps each source type you connect to the mode its events reach Kafka in, using the modes in Source types.

Notes:

  • An entry is required for each source type you connect — see Connect a source.
yaml 
connection_mode:
  web: cloud
  cloud: cloud

Source types

Apache Kafka accepts events from these source types in the mentioned connection modes:

Source type Connection mode
android cloud
android_kotlin cloud
ios cloud
ios_swift cloud
web cloud
unity cloud
cloud cloud
react_native cloud
flutter cloud
cordova cloud
warehouse cloud

Every source type is cloud only — events reach Kafka from RudderStack’s servers, never in device mode.

warehouse is the token a Reverse ETL source resolves to. Rudder CLI manages the destination’s warehouse entry — if the destination is connected to a Reverse ETL source, set connection_mode.warehouse: cloud in its spec, or apply removes it.

Rudder CLI can also declare the Reverse ETL connection itself, for the types listed under Supported destinations — see Manage Reverse ETL Connections using Rudder CLI.

The dashboard additionally offers Apache Kafka to AMP and Shopify sources. Rudder CLI doesn’t manage those connections, so amp and shopify are invalid here.

Connect a source

An event stream connection to this destination is checked against two rules at validate time.

The source’s type must be supported. A source’s type resolves to one of the tokens above before the check — a JavaScript source resolves to web, and webhook and server-side SDK sources resolve to cloud. Every type an event stream source can resolve to is supported here, so this check always passes.

The destination config must carry a connection_mode entry for that source type. This lives on the destination spec, not on the connection spec. Without it:

destination 'kafka-prod' config has no 'connection_mode' entry for source type 'web'

Apache Kafka needs no additional config keys to connect a source of any type.

Secrets

host_name, username, password, and ssh.user are the secret keys. username and password apply only when ssl_enabled and use_sasl are both true, and ssh.user only when use_ssh is true. Write each one you use as a {{ .VAR }} reference and supply the value at apply time:

yaml 
config:
  host_name: "{{ .KAFKA_HOST_NAME }}"
  username: "{{ .KAFKA_USERNAME }}"
  password: "{{ .KAFKA_PASSWORD }}"
bash 
export RUDDER_KAFKA_HOST_NAME="..."
export RUDDER_KAFKA_USERNAME="..."
export RUDDER_KAFKA_PASSWORD="..."
rudder-cli apply

# or
rudder-cli apply --var-file secrets.vars.yaml

Note that:

See How to Use Variable Substitution in Rudder CLI.

Changes by release

No changes since this type became available.

For changes across every destination type, see Destination Type Changes by Release.

See more