Reverse ETL Connection YAML Reference

This guide is the YAML reference for Reverse ETL connection resources in a Rudder CLI project.

Reverse ETL connection support is experimental and requires Rudder CLI v0.27.0 or later.

See Enable Reverse ETL connection support for the flags to set before you validate or apply these specs.

Overview

Set kind to retl-connections and version to rudder/v1. spec.connections is a list. Each entry is one Reverse ETL source to destination link, together with its sync configuration. Store the files anywhere under the project root.

Each source must reference a supported Reverse ETL source, and each destination a supported destination. Unlike Event Stream connections, every entry needs a config block.

Each connection uses one of two mapping flows: object mapping if config.object is set (the dashboard’s Visual Data Mapper), JSON mapper otherwise (the JSON Data Mapper). See Mapping flows. The destination spec must set connection_mode.warehouse; see Destination requirements.

Rudder CLI rejects keys it doesn’t recognize at every level of the spec, so a misspelled key fails validation instead of being ignored.

Examples

version: rudder/v1
kind: retl-connections
metadata:
  name: retl-connections
spec:
  connections:
    - id: users-to-webhook
      source: "#retl-source-sql-model:users-model"
      destination: "#destination:my-http-destination"
      enabled: true
      config:
        sync_behaviour: upsert
        schedule:
          type: basic
          every_minutes: 30
        identifiers:
          - from: user_id
            to: user_id
        mappings:
          - from: email
            to: traits.email
        event:
          type: identify

version: rudder/v1
kind: retl-connections
metadata:
  name: retl-connections
spec:
  connections:
    - id: users-to-hubspot
      source: "#retl-source-sql-model:users-model"
      destination: "#destination:hubspot-prod"
      config:
        object: contacts
        sync_behaviour: upsert
        schedule:
          type: basic
          every_minutes: 60
        identifiers:
          - from: email
            to: email
        mappings:
          - from: first_name
            to: firstname

Connection fields

Each entry in spec.connections has these fields:

Changing source or destination replaces the connection (delete + create). The backend allows only one connection per source–destination pair. See Apply and replace behavior.

Sync configuration

The config object has these fields:

To change a field marked immutable, see Change an immutable field.

Every from value, cursor_column, and event.name_column is a warehouse column name. It must start with a letter or an underscore, followed by letters, digits, underscores, $, or ..

A constants entry can’t use context.mappedToDestination as its key. RudderStack sets that key itself.

Schedule

The schedule object sets the sync schedule:

A manual schedule takes no other fields. The connection syncs only when you trigger it.

Quote a cron expression. YAML treats a leading * or @ as syntax, so an unquoted */10 * * * * or @hourly fails with a YAML parse error:

schedule:
  type: cron
  cron_expression: "*/10 * * * *"

Event

The event object applies to the JSON mapper flow:

If you omit event, RudderStack sends identify events. Rudder CLI also accepts a track event with neither name nor name_column, but RudderStack has no default event name, so set one. event is immutable, so set it when you create the connection.

This track event reads its name from a column, and a constant adds a fixed value to every record:

config:
  event:
    type: track
    name_column: event_name
  constants:
    - key: properties.source
      value: warehouse

See Sync Data from Reverse ETL Sources for how RudderStack sends records as identify and track events.

Sync settings

The sync_settings object controls sync observability. Every field is optional:

config:
  sync_settings:
    sync_logs:
      enabled: true
      retention_days: 7
      snapshots_to_retain: 3
    failed_keys:
      retry: false

Omitted and removed fields behave as follows:

• Omitting sync_settings keeps all four defaults.
• A partial block takes the default for every field you leave out. For example, setting only retention_days: 7 keeps sync logs enabled with five snapshots, and keeps retries on.
• false and 0 are real settings, not omissions. Rudder CLI sends them as written.
• Removing a block that held non-default values resets the stored settings to the defaults on the next apply.
• A table source on s3 doesn’t support sync_settings.

Mapping flows

A connection maps data in one of two flows. No field names the flow. Rudder CLI derives it from whether object is set.

A mappings entry can’t target a field that the flow reserves for identifiers:

Supported sources

Reverse ETL sources created from Audiences aren’t supported.

Supported destinations

A Reverse ETL connection can target these destination types. All of them support the JSON mapper flow. In Rudder CLI v0.27.0, you can connect only HTTP Webhook. The others need v0.28.0 or later.

A destination type that doesn’t accept Reverse ETL sources fails validation. The error lists the source types the destination does accept:

destination 'my-googleads-destination' (type 'googleads') does not accept rETL sources: source type 'warehouse' is not among supported source types: web

Customer.io isn’t supported either. It syncs Reverse ETL data through its own flow, which connection specs can’t express.

Destination requirements

The destination’s config must carry a connection_mode entry for the warehouse source type, which is the source type every Reverse ETL source resolves to:

config:
  connection_mode:
    warehouse: cloud

Without it, validate reports:

destination 'my-http-destination' config has no 'connection_mode' entry for source type 'warehouse'

Some destination types need additional config keys before a Reverse ETL source can connect:

If the destination already has a Reverse ETL connection that you set up in the dashboard, check its spec after you upgrade to v0.28.0. See Where the mode is declared.

Cron expressions

RudderStack stores a cron_expression as written, without checking it. Rudder CLI checks it locally so that a schedule that can never run, or that runs too often, fails at validate instead of after apply.

Rudder CLI evaluates every expression in UTC. Consecutive syncs must be at least five minutes apart.

An error names the problem:

'cron_expression' is not valid: consecutive syncs have a 2-minute gap; the minimum supported interval is 5 minutes

A timezone prefix is a warning because Rudder CLI can’t check the schedule it produces. The apply goes through, so verify the resulting schedule yourself.

Validation rules

rudder-cli validate runs these checks locally. Error blocks validate and apply. Warning does not.

Workspace plan gates and permissions aren’t checked locally; apply surfaces the server error. Rudder CLI also doesn’t check that object or a mapped field exists in the destination.

Topology rules see only connections in the project. Remote-only connections (created in the dashboard and never imported) are invisible at validate time.

Import existing connections

rudder-cli import workspace -l <project-directory>

Enable Reverse ETL connection support before you run the import. Without it, import workspace doesn’t import Reverse ETL connections.

Import writes all importable Reverse ETL connections into one file, imported/retl/connections.yaml. Each connection’s id is generated from the names of its source and destination, for example users-model-to-users-webhook. Related SQL models land under imported/retl/sql-models/. See How to Import Workspace Resources.

Import skips a connection that a spec can’t express and leaves it unchanged in the workspace. Rudder CLI logs each skipped connection, with the reason, to ~/.rudder/cli.log. The most common reasons are:

• Its source isn’t a supported source, or its destination isn’t a supported destination.
• Its source is a table source, and table source support isn’t enabled.
• Its source or destination is neither CLI-managed nor part of the same import.
• It has no identifiers.
• It carries settings that have no spec equivalent, such as destination-specific configuration.

If the destination is already in your project, its spec needs the connection_mode entry for warehouse before the imported connection passes validate. See Destination requirements.

The first apply after an import can fail with setting external ID for rETL connection during import: …. Run apply again, and the retry adopts the connection.

You can’t import a single Reverse ETL connection by ID. import workspace is the only import path.

See more

• Manage Reverse ETL Connections using Rudder CLI for feature overview, and the Reverse ETL connections walkthrough for step-by-step setup
• Destination YAML Reference and SQL Model Resources YAML Reference
• Connection YAML Reference for Event Stream connections