S3 Data Lake Destination Config Reference Beta

S3 Data Lake is a data lake destination. RudderStack writes events as Parquet files into an S3 bucket on a schedule, and can register their schema in the AWS Glue Data Catalog so you can query them with Athena.

In an S3 Data Lake destination spec:

• type: s3_datalake
• definition_version: 1

Sample configuration

version: rudder/v1
kind: destination
metadata:
  name: s3-datalake-prod
spec:
  id: s3-datalake-prod
  display_name: S3 Data Lake Production
  type: s3_datalake
  definition_version: 1
  enabled: true
  config:
    bucket_name: acme-datalake-prod
    prefix: rudder
    namespace: web_events
    enable_sse: false
    cleanup_object_storage_files: false

    use_glue: true
    region: us-east-1
    time_window_layout: dt=2006-01-02

    role_based_auth: true
    iam_role_arn: "arn:aws:iam::123456789012:role/RudderStackDatalake"

    sync_frequency: "180"
    sync_start_at: "01:00"

    skip_users_table: true
    skip_tracks_table: false

    underscore_divide_numbers: false
    allow_users_context_traits: false

    connection_mode:
      web: cloud
      cloud: cloud
    consent_management:
      web:
        - provider: oneTrust
          consents:
            - analytics

The above example registers the schema on AWS Glue with date-only partitions, and uses role-based authentication, so it carries no access keys — see AWS Glue and Authentication.

Config keys

config accepts only the keys listed below. The shared config key rules cover unknown keys, defaults, and immutability.

Storage

RudderStack writes each table to s3://<bucket_name>/<prefix>/rudder-datalake/<namespace>/<table>/. See Find data in S3 data lake.

AWS Glue

Authentication

role_based_auth selects the authentication method, and decides which of the remaining three keys are required.

Rudder CLI checks only that the keys the selected method needs are present. It doesn’t reject the keys belonging to the other method, so a spec carrying both an iam_role_arn and an access key pair passes validate and applies.

Leave the unused method’s keys out — otherwise you store credentials the destination never reads.

Sync scheduling

Table behavior

Legacy column naming

Both keys below preserve the column naming of destinations created before the behavior changed. Leave them at their defaults on a new destination. Neither can be changed once the destination exists — the API rejects the update.

Per-source keys

Both keys are objects keyed by the local source type — the tokens listed under Source types. A key naming a source type this destination doesn’t support fails validation.

connection_mode:
  web: cloud
  cloud: cloud

Source types

S3 Data Lake accepts events from these source types in the mentioned connection modes:

Every source type is cloud only — events reach the bucket from RudderStack’s servers, never in device mode.

The dashboard additionally offers S3 Data Lake to AMP, Shopify, and cloud app sources. Rudder CLI doesn’t manage those connections, so amp, shopify, and cloud_source are invalid here.

Connect a source

An event stream connection to this destination is checked against two rules at validate time.

The source’s type must be supported. A source’s type resolves to one of the tokens above before the check — a JavaScript source resolves to web, and webhook and server-side SDK sources resolve to cloud. An unsupported type reports:

destination 's3-datalake-prod' (type 's3_datalake') does not support source 'my-source':
source type 'amp' is not among supported source types: android, android_kotlin, ...

The destination config must carry a connection_mode entry for that source type. This lives on the destination spec, not on the connection spec. Without it:

destination 's3-datalake-prod' config has no 'connection_mode' entry for source type 'web'

S3 Data Lake needs no additional config keys to connect a source of any type.

Secrets

access_key_id and access_key are the secret keys, and apply only when role_based_auth is false. Write each as a {{ .VAR }} reference and supply the value at apply time:

config:
  bucket_name: acme-datalake-prod
  role_based_auth: false
  access_key_id: "{{ .AWS_ACCESS_KEY_ID }}"
  access_key: "{{ .AWS_SECRET_ACCESS_KEY }}"

export RUDDER_AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export RUDDER_AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
rudder-cli apply

# or
rudder-cli apply --var-file secrets.vars.yaml

Note that:

• A {{ .VAR }} reference satisfies the requirement check, so a spec using access keys validates before the values are supplied.
• The YAML that rudder-cli import writes may or may not include secret keys. Before you apply, make sure every secret key your configuration needs is present and populated through variable substitution.
• iam_role_arn isn’t a secret — an ARN identifies a role but grants nothing on its own.

See How to Use Variable Substitution in Rudder CLI.

See more

• S3 Data Lake Destination for bucket permissions, Glue crawlers, and querying with Athena
• Destination Type Reference for Rudder CLI for the rules shared across destination types
• Destination YAML Reference for the spec envelope