BigQuery Stream Destination Config Reference Beta

BigQuery Stream streams events into an existing BigQuery table through Google’s streaming insert API, so rows are queryable within seconds rather than after a scheduled load.

In a BigQuery Stream destination spec:

• type: bqstream
• definition_version: 1

Sample configuration

version: rudder/v1
kind: destination
metadata:
  name: bigquery-stream-prod
spec:
  id: bigquery-stream-prod
  display_name: BigQuery Stream Production
  type: bqstream
  definition_version: 1
  enabled: true
  config:
    project_id: acme-analytics
    dataset_id: product_events
    table_id: product_inserts
    insert_id: productId
    credentials: "{{ .BQSTREAM_CREDENTIALS }}"

    connection_mode:
      web: cloud
      cloud: cloud
    consent_management:
      web:
        - provider: oneTrust
          consents:
            - analytics

The above example deduplicates on the productId event property. Leave insert_id out to stream without deduplication — see Deduplication.

Config keys

config accepts only the keys listed below. The shared config key rules cover unknown keys, defaults, and immutability.

BigQuery Stream accepts only track events, and RudderStack forwards each event’s properties without fetching or checking the table schema. Make sure the track payload matches the columns of the table named in table_id — see Send events to BigQuery Stream.

Table

Deduplication

insert_id: '{{ message.uniqueId || "productId" }}'

Authentication

Per-source keys

Both keys are objects keyed by the local source type — the tokens listed under Source types. A key naming a source type this destination doesn’t support fails validation.

connection_mode:
  web: cloud
  cloud: cloud

Source types

BigQuery Stream accepts events from these source types in the mentioned connection modes:

Every source type is cloud only — events reach the table from RudderStack’s servers, never in device mode.

The dashboard additionally offers BigQuery Stream to AMP, Shopify, and Reverse ETL sources. Rudder CLI doesn’t manage those connections, so amp, shopify, and warehouse are invalid here.

Connect a source

An event stream connection to this destination is checked against two rules at validate time.

The source’s type must be supported. A source’s type resolves to one of the tokens above before the check — a JavaScript source resolves to web, and webhook and server-side SDK sources resolve to cloud. An unsupported type reports:

destination 'bigquery-stream-prod' (type 'bqstream') does not support source 'my-source':
source type 'amp' is not among supported source types: android, android_kotlin, ...

The destination config must carry a connection_mode entry for that source type. This lives on the destination spec, not on the connection spec. Without it:

destination 'bigquery-stream-prod' config has no 'connection_mode' entry for source type 'web'

BigQuery Stream needs no additional config keys to connect a source of any type.

Secrets

credentials is the only secret key. Write it as a {{ .VAR }} reference and supply the value at apply time:

config:
  credentials: "{{ .BQSTREAM_CREDENTIALS }}"

export RUDDER_BQSTREAM_CREDENTIALS="$(cat service-account.json)"
rudder-cli apply

# or
rudder-cli apply --var-file secrets.vars.yaml

Note that:

• The YAML that rudder-cli import writes may or may not include secret keys. Before you apply, make sure every secret key your configuration needs is present and populated through variable substitution.

See How to Use Variable Substitution in Rudder CLI.

See more

• BigQuery Stream Destination for table schema requirements and deduplication cases
• Destination Type Reference for Rudder CLI for the rules shared across destination types
• Destination YAML Reference for the spec envelope