Select spec version:

BigQuery Stream Destination Config Reference

Complete Rudder CLI reference for the BigQuery Stream destination config keys, deduplication, source types, and secrets.

Available Plans

free
growth
enterprise

6 minute read

BigQuery Stream streams events into an existing BigQuery table through Google’s streaming insert API, so rows are queryable within seconds rather than after a scheduled load.

In a BigQuery Stream destination spec:

type: bqstream
definition_version: 1

Sample configuration

yaml

version: rudder/v1
kind: destination
metadata:
  name: bigquery-stream-prod
spec:
  id: bigquery-stream-prod
  display_name: BigQuery Stream Production
  type: bqstream
  definition_version: 1
  enabled: true
  config:
    project_id: acme-analytics
    dataset_id: product_events
    table_id: product_inserts
    insert_id: productId
    credentials: '{{ .BQSTREAM_CREDENTIALS }}'

    connection_mode:
      web: cloud
      cloud: cloud
    consent_management:
      web:
        - provider: oneTrust
          consents:
            - analytics

The above example deduplicates on the productId event property. Leave insert_id out to stream without deduplication — see Deduplication.

Config keys

config accepts only the keys listed below. The shared config key rules cover unknown keys, defaults, and immutability.

BigQuery Stream accepts only track events, and RudderStack forwards each event’s properties without fetching or checking the table schema. Make sure the track payload matches the columns of the table named in table_id — see Send events to BigQuery Stream.

Table

project_id

Required

Type: string

Description: GCP project ID that holds the dataset.

dataset_id

Required

Type: string

Description: ID of the dataset, within project_id, that holds the table.

table_id

Required

Type: string

Description: ID of the table RudderStack streams events into. The table must already exist.

Deduplication

insert_id

Type: string

Description: Name of the event property whose value BigQuery uses as the insertId to deduplicate rows — for example productId. It names the property, not the value.

Notes:

The property’s value must be a number or a string.
Deduplication applies only when the value is present in the event’s properties.

To pick the property per event, use a template with a fallback:

yaml

insert_id: '{{ message.uniqueId || "productId" }}'

See Configure insertId dynamically for how each case resolves.

Authentication

credentials

RequiredSecret

Type: string

Description: Contents of the JSON key file for a GCP service account that can insert rows into the table.

Supply it as a {{ .VAR }} reference rather than a literal — see Secrets.

See Create a service account for the steps.

Per-source keys

Both keys are objects keyed by the local source type — the tokens listed under Source types. A key naming a source type this destination doesn’t support fails validation.

connection_mode

Type: object

Description: Maps each source type you connect to the mode its events reach BigQuery in, using the modes in Source types.

Notes:

An entry is required for each source type you connect — see Connect a source.

yaml

connection_mode:
  web: cloud
  cloud: cloud

consent_management

Type: object

Description: Consent provider configuration per source type. The entry shape, accepted providers, and the rules on resolution_strategy and consents are shared across all destination types — see Consent management.

Source types

BigQuery Stream accepts events from these source types in the mentioned connection modes:

Source type	Connection mode
`android`	`cloud`
`android_kotlin`	`cloud`
`ios`	`cloud`
`ios_swift`	`cloud`
`web`	`cloud`
`unity`	`cloud`
`react_native`	`cloud`
`flutter`	`cloud`
`cordova`	`cloud`
`cloud`	`cloud`
`warehouse`	`cloud`

Every source type is cloud only — events reach the table from RudderStack’s servers, never in device mode.

warehouse is the token a Reverse ETL source resolves to. Rudder CLI manages the destination’s warehouse entry — if the destination is connected to a Reverse ETL source, set connection_mode.warehouse: cloud in its spec, or apply removes it.

Rudder CLI can also declare the Reverse ETL connection itself, for the types listed under Supported destinations — see Manage Reverse ETL Connections using Rudder CLI.

The dashboard additionally offers BigQuery Stream to AMP and Shopify sources. Rudder CLI doesn’t manage those connections, so amp and shopify are invalid here.

Connect a source

An event stream connection to this destination is checked against two rules at validate time.

The source’s type must be supported. A source’s type resolves to one of the tokens above before the check — a JavaScript source resolves to web, and webhook and server-side SDK sources resolve to cloud. Every type an event stream source can resolve to is supported here, so this check always passes.

The destination config must carry a connection_mode entry for that source type. This lives on the destination spec, not on the connection spec. Without it:

destination 'bigquery-stream-prod' config has no 'connection_mode' entry for source type 'web'

BigQuery Stream needs no additional config keys to connect a source of any type.

Secrets

credentials is the only secret key. Write it as a {{ .VAR }} reference and supply the value at apply time:

yaml

config:
  credentials: '{{ .BQSTREAM_CREDENTIALS }}'

bash

export RUDDER_BQSTREAM_CREDENTIALS="$(cat service-account.json)"
rudder-cli apply

# or
rudder-cli apply --var-file secrets.vars.yaml

Note that:

credentials is JSON, so write its reference in single quotes, as above. Rudder CLI pastes the value into the YAML before parsing it, and the JSON’s double quotes would end a double-quoted string.
rudder-cli import writes only the secrets the API returns, as {{ .VAR }} references, and leaves the others out. Add any secret key it leaves out, then supply the variables before you apply — see Secrets.

See How to Use Variable Substitution in Rudder CLI.

Changes by release

Version	Change
v0.29.0	Breaking: A `null` config value, written as `key: null`, `key: ~`, or `key:` with no value, fails `validate` with `'<key>' must not be null; set a value or remove the key`. Earlier releases accepted it: a top-level null planned an update on every `apply`, and a null inside a list item failed at the API. What to do: Remove the key to leave it unset, or give it a value.
v0.29.0	Changed: `validate` warns when a secret key holds anything but one `{{ .VAR }}` reference with no default, and when one variable fills two secret keys. It also masks secret values in the lines it prints.
v0.29.0	Changed: `rudder-cli import` writes each secret reference single-quoted, as `'{{ .VAR }}'`, so a value containing double quotes, like a JSON key file, substitutes as a string. A single quote in a value that fills such a reference is written as `''`. What to do: In a spec imported by an earlier release, quote each reference to a JSON value by hand: `'{{ .VAR }}'`.
v0.29.0	Changed: `validate` reports a value of the wrong type once per key, such as `'email' must be a string`, instead of one raw decoding error.
v0.28.0	Added: `warehouse` source type, for destinations connected to a Reverse ETL source. Set `connection_mode.warehouse: cloud` for such a destination, or `apply` removes the entry.

For changes across every destination type, see Destination Type Changes by Release.

See more

BigQuery Stream Destination for table schema requirements and deduplication cases
Destination Type Reference for Rudder CLI for the rules shared across destination types
Destination YAML Reference for the spec envelope

BigQuery Stream Destination Config Reference

Sample configuration

Config keys

Table

Deduplication

Authentication

Per-source keys

Source types

Connect a source

Secrets

Changes by release

See more

Questions? Let's figure it out together.