Snowflake Streaming Destination Beta

Snowflake Streaming is a powerful tool for handling real-time data streams. It allows you to stream data rows directly in Snowflake tables with minimal latency.

Find the open source code for this destination in the GitHub repository.

When to use Snowflake Streaming

Snowflake Streaming is ideal for the following scenarios:

• Continuous data streams: If your data sources produce a steady stream of data in small batches, for example, clickstream data, event logs, etc.
• Low latency requirements: When your application requires immediate/frequent updates to the Snowflake table as new data arrives. Some use cases include real-time dashboards, streaming analytics, etc.
• Cost optimization for streaming data: Snowflake Streaming can potentially reduce costs for real-time pipelines as it allows micro-batch ingestion without needing a traditional Snowflake warehouse running constantly.

Prerequisites

Before setting up the Snowflake Streaming destination in RudderStack:

• You will need to have the ACCOUNTADMIN system role or any account with MANAGE GRANTS privilege.
• You will need to set the correct user permissions in Snowflake for RudderStack to send the data correctly.

The following sections walk you through the process of setting up a virtual warehouse, database, role, and user in Snowflake.

1. (Optional) Create a warehouse

Note the following:

• You can skip this step if you wish to use an existing warehouse.
• RudderStack does not use the warehouse for data loading. It only uses the warehouse to run some queries for performing connection validations.

In your Snowflake console, create a X-Small warehouse.

Alternatively, run the following SQL commands to create a new warehouse:

CREATE WAREHOUSE "<WAREHOUSE_NAME>"
  WITH WAREHOUSE_SIZE = 'XSMALL'
    WAREHOUSE_TYPE = 'STANDARD'
    AUTO_SUSPEND = 600
    AUTO_RESUME = TRUE;

Make sure to set AUTO_SUSPEND to ~10 minutes and enable AUTO_RESUME to avoid any extra costs.

2. Create a database

RudderStack recommends creating a new database to avoid conflicts with your existing data. Note that RudderStack creates its own tables within this database while storing your events.

Alternatively, you can create a new database by running the following SQL command:

CREATE DATABASE "<DATABASE_NAME>";

3. Create a role for RudderStack

To create a new role with the required permissions to load your data into the above database, run the following SQL commands in the exact order. Make sure to replace the placeholder names with your preferred names.

1. Create a new role called <ROLE>:

CREATE ROLE "<ROLE>";

2. Grant access to the warehouse <WAREHOUSE_NAME>:

GRANT USAGE ON WAREHOUSE "<WAREHOUSE_NAME>" TO ROLE "<ROLE>";

3. Grant access to the database <DATABASE_NAME>:

GRANT USAGE ON DATABASE "<DATABASE_NAME>" TO ROLE "<ROLE>";
GRANT CREATE SCHEMA ON DATABASE "<DATABASE_NAME>" TO ROLE "<ROLE>";
GRANT ALL ON ALL SCHEMAS IN DATABASE "<DATABASE_NAME>" TO ROLE "<ROLE>";

You can also create a custom role with the necessary permissions and specify it in the dashboard settings while setting up the Snowflake Streaming destination in RudderStack.

4. Create a user

Use the below query to create a Snowflake user that connects RudderStack to your warehouse.

Make sure to replace the placeholder names with your preferred values.

CREATE USER "<USER_NAME>"
  MUST_CHANGE_PASSWORD = FALSE
  DEFAULT_ROLE = "<ROLE>"
  PASSWORD = "<STRONG_PASSWORD>";
GRANT ROLE "<ROLE>" TO USER "<USER_NAME>";

Setup

If your sources send data to different schemas, RudderStack recommends creating multiple Snowflake Streaming destinations (each with a different namespace) and connecting the relevant sources accordingly.

1. In your RudderStack dashboard, add a source. Then select Snowflake Streaming from the list of destinations.
2. Assign a name to your destination and click Continue.

Connection settings

Snowflake account ID examples

The below table illustrates the slight differences in the account IDs depending on the various cloud providers. See the Snowflake documentation for more information on the account locator formats depending on your region or cloud provider.

Advanced settings

Consent settings

IPs to be allowlisted

To enable network access to RudderStack, allowlist the following RudderStack IPs depending on your region and RudderStack plan:

All the outbound traffic is routed through these RudderStack IPs.

Migrate from Snowflake destination

See the Snowflake Streaming Migration Guide for more information on how to migrate from the Snowflake batch destination to the Snowflake Streaming destination.

FAQ

What is the difference between Snowflake Streaming and a traditional Snowflake warehouse?

The following table highlights the key differences between Snowflake Streaming and the traditional Snowflake warehouse:

Why am I not seeing the users table in the schema for identify events?

This integration sends the identify events exclusively to the identifies table and skips the users table entirely.

Questions? Contact us by email or on Slack