Snowflake Streaming Assisted Migration Feature Beta

Migrating from RudderStack’s traditional Snowflake batch destination to the new Snowflake Streaming destination unlocks real-time data ingestion using Snowflake’s Snowpipe Streaming API. However, manually managing this migration — including creating new destinations, reconnecting sources, and coordinating the cutover — can be complex and risky.

With the assisted migration feature, you can now migrate your existing Snowflake pipelines with a guided workflow that handles all the heavy lifting for you. This feature automatically creates and configures your new Snowflake Streaming destination, manages the transition between pipelines, and minimizes duplicate records during the migration process.

Key capabilities

• Automated destination setup: RudderStack creates a new Snowflake Streaming destination using your existing batch destination credentials and configuration, preserving your schema and table mappings.
• Guided verification: A step-by-step workflow prompts you to verify that events flow correctly through the new Snowflake Streaming pipeline before completing the migration.
• Automatic pipeline coordination: The Snowflake batch destination is automatically paused during migration to minimize duplicate records while both pipelines are active.
• Final sync trigger: You can trigger a final sync from the batch destination to ensure any events collected during the pause are loaded to your warehouse.
• Rollback capability: If any errors occur during setup, you can roll back to your previous state without any data loss.
• Deduplication guidance: Receive migration timestamps and documentation to help you deduplicate any records that may have been ingested during the transition.

Why migrate to Snowflake Streaming?

The Snowflake Streaming destination delivers event data to Snowflake in seconds via continuous micro-batches, enabling real-time analytics, alerting, A/B testing, and fresher data for personalization and AI/ML use cases. Because it uses the Snowpipe Streaming API instead of warehouse compute, it maintains the same or lower cost compared to the batch integration while dramatically reducing latency.

Get started

1. Go to your existing Snowflake batch destination in the RudderStack dashboard.
2. Look for the Migrate to Snowflake Streaming banner.
3. Click Proceed to initiate the assisted migration process.

Follow the guided steps to verify your new Snowflake Streaming destination and complete the migration. If needed, reference the provided timestamps to deduplicate records in your target tables.

See the Snowflake Streaming Migration Guide for more information on manual migration options and deduplication queries.

Screenshots

Questions? Contact us by Email or on Slack