Reverse ETL Improvements

We have revamped RudderStack’s Reverse ETL feature with several enhancements for better reliability, observability, and usability. These include faster syncs, improved error handling, and an enhanced UI.

Reliability

Faster syncs

You can now sync data faster with clear indicators marked for the succeeded, failed, and dropped events and the number of deltas in each sync. This also reduces the likelihood of syncs getting stuck due to warehouse errors.

Error categorization

The RudderStack dashboard is now updated to show comprehensive and user-friendly error messages more prominently, with suggested actions to fix them. This makes troubleshooting errors easier for users without needing help or support from the RudderStack team.

Adaptive rate limiting

Whenever a sync hits a destination’s rate limit, RudderStack automatically adjusts the rate of sending events so that you do not encounter any 429 errors and the pipeline continues at a steady pace.

Mirror mode redesign

The mirror mode for syncing events is now faster and more reliable.

Async API

We have introduced the Async API to reduce the likelihood of encountering a timeout error while running the warehouse queries to fetch preview, columns, tables, etc., during the creation of a Reverse ETL model, audience, or a sync.

Scalable retries for failed records

RudderStack now retries syncs for failed records scalably while maintaining a cap on the number of retries. You can also turn on/off your sync retries based on the error category.

Batch support and bulk data upload

We have optimized our top Event Stream destinations to incorporate batch APIs (wherever possible) to handle large volumes of data - a very common scenario in Reverse ETL syncs.

Observability

Health dashboard

You can now monitor the health of your syncs using the Health dashboard, which summarizes the status of the latest sync providing details like:

• Status and duration of the latest sync
• Failures (Percentage of deltas (new rows) that failed to sync)
• Invalids (Invalid records sent from the source)
• Summary of failed or aborted syncs in the selected duration (1 day, 1 week, or 1 month)

Alerts

You can set alerts and notifications to your preferred channel (such as Slack, Microsoft Teams, webhook, email, Pagerduty, etc.) for fatal syncs. This gives greater visibility into the health and status of your Reverse ETL syncs, so you can proactively monitor and take action when required.

Improved syncs page

Reverse ETL syncs page is revamped to provide more visibility into the status and progress of a sync. It displays details about the sync trend over time, so that at any point in time, you can see how many events are queued, succeeded, failed, or retrying.

Usability

One-to-many connections

RudderStack lets you connect a Reverse ETL source to multiple destinations so that you can sync data from the same source to different destinations with different settings.

Cancel and drain ongoing syncs

You can cancel any ongoing sync without turning off the source. RudderStack automatically triggers a draining process in the background once you stop the sync to make sure the next sync is not affected and you don’t have to wait for the events already in the pipeline to be delivered or failed.

Faster and clearer sync statuses

We have made several improvements in the overall event syncing process to improve user experience, like:

• Communicating the sync status clearly by updating the metrics at the same time when a sync is marked as complete.
• Reducing sync completion time significantly.
• Providing real-time feedback on the sync status
• Faster page load times to reduce any likelihood of timeouts when refreshing the syncs page or checking the sync history.
• Better navigation of past syncs, making them easy to find.

Questions? Contact us by email or on Slack