# Configurable Alerts in RudderStack RudderStack's smart alerting capabilities let you set up notifications for critical data issues so you can take appropriate actions immediately before they escalate into major problems. {{< announcement >}} The following features are now available in **Private Beta** — contact [RudderStack Support](mailto:support@rudderstack.com) to get access to these features for your workspace. - Enhanced resource-level overrides for each alert type - Resource-level overrides for alert delivery channels See the [Private Beta features](#private-beta-features) section for more information. {{< /announcement >}} ## Overview With RudderStack's alerting feature, you can configure alerts for: - Event Stream latency - Event delivery failures - Pre-sync or sync failures - Event volume drops - Transformation failures - Tracking Plan violations - Partial row failures and fatal syncs (for Reverse ETL connections) - Profiles run failures You can also use the alerting feature to: - [Set up alert delivery channels](#set-up-alert-delivery-channels) of your choice like email, Slack, PagerDuty, Incident.io, custom webhook, or Microsoft Teams. Once the alert threshold is hit, RudderStack automatically delivers alerts to these systems. - Set failure thresholds at the [workspace](#workspace-level-alerts) and [resource](#resource-level-alerts) level, so you are alerted only when necessary. ## Required permissions - Only [Admins]({{< ref "access-management/member-management.md#member-roles" >}}) can set up workspace-level alerts. - Only [Admins]({{< ref "access-management/member-management.md#member-roles" >}}) and [Members]({{< ref "access-management/member-management.md#member-roles" >}}) with the [**Alert Overrides**]({{< ref "access-management/policies-overview.md#resource-permissions" >}}) permission can set up resource-level alerts. {{< image src="images/access-management/alert-overrides.webp" alt="Alert Overrides permissions in new Access Management system" >}} #### Permissions for legacy RBAC system In the [legacy Permissions Management (RBAC) system]({{< ref "archive/dashboard-guides/user-management.md" >}}): - Only [Org Admins]({{< ref "archive/dashboard-guides/user-management.md#organization-roles" >}}) can set up workspace-level alerts - Only [Org Admins]({{< ref "archive/dashboard-guides/user-management.md#organization-roles" >}}) and members with the **Connections Admin** role in their workspace policy can set up resource-level alerts {{< image src="images/access-management/tracking-plan-permissions-legacy-framework.webp" alt="Data Catalog permissions in the legacy framework" >}} ## Workspace-level alerts Go to **Settings** > **Workspace** and click the **Alerts** tab to set up the workspace-level alerts. {{< image src="images/data-governance/configurable-alerts/alerts-tab.webp" alt="Alerts option in RudderStack dashboard" >}} RudderStack automatically delivers alerts on the [configured channels](#set-up-alert-delivery-channels) if the failures exceed the threshold percentage within the last one hour. {{< warning >}} If you set the error threshold to 0%, even a single failure in processing or delivering events will trigger an alert. {{< /warning >}} The following sections detail the different alert types available per RudderStack feature: ### Event Stream In this section, you can configure alerts and set thresholds for the following incidents: | Failure type |
Description
| Applicable to | | :------ | :--------| :---------| | Cloud destination pipeline failures | Failures in processing or delivering events to a destination due to incorrect credentials, destination downtime, network error, or any other reason. | [Event Stream destinations]({{< ref "destinations/streaming-destinations/" >}}) | | P95 latency
Enterprise plan only | Maximum latency for 95% of the events to reach the destination.

See [P95 latency alerts](#p95-latency-alerts) for more details. | [Event Stream destinations]({{< ref "destinations/streaming-destinations/" >}}) | | Warehouse pre-sync failures | Failures in processing or storing the events in object storage before forwarding them to the warehouse destination. | [Warehouse destinations]({{< ref "destinations/warehouse-destinations/" >}}) | | Warehouse sync failures | Failures in syncing events to a warehouse destination, that is, syncs to a warehouse destination are aborted. Possible reasons include:

| [Warehouse destinations]({{< ref "destinations/warehouse-destinations/" >}}) | | Low event volume | Event volume drop in the last one hour is more than the configured threshold, as compared to the same period from the last week.

See [Low event volume alerts](#low-event-volume-alerts) for more details. | [Event Stream sources]({{< ref "sources/event-streams/" >}}) | | Tracking Plan violations | [Tracking Plan violations]({{< ref "data-governance/tracking-plans/violation-management.md#violation-types" >}}) for a particular source, that is, the incoming source events and properties do not comply with the Tracking Plan connected to that source. | [Event Stream sources]({{< ref "sources/event-streams/" >}}) | | Transformation failures | Failures in processing events due to errors in the transformation.

See [Transformation failure alerts](#transformation-failure-alerts) for more details. | [Transformations]({{< ref "transformations/overview.md" >}}) | #### P95 latency alerts {{< announcement >}} P95 latency alerts are available only in the [Enterprise plan](https://www.rudderstack.com/enterprise-quote/). {{< /announcement >}} RudderStack triggers the **P95 latency** alert if the maximum latency experienced by 95% of the events to reach the destination exceeds the specified threshold. For example, if you send 100 events to an Event Stream destination and the P95 latency alert threshold is set to 15 minutes, then 95 of those events should be delivered in **strictly less than** the specified threshold (that is, 15 minutes). Otherwise, RudderStack triggers an alert. {{< warning >}} **Important considerations** The P95 latency metric and alerts are applicable **only** for [Event Stream destinations]({{< ref "destinations/streaming-destinations/" >}}) connected to sources in [cloud mode]({{< ref "destinations/rudderstack-connection-modes.md#cloud-mode" >}}). They are **not applicable** for: - [Reverse ETL connections]({{< ref "sources/reverse-etl/" >}}) - [Warehouse destinations]({{< ref "destinations/warehouse-destinations/" >}}) - Event Stream destinations connected to sources in [device mode]({{< ref "destinations/rudderstack-connection-modes.md#device-mode" >}}) {{< /warning >}} Go to the **Events** tab of your Event Stream destination and click the **Latency** tab to check the average P95 latency across all Event Stream sources. {{< image src="images/data-governance/configurable-alerts/p95-latency-events.webp" alt="P95 latency for destination" >}} To view the P95 latency for events coming from a particular source, filter the Event Stream source from the dropdown: {{< image src="images/data-governance/configurable-alerts/filter-source-latency.webp" alt="View source-specific P95 latency" >}} #### Low event volume alerts RudderStack triggers the **Low event volume** alert if the event volume drop for an [Event Stream source]({{< ref "sources/event-streams/" >}}) in the last one hour exceeds the configured threshold compared to the event volume for the same time period in the last week. The **Low event volume** alert is triggered based on the below formula: {{< image src="images/data-governance/configurable-alerts/low-event-volume-equation.webp" alt="Equation to calculate low event volume triggers" >}} For example, you will get an alert if: - An Event Stream source ingests 450 events within the last hour (for example, 11 a.m. to 12 p.m.), but it ingested 1000 events from 11 a.m. to 12 p.m. a week before. - The alert threshold was set to 50%. In this case, RudderStack triggers an alert as the volume drop percentage (55%) exceeds the configured threshold (50%). {{< info >}} Note that: - A 0% threshold indicates that RudderStack triggers an alert **even if** the number of ingested events in the past one hour is the same as last week. That is, `last_week_window_count` = `current_window_count`. - A 100% threshold indicates that RudderStack triggers an alert if the source ingested **no** events in the last one hour but some events (>0) exactly a week before. That is, `current_window_count` = 0 and `last_week_window_count` > 0. {{< /info >}} #### Transformation failure alerts RudderStack triggers the **Transformation failures** alert if the [transformation]({{< ref "transformations/overview.md" >}}) failures in the last one hour exceed the configured threshold. For example, you will get an alert if: - You set the alert threshold to 10%. - The transformation ingests 100 events in the past one hour. - 11 events are dropped due to a transformation error, that is, failure to transform the event correctly as per the specified logic. ### Reverse ETL In this section, you can configure alerts for the following incidents applicable to all your [Reverse ETL sources]({{< ref "sources/reverse-etl/" >}}): | Failure type | Description | | :------ | :--------| | Partial row failures | Failures in syncing records from the warehouse source to the connected destination. | | Sync row failures
Beta | Failures in syncing rows from the warehouse source to the connected destination, scoped at an **individual sync level**. | | Fatal syncs | Fatal errors causing a running sync to be aborted. Possible reasons include: | #### Partial row failures vs. Sync row failures This section details the differences between **Partial row failures** and **Sync row failures**. #### Partial row failures **Partial row failures** trigger an alert when the number of failed rows exceeds the configured threshold for a particular connection within a rolling 1 hour time window, **irrespective** of the number of syncs completed within that time. RudderStack evaluates this alert while a sync is still running, so you get feedback without waiting for the sync to finish. **Sync row failures**, by contrast, are evaluated only after each sync completes. #### Sync row failures {{< stability level="beta" >}} **Sync row failures** trigger an alert when the number of failed rows exceeds the configured threshold. RudderStack evaluates the failure count and triggers alerts at the end of each sync. The same [24-hour alert delivery limit](#alert-frequency) applies to this alert type as with other RudderStack alerts. {{< info >}} RudderStack recommends using: - **Partial row failures** to monitor overall connection health and get quick feedback on any failures - **Sync row failures** to monitor the health of a particular sync and get fewer alerts Both alert types follow the same [24-hour alert delivery limit](#alert-frequency) as your other alerts — at most one notification per resource per alert type per channel in any 24-hour period. {{< /info >}} ### Profiles In this section, you can configure alerts for the following incidents applicable to your [Profiles]({{< ref "profiles/overview/" >}}) runs: | Failure type | Description | | :------ | :--------| | Run failures | Failures encountered while running your Profiles project. | ### Custom alerts Click the **Custom alerts** setting present below each failure type to view the resources for which [custom alert overrides](#resource-level-alerts) are configured: {{< image src="images/data-governance/configurable-alerts/custom-alerts-setting.webp" alt="Custom alert setting" >}} The resulting sidebar lists all the resources with custom alerts categorized by failure type. You also see the following information: - **Name**: The resource name. - **Subscribed**: Whether alerts are on or off for that failure type. - **Threshold**: Custom alert threshold value set for that resource. Click on a resource to change these settings. {{< image src="images/data-governance/configurable-alerts/custom-alerts.webp" alt="Custom alerts" >}} ## Set up resource-level alerts {{< warning >}} **Before you set up resource-level alerts** - Once you set the alert overrides for a particular resource, any changes to the workspace-level settings will not be applicable for that resource. - Currently, you cannot change the alert delivery channels for a particular resource. However, this feature is available in **Private Beta** — see [Resource-level alerts (Private Beta)](#resource-level-alerts-private-beta) for more information. {{< /warning >}} To configure resource-level alerts: 1. Go to the resource (source or destination) and click the **Alerts** tab. {{< image src="images/data-governance/configurable-alerts/resource-level-alert-settings.webp" alt="Resource-level alert settings" >}} 2. If configured, you will see the workspace-level alert settings and thresholds enabled for the resource by default. Change these settings and set custom thresholds for this resource. 3. Once you change the settings, you will automatically see the following message pop up: {{< image src="images/data-governance/configurable-alerts/resource-level-alerts.webp" alt="Resource-level alerts" >}} #### Resource-specific alert types The following table lists the alert types applicable to a particular resource: | Resource type | Alert type | | :-----| :-----| | [Event Stream source](#event-stream) | | | [Event Stream destination](#event-stream) | | |[ Warehouse destination](#event-stream) | | | [Reverse ETL source](#reverse-etl) | | | [Profiles project](#profiles) | Run failures | ## Set up alert delivery channels You can set up dedicated alert delivery channels to get notified whenever your sources or destinations have failures or errors. This allows you to take proactive measures to fix the problems before they escalate into major issues. {{< info >}} Note that: - You can set up separate alert delivery channels for your Event Stream and Reverse ETL pipelines. - Toggling off alerts for a channel automatically removes all the configurations. You will have to reconfigure the channel to use it again. - RudderStack limits the alert delivery to one alert per resource per [alert type](#resource-specific-alert-types) for each configured channel every 24 hours. {{< /info >}} RudderStack provides the following alert delivery channel options: ### Slack 1. Toggle on the **Slack** setting in the dashboard. {{< image src="images/data-governance/configurable-alerts/slack-alerts.webp" alt="Slack channel configuration for alerts" >}} 2. Set the Slack channel and authorize RudderStack to post the alerts by clicking **Allow**. {{< info >}} Note that: - You should be an admin of the Slack workspace to grant RudderStack the necessary permissions to post to that channel. - While setting the Slack channel, you will see a **This app is not approved by Slack** ribbon at the top — this is because Slack has not reviewed the app yet. However, it is completely safe to install. {{< /info >}} {{< image src="images/data-governance/configurable-alerts/configure-slack-channel.webp" alt="Slack channel configuration for alerts" >}} Once the alert is triggered, RudderStack automatically sends a notification on the specified Slack channel. Click **Review on RudderStack** to go to the specific resource (source or destination) to investigate and fix the errors. {{< image src="images/data-governance/configurable-alerts/slack-alerts-new.webp" alt="Slack alerts" >}} ### Microsoft Teams 1. [Create an incoming webhook](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook?tabs=newteams%2Cdotnet) for the Teams channel you wish to use. 2. Toggle on the **MS Teams** setting in the dashboard and enter the incoming webhook URL to receive alerts on your preferred Teams channel. {{< image src="images/data-governance/configurable-alerts/teams-alerts.webp" alt="Teams channel configuration for alerts" >}} 3. Once the alert is triggered, RudderStack automatically sends a notification on the specified Teams channel — click **Review on RudderStack** to go to the specific resource (source or destination) to investigate and fix the errors. {{< image src="images/data-governance/configurable-alerts/teams-alerts-2.webp" alt="Teams alerts" >}} ### PagerDuty 1. [Create a service with an Events API v2 integration](https://support.pagerduty.com/main/docs/services-and-integrations#create-a-generic-events-api-integration) in your PagerDuty account and obtain the integration key. 2. Toggle on the **PagerDuty** setting in the dashboard and specify the **Integration Key** to receive alerts as PagerDuty incidents. {{< image src="images/data-governance/configurable-alerts/pagerduty-alerts.webp" alt="PagerDuty alerts" >}} ### Incident.io 1. [Create an HTTP alert source](https://docs.incident.io/alerts/custom-http-sources) in your Incident.io account and obtain the following credentials: - Secret token (API key) - Alert source config ID 2. Toggle on the **Incident.io** setting in the dashboard and specify the **API Key** and **Alert Source Config ID** to receive alerts as Incident.io incidents. {{< image src="images/data-governance/configurable-alerts/incidentio-alerts.webp" alt="Incident.io alerts" >}} ### Webhook Toggle on the **Webhook** setting in the dashboard and specify the webhook URL to receive the alerts. {{< info >}} RudderStack sends the alerts as a `POST` request to the configured endpoint while following the Prometheus styling format. Prometheus is a widely accepted monitoring and alerting tool. Its alert format is compatible with various other monitoring and incident management tools like Squadcast, PagerDuty, etc. {{< /info >}} A sample webhook response is shown: ```json { "alerts": [{ "endsAt": "0001-01-01T00:00:00Z", "labels": { "severity": "critical", "alertname": "partial-row-failures", "workspace": "", "destination": "Failing Webhook", "workspaceId": "", "organization": "", "destinationId": "", "organizationId": "", "configuredThreshold": 61 }, "status": "firing", "startsAt": "2024-02-05T00:02:49.933Z", "annotations": { "description": "Errors in processing or delivering events to Failing Webhook destination have exceeded the configured threshold of 61% within last 1 hour" }, "fingerPrint": "d9885cc7f11b8db0" }], "status": "firing" } ``` #### Send alerts to downstream tools You can also leverage webhooks to forward the alerts to any RudderStack-supported downstream tool: 1. Set up a [webhook source]({{< ref "sources/event-streams/cloud-apps/webhook-source.md" >}}). Note the webhook URL containing the source write key parameter. {{< image src="images/data-governance/configurable-alerts/webhook-source.webp" alt="Webhook source" >}} 2. Set up a [destination integration]({{< ref "destinations/streaming-destinations/" >}}), for example, [PagerDuty]({{< ref "destinations/streaming-destinations/pagerduty.md" >}}). Connect it to the webhook source created in Step 1. 3. Once you set up the connection, specify the webhook source URL obtained in Step 1 in the **Enter URL** field where RudderStack forwards the alerts. {{< image src="images/data-governance/configurable-alerts/webhook-alerts.webp" alt="Webhook configuration for alerts" >}} ### Email Toggle on the **Email** setting in the dashboard and specify comma-separated email addresses of the users who would like to receive the alerts. {{< image src="images/data-governance/configurable-alerts/email.webp" alt="Email alerts" >}} Once the alert is triggered, these users will automatically get email alerts to investigate and fix the errors. {{< image src="images/data-governance/configurable-alerts/email-alert-new.webp" alt="Email alerts" >}} ## Alert delivery frequency RudderStack limits the alert delivery to one alert per resource (source or destination) per [alert type](#resource-specific-alert-types) for each configured channel every 24 hours. This alerting logic ensures you are not spammed with notifications, especially in cases where you have configured multiple [alert types](#resource-specific-alert-types) for your pipelines and some resources have their own overrides (custom alert settings) in place. #### Example Suppose you get a [Partial row failures](#resource-specific-alert-types) alert for a particular Reverse ETL source. You will not get another alert for the same failure type for another 24 hours even if your data syncs are scheduled at a lesser frequency (for example, every one, five, or 12 hours). However, if that source encounters another failure type like a fatal sync, RudderStack will trigger an alert. ## Private Beta features {{< announcement >}} Contact [RudderStack Support](mailto:support@rudderstack.com) to get access to these features for your workspace. {{< /announcement >}} ### Resource-level alerts {{< info >}} In the **Private Beta**, the resource now inherits all workspace-level alert settings except the ones you explicitly override. For example, if you override the setting for the **Low event volume** alert type for an Event Stream source but do not change the **Tracking Plan violations** alert, then the latter will continue to reflect the setting specified at the workspace level. {{< /info >}} To configure resource-level alerts: 1. Go to the resource (source or destination) and click the **Alerts** tab. {{< image src="images/data-governance/configurable-alerts/pb-resource-level-alert-settings.webp" alt="Resource-level alert settings" >}} 2. If configured, you will see the workspace-level alert settings and thresholds for the resource by default. Set custom thresholds for this resource as needed. 3. Click **Create override** in the confirmation dialog to set the override. {{< image src="images/data-governance/configurable-alerts/pb-confirm-override.webp" alt="Confirm resource-level alert override" >}} To revert the alert override back to the default workspace setting: 1. Click the reset icon (rounded arrow) next to the custom setting for the alert. {{< image src="images/data-governance/configurable-alerts/reset-to-workspace-settings.webp" alt="Revert to default workspace setting" >}} 2. Click **Reset** on the resulting confirmation dialog to remove the custom settings. {{< image src="images/data-governance/configurable-alerts/confirmation-dialog.webp" alt="Revert to workspace setting confirmation dialog" >}} ### Resource-level alert delivery channels {{< info >}} In the **Private Beta**, you can now set up specific alert delivery channels for individual resources. This gives you fine-grained control over how alerts are routed, allowing different sources and destinations to send notifications to different channels based on your operational needs. {{< /info >}} To set up resource-level alert delivery channels: 1. Go to the resource and click the **Alerts** tab. 2. Under the **Channels** section, click **Setup** next to the channel you want to configure. 3. Configure your preferred platform just as you would at the [workspace level](#set-up-alert-delivery-channels). Note that: - These resource-level alert delivery channel settings override the workspace-level settings **for that resource only**. - If no channel overrides are set, or you have explicitly disabled all alert delivery channels for that resource, RudderStack automatically forwards all alerts to the channels configured at the [workspace level](#set-up-alert-delivery-channels). {{< image src="images/data-governance/configurable-alerts/no-channel-configured.webp" alt="Dashboard messaging when no channel is configured for a resource" >}} To revert the channel overrides back to the default workspace settings: 1. Click the reset icon (rounded arrow) next to the custom setting for the delivery channel. {{< image src="images/data-governance/configurable-alerts/reset-channel-to-workspace-settings.webp" alt="Revert to default workspace setting" >}} 2. Click **Reset** on the resulting confirmation dialog to remove the custom settings and revert to the default workspace setting. {{< image src="images/data-governance/configurable-alerts/confirmation-dialog-channel-reset.webp" alt="Revert to workspace setting confirmation dialog" >}}