# Databricks Reverse ETL Source

[Databricks](https://databricks.com/) is a data analytics platform that lets you easily integrate with open source libraries. It offers a simple collaborative environment to run interactive and scheduled data analysis workloads.

{{< success >}}
You can now ingest data into RudderStack by running queries on your Databricks cluster or [SQL warehouse](https://docs.databricks.com/en/sql/admin/create-sql-warehouse.html).
{{< /success >}}

## Grant permissions {#granting-permissions}

Before you set up Databricks as a source, you must grant certain permissions in your Databricks instance for RudderStack to access data from it.

Run the queries listed in the following sections in the  **exact order** to grant the required permissions:

### Step 1: Add user {#step-1-add-a-user}

Add a new user (for example, user@example.com) by following the steps in the [Databricks documentation](https://docs.databricks.com/administration-guide/users-groups/users.html#add-a-user).

### Step 2: Create RudderStack schema and grant permissions to the role {#creating-the-rudderstack-schema-and-granting-permissions}

1. Create a dedicated schema `_rudderstack`.

```sql
CREATE SCHEMA `_rudderstack`;
```

{{< warning >}}
The `_rudderstack` schema stores Reverse ETL sync state, snapshots, and related tables. **Do not change this name**.

See [`_rudderstack` Schema Reference]({{< ref "data-pipelines/reverse-etl/developer-guides/rudderstack-schema.md" >}}) for more details.
{{< /warning >}}

2. Grant full access to the schema `_rudderstack` for the user created in step 1.

```sql
GRANT ALL PRIVILEGES ON SCHEMA `_rudderstack` TO `user@example.com`
```

Replace `user@example.com` with the user created in step 1.

## Set up Databricks source in RudderStack {#setting-up-the-databricks-source-in-rudderstack}

1. Log in to your [RudderStack dashboard](https://app.rudderstack.com/).
2. On the **Connections** page, click **Add source**.
3. Under **Sources**, click **Reverse ETL** and select **Databricks**.

### Configure warehouse credentials {#configuring-the-connection-credentials}

You can choose to proceed with your existing warehouse credentials if you have configured them in the RudderStack dashboard previously. Otherwise, click **Add new credentials** to add the connection details of your Databricks cluster or SQL warehouse:

{{< info >}}
For most use cases, RudderStack recommends using a SQL warehouse over a cluster as they generally cost less and are faster to spin up. In contrast, clusters are used for much larger operations that require more resources.
{{< /info >}}

- **Host**: Enter the server hostname.
- **Port**: Enter the port number.
- **Path**: Enter the HTTP path.
- **Authentication Type**: Choose the authentication type for authorizing access to Databricks resources. RudderStack provides two options - **M2M OAuth** and **Personal Access Token**.
    - If you select **M2M OAuth**, then provide the below settings:

        - **Client ID**: The client ID for the service principal.
        - **Client Secret**: The corresponding client secret.

    See the [Databricks documentation](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m#step-2-create-an-oauth-secret-for-a-service-principal) for detailed steps on obtaining the **Client ID** and **Client Secret** fields.

    - If you select **Personal Access Token**, then provide the below setting:

        - **Token**: Enter your Databricks access token. See [Generating the Databricks access token]({{< ref "#generating-the-databricks-access-token" >}}) for more information on generating this token.

{{< warning >}}
Databricks [strongly recommends](https://docs.databricks.com/aws/en/dev-tools/auth/#what-authorization-option-should-i-choose) using OAuth over Personal Access Tokens for authorization.

OAuth tokens are automatically refreshed by default and do not require you to directly manage the access token, thereby improving your security against token hijacking and unauthorized access.
{{< /warning >}}

- **Catalog**: Enter the name of your Unity catalog. See [Databricks documentation](https://docs.databricks.com/en/data-governance/unity-catalog/create-catalogs.html#view-catalog-details) for more information on getting the catalog details.

See the following FAQs for more information on obtaining the host, port, path, and token for your Databricks instance depending on type:

- [Databricks cluster](#databricks-cluster-connection-credentials)
- [SQL warehouse](#databricks-sql-warehouse-connection-credentials)

Click the **Verify** button on the top right. RudderStack will then [verify and validate your credentials](#faq). Once verified, click **Continue** to proceed.

### Specify name and source type

{{< customreadfile "/includes/retl/retl-name-sourcetype.md" >}}

#### **Use warehouse table as source**

{{< customreadfile "/includes/retl/table-as-source.md" >}}

#### **Use model as source**

{{< customreadfile "/includes/retl/model-as-source.md" >}}

#### **Use audience as source**

{{< customreadfile "/includes/retl/audience-as-source.md" >}}

### Review and complete setup

{{< customreadfile "/includes/retl/review-retl-setup.md" >}}

## Connect destination

{{< customreadfile "/includes/retl/connect-retl-destination.md" >}}

## Update source configuration and settings

{{< customreadfile "/includes/retl/update-retl-source-configuration.md" >}}

## FAQ 

#### **Where can I obtain the connection credentials for the Databricks cluster?** {#databricks-cluster-connection-credentials}

To obtain the **Host**, **Path**, and **Port** number, go to your Databricks account and follow these steps:

1. Go to the **Compute** tab and select your Databricks cluster.
2. Click **Advanced options** > **JDBC/ODBC** tab to find the required settings:

{{< image src="images/warehouse-actions-sources/connection-settings-databricks.webp" alt="Select Databricks source in RudderStack" >}}

To obtain the **Token**, go to the **Settings** > **User Settings** in your Databricks account and generate a new personal access token:

{{< image src="images/warehouse-actions-sources/connection-settings-databricks-2.webp" alt="Select Databricks source in RudderStack" >}}

{{< info >}}
See [Databricks documentation](https://docs.databricks.com/dev-tools/api/latest/authentication.html#generate-a-personal-access-token) for more details on generating a personal access token.
{{< /info >}}

#### **Where can I obtain the connection credentials for the SQL warehouse?** {#databricks-sql-warehouse-connection-credentials}

To obtain the **Host**, **Path**, and **Port** number for your SQL warehouse, go to your Databricks account and follow these steps:

1. Go to the **SQL warehouses** tab and select your warehouse.
2. Click the **Connection details** tab to find the **Host**, **Path**, and **Port** number.

{{< image src="images/warehouse-actions-sources/connection-settings-sql-warehouse.webp" alt="SQL warehouse connection details" >}}

To obtain the **Token**, go to the **Settings** > **User Settings** in your Databricks account and generate a new personal access token:

{{< image src="images/warehouse-actions-sources/connection-settings-databricks-azure-token.webp" alt="Databricks access token" >}}

{{< info >}}
See [Databricks documentation](https://docs.databricks.com/dev-tools/api/latest/authentication.html#generate-a-personal-access-token) for more details on generating the access token.
{{< /info >}}

{{< customreadfile "/includes/retl/retl-faq.md" >}}

#### **Does my SQL warehouse/cluster need to be active when running the validations?**

Yes - otherwise, the validations might fail.