# Amazon S3 Reverse ETL Source


{{< danger >}}
AWS has [closed new customer access](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) to Amazon S3 Select, effective July 25, 2024. The Amazon S3 Reverse ETL source depends on S3 Select to read your Parquet files, so this affects new source setups.

You cannot set up a new S3 Reverse ETL source if:

- You have not assigned [S3 Select permissions](https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html) to your S3 bucket role before July 25, 2024.
- You have not set up an S3 source in RudderStack before.

There is no workaround for new sources. Existing S3 sources with S3 Select access are unaffected and will continue to sync normally.

If your data already resides in a supported warehouse ([Snowflake]({{< ref "sources/reverse-etl/snowflake.md" >}}), [Google BigQuery]({{< ref "sources/reverse-etl/google-bigquery.md" >}}), [Amazon Redshift]({{< ref "sources/reverse-etl/amazon-redshift.md" >}}), or [Databricks]({{< ref "sources/reverse-etl/databricks.md" >}})), use it as a Reverse ETL source instead. For help in assessing your options, contact [RudderStack Support](mailto:support@rudderstack.com).
{{< /danger >}}

[Amazon S3](https://aws.amazon.com/s3/) is a cloud-based object storage service that lets businesses securely store their data at scale.

RudderStack supports S3 as a data source from which you can ingest data and route it to your desired downstream destinations.

## Set up S3 source in RudderStack {#setting-up-the-s3-source}

1. Log in to your [RudderStack dashboard](https://app.rudderstack.com/).
2. On the **Connections** page, click **Add source**.
3. Under **Sources**, click **Reverse ETL** and select **Amazon S3**.

### Connection credentials

Configure the following settings to authenticate RudderStack to access your S3 account: 

- **Connection Mode**: RudderStack provides the following options to connect to S3:

  - **Cross-Account Role (recommended)**: This option lets you connect to S3 through an [IAM access role](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html). To do so, you need to first create an IAM role for RudderStack with the required permissions to access your S3 account. See [Creating the RudderStack IAM Role for S3]({{< ref "#creating-the-rudderstack-iam-role" >}}) for the detailed steps.
  - **Access Key**: This option lets you connect to S3 using your AWS access key ID and secret access key.

{{< warning >}}
RudderStack recommends using the **Cross-Account Role** method for connecting to S3 as the **Access Key** method is deprecated and will be discontinued soon.
{{< /warning >}}

- **Account Name**: Specify a name that will be used to identify the connection account.

You will see the below settings depending on the **Connection Mode** selected above:

{{< tabs tabTotal="2" >}}
{{% tab tabName="Cross-Account Role" %}}
- **Role ARN**: Specify the ARN after [creating the RudderStack IAM role]({{< ref "#creating-the-rudderstack-iam-role" >}}).
{{% /tab %}}
{{% tab tabName="Access Key" %}}
- **AWS Access Key ID**: Specify your [AWS access key ID]({{< ref "#faq" >}}).
- **AWS Secret Access Key**: Enter the corresponding [secret access key]({{< ref "#faq" >}}).
{{% /tab %}}
{{< /tabs >}}

### S3 permissions

The **minimum** required permissions for S3 are listed below:

```json
"Action": [
  "s3:GetObject",
  "s3:ListBucket"
],
```

{{< success >}}
Before proceeding, RudderStack verifies if the specified credentials are correct and alerts in case of any errors or permissions issues.
{{< /success >}}

### Specify name, bucket, and prefix

- **Source name**: Assign a name to uniquely identify the source in the RudderStack dashboard.
- **S3 Bucket Name**:  Enter the S3 bucket name.
- **Prefix**: Prefix refers to the path within your S3 bucket from where RudderStack imports the data. For example, if **Prefix** is set to `RUDDER`, then RudderStack imports the data stored in the location `<your_s3_bucket>/RUDDER`.

{{< image src="images/warehouse-actions-sources/s3-bucket-settings.webp" alt="Bucket configuration settings" >}}

{{< warning >}}
Note that:

- Your S3 bucket should only consist of **Apache Parquet** files as RudderStack supports and can extract only Parquet files.
- The first row of the Parquet file should not have a `null` value (empty strings are allowed) for any column. It helps RudderStack to determine the correct schema of the file.
- RudderStack also considers all the nested folders present within the bucket.
{{< /warning >}}

### Review and complete setup

{{< customreadfile "/includes/retl/review-retl-setup.md" >}}

### Update source configuration and settings

Go to the **Configuration** tab of your S3 source to update the configuration settings. Here, you can update the S3 bucket name and prefix.

Go to the **Settings** tab to:

- Get your source ID.
- Change your warehouse credentials.
- [Set up custom alerts]({{< ref "monitor/alerts.md#reverse-etl" >}}) for your Reverse ETL source.
- Delete the source permanently.

{{< warning >}}
You cannot delete a source that is connected to any destination.
{{< /warning >}}

{{< image src="images/retl-sources/source-settings.webp" alt="Edit source settings" >}}

## Sync considerations

Note the following while syncing data from your S3 source:

- RudderStack retrieves and syncs all the files present in the specified S3 bucket during the first sync.
- For subsequent syncs, RudderStack syncs only the new or updated files in the S3 bucket and **ignores** the files that have not changed since the last sync.
- RudderStack always syncs the entire file. To add or update data, create a new file in the S3 bucket to ensure that it is included in the next sync. RudderStack **does not** perform diffing at the record level, nor does it check for any invalid records.
- RudderStack **does not** retry syncs after any failures.
- If you cancel or stop a sync, RudderStack stops the operation immediately. In the next sync, it continues up from the last successfully synced record to avoid any duplication.

#### File handling

- **Updated files**: If you update an existing file present in the bucket before the next sync, RudderStack considers it as a new record and syncs accordingly. However, RudderStack **does not** track the changes made in that file - the entire record is synced again.
- **Changes to a file during sync**: If a file is updated or deleted while RudderStack is in the process of syncing it, the sync might not capture the changes. RudderStack handles the updates/deletes in the next sync.

## Create RudderStack IAM role {#creating-the-rudderstack-iam-role}

Follow the steps in this section to create a RudderStack IAM role and obtain the role ARN.

### Create policy {#creating-the-policy}

To create a managed policy defining the permissions for the RudderStack IAM role, follow these steps:

1. Sign in to your AWS Management Console and open the [IAM console](https://console.aws.amazon.com/iam/).
2. In the left navigation pane, click **Policies** followed by **Create policy**.
3. In the **JSON** tab, paste the following policy:

```json
{
  "Version": "2012-10-17",
  "Statement": [{
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": "*"
    }
  ]
}
```

4. Click **Review policy**. On the **Review** page, enter `read-write-app-bucket`.

### Create IAM role {#creating-the-iam-role}

1. In the left navigation pane, click **Roles** and go to **Create role**.
2. Under **Trusted entity type**, select **AWS account**:

{{< image src="images/destinations/aws-role-1.webp" alt="Setting up AWS IAM Role for RudderStack" >}}

3. Select **Another AWS account** and under **Account ID**, enter `422074288268`, the account ID associated with RudderStack.
4. Under **Options** check **Require external ID** and enter your [workspace ID]({{< ref "get-started/introduction/glossary.md#workspace-id" >}}) as the **External ID**.

{{< image src="images/destinations/aws-role-2.webp" alt="Setting up AWS IAM Role for RudderStack" >}}

5. Review all settings carefully and click **Next** to proceed.
6. In the **Permissions** window, select the check box next to the policy you created in the [Create policy]({{< ref "#creating-the-policy" >}}) section above.
7. Review all settings carefully and click **Next** to proceed.
8. Enter a unique name for your role. Note that this name is **case-insensitive**. For example, you cannot create a role named `RUDDERSTACK` if `rudderstack` already exists.

{{< warning >}}
You cannot edit the name of the role after it has been created.
{{< /warning >}}

9. **Optional**: Enter the description for this role.
10. Click **Create role** to complete the setup.
11. Finally, copy the **ARN** of this newly created role and paste it in the **Role ARN** field in the dashboard settings.

{{< info >}}
See the [AWS IAM tutorial](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html) for more information on delegating access across AWS account using IAM roles.
{{< /info >}}

## Troubleshooting

#### Failing syncs with large row groups

Note that the Reverse ETL syncs will fail if your files contain large row groups with sizes more than 512 MB. This is because S3 cannot process Parquet files with row groups larger than 512 MB.

Make sure that:

- The maximum record length in the input or result is 1 MB.
- The maximum uncompressed row group size is 512 MB.

See the [S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html) for more information on these limits.

## FAQ 

#### Where can I obtain the AWS Access Key ID and the AWS Secret Access Key?

1. Sign in to your [AWS Management Console](http://console.aws.amazon.com/) as the [root user](https://docs.aws.amazon.com/IAM/latest/UserGuide/console.html#root-user-sign-in-page). 
2. From the upper right corner, click your account and go to **Security Credentials**. You can find your access key ID listed here. You can also create a new access key by clicking the **Create access key** button:

{{< image src="images/warehouse-actions-sources/aws-security-credentials-new.webp" alt="AWS security" >}}

See the [AWS documentation](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html) for more information on these credentials.

{{< warning >}}
See [S3 permissions]({{< ref "#s3-permissions" >}}) for more information on the actions must be attached to your access keys required for setting up the S3 source.
{{< /warning >}}