# SFTP Reverse ETL Source


{{< announcement >}}
The SFTP Reverse ETL source is in **Private Beta**, where we work with early users and customers to test new features and get feedback before making them generally available.

Reach out to [Customer Success](mailto:support@rudderstack.com) if you are interested in enabling this feature for your workspace.
{{< /announcement >}}

[SFTP](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol) (SSH File Transfer Protocol) is a standard way for partners and internal systems to exchange files over an encrypted connection.

RudderStack supports SFTP as a data source: it reads the CSV or JSON Lines files a partner drops on a server and routes each row to your downstream destinations.

## Set up SFTP source in RudderStack

1. Log in to your [RudderStack dashboard](https://app.rudderstack.com/).
2. On the **Connections** page, click **Add source**.
3. Under **Sources**, click **Reverse ETL** and select **SFTP**.

### Connection credentials

Configure the following settings to let RudderStack connect to your server:

| Setting | <div style="width: 350px;">Description</div> |
| :--- | :--- |
| **Host** | Hostname or IP address of the SFTP server, without the port. For example, `sftp.example.com`. |
| **Port** | Port the server listens on. Defaults to `22`. |
| **Username** | User RudderStack authenticates as. |
| **Authentication** | Either **Username + SSH Key** (recommended) or **Username + Password**. |
| **Private Key** | For SSH key authentication, the full private key including its `BEGIN` and `END` lines. |
| **Private Key Passphrase** | Passphrase protecting the private key, if it has one. |
| **Password** | For password authentication, the user's password. |

{{< info >}}
Before proceeding, RudderStack connects to the server, reads the configured folder, and reports any credential or permission problems.
{{< /info >}}

### Server permissions

The user RudderStack authenticates as needs **read** access only. RudderStack never writes, renames, or deletes anything on your server.

If your SFTP server is [AWS Transfer Family](https://aws.amazon.com/aws-transfer-family/) backed by an S3 bucket, the minimum permissions on the role behind the server are:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::example-bucket",
      "Condition": {
        "StringLike": { "s3:prefix": "partner-a/exports/*" }
      }
    },
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::example-bucket/partner-a/exports/*"
    }
  ]
}
```

The prefix in this policy is the S3 key prefix, which is not the same as the **Folder Path** you enter in RudderStack. Transfer Family resolves the folder path relative to the user's home directory. 

In the above example, the home directory maps to `example-bucket/partner-a/` and the folder path is `/exports`, so the objects live under the key prefix `partner-a/exports/`.

See [HomeDirectory](https://docs.aws.amazon.com/transfer/latest/userguide/create-user.html) and [HomeDirectoryMappings](https://docs.aws.amazon.com/transfer/latest/userguide/logical-dir-mappings.html) in the AWS documentation.

{{< tip >}}
Add `s3:GetObjectVersion` if the bucket has versioning enabled, and `kms:Decrypt` on the key if objects are encrypted with SSE-KMS. Without the KMS permission, reads fail with `AccessDenied` and no mention of KMS.
{{< /tip >}}

### Name, format, and folder {#specifying-the-data-to-import}

| Setting | <div style="width: 350px;">Description</div> |
| :--- | :--- |
| **Source name** | Assign a name to uniquely identify the source in the RudderStack dashboard. |
| **File Format** | Either **CSV** or **JSON Lines**. RudderStack reads only the files matching this format and ignores the rest of the folder. |
| **Folder Path** | This is an optional setting. Specify the folder to read from, for example, `/data/exports`. Leave it empty to read the directory the user logs in to. |
| **Include subfolders** | Turn on this setting to read files in folders below the one you specified. When off, only files directly in that folder are read. |

{{< warning >}}
**Important considerations**

- CSV files must have a header row. RudderStack reads the column names from it.
- RudderStack reads whichever files match the configured **File Format** and skips any other files in the folder.
- Files compressed with gzip (`.csv.gz`, `.jsonl.gz`) are read and decompressed automatically.
{{< /warning >}}

### Review and complete setup

{{< customreadfile "/includes/retl/review-retl-setup.md" >}}

### Update source configuration and settings

Go to the **Configuration** tab of your SFTP source to update the file format, folder path, and subfolder setting.

Go to the **Settings** tab to:

- Get your source ID.
- Change your connection credentials.
- [Set up custom alerts]({{< ref "monitor/alerts.md#reverse-etl" >}}) for your Reverse ETL source.
- Delete the source permanently.

{{< warning >}}
You cannot delete a source that is connected to any destination.
{{< /warning >}}

## Sync considerations

- RudderStack reads every matching file in the folder during the first sync.
- For subsequent syncs, it reads only the files modified after the most recent file it synced successfully, and **ignores** the rest. A sync where nothing has changed reports 0 delta files.
- RudderStack always reads the entire file. To add or update data, drop a new file rather than editing an existing one. RudderStack **does not** perform diffing at the record level.
- RudderStack reads files oldest first, so a backlog drains in the order it arrived.
- RudderStack does not automatically retry a sync that fails. Resolve the underlying issue (for example, a connection or permission error) and wait for the next scheduled sync, or trigger one manually.
- If you cancel or stop a sync, RudderStack stops immediately. The next sync continues from the last file that completed successfully — rows from that file are not re-sent.

{{< warning >}}
Because RudderStack decides what to read from each file's modification time, a partner that uploads files while **preserving the original timestamps** — for example with `sftp -p` or `rsync -a` — can drop a file that appears older than files already synced, and RudderStack does not read it. 

Ask partners to upload without preserving timestamps, or touch the files after upload to update their modification time.
{{< /warning >}}

#### File handling

- **Updated files**: Editing a file in place updates its modification time, so RudderStack reads it again in full on the next sync and re-sends every row in it.
- **Changes to a file during sync**: If a file is updated or deleted while RudderStack is in the process of syncing it, the sync might not capture the changes. RudderStack handles them in the next sync.

## Troubleshooting


| Issue | <div style="width: 350px;">Solution</div> |
| :--- | :--- |
| The sync reports no files | Check the following: <br /><br /><ul><li>The files in the folder match the configured **File Format**.</li><li>The **Folder Path** is correct.</li><li>**Include subfolders** is on if your files are nested.</li><li>The user RudderStack authenticates as can list that folder.</li></ul> |
| A file is uploaded but never synced | The file's modification time is probably older than the files already synced. See the warning under [Sync considerations]({{< ref "#sync-considerations" >}}). |

## FAQ

#### Which file formats are supported?

CSV and JSON Lines, optionally gzip-compressed. Parquet is not supported by this source. 

Note that:

- If your data is in Parquet, see the [Amazon S3 Source]({{< ref "sources/reverse-etl/amazon-s3.md" >}}) guide for details. 
- New S3 source setups require existing S3 Select access due to [AWS restrictions](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/). 

Contact [RudderStack Support](mailto:support@rudderstack.com) for alternative options.

#### Does the SFTP source need a primary key?

No. Primary keys apply to warehouse tables. A file-based source syncs whole files, so RudderStack does not ask for one.