Amazon S3 is a popular object storage service used to store both structured and unstructured data. With an S3-powered data lake, you can easily use the native AWS services for data processing, analytics, machine learning, and more.

RudderStack lets you configure S3 data lake as a destination to which you can send your event data seamlessly.

For more information on how the events are mapped to the tables in S3 data lake tables, refer to the Warehouse Schema guide .
Find the open source transformer code for this destination in the GitHub repository.

S3 permissions for data lake destination

To successfully send data to your S3 data lake, you need to set the following permissions in your S3 policy:

"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:ListBucket"
]
For detailed steps on creating the IAM user credentials from scratch, refer to the Amazon S3 permissions section.

Configuring S3 Data Lake destination in RudderStack

To set up S3 data lake as a destination in RudderStack, follow these steps:

  1. In your RudderStack dashboard, set up the data source. Then, select S3 Data Lake from the list of destinations.
  2. Assign a name to your destination and then click Continue.

Connection settings

Enter the following settings in the Connection Credentials page:

S3 data lake destination settings in RudderStack
  • S3 Storage Bucket Name: Enter the name of the S3 bucket used to store the data before loading it into the S3 data lake.
  • Register schema on AWS Glue: Enable this option to register the schema of your incoming data on AWS Glue's data catalog.
For more information on registering your schema in AWS Glue, refer to the AWS Glue documentation.
If AWS Glue is enabled, make sure you grant the following permissions to it:
  • glue:CreateTable
  • glue:UpdateTable
  • glue:CreateDatabase
  • glue:GetTables
  • AWS Glue Region: Enter your AWS Glue region. For example, for N.Virginia, it would be us-east-1.
For more information on getting your AWS Glue region and the associated service endpoints, refer to the AWS Glue documentation.
  • S3 Prefix: If specified, RudderStack creates a folder in the S3 bucket with this prefix and pushes all the data within that folder.
  • Namespace: If specified, all the data for the destination will be pushed to the location s3://<bucketName>/<prefix>/rudder-datalake/<namespace>.
If AWS Glue is enabled, all the table definitions are created in a database with the name set to this namespace.
If you don't specify a namespace in the settings, it is set to the source name, by default.
  • AWS Access Key ID: Enter your AWS access key ID.
  • AWS Secret Access Key: Enter AWS secret access key.
For more information on obtaining the AWS access key ID and secret access key, refer to the S3 Permissions section.
Make sure the credentials (Access Key ID and Secret Access Key) have the permissions to read and write into the configured S3 bucket.

Finding your data in S3 data lake

RudderStack converts your events into Apache Parquet files and stores them into the configured S3 bucket. Before storing the events, RudderStack groups them by the event name based on the UTC time they were received.

The folder structure is shown below:

s3://<bucketName>/<prefix>/rudder-datalake/<namespace>/<tableName>/YYYY/MM/DD/HH

As mentioned in the Connection settings section:

  • prefix: This is the S3 prefix in the destination settings. If not specified, RudderStack will omit this value.
  • namespace: The namespace specified in the destination settings. If not specified, RudderStack sets this field to the source name by default.
  • tableName: RudderStack sets this to the event name.
  • YYYY, MM, DD, and HH are replaced by actual time values. A combination of these values represents the UTC time.

For example, suppose RudderStack tracks the following two events:

Event nameTimestamp
Product Purchased"2019-10-12T08:40:50.52Z" UTC
Cart Viewed"2019-11-12T09:34:50.52Z" UTC

RudderStack will convert these events into Parquet files and dump them into the following folders:

Event NameFolder Location
Product Purchaseds3://<bucketName>/<prefix>/rudder-datalake/<namespace>/product_purchased/2019/10/12/08
Cart Vieweds3://<bucketName>/<prefix>/rudder-datalake/<namespace>/cart_viewed/2019/11/12/09
If AWS Glue is enabled, all the table definitions are created in a database with the name set to the namespace specified in the destination settings.

Creating a crawler

Refer to this section only if you haven't enabled the Register Schema on AWS Glue setting while configuring the S3 data lake destination in RudderStack.

In the absence of AWS Glue, you can create a crawler to go through your data and create table definitions out of it. Follow these steps:

  1. Go to your AWS Glue console and select Crawler from the left pane.
  2. Select Add Crawler.
  3. Specify a name for your crawler and click Next, as shown:
Add Crawler
  1. Next, under the Crawler source type section, choose Data stores.
Crawler source type
  1. Configure the Repeat crawls of S3 data stores based on your requirement.
  2. Then, under the Data store section, select S3 from the dropdown for the Choose a data store setting:
Choose a data store
  1. For the Crawl data in setting, choose Specified path in my account.
  2. In the Include path setting, enter the S3 URI of your configured bucket followed by the suffix /<prefix>/rudder-datalake/<namespace>/.
If your S3 bucket name is testBucket and the configured prefix and namespace are testPrefix and testNameSpace respectively, then your path should be:s3://testBucket/testPrefix/rudder-datalake/testNameSpace/
If you have not configured any prefix while setting up the S3 data lake destination in RudderStack, omit the prefix. The path would then be:s3://testBucket/rudder-datalake/testNameSpace/.
  1. Then, under the Add another data store setting, select No, as shown:
Add another data store
  1. In the IAM Role section, configure a suitable IAM role.
IAM Role
  1. In the Schedule section, select the frequency of your crawler from the dropdown options, as shown:
Scheduler
  1. In the Output section, configure the database that stores all the tables. Under the Grouping behavior for S3 data section, enable the Create a single schema for each S3 path option:
Output
  1. Specify the Table level as 5 or 4 (refer to the tips below). This value indicates the absolute level of the table location in the bucket.
The level for the top-level folder is 1. For example, for the path mydataset/a/b, if the level is set to 3, the table will be created at the location mydataset/a/b. Similarly, if the level is set to 2, the table will be created at the location mydataset/a.
Since all tables are created in the path s3://testBucket/<prefix>/rudder-datalake/<namespace>/, make sure the table level is set to:
  • 5: If a prefix is configured.
  • 4: If a prefix is not configured.
  1. Review your crawler configuration and click on Finish to confirm.
Review crawler configuration
  1. Finally, click on your crawler and run it. Wait for the process to finish - you should see some tables created in your configured database.

Querying data using AWS Athena

You can query your S3 data using a tool like AWS Athena which lets you run SQL queries on S3.

Before querying your data on S3, make sure that you have sent some data to S3 and that the sync is completed.

Follow these steps to start querying your data on s3:

  1. Open your AWS Athena console. Then, go to the same AWS region which was used while configuring AWS Glue.
  2. In the left pane, select AwsDataCatalog as your data source, as shown:
AwsDataCatalog
  1. Select your configured namespace (or the database you specified while configuring the crawler) from the database dropdown menu.
Database
By default, RudderStack sets the namespace to your source name if it is not explicitly specified in the destination settings.
  1. You should see some tables already created under the Tables section in the left pane.
  2. You can preview the data by clicking on the three dots next to the table and selecting the Preview Data option. Alternatively, you can run your own SQL queries in the workspace on the right, as shown:
Preview Data

IPs to be allowlisted

To enable network access to RudderStack, you will need to allowlist the following RudderStack IPs:

  • 3.216.35.97
  • 34.198.90.241
  • 54.147.40.62
  • 23.20.96.9
  • 18.214.35.254
  • 35.83.226.133
  • 52.41.61.208
  • 44.227.140.138
  • 54.245.141.180
  • 3.66.99.198
  • 3.64.201.167
If you have your deployment in the EU region, you can allowlist only the following two IPs:
  • 3.66.99.198
  • 3.64.201.167
All the outbound traffic is routed through these RudderStack IPs.

Contact us

For more information on the topics covered on this page, email us or start a conversation in our Slack community.

Contents