Schedule and trigger reverse ETL sources with our Airflow provider.
RudderStack’s Airflow Provider lets you programmatically schedule and trigger your Reverse ETL syncs from outside RudderStack and integrate them with your existing Airflow workflows.
For more information on the codebase and sample implementation, refer to our GitHub Repository.
Using the Airflow Provider
To use the Airflow Provider, you must have a working Apache Airflow installation. For more information, refer to the Airflow documentation.
Follow the steps in the below sections to use the RudderStack Airflow Provider:
Initialize all dependencies by running Apache Airflow via the following command:
The Airflow standalone server is not meant for use in production. It is highly recommended using alternate methods to install and run Airflow in a production environment.
Installing Airflow Provider
Install the RudderStack Airflow Provider by running the following command:
pip install rudderstack-airflow-provider
Creating an Airflow connection
To create a new Airflow connection, follow these steps:
In your Airflow dashboard, go to Admin > Connections:
Add a new connection by configuring the following details:
Connection ID: Specify a unique connection name. RudderstackOperator will pick the connection with the name rudderstack_default by default. If you have created a connection with a different name, make sure that name is passed as a parameter to RudderstackOperator.
Connection Type: For this field, select HTTP from the list.
Host: Set the value for this field to https://api.rudderstack.com.
cookies, the cookies that are categorized as necessary are stored on your browser as they are as
for the working of basic functionalities of the website. We also use third-party cookies that
analyze and understand how you use this website. These cookies will be stored in your browser
consent. You also have the option to opt-out of these cookies. But opting out of some of these
have an effect on your browsing experience.
Necessary cookies are absolutely essential for the website to function properly. This
category only includes cookies that ensures basic functionalities and security
features of the website. These cookies do not store any personal information.
learn more about cookies and why we use them, visit our cookie
policy. We'll assume you're ok with this, but you can opt-out if you wish Cookie Settings.