Transformations

Write functions to transform your source data before sending it to your destinations.

qoute

Transformations Challenge 🏆 💰

Write a transformation and win up to $1,000 cash and more. Join the Transformations Challenge

RudderStack’s Transformations feature lets you write custom functions to implement specific use-cases on your event data, like:

  • Filtering or sampling events
  • Cleaning or aggregating data
  • Data masking or removing sensitive PII to ensure data privacy
  • Enriching events by implementing static logic or leveraging an external API
  • Using an API to implement specific actions on the events
success
See the Transformation Templates guide for some useful prebuilt templates that you can use to create your transformations.
info
The RudderStack Cloud Free and Starter plans let you create 5 JavaScript transformations. However, you can create unlimited transformations in the Growth and Enterprise plans. Refer to the Pricing page for more information.

Key features

  • Easier to build, manage, debug, and reuse.
  • Add custom logic to your source events in real-time, before sending them to your destination.
  • Supports JavaScript and Python.
  • Use transformations across your Event Streams, Cloud Extract, and Reverse ETL pipelines in cloud mode.
  • You can version control your transformations.
  • Create an organization-wide sandbox where your team can store all transformations before publishing them in production.
  • Programmatically manage your transformations using the Transformations API.

Use case

Suppose you want to set the context.os field to Android for all events, irrespective of the actual platform RudderStack tracks the event from. To do this, you can write a simple transformation as shown:

The transformEvent function overrides the event’s context.os.name and sets it as Android, as seen below:

Transformation use-case

Python transformations

info
Python transformations are available only in the RudderStack Cloud Growth and Enterprise plans.

RudderStack now supports writing your transformations in Python, giving you full flexibility to use custom Python code to transform your source events on the fly. This feature is especially useful for the data teams that generally deal with Python.

Limitations

RudderStack supports only some of the built-in Python packages to write your transformations. These are datetime, json, math, random, requests, time, and urllib, along with the external package python-dateutil.

Adding a transformation

  1. Log in to the RudderStack dashboard.
  2. Go to Enhance > Transformations and click New Transformation.
  3. Name your transformation and add an optional description.
  4. In the Transformation window, select the language to write your transformation.
info
RudderStack supports Python transformations only in the Growth and Enterprise plans.
Adding a Transformation
  1. Add your transformation function:
Adding a Transformation
info
You can also add other functions and call them from within the transformEvent function.
  1. To test your transformation, click the Run Test button. By default, RudderStack provides a sample event payload to test if your transformation logic works as expected.
Adding a Transformation
  1. Finally, click Save to save the transformation.

Connecting transformation to a destination

You can connect a transformation to a destination in two cases:

Case 1: Setting up a new destination

You can connect an existing transformation or create a new transformation from scratch while setting up a destination:

Connecting a transformation

Case 2: Connecting to an existing destination

  1. In the dashboard, go to the Transformation tab and click Add a transformation:
Connecting a transformation to existing destination
  1. Select the transformation to connect to the destination and click Choose.
Connecting a transformation to existing destination

Deleting a transformation

warning
You cannot delete a transformation that is connected to a destination.

To delete a transformation, go to Enhance > Transformations and click the Delete button next to the transformation that you want to delete.

Transform event function

While using a transformation, RudderStack applies the transformEvent function on each event that takes the following two arguments:

ArgumentDescription
eventThe input event.
metadataThe JavaScript function to access the event’s metadata. Refer to the Accessing event metadata section below for more information.

After the transformation is complete, the transformEvent function returns the final event to be sent to the destination.

Access event metadata

RudderStack injects a function metadata(event) into your transformations as an argument. This lets you access the event metadata variables to customize your transformations.

info
metadata() takes the event as the input and returns the metadata of the event.

The following properties, if available, are present in the metadata response:

Property
Description
sourceIdThe source ID in the Settings tab of your configured source in the dashboard.
destinationIdThe destination ID in the Settings tab of your configured destination in the dashboard.
messageIdThe unique ID for each event.
sourceTypeThe source type, for example, Android, iOS, etc.
destinationTypeThe destination type where RudderStack sends the transformed event, for example, Snowflake.

Example transformations using the metadata data method are shown below:

Make external API requests

You can make any number of external API requests in your transformation functions and use the fetched responses to enrich your events.

JavaScript

RudderStack injects an asynchronous fetch function in your transformations. It makes an API call to the given URL and returns the response in the JSON format.

You can use the fetch function in your transformations:

export async function transformEvent(event, metadata) {
  const res = await fetch("post_url", {
    method: "POST",  // POST, PUT, DELETE, GET, etc.
    headers: {
      "Content-Type": "application/json;charset=UTF-8",
      Authorization: "Bearer <authorization_token>"
    },
    body: JSON.stringify(event)
  });
  event.response = JSON.stringify(res);
  return event;
}

To see the fetch function in action, refer to the Clearbit enrichment example.

info
For improved performance, it is highly recommended to use the batch API requests wherever possible instead of a separate API request for each event.

Fetch v2

fetchv2 is a wrapper for the fetch call. It enables you to fetch the response properties more efficiently while making the external API calls.

The fetchv2 response contains the following properties:

PropertyDescription
statusStatus code of fetch response, for example, 200.
urlThe URL of the Fetch API.
headersThe response headers
bodyThe response body in JSON or TEXT. By default, it is JSON.

The following example highlights the use of the fetchV2 function in a transformation to capture failure due to a timeout:

export async function transformEvent(event) {
  try {
    const res = await fetchV2("url", { timeout: 1000});
    if (res.status == 200) {
      event.response = JSON.stringify(res.body);
    }
  } catch (err) {
    log(err.message);
  }
  return event;
}

Python

You can use Python’s requests package to fetch response properties while making the external API calls:

import requests

def transformEvent(event, metadata):
    res = requests.get("url")
    if res.status_code == 200:
        event["response"] = res.json();
    return event

FAQ

See the FAQ guide for answers to the commonly asked questions on this feature.


Questions? Contact us by email or on Slack