RudderStack's Transformations feature enables you to write custom JavaScript functions to implement specific use-cases on your event data, such as:

  • Filtering or sampling events
  • Cleaning or aggregating data
  • Data masking or removing sensitive PII to ensure data privacy
  • Enriching events by implementing static logic or leveraging an external API
  • Using an API to implement specific actions on the events

You can use this feature across your Cloud Apps, Reverse ETL, and Cloud Extract pipelines.

Refer to our Sample RudderStack Transformations repository for some useful templates that you can use to create your own transformations.
You can use transformations only when sending events to destinations via cloud mode.

Key features

Some key features of RudderStack transformations are listed below:

  • You can apply the transformations to your events in real-time.
  • You can programmatically add and remove transformations using the Transformations API.
  • They're easier to build, manage, and debug, and reuse.
  • You can create an organization-wide sandbox where your team can store the transformations before publishing them in a production environment.
  • You can version control your transformations.
Read more about why your data pipeline needs to have an efficient event transformations capability in the RudderStack blog.

How do transformations work?

At a high level, once you add a transformation and connect it to a destination, RudderStack:

  1. Tracks and collects events at the source
  2. Applies the transformation logic to the your events
  3. Forwards the transformed event to your connected destination
Transformations workflow
Refer to the Transformations FAQ for commonly asked questions on this feature.

Use case

Suppose you want to set the context.os field to Android for all the events, irrespective of the actual platform RudderStack tracks the event from. You can write a simple transformation to do this:

export function transformEvent(event, metadata) {
event.context.os = { name: "Android"};
return event;
}

The transformEvent function overrides the event's context.os.name and sets it as Android, as seen below:

Transformation use-case

Adding a transformation

To add a new transformation in the RudderStack dashboard, follow these steps:

  1. Log into the RudderStack dashboard.
  2. Go to Enhance > Transformations, as shown:
Adding a Transformation
  1. Click New Transformation.
  2. Add a name for your transformation and an optional description.
  3. Next, add your JavaScript function in the Transformation window, as shown:
Adding a Transformation
You can also add other functions and call them from within transformEvent.
  1. To test your transformation, paste your sample event in the Events block and click Run Test. By default, RudderStack provides some sample events to test if your transformation logic works as expected.

    Adding a Transformation
  2. To save the transformation, click Save.

Debugging with logs

RudderStack enables you to capture any event-related information in the form of logs while running a test on your transformation. You can do this by including the log function in your transformation code.

An example of using the log function is shown below:

export function transformEvent(event, metadata) {
const meta = metadata(event);
event.sourceId = meta.sourceId;
log("Event Name is", event.event, ";", "Message ID is", event.messageId);
log("Source ID is", meta.sourceId);
return event;
}

On adding the above transformation and clicking Run Test, you can see the resulting log in the Logs section of the dashboard, as shown:

Transformation log
You can pass a string, number, or an object as an argument to the log function.

Connecting transformation to a destination

You can connect a transformation to a destination in two cases:

Case 1: While setting up a new destination

RudderStack provides the option to connect an existing transformation or create a new transformation while setting up a destination, as shown:

Connecting a transformation

Case 2: While connecting to an existing destination

To add a transformation to an existing destination, follow these steps:

  1. In the dashboard, go to the Transformation tab in your destination dashboard and click Add a transformation, as shown:
Connecting a transformation to existing destination
  1. Then, choose the transformation to connect to the destination.
Connecting a transformation to existing destination

Deleting a transformation

To delete a transformation, go to Enhance > Transformations and click the Delete button next to the transformation that you want to delete, as shown:

Deleting a transformation
You cannot delete a transformation that is connected to a destination.

transformEvent function

While using a transformation, RudderStack applies the transformEvent function on each event that takes two arguments:

  • event: Corresponds to the input event.
  • metadata (optional): Corresponds to the JavaScript function which you can use to access the metadata of the given event.
For more information on metadata, refer to the Accessing metadata section below.

After the transformation is complete, transformEvent returns the final event to be sent to the destination.

Accessing event metadata

RudderStack injects a function metadata(event) into your transformations as an argument. This allows you to access the event metadata variables that help you customize your transformations.

metadata() takes the event as the input and returns the metadata of the event.
Since you may not need the event metadata in every transformation, it is an optional argument and can be skipped.

The following properties, if available, are present in the metadata response:

PropertyDescription
sourceIdThe source ID in the Settings tab of your configured source in the dashboard.
destinationIdThe destination ID in the Settings tab of your configured destination in the dashboard.
messageIdThe unique ID for each event.

An example of using metadata is shown below:

export function transformEvent(event, metadata) {
const meta = metadata(event);
event.sourceId = meta.sourceId;
return event;
}

Applying transformation on a batch of events

You can also perform any aggregation or roll-up operation on a batch of events using the transformBatch function instead of transformEvent, as shown:

export function transformBatch(events, metadata) {
return events;
}
If you want to ensure event ordering while using transformBatch, make sure you pass the messageId from the input event to the output event. Without the messageId, RudderStack will not guarantee the event ordering. It is highly recommended to use transformEvent as much as possible, as it ensures event ordering.

Libraries

One of the key features of RudderStack transformations is the ability to reuse the transformation code in other transformations using the Libraries feature.

Adding a library

To add a new library, follow these steps:

  1. In the RudderStack dashboard, go to Enhance > Transformations and click the Libraries tab, as shown:
Adding a library
  1. Click New library.
  2. Add the library Name, Description and write the function you want to reuse across different transformations. You can also add multiple functions in a single library, as shown:
Adding a library
  1. Click Run Test to ensure the library code has the correct syntax.

Using libraries in transformations

To use the libraries in your existing transformations, refer to the Import Library Name option in the RudderStack dashboard, as shown:

Using libraries in transformations
RudderStack converts the library name into camel case without spaces; this becomes your library handle which you can use across multiple transformations. For example, if your library name is Sample Transformation Library, then the library handle would be sampleTransformationLibrary.

You can then use the library in a transformation with a simple import statement. Refer to the below use case for more information.

Use case

Suppose you want to import a function rudderEmail from the Is Rudder Email library to filter events that don't have the email address containing the RudderStack domain.

The rudderEmail function is as follows:

export function rudderEmail(email) {
return /@(rudderlabs|rudderstack)| \+ruddertest/.test(email);
}

The following code snippet demonstrates how you can implement this in a transformation:

import { rudderEmail } from "isRudderEmail";
export function transformEvent(event) {
const email =
event.context && event.context.traits && event.context.traits.email;
if (email) {
if (!rudderEmail(email)) return;
}
return event;
}

On clicking Run Test, a sample event not containing the RudderStack email domain is filtered out, as shown:

Filtering out events using libraries in transformation

Importing multiple functions from a single library

The following snippets highlight how to properly import functions from a library:

// ---------------
import { getLoss } from "getFinanceData";
// OR
import { getLoss, getRevenue, getProfit } from "getFinanceData";
import {
getLoss,
getRevenue,
getProfit
} from "getFinanceData";
// For default getPrice import
import getPrice, { getRevenue, getProfit } from "getFinanceData";
// alias imports
import getPrice as gp, { getRevenue as gr, getProfit } from "getFinanceData";
// usage: gp(event), gr(event), getProfit(ev)
import * as GFD from "getFinanceData";
// usage: GFD.getRevenue(ev), GFD.getProfit(ev)
// for default import: GFD.default(ev)

The following snippets highlight the incorrect way of importing the library functions:

// -----------------
import * from "getFinanceData";
getPrice(ev)
// OR
import getPrice as gp from "getFinanceData";
getPrice(ev)

Deleting a library

To delete a transformation, go to Enhance > Transformations and click the Libraries tab. button next to the transformation that you want to delete, as shown:

Deleting a library
You cannot delete a referenced library.

Making external API requests

You can make any number of external API requests in your transformation functions and use the response to enrich your events.

RudderStack injects an asynchronous fetch function in your transformations. It makes an API call to the given URL and returns the response in the JSON format.

You can use the fetch function in your transformations, as shown:

export async function transformEvent(event, metadata) {
const res = await fetch("post_url", {
method: "POST", // POST, PUT, DELETE, GET, etc.
headers: {
"Content-Type": "application/json;charset=UTF-8",
Authorization: "Bearer <authorization_token>"
},
body: JSON.stringify(event)
});
event.response = JSON.stringify(res);
return event;
}
To see the fetch function in action, refer to the Clearbit enrichment sample transformation.
For improved performance, it is highly recommended to use the batch API requests instead of a separate API request for each event wherever possible.

Fetching response properties using fetchV2

FetchV2 is a wrapper for the fetch call. It enables you to fetch the response properties more efficiently while making the external API calls.

The following properties are present in a fetchV2 response:

PropertyDescription
statusStatus code of fetch response, for example, 200.
urlThe URL of the Fetch API.
headersThe response headers
bodyThe response body in JSON or TEXT. By default, it is JSON.

The below example highlights the use of the fetchV2 function in a transformation to capture failure due to a timeout:

export async function transformEvent(event) {
try {
const res = await fetchV2("url", { timeout: 1000});
if (res.status == 200) {
event.response = JSON.stringify(res.body);
}
} catch (err) {
log(err.message);
}
return event;
}

Limits

You should take into account the following memory and time limits when invoking a transformation:

ParameterLimit
Memory limit128 MB
Execution time limit4 seconds
The user transformation fails if these limits are exceeded.

FAQ

For answers to some of the commonly asked questions on this feature, refer to the FAQ guide.


Contact us

For more information on the topics covered on this page, email us or start a conversation in our Slack community.