RudderStack Docs – Concepts

Archive: Entities

Mon, 01 Jan 0001 00:00:00 +0000

This guide introduces you to the concept of entities in Profiles and shows how to define entities in your Profiles project.

Overview

Every business has artifacts that are tracked across systems for creating a complete data picture of that artifact. In Profiles, these are called Entities.

Entities can be as common as users, accounts, etc. It can also expand to be anything that you can track across systems and want a complete picture of, like campaigns, devices, or even month of the year.

Entities are the central concept that a Profiles project is built around. It is where you define what your identity graphs and C360 tables are built around.

SQL Keyword: GROUP BY

Requirements

An entity must have multiple IDs in the data sources, also called as inputs.
Each input must have at least one ID that is shared with another input.
No ID can be shared across any member of an entity (that is, an ID cannot shared by multiple users).

Usage

You can useentities in the pb_project.yaml file present in the top level of every Profiles project, as shown:

name: profiles_project
schema_version: 85
connection: wh_connection_dev
model_folders:
 - models
##########################################
entities:
 - name: user
 id_stitcher: models/user_id_stitcher
 id_types:
 - user_id
 - anonymous_id
 feature_views: # Optional
 using_ids:
 - id: anonymous_id
 name: anonymous_id_360
##########################################
id_types:
 - name: user_id
 - name: anonymous_id
 filters: # Optional
 - type: exclude
 value: ""
python_requirements: # Optional
 - profiles_mlcorelib==0.7.2

Best practices

As entities are the central building block of a Profiles project, you should define them before building the rest of the project.
Most common entities include user, account, organization, and prospect. But you can define an entity for anything that you want to group and aggregate your data around, including marketing campaigns, business units, or even months of the year (for monthly reporting).
When picking id_types, consider the granularity of the entity. At the user grain, you will want to pick unique id_types of the same grain. For higher level grains like organization or account, you can include user level grain id_types as well as org level id_types as long each user only belongs to one org or account.

Archive: IDs

Mon, 01 Jan 0001 00:00:00 +0000

This guide introduces you to the concept of IDs in Profiles and shows how to define id_types and ids in your Profiles project.

Overview

Creating a Customer360 (C360) is primarily combining data from different sources, or inputs. ID fields define how those tables are connected.

You can create the features of a C360 by chaining these IDs from different inputs together. Note that IDs are not always consistent between inputs in name or format.

Profiles has two concepts related to IDs:

id: This is the field within inputs that is an identifier for a given entity. For example a user name, an email, or an anonymous ID.
id_type: These are the categories of an ID. ID fields that have the same id_type mean those inputs can be joined on those id.

SQL Keyword: JOIN ON

Requirements

ID must be unique for the members of an entity.
ID must exist in multiple inputs.
IDs must be strings.

Usage

You can define id_types in the pb_project.yaml file present in the top level of every Profiles project, as shown:

name: llm_sdr_email_content_generation
schema_version: 85
connection: llm-recommend-dev
model_folders:
 - models
entities:
 - name: user
 id_stitcher: models/user_id_stitcher
############### ID Types #################
 id_types:
 - user_id
 - anonymous_id
##########################################
 feature_views: # Optional
 using_ids:
 - id: anonymous_id
 name: anonymous_id_360

############## ID Type Definitions #######
id_types:
 - name: user_id
 - name: anonymous_id
 filters: # Optional
 - type: exclude
 value: ""
##########################################
python_requirements: # Optional
 - profiles_mlcorelib==0.7.2

You can then define ids with the ID types in the inputs.yaml file. Each inputs ID is labeled with one of the id_type defined in pb_project.yaml.

inputs:
 - name: rsIdentifies
 app_defaults:
 table: rudder_autotrack_data.autotrack.identifies
 occurred_at_col: timestamp
########## Map ID to ID Type #############
 ids:
 - select: "anonymous_id"
 type: anonymous_id
 entity: user
 - select: "CAST(user_id AS VARCHAR)" # If data type of user_id column is not a string, you need to cast it to VARCHAR, STRING, or TEXT type depending on your warehouse syntax.
 type: user_id
 entity: user
 - select: "lower(email)"
 type: email
 entity: user
##########################################

Best practices

You can define id_types for the id_graph or to connect additional tables for features.
id_types should only be identifiers that are relevant to your entities and the features you are going to create. For example, if you are looking at customer support tickets, you do not need to include IDs related to your Customer Success systems.
To use email address as an id_type, remove the test and internal domains.
Ensure the id_types you choose are unique. For example, first_name, last_name, or cat(first_name, last_name) would not reasonably be expected to be unique across all users, making them an unsuitable identifier for a user entity. But user_id or emailcould be depending on your product.
When picking id_types, consider the granularity of the entity. At the user grain, you will want to pick unique id_types of the same grain. For higher level grains like organization or account, you can include user level grain id_types as well as org level id_types as long each user only belongs to one org or account.
If any of your ID columns are not of the string data type, then you must cast the columns to string in the select statement.

Archive: Identity Graph

Mon, 01 Jan 0001 00:00:00 +0000

This guide introduces you to the concept of identity graphs in Profiles and shows how to use them in your Profiles project.

Overview

Multiple identities is the biggest obstacle to getting a full picture of your customers, user, or any other entity.

Companies gather user data across digital touchpoints like websites, mobile apps, enterprise systems like CRMs, marketing platforms, etc. During this process, a single user is identified with multiple identifiers across your systems, like email address, phone number, device ID, anonymous ID, user name, etc. To create a unified user profile, it is vital to collect and stitch all of those different identifiers into one canonical identifier.

You can leverage Profiles’ id_stitcher model to take your inputs and stitch all the identifiers together into an identity graph.

An identity graph consists of two main components:

Nodes: Represent individual identifiers that belong to an entity.
Edges: Represent the relationships observed between these identifiers in your input data.

Requirements

Defined id_types for your entity and mapped to your input sources.
Identifiers from your inputs must be of the string data type. If the columns in your input sources are not of string types already, you must cast them explicitly.

Usage

The identity graph is located in the profiles.yaml file as an id_stitcher model.

models:
 - name: user_id_stitcher
 model_type: id_stitcher
 model_spec:
 entity_key: user
 edge_sources:
 - from: inputs/rsIdentifies
 - from: inputs/rsPages

Link the identity graph to your entity

After you define an id_stitcher model in profiles.yaml, you must reference it from the corresponding entity in your pb_project.yaml file. Set the id_stitcher key on the entity to the model path (for example, models/user_id_stitcher for a model named user_id_stitcher).

Always set id_stitcher on each entity that has a custom ID stitcher model defined in profiles.yaml.

If the entity does not specify id_stitcher, the intended id stitcher is not used and you may get unintended results.

Example entity configuration in pb_project.yaml:

entities:
 - name: user
 id_stitcher: models/user_id_stitcher
 id_types:
 - user_id
 - anonymous_id

... # Rest of the configuration

See Entity for full entity configuration options.

Metadata and observability

To provide transparency into the stitching process, you can enable metadata logging using the following flags in your id_stitcher model specification. These flags control what metadata is logged for the identity graph components:

Flag	Description
`log_direct_edge_info`	Logs metadata about the direct relationships (edges) between identifiers.
`log_node_metadata`	Logs metadata for each individual identifier (node).

Both flags are supported on Snowflake and BigQuery. Other supported warehouses do not currently emit the METADATA column.

An example of how to enable these flags is shown below:

models:
 - name: user_id_stitcher
 model_type: id_stitcher
 model_spec:
 entity_key: user
 log_direct_edge_info: true
 log_node_metadata: true
 edge_sources:
 - from: inputs/rsIdentifies
 - from: inputs/rsPages

This metadata helps you understand how identifiers are connected and trace the lineage of your identity graph.

Metadata structure

The generated metadata contains two main objects:

Object	Details
`node_info`	Array of objects containing aggregated metadata for the node (identifier) itself. It includes the `source_model` where the identifier was first seen and the `first_seen_at` timestamp.
`direct_edge_info`	Object describing the direct relationships (edges) of a node. The `edges_by_type` field is an array of buckets — one bucket per identifier type. Each bucket contains a `count` of edges and an `edges` array of the connected nodes, with the per-source provenance of each edge.

The following example shows one node with a single email edge from one input model:

{
 "direct_edge_info": {
 "edges_by_type": [
 {
 "count": 1,
 "edges": [
 {
 "first_seen_at": "2022-01-01 10:00:00.000",
 "id": "s1_email@test.com",
 "sources": [
 {
 "first_seen_at": "2022-01-01 10:00:00.000",
 "source_model": "events_app"
 }
 ]
 }
 ],
 "type": "email"
 }
 ]
 },
 "node_info": [
 {
 "first_seen_at": "2022-01-01 10:00:00.000",
 "source_model": [
 "events_app"
 ]
 }
 ]
}

When the same edge is observed in more than one input model, each contributing model appears as a separate entry in the edge’s sources array. The edge-level first_seen_at is the earliest timestamp across all sources:

{
 "direct_edge_info": {
 "edges_by_type": [
 {
 "count": 1,
 "edges": [
 {
 "first_seen_at": "2022-02-10 08:00:00.000",
 "id": "ms_email@test.com",
 "sources": [
 {
 "first_seen_at": "2022-02-15 10:00:00.000",
 "source_model": "events_app"
 },
 {
 "first_seen_at": "2022-02-10 08:00:00.000",
 "source_model": "events_web"
 }
 ]
 }
 ],
 "type": "email"
 }
 ]
 },
 "node_info": [
 {
 "first_seen_at": "2022-02-15 10:00:00.000",
 "source_model": ["events_app"]
 },
 {
 "first_seen_at": "2022-02-10 08:00:00.000",
 "source_model": ["events_web"]
 }
 ]
}

Incremental runs do not merge metadata across runs

On each incremental run, the per-node METADATA column reflects only the source attribution observed during the current run’s data window. If a node was contributed by events_app in a previous run and only by events_mobile in the current run, the metadata column shows events_mobile alone — the previous run’s source list is not carried forward.

This applies to both node_info[].source_model and direct_edge_info.edges_by_type[].edges[].sources[]. The stitched ID graph itself (the id and other_id columns) does carry forward across incremental runs as expected; only the metadata column resets per run.

Reading the metadata column

The METADATA column is a per-row JSON document attached to every row of the ID stitcher output. Each row describes one identifier (other_id) and how it was observed during the current run.

The top-level structure is fixed:

node_info is a deduplicated array, one entry per input model that contributed the node. Each entry carries the first_seen_at timestamp for that model and a one-element source_model array.
direct_edge_info.edges_by_type is an array of buckets. Each bucket has a type (the ID type of the connected nodes), a count, and an edges array.
Each edge in edges has the connected node’s id, the edge-level first_seen_at, and a sources array. The sources array carries one entry per contributing input model with that model’s own first_seen_at.

Multi-source edges

The same edge can appear in multiple inputs. The number of entries in an edge’s sources array shows how many inputs reported the edge in this run — the source_model and first_seen_at per entry trace the edge back to a specific table and time.

A single entry in sources means only one input model contributed the edge in this run — not that the edge is weaker.

Incremental run behavior

Two incremental runs of the same project can produce different sources arrays for the same edge, depending on which inputs landed new rows in each run.

When you compare metadata across runs, treat the sources array as a per-run snapshot, not as lifetime history. To preserve cross-run source history, snapshot the column into a downstream table on each run.

For more information on how violator edges interact with the metadata column under cardinality rules, see ID Graph Cardinality Rules.

Metadata size and limits

Each row’s METADATA JSON is bounded by structure, not by data scale. Two internal caps keep each row’s JSON small even for highly-connected nodes:

Field	Limit	What it means
`direct_edge_info.edges_by_type` length	10 buckets	At most 10 distinct identifier types per node appear in the metadata.
`edges_by_type[].edges` length	10 edges per bucket	At most 10 direct neighbors of a given type are listed per node.
`edges[].sources` length	Less than or equal to the number of `edge_sources:` in your project YAML	One entry per contributing input model.
`node_info` length	Less than or equal to the number of `edge_sources:` in your project YAML	One entry per contributing input model.

The two hardcoded limits are not currently configurable from YAML. In practice, the defaults are large enough for almost all production identity graphs. If you lose meaningful detail, consider whether ID Graph Cardinality Rules on the entity can tighten the underlying graph rather than expanding the metadata caps.

Detecting edges hidden by the cap

Each edges_by_type[] bucket reports the uncapped total in its count field, while the edges array is capped at 10. A row where count exceeds the length of edges means the metadata column shows a truncated view of that node’s neighbors.

You can run this check directly against the METADATA column without consulting the cardinality audit table — it complements the cardinality_info field used for rule-capped edges. See ID Graph Cardinality Rules for the rule-capped case.

Best practices

For the identity_stitcher model, use only the id_types and inputs necessary to stitch the identifiers relevant to an entity. For example, if you don’t care about anonymous traffic, do not add anonymous_id as an id_type or any inputs related to anonymous traffic.
Conceptually, you can think of your identity_graph table as a kind of fact table in star or Snowflake schema. All of your core identifiers are in the center table and are used to connect to your inputs.
Only include inputs as edge sources that include id_types directly related to the entity (the first level out from the fact table in a Snowflake schema). For example, you may need to include an id_type for product names because they are in a separate table but you do not want to include them in the identity_graph.

Archive: Inputs

Mon, 01 Jan 0001 00:00:00 +0000

This guide introduces you to the concept of inputs in Profiles and shows how to use them in your Profiles project.

Overview

The foundation of every companies data is the source files and data tables - these are the inputs that go into every data product a company produces.

In Profiles, inputs contain the IDs for identity graphs and the fields that build into a Customer360 (C360). They can be database tables or views, S3 buckets, and/or CSV files.

SQL Keyword: FROM

Usage

You can define inputs in the inputs.yaml file, as shown:

inputs:
 - name: rsIdentifies
 app_defaults:
 table: autotrack_data.autotrack.identifies
 occurred_at_col: timestamp
 ids:
 - select: "user_id"
 type: user_id
 entity: user
 - select: "anonymous_id"
 type: anonymous_id
 entity: user
 - select: "lower(email)"
 type: email
 entity: user
 - name: rsPages
 app_defaults:
 table: autotrack_data.autotrack.pages
 occurred_at_col: timestamp
 ids:
 - select: "user_id"
 type: user_id
 entity: user
 - select: "anonymous_id"
 type: anonymous_id
 entity: user

Best practices

Though it is possible to bring in data sources that only have one identifier column for your entity, it is best to have two or more columns, especially for inputs that you want to use as a source for the id_stitcher model.
If any of your ID columns are not of the string data type, then you must case them explicitly, as shown below:

 ids:
 - select: "CAST(user_id AS VARCHAR)"
 type: user_id
 entity: user

Archive: Features

Mon, 01 Jan 0001 00:00:00 +0000

This guide introduces you to the concept of features in Profiles and explains how to define them in your Profiles project.

Overview

A Customer360 (C360) table contains various customer features. In Profiles, you can create these features using entity_vars which, in turn, can be created using an input, model or other entity_vars.

Usage

You can define features in the profiles.yaml file, as shown:

var_groups:
 - name: user_features
 entity_key: user
 vars:
 - entity_var:
 name: first_seen
 select: min(cast(timestamp as date))
 from: inputs/rsPages
 is_feature: false
 description: First seen date
 - entity_var:
 name: last_seen
 select: max(cast(timestamp as date))
 from: inputs/rsPages
 description: Last seen date
 - entity_var:
 name: user_lifespan
 select: '{{ user.last_seen }} - {{ user.first_seen }}'
 description: User lifespan

Incremental features

This is a beta feature.

Profiles also supports Incremental Features to help reduce warehouse costs, especially in case of large datasets.

By adding a merge: clause to your entity_var definitions and marking the input model as is_append_only: true, you enable Profiles to process only new data and merge it with previously computed values — improving performance and reducing costs considerably.

Controlling update frequency

Not all features need to be computed on every run — use Timegrains to control how often features are updated:

var_groups:
 - name: realtime_signals
 entity_key: user
 time_grain: day  # Computed daily
 vars:
 - entity_var:
 name: last_active_date
 select: max(timestamp::date)
 from: inputs/rsPages

 - name: monthly_aggregates
 entity_key: user
 time_grain: month  # Computed monthly
 vars:
 - entity_var:
 name: favorite_category
 select: mode(category)
 from: inputs/orders

This helps reduce compute costs for features that don’t need frequent updates.

Best practices

An entity_var will automatically group records from a source input or model by customer, so you need only to specify an aggregate function in the select field.
Every entity_var is included in feature_views by default.
To use an entity_var to create another entity_var but not include it as a feature, add is_feature: false to the entity_var definition.
For large datasets, consider making features incremental to improve performance.
Group features by update frequency using time_grain on var_group to optimize compute costs.

Archive: Incremental Features

Mon, 01 Jan 0001 00:00:00 +0000

This guide introduces you to the concept of incremental features in Profiles and explains how they improve performance when processing large datasets.

Overview

Incremental features update existing feature values with newly arrived event data instead of recalculating from the entire historical dataset.

Instead of scanning billions of rows on every run, incremental features reuse previously computed results and only process new data that has arrived since the last run.

How incremental features work

When you define an incremental feature, Profiles stores previous results in a checkpoint, identifies new data since the last checkpoint, merges previous values with new calculations using your merge logic, and produces updated feature values.

Mandatory requirement:

Inputs used by incremental features require contract.is_append_only: true to be configured. Without this configuration, the project may throw an error.

See Make Features Incremental for configuration details.

Features built incrementally may drift slightly with every run due to late-landing data.

When to use incremental features

Use incremental features when:

You want to reduce warehouse costs when processing large historical datasets (millions or billions of rows)
New data arrives regularly (daily, hourly, or more frequently)
The volume of new data is significantly smaller than your total dataset
Your features can be expressed using supported incremental patterns

Not all features can be computed incrementally.

Features that require scanning all historical data (such as median or percentile calculations) cannot use incremental computation. For such cases, consider using Timegrains and bundling.

Relationship to other Profiles features

Incremental features work alongside other Profiles capabilities:

Incremental ID stitching: The identity graph can also run incrementally, processing only new identity relationships.
Timegrains: For features that cannot be incremental, timegrains control how frequently they recompute.
Entity Var Bundling: Groups the execution of entity_vars based on their definitions to improve efficiency. This optimization complements incremental features and can help optimize performance for features that cannot be incremental.

Incremental entity vars do not support entity var bundling yet. The bundling algorithm will only bundle non incremental entity vars. Incremental entity vars will run one at a time.

Tip: Input vars are generally discouraged due to performance considerations. For incremental processing, consider using entity_vars with merge property or incremental SQL models.

Contact RudderStack Support to discuss specific use cases that may require input vars.

Understanding composable functions

Composable functions are functions that can be merged incrementally by combining results from different time periods:

Simple composable: SUM, MIN, MAX, COUNT - can be merged directly (e.g., SUM(period1) + SUM(period2) = SUM(all))
Compound composable: AVG, WEIGHTED_AVG - require breaking into multiple simple composable functions (e.g., AVG = SUM / COUNT)

Note that non-composable functions (e.g., Median, percentiles, DISTINCT COUNT (in most warehouses)) cannot be computed incrementally and require full scans.

AVG is not composable directly: You cannot average two averages correctly without knowing the counts. For example, averaging 10 (from 5 values) and 20 (from 3 values) doesn’t give you the correct overall average. You need: (SUM1 + SUM2) / (COUNT1 + COUNT2).

Supported patterns

Profiles supports incremental computation through three main pattern categories:

Pattern	Description
Simple aggregations	Direct composable aggregations using `SUM`, `MIN`, `MAX`, `COUNT` with merge property
Compound aggregations	Aggregations requiring multiple intermediate simple composable aggregations, such as: `AVG(=SUM/COUNT)` `WEIGHTED_AVG(=SUM_OF_WEIGHTED_VALUES/SUM_OF_WEIGHTS)` `LAST_KNOWN_LOCATION(=MAX_BY(location update, update timestamp))` arrays(e.g. `LAST_10_TOUCHPOINTS`)
Patterns requiring input pre-processing before aggregation	An incremental SQL model maintains a running snapshot of input dataset. Non-incremental entity vars are then defined on top of this processed input. For example, window function cases can be made incremental using an intermediate incremental input window model, like `user_touchpoints_in_last_30_days`, to subsequently define `is_daily_active_user` and `is_monthly_active_user` over it

Archive: Feature Views

Mon, 01 Jan 0001 00:00:00 +0000

This guide introduces you to the concept of feature views in Profiles and shows how to use them in your Profiles project.

Video walkthrough

Overview

Profiles adds a unique ID when you create a Customer360 (C360) table. But for most activation use cases, you need to use a specific ID from your data warehouse, like user_name, email, or anonymous ID. You can achieve this using SQL, the ID graph table, and the C360 table, but it is complicated and can be difficult to maintain.

Profiles can create these pivots automatically with feature views. They include the same entity_vars as the C360 table but with the chosen ID as the primary key.

SQL Keyword: VIEW

Requirements

entity_vars where is_feature is set to true.
View primary ID is included in ID graph.

Usage

Feature views (feature_views) are in the pb_project.yaml file within the entities block, as shown:

name: llm_sdr_email_content_generation
schema_version: 85
connection: llm-recommend-dev
model_folders:
 - models
entities:
 - name: user
 id_stitcher: models/user_id_stitcher
 id_types:
 - user_id
 - anonymous_id**
###########################################
 feature_views:
 using_ids:
 - id: anonymous_id
 name: anonymous_id_360
###########################################
id_types:
 - name: user_id
 - name: anonymous_id
 filters: # Optional
 - type: exclude
 value: ""
python_requirements: # Optional
 - profiles_mlcorelib==0.7.2

Best practices

Profiles automatically creates a feature view where the main_id is the primary key, called {entity_name}_feature_view. It is a best practice to create another feature view using a desired ID type as the primary key for data activation.
All features will be included in all feature views for an entity, unless you set is_feature to false for a specific entity_var.
Feature views are defined in pb_project.yaml for the user/all cohort. For individual cohorts, they are defined in the cohort definition in profiles.yaml.
Cohorts inherit the features from their parent, so you do not need to define them at the cohort level.

Archive: Cohorts

Mon, 01 Jan 0001 00:00:00 +0000

This guide introduces you to the concept of cohorts in Profiles and shows how to use them in your Profiles project.

Overview

All businesses have core entities like customers. Within those entities, there are core segments that are tracked and interacted with across the business, for example, high value customers, brands, business segments, regions, etc.

In Profiles, you can create these enduring subsets by leveraging cohorts. You use the features of your Customer360 (C360) to filter entities into these cohorts. This gives you a stable definition for these important segments within the warehouse that you can leverage across the business.

Requirements

Features used for filtering must be defined in the entity/all cohort or other parent cohort var_group.

Usage

You can define cohorts in the profiles.yaml file, as shown:

# Cohort definition
models:
 - name: credit_card_user
 model_type: entity_cohort
 model_spec:
 extends: customers/all
 materialization:
 output_type: view
 filter_expression:
 AND:
 - "{{ customers.has_credit_card }} = 1"
 feature_views:
 - id: email
 name: credit_card_user_by_email

# Cohort Feature Definition
var_groups:
 - name: credit_card_user_vars
 entity_cohort: models/credit_card_user
 vars:
 - entity_var:
 name: total_sales
 select: sum(total)
 from: inputs/orders
 - entity_var:
 name: total_orders
 select: count(order_id)
 from: inputs/orders

Best practices

Use cohorts as sub-groups that require a standard definition, can be used across teams, and are needed to exist for a long period of time. For example, key customer segments, brands, or business units, but not a campaign audience.
Cohorts can also be a subset of another cohort. For example, you can create a cohort named known_mobile_users by filtering the known_users cohort.
A child cohort inherits all the features from their parent and can have features and feature views defined at the individual cohort level.

Archive: Timegrains

Mon, 01 Jan 0001 00:00:00 +0000

This guide introduces you to the concept of Timegrains in Profiles and shows how to define them in your Profiles project.

Overview

Timegrains gives you the ability to control the frequency at which certain models run irrespective of the frequency at which you trigger the project runs. This is useful for:

Reducing compute costs: Features that don’t need real-time updates can be computed less frequently.
Organizing features by freshness: Group real-time signals separately from slow-changing attributes.
Managing non-composable functions: Features like percentiles or distinct counts that can’t be computed incrementally can use longer time grains.

You can define the following timegrains in your model’s model_spec file:

tick, 10minutes, hour, day, week, month, year

For example, suppose you have defined a feature for which you do not need the most updated data, for example, computing a feature for each user every month. You can then set the time_grain : month and run the project. Profiles Builder ensures that this feature is computed only once a month. The computed values from the month boundary are used in all the subsequent feature_views. Any new identified user identified after the start of the month boundary will have the value of that feature set to null.

Note that:

You will get the same output if you specify a model’s time_grain to be day and you run this model multiple times a day.
If a model’s time_grain is specified to be day, it will only consider input data until that day’s boundary (00:00 UTC). Any data ingested after this timestamp is not considered in the computation.
The boundary for a particular month is the first Monday of that month. For example, if March 1 is a Friday, then it’s boundary will be March 4th. Similarly, the boundary for the year is the first Monday of that year.

Usage

You can specify the timegrain in model_spec of any model. An example is shown below:

models:
 - name: users_with_valid_email
 model_type: entity_cohort
 model_spec:
 extends: user/all
 time_grain: "day"
 filter_expression:
 AND:
 - "NOT {{ user.Var('id_type_email_count') }} = 0"
 feature_views:
 name: users_with_valid_email_feature_view

You can also specify it at var_group level. For example:

 - name: weekly_user_vars
 entity_cohort: models/new_users
 time_grain: "week"
 vars:
 - entity_var:
 name: campaign_sources
 select: "{{list_agg('context_campaign_source', ',')}}"
 from: inputs/rsTracks

Note if you have specified time_grain on any of the model, PB will need you to specify a default_time_grain in your pb_project.yaml file. Its value should be equal to or lower than the finest time_grain you have used in any of the models. This default_time_grain will be applied to all the models which don’t have a specified time_grain.

default_time_grain: day

For example, if you have specified day, week, and month as your timegrains for all your models, you will need to specify default_time_grain: day . You can also set default_time_grain to tick, 10minutes, or hour - however, this is not recommended as it may negatively impact the performance.

As a general rule, note that you should specify the default_time_grain to match with your run schedule. If you are scheduling PB to run daily, its best to keep default_time_grain: day. You can then accordingly schedule the run at 00:00 UTC so that the latest data is considered in the run.

Features: Learn about defining features in Profiles.
Var Groups: Reference for var_group configuration including time_grain.
Incremental Features: Optimize feature computation for large datasets.

Archive: SQL Models

Mon, 01 Jan 0001 00:00:00 +0000

This guide introduces you to the concept of SQL models in Profiles and shows how to use them in your Profiles project.

Overview

Profiles projects allows for the importing and transformation of almost any source. However, there are edge cases that cannot be completed, or are overly onerous in case of Profiles. You can use SQL models in such cases.

SQL models allow you to model data using plain or templated SQL. This is useful for complex use cases where you need to join, union, or intermediately transform data.

You can use a SQL model as an ephemeral source for ID stitcher and feature creation models, materialize it as a table or view.

Usage

You can define SQL models in the sql_model.yaml file, as shown:

models:
 - name: tracks_union_pages
 model_type: sql_template
 model_spec:
 materialization:
 output_type: view
 run_type: discrete
 single_sql: |
 {% with tracks = this.DeRef("inputs/tracks") pages = this.DeRef("inputs/pages") %}
 SELECT
 anonymous_id,
 event,
 timestamp
 FROM {{ tracks }}
 UNION
 SELECT
 anonymous_id,
 'page_viewed' AS event,
 timestamp
 FROM {{ pages }}
 {% endwith %}

Relationship to other Profiles features

SQL models integrate with other Profiles capabilities:

Incremental SQL Models: Regular SQL models can be made incremental by dereferencing their own previous materialization. See Incremental SQL Models for details on enabling incremental processing for complex stateful patterns.
ID Stitcher: SQL models can serve as ephemeral sources for ID stitcher, providing transformed data for identity resolution.
Feature Creation: SQL models can be used as intermediate data sources for entity vars and feature creation, enabling complex data transformations before aggregation.
Materialization Options: SQL models can be materialized as tables (persistent storage), views (virtual tables), or ephemeral sources (temporary computation).

Best practices

Verify the outputs of any SQL model in your warehouse before adding to a Profiles project.

Archive: Cleanup

Mon, 01 Jan 0001 00:00:00 +0000

This guide introduces you to the concept of cleanup in Profiles.

Overview

You can use the pb cleanup materials command in RudderStack Profiles to remove any outdated materials (tables, views) from the data warehouse - thereby optimizing storage and ensuring that unnecessary historical materials do not accumulate over time.

There are two ways to perform a material cleanup - old and new.

Old version

This version of cleanup decides which materials to delete depending on the time when PB last accessed it.

This is the default mode now, meaning running pb cleanup materials triggers this mode automatically.

You can also specify a retention period in the above command by adding any of the following three flags:

-r
--retention_time_in_ms
--retention_time_in_hours

If no flag is passed, RudderStack considers a default retention period of 180 days.

Usage

Some examples of using the cleanup command are shown:

$pb cleanup materials # Deletes materials whose last access time is greater than 30 days.

$pb cleanup materials -r 1 # Deletes materials whose last access time is greater than 1 day.

$pb cleanup materials --retention_time_in_ms 5 # Deletes materials whose last access time is greater than 5 milliseconds.

$pb cleanup materials --retention_time_in_hours 3 # Deletes materials whose last access time is greater than 3 hours.

Limitations

Note that the old version create issues while performing cleanups in cases where you are using timegrains.

Suppose you have a Profiles project which has:

A weekly timegrain model that runs once per week and no other models of a finer timegrain depend on it.
An hourly timegrain model that runs more frequently.
Some models with a default timegrain of tick.

When you run this project daily, models only with timegrain finer than or equal to a day run everyday. However, the weekly model runs only once per week.

If you run the cleanup on the fourth day after the weekly model was run with a retention period of 3 days, then the cleanup will delete its materials, even though the frontier views still need them.

A project run after this incident will again run the weekly timegrain model because it is deleted - this is inefficient. Also, the frontier views become useless as underlying materials are deleted.

A new version of cleanup is available to overcome the above limitations.

New version

Use this mode if you are using timegrains in your Profiles project.

When PB runs, the retention_period parameter defined in the pb_project.yaml is recorded in the material registry against the models. The new cleanup version uses this value and also does dependency tracking when deciding which materials to delete.

You can specify the retention_period in the project as follows:

In the pb_project.yaml:

name: profiles_project
schema_version: 85
connection: wh_connection_dev

retention_period: 24h # 1 day

You can also specify retention_peroid at the model level as well:

models:
 - name: user_id_stitcher
 model_type: id_stitcher
 model_spec:
 entity_key: user
 retention_period: 24h
 edge_sources:
 - from: inputs/rsIdentifies
 - from: inputs/rsPages

Note that:

retention_period should be strictly a non-zero positive value.
RudderStack considers a default retention period of 30 days if not explicitly defined in the YAML using the retention_period key.

If you have set up the Profiles project in the RudderStack dashboard, then the retention period specified in the project’s settings will be used during cleanup.

Usage

Pass the --expired flag in the pb cleanup materials command to trigger this mode.

pb cleanup materials --expired

Deletion criteria

The criteria this cleanup mode uses to determine which materials to delete is explained below:

Expiration criteria

Every model has a retention period, which you can set at the model level or for the entire project.
A material is considered expired if all of the below conditions are met:
- Its last access/reference time is before the current time - retention period.
- There exists a newer run of the same model.
- No other non-expired material depends on this material.

Preventing accidental deletion

This mode ensures that cleanup happens safely without unintentionally deleting materials that are still in use.

RudderStack Docs – Concepts

Archive: Entities

Overview

Requirements

Usage

Best practices

Archive: IDs

Overview

Requirements

Usage

Best practices

Archive: Identity Graph

Overview

Requirements

Usage

Link the identity graph to your entity

Metadata and observability

Metadata structure

Reading the metadata column

Multi-source edges

Incremental run behavior

Metadata size and limits

Best practices

Archive: Inputs

Overview

Usage

Best practices

Archive: Features

Overview

Usage

Incremental features

Controlling update frequency

Best practices

Archive: Incremental Features

Overview

How incremental features work

When to use incremental features

Relationship to other Profiles features

Understanding composable functions

Supported patterns

See also

Archive: Feature Views

Video walkthrough

Overview

Requirements

Usage

Best practices

Archive: Cohorts

Overview

Requirements

Usage

Best practices

Archive: Timegrains

Overview

Usage

Related resources

Archive: SQL Models

Overview

Usage

Relationship to other Profiles features

Best practices

Archive: Cleanup

Overview

Old version

Usage

Limitations

New version

Usage

Deletion criteria

Expiration criteria

Preventing accidental deletion