RudderStack Docs – Incremental Features

Archive: Simple Aggregations

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains simple aggregations and how to implement them as incremental features in Profiles.

Overview

Simple aggregations are direct composable aggregations that combine previous values with new data using straightforward merge functions.

Only composable functions work: SUM, MIN, MAX, COUNT are composable because results from different time periods can be merged correctly. AVG is NOT composable - you cannot average two averages correctly. For example, averaging 10 (from 5 values) and 20 (from 3 values) doesn’t give the correct overall average of 13.75. Use Compound Aggregations for AVG.

Motivation: When and why to use simple aggregations

Use simple aggregations when:

You need to compute totals, counts, or min/max values
Your aggregation uses composable functions: SUM, MIN, MAX, or COUNT
The merge operation is straightforward (sum previous + new, take min/max of previous and new)

Simple aggregations are the first pattern to try because they:

Require minimal configuration (just add a merge: clause)
Have predictable behavior
Are the most performant incremental pattern
Work across all data warehouses

If your use case doesn’t fit simple aggregations, consider Compound Aggregations or Incremental SQL Models.

Real-world examples

Here are common use cases that map to simple aggregations:

Use case	Aggregation	Example
Total lifetime value	`SUM`	Sum of all purchases across a customer’s lifetime
First purchase date	`MIN`	Earliest timestamp when a customer made their first purchase
Last login time	`MAX`	Most recent timestamp when a user logged in
Total event count	`COUNT`	Total number of events a user has generated

Supported simple aggregators

Profiles supports the following simple aggregators for incremental features:

SUM

Use SUM for totals and cumulative values.

Example: Total lifetime value

- entity_var:
 name: total_purchases
 select: SUM({{orders.value}})
 merge: SUM({{rowset.total_purchases}})
 from: inputs/orders
 description: Total purchase value across all orders

How it works: The merge property adds the previous total to the new sum.

MIN

Use MIN for earliest values or minimum values.

Example: First purchase date

- entity_var:
 name: first_purchase_date
 select: MIN({{orders.order_date}})
 merge: MIN({{rowset.first_purchase_date}})
 from: inputs/orders
 where: {{orders.status}} = "completed"
 description: Date of first completed purchase

How it works: The merge property takes the minimum of the previous and new values.

MAX

Use MAX for latest values or maximum values.

Example: Last login time

- entity_var:
 name: last_purchase_date
 select: MAX({{orders.order_date}})
 merge: MAX({{rowset.last_purchase_date}})
 from: inputs/orders
 description: Most recent purchase date

How it works: The merge property takes the maximum of the previous and new values.

COUNT

Use COUNT for counting events or rows.

Example: Total event count

- entity_var:
 name: event_count
 select: COUNT(*)
 merge: SUM({{rowset.event_count}})
 from: inputs/events
 description: Total number of events

Important: For COUNT aggregations, always use SUM in the merge property, not COUNT. You’re summing the counts from previous runs and new data, not counting counts.

How it works: The merge property adds the previous count to the new count.

`merge` property syntax

The merge property defines how to combine previous feature values (from the checkpoint) with new calculations.

Basic syntax

- entity_var:
 name: feature_name
 select: AGGREGATION_FUNCTION(...)
 merge: MERGE_FUNCTION({{rowset.feature_name}})
 from: inputs/table_name

Merge function reference

Select function	Merge function	Notes
`MIN(...)`	`MIN({{rowset.feature_name}})`	Takes minimum of previous and new values
`MAX(...)`	`MAX({{rowset.feature_name}})`	Takes maximum of previous and new values
`SUM(...)`	`SUM({{rowset.feature_name}})`	Sums previous and new values
`COUNT(...)`	`SUM({{rowset.feature_name}})`	Sums counts (not COUNT of counts)

Rowset reference

In merge property, use {{rowset.feature_name}} to reference one or more base aggregations from the baseline checkpoint, and the delta aggregation from current checkpoint.

For example, if baseline ID1, ID2 and ID3 merge into current checkpoint ID2, then you would be merging the following rowsets:

All input rows associated with ID1 and part of baseline.
All input rows associated with ID2 and part of baseline.
All input rows associated with ID3 and part of baseline.
All input rows landing between baseline and current checkpoint, and mapped to the merged ID2.

Key points:

{{rowset.feature_name}}: References the previous value of feature_name from the checkpoint
{{rowset.column_name}}: References any column from the previous checkpoint
{{this.feature_name}}: References the new value being computed (used in merge_where clauses)

When simple aggregations don’t work

Simple aggregations may not be sufficient when:

Your aggregation isn’t directly composable: AVG(WEIGHTED_AVG) cannot be computed directly. Use Compound Aggregations and combine helper aggregations like SUM(weighted sum) and COUNT(sum_of_weights).
You need MIN_BY/MAX_BY aggregations: Selecting a value based on the min/max of another column (for example, LAST_KNOWN_LOCATION using MAX_BY(location, timestamp) or FIRST_PURCHASE_STORE using MIN_BY(store, purchase_date)) requires a _by_param helper var. See MIN_BY/MAX_BY with ordering helper.
You need to replace window functions: LAST_VALUE, FIRST_VALUE, and other window functions are not composable — replace them with MAX_BY/MIN_BY. See Replace window functions.
You need array aggregations: Lists and arrays require union operations. Use Compound Aggregations.
You need time window aggregations: When you need to aggregate running totals, balances including or multi-step aggregations within a time window, you can create an incremental input window using Incremental SQL Models.
You need PIVOT operations: When you need to transform rows into columns (for example, last_login_by_device_id to track the last login time for each device separately). You can use Incremental SQL Models to achieve this.

`merge_where` for conditional merging

You can use the merge_where: clause to restrict merging to only those rowsets that meet specific conditions:

- entity_var:
 name: timestamp
 select: MAX({{game_events.timestamp}})
 merge: MAX({{rowset.timestamp}})
 from: inputs/game_events
 where: {{game_events.event_type}} = "BREAK"
 is_feature: false
 description: Timestamp of the most recent break event (used for score lookup)

- entity_var:
 name: score_at_last_break
 select: any_value({{game_events.score}})
 where: {{game_events.timestamp}} = {{user.timestamp}}
 merge: any_value({{game_events.score}})
 merge_where: {{rowset.timestamp}} = {{user.timestamp}}
 from: inputs/game_events
 description: Score at the most recent break event

Use {{rowset.column}} for previous values and {{this.column}} for new values in the condition.

Best practices

Start with simple aggregations before moving to more complex patterns
Ensure merge property matches the aggregation function
For COUNT, always use SUM in the merge property
Use merge_where: only when you need conditional merging logic
Verify incremental results match full refresh results

Troubleshooting

Issue	Solution
`merge` function does not match aggregation type	Ensure `merge` function matches select aggregation (`SUM` for `SUM`, `MIN` for `MIN`, etc.)
Missing rowset reference	Use `{{rowset.feature_name}}` to reference previous values from checkpoint
`COUNT` merge using `COUNT` instead of `SUM`	Use `SUM({{rowset.feature_name}})` for COUNT aggregations
Incorrect merge results	Verify `merge_where:` conditions are logically correct, if used

See more

Archive: Compound Aggregations

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains compound aggregations and how to implement them as incremental features in Profiles.

Overview

Compound aggregations require intermediate calculations or multiple steps, each implemented as a simple composable aggregation. They break complex aggregations into component parts that can be maintained incrementally.

Motivation: When and why to use compound aggregations

Use compound aggregations when:

You need to compute averages (AVG, WEIGHTED_AVG), which cannot be computed directly incrementally
You need to select values based on min/max of another column (MIN_BY, MAX_BY)
You need to replace window functions (LAST_VALUE, FIRST_VALUE) with incremental alternatives
You need to aggregate arrays or lists that require union and sorting operations
You need boolean aggregations (BOOLOR_AGG, BOOLAND_AGG)
Your aggregation requires multiple intermediate calculations

Compound aggregations use:

Component parts (like sum and count for averages)
Intermediate entity vars maintained incrementally
Final calculations that combine components

Tip: If compound aggregations don’t fit your use case, consider Incremental SQL Models for stateful processing or complex transformations.

Real-world examples

Here are common use cases that map to compound aggregations:

Use case	Pattern	Example
Average order value	AVG using sum/count	Average purchase amount across all orders
Average session duration	AVG using sum/count	Average time spent per session
Weighted average rating	WEIGHTED_AVG using weighted sum/weight sum	Average product rating weighted by number of reviews
List of countries visited	Array aggregation	Sorted list of unique countries a user has visited
Array of product IDs	Array aggregation	List of all product IDs a user has purchased
Most recent login platform	MIN_BY/MAX_BY with helper	Platform of the most recent login event
First campaign source	MIN_BY/MAX_BY with helper	Campaign source from the earliest identification
Has ever enabled push	Boolean aggregation	Whether a user has ever enabled push notifications

Average (AVG) using component features

AVG cannot be computed directly incrementally. Express it using sum and count components:

Sum component: Maintain sum incrementally using SUM with SUM merge
Count component: Maintain count incrementally using COUNT with SUM merge (counts from previous checkpoints are summed together)
Average calculation: Divide sum by count in a final entity var with division by zero protection

Example: Average order value

- entity_var:
 name: sum_of_order_values
 select: SUM({{orders.amount}})
 merge: SUM({{rowset.sum_of_order_values}})
 from: inputs/orders
 is_feature: false
 description: Sum of order values (used for average calculation)

- entity_var:
 name: count_of_orders
 select: COUNT({{orders.order_id}})
 merge: SUM({{rowset.count_of_orders}})
 from: inputs/orders
 is_feature: false
 description: Count of orders (used for average calculation)

- entity_var:
 name: avg_order_value
 select: "CASE WHEN {{user.count_of_orders}} > 0 THEN {{user.sum_of_order_values}} / {{user.count_of_orders}} ELSE 0 END"
 description: Average order value across all orders

How it works:

Sum component: Maintains total sum incrementally using SUM with SUM merge
Count component: Maintains total count incrementally using COUNT with SUM merge (counts from previous checkpoints are summed together)
Average calculation: Divides sum by count using {{user.var_name}} syntax, with division by zero protection

Tip: Mark sum and count components with is_feature: false since they’re intermediate calculations. The average is computed on-the-fly from the incrementally maintained components.

Workflow

Input Data → [Sum Component (Incremental)] → Final Calculation
 → [Count Component (Incremental)] ↗

Profiles creates intermediate tables for sum and count components
Both components are maintained incrementally using simple aggregation patterns
The average is computed by dividing sum by count (with division by zero protection)

Weighted Average (WEIGHTED_AVG) using component features

WEIGHTED_AVG cannot be computed directly incrementally. You can express it using weighted sum and sum of weights components:

Weighted sum component: Maintain sum of (value × weight) incrementally using SUM with SUM merge
Sum of weights component: Maintain sum of weights incrementally using SUM with SUM merge
Weighted average calculation: Divide weighted sum by sum of weights in a final entity var with division by zero protection

Example: Weighted average product rating

- entity_var:
 name: weighted_sum_of_ratings
 select: SUM({{reviews.rating}} * {{reviews.review_count}})
 merge: SUM({{rowset.weighted_sum_of_ratings}})
 from: inputs/reviews
 is_feature: false
 description: Sum of (rating × review_count) for weighted average calculation

- entity_var:
 name: sum_of_review_counts
 select: SUM({{reviews.review_count}})
 merge: SUM({{rowset.sum_of_review_counts}})
 from: inputs/reviews
 is_feature: false
 description: Sum of review counts (weights) for weighted average calculation

- entity_var:
 name: weighted_avg_rating
 select: "CASE WHEN {{user.sum_of_review_counts}} > 0 THEN {{user.weighted_sum_of_ratings}} / {{user.sum_of_review_counts}} ELSE 0 END"
 description: Weighted average product rating across all reviews

How it works:

Weighted sum component: Maintains sum of (value × weight) incrementally using SUM with SUM merge -Sum of weights component: Maintains total sum of weights incrementally using SUM with SUM merge
Weighted average calculation: Divides weighted sum by sum of weights using {{user.var_name}} syntax, with division by zero protection

Tip: Mark weighted sum and sum of weights components with is_feature: false since they’re intermediate calculations. The weighted average is computed on-the-fly from the incrementally maintained components.

Workflow

Input Data → [Weighted Sum Component (Incremental)] → Final Calculation
 → [Sum of Weights Component (Incremental)] ↗

Profiles creates intermediate tables for weighted sum and sum of weights components
Both components are maintained incrementally using simple aggregation patterns
The weighted average is computed by dividing weighted sum by sum of weights (with division by zero protection)

Array aggregations

For array aggregations, use ARRAY_UNION_AGG or ARRAY_SORT in the merge property to combine arrays from previous checkpoints with new arrays.

Warehouse-specific syntax: Array aggregation syntax varies by warehouse. The examples in this section are specific to Snowflake. Adjust syntax for your warehouse.

Example: List of countries visited

- entity_var:
 name: countries_list
 select: ARRAY_AGG(DISTINCT {{tracks.country}}) WITHIN GROUP (ORDER BY {{tracks.country}})
 merge: ARRAY_SORT(ARRAY_UNION_AGG({{rowset.countries_list}}))
 from: models/tracks
 where: country IS NOT NULL
 description: Sorted list of unique countries

How it works:

Select clause: Creates an array of distinct countries from new data, sorted
Merge property:
- ARRAY_UNION_AGG({{rowset.countries_list}}): Unions previous and new arrays, removing duplicates
- ARRAY_SORT(...): Sorts the final array

Array merge functions

Select function	Merge function	Notes
`ARRAY_AGG(...)`	`ARRAY_UNION_AGG({{rowset.feature_name}})`	Unions arrays, removing duplicates
`ARRAY_AGG(...)`	`ARRAY_SORT(ARRAY_UNION_AGG({{rowset.feature_name}}))`	Unions and sorts arrays

MIN_BY / MAX_BY with ordering helper

MIN_BY and MAX_BY select a value from the row with the minimum or maximum of another column. These are common replacements for window functions like FIRST_VALUE and LAST_VALUE.

Required: _by_param helper var.

When using MIN_BY or MAX_BY in a merge clause, you must define a companion entity var (conventionally suffixed _by_param) that tracks the ordering column. Omitting this helper breaks incremental runs because the merge has no ordering context from previous checkpoints.

- entity_var:
 name: last_login_platform
 select: MAX_BY({{logins.platform}}, {{logins.timestamp}})
 merge: MAX_BY({{rowset.last_login_platform}}, {{rowset.last_login_platform_by_param}})
 from: inputs/logins
 description: Platform of the most recent login

- entity_var:
 name: last_login_platform_by_param
 select: MAX({{logins.timestamp}})
 merge: MAX({{rowset.last_login_platform_by_param}})
 from: inputs/logins
 is_feature: false
 description: Ordering helper for last_login_platform

How it works:

Main var: MAX_BY(platform, timestamp) selects the platform from the row with the latest timestamp
Helper var: MAX(timestamp) tracks the latest timestamp across checkpoints
Merge: MAX_BY({{rowset.last_login_platform}}, {{rowset.last_login_platform_by_param}}) compares the previous value’s timestamp with the new value’s timestamp and keeps the one with the later timestamp

Example: First purchase store

- entity_var:
 name: first_purchase_store
 select: MIN_BY({{orders.store_name}}, {{orders.order_date}})
 merge: MIN_BY({{rowset.first_purchase_store}}, {{rowset.first_purchase_store_by_param}})
 from: inputs/orders
 description: Store where the user made their first purchase

- entity_var:
 name: first_purchase_store_by_param
 select: MIN({{orders.order_date}})
 merge: MIN({{rowset.first_purchase_store_by_param}})
 from: inputs/orders
 is_feature: false
 description: Ordering helper for first_purchase_store

MIN_BY / MAX_BY merge reference

Select function	Merge function	Helper var
`MIN_BY(value_col, order_col)`	`MIN_BY({{rowset.feature}}, {{rowset.feature_by_param}})`	`MIN(order_col)` with `MIN` merge
`MAX_BY(value_col, order_col)`	`MAX_BY({{rowset.feature}}, {{rowset.feature_by_param}})`	`MAX(order_col)` with `MAX` merge

When you can skip the `_by_param` helper

You can skip the helper only when the ordering column is irrelevant to the merge. For example, when you always want the latest value and don’t need to track when it changed. In practice, this is rare — however when in doubt, include the helper.

Boolean aggregations

For boolean OR aggregations, use BOOLOR_AGG in both select and merge:

Example: Has user ever enabled push notifications

- entity_var:
 name: has_ever_enabled_push
 select: BOOLOR_AGG(COALESCE({{settings.push_enabled}}, false))
 merge: BOOLOR_AGG({{rowset.has_ever_enabled_push}})
 from: inputs/settings
 description: Whether the user has ever enabled push notifications

How it works: BOOLOR_AGG returns true if any value in the group is true. Since boolean OR is composable (once true, always true), previous and new results can be merged directly.

Boolean merge reference

Select function	Merge function	Notes
`BOOLOR_AGG(...)`	`BOOLOR_AGG({{rowset.feature_name}})`	Boolean OR: true if any value is true
`BOOLAND_AGG(...)`	`BOOLAND_AGG({{rowset.feature_name}})`	Boolean AND: true only if all values are true

Warehouse-specific syntax: BOOLOR_AGG and BOOLAND_AGG are Snowflake functions.

For other warehouses, use equivalent functions (for example, LOGICAL_OR / LOGICAL_AND in BigQuery, BOOL_OR / BOOL_AND in Redshift and Databricks).

Replace window functions

Window functions like LAST_VALUE, FIRST_VALUE, LAG, and LEAD cannot be used in incremental entity vars because they require scanning all data to determine row ordering. Replace them with composable alternatives.

Common replacements

Window function	Incremental replacement	Pattern
`LAST_VALUE(col) OVER (ORDER BY ts)`	`MAX_BY(col, ts)`	MIN_BY/MAX_BY with helper
`FIRST_VALUE(col) OVER (ORDER BY ts)`	`MIN_BY(col, ts)`	MIN_BY/MAX_BY with helper
`ROW_NUMBER()`, `RANK()`, `DENSE_RANK()`	Not directly replaceable	Use Incremental SQL Models
`LAG()`, `LEAD()`	Not directly replaceable	Use Incremental SQL Models

Example: Replace LAST_VALUE with MAX_BY

# Before (non-incremental, uses window function)
- entity_var:
 name: last_known_city
 select: last_value(city)
 from: inputs/events
 window:
 order_by:
 - timestamp
 where: city IS NOT NULL

# After (incremental, uses MAX_BY with helper)
- entity_var:
 name: last_known_city
 select: MAX_BY({{events.city}}, {{events.timestamp}})
 merge: MAX_BY({{rowset.last_known_city}}, {{rowset.last_known_city_by_param}})
 from: inputs/events
 where: {{events.city}} IS NOT NULL
 description: Most recent city from events

- entity_var:
 name: last_known_city_by_param
 select: MAX({{events.timestamp}})
 merge: MAX({{rowset.last_known_city_by_param}})
 from: inputs/events
 where: {{events.city}} IS NOT NULL
 is_feature: false
 description: Ordering helper for last_known_city

Example: Replace FIRST_VALUE with MIN_BY

# Before (non-incremental, uses window function)
- entity_var:
 name: first_campaign_source
 select: first_value(campaign_source)
 from: inputs/identifies
 window:
 order_by:
 - timestamp
 where: campaign_source IS NOT NULL

# After (incremental, uses MIN_BY with helper)
- entity_var:
 name: first_campaign_source
 select: MIN_BY({{identifies.campaign_source}}, {{identifies.timestamp}})
 merge: MIN_BY({{rowset.first_campaign_source}}, {{rowset.first_campaign_source_by_param}})
 from: inputs/identifies
 where: {{identifies.campaign_source}} IS NOT NULL
 description: Campaign source from first identification

- entity_var:
 name: first_campaign_source_by_param
 select: MIN({{identifies.timestamp}})
 merge: MIN({{rowset.first_campaign_source_by_param}})
 from: inputs/identifies
 where: {{identifies.campaign_source}} IS NOT NULL
 is_feature: false
 description: Ordering helper for first_campaign_source

When compound aggregations don’t work

Compound aggregations may not be sufficient when:

You need activity windows: Tracking recent activity over time windows (like MAU/DAU) requires Incremental SQL Models with Incremental Input Windows.
You need per-object aggregations: Instead of one value per user, you need one value per (user_id, object_id) pair, where object_id could be product_id, location_id, or any other dimension. This requires Incremental SQL Models.
You need complex business logic: Multi-step transformations with conditional logic may require Incremental SQL Models.

Input-vars cannot be built incrementally

An entity_var dependent on an input_var cannot be incremental. To enable incremental features, replace the input_var with an Incremental SQL Model and reference the model from the entity_var instead.

Best practices

Break AVG into sum and count components with separate merge properties
Break WEIGHTED_AVG into weighted sum (value × weight) and sum of weights components with separate merge properties
Always define a _by_param helper var when using MIN_BY/MAX_BY in merge clauses
Replace LAST_VALUE window functions with MAX_BY, and FIRST_VALUE with MIN_BY
Mark intermediate entity vars (helpers, components) with is_feature: false to avoid exposing them as features
Add division by zero protection when computing averages (use CASE WHEN count > 0 THEN sum/count ELSE 0 END for AVG, or CASE WHEN sum_of_weights > 0 THEN weighted_sum / sum_of_weights ELSE 0 END for WEIGHTED_AVG)
Use ARRAY_UNION_AGG for array aggregations to maintain deduplication
Be aware of warehouse-specific syntax differences for array and boolean operations
Verify incremental results match full refresh results

Troubleshooting

Issue	Solution
Average calculation returns incorrect values	Verify that sum and count components are correctly maintained incrementally and include division by zero protection
Weighted average calculation returns incorrect values	Verify that weighted sum (value × weight) and sum of weights components are correctly maintained incrementally and include division by zero protection
Division by zero errors	Add `CASE WHEN count > 0 THEN sum/count ELSE 0 END` for AVG, or `CASE WHEN sum_of_weights > 0 THEN weighted_sum / sum_of_weights ELSE 0 END` for WEIGHTED_AVG
Array aggregation syntax errors	Check warehouse-specific syntax requirements (Snowflake, BigQuery, etc.)
Duplicate values in arrays	Ensure `ARRAY_UNION_AGG` is used in `merge` property to remove duplicates
Intermediate features appearing in output	Mark intermediate entity vars with `is_feature: false`
Missing `_by_param` helper for MIN_BY/MAX_BY	Define a companion entity var (suffixed `_by_param`) that tracks the ordering column with `is_feature: false`
MIN_BY/MAX_BY merge returns wrong value	Verify the `_by_param` helper uses the same `from`, `where`, and ordering column as the main var
Boolean aggregation syntax errors	Use warehouse-appropriate function: `BOOLOR_AGG` (Snowflake), `LOGICAL_OR` (BigQuery), `BOOL_OR` (Redshift/Databricks)

See more

Archive: Incremental SQL Models

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains incremental SQL models and how they enable complex stateful processing in Profiles.

Overview

Incremental SQL models are SQL models that dereference their own previous materialization to enable incremental processing. They enable complex patterns requiring stateful processing, running totals, or multi-step transformations that cannot be expressed using simple incremental entity vars.

See Incremental SQL Models Overview for conceptual information.

What are incremental SQL models?

An incremental SQL model is a SQL-template model that:

Dereferences a previous materialization of itself using this.DeRef() with pre_existing=true
Produces the same final result whether the previous state is available or not (idempotence)
Operates with two checkpoints:
- Current checkpoint: The checkpoint being produced now
- Baseline checkpoint: A previous checkpoint represented by the previous materialization

When the previous state is present, the model merges its prior state with new/changed inputs; otherwise it runs a full scan.

When and why to use incremental SQL models

Use incremental SQL models when:

Simple aggregations don’t work: Your use case requires stateful processing that cannot be expressed with SUM, MIN, MAX, or COUNT in your merge property
Compound aggregations don’t work: Your use case requires complex transformations beyond averages or arrays
You need activity windows: Tracking recent activity over time windows (like MAU/DAU)
You need window functions or composite key aggregations: Per-dimension aggregations or window-based calculations
You need multi-step transformations: Complex business logic that requires multiple steps or conditional processing

Incremental SQL models let you:

Reference previous computation results using this.DeRef()
Maintain stateful aggregations across runs
Implement complex business logic with full SQL control
Handle both incremental and full refresh modes

Tip: For simple aggregations like SUM, MIN, MAX, COUNT, prefer using Simple Aggregations with merge property. For averages and arrays, use Compound Aggregations.

Incremental SQL models are best for complex scenarios.

Real-world examples

Some common use cases that map to incremental SQL models are listed below:

Use case	Pattern	Example
Monthly Active Users (MAU)	Incremental Input Window Models	Count of users active in the past 30 days
Daily Active Users (DAU)	Incremental Input Window Models	Count of users active today
Account balances by wallet	Composite Key Models	Account balance per (user_id, wallet_id) composite key maintained across transactions - each user has multiple wallets with individual balances
Recent activity tracking	Incremental Input Window Models	Tracking recent interactions within a time window

Incremental SQL models subcategories

Incremental SQL models can be categorized into two main patterns:

Incremental input window models

When to use: Tracking recent activity or maintaining rolling windows of data.

Characteristics

Maintain a rolling window of recent interactions (e.g., last 90 days of activity)
Track activity over time windows (MAU, DAU, weekly active users)
Filter and maintain only recent data incrementally

Example: Monthly Active Users (MAU) or Daily Active Users (DAU) where you need to track which users were active in a recent time window.

Composite key models

When to use: Per-dimension aggregations where you need one value per (entity_id, dimension_id) composite key pair.

Characteristics

Maintain aggregations per (entity_id, dimension_id) composite key instead of just per entity_id
Handle multi-dimensional state with composite key granularity
Support complex logic where each entity has multiple related objects to track

Example: Account balances by wallet where each user (entity_id) has multiple wallets (dimension_id), and you need to maintain a separate balance for each wallet. This is a composite key pattern common in financial applications.

How incremental SQL models work

Incremental SQL models follow this workflow:

Reference previous state using this.DeRef() with pre_existing=true
Reference delta models that represent changes since the baseline checkpoint
Collect new rows from upstream sources that arrived since the baseline checkpoint
If previous state exists, merge it with new data; otherwise perform a full scan
Materialize the result as a table, view, or ephemeral model

Key advantages

Profiles handles the complexity for you:

Automatic creation: Delta models are automatically generated for append-only inputs
Automatic cleanup: Old materials are cleaned up based on retention periods
Automatic build order: Dependency resolution ensures correct execution order
No manual backfill required: Full refresh mode handles first run automatically
Per-model incremental base: If one entity var changes, others can still run incrementally
Efficient time management: Large changes may take longer, but subsequent runs remain fast

Basic template structure

models:
 - name: example_incremental_model
 model_type: sql_template
 model_spec:
 occurred_at_col: timestamp
 materialization:
 output_type: table
 single_sql: |
 {%- set rootIncrementalPreReqs = this.RootIncrementalPreReqs() -%}
 {%- set lastThis = this.DeRef(pre_existing=true, dependency="optional", checkpoint_name="baseline", prereqs=rootIncrementalPreReqs) -%}

 {% set events_delta = this.DeRef("inputs/events/incr_delta_tbl", dependency="coercive", baseline_name="baseline", prereqs=[lastThis]) %}

 {% set events_full = this.DeRef("inputs/events", prereqs=[events_delta.Except()]) %}

 {# All DeRef before this line. Note that DeRef should not be put in conditionals. DeRef mark dependencies — use prereqs for conditional dependencies #}

 {% if lastThis.IsEnabled() && events_delta.IsEnabled() %}
 -- Incremental mode: merge with baseline
 SELECT * FROM {{ lastThis }}
 UNION ALL
 SELECT * FROM {{ events_delta }}
 {% else %}
 -- Full refresh mode: process all data
 SELECT * FROM {{ events_full }}
 {% endif %}
 contract:
 is_append_only: false

Tip: Note the following:

Incremental SQL models work on append-only inputs: The upstream inputs must have contract: is_append_only: true to enable delta model generation.

Also works on append-only SQL models: Incremental SQL models can build on other append-only SQL models, even ephemeral ones.

Set is_append_only: false for incremental SQL models that merge or update data.

Set is_append_only: true only if your model truly only appends new rows without modifying existing ones.

Profiles also supports keeping the SQL template in another file and referencing it using {% exec %} {{ this.ReadFile("file/path") }} {% endexec %}.

Key concepts

This section explains the key concepts of incremental SQL models.

Previous state reference

Use this.DeRef() with pre_existing=true to reference the previous materialization of the current model:

{%- set lastThis = this.DeRef(
 pre_existing=true,
 dependency="optional",
 checkpoint_name="baseline") -%}

This looks for a previous materialization, uses it as a baseline checkpoint, and returns nil if no baseline exists (enabling full refresh mode).

See DeRef Reference for complete syntax and usage.

Collecting new data

When running incrementally, collect only new data that arrived since the baseline checkpoint. Profiles automatically handles this for append-only inputs when you reference them with the appropriate baseline context using baseline_name that matches the checkpoint_name used for lastThis.

Incremental vs. full refresh modes

Incremental mode (when baseline exists): Merges previous state with new data, processing only changes since the last checkpoint.
Full refresh mode (when no baseline exists): Processes all historical data, used on first run or when baseline is unavailable.

See the Checkpoints and Baselines guide for more information on these modes.

Example: Monthly Active Users (MAU)

This example demonstrates an incremental input window model pattern for tracking monthly active users, where an incremental SQL model maintains a rolling window of active days, and an entity var references it to compute MAU.

Input configuration (inputs.yaml):

inputs:
 - name: events
 table: PROD.ANALYTICS.EVENTS
 contract:
 is_append_only: true
 is_event_stream: true
 occurred_at_col: timestamp

Incremental SQL model (sql_models.yaml):

models:
 - name: recent_user_activity
 model_type: sql_template
 model_spec:
 occurred_at_col: timestamp
 materialization:
 output_type: table
 single_sql: |
 {%- set rootIncrementalPreReqs = this.RootIncrementalPreReqs() -%}
 {%- set lastThis = this.DeRef(pre_existing=true, dependency="optional", checkpoint_name="baseline", prereqs=rootIncrementalPreReqs) -%}

 {% set events_delta = this.DeRef("inputs/events/incr_delta_tbl", dependency="coercive", baseline_name="baseline", prereqs=[lastThis]) %}

 {% set events_full = this.DeRef("inputs/events", prereqs=[events_delta.Except()]) %}

 {# All DeRef before this line. Note that DeRef should not be put in conditionals. DeRef mark dependencies — use prereqs for conditional dependencies #}

 SELECT user_id, active_day
 FROM (
 {%- if lastThis.IsEnabled() %}
 -- Incremental mode: merge previous active days with new events
 SELECT DISTINCT user_id, CAST(timestamp AS DATE) AS active_day
 FROM (
 SELECT user_id, timestamp FROM {{ events_delta }}
 UNION ALL
 SELECT user_id, active_day AS timestamp FROM {{ lastThis }}
 )
 {%- else %}
 -- Full refresh mode: process all events
 SELECT DISTINCT user_id, CAST(timestamp AS DATE) AS active_day
 FROM {{ events_full }}
 {%- endif %}
 )
 WHERE active_day >= DATEADD('day', -365, {{ end_time_sql }})
 ids:
 - select: user_id
 type: user_id
 entity: user
 contract:
 is_append_only: false

This incremental SQL model has is_append_only: false because it maintains a rolling window that removes old data (older than 365 days). Incremental SQL models that merge or update data should set is_append_only: false — only set is_append_only: true if your model truly only appends new rows without modifying existing ones.

Entity variable (profiles.yaml):

- entity_var:
 name: monthly_active_users
 select: COUNT(DISTINCT user_id)
 from: models/recent_user_activity
 where: active_day >= DATEADD('day', -30, {{ end_time_sql }})
 description: Number of distinct users active in the past 30 days

How it works

Input configuration: The events input has contract: is_append_only: true, enabling incremental processing.
Incremental SQL model: The recent_user_activity model:
- References its previous materialization using lastThis with pre_existing=true
- Collects new events from events_delta when running incrementally
- Merges previous active days with new event dates using UNION ALL and DISTINCT
- Filters to maintain only the last 365 days of activity
- Handles full refresh mode when no baseline exists
Entity var reference: The monthly_active_users entity var references the incremental SQL model and filters to the past 30 days, counting distinct users from the pre-computed model.

This pattern is efficient because the SQL model maintains state incrementally (only processing new events), and the entity var performs a simple aggregation on the pre-computed model.

Troubleshooting

Issue	Solution
Model fails on first run	Ensure your model handles cases when `lastThis.IsEnabled()` is both false and true. Use `events_full` with `prereqs=[events_delta.Except()]` to ensure proper fallback when baseline doesn’t exist
New data not being processed	Verify that upstream inputs have `contract: is_append_only: true` configured and that your `baseline_name` matches the `checkpoint_name` used for `lastThis`
Incorrect merge results	Check that your merge logic correctly handles both existing and new entities. Use appropriate SQL joins, UNION ALL patterns, or ordering/de-duplication functions based on your use case
Performance not improving	Ensure your filtering logic correctly identifies only new data and that your merge logic is efficient. Verify that delta models are being used instead of full inputs
`DeRef` errors	See DeRef Reference for syntax and troubleshooting
SQL syntax errors	Note that SQL syntax varies by warehouse. `DATEADD` is Snowflake/Redshift syntax. For BigQuery, use `DATE_SUB`; for Databricks, use `date_sub`

Avoiding cycles and infinite regress

Incremental features are powerful but need to be used carefully to avoid cycles and infinite regress.

Important: All DeRef before conditionals

All DeRef calls must be placed before any conditional logic (if statements). Profiles needs to discover all dependencies during the dependency discovery phase, which happens before SQL execution. If DeRef is inside a conditional, Profiles cannot determine dependencies correctly.

{# CORRECT: All DeRef at the top #}
{% set lastThis = this.DeRef(pre_existing=true, dependency="optional") %}
{% set events_delta = this.DeRef("inputs/events/incr_delta_tbl", prereqs=[lastThis]) %}

{% if lastThis.IsEnabled() %}
 {# Use materials here #}
{% endif %}

{# WRONG: DeRef inside conditional - breaks dependency discovery #}
{% if some_condition %}
 {% set material = this.DeRef("some/model") %} {# ERROR! #}
{% endif %}

Common mistakes and how to avoid them:

Infinite regress: Self pre-existing DeRef must be optional

{# WRONG: mandatory self-reference creates infinite regress #}
{% set lastThis = this.DeRef(pre_existing=true, dependency="normal") %}

{# CORRECT: self-reference must be optional #}
{% set lastThis = this.DeRef(pre_existing=true, dependency="optional", checkpoint_name="baseline") %}

Why: Without dependency="optional", Profiles tries to create infinite historical chain of materials.

Cycle: Missing pre_existing=true flag

{# WRONG: missing pre_existing creates cycle #}
{% set lastThis = this.DeRef(checkpoint_name="baseline") %}

{# CORRECT: must specify pre_existing=true #}
{% set lastThis = this.DeRef(pre_existing=true, dependency="optional", checkpoint_name="baseline") %}

Why: Without pre_existing=true, the model depends on itself in the same time context, creating a cycle.

Inefficient: Delta derefs must be conditional

{# INEFFICIENT: delta runs even in full refresh mode #}
{% set events_delta = this.DeRef("inputs/events/incr_delta_tbl", baseline_name="baseline") %}

{# CORRECT: delta only runs when baseline exists #}
{% set lastThis = this.DeRef(pre_existing=true, dependency="optional", checkpoint_name="baseline") %}
{% set events_delta = this.DeRef("inputs/events/incr_delta_tbl", dependency="coercive", baseline_name="baseline", prereqs=[lastThis]) %}

Why: Without prereqs=[lastThis], delta model runs even during full refresh, wasting compute. Var tables may also run unnecessarily.

Profiles protections:

Profiles has built-in protections to catch these issues:

Cycle detection: Graph validation catches same-time-context cycles during dependency resolution
Infinite regress detection: If >10,000 temporary sequence numbers are created, Profiles throws an error with diagnostic information
Helpful error messages: Errors include the call chain, recent material allocations, and suggestions for fixing the issue

Debug tips:

Check that all self-referencing DeRefs have dependency="optional"
Verify all pre-existing references include pre_existing=true
Ensure delta model DeRefs include prereqs=[lastThis] to make them conditional
Use baseline_name matching the checkpoint_name for consistency

See more

Incremental SQL Models Overview: Conceptual introduction to incremental SQL models
How to Make Features Incremental: Step-by-step guide to convert features to incremental
DeRef Reference: Complete reference for this.DeRef() syntax and usage
Checkpoints and Baselines: Understanding checkpoints and how they enable incremental computation

Archive: How to Make Features Incremental

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks you through converting features to incremental mode in Profiles.

Prerequisites

Before making a feature incremental, make sure that:

Your input sources are append-only and have reliable occurred_at_col timestamps
The feature can be expressed using supported incremental patterns
You understand how Checkpoints and Baselines work in Profiles

Not all features can be incremental.

Features that require scanning all historical data (such as median or percentile calculations) cannot use incremental computation. For such cases, consider using Timegrains and bundling.

Step 1: Configure input contracts

Before making features incremental, you must configure all input sources and SQL models with the is_append_only contract. This applies to inputs and SQL models (which can also be ephemeral).

Verify append-only behavior

Before marking an input as append-only, verify that it truly only appends new rows and never updates or deletes existing ones:

Check business semantics: Event/fact tables (page views, purchases, clicks) are typically append-only. Dimension tables (user profiles, account settings) that update rows in place are not append-only.

Run a warehouse query: Check for duplicate primary keys that would indicate updates:

SELECT entity_id_column, COUNT(*)
FROM your_input_table
GROUP BY entity_id_column
HAVING COUNT(*) > 1;

Tip: Inspect results to distinguish genuine duplicate events from row updates.

Check downstream usage: If a downstream model selects the latest attribute value (like current subscription status), the input likely receives updates and should not be marked append-only.

When in doubt, treat an input as not append-only.

Incorrectly marking a mutable input as is_append_only: true leads to incorrect incremental results that are difficult to diagnose.

Input example

Add the following to each verified append-only input in your inputs.yaml:

inputs:
 - name: orders
 table: PROD.ANALYTICS.ORDERS
 contract:
 is_append_only: true
 is_event_stream: true
 occurred_at_col: order_timestamp

SQL model example

You can also create append-only SQL models that filter or transform inputs. Add this to your sql_models.yaml:

models:
 - name: purchase_events
 model_type: sql_template
 model_spec:
 occurred_at_col: timestamp
 materialization:
 output_type: table
 single_sql: |
 {% with events = this.DeRef("inputs/events") %}
 SELECT * FROM {{ events }}
 WHERE event_name = 'purchase'
 {% endwith %}
 contract:
 is_append_only: true
 is_event_stream: true

This SQL model filters events to only purchases and is marked as is_append_only: true, making it usable as an incremental source.

Ephemeral SQL models

Ephemeral SQL models (output_type: ephemeral) that sit on top of append-only inputs can also be marked as is_append_only: true. This is useful for filtering or transforming inputs without materializing intermediate tables.

Only mark an ephemeral model as append-only if:

All upstream inputs are append-only
The model only filters, renames, or transforms columns (no joins with mutable dimension tables)
The output is itself append-only (no aggregations that change with updates)

All inputs and SQL models used by incremental features must have contract: is_append_only: true. SQL models can also be ephemeral. Without this configuration, the project will throw an error when you try to run incremental features.

Complete example: Simple incremental feature

The following example shows a complete setup for a simple incremental feature that tracks total purchase value:

Input configuration (inputs.yaml):

inputs:
 - name: orders
 table: PROD.ANALYTICS.ORDERS
 contract:
 is_append_only: true
 is_event_stream: true
 occurred_at_col: order_timestamp

Entity var (profiles.yaml):

- entity_var:
 name: total_purchases
 select: SUM({{orders.value}})
 merge: SUM({{rowset.total_purchases}})
 from: inputs/orders
 description: Total purchase value across all orders

This example shows:

Input configured with contract: is_append_only: true
Entity var using SUM aggregation with merge: SUM({{rowset.total_purchases}})
Profiles automatically handles incremental computation

Step 2: Identify the feature pattern

Determine which incremental pattern your feature matches:

Simple aggregations: MIN, MAX, SUM, COUNT - See Simple Aggregations
Compound aggregations: AVG, MIN_BY/MAX_BY, arrays, booleans - See Compound Aggregations
Window function replacements: LAST_VALUE -> MAX_BY, FIRST_VALUE -> MIN_BY - See Replacing window functions
Complex patterns: Stateful processing, running totals, activity windows - See Incremental SQL Models

See Incremental Features for a pattern selection guide and detailed examples.

Step 3: Add the `merge` property

Add a merge: property to your entity_var definition in your profiles.yaml file. This property defines how to combine previous feature values with new data.

Simple aggregators

For MIN, MAX, SUM, COUNT, and similar functions:

- entity_var:
 name: first_event_time
 select: MIN({{events.timestamp}})
 merge: MIN({{rowset.first_event_time}})
 from: inputs/events
 description: Earliest timestamp of any event

Average using component features

For AVG, break it into sum and count components. Define three separate entity_var entries at the same level in your profiles.yaml:

- entity_var:
 name: sum_of_scores
 select: SUM({{game_events.score}})
 merge: SUM({{rowset.sum_of_scores}})
 from: inputs/game_events
 is_feature: false
 description: Sum of scores (used for average calculation)

- entity_var:
 name: count_of_scores
 select: COUNT({{game_events.score}})
 merge: SUM({{rowset.count_of_scores}})
 from: inputs/game_events
 is_feature: false
 description: Count of scores (used for average calculation)

- entity_var:
 name: avg_score
 select: "CASE WHEN {{user.count_of_scores}} > 0 THEN {{user.sum_of_scores}} / {{user.count_of_scores}} ELSE 0 END"
 description: Average score across all games

Features referencing other features

When a feature references another feature (like {{user.sum_of_scores}}), it doesn’t need a merge: clause if it’s just performing a calculation on incrementally maintained features. The referenced features (sum_of_scores and count_of_scores) maintain their incremental state, and the calculation (avg_score) is computed on-the-fly from those values.

Use incremental sources

You can also make features incremental by pointing to an incremental source (such as a delta model or incremental input window). Even if the entity_var SQL runs a full aggregation, the scan volume stays small because the source is already delta-restricted:

- entity_var:
 name: recent_purchases
 select: SUM({{recent_orders.value}})
 from: inputs/orders/incr_delta_tbl
 description: Purchases from recent data only

Step 4: Define merge logic

The merge: clause must match the aggregation function in select:. Common patterns:

Select function	Merge function
`MIN(...)`	`MIN({{rowset.feature_name}})`
`MAX(...)`	`MAX({{rowset.feature_name}})`
`SUM(...)`	`SUM({{rowset.feature_name}})`
`COUNT(...)`	`SUM({{rowset.feature_name}})`
`MAX_BY(val, ord)`	`MAX_BY({{rowset.feature}}, {{rowset.feature_by_param}})` + helper var
`MIN_BY(val, ord)`	`MIN_BY({{rowset.feature}}, {{rowset.feature_by_param}})` + helper var
`ARRAY_AGG(...)`	`ARRAY_UNION_AGG({{rowset.feature_name}})`
`BOOLOR_AGG(...)`	`BOOLOR_AGG({{rowset.feature_name}})`

For complete merge syntax, see Simple Aggregations and Compound Aggregations.

Which `merge` function should I use?

The merge function must match your aggregation type. For COUNT, always use SUM in the merge property (summing counts, not counting counts). See the table above for the correct merge function for each aggregation type.

Step 5: Use `merge_where` when needed

For complex merges, use merge_where: to restrict merging to only changed rows. Define separate entity_var entries at the same level in your profiles.yaml:

- entity_var:
 name: timestamp
 select: MAX({{game_events.timestamp}})
 merge: MAX({{rowset.timestamp}})
 from: inputs/game_events
 where: {{game_events.event_type}} = "BREAK"
 is_feature: false
 description: Timestamp of the most recent break event (used for score lookup)

- entity_var:
 name: score_at_last_break
 select: any_value({{game_events.score}})
 where: {{game_events.timestamp}} = {{user.timestamp}}
 merge: any_value({{game_events.score}})
 merge_where: {{rowset.timestamp}} = {{user.timestamp}}
 from: inputs/game_events
 description: Score at the most recent break event

The timestamp entity var is marked with is_feature: false since it’s an intermediate calculation used by score_at_last_break.

Tip: Mark helper entity vars with is_feature: false when they’re intermediate calculations used by other features. This prevents them from appearing as features in your output while still allowing them to be referenced by other entity vars.

When merge is not needed

Not every entity var requires a merge clause to participate in an incremental project:

Derived-only vars (no from clause) that reference other entity vars are recomputed from their incrementally maintained dependencies. Since they don’t read from input sources directly, they don’t need their own merge logic:
```
- entity_var:
 name: days_since_last_seen
 select: "DATEDIFF(day, {{user.last_seen}}, {{ end_time_sql }})"
 description: Days since the user was last seen
```

Vars from window-bounded incremental tables that read from an Incremental SQL Model maintaining a sliding window. The SQL model handles state incrementally, and the entity var recomputes from the bounded window each run:

- entity_var:
 name: active_days_in_past_7_days
 select: count(distinct active_day)
 from: models/recent_active_days
 where: active_day >= DATEADD(day, -7, {{ end_time_sql }})
 description: Days active in the last 7 days

Step 6: Test your implementation

Run pb run to verify the feature computes correctly
Compare incremental results with a full refresh to ensure accuracy
Monitor run times to verify improvement

The first run performs a full refresh to establish the baseline checkpoint. Subsequent runs use incremental computation.

Verifying incremental behavior

To verify your feature is running incrementally:

Check run times: Incremental runs should be faster than full refresh runs (after the first run)
Compare results: Run with incremental mode and compare results with a full refresh to ensure they match
Monitor checkpoint usage: Verify that checkpoints are being created and used in subsequent runs

Troubleshooting

Issue	Solution
Feature values are incorrect	Verify that your `merge:` clause correctly matches the aggregation logic Ensure that your input sources are append-only or have reliable timestamps Check that you are not missing edge cases in merge logic Compare incremental results with a full refresh to identify discrepancies
Performance not improving	Verify that your input sources are properly configured for incremental processing Check that delta models are being used correctly Ensure your data warehouse supports efficient timestamp filtering Monitor checkpoint creation and usage
Merge errors	Verify that all aggregators in `select:` have corresponding merge logic Check that merge functions match the aggregation type Ensure `merge_where:` conditions are correct if used Review Simple Aggregations and Compound Aggregations for correct syntax
The first run is slow	This is expected. The first run always performs a full refresh to establish the baseline checkpoint. Subsequent runs use incremental computation and should be faster.

See more

Explore Incremental Features for pattern selection guide
Learn about Simple Aggregations
Learn about Compound Aggregations
Learn about Incremental SQL Models
Learn about Checkpoints and Baselines

Archive: How to Migrate a Project to Incremental

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial walks you through migrating an entire Profiles project from non-incremental to incremental mode. It covers the systematic workflow: auditing your project, configuring inputs, converting features, eliminating input vars, and validating results.

For converting individual features, see How to Make Features Incremental.

Overview

An incremental Profiles project processes only new and changed data on each run, rather than reprocessing all historical data. This reduces compute costs, improves latency, and enables faster data freshness.

Migration follows a specific order of operations:

Audit your project
Verify and mark append-only inputs
Handle ephemeral SQL models
Convert entity vars (add merge clauses)
Eliminate input vars
Validate with --rebase_incremental

Prerequisites

Before starting migration, make sure that:

You have a working, non-incremental Profiles project that runs successfully
You understand Incremental Features concepts
You understand Checkpoints and Baselines
You have access to run queries against your data warehouse

1: Audit your project

Before making any changes, inventory your project to understand what needs to migrate.

Identify all components

Review your project files and list:

Resource	File name	Description
Inputs	`inputs.yaml`	All input sources and their table types
SQL models	`sql_models.yaml`	All SQL template models, noting which are ephemeral
Entity vars	`profiles.yaml`	All entity vars, their aggregation functions, and `from` sources
Input vars	`profiles.yaml`	All input vars (must be eliminated or converted)

Classify inputs

For each input, determine whether it is append-only or mutable:

Input type	Append-only	Examples
Event streams (facts)	Yes	Page views, purchases, clicks, logins
Slowly-changing dimensions	No	User profiles, account settings, subscription status
Snapshot tables	No	Daily balance snapshots, inventory levels

Check for input vars

Input vars scan all input data on every run and cannot be made incremental. If your project has input vars, you must convert them (see Step 5).

2: Verify and mark append-only inputs

For a model to participate in incremental processing, its upstream inputs must be append-only. This step ensures you correctly identify and mark them.

Verify append-only behavior

Use these three checks to verify an input is truly append-only:

1. Business semantics check

Ask: Is the underlying source a fact table (events) or a slowly-changing dimension?

Fact tables (events, transactions, clicks) are typically append-only — rows represent immutable events
Dimension tables (user profiles, account settings) are not append-only — rows are updated in place

2. Warehouse verification query

Run this query against your warehouse to check for duplicate or updated rows:

SELECT entity_id_column, COUNT(*)
FROM your_input_table
GROUP BY entity_id_column
HAVING COUNT(*) > 1;

If this returns rows, inspect whether duplicates represent distinct events (append-only) or updates to the same record (not append-only).

3. Downstream usage inspection

Check how the input is used in SQL models and entity vars. If a model selects the “latest” attribute value (such as current subscription status or country), that suggests the input receives updates and is NOT append-only.

When in doubt about whether an input is append-only, treat it as not append-only.

Incorrectly marking a mutable input as append-only leads to incorrect incremental results that are difficult to diagnose.

Mark append-only inputs

For each verified append-only input, add the contract:

inputs:
 - name: events
 table: PROD.ANALYTICS.EVENTS
 contract:
 is_append_only: true
 is_event_stream: true
 occurred_at_col: event_timestamp

Some key fields in the contract are listed below:

Field	Description
`is_append_only`	Enables delta model generation for incremental processing
`is_event_stream`	Indicates this is an event stream
`occurred_at_col`	The timestamp column used for time-range filtering. Required for incremental inputs.

3: Handle ephemeral SQL models

Ephemeral SQL models defined on top of append-only inputs can also participate in incremental processing.

Identify eligible ephemeral models

Look for SQL models with output_type: ephemeral that:

Read from append-only inputs
Only filter, rename, or transform columns (no aggregations or joins with mutable sources)
Produce output that is itself append-only

Mark ephemeral models as append-only

models:
 - name: purchase_events
 model_type: sql_template
 model_spec:
 occurred_at_col: timestamp
 materialization:
 output_type: ephemeral
 single_sql: |
 {% with events = this.DeRef("inputs/events") %}
 SELECT * FROM {{ events }}
 WHERE event_name = 'purchase'
 {% endwith %}
 contract:
 is_append_only: true
 is_event_stream: true

Only mark an ephemeral model as is_append_only: true if all of its upstream inputs are also append-only and the model’s SQL only filters or transforms without introducing mutations (no joins with mutable dimension tables, no aggregations that change with updates).

4: Convert entity vars

Add merge clauses to entity vars to enable incremental computation. The merge clause defines how to combine previous feature values with new calculations.

Choose the right pattern

Use the Incremental Features decision guide to determine which pattern applies:

Pattern	When to use	Reference
Simple aggregation	`SUM`, `MIN`, `MAX`, `COUNT`	Simple Aggregations
Compound aggregation	`AVG`, `WEIGHTED_AVG`, `MIN_BY`/`MAX_BY`, arrays, booleans	Compound Aggregations
Incremental SQL model	Stateful processing, rolling windows, multi-step logic	Incremental SQL Models

Quick reference: Common migrations

Simple aggregations: Add a matching merge:

# Before
- entity_var:
 name: first_purchase_time
 select: min(timestamp)
 from: inputs/orders

# After
- entity_var:
 name: first_purchase_time
 select: MIN({{orders.timestamp}})
 merge: MIN({{rowset.first_purchase_time}})
 from: inputs/orders

Window functions: Replace with MAX_BY/MIN_BY:

# Before (non-incremental window function)
- entity_var:
 name: last_login_platform
 select: last_value(platform)
 from: inputs/logins
 window:
 order_by:
 - timestamp

# After (incremental with MAX_BY + helper)
- entity_var:
 name: last_login_platform
 select: MAX_BY({{logins.platform}}, {{logins.timestamp}})
 merge: MAX_BY({{rowset.last_login_platform}}, {{rowset.last_login_platform_by_param}})
 from: inputs/logins

- entity_var:
 name: last_login_platform_by_param
 select: MAX({{logins.timestamp}})
 merge: MAX({{rowset.last_login_platform_by_param}})
 from: inputs/logins
 is_feature: false

See Compound Aggregations for the full MIN_BY/MAX_BY pattern and _by_param helper documentation.

When merge is not needed

Not every entity var requires a merge clause:

Derived-only vars that reference other entity vars (no from clause) are recomputed from their incrementally maintained dependencies:

- entity_var:
 name: days_since_last_seen
 select: "DATEDIFF(day, {{user.last_seen}}, {{ end_time_sql }})"
 description: Days since the user was last seen

Vars from window-bounded incremental tables that read from an incremental SQL model with a sliding window. The model already maintains state incrementally, and the var recomputes from the bounded window each run:
```
- entity_var:
 name: active_days_in_past_7_days
 select: count(distinct active_day)
 from: models/recent_active_days
 where: active_day >= DATEADD(day, -7, {{ end_time_sql }})
```

5: Eliminate input vars

Input vars scan all data on every run and block incremental processing. You must convert or replace every input var before the project can run incrementally.

Decision tree

For each input var, follow this decision tree:

Input Var
+-- Can this logic be expressed as an entity var?
| +-- Yes: Convert to entity_var with merge clause
| | (common for window functions like LAST_VALUE, FIRST_VALUE)
| +-- No
| +-- Can the input var be replaced by a SQL model?
| | +-- Yes: Create a SQL model (possibly incremental) and
| | | reference it from entity vars
| | +-- No
| | +-- Create an incremental SQL model with full
| | DeRef pattern

Example: Convert input var to entity var

This is the most common conversion. Window functions like LAST_VALUE and FIRST_VALUE can typically be replaced with MAX_BY and MIN_BY:

# Before (input var with window function)
- input_var:
 name: last_known_city
 select: last_value(city)
 from: inputs/events
 window:
 partition_by:
 - user_id
 order_by:
 - timestamp
 where: city IS NOT NULL

# After (entity var with MAX_BY)
- entity_var:
 name: last_known_city
 select: MAX_BY({{events.city}}, {{events.timestamp}})
 merge: MAX_BY({{rowset.last_known_city}}, {{rowset.last_known_city_by_param}})
 from: inputs/events
 where: {{events.city}} IS NOT NULL

- entity_var:
 name: last_known_city_by_param
 select: MAX({{events.timestamp}})
 merge: MAX({{rowset.last_known_city_by_param}})
 from: inputs/events
 where: {{events.city}} IS NOT NULL
 is_feature: false

Example: Replace input var with incremental SQL model

When input var logic is too complex for an entity var (multi-step transformations, complex joins), create an incremental SQL model:

# In sql_models.yaml
models:
 - name: recent_active_days
 model_type: sql_template
 model_spec:
 occurred_at_col: timestamp
 materialization:
 output_type: table
 single_sql: |
 {%- set rootIncrementalPreReqs = this.RootIncrementalPreReqs() -%}
 {%- set lastThis = this.DeRef(pre_existing=true, dependency="optional", checkpoint_name="baseline", prereqs=rootIncrementalPreReqs) -%}
 {% set events_delta = this.DeRef("inputs/events/incr_delta_tbl", dependency="coercive", baseline_name="baseline", prereqs=[lastThis]) %}
 {% set events_full = this.DeRef("inputs/events", prereqs=[events_delta.Except()]) %}

 SELECT user_id, active_day
 FROM (
 {%- if lastThis.IsEnabled() %}
 SELECT user_id, active_day FROM {{ lastThis }}
 UNION ALL
 SELECT DISTINCT user_id, CAST(timestamp AS DATE) AS active_day
 FROM {{ events_delta }}
 {%- else %}
 SELECT DISTINCT user_id, CAST(timestamp AS DATE) AS active_day
 FROM {{ events_full }}
 {%- endif %}
 )
 WHERE active_day >= DATEADD('day', -90, {{ end_time_sql }})
 ids:
 - select: user_id
 type: user_id
 entity: user
 contract:
 is_append_only: false

Then reference the model from an entity var:

- entity_var:
 name: active_days_last_30
 select: COUNT(DISTINCT active_day)
 from: models/recent_active_days
 where: active_day >= DATEADD('day', -30, {{ end_time_sql }})

6: Validate with `rebase_incremental`

After completing the migration, validate that incremental results match full refresh results.

Validation workflow

Run a full refresh to establish a baseline:
```
pb run
```
Run again to trigger incremental processing (if new data has arrived):
```
pb run
```
Run with --rebase_incremental to force a full rebuild and compare:
```
pb run --rebase_incremental
```
Compare results: Verify that incremental and full refresh outputs match by comparing row counts, aggregated values, and spot-checking individual entities.

Checklist

Row counts: Entity var tables should have the same number of rows
Value consistency: Feature values should match between incremental and full refresh runs
Performance: Incremental runs should be faster than the full refresh baseline (after the first run)
No errors: Check logs for warnings or errors related to incremental processing

The first run after migration always performs a full refresh to establish the baseline checkpoint. Performance improvements are visible starting from the second run.

Migration checklist

This section contains a detailed migration checklist to convert your project to incremental mode.

Pre-migration

Inventory all inputs, SQL models, entity vars, and input vars
Classify each input as append-only or mutable
Verify append-only inputs with warehouse queries
Identify all input vars that need conversion

During migration

Mark all append-only inputs with is_append_only: true and occurred_at_col
Mark eligible ephemeral SQL models with is_append_only: true
Add merge clauses to all entity vars that read from append-only sources
Replace window functions with MAX_BY/MIN_BY patterns
Convert or replace all input vars
Create incremental SQL models for complex patterns
Use {{ end_time_sql }} instead of CURRENT_DATE or CURRENT_TIMESTAMP() in rolling-window filters

Post-migration

Run pb run to establish baseline
Run pb run again to verify incremental processing
Run pb run --rebase_incremental and compare results
Verify performance improvement on subsequent runs
Remove any unused input vars or deprecated models

Common pitfalls

Pitfall	Description	Solution
Missing `merge` clause	Entity vars without `merge` cannot run incrementally	Add appropriate `merge` clause matching the `select` aggregation
Incorrect merge function	Using `COUNT` instead of `SUM` for count merges	For `COUNT(...)` in `select`, use `SUM({{rowset.feature}})` in `merge`
Missing `_by_param` helper	Using `MAX_BY`/`MIN_BY` without the ordering companion var	Always define a `_by_param` helper var with `is_feature: false`
Mutable input marked append-only	Dimension table incorrectly marked as `is_append_only: true`	Verify append-only behavior before marking; dimension tables should NOT be append-only
Using `CURRENT_DATE` in filters	Non-deterministic results across re-runs	Use `{{ end_time_sql }}` instead
Input vars still present	Input vars block incremental processing	Convert to entity vars or incremental SQL models
Non-composable merge function	Using `AVG` directly in `merge`	Decompose into `SUM`/`COUNT` components via Compound Aggregations

See more

How to Make Features Incremental: Feature-level migration guide
Incremental Features: Pattern selection guide
Simple Aggregations: SUM, MIN, MAX, COUNT patterns
Compound Aggregations: AVG, MIN_BY/MAX_BY, array, boolean patterns
Incremental SQL Models: Stateful processing patterns
Input Var: Input var reference and migration guidance

Archive: How to Use Checkpoints

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains how to reference and use checkpoints when building incremental features and SQL models.

Overview

Checkpoints represent point-in-time snapshots of warehouse data and their computed outputs from previous runs, enabling incremental computation. How you reference checkpoints depends on whether you’re using incremental entity vars or incremental SQL models.

See Checkpoints and Baselines for conceptual information.

Usage with incremental entity vars

When using a merge: clause in an incremental entity var, Profiles uses the checkpoint implicitly. You don’t need to explicitly reference the checkpoint.

Example

- entity_var:
 name: total_purchases
 select: SUM({{orders.value}})
 merge: SUM({{rowset.total_purchases}})
 from: inputs/orders

How it works

The {{rowset.total_purchases}} reference automatically pulls the value from the baseline checkpoint. Profiles:

Retrieves the previously computed value from the baseline checkpoint
Computes the delta (new rows since the baseline)
Merges them using the merge property

The {{rowset.}} syntax refers to the changed rowset from the baseline checkpoint, giving you access to previous checkpoint values during the merge operation.

Usage with incremental SQL models

When building an incremental SQL model, you must reference the previous checkpoint explicitly using this.DeRef() with pre_existing=true.

Reference past version of current model

To reference the previous materialization of the current model:

{%- set rootIncrementalPreReqs = this.RootIncrementalPreReqs() -%}
{%- set lastThis = this.DeRef(pre_existing=true, dependency="optional", checkpoint_name="baseline", prereqs=rootIncrementalPreReqs) -%}

{# All DeRef before this line. Note that DeRef should not be put in conditionals. DeRef mark dependencies — use prereqs for conditional dependencies #}

{% if lastThis.IsEnabled() %}
 -- Incremental mode: merge with baseline
 SELECT * FROM {{ lastThis }}
 UNION ALL
 SELECT * FROM {{ events_delta }}
{% else %}
 -- Full refresh mode: process all data
 SELECT * FROM {{ this.DeRef("inputs/events") }}
{% endif %}

Reference another model’s previous materialization

To reference another model’s previous materialization:

{%- set rootIncrementalPreReqs = this.RootIncrementalPreReqs() -%}
{%- set previousModel = this.DeRef("models/previous_model", pre_existing=true, dependency="optional", prereqs=rootIncrementalPreReqs) -%}

Key points

Use pre_existing=true to reference previous materializations. This indicates that the referenced material will NOT be created in the current run. If it doesn’t exist, it will be disabled. In SQL templates, you can check if the material is enabled using <material>.IsEnabled().
Use dependency="optional" to handle cases where no baseline exists
Always check if <reference>.IsEnabled() is true/false to support full refresh mode (when baseline doesn’t exist, IsEnabled() returns false)
Use checkpoint_name to specify which baseline to use (defaults to "baseline")

See DeRef Reference for complete syntax and usage.

Delta models

Profiles automatically creates delta models for append-only inputs at the path /path/to/input/incr_delta_tbl. These models contain only rows that arrived after the baseline checkpoint timestamp.

Reference delta models

Reference delta models in your incremental SQL templates:

{%- set lastThis = this.DeRef(
 pre_existing=true,
 dependency="optional",
 checkpoint_name="baseline") -%}

{% set events_delta = this.DeRef("inputs/events/incr_delta_tbl",
 dependency="coercive",
 baseline_name="baseline",
 prereqs=[lastThis]) %}

{% set events_full = this.DeRef("inputs/events",
 prereqs=[events_delta.Except()]) %}

{# All DeRef before this line. Note that DeRef should not be put in conditionals. DeRef mark dependencies — use prereqs for conditional dependencies #}

{% if lastThis.IsEnabled() %}
 -- Incremental mode: use delta
 SELECT * FROM {{ events_delta }}
{% else %}
 -- Full refresh mode: use full input
 SELECT * FROM {{ events_full }}
{% endif %}

How delta models work

Delta models contain only rows that arrived after the baseline checkpoint timestamp
They’re automatically enabled when a baseline exists, disabled otherwise
Use prereqs=[lastThis] to ensure the delta model is created only when a baseline exists
Use baseline_name to specify which baseline to use for delta computation (must match checkpoint_name)

Tip: Always declare all DeRefs together at the top of your template, not inside conditionals. Use prereqs to chain conditional evaluation instead.

Checkpoint management

Profiles automatically creates checkpoints after each successful run. Each checkpoint is identified by a sequence number and associated with specific time boundaries. Model output tables reference their checkpoint through their materialized table names (e.g., Material_<modelname>_<modelhash>_<checkpoint_seqno>).

Troubleshooting

Issue	Solution
No baseline checkpoint available	Profiles automatically performs a full refresh to create the baseline. Ensure regular runs to maintain baseline availability.
Baseline checkpoint incompatible	When new entity vars are added or existing definitions change, Profiles automatically discovers the model hash change and runs a full refresh.
Checkpoints are too old	Profiles can use older checkpoints, but performance may degrade. Maintain regular run schedules.
Materials consume excessive storage	Profiles automatically cleans up old materials based on your model retention periods. Review retention settings to adjust cleanup behavior.
`pre_existing=true` returns `nil`	This is expected on first run or when no baseline exists. Always handle this case with conditional logic for full refresh mode.
Delta model not available	Ensure your input has `contract: is_append_only: true` configured. Delta models are only created for append-only inputs.

See more

Checkpoints and Baselines: Conceptual overview
Incremental SQL Models
DeRef Reference
How to Make Features Incremental

RudderStack Docs – Incremental Features

Archive: Simple Aggregations

Overview

Motivation: When and why to use simple aggregations

Real-world examples

Supported simple aggregators

SUM

MIN

MAX

COUNT

merge property syntax

Basic syntax

Merge function reference

Rowset reference

When simple aggregations don’t work

merge_where for conditional merging

Best practices

Troubleshooting

See more

Archive: Compound Aggregations

Overview

Motivation: When and why to use compound aggregations

Real-world examples

Average (AVG) using component features

Example: Average order value

Workflow

Weighted Average (WEIGHTED_AVG) using component features

Example: Weighted average product rating

Workflow

Array aggregations

Example: List of countries visited

Array merge functions

MIN_BY / MAX_BY with ordering helper

Example: Most recent login platform

Example: First purchase store

MIN_BY / MAX_BY merge reference

When you can skip the _by_param helper

Boolean aggregations

Example: Has user ever enabled push notifications

Boolean merge reference

Replace window functions

Common replacements

Example: Replace LAST_VALUE with MAX_BY

Example: Replace FIRST_VALUE with MIN_BY

When compound aggregations don’t work

Best practices

Troubleshooting

See more

Archive: Incremental SQL Models

Overview

What are incremental SQL models?

When and why to use incremental SQL models

Real-world examples

Incremental SQL models subcategories

Incremental input window models

Composite key models

How incremental SQL models work

Key advantages

Basic template structure

Key concepts

Previous state reference

Collecting new data

Incremental vs. full refresh modes

Example: Monthly Active Users (MAU)

How it works

Troubleshooting

Avoiding cycles and infinite regress

See more

Archive: How to Make Features Incremental

Prerequisites

Step 1: Configure input contracts

Verify append-only behavior

Input example

SQL model example

Ephemeral SQL models

Complete example: Simple incremental feature

Step 2: Identify the feature pattern

Step 3: Add the merge property

Simple aggregators

Average using component features

`merge` property syntax

`merge_where` for conditional merging

When you can skip the `_by_param` helper

Step 3: Add the `merge` property

Which `merge` function should I use?

Step 5: Use `merge_where` when needed

6: Validate with `rebase_incremental`