RudderStack Docs – Incremental Features

Archive: Checkpoints and Baselines

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains checkpoints and baselines and how they enable incremental feature computation.

Overview

Checkpoints represent point-in-time snapshots of all Profiles inputs and their computed outputs. Each checkpoint is identified by a sequence number and references a specific time boundary. When running incrementally, Profiles builds models using:

Stored computation from a baseline checkpoint (a previous successful run)
New warehouse data that arrived after the baseline checkpoint’s timestamp

The baseline checkpoint provides the reference point for identifying what data has changed and what computations can be reused.

What are checkpoints?

A checkpoint is a conceptual snapshot of all Profiles inputs data that landed before a specific timestamp, along with all computed outputs generated from that data. Profiles uses checkpoints to:

Reference a specific point-in-time state of your warehouse data
Identify new data that has arrived since the last checkpoint
Enable incremental computation by reusing previous results

Each checkpoint is identified by a unique sequence number (seqno). Every model output table is associated with a checkpoint and follows the naming pattern Material_<modelname>_<modelhash>_<checkpoint_seqno>.

Current checkpoint

The current checkpoint is the checkpoint being produced during the current run.

Baseline checkpoint

The baseline checkpoint is a previous checkpoint used as a reference point for incremental computation. In incremental runs, each model selects a named baseline to use. Once a baseline corresponding to a particular (checkpoint seqno, baseline name) pair is chosen, it remains fixed for that and all subsequent runs.

When Profiles runs incrementally, it:

Selects a baseline checkpoint for each incrementally-run model
Loads computed values from the baseline checkpoint’s model output tables
Identifies new data that has arrived since the baseline
Merges baseline values with new calculations
Produces the current checkpoint

How checkpoints work

This section explains how checkpoints and baselines work in Profiles.

Checkpoint creation

Profiles automatically creates checkpoints after each successful run:

Checkpoint assignment: A new sequence number (seqno) is assigned to represent this checkpoint
During run: Profiles computes feature values from your warehouse data up to the checkpoint’s end timestamp
After computation: Model output tables are materialized with names containing the checkpoint seqno (e.g., Material_user_features_a1b2c3_42)
Checkpoint registration: The checkpoint seqno, along with its time boundaries and baseline references, is recorded in the registry
Next run: The checkpoint becomes available as a baseline for subsequent incremental runs

Baseline selection

When running incrementally, each model independently:

Identifies available baselines (previous checkpoints with matching time boundaries)
Selects an appropriate checkpoint as the baseline for the named baseline context
Once selected, the baseline (checkpoint seqno, baseline name) pair remains fixed and is never changed
Delta models and incremental models then compute changes since the baseline
New calculations are merged with baseline values to produce the current checkpoint output

Incremental vs full refresh

This section explains the difference between incremental and full refresh modes in Profiles.

Incremental mode

When a baseline checkpoint is available, each incremental model:

References the baseline checkpoint’s model output table (e.g., Material_user_features_a1b2c3_38 for baseline seqno 38)
Processes only new warehouse data that has arrived since the baseline checkpoint’s end timestamp
Merges baseline values with new calculations using your merge logic
Produces updated feature values in a new output table for the current checkpoint (e.g., Material_user_features_a1b2c3_42 for current seqno 42)

Full refresh mode

When no baseline checkpoint is available, Profiles:

Processes all data from scratch
Computes all feature values from the beginning
Creates a new checkpoint for future incremental runs

The first run of an incremental feature always performs a full refresh to establish the baseline checkpoint. Subsequent runs use incremental computation.

See more

How to Use Checkpoints: Implementation guide for referencing checkpoints
Incremental Features Overview
Make Features Incremental

Archive: Incremental SQL Models

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains incremental SQL models and how they enable incremental processing for complex stateful patterns.

Overview

Incremental SQL models are SQL models that dereference their own previous materialization to enable incremental processing. Incremental SQL models are the raw foundation of incremental features. They are general-purpose and enable any incremental SQL pattern, including advanced use cases like windowed incremental features that require complex stateful processing beyond what simple incremental entity vars can express.

Instead of recalculating from all historical data on every run, incremental SQL models reuse previously computed results and only process new data that has arrived since the last checkpoint.

What are incremental SQL models?

An incremental SQL model is a SQL-template model that:

Uses this.DeRef() with pre_existing=true to reference a previous materialization of itself
Merges previous state with new data instead of recalculating from scratch
Handles both incremental processing (when baseline exists) and full refresh (when no baseline exists)

Unlike regular SQL models that process all data on each run, incremental SQL models process only new data that arrived since the last checkpoint and merge it with previous state.

How incremental SQL models work

When you define an incremental SQL model, Profiles stores previous materializations as checkpoints, identifies new data that has arrived since the baseline checkpoint, merges the previous state with new data using your SQL logic, and produces updated results that become the new checkpoint.

Mandatory requirement:

All inputs used by incremental SQL models must have contract: is_append_only: true configured in the input’s contract. Without this configuration, delta models cannot be generated and incremental processing will not work.

See Incremental SQL Models Developer Documentation for configuration details.

When to use incremental SQL models

Use incremental SQL models when:

You want to reduce warehouse costs when processing large historical datasets (millions or billions of rows)
New data arrives regularly (daily, hourly, or more frequently)
The volume of new data is significantly smaller than your total dataset
Your use case requires stateful processing that cannot be expressed with simple aggregations

Tip: For simple aggregations like SUM, MIN, MAX, COUNT, prefer using Simple Aggregations with merge property. For averages and arrays, use Compound Aggregations.

Incremental SQL models are best for complex scenarios requiring stateful processing.

Relationship to other Profiles features

Incremental SQL models work alongside other Profiles capabilities:

SQL Models: Incremental SQL models are specialized SQL models that dereference their own past, using the same sql_models.yaml structure and model_spec configuration as regular SQL models. Both regular and incremental SQL models can be materialized as tables, views, or ephemeral sources.
Incremental Features: Incremental SQL models enable incremental processing for complex patterns that cannot be expressed using simple incremental entity vars.
Checkpoints: Incremental SQL models rely on checkpoints to reference previous materializations and enable incremental processing.

Supported patterns

Incremental SQL models support two main pattern categories:

Pattern	Description
Incremental input window models	Tracking recent activity or maintaining rolling windows of data (e.g., MAU/DAU)
Composite key models	Per-dimension aggregations where you need one value per (entity_id, dimension_id) composite key pair

RudderStack Docs – Incremental Features

Archive: Checkpoints and Baselines

Overview

What are checkpoints?

Current checkpoint

Baseline checkpoint

How checkpoints work

Checkpoint creation

Baseline selection

Incremental vs full refresh

Incremental mode

Full refresh mode

See more

Archive: Incremental SQL Models

Overview

What are incremental SQL models?

How incremental SQL models work

When to use incremental SQL models

Relationship to other Profiles features

Supported patterns

See also