You are viewing documentation for an older version.

Click here to view the latest documentation.

Incremental Features Beta

Making a feature incremental improves performance and reduces costs by processing only new data instead of recomputing everything from scratch.

See Incremental Features Overview for more information.

Quick decision guide

Follow this decision tree if you’re unsure which approach to use:

Understand composable functions

Composable functions can be computed incrementally by combining results from different time periods:

• SUM, MIN, MAX, COUNT are composable - they can merge results across checkpoints
• AVG is NOT composable (you cannot average two averages correctly without knowing the counts)

Compound composable functions can be expressed as combinations of simple composable functions:

• AVG(x) = SUM(x) / COUNT(x) - requires two composable components
• WEIGHTED_AVG(x, w) = SUM(x * w) / SUM(w) - requires two composable components

This distinction matters because composable functions can use simple merge logic, while non-composable functions like AVG require compound aggregations.

Simple aggregations

Best for: Direct aggregations using a single composable merge function

When to use

• Computing totals, counts, min/max values
• Using SUM, MIN, MAX, or COUNT
• Straightforward merge operations (for example, sum of previous + new, min/max of previous and new)

Why start here

• Minimal configuration (just add a merge property)
• Predictable behavior
• Best performance
• Works across all warehouses

Use cases

Quick example

- entity_var:
    name: total_purchases
    select: SUM({{orders.value}})
    merge: SUM({{rowset.total_purchases}})  # ← Key: merge mirrors select
    from: inputs/orders

Detailed reference

See Simple Aggregations for implementation and detailed examples.

Compound aggregations

Best for: Metrics combining multiple simple aggregations

When to use

• Need to calculate averages (sum ÷ count)
• Combining multiple aggregations
• Array operations (union, distinct, sort)

How it works

This approach breaks down complex metrics into simple components, then combines them:

# Components
- entity_var:
    name: order_value_sum
    select: SUM({{orders.value}})
    merge: SUM({{rowset.order_value_sum}})

- entity_var:
    name: order_count
    select: COUNT(*)
    merge: SUM({{rowset.order_count}})

# Combination
- entity_var:
    name: avg_order_value
    select: "{{user.order_value_sum}} / NULLIF({{user.order_count}}, 0)"

Use cases

Detailed reference

See Compound Aggregations for detailed examples and implementation.

Incremental SQL models

Best for: Stateful processing and advanced transformations

When to use

Use this approach only when entity vars cannot express your logic. For example:

• Tracking activity windows (MAU/DAU with rolling dates)
• Maintaining running balances or state
• Multi-step transformations with conditional logic
• Complex business rules requiring temporary tables

How it works

Reference previous model state and merge with new data:

-- Previous state
{{#with this.DeRef()}}
  SELECT * FROM {{this}}
{{/with}}

UNION ALL

-- New incremental data
SELECT * FROM new_data

Use cases

Detailed reference

See Incremental SQL Models for detailed examples and implementation.

Quick reference

Start with simple aggregations first. If your use case doesn’t fit, move to compound aggregations. Use incremental SQL models only when simple and compound aggregations cannot express your requirements.

See more

• How to Make Features Incremental: Feature-level migration guide
• Entity Vars
• Features
• Checkpoints and Baselines