RudderStack Docs – Manage SQL Models using Rudder CLI

Archive: Create New SQL Model Resource using Rudder CLI

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks you through the process of creating a new SQL model resource from scratch using Rudder CLI.

Prerequisites

Rudder CLI tool installed and authenticated with your workspace-level Service Access Token
A Rudder CLI project directory for storing the SQL model YAML resources
Important: At least one Reverse ETL source in the RudderStack dashboard using the data warehouse you want to query — RudderStack uses the connection credentials used to set up the warehouse account this source to fetch the account ID. An example is shown below:

List Reverse ETL accounts

Before creating a SQL model resource, identify the account ID for the warehouse you want to connect to.

Use the following command to list all accounts linked to your workspace:

rudder-cli workspace accounts list

This command displays all available accounts along with their IDs. Make note of the account ID that corresponds to your target warehouse — you will need this for the resource configuration.

Create the resource file

Create a new YAML file in your project folder containing the SQL model configuration. This file should have the following structure:

version: "rudder/v0.1"
kind: "retl-source-sql-model"
metadata:
 name: "product-purchases-model"
spec:
 id: "product-purchases-model"
 display_name: "Product Purchases SQL Model"
 primary_key: "user_id"
 description: "SQL model for product purchases"
 source_definition: "postgres"
 enabled: true
 account_id: "2rzPI1ARibivIvNH5DSpAKGTATy"
 sql: |
 SELECT
 user_id,
 event_name,
 event_type,
 timestamp,
 properties,
 pruduct_id
 FROM
 user_events
 WHERE
 timestamp >= CURRENT_DATE() - INTERVAL 180 DAY
 ORDER BY
 timestamp DESC

See the SQL Model YAML reference for more details on the parameters.

SQL query options

RudderStack provides two options for defining your SQL query:

Inline SQL

sql: |
 SELECT
 user_id,
 event_name
 FROM
 user_events

External file reference

file: "./queries/my-query.sql"

You must use exactly one of sql or file — they are mutually exclusive.

Supported sources

The source_definition field in the spec parameter supports the below warehouses:

Warehouse	`source_definition` value
PostgreSQL	`postgres`
MySQL	`mysql`
Snowflake	`snowflake`
Redshift	`redshift`
BigQuery	`bigquery`
Databricks	`databricks`
Trino	`trino`

Apply the SQL model

Once you have created the resource file, use the following commands to validate and create the SQL model:

Validate configuration

rudder-cli validate -l path/to/project

Dry run

rudder-cli apply -l path/to/project --dry-run

The above command lists the changes that would be applied without actually making them.

Apply changes

rudder-cli apply -l path/to/project

After successful application, the SQL model will be created in your workspace and you can manage it through the CLI.

Update the SQL model

You can modify any field in the YAML file of your existing SQL model — including source_definition.

Changing the id field will delete the existing resource and create a new one.

To apply the updates, repeat the validation and apply commands listed above:

rudder-cli validate -l path/to/project
rudder-cli apply -l path/to/project --dry-run # Optional: Preview changes
rudder-cli apply -l path/to/project

Best practices

Use external files: For complex queries, use the file option to keep your SQL in separate .sql files for better syntax highlighting and version control.
Descriptive IDs: Choose clear, descriptive IDs that reflect the model’s purpose.
Test first: Always run the validation and dry-run commands before applying any changes.

Next steps

Archive: Import Existing SQL Model Resource using Rudder CLI

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains how to import existing SQL model resources from your workspace into your CLI project. This allows you to bring existing resources under version control and manage them through Git workflows.

Prerequisites

Rudder CLI tool installed and authenticated with your access token
A CLI project directory for storing the SQL model YAML resources
Existing SQL model resources in your workspace

List available SQL model resources

First, identify which SQL model resources are available in your workspace for import:

rudder-cli workspace retl-sources list

This command lists all SQL model resources in your workspace. Make note of the source_id for the resource you want to import — you will need this in the next step.

Import the resource

Use the import command to bring an existing resource into your CLI project — this command creates a YAML file that describes the imported resource.

RudderStack provides two ways to import an existing resource:

Import the resource with inline SQL
Import the resource with an external SQL file

Import the resource with inline SQL

To import the resource with inline SQL, use the import command without the --sql-location flag, as shown:

rudder-cli import retl-sources --local-id api-test-10 --remote-id 2zMxxCKnGqkQLHffzCCIAvvxBso -l ./path/to/project

By default, the generated file will have the following format with inline SQL:

version: "rudder/v0.1"
kind: "retl-source-sql-model"
metadata:
 import:
 workspaces:
 - workspace_id: "2dgUfFqnI6LFhlDmYPd8mA4Xrnc"
 resources:
 - local_id: "api-test-10"
 remote_id: "2zMwvwOG8zYJoJRHdmtN2ka1VUY"
 name: "api-test-10"
spec:
 account_id: "2rzPI1ARibivIvNH5DSpAKGTATy"
 description: ""
 display_name: "api-test-10"
 enabled: true
 id: "api-test-10"
 primary_key: "id"
 source_definition: "postgres"
 sql: |
 SELECT
 user_id,
 event_name,
 timestamp
 FROM
 user_events
 WHERE
 timestamp >= CURRENT_DATE() - INTERVAL 30 DAY

Import the resource with an external SQL file

To store the SQL query in a separate file while importing the resource, use the --sql-location parameter, as shown:

rudder-cli import retl-sources --local-id api-test-10 --remote-id 2zMxxCKnGqkQLHffzCCIAvvxBso -l ./path/to/project --sql-location ./sql/api-test-10.sql

This command creates a YAML file that references the external SQL file:

version: "rudder/v0.1"
kind: "retl-source-sql-model"
metadata:
 import:
 workspaces:
 - workspace_id: "2dgUfFqnI6LFhlDmYPd8mA4Xrnc"
 resources:
 - local_id: "api-test-10"
 remote_id: "2zMwvwOG8zYJoJRHdmtN2ka1VUY"
 name: "api-test-10"
spec:
 account_id: "2rzPI1ARibivIvNH5DSpAKGTATy"
 description: ""
 display_name: "api-test-10"
 enabled: true
 id: "api-test-10"
 primary_key: "id"
 source_definition: "postgres"
 file: "./sql/api-test-10.sql"

See the SQL Model YAML reference for more details on the above parameters.

Important considerations

The metadata.import section tracks the relationship between your local CLI resource and the remote workspace resource. This enables the CLI to sync changes properly.
By default, the SQL query is stored inline in the YAML file using the sql field.
If you use the --sql-location parameter, the import process creates a separate .sql file containing the SQL query and references it using the file field.
All current settings from your workspace resource are captured in the spec section.

Import command parameters

The following table lists the parameters for the import command:

Parameter	Description
`--local-id` Required	The name you want to use for this resource in your CLI project
`--remote-id` Required	The source ID from your workspace obtained from the list command
`-l` or `--location` Required	Path to your project directory
`--sql-location`	Path where you want to save the SQL query as an external file. If not provided, the SQL query will be stored inline in the YAML file

Sync the imported resource

After generating the resource file, run the apply command to complete the import process:

rudder-cli validate -l path/to/project # Validate changes
rudder-cli apply -l path/to/project --dry-run # Preview changes
rudder-cli apply -l path/to/project # Apply changes

After successful application, the SQL model will be created in your workspace and you can manage it through the CLI.

Once imported, you can modify any field in the YAML file. Changes are applied to the workspace resource when you run the apply command.

Delete resources

If you remove the YAML file from your project, the next apply command will delete the SQL model from your workspace. Therefore, be cautious with this operation.

Troubleshooting

Issue	Resolution steps
Import fails	Verify the `remote-id` exists and is accessible in your workspace Ensure you have proper permissions to read the resource Confirm that the `local-id` does not conflict with existing resources
Apply command fails after import	Ensure the YAML file is present and properly formatted If using the `--sql-location` flag, ensure both the YAML file and the generated SQL file are present Verify your CLI is authenticated and has access to the workspace Confirm that no manual changes were made to the resource in the UI after import

Best practices

Consistent naming: Use descriptive local-id values that reflect the resource’s purpose.
Immediate sync: Always run the apply command immediately after import to establish proper tracking.
Commit together: If using the --sql-location flag, commit both the YAML and SQL files together to maintain consistency.
Environment planning: Consider how to handle different environments before importing multiple resources.

Next steps

Archive: Validate SQL Model Resource using Rudder CLI

Mon, 01 Jan 0001 00:00:00 +0000

This guide covers how to validate configurations and preview query results using Rudder CLI.

Prerequisites

Rudder CLI tool installed and authenticated with your access token
A CLI project directory with SQL model resources
Access to the data warehouses referenced by your models

Overview

The Rudder CLI provides powerful validation and preview capabilities that help you test your SQL models before deploying them to production.

The preview feature allows you to execute your SQL queries and see the results before applying changes to your workspace — this is helpful for testing query logic and ensuring data quality.

Preview command

Run the below command to preview a specific SQL model resource:

rudder-cli retl-sources preview campaigns-cli -l path/to/project --limit 10

When you run a preview command, Rudder CLI:

Validates connection: Confirms it can connect to the specified data warehouse.
Executes query: Runs your SQL query against the warehouse.
Returns results: Shows you a sample of the query results.

You can also use the preview command to validate that the primary key column is present in the result set.

Command parameters

The following table lists the parameters for the preview command:

Parameter	Description
Resource ID	The first argument specifies which SQL model to preview — it uses the `id` value from your YAML file
`-l` or `--location`	Path to your project directory
`--limit`	Number of rows to fetch Default value: 100 — RudderStack returns an error if this value is exceeded
`-j` or `--json`	Output results in JSON format instead of table format
`--interactive`	Controls whether output should be in interactive mode Default value: true

Preview command examples

The following example demonstrates the key parameters available with the preview command:

rudder-cli retl-sources preview api-test-10 --location path/to/project --limit 5 -j --interactive=false

The above command:

Previews the SQL model with ID api-test-10
Uses path/to/project as the location for the CLI project directory
Limits results to 5 rows using --limit 5
Outputs results in JSON format using -j
Runs in non-interactive mode using --interactive=false

Validation

Validation ensures your SQL model configurations are correct before deployment. It performs comprehensive checks without executing the full query.

Validate a specific resource

rudder-cli retl-sources validate campaigns-cli -l path/to/project

What validation checks

The validation process verifies:

SQL Syntax

Ensures your SQL query is syntactically correct
Validates it’s compatible with your specified warehouse type

Connectivity

Tests connection to the specified account/warehouse
Verifies authentication and permissions

Configuration integrity

Validates all required fields are present and properly formatted
Ensures account ID references exist and are accessible

Query execution (limited)

Tests that the query can execute without errors
May run a limited version of the query to validate structure

Best practices

This section provides best practices for using validation and preview in your development and CI/CD workflows.

Development workflow

Write query: Create or modify your SQL in the YAML or SQL file.
Validate first: Run validation to check syntax and configuration.
Dry run: Use --dry-run with apply commands to see what will change.
Preview results: Use preview to verify query logic and results.
Iterate: Refine your query based on preview results.
Final validation: Run validation once more before committing.

CI/CD pipeline

Always validate: Include validation in your automated checks.
Limited preview: Use preview sparingly in CI/CD due to performance implications.

Troubleshooting

This section provides troubleshooting tips for common issues you may encounter when using validation and preview.

Common validation errors

Issue	Resolution steps
Account not found	Verify the `account_id` exists in your workspace Check your CLI is authenticated with the correct workspace
SQL Syntax Error	Review your query syntax for the specific warehouse type Test the query directly in your warehouse’s query interface
Primary Key Column Missing	Ensure your query returns a column with the name specified in `primary_key` Check for typos in column names
Connection Timeout	Verify network connectivity to your warehouse Check if warehouse credentials need renewal

Error handling

Error	Resolution steps
Connection issues	Check account credentials and network connectivity
SQL errors	Review query syntax and warehouse-specific SQL dialect requirements
Primary key issues	Ensure the specified primary key column exists in your query results
Timeout issues	Simplify complex queries for validation, optimize for production

Performance considerations

Issue	Resolution steps
Large result sets	Use smaller `--limit` values for initial testing Consider adding `WHERE` clauses to reduce data volume during development
Complex queries	Break down complex queries into simpler parts for testing Use preview to validate intermediate steps
Resource usage	Preview commands consume warehouse compute resources Be mindful of costs when running against production warehouses

RudderStack Docs – Manage SQL Models using Rudder CLI

Archive: Create New SQL Model Resource using Rudder CLI

Prerequisites

List Reverse ETL accounts

Create the resource file

SQL query options

Inline SQL

External file reference

Supported sources

Apply the SQL model

Validate configuration

Dry run

Apply changes

Update the SQL model

Best practices

Next steps

Archive: Import Existing SQL Model Resource using Rudder CLI

Prerequisites

List available SQL model resources

Import the resource

Import the resource with inline SQL

Import the resource with an external SQL file

Important considerations

Import command parameters

Sync the imported resource

Delete resources

Troubleshooting

Best practices

Next steps

Archive: Validate SQL Model Resource using Rudder CLI

Prerequisites

Overview

Preview command

Command parameters

Preview command examples

Validation

Validate a specific resource

What validation checks

Best practices

Development workflow

CI/CD pipeline

Troubleshooting

Common validation errors

Error handling

Performance considerations

See also