RudderStack Docs – Rudder CLI v0.1

Archive: Install Rudder CLI

Mon, 01 Jan 0001 00:00:00 +0000

Rudder CLI is a command-line interface tool designed to interact with RudderStack’s services, facilitating tasks like CLI-based configuration, management, and deployment of Tracking Plans and Data Catalog. See CLI-based Tracking Plan Management for more information on this feature.

This tool is currently under active development and there may be frequent changes and updates to it.

Also, backward compatibility is not guaranteed at this stage.

Install Rudder CLI

This section contains the Rudder CLI installation steps for your preferred platform.

macOS

You can install the Rudder CLI on macOS devices depending on your chip configuration:

curl -L https://github.com/rudderlabs/rudder-iac/releases/latest/download/rudder-cli_Darwin_arm64.tar.gz | tar -xz rudder-cli
sudo mv rudder-cli /usr/local/bin/

curl -L https://github.com/rudderlabs/rudder-iac/releases/latest/download/rudder-cli_Darwin_x86_64.tar.gz | tar -xz rudder-cli
sudo mv rudder-cli /usr/local/bin/

Linux

Run the following command to install Rudder CLI in your Linux machine:

curl -L https://github.com/rudderlabs/rudder-iac/releases/latest/download/rudder-cli_Linux_x86_64.tar.gz | tar -xz rudder-cli
sudo mv rudder-cli /usr/local/bin/

Docker

You can run the Rudder CLI tool directly via Docker by running the following command:

docker run rudderlabs/rudder-cli

If you need to persist your configuration or provide an external configuration file, you can mount your local configuration directory into the container.

Assuming your configuration directory is located at ~/.rudder, you can run the following command:

docker run -v ~/.rudder:/.rudder rudderlabs/rudder-cli

To run commands with the local data catalog files, mount the directory containing your files and use the -l flag, as shown:

docker run -v ~/.rudder:/.rudder -v ~/my-catalog:/catalog rudderlabs/rudder-cli apply --dry-run -l /catalog

The above command uses the access token from your local configuration file and the Catalog files from the /catalog directory.

You can also use the RUDDERSTACK_ACCESS_TOKEN environment variable to provide the access token, as shown:

docker run -v ~/my-catalog:/catalog -e RUDDERSTACK_ACCESS_TOKEN=your-access-token rudderlabs/rudder-cli apply --dry-run -l /catalog

Build Rudder CLI from source

Make sure to have Go installed and set up in your environment before following the steps in this section.

Run the following commands in order:

git clone https://github.com/rudderlabs/rudder-iac.git
cd rudder-iac
make build
sudo mv bin/rudder-cli /usr/local/bin/

The make build command is used to run the build process.

Archive: Data Catalog Project Setup

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to set up your project containing the definitions of your Data Catalog resources.

Overview

In the context of the Rudder CLI tool, a Data Catalog project typically consists of a root directory that contains all the project files. Within this root directory, each YAML file can contain definitions for resources of a particular type, for example, events, properties, custom data types, and Tracking Plans.

The location and naming of these YAML files is flexible, as you can store the YAML files anywhere within the project’s root directory or subdirectories.

Set up your project

The Rudder CLI tool expects the Data Catalog resources to be described in YAML files present in your system.

If you’re working with a Unix-compatible shell, run the following command to create a tutorial-catalog folder in your home directory, which you can use to populate the Data Catalog resources as YAML files:

mkdir ~/tutorial-catalog

The Rudder CLI tool automatically processes all valid YAML files within the project structure to recognize the defined resources.

Next steps

See Data Catalog YAML Reference for the detailed YAML spec containing the definitions of these Data Catalog resources.

Archive: Recommended Project Directory Structure

Mon, 01 Jan 0001 00:00:00 +0000

This guide outlines a recommended directory structure for organizing your RudderStack Data Catalog resources and Tracking Plans as YAML files in a Git repository.

How Rudder CLI works

The Rudder CLI tool (rudder-cli) is flexible in how it processes your YAML files:

File discovery: The CLI recursively scans the specified directory (or project root) for all .yaml and .yml files.
Flexible organization: You can organize files in any directory structure that makes sense for your project.
No naming requirements: File names and directory names are completely flexible—the CLI identifies resources by their kind and content, not file location.

Important YAML file requirements

Before organizing your files, understand these critical requirements:

Single YAML object per file

Each YAML file must contain exactly one YAML document/object with the standard structure:

version: rudder/v0.1
kind: [events|properties|custom-types|tp]
metadata:
 name: [unique-name]
spec:
 # Resource definitions

Collection capabilities by kind

Kind	Description
`kind: events`	Can contain multiple event definitions in the `spec.events` array
`kind: categories`	Can contain multiple event category definitions in the `spec.categories` array
`kind: properties`	Can contain multiple property definitions in the `spec.properties` array
`kind: custom-types`	Can contain multiple custom type definitions in the `spec.types` array
`kind: tp`	Contains exactly one Tracking Plan definition

Unique resource identification

Each resource (event, property, custom type, Tracking Plan) must have a unique id across the entire project.
No duplicate id values are allowed, even across different YAML files.
Rudder CLI uses kind + metadata.name + resource id to create unique references.

Avoid resource duplication

Do not define the same resource (event, property, or custom type) in multiple YAML files. Each resource should exist in only one file to prevent duplication and potential confusion.

If you need to reference a resource from another file, use the $ref syntax instead of redefining it.

Tracking Plan limitation

Each Tracking Plan YAML file (kind: tp) can define only one Tracking Plan.
If you need multiple Tracking Plans, create separate YAML files for each.

Recommended directory structure

The recommended directory structure offers developers a clear organizational hierarchy for configuration files while maintaining high-level alignment with the Data Governance structure in the RudderStack UI:

The Data Catalog and Tracking Plans are managed in separate, dedicated folders
Business logic is reflected by grouping events and properties into business-specific folders (for example, /product and /marketing)
- In complex environments, you can maintain clarity by using separate files for similar assets within a business-specific folder (for example, the events/checkout folder has separate files for cart-events and purchase-events)
Tracking Plans are grouped by source
- In complex environments, you can create dedicated Tracking Plans for use cases within a source folder (for example, /tracking-plans/web/checkout-flow.yaml)

project-root/
├── .github/
│ └── workflows/
│ ├── tracking-plan-validate.yml # PR validation workflow
│ └── tracking-plan-sync.yml # Main branch deployment workflow
├── data-catalog/
│ ├── custom-types/
│ │ ├── identifiers.yaml # SKU, product IDs, etc.
│ │ ├── user-types.yaml # User-related custom types
│ │ ├── commerce-types.yaml # E-commerce specific types
│ │ └── analytics-types.yaml # Analytics-specific types
│ ├── properties/
│ │ ├── common/
│ │ │ ├── user-properties.yaml # User ID, email, traits, etc.
│ │ │ ├── session-properties.yaml # Session ID, timestamp, etc.
│ │ │ └── device-properties.yaml # Device info, browser, etc.
│ │ ├── product/
│ │ │ ├── product-properties.yaml # product_id, name, price, etc.
│ │ │ └── catalog-properties.yaml # category, brand, etc.
│ │ ├── checkout/
│ │ │ ├── cart-properties.yaml # Cart-related properties
│ │ │ ├── order-properties.yaml # Order-related properties
│ │ │ └── payment-properties.yaml # Payment-related properties
│ │ └── marketing/
│ │ ├── campaign-properties.yaml # UTM parameters, etc.
│ │ └── attribution-properties.yaml # Attribution parameters
│ ├── categories/
│ │ ├── user-categories.yaml # User-related categories
│ │ ├── product-categories.yaml # Product-related categories
│ │ ├── checkout-categories.yaml # Checkout-related categories
│ │ ├── marketing-categories.yaml # Marketing-related categories
│ │ └── analytics-categories.yaml # Analytics-related categories
│ └── events/
│ ├── user/
│ │ ├── authentication-events.yaml # Signup, login, logout, etc.
│ │ ├── profile-events.yaml # Profile updates, preferences
│ │ └── onboarding-events.yaml # User onboarding flow
│ ├── product/
│ │ ├── catalog-events.yaml # Product viewed, searched, etc.
│ │ ├── engagement-events.yaml # Product liked, shared, etc.
│ │ └── comparison-events.yaml # Product compared, listed, etc.
│ ├── checkout/
│ │ ├── cart-events.yaml # Cart events
│ │ ├── checkout-events.yaml # Checkout flow events
│ │ └── purchase-events.yaml # Purchase and payment events
│ ├── engagement/
│ │ ├── page-events.yaml # Page views, navigation
│ │ ├── interaction-events.yaml # Clicks, form submissions
│ │ └── content-events.yaml # Content engagement
│ └── marketing/
│ ├── campaign-events.yaml # Campaign interactions
│ └── conversion-events.yaml # Goal completions
├── tracking-plans/
│ ├── web/
│ │ ├── main-website.yaml # Main website Tracking Plan
│ │ ├── checkout-flow.yaml # Checkout-specific plan
│ │ └── marketing-pages.yaml # Landing pages, campaigns
│ ├── mobile/
│ │ ├── ios-app.yaml # iOS app Tracking Plan
│ │ ├── android-app.yaml # Android app Tracking Plan
│ │ └── mobile-shared.yaml # Shared mobile events
│ ├── backend/
│ │ ├── api-events.yaml # Server-side events
│ │ └── system-events.yaml # System/operational events
│ └── integrations/
│ ├── crm-integration.yaml # CRM-specific events
│ ├── marketing-tools.yaml # Marketing automation events
│ └── analytics-platforms.yaml # Analytics platform events

Alternative organization approaches

Since the CLI is flexible, you could also organize files differently based on your team’s preferences:

1. Flat structure

tracking-plans/
├── all-events.yaml
├── all-properties.yaml
├── all-custom-types.yaml
├── web-tracking-plan.yaml
└── mobile-tracking-plan.yaml

2. Feature-based organization

tracking-plans/
├── user-authentication/
│ ├── events.yaml
│ ├── properties.yaml
│ └── tracking-plan.yaml
├── e-commerce/
│ ├── events.yaml
│ ├── properties.yaml
│ ├── custom-types.yaml
│ └── tracking-plan.yaml
└── analytics/
 ├── events.yaml
 └── tracking-plan.yaml

3. Platform-first organization

tracking-plans/
├── web/
│ ├── events.yaml
│ ├── properties.yaml
│ └── tracking-plan.yaml
├── mobile/
│ ├── events.yaml
│ ├── properties.yaml
│ └── tracking-plan.yaml
└── shared/
 ├── common-events.yaml
 ├── common-properties.yaml
 └── custom-types.yaml

Reference system

When creating Tracking Plans, you can reference resources using the following format:

# Reference to an event
$ref: "#/events/[metadata.name]/[event.id]"

# Reference to a property
$ref: "#/properties/[metadata.name]/[property.id]"

# Reference to a custom type (in property definitions)
type: "#/custom-types/[metadata.name]/[type.id]"

Example file contents

The following are some example file contents for the recommended directory structure:

version: rudder/v0.1
kind: custom-types
metadata:
 name: identifier-types
spec:
 types:
 - id: sku_type
 name: "SKUType"
 description: "Custom type for SKU validation"
 type: string
 config:
 minLength: 5
 maxLength: 255
 pattern: "^SKU-[0-9]+$"

version: rudder/v0.1
kind: categories
metadata:
 name: product-categories
spec:
 categories:
 - id: product_viewed_category
 name: "Product Viewed"
 description: "Category for product viewing events"
 - id: product_interaction_category
 name: "Product Interaction"
 description: "Category for product interaction events"
 - id: product_purchase_category
 name: "Product Purchase"
 description: "Category for product purchase events"

version: rudder/v0.1
kind: events
metadata:
 name: product-events
spec:
 events:
 - id: product_viewed
 name: "Product Viewed"
 event_type: track
 description: "User viewed a product"
 category: "#/categories/product-categories/product_viewed_category"
 - id: product_added_to_cart
 name: "Product Added to Cart"
 event_type: track
 description: "User added product to cart"
 category: "#/categories/product-categories/product_interaction_category"
 - id: product_purchased
 name: "Product Purchased"
 event_type: track
 description: "User completed a product purchase"
 category: "#/categories/product-categories/product_purchase_category"

version: rudder/v0.1
kind: properties
metadata:
 name: product-properties
spec:
 properties:
 - id: product_sku
 name: "product_sku"
 type: "#/custom-types/identifier-types/sku_type"
 description: "Product SKU"
 - id: product_name
 name: "product_name"
 type: string
 description: "Product name"
 - id: product_price
 name: "product_price"
 type: number
 description: "Product price"
 - id: product_quantity
 name: "product_quantity"
 type: integer
 description: "Product quantity"

version: rudder/v0.1
kind: tp
metadata:
 name: web-ecommerce-plan
spec:
 id: web_ecommerce_tracking_plan
 display_name: "Web E-commerce Tracking Plan"
 description: "Tracking Plan for web e-commerce events"
 rules:
 - type: event_rule
 id: product_viewed_rule
 event:
 $ref: "#/events/product-events/product_viewed"
 allow_unplanned: false
 properties:
 - $ref: "#/properties/product-properties/product_sku"
 required: true
 - $ref: "#/properties/product-properties/product_name"
 required: true
 - $ref: "#/properties/product-properties/product_price"
 required: false

Benefits of the recommended structure

Scalability: It is easy to add new domains, platforms, or resource types.
Maintainability: Clear organization makes finding and updating resources simple.
Reusability: Common properties and custom types can be easily referenced across different Tracking Plans.
Collaboration: Team members can work on different domains without merge conflicts.
Flexibility: While providing structure, allows teams to adapt the organization to their specific needs.

Get started

Choose an organizational approach that fits your team’s workflow.
Ensure all resource id values are unique across your entire project.
Set up GitHub Actions workflows for validation and deployment.
Start with a simple structure and evolve as your needs grow.

The most important aspect is consistency within your chosen approach, not following any specific directory structure.

Archive: How to Import Workspace Resources into Your Rudder CLI Project

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks you through importing workspace resources into a Rudder CLI project and running apply so the CLI manages them.

Overview

Rudder CLI manages Data Catalog, Tracking Plans, SQL Models, and Event Stream Sources as YAML.

If those resources already exist in your workspace, you can use the import workspace command to generate files under imported/, adjust them if needed, then run apply to attach them to your CLI project — you do not need to recreate everything manually.

Required permissions

Generate a workspace-level Service Access Token with the permissions required for each resource type you want to import and manage:

Tip: Include permissions only for the resource types you use. The same scope applies for apply locally or in CI/CD. See Manage Workspaces for a deployment-oriented workflow.

Resource type	Resource	Permissions
Data Catalog and Tracking Plans	Data Catalog	Edit
	Tracking Plans	Edit
SQL Models	SQL Models	Create & Delete, Edit
Event Stream Sources	Event Stream Sources	Create & Delete, Edit, Connect
	Tracking Plans	Edit, Connect

For testing or development only: Use a Personal Access Token with the Read-Write role.

RudderStack recommends using a workspace-level Service Access Token for authentication.

Any action authenticated by a Personal Access Token will break if the user is removed from the organization or a breaking change is made to their permissions.

Token permissions for legacy RBAC system

If you are on the legacy Permissions Management (RBAC) system, your workspace-level Service Access Token should have minimum Admin permissions.

See Generate a workspace-level Service Access Token for steps to create the token.

Import workspace resources

Before you import workspace resources

Before starting the import process, make sure:

You have a Rudder CLI project directory (can be empty)

You have no unsynced changes in your project — apply any existing changes first by running the apply command.

You don’t have an existing directory named imported in your CLI project.

Otherwise, the import process will fail.

To import all resources from your current workspace into a CLI project, run the import workspace command — replace <project-directory> with the path to your CLI project directory.

rudder-cli import workspace -l <project-directory>

The command scans your workspace for resources not yet managed by Rudder CLI, writes YAML under imported/, and adds import metadata that maps local IDs to workspace IDs.

What gets imported and where

project/
├── imported/
│ ├── data-catalog/
│ │ ├── events/
│ │ ├── properties/
│ │ ├── categories/
│ │ ├── trackingplans/
│ │ └── custom-types/
│ ├── retl-sources/
│ │ ├── <model-name>.yaml
│ │ └── sql/
│ │ └── <model-name>.sql
│ ├── sources/
│ │ └── <source-name>.yaml

If a resource type has no instances, its folder is empty or omitted.

Resource	Path (under `imported/`)	Notes
Events	`data-catalog/events/`	One YAML file per event definition
Properties	`data-catalog/properties/`	One YAML file per property definition
Event Categories	`data-catalog/categories/`	One YAML file per category
Custom Types	`data-catalog/custom-types/`	One YAML file per custom type
Tracking Plans	`data-catalog/trackingplans/`	One YAML per plan Supports plans from the RudderStack Dashboard and Data Catalog API Does not support non-migrated legacy plans from Google Sheets or the deprecated Tracking Plan API
Event Stream Sources	`sources/`	One YAML per source SDK Sources only (client or server) Cloud and webhook sources are not supported
SQL Models	`retl-sources/`	One YAML per model — SQL inline by default, or under `sql/` when using external files

Folder names under imported/ follow the layout your Rudder CLI version generates. If you upgrade the CLI and the structure changes, compare the new output with this table and move files as needed before you run apply.

Import metadata

YAML from an import includes metadata.import so Rudder CLI can match local definitions to workspace resources.

For how workspace_id behaves when you target another workspace, see Manage Workspaces.

Example:

version: "rudder/v0.1"
kind: "categories"
metadata:
 import:
 workspaces:
 - workspace_id: "3NrueK2Hu7ooXVQqQJhKqKlnofE"
 resources:
 - local_id: "abc"
 remote_id: "cat_343HNkcWRt8YXHphthHwa8QEdXE"
 - local_id: "webapp"
 remote_id: "cat_2ohsVV9iKuw7GfLFITwsVLn6Nhy"
 name: "categories"
spec:
 categories:
 - id: "abc"
 name: "ABC"
 - id: "webapp"
 name: "Webapp"

Property	Type	Description
`workspace_id`	String	Workspace the resources were imported from.
`resources`	Array	Maps each `local_id` to a workspace `remote_id`. Do not change `remote_id` after import; see Key considerations.

For the full metadata schema, see the CLI Project Resources YAML Reference.

Complete the import

The import workspace command only writes files; it does not register them with your CLI project.

Review and adjust files

Confirm the generated YAML matches what you expect. You may move files out of imported/ to match your layout and edit fields such as names or descriptions. Before you run apply, read Key considerations for rules on local_id, remote_id, import metadata, and dashboard changes.

Apply imported resources

Run:

rudder-cli apply -l <project-directory>

apply finds resources marked for import, links them to the CLI project, prints a summary, and asks you to confirm.

Example:

$ rudder-cli apply -l ./my-rudder-project

Importable resources:
 - category:abc
 - category:webapp

? Do you want to apply these changes? (y/N)

After you confirm, you manage those resources through Rudder CLI.

Important considerations

This section covers important considerations when importing workspace resources and applying them to your CLI project.

Use Rudder CLI as the source of truth

For CLI-managed resources, use only Rudder CLI (not the dashboard or APIs) unless you intend to reconcile outside changes. After a successful apply, Rudder CLI is the source of truth for those resources.

IDs, metadata, and the dashboard

Do not change remote_id values in imported YAML — updates will fail.
You may change local_id only before apply. Keep the import metadata mapping in sync with any local_id you change.
After apply, do not change local_id.
Do not delete a resource in the dashboard after import but before apply — otherwise apply will fail with Resource with ID not found. See Resource Deleted from Dashboard Before Apply.

Workflow tips

Try the flow in a development workspace before production.
Commit imported YAML to version control so changes stay reviewable and reversible.

Next steps

See Manage Workspaces for workspace promotion and CI/CD
See Troubleshooting for common import and apply errors

Archive: Core Rudder CLI Commands Reference

Mon, 01 Jan 0001 00:00:00 +0000

This guide covers the core workflow commands available in the Rudder CLI tool for managing resources like Tracking Plans, Data Catalog, SQL Models, and Event Stream sources.

For resource-specific commands (for example SQL model preview), see the sections below and the linked how-to guides.

Authentication

Before running any commands, you must authenticate the CLI tool with your RudderStack workspace.

rudder-cli auth login

You will be prompted to enter your Service Access Token.

Commands

Use the following commands to manage various RudderStack resources.

`import workspace`

Import existing resources from your RudderStack workspace into your local project directory. This is useful when you want to start managing existing dashboard configurations as code.

rudder-cli import workspace -l <project-directory>

Supported resources for import

Data Catalog (Events, Properties, Categories, Custom Types)
Tracking Plans
Event Stream sources (SDK-based)
SQL Models

See Import Workspace Resources for prerequisites, metadata, and troubleshooting.

`validate`

Validate your local YAML configurations for syntax errors, structural integrity, and resource-specific constraints.

rudder-cli validate -l <project-directory>

What it checks

YAML syntax and schema compliance
Resource referencing (for example, ensuring an event references a valid property)
SQL syntax and warehouse connectivity

`apply`

Sync your local configurations to your RudderStack workspace.

rudder-cli apply -l <project-directory>

Dry run

RudderStack recommends running the apply command with the --dry-run flag first to preview the changes that will be made to your workspace.

rudder-cli apply -l <project-directory> --dry-run

`destroy`

Use this command to permanently remove resources that Rudder CLI manages from your workspace, based on your local project state.

This is a destructive operation.

rudder-cli destroy

Rudder CLI lists what it plans to remove and asks you to confirm before anything is deleted.

Tip: Flags and behavior can change between CLI releases — run rudder-cli destroy --help for the options your build supports, and prefer dry runs (validate, apply --dry-run) in CI before you use destroy in production.

Additional command groups

The CLI also exposes subcommands for specific workflows. See the linked docs for syntax, flags, and examples:

Resource	Example commands	Documentation
SQL Models (Reverse ETL)	`rudder-cli workspace accounts list` `rudder-cli workspace retl-sources list` `rudder-cli import retl-sources ...` `rudder-cli retl-sources preview ...` `rudder-cli retl-sources validate ...`	Create SQL Models Import SQL Models Validate and Preview SQL Models

Run rudder-cli --help and rudder-cli <command> --help locally for the full list of subcommands and flags for your CLI version.

Archive: Validate Rudder CLI Projects

Mon, 01 Jan 0001 00:00:00 +0000

Archive: CLI-based Data Catalog and Tracking Plan Management

Mon, 01 Jan 0001 00:00:00 +0000

This feature is in Alpha as part of RudderStack’s Early Access Program, where we work with early users and customers to test new features and get feedback before making them generally available. Note that these features are functional but can change as we improve them.

Before you get started, note the following:

See the limitations section to learn more about current differences between the CLI and UI.

RudderStack recommends testing this feature in a clean Dev workspace.

Contact the Product team if you have any questions.

RudderStack’s CLI-based Tracking Plans management feature helps you manage Tracking Plans and Data Catalog as code by leveraging the new CLI-based workflow. It helps you bring data governance into your existing development workflows through code-first management.

With this feature, you can bring the same rigor and process to your Tracking Plans that you apply to your application code, ensuring consistency and quality in your data collection strategy.

Key features

With RudderStack’s CLI-based Tracking Plan management, you get the following features:

Declarative YAML configuration

Define and manage events, properties, and Tracking Plans directly in your codebase using simple YAML files.
Create reusable property definitions with advanced validation rules like minimum/maximum length, patterns, enums, etc. across multiple events.
Define dynamic validation rules using Conditional Validation that adapt based on context.
Leverage a structured referencing system to associate events and properties to a Tracking Plan.

Powerful Rudder CLI tool

This feature comes with a powerful Rudder CLI utility that lets you:

Create, update, and delete Tracking Plan resources via command line.
Verify your YAML configurations locally before pushing changes to production.
Sync the local configuration files with your RudderStack workspace.

In contrast to how Infrastructure-as-Code (IaC) tools typically depend on external object storage like S3 to maintain their state, you can store the relevant state directly in RudderStack, thereby significantly reducing configuration overhead.

You only need the YAML configuration files and a RudderStack token to use Rudder CLI.

Git integration and CI/CD support

Leverage GitHub Actions to automatically validate and deploy Tracking Plans from your repository.
Use the standard Git workflows to track changes, implement version control, manage branches, and make any other modifications.

Who can use this feature?

The CLI-based Data Catalog and Tracking Plan management feature is helpful for data teams who need to:

Manage their Tracking Plans and Data Catalog programmatically with consistent tracking standards.
Integrate their Tracking Plan changes into their existing development workflows.
Validate their Tracking Plan changes before deployment.

Get started

Install the Rudder CLI tool in your preferred environment.
Generate a workspace-level Service Access Token in the RudderStack dashboard with the following permissions to manage Data Catalog and Tracking Plans:

Resource	Permissions
Tracking Plans	Create & Delete, Edit
Data Catalog	Edit

Token permissions for legacy RBAC system

If you are on the legacy Permissions Management (RBAC) system, your workspace-level Service Access Token should have Admin permissions.

See this documentation for more information on generating the token.

For testing or development purposes only: Generate a Personal Access Token with Read-Write role

RudderStack recommends using a workspace-level Service Access Token for authentication.

Any action authenticated by a Personal Access Token will break if the user is removed from the organization or a breaking change is made to their permissions.

Initialize a new Tracking Plan project with YAML definitions. Then, sync the changes to your workspace.
Automate Tracking Plan management using GitHub Actions.

Limitations

Rudder CLI only supports pushing updates to RudderStack, not pulling. This means you will need to test with a clean Dev workspace where the Data Catalog is empty and there are no Tracking Plans.

Contact the Product team or your Technical Account Manager if you need a new workspace.

Archive: Manage SQL Models using Rudder CLI

Mon, 01 Jan 0001 00:00:00 +0000

This feature is in Alpha as part of RudderStack’s Early Access Program, where we work with early users and customers to test new features and get feedback before making them generally available.

Note that these features are functional but can change as we improve them. Make sure to contact the RudderStack team before using them in production.

The SQL Models feature in Rudder CLI lets you manage your Reverse ETL SQL model sources through a Git-based workflow. It lets you store your SQL model configurations as YAML files in Git repositories, and use standard Git workflows to collaborate on any changes.

This approach brings version control, collaboration, and review processes to your SQL model configurations, addressing key limitations of managing these resources solely through the UI.

Key features

Bi-directional management

Create new SQL model resources directly from CLI
Import existing SQL model sources from your workspace into Git

Validation and preview

Validate SQL syntax and connectivity before deployment
Preview query results to ensure correctness
Check primary key constraints and column mappings

Flexible configuration

Define SQL queries inline in YAML files or reference external .sql files
Support for multiple warehouse types (listed below)

Automation ready

GitHub Actions integration for CI/CD workflows
Dry-run capabilities to preview changes before applying them
Cross-domain resource management alongside Data Catalog and Tracking Plan resources

Supported warehouses

The SQL Models feature supports the following data warehouses:

PostgreSQL
MySQL
Snowflake
Amazon Redshift
Google BigQuery
Databricks
Trino

Workflow overview

Configure your Git repository and Rudder CLI project.
Create new SQL models or import existing ones from your workspace.
Make changes to your SQL models using your preferred editor.
Validate your changes using the validate command and apply command in dry-run mode (via the --dry-run flag).
Use the preview command for SQL validation and connection validation.
Apply your changes using the apply command.
Create pull requests for team review and approval.
Automatically sync approved changes to your workspace via GitHub Actions.

Before you begin

Before you begin, make sure you have:

Rudder CLI installed and configured locally
A Rudder CLI project directory for storing the SQL model YAML resources
GitHub Actions configured if you want automated syncing in CI/CD
A workspace-level Service Access Token in the RudderStack dashboard with the following permissions to manage SQL models:

Resource	Permissions
SQL Models	Create & Delete, Edit

For testing or development purposes only: Generate a Personal Access Token with Read-Write role

RudderStack recommends using a workspace-level Service Access Token for authentication.

Any action authenticated by a Personal Access Token will break if the user is removed from the organization or a breaking change is made to their permissions.

Token permissions for legacy RBAC system

If you are on the legacy Permissions Management (RBAC) system, your workspace-level Service Access Token should have minimum Admin permissions.

See this documentation for more information on generating the token.

Recommended project structure

The SQL model resources need to be in the location you pass to the apply command. If you don’t specify a location, the current directory is used as the project location.

The apply command syncs all Rudder CLI resources it finds under that path (SQL models, Data Catalog, and so on).

A recommended directory structure for your CLI project is shown:

my-rudder-project/
├── sql-models/
│ ├── user-analytics.yaml
│ ├── product-models.yaml
│ └── sql/
│ ├── user-analytics.sql
│ └── product-views.sql
├── data-catalog/
│ ├── events/
│ │ └── product-events.yaml
│ └── properties/
│ └── user-properties.yaml
└── README.md

Key considerations

File discovery: The CLI recursively scans the specified directory for all .yaml and .yml files.
Flexible organization: You can organize files in any directory structure that makes sense for your project.
External SQL files: When using the file option in your YAML configurations, ensure the SQL files are accessible relative to the YAML file location.
Mixed resources: The CLI can manage SQL models alongside Data Catalog resources (events, properties, custom types, Tracking Plans) from the same project directory.

Get started

Follow the End-to-End Walkthrough: SQL Models with Rudder CLI for a single guided path through create, import, validate, preview, and apply
See Create New SQL Model Resources, Import Existing SQL Model Resources, and Validate and Preview Your Models for focused how-tos
See the SQL Model Resources YAML Reference for field-level specs
Automate with GitHub Actions for Rudder CLI

Archive: Manage Event Stream Sources using Rudder CLI

Mon, 01 Jan 0001 00:00:00 +0000

This feature is in Alpha as part of RudderStack’s Early Access Program, where we work with early users and customers to test new features and get feedback before making them generally available. Note that these features are functional but can change as we improve them.

Contact the Product team if you have any questions.

RudderStack’s CLI-based Event Stream source management feature helps you manage Event Stream sources as code by leveraging the new CLI-based workflow. It helps you bring source configuration into your existing development workflows through code-first management.

With this feature, you can apply the same rigor and process to your source configuration that you apply to your application code, ensuring consistency and quality in your event collection strategy.

Key features

With RudderStack’s CLI-based Event Stream source management, you get the following features:

Declarative YAML configuration

Define and manage Event Stream sources directly in your codebase using simple YAML files.
Configure source-level data governance settings by associating Tracking Plans with sources.
Control violation handling behavior for different event types (track, identify, group, page, screen) through granular configuration options.

Git integration and CI/CD support

Leverage GitHub Actions to automatically validate and deploy source configurations from your repository.
Use the standard Git workflows to track changes, implement version control, manage branches, and make any other modifications.

Who can use this feature?

The CLI-based Event Stream source management feature is helpful for data teams who need to:

Manage their Event Stream sources programmatically with consistent configuration standards.
Integrate their source configuration changes into their existing development workflows.
Validate their source configuration changes before deployment.
Apply data governance policies consistently across multiple sources.

Supported sources

You can define Event Stream source YAMLs only for client-side and server-side SDK sources.

Rudder CLI does not support cloud apps and webhook sources currently.

Click here to view the full list of sources.

Web

JavaScript

Mobile

Android
iOS
React Native
Flutter
Cordova
Unity

Server

Java
.NET
PHP
Rust
Python
Go
Node
Ruby
Unity

Get started

Follow the End-to-End Walkthrough: Event Stream Sources with Rudder CLI for a guided path from authentication through validate and deploy
See the Event Stream Sources YAML Reference for complete YAML schemas and field definitions
See CLI-based Tracking Plan Management to create Tracking Plans you reference in source governance settings
Automate with GitHub Actions for Rudder CLI

Archive: End-to-End Walkthrough: Data Governance with Rudder CLI

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial shows you how to use the Rudder CLI tool to:

Create new events, properties, and custom data types in your Data Catalog
Create a Tracking Plan from scratch using the events and properties
Deploy the Tracking Plan to your RudderStack workspace
Update your Tracking Plan

Prerequisites

Rudder CLI tool (rudder-cli) installed locally and available in your path

Generate a workspace-level Service Access Token in the RudderStack dashboard with the following permissions to manage Data Catalog and Tracking Plans:

Resource	Permissions
Tracking Plans	Create & Delete, Edit
Data Catalog	Edit

For testing or development purposes only: Generate a Personal Access Token with Read-Write role

RudderStack recommends using a workspace-level Service Access Token for authentication.

Any action authenticated by a Personal Access Token will break if the user is removed from the organization or a breaking change is made to their permissions.

Token permissions for legacy RBAC system

If you are on the legacy Permissions Management (RBAC) system, your workspace-level Service Access Token should have Admin permissions.

See this documentation for more information on generating the token.

1. Authenticate the CLI tool

You need to first authenticate the Rudder CLI tool (rudder-cli) with the Service Access Token - this is required to perform operations on the target RudderStack workspace. Run the below command:

rudder-cli auth login

You will then see a prompt to enter your Service Access Token. Specify the token and press Enter to continue—this stores the token locally and makes it available for all the future rudder-cli commands that need access to RudderStack.

The CLI maintains a local configuration file in your home directory (~/.rudder/config.json). After authentication, the configuration file will contain the unencrypted Service Access Token.

2. Prepare your Data Catalog project

The rudder-cli tool expects the Data Catalog resources to be described in YAML files present in your system. If you’re working with a Unix-compatible shell, run the following command to create a directory to contain these YAML files:

mkdir ~/tutorial-catalog

The above command creates a tutorial-catalog folder in your home directory, which you will use in this walkthrough to start populating the Data Catalog files.

See Data Catalog YAML Reference for more information on the YAML files containing the definitions of your Data Catalog resources.

3. Create events

Before defining your Tracking Plan, you need to define the events you want to track. Using your preferred text editor, create a file ~/tutorial-catalog/my-events.yaml and add the following lines:

rudder-cli does not have any requirements for the file names or their location in the project directory, other than them having the yaml or .yml suffix. You can use any names or structure that makes sense for your project.

version: rudder/v0.1
kind: events
metadata:
 name: myeventgroup
spec:
 events:
 - id: product_viewed
 name: "Product Viewed"
 event_type: track
 description: "This event is triggered every time the user views a product."

The above file defines an event you want to track in your application (Product Viewed). Before continuing, run the below command that validates the contents of the above file to make sure there are no issues:

rudder-cli validate -l ~/tutorial-catalog

Note that:

If the file has no errors, the command returns no output.
If there is any problem with the contents of the file, it outputs an error in the console.

4. Create event categories

Referencing categories in events is optional — you can skip this step if you don’t want to specify categories for your events.

See Manage Event Categories using Rudder CLI for more information.

You can also define event categories in YAML files with the required validation rules and reference them in your event definitions.

Using your preferred text editor, create a file ~/tutorial-catalog/my-event-categories.yaml that defines the event categories, as shown:

version: rudder/v0.1
kind: categories
metadata:
 name: event-categories
spec:
 categories:
 - id: signup_category
 name: Signup
 - id: login_category
 name: Login
 - id: browsing_category
 name: Browsing
 - id: miscellaneous_category
 name: Miscellaneous

You can later reference these categories in your event definitions, as shown:

version: rudder/v0.1
kind: events
metadata:
 name: myevents
spec:
 events:
 - id: product_viewed
 name: "Product Viewed"
 event_type: track
 description: "Triggered when a user views a product"
 category: "#/categories/event-categories/browsing_category" # Reference to the Browsing category

 - id: signup_completed
 event_type: track
 name: "Signup Completed"
 category: "#/categories/event-categories/signup_category" # Reference to the Signup category

Before continuing, run the below command that validates the contents of the above file to make sure there are no issues:

rudder-cli validate -l ~/tutorial-catalog

Note that:

If the file has no errors, the command returns no output.
If there is any problem with the contents of the file, it outputs an error in the console.

5. Create properties

Each event in the Tracking Plan can be associated with various properties. Also, some properties might be shared across events. Create a file ~/tutorial-catalog/my-properties.yaml that defines the properties to track, as shown:

version: rudder/v0.1
kind: properties
metadata:
 name: mypropertygroup
spec:
 properties:
 - id: product_sku
 name: "SKU"
 type: string
 description: "Product SKU"
 - id: category
 name: "Category"
 type: string
 description: "Product's category"

This file defines two string properties that you can later associate with our Events, SKU and Category. Validate the project again using the same command (mentioned in Step 3):

rudder-cli validate -l ~/tutorial-catalog

Again, the command gives no output if there are no issues with the file’s content.

6. Create custom types

You can define custom data types in YAML files with the required validation rules and reference them in your property definitions.

Create a file ~/tutorial-catalog/my-custom-types.yaml that defines the custom data types, as shown:

version: rudder/v0.1
kind: custom-types
metadata:
 name: identifier-types
spec:
 types:
 - id: sku_type
 name: "SKUType"
 description: "Custom type for SKU validation"
 type: string
 config:
 minLength: 5
 maxLength: 255
 pattern: "^SKU-[0-9]+$"
 - id: category_type
 name: "CategoryType"
 description: "Custom type for product identifiers"
 type: string
 config:
 minLength: 10
 maxLength: 20
 pattern: "^PROD-[0-9]+$"

This file defines two custom types (SKU Type and Category Type) that you can later reference in your properties, as shown:

version: rudder/v0.1
kind: properties
metadata:
 name: mypropertygroup
spec:
 properties:
 - id: product_sku
 name: "SKU"
 type: "#/custom-types/identifier-types/sku_type" # Reference to custom type
 description: "Product SKU"
 - id: category
 name: "Category"
 type: "#/custom-types/identifier-types/category_type" # Reference to custom type
 description: "Product's category"

Validate the project again using the same command (mentioned in Step 3):

rudder-cli validate -l ~/tutorial-catalog

See the Custom Data Types guide for more information.

6.1 Define custom type variants

Custom Type Variants let you define different property requirements for reusable object types based on a discriminating property value. This is particularly useful when an object needs different validation rules depending on the context in which it is used.

For example, you can create an address type with variants based on the country:

version: rudder/v0.1
kind: custom-types
metadata:
 name: address_types
spec:
 types:
 - id: address_object
 name: "AddressObject"
 type: object
 description: "Address with region-specific requirements"

 properties:
 - $ref: "#/properties/address_props/country"
 required: true
 - $ref: "#/properties/address_props/street_address"
 required: true
 - $ref: "#/properties/address_props/city"
 required: true
 - $ref: "#/properties/address_props/postal_code"
 - $ref: "#/properties/address_props/state_province"

 variants:
 - type: discriminator
 discriminator: "#/properties/address_props/country"
 cases:
 - display_name: "US Address"
 match:
 - "US"
 - "USA"
 - "United States"
 description: "US address format with state and ZIP"
 properties:
 - $ref: "#/properties/address_props/state_province"
 required: true
 - $ref: "#/properties/address_props/postal_code"
 required: true

 - display_name: "UK Address"
 match:
 - "UK"
 - "GB"
 - "United Kingdom"
 description: "UK address format with postcode"
 properties:
 - $ref: "#/properties/address_props/postal_code"
 required: true

 default:
 - $ref: "#/properties/address_props/postal_code"
 required: false

Validate the project again using the same command:

rudder-cli validate -l ~/tutorial-catalog

See Custom Type Variants for more examples and detailed information.

7. Create Tracking Plan

With both events and properties defined, you can now create a Tracking Plan that references them. Create a file ~/tutorial-catalog/my-tracking-plan.yaml and add the following:

version: rudder/v0.1
kind: tp
metadata:
 name: mytrackingplan
spec:
 id: mytrackingplan
 display_name: "Product Tracking Plan"
 description: "Contains all the events and properties for the Product Tracking Plan."
 rules:
 - type: event_rule
 id: product_viewed_rule
 event:
 $ref: "#/events/myeventgroup/product_viewed"
 allow_unplanned: false
 properties:
 - $ref: "#/properties/mypropertygroup/product_sku"
 required: true
 - $ref: "#/properties/mypropertygroup/category"
 required: false
 - $ref: "#/properties/mypropertygroup/product_details" # Object property
 required: true
 properties: # Nested properties
 - $ref: "#/properties/mypropertygroup/product_name"
 required: true
 - $ref: "#/properties/mypropertygroup/product_price"
 required: true

The above file defines a Tracking Plan that contains the event (Product Viewed) that you defined previously along with the properties SKU, Category, and Product Details containing two nested properties Product Name and Product Price.

Note that you can associate the events and properties with the Tracking Plan by providing a reference to them using $ref. The references point to specific entities defined in the project without strictly depending on the files they were defined in. Instead, they rely on the on the kind (events / properties), metadata.name, and the entity’s id fields to uniquely refer to them.

For example, the Product Viewed event is defined with:

kind: events
metadata.name: myeventgroup
id: product_viewed

Therefore, you can refer to it using the #/events/myeventgroup/product_viewed reference.

Validate the Tracking Plan again before proceeding to the next step:

rudder-cli validate -l ~/tutorial-catalog

7.1 Define event rule variants

Event Rule Variants let you define different property requirements for the same event based on a discriminating property value. This is useful when an event needs different validation rules depending on the context in which it occurs.

For example, you can extend the product_viewed event to have different requirements based on the page type.

Add the required properties:

version: rudder/v0.1
kind: properties
metadata:
 name: mypropertygroup
spec:
 properties:
 - id: page_type
 name: "page_type"
 type: string
 description: "Type of page being viewed"
 - id: page_url
 name: "page_url"
 type: string
 description: "URL of the page"
 - id: search_term
 name: "search_term"
 type: string
 description: "Search query entered by user"
 - id: product_sku
 name: "SKU"
 type: string
 description: "Product SKU"
 - id: category
 name: "Category"
 type: string
 description: "Product's category"

Update the Tracking Plan to use variants:

version: rudder/v0.1
kind: tp
metadata:
 name: mytrackingplan
spec:
 id: mytrackingplan
 display_name: "Product Tracking Plan"
 description: "Contains all the events and properties for the Product Tracking Plan."
 rules:
 - type: event_rule
 id: product_viewed_rule
 event:
 $ref: "#/events/myeventgroup/product_viewed"
 allow_unplanned: false

 properties:
 - $ref: "#/properties/mypropertygroup/page_type"
 required: true
 - $ref: "#/properties/mypropertygroup/product_sku"
 required: false
 - $ref: "#/properties/mypropertygroup/category"
 required: false
 - $ref: "#/properties/mypropertygroup/search_term"
 required: false

 variants:
 - type: discriminator
 discriminator: "#/properties/mypropertygroup/page_type"
 cases:
 - display_name: "Search Results Page"
 match:
 - "search"
 - "search_results"
 - "category_search"
 description: "When user is on search or category pages"
 properties:
 - $ref: "#/properties/mypropertygroup/search_term"
 required: true

 - display_name: "Product Detail Page"
 match:
 - "product"
 - "product_detail"
 description: "When user is viewing a specific product"
 properties:
 - $ref: "#/properties/mypropertygroup/product_sku"
 required: true
 - $ref: "#/properties/mypropertygroup/category"
 required: true

 default:
 - $ref: "#/properties/mypropertygroup/page_url"
 required: true

Make sure to:

Define all properties used in variants in the common properties section

Mark the discriminator property (page_type) as required

Set allow_unplanned: false in the event definition

Validate the project again using the same command:

rudder-cli validate -l ~/tutorial-catalog

See Event Rule Variants for more examples and detailed information.

8. Deploy Data Catalog resources

With your events, properties, and Tracking Plan defined, deploy them to your RudderStack workspace.

Review the changes to ensure there are no mistakes by running the following command:

rudder-cli apply -l ~/tutorial-catalog --dry-run

The --dry-run option simulates the application of changes without making actual updates to your workspace. Instead, the rudder-cli tool reports all the resources as New resources, as seen in the below output:

New resources:
 - event:product_viewed
 - property:category
 - property:product_sku
 - tracking_plan:mytrackingplan

If the list matches your expectations, run the following command to apply the changes to your workspace:

rudder-cli apply -l ~/tutorial-catalog

The output will be similar, but with an additional confirmation prompt:

New resources:
 - event:product_viewed
 - property:category
 - property:product_sku
 - tracking_plan:mytrackingplan

? Do you want to apply these changes? (y/N)

Continue by typing y and pressing the Enter key. rudder-cli then starts creating the new resources one by one in your Data Catalog and reports if the process was successful, as shown:

✔ Create event:product_viewed
✔ Create property:category
✔ Create property:product_sku
✔ Create tracking_plan:mytrackingplan

9. Update Tracking Plan

In the above steps, you created and deployed a Tracking Plan containing rules for the Product Viewed event. You can use the rudder-cli tool to update the Tracking Plan by adding another event (Added to Cart) and property (price), and deploy the changes to your workspace.

Define a new event Added to Cart by opening the ~/tutorial-catalog/my-events.yaml file and adding the following content under spec.events:

version: rudder/v0.1
kind: events
metadata:
 name: myeventgroup
spec:
 events:
 - id: product_viewed
 name: "Product Viewed"
 event_type: track
 description: "This event is triggered every time a user views a product."
 - id: added_to_cart
 name: "Added To Cart"
 event_type: track
 description: "This event is triggered everytime the user adds a product to their cart."

Note that you can also define the new event in a different file. In this case, you can add it in the existing file to group together events that are used in the same context.

Define a new property price by opening the ~/tutorial-catalog/my-properties.yaml file and adding the following details under spec.properties:

version: rudder/v0.1
kind: properties
metadata:
 name: mypropertygroup
spec:
 properties:
 - id: product_sku
 name: "SKU"
 type: string
 description: "Product SKU"
 - id: category
 name: "Category"
 type: string
 description: "Product's category"
 - id: price
 name: "Price"
 type: integer
 description: "Product's price"

Add the newly-defined event and property in the Tracking Plan by updating the ~/tutorial-catalog/my-tracking-plan.yaml file by adding the new event rule and property under spec.rules, as shown:

version: rudder/v0.1
kind: tp
metadata:
 name: mytrackingplan
spec:
 id: mytrackingplan
 display_name: "Product Tracking Plan"
 description: "Contains all the events and properties for the Product Tracking Plan."
 rules:
 - type: event_rule
 id: product_viewed_rule
 event:
 $ref: "#/events/myeventgroup/product_viewed"
 allow_unplanned: false
 properties:
 - $ref: "#/properties/mypropertygroup/product_sku"
 required: true
 - $ref: "#/properties/mypropertygroup/category"
 required: false
 - $ref: "#/properties/mypropertygroup/product_details" # Object property
 required: true
 properties: # Nested properties
 - $ref: "#/properties/mypropertygroup/product_name"
 required: true
 - $ref: "#/properties/mypropertygroup/product_price"
 required: true
 - type: event_rule # New event rule
 id: added_to_cart_rule
 event:
 $ref: "#/events/myeventgroup/added_to_cart"
 properties:
 - $ref: "#/properties/mypropertygroup/product_sku"
 required: true
 - $ref: "#/properties/mypropertygroup/price" # New property
 required: true

As described in Step 8: Deploy Data Catalog resources, review the changes and apply them to your workspace by running the below commands:

rudder-cli apply -l ~/tutorial-catalog --dry-run

rudder-cli apply -l ~/tutorial-catalog

You can also run the below command to deploy the Tracking Plan changes to your workspace directly and skip any confirmation:

rudder-cli apply -l ~/tutorial-catalog --confirm=false

The output of this command should look like:

New resources:
 - event:added_to_cart
 - property:price

Updated resources:
 - tracking_plan:mytrackingplan

✔ Create event:added_to_cart
✔ Create property:price
✔ Update tracking_plan:mytrackingplan

The above output informs that a new event and property along with a change in the Tracking Plan is detected. It also informs that the new resources are created and the update operation is completed successfully.

Archive: End-to-End Walkthrough: SQL Models with Rudder CLI

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial shows you how to use Rudder CLI to:

Authenticate and prepare a project directory for SQL model YAML
Resolve the warehouse account ID your models use
Create a new SQL model from YAML and sync it to your workspace
Import an existing workspace SQL model into Git (inline SQL or external .sql file)
Validate configuration and preview query results before you apply changes

For feature overview, supported warehouses, and recommended repository layout, see Manage SQL Models using Rudder CLI.

Prerequisites

Rudder CLI installed locally
A directory you will use as the CLI project root (this tutorial uses ~/tutorial-sql-models; replace it with your path)
At least one Reverse ETL source in the RudderStack dashboard for the warehouse you query — RudderStack uses that connection to resolve account IDs (see List warehouse accounts)
A workspace-level Service Access Token with the following permissions:

Resource	Permissions
SQL Models	Create & Delete, Edit

For testing or development only: A Personal Access Token with the Read-Write role

RudderStack recommends using a workspace-level Service Access Token for authentication.

Any action authenticated by a Personal Access Token will break if you’re removed from the organization or RudderStack makes a breaking change to your permissions.

Token permissions for legacy RBAC system

If you are on the legacy Permissions Management (RBAC) system, your workspace-level Service Access Token should have minimum Admin permissions.

See Generate a workspace-level Service Access Token for steps to create the token.

1. Authenticate the CLI tool

Run the following command and enter your workspace-level Service Access Token when prompted:

rudder-cli auth login

2. Create a project directory

Create a folder for YAML and optional SQL files:

mkdir -p ~/tutorial-sql-models/sql-models/sql

If you already manage Data Catalog or Tracking Plan YAML in another folder, you can add SQL model files there instead — Rudder CLI discovers .yaml / .yml files recursively under the path you pass to -l / --location.

3. List warehouse accounts

Each SQL model references an account_id for the warehouse connection. List accounts linked to your workspace:

rudder-cli workspace accounts list

Pick the account ID that matches your Reverse ETL warehouse setup. You will use it in the YAML spec in the next sections.

You need a configured Reverse ETL source in the dashboard so warehouse credentials exist for that account.

4. Create a new SQL model

Add a YAML file (for example ~/tutorial-sql-models/sql-models/product-purchases.yaml) with a retl-source-sql-model resource. Replace account_id with the value from List warehouse accounts.

version: "rudder/v0.1"
kind: "retl-source-sql-model"
metadata:
 name: "product-purchases-model"
spec:
 id: "product-purchases-model"
 display_name: "Product Purchases SQL Model"
 primary_key: "user_id"
 description: "SQL model for product purchases"
 source_definition: "postgres"
 enabled: true
 account_id: "YOUR_ACCOUNT_ID"
 sql: |
 SELECT
 user_id,
 event_name,
 event_type,
 timestamp,
 properties
 FROM
 user_events
 WHERE
 timestamp >= CURRENT_DATE - INTERVAL '180 days'
 ORDER BY
 timestamp DESC

Query shape

Inline SQL: use sql: | as above.
External file: use file: "./sql/my-query.sql" instead of sql — use one of sql or file, not both.

source_definition values (examples): postgres, mysql, snowflake, redshift, bigquery, databricks, trino. See the SQL Model YAML reference for the full spec.

Validate and apply (create path)

From your machine (adjust the path):

rudder-cli validate -l ~/tutorial-sql-models
rudder-cli apply -l ~/tutorial-sql-models --dry-run
rudder-cli apply -l ~/tutorial-sql-models

After a successful apply, the model exists in the workspace and you can keep editing YAML and re-running validate / apply.

5. Import an existing SQL model

Use this path when the model already exists in the workspace and you want it under Git.

List models in the workspace

rudder-cli workspace retl-sources list

Note the source_id (remote ID) for the model you want.

Run import

Pick a local_id (name for the resource in your project). Replace placeholders with your IDs and paths:

Inline SQL in YAML (default)

rudder-cli import retl-sources --local-id my-imported-model --remote-id YOUR_SOURCE_ID -l ~/tutorial-sql-models

SQL in an external file

rudder-cli import retl-sources --local-id my-imported-model --remote-id YOUR_SOURCE_ID -l ~/tutorial-sql-models --sql-location ~/tutorial-sql-models/sql-models/sql/my-imported-model.sql

Rudder CLI writes YAML under your project (and a .sql file when you use --sql-location). The metadata.import section maps your local_id to the workspace remote_id so later apply calls stay aligned.

Finish linking the import

Validate and apply so the CLI project owns the resource:

rudder-cli validate -l ~/tutorial-sql-models
rudder-cli apply -l ~/tutorial-sql-models --dry-run
rudder-cli apply -l ~/tutorial-sql-models

If you use --sql-location, keep the YAML and the .sql file together in Git; both must be present for validate / apply.

For parameter details and troubleshooting, see Import existing SQL model resources.

6. Validate and preview

Project-wide validation

Checks YAML structure, references, and related rules for everything under the project path:

rudder-cli validate -l ~/tutorial-sql-models

Validate a single SQL model

Targets one model by the spec.id in your YAML:

rudder-cli retl-sources validate YOUR_MODEL_ID -l ~/tutorial-sql-models

This checks SQL syntax for the warehouse, connectivity to the account, required fields, and that the query can run (limited execution), as described in Validate SQL model resources.

Preview query results

Runs the query and prints sample rows (use your model’s spec.id):

rudder-cli retl-sources preview YOUR_MODEL_ID -l ~/tutorial-sql-models --limit 10

Useful flags: -l / --location for the project root, --limit for row cap (default 100), -j for JSON output, --interactive=false for scripts. See Validate and preview your models for examples and troubleshooting.

Deploy changes

After you are satisfied:

rudder-cli apply -l ~/tutorial-sql-models --dry-run
rudder-cli apply -l ~/tutorial-sql-models

apply syncs SQL models together with any other Rudder CLI resources in the same project directory.

See more

Manage SQL Models using Rudder CLI
Create New SQL Model Resources — deeper how-to for new resources
Import Existing SQL Model Resources — import parameters and edge cases
Validate and Preview Your Models — preview parameters, CI/CD notes, error tables
SQL Model Resources YAML Reference
GitHub Actions for Rudder CLI for automating validate / apply

Archive: End-to-End Walkthrough: Event Stream Sources with Rudder CLI

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial shows you how to use Rudder CLI to:

Authenticate against your RudderStack workspace
Create a project directory and define an Event Stream source in YAML
Optionally attach a Tracking Plan for data governance
Validate and deploy the source to your workspace

For feature overview and supported source types, see Manage Event Stream Sources using Rudder CLI.

Prerequisites

Rudder CLI tool (rudder-cli) installed locally
An existing Tracking Plan defined in your CLI project (optional, but recommended for Data Governance)
In your RudderStack workspace, create a workspace-level Service Access Token with the following permissions to manage Event Stream sources:

Resource	Permissions	Description
Event Stream Sources	Create & Delete	Create or delete Event Stream sources in the workspace
Event Stream Sources	Edit	Make changes to the configuration of Event Stream sources
Event Stream Sources	Connect	Connect an Event Stream source to a Tracking Plan
Tracking Plans	Edit, Connect	Connect a Tracking Plan to an Event Stream source

Click here to see how these permissions appear in the workspace policy.

Token permissions for legacy RBAC system

If you are on the legacy Permissions Management (RBAC) system, your workspace-level Service Access Token should have minimum Admin permissions.

See Generate a workspace-level Service Access Token for steps to create the token.

1. Authenticate the CLI tool

Run the following command and enter your access token when prompted:

rudder-cli auth login

2. Create a project directory

Create a project directory to store your source YAML files:

mkdir ~/tutorial-sources

If you are already managing Tracking Plans or Data Catalog resources via CLI, you can add source definitions to the same project directory. The CLI processes all YAML files in the directory recursively.

3. Review the YAML reference

Before defining your Event Stream source, review the Source YAML Reference to understand:

The complete YAML structure for Event Stream sources
Required and optional fields for source configuration
Governance validation settings and violation rules
Reference syntax for linking sources to Tracking Plans

Understanding the YAML reference will help you create properly structured files and avoid common validation errors when defining your sources.

4. Define an Event Stream source

Create a YAML file for your Event Stream source (~/tutorial-sources/ios-source.yaml):

version: rudder/v0.1
kind: event-stream-source
metadata:
 name: ios-source
spec:
 id: "my-ios-source"
 type: "ios"
 name: "iOS Source"
 enabled: true

This example creates a basic iOS source without governance configuration. The source accepts events and is enabled by default.

5. Optional: Add governance configuration

If you want to associate a Tracking Plan with your source for data governance, update the YAML file to include governance settings:

version: rudder/v0.1
kind: event-stream-source
metadata:
 name: ios-source
spec:
 id: "my-ios-source"
 type: "ios"
 name: "iOS Source"
 enabled: true
 governance:
 validations:
 tracking_plan: "#/tp/my-tracking-plan/my-tracking-plan"
 config:
 track:
 propagate_violations: true
 drop_unplanned_events: true
 drop_unplanned_properties: true
 drop_other_violations: true
 identify:
 propagate_violations: true
 drop_unplanned_properties: true
 drop_other_violations: true

In this example:

The source is linked to a Tracking Plan referenced as #/tp/my-tracking-plan/my-tracking-plan.
For track events, violations are propagated, unplanned events are dropped, unplanned properties are dropped, and other violations are dropped.
For identify events, violations are propagated, unplanned properties are dropped, and other violations are dropped.

The Tracking Plan reference follows the format #/tp/[metadata.name]/[tracking-plan.id], where metadata.name is the metadata.name value in your Tracking Plan YAML file and tracking-plan.id is the spec.id value of the Tracking Plan.

See Source YAML Reference for detailed information about these configuration options.

6. Validate and deploy

Validate your files:

rudder-cli validate -l ~/tutorial-sources

Optional: Review changes before deploying:

rudder-cli apply -l ~/tutorial-sources --dry-run

Deploy changes to your workspace:

rudder-cli apply -l ~/tutorial-sources

See more

Archive: Data Catalog and Tracking Plans YAML Reference

Mon, 01 Jan 0001 00:00:00 +0000

This guide serves as a detailed reference for the CLI project YAML files that contain definitions of your Data Catalog and Tracking Plan resources.

Overview

In the context of the Rudder CLI (rudder-cli) tool, a project typically consists of a root directory that contains all the project files. Within this root directory, each YAML file can contain definitions for resources of a particular type, for example, events, properties, custom data types, and Tracking Plans.

The location and naming of these YAML files is flexible, as you can store the YAML files anywhere within the project’s root directory or subdirectories.

You can also group some resources of the same type in the same file, allowing structures that can best serve your project’s requirements. For example, you could have:

A events.yaml file in the project’s root directory that defines multiple events
Another file subdirectory/user-events.yaml that defines additional events

The Rudder CLI tool processes all valid YAML files within the project structure to recognize the defined resources.

The following sections detail the specific YAML formats and parameter definitions for each resource type.

For discriminator-based validation in Tracking Plan event rules and custom types, see the Conditional Validation YAML Reference.

Events

You can define one or more events in the YAML file by setting kind to events.

The spec parameter of the YAML file has the following structure:

Property	Type	Description
`events` Required	Array of event definitions	An array of event definitions grouped together in the same file.

Event definition

The event definitions have a structure that depends on the event type. All definitions share some common properties, as listed in the below table:

Property	Type	Description
`id` Required	String	Unique identifier for the event within the project. This parameter must be unique across all events.
`event_type` Required	String	Event type. Acceptable values are `track`, `identify`, `page`, `screen`, and `group`.
`description`	String	Event description.
`category`	String	Reference to an existing event category. See Manage Event Categories using Rudder CLI for more information.

Additionally, track events (event_type: track) also support the following property:

Property	Type	Description
`name` Required	String	The `track` event name. In other words, this parameter corresponds to the `event` property of the corresponding RudderStack `track` event.

Example

version: rudder/v0.1
kind: events
metadata:
 name: myeventgroup
spec:
 events:
 - id: product_viewed
 name: "Product Viewed"
 event_type: track
 description: "This event is triggered every time a user views a product."
 category: "#/categories/event-categories/browsing_category" # Reference to the Browsing category
 - id: added_to_cart
 name: "Added To Cart"
 event_type: track
 description: "This event is triggered every time the user adds a product to their cart."
 - id: identify
 event_type: identify
 description: "An event that identifies the user."
 - id: page
 event_type: page

Event categories

You can define event categories in the YAML file by setting kind to categories.

The spec parameter of the YAML file has the following structure:

Property	Type	Description
`categories` Required	Array of category definitions	An array of category definitions grouped together in the same file.

Category definition

All category definitions share the following properties:

Property	Type	Description
`id` Required	String	Unique identifier for the category within the project. This parameter must be unique across all categories in all the YAML files within the project.
`name` Required	String	Display name of the category.

Example

version: rudder/v0.1
kind: categories
metadata:
 name: event-categories
spec:
 categories:
 - id: signup_category
 name: Signup
 - id: login_category
 name: Login
 - id: browsing_category
 name: Browsing
 - id: miscellaneous_category
 name: Miscellaneous

Properties

You can define one or more properties in the YAML file by setting kind to properties.

The spec parameter of the YAML file has the following structure:

Property	Type	Description
`properties` Required	Array of property definitions	An array of property definitions grouped together in the same file.

Property definition

A property definition has the following structure:

Property	Type	Description
`id` Required	String	Unique identifier for the property within the project. This parameter must be unique across all properties in all the YAML files within the project.
`name` Required	String	This parameter corresponds to the field inside an event’s `properties` or `traits` JSON.
`type` Required	String	Property type. Acceptable values are: `string`, `integer`, `number`, `object`, `array`, `boolean`, and `null`.
`description`	String	Property description.
`propConfig`	`propConfig` object	Additional validation rules for the property’s values.

Property config

Property	Type	Description
`minLength`	Integer	Minimum length of the property’s string value.
`maxLength`	Integer	Maximum length of the property’s string value.
`pattern`	String	Regular expression that the property’s string values need to match with.
`enum`	Array of strings	List of all valid values for the property.

Example

version: rudder/v0.1
kind: properties
metadata:
 name: ecommerce_properties
spec:
 properties:
 - id: product_id
 name: "product_id"
 type: string
 description: "Unique identifier for the product."
 propConfig:
 minLength: 3
 maxLength: 64
 - id: product_name
 name: "product_name"
 type: string
 description: "Name of the product."
 propConfig:
 minLength: 2
 maxLength: 255
 - id: product_price
 name: "product_price"
 type: number
 description: "Price of the product in the store's currency."
 - id: product_category
 name: "product_category"
 type: string
 description: "Category the product belongs to."
 propConfig:
 enum:
 - "clothing"
 - "electronics"
 - "home_goods"
 - "beauty"
 - "accessories"
 maxLength: 60
 - id: cart_items
 name: "cart_items"
 description: "Items present in the cart."
 type: array
 propConfig:
 minItems: 1
 uniqueItems: false

Custom data types

You can define custom data types in the YAML file by setting kind to custom-types.

The spec parameter of the YAML file has the following structure:

Property	Type	Description
`types` Required	Array of custom type definitions	An array of custom type objects grouped together in the same file.

Custom type definition

Property	Type	Description
`id` Required	String	Unique identifier for the custom type within the project. This parameter must be unique across all custom types in all the YAML files within the project.
`name` Required	String	Display name of the custom type.
`type` Required	String	Base data type for the custom type. Acceptable values are: `string`, `integer`, `number`, `object`, `array`, `boolean`.
`config` Required	`config` object	Validation rules for the custom type. The configuration options vary depending on the `type` parameter.
`description`	String	Description of the custom type.

`config` options

The config object’s configuration varies depending on the type parameter.

Parameter	Description
`minLength`	Minimum string length
`maxLength`	Maximum string length
`pattern`	Regular expression pattern
`format`	Predefined format like email, date, etc.
`enum`	Array of allowed values

Parameter	Description
`minimum`	Minimum value
`maximum`	Maximum value
`exclusiveMinimum`	Exclusive minimum value
`exclusiveMaximum`	Exclusive maximum value
`multipleOf`	Multiple of value

Parameter	Description
`itemTypes`	List of acceptable types for array items
`minItems`	Minimum number of items
`maxItems`	Maximum number of items
`uniqueItems`	Boolean requiring uniqueness of items

Example

version: rudder/v0.1
kind: custom-types
metadata:
 name: email-types
spec:
 types:
 - id: emailtype
 name: "EmailType"
 description: "Custom type for email validation"
 type: string
 config:
 format: "email"
 minLength: 5
 maxLength: 255
 pattern: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
 - id: addresstype
 name: "AddressType"
 description: "Physical address information"
 type: object
 properties:
 - id: street
 type: string
 required: true
 - id: city
 type: string
 required: true
 - id: country
 type: string
 required: false

Tracking Plans

You can define a Tracking Plan in the YAML file by setting kind to tp.

The spec parameter of the YAML file has the following structure:

Property	Type	Description
`id` Required	String	Unique identifier for the Tracking Plan within the project. This parameter must be unique across all the Tracking Plans in the project.
`display_name` Required	String	A readable short name for the Tracking Plan.
`description`	String	Tracking plan description.
`rules` Required	Array of rules definitions	Contains the list of events in the Tracking Plan along with the rules for their expected properties.

Rules definition

Property	Type	Description
`type` Required	String	The rule type. The only acceptable value currently is `event_rule`.
`id` Required	String	Rule ID.
`event` Required	Rule event definition object	Event definition associated with the rule along with the validation rules for the Tracking Plan.
`properties` Required	Array of rule property definitions	List of properties associated with the rule’s event along with the validation rules for the Tracking Plan.

Rule event definition

Property	Type	Description
`$ref` Required	String	Reference to an existing event definition. See Reference catalog resources for more information on how to work with references.
`allow_unplanned`	Boolean	Validation rule that checks if the event can have properties other than those defined in the rule’s `properties` section. Default value: `false`
`identity_section` Required, for non-track events	String	Defines in which field of the corresponding RudderStack event payload the rule’s properties should be included. Acceptable values are: `properties`, `traits`, and `context.traits`.

Rule property definition

Property	Type	Description
`$ref` Required	String	Reference to an existing property definition. See Reference catalog resources for more information on how to work with references.
`required`	Boolean	Validation rule that determines whether the property should always be present in the RudderStack event. Default value: `false`
`properties` Applicable only for properties of `object` type	Array of rule property definitions	Defines which nested properties should be included when this property is used in the Tracking Plan.

Example

version: rudder/v0.1
kind: tp
metadata:
 name: ecommerce_tracking_plan
spec:
 id: ecommerce_tracking_plan
 display_name: "E-commerce Tracking Plan"
 description: "Tracking plan for an e-commerce application."
 rules:
 - type: event_rule
 id: product_viewed_rule
 event:
 $ref: "#/events/myeventgroup/product_viewed"
 allow_unplanned: false
 properties:
 - $ref: "#/properties/ecommerce_properties/product_id"
 required: true
 - $ref: "#/properties/ecommerce_properties/product_name"
 required: true
 - $ref: "#/properties/ecommerce_properties/product_price"
 required: true
 - $ref: "#/properties/ecommerce_properties/product_category"
 required: false

Reference Catalog resources

Definitions in a YAML file can refer to definitions in other files by using the resource reference ($ref) strings — this is useful while defining resources like Tracking Plans which need to be associated with events and properties defined in other files.

Note that references must always start with a # character followed by the path using the / delimiter. The path can point to a unique resource within a project’s file and is expected to have the following components in order:

kind value of the target resource’s file, that is, events, properties, or tp.
metadata.name value of the target resource’s file.
id value of the target resource.

For example, you can refer an event inside a file with metadata.name set to examples and having id as example_id as follows:

- $ref: "#/events/examples/example_id"

Import metadata

When you import resources from a workspace using the import workspace command, the generated YAML files contain special import metadata that tells Rudder CLI how to link local resources to workspace resources.

Structure

The import metadata is located in the metadata.import section of the YAML file:

version: "rudder/v0.1"
kind: "categories"
metadata:
 import:
 workspaces:
 - workspace_id: "3NrueK2Hu7ooXVQqQJhKqKlnofE"
 resources:
 - local_id: "abc"
 remote_id: "cat_343HNkcWRt8YXHphthHwa8QEdXE"
 - local_id: "webapp"
 remote_id: "cat_2ohsVV9iKuw7GfLFITwsVLn6Nhy"
 name: "categories"
spec:
 categories:
 - id: "abc"
 name: "ABC"
 - id: "webapp"
 name: "Webapp"

Properties

The metadata.import section contains the following properties:

Property	Type	Description
`workspaces` Required	Array	Array of workspace import configurations. Each workspace configuration contains the workspace ID and resource mappings.

Workspace configuration

Each workspace configuration in the workspaces array has the following structure:

Property	Type	Description
`workspace_id` Required	String	The ID of the workspace where resources were imported from.
`resources` Required	Array	Array of mappings between local resource IDs and their corresponding workspace IDs.

Resource mapping

Each resource mapping in the resources array has the following structure:

Property	Type	Description
`local_id` Required	String	The local resource ID used within your CLI project. This corresponds to the `id` field in the resource definition.
`remote_id` Required	String	The remote resource ID from the workspace where the resource was imported from.

The import metadata serves two key purposes:

Resource linking: Enables Rudder CLI to link local resource definitions to existing workspace resources when you run the apply command.

Workspace-aware operations: Tracks which workspace resources were imported from, allowing you to apply the same project to different workspaces. When you apply a project to a workspace different from the one specified in workspace_id, resources are treated as new resources to be created rather than imported.

See Manage Workspaces for more information about how to use import metadata in workspace management workflows.

Archive: SQL Model Resources YAML Reference

Mon, 01 Jan 0001 00:00:00 +0000

This guide serves as a detailed reference for the CLI project YAML files that contain definitions of your Reverse ETL SQL model resources.

Overview

In the context of the Rudder CLI (rudder-cli) tool, SQL model resources are defined using YAML files that specify the configuration for your Reverse ETL SQL model sources. These YAML files enable you to manage your SQL models through a Git-based workflow with version control and collaboration features.

The location and naming of these YAML files is flexible, as you can store the YAML files anywhere within the project’s root directory or subdirectories.

You can also group multiple SQL model resources in the same file, allowing structures that can best serve your project’s requirements.

The Rudder CLI tool processes all valid YAML files within the project structure to recognize the defined resources.

The following sections detail the specific YAML formats and parameter definitions for SQL model resources.

SQL model resources

You can define one or more SQL model resources in the YAML file by setting kind to retl-source-sql-model.

The spec parameter of the YAML file has the following structure:

Property	Type	Description
`id` Required	String	Unique identifier for the SQL model resource within the project. This parameter must be unique across all SQL model resources.
`display_name` Required	String	Display name for the SQL model resource as shown in the RudderStack dashboard.
`primary_key` Required	String	Column name to use as the primary key for the model. This column must exist in your SQL query results.
`source_definition` Required	String	Warehouse type identifier. See Supported warehouse types for values that you can use.
`enabled` Required	Boolean	Whether the SQL model resource is active. Set to `true` to enable the resource.
`account_id` Required	String	The account ID for the data warehouse connection. You can obtain this using `./rudder-cli workspace accounts list`.
`description`	String	Optional description for the SQL model resource.
`sql`	String	Inline SQL query definition. Use the YAML literal block scalar `
`file`	String	Path to an external SQL file containing the query. Path is relative to the YAML file location.

You must use exactly one of sql or file — they are mutually exclusive. Use sql for inline queries or file to reference external SQL files.

Supported warehouse types

The source_definition field supports the following data warehouse types:

Warehouse	`source_definition` value
PostgreSQL	`postgres`
MySQL	`mysql`
Snowflake	`snowflake`
Amazon Redshift	`redshift`
Google BigQuery	`bigquery`
Databricks	`databricks`
Trino	`trino`

Example definitions

The following examples highlight the YAML configurations for SQL model resources defined using inline SQL and external SQL files. You can group multiple resources in the same YAML file to manage them together.

version: "rudder/v0.1"
kind: "retl-source-sql-model"
metadata:
 name: "product-purchases-model"
spec:
 id: "product-purchases-model"
 display_name: "Product Purchases SQL Model"
 primary_key: "user_id"
 description: "SQL model for tracking product purchases over the last 180 days"
 source_definition: "postgres"
 enabled: true
 account_id: "2rzPI1ARibivIvNH5DSpAKGTATy"
 sql: |
 SELECT
 user_id,
 event_name,
 event_type,
 timestamp,
 properties,
 product_id
 FROM
 user_events
 WHERE
 timestamp >= CURRENT_DATE() - INTERVAL 180 DAY
 AND event_name = 'Product Purchased'
 ORDER BY
 timestamp DESC

version: "rudder/v0.1"
kind: "retl-source-sql-model"
metadata:
 name: "user-analytics-model"
spec:
 id: "user-analytics-model"
 display_name: "User Analytics Model"
 primary_key: "user_id"
 description: "Comprehensive user analytics model"
 source_definition: "snowflake"
 enabled: true
 account_id: "2rzPI1ARibivIvNH5DSpAKGTATy"
 file: "./queries/user-analytics.sql"

In this example, the SQL query would be stored in a separate file at ./queries/user-analytics.sql relative to the YAML file location.

version: "rudder/v0.1"
kind: "retl-source-sql-model"
metadata:
 name: "ecommerce-models"
spec:
 - id: "product-views-model"
 display_name: "Product Views Model"
 primary_key: "user_id"
 description: "Model for tracking product page views"
 source_definition: "bigquery"
 enabled: true
 account_id: "2rzPI1ARibivIvNH5DSpAKGTATy"
 sql: |
 SELECT
 user_id,
 product_id,
 view_timestamp,
 session_id
 FROM
 product_views
 WHERE
 view_timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
 - id: "cart-abandonment-model"
 display_name: "Cart Abandonment Model"
 primary_key: "session_id"
 description: "Model for identifying cart abandonment events"
 source_definition: "bigquery"
 enabled: true
 account_id: "2rzPI1ARibivIvNH5DSpAKGTATy"
 file: "./queries/cart-abandonment.sql"

Best practices

When defining SQL model resources in YAML files, follow these best practices:

File organization

Logical grouping: Group related SQL models in the same YAML file when they share similar purposes or data sources.
Clear naming: Use descriptive file names that reflect the models’ purpose, such as user-analytics-models.yaml or ecommerce-models.yaml.
Directory structure: Organize SQL files in subdirectories (like ./queries/ or ./sql/) to keep your project structure clean.

Resource configuration

Unique IDs: Choose clear, descriptive IDs that reflect the model’s purpose and avoid conflicts.
Meaningful names: Use display names that clearly describe what the model does for dashboard users.
Documentation: Always include descriptions to help team members understand the model’s purpose.
External files: For complex queries, use the file option to keep SQL in separate files for better syntax highlighting and version control. When importing resources, use the --sql-location parameter if you prefer external SQL files.

Environment management

Variable substitution: Consider using variable substitutions for account_id values when working across multiple environments (development, staging, or production).
Environment-specific configurations: Maintain separate configuration files or use templating for environment-specific settings.

Query optimization

Primary key selection: Choose primary keys that are unique, stable, and efficiently indexed in your warehouse.
Performance considerations: Write efficient queries that minimize warehouse compute costs and execution time.
Data freshness: Include appropriate date filters to control data volume and ensure relevance.

Archive: Event Stream Source YAML Reference

Mon, 01 Jan 0001 00:00:00 +0000

This guide serves as a detailed reference for the CLI project YAML files that contain definitions of your Event Stream source resources.

Overview

In the context of the Rudder CLI (rudder-cli) tool, you can define Event Stream sources as YAML files within your project directory. The location and naming of these YAML files is flexible, as you can store the YAML files anywhere within the project’s root directory or subdirectories.

The Rudder CLI tool processes all valid YAML files within the project structure to recognize the defined resources.

Event Stream Sources

You can define an Event Stream source in the YAML file by setting kind to event-stream-source.

The spec parameter of the YAML file has the following structure:

Property	Type	Description
`id` Required	String	Unique identifier for the source within the project. This parameter must be unique across all sources in the project.
`type` Required	String	Source type identifier. See Supported source types for the list of acceptable values.
`name` Required	String	Name for the source. It must be non-empty.
`enabled`	Boolean	Determines whether this source is accepting events. Default value: `true`
`governance`	Data governance configuration object	Contains data governance-related settings for the source.

Data governance configuration

The governance object contains validation settings that control how the source handles any Tracking Plan violations:

Property	Type	Description
`validations` Required	Validation properties object	Contains validation configuration including Tracking Plan reference and violation rules.

Validation properties

The validations object contains the Tracking Plan reference and violation handling rules:

Property	Type	Description
`tracking_plan` Required	String	Reference to a Tracking Plan resource — the format is `#/tp/[metadata.name]/[tracking-plan.id]`, where `metadata.name` is the `metadata.name` value in your Tracking Plan YAML file and `tracking-plan.id` is the `spec.id` value of the Tracking Plan resource.
`config`	Violation rules configuration object	Contains violation rules for all event types. If not specified, default violation rules apply to all event types.

You must have a Tracking Plan defined in your CLI project before you can reference it in a source configuration.

See CLI-based Tracking Plan Management for more information on creating Tracking Plans.

Violation rules configuration

The config object contains violation handling rules for different event types. Each event type configuration is optional; if not specified, default violation rules apply:

Property	Type	Description
`track`	Violation rules object	Violation rules for `track` events.
`identify`	Violation rules object	Violation rules for `identify` events.
`group`	Violation rules object	Violation rules for `group` events.
`page`	Violation rules object	Violation rules for `page` events.
`screen`	Violation rules object	Violation rules for `screen` events.

Violation rules properties

Each violation rules object contains the following properties:

Property	Type	Description
`propagate_violations`	Boolean	Determines whether to add violations in event context. When enabled, violation information is included in the event payload’s context for downstream processing. Default value: `true`
`drop_unplanned_events`	Boolean	Only applies to `track` configuration. Determines whether to drop events without a corresponding rule in the associated Tracking Plan. Default value: `false`
`drop_unplanned_properties`	Boolean	Determines whether to drop properties that are not defined in the Tracking Plan for the event. Default value: `false`
`drop_other_violations`	Boolean	Determines whether to drop events with any other validation violations. Default value: `false`

If you set drop_unplanned_events, drop_unplanned_properties, or drop_other_violations to true, events that violate these rules are dropped and not forwarded to destinations.

Make sure you understand the impact of these settings before deploying your source configuration.

Supported source types

The following table lists all supported source types and their corresponding type values:

Source	`type` value
Java	`java`
.NET	`dotnet`
PHP	`php`
Flutter	`flutter`
Cordova	`cordova`
Rust	`rust`
React Native	`react_native`
Python	`python`
iOS	`ios`
Android	`android`
JavaScript	`javascript`
Go	`go`
Node	`node`
Ruby	`ruby`
Unity	`unity`

Examples

Basic source configuration

This example shows a minimal source configuration without governance settings:

version: rudder/v0.1
kind: event-stream-source
metadata:
 name: ios-source
spec:
 id: "my-ios-source"
 type: "ios"
 name: "iOS Source"
 enabled: true

Source with governance configuration

This example shows a source with governance configuration that links to a Tracking Plan and defines violation rules for different event types:

version: rudder/v0.1
kind: event-stream-source
metadata:
 name: ios-source
spec:
 id: "my-ios-source"
 type: "ios"
 name: "iOS Source"
 enabled: true
 governance:
 validations:
 tracking_plan: "#/tp/my-tracking-plan/ecommerce-tracking-plan"
 config:
 track:
 propagate_violations: true
 drop_unplanned_events: true
 drop_unplanned_properties: true
 drop_other_violations: true
 identify:
 propagate_violations: true
 drop_unplanned_properties: true
 drop_other_violations: true
 group:
 propagate_violations: true
 drop_unplanned_properties: true
 drop_other_violations: true
 page:
 propagate_violations: true
 drop_unplanned_properties: true
 drop_other_violations: true
 screen:
 propagate_violations: true
 drop_unplanned_properties: true
 drop_other_violations: true

Source with partial governance configuration

This example shows a source that only defines violation rules for track events, letting other event types use default rules:

version: rudder/v0.1
kind: event-stream-source
metadata:
 name: web-source
spec:
 id: "my-web-source"
 type: "javascript"
 name: "Web Source"
 enabled: true
 governance:
 validations:
 tracking_plan: "#/tp/web-tracking-plan/web-tracking-plan"
 config:
 track:
 propagate_violations: false
 drop_unplanned_events: false
 drop_unplanned_properties: false
 drop_other_violations: false

Disabled source

This example shows a source that is disabled and does not accept events:

version: rudder/v0.1
kind: event-stream-source
metadata:
 name: test-source
spec:
 id: "test-source"
 type: "python"
 name: "Test Source"
 enabled: false

Reference Tracking Plans

When configuring data governance validation, you can reference Tracking Plans using the following format:

tracking_plan: "#/tp/[metadata.name]/[tracking-plan.id]"

Where:

metadata.name is the metadata.name value in your Tracking Plan YAML file
tracking-plan.id is the spec.id value of the Tracking Plan

For example, if your Tracking Plan YAML file looks like this:

version: rudder/v0.1
kind: tp
metadata:
 name: ecommerce-tracking-plan
spec:
 id: product-tracking-plan
 display_name: "Product Tracking Plan"
 # ... rest of the configuration

You would reference it in your source configuration as:

tracking_plan: "#/tp/ecommerce-tracking-plan/product-tracking-plan"

References must always start with a # character followed by the path using the / delimiter. The path points to a unique Tracking Plan resource within a project’s file and uses the kind value (tp), metadata.name value, and spec.id value to create the reference.

See more

See Manage Event Stream Sources using Rudder CLI for feature overview, and the Event Stream Sources walkthrough for step-by-step setup
See CLI-based Tracking Plan Management to create the Tracking Plans that you can reference in your Event Stream source configurations
See Data Catalog and Tracking Plans YAML Reference for information about Tracking Plan YAML structure

Archive: Automate Project Management with Rudder CLI and GitHub Actions

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains how to validate and manage your Rudder CLI projects directly via GitHub workflows.

Key features

By leveraging the Rudder CLI Project Manager Action, you can:

Validate your Rudder CLI project files
Perform a dry run of any changes to your project files
Apply the changes to your RudderStack workspace

Prerequisites

A GitHub repository containing your Tracking Plan YAML files

Generate a workspace-level Service Access Token in the RudderStack dashboard with the following permissions to manage Data Catalog and Tracking Plans:

Resource	Permissions
Tracking Plans	Create & Delete, Edit
Data Catalog	Edit

For testing or development purposes only: Generate a Personal Access Token with Read-Write role

RudderStack recommends using a workspace-level Service Access Token for authentication.

Any action authenticated by a Personal Access Token will break if the user is removed from the organization or a breaking change is made to their permissions.

Token permissions for legacy RBAC system

If you are on the legacy Permissions Management (RBAC) system, your workspace-level Service Access Token should have Admin permissions.

See this documentation for more information on generating the token.

Setup

Follow the steps in the below sections in the exact order to set up the GitHub Actions workflow.

Step 1: Configure repository secrets

In your GitHub repository, go to Settings > Secrets and variables > Actions.
Add a new repository secret as follows:

Name: RUDDERSTACK_ACCESS_TOKEN (use this exact name so it matches the workflow below)
Value: The Service Access Token generated in the Prerequisites section

RudderStack recommends storing this token in GitHub Secrets and referencing it in your workflow using ${{ secrets.RUDDERSTACK_ACCESS_TOKEN }}.

Do not expose the token directly in your workflow files.

Step 2: Create Actions workflow

Create the following workflow in your .github/workflows/ directory:

name: Manage Rudder CLI projects

on:
 push:
 branches: [main]
 pull_request:
 branches: [main]

jobs:
 validate:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - name: Validate Project Files
 uses: rudderlabs/rudder-cli-action@v1.0.1
 env:
 RUDDERSTACK_ACCESS_TOKEN: ${{ secrets.RUDDERSTACK_ACCESS_TOKEN }}
 with:
 location: "<path_to_root_project_folder>/"
 mode: "validate"

 apply:
 runs-on: ubuntu-latest
 needs: validate
 if: github.ref == 'refs/heads/main'
 steps:
 - uses: actions/checkout@v4
 - name: Apply Project Files
 uses: rudderlabs/rudder-cli-action@v1.0.1
 env:
 RUDDERSTACK_ACCESS_TOKEN: ${{ secrets.RUDDERSTACK_ACCESS_TOKEN }}
 with:
 location: "<path_to_root_project_folder>/" #Required
 mode: "apply" #Required

Note that:

The previous Action rudderlabs/rudder-tracking-plan-action@v1.0.0 is now deprecated.

You can update the CLI action version by modifying the uses parameter in the workflow file — see the Rudder CLI Project Manager Action releases for the latest version.

Some of the key inputs are described below:

Input	Description	Default value
`location` Required	Path to the folder containing the Rudder CLI project files.	-
`mode` Required	Operation mode — acceptable values are `validate`, `dry-run`, and `apply`. See Modes for more information.	-
`cli_version`	Version of the Rudder CLI tool to use.	`v0.10.0`
`RUDDERSTACK_ACCESS_TOKEN` Required	The access token for the RudderStack workspace.	-

Modes

The mode parameter defines the operation mode of the GitHub Action. You can specify either of the following values:

Value	Notes
`validate`	Validates Tracking Plan syntax and structure. No changes are applied to your configuration.
`dry-run`	Simulates the application of changes. Shows what would be modified without actually making the changes.
`apply`	Applies the relevant changes to your RudderStack workspace.

How it works

This section explains the GitHub Actions workflow:

The action only triggers when files in your root project directory are modified.
The action automatically syncs the changes with RudderStack when you merge them with the main branch. It uses the apply mode (mode: apply) to push the relevant changes.

Archive: Automate Project Management with Rudder CLI and GitLab CI/CD

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows how to run the same validate and apply workflow you use locally inside a GitLab pipeline. The pattern mirrors GitHub Actions for Rudder CLI: check out your repository, install or invoke Rudder CLI, pass a workspace token, then run rudder-cli validate and rudder-cli apply (often with --dry-run on merge requests).

Prerequisites

A GitLab project with your Rudder CLI YAML project checked in
A workspace-level Service Access Token with permissions for the resources you manage (see Import Workspace Resources for a full permission matrix)
Store the token as a masked CI/CD variable (for example RUDDERSTACK_ACCESS_TOKEN)

Example pipeline

The following .gitlab-ci.yml sketches a two-stage flow: validate on every branch, apply only on the default branch after validation succeeds. Adjust image, paths, and branch rules to match your environment.

stages:
 - validate
 - apply

variables:
 RUDDER_PROJECT_PATH: "$CI_PROJECT_DIR/my-rudder-project"

validate:
 stage: validate
 image: rudderlabs/rudder-cli
 script:
 - rudder-cli validate -l "$RUDDER_PROJECT_PATH"
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 - if: $CI_COMMIT_BRANCH

apply:
 stage: apply
 image: rudderlabs/rudder-cli
 script:
 - rudder-cli apply -l "$RUDDER_PROJECT_PATH" --dry-run
 - rudder-cli apply -l "$RUDDER_PROJECT_PATH"
 rules:
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
 needs: ["validate"]

Use the official CLI image only if it matches the version you run locally. Alternatively install the binary in the job (for example from GitHub releases) so versions stay aligned with Install Rudder CLI.

See more

GitHub Actions for Rudder CLI for the maintained GitHub Action and workflow patterns
Commands for core rudder-cli subcommands