Event Stream Source YAML Reference Beta

This guide serves as a detailed reference for the CLI project YAML files that contain definitions of your Event Stream source resources.

Overview

In the context of the Rudder CLI (rudder-cli) tool, you can define Event Stream sources as YAML files within your project directory. The location and naming of these YAML files is flexible, as you can store the YAML files anywhere within the project’s root directory or subdirectories.

The Rudder CLI tool processes all valid YAML files within the project structure to recognize the defined resources.

Event Stream Sources

You can define an Event Stream source in the YAML file by setting kind to event-stream-source.

The spec parameter of the YAML file has the following structure:

Data governance configuration

The governance object contains validation settings that control how the source handles any Tracking Plan violations:

Validation properties

The validations object contains the Tracking Plan reference and violation handling rules:

You must have a Tracking Plan defined in your CLI project before you can reference it in a source configuration.

See CLI-based Tracking Plan Management for more information on creating Tracking Plans.

Violation rules configuration

The config object contains violation handling rules for different event types. Each event type configuration is optional; if not specified, default violation rules apply:

Violation rules properties

Each violation rules object contains the following properties:

If you set drop_unplanned_events, drop_unplanned_properties, or drop_other_violations to true, events that violate these rules are dropped and not forwarded to destinations.

Make sure you understand the impact of these settings before deploying your source configuration.

Validation rules

version: rudder/v1
kind: event-stream-source
metadata:
  name: my-source
spec:
  id: src-1
  name: My Python Source
  type: python

version: rudder/v1
kind: event-stream-source
metadata:
  name: my-source
spec:
  id: src-1
  name: My Python Source
  type: python
  governance:
    validations:
      tracking_plan: "#tracking-plan:tp-main"
      config: {}

version: rudder/v1
kind: tracking-plan
metadata:
  name: tp-main
spec:
  id: tp-main
  display_name: Main Tracking Plan
  description: Default plan
  rules: []

version: rudder/v1
kind: event-stream-source
metadata:
  name: my-source
spec:
  id: src-1
  name: My Python Source
  type: python
  governance:
    validations:
      tracking_plan: "#tracking-plan:tp-missing"
      config: {}

version: rudder/v1
kind: event-stream-source
metadata:
  name: source-a
spec:
  id: src-a
  name: Shared Name
  type: javascript

version: rudder/v1
kind: event-stream-source
metadata:
  name: source-b
spec:
  id: src-b
  name: Shared Name
  type: node

version: rudder/0.1
kind: event-stream-source
metadata:
  name: my-source
spec:
  id: src-1
  name: My JavaScript Source
  type: javascript

version: rudder/0.1
kind: event-stream-source
metadata:
  name: my-source
spec:
  id: src-1
  name: My JavaScript Source
  type: javascript
  governance:
    validations:
      tracking_plan: "#/tp/my-group/tp-main"
      config: {}

version: rudder/0.1
kind: tp
metadata:
  name: tp-main
spec:
  id: tp-main
  display_name: Main Tracking Plan
  description: Default plan
  rules: []

version: rudder/0.1
kind: event-stream-source
metadata:
  name: my-source
spec:
  id: src-1
  name: My JavaScript Source
  type: javascript
  governance:
    validations:
      tracking_plan: "#/tp/my-group/tp-missing"
      config: {}

version: rudder/0.1
kind: event-stream-source
metadata:
  name: source-a
spec:
  id: src-a
  name: Shared Name
  type: javascript

version: rudder/0.1
kind: event-stream-source
metadata:
  name: source-b
spec:
  id: src-b
  name: Shared Name
  type: node

version: rudder/v1
kind: event-stream-source
metadata:
  name: my-python-source
spec:
  id: src-py-1
  name: My Python Source
  type: python

version: rudder/v1
kind: event-stream-source
metadata:
  name: my-android-source
spec:
  id: src-android-1
  name: My Android Source
  type: android
  governance:
    validations:
      tracking_plan: "#tracking-plan:tp-main"
      config: {}

version: rudder/v1
kind: event-stream-source
metadata:
  name: my-source
spec:
  id: src-1
  type: javascript

version: rudder/v1
kind: event-stream-source
metadata:
  name: my-source
spec:
  id: src-1
  name: My Source
  type: javascript
  governance:
    validations:
      tracking_plan: "#/tp/my-group/tp-1"
      config: {}

version: rudder/0.1
kind: event-stream-source
metadata:
  name: my-js-source
spec:
  id: src-js-1
  name: My JavaScript Source
  type: javascript

version: rudder/0.1
kind: event-stream-source
metadata:
  name: my-node-source
spec:
  id: src-node-1
  name: My Node.js Source
  type: node
  governance:
    validations:
      tracking_plan: "#/tp/my-group/tp-main"
      config: {}

version: rudder/0.1
kind: event-stream-source
metadata:
  name: my-source
spec:
  name: My Source
  type: javascript

version: rudder/0.1
kind: event-stream-source
metadata:
  name: my-source
spec:
  id: src-1
  name: My Source
  type: unknown_sdk

version: rudder/0.1
kind: event-stream-source
metadata:
  name: my-source
spec:
  id: src-1
  name: My Source
  type: javascript
  governance:
    validations:
      tracking_plan: "#tracking-plan:tp-1"
      config: {}

Supported source types

The following table lists all supported source types and their corresponding type values:

Examples

Basic source configuration

This example shows a minimal source configuration without governance settings:

version: rudder/v1
kind: event-stream-source
metadata:
  name: ios-source
spec:
  id: "my-ios-source"
  type: "ios"
  name: "iOS Source"
  enabled: true

Source with governance configuration

This example shows a source with governance configuration that links to a Tracking Plan and defines violation rules for different event types:

version: rudder/v1
kind: event-stream-source
metadata:
  name: ios-source
spec:
  id: "my-ios-source"
  type: "ios"
  name: "iOS Source"
  enabled: true
  governance:
    validations:
      tracking_plan: "#tracking-plan:ecommerce-tracking-plan"
      config:
        track:
          propagate_violations: true
          drop_unplanned_events: true
          drop_unplanned_properties: true
          drop_other_violations: true
        identify:
          propagate_violations: true
          drop_unplanned_properties: true
          drop_other_violations: true
        group:
          propagate_violations: true
          drop_unplanned_properties: true
          drop_other_violations: true
        page:
          propagate_violations: true
          drop_unplanned_properties: true
          drop_other_violations: true
        screen:
          propagate_violations: true
          drop_unplanned_properties: true
          drop_other_violations: true

Source with partial governance configuration

This example shows a source that only defines violation rules for track events, letting other event types use default rules:

version: rudder/v1
kind: event-stream-source
metadata:
  name: web-source
spec:
  id: "my-web-source"
  type: "javascript"
  name: "Web Source"
  enabled: true
  governance:
    validations:
      tracking_plan: "#tracking-plan:web-tracking-plan"
      config:
        track:
          propagate_violations: false
          drop_unplanned_events: false
          drop_unplanned_properties: false
          drop_other_violations: false

Disabled source

This example shows a source that is disabled and does not accept events:

version: rudder/v1
kind: event-stream-source
metadata:
  name: test-source
spec:
  id: "test-source"
  type: "python"
  name: "Test Source"
  enabled: false

Reference Tracking Plans

When configuring data governance validation, you can reference Tracking Plans using the following URN format:

tracking_plan: "#tracking-plan:[tracking-plan.id]"

Where tracking-plan.id is the spec.id value of the Tracking Plan.

For example, if your Tracking Plan YAML file looks like this:

version: rudder/v1
kind: tracking-plan
metadata:
  name: ecommerce-tracking-plan
spec:
  id: product-tracking-plan
  display_name: "Product Tracking Plan"
  # ... rest of the configuration

You would reference it in your source configuration as:

tracking_plan: "#tracking-plan:product-tracking-plan"

See more

• See Manage Event Stream Sources using Rudder CLI for feature overview, and the Event Stream Sources walkthrough for step-by-step setup
• See CLI-based Tracking Plan Management to create the Tracking Plans that you can reference in your Event Stream source configurations
• See Data Catalog and Tracking Plans YAML Reference for information about Tracking Plan YAML structure
• See Manage Destinations using Rudder CLI and Connection YAML Reference to send events from this source to a destination