Skip to content

Concepts

Sources

Declare external data inputs for your pipeline.

Sources declare external data that your models depend on. They are defined in YAML files under sources/ in your project directory.

Point at an existing table or view in your warehouse:

sources:
- name: raw_events
database: analytics
schema: raw
table: events

Reference it in models with __source("raw_events").

Define source data inline as a SQL expression. No external tables or setup scripts needed:

sources:
- name: raw__customers
expression: |
SELECT * FROM (VALUES
(1, 'Leslie', 'Knope', 'leslie@pawnee.gov', TIMESTAMP '2026-01-15 09:00:00'),
(2, 'Ron', 'Swanson', 'ron@pawnee.gov', TIMESTAMP '2026-02-01 08:00:00')
) AS raw__customers(id, first_name, last_name, email, created_at)

Expression sources are resolved at compile time. They’re the escape hatch for anything the framework doesn’t natively model: external tables, warehouse-specific syntax, function calls, or any other relation type that doesn’t fit a standard table reference.

An inline source expression uses macros, constants, and enums available from its source definition under sources/. Declarations limited to one source folder are not available in sibling folders. See How Visibility Works.

Sources support the same audit system as models. Audits attached to sources run before any dependent model is built:

sources:
- name: raw_orders
columns:
- name: id
audits:
- not_null
- unique
audits:
- expression_is_true:
name: no future orders
expression: "ordered_at <= CURRENT_TIMESTAMP"

If a source audit with error severity fails, all downstream models that depend on that source are blocked.

Type enforcement is implicit for sources. If any column declares a type, SQLBuild activates source type enforcement, casts that column during source resolution, and uses declared types for schema-change detection:

sources:
- name: raw__customers
expression: |
SELECT 1 AS id, 'Leslie' AS first_name, 'Knope' AS last_name
columns:
- name: id
type: INTEGER
- name: first_name
- name: last_name

In this example, only id has a type declared, so only id is cast. Columns without a type are passed through unchanged.

For expression sources, SQLBuild probes the expression’s output columns and builds a projection that casts typed columns while preserving the rest. For table sources, it uses warehouse metadata to validate that declared column names exist and applies casts accordingly.

You can explicitly set type_enforcement: false on a source to disable casting even when column types are declared.

Sources can be loaded by Python functions instead of pointing at existing tables or inline expressions. Set managed: true to bind a source to the @loader function of the same name, and SQLBuild will call it to populate the source table:

sources:
- name: raw_customers
managed: true
write_strategy: table
columns:
- name: id
type: INTEGER
- name: name
type: VARCHAR

Managed sources support incremental write strategies (table, append, delete_insert, merge), cursor-based loading, and concurrent execution.

See Loaders for the full guide on writing loader functions, write strategies, the loader context API, and auto-load behavior during builds.

For common ingestion you can declare the source entirely in YAML, with no @loader function. SQLBuild generates the loader for you and runs it during the build:

  • dlt - declare dlt_sources for rest_api, sql_database, and filesystem sources.
  • ingestr - add an ingestr block to a source to pull from 50+ sources.

Source freshness lets SQLBuild observe whether a source’s data changed between runs. This feeds into planning and change detection: changed observations propagate to downstream models and can alter their planned action.

Configure freshness per source with a freshness: block:

sources:
- name: raw_events
schema: raw
table: events
freshness:
strategy: column
column: updated_at
type: timestamp
lag_tolerance: 15m
Strategy Description Required fields
adapter Uses warehouse metadata (e.g. Snowflake LAST_ALTERED). No query against the source table. None
column Reads MAX(column) from the source table. column, type
sql Runs a custom query that returns a single scalar value. query, type
freshness:
strategy: adapter

Uses adapter-level table metadata. Supported on Snowflake, BigQuery, Databricks, PostgreSQL, DuckDB, and SQL Server. Does not support type, column, or query.

freshness:
strategy: column
column: updated_at
type: timestamp

Queries MAX(column) from the source table. The column must be a plain column name (no expressions; use sql strategy for those). Requires type.

freshness:
strategy: sql
query: "SELECT MAX(version_id) FROM raw.events"
type: integer

Runs an arbitrary SQL query that returns a single scalar. Requires type. Does not support column.

The type field declares the value kind for comparison:

Type Description
timestamp Datetime value. Supports lag_tolerance.
integer Integer value. Change detected by exact comparison.
string String value. Change detected by exact comparison.

lag_tolerance is optional and only valid with type: timestamp. It declares how much the observed timestamp can drift from the previous observation before being treated as a change:

freshness:
strategy: column
column: updated_at
type: timestamp
lag_tolerance: 2h

Accepts positive durations: 15m (minutes), 2h (hours), 1d (days). If the current observation is within the tolerance of the previous one, the source is treated as unchanged.

Sources without an explicit freshness: block are auto-observed using the adapter strategy if:

  • The source has a physical table (not an expression source)
  • The source is not managed
  • The adapter supports table freshness metadata

This means most unmanaged table sources get freshness tracking automatically on adapters that support it, with no configuration needed.

Use sqb freshness to observe source freshness on demand without triggering a build. See Source freshness for how freshness feeds into change-aware builds.

Field Description
strategy Observation strategy: adapter, column, or sql
type Value kind: timestamp, integer, or string
column Column name for column strategy
query SQL query for sql strategy
lag_tolerance Duration tolerance for timestamp comparisons (e.g. 15m, 2h, 1d). Only valid with type: timestamp.
Field Description
name Source name, used in __source("name") references
database Target database (optional)
schema Physical source schema (optional). For managed sources, overrides target loader_schema; otherwise managed loaders fall back to loader_schema, then target schema.
table Target table name (defaults to name if omitted)
expression Inline SQL expression (alternative to table reference)
managed Set to true to bind the source to the @loader function of the same name (see Loaders)
write_strategy How the loader writes data: table, append, delete_insert, or merge (requires managed: true)
cursor_column Column for incremental cursor tracking (required for delete_insert and merge)
unique_key Merge key column(s) (required for merge)
freshness Source freshness observation config (see Source freshness)
description Human-readable description
type_enforcement Override implicit type enforcement (true/false). Defaults to true when any column declares a type.
contract enforced or none. When enforced, downstream models validate configured column references against source columns.
columns Column declarations with optional types and audits
audits Source-level audits