ingestr

ingestr is an open-source CLI tool by Bruin that copies data from any source to any destination using a single command. SQLBuild integrates with ingestr as a declarative source loader - you configure the ingestion directly in your source YAML, and SQLBuild handles execution as part of the build lifecycle.

Install

pip install 'sqlbuild[ingestr]'
# or
uv pip install 'sqlbuild[ingestr]'

This installs ingestr alongside SQLBuild. The ingestr CLI must be available on PATH.

ingestr reads and writes DuckDB through an ADBC driver. Before the first ingestr run, SQLBuild installs the DuckDB driver that matches its own duckdb package (with dbc, included in the extra) into <environment>/etc/adbc/drivers, so the files ingestr writes stay readable by SQLBuild. Set ADBC_DRIVER_PATH to manage the driver yourself.

How it works

1. Declare an ingestr block on a source in sources/*.yml
2. SQLBuild generates a synthetic loader that calls ingestr ingest as a subprocess
3. ingestr reads from the configured source and writes directly to the SQLBuild adapter’s database
4. The destination URI is built automatically from your SQLBuild connection config - no manual destination setup

No Python loader code is needed. The YAML declaration is the entire configuration.

Example: PostgreSQL to DuckDB

Replicate a table from PostgreSQL into your local DuckDB project:

sources/raw.yml

sources:
  - name: raw_orders
    description: Orders
    table: orders
    ingestr:
      source_uri: "postgresql://user:pass@host:5432/mydb"
      source_table: "public.orders"

That’s it. Run sqb load or sqb build and ingestr copies the orders table into your project.

Example: Stripe with incremental merge

Load Stripe charges with incremental merge on a primary key:

sources:
  - name: raw_stripe_charges
    description: Stripe charges
    table: charges
    ingestr:
      source_uri: "stripe://${stripe_api_key}"
      source_table: "charges"
      strategy: merge
      primary_key: id
      incremental_key: created

On subsequent runs, ingestr merges new and updated records based on the id column, using created to detect changes.

Configuration reference

The ingestr block on a source supports the following fields:

Field	Required	Description
source_uri	Yes	ingestr source connection URI (e.g. postgresql://..., stripe://..., shopify://...)
source_table	Yes	Source table or resource name to ingest
strategy	No	Ingestion strategy: replace, append, merge, delete+insert, or truncate+insert
incremental_key	No	Column used for incremental change detection
primary_key	No	Primary key column(s) for merge strategy (string or list)
columns	No	Comma-separated column list to select from the source
extra_args	No	Additional CLI arguments passed to ingestr ingest (list of strings)

Strategy mapping

Strategy	Behavior
replace	Drop and recreate the destination table (default when no strategy is set)
append	Insert all rows without deduplication
merge	Upsert based on primary_key, using incremental_key for change detection
delete+insert	Delete matching rows by incremental_key range, then insert replacements
truncate+insert	Truncate destination table, then insert all rows

Template variables

All string fields in the ingestr block support SQLBuild project variable substitution and context templates:

sources:
  - name: raw_orders
    description: Orders
    ingestr:
      source_uri: "postgresql://${pg_user}:${pg_password}@${pg_host}:5432/${pg_database}"
      source_table: "public.orders"

Variables are resolved from the project’s merged variable config (project + target + local).

Set sensitive values in sqlbuild_local.toml (gitignored):

[vars]
pg_user = "readonly"
pg_password = "secret"
pg_host = "prod-db.example.com"
pg_database = "analytics"
stripe_api_key = "sk_live_..."

Destination URI

SQLBuild automatically builds the ingestr destination URI from your adapter connection config. All supported adapters work without manual destination configuration:

Adapter	Destination URI format
DuckDB	duckdb:///path/to/db.duckdb
SQLite	sqlite:///path/to/db.sqlite
MotherDuck	motherduck://database
PostgreSQL	postgresql://user:pass@host:port/db
Snowflake	snowflake://user:pass@account/db/schema
BigQuery	bigquery://project
Databricks	databricks://token:...@host
SQL Server	mssql://user:pass@host:port/db

Reload

When --reload is passed, ingestr runs with --full-refresh, forcing a complete reload regardless of the configured strategy:

sqb load --reload
sqb build --reload

Build integration

ingestr sources are managed sources - they participate in the same lifecycle as Python loaders:

# ingestr runs automatically before dependent models
sqb build

# run ingestr sources standalone
sqb load

# skip loading
sqb build --no-load

See Loaders for details on auto-load behavior, source deferral, and the --load / --no-load / --reload flags.

Supported sources

ingestr supports 50+ sources including databases, SaaS APIs, and file systems. See the ingestr documentation for the full list of supported sources and their URI formats.