Starflow

Documentation • Installation • Discord • VS Code Extension • Starlake Skills • Data Stacks • Contributing

Test on your laptop. Ship to your warehouse. Same SQL, same YAML, no rewrite.

Starflow is a declarative Extract · Load · Transform · Orchestrate engine: describe what your pipeline does in YAML and plain SQL, and Starflow works out how: schemas, merges, quality checks, lineage, and the DAGs to run it all.

And because it transpiles your warehouse SQL to DuckDB, the whole pipeline, loads included, runs and tests locally in seconds. Engine choice becomes an environment variable, not a replatforming program.

Describe the table once: file pattern, merge strategy, validation:

# metadata/load/crm/customers.sl.yml
table:
  pattern: "customers.*.csv"              # files landing in your incoming folder
  metadata:
    writeStrategy:
      type: UPSERT_BY_KEY_AND_TIMESTAMP   # incremental merge, no MERGE SQL to write
      key: [id]
      timestamp: signup
  attributes:
    - name: id
      type: string
      required: true
    - name: signup
      type: timestamp
    - name: email
      type: string                        # or one of your own semantic types, validated at load time

Then let Starflow do the engineering:

starlake bootstrap                        # scaffold a project
starlake load                             # infer, validate and merge into your warehouse
starlake transform --name kpi.revenue     # run SQL with the right MERGE/INSERT logic

The Airflow DAG above was generated from the SQL dependencies. Nobody wrote it.

Your transforms are written for BigQuery or Snowflake. Starflow transpiles them to DuckDB, so the whole pipeline runs on your laptop. No dev warehouse. No waiting. No bill.

A test is just a folder: the input you feed in, the output you expect back.

metadata/tests/transform/sales_kpi/byseller_kpi/test1/
├── sales.orders.csv        # input fixture
├── sales.customers.json    # input fixture
└── _expected.csv           # what the transform must produce

starlake test                                  # every load and transform, on local DuckDB
starlake test --transform --domain sales_kpi   # just this one
starlake test --site                           # HTML report with coverage

• No SQL rewriting: your warehouse dialect is transpiled, not reimplemented by hand.
• Loads are tested too: parsing, type validation, merge strategy and rejected rows, not just SQL.
• One project, any engine: SL_ENV=DUCKDB on the laptop, SL_ENV=BQ in production. Same YAML, same SQL.

• Local-first: the full pipeline runs on DuckDB in CI and on your laptop, on your warehouse in production, from one codebase.
• Config, not code: YAML and plain SQL replace bespoke ETL scripts and orchestration glue.
• Any source, any warehouse, any orchestrator: files, JDBC databases and Kafka into BigQuery, Snowflake, Redshift, DuckDB, PostgreSQL, Delta Lake or Iceberg, scheduled on Airflow, Dagster or Snowflake Tasks with generated DAGs.
• Quality and lineage built in: expectations run at load time, and table- and column-level lineage falls out of your SQL automatically.
• Privacy by declaration: column-level encryption, row- and column-level security policies applied from the same YAML.
• AI-assistant-ready: MCP-based Starlake Skills teach Claude Code and GitHub Copilot to build and debug pipelines with you.

macOS / Linux:

bash <(curl -sL https://starlake.ai/setup.sh)

Windows (PowerShell):

Invoke-Expression (Invoke-WebRequest -Uri "https://raw.githubusercontent.com/starlake-ai/starflow/master/distrib/setup.ps1" -UseBasicParsing).Content

Docker:

docker run -it starlakeai/starlake:latest starlake bootstrap

The product is Starflow; the CLI keeps its historical name starlake.

For pre-built production-ready data stacks, see Starlake Pragmatic Data Stacks.

Pull data from any JDBC source with a few lines of YAML:

extract:
  connectionRef: "pg-adventure-works-db"
  jdbcSchemas:
    - schema: "sales"
      tables:
        - name: "salesorderdetail"
          partitionColumn: "salesorderdetailid"  # parallel extraction
          timestamp: salesdatetime               # incremental

Point Starflow at your files: it infers schemas, validates every row against the declared types and expectations, and applies the merge strategy you declared. Malformed lines are quarantined into an audit trail and a replay file instead of failing the load.

Write plain SQL; Starflow wraps it in the correct MERGE/INSERT/OVERWRITE logic for your warehouse:

SELECT
  productid,
  SUM(unitprice * orderqty) AS total_revenue
FROM salesorderdetail
GROUP BY productid
ORDER BY total_revenue DESC

transform:
  tasks:
    - name: most_profitable_products
      writeStrategy:
        type: "UPSERT_BY_KEY_AND_TIMESTAMP"
        timestamp: order_date
        key: [productid]

Starflow extracts the dependencies between your loads and transforms and generates the DAGs. Reference built-in templates for Airflow, Dagster, or Snowflake Tasks in your YAML. No custom DAG code required.

The Starlake VS Code Extension brings Starflow into your editor: schema inference, SQL transformations, ER diagrams, lineage visualization, and workflow orchestration without leaving VS Code.

It ships with Starlake Skills, MCP-based skills that give AI coding assistants like Claude Code and GitHub Copilot deep knowledge of the platform, so your assistant builds, debugs, and optimizes pipelines using Starflow best practices.

• Discord: questions, feedback and release news. Join us
• Docs: guides, concepts, and the full configuration reference at docs.starlake.ai
• Contributing: see the Contributing Guide and Code of Conduct

Apache License 2.0. See LICENSE for details.