ELT pipeline example: Getting started with Kestra

A getting started ELT pipeline that runs successfully on the very first execution with zero secrets. It downloads a public orders CSV, aggregates it in in-process DuckDB, and logs the top products by revenue. Once you have seen it run green, commented connector blocks let you point the same pipeline at your own warehouse (Snowflake, Postgres, BigQuery, or Redshift) and add a Slack alert. It is the fastest way to see the Extract, Load, Transform pattern working end to end in Kestra before wiring in real credentials.

How it works

1. The extract task (io.kestra.plugin.core.http.Download) fetches a public sample orders.csv from the Kestra datasets repository. No authentication is required.
2. The transform task (io.kestra.plugin.jdbc.duckdb.Query) loads that CSV into embedded DuckDB via inputFiles and runs a GROUP BY product_id aggregation, ranking products by revenue with fetchType: FETCH. communityExtensions: [] skips a version-mismatched bundled extension so the query runs cleanly.
3. The report task (io.kestra.plugin.core.log.Log) logs the processed row count and the top seller so the first run shows a real, human-readable result.
4. The optional notify task posts a Slack summary once you enable it.

What you get

• A working ELT run on the first click, with no accounts or keys to configure.
• A realistic aggregation in embedded DuckDB you can read and adapt.
• Drop-in connector blocks for Snowflake, Postgres, BigQuery, and Redshift.
• Optional Slack alerting and webhook or schedule triggers.

Who it's for

• Data engineers evaluating Kestra for ELT and warehouse loading.
• Anyone new to Kestra who wants a green first run before adding credentials.
• Teams standardizing a reusable extract, transform, report skeleton.

Why orchestrate this with Kestra

A one-off script can download a CSV and run SQL, but it cannot version the pipeline, retry a failed load, track execution lineage, or swap the compute engine without a rewrite. Kestra models the whole flow as declarative YAML: the same report step reads outputs.transform no matter which warehouse produces it, so moving from DuckDB to Snowflake is a single task swap. You also get scheduling, event triggers, full logs, and replay out of the box.

Prerequisites

• A running Kestra instance with the DuckDB JDBC plugin available.
• (Optional) A warehouse and its credentials if you swap in your own connector.
• (Optional) A Slack incoming webhook for notifications.

Secrets

The default green path uses no secrets. Optional steps use:

• SLACK_WEBHOOK: Slack incoming webhook URL for the notify task.
• SNOWFLAKE_PASSWORD / POSTGRES_PASSWORD / REDSHIFT_PASSWORD / BQ_SERVICE_ACCOUNT: credentials for whichever warehouse connector you enable.

Quick start

1. Add this flow to your Kestra instance and Execute it. It runs green with no setup.
2. To read from your own warehouse, copy ONE connector block from the source comments and paste it OVER the DuckDB transform task (keep id: transform so report still works).
3. Set the matching secret under Tenant > Secrets.
4. (Optional) Remove disabled: true from notify and from a trigger to alert and automate.

How to extend

• Replace the sample CSV with your own source (S3, GCS, an API).
• Add data-quality checks (row counts, null checks) before report.
• Fan out to multiple warehouses or add a dbt transformation step.
• Enable the schedule or webhook trigger to run it automatically.

Links

• DuckDB plugin
• HTTP Download task