Schema Contract Guard for dlt Pipelines

Schema drift is how upstream teams break your dashboards without telling you. This blueprint runs a dlt load under a frozen schema contract: schema_contract={"columns": "freeze", "data_type": "freeze"} makes any new column or changed data type raise inside dlt's normalize step, before a single row reaches the destination. The flow fails loudly, Discord gets a page explaining that the destination is untouched, and the team decides consciously whether to fix the upstream or evolve the contract, instead of discovering a mystery column in production three weeks later.

How it works

1. load_with_contract (io.kestra.plugin.dlt.Run) declares the resource with a schema_contract freezing both columns and data types; the first run establishes the baseline schema, and every subsequent run is validated against it.
2. When the incoming data matches, the load merges normally and the script emits row counts as Kestra outputs; confirm posts a quiet success note to Discord.
3. When the source drifts, dlt raises a contract violation, the task fails, and the errors block pages Discord with instructions: the destination was not modified, check the logs for the offending column.
4. A disabled-by-default daily Schedule turns schema drift into a failed morning execution instead of silently corrupted downstream models.

What you get

• A hard guarantee that the destination schema only changes when a human decides it should.
• Violations caught at normalize time, before any write, so there is nothing to roll back.
• An alert that distinguishes schema drift from ordinary load failures.
• A one-line dial per resource: freeze, evolve, or discard rows and values selectively.

Who it's for

• Data teams whose upstream services ship column changes without notice.
• Platform engineers formalizing data contracts between producers and the warehouse.
• Anyone who has debugged a silently added column that broke a downstream join.

Why orchestrate this with Kestra

A contract violation is only useful if someone hears about it. Kestra turns dlt's exception into an operational event: a failed execution with full logs, a Discord page from the errors block, retries that make sense for transient failures but not for contract violations, and a history showing exactly when the source started drifting. The governance policy lives in twenty lines of version-controlled YAML plus Python.

Prerequisites

• A persistent path for the DuckDB file (the warehouse_path input); the baseline schema builds on the first run.
• A Discord incoming webhook routed to the data-platform channel.
• For real sources, credentials passed through task env from secrets.

Secrets

• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.

Quick start

1. Add the DISCORD_WEBHOOK_URL secret to your Kestra namespace.
2. Execute once to establish the baseline schema and see the success note.
3. Point the resource at your real source, keep the contract, and set disabled: false on the daily trigger.
4. Test the guard by adding a field upstream: the next run must fail with a contract violation.

How to extend

• Relax selectively: {"columns": "evolve", "data_type": "freeze"} allows new columns while still blocking type changes.
• Use discard_row or discard_value modes to quarantine non-conforming data instead of failing the load.
• Route violations to PagerDuty instead of Discord for on-call escalation.
• Pair with the dlt incremental sync blueprint to put contracts on continuously replicated tables.

Links

• dlt plugin documentation
• dlt schema contracts