Gate Downstream Flows on Dataform Assertions

Dataform assertions are data quality tests written next to the models, uniqueness, non-null, or any custom SQL condition, and dataform run executes them as part of the DAG, exiting non-zero when any assertion fails. What the CLI cannot do is decide what happens next. This blueprint makes the decision explicit through io.kestra.plugin.dataform.cli.DataformCLI: the run's exit code is captured as a boolean flow output, Discord receives a verdict either way, and io.kestra.plugin.core.execution.Fail turns broken assertions into a hard red execution that dependent flows and schedules cannot miss.

How it works

1. run_with_assertions (io.kestra.plugin.dataform.cli.DataformCLI) loads the project from namespace files, installs the pinned @dataform/core dependency in beforeCommands, and runs dataform run, which executes models and assertions in dependency order against BigQuery.
2. The command runs with || rc=$?, so a failing assertion records the exit code instead of failing the task before the verdict is posted. A shell conditional converts the code to a boolean, emitted through Kestra's output protocol as {{ outputs.run_with_assertions.vars.run_ok }}.
3. gate (io.kestra.plugin.core.flow.If) branches on the flag. On success, Discord gets a pass message. On failure, Discord gets the reason first, then io.kestra.plugin.core.execution.Fail fails the execution with an explicit error message.
4. The errors block posts a distinct alert when the flow crashes outside the controlled gate path.

What you get

• Assertions promoted from log lines to a hard gate, a failed check produces a failed execution.
• A Discord verdict on every run, pass or fail, with the failure reason posted before the execution goes red.
• Downstream flows that can safely depend on this execution's state instead of re-checking data quality themselves.
• The exit-code capture pattern, reusable for any CLI whose exit code should become a flow decision.

Who it's for

• Analytics engineers who write Dataform assertions and want them to actually stop bad data, not just log it.
• Data platform teams wiring quality gates between transformation and the flows that consume the results.
• Anyone whose dashboards have confidently displayed data that a failed assertion had already flagged.

Why orchestrate this with Kestra

A cron job running dataform run exits non-zero into the void, nothing downstream notices, and the failure lives only in a log file. In Kestra, the exit code becomes a structured output, the decision becomes a visible If branch, the failure becomes a red execution with an explicit message, and everything downstream, dependent flows, schedules, alerting, reacts to that state automatically.

Prerequisites

• A Dataform project with assertions, assertions in config blocks or standalone assertion SQLX files, stored as namespace files.
• A BigQuery service account with permission to run jobs and write to the target and assertion datasets.
• A Discord incoming webhook for verdicts and crash alerts.

Secrets

• GCP_SERVICE_ACCOUNT_JSON: the service account key JSON, delivered to the task as sa.json.
• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.

Quick start

1. Add the GCP_SERVICE_ACCOUNT_JSON and DISCORD_WEBHOOK_URL secrets to your Kestra namespace.
2. Upload your Dataform project to the namespace files.
3. Execute the flow and check that Discord posts a pass verdict.
4. Break an assertion on purpose, rerun, and confirm Discord reports the failure and the execution ends in a failed state.

How to extend

• Run assertions on a tagged slice only by adding --tags to the command, as shown in the tag-scoped run blueprint.
• Trigger the gated run when upstream data lands, as shown in the incremental refresh blueprint.
• Add --actions to rerun a single failing assertion during incident triage without rebuilding the DAG.
• Chain a downstream flow on the SUCCESS state of this one with a Flow trigger, so consumers start only behind a green gate.

Links

• Dataform plugin documentation
• Dataform documentation