Weekend Full Refresh of Incremental Dataform Models

Incremental models drift. Late-arriving rows miss their window, upstream backfills rewrite history the merge never sees, and a changed transformation only applies to rows processed after the change. The standard correction is a periodic full refresh, dataform run --full-refresh rebuilds incremental tables from scratch instead of appending to them, but it is the most expensive run a project has, which is exactly why it should happen in a controlled window rather than whenever someone remembers. This blueprint schedules it through io.kestra.plugin.dataform.cli.DataformCLI for early Sunday and reports to Discord.

How it works

1. The sunday_window trigger (io.kestra.plugin.core.trigger.Schedule, disabled by default) fires at 03:00 on Sunday, a slot where a slot-hungry rebuild competes with no dashboards and no business-hours pipelines.
2. full_refresh (io.kestra.plugin.dataform.cli.DataformCLI) loads the project from namespace files, installs the pinned @dataform/core dependency in beforeCommands, and runs dataform run --full-refresh with credentials from inputFiles and GOOGLE_APPLICATION_CREDENTIALS.
3. The full-refresh flag makes Dataform recompute every incremental model from its full source query, erasing accumulated drift, while non-incremental models rebuild as they would in any run.
4. notify posts the completed rebuild to Discord, and the errors block posts a distinct alert, early enough on Sunday that someone can intervene before Monday's consumers arrive.

What you get

• Incremental drift erased on a predictable weekly cadence instead of accumulating until someone notices wrong numbers.
• The most expensive run in the project confined to an agreed low-traffic window.
• A Monday-morning warehouse in a known-clean state, with the rebuild's logs and timing in the execution history.
• A visible weekly record of how long the full rebuild takes, which is the number you need before changing warehouse sizing or partitioning.

Who it's for

• Analytics engineers running incremental Dataform models who need a periodic correction for drift and backfills.
• Data platform teams who want expensive rebuilds governed by a schedule and a window, not by ad hoc terminal commands.
• Anyone who has run a full refresh at 11:00 on a Tuesday and spent the afternoon explaining slow dashboards.

Why orchestrate this with Kestra

A full refresh run by hand happens at random times, from random machines, with no record beyond a terminal scrollback. In Kestra, the window is explicit in the cron expression, the run is an execution with logs and duration, failures alert Discord while there is still weekend left to react, and pausing the rebuild during a freeze is flipping disabled rather than remembering not to type a command.

Prerequisites

• A Dataform project with incremental models, stored as namespace files.
• A BigQuery service account with permission to run jobs and write to the target datasets.
• A Discord incoming webhook for run reports.
• An agreed low-traffic window, adjust the cron expression to yours.

Secrets

• GCP_SERVICE_ACCOUNT_JSON: the service account key JSON, delivered to the task as sa.json.
• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.

Quick start

1. Add the GCP_SERVICE_ACCOUNT_JSON and DISCORD_WEBHOOK_URL secrets to your Kestra namespace.
2. Upload your Dataform project to the namespace files.
3. Execute the flow manually once and compare the rebuilt incremental tables against expectations.
4. Set disabled: false on the sunday_window trigger and adjust the cron to your window.

How to extend

• Scope the rebuild to one team's models by adding --tags to the command, as shown in the tag-scoped run blueprint.
• Follow the rebuild with an assertion pass so the clean state is verified, as shown in the assertions gate blueprint.
• Keep weekday freshness event-driven between the weekly rebuilds, as shown in the incremental refresh blueprint.
• Post the run duration to the notification payload from the execution context to track rebuild growth week over week.

Links

• Dataform plugin documentation
• Dataform documentation