Publish dbt Artifacts to DataHub After Every Run

dbt already knows your models, their lineage, and their documentation; it writes all of it into manifest.json and catalog.json on every run. This blueprint moves that knowledge into DataHub the moment it exists. io.kestra.plugin.datahub.Ingestion stages both artifacts into the ingestion container through inputFiles, a dbt source recipe parses them, and the datahub-rest sink publishes models, column schemas, lineage edges, and docs to your GMS server. Run it right after dbt build and the catalog never lags the transformation layer.

How it works

1. The flow takes manifest.json and catalog.json as FILE inputs. When you chain this flow after a dbt task, pass the artifacts from the dbt task's output files instead.
2. ingest_dbt_artifacts (io.kestra.plugin.datahub.Ingestion) writes both files into the container's working directory through inputFiles, so the recipe can reference them by plain relative paths.
3. The dbt source reads manifest_path and catalog_path, maps every model to a dataset on the target_platform, and derives lineage from the dbt graph.
4. The datahub-rest sink pushes everything to the GMS server, with the URL and token resolved from Kestra secrets.
5. notify tells Discord the catalog is current, and the errors block sends a distinct alert when parsing or publishing fails.

What you get

• Model definitions, column docs, and lineage in DataHub seconds after each dbt run.
• The inputFiles pattern for feeding run artifacts into the ingestion container without a shared filesystem.
• A target_platform input, so the same flow serves projects targeting Postgres, Snowflake, or BigQuery.
• Discord messages that distinguish a current catalog from a stale one.

Who it's for

• Analytics engineers who maintain dbt docs and want them discoverable in DataHub, not only in dbt's own site.
• Data platform teams stitching dbt lineage into a company-wide catalog next to Kafka, S3, and warehouse metadata.
• Governance owners who need lineage that updates with every deployment instead of a quarterly diagram.

Why orchestrate this with Kestra

The hard part of dbt-to-DataHub sync is not the recipe, it is the handoff: the artifacts exist inside whatever ran dbt, and the ingestion CLI runs somewhere else. Kestra makes the handoff explicit. Artifacts travel through internal storage as typed files, inputFiles places them where the recipe expects them, and the execution history records exactly which artifacts produced which catalog state. Chain this flow as a subflow after your dbt task and every transformation deploy updates the catalog automatically.

Prerequisites

• A running DataHub instance with its GMS endpoint reachable from Kestra.
• dbt artifacts from a completed run, manifest.json at minimum and catalog.json for column-level detail.
• Docker available to the Kestra worker for the acryldata/datahub-ingestion container image.
• A Discord incoming webhook.

Secrets

• DATAHUB_GMS_URL: URL of the DataHub GMS server.
• DATAHUB_TOKEN: DataHub personal access token with permission to ingest metadata.
• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.

Quick start

1. Add the three secrets to your Kestra namespace.
2. Run dbt build and dbt docs generate locally, then execute this flow and upload the two artifacts as inputs.
3. Open a dbt model in DataHub and confirm its lineage and docs, then wire the flow after your dbt task.

How to extend

• Chain it after io.kestra.plugin.dbt.cli.DbtCLI with a io.kestra.plugin.core.flow.Subflow task, passing the dbt task's output files as inputs.
• Add run_results.json handling through the dbt source's test results options to surface dbt test outcomes as DataHub assertions.
• Set target_platform per environment and run the flow once per dbt target.
• Pair it with the Postgres ingestion blueprint so warehouse datasets and dbt models meet in one lineage graph.

Links

• DataHub plugin documentation
• DataHub dbt source documentation