Re-Catalog a Postgres Schema Right After a Migration

The gap between a nightly catalog sync and a morning migration is where wrong decisions get made: a column was renamed at 09:00, the catalog says otherwise until 05:00 tomorrow, and every consumer in between reads stale documentation. This blueprint closes that gap. Your migration pipeline calls a Kestra webhook with the schema it just changed, and io.kestra.plugin.datahub.Ingestion runs a postgres recipe scoped by schema_pattern to exactly that schema. The sync is small, fast, and immediate, and Slack confirms the catalog caught up.

How it works

1. The migration pipeline, a CI job running Flyway, Liquibase, or plain SQL, finishes applying DDL and calls the webhook URL with a JSON body such as a schema field naming the changed schema.
2. The migration_webhook trigger starts the flow, and every reference to the schema uses {{ trigger.body.schema ?? inputs.schema_name }}, so manual executions fall back to the schema_name input.
3. ingest_changed_schema (io.kestra.plugin.datahub.Ingestion) runs a postgres source whose schema_pattern allow list is anchored to that single schema, re-extracting only the tables and views the migration could have touched.
4. The datahub-rest sink pushes the fresh metadata to the GMS server, with all connection details resolved from secrets.
5. notify names the re-cataloged schema in Slack so the migration author can verify their DDL in DataHub, and the errors block alerts when the sync fails, the one moment when catalog and database are known to disagree.

What you get

• A catalog that updates within seconds of DDL, not at the next nightly sync.
• Scoped ingestion through schema_pattern, so a migration to one schema never triggers a full-database scan.
• A webhook contract simple enough to add to any migration pipeline as a one-line curl call.
• The trigger.body with input fallback pattern, keeping the flow runnable by hand for backfills.

Who it's for

• Platform teams whose migration pipeline already notifies chat and should notify the catalog too.
• Data engineers who field questions about columns that exist in the database but not in DataHub.
• Governance owners who want catalog freshness tied to change events instead of wall-clock schedules.

Why orchestrate this with Kestra

Event-driven cataloging needs something listening for the event, holding the database and DataHub credentials, and owning the failure path. Kestra's webhook trigger gives the migration pipeline a stable URL with no credentials to manage on the caller's side beyond the webhook key, secrets stay in Kestra, and every migration-triggered sync is an execution you can audit later. Combine it with a scheduled full sync and you get fast reaction to changes plus a periodic safety net.

Prerequisites

• A running DataHub instance with its GMS endpoint reachable from Kestra.
• A migration pipeline able to send an HTTP POST after applying DDL.
• Docker available to the Kestra worker for the acryldata/datahub-ingestion container image.
• A Slack incoming webhook.

Secrets

• DATAHUB_GMS_URL: URL of the DataHub GMS server.
• DATAHUB_TOKEN: DataHub personal access token with permission to ingest metadata.
• POSTGRES_HOST_PORT: host and port of the migrated database.
• POSTGRES_DATABASE: database name.
• POSTGRES_USERNAME: Postgres user for metadata extraction.
• POSTGRES_PASSWORD: password for that user.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the seven secrets to your Kestra namespace.
2. Execute the flow manually with schema_name set to a real schema and confirm it appears fresh in DataHub.
3. Add a POST to the webhook URL, including the datahub_after_migration key, as the last step of your migration pipeline, sending the changed schema in the JSON body.

How to extend

• Send a table list in the webhook body and scope further with table_pattern for very large schemas.
• Chain a io.kestra.plugin.core.flow.Subflow call from your Kestra-managed migration flow instead of the webhook when migrations already run in Kestra.
• Change the webhook key to a random value before production and rotate it like any credential.
• Pair it with the scheduled Postgres ingestion blueprint, event-driven for speed, nightly for completeness.

Links

• DataHub plugin documentation
• DataHub postgres source documentation