Catalog Kafka Topics and Schemas in DataHub

Streaming data is the least discoverable data in most companies: topics are named by convention, schemas live in a registry nobody browses, and the only documentation is the producer's source code. This blueprint catalogs the bus. io.kestra.plugin.datahub.Ingestion runs a kafka source recipe that lists topics matching an allow pattern, resolves each topic's Avro or JSON schema from the schema registry, and publishes both to DataHub through the datahub-rest sink. It also demonstrates the task's env property: secrets become environment variables that the recipe references with ${VAR} expansion, keeping the recipe body free of any inline credential expression.

How it works

1. ingest_kafka_metadata (io.kestra.plugin.datahub.Ingestion) receives DATAHUB_GMS_URL and DATAHUB_TOKEN through its env property, resolved from Kestra secrets at runtime.
2. The recipe's sink references them as ${DATAHUB_GMS_URL} and ${DATAHUB_TOKEN}, which the DataHub CLI expands inside the container. This is the pattern to use when you want recipes portable between Kestra and manual datahub ingest runs.
3. The kafka source connects to the brokers from the bootstrap_servers input, lists topics matching the topic_allow_pattern regex, and pulls schemas from the registry URL held in a secret.
4. The datahub-rest sink publishes each topic as a dataset with its schema fields.
5. notify reports every sync to Slack, the errors block alerts on failure, and a disabled-by-default Schedule trigger reruns the sync every six hours.

What you get

• Every allowed topic in DataHub search, with fields taken from the registered schema rather than guesswork.
• The env plus ${VAR} expansion pattern, so recipe files stay credential-free and portable.
• An allow pattern input that keeps internal, test, and compacted changelog topics out of the catalog.
• A six-hour cadence that tracks schema evolution the same day it happens.

Who it's for

• Streaming platform teams asked what is on the bus and tired of answering from memory.
• Data engineers wiring consumers who need field-level schema detail before writing a deserializer.
• Governance owners extending the catalog from batch datasets to event streams.

Why orchestrate this with Kestra

Topic metadata changes with every deploy that adds an event type or evolves a schema, so cataloging it is a recurring job, not a one-off script. Kestra runs the unchanged DataHub recipe on a schedule, injects broker and registry credentials from secrets, retries transient broker or GMS failures, and alerts the channel that owns the bus when a sync fails. The execution history doubles as an audit log of when the catalog last matched the cluster.

Prerequisites

• A running DataHub instance with its GMS endpoint reachable from Kestra.
• Network access from the Kestra worker to the Kafka brokers and the schema registry.
• Docker available to the Kestra worker for the acryldata/datahub-ingestion container image.
• A Slack incoming webhook.

Secrets

• DATAHUB_GMS_URL: URL of the DataHub GMS server.
• DATAHUB_TOKEN: DataHub personal access token with permission to ingest metadata.
• SCHEMA_REGISTRY_URL: URL of the schema registry, including credentials if it requires them.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the four secrets to your Kestra namespace.
2. Set bootstrap_servers and topic_allow_pattern for your cluster and execute the flow.
3. Search DataHub for one of your topic names, confirm its schema fields, then set disabled: false on the every_six_hours trigger.

How to extend

• Add consumer_config under connection for SASL or TLS clusters, keeping passwords in secrets passed through env.
• Add a deny list under topic_patterns to exclude specific noisy topics from an otherwise broad allow pattern.
• Point the same recipe at Confluent Cloud by switching the bootstrap servers and adding the required consumer config.
• Pair it with the dbt artifacts blueprint so stream schemas and warehouse models share one lineage graph.

Links

• DataHub plugin documentation
• DataHub kafka source documentation