Catalog an S3 Data Lake in DataHub Weekly

Warehouses catalog themselves through their information schema; a data lake is just files, and files answer no questions. This blueprint points DataHub's s3 source at your lake through io.kestra.plugin.datahub.Ingestion. The recipe scans a path spec, groups files into datasets using the {table} placeholder, infers each dataset's schema from the Parquet files themselves, and publishes the result to your GMS server. A weekly schedule keeps the catalog tracking the lake as new datasets land, and Discord hears about every scan and every failure.

How it works

1. ingest_lake_metadata (io.kestra.plugin.datahub.Ingestion) runs the standard DataHub ingestion recipe in its container image.
2. The s3 source scans the path_specs entry built from the path_spec input. The {table} placeholder makes every matching folder one DataHub dataset, so curated/orders/*.parquet becomes a dataset named orders with a schema read from the files.
3. AWS credentials come from Kestra secrets through aws_config, and the region stays an input so the same flow covers buckets in different regions.
4. The datahub-rest sink publishes the datasets, with the GMS URL and token resolved from secrets.
5. notify posts the scan confirmation to Discord, the errors block alerts on failure, and a disabled-by-default Schedule trigger reruns the scan every Monday at 06:00.

What you get

• Lake datasets that show up in DataHub search with real schemas, not tribal knowledge about folder layouts.
• Schema inference straight from Parquet files, no Glue crawler or external metastore required.
• A path spec input, so scanning another zone of the lake is a parameter change, not a new flow.
• A weekly cadence with execution history proving when each scan ran.

Who it's for

• Data platform teams whose lake grew faster than anyone documented it.
• Analytics engineers who need to find lake datasets by column name instead of asking in a channel.
• Governance owners extending catalog coverage beyond the warehouse to raw and curated zones.

Why orchestrate this with Kestra

A lake scan is exactly the kind of job that dies quietly in cron: it takes minutes, it fails on expired credentials or a renamed prefix, and nobody notices until someone searches for a dataset that never got cataloged. Kestra gives the recipe a schedule, secrets that rotate without touching the flow, retries for transient S3 or GMS errors, and an alert path that names the failure. The recipe itself stays the plain DataHub YAML you would run with datahub ingest.

Prerequisites

• A running DataHub instance with its GMS endpoint reachable from Kestra.
• An AWS access key with read and list permissions on the lake bucket.
• Docker available to the Kestra worker for the acryldata/datahub-ingestion container image.
• A Discord incoming webhook.

Secrets

• DATAHUB_GMS_URL: URL of the DataHub GMS server.
• DATAHUB_TOKEN: DataHub personal access token with permission to ingest metadata.
• AWS_ACCESS_KEY_ID: AWS access key with read access to the bucket.
• AWS_SECRET_ACCESS_KEY: matching AWS secret key.
• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.

Quick start

1. Add the five secrets to your Kestra namespace.
2. Set path_spec to a real prefix in your lake and execute the flow.
3. Search DataHub for a folder name from that prefix, confirm its schema, then set disabled: false on the weekly trigger.

How to extend

• Add more path_specs entries to cover raw, staging, and curated zones in one scan.
• Enable profiling for row counts and null ratios on smaller datasets.
• Use {partition_key[0]}={partition[0]} placeholders in the path spec so partitioned tables catalog as single datasets.
• Run the same recipe against MinIO or another S3-compatible store by adding an endpoint override to aws_config.

Links

• DataHub plugin documentation
• DataHub s3 source documentation