# Kestra Complete Documentation
> Full content snapshot of all Kestra documentation pages. For the curated index, see /llms.txt.
> Append .md to any kestra.io/docs/* URL to retrieve that page as plain Markdown.
Total pages: 518
---
# Kestra Docs: Your Infinitely-Scalable Orchestration Platform
URL: https://kestra.io/docs
> The official documentation for Kestra, an open-source orchestration platform to automate business-critical workflows.
import HomePageButtons from "~/components/content/HomePageButtons.astro"
import WhatsNew from "~/components/common/WhatsNew.astro"
import SupportLinks from "~/components/content/SupportLinks.astro"
import PluginCount from "~/components/content/PluginCount.vue"
import ChildCard from "~/components/docs/ChildCard.astro"
What is Kestra?
Kestra is an open-source, infinitely-scalable **orchestration platform** that enables all engineers
to manage **business-critical workflows** declaratively in code.
Thanks to plugins and an embedded code editor with Git and Terraform integrations, Kestra makes scheduled and event-driven workflows easy.
---
# Administrator Guide: Operate and Secure Your Cluster
URL: https://kestra.io/docs/administrator-guide
> The complete Administrator Guide for operating, securing, and scaling your Kestra cluster in production.
import ChildCard from "~/components/docs/ChildCard.astro"
The Administrator Guide covers everything you need to know about managing your Kestra cluster.
- Check the **[Installation Guide](../02.installation/index.mdx)** for details on how to **install Kestra** to your preferred environment.
- Check the **[Configuration guide](../configuration/index.mdx)** for details on how to **configure Kestra** based your specific needs.
---
# Backup and Restore Kestra: Flows, Secrets, and Executions
URL: https://kestra.io/docs/administrator-guide/backup-and-restore
> Learn how to perform full or metadata-only backups and restores of your Kestra instance for disaster recovery and migration.
Back up and restore your Kestra instance.
Kestra provides a built-in metadata backup feature in Enterprise Edition. For full disaster recovery, you can also back up the underlying database and internal storage directly using backend tools.
Backup and restore are CLI-only operations — there is no REST API, no UI, and no built-in scheduler. Wire the CLI commands into your own cron, CI/CD, or disaster-recovery tooling.
:::alert{type="info"}
The commands in the next section assume Kestra runs locally on the host. If you run Kestra in Docker, see the [container example](#example-backup-and-restore-inside-docker) below.
:::
## Metadata backup and restore (Enterprise Edition)
[Kestra Enterprise Edition](../../oss-vs-paid/index.md) provides metadata backup and restore. You can back up metadata from one Kestra instance and restore it into another — even across different Kestra versions or repository/queue backends.
Perform metadata backup and restore while Kestra is paused to ensure consistency. As a best practice, enable [Maintenance Mode](../../07.enterprise/05.instance/maintenance-mode/index.md) before starting.
### What is included
A metadata backup exports all resource types except execution history. By default, it includes:
apps, banners, bindings, blueprints, credentials, dashboards, flows, groups, invitations, kill switches, KV entries, namespace files, namespaces, roles, secrets (when using a compatible secret backend), security integrations, settings, tenants, tenant accesses, test suites, triggers, users, and worker groups.
Execution data (executions, logs, metrics, audit logs, test runs) is excluded by default due to size. Use `--include-data` to include it.
Each backup run writes a new timestamped archive and never overwrites existing files. Archives are not pruned automatically — removing old archives is your responsibility.
### Metadata backup
To back up instance metadata, run:
```bash
kestra backups create FULL
```
`FULL` backs up the entire instance including all tenants and instance-level resources (users, settings, worker groups, kill switches). To back up a single tenant, use `TENANT` — only that tenant's data is included; instance-level resources are excluded.
To back up only specific resource types, use `--resources`:
```bash
kestra backups create FULL --resources FLOW,KV_STORE,SECRET
```
Available resource names include: `APP`, `BANNER`, `BINDING`, `BLUEPRINT`, `CREDENTIAL`, `DASHBOARD`, `EXECUTION`, `FLOW`, `GROUP`, `INVITATION`, `KILL_SWITCH`, `KV_STORE`, `LOG`, `METRIC`, `NAMESPACE`, `NAMESPACE_FILE`, `ROLE`, `SECRET`, `SECURITY_INTEGRATION`, `SETTING`, `TENANT`, `TENANT_ACCESS`, `TEST_SUITE`, `TRIGGER`, `USER`, `WORKER_GROUP`.
**Backup options:**
- `--tenant`: (for `TENANT` backups) the tenant to back up. Defaults to the `default` tenant.
- `--encryption-key`: a custom encryption key. If omitted, the instance encryption key (`kestra.encryption.secret-key`) is used. If neither is configured, the command fails.
- `--no-encryption`: disable encryption. Use only for non-sensitive content such as seeding a demo instance.
- `--include-data`: include execution data (executions, logs, metrics, audit logs, test runs). Excluded by default due to size.
When the backup completes, the CLI logs a summary and the internal storage URI of the archive:
```plaintext
2024-09-17 16:33:12,706 INFO create io.kestra.ee.backup.BackupService Backup summary: [BINDING: 3, BLUEPRINT: 1, FLOW: 13, GROUP: 1, NAMESPACE: 1, ROLE: 6, SECRET: 1, SECURITY_INTEGRATION: 0, SETTING: 1, TENANT: 1, TENANT_ACCESS: 2, TRIGGER: 2, USER: 1]
2024-09-17 16:33:12,706 INFO create io.kestra.ee.backup.BackupService Backup instance created in 508 ms
Backup created: kestra:///backups/full/backup-20240917163312.kestra
```
Full archives are stored under `kestra:///backups/full/`; tenant archives under `kestra:///backups/tenants//`.
### Metadata restore
To restore from a metadata backup, run the command using the URI returned by the backup:
```bash
kestra backups restore kestra:///backups/full/backup-20240917163312.kestra
```
Restore is idempotent at the record level — every record is upserted, so a partially completed restore can be safely re-run.
The restore command checks the archive's Kestra version against the running instance and logs a warning when they differ. This does not block the restore, but review the warning before proceeding in production.
**Restore options:**
- `--encryption-key`: decryption key. Falls back to the instance key if omitted; fails if neither is available.
- `--to-tenant`: restore a tenant backup into a different tenant. Rewrites every record's tenant ID to the target tenant. Not supported for full archives — the restore command fails if `--to-tenant` is used with a full archive.
- `--resources`: restore only specific resource types (same names as `create`).
The CLI logs a summary when complete:
```plaintext
2024-09-17 16:41:06,065 INFO restore io.kestra.ee.backup.BackupService Restoring kestra:///backups/full/backup-20240917163312.kestra
2024-09-17 16:41:06,149 INFO restore io.kestra.ee.backup.BackupService Restoring FULL backup from Kestra version 2.0.0 created at 2024-09-17T16:33:12.700099909
2024-09-17 16:41:06,150 INFO restore io.kestra.ee.backup.BackupService Backup summary: [BINDING: 3, BLUEPRINT: 1, FLOW: 13, GROUP: 1, NAMESPACE: 1, ROLE: 6, SECRET: 1, SECURITY_INTEGRATION: 0, SETTING: 1, TENANT: 1, TENANT_ACCESS: 2, TRIGGER: 2, USER: 1]
2024-09-17 16:41:07,182 INFO restore io.kestra.ee.backup.BackupService Restore summary: [BINDING: 3, BLUEPRINT: 1, FLOW: 13, GROUP: 1, NAMESPACE: 1, ROLE: 6, SECRET: 1, SECURITY_INTEGRATION: 0, SETTING: 1, TENANT: 1, TENANT_ACCESS: 2, USER: 1, TRIGGER: 2]
Backup restored from URI: kestra:///backups/full/backup-20240917163312.kestra
```
### Example: Backup and restore inside Docker
If Kestra runs in Docker, use `docker exec` and `docker cp` to move the archive in and out of the container:
```bash
## Create a full backup (with execution data) from inside the container
docker exec your_container bash -c "./kestra backups create FULL --include-data --no-encryption"
## Copy the backup file from the container to a local directory
docker cp your_container:/app/storage/backups/full/backup123.kestra .
## After upgrading Kestra, copy the backup back into the container
docker cp ./backup123.kestra your_container:/app/storage/backups/full/
## Restore the backup from inside the container
docker exec your_container bash -c "./kestra backups restore kestra:///backups/full/backup123.kestra"
```
## Full backup and restore with backend tools
### Backup and restore with the JDBC backend
Use the database's native backup tools to back up and restore Kestra when running the JDBC backend.
#### PostgreSQL
Stop Kestra first to ensure the database is in a stable state.
```bash
pg_dump -h localhost -p 5432 -U -d -F tar -f kestra.tar
```
To restore:
```bash
pg_restore -h localhost -p 5432 -U -d kestra.tar
```
Restart Kestra after the restore completes.
#### MySQL
Stop Kestra first.
```bash
mysqldump -h localhost -P 3306 -u -p'' > kestra.sql
```
To restore:
```bash
mysql -h localhost -P 3306 -u -p'' < kestra.sql
```
Restart Kestra after the restore completes.
#### SQL Server
Stop Kestra first, then create a backup using SQL Server Management Studio or `sqlcmd`:
```sql
BACKUP DATABASE [kestra] TO DISK = '/var/opt/mssql/backup/kestra.bak' WITH INIT;
```
To restore:
```sql
RESTORE DATABASE [kestra] FROM DISK = '/var/opt/mssql/backup/kestra.bak' WITH REPLACE;
```
Restart Kestra after the restore completes.
### Backup and restore with the Elasticsearch and Kafka backend
With the Elasticsearch or OpenSearch backend, back up and restore using Elasticsearch snapshots. Kafka is reinitialized from Elasticsearch after the restore.
This guide assumes you have configured a snapshot repository named `my_snapshot_repository`. See the [Elasticsearch snapshot documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html) for setup options.
Create a snapshot:
```bash
## Kibana Dev Tools (Console) or curl (adjust host/auth as needed)
PUT _snapshot/my_snapshot_repository/kestra?wait_for_completion=true
```
Delete all Kestra indices and restore from the snapshot:
```bash
POST _snapshot/my_snapshot_repository/kestra/_restore
{
"indices": "kestra_*"
}
```
If starting from a fresh Kafka cluster, reinitialize Kafka from Elasticsearch:
```bash
kestra sys-ee restore-queue
```
Some in-flight execution state is stored only in Kafka — pending executions may not be fully recovered.
Restart Kestra after the restore completes.
### Backup and restore of internal storage
Kestra's internal storage can be a local filesystem or object storage.
- **Local filesystem**: back up and restore the storage directory with your standard filesystem tools.
- **Managed object storage**: enable cross-region replication (often sufficient for DR) or use the provider's backup tooling.
- **Self-hosted object storage (e.g., MinIO)**: use a tool like [Restic](https://blog.min.io/back-up-restic-minio/) and/or configure replication.
---
# Fix Basic Authentication Issues in Kestra
URL: https://kestra.io/docs/administrator-guide/basic-auth-troubleshooting
> Troubleshoot common issues with Basic Authentication in Kestra, including configuration and login problems.
Troubleshoot issues with Basic Authentication.
Every open-source instance of Kestra requires Basic Authentication (`username` and `password`). You can configure credentials via the Setup page in the UI (http://localhost:8080/ui/main/setup) or manually in the configuration file under `basic-auth` (recommended for production):
```yaml
kestra:
server:
basic-auth:
username: admin@kestra.io
password: Admin1234
```
Since Basic Authentication is now required, the `enabled` flag is ignored and should no longer be used. Credentials must be configured to access the Kestra UI or API. For new users, follow the Setup page that appears when you start the Kestra UI.
For production deployments, set a valid email address and a strong password in the configuration file.
If you use the Setup page to create credentials, the password must satisfy the password policy configured under `kestra.security.basic-auth`. See [Security and Secrets configuration](../../configuration/05.security-and-secrets/index.md) for the available password policy settings.
There are four possible scenarios for existing users.
### Scenario 1: The `enabled` flag is set to `true`
```yaml
kestra:
server:
basic-auth:
enabled: true
username: admin@kestra.io
password: Admin1234
```
In this case, the following occurs:
- Now that authentication is required, it is always enabled. Therefore, the `enabled` flag is ignored regardless of `true` or `false`.
- The user Setup page **will not** appear when starting Kestra because `username` and `password` are set. You will be prompted to log in with those credentials.
- If either `username` or `password` is missing, Kestra starts with the Setup page and prompts you to create credentials. These values will be used for all future logins.
- If `username` or `password` is invalid, Kestra will show an error and prompt you to update the credentials to valid values.
### Scenario 2: The `enabled` flag is set to `false`
```yaml
kestra:
server:
basic-auth:
enabled: false
```
In this case, the following occurs:
- Now that authentication is required, it is always enabled. Therefore, the `enabled` flag is ignored regardless of `true` or `false`.
- On first startup, the Setup page appears and prompts you to create credentials. These are stored in the Kestra database in the **Settings** table under the key `kestra.server.basic-auth` and are used for all subsequent logins.
### Scenario 3: No `basic-auth` configuration is added
If no `basic-auth` configuration is defined:
- The Setup page will appear the first time starting Kestra, and you will need to create valid credentials. The authentication credentials are stored in your Kestra database in the **Settings** table under the key `kestra.server.basic-auth`. This is how you log in for all future sessions.
:::alert{type="warning"}
In this scenario, the Setup page is publicly reachable until credentials are submitted. For production deployments, configure `username` and `password` in the application configuration file **before** starting Kestra to skip the Setup page and ensure the instance is never unprotected.
:::
:::alert{type="info"}
If you forget your credentials, update the `username` and `password` in the configuration file. The configuration file always takes precedence over values set from the Setup page.
:::
### Scenario 4: Using Authorization headers instead of cookies
Kestra’s API accepts both `Authorization: Basic ...` headers and cookies for authentication.
However, the **UI only works with cookie-based authentication** and will ignore the `Authorization` header.
If you are running in an environment where headers are injected automatically (e.g. via proxy or authentication middleware), you have two options:
- Use a proxy or middleware to translate the `Authorization` header into a `BASIC_AUTH` cookie before forwarding to the Kestra UI.
- Use a browser extension (e.g. [ModHeader](https://modheader.com/)) to inject the `BASIC_AUTH` cookie directly.
This limitation does not affect API usage, which continues to accept both headers and cookies.
---
# Adding a Self-Signed Certificate to Kestra on Kubernetes
URL: https://kestra.io/docs/administrator-guide/custom-ca-kubernetes
> Add a self-signed or internal CA certificate to the JVM truststore in a Kubernetes-based Kestra deployment.
Add a self-signed or internal CA certificate to the JVM truststore so Kestra tasks can make outbound calls to services secured by that CA.
This is common in environments where internal services — such as databases, APIs, or artifact registries — use a private or self-signed certificate authority not included in the JVM's default truststore. Without this configuration, tasks that connect to those services fail with:
```plaintext
javax.net.ssl.SSLHandshakeException: PKIX path building failed: unable to find valid certification path to requested target
```
## Prerequisites
You need `kubectl`, `keytool` (bundled with the JDK), and `helm`.
The commands use the following variables:
| Variable | Description |
| --- | --- |
| `$NAMESPACE` | Kubernetes namespace where Kestra is running |
| `$WEBSERVER_POD_NAME` | Name of a running Kestra pod (e.g., the webserver or executor) |
| `$ALIAS_FOR_YOUR_DOMAIN` | Unique alias for your certificate (e.g., `my-internal-ca`) |
| `$RELEASE_NAME` | Helm release name for your Kestra installation |
## Steps
1. **Prepare a working directory**:
```bash
mkdir ssl
cp /path/to/self-signed-certificate.pem ./ssl/
```
2. **Retrieve the base keystore from the running pod**: Pull the default Java truststore (`cacerts`) from a running pod to preserve all standard public CAs while you add your own.
```bash
kubectl get po -n $NAMESPACE
WEBSERVER_POD_NAME=
kubectl cp $NAMESPACE/$WEBSERVER_POD_NAME:/opt/java/openjdk/lib/security/cacerts ./ssl/cacerts
ls -l ./ssl
```
3. **Import and convert the certificate**: Import your PEM file into the downloaded truststore:
```bash
keytool -importcert \
-trustcacerts \
-file ssl/self-signed-certificate.pem \
-keystore ssl/cacerts \
-alias $ALIAS_FOR_YOUR_DOMAIN \
-storepass changeit \
-noprompt
```
Convert the modified keystore to PKCS12 format:
```bash
keytool -importkeystore \
-srckeystore ssl/cacerts \
-destkeystore ssl/truststore.p12 \
-deststoretype PKCS12 \
-srcstorepass changeit \
-deststorepass changeit
```
Verify your alias is present in the new truststore:
```bash
keytool -list -keystore ssl/truststore.p12 -storetype PKCS12 -storepass changeit | grep $ALIAS_FOR_YOUR_DOMAIN
```
If the command returns a line containing your alias, the import was successful.
4. **Create a Kubernetes Secret**:
```bash
kubectl create secret generic kestra-ssl \
--from-file=truststore.p12=ssl/truststore.p12 \
-n $NAMESPACE
```
The `--from-file=key=path` syntax sets the key name inside the Secret to `truststore.p12`, so the mounted filename matches the JVM path configured in step 5.
5. **Configure Helm**: Add the following to your `values.yaml` to mount the secret and point the JVM to it:
```yaml
common:
extraVolumeMounts:
- name: ssl-secret
mountPath: "/app/ssl"
readOnly: true
extraVolumes:
- name: ssl-secret
secret:
secretName: kestra-ssl
extraEnv:
- name: JAVA_OPTS
value: >-
-Djavax.net.ssl.trustStore=/app/ssl/truststore.p12
-Djavax.net.ssl.trustStorePassword=changeit
-Djavax.net.ssl.trustStoreType=PKCS12
```
Then apply the changes:
```bash
helm upgrade --install $RELEASE_NAME kestra/kestra -n $NAMESPACE -f /path/to/values.yaml
```
:::alert{type="info"}
The default Java keystore password is `changeit`. If your organization uses a different password, update it consistently across all `keytool` and `JAVA_OPTS` references above.
:::
---
# Docker-in-Docker Behind a Proxy: Kestra on Kubernetes
URL: https://kestra.io/docs/administrator-guide/dind-behind-proxy
> Configure Docker-in-Docker (DinD) to run securely behind a corporate or MITM proxy within your Kestra deployment.
Configure Docker-in-Docker (DinD) to work behind a corporate or MITM proxy in a rootless Kubernetes deployment.
## Why configure CA certs and proxies for DinD?
Docker-in-Docker (DinD) runs a Docker daemon inside a container, allowing it to build and run other containers. Kestra relies on DinD for certain task types that require Docker runtime isolation.
If your environment uses a proxy that intercepts HTTPS traffic (such as an MITM proxy), Docker must **trust the proxy’s CA certificate** when pulling images from remote registries (like Docker Hub or private registries).
Without this, you'll see errors like:
```plaintext
x509: certificate signed by unknown authority
```
## Prerequisites
1. Create a ConfigMap for the Docker daemon configuration.
This should include your `daemon.json` with proxy settings.
Create a file `daemon.json`:
```json
{
"proxies": {
"http-proxy": "http://mitmproxy.default.svc.cluster.local:8000",
"https-proxy": "http://mitmproxy.default.svc.cluster.local:8000",
"no-proxy": "localhost,127.0.0.1,.svc,.cluster.local,your.nexus.domain.com,kestra-minio"
}
}
```
Apply the configmap:
```bash
kubectl create configmap dind-daemon-config \
--from-file=daemon.json=./daemon.json \
-n kestra
```
2. Create a ConfigMap for the MITM Proxy CA certificate.
Assuming you have the CA file saved as `mitmproxy-ca.crt`, run:
```bash
kubectl create configmap dind-ca-certs \
--from-file=ca.crt=./mitmproxy-ca.crt \
-n kestra
```
3. Kestra Configuration
Here is a configuration sample you can include in your Helm `values.yaml`:
```yaml
configurations:
application:
kestra:
plugins:
configurations:
- type: io.kestra.plugin.scripts.runner.docker.Docker
values:
volume-enabled: true
common:
extraVolumes:
- name: docker-daemon-config
configMap:
name: dind-daemon-config
- name: ca-cert-volume
configMap:
name: dind-ca-certs
extraVolumeMounts:
- name: docker-daemon-config
mountPath: /home/rootless/.config/docker
readOnly: true
- name: ca-cert-volume
mountPath: /home/rootless/.config/docker/certs.d/mitmproxy.default.svc.cluster.local:8000
readOnly: true
- name: ca-cert-volume
mountPath: /home/rootless/mitmproxy
readOnly: true
dind:
enabled: true
base:
rootless:
image:
repository: docker
tag: dind-rootless
pullPolicy: IfNotPresent
securityContext:
runAsUser: 1000
runAsGroup: 1000
args:
- --log-level=fatal
- --group=1000
socketPath: /dind/
tmpPath: /tmp/
resources: {}
extraEnv:
- name: SSL_CERT_FILE
value: /home/rootless/mitmproxy/ca.crt
```
Here, `volume-enabled: true` ensures that the CA certificate is mounted from the DinD pod into any container deployed by a Kestra task.
## DinD in action
This configuration will help the DinD pod pull the required container images successfully through the MITM proxy.
For Kestra tasks that run in Docker containers (e.g., `io.kestra.plugin.scripts.shell.Script`), you also need to set the `HTTPS_PROXY` environment variable and trust the certificate using `beforeCommands` as shown below.
For consistency across tasks, consider configuring these settings as plugin defaults.
```yaml
id: mitm_proxy
namespace: company.team
tasks:
- id: shell
type: io.kestra.plugin.scripts.shell.Script
containerImage: alpine/curl
beforeCommands:
- apk add --no-cache ca-certificates
- update-ca-certificates
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
volumes:
- /home/rootless/mitmproxy/ca.crt:/usr/local/share/ca-certificates/mitmproxy.crt
env:
HTTPS_PROXY: "mitmproxy.default.svc.cluster.local:8000"
script: |
curl https://httpbin.org/get
```
## How it works
- `daemon.json`: tells Docker which proxy settings to use.
- `certs.d`: directory where Docker looks for custom CA certificates to trust registries.
- `SSL_CERT_FILE`: overrides the TLS stack used by the Docker daemon to trust the MITM CA.
- `HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY`: standard proxy env vars for networking.
---
# High Availability in Kestra: Scale Workers and Webservers
URL: https://kestra.io/docs/administrator-guide/high-availability
> Design and configure Kestra for High Availability (HA) to ensure fault tolerance and continuous operation in production.
Kestra is designed for high availability and fault tolerance. This page explains how to configure your deployment to ensure continuous operation.
Highly available systems are built to keep running even in the event of component or infrastructure failures. This is achieved by eliminating single points of failure and introducing redundancy across critical services.
In Kestra, high availability is achieved by running multiple instances of each core component — including the `webserver` (API), `scheduler`, `executor`, `indexer`, and `workers`. This ensures that if one instance fails, the system can continue to operate without interruption.
:::alert{type="info"}
This architecture requires a [Kafka and Elasticsearch deployment](../../08.architecture/index.mdx), which is designed to be highly available and fault-tolerant.
:::
## Scaling the components
The following components can be scaled horizontally by increasing the number of replicas in your [Helm chart values](https://github.com/kestra-io/kestra/blob/develop/charts/kestra/values.yaml):
- Webserver
- Scheduler
- Executor
- Worker
- Indexer
Additionally, the Elasticsearch and Kafka clusters can be scaled out as needed to handle large volumes of data.
Finally, internal storage (such as S3) is highly available and fault-tolerant by design.
:::alert{type="info"}
Ensure that the underlying host system is also tuned for high availability and fault tolerance. For example, adjusting the Linux kernel parameter `net.ipv4.tcp_retries2` can reduce [TCP retransmission times](https://access.redhat.com/solutions/726753).
:::
## Load balancing
To guarantee high availability, deploy a load balancer to distribute incoming requests across multiple `webserver` instances. This prevents downtime if any single instance fails, allowing the system to continue operating without interruption.
---
# JVM CPU Limits for Kestra on Kubernetes
URL: https://kestra.io/docs/administrator-guide/jvm-cpu-limits
> Configure the Kestra Helm chart to force the JVM to honor Kubernetes CPU limits, preventing pods from over-consuming resources.
Force the JVM to match Kubernetes CPU limits through the Kestra Helm chart.
Kestra pods on some Kubernetes clusters can exceed their CPU and memory limits because the Java Virtual Machine (JVM) does not always read cgroup data correctly. This guide explains how to align the JVM with [Kubernetes](../../02.installation/03.kubernetes/index.md) constraints using the Kestra Helm chart and the `-XX:ActiveProcessorCount` flag. For broader deployment guidance, see [High Availability](../high-availability/index.md) and [Monitoring](../03.monitoring/index.md).
## Why JVM sizing can ignore Kubernetes limits
Kubernetes enforces container limits with cgroups, but the JVM may still detect the host capacity instead of the constrained container. When that happens:
- Netty and other pools size themselves for the full node.
- The pod uses more memory and threads than its limits allow and can be killed by the OOM killer.
This behavior depends on the cluster runtime and cgroup configuration, so a consistent fix must live in the Helm chart rather than in the application code.
## Use `-XX:ActiveProcessorCount` to align CPU detection
`-XX:ActiveProcessorCount` lets you tell the JVM how many CPUs to see. Using it inside the Kestra container makes internal pools scale to the CPU count that matches your Kubernetes limits:
```bash
java -XX:ActiveProcessorCount=2 -jar kestra.jar
```
Because many clusters already expose cgroup data correctly, the Helm chart keeps this flag optional and configurable.
## Configure the Kestra Helm chart
The chart adds a dedicated JVM section in `values.yaml`:
```yaml
common:
jvm:
forceActiveProcessors:
enabled: false
count: "auto" # "auto" or "value"
value: 2 # only used when count = "value"
extraOpts: ""
```
- `enabled`: toggle the feature (disabled by default).
- `count`:
- `"auto"` derives the CPU count from `resources.limits.cpu`.
- `"value"` uses a fixed number.
- `value`: CPU count when `count` is set to `"value"`.
- `extraOpts`: additional JVM flags; the chart prepends `-XX:ActiveProcessorCount` when enabled.
### Derive CPU count automatically
Auto mode reads `resources.limits.cpu`, supports values such as `"250m"`, `"1"`, or `"1.5"`, converts them to an integer CPU count (minimum 1), and injects:
```plaintext
-XX:ActiveProcessorCount=
```
Example:
```yaml
common:
resources:
limits:
cpu: "250m"
jvm:
forceActiveProcessors:
enabled: true
count: "auto"
```
This yields `KESTRA_JAVA_OPTS="-XX:ActiveProcessorCount=1"` for the pod.
### Provide an explicit CPU value
If you prefer a fixed number, switch to `"value"`:
```yaml
common:
jvm:
forceActiveProcessors:
enabled: true
count: "value"
value: 3
```
This sets `KESTRA_JAVA_OPTS="-XX:ActiveProcessorCount=3"`, which can be useful to keep the JVM more conservative than the container limit.
### Override per component
Different components can use different CPU counts. Component overrides take precedence over the global setting:
```yaml
common:
jvm:
forceActiveProcessors:
enabled: true
count: "value"
value: 2
deployments:
standalone:
enabled: true
jvm:
forceActiveProcessors:
enabled: true
count: "value"
value: 5
```
## How the chart applies the setting
- The Helm helper computes the CPU count from the global `common.jvm.forceActiveProcessors`, any component override, and the component `resources.limits.cpu` (falling back to `common.resources.limits.cpu`).
- It builds `KESTRA_JAVA_OPTS`, adding `-XX:ActiveProcessorCount=` when enabled and appending `extraOpts`.
- The container exports `KESTRA_JAVA_OPTS`, and the Kestra start script runs `exec java ${KESTRA_JAVA_OPTS} ${JAVA_OPTS} ...`.
## When to enable it
Enable `forceActiveProcessors` when pods hit OOMs or thread pools scale as if the full node is available. Start with auto mode so the JVM mirrors your Kubernetes CPU limits:
```yaml
common:
resources:
limits:
cpu: "2"
jvm:
forceActiveProcessors:
enabled: true
count: "auto"
```
If your pods already respect limits, keep the feature disabled. Combine this setting with your existing Helm configuration in [High Availability](../high-availability/index.md) to scale components safely and monitor the impact using [Prometheus metrics](../prometheus-metrics/index.md).
---
# External Log Data Store in Kestra
URL: https://kestra.io/docs/administrator-guide/log-data-store
> Route Kestra execution logs to a dedicated JDBC database or Elasticsearch, separate from your main backend, to reduce database size and speed up schema migrations.
The external log data store routes Kestra execution logs to a dedicated JDBC database or Elasticsearch, separate from the main backend.
By default, Kestra stores execution logs in the same database as flows, executions, and state.
## Why move logs out of the main database
Logs are high-volume, write-heavy data. Keeping them in the main database creates three operational problems:
- **Database bloat**: Log volume grows independently of flows and executions, inflating main database size over time.
- **Migration risk**: Kestra schema migrations run across every table, including logs. Large log tables extend upgrade downtime.
- **Retention mismatch**: Flows, executions, and logs typically have different retention requirements but share the same database and backup lifecycle.
## How it works
The Indexer writes logs to the configured log store in real time. The log UI and dashboards read from the same store. The main database stores only flows, executions, and state.
```mermaid
graph LR
I[Kestra Indexer] -->|writes state| M[Main DB]
I -->|writes logs| L[Log data store]
```
### Default behavior
If `kestra.logs.type` is not set, logs stay in the main backend (`kestra.repository.type`). Existing installations see no change on upgrade.
The full resolution order is:
1. `kestra.logs.type` — explicit log store selection
2. `kestra.repository.type` — fallback: logs stay in the main backend (the default)
3. If neither is configured, Kestra fails fast at startup with a clear error
### Migration note
Configuring an external log store applies to **new executions only**. Historical logs remain in the main database.
## Configure the JDBC log store
Select a JDBC store by setting `kestra.logs.type` to `h2`, `postgres`, or `mysql`. The store can connect to a dedicated log database or reuse the main datasource.
:::alert{type="info"}
There is no `datasources.logs` block. The dedicated log database is configured entirely under `kestra.logs..*`, not alongside the main `datasources.*` config.
:::
### Config reference
| Key | Description |
|-----|-------------|
| `kestra.logs.type` | Store type: `h2`, `postgres`, `mysql`, or `elasticsearch` |
| `kestra.logs..url` | JDBC URL of the dedicated log database. When set, Kestra opens its own HikariCP connection pool and runs log-table migrations against this database. |
| `kestra.logs..username` | Username for the dedicated log database |
| `kestra.logs..password` | Password for the dedicated log database |
| `kestra.logs..table` | Log table name (default: `logs`) |
When `kestra.logs..url` is **not** set, the JDBC store reuses the primary datasource and `kestra.logs.type` must match `kestra.repository.type`. This is useful for verifying config shape without standing up a second database, but it does not move logs out of the main database.
### Example: PostgreSQL backend with a separate PostgreSQL log database
Two independent Postgres databases: one for the main backend, one for logs only.
```yaml
services:
postgres: # main backend DB
image: postgres:16
environment:
POSTGRES_DB: kestra
POSTGRES_USER: kestra
POSTGRES_PASSWORD: k3str4
volumes:
- postgres-data:/var/lib/postgresql/data
postgres-logs: # dedicated log DB
image: postgres:16
environment:
POSTGRES_DB: kestra_logs
POSTGRES_USER: kestra
POSTGRES_PASSWORD: k3str4
volumes:
- postgres-logs-data:/var/lib/postgresql/data
kestra:
image: kestra/kestra:latest
command: server standalone
depends_on: [postgres, postgres-logs]
ports: ["8080:8080"]
environment:
KESTRA_CONFIGURATION: |
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driverClassName: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
repository:
type: postgres
queue:
type: postgres
storage:
type: local
logs:
type: postgres
postgres:
url: jdbc:postgresql://postgres-logs:5432/kestra_logs
username: kestra
password: k3str4
table: logs
volumes:
postgres-data: {}
postgres-logs-data: {}
```
The main backend keeps using `datasources.postgres`. The log store opens its own connection pool against `postgres-logs`, runs log-table migrations there, and routes all log reads and writes to that database.
## Configure the Elasticsearch log store
The Elasticsearch log store reuses the existing `kestra.elasticsearch.*` connection config; you add a `kestra.logs` block naming the index.
:::alert{type="warning"}
Kestra fails at startup if no Elasticsearch client is configured or if the required index declaration (with `cls: io.kestra.core.models.executions.LogEntry`) is missing.
:::
### Config reference
| Key | Description |
|-----|-------------|
| `kestra.logs.type` | `elasticsearch` |
| `kestra.logs.elasticsearch.index` | Elasticsearch index or alias name (default: `logs`) |
| `kestra.elasticsearch.client.*` | ES connection config (hosts, auth, TLS) — same block used by the main ES backend |
| `kestra.elasticsearch.indices..cls` | Must be `io.kestra.core.models.executions.LogEntry` |
| `kestra.elasticsearch.indices..index` | Must match `kestra.logs.elasticsearch.index` |
### Example: PostgreSQL backend with Elasticsearch log store
The main backend stays Postgres; execution logs go to Elasticsearch.
```yaml
services:
postgres:
image: postgres:16
environment:
POSTGRES_DB: kestra
POSTGRES_USER: kestra
POSTGRES_PASSWORD: k3str4
volumes:
- postgres-data:/var/lib/postgresql/data
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
environment:
discovery.type: single-node
xpack.security.enabled: "false"
ES_JAVA_OPTS: "-Xms512m -Xmx512m"
ports: ["9200:9200"]
kestra:
image: kestra/kestra:latest-ee # EE image required
command: server standalone
depends_on: [postgres, elasticsearch]
ports: ["8080:8080"]
environment:
KESTRA_CONFIGURATION: |
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driverClassName: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
repository:
type: postgres
queue:
type: postgres
storage:
type: local
logs:
type: elasticsearch
elasticsearch:
index: logs
elasticsearch:
client:
http-hosts:
- http://elasticsearch:9200
# basic-auth:
# username: elastic
# password: changeme
indices:
logs:
cls: io.kestra.core.models.executions.LogEntry
index: logs
volumes:
postgres-data: {}
```
## Store capabilities
JDBC and Elasticsearch are complete implementations: they support offset pagination with exact totals, dashboard aggregation, and on-demand purge.
Each store declares its capabilities, and Kestra adapts gracefully:
| Capability | What it controls | JDBC | Elasticsearch |
|---|---|---|---|
| Aggregation | Whether log count charts and KPI tiles on dashboards query this store | ✓ | ✓ |
| Pagination type | `OFFSET`: numbered pages with exact totals. `CURSOR`: forward-only, no total. | OFFSET | OFFSET |
| Purge | Whether Kestra can delete logs on demand via [`PurgeLogs`](../purge/index.md) or the UI | ✓ | ✓ |
When a store does not support aggregation, dashboard log charts display "No data" rather than erroring. When a store does not support purge, Kestra's `PurgeLogs` operations no-op for logs — the backend's own retention or TTL policy governs deletion.
:::alert{type="warning"}
If you configure a store that cannot purge, set up a retention policy in that backend before switching. Kestra will not delete logs on your behalf.
:::
## Log Shipper vs External Log Data Store
| | External Log Data Store | Log Shipper |
|---|---|---|
| **What it is** | The primary store Kestra writes and reads logs from | A Kestra task that copies logs to observability platforms |
| **How it runs** | Infrastructure-level config, always active | A scheduled flow |
| **Use case** | Keep logs out of the main database; reduce migration risk | Send logs to Datadog, Splunk, Elastic, CloudWatch for alerting and search |
| **Requires a sidecar?** | No | No |
Use the External Log Data Store when you want logs **out of the main database**. Use the [Log Shipper](../../07.enterprise/02.governance/logshipper/index.md) when you want logs **in a third-party observability platform** as well, regardless of where Kestra stores them internally.
## Build a new log store plugin
A new log store is a Kestra plugin that implements `LogDataStoreInterface`. Two reference implementations are available:
- [log-data-store-splunk](https://github.com/kestra-io/log-data-store-splunk) — offset pagination, full aggregation support, `canPurge() == true`
- [log-data-store-datadog](https://github.com/kestra-io/log-data-store-datadog) — cursor pagination, `canPurge() == false` (logs expire via index retention)
### 1. Implement the interface
Implement `io.kestra.core.repositories.LogDataStoreInterface`. For a new JDBC dialect, extend `AbstractJdbcLogDataStore` — it provides pagination, aggregation, and purge. Provide a no-arg constructor (the factory deserializes config onto the instance). If you need runtime Micronaut beans, implement `ApplicationContextInitializable` and wire dependencies in `init(ApplicationContext)`.
### 2. Declare the plugin id
```java
@Plugin
@Plugin.Id("mystore") // the value operators set in kestra.logs.type
public class MyLogDataStore implements LogDataStoreInterface, ApplicationContextInitializable {
@PluginProperty private String someOption; // bound from kestra.logs.mystore.someOption
...
}
```
### 3. Declare capabilities
The defaults are `canPurge() == true`, `canAggregate() == true`, and `paginationType() == OFFSET`. Override only what your store cannot support:
```java
@Override public boolean canPurge() { return false; } // backend handles TTL
@Override public boolean canAggregate() { return false; } // no server-side aggregation
@Override public PaginationType paginationType() { return PaginationType.CURSOR; }
```
A store declaring `canPurge() == false` must no-op all delete/purge methods: return `0` for integer-return methods and provide an empty body for void ones. A store declaring `CURSOR` must return a `CursoredPage` with a `nextPageable()` cursor token from paginated `find(...)` methods.
When the backend has no further results, emit an end-of-results sentinel as the next cursor token. If a caller sends that sentinel back, return an empty page without hitting the backend. The exact sentinel value is an implementation detail; your store just needs to recognize it on inbound requests and short-circuit.
### 4. Pass the contract test
Every log store implementation must pass `io.kestra.core.repositories.AbstractLogDataStoreTest` from the `:tests` module. The base class exercises every interface method and asserts every capability branch. Extend it with an empty class:
```java
class MyLogDataStoreTest extends AbstractLogDataStoreTest {}
```
Wire your store as the injected `LogDataStoreInterface` bean via test config. The suite adapts to whatever your store declares, but your store must genuinely honor its declarations.
:::alert{type="info"}
Most external backends have an indexing delay — writes are not immediately queryable. Override `awaitIndexing(BooleanSupplier ready)` in your test class to poll until data is visible before assertions run. Polling until a probe write is visible is not always sufficient: different fixture groups can have slightly different indexing lag, so the first visible write does not guarantee all writes are queryable. Add a short settle period after the probe goes green to absorb that skew.
:::
Two testing strategies are viable: run the contract suite against a real backend via Testcontainers (the Splunk approach, using the same Docker Compose stack as local QA), or inject a mock client through a package-private test constructor and gate live backend testing in a separate integration test class (the Datadog approach). Testcontainers gives higher confidence in query assembly and wire behavior; mock injection keeps CI fast when a live account is not available.
---
# MITM Proxy: Inspect Kestra's Outbound HTTPS Traffic
URL: https://kestra.io/docs/administrator-guide/mitm-proxy-configuration
> Configure Kestra to route outbound HTTPS traffic through a Man-in-the-Middle (MITM) proxy for secure environments.
Route and inspect Kestra's outbound HTTP/S traffic through an MITM proxy.
## Why use an MITM proxy
In secured or restricted environments it’s common to route outbound HTTP/S traffic through a **Man-in-the-Middle (MITM) proxy** for auditing, inspection, or policy enforcement. For this to work, clients (Kestra) must:
- Trust the proxy’s CA certificate.
- Route outbound traffic through the proxy.
- Configure the JVM and any auxiliary daemons (e.g., Docker daemon) to use the proxy and truststore.
:::alert{type="info"}
**Security note:** An MITM proxy intercepts TLS traffic. Only enable this in controlled environments and with appropriate approvals.
:::
## Prerequisites
### 1. Create a Java truststore with the MITM CA certificate
Import the MITM CA certificate into a Java keystore so the JVM trusts intercepted TLS connections:
```bash
keytool -importcert -alias mitmproxy-ca -storepass changeit -keystore truststore.jks -trustcacerts -file mitmproxy-ca.crt -noprompt
```
:::alert{type="info"}
Tip: prefer a strong password instead of `changeit` in production. You can also use PKCS12 by setting `-deststoretype PKCS12`.
:::
### 2. (Kubernetes) Create a Secret containing the truststore
Create a Kubernetes secret from the `truststore.jks`:
```bash
kubectl create secret generic kestra-ssl --from-file=truststore.jks -n kestra
```
This secret will be mounted into Kestra pods.
## Configuring Kestra to use the MITM proxy
You must update the [Observability and Networking configuration](../../configuration/03.observability-and-networking/index.md) and ensure the truststore is available inside the container. Below are suggested changes for both Kubernetes (Helm) and Docker Compose deployments.
### 1. Micronaut / Kestra configuration
Add proxy settings and truststore configuration to your [Observability and Networking configuration](../../configuration/03.observability-and-networking/index.md) (merged via Helm `configurations.application` or a config file):
```yaml
## values.yaml
configurations:
application:
micronaut:
http:
client:
proxy-address: "your.proxy.net:8000"
proxy-type: HTTP
server:
ssl:
clientAuthentication: want
trustStore:
path: "file:/app/ssl/truststore.jks"
password: "changeit"
type: "JKS"
```
### 2. Mount the truststore inside the container
**Kubernetes (Helm `values.yaml`)**
```yaml
common:
extraVolumeMounts:
- name: ssl-secret
mountPath: "/app/ssl"
readOnly: true
extraVolumes:
- name: ssl-secret
secret:
secretName: kestra-ssl
```
**Docker Compose**
```yaml
services:
kestra:
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- tmp-kestra:/tmp/kestra-wd
- ./ssl:/app/ssl # ensure ./ssl/truststore.jks exists on host
```
### 3. JVM environment variables (JAVA_OPTS)
**Kubernetes (values.yaml)**
```yaml
common:
extraEnv:
- name: JAVA_OPTS
value: >-
-Djavax.net.ssl.trustStore=/app/ssl/truststore.jks
-Djavax.net.ssl.trustStorePassword=changeit
-Djavax.net.ssl.trustStoreType=JKS
-Dhttp.proxyHost=your.proxy.net
-Dhttp.proxyPort=8000
-Dhttps.proxyHost=your.proxy.net
-Dhttps.proxyPort=8000
-Dhttp.nonProxyHosts=localhost|127.0.0.1|kubernetes.default.svc|.svc|.cluster.local|your.nexus.domain.com|kestra-minio
```
**Docker Compose**
```yaml
services:
kestra:
environment:
- JAVA_OPTS=-Djavax.net.ssl.trustStore=/app/ssl/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStoreType=JKS -Dhttp.proxyHost=your.proxy.net -Dhttp.proxyPort=8000 -Dhttps.proxyHost=your.proxy.net -Dhttps.proxyPort=8000 -Dhttp.nonProxyHosts=localhost|127.0.0.1|your.nexus.domain.com
```
## Troubleshooting
1. **TLS handshake errors**
Verify `truststore.jks` contains the correct CA (`keytool -list -keystore truststore.jks`).
2. **Requests not reaching the proxy**
Confirm `http.proxyHost` / `https.proxyHost` and `http.nonProxyHosts` are correct.
3. **Docker image pull failures**
Add the MITM CA to Docker daemon certs (`/etc/docker/certs.d/.../ca.crt`).
4. **Debugging TLS**
Temporarily enable: `-Djavax.net.debug=ssl,handshake`.
---
# Kestra Monitoring: Prometheus, Alerts, and Health Checks
URL: https://kestra.io/docs/administrator-guide/monitoring
> Monitor and alert on Kestra health. Best practices for setting up Prometheus metrics, health checks, and failure notifications for your instance.
Set up failure alerting and Prometheus-based monitoring for your Kestra instance.
Failure alerts are essential. When a production workflow fails, you should be notified immediately. To implement failure alerting, you can use Kestra’s built-in notification tasks, such as:
- [Slack](/plugins/plugin-slack)
- [Microsoft Teams](/plugins/plugin-microsoft365)
- [Email](/plugins/plugin-email)
Technically, you can add custom failure alerts to each flow separately using the `errors` tasks:
```yaml
id: onFailureAlert
namespace: company.team
tasks:
- id: fail
type: io.kestra.plugin.core.execution.Fail
errors:
- id: slack
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
url: "{{ secret('SLACK_WEBHOOK') }}"
messageText: "Failure alert for flow `{{ flow.namespace }}.{{ flow.id }}` with ID `{{ execution.id }}`. Here is a bit more context about why the execution failed: `{{ errorLogs() }}`"
```
However, this can lead to boilerplate code when this `errors` configuration is duplicated across multiple flows.
For centralized namespace-level alerting, create a dedicated monitoring workflow with a notification task and a Flow trigger. Below is an example workflow that automatically sends a Slack alert as soon as any flow in a namespace `company.analytics` fails or finishes with warnings.
```yaml
id: failureAlertToSlack
namespace: company.monitoring
tasks:
- id: send
type: io.kestra.plugin.slack.notifications.SlackExecution
url: "{{ secret('SLACK_WEBHOOK') }}"
channel: "#general"
executionId: "{{trigger.executionId}}"
triggers:
- id: listen
type: io.kestra.plugin.core.trigger.Flow
dependsOn:
- states: [FAILED, WARNING]
when: "{{ namespace | startsWith('company.analytics') }}"
```
Adding this single flow will ensure that you receive a Slack alert on any flow failure in the `company.analytics` namespace. Here is an example alert notification:

:::alert{type="info"}
To alert on failures across multiple namespaces or specific flows, use `mode: ANY` with multiple `dependsOn` entries. The trigger fires when any entry is satisfied:
```yaml
id: alert
namespace: company.system
tasks:
- id: send
type: io.kestra.plugin.slack.notifications.SlackExecution
url: "{{ secret('SLACK_WEBHOOK') }}"
channel: "#general"
executionId: "{{trigger.executionId}}"
triggers:
- id: listen
type: io.kestra.plugin.core.trigger.Flow
mode: ANY
dependsOn:
- states: [FAILED, WARNING]
when: "{{ namespace | startsWith('company.product') }}"
- flowId: cleanup
namespace: company.system
states: [FAILED, WARNING]
```
:::
The example above fires when either any `company.product` flow fails or the specific `cleanup` flow in `company.system` fails. `mode: ANY` means the trigger fires as soon as one entry is satisfied — you do not need to combine everything into a single expression.
## Monitoring
Kestra exposes a monitoring endpoint on port 8081 by default. You can change this port using the `endpoints.all.port` property in the [Observability and Networking configuration](../../configuration/03.observability-and-networking/index.md).
This monitoring endpoint provides invaluable information for troubleshooting and monitoring, including Prometheus metrics and several Kestra's internal routes. For instance, the `/health` endpoint exposed by default on port 8081 (e.g., http://localhost:8081/health) generates a similar response as shown below as long as your Kestra instance is healthy:
```json
{
"name": "kestra",
"status": "UP",
"details": {
"jdbc": {
"name": "kestra",
"status": "UP",
"details": {
"jdbc:postgresql://postgres:5432/kestra": {
"name": "kestra",
"status": "UP",
"details": {
"database": "PostgreSQL",
"version": "15.3 (Debian 15.3-1.pgdg110+1)"
}
}
}
},
"compositeDiscoveryClient()": {
"name": "kestra",
"status": "UP",
"details": {
"services": {
}
}
},
"service": {
"name": "kestra",
"status": "UP"
},
"diskSpace": {
"name": "kestra",
"status": "UP",
"details": {
"total": 204403494912,
"free": 13187035136,
"threshold": 10485760
}
}
}
}
```
## Prometheus
Kestra exposes [Prometheus](https://prometheus.io/) metrics on the endpoint `/prometheus`. This endpoint is compatible with Prometheus and can be scraped by any Prometheus-based monitoring system.
For more details about Prometheus setup, refer to the [Monitoring with Grafana & Prometheus](../../15.how-to-guides/monitoring/index.md) article.
:::alert{type="info"}
For a complete list of available metrics, refer to the [Prometheus metrics page](../prometheus-metrics/index.md).
:::
### Kestra's metrics
Use Kestra's internal metrics to configure custom alerts. Each metric exposes time-series data tagged by at least `namespace` and `flow_id`, with additional tags depending on the task type.
Kestra metrics use the prefix `kestra`. This prefix can be changed using the `kestra.metrics.prefix` property in the [Observability and Networking configuration](../../configuration/03.observability-and-networking/index.md).
Each task type can expose custom metrics that will be also exposed on Prometheus.
#### Worker
|Metrics|Type|Description|
|-|-|-|
|worker.running.count|`GAUGE`|Number of tasks currently running|
|worker.started.count|`COUNTER`|Count of tasks started|
|worker.retried.count|`COUNTER`|Count of tasks retried|
|worker.ended.count|`COUNTER`|Count of tasks completed|
|worker.ended.duration|`TIMER`|Duration of tasks completed|
|worker.job.running|`GAUGE`|Count of currently running worker jobs|
|worker.job.pending|`GAUGE`|Count of currently pending worker jobs|
|worker.job.thread|`GAUGE`|Total worker job thread count|
:::alert{type="info"}
The `worker.job.pending`, `worker.job.running`, and `worker.job.thread` metrics are intended for autoscaling [worker servers](../../08.architecture/02.server-components/index.md#worker).
:::
#### Executor
|Metrics|Type|Description|
|-|-|-|
|executor.taskrun.next.count|`COUNTER`|Count of tasks found|
|executor.execution.end.count|`COUNTER`|Count of completed executions|
|executor.taskrun.ended.duration|`TIMER`|Duration of tasks completed|
|executor.workertaskresult.count|`COUNTER`|Count of task results sent by a worker|
|executor.execution.started.count|`COUNTER`|Count of executions started|
|executor.execution.end.count|`COUNTER`|Count of executions completed|
|executor.execution.duration|`TIMER`|Duration of executions completed|
|executor.flowable.execution.count|`COUNTER`|Count of flowable tasks executed|
|executor.execution.popped.count|`COUNTER`|Count of executions popped|
|executor.execution.queued.count|`COUNTER`|Count of executions queued|
|executor.thread.count|`COUNTER`|Count of executor threads|
#### Indexer
|Metrics|Type| Description |
|-|-|-------------------------------------------|
|indexer.count|`COUNTER`| Count of index requests sent to a repository |
|indexer.duration|`DURATION`| Duration of index requests sent to a repository |
#### Scheduler
|Metrics|Type| Description |
|-|-|-----------------------------------------------------------------------------------------------------|
|scheduler.trigger.count|`COUNTER`| Count of triggers |
|scheduler.evaluate.running.count|`COUNTER`| Evaluation of triggers actually running |
|scheduler.evaluate.duration|`TIMER`| Duration of trigger evaluation |
#### JDBC Queue
|Metrics|Type|Description|
|-|-|-|
|queue.big_message.count|`COUNTER`|Count of big messages|
|queue.produce.count|`COUNTER`|Count of produced messages|
|queue.receive.duration|`TIMER`|Duration to receive and consume a batch of messages|
|queue.poll.size|`GAUGE`|Size of a poll to the queue (message batch size)|
### Other metrics
Kestra also exposes all internal metrics from the following sources:
- [Micronaut](https://micronaut-projects.github.io/micronaut-micrometer/latest/guide/)
- [Kafka](https://kafka.apache.org/documentation/#remote_jmx)
- Thread pools of the application
- JVM
See the [Micronaut documentation](https://micronaut-projects.github.io/micronaut-micrometer/latest/guide/) for more information.
## Grafana and Kibana
You can create a Grafana dashboard backed by Prometheus metrics to monitor the health of your Kestra instance. If you use the Elasticsearch backend (Enterprise Edition), Kibana is another option.
Share your dashboard with [the community](/slack). Below is an example Grafana dashboard you can use as a starting point:

:::collapse{title="Grafana Dashboard JSON"}
```json
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 1862,
"links": [],
"panels": [
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 3,
"panels": [],
"repeat": "namespace",
"title": "INSTANCE: $namespace",
"type": "row"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "light-blue"
}
]
},
"unit": "core"
},
"overrides": []
},
"gridPos": {
"h": 3,
"w": 4,
"x": 0,
"y": 1
},
"id": 6,
"options": {
"colorMode": "background_solid",
"graphMode": "area",
"justifyMode": "center",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"exemplar": false,
"expr": "sum(kube_pod_container_resource_requests{namespace=\"$namespace\", resource=\"cpu\"})",
"instant": true,
"legendFormat": "{{container}}",
"range": false,
"refId": "A"
}
],
"title": "cpu requests",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "light-blue"
}
]
},
"unit": "decbytes"
},
"overrides": []
},
"gridPos": {
"h": 3,
"w": 4,
"x": 4,
"y": 1
},
"id": 5,
"options": {
"colorMode": "background_solid",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"exemplar": false,
"expr": "sum(kube_pod_container_resource_requests{namespace=\"$namespace\", resource=\"memory\"})",
"instant": true,
"legendFormat": "{{container}}",
"range": false,
"refId": "A"
}
],
"title": "memory requests",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"decimals": 2,
"mappings": [],
"max": 1,
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "light-green"
},
{
"color": "light-orange",
"value": 0.25
},
{
"color": "light-red",
"value": 0.75
}
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": {
"h": 6,
"w": 8,
"x": 8,
"y": 1
},
"id": 4,
"options": {
"displayMode": "lcd",
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"maxVizHeight": 20,
"minVizHeight": 20,
"minVizWidth": 8,
"namePlacement": "auto",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showUnfilled": true,
"sizing": "manual",
"text": {
"titleSize": 12,
"valueSize": 16
},
"valueMode": "color"
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"exemplar": false,
"expr": "avg(rate(container_cpu_usage_seconds_total{namespace=\"$namespace\", container!=\"\"}[2m])) by (container) / avg(kube_pod_container_resource_requests{namespace=\"$namespace\", resource=\"cpu\"}) by (container)",
"instant": true,
"legendFormat": "{{container}}",
"range": false,
"refId": "A"
}
],
"title": "cpu consumptions / usages vs requests",
"type": "bargauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"decimals": 2,
"mappings": [],
"max": 1,
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "light-green"
},
{
"color": "light-orange",
"value": 0.25
},
{
"color": "light-red",
"value": 0.75
}
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": {
"h": 6,
"w": 8,
"x": 16,
"y": 1
},
"id": 12,
"options": {
"displayMode": "lcd",
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"maxVizHeight": 20,
"minVizHeight": 20,
"minVizWidth": 8,
"namePlacement": "auto",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showUnfilled": true,
"sizing": "manual",
"text": {
"titleSize": 12,
"valueSize": 16
},
"valueMode": "color"
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"exemplar": false,
"expr": "avg(rate(container_cpu_usage_seconds_total{namespace=\"$namespace\", container!=\"\"}[2m])) by (container) / avg(kube_pod_container_resource_limits{namespace=\"$namespace\", resource=\"cpu\"}) by (container)",
"instant": true,
"legendFormat": "{{container}}",
"range": false,
"refId": "A"
}
],
"title": "cpu overloads / usages vs limits",
"type": "bargauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "light-blue"
}
]
},
"unit": "currencyEUR"
},
"overrides": []
},
"gridPos": {
"h": 3,
"w": 4,
"x": 0,
"y": 4
},
"id": 16,
"options": {
"colorMode": "background_solid",
"graphMode": "area",
"justifyMode": "center",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"exemplar": false,
"expr": "monthly_namespace_cost{exported_namespace=\"$namespace\"}",
"instant": true,
"legendFormat": "{{container}}",
"range": false,
"refId": "A"
}
],
"title": "monthly cost",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "light-blue"
}
]
},
"unit": "currencyEUR"
},
"overrides": []
},
"gridPos": {
"h": 3,
"w": 4,
"x": 4,
"y": 4
},
"id": 17,
"options": {
"colorMode": "background_solid",
"graphMode": "area",
"justifyMode": "center",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"exemplar": false,
"expr": "monthly_namespace_slack{exported_namespace=\"$namespace\"}",
"instant": true,
"legendFormat": "{{container}}",
"range": false,
"refId": "A"
}
],
"title": "monthly slack",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"axisPlacement": "auto",
"fillOpacity": 50,
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineWidth": 0,
"spanNulls": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 1
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 6,
"w": 8,
"x": 0,
"y": 7
},
"id": 8,
"options": {
"alignValue": "left",
"legend": {
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"mergeValues": true,
"rowHeight": 0.8,
"showValue": "never",
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "count(kube_pod_container_status_last_terminated_reason{reason=\"OOMKilled\", namespace=\"$namespace\"}) or vector(0)",
"legendFormat": "OOM Killed",
"range": true,
"refId": "OOMKilled"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "count(ALERTS{alertstate=\"pending\", namespace=\"$namespace\"}) or vector(0)",
"hide": false,
"instant": false,
"legendFormat": "Alerts Pending",
"range": true,
"refId": "Alerts Pending"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "count(ALERTS{alertstate=\"firing\", namespace=\"$namespace\"}) or vector(0)",
"hide": false,
"instant": false,
"legendFormat": "Alerts Firing",
"range": true,
"refId": "Alerts Firing"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "count(avg(container_memory_usage_bytes{namespace=\"$namespace\", container!=\"\"}) by (container) / avg(kube_pod_container_resource_limits{namespace=\"$namespace\", resource=\"memory\"}) by (container) > 0.8) or vector(0)",
"hide": false,
"instant": false,
"legendFormat": "Memory Limits 80%",
"range": true,
"refId": "Memory Limits"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "count(ALERTS{alertname=\"ProbesStatusError\", alertstate=\"firing\", job=\"blackbox-$namespace\"} == 1) or vector(0)",
"hide": false,
"instant": false,
"legendFormat": "Blackbox",
"range": true,
"refId": "Blackbox"
}
],
"title": "sanity checks",
"type": "state-timeline"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"decimals": 2,
"mappings": [],
"max": 1,
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "light-orange",
"value": 0.25
},
{
"color": "light-red",
"value": 0.75
}
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": {
"h": 6,
"w": 8,
"x": 8,
"y": 7
},
"id": 7,
"options": {
"displayMode": "lcd",
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"maxVizHeight": 20,
"minVizHeight": 20,
"minVizWidth": 8,
"namePlacement": "auto",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showUnfilled": true,
"sizing": "manual",
"text": {
"titleSize": 12,
"valueSize": 16
},
"valueMode": "color"
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"exemplar": false,
"expr": "avg(container_memory_usage_bytes{namespace=\"$namespace\", container!=\"\"}) by (container) / avg(kube_pod_container_resource_requests{namespace=\"$namespace\", resource=\"memory\"}) by (container)",
"instant": true,
"legendFormat": "{{container}}",
"range": false,
"refId": "A"
}
],
"title": "memory consumptions / usages vs requests",
"type": "bargauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"decimals": 2,
"mappings": [],
"max": 1,
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "light-orange",
"value": 0.25
},
{
"color": "light-red",
"value": 0.75
}
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": {
"h": 6,
"w": 8,
"x": 16,
"y": 7
},
"id": 10,
"options": {
"displayMode": "lcd",
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"maxVizHeight": 20,
"minVizHeight": 20,
"minVizWidth": 8,
"namePlacement": "auto",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showUnfilled": true,
"sizing": "manual",
"text": {
"titleSize": 12,
"valueSize": 16
},
"valueMode": "color"
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"exemplar": false,
"expr": "avg(container_memory_usage_bytes{namespace=\"$namespace\", container!=\"\"}) by (container) / avg(kube_pod_container_resource_limits{namespace=\"$namespace\", resource=\"memory\"}) by (container)",
"instant": true,
"legendFormat": "{{container}}",
"range": false,
"refId": "A"
}
],
"title": "memory overloads / usages vs limits",
"type": "bargauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "smooth",
"lineWidth": 2,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
}
]
},
"unit": "core"
},
"overrides": []
},
"gridPos": {
"h": 6,
"w": 8,
"x": 0,
"y": 13
},
"id": 2,
"options": {
"legend": {
"calcs": [],
"displayMode": "table",
"placement": "right",
"showLegend": true,
"width": 200
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "avg(rate(container_cpu_usage_seconds_total{namespace=\"$namespace\", container!=\"\", pod!~\"^app-bootstrap.*\"}[2m])) by (container)",
"legendFormat": "{{container}}",
"range": true,
"refId": "A"
}
],
"title": "cpu usages",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "smooth",
"lineWidth": 2,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "decbytes"
},
"overrides": []
},
"gridPos": {
"h": 6,
"w": 8,
"x": 8,
"y": 13
},
"id": 1,
"options": {
"legend": {
"calcs": [],
"displayMode": "table",
"placement": "right",
"showLegend": true,
"width": 200
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "avg(container_memory_usage_bytes{namespace=\"$namespace\", container!=\"\", container!=\"certresolver\", pod!~\"^app-bootstrap.*\"}) by (namespace, container)",
"legendFormat": "{{container}}",
"range": true,
"refId": "A"
}
],
"title": "memory usages",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "smooth",
"lineWidth": 2,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
}
]
},
"unit": "binBps"
},
"overrides": []
},
"gridPos": {
"h": 6,
"w": 8,
"x": 16,
"y": 13
},
"id": 13,
"options": {
"legend": {
"calcs": [],
"displayMode": "table",
"placement": "right",
"showLegend": true,
"width": 200
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "avg by (namespace) (\n rate(container_network_receive_bytes_total{namespace=\"$namespace\"}[2m])\n * on (namespace,pod) group_left ()\n topk by (namespace,pod) (\n 1,\n max by (namespace,pod) (kube_pod_info{host_network=\"false\"})\n )\n)",
"legendFormat": "receive_bytes",
"range": true,
"refId": "receive_bytes"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "avg by (namespace) (\n rate(container_network_transmit_bytes_total{namespace=\"$namespace\"}[2m])\n * on (namespace,pod) group_left ()\n topk by (namespace,pod) (\n 1,\n max by (namespace,pod) (kube_pod_info{host_network=\"false\"})\n )\n)",
"hide": false,
"instant": false,
"legendFormat": "transmit_bytes",
"range": true,
"refId": "transmit_bytes"
}
],
"title": "network",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineStyle": {
"fill": "solid"
},
"lineWidth": 2,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "light-green"
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 19
},
"id": 14,
"options": {
"legend": {
"calcs": [],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "avg(increase(kestra_worker_ended_duration_seconds_count{namespace=\"$namespace\"}[5m])) by (flow_id) > 0",
"legendFormat": "{{ flow_id }}",
"range": true,
"refId": "indexer_message_in_count"
}
],
"title": "flows activities",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "smooth",
"lineWidth": 2,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "light-green"
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 19
},
"id": 20,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "sum(kestra_worker_job_running{namespace=\"$namespace\"})",
"legendFormat": "{{ flow_id }}",
"range": true,
"refId": "indexer_message_in_count"
}
],
"title": "worker jobs running",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
}
]
},
"unit": "decbytes"
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 4,
"x": 0,
"y": 27
},
"id": 19,
"options": {
"colorMode": "background_solid",
"graphMode": "area",
"justifyMode": "center",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "sum(jvm_memory_used_bytes{namespace=\"$namespace\"})",
"hide": false,
"instant": false,
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "jvm memory",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
}
]
},
"unit": "decbytes"
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 20,
"x": 4,
"y": 27
},
"id": 18,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "12.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "avg(jvm_memory_used_bytes{namespace=\"$namespace\"}) by (id)",
"hide": false,
"instant": false,
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "jvm memory used",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 32
},
"id": 21,
"options": {
"legend": {
"calcs": [],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.0.1",
"targets": [
{
"editorMode": "code",
"expr": "avg(rate(http_server_requests_seconds_count{namespace=\"$namespace\", uri!~\"(UNMATCHED_URI|REDIRECTION|/health|/prometheus|/health/{selector})\"}[2m])) by (uri)",
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "http_server_requests_seconds_count",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 32
},
"id": 22,
"options": {
"legend": {
"calcs": [],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.0.1",
"targets": [
{
"editorMode": "code",
"expr": "avg(rate(http_server_requests_seconds_count{namespace=\"$namespace\", uri=~\"(REDIRECTION|UNMATCHED_URI)\"}[2m])) by (uri)",
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "http_server_requests_seconds_count",
"type": "timeseries"
}
],
"preload": false,
"refresh": "10s",
"schemaVersion": 41,
"tags": [],
"templating": {
"list": [
{
"current": {
"text": "Prometheus Sample",
"value": "Prometheus Sample"
},
"label": "Datasource",
"name": "datasource",
"options": [
{
"selected": false,
"text": "Prometheus Sample 1",
"value": "Prometheus Sample 1"
},
{
"selected": false,
"text": "Prometheus Sample 2",
"value": "Prometheus Sample 2"
},
{
"selected": true,
"text": "Prometheus Sample",
"value": "Prometheus Sample"
}
],
"query": "Prometheus Sample 1,Prometheus Sample 2,Prometheus Sample",
"type": "custom"
},
{
"current": {
"text": "kestra",
"value": "kestra"
},
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"definition": "label_values(kestra_jdbc_query_duration_seconds_count,kestra_instance)",
"includeAll": false,
"label": "KestraServer",
"name": "KestraServer",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(kestra_jdbc_query_duration_seconds_count,kestra_instance)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"type": "query"
},
{
"current": {
"text": "All",
"value": [
"$__all"
]
},
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"definition": "label_values(kestra_jdbc_query_duration_seconds_count{kestra_instance=\"$KestraServer\"},kestra_cloud_instance_name)",
"includeAll": true,
"label": "Instance",
"multi": true,
"name": "instance",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(kestra_jdbc_query_duration_seconds_count{kestra_instance=\"$KestraServer\"},kestra_cloud_instance_name)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"type": "query"
},
{
"current": {
"text": "All",
"value": "$__all"
},
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"definition": "label_values(kestra_jdbc_query_duration_seconds_count{kestra_instance=~\"$KestraServer\", kestra_cloud_instance_name=~\"$instance\"},namespace)",
"hide": 2,
"includeAll": true,
"label": "Namespace",
"multi": true,
"name": "namespace",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(kestra_jdbc_query_duration_seconds_count{kestra_instance=~\"$KestraServer\", kestra_cloud_instance_name=~\"$instance\"},namespace)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"type": "query"
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {},
"timezone": "UTC",
"title": "Sample Kestra Dashboard",
"uid": "sample_dashboard_uid",
"version": 1
}
```
:::
## Kestra endpoints
Kestra exposes internal endpoints on the management port (8081 by default) to provide status corresponding to the [server type](../../08.architecture/02.server-components/index.md):
* `/worker`: will expose all currently running tasks on this worker.
* `/scheduler`: will expose all currently scheduled flows on this scheduler with the next date.
* `/kafkastreams`: will expose all [Kafka Streams](https://kafka.apache.org/documentation/streams/) states and aggregated store lag.
* `/kafkastreams/{clientId}/lag`: will expose details lag for a `clientId`.
* `/kafkastreams/{clientId}/metrics`: will expose details metrics for a `clientId`.
## Other Micronaut default endpoints
Since Kestra is based on [Micronaut](https://micronaut.io), the [default Micronaut endpoints](https://docs.micronaut.io/latest/guide/index.html#providedEndpoints) are enabled by default on port 8081:
* `/info` [Info Endpoint](https://docs.micronaut.io/snapshot/guide/index.html#infoEndpoint) with git status information.
* `/health` [Health Endpoint](https://docs.micronaut.io/snapshot/guide/index.html#healthEndpoint) usable as an external heathcheck for the application.
* `/loggers` [Loggers Endpoint](https://docs.micronaut.io/snapshot/guide/index.html#loggersEndpoint) allows changing logger level at runtime.
* `/metrics` [Metrics Endpoint](https://docs.micronaut.io/snapshot/guide/index.html#metricsEndpoint) metrics in JSON format.
* `/env` [Environment Endpoint](https://docs.micronaut.io/snapshot/guide/index.html#environmentEndpoint) to debug configuration files.
You can disable some endpoints following the above Micronaut configuration.
:::alert{type="warning"}
The management port is unauthenticated by default. Do not expose port `8081` outside your internal network. See [Management endpoint access](../security-hardening/index.md#management-endpoint-access) for hardening guidance.
:::
## Debugging techniques
Here are several debugging techniques administrators can use to investigate issues:
## Enable verbose log
Kestra has some [management endpoints](../03.monitoring/index.md#other-micronaut-default-endpoints) including one that allows changing logging verbosity at run time.
Inside the container (or locally if standalone jar is used), send this command to enable very verbose logging:
```shell
curl -i -X POST -H "Content-Type: application/json" \
-d '{ "configuredLevel": "TRACE" }' \
http://localhost:8081/loggers/io.kestra
```
Alternatively, you can change logging levels on configuration files:
```yaml
logger:
levels:
io.kestra.core.runners: TRACE
```
## Capture some java dump
Kestra runs on a JRE rather than a JDK, so JVM monitoring tools are not included. Install [Jattach](https://github.com/jattach/jattach#installation) first:
:::alert{type="info"}
Jattach is included in the Kestra image, so there is no need to install it separately. If you're running an older version, continue to follow the steps below.
:::
```shell
curl -L -o jattach https://github.com/jattach/jattach/releases/download/v2.2/jattach
chmod +x jattach
```
- You need to find the pid of the Kestra process, it's usually `1` on docker installation.
- You can get JVM information with `jattach jcmd VM.info > vminfo`
- You can get a heap history via `jattach inspectheap > inspectheap`
- You can get a heap dump via `jattach dumpheap > dumpheap`
- You can get a thread dump via `jattach threaddump > threaddump`
Alternatively, you can request a thread dump via the `/threaddump` endpoint available on the management port (8081 if not configured otherwise).
---
# OpenTelemetry for Kestra: Traces, Metrics, and Logs
URL: https://kestra.io/docs/administrator-guide/open-telemetry
> Implement observability in Kestra with OpenTelemetry to export traces, metrics, and logs to your preferred monitoring tools.
**Observability** refers to understanding a system's internal state by analyzing its outputs. In software, this means examining telemetry data — such as traces, metrics, and logs — to gain insights into system behavior.
**OpenTelemetry** is a vendor-neutral, tool-agnostic framework and toolkit for creating and managing telemetry data.
It helps implement observability in software applications.
OpenTelemetry defines three different kinds of telemetry data:
- **Traces** provide a high-level view of what happens when a request is made to an application. A trace can contain multiple [spans](https://opentelemetry.io/docs/concepts/signals/traces/#spans).
- **Metrics** are measurements of a service captured at runtime.
- **Logs** are timestamped text records, either structured (recommended) or unstructured, with optional metadata.
Kestra supports all three kinds of telemetry data via OpenTelemetry-compatible exporters. For more details, see the [OpenTelemetry official documentation](https://opentelemetry.io/docs/).
## Traces
The first step is to enable distributed traces inside the [Observability and Networking configuration](../../configuration/03.observability-and-networking/index.md) file:
```yaml
micronaut:
otel:
enabled: true
kestra:
traces:
root: DEFAULT # Enable traces inside Kestra flow executions
otel:
traces:
exporter: otlp # Only otlp is supported for now
exporter:
otlp:
endpoint: http://localhost:4317 # Replace with the address of your own collector
```
When enabled, Kestra instruments:
- All calls to its API
- All flow executions (one span per task execution, plus one span for each execution message processed by the Executor)
- External HTTP calls made by the HTTP tasks (including tasks that use the Kestra HTTP client)
### Trace correlation
Kestra propagates the trace context so that traces are correlated:
- The API call trace correlates with the execution it creates.
- Flow execution traces correlate with parent flows when the `Subflow` or `Loop` task is used.
- External HTTP calls include the standard propagation header for downstream correlation.
### Propagate trace context to scripts
Scripts run in isolated containers, so OTel spans they generate start a new root trace by default. Pass `{{ trace.parent }}` as the `TRACEPARENT` environment variable to parent those spans under the Kestra task span.
`{{ trace.parent }}` holds the W3C [traceparent](https://www.w3.org/TR/trace-context/) header; it is empty when tracing is disabled.
```yaml
id: traced_script
namespace: company.team
tasks:
- id: run_python
type: io.kestra.plugin.scripts.python.Script
env:
TRACEPARENT: "{{ trace.parent }}"
script: |
from opentelemetry.propagate import extract
from opentelemetry.sdk.trace import TracerProvider
import os
ctx = extract({"traceparent": os.environ.get("TRACEPARENT", "")})
tracer = TracerProvider().get_tracer(__name__)
with tracer.start_as_current_span("my-span", context=ctx):
pass # spans here appear as children of the Kestra task span
```
`TRACEPARENT` is recognized by all major OTel SDKs and works the same way for Node.js, Bash, and any other script type. The variable is also usable in HTTP task headers and any other [expression-capable property](../../expressions/index.md#default-execution-context-variables).
### Example: Jaeger with Docker Compose
Enable [Jaeger](https://www.jaegertracing.io), an OpenTelemetry-compatible tracing platform, with Kestra in a Docker Compose configuration file:
```yaml
services:
# Postgres is included here as a dependency for Kestra during local testing
postgres:
image: postgres:14.13
environment:
POSTGRES_DB: kestra_unit
POSTGRES_USER: kestra
POSTGRES_PASSWORD: k3str4
ports:
- 5432:5432
restart: on-failure
jaeger-all-in-one:
image: jaegertracing/all-in-one:latest
ports:
- "16686:16686" # Jaeger UI
- "14268:14268" # OpenTracing (optional)
- "4317:4317" # OTLP gRPC receiver
- "4318:4318" # OTLP HTTP receiver
- "14250:14250" # External otel-collector (optional)
environment:
- COLLECTOR_OTLP_ENABLED=true
restart: on-failure
```
The following screenshot shows three correlated traces:
- One created from the API call that creates the execution
- One created from an execution of a flow named `opentelemetry_parent` which has spans for tasks including a `Subflow`
- One created from the `opentelemetry_basic` flow execution

### Disabling traces
You can disable traces for flows while keeping API traces:
```yaml
kestra:
traces:
root: DISABLED
```
You can also disable traces per component (experimental). For example, disabling only Executor spans:
```yaml
kestra:
traces:
root: DEFAULT
categories:
io.kestra.core.runners.Executor: DISABLED
```
#### Supported categories
| Category | Description |
|----------------------------------------|---------------------------------------------------|
| `io.kestra.core.runners.Executor` | Spans for each message in the execution queue |
| `io.kestra.core.runners.Worker` | Spans for each runnable task execution |
| `io.kestra.plugin.core.flow.Subflow` | Spans for each `Subflow` task execution |
| `io.kestra.plugin.core.flow.Loop` | Spans for each `Loop` sub-execution |
## Metrics
To send metrics to an OpenTelemetry-compatible collector, add the following parameters to your [Observability and Networking configuration](../../configuration/03.observability-and-networking/index.md) file:
```yaml
micronaut:
metrics:
export:
otlp:
enabled: true
url: http://localhost:4318/v1/metrics # Replace with your collector URL
```
For example, you can configure an OpenTelemetry Collector to forward metrics to Prometheus:
```yaml
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
exporters:
prometheus:
endpoint: "0.0.0.0:9464"
```
## Logs
To send logs to an OpenTelemetry-compatible collector, use the [LogShipper](../../07.enterprise/02.governance/logshipper/index.md) with the built-in OpenTelemetry log exporter.
:::alert{type="warning"}
LogShipper is only available in the Kestra **Enterprise Edition**.
:::
The following flow sends logs from all flows to a collector daily:
```yaml
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logExporters:
- id: OTLPLogExporter
type: io.kestra.plugin.ee.opentelemetry.LogExporter
otlpEndpoint: http://localhost:4318/v1/logs # Replace with your collector URL
```
---
# Prometheus Metrics for Kestra: Reference and /prometheus Endpoint
URL: https://kestra.io/docs/administrator-guide/prometheus-metrics
> Explore the available Prometheus metrics in Kestra to monitor the performance and health of your orchestration cluster.
This page provides an overview of all available [Prometheus](https://prometheus.io/) metrics in Kestra.
Prometheus metrics are enabled by default in Kestra, in contrast to [OpenTelemetry](../open-telemetry/index.md), which must be explicitly enabled in the configuration file.
Metrics include custom metrics defined within the application and framework-provided metrics collected via [Micrometer](https://micrometer.io/).
Each Prometheus metric is described with its purpose and the type of data it represents. You can access these metrics via the `http://localhost:8081/prometheus` endpoint in Kestra.
Example output from the Prometheus endpoint:
```plaintext
## HELP executor_active_threads The approximate number of threads that are actively executing tasks
## TYPE executor_active_threads gauge
executor_active_threads 4
```
:::alert{type="info"}
For deeper details on Micrometer metrics integration, see the [Micronaut Micrometer documentation](https://micronaut-projects.github.io/micronaut-micrometer/latest/guide/).
:::
## Kestra
### Kestra Executor Metrics
Executor server exclusive:
* `kestra_executor_execution_delay_created_count_total` (counter): The total number of execution delays created by the Executor.
* `kestra_executor_execution_duration_seconds` (summary): Execution duration inside the Executor.
* `kestra_executor_execution_duration_seconds_max` (gauge): Maximum observed execution duration inside the Executor.
* `kestra_executor_execution_end_count_total` (counter): The total number of executions ended by the Executor.
* `kestra_executor_execution_message_process_seconds` (summary): Duration of a single execution message processed by the Executor.
* `kestra_executor_execution_message_process_seconds_max` (gauge): Maximum observed duration of a single execution message processed by the Executor.
* `kestra_executor_execution_started_count_total` (counter): The total number of executions started by the Executor.
* `kestra_executor_flowable_execution_count_total` (counter): The total number of flowable tasks executed by the Executor
* `kestra_executor_loop_delay_duration_seconds` (summary): Execution delay loop duration inside the Executor.
* `kestra_executor_loop_delay_duration_seconds_max` (gauge): Maximum observed execution delay loop duration inside the Executor.
* `kestra_executor_loop_sla_duration_seconds` (summary): SLA monitor loop duration inside the Executor.
* `kestra_executor_loop_sla_duration_seconds_max` (gauge): Maximum observed SLA monitor loop duration inside the Executor.
* `kestra_executor_processing_flow_trigger_duration_seconds` (summary): Flow trigger processing duration inside the Executor.
* `kestra_executor_processing_flow_trigger_duration_seconds_max` (gauge): Maximum observed flow trigger processing duration inside the Executor.
* `kestra_executor_taskrun_created_count_total` (counter): The total number of tasks created by the Executor.
* `kestra_executor_taskrun_ended_count_total` (counter): he total number of tasks ended by the Executor.
* `kestra_executor_taskrun_ended_duration_seconds` (summary): Task duration inside the Executor.
* `kestra_executor_taskrun_ended_duration_seconds_max` (gauge): Maximum observed task duration inside the Executor.
* `kestra_executor_thread_count` (gauge): The number of executor threads.
* `kestra_executor_worker_job_resubmit_count_total` (counter): The total number of worker jobs resubmitted to the Worker by the Executor.
### Kestra Indexer Metrics
Indexer server exclusive:
* `kestra_indexer_message_in_count_total` (counter): Total number of records received by the Indexer
* `kestra_indexer_message_out_count_total` (counter): Total number of records indexed by the Indexer
* `kestra_indexer_request_count_total` (counter): Total number of batches of records received by the Indexer
* `kestra_indexer_request_duration_seconds` (summary): Batch of records duration inside the Indexer.
* `kestra_indexer_request_duration_seconds_max` (gauge): Maximum observed batch of records duration inside the Indexer.
### Kestra Scheduler Metrics
Scheduler server exclusive:
* `kestra_scheduler_evaluate_count_total` (counter): Total number of triggers evaluated by the Scheduler.
* `kestra_scheduler_evaluation_loop_duration_seconds` (summary): Trigger evaluation loop duration inside the Scheduler.
* `kestra_scheduler_evaluation_loop_duration_seconds_max` (gauge): Maximum observed trigger evaluation loop duration inside the Scheduler.
* `kestra_scheduler_loop_count_total` (counter): Total number of evaluation loops executed by the Scheduler.
### Kestra Worker Metrics
Worker server exclusive:
* `kestra_worker_ended_count_total` (counter): The total number of tasks ended by the Worker.
* `kestra_worker_ended_duration_seconds` (summary): Task run duration inside the Worker.
* `kestra_worker_ended_duration_seconds_max` (gauge): Maximum observed task run duration inside the Worker.
* `kestra_worker_job_pending` (gauge): The number of jobs (tasks or triggers) pending to be run by the Worker.
* `kestra_worker_job_running` (gauge): The number of jobs (tasks or triggers) currently running inside the Worker.
* `kestra_worker_job_thread` (gauge): The number of worker threads.
* `kestra_worker_queued_duration_seconds` (summary): Task queued duration inside the Worker.
* `kestra_worker_queued_duration_seconds_max` (gauge): Maximum observed task queued duration inside the Worker.
* `kestra_worker_running_count` (gauge): The number of tasks currently running inside the Worker.
* `kestra_worker_started_count_total` (counter): The total number of tasks started by the Worker.
### Kestra JDBC Metrics
Various Kestra-specific database queries:
* `kestra_jdbc_query_duration_seconds` (summary): Duration of database queries.
* `kestra_jdbc_query_duration_seconds_max` (gauge): Maximum observed query duration.
### Kestra Queue Metrics
For each internal queue:
* `kestra_queue_big_message_count_total` (counter): Big messages in the queue.
* `kestra_queue_message_lag_count` (gauge): Total number of messages in the queue that are not yet consumed.
* `kestra_queue_poll_size` (gauge): Size of a poll to the queue (message batch size).
* `kestra_queue_produce_count_total` (counter): Total number of produced messages.
* `kestra_queue_receive_duration_seconds` (summary): Queue duration to receive and consume a batch of messages.
* `kestra_queue_receive_duration_seconds_max` (gauge): Maximum observed queue duration to receive and consume a batch of messages.
## Cache metrics
Micronaut web server caching overview:
* `cache_size` (gauge): Current number of entries in the cache. Approximate depending on cache type.
## HikariCP Connection Pool Metrics
Database connection pool status:
* `hikaricp_connections` (gauge): Total number of connections in the pool.
* `hikaricp_connections_acquire_seconds` (summary): Time taken to acquire connections.
* `hikaricp_connections_acquire_seconds_max` (gauge): Maximum time observed for acquiring a connection.
* `hikaricp_connections_active` (gauge): Number of currently active connections.
* `hikaricp_connections_creation_seconds` (summary): Time taken to create new connections.
* `hikaricp_connections_creation_seconds_max` (gauge): Maximum observed connection creation time.
* `hikaricp_connections_idle` (gauge): Number of idle connections.
* `hikaricp_connections_max` (gauge): Maximum connections allowed in the pool.
* `hikaricp_connections_min` (gauge): Minimum connections maintained in the pool.
* `hikaricp_connections_pending` (gauge): Threads waiting to acquire a connection.
* `hikaricp_connections_timeout_total` (counter): Total count of connection timeouts.
* `hikaricp_connections_usage_seconds` (summary): Time connections are in use.
* `hikaricp_connections_usage_seconds_max` (gauge): Maximum observed connection usage time.
## HTTP Client Metrics
Outbound HTTP requests:
* `http_client_requests_seconds` (summary): Duration of HTTP client requests.
* `http_client_requests_seconds_max` (gauge): Maximum observed client request duration.
## HTTP Server Metrics
Inbound HTTP requests to Micronaut web server endpoints:
* `http_server_requests_seconds` (summary): Duration of HTTP server requests.
* `http_server_requests_seconds_max` (gauge): Maximum observed server request duration.
## JVM
## Java executor pool metrics
Various asynchronous task executors:
* `executor_active_threads` (gauge): The approximate number of threads that are actively executing tasks.
* `executor_completed_tasks_total` (counter): The approximate total number of tasks that have completed execution.
* `executor_idle_seconds` (summary): Time threads have spent idle in the executor pool.
* `executor_idle_seconds_max` (gauge): Maximum idle time observed for a thread.
* `executor_pool_core_threads` (gauge): The core number of threads for the pool.
* `executor_pool_max_threads` (gauge): The maximum allowed number of threads in the pool.
* `executor_pool_size_threads` (gauge): The current number of threads in the pool.
* `executor_queue_remaining_tasks` (gauge): The number of additional elements that this queue can ideally accept without blocking.
* `executor_queued_tasks` (gauge): The approximate number of tasks that are queued for execution.
* `executor_seconds` (summary): Time tasks have spent executing.
* `executor_seconds_max` (gauge): Maximum execution time observed for a task.
### JVM Buffer Pool Metrics
Overview of Java buffer pools type statistics:
* `jvm_buffer_count_buffers` (gauge): An estimate of the number of buffers in the pool.
* `jvm_buffer_memory_used_bytes` (gauge): An estimate of the memory that the Java virtual machine is using for this buffer pool.
* `jvm_buffer_total_capacity_bytes` (gauge): An estimate of the total capacity of the buffers in this pool.
### JVM Class Loading Metrics
Overview of Java class loading activity:
* `jvm_classes_loaded_classes` (gauge): The number of classes that are currently loaded in the Java virtual machine.
* `jvm_classes_unloaded_classes_total` (counter): The number of classes unloaded in the Java virtual machine.
### JVM Garbage Collection (GC) Metrics
Overview of runtime Java GC:
* `jvm_gc_concurrent_phase_time_seconds` (summary): Time spent in concurrent phase.
* `jvm_gc_concurrent_phase_time_seconds_max` (gauge): Maximum observed time spent in concurrent phase.
* `jvm_gc_live_data_size_bytes` (gauge): Size of long-lived heap memory pool after reclamation.
* `jvm_gc_max_data_size_bytes` (gauge): Max size of long-lived heap memory pool.
* `jvm_gc_memory_allocated_bytes_total` (counter): Incremented for an increase in the size of the (young) heap memory pool after one GC to before the next.
* `jvm_gc_memory_promoted_bytes_total` (counter): Count of positive increases in the size of the old generation memory pool before GC to after GC.
* `jvm_gc_pause_seconds` (summary): Time spent in GC pause.
* `jvm_gc_pause_seconds_max` (gauge): Maximum observed time spent in GC pause.
### JVM Memory Metrics
Overview of various Java memory regions:
* `jvm_memory_committed_bytes` (gauge): The amount of memory in bytes that is committed for the Java virtual machine to use.
* `jvm_memory_max_bytes` (gauge): The maximum amount of memory in bytes that can be used for memory management.
* `jvm_memory_used_bytes` (gauge): The amount of used memory.
### JVM Thread Metrics
Java threading model:
* `jvm_threads_daemon_threads` (gauge): The current number of live daemon threads.
* `jvm_threads_live_threads` (gauge): The current number of live threads including both daemon and non-daemon threads.
* `jvm_threads_peak_threads` (gauge): The peak live thread count since the Java virtual machine started or peak was reset.
* `jvm_threads_started_threads_total` (counter): The total number of application threads started in the JVM.
* `jvm_threads_states_threads` (gauge): The current number of threads.
## Logback metrics
Logger emitted events by log level:
* `logback_events_total` (counter): Log events enabled by the effective log level.
## Runtime metrics
### Process Metrics
Kestra from OS process point of view:
* `process_cpu_time_ns_total` (counter): The "cpu time" used by the Java Virtual Machine process.
* `process_cpu_usage` (gauge): The "recent cpu usage" for the Java Virtual Machine process.
* `process_files_max_files` (gauge):The maximum file descriptor count.
* `process_files_open_files` (gauge): The open file descriptor count.
* `process_start_time_seconds` (gauge): Start time of the process since unix epoch.
* `process_uptime_seconds` (gauge): The uptime of the Java virtual machine.
### System Metrics
Runtime resources overview:
* `system_cpu_count` (gauge): The number of processors available to the Java virtual machine.
* `system_cpu_usage` (gauge): The "recent cpu usage" of the system the application is running in.
* `system_load_average_1m` (gauge): The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time.
---
# Purge Executions, Logs, and Files in Kestra
URL: https://kestra.io/docs/administrator-guide/purge
> Reclaim storage by purging old executions, logs, KV entries, and orphaned execution files in Kestra. Configure scheduled purge jobs to keep your database lean in production.
Use purge tasks to remove old executions, logs, and key-value pairs and reduce storage usage.
To keep storage optimized, use [`PurgeExecutions`](/plugins/core/execution/io.kestra.plugin.core.execution.purgeexecutions), [`PurgeLogs`](/plugins/core/log/io.kestra.plugin.core.log.purgelogs), [`PurgeKV`](/plugins/core/kv/io.kestra.plugin.core.kv.purgekv), and [`PurgeStorage`](/plugins/core/storage/io.kestra.plugin.core.storage.purgestorage).
- `PurgeExecutions`: deletes execution records from the database and their associated storage files
- `PurgeLogs`: removes execution logs and non-execution logs (e.g. trigger logs) in bulk; use `purgeExecutionLogs` and `purgeNonExecutionLogs` to target each type independently. If you have configured an [external log data store](../log-data-store/index.md) that does not support purge, `PurgeLogs` is a no-op for logs — manage retention directly in that backend.
- `PurgeKV`: deletes expired keys globally for a specific namespace
- `PurgeStorage`: removes orphaned execution files from internal storage — files that exist on disk but whose execution records are no longer in the database
`PurgeExecutions`, `PurgeLogs`, and `PurgeKV` replace the legacy `io.kestra.plugin.core.storage.Purge` task. `PurgeStorage` is a new addition that handles orphaned files the other tasks cannot reach.
:::alert{type="info"}
The [Enterprise Edition](../../07.enterprise/index.mdx) also includes [`PurgeAuditLogs`](../../07.enterprise/02.governance/06.audit-logs/index.md#how-to-purge-audit-logs).
:::
## Purge executions and logs
Use a multi-step log purge that applies progressively shorter retention windows by log level. Verbose logs accumulate far faster than errors or warnings, so keeping them longer than necessary inflates storage without adding much value:
- All logs: purge anything older than **1 month**
- DEBUG logs: purge anything older than **1 week** — error stacktraces are often logged at DEBUG level, so this also removes them; extend the window if you need those for post-incident debugging
- TRACE logs: purge anything older than **1 day**
```yaml
id: purge
namespace: system
description: |
Multi-step purge: removes all logs older than one month, DEBUG logs older
than one week, and TRACE logs older than one day. Run daily to prevent
storage issues.
tasks:
- id: purge_executions
type: io.kestra.plugin.core.execution.PurgeExecutions
endDate: "{{ now() | dateAdd(-1, 'MONTHS') }}"
purgeLog: false
- id: purge_logs
type: io.kestra.plugin.core.log.PurgeLogs
endDate: "{{ now() | dateAdd(-1, 'MONTHS') }}"
# DEBUG logs often include error stacktraces; this shorter window keeps
# storage lean while still retaining recent failures for debugging.
- id: purge_debug_logs
type: io.kestra.plugin.core.log.PurgeLogs
endDate: "{{ now() | dateAdd(-1, 'WEEKS') }}"
logLevels:
- DEBUG
- id: purge_trace_logs
type: io.kestra.plugin.core.log.PurgeLogs
endDate: "{{ now() | dateAdd(-1, 'DAYS') }}"
logLevels:
- TRACE
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
```
### Selectively purge execution or trigger logs
Both `purgeExecutionLogs` and `purgeNonExecutionLogs` default to `true`. Set either to `false` to exclude that log type — for example, to retain execution logs for debugging while still clearing trigger logs.
Purge only trigger (non-execution) logs:
```yaml
id: purge-trigger-logs
namespace: company.myteam
tasks:
- id: purge_logs
type: io.kestra.plugin.core.log.PurgeLogs
endDate: "{{ now() | dateAdd(-1, 'MONTHS') }}"
purgeExecutionLogs: false
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
```
Purge only execution logs:
```yaml
id: purge-execution-logs
namespace: company.myteam
tasks:
- id: purge_logs
type: io.kestra.plugin.core.log.PurgeLogs
endDate: "{{ now() | dateAdd(-1, 'MONTHS') }}"
purgeNonExecutionLogs: false
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
```
The task outputs `executionLogsCount` and `nonExecutionLogsCount` alongside the existing `count` (total), so you can log or alert on how many of each type were removed.
### Control deletion batch size
By default, `PurgeLogs` deletes all matching rows in a single transaction. Use `batchSize` to split the deletion into smaller batches — useful when purging a large volume of logs to limit transaction size:
```yaml
id: purge-logs-batched
namespace: company.myteam
tasks:
- id: purge_logs
type: io.kestra.plugin.core.log.PurgeLogs
endDate: "{{ now() | dateAdd(-1, 'MONTHS') }}"
batchSize: 1000
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
```
## Purge orphaned execution files
`PurgeStorage` removes execution files from internal storage whose last-modified timestamp falls within a date window. Unlike `PurgeExecutions`, which is database-driven (it looks up execution records and deletes their files), `PurgeStorage` is storage-driven — it walks the storage tree directly and deletes files regardless of whether a matching execution record exists. This makes it the right tool for reclaiming storage that `PurgeExecutions` cannot reach.
:::alert{type="warning"}
`PurgeStorage` permanently deletes files. Always run with `dryRun: true` first and review the output counts before switching to `dryRun: false`. The `dryRun` property defaults to `true`.
:::
### Isolated worker groups and orphaned files
The most common use case is deployments with remote worker groups using dedicated internal storage that the primary Kestra cluster cannot access. When `PurgeExecutions` runs on the primary cluster, it deletes execution records from the database. Any subsequent `PurgeExecutions` run targeted at the remote worker group finds no execution records to match and never touches the isolated storage — leaving orphaned files behind.
There are two strategies depending on whether orphaned files already exist:
**Prevention — order your purge tasks correctly.** Run a worker-group-scoped `PurgeExecutions` with `purgeExecution: false` before the primary purge. This deletes files from the remote storage while the execution records still exist:
```yaml
id: purge_isolated_storage
namespace: system
tasks:
- id: purge_remote_files_first
type: io.kestra.plugin.core.execution.PurgeExecutions
endDate: "{{ now() | dateAdd(-1, 'MONTHS') }}"
purgeExecution: false
purgeStorage: true
workerSelector:
tags:
- my-worker-group
- id: purge_executions
type: io.kestra.plugin.core.execution.PurgeExecutions
endDate: "{{ now() | dateAdd(-1, 'MONTHS') }}"
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
```
**Remediation — clean up existing orphans with `PurgeStorage`.** If orphaned files already exist (execution records deleted but files remain), use `PurgeStorage` targeted at the remote worker group:
```yaml
id: purge_orphan_storage
namespace: system
tasks:
- id: dry_run
type: io.kestra.plugin.core.storage.PurgeStorage
namespace: company.team
endDate: "{{ now() | dateAdd(-1, 'MONTHS') }}"
dryRun: true
workerSelector:
tags:
- my-worker-group
- id: real_run
type: io.kestra.plugin.core.storage.PurgeStorage
namespace: company.team
endDate: "{{ now() | dateAdd(-1, 'MONTHS') }}"
dryRun: false
workerSelector:
tags:
- my-worker-group
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
```
Both tasks run sequentially in the same execution. Check the `dry_run` task output in the execution logs — when the counts look correct, set `dryRun: false` on the `real_run` task (or remove `dry_run` entirely for scheduled runs). Outputs are:
| Output | Description |
|---|---|
| `scannedCount` | Total execution storage directories found |
| `purgedCount` | Execution directories matched within the date window (preview when `dryRun: true`) |
| `deletedFilesCount` | Files actually deleted (`0` when `dryRun: true`) |
### Namespace scoping
`namespace` matching is recursive — `namespace: company` also reaches `company.team` and `company.team.prod`. Omitting `namespace` purges across every namespace under the tenant (requires tenant-admin level access). Narrowing with `flowId` is also supported but requires `namespace` to be set.
:::alert{type="info"}
In the Enterprise Edition, sub-namespaces configured with their own dedicated storage are **not** reached by recursive namespace scoping — they must be targeted explicitly by setting `namespace` to that sub-namespace. This applies directly to the isolated worker group pattern above.
:::
## Purge key-value pairs
The example below purges expired key-value pairs from the `company` namespace. It's set up as a flow in the [`system`](../../06.concepts/system-flows/index.md) namespace.
```yaml
id: purge_kv_store
namespace: system
tasks:
- id: purge_kv
type: io.kestra.plugin.core.kv.PurgeKV
expiredOnly: true
namespaces:
- company
includeChildNamespaces: true
```
:::alert{type="warning"}
Purge tasks permanently delete data. Always test in non-production environments first.
:::
## Auto-delete expired key-value pairs
Rather than scheduling a flow to purge key-value pairs, you can configure Kestra to delete expired entries automatically:
```yaml
kestra:
kv:
purge-expired:
enabled: true # default true
initial-delay: PT5S # default PT6H
fixed-delay: PT5S # default PT6H
batch-size: 10 # default 1000
```
## Purge namespace files
The example below purges old versions of namespace files for a namespace tree (parent and child namespaces). Use `filePattern` and `behavior` to keep the last N versions or delete versions older than a given date:
```yaml
id: purge_namespace_files
namespace: system
tasks:
- id: purge_files
type: io.kestra.plugin.core.namespace.PurgeFiles
namespaces:
- company
includeChildNamespaces: true
filePattern: "**/*.sql"
behavior:
type: version
before: "2025-01-01T00:00:00Z"
```
Refer to the [PurgeFiles documentation](/plugins/core/namespace/io.kestra.plugin.core.namespace.purgefiles) for more details.
## Purge assets and lineage (retention)
Use the `io.kestra.plugin.ee.assets.PurgeAssets` task to enforce asset retention without touching executions or logs. By default, this task purges assets, asset usage events (execution view), and asset lineage events (for asset exporters) matching the filters. You can configure it to only purge specific types of records.
**Filters:**
| Property | Description |
| --- | --- |
| `namespace` | Filter by namespace. Supports prefix matching (e.g., `company.data` matches `company.data.staging`). |
| `assetId` | Filter by a specific asset ID. |
| `assetType` | Filter by one or more asset types (e.g., `io.kestra.plugin.ee.assets.Table`). |
| `metadataQuery` | Filter by metadata key-value pairs. |
| `endDate` | **(required)** Purge records created or updated before this date (ISO 8601). |
**Purge scope:**
| Property | Default | Description |
| --- | --- | --- |
| `purgeAssets` | `true` | Whether to purge the asset records themselves. |
| `purgeAssetUsages` | `true` | Whether to purge asset usage events (execution view). |
| `purgeAssetLineages` | `true` | Whether to purge asset lineage events. |
**Outputs:** `purgedAssetsCount`, `purgedAssetUsagesCount`, `purgedAssetLineagesCount`.
Example: purge old VM assets on a monthly schedule.
```yaml
id: asset_retention_policy
namespace: company.infra
triggers:
- id: monthly_cleanup
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 0 1 * *"
tasks:
- id: purge_old_vms
type: io.kestra.plugin.ee.assets.PurgeAssets
assetType:
- io.kestra.plugin.ee.assets.VM
endDate: "{{ now() | dateAdd(-180, 'DAYS') }}"
```
## Purge tasks vs. UI deletion
Purge tasks perform **hard deletion**, permanently removing records and reclaiming storage. In contrast, deleting items in the UI is a **soft deletion** — the data is hidden but retained (e.g., revision history and past executions can reappear if a flow with the same ID is recreated).
This distinction matters for compliance and troubleshooting: purge flows are best for cleaning up space, while UI deletions preserve history for auditability.
:::alert{type="warning"}
Purge tasks do not affect Kestra’s [internal queues](../../08.architecture/01.main-components/index.md#queue). Queue retention is managed separately via the [Runtime and Storage configuration](../../configuration/02.runtime-and-storage/index.md) for JDBC or the [Enterprise and Advanced configuration](../../configuration/06.enterprise-and-advanced/index.md) for Kafka.
:::
---
# Software and Hardware Requirements to Run Kestra
URL: https://kestra.io/docs/administrator-guide/requirements
> Check Kestra system requirements. Verify software prerequisites (Java, DB) and hardware recommendations for running Kestra effectively.
This page outlines the software and hardware requirements for running Kestra.
## Software requirements
The table below lists the software requirements for Kestra.
### Java Runtime
| Kestra Edition | Required version | Note |
|----------------|------------------|------|
| Open Source / Enterprise | Runtime JDK 25; source/target 21 | Default: Java 25 (Eclipse Temurin); compiled with `--release 21` |
### Queue and Repository
Kestra Open Source supports PostgreSQL or MySQL for the queue and repository components.
Kestra Enterprise Edition (EE) provides two options:
- Use the same JDBC configuration as Open Source for standard deployments
- Use Kafka with Elasticsearch or OpenSearch for large-scale deployments
| Kestra Edition | Database | Required version | Note |
|-------------------|--------------------------|---------------------------|----------------|
| OSS / Enterprise | **PostgreSQL** | >=14 | Default `latest` |
| OSS / Enterprise | **MySQL** | >= 8 (except version 8.0.31) | Default: 8.3.2 |
| Enterprise | **Apache Kafka** | >=3 | |
| Enterprise | **Elasticsearch** | >=7 | |
| Enterprise | **Opensearch** | >=2 | |
:::alert{type="warning"}
MySQL deployments must have the **time zone tables loaded**. If the time zone data is missing, the scheduler can misfire or skip runs. Follow the MySQL guide to install time zone information to avoid deployment issues: [Time Zone Support → Load the Time Zone Tables](https://dev.mysql.com/doc/refman/8.0/en/time-zone-support.html#time-zone-installation).
:::
### Internal Storage
| Kestra Edition | Storage Provider | Required version | Note |
|-------------------|--------------------|---------------------------|----------------------------------|
| OSS/Enterprise | MinIO | >=8 | |
| OSS/Enterprise | Google Cloud GCS | N/A | |
| OSS/Enterprise | AWS S3 | N/A | |
| OSS/Enterprise | Azure Blob Storage | N/A | |
## Hardware requirements
A Kestra standalone server requires at least 4 GiB of memory and 2 vCPUs. To use script tasks, the server must support Docker-in-Docker (this is why, for example, AWS ECS Fargate is not supported).
For guidance on allocating memory and CPU for different architecture components, [contact us](/demo). We can help size your deployment based on your expected workload.
---
# Security Hardening for Kestra: Network and Process Isolation
URL: https://kestra.io/docs/administrator-guide/security-hardening
> Best practices for hardening Kestra security, including network isolation, host-level controls, and plugin validation.
Configure network isolation, endpoint authentication, and upload protection to harden a Kestra deployment.
By design, Kestra allows arbitrary HTTP calls and script execution. To prevent misuse of link-local metadata services (IMDS), isolate and block access at the network layer:
- Network ACLs or security groups: configure your VPC or firewall to deny all requests to link-local ranges (e.g., `169.254.169.254/32`).
- Dedicated orchestration subnet: place Kestra workers in a private subnet with no route to management or metadata services.
- Egress proxy or NAT gateway filtering: route all outbound traffic through a proxy or gateway that can enforce allow-lists and block link-local IPs.
## Host-level isolation
Running workflows in isolated environments reduces the impact of potentially malicious flows:
- Container sandboxes: launch each flow execution in its own container (for example, Docker or Kubernetes Pod) with minimal privileges.
- Ephemeral compute: use Kestra's native [Task Runners](../../07.enterprise/04.scalability/task-runners/index.md) to auto-scale ephemeral compute nodes that are destroyed after each run, leaving no residual state.
- Minimum host permissions: grant only the OS-level rights required for the runtime; avoid mounting cloud credential files or granting host-level IAM roles directly.
## Transport security (EE only)
In distributed deployments, Worker Controllers communicate with Workers over gRPC. By default this channel is plaintext. Enterprise Edition supports TLS encryption and mutual TLS (mTLS) to authenticate both sides of the connection:
- **One-way TLS** — the controller presents a certificate; workers verify it. Encrypts the channel without requiring worker certificates.
- **Mutual TLS (mTLS)** — both controller and worker present certificates. Use this when you need strong identity verification between components, not just encryption.
See [gRPC TLS/mTLS configuration](../../configuration/06.enterprise-and-advanced/index.md#grpc-tlsmtls-ee-only) for setup instructions and a full property reference.
## HTTP task URL filtering
HTTP plugin tasks (`Request`, `Download`, `SseRequest`, and `Trigger`) make server-side HTTP calls to URIs controlled by flow authors. Without restrictions, a flow author can reach cloud metadata endpoints (such as `169.254.169.254` on AWS, GCP, and Azure), internal management APIs, or private services not reachable from the internet.
Configure an allow-list, a deny-list, or both under `kestra.tasks.http`:
```yaml
kestra:
tasks:
http:
allowed-list:
- https://api.example.com
- https://data.partner.io
denied-list:
- http://169.254.169.254
- http://localhost
- http://127.0.0.1
```
| Property | Default | Description |
|---|---|---|
| `kestra.tasks.http.allowed-list` | `[]` | When non-empty, a request URI must start with at least one entry or the task fails. |
| `kestra.tasks.http.denied-list` | `[]` | A request URI that starts with any entry causes the task to fail. Evaluated after the allowed-list. |
Both lists are empty by default — no filtering is applied unless you configure them.
When both lists are set, the allowed-list is checked first. A URI that matches an allowed-list entry but also matches a denied-list entry is still blocked.
**Matching is prefix-based**, not glob or CIDR. Each entry is a literal string prefix, so:
- `http://169.254.169.254` blocks `http://169.254.169.254/latest/meta-data/...`
- `http://10.` blocks `http://10.0.0.1/admin` but not `https://10.0.0.1/admin` because the scheme differs
When a URI is blocked, the task fails with an error that identifies the matching config key:
```
The URI http://169.254.169.254/... is in the configured denied list (kestra.tasks.http.denied-list).
```
:::alert{type="info"}
This filter applies to HTTP plugin tasks only. The `http()` Pebble expression function makes independent server-side HTTP calls and is not covered by this configuration.
:::
## Plugin and code validation
- Plugin configuration: use Kestra’s plugin architecture, including [Plugin Versioning](../../07.enterprise/05.instance/versioned-plugins/index.md), to control which plugins are allowed and [which should be prohibited](../../07.enterprise/02.governance/worker-isolation/index.md).
- CI/CD validation: add a [Flow Validation step in your CI/CD pipeline](../../version-control-cicd/cicd/index.md) to scan task definitions for disallowed patterns (e.g., `169.254.169.254`) and block merging if detected.
- Java Security (EE): Enterprise Edition users can define security policies to restrict access to untrusted files, plugins, or network resources.
## Credential initialization
On Enterprise Edition, use [OIDC/SSO](../../07.enterprise/03.auth/sso/index.md) or [LDAP](../../07.enterprise/03.auth/sso/ldap/index.md) instead of Basic Authentication. These integrate with your existing identity provider, support MFA, and remove the risk of locally managed credentials.
If you use Basic Authentication on OSS or EE:
:::alert{type="warning"}
On a fresh installation with no `basic-auth` credentials configured, the Setup page at `/ui/main/setup` is publicly reachable. Any user who reaches it first can set credentials and lock out the intended administrator.
**Configure `username` and `password` in the application configuration file before starting Kestra in production.** This skips the Setup page entirely and ensures the instance is never in an unprotected state.
```yaml
kestra:
server:
basic-auth:
username: admin@kestra.io
password: "{{ your-strong-password }}"
```
:::
## Management endpoint access
Kestra exposes internal endpoints on a separate management port (default `8081`). These include health checks, metrics (`/metrics`), and runtime log level inspection (`/loggers`). This port is unauthenticated by default.
The primary protection is network isolation: do not expose port `8081` outside the internal network. Firewall or security-group rules should restrict access to the management port to trusted internal hosts only (monitoring agents, load balancer health checkers, operations tooling).
If network isolation is not sufficient, you can add Basic Auth to the management port and optionally move it to a non-default port by configuring `endpoints.all` in your `application.yml`:
```yaml
endpoints:
all:
port: 8084 # move away from the default 8081
basic-auth:
username:
password:
```
When `basic-auth` credentials are present, Kestra's management endpoint filter requires Basic Auth on every request to that port.
See [Kestra endpoints](../03.monitoring/index.md#kestra-endpoints) for the full list of what is exposed on the management port.
## ZIP bomb protection
Kestra guards against ZIP bomb attacks on the two endpoints that accept user-uploaded ZIP archives: flow import and namespace file upload. The protection is opt-in and disabled by default.
Enable it under `kestra.security.zip-bomb-protection` in your `application.yml`:
```yaml
kestra:
security:
zip-bomb-protection:
enabled: true
max-number-of-entries: # maximum number of entries in the archive
max-entry-size: # maximum uncompressed size of a single entry
```
Both `max-number-of-entries` and `max-entry-size` are required when `enabled: true`. Set them based on the largest legitimate ZIP archives your users are expected to upload. When a ZIP exceeds either limit, the request is rejected with an HTTP 422 error.
:::alert{type="info"}
The protection applies only to user-uploaded ZIPs (flow import and namespace file upload). Internal Kestra archives used for task execution caching are not affected.
:::
## Documentation and audit
- User guidance: update onboarding materials and runbooks to highlight metadata-blocking best practices when deploying a new Kestra environment.
- Periodic review: include network and host configuration checks in your security audit cycle to verify link-local ranges remain blocked.
---
# Server Heartbeats and Job Recovery in Kestra
URL: https://kestra.io/docs/administrator-guide/server-lifecycle
> Understand Kestra's server liveness mechanism, heartbeats, and how it handles component failures and recovery.
Kestra is separated into several components that can be deployed independently or inside a single process (a standalone deployment).
These components are called **server components** or just **servers**.
See the [server components](../../08.architecture/02.server-components/index.md) and [deployment](../../08.architecture/03.deployment-architecture/index.md) sections for more information.
Kestra has a built-in liveness mechanism. Each server sends a periodic heartbeat stored inside the database. The Executor runs the cluster-wide liveness coordinator: it reviews every registered service instance on a scheduled tick and drives state transitions when heartbeats are missed.
When a server is not alive, Kestra runs maintenance routines such as [worker job resubmission](#worker-job-resubmission).
## The liveness mechanism
When a server starts, it sends a heartbeat to the database with a `RUNNING` status.
When it stops, it first transitions to `TERMINATING`. If the server has pending tasks, it waits up to the configured `kestra.server.terminationGracePeriod` for them to finish. If it completes within that window, it sends a `TERMINATED_GRACEFULLY` heartbeat; otherwise the process is terminated with status `TERMINATED_FORCED`.
Other servers detect missing heartbeats and run maintenance tasks:
- Workers resubmit pending jobs to another worker before transitioning to `NOT_RUNNING`.
- Other server types transition to `NOT_RUNNING` immediately.
By default, liveness checks run every 10 seconds.
`NOT_RUNNING` servers transition to `INACTIVE` at the next liveness check.
If a server does not send a heartbeat within `kestra.server.liveness.timeout`, it is marked `DISCONNECTED` and then `NOT_RUNNING` at the next check. If that server is still alive, it self-terminates after detecting that other components classified it as `NOT_RUNNING`, preventing “resurrection.”
For configuration details, see the [Runtime and Storage configuration](../../configuration/02.runtime-and-storage/index.md).
## Worker job resubmission
The Worker has a special behavior that allows it to resubmit jobs that were not completed due to a server termination or any kind of failures.
**Worker jobs** are tasks or triggers currently executing on a worker.
When the Executor sends a task to a worker, it creates an entry in the worker job store. Workers remove the entry once they complete the task. The same logic applies to triggers evaluated by the Scheduler.
The liveness mechanism resubmits pending jobs from a terminating worker to another worker, ensuring each job runs at least once.
Configure this behavior via `kestra.server.workerTaskRestartStrategy`:
- `AFTER_TERMINATION_GRACE_PERIOD` (default): wait another grace period before resubmitting jobs, preventing a terminated worker from returning.
- `IMMEDIATELY`: resubmit jobs right away.
- `NEVER`: never resubmit jobs (tasks remain incomplete and flows stay `RUNNING`).
::alert{type="info"}
This resubmission mechanism also applies to **Realtime Triggers**. If the worker running a Realtime Trigger listener is stopped gracefully, Kestra waits for the `terminationGracePeriod` before reassigning the trigger to another worker. See [Worker failover for Realtime Triggers](../../05.workflow-components/07.triggers/05.realtime-trigger/index.md#worker-failover-for-realtime-triggers) for more details.
::
Resubmitted task runs show multiple attempts in the UI.

In the timeline, one of the states will be `RESUBMITTED`.

## Instance view (EE only)
Kestra Enterprise exposes an instance dashboard accessible via the [Instance Owner console](../../07.enterprise/05.instance/00.instance-owner/index.md) that summarizes heartbeats, liveness status, and maintenance activity across clusters. See the [instance dashboard documentation](../../07.enterprise/05.instance/index.mdx) for a walkthrough.
---
# Service Instance Metrics in Kestra – Expose via the Webserver
URL: https://kestra.io/docs/administrator-guide/service-instance-metrics
> Configure the webserver to collect and expose metrics from other Kestra service instances, making them available through a single monitoring endpoint.
Configure which service instance metrics the webserver collects and re-exposes through its monitoring endpoint.
## Service instance metrics – expose via the webserver
By default, each Kestra service (Worker, Executor, Scheduler, and so on) publishes its own metrics on its own management port. In deployments where only the webserver is reachable from the outside — for example, behind an ingress or a load balancer — metrics from other service instances are not directly accessible.
The `kestra.metrics.sharedServiceInstanceMetrics` configuration lets you define a set of metrics per service type that the webserver will collect from all running instances and re-expose on its own monitoring endpoint. This gives any compatible monitoring consumer a single scrape target for the metrics you choose.
## How it works
The webserver runs a background task that polls the service instance repository every 30 seconds. For each running service instance whose type is listed in `sharedServiceInstanceMetrics`, it collects the configured metrics, sums the values across all instances of that type, and registers the results as Micrometer gauges. The webserver itself is excluded from this aggregation.
Metric tags are preserved, so you can still filter by dimensions such as `worker_group`.
When a service instance is no longer active, its contribution is set to zero at the next poll cycle.
## How to configure `sharedServiceInstanceMetrics`
Set `kestra.metrics.sharedServiceInstanceMetrics` in your [configuration file](../../configuration/03.observability-and-networking/index.md). The value is a map where each key is a service type and each value is a list of fully-qualified metric names to collect from instances of that type. Kestra ships with the following defaults:
```yaml
kestra:
metrics:
prefix: kestra
sharedServiceInstanceMetrics:
WORKER:
- kestra.worker.job.pending
- kestra.worker.job.thread
- kestra.worker.job.running
```
You can extend or replace these entries to expose any metric listed in the [Alerting and Monitoring](../03.monitoring/index.md) reference.
:::alert{type="info"}
Metric names must include the prefix (e.g. `kestra.`) as configured in `kestra.metrics.prefix`.
:::
### Configuration properties
| Property | Type | Description |
|---|---|---|
| `kestra.metrics.sharedServiceInstanceMetrics` | `Map>` | Maps each service type to the metric names to collect and expose via the webserver. Defaults to an empty map. |
| Key | `ServiceType` | One of `EXECUTOR`, `INDEXER`, `SCHEDULER`, `WEBSERVER`, `WORKER` |
| Value | List of `String` | Fully-qualified metric names to collect from instances of that type, including the configured prefix |
## Next steps
- [Prometheus Metrics](../prometheus-metrics/index.md) — scrape Kestra metrics using Prometheus
- [OpenTelemetry](../open-telemetry/index.md) — export Kestra metrics using OpenTelemetry
- [Alerting and Monitoring](../03.monitoring/index.md) — full metrics reference for each service component
- [Observability and Networking Configuration](../../configuration/03.observability-and-networking/index.md) — configure the metrics prefix, tags, and label-based metrics
- [Server Components](../../08.architecture/02.server-components/index.md) — understand the role of each Kestra service
---
# SSL/TLS Configuration: Enable HTTPS for Kestra
URL: https://kestra.io/docs/administrator-guide/ssl-configuration
> Configure SSL/TLS encryption for Kestra to secure the UI and API access using self-signed or CA-signed certificates.
Configure secure access to the Kestra UI via HTTPS. The right approach depends on your deployment type:
## Why use SSL/TLS encryption
Adding TLS encryption to your environment provides the following benefits:
- Data is encrypted in transit, preventing sensitive data from being intercepted in "man-in-the-middle" attacks.
- TLS adds a layer of trust by ensuring users know the URL they access is genuine (e.g., `https://mycompany.kestra.com/ui` is verified as an internal site).
For further details, see [Why use HTTPS?](https://www.cloudflare.com/en-gb/learning/ssl/why-use-https/) on the Cloudflare documentation.
## Using Caddy as a reverse proxy
[Caddy](https://caddyserver.com/) is a reverse proxy that manages HTTPS automatically. For public domains it obtains and renews Let's Encrypt certificates without any manual steps; for local development it generates a local CA and prompts once for system trust.
### Local development (macOS)
This setup gives multiple local Kestra instances clean HTTPS URLs (e.g. `https://oss`, `https://ee`) with a single local CA prompt.
1. **Install Caddy**:
```bash
brew install caddy
```
2. **Create a `~/Caddyfile`** with one `reverse_proxy` block per Kestra instance (adjust ports to match your instances):
```
oss {
reverse_proxy localhost:8080
}
oss-dev {
reverse_proxy localhost:8081
}
ee {
reverse_proxy localhost:8082
}
ee-dev {
reverse_proxy localhost:8083
}
```
3. **Run Caddy**:
```bash
sudo caddy run --config ~/Caddyfile
```
Caddy generates a local CA on first run and prompts once to trust it in the macOS Keychain. After that, all configured hostnames are available over HTTPS with no browser warnings.
:::alert{type="info"}
For the short hostnames (e.g. `oss`, `ee`) to resolve, add them to `/etc/hosts`:
```
127.0.0.1 oss oss-dev ee ee-dev
```
:::
### Production (public domain)
For a publicly accessible Kestra instance, point your domain at the server and Caddy handles certificate issuance and renewal automatically via Let's Encrypt.
```
kestra.yourdomain.com {
reverse_proxy localhost:8080
}
```
Run Caddy as a background service so it persists across reboots:
```bash
sudo caddy start --config /etc/caddy/Caddyfile
```
With this setup, Kestra itself runs on plain HTTP internally (`localhost:8080`) and Caddy terminates TLS at the edge — no changes to Kestra's Micronaut SSL configuration are needed.
:::alert{type="info"}
Kestra uses WebSockets for real-time execution log streaming. Caddy's `reverse_proxy` directive supports WebSocket upgrades by default, so no extra configuration is required.
:::
## Creating self-signed certificates
To get started in lower environments, create self-signed certificates using the OpenSSL library. For more detail on examining certificates and keys, see this [Micronaut article](https://guides.micronaut.io/latest/micronaut-security-x509-maven-groovy.html).
:::alert{type="info"}
While self-signed certificates encrypt traffic, they are considered unsuitable for production usage. They are deemed untrustworthy, as they do not come from a trusted Certificate Authority (CA) such as [Let's Encrypt](https://letsencrypt.org/). Follow your organization's best practices when choosing a CA provider.
:::
```bash
## Create a folder which will be later mounted to the kestra container
mkdir -p /app/ssl
cd /app/ssl
```
```bash
## Create CA in PEM format along with private key
openssl req -x509 -sha256 -days 365 -newkey rsa:4096 \
-keyout cacert.key -out cacert.pem \
-subj '/CN=example.kestra.com/C=IE/O=kestra' \
-passout pass:changeit
## Create certificate signing request
openssl req -newkey rsa:4096 \
-keyout server.key -out server.csr \
-subj '/CN=example.kestra.com/C=IE/O=kestra' \
-passout pass:changeit
## Create the server configuration which will be used to sign the certificate
cat <<< 'authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost' > server.conf
## sign certificate
openssl x509 -req -CA cacert.pem -CAkey cacert.key \
-in server.csr -out server.pem -days 365 \
-CAcreateserial -extfile server.conf \
-passin pass:changeit
## Create server.p12
openssl pkcs12 -export -out server.p12 -name "localhost" \
-inkey server.key -in server.pem \
-passin pass:changeit \
-passout pass:changeit
## Create keystore.p12 with JDK keytool
keytool -importkeystore -srckeystore server.p12 \
-srcstoretype pkcs12 -destkeystore keystore.p12 \
-deststoretype pkcs12 \
-deststorepass changeit -srcstorepass changeit
## Create truststore.jks
keytool -import -trustcacerts -noprompt -alias ca \
-ext san=dns:localhost,ip:127.0.0.1 \
-file cacert.pem -keystore truststore.jks \
-storepass changeit -keypass changeit
```
## Micronaut SSL configuration
Configure HTTPS through the `micronaut` settings in the [Observability and Networking configuration](../../configuration/03.observability-and-networking/index.md).
:::alert{type="info"}
Ensure that you expose the secure port of the connection if different from the default port.
:::
```yaml
kestra:
image: registry.kestra.io/docker/kestra:latest
pull_policy: always
user: "root"
command: server standalone --worker-thread=128
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- tmp-kestra:/tmp/kestra-wd
- /app/ssl:/app/ssl
ports:
- "8443:8443"
environment:
KESTRA_CONFIGURATION: |
micronaut:
security:
x509:
enabled: false
ssl:
enabled: true
server:
ssl:
port: 8443
enabled: true
clientAuthentication: want
keyStore:
path: file:/app/ssl/server.p12
password: changeit
type: PKCS12
trustStore:
path: file:/app/ssl/truststore.jks
password: changeit
type: JKS
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driver-class-name: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
server:
basic-auth:
enabled: false
username: "admin@kestra.io" # it must be a valid email address
password: kestra
repository:
type: postgres
storage:
type: local
local:
base-path: "/app/storage"
queue:
type: postgres
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
ports:
- "8443:8443"
```
## Outbound SSL configuration
If Kestra tasks make outbound calls to services that require SSL, configure the JVM to trust your certificates by setting the following options in the `JAVA_OPTS` environment variable in your [Observability and Networking configuration](../../configuration/03.observability-and-networking/index.md):
```yaml
JAVA_OPTS: "-Djavax.net.ssl.trustStore=/app/ssl/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit"
```
The following example shows the full service configuration with `JAVA_OPTS` set:
```yaml
kestra:
image: registry.kestra.io/docker/kestra:latest
pull_policy: always
user: "root"
command: server standalone --worker-thread=128
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- tmp-kestra:/tmp/kestra-wd
- /app/ssl:/app/ssl
ports:
- "8443:8443"
environment:
JAVA_OPTS: "-Djavax.net.ssl.trustStore=/app/ssl/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit"
KESTRA_CONFIGURATION: |
micronaut:
security:
x509:
enabled: false
ssl:
enabled: true
server:
ssl:
port: 8443
enabled: true
clientAuthentication: want
keyStore:
path: file:/app/ssl/server.p12
password: changeit
type: PKCS12
trustStore:
path: file:/app/ssl/truststore.jks
password: changeit
type: JKS
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driver-class-name: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
server:
basic-auth:
enabled: false
username: "admin@kestra.io" # it must be a valid email address
password: kestra
repository:
type: postgres
storage:
type: local
local:
base-path: "/app/storage"
queue:
type: postgres
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
ports:
- "8443:8443"
```
## Enabling CSRF protection
Cross-site request forgery (CSRF) is an attack where a malicious website or email tricks a user's browser into performing unwanted actions on a trusted site while authenticated.
CSRF protection requires TLS/SSL to be enabled on your instance. Once TLS is configured, add the following to your configuration file:
```yaml
micronaut:
security:
csrf:
enabled: true
```
This setting enables CSRF protection on all endpoints that reach `/api/.*`.
## Configuring SSL with Kubernetes
For Kubernetes deployments, you can enable HTTPS either by configuring TLS at the Ingress level or by using self-signed certificates at the application level.
### Using ingress with TLS termination (recommended for production)
Most cloud providers expect TLS termination at the ingress controller. Here's how to configure HTTPS using Let's Encrypt certificates:
1. **Install cert-manager**. To use a different version, see [available releases on GitHub](https://github.com/cert-manager/cert-manager/releases):
```bash
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.1/cert-manager.yaml
```
2. **Create a Let's Encrypt issuer** (replace `your-email@example.com`):
```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: your-email@example.com
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginx # Update for your ingress controller
```
3. **Configure Ingress with TLS** (Azure AKS example):
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kestra-ingress
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
tls:
- hosts:
- kestra.yourdomain.com
secretName: kestra-tls
rules:
- host: kestra.yourdomain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kestra-service
port:
number: 80
```
### Using self-signed certificates (for testing)
1. **Generate certificates** using the OpenSSL commands in the [Creating self-signed certificates](#creating-self-signed-certificates) section above.
2. **Create TLS secret**:
```bash
kubectl create secret tls kestra-tls \
--cert=server.pem \
--key=server.key
```
3. **Reference the secret in your Ingress**:
```yaml
spec:
tls:
- hosts:
- kestra.yourdomain.com
secretName: kestra-tls
```
### Application-level SSL configuration
For environments where ingress TLS termination isn't available:
1. **Create secret with SSL files**:
```bash
kubectl create secret generic kestra-ssl \
--from-file=keystore.p12 \
--from-file=truststore.jks
```
2. **Configure Kestra deployment**:
```yaml
env:
- name: KESTRA_CONFIGURATION
value: |
micronaut:
server:
ssl:
enabled: true
port: 8443
keyStore:
path: file:/app/ssl/keystore.p12
password: changeit
type: PKCS12
volumeMounts:
- name: ssl-secret
mountPath: "/app/ssl"
volumes:
- name: ssl-secret
secret:
secretName: kestra-ssl
```
3. **Expose HTTPS port** in your service:
```yaml
ports:
- name: https
port: 8443
targetPort: 8443
```
:::alert{type="warning"}
Production deployments on cloud platforms such as Azure AKS typically require valid certificates from trusted CAs for SSO integration. Self-signed certificates may work for testing but aren't suitable for production use.
:::
### Verifying the configuration
Check certificate validity with:
```bash
kubectl get certificate kestra-tls -w
```
Expected output:
```plaintext
NAME READY SECRET AGE
kestra-tls True kestra-tls 5m
```
For a step-by-step guide to adding a self-signed or internal CA to the JVM truststore in a Kubernetes deployment, see [Trusting a custom CA for outbound connections on Kubernetes](../custom-ca-kubernetes/index.md).
---
# Troubleshoot Kestra: Kubernetes, Docker, and Startup Issues
URL: https://kestra.io/docs/administrator-guide/troubleshooting
> Solutions for common Kestra issues, including pod restarts, unprocessable executions, and Docker-in-Docker problems.
Common issues and fixes for Kestra deployments.
## CrashLoopBackoff when restarting all pods
**Question:** "When I restart all Kubernetes pods at once, they get stuck in a `CrashLoopBackoff` for a number of minutes before eventually resolving — why does it happen?"
This is likely caused by Java startup behavior, which can consume a lot of resources and cause liveness probes to fail. Since Java loads many classes at startup, pods may restart multiple times before stabilizing. Setting the CPU limit to 2 times the request can improve startup time and resolve failing health checks.
## Unprocessable execution
Sometimes, executions cannot be processed. In such cases, you can instruct Kestra to skip them.
Start the executor server (or the standalone server if not using a multi-component deployment) with a list of execution identifiers to skip:
```sh
kestra server executor --skip-executions 6FSPERUe1JwbYmMmdwRlgV,5iLGjTLOHAVGUGlsesFaMb
```
You can also skip executions at broader levels:
1. **Flows** — Skip all executions of one or more flows:
```sh
kestra server executor "--skip-flows=tenant|namespace|flowA,tenant|namespace|flowB"
```
Example:
```sh
kestra server executor "--skip-flows=companyA|production-data|daily-data-sync"
```
:::alert{type="info"}
Replace `tenant` and `namespace` with the correct values for the flow.
:::
2. **Namespaces** — Skip all executions within specific namespaces:
```sh
kestra server executor "--skip-namespaces=tenant|myNamespaceA,tenant|myNamespaceB"
```
Example:
```sh
kestra server executor "--skip-namespaces=companyA|production-data"
```
:::alert{type="info"}
Replace `tenant` with the correct values for the namespace.
:::
3. **Tenants** — Skip all executions associated with specific tenants:
```sh
kestra server executor "--skip-tenants=tenantA,tenantB"
```
Example:
```sh
kestra server executor "--skip-tenants=companyA"
```
## Docker in Docker (DinD)
If you face issues using Docker in Docker (e.g., with [Script tasks](../../16.scripts/index.mdx) using the `DOCKER` runner), start troubleshooting by attaching to the DinD container:
```sh
docker run -it --privileged docker:dind sh
```
From there, use:
- `docker logs ` to view logs
- `docker inspect ` to get environment, network, and configuration details
These commands help diagnose misconfigurations.
## Docker in Docker using Helm charts
On some Kubernetes deployments, using DinD with our default Helm charts can result in errors such as:
```bash
Device "ip_tables" does not exist.
ip_tables 24576 4 iptable_raw,iptable_mangle,iptable_nat,iptable_filter
modprobe: can't change directory to '/lib/modules': No such file or directory
error: attempting to run rootless dockerd but need 'kernel.unprivileged_userns_clone' (/proc/sys/kernel/unprivileged_userns_clone) set to 1
```
To fix this, switch DinD to insecure (privileged) mode by setting the following values:
```yaml
dind:
mode: 'insecure'
base:
insecure:
image:
tag: dind
args:
- --log-level=fatal
securityContext:
runAsUser: 0
runAsGroup: 0
```
## DinD on a Mac with Apple silicon (ARM)
If you see errors like:
```plaintext
java.io.IOException: com.sun.jna.LastErrorException: [111] Connection refused
```
it may be caused by running Docker in Docker on ARM-based Macs.
Try using an embedded Docker server as shown below:
:::collapse{title="Example docker-compose.yml"}
```yaml
## volumes omitted for brevity
services:
postgres:
image: postgres
# ...
dind:
image: docker:dind
privileged: true
environment:
DOCKER_HOST: unix://dind/docker.sock
command:
- --log-level=fatal
volumes:
- dind-socket:/dind
- tmp-data:/tmp/kestra-wd
kestra:
image: kestra/kestra:latest
entrypoint: /bin/bash
user: "root" # dev only — not for production
command:
- -c
- /app/kestra server standalone --worker-thread=128
volumes:
- kestra-data:/app/storage
- dind-socket:/dind
- tmp-data:/tmp/kestra-wd
environment:
KESTRA_CONFIGURATION: |
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driver-class-name: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
storage:
type: local
local:
base-path: "/app/storage"
queue:
type: postgres
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
ports:
- "8080:8080"
- "8081:8081"
```
:::
## tmp directory errors ("No such file or directory")
If you encounter errors such as `"No such file or directory"` related to the tmp directory, it usually means the directory is not mounted correctly.
In your `docker-compose.yml`, ensure the `tmpDir` path matches the mounted volume:
```yaml
kestra:
tasks:
tmp-dir:
path: /home/kestra/tmp
```
Example volume configuration:
```yaml
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /home/kestra:/home/kestra
```
This ensures Kestra can properly access the tmp directory.
---
# Upgrade Kestra: Rolling Updates, Migrations, and Rollback
URL: https://kestra.io/docs/administrator-guide/upgrades
> Best practices for upgrading Kestra, performing rolling updates, and rolling back to previous versions safely.
Kestra evolves quickly. This page explains how to upgrade your installation.
## How to upgrade Kestra
To upgrade Kestra, follow these steps:
1. Perform a database backup (optional but recommended).
2. Read the [release notes](https://github.com/kestra-io/kestra/releases) to understand the changes in the new version.
3. Perform a rolling upgrade of Kestra components. For Kubernetes, upgrade the Kestra Helm chart as described in “Rolling upgrades in Kubernetes,” below.
4. Apply any actions noted in the release notes (for example, update configuration files or adjust deprecated features).
## How to rollback Kestra to a previous version
Sometimes you might need to roll back Kestra to a previous version. Follow these steps:
1. Perform a database backup (optional but recommended).
2. Stop all Kestra components.
3. Restore from a backup.
4. Restart with the older version.
Check the [Backup and Restore](../backup-and-restore/index.md) section for more information on how to backup and restore Kestra, and [Maintenance Mode](../../07.enterprise/05.instance/maintenance-mode/index.md) to pause your Kestra instance for maintenance, upgrade, and backup tasks.
:::alert{type="warning"}
We strongly recommend avoiding downgrades. To prevent surprises, test the new version in a non-production environment before upgrading. If you must roll back, closely follow the steps above.
:::
## Where you can find the release changelog
You can find the changelog on the main repository’s [Releases](https://github.com/kestra-io/kestra/releases) page. It lists changes, new features, and bug fixes for each release, as well as any breaking changes. For a high-level overview, see the release [blog posts](/blogs).
## How to identify breaking changes in a release
In addition to bug fixes and enhancements, the release notes include a `Breaking Changes` section. It lists changes that may require adjustments to your code or Kestra configuration, with links to [migration docs](../../11.migration-guide/index.mdx).
:::alert{type="warning"}
The `Breaking Changes` section appears at the end of the [release notes](https://github.com/kestra-io/kestra/releases). Review it before upgrading.
:::
## How to minimize downtime when updating Kestra
If you run Kestra as separate components, you should:
- Stop the executors and the scheduler
- Stop the workers — a graceful shutdown waits for active jobs to finish. The default is `kestra.server.terminationGracePeriod = 'PT5M'`, configurable in your [Runtime and Storage configuration](../../configuration/02.runtime-and-storage/index.md).
- If the job finishes within five minutes, the worker shuts down immediately. Otherwise, the task is killed and restarts when the worker restarts.
- Stop the webserver (and the indexer if using EE with Kafka).
All components support graceful shutdown, so no data is lost. Afterward, update and restart everything in the opposite order (or in any order, as components are independent).
:::alert{type="info"}
The webserver hosts the API, so stop and then start it immediately to avoid downtime. After that, restart the other components so flow executions can resume.
:::
## How to stick to a specific Kestra version
If you want to stick to a specific Kestra version, you can pin the [Docker image tag](https://hub.docker.com/r/kestra/kestra/tags) to a specific release. Here are some examples:
- `kestra/kestra:v2.0.0-slim` — the 2.0.0 release without bundled plugins
- `kestra/kestra:v2.0.0` — the 2.0.0 release with all plugins
The `-slim` suffix replaces the former `-no-plugins` suffix; releases older than the rename only exist under `-no-plugins` (e.g. `kestra/kestra:v0.21.4-no-plugins`).
You can also create a custom image with your own plugins and dependencies, as explained in the [Docker installation](../../02.installation/02.docker/index.md).
## Migrating a standalone installation
If you use a manual standalone installation with Java, you can download the Kestra binary for a specific version from the Assets menu of a specific [Release](https://github.com/kestra-io/kestra/releases) page. The image below shows how you can download the binary for the 0.14.1 release.

Once you’ve downloaded the binary, start Kestra with the following command:
```bash
./kestra-VERSION server standalone
```
## Migrating an installation with Docker
If you use Docker, change the [Docker image tag](https://hub.docker.com/r/kestra/kestra/tags) to the desired version and restart the container(s) or Kubernetes pod(s).
### Docker Compose
If you use Docker Compose, update your compose file to the desired [Docker image tag](https://hub.docker.com/r/kestra/kestra/tags) and run `docker compose up -d` to restart the container(s).
## Migration in Kubernetes using Helm
If you use Helm, set the [Helm chart `image.tag` value](https://github.com/kestra-io/kestra/blob/develop/charts/kestra/values.yaml) to the desired version. For example:
```bash
helm upgrade kestra kestra/kestra --set image.tag=v1.0.0
```
For more complex configurations that include multiple changes, consider using a custom values file:
1. First, create a `values.yaml` file that contains the settings you want to adjust.
2. Then, use the `helm upgrade` command with the `-f` flag to specify your custom values file:
```sh
helm upgrade kestra kestra/kestra -f values.yaml
```
## Rolling upgrades in Kubernetes
Upgrading Kestra on Kubernetes depends on your deployment rollout strategy. Every service can be rolled out without downtime, except workers, which need special attention.
During rollout, each component creates a new pod (the old one keeps running). After the new pod passes health checks, Kubernetes shuts down the previous pod, resulting in zero downtime.
Upgrading workers is more involved because they handle data-processing tasks that can run from seconds to hours. Define the desired behavior for in-flight tasks.
By default, Kestra workers wait for all task runs to complete before shutting down during a migration. You can override this behavior if needed. Kestra [Helm charts](https://github.com/kestra-io/kestra/blob/develop/charts/kestra/values.yaml) provide a configuration of a `terminationGracePeriodSeconds` (set to 60 seconds by default) that allows you to define the amount of time you want to wait before force-killing the worker.
If the worker has no running tasks, or finishes them before the grace period, it shuts down immediately. If the pod cannot finish tasks before `terminationGracePeriodSeconds`, Kubernetes kills the pod, and those tasks are resubmitted to another worker.
If a worker exits unexpectedly, the executor detects it and resubmits unfinished task runs to a new worker. The same behavior applies when a pod is terminated at `terminationGracePeriodSeconds`.
## Where can I find migration guides
The [Migrations section](../../11.migration-guide/index.mdx) details deprecated features and explains how to migrate to the new behavior.
For all breaking changes, the migration guides are linked in the [release notes](https://github.com/kestra-io/kestra/releases).
## How to stay informed about new releases
You can get notified about new releases in the following ways:
1. Subscribe to notifications in the `#announcements` channel in the [Slack](/slack) community.
2. Follow us on [X (Twitter)](https://twitter.com/kestra_io)
3. Follow us on [LinkedIn](https://www.linkedin.com/company/kestra/)
4. Subscribe to the [Kestra newsletter](/blogs)
5. Subscribe to Release notifications on the [main GitHub repository](https://github.com/kestra-io/kestra), as shown in the image below:

## Database migrations
There are two types of database migrations: automatic and manual.
### Automatic database migration
Kestra uses [Flyway](https://flywaydb.org/) to automatically perform database migrations when the server starts. Flyway version-controls schema changes and stores the current version in the `flyway_schema_history` table. On startup, it compares the current version with the target and runs any required migrations — no manual intervention needed.
### Manual database migration
Sometimes a manual database migration is useful, especially when you have a large database and you want to perform the migration before upgrading Kestra to avoid a long downtime.
For example, when migrating from v0.12.0 to v0.13.0, all indexes are rebuilt due to multi-tenancy (`tenant_id` is added to most tables). With a large JDBC-backed database, this can take hours. In such cases, run `kestra sys database migrate` manually before starting Kestra.
This command should use the same configuration as configured on your Kestra instance. Depending on whether you deploy Kestra using Docker or Kubernetes, this command can be launched via a `docker exec` or a `kubectl exec` command.
There are two ways to initiate the manual database migration:
1. Keep Kestra running in an old version. Then, stop Kestra and launch the command on the new version.
2. Start Kestra on the new version with automatic schema migration disabled: `flyway.datasources.postgres.enabled=false` (if your database is not Postgres, replace `postgres` with your DB type). Then run: `kestra sys database migrate`.
Example: run the command via `docker exec`:
```bash
docker exec your_container_id bash ./kestra sys database migrate --help
```
Here is the output of that `--help` command:
```bash
Usage: kestra sys database migrate [-hVv] [--internal-log] [-c=]
[-l=] [-p=]
Force database schema migration.
Kestra uses Flyway to manage database schema evolution, this command will run
Flyway then exit.
-c, --config= Path to a configuration file, default: /root/.
kestra/config.yml)
-h, --help Show this help message and exit.
--internal-log Change also log level for internal log, default:
false)
-l, --log-level=
Change log level (values: TRACE, DEBUG, INFO, WARN,
ERROR; default: INFO)
-p, --plugins=
Path to plugins directory , default: ./plugins)
-v, --verbose Change log level. Multiple -v options increase the
verbosity.
-V, --version Print version information and exit.
```
## Getting help
If you have questions about the upgrade process:
- If you are a [Kestra Enterprise](/enterprise) customer, submit a [support ticket](https://support.kestra.io/).
- Or reach out [via Slack](/slack).
For further help, [contact us](/contact-us) for assistance with migration based on your environment and use case.
---
# Anonymous Usage Reporting in Kestra: Enable or Disable
URL: https://kestra.io/docs/administrator-guide/usage
> Learn about anonymous usage reporting in Kestra and how to configure or disable data collection.
Configure whether Kestra shares anonymous usage data and how frequently.
The `kestra.anonymous-usage-report.enabled` option controls the server-side report — decide whether to share anonymous data to help improve Kestra.
- `kestra.anonymous-usage-report.enabled`: (default true)
- `kestra.anonymous-usage-report.initial-delay`: (default 5m)
- `kestra.anonymous-usage-report.fixed-delay`: (default 1h)
Kestra collects anonymous usage through **two independent streams**, each controlled by its own option:
- `kestra.anonymous-usage-report.enabled`: the **server-side** report (host, plugins, flow, and execution data listed below).
- `kestra.ui-anonymous-usage-report.enabled`: (default true) the **UI (frontend)** usage report, sent separately from the browser to help us understand user experience in the interface.
To disable **all** anonymous usage reporting, set **both** options to `false`:
```yaml
kestra:
anonymous-usage-report:
enabled: false
ui-anonymous-usage-report:
enabled: false
```
Setting only `kestra.anonymous-usage-report.enabled` to `false` stops the server-side report but leaves UI reporting enabled (and vice versa).
The collected data can be found [here](https://github.com/kestra-io/kestra/tree/develop/core/src/main/java/io/kestra/core/models/collectors). We collect only **anonymous data** that allows us to understand how you use Kestra. The data collected includes:
- **host data:** CPU, RAM, OS, JVM, and a machine fingerprint.
- **plugins data:** plugins installed and their current versions.
- **flow data:** namespace count, flow count, the task type and the trigger type used.
- **execution data:** execution and task run counts for the last two days, with counts and durations grouped by status.
- **UI interaction:** data to help us understand user experience in the interface.
- **common data:** server type, version, time zone, environment, start time, and URL.
---
# Webserver URL, Reverse Proxy, and Forward Proxy Setup
URL: https://kestra.io/docs/administrator-guide/webserver-url
> Configure the Kestra webserver URL and proxy settings to ensure correct link generation and access behind reverse proxies.
Configure the URL of your Kestra webserver.
Some notification services require a URL configuration to add links from alert messages. Use a full URI with a trailing `/` (excluding `ui` or `api`).
```yaml
kestra:
url: https://www.my-host.com/kestra/
```
## Proxy configuration
In networking, a **forward proxy** acts on behalf of clients to control **outbound traffic**, while a **reverse proxy** acts on behalf of servers to control **inbound traffic** and may also provide features such as load balancing and SSL encryption.
A forward proxy serves as an intermediary for requests from clients seeking resources from other servers (such as the Kestra API for retrieving blueprints and plugin documentation), while a reverse proxy sits in front of one or more web servers, intercepting client requests before they reach the server.
### Forward proxy configuration
In a forward proxy, the client connects to the proxy server, requesting some service (such as Kestra API) available from a different server.
To set up a proxy in your Kestra installation, adjust the `micronaut.http.services.api` configuration to include a proxy address, username, and password. This will allow you to make requests to the Kestra API through the proxy to fetch data for the Kestra UI, such as Blueprints. Here is how you can adjust your `config.yml` file to include the necessary configuration:
```yaml
micronaut:
http:
services:
api:
url: https://api.kestra.io
proxy-type: http
proxy-address: my.company.proxy.address:port
proxy-username: "username"
proxy-password: "password"
follow-redirects: true
```
See the [Micronaut HttpClient Configuration](https://docs.micronaut.io/latest/guide/configurationreference.html#io.micronaut.http.client.DefaultHttpClientConfiguration) for more details on configuring `DefaultHttpClientConfiguration` in your `config.yml` file.
Another way to authenticate is by providing `micronaut.http.client.proxy-authorization: Basic ` and `micronaut.http.services.*.proxy-authorization: Basic `, which prevents the password from being displayed in plain text in the config file.
### Reverse proxy configuration
Reverse proxies hide the server’s identity from clients and may perform tasks such as load balancing, authentication, decryption, and caching. A reverse proxy acts on behalf of the server, taking requests from the external network, and directing them to the internal server(s) that can fulfill those requests.
To display executions in real-time when hosting Kestra behind a reverse proxy, enable [Server-sent events (SSE)](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events).
On some reverse proxies, such as Nginx, you need to disable buffering to enable real-time updates.
Here is a working configuration:
```bash
location / {
proxy_pass http://localhost:;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 600s;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Protocol $scheme;
# Needed for SSE
proxy_buffering off;
proxy_cache off;
}
```
To access Kestra via a separate context path, add the following to your Kestra startup configuration (for example, to serve the UI at `mycompany.com/kestra`):
```yaml
micronaut:
server:
context-path: "/kestra"
```
Then, modify your above nginx configuration to the following
```bash
server {
listen 80;
server_name mycompany.com;
location /kestra {
proxy_pass http://:/kestra;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 600s;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Protocol $scheme;
# Needed for SSE
proxy_buffering off;
proxy_cache off;
}
}
```
---
# AI in Kestra: Copilot, Coding Agents, MCP & AI Agents
URL: https://kestra.io/docs/ai-tools
> Compare every way to use AI with Kestra — AI Copilot in the UI, your own coding agent via MCP and Agent Skills, and AI Agents running inside your flows.
import ChildCard from "~/components/docs/ChildCard.astro"
Compare every way to use AI with Kestra and pick the one that fits how you work.
AI shows up in Kestra in two unrelated ways, and it helps to keep them apart:
- **AI that helps you build and operate Kestra** — writing flow YAML, fixing a failed execution, deploying to a namespace. The AI is a way to drive Kestra faster; it never runs as part of your workflow.
- **AI that runs inside your flows** — a task calls a model, an agent decides what to do next, a flow answers questions over your documents. Here the AI is part of what you orchestrate.
None of this is required. Kestra is an orchestrator, and every flow you can generate with AI you can also write by hand in YAML or build in the No-Code editor.
## Build and operate Kestra with AI
Start from where you already work:
| Where you work | What to use | What it gives you |
| --- | --- | --- |
| The Kestra UI | [AI Copilot](./01.ai-copilot/index.md) | Generate and refine flows from a prompt in the editor, and "Fix with AI" on a failed execution |
| Your terminal or editor (Claude Code, Cursor, Codex, Windsurf) | [MCP server](./02.kestra-mcp-resources/index.md) + [Agent Skills](./04.agent-skills/index.md) | Your existing coding agent writes, validates, and deploys flows alongside the rest of your codebase |
| Scripts and CI | [`kestractl`](../kestra-cli/kestractl/index.md) | Deterministic flow, execution, and namespace operations with no AI in the loop |
### AI Copilot in the Kestra UI
[AI Copilot](./01.ai-copilot/index.md) is built into the Kestra editor. Describe what you want — for example, _"make a REST API call to https://kestra.io/api/mock and allow failure"_ — and Copilot generates the flow YAML for you to review, accept, or edit. It also modifies existing flows incrementally, touching only the part you asked about, and powers **Fix with AI** on failed tasks.
Copilot is the fastest path when you are already in the UI and want the change applied to the flow in front of you.
### Your own coding agent
If your flows live in Git next to your application code, you may prefer to stay in Claude Code, Cursor, Codex, or Windsurf. Three separate pieces make that work, and each covers a different gap:
- **[MCP server](./02.kestra-mcp-resources/index.md) — context.** A hosted [Model Context Protocol](https://modelcontextprotocol.io/) endpoint that gives your agent live access to task schemas for all 1,400+ plugins, the blueprints library, and documentation search. Without it, an agent invents plugin properties from stale training data.
- **[Agent Skills](./04.agent-skills/index.md) — know-how.** `SKILL.md` files that teach your agent how to author valid flows (`kestra-flow`), harden them for production (`kestra-flow-hardening`), operate an environment (`kestra-ops`), and port Airflow DAGs (`migrate-airflow-kestra`). Skills carry the guardrails: no invented task types, no hardcoded secrets, validate before deploy.
- **[`kestractl`](../kestra-cli/kestractl/index.md) — hands.** The Kestra CLI the agent actually calls to validate, deploy, and run flows against a real instance. The `kestra-ops` skill is a wrapper around these commands, so an agent without `kestractl` can write flows but cannot deploy them.
#### Set it up
Connect the MCP server. For Claude Code:
```bash
claude mcp add kestra --transport http https://api.kestra.io/v1/mcp
```
Any MCP-compatible tool can connect to `https://api.kestra.io/v1/mcp` over HTTP — see [MCP server](./02.kestra-mcp-resources/index.md) for Cursor and other configurations.
Install the skills. [skills.sh](https://skills.sh) detects your agent and writes the files to the right location:
```bash
npx skills add kestra-io/agent-skills
```
Install `kestractl` and point it at your instance:
```bash
curl -fsSL https://raw.githubusercontent.com/kestra-io/kestractl/main/install-scripts/install.sh | bash
kestractl config add default http://localhost:8080 main --username YOUR_USERNAME --password YOUR_PASSWORD --default
```
See [`kestractl`](../kestra-cli/kestractl/index.md) for Enterprise token auth and the full command reference.
#### What it looks like end to end
With all three in place, a single prompt covers authoring and deployment:
```plaintext
Use kestra-flow to write a flow in namespace company.data that fetches
https://api.example.com/metrics every 30 minutes and stores the response in
the KV store under the key last_metrics. Validate it with kestractl, then
deploy it to the dev instance.
```
The agent queries the MCP server for the correct plugin schema, follows the skill's guardrails (no hardcoded credentials, valid YAML structure), runs `kestractl flow validate`, and deploys with `kestractl flow create`.
### Expose flows as AI tools
[MCP Server](./03.mcp-server/index.md) lets you go the other direction: instead of an agent driving Kestra, Kestra flows become callable tools for any MCP-compatible AI agent. Add an `McpToolTrigger` to any flow and it is automatically registered as a named tool. AI agents such as Claude Desktop, Claude Code, and Cursor can then discover and invoke your flows directly, with inputs and outputs mapped to a JSON schema.
## Run AI inside your flows
- **[AI Agents](./05.ai-agents/index.md)**: autonomous orchestration where an LLM decides which actions to take at runtime rather than following a fixed sequence.
- **[AI Workflows](./06.ai-workflows/index.md)**: pre-built flow templates for common AI patterns such as summarization, classification, and structured extraction.
- **[RAG Workflows](./07.ai-rag-workflows/index.md)**: retrieval-augmented generation pipelines that connect your documents to a language model.
---
# Agent Skills – Operate Kestra from AI Coding Agents
URL: https://kestra.io/docs/ai-tools/agent-skills
> Give AI coding agents like Claude Code, Cursor, and Windsurf structured knowledge to generate Kestra flows and operate Kestra environments using kestractl.
Give AI coding agents structured knowledge to generate Kestra flows and operate Kestra environments.
## What are Agent Skills
Agent Skills are structured knowledge files (`SKILL.md`) that teach external AI coding agents how to work with Kestra. They provide the context, commands, and guardrails an agent needs to generate valid flow YAML or operate a Kestra environment via the CLI.
Unlike [AI Copilot](../01.ai-copilot/index.md), which works inside the Kestra UI, Agent Skills bring Kestra expertise to the tools you already use in your editor or terminal: Claude Code, Cursor, Windsurf, OpenAI Codex, and others.
Unlike [AI Agents](../05.ai-agents/index.md), which are autonomous tasks running inside Kestra flows, Agent Skills equip your external coding agent with Kestra-specific knowledge so it can help you build and operate flows from your development environment.
Agent Skills follow an emerging standard for giving AI tools domain-specific knowledge. Learn more at [agentskills.io](https://agentskills.io/home), the community hub for agent skills across tools and domains.
## Available Skills
Kestra provides multiple skills in the [kestra-io/agent-skills](https://github.com/kestra-io/agent-skills) repository.
### kestra-flow
Generate, modify, or debug Kestra Flow YAML grounded in the live flow schema (the same approach used by Kestra's AI Copilot).
**Use when:**
- Generating a new flow from a description
- Modifying or extending an existing flow
- Debugging invalid YAML or incorrect task/trigger references
**Covers:**
- Fetching and validating against the live flow schema from `https://api.kestra.io/v1/plugins/schemas/flow`
- Schema-validated task and trigger generation
- Partial modifications that touch only the relevant part of a flow
- Guardrails: no invented types, no hardcoded secrets, correct looping and trigger patterns
**Example prompt:**
```plaintext
Use kestra-flow to write a flow that polls a REST API every 30 minutes and stores the result in KV store.
```
### kestra-flow-hardening
Audit existing flows and add production-hardening controls, the consulting counterpart to `kestra-flow`. It is **audit-first**: it produces a severity-ranked findings report (risk, caveat, and proposed fix per finding), then applies only the edits you confirm.
**Use when:**
- Hardening one or more flows for production
- Auditing flows for resilience, idempotency, and guardrail gaps
- Adding retries, timeouts, error handling, concurrency limits, SLAs, or idempotency guards
**Covers:**
- Severity-ranked audit report (Critical / High / Medium / Low) with risk, caveat, and proposed fix per finding
- Idempotency judgment — never recommends a blind retry on a non-idempotent write; flags the dedup-guard vs. retry-if-safe branches
- Proportional auditing calibrated by flow signals (triggers, side-effects, namespace environment); "already sound" is a valid result
- Surgical, schema-validated edits applied on confirmation, inline and structure-preserving
- Version- and edition-aware (OSS / EE), with EE-only patterns labeled and given an OSS fallback
**Example prompt:**
```plaintext
Use kestra-flow-hardening to audit the flows in ./flows and add retries, timeouts, and failure alerting where they're missing.
```
### kestra-ops
Operate Kestra using `kestractl` for flow, execution, namespace, and namespace-file operations.
**Use when:**
- Validating or deploying flows
- Triggering executions and checking status
- Managing namespaces and namespace files (`nsfiles`)
- Configuring or switching CLI contexts
**Covers:**
- Context and auth setup (`config add`, `config use`, `config show`)
- Flow operations: list, get, validate, deploy
- Execution monitoring: run with `--wait`, get status
- Namespace file management: list, get, upload, delete
- Production guardrails: validate before deploy, confirm destructive actions, avoid exposing credentials
**Example prompt:**
```plaintext
Use kestra-ops to validate and deploy all flows in ./flows to prod.namespace with fail-fast enabled.
```
### migrate-airflow-kestra
Migrate an Apache Airflow DAG to a production-ready Kestra flow, preserving task dependencies and parallelism.
**Use when:**
- Converting an Airflow DAG (`.py`) to a Kestra flow YAML
- Translating `@task`-decorated functions or operators into Kestra tasks
- Preserving Airflow parallel execution (fan-out/fan-in) in Kestra
**Covers:**
- Reading and analysing the Airflow DAG structure, tasks, and dependencies
- Fetching the live Kestra schema from `https://api.kestra.io/v1/plugins/schemas/flow`
- Extracting Python business logic into namespace files
- Generating schema-validated Kestra flow YAML with correct task ordering and parallelism
- Mapping Airflow XCom data passing to Kestra `outputFiles`/`inputFiles`
**Example prompt:**
```plaintext
Use migrate-airflow-kestra to migrate dags/ingest_pipeline.py from Airflow to Kestra, output to kestra/.
```
## Prerequisites
- **AI coding agent**: Claude Code, Cursor, Windsurf, OpenAI Codex, OpenCode, or any agent that supports skill files
- **For kestra-flow, kestra-flow-hardening, and migrate-airflow-kestra**: `curl` and network access to `https://api.kestra.io`
- **For kestra-ops**: [`kestractl`](../../kestra-cli/kestractl/index.md) installed with valid credentials
## Setup
The easiest way to install Kestra agent skills is with [skills.sh](https://skills.sh); it auto-detects your AI coding agent and places the skill files in the right location:
```bash
npx skills add kestra-io/agent-skills
```
This works with Claude Code, Cursor, Windsurf, OpenAI Codex, and other agents that support skill files. The CLI detects which agent you’re using and installs the `SKILL.md` files into the correct directory (e.g. `.claude/skills/` for Claude Code, `.cursor/rules/` for Cursor).
### Manual installation
You can also manually download skill files from the [kestra-io/agent-skills](https://github.com/kestra-io/agent-skills) repository. Each skill is a `SKILL.md` file under `skills//`.
For example, to add the `kestra-ops` skill to Claude Code:
```bash
mkdir -p .claude/skills/kestra-ops
curl -sL https://raw.githubusercontent.com/kestra-io/agent-skills/main/skills/kestra-ops/SKILL.md \
-o .claude/skills/kestra-ops/SKILL.md
```
Repeat for any other skill you need (e.g. `kestra-flow`). Adjust the target directory for your agent: `.cursor/rules/` for Cursor, `.agents/skills/` for OpenAI Codex, etc. Some skills ship supporting files alongside `SKILL.md` (for example, `kestra-flow-hardening` includes `references/hardening-patterns.md`); copy the whole skill directory so those files are available. Using `skills.sh` handles this automatically.
## Example Workflows
### Generate a flow with kestra-flow
Ask your agent to create a flow that polls an API on a schedule and persists the result:
```plaintext
Use kestra-flow to write a flow in namespace company.data that fetches
https://api.example.com/metrics every 30 minutes and stores the response
in KV store under the key "latest_metrics".
```
The agent will fetch the live schema, generate valid YAML with a `Schedule` trigger and `io.kestra.plugin.core.kv.Set` task, and output ready-to-deploy flow code.
### Harden a flow for production with kestra-flow-hardening
Ask your agent to audit an existing flow and add resilience controls before promoting it to production:
```plaintext
Use kestra-flow-hardening to audit ./flows/extract.yaml and add retries,
timeouts, and failure alerting where they're missing.
```
The agent will produce a severity-ranked findings report (for example, flagging missing retries on external HTTP calls (Medium), a silent fallback masking a geocoding failure (data-correctness risk), and no failure alerting on a scheduled flow (High)), then apply only the fixes you confirm, validating each edit against the live schema.
### Validate and deploy with kestra-ops
Ask your agent to validate local flow files and deploy them:
```plaintext
Use kestra-ops to validate all flows in ./flows, then deploy them to
prod.pipelines namespace with --override and --fail-fast.
```
The agent will run `kestractl flows validate ./flows/`, confirm results, and then run `kestractl flows deploy` with the requested flags.
### Run a flow and report results with kestra-ops
Ask your agent to trigger an execution and summarize the outcome:
```plaintext
Use kestra-ops to run nightly-refresh in analytics.jobs namespace,
wait for completion, and report the execution status.
```
The agent will run `kestractl executions run analytics.jobs nightly-refresh --wait`, then summarize the execution result.
## Creating Custom Skills
You can create your own skills following the same `SKILL.md` format. Each skill file should include:
- **Frontmatter** with `name`, `description`, and `compatibility`
- **When to use** — trigger conditions for the skill
- **Required inputs** — what context the agent needs
- **Workflow** — step-by-step instructions
- **Guardrails** — safety rules and constraints
- **Example prompts** — realistic usage examples
See the [contributing guidelines](https://github.com/kestra-io/agent-skills) in the repository for more details.
---
# AI Agents in Kestra – Autonomous Orchestration
URL: https://kestra.io/docs/ai-tools/ai-agents
> Build autonomous AI agents in Kestra for LLM-powered orchestration. Create agents that think, remember, and use tools like web search for complex workflows.
Launch autonomous processes with an LLM, memory, and tools.
## Build autonomous AI agents in Kestra
Add autonomous AI-driven tasks to flows that can think, remember, and dynamically orchestrate tools and tasks.
An AI Agent is an autonomous system that uses a Large Language Model (LLM). Each run combines a **system message** and a **prompt**. The system message defines the agent's role and behavior, while the prompt carries the actual user input for that execution. Together, they guide the agent's response.
With AI Agents, workflows are no longer limited to a predefined sequence of tasks. An AI Agent task launches an autonomous process with the help of an LLM, memory, and tools such as web search, task execution, and flow calling, and can dynamically decide which actions to take and in what order. Unlike traditional flows, an AI Agent can loop tasks until a condition is met, adapt to new information, and orchestrate complex multi-step objectives on its own. This enables agentic orchestration patterns in Kestra, where agents can operate independently or collaborate in multi-agent systems, all while remaining fully observable and manageable in code.
To start using this feature, you can add an [**AI Agent**](/plugins/plugin-ai/agent) task to your flow. The AI Agent will then use the tools you provide to achieve its goal, leveraging capabilities such as web search, task execution, and flow calling. Thanks to memory, your AI Agent can remember information across executions to provide context for future tasks and subsequent prompts.
## AI Agent flow example
The following flow summarizes arbitrary text with controllable length and language. Each component of the flow is broken down below.
```yaml
id: simple_summarizer_agent
namespace: company.ai
inputs:
- id: summary_length
displayName: Summary Length
type: SELECT
defaults: medium
values:
- short
- medium
- long
- id: language
displayName: Language ISO code
type: SELECT
defaults: en
values:
- en
- fr
- de
- es
- it
- ru
- ja
- id: text
type: STRING
displayName: Text to summarize
defaults: |
Kestra is an open-source orchestration platform that:
- Allows you to define workflows declaratively in YAML
- Allows non-developers to automate tasks with a no-code interface
- Keeps everything versioned and governed, so it stays secure and auditable
- Extends easily for custom use cases through plugins and custom scripts.
Kestra follows a "start simple and grow as needed" philosophy. You can schedule a basic workflow in a few minutes, then later add Python scripts, Docker containers, or complicated branching logic if the situation calls for it.
tasks:
- id: multilingual_agent
type: io.kestra.plugin.ai.agent.AIAgent
provider:
type: io.kestra.plugin.ai.provider.GoogleGemini
modelName: gemini-3.5-flash-lite
apiKey: "{{ secret('GEMINI_API_KEY') }}"
configuration:
logRequests: true
logResponses: true
responseFormat:
type: TEXT
systemMessage: |
You are a precise technical assistant.
Produce a {{ inputs.summary_length }} summary in {{ inputs.language }}.
Keep it factual, remove fluff, and avoid marketing language.
If the input is empty or non-text, return a one-sentence explanation.
Output format:
- 1-2 sentences for 'short'
- 2-5 sentences for 'medium'
- Up to 5 paragraphs for 'long'
prompt: |
Summarize the following content: {{ inputs.text }}
- id: english_brevity
type: io.kestra.plugin.ai.agent.AIAgent
provider:
type: io.kestra.plugin.ai.provider.GoogleGemini
modelName: gemini-3.5-flash-lite
apiKey: "{{ secret('GEMINI_API_KEY') }}"
configuration:
logRequests: true
logResponses: true
responseFormat:
type: TEXT
prompt: Generate exactly 1 sentence English summary of "{{ outputs.multilingual_agent.textOutput }}"
```
### Inputs
The flow uses three inputs (`summary_length`, `language`, and `text`) to control the summary length, language, and source text.
All inputs have a default value. Any of them can be referenced in downstream tasks with [expressions](../../expressions/index.mdx). When executing the flow, any input can be selected or modified from its default.

The example selects `short` for the summary length and German (`de`) for the summary language.
### Tasks
The flow has two tasks using the [AI Agent plugin](/plugins/plugin-ai/agent): `multilingual_agent` and `english_brevity`. The first task, `multilingual_agent`, uses the `systemMessage` property to set the agent's role and behavior. The system message references the input selections for summary length and language, and defines what to output for each length option.
The `prompt` property instructs the agent to summarize the input text. For a short summary, `multilingual_agent` produces a 1–2 sentence German summary of Kestra.

The `english_brevity` task only needs a `prompt` because the `systemMessage` is inherited from plugin defaults. Whether the original output is in a different language or needs shortening, `english_brevity` produces a one-sentence English summary.

These outputs can then be passed on as notifications or system messages to external tools or subflows within Kestra. Other useful outputs include `tokenUsage` to compare different providers for the same tasks. At runtime, Kestra also emits counter metrics — `ai.agent.tool.calls`, `ai.provider.calls`, and `ai.embedding.store.calls` — tagged by class name, which you can scrape with Prometheus or export via OpenTelemetry to monitor AI task usage. For more examples and details about properties, outputs, and definitions, refer to the AI [Agent plugin documentation](/plugins/plugin-ai/agent).
### Centralizing provider configuration
Each task using the AI Agent requires the `provider` property. To avoid repeating it on every task, use a [Policy](../../07.enterprise/02.governance/policies/index.md) with an `Add` rule to inject the `provider` block into all `AIAgent` tasks across a namespace — this is an Enterprise Edition and Cloud feature. For your provider API key, store it as a [Secret](../../06.concepts/04.secret/index.md) and reference it with `{{ secret('...') }}`.
## Agent tools
The AI Agent can be extended with **tools** — capabilities the LLM can choose to invoke at runtime to complete its task. Tools are listed under the `tools` property of an `AIAgent` task.
### Skills
The [**Skill**](/plugins/plugin-ai/tool/skill) tool lets you attach structured instructions to an agent that it can activate on demand. Rather than including all instructions in the system message, skills let you define discrete, reusable knowledge blocks — each with a name, a description the LLM uses to decide when to activate it, and the actual instruction content.
This is useful when an agent has multiple possible modes of operation, such as translating text, reviewing code, or formatting data, where you want the LLM to select and apply the right instructions based on context rather than always receiving all instructions at once.
Each skill requires:
- `name` — a unique identifier for the skill
- `description` — explains to the LLM when to activate the skill
- `content` or `contentUri` — the instruction content, either inline or loaded from Kestra internal storage
#### Inline skill content
The simplest way to define a skill is with inline `content`:
```yaml
id: agent_with_skills
namespace: company.ai
tasks:
- id: agent
type: io.kestra.plugin.ai.agent.AIAgent
prompt: Translate the following text to French - "Hello, how are you today?"
provider:
type: io.kestra.plugin.ai.provider.GoogleGemini
modelName: gemini-3.5-flash-lite
apiKey: "{{ secret('GEMINI_API_KEY') }}"
tools:
- type: io.kestra.plugin.ai.tool.Skill
skills:
- name: translation_expert
description: Expert translator for multiple languages
content: |
You are an expert translator. When translating text:
1. Preserve the original meaning and tone
2. Use natural phrasing in the target language
3. Keep proper nouns unchanged
```
#### Loading skill content from storage
For longer or reusable instructions, store the skill content as a file in Kestra internal storage and reference it with `contentUri`. This is especially useful when skill content is generated or updated by an earlier task in the same flow:
```yaml
id: agent_with_skill_from_storage
namespace: company.ai
tasks:
- id: write_instructions
type: io.kestra.plugin.core.storage.Write
content: |
You are a senior code reviewer. When reviewing code:
1. Check for security vulnerabilities
2. Ensure proper error handling
3. Verify naming conventions are followed
4. Flag any code duplication
- id: agent
type: io.kestra.plugin.ai.agent.AIAgent
prompt: "Review this Python function - 'def add(a, b): return a + b'"
provider:
type: io.kestra.plugin.ai.provider.GoogleGemini
modelName: gemini-3.5-flash-lite
apiKey: "{{ secret('GEMINI_API_KEY') }}"
tools:
- type: io.kestra.plugin.ai.tool.Skill
skills:
- name: code_review_expert
description: Expert code reviewer with strict guidelines
contentUri: "{{ outputs.write_instructions.uri }}"
```
A single `Skill` tool can define multiple skills. Each skill must have a unique name. `content` and `contentUri` are mutually exclusive — exactly one must be set per skill. For more details on all available properties, refer to the [Skill plugin documentation](/plugins/plugin-ai/tool/skill).
### Kestra-native tools
- [**KestraFlow**](/plugins/plugin-ai/tool/kestraflow) — triggers a Kestra flow as a tool, either with a predefined namespace and flow ID or dynamically based on the agent's prompt.
- [**KestraTask**](/plugins/plugin-ai/tool/kestratask) — exposes one or more Kestra runnable tasks as tools, letting the agent supply values for properties left unset.
### Web search
- [**TavilyWebSearch**](/plugins/plugin-ai/tool/tavilywebsearch) — gives the agent access to live web results via the Tavily search API.
- [**GoogleCustomWebSearch**](/plugins/plugin-ai/tool/googlecustomwebsearch) — gives the agent access to live web results via a Google Custom Search Engine.
### Code execution
- [**CodeExecution**](/plugins/plugin-ai/tool/codeexecution) — lets the agent write and run JavaScript snippets in a Judge0 sandbox (via RapidAPI).
### Nested agents
- [**AIAgent**](/plugins/plugin-ai/tool/aiagent) — wraps another AI agent as a callable tool so a parent agent can delegate sub-tasks to a specialized child agent.
- [**A2AClient**](/plugins/plugin-ai/tool/a2aclient) — forwards prompts to a remote AI agent over the Agent-to-Agent (A2A) protocol and returns its response.
### MCP clients
Kestra supports MCP in two directions. These clients cover the **Kestra-as-client** direction: your flow calls tools on an *external* MCP server. For the opposite direction — exposing your flows *as* MCP tools for external AI agents to call — see [MCP Server](../03.mcp-server/index.md) and the [McpToolTrigger](../../05.workflow-components/07.triggers/06.mcp-tool-trigger/index.md).
Connect the agent to any [Model Context Protocol (MCP)](https://modelcontextprotocol.io) server to expose its tools:
- [**DockerMcpClient**](/plugins/plugin-ai/tool/dockermcpclient) — runs an MCP server inside a Docker container.
- [**SseMcpClient**](/plugins/plugin-ai/tool/ssemcpclient) — connects to a remote MCP server over Server-Sent Events (SSE).
- [**StdioMcpClient**](/plugins/plugin-ai/tool/stdiomcpclient) — spawns a local MCP server process and communicates over stdio.
- [**StreamableHttpMcpClient**](/plugins/plugin-ai/tool/streamablehttpmcpclient) — connects to an MCP server over HTTP streaming.
The [Kestra Python MCP server](https://github.com/kestra-io/mcp-server-python) is an example of an external MCP server you can connect to from a Kestra AI Agent task using one of the clients above.
## Execution details
When you open an execution in the topology view, the details panel for `AIAgent`, `ChatCompletion`, and `rag.ChatCompletion` tasks shows the LLM configuration and post-execution context for each call.
**Pre-execution:**
- Model name and provider
- System prompt (collapsible)
- Tools available to the agent
- RAG retriever and embedding store configuration (when applicable)
**Post-execution:**
| Signal | Description |
|---|---|
| LLM response | The final text or JSON output rendered inline |
| Tool call timeline | Each tool invocation in order: name, arguments, and result |
| Token usage | Input tokens, output tokens, total, and an estimated cost by provider and model |
| Reasoning chain | Intermediate responses and extended thinking steps when present |
| RAG sources | Retrieved chunks ranked by similarity score, showing which context grounded the answer |
| Finish reason | Why the model stopped: natural stop, max tokens reached, or a guardrail trigger |
---
# AI Copilot in Kestra – Generate and Edit Flows
URL: https://kestra.io/docs/ai-tools/ai-copilot
> Use Kestra AI Copilot to generate and edit flows with natural language prompts. Get AI-assisted suggestions for tasks, triggers, and configurations.
Build and modify flows, ask questions about Kestra, and get AI-driven plans — all from a persistent chat sidebar.
The AI Copilot opens as a right-side panel from the **AI** button in the top toolbar. Click **New chat +** to start a conversation, or use **Recents** to return to a previous one. Conversations persist for the browser session. You can type prompts or click the microphone button to dictate with speech-to-text.
## Modes
The Copilot has three modes, selectable from the dropdown at the bottom left of the chat panel:
| Mode | What it does |
|---|---|
| **Ask** | Answers questions about Kestra using docs-grounded responses. Ask about features, configuration, plugin options, or get help diagnosing a failed execution. |
| **Edit** | Generates and iteratively edits flow YAML. Describe what you want to build; the Copilot drafts the change and asks for confirmation before applying it. |
| **Plan** | Proposes a step-by-step plan for a complex task. Each step requires individual approval before the Copilot executes it. Rejecting any step cancels the plan. |
Switch modes at any point in a conversation — the Copilot carries the conversation history across mode switches.
| If you want to… | Use |
|---|---|
| Build, modify, or refactor a flow | Edit |
| Diagnose a failed execution | Ask |
| Ask about Kestra features, plugins, or configuration | Ask |
| Complete a multi-step task with approval at each step | Plan |
## Context
The Copilot automatically attaches the resource you are viewing as context when you open the panel. Attached resources appear as dismissible pills above the input. You can remove any pill to narrow the Copilot’s focus, and the transcript records each add and remove so you always know what the agent is looking at.
Resources that can be attached as context:
- Flow
- Namespace
- Execution
- Dashboard
- App
- Test suite
- Blueprint
- Plugin
Copilot also reads Namespace metadata — Policies, Variables, Secrets, and Key-Value pairs — so prompts like "Create a task that integrates with MongoDB" can reuse your configured credentials and variables without extra hints.
## Confirmation
In **Edit** and **Plan** modes, actions that modify resources (creating or updating a flow, restarting an execution) require explicit confirmation before the Copilot executes them. A confirmation prompt appears in the chat with an optional field to steer the next step. Approving executes the action; rejecting resumes the conversation in Edit mode or cancels the current plan in Plan mode.
## Edit mode
Edit mode generates and iteratively refines declarative flow YAML. Describe what you want to build; the Copilot searches available plugins, validates the generated YAML, and proposes the change for your approval. Once accepted, you can keep iterating — adding triggers, adjusting tasks, or refactoring a section — without the Copilot touching unrelated parts of the flow.
Edit mode is available anywhere you build in Kestra — Flows, Apps, Unit tests, and Dashboards.
## Usage limits
When no custom provider is configured, Kestra uses a built-in AI service with a daily generation limit per instance. The UI shows how many generations you have left, and the limit resets daily at midnight UTC.
To remove the limit, configure your own LLM provider in the `kestra.ai.providers` block. See [Configuration](#configuration) below.
## Configuration
To add Copilot to your flow editor, add the following to your [Enterprise and Advanced configuration](../../configuration/06.enterprise-and-advanced/index.md). The `providers` array lets you register multiple LLMs and pick a default (`is-default: true`):
```yaml
kestra:
ai:
enabled: true # set to false to disable AI Copilot entirely
providers:
- id: gemini
display-name: Gemini - Private
type: gemini
configuration:
model-name: gemini-2.5-flash
api-key: YOUR_GEMINI_API_KEY
- id: gpt
display-name: OpenAI
type: openai
is-default: true
configuration:
model-name: gpt-4o
api-key: YOUR_OPENAI_API_KEY
```
:::alert{type="info"}
Legacy single-provider configs (`kestra.ai.type` + provider block) still work, but the `providers` array lets you register multiple providers and choose a default (`is-default: true`).
:::
### Disabling AI Copilot
To fully disable the AI Copilot — including the built-in fallback to the `api.kestra.io` service — set `kestra.ai.enabled` to `false`:
```yaml
kestra:
ai:
enabled: false
```
When disabled, the Copilot UI will not appear and all AI endpoints will be deactivated. The property defaults to `true`.
### Multiple providers
When multiple providers are configured, users can switch models from a dropdown in the Copilot UI instead of relying only on the default.
For a full reference of every configuration property — including generation parameters, extended reasoning, mTLS, custom headers, and per-provider availability — see the [AI Copilot configuration reference](../../configuration/06.enterprise-and-advanced/index.md#ai-copilot).
- `timeout`: Specifies the maximum duration to wait for an AI model API request to complete before timing out. ISO 8601 duration format (Java Duration): `PT30S` = 30 seconds. You can set it per provider to enforce strict SLAs.
:::alert{type="info"}
Enterprise Edition includes an [RBAC permission](../../07.enterprise/03.auth/rbac/index.md) that lets administrators allow or disallow Copilot usage per role at tenant or namespace scope.
:::
:::alert{type="info"}
The open-source version supports only Google Gemini models. Enterprise Edition users can configure any LLM provider, including Amazon Bedrock, Anthropic, Azure OpenAI, DeepSeek, Google Gemini, Google Vertex AI, Mistral, OpenAI, OpenRouter, and all open-source models supported by Ollama. See [Enterprise Edition Copilot configurations](#enterprise-edition-copilot-configurations) below. If you use a different provider, [reach out to us](https://kestra.io/demo) and we'll add it.
:::
## Build flows with Edit mode
Open the Copilot sidebar, select **Edit** mode, and describe what you want to build. The Copilot searches for the right plugins, generates validated YAML, and proposes the change for your approval. The flow is marked **Valid** before the proposal is shown — you will not be asked to apply broken YAML.
**Step 1: Build the initial flow**
```txt
Create a flow that downloads a CSV from S3 and loads it into Postgres
```

The Copilot searches for the S3 and Postgres plugins, authors the flow with secrets referenced via `{{ secret('...') }}`, and presents the proposal. Select **Apply** to write it to the editor, or **Open in editor** to review the diff before accepting.
**Step 2: Add error handling**
```txt
Add error handling that sends a Slack alert if any task fails
```

The Copilot updates only the `errors` block — the existing `download_from_s3` and `load_to_postgres` tasks are untouched. The Copilot explains what it changed before presenting the proposal.
**Step 3: Parameterize hardcoded values**
```txt
Parameterize the S3 bucket name and Postgres table as flow inputs
```

The Copilot reads the current flow (note the `read-flow` step in the sidebar), adds an `inputs` block with `s3_bucket`, `s3_key`, and `postgres_table`, and rewires the hardcoded values to `{{ inputs.* }}` references throughout the flow. The flow and namespace context pills are attached automatically while working inside the editor.
Each accepted change is saved as a revision. You can track the full edit history from the **Revisions** tab, or use [Git sync](../../version-control-cicd/04.git/index.md) to push revisions to your repository.
## Ask mode
Use Ask mode to ask natural-language questions about Kestra without generating any code. Ask mode grounds its answers in the Kestra documentation and can analyze execution failures by reading the execution logs directly.
**Diagnosing a failed execution**
When you open the Copilot from a failed execution view, the execution is automatically attached as context. Ask "Why did this execution fail?" and the Copilot reads the execution metadata and logs, then gives a structured answer: which task failed, the root-cause error, and what to fix.

In the example above, the Copilot ran `read-execution` and `read-execution-logs`, identified that the `download_from_s3` task failed due to `SecretNotFoundException`, and listed exactly which secrets — `AWS_ACCESS_KEY_ID`, `AWS_SECRET_KEY_ID`, `POSTGRES_USERNAME`, `POSTGRES_PASSWORD`, `POSTGRES_HOST`, and `SLACK_WEBHOOK` — need to be configured before running the flow again.
Other example questions:
- "What is the difference between a Worker Group and a Task Runner?"
- "How do I configure namespace-level Policies?"
- "What secrets and variables are available in this namespace?" (with a namespace attached as context)
Ask mode is also a useful starting point before switching to Edit or Plan — use it to understand your options, then switch modes to act on the answer.
## Plan mode
Use Plan mode when a task involves multiple ordered steps that you want to approve individually before the Copilot executes them. Plan mode presents the full plan upfront as a numbered list, then waits for your confirmation before starting. You can approve and execute the plan, or reply to revise it before anything runs.

In the example above, the prompt "Build an ELT pipeline: extract from Salesforce, transform with dbt on DuckDB, load into Snowflake, and send a Slack summary on completion or failure" produced a four-step plan. The `company.team` namespace pill is attached, so the Copilot can reference available plugins and credentials in that namespace.
Rejecting a step cancels the remaining steps. If you want to adjust the plan before it runs, use **Reply to revise** to send feedback and get a revised plan.
Use Plan mode for tasks like:
- Building a multi-stage pipeline where you want to review the structure before any YAML is generated
- Migrating flows from one pattern to another (for example, from `ForEach` to the Loop task) across multiple steps
- Setting up namespaces, variables, and RBAC in sequence for a new team
## Fix with AI
From the Logs and Gantt views, click the three-dot menu on any failed task and select **Fix with AI**. The flow editor opens with the Copilot pre-loaded with the error context in Edit mode, ready to propose a fix.
## Starter prompts
:::collapse{title="Edit mode prompts"}
```markdown
- Create a flow that runs a dbt build command on DuckDB
- Create a flow cloning https://github.com/kestra-io/dbt-example Git repository from a main branch, then add a dbt CLI task using DuckDB backend that will run dbt build command for that cloned repository using my_dbt_project profile and dev target. The dbt project is located in the root directory so no dbt project needs to be configured.
- Create a flow that sends a POST request to https://dummyjson.com/products/add
- Send a POST request to https://dummyjson.com/products/add
- Write a Python script that sends a POST request to https://dummyjson.com/products/add
- Write a Node.js script that sends a POST request to https://dummyjson.com/products/add
- Create a flow with a Python script that fetches weather data for New York City
- Make a REST API call to https://kestra.io/api/mock and allow failure
- Create a flow that logs "Hello from AI" to the console
- Create a flow that returns Hello as output
- Create a flow that outputs Hello as value
- Run a flow every 10 minutes
- Run a flow every day at 9 AM
- Run a shell command echo 'Hello Docker' in a Docker container
- Run a command python main.py in a Docker container
- Run a script main.py stored as namespace file
- Build a Docker image from an inline Dockerfile and push it to a GitHub Container Registry
- Build a Docker image from an inline Dockerfile and push it to a DockerHub Container Registry
- Create a flow that adds a string KV pair called MYKEY with value myvalue to namespace company
- Fetch value for KV pair called MYKEY from namespace company
- Create a flow that downloads a file mydata.csv from S3 bucket named mybucket
- Create a flow that downloads all files from the folder kestra/plugins/ from S3 bucket mybucket in us-east-1
- Send a Slack notification that approval is needed and Pause the flow for manual approval
- Send a Slack alert whenever any execution from namespace company fails
- Fetch value for string kv pair called mykey from Redis
- Fetch value for mykey from Redis
- Set value for mykey in Redis to myvalue
- Sync all flows and scripts for selected namespaces from Git to Kestra
- Create a flow that clones a Git repository and runs a Python script
- Export a Postgres table called mytable to a CSV file
- Query a Postgres table called mytable
- Find documents in a MongoDB collection called mycollection
- Load documents into a MongoDB mycollection using a file from input mydata
- Trigger an Airbyte connection sync and retry it up to 3 times
- Run an Airflow DAG called mydag
- Orchestrate an Ansible playbook stored in Namespace Files
- Run a DuckDB query that reads a CSV file
- Fetch AWS ECR authorization token to push Docker images to Amazon ECR
- Run a flow whenever 5 records are available in Kafka topic mytopic
- Submit a run for a Databricks job
```
:::
:::collapse{title="Ask mode prompts"}
```markdown
- Why did this execution fail? (attach the execution as context)
- What secrets and variables are available in this namespace? (attach the namespace as context)
- What is the difference between a Worker Group and a Task Runner?
- What plugins are available for working with Kafka?
- How do I configure RBAC so developers can run flows but not edit them?
- What is the best way to handle retries for a flaky HTTP API?
- How do I pass outputs from one task to the next?
- What does the errors block do and when should I use it?
- How do I schedule a flow to run only on weekdays?
- What is the difference between Namespace Variables and the KV Store?
```
:::
:::collapse{title="Plan mode prompts"}
```markdown
- Build an ELT pipeline: extract from Salesforce, transform with dbt on DuckDB, load into Snowflake, and send a Slack summary on completion or failure
- Migrate all ForEach tasks in this flow to use the Loop task
- Add retry logic, error notifications, and a timeout to every task in this flow
- Set up namespaces for dev, staging, and prod with RBAC roles for the engineering team
- Create a flow that ingests data from five different S3 paths in parallel, merges the results, and loads them into BigQuery
```
:::
## Enterprise Edition Copilot configurations
Enterprise Edition supports Amazon Bedrock, Anthropic, Azure OpenAI, DeepSeek, Google Gemini, Google Vertex AI, Mistral, OpenAI, OpenRouter, and all open-source models via Ollama. Add one or more provider blocks inside `kestra.ai.providers` and set `is-default: true` on the one Copilot should use by default.
:::alert{type="warning"}
Some Ollama model tags resolve to thinking models behind the scenes (for example, `qwen3:30b-a3b` points to a thinking variant). Thinking models that cannot have thinking disabled will produce incorrect YAML. Check that your chosen model supports a non-thinking mode, or use `thinking-enabled: false` where the provider supports it.
:::
### Amazon Bedrock
```yaml
kestra:
ai:
providers:
- id: bedrock
display-name: Amazon Bedrock
type: bedrock
configuration:
model-name: amazon.nova-lite-v1:0
access-key-id: BEDROCK_ACCESS_KEY_ID
secret-access-key: BEDROCK_SECRET_ACCESS_KEY
```
### Anthropic
Anthropic does not accept an `api-key` configuration field. Set the `ANTHROPIC_API_KEY` environment variable on the Kestra server instead.
```yaml
kestra:
ai:
providers:
- id: anthropic
display-name: Anthropic
type: anthropic
configuration:
model-name: claude-opus-4-5
```
### Azure OpenAI
```yaml
kestra:
ai:
providers:
- id: azure-openai
display-name: Azure OpenAI
type: azure-openai
configuration:
model-name: gpt-4o-2024-11-20
api-key: AZURE_OPENAI_API_KEY
tenant-id: AZURE_TENANT_ID
client-id: AZURE_CLIENT_ID
client-secret: AZURE_CLIENT_SECRET
endpoint: "https://your-resource.openai.azure.com/"
```
### Deepseek
```yaml
kestra:
ai:
providers:
- id: deepseek
display-name: DeepSeek
type: deepseek
configuration:
model-name: deepseek-chat
api-key: DEEPSEEK_API_KEY
base-url: "https://api.deepseek.com/v1"
```
### Google Gemini
```yaml
kestra:
ai:
providers:
- id: gemini
display-name: Google Gemini
type: gemini
configuration:
model-name: gemini-2.5-flash
api-key: YOUR_GEMINI_API_KEY
```
### Google Vertex AI
Authenticates via Application Default Credentials; no `api-key` field is needed. Ensure the Kestra runtime has ADC configured (e.g. `GOOGLE_APPLICATION_CREDENTIALS` env var or Workload Identity).
```yaml
kestra:
ai:
providers:
- id: vertex
display-name: Google Vertex AI
type: googlevertexai
configuration:
model-name: gemini-2.5-flash
project: GOOGLE_PROJECT_ID
location: us-central1
```
### Mistral
```yaml
kestra:
ai:
providers:
- id: mistral
display-name: Mistral
type: mistralai
configuration:
model-name: mistral:7b
api-key: MISTRALAI_API_KEY
base-url: "https://api.mistral.ai/v1"
```
### Ollama
```yaml
kestra:
ai:
providers:
- id: ollama
display-name: Ollama
type: ollama
configuration:
model-name: llama3
base-url: http://localhost:11434
```
:::alert{type="info"}
If Ollama is running locally on your host machine while Kestra is running inside a container, connection errors may occur when using `localhost`. In this case, use the Docker internal network URL instead. For example, set the base URL to `http://host.docker.internal:11434`.
:::
### OpenAI
```yaml
kestra:
ai:
providers:
- id: openai
display-name: OpenAI
type: openai
configuration:
model-name: gpt-5-nano
api-key: OPENAI_API_KEY
base-url: https://api.openai.com/v1
```
### OpenRouter
```yaml
kestra:
ai:
providers:
- id: openrouter
display-name: OpenRouter
type: open-router
configuration:
api-key: OPENROUTER_API_KEY
model-name: "anthropic/claude-sonnet-4"
```
---
# RAG Workflows in Kestra – Retrieval-Augmented Generation
URL: https://kestra.io/docs/ai-tools/ai-rag-workflows
> Build Retrieval-Augmented Generation (RAG) workflows in Kestra to ground LLM responses in your own data or web search results.
Ask questions, get data-backed answers with RAG.
## Build retrieval-augmented generation workflows
Retrieval Augmented Generation (RAG) enhances LLM responses by grounding them in your own data instead of relying solely on the model’s internal knowledge. It works by retrieving relevant document embeddings and combining them with the user’s prompt to produce accurate, context-aware outputs. Chat with your data using RAG in Kestra.
This example shows how to use **Retrieval Augmented Generation (RAG)** in Kestra to ground Large Language Model (LLM) responses in your own data. The flow ingests documents, stores embeddings in the KV Store, and contrasts responses from a plain LLM prompt with RAG-enabled responses, demonstrating how RAG reduces hallucinations and improves accuracy.
## RAG flow example
```yaml
id: rag
namespace: company.ai
tasks:
- id: ingest
type: io.kestra.plugin.ai.rag.IngestDocument
provider:
type: io.kestra.plugin.ai.provider.GoogleGemini
modelName: gemini-embedding-exp-03-07
apiKey: "{{ secret('GEMINI_API_KEY') }}"
embeddings:
type: io.kestra.plugin.ai.embeddings.KestraKVStore
drop: true
fromExternalURLs:
- https://raw.githubusercontent.com/kestra-io/docs/refs/heads/main/content/blogs/release-0-24.md
- id: parallel
type: io.kestra.plugin.core.flow.Parallel
tasks:
- id: chat_without_rag
type: io.kestra.plugin.ai.completion.ChatCompletion
provider:
type: io.kestra.plugin.ai.provider.GoogleGemini
apiKey: "{{ secret('GEMINI_API_KEY') }}"
modelName: gemini-3.5-flash-lite
messages:
- type: USER
content: Which features were released in Kestra 0.24?
- id: chat_with_rag
type: io.kestra.plugin.ai.rag.ChatCompletion
chatProvider:
type: io.kestra.plugin.ai.provider.GoogleGemini
apiKey: "{{ secret('GEMINI_API_KEY') }}"
modelName: gemini-3.5-flash-lite
embeddingProvider:
type: io.kestra.plugin.ai.provider.GoogleGemini
apiKey: "{{ secret('GEMINI_API_KEY') }}"
modelName: gemini-embedding-exp-03-07
embeddings:
type: io.kestra.plugin.ai.embeddings.KestraKVStore
systemMessage: You are a helpful assistant that can answer questions about Kestra.
prompt: Which features were released in Kestra 0.24?
```
### How it works
This flow first ingests external documents into the Kestra KV Store by generating embeddings with a chosen LLM provider. Those embeddings act as a searchable index. When you ask a question, Kestra can either pass the raw prompt directly to the LLM (without RAG) or augment the prompt with the most relevant information retrieved from the embeddings (with RAG). By supporting the model’s response in actual data, Kestra reduces the likelihood of hallucinations and ensures answers remain accurate and contextual to your source material.
### Without RAG vs. with RAG
Without RAG, the model answers based only on its pretraining and may produce plausible but inaccurate results if the requested details are not part of its training knowledge. With RAG, the model supplements its reasoning by retrieving embeddings stored in the KV Store and using them as context, producing responses directly tied to the ingested documents.
Use RAG when you need AI responses anchored in current, domain-specific, or external data sources.
## RAG with web search example
This example shows how to combine Retrieval Augmented Generation (RAG) with a web search content retriever to answer questions using both stored knowledge and up-to-date external sources.
```yaml
id: rag_with_websearch_content_retriever
namespace: company.ai
tasks:
- id: chat_with_rag_and_websearch_content_retriever
type: io.kestra.plugin.ai.rag.ChatCompletion
chatProvider:
type: io.kestra.plugin.ai.provider.GoogleGemini
modelName: gemini-3.5-flash-lite
apiKey: "{{ secret('GEMINI_API_KEY') }}"
contentRetrievers:
- type: io.kestra.plugin.ai.retriever.TavilyWebSearch
apiKey: "{{ secret('TAVILY_API_KEY') }}"
systemMessage: You are a helpful assistant that can answer questions about Kestra.
prompt: What is the latest release of Kestra?
```
The flow uses the `TavilyWebSearch` ([Tavily](https://www.tavily.com/)) retriever to fetch the latest information from the web and provides it as context to the `ChatCompletion` task. By grounding the LLM’s response in real-time search results, Kestra can answer questions such as “What is the latest release of Kestra?” with accurate, current data.
### Comparison: Static RAG vs. Web Search RAG
- Static RAG (e.g., with document ingestion) is ideal when you want to ground responses in a fixed knowledge base, such as internal documentation or policies.
- Web Search RAG extends this by retrieving fresh, dynamic content from the internet, making it better for answering time-sensitive or evolving questions like product releases or recent events.
For more properties, examples, and implementations, refer to the [Kestra RAG documentation](/plugins/plugin-ai/rag).
---
# AI Workflows in Kestra: Orchestrate with Any LLM
URL: https://kestra.io/docs/ai-tools/ai-workflows
> Orchestrate AI workflows in Kestra with any LLM provider. Connect to OpenAI, Anthropic, Google, and more to build intelligent automation pipelines.
Build AI workflows with your preferred LLM.
## Orchestrate AI workflows with your preferred LLM
Kestra provides plugins for multiple LLM providers and continues to add more with each release. You can design flows that use your chosen model and integrate AI into orchestration workflows.
The following examples demonstrate Kestra AI plugins for a variety of workflows. You can adapt each example to your chosen provider. Three key properties are important to understand:
- `type`: Defines the LLM provider plugin and task (e.g., `ChatCompletion` with OpenAI).
- `apiKey`: Access key for the provider – store this as a [key-value pair](../../06.concepts/05.kv-store/index.md) in Kestra Open Source or as a [secret](../../06.concepts/04.secret/index.md) in Enterprise Edition.
- `model`: Specifies the provider model. Models vary in performance, cost, and capabilities, so choose the one that best fits your use case.
Different provider plugins may include additional properties beyond those shown in the examples. Refer to each plugin’s documentation for a complete list. Common properties to be aware of include `prompt`, `messages`, `jsonResponseSchema`, to name a few.
:::collapse{title="Check the weather is suitable for sports every day using Gemini"}
This flow checks the daily wind conditions in Cambridgeshire and uses Google Gemini to decide whether it is suitable to go sailing. If the wind speed falls within the preferred range (above 10 knots and below 30 knots), the flow notifies you in Slack with the recommendation and automatically blocks your calendar for the day with an 'Out of office – gone sailing' event. It runs every morning at `8:00` AM on a schedule.
```yaml
id: check_weather
namespace: company.ai
tasks:
- id: ask_ai
type: io.kestra.plugin.gemini.StructuredOutputCompletion
apiKey: "{{ secret('GEMINI_API_KEY') }}"
model: "gemini-3.5-flash-lite"
prompt: "I like to go sailing when the wind is above 10 knots but below 30 knots. I sail in Cambridgeshire. If the wind is within that range, I want to know if I should go sailing or not. Also tell me the current wind speed speeds"
jsonResponseSchema: |
{
"type": "object",
"properties": {
"content": {
"type": "string"
},
"wind": {
"type": "number"
},
"go_sailing": {
"type": "boolean"
}
}
}
}
- id: if
type: io.kestra.plugin.core.flow.If
condition: "{{ outputs.ask_ai['predictions'] | first | jq('.go_sailing') | first }}"
then:
- id: notify_me
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
url: "{{ secret('SLACK_WEBHOOK') }}"
payload: |
{
"text": "{{ outputs.ask_ai['predictions'] | first | jq('.content') | first }}"
}
- id: block_calendar
type: io.kestra.plugin.googleworkspace.calendar.InsertEvent
calendarId: "{{ secret('CALENDAR_ID') }}"
serviceAccount: "{{ secret('GOOGLE_SA') }}"
summary: Out of office
description: "Gone sailing because the wind is {{ outputs.ask_ai['predictions'] | first | jq('.wind') | first }} knots"
startTime:
dateTime: "{{ now() | date(\"yyyy-MM-dd'T'09:00:00+01:00\") }}"
timeZone: "Europe/London"
endTime:
dateTime: "{{ now() | date(\"yyyy-MM-dd'T'18:00:00+01:00\") }}"
timeZone: "Europe/London"
creator:
email: wrussell@kestra.io
triggers:
- id: check_daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "* 8 * * *"
```
:::
:::collapse{title="Create tasks with natural language prompts using DeepSeek and Todoist"}
This flow turns natural language prompts into structured Todoist tasks using an AI model. Each item is parsed into a title, description, and due date, then automatically created in your Todoist workspace via the REST API.
```yaml
id: add_tasks_to_todoist
namespace: company.ai
inputs:
- id: prompt
type: STRING
displayName: What would you like to add to your task list?
description: List out all the things you need to get done
defaults: I need to get my prescription on Friday afternoon and go shopping afterwards
tasks:
- id: create_task_fields
type: io.kestra.plugin.deepseek.ChatCompletion
apiKey: '{{ secret("DEEPSEEK_API_KEY") }}'
modelName: deepseek-chat
messages:
- type: SYSTEM
content: You are going to help to write a todo list inside of Todoist. I need you to return any user messages as tasks in JSON format only. There might be multiple tasks. The current time is '{{ now() }}'
- type: USER
content: "{{ inputs.prompt }}"
jsonResponseSchema: |
{
type: "object",
"properties": {
"tasks": {
"type": "array",
"items": {
"type": "object",
"properties": {
"title": "string",
"description": "string",
"due_date": {
"type": "string",
"format": "date-time"
"description": "Due date of the task (as a RFC 3339 timestamp).",
}
}
}
}
}
}
- id: create_tasks
type: io.kestra.plugin.core.flow.Loop
values: "{{ outputs.create_task_fields.response | jq('.tasks') | first }}"
tasks:
- id: create_task
type: io.kestra.plugin.core.http.Request
uri: https://api.todoist.com/rest/v2/tasks
method: POST
contentType: application/json
headers:
Authorization: "Bearer {{ secret('TODOIST_API_TOKEN') }}"
body: |
{
"content": "{{ item.value | jq('.title') | first }}",
"description": "{{ item.value | jq('.description') | first }}",
"due_datetime": "{{ item.value | jq('.due_date') | first }}"
}
```
:::
:::collapse{title="Generate an image with OpenAI with human approval"}
This flow generates an image from a user prompt, sends it to a Discord channel for review, and waits for approval. If approved, the image is finalized and logged; if rejected, the user can provide feedback to regenerate a new image, which is then shared again on Discord.
```yaml
id: gen_img_approval
namespace: company.ai
inputs:
- id: image_prompt
type: STRING
variables:
discord_webhook: "https://discord.com/api/webhooks/URL"
tasks:
- id: gen_img
type: io.kestra.plugin.core.flow.Subflow
namespace: demo
flowId: generate_image
inputs:
openai_prompt: "{{ inputs.image_prompt }}"
- id: send_image
type: io.kestra.plugin.discord.DiscordExecution
content: "Are you happy with the image: {{ outputs.gen_img.outputs.image }}. Approve it here: http://localhost:8082/ui/executions/{{flow.namespace}}/{{flow.id}}/{{execution.id}} "
url: "{{ vars.discord_webhook }}"
- id: wait_for_approval
type: io.kestra.plugin.core.flow.Pause
onResume:
- id: approve
description: Are you happy with the photo or not?
type: BOOLEAN
- id: feedback
description: Write the prompt again with more detail
type: STRING
- id: try_again
type: io.kestra.plugin.core.flow.If
condition: "{{ outputs.wait_for_approval.onResume.approve }}"
then:
- id: approved
type: io.kestra.plugin.core.log.Log
message: "Final photo: {{ outputs.gen_img.outputs.image }}"
else:
- id: retry
type: io.kestra.plugin.core.flow.Subflow
namespace: demo
flowId: generate_image
inputs:
openai_prompt: "{{ outputs.wait_for_approval.onResume.feedback }}"
- id: send_new_image
type: io.kestra.plugin.discord.DiscordExecution
content: "Here's the new image with your feedback: {{ outputs.retry.outputs.image }}"
url: "{{ vars.discord_webhook }}"
```
:::
:::collapse{title="Summarize Git commits from the past week using Ollama"}
This flow automatically summarizes Git commits from the past week in a specified repository and branch. Each Friday at `15:00` UTC, it generates a plain-text summary using Ollama and posts it to Slack, keeping teams updated on project progress.
```yaml
id: ai-summarize-weekly-git-commits
namespace: company.ai
inputs:
- id: repository
type: URI
defaults: https://github.com/kestra-io/blueprints
description: Repository to summarize last week's progress
- id: branch
type: STRING
defaults: main
description: Git branch to summarize last week's progress
tasks:
- id: wdir
type: io.kestra.plugin.core.flow.WorkingDirectory
tasks:
- id: clone_repo
type: io.kestra.plugin.git.Clone
branch: "{{ inputs.branch }}"
url: "{{ inputs.repository }}"
- id: fetch_commits
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: bitnami/git:latest
commands:
# 0. Set safe.directory for Git to avoid "dubious ownership" errors
- git config --global --add safe.directory "$(pwd)"
# 1. Deepen clone if shallow
- git fetch --unshallow origin {{ inputs.branch }} || true
# 2. Update main branch
- git fetch origin {{ inputs.branch }}
# 3. Fetch commits from the last 7 days (weekly)
- git log origin/{{ inputs.branch }} --since="7 days ago" --pretty=format:"%h %ad %s" --date=short > commits.txt
# 4. Show how many were found
- echo "Fetched $(wc -l < commits.txt) commits from the last 7 days"
outputFiles:
- commits.txt
- id: summarize_commits
type: io.kestra.plugin.ollama.cli.OllamaCLI
enableModelCaching: true
modelCachePath: "{{ kv('OLLAMA_CACHE_PATH') }}"
commands:
- "ollama run gemma3:1b \"Summarize the following Git commits into a clear and concise weekly development update for users. Output plain text for Slack, no markdown or extra formatting. Ensure no markdown syntax like **bold text** in the response — stick to plain text! Here are the commit messages: {{ read(outputs.fetch_commits.outputFiles['commits.txt']) }}\" > output.txt"
outputFiles:
- output.txt
- id: slack
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
url: "{{ secret('SLACK_WEBHOOK') }}"
payload: |
{{
{
"text": "This week's repository updates for " ~ inputs.repository ~ ". " ~ read(outputs.summarize_commits.outputFiles['output.txt'])
}
}}
triggers:
- id: weekly-trigger
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 15 * * 5" # Every Friday at 15:00 (3:00 PM) UTC
```
:::
---
# Kestra MCP Server: Plugins, Blueprints & Docs for AI Tools
URL: https://kestra.io/docs/ai-tools/kestra-mcp-resources
> Use the Kestra MCP server to give AI tools like Claude Code and Cursor live access to Kestra plugin documentation, blueprints, and product docs.
Give AI tools direct access to Kestra's plugin reference, blueprints library, and product documentation through a single MCP server.
## What is the Kestra MCP server
The Kestra MCP server implements the [Model Context Protocol](https://modelcontextprotocol.io/), a standard that lets AI tools query external data sources at runtime. Connect an AI coding agent to the Kestra MCP server to access:
- **Plugin reference**: task schemas, properties, inputs, outputs, and version history for all 1,400+ Kestra plugins
- **Blueprints**: ready-to-use flow templates for common integration patterns
- **Documentation**: searchable product docs — concepts, how-to guides, and reference pages
An agent configured with the Kestra MCP server retrieves current information about plugin behavior, configuration options, and usage patterns — rather than relying on static training data.
## Connect the MCP server
The Kestra MCP server is available at:
```plaintext
https://api.kestra.io/v1/mcp
```
### Claude Code
Add the server with the Claude Code CLI:
```bash
claude mcp add kestra --transport http https://api.kestra.io/v1/mcp
```
Or add it directly to `~/.claude.json`:
```json
{
"mcpServers": {
"kestra": {
"type": "http",
"url": "https://api.kestra.io/v1/mcp"
}
}
}
```
### Cursor
Add the following to your Cursor MCP configuration (`.cursor/mcp.json` in your project or `~/.cursor/mcp.json` globally):
```json
{
"mcpServers": {
"kestra": {
"url": "https://api.kestra.io/v1/mcp"
}
}
}
```
### Other AI tools
Any MCP-compatible tool can connect using the HTTP transport and the URL above. Refer to your tool's documentation for the exact configuration format.
## Available tools
The Kestra MCP server exposes the following tools to your agent once connected:
| Category | Tools |
| --- | --- |
| Plugins | List plugins, browse tasks by plugin, retrieve task schemas, check plugin versions |
| Blueprints | Search and browse blueprints, retrieve full blueprint flow YAML |
| Docs | Search documentation, retrieve pages, navigate doc hierarchy |
| Reference | List task runners, triggers, secret managers, storage backends, log exporters |
## Example usage
### Plugin schema lookup
Ask your agent about a specific task's properties:
> "What properties does the `io.kestra.plugin.jdbc.postgresql.Query` task accept?"
The agent retrieves the live task schema and returns a structured summary:
```
io.kestra.plugin.jdbc.postgresql.Query — key properties:
- url (string, required): JDBC connection string, e.g. jdbc:postgresql://host:5432/db
- username (string, required)
- password (string, required): use {{ secret('KEY') }} to avoid storing credentials in plain text
- sql (string, required): the SQL query to execute
- fetchType (enum): FETCH, FETCH_ONE, STORE, or NONE — controls how results are returned
- fetchSize (integer): rows per fetch batch, default 1000
```
### Blueprint retrieval
Ask your agent to find a blueprint by use case:
> "Find a blueprint for triggering a flow when a file lands in S3."
The agent searches the blueprints library and returns matching flow YAML you can paste directly into the Kestra editor:
```yaml
id: s3_trigger
namespace: company.team
tasks:
- id: process
type: io.kestra.plugin.scripts.python.Script
# ...
triggers:
- id: watch
type: io.kestra.plugin.aws.s3.Trigger
bucket: your-bucket
prefix: incoming/
interval: PT1M
```
### Documentation lookup
For prose questions — such as _"What does the `whenMissingInSource` property do in NamespaceSync?"_ — the agent retrieves and summarizes the relevant docs page directly.
## Relationship to other AI tools
The Kestra MCP server complements the other AI tools in this section:
- **[AI Copilot](../01.ai-copilot/index.md)**: generates and refines flows from natural language inside the Kestra UI.
- **[AI Agents](../05.ai-agents/index.md)**: autonomous task execution inside Kestra flows.
- **[Agent Skills](../04.agent-skills/index.md)**: structured knowledge files that teach coding agents how to build Kestra flows and operate environments.
If you primarily work in an AI coding agent like Claude Code or Cursor, the MCP server gives you current Kestra context while building flows.
---
# MCP Server in Kestra – Expose Flows as AI Tools
URL: https://kestra.io/docs/ai-tools/mcp-server
> Configure Kestra MCP servers to expose flows as tools for AI agents. Learn how to create servers, set authentication, and connect Claude Desktop, Claude Code, and Cursor.
A Kestra MCP server exposes flows as named tools over HTTP for AI agents to discover and call.
A Kestra MCP server is a tenant-scoped entity that uses the [Model Context Protocol](https://modelcontextprotocol.io). Any flow with an [`McpToolTrigger`](../../05.workflow-components/07.triggers/06.mcp-tool-trigger/index.md) is automatically registered as a named tool on its target server. AI agents discover the tool list at connection time, so adding or removing triggers takes effect without restarting clients.
## Two directions: Kestra as server vs. Kestra as client
Kestra supports MCP in both directions:
| Direction | How | When to use |
|---|---|---|
| **Kestra as MCP server** | `McpToolTrigger` + MCP server entity | AI agents (Claude, Cursor) call your flows as tools |
| **Kestra as MCP client** | MCP client tasks (`SseMcpClient`, `StreamableHttpMcpClient`, `StdioMcpClient`, `DockerMcpClient`) | Your flows call external MCP servers as part of an AI Agent task |
This page covers Kestra as an MCP server. For using external MCP servers from within flows, see [AI Agents](../05.ai-agents/index.md).
## Default server
A `default` MCP server is automatically provisioned for every tenant on startup. You can use it immediately with no setup needed. The `McpToolTrigger`'s `mcpServer` property defaults to `"default"`, so a minimal trigger requires no explicit server reference.
## Managing MCP servers
Navigate to **Tenant → MCP Servers** in the left sidebar to view, create, edit, and manage MCP servers.
Each server has the following fields:
| Field | Description |
|---|---|
| `name` | Display name for the server. |
| `description` | Optional description shown in the UI. |
| `systemPrompt` | Instructions prepended to every AI agent session connected to this server. Use this to guide agent behavior, for example to restrict which tools to call or define the agent's persona. |
| `serverType` | `PRIVATE` (default) or `PUBLIC`. A private server requires authentication; a public server accepts unauthenticated connections. |
| `authType` | `BASIC` (username/password, available in OSS and EE), `API_TOKEN` (EE and Cloud only), or `OAUTH` (EE and Cloud only). |
### Authentication types
| Auth type | Available in | Notes |
|---|---|---|
| `BASIC` | OSS, EE | Username and password required on connect. |
| `API_TOKEN` | EE and Cloud | API token required on connect. Rejected on OSS. |
| `OAUTH` | EE and Cloud | OAuth 2.0 flow. Required for browser-based MCP clients such as Claude web. Configure the OAuth provider name via `oauthProvider`. |
Keep servers private unless you have a specific reason to expose them publicly. A public server allows any MCP client to call any flow registered on it without authentication.
### Configuring OAuth authentication
`OAUTH` auth requires an OIDC provider configured in your Kestra instance (the same configuration used for SSO). See [SSO configuration](../../07.enterprise/03.auth/sso/index.md) for how to set up an OIDC provider under `micronaut.security.oauth2.clients`.
Once a provider is configured, set `authType` to `OAUTH` on the MCP server. The `oauthProvider` field specifies which provider to use by its configured name:
```yaml
# kestra.yml
micronaut:
security:
oauth2:
clients:
my-oidc-provider:
client-id:
client-secret:
openid:
issuer: https://accounts.example.com
```
When a client connects to an `OAUTH`-protected MCP server and presents no token, the server responds with a `WWW-Authenticate` header pointing to the OAuth Protected Resource Metadata endpoint (RFC 9728). MCP-compliant clients such as Claude web discover the OIDC provider from this automatically and initiate the authorization code + PKCE flow; no manual client configuration is needed beyond the MCP server URL.
:::alert{type="info"}
The authenticating user must already have a Kestra account. The OAuth token's `email` or `preferred_username` claim is matched against existing Kestra users. If no matching user is found, the connection is rejected.
:::
If multiple OIDC providers are configured, set `oauthProvider` to the name of the specific provider (e.g. `my-oidc-provider` from the example above). If `oauthProvider` is omitted, the provider is matched automatically by the JWT's issuer claim.
## Connecting an AI agent client
Open a server in the UI and click the **Connect** tab. It shows the server URL and ready-to-paste configuration snippets for each supported client:
- **Claude Desktop** — JSON block to add to `claude_desktop_config.json`. Claude Desktop does not natively support HTTP MCP servers, so the snippet uses `npx mcp-remote` as a bridge over SSE.
- **Claude Code** — `claude mcp add` command using `--transport http`. Run it in your terminal, then start Claude Code from the same terminal session so the auth header is in scope.
- **Cursor** — server URL to paste into Cursor Settings → MCP → Add new MCP server
- **Codex** — connection configuration
For Basic Auth, the snippet references `${KESTRA_BASIC_AUTH}` as a placeholder. Replace it with the actual base64-encoded `username:password` value inline; the variable reference does not expand at connection time:
```bash
claude mcp add \
--transport http \
--header "Authorization: Basic $(echo -n 'username:password' | base64)"
```
## Viewing registered tools
The **Tool Flows** tab on each server lists all flows that have an `McpToolTrigger` pointing at that server. Use this to audit which flows are exposed and to navigate directly to a flow's trigger configuration.
## RBAC (Enterprise)
In the Enterprise Edition, `MCP_SERVER` is a first-class RBAC resource. See [RBAC](../../07.enterprise/03.auth/rbac/index.md#mcp-server-permissions) for the default role assignments.
Access to a private server is also flow-scoped: a user can connect to a private MCP server only if they have `FLOW: EXECUTE` permission on at least one namespace that has a flow with an `McpToolTrigger` pointing at that server.
## MCP server cache configuration
By default, each webserver node caches MCP server configuration in memory and hot-reloads it when a server is created, updated, or deleted. Two optional properties control the cache behavior:
| Property | Default | Description |
|---|---|---|
| `kestra.mcp.server-cache-config.maximum-size` | `500` | Maximum number of MCP server entries to cache. |
| `kestra.mcp.server-cache-config.expire-after-access` | `PT5M` | How long a cache entry remains valid after last access. |
Example configuration:
```yaml
kestra:
mcp:
server-cache-config:
maximum-size: 200
expire-after-access: PT10M
```
Adjust these only if you have a large number of MCP servers or tight memory constraints. The defaults are sufficient for most deployments.
---
# API Reference: Enterprise and Open Source Editions
URL: https://kestra.io/docs/api-reference
> Access the complete API reference for both Kestra Open Source and Enterprise editions to integrate and automate your workflows.
import ChildTableOfContents from "~/components/content/ChildTableOfContents.astro"
## Choose the right Kestra API reference
---
# Cloud & Enterprise API Reference for Kestra
URL: https://kestra.io/docs/api-reference/enterprise
> Comprehensive API reference for Kestra Cloud and Enterprise editions, including advanced features like authentication and governance.
import ApiDocEE from "~/components/content/ApiDocee.astro"
API Reference of Kestra Cloud & Enterprise.
## Explore the Kestra Cloud and Enterprise API
---
# SDK Language Clients for the Kestra API
URL: https://kestra.io/docs/api-reference/kestra-sdk
> Explore official Kestra SDKs for Java, Python, and Node.js to interact with the Kestra API and build custom applications.
import ChildCard from "~/components/docs/ChildCard.astro"
Interact with Kestra's API via language SDKs. There are [official Kestra SDKs](https://github.com/kestra-io/client-sdk) for Java, JavaScript, and Python.
SDK-based plugins support a `DEFAULT`/`AUTO` authentication mode that pulls a default service account globally or from the current [Namespace](../../07.enterprise/02.governance/07.namespace-management/index.md#default-service-account-for-sdk-plugins) (or [Tenant](../../07.enterprise/02.governance/tenants/index.md#default-service-account-for-sdk-plugins)). Configure those defaults in the UI, or set a global fallback under `tasks.sdk.authentication` in your [Configuration Basics](../../configuration/01.configuration-basics/index.md).
---
# Go SDK for Kestra: Client Setup and Examples
URL: https://kestra.io/docs/api-reference/kestra-sdk/go-sdk
> Integrate Kestra with Go using the official SDK. Learn to set up the client, configure authentication, and programmatically create and execute workflows.
Use the Kestra Go SDK to interact with the Kestra API from Go applications.
## Install the Go SDK
Install the Go module:
```shell
go get github.com/kestra-io/client-sdk/go-sdk
```
Import the package in your code:
```go
import openapiclient "github.com/kestra-io/client-sdk/go-sdk"
```
---
## Configure the client
Define two helpers — `newClient()` to build the `APIClient` and `newContext()` to attach credentials — then reuse them across your application. Read configuration from environment variables.
```go
package main
import (
"context"
"os"
openapiclient "github.com/kestra-io/client-sdk/go-sdk"
)
func getenv(key, fallback string) string {
if v := os.Getenv(key); v != "" {
return v
}
return fallback
}
func newClient() *openapiclient.APIClient {
cfg := openapiclient.NewConfiguration()
cfg.Servers = openapiclient.ServerConfigurations{
{URL: getenv("KESTRA_URL", "http://localhost:8080")},
}
return openapiclient.NewAPIClient(cfg)
}
func newContext() context.Context {
return context.WithValue(
context.Background(),
openapiclient.ContextBasicAuth,
openapiclient.BasicAuth{
UserName: getenv("KESTRA_USER", "root@root.com"),
Password: getenv("KESTRA_PASS", "Root!1234"),
},
)
}
```
:::alert{type="info"}
To use bearer token authentication instead, pass `openapiclient.ContextAccessToken` with your token string as the context value. The examples below receive `ctx` and `apiClient` from `newContext()` and `newClient()` above.
:::
---
## Create a flow
Send the flow definition as a YAML string.
```go
func createFlow(ctx context.Context, apiClient *openapiclient.APIClient) {
tenant := "main"
body := `id: my_flow
namespace: my_namespace
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Hello World!
`
flow, _, err := apiClient.FlowsAPI.CreateFlow(ctx, tenant).Body(body).Execute()
if err != nil {
fmt.Printf("Error creating flow: %v\n", err)
return
}
fmt.Println("Flow created:", flow.GetId())
}
```
:::alert{type="info"}
`body` must be valid flow YAML. If a flow with the same `id` and `namespace` already exists, use `UpdateFlow` instead.
:::
---
## Update a flow
Send the full YAML — including the same `id` and `namespace` — to replace an existing flow.
```go
func updateFlow(ctx context.Context, apiClient *openapiclient.APIClient) {
tenant := "main"
namespace := "my_namespace"
id := "my_flow"
body := `id: my_flow
namespace: my_namespace
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Updated message!
`
updated, _, err := apiClient.FlowsAPI.UpdateFlow(ctx, tenant, namespace, id).Body(body).Execute()
if err != nil {
fmt.Printf("Error updating flow: %v\n", err)
return
}
fmt.Println("Flow updated:", updated.GetId())
}
```
---
## Delete a flow
Remove a flow by its `namespace`, `id`, and `tenant`.
```go
func deleteFlow(ctx context.Context, apiClient *openapiclient.APIClient) {
tenant := "main"
namespace := "my_namespace"
id := "my_flow"
_, err := apiClient.FlowsAPI.DeleteFlow(ctx, namespace, id, tenant).Execute()
if err != nil {
fmt.Printf("Error deleting flow: %v\n", err)
return
}
fmt.Println("Flow deleted:", namespace+"/"+id)
}
```
:::alert{type="info"}
Deleting a flow removes its definition. Execution history is retained unless you delete executions separately.
:::
---
## Execute a flow
Trigger an execution and optionally wait for it to complete.
```go
func createExecution(ctx context.Context, apiClient *openapiclient.APIClient) {
tenant := "main"
namespace := "my_namespace"
id := "my_flow"
wait := true
execs, _, err := apiClient.ExecutionsAPI.
CreateExecution(ctx, namespace, id, tenant).
Wait(wait).
Execute()
if err != nil {
fmt.Printf("Error creating execution: %v\n", err)
return
}
if len(execs) > 0 {
fmt.Println("Execution started:", execs[0].GetId())
}
}
```
:::alert{type="info"}
`Wait(true)` blocks until the execution finishes. Use `Wait(false)` for non-blocking calls. Pass `.Labels([]string{"team:platform"})` to attach labels to the execution. `CreateExecution` returns a slice — the SDK wraps the single execution in an array. Always check `len(execs) > 0` before accessing `execs[0]`.
:::
---
## KV Store
The KV Store lets you read and write key-value pairs scoped to a namespace.
### List keys
```go
func listKVKeys(ctx context.Context, apiClient *openapiclient.APIClient) {
tenant := "main"
keys, _, err := apiClient.KVAPI.ListKeys(ctx, "my_namespace", tenant).Execute()
if err != nil {
fmt.Printf("Error listing keys: %v\n", err)
return
}
for _, entry := range keys {
fmt.Println("Key:", entry.GetKey())
}
}
```
### Get a value
```go
func getKVValue(ctx context.Context, apiClient *openapiclient.APIClient) {
tenant := "main"
result, _, err := apiClient.KVAPI.
GetKeyValue(ctx, "my_namespace", "my_key", tenant).
Execute()
if err != nil {
fmt.Printf("Error getting key: %v\n", err)
return
}
fmt.Println("Value:", result.GetValue())
}
```
### Set a value
```go
func setKVValue(ctx context.Context, apiClient *openapiclient.APIClient) {
tenant := "main"
_, _, err := apiClient.KVAPI.
SetKeyValue(ctx, "my_namespace", "my_key", tenant).
Body("my_value").
Execute()
if err != nil {
fmt.Printf("Error setting key: %v\n", err)
return
}
fmt.Println("Key set")
}
```
### Delete a key
```go
func deleteKVKey(ctx context.Context, apiClient *openapiclient.APIClient) {
tenant := "main"
_, _, err := apiClient.KVAPI.
DeleteKeyValue(ctx, "my_namespace", "my_key", tenant).
Execute()
if err != nil {
fmt.Printf("Error deleting key: %v\n", err)
return
}
fmt.Println("Key deleted")
}
```
---
## Read execution logs
Fetch all log entries for a completed execution. Log operations use `KestraClient` from the same package — not the generated `APIClient`.
```go
import (
"context"
"fmt"
kestra "github.com/kestra-io/client-sdk/go-sdk/kestra_api_client"
)
func listLogs() {
ctx := context.Background()
client := kestra.NewClient("http://localhost:8080",
kestra.WithBasicAuth("root@root.com", "Root!1234"))
logs, err := client.Logs().ListLogsFromExecution(ctx, "your-execution-id", "main", nil, nil, nil, nil)
if err != nil {
fmt.Printf("Error fetching logs: %v\n", err)
return
}
for _, log := range logs {
fmt.Printf("[%s] %s\n", log.GetLevel(), log.GetMessage())
}
}
```
Pass a minimum log level to filter results — for example `kestra.PtrString("INFO")` as the fourth argument. Remaining arguments (`taskRunId`, `taskId`, `attempt`) narrow by task; pass `nil` to skip each filter.
---
## Stream execution logs (SSE)
Stream logs from a running execution in real time. `FollowLogsFromExecution` opens an SSE connection and returns a `<-chan *LogEntry`. Entries arrive as the execution produces them. The channel closes when the execution ends or the context is cancelled.
```go
func followLogs() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
client := kestra.NewClient("http://localhost:8080",
kestra.WithBasicAuth("root@root.com", "Root!1234"))
ch, err := client.Logs().FollowLogsFromExecution(ctx, "your-execution-id", "main", nil)
if err != nil {
fmt.Printf("Error starting log stream: %v\n", err)
return
}
for entry := range ch {
if entry.GetExecutionId() == "" {
continue // skip the synthetic "start" entry the server sends on stream open
}
fmt.Printf("[%s] %s\n", entry.GetLevel(), entry.GetMessage())
}
}
```
To stop streaming early, cancel the context. The server-side SSE connection closes and the channel drains within milliseconds:
```go
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
ch, err := client.Logs().FollowLogsFromExecution(ctx, "your-execution-id", "main", nil)
if err != nil {
fmt.Printf("Error starting log stream: %v\n", err)
return
}
for entry := range ch {
if entry.GetExecutionId() == "" {
continue
}
fmt.Printf("[%s] %s\n", entry.GetLevel(), entry.GetMessage())
}
```
:::alert{type="info"}
The fourth argument is an optional minimum log level filter (`*string`). Pass `kestra.PtrString("INFO")` to receive only INFO and above, or `nil` to receive all levels.
:::
---
## Manage triggers
Search, enable or disable, unlock, and restart triggers for flows.
### Search triggers
```go
func searchTriggers(ctx context.Context, apiClient *openapiclient.APIClient) {
tenant := "main"
result, _, err := apiClient.TriggersAPI.
SearchTriggers(ctx, tenant).
Namespace("my_namespace").
Execute()
if err != nil {
fmt.Printf("Error searching triggers: %v\n", err)
return
}
for _, t := range result.GetResults() {
fmt.Printf("%s: disabled=%v\n", t.GetTriggerId(), t.GetDisabled())
}
}
```
### Disable or enable a trigger
Add `"time"` to your import block for `time.Time{}`.
```go
func disableTrigger(ctx context.Context, apiClient *openapiclient.APIClient) {
tenant := "main"
trigger := openapiclient.NewTrigger("my_namespace", "my_flow", "my_schedule", time.Time{})
request := openapiclient.NewTriggerControllerSetDisabledRequest(
[]openapiclient.Trigger{*trigger},
true, // disabled=true; pass false to re-enable
)
_, _, err := apiClient.TriggersAPI.
DisabledTriggersByIds(ctx, tenant).
TriggerControllerSetDisabledRequest(*request).
Execute()
if err != nil {
fmt.Printf("Error disabling trigger: %v\n", err)
return
}
fmt.Println("Trigger disabled")
}
```
### Unlock a trigger
```go
func unlockTrigger(ctx context.Context, apiClient *openapiclient.APIClient) {
tenant := "main"
_, _, err := apiClient.TriggersAPI.
UnlockTrigger(ctx, "my_namespace", "my_flow", "my_schedule", tenant).
Execute()
if err != nil {
fmt.Printf("Error unlocking trigger: %v\n", err)
return
}
fmt.Println("Trigger unlocked")
}
```
### Restart a trigger
```go
func restartTrigger(ctx context.Context, apiClient *openapiclient.APIClient) {
tenant := "main"
_, _, err := apiClient.TriggersAPI.
RestartTrigger(ctx, "my_namespace", "my_flow", "my_schedule", tenant).
Execute()
if err != nil {
fmt.Printf("Error restarting trigger: %v\n", err)
return
}
fmt.Println("Trigger restarted")
}
```
---
## Dashboards
The following examples use `NewClient`, which provides a higher-level client for APIs not available in the generated client.
```go
func newKestraClient() *openapiclient.KestraClient {
return openapiclient.NewClient(
getenv("KESTRA_URL", "http://localhost:8080"),
openapiclient.WithBasicAuth(
getenv("KESTRA_USER", "root@root.com"),
getenv("KESTRA_PASS", "Root!1234"),
),
)
}
```
### Create a dashboard
```go
func createDashboard(ctx context.Context) {
tenant := "main"
body := `id: my_dashboard
title: My Dashboard
`
kestraClient := newKestraClient()
dashboard, err := kestraClient.Dashboards().CreateDashboard(ctx, tenant, body)
if err != nil {
fmt.Printf("Error creating dashboard: %v\n", err)
return
}
fmt.Println("Dashboard created:", dashboard.GetId())
}
```
### Search dashboards
```go
func searchDashboards(ctx context.Context) {
tenant := "main"
kestraClient := newKestraClient()
result, err := kestraClient.Dashboards().SearchDashboards(ctx, tenant, nil, nil, nil, nil)
if err != nil {
fmt.Printf("Error searching dashboards: %v\n", err)
return
}
for _, d := range result.GetResults() {
fmt.Println("Dashboard:", d.GetId())
}
}
```
### Delete a dashboard
```go
func deleteDashboard(ctx context.Context) {
tenant := "main"
kestraClient := newKestraClient()
err := kestraClient.Dashboards().DeleteDashboard(ctx, "my_dashboard_id", tenant)
if err != nil {
fmt.Printf("Error deleting dashboard: %v\n", err)
return
}
fmt.Println("Dashboard deleted")
}
```
---
## Namespace files
List, read, and delete files stored in a namespace.
### List files
```go
func listFiles(ctx context.Context) {
tenant := "main"
kestraClient := newKestraClient()
path := "/"
files, err := kestraClient.Files().ListNamespaceDirectoryFiles(ctx, "my_namespace", tenant, &path)
if err != nil {
fmt.Printf("Error listing files: %v\n", err)
return
}
for _, f := range files {
fmt.Println("File:", f.GetFileName())
}
}
```
### Read file content
```go
func readFile(ctx context.Context) {
tenant := "main"
kestraClient := newKestraClient()
content, err := kestraClient.Files().FileContent(ctx, "my_namespace", tenant, "/scripts/main.py", nil)
if err != nil {
fmt.Printf("Error reading file: %v\n", err)
return
}
defer content.Close()
fmt.Println("Downloaded to:", content.Name())
}
```
### Delete a file
```go
func deleteFile(ctx context.Context) {
tenant := "main"
kestraClient := newKestraClient()
err := kestraClient.Files().DeleteFileDirectory(ctx, "my_namespace", tenant, "/scripts/main.py")
if err != nil {
fmt.Printf("Error deleting file: %v\n", err)
return
}
fmt.Println("File deleted")
}
```
---
## Test suites
:::alert{type="warning"}
Test suites require Kestra Enterprise Edition.
:::
Create, run, and fetch results for unit test suites.
### Create a test suite
```go
func createTestSuite(ctx context.Context) {
tenant := "main"
body := `id: my_tests
namespace: my_namespace
flows:
- flowId: my_flow
`
kestraClient := newKestraClient()
suite, err := kestraClient.TestSuites().CreateTestSuite(ctx, tenant, body)
if err != nil {
fmt.Printf("Error creating test suite: %v\n", err)
return
}
fmt.Println("Test suite created:", suite.GetId())
}
```
### Run a test suite
```go
func runTestSuite(ctx context.Context) {
tenant := "main"
kestraClient := newKestraClient()
result, err := kestraClient.TestSuites().RunTestSuite(ctx, "my_namespace", "my_tests", tenant, nil)
if err != nil {
fmt.Printf("Error running test suite: %v\n", err)
return
}
fmt.Println("State:", result.GetState())
}
```
### Get test results
```go
func getTestResult(ctx context.Context) {
tenant := "main"
kestraClient := newKestraClient()
result, err := kestraClient.TestSuites().TestResult(ctx, "run-id", tenant)
if err != nil {
fmt.Printf("Error fetching test result: %v\n", err)
return
}
fmt.Println("State:", result.GetState())
}
```
---
## Apps
:::alert{type="warning"}
Apps require Kestra Enterprise Edition.
:::
Create, enable, disable, and delete apps.
### Create an app
```go
func createApp(ctx context.Context) {
tenant := "main"
body := `id: my_app
title: My App
`
kestraClient := newKestraClient()
app, err := kestraClient.Apps().CreateApp(ctx, tenant, body)
if err != nil {
fmt.Printf("Error creating app: %v\n", err)
return
}
fmt.Println("App created:", app.GetUid())
}
```
### Enable or disable an app
```go
func enableApp(ctx context.Context) {
tenant := "main"
kestraClient := newKestraClient()
_, err := kestraClient.Apps().EnableApp(ctx, "app-uid", tenant)
if err != nil {
fmt.Printf("Error enabling app: %v\n", err)
return
}
fmt.Println("App enabled")
}
func disableApp(ctx context.Context) {
tenant := "main"
kestraClient := newKestraClient()
_, err := kestraClient.Apps().DisableApp(ctx, "app-uid", tenant)
if err != nil {
fmt.Printf("Error disabling app: %v\n", err)
return
}
fmt.Println("App disabled")
}
```
### Delete an app
```go
func deleteApp(ctx context.Context) {
tenant := "main"
kestraClient := newKestraClient()
err := kestraClient.Apps().DeleteApp(ctx, "app-uid", tenant)
if err != nil {
fmt.Printf("Error deleting app: %v\n", err)
return
}
fmt.Println("App deleted")
}
```
---
## Best practices
- **Reuse your client:** construct one `APIClient` at startup and share it via dependency injection or a package-level variable.
- **Externalize credentials:** read URL and auth from environment variables.
- **Handle errors explicitly:** all SDK methods return an error value — always check it.
- **Use context for timeouts:** pass a `context.WithTimeout` to control request deadlines.
---
# Java SDK for Kestra: Client Setup and Examples
URL: https://kestra.io/docs/api-reference/kestra-sdk/java-sdk
> Integrate Kestra with Java using the official SDK. Learn to set up the client, configure authentication, and programmatically create and execute workflows.
Use the Kestra Java SDK to interact with the Kestra API from Java applications.
## Install the Java SDK
Choose the installation method that matches your build tool.
### Maven
Add this dependency to your `pom.xml`:
```xml
io.kestrakestra-api-client1.0.0compile
```
### Gradle
Add this dependency to your `build.gradle`:
```groovy
implementation "io.kestra:kestra-api-client:1.0.0"
```
---
## Configure the client
Initialize `KestraClient` once using the builder, then reuse it across your application. Read credentials from environment variables rather than hardcoding them.
```java
import io.kestra.sdk.KestraClient;
public final class KestraClients {
private KestraClients() {}
public static final KestraClient INSTANCE = KestraClient.builder()
.url(System.getenv().getOrDefault("KESTRA_URL", "http://localhost:8080"))
.basicAuth(
System.getenv().getOrDefault("KESTRA_USER", "root@root.com"),
System.getenv().getOrDefault("KESTRA_PASS", "Root!1234")
)
// .tokenAuth(System.getenv("KESTRA_TOKEN")) // use instead for bearer auth
.build();
}
```
:::alert{type="info"}
Construct the client once (singleton or DI) and reuse it. Use either `.basicAuth(...)` or `.tokenAuth(...)`, not both.
:::
---
## Configure timeouts
By default, both connect and read timeouts are infinite. Configure them on the builder before calling long-running endpoints such as `runTestSuite`:
```java
import io.kestra.sdk.KestraClient;
import java.time.Duration;
KestraClient client = KestraClient.builder()
.url("https://kestra.example.com")
.tokenAuth(System.getenv("KESTRA_TOKEN"))
.connectTimeout(Duration.ofSeconds(10)) // time to establish the connection
.readTimeout(Duration.ofMinutes(30)) // time to wait for a response
.build();
```
Pass `Duration.ZERO` or omit the call to keep the default (infinite). Both methods accept any `java.time.Duration`.
---
## Create a flow
Send the flow definition as a YAML string. This matches what you would define in the UI.
```java
import io.kestra.sdk.KestraClient;
public class FlowsExamples {
public static void createFlow() {
String tenant = "main";
String body = """
id: my_flow
namespace: my_namespace
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Hello World!
""";
KestraClients.INSTANCE.flows().createFlow(tenant, body);
System.out.println("Flow created: my_namespace/my_flow");
}
}
```
:::alert{type="info"}
`body` must be valid YAML. If a flow with the same `id` and `namespace` already exists, use `updateFlow` instead.
:::
---
## Update a flow
Send the full YAML — including the same `id` and `namespace` — to replace an existing flow.
```java
public class FlowsExamples {
public static void updateFlow() {
String id = "my_flow";
String namespace = "my_namespace";
String tenant = "main";
String body = """
id: my_flow
namespace: my_namespace
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Updated message!
""";
KestraClients.INSTANCE.flows().updateFlow(id, namespace, tenant, body);
System.out.println("Flow updated: my_namespace/my_flow");
}
}
```
---
## Delete a flow
Remove a flow by its `namespace`, `id`, and `tenant`.
```java
public class FlowsExamples {
public static void deleteFlow() {
String namespace = "my_namespace";
String id = "my_flow";
String tenant = "main";
KestraClients.INSTANCE.flows().deleteFlow(namespace, id, tenant);
System.out.println("Flow deleted: my_namespace/my_flow");
}
}
```
:::alert{type="info"}
Deleting a flow removes its definition. Execution history is retained unless you delete executions separately.
:::
---
## Execute a flow
Trigger an execution and optionally pass labels or scheduling parameters.
```java
import java.util.List;
import java.time.OffsetDateTime;
import io.kestra.sdk.ApiException;
import io.kestra.sdk.model.ExecutionKind;
public class ExecutionsExamples {
public static void createExecution() {
String namespace = "my_namespace";
String id = "my_flow";
String tenant = "main";
Boolean wait = false;
try {
KestraClients.INSTANCE.executions()
.createExecution(
namespace, id, wait, tenant,
List.of("team:platform"), // labels
null, // revision (null = latest)
(OffsetDateTime) null, // scheduleDate
null, // breakpoint task ID
ExecutionKind.NORMAL
);
} catch (ApiException e) {
if (e.getCode() != 0) throw e; // code 0 = deserialization-only; execution ran normally
}
System.out.println("Execution triggered");
}
}
```
:::alert{type="warning"}
In SDK 1.0.0, `createExecution` successfully triggers the execution but throws a deserialization exception when reading the response — the server returns a single JSON object while the SDK expects an array. The execution runs normally. The `try/catch` above suppresses this by re-throwing only on a non-zero status code.
:::
---
## Delete an execution
Delete an execution and optionally purge its logs, metrics, and storage.
```java
public class ExecutionsExamples {
public static void deleteExecution() {
String executionId = "your-execution-id";
String tenant = "main";
KestraClients.INSTANCE.executions()
.deleteExecution(executionId, true, true, true, tenant);
System.out.println("Execution deleted");
}
}
```
---
## Follow an execution
`followExecution` returns a reactive `Flux` that streams state updates as the execution progresses. Each emission is an `Execution` object — skip entries where `getState()` is `null` (keepalive frames emitted by the server before the first real update).
```java
public class ExecutionsExamples {
public static void followExecution() {
String executionId = "your-execution-id";
String tenant = "main";
KestraClients.INSTANCE.executions()
.followExecution(executionId, tenant)
.filter(execution -> execution.getState() != null) // skip keepalive frames
.doOnNext(execution -> System.out.printf("[%s] %s%n",
execution.getId(), execution.getState().getCurrent()))
.blockLast(); // blocks until the execution stream ends
}
}
```
---
## KV Store
The KV Store lets you read and write key-value pairs scoped to a namespace.
### List keys
```java
public class KVExamples {
public static void listKeys() {
String namespace = "my_namespace";
String tenant = "main";
var keys = KestraClients.INSTANCE.kv().listKeys(namespace, tenant);
keys.forEach(entry -> System.out.println("Key: " + entry.getKey()));
}
}
```
### Get a value
```java
public class KVExamples {
public static void getKeyValue() {
String namespace = "my_namespace";
String tenant = "main";
var result = KestraClients.INSTANCE.kv()
.getKeyValue(namespace, "my_key", tenant);
System.out.println("Value: " + result.getValue());
}
}
```
### Set a value
```java
public class KVExamples {
public static void setKeyValue() {
String namespace = "my_namespace";
String tenant = "main";
KestraClients.INSTANCE.kv()
.setKeyValue(namespace, "my_key", tenant, "my_value");
System.out.println("Key set");
}
}
```
### Delete a key
```java
public class KVExamples {
public static void deleteKey() {
String namespace = "my_namespace";
String tenant = "main";
KestraClients.INSTANCE.kv().deleteKeyValue(namespace, "my_key", tenant);
System.out.println("Key deleted");
}
}
```
---
## Read execution logs
Fetch or stream logs for an execution.
### List logs
```java
import io.kestra.sdk.model.Level;
public class LogsExamples {
public static void listLogs() {
String executionId = "your-execution-id";
String tenant = "main";
var logs = KestraClients.INSTANCE.logs()
.listLogsFromExecution(executionId, tenant,
null, // minLevel (null = all levels)
null, // taskRunId
null, // taskId
null // attempt
);
logs.forEach(log ->
System.out.printf("[%s] %s%n", log.getLevel(), log.getMessage()));
}
}
```
### Stream logs live
`followLogsFromExecution` returns a reactive `Flux`. Each `FollowLogEvent` carries the same fields as `LogEntry` (plus `tenantId`). The server sends an initial keepalive frame with all fields `null` — filter it out before processing.
```java
public class LogsExamples {
public static void followLogs() {
String executionId = "your-execution-id";
String tenant = "main";
KestraClients.INSTANCE.logs()
.followLogsFromExecution(executionId, tenant, null) // null = no filters
.filter(event -> event.getExecutionId() != null) // skip keepalive frames
.doOnNext(event -> System.out.printf("[%s] %s%n",
event.getLevel(), event.getMessage()))
.blockLast(); // blocks until the stream ends
}
}
```
:::alert{type="info"}
Use `listLogsFromExecution` after an execution finishes. Use `followLogsFromExecution` to stream logs in real time from a running execution.
:::
---
## Manage triggers
Search, enable or disable, unlock, and restart triggers for flows.
### Search triggers
```java
public class TriggersExamples {
public static void searchTriggers() {
String tenant = "main";
var result = KestraClients.INSTANCE.triggers()
.searchTriggers(1, 50, tenant,
null, // sort
null, // filters
null, // query string
"my_namespace", // namespace filter
null, // workerId
null // flowId
);
result.getResults().forEach(t -> {
var ctx = t.getTriggerContext();
System.out.printf("%s: disabled=%s%n",
ctx.getTriggerId(), ctx.getDisabled());
});
}
}
```
### Disable or enable a trigger
```java
import io.kestra.sdk.model.TriggerControllerSetDisabledRequest;
import io.kestra.sdk.model.Trigger;
public class TriggersExamples {
public static void disableTrigger() {
String tenant = "main";
var request = new TriggerControllerSetDisabledRequest()
.addTriggersItem(new Trigger()
.namespace("my_namespace")
.flowId("my_flow")
.triggerId("my_schedule"))
.disabled(true); // pass false to re-enable
KestraClients.INSTANCE.triggers()
.disabledTriggersByIds(tenant, request);
System.out.println("Trigger disabled");
}
}
```
### Unlock a trigger
Use `unlockTrigger` to unlock a trigger that is stuck in a locked state.
```java
public class TriggersExamples {
public static void unlockTrigger() {
String tenant = "main";
KestraClients.INSTANCE.triggers()
.unlockTrigger("my_namespace", "my_flow", "my_schedule", tenant);
System.out.println("Trigger unlocked");
}
}
```
### Restart a trigger
```java
public class TriggersExamples {
public static void restartTrigger() {
String tenant = "main";
KestraClients.INSTANCE.triggers()
.restartTrigger("my_namespace", "my_flow", "my_schedule", tenant);
System.out.println("Trigger restarted");
}
}
```
---
## Dashboards
Create, search, and delete dashboards.
### Create a dashboard
```java
public class DashboardsExamples {
public static void createDashboard() {
String tenant = "main";
String body = """
id: my_dashboard
title: My Dashboard
""";
var dashboard = KestraClients.INSTANCE.dashboards().createDashboard(tenant, body);
System.out.println("Dashboard created: " + dashboard.getId());
}
}
```
### Search dashboards
```java
public class DashboardsExamples {
public static void searchDashboards() {
String tenant = "main";
var result = KestraClients.INSTANCE.dashboards().searchDashboards(tenant, null, null, null, null);
result.getResults().forEach(d -> System.out.println(d.getId()));
}
}
```
### Delete a dashboard
```java
public class DashboardsExamples {
public static void deleteDashboard() {
String tenant = "main";
KestraClients.INSTANCE.dashboards().deleteDashboard("my_dashboard_id", tenant);
System.out.println("Dashboard deleted");
}
}
```
---
## Namespace files
List, read, and delete files stored in a namespace.
### List files
```java
public class FilesExamples {
public static void listFiles() {
var files = KestraClients.INSTANCE.files()
.listNamespaceDirectoryFiles("my_namespace", "main", "/");
files.forEach(f -> System.out.println(f.getFileName()));
}
}
```
### Read file content
```java
import java.io.File;
public class FilesExamples {
public static void readFile() {
String tenant = "main";
File content = KestraClients.INSTANCE.files()
.fileContent("my_namespace", "/scripts/main.py", tenant, null);
System.out.println("Downloaded to: " + content.getAbsolutePath());
}
}
```
### Delete a file
```java
public class FilesExamples {
public static void deleteFile() {
String tenant = "main";
KestraClients.INSTANCE.files()
.deleteFileDirectory("my_namespace", "/scripts/main.py", tenant);
System.out.println("File deleted");
}
}
```
---
## Test suites
:::alert{type="warning"}
Test suites require Kestra Enterprise Edition.
:::
Create, run, and fetch results for unit test suites.
### Create a test suite
```java
public class TestSuitesExamples {
public static void createTestSuite() {
String tenant = "main";
String body = """
id: my_tests
namespace: my_namespace
flows:
- flowId: my_flow
""";
var suite = KestraClients.INSTANCE.testSuites().createTestSuite(tenant, body);
System.out.println("Test suite created: " + suite.getId());
}
}
```
### Run a test suite
```java
public class TestSuitesExamples {
public static void runTestSuite() {
String tenant = "main";
var result = KestraClients.INSTANCE.testSuites()
.runTestSuite("my_namespace", "my_tests", tenant, null);
System.out.println("State: " + result.getState());
}
}
```
### Get test results
```java
public class TestSuitesExamples {
public static void getTestResult() {
String tenant = "main";
var result = KestraClients.INSTANCE.testSuites().testResult("run-id", tenant);
System.out.println("State: " + result.getState());
}
}
```
---
## Apps
:::alert{type="warning"}
Apps require Kestra Enterprise Edition.
:::
Create, enable, disable, and delete apps.
### Create an app
```java
public class AppsExamples {
public static void createApp() {
String tenant = "main";
String body = """
id: my_app
title: My App
""";
var app = KestraClients.INSTANCE.apps().createApp(tenant, body);
System.out.println("App created: " + app.getUid());
}
}
```
### Enable or disable an app
```java
public class AppsExamples {
public static void enableApp() {
String tenant = "main";
KestraClients.INSTANCE.apps().enableApp("app-uid", tenant);
System.out.println("App enabled");
}
public static void disableApp() {
String tenant = "main";
KestraClients.INSTANCE.apps().disableApp("app-uid", tenant);
System.out.println("App disabled");
}
}
```
### Delete an app
```java
public class AppsExamples {
public static void deleteApp() {
String tenant = "main";
KestraClients.INSTANCE.apps().deleteApp("app-uid", tenant);
System.out.println("App deleted");
}
}
```
---
## Best practices
- **Reuse your client:** construct one `KestraClient` per application (singleton or DI).
- **Externalize credentials:** read URL and auth from environment variables or your config system.
- **Keep flow YAML as code:** store YAML strings or templates under version control for traceability.
- **Use labels** for governance, search, and routing across executions.
---
# JavaScript SDK for Kestra: Client Setup and Examples
URL: https://kestra.io/docs/api-reference/kestra-sdk/javascript-sdk
> Integrate Kestra with JavaScript using the official SDK. Install the library, configure the client, and programmatically create and execute workflows.
Use the Kestra JavaScript SDK to interact with the Kestra API from Node.js applications.
## Install the JavaScript SDK
Store credentials in environment variables:
```bash
KESTRA_BASE_URL=http://localhost:8080
KESTRA_USERNAME=root@root.com
KESTRA_PASSWORD=Root!1234
```
Install the SDK:
```shell
npm install @kestra-io/kestra-sdk
```
## Configure the client
Call `configureClient` once at application startup, then call `setSelectedTenant` to set the active tenant. Both are applied globally — you do not need to pass connection details or tenant to individual method calls.
```javascript
import { configureClient } from "@kestra-io/kestra-sdk";
import { setSelectedTenant } from "@kestra-io/kestra-sdk/shared";
configureClient({
baseURL: process.env.KESTRA_BASE_URL ?? "http://localhost:8080",
auth: () => `${process.env.KESTRA_USERNAME}:${process.env.KESTRA_PASSWORD}`,
});
setSelectedTenant("main");
```
:::alert{type="info"}
For bearer token authentication, set `auth: () => process.env.KESTRA_TOKEN` and pass the token directly.
:::
Each API group is a separate subpath module. Import only what you need:
```javascript
import * as Flows from "@kestra-io/kestra-sdk/flows";
import * as Executions from "@kestra-io/kestra-sdk/executions";
import * as Kv from "@kestra-io/kestra-sdk/kv";
import * as Triggers from "@kestra-io/kestra-sdk/triggers";
```
---
## Create a flow
Send the flow definition as a YAML string:
```javascript
import * as Flows from "@kestra-io/kestra-sdk/flows";
async function createFlow() {
const body = `id: my_flow
namespace: my_namespace
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Hello World!
`;
const created = await Flows.createFlow({ body });
console.log("Flow created:", created.id);
}
```
:::alert{type="info"}
`body` must be valid flow YAML. If a flow with the same `id` and `namespace` already exists, use `updateFlow` instead.
:::
---
## Delete a flow
Remove a flow by its `namespace` and `id`:
```javascript
import * as Flows from "@kestra-io/kestra-sdk/flows";
async function deleteFlow() {
await Flows.deleteFlow({ namespace: "my_namespace", id: "my_flow" });
console.log("Flow deleted");
}
```
:::alert{type="info"}
Deleting a flow removes its definition. Execution history is retained unless you delete executions separately.
:::
---
## Execute a flow
Trigger an execution and optionally wait for it to complete:
```javascript
import * as Executions from "@kestra-io/kestra-sdk/executions";
async function executeFlow() {
const exec = await Executions.createExecution({
namespace: "my_namespace",
id: "my_flow",
wait: true, // set false for a non-blocking call
});
console.log("Execution started:", exec.id);
}
```
---
## Delete an execution
Delete an execution and optionally purge its logs, metrics, and storage:
```javascript
import * as Executions from "@kestra-io/kestra-sdk/executions";
async function deleteExecution() {
await Executions.deleteExecution({
executionId: "your-execution-id",
deleteLogs: true,
deleteMetrics: true,
deleteStorage: true,
});
console.log("Execution deleted");
}
```
---
## Follow an execution
Stream live execution state updates. `followExecution` returns a `{ stream }` object where `stream` is an async iterable of execution events:
```javascript
import * as Executions from "@kestra-io/kestra-sdk/executions";
async function followExecution() {
const { stream } = await Executions.followExecution({
executionId: "your-execution-id",
});
for await (const evt of stream) {
if (!evt.state) continue; // skip keepalive frames
console.log(`Status: ${evt.state.current}`);
if (evt.state.current === "SUCCESS" || evt.state.current === "FAILED") break;
}
}
```
:::alert{type="info"}
The server emits an initial keepalive event with no `state` — skip it before processing updates.
:::
---
## Read execution logs
### List logs
Fetch all log entries for a completed execution:
```javascript
import * as Logs from "@kestra-io/kestra-sdk/logs";
async function listLogs() {
const logs = await Logs.listLogsFromExecution({
executionId: "your-execution-id",
});
logs?.forEach(entry => console.log(`[${entry.level}] ${entry.message}`));
}
```
To filter by minimum log level, pass a `filters` array:
```javascript
const logs = await Logs.listLogsFromExecution({
executionId: "your-execution-id",
filters: [{ field: "LEVEL", operation: "GREATER_THAN_OR_EQUAL_TO", value: "INFO" }],
});
```
### Search logs
Search across all executions with pagination:
```javascript
import * as Logs from "@kestra-io/kestra-sdk/logs";
async function searchLogs() {
const result = await Logs.searchLogs({ page: 1, size: 50 });
result?.results?.forEach(entry => console.log(`[${entry.level}] ${entry.message}`));
}
```
---
## KV Store
The KV Store lets you read and write key-value pairs scoped to a namespace.
### Set a value
```javascript
import * as Kv from "@kestra-io/kestra-sdk/kv";
async function setKvValue() {
await Kv.setKeyValue({ namespace: "my_namespace", key: "my_key", body: "my_value" });
console.log("Key set");
}
```
### Get a value
```javascript
import * as Kv from "@kestra-io/kestra-sdk/kv";
async function getKvValue() {
const result = await Kv.keyValue({ namespace: "my_namespace", key: "my_key" });
console.log("Value:", result?.value);
}
```
### Delete a key
```javascript
import * as Kv from "@kestra-io/kestra-sdk/kv";
async function deleteKvKey() {
await Kv.deleteKeyValue({ namespace: "my_namespace", key: "my_key" });
console.log("Key deleted");
}
```
---
## Manage triggers
### Search triggers
```javascript
import * as Triggers from "@kestra-io/kestra-sdk/triggers";
async function searchTriggers() {
const result = await Triggers.searchTriggers({ page: 1, size: 50 });
result.results?.forEach(t => {
console.log(`${t.triggerContext?.triggerId}: disabled=${t.triggerContext?.disabled}`);
});
}
```
### Disable or enable a trigger
```javascript
import * as Triggers from "@kestra-io/kestra-sdk/triggers";
async function disableTrigger() {
await Triggers.disabledTriggersByIds({
triggers: [{ namespace: "my_namespace", flowId: "my_flow", triggerId: "my_schedule" }],
disabled: true, // pass false to re-enable
});
console.log("Trigger disabled");
}
```
### Restart a trigger
```javascript
import * as Triggers from "@kestra-io/kestra-sdk/triggers";
async function restartTrigger() {
await Triggers.restartTrigger({
namespace: "my_namespace",
flowId: "my_flow",
triggerId: "my_schedule",
});
console.log("Trigger restarted");
}
```
### Unlock a trigger
```javascript
import * as Triggers from "@kestra-io/kestra-sdk/triggers";
async function unlockTrigger() {
await Triggers.unlockTrigger({
namespace: "my_namespace",
flowId: "my_flow",
triggerId: "my_schedule",
});
console.log("Trigger unlocked");
}
```
---
## Dashboards
Create, search, and delete dashboards.
### Create a dashboard
```javascript
import * as Dashboards from "@kestra-io/kestra-sdk/dashboards";
async function createDashboard() {
const body = `id: my_dashboard
title: My Dashboard
`;
const dashboard = await Dashboards.createDashboard({ body });
console.log("Dashboard created:", dashboard.id);
}
```
### Search dashboards
```javascript
import * as Dashboards from "@kestra-io/kestra-sdk/dashboards";
async function searchDashboards() {
const result = await Dashboards.searchDashboards({});
result.results?.forEach(d => console.log(d.id));
}
```
### Delete a dashboard
```javascript
import * as Dashboards from "@kestra-io/kestra-sdk/dashboards";
async function deleteDashboard() {
await Dashboards.deleteDashboard({ id: "my_dashboard_id" });
console.log("Dashboard deleted");
}
```
---
## Namespace files
List, read, and delete files stored in a namespace.
### List files
```javascript
import * as Files from "@kestra-io/kestra-sdk/files";
async function listFiles() {
const files = await Files.listNamespaceDirectoryFiles({
namespace: "my_namespace",
path: "/",
});
files?.forEach(f => console.log(f.fileName));
}
```
### Read file content
```javascript
import * as Files from "@kestra-io/kestra-sdk/files";
async function readFile() {
const content = await Files.fileContent({
namespace: "my_namespace",
path: "/scripts/main.py",
});
console.log("Content received:", content?.size, "bytes");
}
```
### Delete a file
```javascript
import * as Files from "@kestra-io/kestra-sdk/files";
async function deleteFile() {
await Files.deleteFileDirectory({
namespace: "my_namespace",
path: "/scripts/main.py",
});
console.log("File deleted");
}
```
---
## Test suites
:::alert{type="warning"}
Test suites require Kestra Enterprise Edition.
:::
Create, run, and fetch results for unit test suites.
### Create a test suite
```javascript
import * as TestSuites from "@kestra-io/kestra-sdk/test-suites";
async function createTestSuite() {
const body = `id: my_tests
namespace: my_namespace
flows:
- flowId: my_flow
`;
const suite = await TestSuites.createTestSuite({ body });
console.log("Test suite created:", suite.id);
}
```
### Run a test suite
```javascript
import * as TestSuites from "@kestra-io/kestra-sdk/test-suites";
async function runTestSuite() {
const result = await TestSuites.runTestSuite({
namespace: "my_namespace",
id: "my_tests",
});
console.log("State:", result.state);
}
```
### Get test results
```javascript
import * as TestSuites from "@kestra-io/kestra-sdk/test-suites";
async function getTestResult() {
const result = await TestSuites.testResult({ id: "run-id" });
console.log("State:", result.state);
}
```
---
## Apps
:::alert{type="warning"}
Apps require Kestra Enterprise Edition.
:::
Create, enable, disable, and delete apps.
### Create an app
```javascript
import * as Apps from "@kestra-io/kestra-sdk/apps";
async function createApp() {
const body = `id: my_app
title: My App
`;
const app = await Apps.createApp({ body });
console.log("App created:", app.uid);
}
```
### Enable or disable an app
```javascript
import * as Apps from "@kestra-io/kestra-sdk/apps";
async function enableApp() {
await Apps.enableApp({ uid: "app-uid" });
console.log("App enabled");
}
async function disableApp() {
await Apps.disableApp({ uid: "app-uid" });
console.log("App disabled");
}
```
### Delete an app
```javascript
import * as Apps from "@kestra-io/kestra-sdk/apps";
async function deleteApp() {
await Apps.deleteApp({ uid: "app-uid" });
console.log("App deleted");
}
```
---
## Best practices
- **Configure once:** call `configureClient` and `setSelectedTenant` once at startup and reuse them globally.
- **Externalize config:** keep URL and auth in environment variables.
- **Validate YAML:** invalid flow YAML returns `422` responses.
- **Use labels** for governance, search, and routing across executions.
---
# Python SDK for Kestra: Client Setup and Examples
URL: https://kestra.io/docs/api-reference/kestra-sdk/python-sdk
> Integrate Kestra with your Python applications. Learn to set up the Kestra Python SDK, configure the client, and programmatically create and execute workflows.
Use the Kestra Python SDK (`kestrapy`) to interact with the Kestra API from Python applications.
## Install the Python SDK
Before starting, make sure your Kestra instance is running. Store credentials in an `.env` file:
```bash
KESTRA_HOST=http://localhost:8080
KESTRA_USERNAME=root@root.com
KESTRA_PASSWORD=Root!1234
```
Create a virtual environment and install the [Kestra Python SDK](https://github.com/kestra-io/client-sdk/blob/main/README_PYTHON_SDK.md).
```shell
uv venv
source .venv/bin/activate
uv pip install kestrapy
uv pip install python-dotenv # optional: loads .env automatically
```
## Configure the client
Import and initialize the client with your Kestra credentials. Construct `KestraClient` once and reuse it throughout your application.
```python
from kestrapy import Configuration, KestraClient
configuration = Configuration(
host="http://localhost:8080",
username="root@root.com",
password="Root!1234"
)
kestra_client = KestraClient(configuration)
```
:::alert{type="info"}
Use environment variables rather than hardcoding credentials. You can also use token-based authentication by setting `access_token` on the `Configuration` object instead of `username`/`password`.
:::
---
## Configure timeouts
By default, requests wait indefinitely. Pass a `timeout` argument to `KestraClient` to limit how long requests wait before raising `requests.Timeout`.
Using a `Configuration` object:
```python
from kestrapy import Configuration, KestraClient
configuration = Configuration(host="http://localhost:8080", username="root@root.com", password="Root!1234")
kestra_client = KestraClient(configuration, timeout=30.0) # float: connect + read combined
kestra_client = KestraClient(configuration, timeout=(10.0, 300.0)) # tuple: (connect, read)
kestra_client = KestraClient(configuration, timeout=None) # None: no timeout (default)
```
Using keyword arguments directly:
```python
from kestrapy import KestraClient
kestra_client = KestraClient(host="http://localhost:8080", token="your-api-token", timeout=30.0)
kestra_client = KestraClient(host="http://localhost:8080", token="your-api-token", timeout=(10.0, 300.0))
kestra_client = KestraClient(host="http://localhost:8080", token="your-api-token", timeout=None)
```
The `timeout` value is forwarded directly to [`requests`](https://docs.python-requests.org/en/latest/user/advanced/#timeouts), so any form that `requests` accepts is valid.
---
## Create a flow
Pass the flow definition as a YAML string to [`create_flow`](https://github.com/kestra-io/client-sdk/blob/main/python-sdk/docs/FlowsApi.md#create_flow).
```python
def create_flow():
tenant = "main"
body = """
id: my_flow
namespace: my_namespace
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: "Hello World!"
"""
created = kestra_client.flows.create_flow(tenant=tenant, body=body)
print(f"Flow created: {created.id}")
```
:::alert{type="info"}
`body` must be valid YAML. If a flow with the same `id`, `namespace`, and `tenant` already exists, use `update_flow` instead.
:::
---
## Update a flow
Send the full YAML — including the same `id` and `namespace` — to replace an existing flow.
```python
def update_flow():
tenant = "main"
body = """
id: my_flow
namespace: my_namespace
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: "Updated message!"
"""
updated = kestra_client.flows.update_flow(
id="my_flow",
namespace="my_namespace",
tenant=tenant,
body=body
)
print(f"Flow updated: {updated.id}")
```
---
## Delete a flow
Remove a flow by its `namespace`, `id`, and `tenant`.
```python
def delete_flow():
tenant = "main"
kestra_client.flows.delete_flow(
namespace="my_namespace",
id="my_flow",
tenant=tenant
)
print("Flow deleted")
```
:::alert{type="info"}
Deleting a flow removes its definition. Execution history is retained unless you delete executions separately.
:::
---
## Execute a flow
Trigger an execution using [`create_execution`](https://github.com/kestra-io/client-sdk/blob/main/python-sdk/docs/ExecutionsApi.md#create_execution).
The first three positional arguments are `namespace`, `id` (the flow ID), and `wait`.
```python
def create_execution():
tenant = "main"
execution = kestra_client.executions.create_execution(
namespace="my_namespace",
id="my_flow",
wait=True,
tenant=tenant
)
print(f"Execution started: {execution.id}")
```
To pass inputs, use `additional_form_datas` with a dictionary keyed by input ID:
```python
def create_execution_with_inputs():
tenant = "main"
execution = kestra_client.executions.create_execution(
namespace="my_namespace",
id="my_flow",
wait=True,
tenant=tenant,
additional_form_datas={"input_id": "value"}
)
print(f"Execution started: {execution.id}")
```
:::alert{type="info"}
`wait=True` blocks until the execution completes. Use `wait=False` for non-blocking calls. The `additional_form_datas` keys must match the flow's defined input IDs.
:::
---
## Follow an execution
Stream live execution updates using `follow_execution`.
```python
def follow_execution():
tenant = "main"
execution = kestra_client.executions.create_execution(
namespace="my_namespace",
id="my_flow",
wait=False,
tenant=tenant
)
for event in kestra_client.executions.follow_execution(
execution_id=execution.id,
tenant=tenant
):
if event.state is None:
continue # keepalive frame
print(event.state.current)
```
:::alert{type="info"}
The first SSE payload is an empty keepalive — skip it before processing subsequent events. Use `follow_execution` for CI/CD pipelines or real-time dashboards.
:::
---
## Read execution logs
### List logs
Fetch all log entries for a completed execution:
```python
def list_logs():
tenant = "main"
logs = kestra_client.logs.list_logs_from_execution(
execution_id="your-execution-id",
tenant=tenant,
min_level="INFO", # optional; filters to INFO and above
)
for entry in logs:
print(f"[{entry.level}] {entry.message}")
```
### Stream logs live
`follow_logs_from_execution` yields `LogEntry` items as the execution produces them. The server sends an initial keepalive frame with all fields `None` — skip entries where `execution_id` is `None`.
```python
def follow_logs():
tenant = "main"
for entry in kestra_client.logs.follow_logs_from_execution(
execution_id="your-execution-id",
tenant=tenant,
):
if entry.execution_id is None:
continue # keepalive frame
print(f"[{entry.level}] {entry.message}")
```
:::alert{type="info"}
The `min_level` parameter on `follow_logs_from_execution` is not applied by the Kestra 2.0 server — pass no filter and handle level filtering in the consumer loop if needed.
:::
---
## KV Store
The KV Store lets you read and write key-value pairs scoped to a namespace.
### List keys
Use `list_all_keys` to get a paged list of keys across the tenant:
```python
def list_kv_keys():
tenant = "main"
result = kestra_client.kv.list_all_keys(
page=1,
size=50,
tenant=tenant
)
for entry in result.results:
print(f"Key: {entry.key}")
```
### Get a value
```python
def get_kv_value():
tenant = "main"
result = kestra_client.kv.key_value(
namespace="my_namespace",
key="my_key",
tenant=tenant
)
print(f"Value: {result.value}")
```
### Set a value
```python
def set_kv_value():
tenant = "main"
kestra_client.kv.set_key_value(
namespace="my_namespace",
key="my_key",
tenant=tenant,
body="my_value"
)
print("Key set")
```
### Delete a key
```python
def delete_kv_key():
tenant = "main"
kestra_client.kv.delete_key_value(
namespace="my_namespace",
key="my_key",
tenant=tenant
)
print("Key deleted")
```
---
## Manage triggers
Search, enable or disable, unlock, and restart triggers for flows.
### Search triggers
`search_triggers` is paginated and requires `page` and `size`:
```python
def search_triggers():
tenant = "main"
result = kestra_client.triggers.search_triggers(
page=1,
size=50,
tenant=tenant
)
for t in result.results:
print(f"{t.trigger_context.trigger_id}: disabled={t.trigger_context.disabled}")
```
### Disable or enable a trigger
```python
import datetime
from kestrapy.models import TriggerControllerSetDisabledRequest, Trigger
def disable_trigger():
tenant = "main"
request = TriggerControllerSetDisabledRequest(
triggers=[
Trigger(
namespace="my_namespace",
flow_id="my_flow",
trigger_id="my_schedule",
var_date=datetime.datetime.now(datetime.timezone.utc)
)
],
disabled=True # pass False to re-enable
)
kestra_client.triggers.disabled_triggers_by_ids(
tenant=tenant,
trigger_controller_set_disabled_request=request
)
print("Trigger disabled")
```
### Unlock a trigger
Use `unlock_trigger` to unlock a trigger that is stuck in a locked state:
```python
def unlock_trigger():
tenant = "main"
kestra_client.triggers.unlock_trigger(
namespace="my_namespace",
flow_id="my_flow",
trigger_id="my_schedule",
tenant=tenant
)
print("Trigger unlocked")
```
### Restart a trigger
```python
def restart_trigger():
tenant = "main"
kestra_client.triggers.restart_trigger(
namespace="my_namespace",
flow_id="my_flow",
trigger_id="my_schedule",
tenant=tenant
)
print("Trigger restarted")
```
---
## Dashboards
Create, search, and delete dashboards.
### Create a dashboard
```python
def create_dashboard():
tenant = "main"
body = """
id: my_dashboard
title: My Dashboard
"""
dashboard = kestra_client.dashboards.create_dashboard(tenant=tenant, yaml_body=body)
print(f"Dashboard created: {dashboard.get('id')}")
```
### Search dashboards
```python
def search_dashboards():
tenant = "main"
result = kestra_client.dashboards.search_dashboards(tenant=tenant)
for d in result.get('results', []):
print(d.get('id'))
```
### Delete a dashboard
```python
def delete_dashboard():
tenant = "main"
kestra_client.dashboards.delete_dashboard(id="my_dashboard_id", tenant=tenant)
print("Dashboard deleted")
```
---
## Namespace files
List, read, and delete files stored in a namespace.
### List files
```python
def list_files():
tenant = "main"
files = kestra_client.files.list_namespace_directory_files(
namespace="my_namespace",
tenant=tenant,
path="/"
)
for f in files:
print(f.file_name)
```
### Read file content
```python
def read_file():
tenant = "main"
content = kestra_client.files.file_content(
namespace="my_namespace",
path="/scripts/main.py",
tenant=tenant,
revision=None
)
print(f"Downloaded {len(content)} bytes")
```
### Delete a file
```python
def delete_file():
tenant = "main"
kestra_client.files.delete_file_directory(
namespace="my_namespace",
path="/scripts/main.py",
tenant=tenant
)
print("File deleted")
```
---
## Test suites
:::alert{type="warning"}
Test suites require Kestra Enterprise Edition.
:::
Create, run, and fetch results for unit test suites.
### Create a test suite
```python
def create_test_suite():
tenant = "main"
body = """
id: my_tests
namespace: my_namespace
flows:
- flowId: my_flow
"""
suite = kestra_client.test_suites.create_test_suite(tenant=tenant, yaml_body=body)
print(f"Test suite created: {suite.id}")
```
### Run a test suite
```python
def run_test_suite():
tenant = "main"
result = kestra_client.test_suites.run_test_suite(
namespace="my_namespace",
id="my_tests",
tenant=tenant
)
print(f"State: {result.state}")
```
### Get test results
```python
def get_test_result():
tenant = "main"
result = kestra_client.test_suites.test_result(id="run-id", tenant=tenant)
print(f"State: {result.state}")
```
---
## Apps
:::alert{type="warning"}
Apps require Kestra Enterprise Edition.
:::
Create, enable, disable, and delete apps.
### Create an app
```python
def create_app():
tenant = "main"
body = """
id: my_app
title: My App
"""
app = kestra_client.apps.create_app(tenant=tenant, yaml_body=body)
print(f"App created: {app.uid}")
```
### Enable or disable an app
```python
def enable_app():
tenant = "main"
kestra_client.apps.enable_app(uid="app-uid", tenant=tenant)
print("App enabled")
def disable_app():
tenant = "main"
kestra_client.apps.disable_app(uid="app-uid", tenant=tenant)
print("App disabled")
```
### Delete an app
```python
def delete_app():
tenant = "main"
kestra_client.apps.delete_app(uid="app-uid", tenant=tenant)
print("App deleted")
```
---
## Best practices
- **Reuse your client:** construct one `KestraClient` per application and share it.
- **Avoid hardcoding credentials:** use environment variables or a secrets manager.
- **Validate YAML before submission:** invalid syntax causes `422` responses.
- **Combine `create_flow` and `create_execution`** for end-to-end CI/CD automation.
---
# Open Source API Reference for Kestra
URL: https://kestra.io/docs/api-reference/open-source
> Detailed API documentation for Kestra Open Source edition, covering endpoints for flows, executions, and triggers.
import ApiDoc from "~/components/content/ApiDoc.astro"
API Reference of the Open-Source edition of Kestra.
## Explore the Kestra Open Source API
---
# Architecture in Kestra: Components and Deployment Models
URL: https://kestra.io/docs/architecture
> Overview of Kestra's Architecture. Explore the scalable, event-driven design connecting server components, storage, and external systems.
Kestra is built on a single unified, pluggable backend. One set of queue and persistence contracts is satisfied by your chosen backend — a relational database by default, or a broker-backed alternative for higher throughput. All server roles are stateless and communicate only through the queue layer; no role calls another directly.

## Control plane and data plane
Kestra's server roles divide into two planes:
- **Control plane** — the stateless coordination roles: Executor, Worker Controller, Scheduler, Webserver, and Indexer. These roles never execute user code and have no access to user infrastructure. They communicate exclusively through the queue.
- **Data plane** — the Worker. Workers execute runnable tasks and polling triggers, and are the only roles that access user infrastructure and internal storage. A Worker connects to the control plane's Worker Controller over a bidirectional gRPC stream and can run in cloud, on-premises, or hybrid environments independently of the rest of the deployment.
This separation enables hybrid deployments where the control plane is managed by Kestra while Workers run inside your own infrastructure, or both are self-hosted.
## Server roles
Kestra has six server roles. In a standalone deployment, all six run as threads inside a single process. In a distributed deployment, each role runs as its own independently scaled process.
| Role | Responsibility |
|------|---------------|
| **Executor** | Drives the execution state machine. Consumes execution events and worker results, determines the next task to run, and dispatches work. Runs no user code. |
| **Worker Controller** | The sole communication point for workers. Dispatches jobs to workers over a bidirectional gRPC stream; accepts results, logs, and metrics back. Workers never touch the queue or database directly. |
| **Worker** | Executes runnable tasks and polling triggers. Connects to the Worker Controller via gRPC. The only role that accesses user infrastructure and [internal storage](./data-components/index.md#internal-storage). |
| **Scheduler** | Evaluates trigger conditions (except flow triggers, which the Executor handles) and submits executions to the queue. |
| **Webserver** | Serves the [REST API](../api-reference/index.mdx) and [UI](../09.ui/index.mdx). |
| **Indexer** | Reads from the queue and writes indexed content — flows, executions, logs, metrics, and audit logs — to the repository backend. Required in all deployments. |
## Queue and repository
The queue layer is the only channel through which server roles communicate. A server emits a typed message onto a named queue; another server consumes it. One backend implementation satisfies all queue contracts:
- **JDBC** (default) — backed by any supported relational database (PostgreSQL, MySQL). Available in all editions.
- **Kafka** — Enterprise Edition. Pairs with Elasticsearch for the search and read model.
- **Redis**, **AMQP**, **GCP Pub/Sub** — Enterprise Edition. Additional broker-backed options for higher throughput.
The repository stores all domain entities: flows, executions, logs, triggers, and secrets. In a JDBC deployment, the relational database handles both the queue and repository. In a Kafka deployment, Elasticsearch backs the high-volume read model; the Indexer keeps it in sync.
## Worker communication
Workers do not subscribe to the job queue directly. Each worker opens a persistent bidirectional gRPC stream to the Worker Controller and uses that stream for the lifetime of its connection:
- The Worker Controller dispatches jobs from the queue onto the stream.
- Workers return results, logs, and metrics over the same stream.
- Kill signals and metadata changes are broadcast to all connected workers.
gRPC transport is available in all editions. TLS and mTLS secure the connection in all editions; JWT-based worker authentication is an Enterprise Edition feature.

## How an execution runs
1. A user or client defines a flow through the Webserver's REST API. The flow is validated and stored in the repository under its namespace.
2. A trigger fires — a schedule comes due, an external event matches, or an API call requests a run. The Scheduler (for schedule and polling triggers) or the Webserver (for manual runs) emits a new execution onto the queue.
3. The Executor picks up the execution and runs its state machine to decide the next task.
4. For each task, the Executor asks the worker-queue resolver for a routing decision, then dispatches a worker task onto the queue.
5. The Worker Controller routes the job to a matching worker. The worker loads the task plugin, reads any inputs from internal storage, executes the task code, writes outputs back to internal storage, and emits the result.
6. The Executor joins the result back into the execution, advances the state machine, and either dispatches the next task or terminates the execution.
7. The Webserver serves the evolving execution state and logs to the UI by reading the repository. The Indexer keeps the read model in sync.
## Execution context
Task outputs are stored separately from the execution row and fetched on demand. This keeps the execution record small as flows grow in complexity and improves performance at high concurrency.
## Enterprise Edition
Enterprise Edition is an additive overlay on the open-source core — not a fork. It adds multi-tenancy, action-based RBAC, identity-provider integration, audit logging, and additional queue backends by supplying alternative implementations of the same core contracts the open-source engine depends on. All open-source behavior is preserved.
## Monitoring
Every server role exposes a Prometheus scrape endpoint and OpenTelemetry traces. Three platform-wide metrics serve as primary health signals:
| Metric | Description |
|--------|-------------|
| `kestra.queue.message.lag.count` | Backlog of unprocessed messages, tagged by worker queue. A steadily rising lag means consumers cannot keep up with producers. |
| `kestra.worker.job.pending` | Worker jobs waiting for a free worker thread across the cluster. |
| `kestra.worker.job.running` | Worker jobs currently executing across the cluster. |
Per-role metrics are documented on each component's page.
## Components in detail
import ChildCard from "~/components/docs/ChildCard.astro"
---
# Data Storage Components in Kestra Architecture
URL: https://kestra.io/docs/architecture/data-components
> Dive into Kestra's Data Architecture. Learn how inputs, outputs, logs, and metadata are stored across Repositories and Internal Storage systems.
Understand where different data components ([inputs](../../05.workflow-components/05.inputs/index.md), [outputs](../../05.workflow-components/06.outputs/index.md), logs, and more) are stored in Kestra’s architecture.
Kestra processes and stores a variety of data, including [flow definitions](../../05.workflow-components/01.flow/index.md), workflow inputs, outputs, logs, execution metadata, and more. Understanding how these components are stored helps optimize performance, configure persistence, and integrate with external storage systems.
Kestra data is stored in either the [repository](../01.main-components/index.md#repository), such as PostgreSQL, or in [internal storage](../data-components/index.md#internal-storage). By default, internal storage is local, but you can configure it to use cloud or S3-compatible object storage such as [AWS S3](https://aws.amazon.com/s3/), [Google Cloud Storage](https://cloud.google.com/storage), or [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/).
:::alert{type="info"}
See [Kestra architecture](../../08.architecture/03.deployment-architecture/index.md) and [internal storage](../data-components/index.md#internal-storage) for more details.
:::
## Data storage components
The table below outlines key data components, where they are stored, and their purpose.
| Data component | Storage location | Description |
|--------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------| --- |
| **Flows & definitions** | Repository | Stores flows, tasks, and their configurations. |
| **Namespaces** | Repository | Organizes workflows and manages secrets, variables, and KV store entries. |
| **Namespace files** | Internal storage | Stores code and configuration files in Kestra’s storage backend. |
| **Executions & metadata** | Repository | Stores execution details including status, timestamps, and metadata. |
| **Input values** (non-FILE types) | Repository (executions table) | Scalar input values stored in the executions table. Non-sensitive types (STRING, INTEGER, etc.) are stored as plaintext; `SECRET` type inputs are stored encrypted. |
| **Input files** (FILE type) | Internal storage | FILE-type inputs and files passed to script or CLI tasks, stored at `/{namespace}/{flow-id}/executions/{execution-id}/inputs/{input-name}/{file-name}`. |
| **Output values** | Repository (task_outputs table) | Scalar task outputs stored in a dedicated task_outputs table. In Enterprise Edition, values emitted via `encryptedOutputs` are stored as encrypted strings rather than plaintext. |
| **Output files** | Internal storage | Generated files available for download and reuse in downstream tasks. |
| **Key-value pairs** | Internal storage & repository (metadata only) | KV store holds data in key-value format. Metadata is recorded in the repository. |
| **Logs & [audit logs](../../07.enterprise/02.governance/06.audit-logs/index.md) (Enterprise Edition)** | Repository or [external log data store](../../10.administrator-guide/log-data-store/index.md) | Stores logs generated by tasks. Audit logs always use the repository. |
| **Task state & variables** | Repository | Stores dynamic variables and task states during executions. |
| **Secrets** | Repository or external [secret manager](../../07.enterprise/02.governance/secrets-manager/index.md) | Stores secrets internally or integrates with services like AWS Secrets Manager, Vault, or Google Secret Manager. |
| **Queues** | Database (default), or Kafka / Redis / AMQP / GCP Pub/Sub (Enterprise Edition) | Handles internal communication between Kestra components. |
| **Triggers** | Repository | Stores definitions of event-based triggers. |
| **User administration** | Repository | Stores RBAC, user management, and related metadata. |
## Internal storage
**Internal storage** is used for handling files during executions. It ensures efficient input and output management without burdening the database.
- **Purpose**: Handles inputs, outputs, temporary execution data, and artifacts such as [namespace files](../../06.concepts/02.namespace-files/index.md).
- **KV store**: Stores key-value pairs in internal storage, with metadata in the repository. Metadata includes the key, URI, TTL, and timestamps.
- **Backends**: By default, Kestra uses local storage, but for production you can configure cloud or S3-compatible object storage:
- [AWS S3](https://aws.amazon.com/s3/)
- [Google Cloud Storage](https://cloud.google.com/storage)
- [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/)
- [Cloudflare R2](https://www.cloudflare.com/products/r2/)
- [Huawei OBS](https://www.huaweicloud.com/en-us/product/obs.html)
- Any S3-compatible service (Ceph, SeaweedFS, Garage, MinIO)
### Configuring internal storage
Example configuration for AWS S3:
```yaml
kestra:
storage:
type: s3
s3:
bucket: "kestra-internal-storage"
region: "us-east-1"
```
For full details, see [internal storage configuration](../../configuration/02.runtime-and-storage/index.md#internal-storage).
## Additional information
### Flows and execution metadata
- Stored in **PostgreSQL**, **MySQL**, or **H2** (not recommended for distributed components).
- Includes:
- Flow definitions
- Execution details
- Execution queues
- Historical metadata
- Accessible via the [Kestra API](../../api-reference/index.mdx) and [UI](../../09.ui/index.mdx).
### Logs
- By default, logs are stored in the repository. In Kestra 2.0+ (EE), you can route them to a separate JDBC database or Elasticsearch using the [external log data store](../../10.administrator-guide/log-data-store/index.md).
- Audit logs are always stored in the repository.
- Logs can be accessed through the API, UI, or external logging integrations such as the [Log Shipper](../../07.enterprise/02.governance/logshipper/index.md).
### Queues
- **Open source**: Backed by the relational database (PostgreSQL or MySQL).
- **Enterprise Edition**: Can use Kafka, Redis, AMQP, or GCP Pub/Sub for higher-throughput inter-component messaging.
### Secrets management
Secrets can be stored in Kestra’s database or in external managers like [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/), [Google Secret Manager](https://cloud.google.com/secret-manager), [Azure Key Vault](https://azure.microsoft.com/products/key-vault/), or [HashiCorp Vault](https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2).
Example configuration for AWS Secret Manager:
```yaml
kestra:
secret:
type: aws-secret-manager
aws-secret-manager:
access-key-id: my-access-key
secret-key-id: my-secret-key
sessionToken: my-session-token
region: us-east-1
```
See [secret managers](../../07.enterprise/02.governance/secrets-manager/index.md) for more.
### Handling sensitive data and PII
Understanding where data is persisted is critical when flows process personally identifiable information (PII) or other sensitive values.
**Stored as plaintext in the database:**
- Non-sensitive scalar inputs (STRING, INTEGER, etc.) — stored in the executions table
- Task output values emitted via `outputs` in the script output protocol — stored in the task_outputs table
- Log messages — stored in the logs table
**Stored encrypted in the database:**
- `SECRET` type inputs — stored in the executions table as an encrypted value (requires encryption to be configured); the value is also automatically masked in logs
- `encryptedOutputs` values (Enterprise Edition) — stored in the task_outputs table
**Not stored in the database:**
- FILE-type inputs and output files — these go to internal storage (your configured S3, GCS, Azure Blob, etc.).
#### Encrypting sensitive task outputs
:::alert{type="info"}
`encryptedOutputs` is an Enterprise Edition and Cloud feature.
:::
In Enterprise Edition and Cloud, script tasks support an `encryptedOutputs` key in the `::{}::` output protocol. Values written this way are wrapped in an `EncryptedString` and stored encrypted in the task_outputs table rather than as plaintext. They are merged into the same outputs map as regular outputs and are decrypted by Kestra at evaluation time.
```yaml
id: sensitive_data_flow
namespace: company.team
inputs:
- id: ssn
type: SECRET
tasks:
- id: process_pii
type: io.kestra.plugin.scripts.shell.Script
script: |
echo '::{"encryptedOutputs":{"ssn":"{{ inputs.ssn }}"}}::'
echo '::{"outputs":{"status":"processed"}}::'
```
Both plaintext and encrypted outputs are accessible in downstream tasks via `{{ outputs.process_pii.ssn }}` — Kestra decrypts the value at expression evaluation time.
Lines matching the `::{}::` protocol are consumed by the output processor and never written to the logs table, so the plaintext value is not exposed in log storage.
#### Best practices for PII-sensitive flows
- **Use `type: SECRET` for sensitive inputs.** `SECRET` inputs are encrypted at rest in the executions table and automatically masked in log output. Requires [`kestra.encryption.secret-key`](../../configuration/05.security-and-secrets/index.md) to be configured — without it, `SECRET` inputs fail at runtime.
- **Use `encryptedOutputs` for sensitive task outputs** (Enterprise Edition). Values are stored encrypted in the task_outputs table and decrypted transparently when referenced in downstream tasks.
- **Use [Secrets](../../07.enterprise/02.governance/secrets-manager/index.md)** for credentials and configuration values that should never appear in execution records.
- **Log carefully.** Only `SECRET` inputs are automatically masked in logs. Any sensitive value logged directly — via `print`, `echo`, or a logging statement — is stored as plaintext in the logs table.
### Database maintenance
Use [purge tasks](../../10.administrator-guide/purge/index.md) to free up storage and maintain performance as databases accumulate execution and log data.
To further separate data across business units or environments, see the [governance features](../../07.enterprise/02.governance/index.mdx) in the [Enterprise Edition](../../07.enterprise/01.overview/01.enterprise-edition/index.md), including [tenants](../../07.enterprise/02.governance/tenants/index.md).
---
# Deployment Architectures in Kestra: JDBC and Kafka
URL: https://kestra.io/docs/architecture/deployment-architecture
> Choose your Kestra deployment architecture. Compare Standalone (JDBC), Medium (Database), and High-Availability (Kafka & Elasticsearch) models.
Kestra is a Java application distributed as an executable. It supports multiple deployment options:
- [Docker](../../02.installation/02.docker/index.md)
- [Kubernetes](../../02.installation/03.kubernetes/index.md)
- Manual deployment
Kestra's plugin system allows you to choose the dependency types that best match your requirements.
## Small-sized deployment

For small-scale deployments, you can use the Kestra **standalone server**, which runs all server components as threads inside a single process. This architecture has no scaling capability but behaves identically to a distributed cluster — the same components run, collocated in one JVM.
For quick local experimentation, Kestra also offers a **local mode** (`server local`) that reduces the footprint further: it uses an embedded H2 database with no external dependencies, requiring no infrastructure setup.
In standalone mode, a database is the only dependency. Supported databases include:
- PostgreSQL
- MySQL
- H2
## Medium-sized deployment

For medium-scale deployments where high availability is not required, Kestra can be run with a relational database (PostgreSQL or MySQL) as the only dependency. H2 is not recommended in distributed setups.
- Supported databases: PostgreSQL and MySQL
- All server components communicate through the database queue
- Each server role runs as its own process and can be scaled independently
- Workers communicate with the Worker Controller via gRPC; they never access the queue or database directly
If components are distributed across multiple hosts, use a shared [internal storage](../data-components/index.md#internal-storage) implementation such as [Google Cloud Storage](../../02.installation/09.gcp-vm/index.md), [AWS S3](../../02.installation/08.aws-ec2/index.md), or [Azure Blob Storage](../../02.installation/10.azure-vm/index.md).
## High-availability deployment

For high throughput and full horizontal and vertical scaling, replace the database queue with Kafka and Elasticsearch. This architecture removes single points of failure and enables scaling of all server components.
- Dependencies: Kafka and Elasticsearch
- Available only in the [Enterprise Edition](../../07.enterprise/01.overview/01.enterprise-edition/index.md)
As with medium deployments, a distributed [internal storage](../data-components/index.md#internal-storage) solution is required if components run on different hosts.
### Kafka
[Kafka](https://kafka.apache.org/) is the queue backbone of the high-availability deployment. The Executor, Scheduler, Worker Controller, Webserver, and Indexer emit to and subscribe from named Kafka topics — no two roles call each other directly.
Workers do not subscribe to Kafka topics. They connect to the Worker Controller via gRPC, and all job dispatch, result intake, and broadcast events travel over that stream.
Executors scale horizontally — each instance subscribes to the queue and processes the executions assigned to it. Because the executor performs lightweight orchestration work (state transitions, dispatch decisions), it typically requires minimal resources.
### Elasticsearch
[Elasticsearch](https://www.elastic.co/elasticsearch) acts as the search and read backend for Kestra's webserver, providing fast retrieval and aggregation of flows, executions, and logs. It is used exclusively by the API and UI.
The Indexer subscribes to Kafka topics and writes to Elasticsearch, keeping the search index in sync. Because the queue and search index are separate, executions continue processing even if Elasticsearch is temporarily unavailable.
---
# Main Components of Kestra Architecture
URL: https://kestra.io/docs/architecture/main-components
> Understand Kestra's core architecture. Dive into main components like the Repository, Queue, Internal Storage, and Plugin system.
Kestra relies on the following internal components:
- **Internal storage**: stores flow data such as task outputs and flow inputs.
- **Queue**: enables internal communication between Kestra server components.
- **Repository**: persists flows, executions, logs, and all other internal objects.
- **Plugins**: extend Kestra’s core with additional task and trigger types, storage implementations, and data transformations.
Each component has multiple implementations depending on deployment architecture. Some require additional plugins.
## Internal storage
The **internal storage** is a dedicated system that handles files of any size during flow executions. It manages both inputs and outputs, enabling scalable file sharing between tasks.
### Purpose
Internal storage is used to:
- Save files generated during a [flow’s execution](../../05.workflow-components/03.execution/index.md) and pass them between tasks via [outputs](../../05.workflow-components/06.outputs/index.md).
- Automatically persist [flow inputs](../../05.workflow-components/05.inputs/index.md) of type `FILE`.
- Provide download links for stored files in the **Outputs** tab of an execution.
Files are addressed by stable `kestra://` URIs that the engine resolves on demand. A worker on one host and the Webserver on another can both reach the same artifact through the same URI. Files can be retrieved in the execution context using `{{ outputs.task_id.output_attribute }}` (often the `uri` property). Kestra fetches the file automatically when referenced.
Execution metadata — including storage file paths — is recorded in the **repository**.
### Storage types
By default, Kestra uses **local storage**, which stores files on the host filesystem. Local storage is not recommended for production distributed deployments — use cloud object storage or a self-hosted alternative instead.
:::alert{type="warning"}
Local storage behavior differs between standalone and distributed deployments:
- ✅ **Standalone**: Local storage with persistent volumes is OK
- ❌ **Distributed on single-writer storage**: NOT recommended — storage that only one instance can mount read-write at a time (in Kubernetes, `ReadWriteOnce`) cannot back a distributed service
- ✅ **Distributed on shared storage**: OK when all instances can mount it read-write at once (in Kubernetes, `ReadWriteMany`), though this is rarely available
- ❌ **Host storage sharing**: NOT recommended — difficult to achieve reliably
When shared read-write storage (`ReadWriteMany`) is unavailable, use cloud storage (S3, GCS, Azure) or self-hosted S3-compatible object storage (Ceph, SeaweedFS, Garage, MinIO).
:::
Scalable alternatives are available as plugins:
- [Storage MinIO](https://github.com/kestra-io/storage-minio) — for [AWS S3](https://aws.amazon.com/s3/) and any S3-compatible object storage (Ceph, SeaweedFS, Garage, MinIO).
- [Storage GCS](https://github.com/kestra-io/storage-gcs) — for [Google Cloud Storage](https://cloud.google.com/storage).
- [Storage Azure](https://github.com/kestra-io/storage-azure) — for [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/).
- [Storage Cloudflare](https://github.com/kestra-io/storage-cloudflare) — for [Cloudflare R2](https://www.cloudflare.com/products/r2/).
- [Storage OBS](https://github.com/kestra-io/storage-obs) — for [Huawei Cloud OBS](https://www.huaweicloud.com/en-us/product/obs.html).
For details, see [Runtime and Storage](../../configuration/02.runtime-and-storage/index.md).
## Queue
The **queue** is the internal communication channel between Kestra’s server components. Server roles emit typed messages onto named queues and subscribe to the queues they consume — no role calls another directly. The full queue surface is defined once as an abstract contract, satisfied by one chosen backend:
- **Database queue** (default) — backed by PostgreSQL or MySQL. Available in all editions.
- **In-memory queue** — for testing and ephemeral use only.
- **Kafka queue** — Enterprise Edition. Higher throughput; pairs with the Elasticsearch repository.
- **Redis queue** — Enterprise Edition.
- **AMQP queue** — Enterprise Edition.
- **GCP Pub/Sub queue** — Enterprise Edition.
The queue surface covers four delivery families:
- **Dispatch** — point-to-point; exactly one subscriber processes each message. Used for executions, execution events and commands, worker task results, logs, and metrics.
- **Keyed dispatch** — point-to-point partitioned by a routing key so a subscriber receives only messages for its key. Used for worker job routing: each Worker Queue is a key, and a worker subscribes only to the queues its group covers.
- **VNode dispatch** — sharded across a fixed set of virtual nodes so a scaled component can divide a single logical stream deterministically. Used for trigger evaluation across a Scheduler fleet.
- **Broadcast** — fan-out; every active subscriber receives every message. Used for kill signals, flow and metadata change notifications, follow-execution streams, and cluster-wide events.
Messages above a configurable size limit are rejected before reaching the backend, protecting it from oversized payloads. Terminal execution states are always allowed through regardless of size. Message protection is enabled by default with a 1 MB limit and can be adjusted under `kestra.queue.message-protection`.
## Repository
The **repository** persists all domain entities, including flows, executions, logs, and triggers. The backend is chosen alongside the queue:
- **Database repository** (default) — backed by PostgreSQL, MySQL, or H2. Available in all editions.
- **In-memory repository** — for testing only.
- **Elasticsearch repository** — Enterprise Edition. Backs the high-volume search and read model; requires the Kafka queue and the Indexer server role to keep it in sync.
## Plugins
Kestra’s core only provides basic functionality. A [plugin ecosystem](/plugins) extends the platform with:
- New task and trigger types.
- Alternative implementations of core components (e.g., storage backends).
- Integrations with external systems and data transformation utilities.
A wide range of plugins is already available, and the ecosystem continues to expand.
---
# Multi-Tenancy in Kestra: Tenant Isolation Model
URL: https://kestra.io/docs/architecture/multi-tenancy
> Understand Kestra's Multi-tenancy architecture. Learn how tenant isolation works for flows, data, and resources in a single Enterprise instance.
Multi-tenancy allows you to manage **multiple environments** (e.g., dev, staging, prod) in a single Kestra instance.
Multi-tenancy is a software architecture in which a single instance of software serves multiple tenants. You can think of it as running multiple virtual instances in a single physical instance. You can use multi-tenancy to **separate resources** between business units, teams, or customers.
This feature requires the [Enterprise Edition](../../07.enterprise/index.mdx).
## How does multi-tenancy work in Kestra
Every resource in Kestra belongs to exactly one tenant. The following are fully isolated per tenant:
| Resource | Description |
|---|---|
| [Flows](../../05.workflow-components/01.flow/index.md), [triggers](../../05.workflow-components/07.triggers/index.mdx), [executions](../../05.workflow-components/03.execution/index.md) | Core workflow resources — the same flow ID and namespace can exist independently in multiple tenants |
| [Namespaces](../../07.enterprise/02.governance/07.namespace-management/index.md) | Namespace hierarchy, variables, KV store, namespace files, and task defaults |
| [RBAC](../../07.enterprise/03.auth/rbac/index.md) — roles, users, groups, service accounts | Access control is fully scoped to the tenant |
| [Secrets](../../07.enterprise/02.governance/secrets-manager/index.md) | Secret keys and values are never shared across tenants |
| [Policies](../../07.enterprise/02.governance/policies/index.md) | Governance rules (injection, validation, enforcement) are scoped to tenant and namespace |
| [Worker Queues](../../07.enterprise/04.scalability/worker-group/index.md) | Task routing rules are tenant-scoped |
| [Audit logs](../../07.enterprise/02.governance/06.audit-logs/index.md) | Activity logs are isolated and queryable per tenant |
| [Internal storage](../data-components/index.md#internal-storage) | Execution outputs and task data are stored in tenant-specific paths |
Instance-level resources — configuration, license, static policies, and instance owner banners — sit above the tenant layer and require Instance Owner access.
Users switch between tenants using the tenant dropdown in the bottom-left corner of the UI. The dropdown lists every tenant the user has access to; the active tenant is indicated with a checkmark. Each UI page also includes the tenant ID in the URL (e.g., `https://demo.kestra.io/ui/yourTenantId/executions/namespace/flow/executionId`).

Tenants are created and managed through the **Instance Owner console** (**Instance Owner → Tenants**) — only users with the Instance Owner privilege can create, edit, or delete tenants. Users must be granted access to a tenant before they can switch to it. See [Tenants](../../07.enterprise/02.governance/tenants/index.md) for configuration details.
Most [API](../../api-reference/index.mdx) endpoints are scoped to a tenant and include the tenant identifier in the path — for example, `/api/v1/{tenant_id}/flows/products` to list flows in the `products` namespace. Instance-level endpoints such as `/api/v1/configs` or `/api/v1/license-info` have no tenant segment. See the [Enterprise Edition API Guide](../../api-reference/01.enterprise/index.mdx) for the full reference.
---
# Server Components in Kestra Architecture Explained
URL: https://kestra.io/docs/architecture/server-components
> Explore Kestra server components. Learn about the Executor, Worker, Scheduler, Webserver, and Indexer roles in the orchestration engine.
Kestra runs six server roles that can be deployed as a single process or as independent, separately scaled services. Every role communicates exclusively through the [Queue](../01.main-components/index.md#queue) and reads shared state from the [Repository](../01.main-components/index.md#repository). Workers are the only role that accesses [Internal Storage](../data-components/index.md#internal-storage) and user infrastructure directly.
The Webserver is the only role reachable from outside the cluster. The orchestration roles — Executor, Worker Controller, Scheduler, Indexer, and Workers — cannot be reached from the network; they only produce and consume queue messages. All external actors reach the platform through the Webserver's authenticated API.
## Executor
The **Executor** is a lightweight server component responsible for driving the execution state machine. Given a flow and an execution, it decides which task runs next, what state the execution is in, what to dispatch, what to retry, and when an execution terminates. It does not perform heavy computation itself — runnable tasks are dispatched to [Workers](#worker) via the [Worker Controller](#worker-controller).
The Executor subscribes to the queue and handles:
- [Flowable Tasks](../../05.workflow-components/01.tasks/00.flowable-tasks/index.md)
- [Flow Triggers](../../05.workflow-components/07.triggers/02.flow-trigger/index.md)
- Subflow and loop coordination
- Concurrency limits, retries, SLA monitoring, and kill signals
Before dispatching a task, the Executor consults a worker-queue resolver that returns one of four decisions: `DISPATCH` (send immediately), `WAIT_AND_DISPATCH` (enqueue for a worker not yet connected), `FAIL` (fail the task run), or `CANCEL` (cancel it). The resolver — not the Executor — performs the worker availability check.
A one-second delay loop re-injects executions when their scheduled wakeup arrives, driving task retries, paused-flow resumption, and `LoopUntil` iterations.
The Executor also runs the cluster-wide **service liveness coordinator**: it periodically reviews every registered service instance — workers, schedulers, and peer executors — drives state transitions on missed heartbeats, and releases orphaned worker jobs back to the queue for reprocessing. See [Cluster liveness model](#cluster-liveness-model).
The Executor never interacts directly with user data or infrastructure. Because of its low resource usage, it rarely needs to be scaled. In deployments with very high execution volume, Executors can scale horizontally.
## Worker Controller
The **Worker Controller** is the sole communication point between the cluster and its workers. Workers never subscribe to the job queue or access the database directly — all job dispatch and result intake are funnelled through the Worker Controller.
Each worker opens a persistent bidirectional gRPC stream to a Worker Controller and uses that stream for its lifetime:
- The Worker Controller dispatches jobs from the queue onto the stream.
- Workers return results, logs, and metrics over the same stream.
- Kill signals and metadata changes are broadcast to all connected workers.
Dispatch is partitioned by **Worker Queue** — a stable identifier derived from the tag set a task or trigger declares through its worker selector. Two queues are always reserved: the **default queue** carries untagged work; the **system queue** carries platform-internal tasks and is served exclusively by the [system worker](#worker). A worker connects under a **worker group** that maps it to one or more Worker Queue subscriptions. Each subscription carries a reserved-capacity percentage so a busy queue cannot starve one to which the group has committed capacity. In the open-source build, there is a single implicit default group subscribed only to the default queue; in Enterprise Edition, groups are persisted entities with their own authorization tokens.
Before dispatching a job, the Worker Controller writes it to a durable **running state store**. If the controller crashes after persisting but before the worker receives the job, the Executor recovers and re-dispatches from that store. The capacity slot reserved for a dispatched job is held for its entire lifetime — not just until delivery — so the permit count accurately reflects in-flight load.
Multiple Worker Controller instances can run in parallel. Workers discover available controllers through static endpoint lists, DNS, or self-registration in internal storage. The controller periodically recycles long-lived streams so that newly deployed controller instances pick up traffic without requiring worker restarts.
gRPC transport is available in all editions. TLS and mTLS secure the connection in all editions; JWT-based worker authentication is an Enterprise Edition feature.
## Worker
The **Worker** is the server component responsible for executing all [runnable tasks](../../05.workflow-components/01.tasks/01.runnable-tasks/index.md) and [Polling Triggers](../../05.workflow-components/07.triggers/04.polling-trigger/index.md). Workers are the only roles that load user plugins, access user infrastructure, and consume CPU on user code.
Workers come in two shapes:
- **Worker agent** — a dedicated process that connects to a Worker Controller over gRPC. This is the standard deployment unit.
- **System worker** — an in-process variant embedded in the Executor (or the standalone server). It serves the reserved system queue for platform-internal tasks and starts automatically; it does not require separate deployment.
Internally, each worker agent runs as a configurable thread pool. Between the gRPC stream fetcher and the thread pool sits a bounded in-memory **buffer queue**. When the buffer fills, the worker stops pulling new jobs from the stream, letting the distributed queue's lag metric reflect a saturated worker rather than silently overloading it. Set the thread count based on your workload — more threads for I/O-bound tasks, fewer for memory-intensive ones.
A task can declare a **worker selector**: a set of tags, a match strategy (`all` tags must match, or `any`), and a fallback policy for when no matching worker is available (`fail`, `wait`, `cancel`, or fall back to the default queue). The Worker Controller routes the task to the Worker Queue whose tags match the selector.
Workers optionally support a **task output cache**: task outputs are stored in internal storage keyed by a hash of the task definition and its inputs. On a cache hit, the worker emits the cached outputs without running the task.
Worker shutdown is two-phase: the worker stops fetching new jobs and waits up to the configured grace period for in-flight jobs to finish, then drains the outbound result senders before exiting.
Deploy multiple worker agent instances to scale horizontally. Each handles its assigned tasks independently, so adding workers increases throughput without coordination overhead.
:::alert{type="info"}
Looking for runtime status? The **Instance – Services** view shows live health for each component. See [Instance – services](../../07.enterprise/05.instance/index.mdx#services).
:::
## Worker Group (EE)
In the [Enterprise Edition](../../07.enterprise/01.overview/01.enterprise-edition/index.md), [Worker Groups](../../07.enterprise/04.scalability/worker-group/index.md) are persisted entities that define which Worker Queues a connected worker serves and how much of its capacity each queue is guaranteed. Each subscription in a group pairs a Worker Queue with a reserved-capacity percentage and a reservation mode — **strict** (reserved slots are exclusive to that queue) or **elastic** (reserved slots can be borrowed by other subscriptions when idle).
Worker Groups enable scenarios such as GPU workloads, OS-specific execution, infrastructure access restrictions, and region-based routing. Every worker agent connects under a group; the group determines its Worker Queue subscriptions and carries its own authorization tokens.
To route a task to a specific queue, set `workerSelector.tags` on the task definition with the tags matching the target Worker Queue. Tasks without a `workerSelector` run on the default queue.
:::alert{type="info"}
Worker Groups are available in Kestra Enterprise Edition only, not in Kestra Cloud.
:::
## Scheduler
The **Scheduler** is a server component responsible for managing all [triggers](../../05.workflow-components/07.triggers/index.mdx) — except for [Flow Triggers](../../05.workflow-components/07.triggers/02.flow-trigger/index.md), which are handled by the [Executor](#executor).
The Scheduler continuously evaluates trigger conditions and determines when a flow should start. When a trigger is satisfied, the Scheduler submits the flow to the Executor for execution.
For [Polling Triggers](../../05.workflow-components/07.triggers/04.polling-trigger/index.md), the Scheduler checks them at their configured evaluation interval. If the polling conditions are met, it sends the execution — along with trigger metadata — to the [Worker](#worker) for execution.
Polling Triggers have specific constraints:
- They cannot be evaluated concurrently.
- They cannot be reevaluated while a previous execution from the same trigger is still running.
Internally, the Scheduler checks every second to determine whether any trigger needs evaluation.
:::alert{type="info"}
By default, Kestra handles all date and time values using your system's timezone. You can override this behavior using [JVM options](../../configuration/02.runtime-and-storage/index.md)
:::
## Indexer
The **Indexer** reads content from the queue — flows, executions, logs, metrics, and audit logs — and writes it to the repository backend. It is required in all deployments.
By default, the Indexer runs embedded in the [Webserver](#webserver). You can disable it there with the `server webserver --no-indexer` CLI option and run it as a standalone process — useful when the Webserver and Indexer need access to different infrastructure, or when you want to isolate the Webserver behind a bastion host.
## Webserver
The **Webserver** is the entry point for all external communications with Kestra. It is responsible for serving both the [User Interface (UI)](../../09.ui/index.mdx) and the [REST API](../../api-reference/index.mdx).
It consists of two main modules:
- **API**: Exposes all [REST endpoints](../../api-reference/index.mdx) for interacting with Kestra — including triggering executions, retrieving flow data, managing tasks, and more.
- **UI**: Serves the [Kestra web interface](../../09.ui/index.mdx), enabling users to design, monitor, and manage workflows visually.
The Webserver primarily interacts with the [Repository](../01.main-components/index.md#repository) to serve content through the API and UI. It also connects to the [Queue](../01.main-components/index.md#queue) to submit new executions and stream real-time updates on flow progress.
:::alert{type="info"}
As long as the [Queue](../01.main-components/index.md#queue) is operational, most server components — including the Webserver — will continue to function. While the Repository is essential for rendering the UI, workloads can still be processed even if the Repository is temporarily unavailable.
:::
## Cluster liveness model
Every running server registers itself as a service and sends heartbeats at a fixed interval. The Executor runs the cluster-wide liveness coordinator: on a scheduled tick it reviews every registered service instance, drives state transitions when heartbeats are missed (`running → disconnected → not-running`), and releases any work orphaned by a vanished server — such as a worker's in-flight jobs — back onto the queue so a healthy server picks it up.
The same model coordinates **maintenance mode**: on maintenance entry, every server pauses its queue subscribers while in-flight work drains, then resumes when maintenance exits. Old service rows are purged on a schedule to keep the registry bounded.
---
# Workflow Best Practices in Kestra: Design & Patterns
URL: https://kestra.io/docs/best-practices
> Best practices for building reliable workflows in Kestra, including guidance on choosing the right patterns and making sound design decisions.
import ChildCard from "~/components/docs/ChildCard.astro"
Best practices for building reliable workflows in Kestra, including guidance on choosing the right patterns for common design decisions.
## Apply best practices for Kestra workflows
Kestra often provides multiple ways to achieve the same outcome. This section helps you choose the approach that best fits your use case, with guidance on design decisions, implementation patterns, and tradeoffs.
---
# Business Unit Separation in Kestra Enterprise
URL: https://kestra.io/docs/best-practices/business-unit-separation
> Strategies for isolating business units in Kestra Enterprise using Tenants and Namespaces for security and governance.
Kestra Enterprise provides two primary levels of isolation within an instance: tenants and namespaces.
## Separate business units with tenants and namespaces
Choosing between tenants and namespaces for separating business units depends on the required level of data isolation, access control, and visibility into cross-workflow dependencies.
## When to use multiple tenants
A [tenant](../../07.enterprise/02.governance/tenants/index.md) is an **isolated environment** within a Kestra instance. Tenants have their own **fully isolated resources**, including flows, RBAC policies, secrets, variables, plugin defaults, and more. Users exist globally across the instance but can have different roles and permissions per tenant.
You can configure dedicated resources for each tenant:
- Dedicated internal storage (e.g., a separate S3 bucket per tenant)
- Dedicated secrets manager backend (e.g., a separate Vault or AWS Secrets Manager per tenant)
- Dedicated worker groups (e.g., a pool of workers used exclusively by a specific tenant)
- Flows, executions, and logs are **isolated** between tenants by default
:::alert{type="info"}
Worker groups are not yet available in Kestra Cloud; they are supported only in Kestra Enterprise Edition.
:::
### Use cases for tenants
- **Customer separation**: For multi-tenant SaaS setups requiring strict isolation between customers, assign each customer its own tenant rather than a dedicated namespace.
- **Fully isolated teams**: When even administrators should not have visibility into other teams' workflows, tenants provide the highest level of isolation.
**Note:** Since tenants are **fully isolated**, there is no cross-tenant visibility. If you need to share flows (e.g., team A runs a subflow from team B) or manage interdependent workflows, namespaces are a better option.
## When to use multiple namespaces
A [namespace](../../07.enterprise/02.governance/07.namespace-management/index.md) is a logical grouping within a tenant that organizes teams and projects while providing fine-grained access control. Unlike tenants, namespaces allow cross-visibility of flows and dependencies.
You can configure dedicated resources for each namespace:
- Dedicated internal storage (e.g., a separate S3 bucket per namespace)
- Dedicated secrets manager backend (e.g., a separate Vault or AWS Secrets Manager per namespace)
- Dedicated worker groups (e.g., a pool of workers used exclusively by a specific namespace)
- Flows, executions, and logs are **shared** across namespaces; isolation is managed through RBAC permissions
### Use cases for namespaces
- **Team-based organization**: Separate flows and resources by team within the same tenant, maintaining visibility for users with appropriate permissions.
- **Project-based organization**: Create separate namespaces for projects that need limited isolation while retaining workflow visibility.
- **Dependency management**: Namespaces support cross-team dependencies (e.g., subflows or triggers), simplifying dependency tracking.
- **RBAC control**: Namespaces allow granular role-based access. A user might have view-only access in one namespace and full management permissions in another.
## Summary of when to use tenants vs. namespaces
| Feature | Tenant | Namespace |
|------------------------------|------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|
| **Full isolation** | ✅ Yes | ❌ No (unless configured via RBAC) |
| **Cross-visibility** | ❌ No (tenants don't share flows) | ✅ Yes (namespaces can share flows and dependencies) |
| **RBAC control** | ✅ Yes (separate roles per tenant) | ✅ Yes (roles can be restricted to namespaces) |
| **Secrets manager backend** | ✅ Optional dedicated backend per tenant | ✅ Optional dedicated backend per namespace |
| **Internal storage backend** | ✅ Optional dedicated backend per tenant | ✅ Optional dedicated backend per namespace |
| **Best for** | Customer separation or strict business unit isolation | Team or project isolation with centralized governance and shared dependencies |
## Recommendations
- Use **tenants** when strict isolation is required, and no cross-team dependencies exist.
- Use **namespaces** to organize business units or teams that require centralized governance and cross-team visibility.
- Configure **dedicated secrets and internal storage backends** at the lowest necessary level (tenant or namespace) to follow the principle of least privilege and simplify management (e.g., applying S3 lifecycle policies per team).
- Prefer **namespace isolation** over tenant isolation when workflows depend on each other to maintain visibility across dependencies.
For a hands-on guide on how to use RBAC to separate business units at the namespace level, see the video below:
---
# Choosing Where to Store Sensitive and Shared Values
URL: https://kestra.io/docs/best-practices/credentials-vs-secrets-vs-kv-store
> Learn how to choose between Kestra credentials, secrets, and the KV Store for authentication, sensitive values, and runtime state.
How to choose between credentials, secrets, and the KV Store for sensitive values, authentication, and shared state in Kestra.
## Choose the right place for sensitive values and shared state
Kestra gives you multiple ways to manage authentication and sensitive values:
- [credentials](../../07.enterprise/03.auth/credentials/index.md)
- [secrets](../../06.concepts/04.secret/index.md)
- the [KV Store](../../06.concepts/05.kv-store/index.md)
These options serve different purposes. Choosing the right one improves security, reduces duplication, and makes workflows easier to maintain.
## Quick recommendation
Use this rule of thumb:
- Use **credentials** for reusable server-to-server authentication where Kestra should mint or refresh short-lived access tokens for you.
- Use **secrets** for protected static values such as API keys, passwords, client secrets, private keys, and connection strings.
- Use the **KV Store** for dynamic values that must be created, updated, or read at runtime across executions or flows.
## Comparison table
| If you need to store... | Prefer | Why |
| --- | --- | --- |
| A reusable OAuth2 or GitHub App authentication object | Credentials | Kestra can mint and refresh access tokens at runtime |
| A password, API key, client secret, private key, or connection string | Secrets | Sensitive static values should be protected and not stored in flow code |
| A value created by one flow and reused later by another flow | KV Store | It is designed for runtime state shared across flows |
| A value that changes during execution and must be updated programmatically | KV Store | Flows can read and write KV pairs dynamically |
| Sensitive material used by a credential, such as a client secret or private key | Secrets | Credentials should reference secrets rather than embed raw secret values |
| A non-sensitive setting such as region, endpoint, or bucket name | Task properties, [variables](../../05.workflow-components/04.variables/index.md), or [Policies](../../07.enterprise/02.governance/policies/index.md) (EE) | These are configuration values, not authentication objects |
## When to use credentials
Use credentials when you need a reusable server-to-server authentication object that Kestra manages for you.
Credentials are designed for authentication patterns such as:
- OAuth2 `client_credentials`
- OAuth2 JWT Bearer
- OAuth2 `private_key_jwt`
- GitHub App authentication
Why credentials are the right choice here:
- They let you define the authentication object once and reuse it across tasks.
- Kestra can mint and refresh access tokens at runtime.
- Workflows only need the current access token via `{{ credential('name') }}`.
- Sensitive material used by the credential can stay in secrets.
Best practice:
- Use credentials for managed authentication objects, not as a generic replacement for secrets.
- Scope credentials at the namespace or [tenant](../../07.enterprise/02.governance/tenants/index.md) level based on the required reuse. For multi-team setups, align that scope with your [namespace management](../../07.enterprise/02.governance/07.namespace-management/index.md) model.
- Reference secrets inside credentials for client secrets, private keys, or certificates.
For more details, see [Credentials](../../07.enterprise/03.auth/credentials/index.md).
## When to use secrets
Use secrets when the value is sensitive and should not be committed to source control or exposed broadly in flow definitions.
Typical examples:
- database passwords
- API keys
- cloud access keys
- OAuth client secrets
- SSH private keys
- service account JSON
- webhook secrets
Why secrets are the default for sensitive values:
- They keep sensitive data out of flow YAML.
- They can be managed centrally, including through an [external secrets manager](../../07.enterprise/02.governance/secrets-manager/index.md).
- They reduce accidental exposure when flows are shared across teams.
- They are the right source for secret values referenced by credentials and task properties.
Best practice:
- Store secrets at the lowest [namespace](../../07.enterprise/02.governance/07.namespace-management/index.md) level that still supports the required reuse.
- Avoid placing broadly scoped secrets at the root [namespace](../../07.enterprise/02.governance/07.namespace-management/index.md) unless they truly need to be inherited everywhere. If you need inheritance without allowing downstream edits, consider [read-only secrets](../../07.enterprise/02.governance/secrets-manager/index.md).
- Avoid logging secrets or transforming them in ways that could bypass masking.
For more details, see [Secrets](../../06.concepts/04.secret/index.md) and [Best Practices for Secrets in Kestra](../9.secrets-management/index.md).
## When to use the KV Store
Use the KV Store for dynamic state, not for secret management.
The KV Store is a good fit when a value:
- is created during an execution
- must be updated by flows
- needs to be read later by other flows or later executions
- represents state rather than static configuration
Typical examples:
- checkpoints
- cursors and offsets
- last processed timestamp
- feature flags or runtime switches managed by workflows
- identifiers of external resources created by one execution and reused later
Why the KV Store is usually the wrong place for secrets:
- Secrets and credentials are usually static or centrally managed, while KV pairs are designed for runtime mutation.
- Flows can update KV pairs, which increases the risk of accidental overwrite for sensitive data.
- The KV Store is better suited to application state than authentication material.
If you need a value to be both sensitive and mutable, stop and review the design carefully. In most cases, that indicates a workflow state problem or a secret-lifecycle problem that should be solved more explicitly.
## Recommended patterns
### Pattern 1: Credential plus secrets
Use a credential for the authentication flow and secrets for the sensitive material that backs it.
Example:
- OAuth2 credential stored in Kestra
- `client_secret` or private key referenced from a secret
This is the preferred pattern when Kestra should mint or refresh tokens for you.
### Pattern 2: Secrets plus non-sensitive configuration
Use secrets for the confidential part and task properties or [variables](../../05.workflow-components/04.variables/index.md) for the rest. In Enterprise Edition, use [Policies](../../07.enterprise/02.governance/policies/index.md) to inject repeated non-sensitive settings across all tasks of a type.
Example:
- `password` from `secret('DB_PASSWORD')`
- `host`, `port`, and `database` from [variables](../../05.workflow-components/04.variables/index.md) or task properties
### Pattern 3: Secret plus KV Store
Use secrets for authentication and the KV Store for runtime state.
Example:
- API key from a secret
- `last_processed_cursor` from the KV Store
This is common in [polling triggers](../../05.workflow-components/07.triggers/04.polling-trigger/index.md), ingestion, and synchronization flows.
### Pattern 4: Policies plus secrets (Enterprise Edition)
Use [Policies](../../07.enterprise/02.governance/policies/index.md) to inject repeated connection settings into all tasks of a given type, while referencing secrets for the sensitive fields.
This is often the cleanest approach for large teams because it reduces duplication without putting secret material in the flow body.
## Anti-patterns
Avoid these patterns:
- Storing passwords, API keys, or client secrets directly in flow YAML
- Using secrets where a managed credential would better handle token minting and refresh
- Using the KV Store as the default place for secret material
- Putting broad secrets at the root namespace when only one team or project needs them
- Mixing static configuration, secret material, and mutable runtime state without a clear reason
## Decision guide
Ask these questions:
1. Is the value sensitive?
If yes, start with **secrets**.
2. Do you need Kestra to mint or refresh an access token for a supported authentication flow?
If yes, use **credentials**, backed by **secrets** for the sensitive inputs.
3. Does the value need to be updated dynamically by flows?
If yes, consider the **KV Store**.
4. Is the value just a protected static value such as an API key, password, or private key?
If yes, use **secrets**.
5. Is the value stable non-sensitive configuration reused across many tasks?
If yes, consider [**Policies**](../../07.enterprise/02.governance/policies/index.md) (Enterprise Edition), [variables](../../05.workflow-components/04.variables/index.md), or task properties.
## Summary
- **Credentials** are for reusable managed authentication objects that mint or refresh tokens.
- **Secrets** are for sensitive static values and for secret inputs referenced by credentials.
- **KV Store** is for dynamic runtime state shared across flows or executions.
In most cases, the right answer is not one feature alone, but a combination:
- credentials for token-based authentication
- secrets for sensitive inputs
- [Policies](../../07.enterprise/02.governance/policies/index.md) (Enterprise Edition) or [variables](../../05.workflow-components/04.variables/index.md) for non-sensitive configuration
- KV Store for changing state
---
# Expressions with Namespace Files in Kestra
URL: https://kestra.io/docs/best-practices/expressions-with-namespace-files
> Learn how to pass Kestra expressions to scripts stored in Namespace Files using environment variables or CLI arguments.
Learn how to pass expressions to Namespace Files.
## Use expressions with namespace files
You can write scripts inline in your flow and include expressions within them. The following example shows a flow that contains an expression in the inline script:
```yaml
id: expressions_inline
namespace: company.team
inputs:
- id: uri
type: URI
defaults: https://www.google.com/
tasks:
- id: inline_script
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/pydata:latest
script: |
import requests
url = "{{ inputs.uri }}"
response = requests.get(url)
if response.status_code == 200:
print(response.text)
else:
print(f"Failed to retrieve the webpage. Status code: {response.status_code}")
```
This approach is convenient for scripts specific to a flow, but it does not allow the use of separate files for your code. Using separate files has several benefits:
- Multiple files can be used, which is common in larger projects.
- Long code blocks are easier to maintain when separated from the workflow.
- Files can be written and tested locally, then synced to Kestra through Git.
- The same files can be reused across multiple flows, avoiding code duplication.
You cannot directly use [expressions](../../expressions/index.mdx) inside [Namespace Files](../../06.concepts/02.namespace-files/index.md), as they will not be rendered or executed outside of Kestra. With that being said, you can use the combination of the `render()` and `read()` functions in script tasks like in the following example (see the `render()` function [migration guide](../../11.migration-guide/v0.14.0/recursive-rendering/index.md)):
```yaml
id: expression_render_example
namespace: company.team
inputs:
- id: param1
type: STRING
defaults: myInput
tasks:
- id: hello
type: io.kestra.plugin.scripts.python.Script
script: "{{ render(read('main.py')) }}"
```
With a Python namespace file using:
```python
print("Here is my input displayed using expression in the python script: {{ inputs.param1 }}")
```
The expressions will be rendered in the logs. However, while possible, this is not necessarily best practice, as the script would only work inside Kestra. The following two methods are recommended best practice for passing expressions to your code:
1. [Using Namespace Files with environment variables](#using-namespace-files-with-environment-variables)
2. [Using Namespace Files with CLI arguments](#using-namespace-files-with-cli-arguments)
In either case, you need to add your code as a [Namespace File](../../06.concepts/02.namespace-files/index.md). You can do this using the [Editor](../../09.ui/01.flows/index.md) or by importing it directly.

## Using namespace files with environment variables
You can pass inputs as environment variables using expressions.
The following example uses the input `uri` and passes it to the task `code` as an environment variable so the Python code can access it:
```yaml
id: expressions_env_vars
namespace: company.team
inputs:
- id: uri
type: URI
defaults: https://www.google.com/
tasks:
- id: code
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/pydata:latest
commands:
- python main.py
env:
URI: "{{ inputs.uri }}"
```
Inside the Python code, use `os.environ` to retrieve the environment variable:
```python
import requests
import os
## Perform the GET request
response = requests.get(os.environ['URI'])
## Check if the request was successful
if response.status_code == 200:
print(response.text)
else:
print(f"Failed to retrieve the webpage. Status code: {response.status_code}")
```
This method keeps the code in a separate file while avoiding the maintenance challenges of long inline scripts.
## Using namespace files with CLI arguments
You can also pass arguments directly to your code during execution. In Python, the `argparse` module can be used to handle these arguments.
First, modify your code to accept arguments as follows:
```python
import argparse
import requests
## Set up command-line argument parsing
parser = argparse.ArgumentParser(description="Fetch the content of a given URL")
parser.add_argument("url", type=str, help="The URL to fetch")
args = parser.parse_args()
## Perform the GET request
response = requests.get(args.url)
## Check if the request was successful
if response.status_code == 200:
print(response.text)
else:
print(f"Failed to retrieve the webpage. Status code: {response.status_code}")
```
Next, pass the arguments to your code using expressions. The expressions will be rendered, and the evaluated values passed to the script via `argparse`:
```yaml
id: expressions_argparse
namespace: company.team
inputs:
- id: uri
type: URI
defaults: https://www.google.com/
tasks:
- id: hello
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/pydata:latest
commands:
- python main.py "{{ inputs.uri }}"
```
This method makes the code slightly longer due to the argument-handling logic, but it offers better reusability. The same script can be used in multiple flows without duplicating code.
---
# Choose the Right Fetch Pattern in Kestra
URL: https://kestra.io/docs/best-practices/fetch-patterns
> Learn when to use Download, HTTP Request, or script-based fetching in Kestra, and how to choose the right pattern for files, APIs, and custom integrations.
Choose the simplest fetch pattern that matches the shape, size, and the amount of control you need.
## Decision guide
Use `Download` when:
- You need to retrieve a file over HTTP or HTTPS.
- The result is naturally a file, such as CSV, JSON, ZIP, or a binary artifact.
- The result is a large payload.
- You want the response body streamed directly to Kestra internal storage.
- Downstream tasks should consume a file URI rather than an in-memory response body.
Use `Request` when:
- You need to call an HTTP API and inspect the response directly.
- You need to work with status codes, headers, form data, JSON payloads, or authentication options.
- The response is small enough to treat as task output, less than 10 MB.
- You are orchestrating an API call, not implementing a full client.
Use a script task when:
- You need custom pagination, signing, retries, throttling, or SDK-specific logic.
- You need to combine multiple requests before producing one output.
- The integration logic is complex enough that inline HTTP task configuration becomes harder to maintain than code.
- You want to write a file after custom processing and pass that file to downstream tasks.
:::alert{type="info"}
Prefer built-in tasks first. Use scripts when you genuinely need custom fetching logic, not just because HTTP can also be called from Python or Shell.
:::
## Start with dedicated plugins
If Kestra already provides a plugin for the system you want to integrate with, prefer that plugin over generic HTTP tasks or scripts.
Dedicated plugins are usually the best choice because they:
- model the external system more clearly
- reduce custom request construction and parsing
- make flows easier to read and maintain
- often provide better task outputs and task-specific validation
Use `Request` or script-based fetching when:
- no dedicated plugin exists
- the plugin does not yet support the endpoint or behavior you need
- you need temporary or highly custom integration logic
If a dedicated plugin exists but does not meet your needs, open a [GitHub issue](https://github.com/kestra-io) on that plugin repository rather than silently replacing it with long-term custom code. That helps improve the plugin for the next user facing the same limitation.
## Quick rule of thumb
- If you are fetching a file, start with `Download`.
- If you are calling an API endpoint, start with `Request`.
- If you need custom client behavior, move to a script task.
## Comparison table
| If your goal is... | Prefer | Why |
| --- | --- | --- |
| Download a file and store it for downstream tasks | [`Download`](/plugins/core/http/io.kestra.plugin.core.http.download) | It streams the response body directly to internal storage and returns a file URI |
| Call an API and inspect status, headers, or body | [`Request`](/plugins/core/http/io.kestra.plugin.core.http.request) | It exposes structured HTTP outputs such as `code`, `headers`, and `body` |
| Implement custom pagination, retries, signing, or SDK logic | A script task from [`plugin-scripts`](/docs/scripts) | It gives you full code control and can still emit files or structured outputs |
## Use `Download` for file-oriented retrieval
`Download` is the best default when the remote resource should be treated as a file.
This task is implemented in Kestra's core to stream the response body directly to Kestra internal storage and return a `uri`, `code`, `headers`, and `length`. That makes it a good fit for file ingestion pipelines.
Best practice:
- Use `Download` when the next task expects a file URI.
- Prefer it over `Request` for large payloads or binary files.
- Let downstream conversion or processing tasks read the file from internal storage.
### Example: download a CSV file and convert it
```yaml
id: fetch_with_download
namespace: company.team
tasks:
- id: download_orders
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: csv_to_ion
type: io.kestra.plugin.serdes.csv.CsvToIon
from: "{{ outputs.download_orders.uri }}"
- id: log_download_status
type: io.kestra.plugin.core.log.Log
message: "Downloaded {{ outputs.download_orders.length }} bytes with status {{ outputs.download_orders.code }}"
```
Use this pattern when the remote system is serving a file and your workflow should continue from a persisted file URI.
## Use `Request` for API-oriented retrieval
`Request` is the right fit when you need HTTP semantics, not just file retrieval.
This task is implemented in Kestra's core as a generic HTTP client. It is designed to render URLs, headers, request bodies, form data, and auth options, then expose structured outputs such as response code, headers, and body.
Best practice:
- Use `Request` for JSON APIs, form posts, or authenticated HTTP endpoints.
- Keep response bodies small enough to work with as task output.
- Prefer dedicated plugins when one exists for the external system.
- Prefer `Download` instead when the response is really a file or a large payload.
### Example: call an API and use the JSON response
```yaml
id: fetch_with_request
namespace: company.team
tasks:
- id: api
type: io.kestra.plugin.core.http.Request
uri: https://dummyjson.com/products
method: GET
- id: log_status
type: io.kestra.plugin.core.log.Log
message: "API status: {{ outputs.api.code }}"
- id: extract_first_product
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.api.body | jq('.products[0].title') }}"
```
Use this pattern when the API response itself is the thing you want to orchestrate around.
## Use script-based fetching for custom integration logic
[Script tasks](../../16.scripts/index.mdx) are the right choice when the fetch logic starts to look like application code.
This is especially true for:
- multi-step authentication
- cursor-based or token-based pagination
- API-specific retry or backoff behavior
- combining multiple responses into one normalized output
Best practice:
- Keep orchestration in YAML and only move the integration-specific logic into code.
- Emit files with `outputFiles` when the result should be persisted and reused.
- Use explicit dependencies rather than assuming packages are already installed.
- Do not hide simple one-request logic inside a script when `Request` is enough.
### Example: fetch paginated API data with Python and emit one file
```yaml
id: fetch_with_script
namespace: company.team
tasks:
- id: fetch_all_products
type: io.kestra.plugin.scripts.python.Script
containerImage: python:3.13-slim
dependencies:
- requests
outputFiles:
- all_products.json
script: |
import json
import requests
all_products = []
limit = 10
skip = 0
while True:
response = requests.get(
"https://dummyjson.com/products",
params={"limit": limit, "skip": skip},
timeout=30
)
response.raise_for_status()
payload = response.json()
products = payload.get("products", [])
if not products:
break
all_products.extend(products)
skip += limit
if skip >= payload.get("total", 0):
break
with open("all_products.json", "w") as f:
json.dump(all_products, f)
- id: preview_output
type: io.kestra.plugin.core.log.Log
message: "{{ read(outputs.fetch_all_products.outputFiles['all_products.json']) }}"
```
Use this pattern when your fetch logic needs real code, but the workflow should still continue from a persisted file in Kestra.
## `Download` vs `Request`
Choose between them based on the shape of the response:
- If the response should be treated as a file, use `Download`.
- If the response should be treated as HTTP output data, use `Request`.
For example:
- fetching `orders.csv` for downstream conversion is a `Download` use case
- calling `/products` and branching on the JSON payload is a `Request` use case
## `Request` vs script-based fetching
Choose between them based on how much client logic you need.
Use `Request` when:
- one request is enough
- auth and headers are straightforward
- the response can be handled directly with outputs and expressions
Use a script when:
- you need loops, pagination, or multiple dependent requests
- the external API requires custom signing or SDK usage
- you want to normalize or aggregate responses before downstream tasks
Rule of thumb:
- `Request` is for orchestrating API calls.
- A script is for implementing API client behavior.
## Anti-patterns
Avoid these patterns:
- using a Python or Shell script just to download one file from a public URL
- using `Request` for very large file downloads that should go straight to storage
- storing large API payloads in task outputs when a file-based pattern would scale better
- hiding complex integration logic in a long inline script without clear output artifacts
- skipping dedicated plugins when they already solve the target integration cleanly
## Recommended patterns
### Pattern 1: File ingestion
Use `Download`, then pass the returned URI to a conversion or processing task.
### Pattern 2: API orchestration
Use `Request`, then branch, transform, or log based on `code`, `headers`, or `body`.
### Pattern 3: Custom fetch client
Use a script task, then persist the result with `outputFiles` so downstream tasks still operate on Kestra-managed files.
## Summary
- Start with built-in tasks.
- Use `Download` for file retrieval.
- Use `Request` for API calls and structured HTTP responses.
- Use script-based fetching only when you need custom client logic that built-in tasks do not provide.
For task-level details, see [`Download`](/plugins/core/http/io.kestra.plugin.core.http.download), [`Request`](/plugins/core/http/io.kestra.plugin.core.http.request), and the [`plugin-scripts` documentation](/docs/scripts).
---
# Flow Best Practices: Performance and Reliability
URL: https://kestra.io/docs/best-practices/flows
> Design Kestra flows for optimal performance and reliability by managing task count, data volume, and parallelism.
How to design your workflows for optimal performance.
## Understanding what an execution is in Kestra
A flow execution in Kestra is an object that contains:
- All **TaskRuns** for that flow, each with:
- Their attempts, including:
- Metrics
- State history
- Their outputs
- Their state history
Internally:
- Each TaskRun belongs to the same execution context, which holds all task data for the entire flow.
- The Kestra Executor reads each TaskRun’s status changes (typically three per task: **CREATED**, **RUNNING**, and **SUCCESS**).
- For each state transition, the Executor must:
- Fetch the serialized flow execution context over the network.
- Deserialize it, determine the next task(s), then reserialize the context.
- Send the updated serialized context back over the network.
The larger the flow execution context, the longer the serialization process takes.
Depending on the internal queue and repository implementation, there may be a hard limit on the size of this context, since it’s stored as a single row or message (often around **1 MB**). To avoid hitting this limit, do not store large amounts of data inside the flow execution context.
## Tasks in the same execution
While a flow can contain many tasks, it’s not recommended to include a large number of tasks within a single execution.
A flow can contain either manually defined tasks or dynamically generated ones. While the [Loop task](/plugins/core/flow/io.kestra.plugin.core.flow.loop) runs each iteration as an isolated sub-execution, a deeply nested loop can still generate a large number of sub-executions. For example, a nested loop of 20 × 20 tasks results in **400 sub-executions**.
:::alert{type="warning"}
Flows with **over 100 tasks** tend to experience performance degradation and longer execution times.
:::
To avoid this, consider breaking the workflow into subflows using the [Subflow task](../../05.workflow-components/10.subflows/index.md). Since a `Subflow` task creates a new execution, its tasks are **isolated** and do not affect the parent flow’s performance.
## Managing output data volume
Some tasks allow you to fetch outputs from previous tasks and reuse them in subsequent ones.
While powerful, this feature **should not be used to transfer large amounts of data**.
For example, the [Query](/plugins/plugin-gcp/google-cloud-bigquery/io.kestra.plugin.gcp.bigquery.query) task in BigQuery has a `fetch` property that retrieves query results as an output attribute. If the query returns a large dataset, the result will be stored in the execution context — meaning it will be serialized and deserialized on each task state change, severely impacting performance.
This feature is best suited for small datasets, such as querying a few rows to feed into a [Switch](/plugins/core/flow/io.kestra.plugin.core.flow.switch) or [Loop](/plugins/core/flow/io.kestra.plugin.core.flow.loop) task.
:::alert{type="info"}
For large data volumes, use the `stores` property instead. Stored outputs are written to Kestra’s internal storage, and only the file URL is referenced in the execution context.
:::
Some plugins have outputs that include both a `value` and a `uri`. The `store` property for these plugins is set to `false` by default and should typically only be used with small data volumes. This property should be adjusted for larger data volumes to make file URIs available.
When `store` is set to `false` or the default value, the output will include a `value`, which is accessible through a Pebble expression like `"{{ outputs.task.value }}"`. When `store` is set to `true`, `value` is not accessible but instead the file URI is accessible through a Pebble expression like `"{{ outputs.task.uri }}"`. `value` and `uri` are not available outputs at the same time. Trying to access `value` when `store: true` will cause an execution error.
## Parallel tasks
The [Parallel](/plugins/core/flow/io.kestra.plugin.core.flow.parallel) task helps reduce overall flow duration by running multiple branches simultaneously.
By default, **all parallel tasks start at the same time**, unless you set the `concurrent` property. The only limit is the number of worker threads configured in your environment.
Be mindful of external system limits such as API rate restrictions or connection quotas — running too many parallel tasks may overload those systems.
## Task duration
By default, Kestra **does not limit task duration** unless explicitly stated in a task’s documentation. Long-running or infinite processes will continue indefinitely.
You can control task runtime for [Runnable Tasks](../../05.workflow-components/01.tasks/01.runnable-tasks/index.md) using the `timeout` property, which accepts [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) durations (e.g., `PT5M` for five minutes).
This helps prevent stalled executions and ensures resource efficiency.
## Flow trigger on state change
Kestra can automatically start a flow as soon as another flow completes. This makes it easy to create dependencies between flows, even when they are owned by different teams. Use `dependsOn` to declare the upstream flow and the required states:
```yaml
triggers:
- id: release
type: io.kestra.plugin.core.trigger.Flow
dependsOn:
- namespace: company.release
flowId: parent
states: [SUCCESS]
```
`states` defaults to all terminal states and `PAUSED` when omitted. Declare it explicitly when you need a narrower set.
---
# Dev to Production in Kestra: Promote Flows Safely
URL: https://kestra.io/docs/best-practices/from-dev-to-prod
> Recommended patterns for promoting Kestra flows from development to production environments using Git and CI/CD.
Common patterns for deploying flows from development to production environments.
## Move flows from development to production safely
## Development environment
A best practice with Kestra is to maintain a dedicated **development instance** where users can create and test flows safely. This environment acts as a sandbox, allowing experimentation without impacting production or critical business operations.
You can set up a development environment in one of the following ways:
- Install Kestra locally using [Docker Compose](../../02.installation/03.docker-compose/index.md)
- Deploy Kestra on a [Kubernetes cluster](../../02.installation/03.kubernetes/index.md) accessible to users and isolated from production workloads
## Production environment
The production instance should be secured and tightly controlled, as it runs critical workflows that directly impact end users.
A common best practice is to **limit access** to production systems. Two areas should be considered:
- User access
- Flow deployments
### User access
In **Kestra Enterprise**, user management is streamlined through [RBAC](../../07.enterprise/03.auth/rbac/index.md) and [SSO](../../07.enterprise/03.auth/sso/index.md).
Administrators can define fine-grained access using role policies such as *Admin* or *Viewer*, ensuring proper access control across all resources. Learn more in the [Enterprise documentation](../../07.enterprise/index.mdx).
For open-source users, it’s recommended to run a **restricted production instance**, accessible only to administrators and your [CI/CD system](../../version-control-cicd/cicd/index.md).
### Flow deployments
Kestra supports several deployment strategies:
- [Via the UI](../../09.ui/01.flows/index.md)
- [Promote](../../07.enterprise/02.governance/promote/index.md) (Enterprise Edition) — move flows between environments directly from the UI with a diff review and confirmation gate, no Git or pipeline required
- [Git synchronization](../../version-control-cicd/04.git/index.md)
- [CI/CD pipelines](../../version-control-cicd/cicd/index.md)
- [Terraform](../../13.terraform/index.mdx)
- [API](../../api-reference/index.mdx)
Choose a method that aligns with your organization’s existing deployment processes.
A common approach is to move flows from development to production using a **version control system** (such as Git) combined with **CI/CD automation**.
In this pattern, developers commit flow changes to Git. Once the pull request is approved, the CI/CD system deploys the updated flows to the production instance.
Flows can be committed to Git using:
- Manual export or copy-paste from the UI
- The [`git.PushFlows` task](/plugins/plugin-git/io.kestra.plugin.git.pushflows)
CI/CD deployment to production can be automated with:
- GitHub Actions, GitLab CI/CD, Jenkins, or Azure DevOps
- Terraform
- The Kestra CLI
For more details on CI/CD automation, refer to the [CI/CD guide](../../version-control-cicd/cicd/index.md).
## Git example
You can use the [`git.SyncFlows` task](/plugins/plugin-git/io.kestra.plugin.git.syncflows) together with a [trigger](../../05.workflow-components/07.triggers/index.mdx) to automatically pull flows from the `main` branch of your Git repository.
This enables Kestra to manage the synchronization process directly, minimizing the need for external tools.
You can schedule synchronization using:
- A [Schedule trigger](../../05.workflow-components/07.triggers/01.schedule-trigger/index.md) to pull flows at regular intervals (e.g., nightly)
- A [Webhook trigger](../../05.workflow-components/07.triggers/03.webhook-trigger/index.md) to pull updates whenever new commits are pushed to `main`
See the [dedicated SyncFlows guide](../../15.how-to-guides/syncflows/index.md) for details.
To push flows from development to Git, use the [`git.PushFlows` task](/plugins/plugin-git/io.kestra.plugin.git.pushflows).
This ensures flows are validated before being saved — Kestra will reject invalid flows automatically.
You can also automate pull requests with the [`create.Pulls` task](/plugins/plugin-github/github-pull-requests/io.kestra.plugin.github.pulls.create), which creates a PR to `main` for review before deploying to production.
:::alert{type="info"}
While Kestra validates flow syntax, it does not detect logical or runtime errors. Always test flows thoroughly before promoting them to production.
:::
## CI/CD example
CI/CD pipelines can automatically deploy flows from Git to your Kestra production instance when changes are merged into the `main` branch.
For GitHub, Kestra provides an official [Deploy Action](../../version-control-cicd/cicd/01.github-action/index.md) that uses the Kestra Server CLI behind the scenes to perform deployments.
You can pair this with the [Validate Action](../../version-control-cicd/cicd/01.github-action/index.md), which checks that flows are valid before merging.
By enforcing required status checks on pull requests, you can prevent invalid flows from being merged and deployed to production.
:::alert{type="info"}
If a flow contains invalid syntax, the **Deploy Action** will fail.
:::
---
# Version Control with Git in Kestra
URL: https://kestra.io/docs/best-practices/git
> Best practices for using Git with Kestra for version control, including SyncFlows, PushFlows, and CI/CD integration.
Best practices for version control with Git in Kestra.
## Use Git effectively with Kestra
By default, **all Kestra flows are automatically versioned** using [Revisions](../../06.concepts/03.revision/index.md). You don't need an additional version control system to track changes to your flows. Kestra automatically creates a new revision each time you save a flow, allowing you to view change history, compare revisions, and restore previous versions at any time.
However, if you want to use Git to manage your [flows](../../05.workflow-components/01.flow/index.md) and [namespace files](../../06.concepts/02.namespace-files/index.md), you can do so with Kestra’s built-in Git integration.
There are multiple ways to use Git with Kestra:
- The [git.SyncFlows](/plugins/plugin-git/io.kestra.plugin.git.syncflows) pattern enables a GitOps approach, using Git as the single source of truth for your flows.
- The [git.SyncNamespaceFiles](/plugins/plugin-git/io.kestra.plugin.git.syncnamespacefiles) pattern enables GitOps for your namespace files.
- The [git.PushFlows](/plugins/plugin-git/io.kestra.plugin.git.pushflows) pattern allows you to edit flows from the UI and automatically commit and push changes to Git — ideal if you prefer using the built-in editor while keeping your code synchronized with Git.
- The [git.PushNamespaceFiles](/plugins/plugin-git/io.kestra.plugin.git.pushnamespacefiles) pattern allows you to edit namespace files from the UI and push updates to Git.
- The [CI/CD](../../version-control-cicd/cicd/index.md) pattern is ideal if you prefer managing the CI/CD process manually (for example, using GitHub Actions or Terraform) while keeping Git as the single source of truth for your code.
- [Clone](https://kestra.io/plugins/plugin-git/io.kestra.plugin.git.clone) task clones a repository directly into a flow, making scripts available for execution.
- [TenantSync](/plugins/plugin-git/io.kestra.plugin.git.tenantsync) task synchronizes all namespaces in a tenant, including flows, files, apps, tests, and dashboards.
- [NamespaceSync](/plugins/plugin-git/io.kestra.plugin.git.namespacesync) task synchronizes objects within a single namespace with your Git repository.
The diagram below illustrates how to choose the right pattern for your workflow:

For a detailed comparison of these patterns, see the [Version Control with Git](../../version-control-cicd/04.git/index.md) page.
---
# Loop Task Best Practices in Kestra
URL: https://kestra.io/docs/best-practices/loop
> Best practices for using the Loop task in Kestra — output collection, concurrency, map-reduce patterns, error handling, large-file processing, and subflow isolation.
Use `Loop` for all iteration needs in Kestra.
## Choose the right iteration pattern
`Loop` runs child tasks for each item in a list, map, file, or URI list. Every iteration is an isolated sub-execution.
Use a plain `Loop` when:
- You need to run the same tasks for each item in a list or dataset.
- Each iteration should process one value, one object, or one file chunk.
- You want parallel iteration with controlled concurrency.
- You want per-iteration failure handling without stopping the entire loop.
Use `Loop` + `Subflow` when:
- You need full execution isolation per batch — own retries, own logs, own failure state.
- Each batch should be independently restartable.
## Access the iteration value
Inside a Loop, use `item.value` for the current value and `item.index` for the zero-based position. These are available in every child task, including those nested inside `If`, `Parallel`, or other flowable tasks — no parent traversal needed.
```yaml
tasks:
- id: loop
type: io.kestra.plugin.core.flow.Loop
values: ["north", "south", "west"]
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "region={{ item.value }} index={{ item.index }}"
```
When iterating over JSON objects, `item.value` is a JSON string. Use `fromJson(item.value).field` to access properties — `item.value.field` does not work.
```yaml
tasks:
- id: loop
type: io.kestra.plugin.core.flow.Loop
values:
- {"id": 101, "email": "a@example.com"}
- {"id": 102, "email": "b@example.com"}
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "User {{ fromJson(item.value).id }} -> {{ fromJson(item.value).email }}"
```
## Expose outputs explicitly
Task outputs inside a loop are not visible outside it by default. Declare an `outputs:` block on the Loop task to surface values. Choose `fetchType` based on data volume:
- `AUTO` — default, switches automatically based on whether `values` is a URI
- `FETCH` — collects all iteration results inline (suitable for small iteration counts)
- `STORE` — writes results to internal storage and exposes a URI (preferred for large iteration counts)
After the loop:
- `outputs..outputs` is a list of per-iteration results
- `outputs..outputs[n].outputs.` accesses a specific iteration by index
- `loopOutputs(outputs..outputs, '')` extracts one field across all iterations as a flat list
### Example: collect outputs and read them downstream
```yaml
id: loop_outputs
namespace: company.team
tasks:
- id: enrich_regions
type: io.kestra.plugin.core.flow.Loop
values: ["north", "south", "west"]
concurrencyLimit: 2
fetchType: AUTO
outputs:
- id: bucket
type: STRING
value: "landing-{{ item.value }}"
- id: message
type: STRING
value: "{{ outputs.build_message.value }}"
tasks:
- id: build_message
type: io.kestra.plugin.core.debug.Return
format: "Load {{ item.value }} into landing-{{ item.value }}"
- id: log_first
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.enrich_regions.outputs[0].outputs.message }}"
- id: log_all
type: io.kestra.plugin.core.log.Log
message: "{{ loopOutputs(outputs.enrich_regions.outputs, 'message') }}"
```
Inside the loop, sibling task outputs are accessed with plain `outputs.task_id.attribute` syntax — each iteration runs in its own isolated context, so there is no ambiguity.
## Use `concurrencyLimit` deliberately
- `1` (default) — sequential execution
- A positive integer — bounded parallelism; prefer this for heavy workloads
- `0` — unlimited; all iterations run simultaneously; avoid for large datasets unless you understand the resource implications
## Process large files with Split and Loop
For file-backed datasets, use `Split` to break the file into chunk URIs, then loop over the URI list. Each `item.value` is one chunk URI.
Passing `values: "{{ outputs.split.uris }}"` where `outputs.split.uris` is a **list** is different from passing a single file URI string. A list iterates over elements; a single URI string iterates line-by-line through that file.
```yaml
tasks:
- id: split
type: io.kestra.plugin.core.storage.Split
from: "{{ inputs.file }}"
rows: 100
- id: per_chunk
type: io.kestra.plugin.core.flow.Loop
values: "{{ outputs.split.uris }}"
concurrencyLimit: 4
fetchType: FETCH
outputs:
- id: result_uri
type: STRING
value: "{{ outputs.process.value }}"
tasks:
- id: process
type: io.kestra.plugin.core.debug.Return
format: "processed chunk {{ item.index }}: {{ item.value }}"
- id: summary
type: io.kestra.plugin.core.log.Log
message: "{{ loopOutputs(outputs.per_chunk.outputs, 'result_uri') }}"
```
## Map-reduce: collect and aggregate file outputs across iterations
Use the map-reduce pattern when each iteration produces a file output that you need to combine and reduce after the loop.
The structure is always the same:
1. **Split** — break the input into chunk URIs
2. **Loop** — process each chunk in parallel; declare a file URI as a `STRING` output
3. **Concat** — stitch the per-iteration output files into one
4. **Aggregate** (or any reduce task) — merge results across chunks
The key is declaring the per-iteration output URI as a `STRING` in the Loop `outputs:` block, then using `loopOutputs()` to extract that list and pass it directly to `Concat`.
```yaml
id: elt_csv_split_loop_aggregate
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: to_ion
type: io.kestra.plugin.serdes.csv.CsvToIon
from: "{{ outputs.download.uri }}"
header: true
- id: split
type: io.kestra.plugin.core.storage.Split
from: "{{ outputs.to_ion.uri }}"
rows: 25
- id: per_chunk
type: io.kestra.plugin.core.flow.Loop
values: "{{ outputs.split.uris }}"
concurrencyLimit: 4
fetchType: FETCH
outputs:
- id: data
type: STRING
value: "{{ outputs.aggregate.uri }}"
tasks:
- id: filter
type: io.kestra.plugin.transform.Filter
from: "{{ item.value }}"
where: todecimal(total) > 10
- id: aggregate
type: io.kestra.plugin.transform.Aggregate
from: "{{ outputs.filter.uri }}"
outputType: STORE
groupBy: [customer_email]
aggregates:
orders:
expr: count()
type: INT
revenue:
expr: sum(todecimal(total))
type: DECIMAL
- id: concat
type: io.kestra.plugin.core.storage.Concat
files: "{{ loopOutputs(outputs.per_chunk.outputs, 'data') }}"
extension: .ion
- id: reduce
type: io.kestra.plugin.transform.Aggregate
from: "{{ outputs.concat.uri }}"
outputType: STORE
groupBy: [customer_email]
aggregates:
orders:
expr: sum(orders)
type: INT
revenue:
expr: sum(revenue)
type: DECIMAL
```
Key points:
- The Loop output declares type `STRING` even though the value is a file URI. `loopOutputs()` extracts strings, and `Concat` accepts a list of URI strings.
- `fetchType: FETCH` keeps the per-iteration output objects in memory so `loopOutputs()` can read them immediately after the loop. Use `STORE` instead if you have hundreds of iterations.
- The final `reduce` task re-aggregates because the same key (e.g. a customer) may appear in multiple chunks. The two-pass structure — per-chunk aggregate then cross-chunk reduce — is the canonical map-reduce pattern.
## Handle per-iteration failures
Set `transmitFailed: false` to continue the loop when individual iterations fail. Use `errors:` to run tasks per failed iteration, and `finally:` for a one-time cleanup block after all iterations finish. `errors:` runs on failure regardless of `transmitFailed` — with `transmitFailed: false`, the loop continues to subsequent iterations after handling the error. `finally:` always runs regardless.
```yaml
tasks:
- id: loop
type: io.kestra.plugin.core.flow.Loop
values: ["ok", "boom", "ok"]
transmitFailed: false
tasks:
- id: maybe_fail
type: io.kestra.plugin.scripts.shell.Commands
commands:
- |
if [ "{{ item.value }}" = "boom" ]; then exit 1; fi
echo "ok {{ item.value }}"
errors:
- id: handle_error
type: io.kestra.plugin.core.log.Log
message: "Iteration {{ item.index }} ({{ item.value }}) failed"
finally:
- id: cleanup
type: io.kestra.plugin.core.log.Log
message: "Loop finished (with or without failures)"
```
## Use Loop + Subflow for isolated per-batch execution
When each batch needs its own execution — independent retries, logs, and failure ownership — pair `Loop` with `Subflow`. The parent splits the data and fans out; the child flow receives one chunk URI per invocation and returns its result as a flow-level output.
```yaml
# Parent flow
tasks:
- id: split
type: io.kestra.plugin.core.storage.Split
from: "{{ inputs.file }}"
rows: 100
- id: per_batch
type: io.kestra.plugin.core.flow.Loop
values: "{{ outputs.split.uris }}"
concurrencyLimit: 4
fetchType: FETCH
outputs:
- id: result_uri
type: STRING
value: "{{ outputs.run_child.outputs.uri }}"
tasks:
- id: run_child
type: io.kestra.plugin.core.flow.Subflow
namespace: company.team
flowId: process_batch
wait: true
transmitFailed: true
inputs:
batch_uri: "{{ item.value }}"
- id: concat
type: io.kestra.plugin.core.storage.Concat
files: "{{ loopOutputs(outputs.per_batch.outputs, 'result_uri') }}"
extension: .ion
```
## Compose Loop with supporting tasks
`Loop` iterates. These tasks handle the rest — batching, transforming, stitching, and reducing. Each does one thing well; combine them around `Loop` to build larger pipelines.
| Task | Role | When to reach for it |
|---|---|---|
| `io.kestra.plugin.core.storage.Split` | Batching | Split a single file into chunk URIs by `rows`, `bytes`, `partitions`, or `separator`. Feeds `Loop.values` for map-reduce. |
| `io.kestra.plugin.core.storage.Concat` | Stitching | Concatenate per-iteration output files into one before a reduce step. |
| `io.kestra.plugin.transform.Aggregate` | Reduce | Group records by one or more keys with `count()`, `sum()`, `max()`, and more. The reduce side of map-reduce. |
| `io.kestra.plugin.transform.Filter` | Predicate | Keep only rows where a boolean expression holds. |
| `io.kestra.plugin.transform.Map` | Project | Per-record rename, drop, or compute fields — SQL `SELECT`-style. |
| `io.kestra.plugin.transform.Unnest` | Explode | Flatten an array field into one row per element, carrying sibling fields through. |
| `io.kestra.plugin.core.flow.Subflow` | Isolate | Spawn a separate execution per iteration — own retries, own logs, own failure state. |
| `io.kestra.plugin.core.flow.Parallel` | Fan-out | Run independent task groups concurrently inside a single iteration. |
## Common mistakes to avoid
- Do not use `taskrun.value` or `taskrun.iteration` — use `item.value` and `item.index`.
- Do not access `item.value.field` directly on object values — use `fromJson(item.value).field`.
- Do not expect loop outputs to be visible downstream without declaring an `outputs:` block.
- Do not use `outputs.task_id[item.value]` inside a loop — sibling outputs are accessed with plain `outputs.task_id.attribute`.
- Do not set `concurrencyLimit: 0` on very large datasets without considering memory and worker capacity.
For more details, see the [Loop task documentation](/plugins/core/flow/io.kestra.plugin.core.flow.loop) and the [Flowable Tasks reference](../../05.workflow-components/01.tasks/00.flowable-tasks/index.md#loop).
---
# Managing Environments in Kestra: Dev, Prod & Tenants
URL: https://kestra.io/docs/best-practices/manage-environments
> Best practices for managing Kestra environments, including separate instances for development and production, and using tenants.
Kestra users can manage their environments with different levels of granularity. Kestra has three main concepts: instance, tenant, and namespace.
## Manage Kestra instances, tenants, and namespaces
An instance is a full deployment of Kestra. A best practice is to have at least two separate instances: one for development and one for production.
The development instance serves as a sandbox for testing and experimentation, while the production instance supports critical operations and should only be accessible to administrators.
Large organizations sometimes have three or four environments. In such cases, it's best to use the [Kestra Enterprise Edition](../../oss-vs-paid/index.md) to manage all instances effectively, benefiting from improved governance, security, and scalability.
## When to use multiple tenants
A [tenant](../../07.enterprise/02.governance/tenants/index.md) is a logical separation within an instance. You can think of tenants as isolated Kestra projects that share instance resources. A single instance can have multiple tenants.
Tenants are useful when Kestra manages operations for different customers or teams. For example, a company with ten customers could assign each one to a separate tenant. Similarly, an international organization could use tenants to separate workflows by country.
Tenants can also be used to isolate environments for different engineering teams within the same development instance.
:::alert{type="info"}
Each tenant uses the same underlying instance resources. Therefore, it is not recommended to use tenants to separate development and production environments. If the underlying instance goes down, all tenants will be affected.
:::
## When to use multiple namespaces
Namespaces are useful for organizing your flows. They can help structure projects by domain or team.
Namespaces can also be used as lightweight “environments” for getting started, especially for open-source users who don’t need to manage multiple instances. However, this approach is not recommended for critical operations, since an issue in one namespace could impact production flows.
---
# Managing pip Dependencies in Kestra: Docker & Caching
URL: https://kestra.io/docs/best-practices/managing-pip-dependencies
> Efficiently manage Python pip dependencies in Kestra using custom Docker images, server startup installs, or caching.
Learn how to manage pip package dependencies in your flows.
## Manage Python dependencies efficiently
Your Python code may require `pip` package dependencies. How you manage these dependencies can affect the execution time of your flows.
If you install `pip` packages within `beforeCommands`, the packages will be downloaded and installed each time the task runs. This can significantly increase the duration of workflow executions. The following sections describe several ways to manage `pip` package dependencies efficiently in your flows.
## Using a custom Docker image
Instead of using the base Python Docker image and installing dependencies through `beforeCommands`, you can create a custom Docker image that includes Python and all required `pip` packages. Since the dependencies are built into the image, they do not need to be downloaded and installed at runtime. This reduces overhead and ensures that execution time is dedicated solely to running your Python code.
For example, if your Python script depends on `pandas`, you can use a container image that already includes it, such as `ghcr.io/kestra-io/pydata:latest`. This eliminates the need to install dependencies using `beforeCommands`:
```yaml
id: docker_dependencies
namespace: company.team
tasks:
- id: code
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/pydata:latest
script: |
import pandas as pd
df = pd.read_csv('https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv')
total_revenue = df['total'].sum()
```
## Installing pip package dependencies at server startup
Another way to avoid installing dependencies during every execution is to preinstall them before starting the Kestra server. For a standalone Kestra server, you can run:
```bash
pip install requests pandas polars && ./kestra server standalone --worker-thread=16
```
If you are running Kestra with Docker, create a Dockerfile and install dependencies using the `RUN` command. Set the `USER` to `root` to allow package installation:
```dockerfile
FROM kestra/kestra:latest
USER root
RUN pip install requests pandas polars
CMD ["server", "standalone"]
```
In your Docker Compose configuration, replace the `image` property with `build: .` to use your custom Dockerfile instead of the official image from Docker Hub. Also, remove the `command` property, since the `CMD` instruction in your Dockerfile now handles it:
```yaml
services:
...
kestra:
build: .
...
```
When you start Kestra using Docker Compose, the Python dependencies will already be included in the container.
In both installation methods, you must run Python tasks using the [Process Task Runner](../../task-runners/04.types/01.process-task-runner/index.md) to ensure the code can access the dependencies installed in the Kestra server process.
You can verify that the dependencies are installed with the following example:
```yaml
id: list_dependencies
namespace: company.team
tasks:
- id: check
type: io.kestra.plugin.scripts.python.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- pip list
```
## Using cache files
In a `WorkingDirectory` task, you can install all required `pip` dependencies into a local `deps` folder using the [Process Task Runner](../../task-runners/04.types/01.process-task-runner/index.md), and cache that folder. This ensures the dependencies are reused in subsequent executions, eliminating the need for repeated installations. For more details, see the [caching](../../06.concepts/12.caching/index.md) page.
The example below demonstrates how to cache the `deps` folder:
```yaml
id: python_cached_dependencies
namespace: company.team
tasks:
- id: working_dir
type: io.kestra.plugin.core.flow.WorkingDirectory
tasks:
- id: python_script
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.core.runner.Process
beforeCommands:
- pip install --target=./deps pandas
env:
PYTHONPATH: "./deps"
script: |
import pandas as pd
print(pd.__version__)
cache:
patterns:
- deps/**
ttl: PT24H
```
By using one of these techniques, you can avoid reinstalling dependencies for each execution and reduce overall execution time.
---
# Naming Conventions in Kestra: Flows and Namespaces
URL: https://kestra.io/docs/best-practices/naming-conventions
> Learn the best practices for naming namespaces, flows, tasks, and other identifiers in Kestra for a clean and scalable hierarchy.
Common naming conventions to keep your flows and tasks well-organized and consistent in Kestra.
## Name namespaces and flows consistently
Follow a `company.team` structure for namespaces to maintain a clean, scalable, and consistent hierarchy across your workflows.
This approach helps with:
1. Centralized governance for credentials and configurations
2. Easier sharing of variables, plugin defaults, and secrets across teams
3. Simplified Git synchronization
## Why use the `company.team` structure
By defining a **root namespace named after your company**, you can centralize management of [Policies](../../07.enterprise/02.governance/policies/index.md) (Enterprise Edition), [variables](../../05.workflow-components/04.variables/index.md), and [secrets](../../06.concepts/04.secret/index.md).
These configurations can then be inherited by all namespaces under that root.
This structure also simplifies [Git synchronization](../../version-control-cicd/04.git/index.md).
You can maintain a single synchronization flow that updates all namespaces under your company root.
The next level — named after your team (e.g., `company.team`) — allows for shared governance and visibility at the team level.
From there, you can further divide namespaces by project, system, or other logical hierarchies.
When synced with Git, this nested structure maps directly to nested directories in your repository.
## Example namespace structure
```plaintext
mycompany
├── mycompany.marketing
│ ├── mycompany.marketing.projectA
│ └── mycompany.marketing.projectB
└── mycompany.sales
├── mycompany.sales.projectC
└── mycompany.sales.projectD
```
## Should you use environment-specific namespaces?
Avoid **environment-specific namespaces** (e.g., `dev`, `staging`, `prod`) because they can introduce several issues:
- **Shared risk:** Development workflows can unintentionally impact production.
- **Configuration drift:** Duplicating configurations across environments can lead to inconsistencies.
Instead, run **separate Kestra instances** (or tenants in Enterprise Edition) for development and production.
## Summary
Using a `company.team` namespace structure creates a clear, maintainable hierarchy that mirrors your organization’s structure and simplifies Git synchronization.
To separate environments reliably, use distinct Kestra instances or tenants rather than environment-based namespaces.
## ID naming convention
Use a consistent naming pattern across all identifiers in Kestra, including:
- Flows
- Tasks
- Inputs
- Outputs
- Triggers
### Valid characters and subscript notation
Kestra does not enforce a specific naming style, but IDs must match the regex pattern:
`^[a-zA-Z0-9][a-zA-Z0-9_-]*`
This means:
- Only letters, numbers, underscores `_`, and hyphens `-` are allowed.
- If you use hyphens (e.g., `kebab-case`), reference IDs using **subscript notation**, such as:
`{{ outputs.task_id["your-custom-value"].attribute }}`
:::alert{type="info"}
Use **snake_case** or **camelCase** instead of `kebab-case` to avoid the need for subscript notation and improve readability.
:::
### Snake case
Snake case is popular among Python developers, especially in data and AI workflows.
Here’s an example using `snake_case` for all IDs:
```yaml
id: api_python_sql
namespace: prod.marketing.attribution
inputs:
- id: api_endpoint
type: URL
defaults: https://dummyjson.com/products
tasks:
- id: fetch_products
type: io.kestra.plugin.core.http.Request
uri: "{{ inputs.api_endpoint }}"
- id: transform_in_python
type: io.kestra.plugin.scripts.python.Script
containerImage: python:slim
beforeCommands:
- pip install polars
outputFiles:
- "products.csv"
script: |
import polars as pl
data = {{ outputs.fetch_products.body | jq('.products') | first }}
df = pl.from_dicts(data)
df.glimpse()
df.select(["brand", "price"]).write_csv("products.csv")
- id: sql_query
type: io.kestra.plugin.jdbc.duckdb.Query
inputFiles:
in.csv: "{{ outputs.transform_in_python.outputFiles['products.csv'] }}"
sql: |
SELECT brand, round(avg(price), 2) as avg_price
FROM read_csv_auto('{{ workingDir }}/in.csv', header=True)
GROUP BY brand
ORDER BY avg_price DESC;
fetchType: STORE
outputs:
- id: final_result
value: "{{ outputs.sql_query.uri }}"
triggers:
- id: daily_at_9am
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 9 * * *"
```
### Camel case
Camel case is common in Java and JavaScript ecosystems.
Here’s the same example using `camelCase`:
```yaml
id: apiPythonSql
namespace: prod.marketing.attribution
inputs:
- id: apiEndpoint
type: URL
defaults: https://dummyjson.com/products
tasks:
- id: fetchProducts
type: io.kestra.plugin.core.http.Request
uri: "{{ inputs.apiEndpoint }}"
- id: transformInPython
type: io.kestra.plugin.scripts.python.Script
containerImage: python:slim
beforeCommands:
- pip install polars
outputFiles:
- "products.csv"
script: |
import polars as pl
data = {{ outputs.fetchProducts.body | jq('.products') | first }}
df = pl.from_dicts(data)
df.glimpse()
df.select(["brand", "price"]).write_csv("products.csv")
- id: sqlQuery
type: io.kestra.plugin.jdbc.duckdb.Query
inputFiles:
in.csv: "{{ outputs.transformInPython.outputFiles['products.csv'] }}"
sql: |
SELECT brand, round(avg(price), 2) as avgPrice
FROM read_csv_auto('{{ workingDir }}/in.csv', header=True)
GROUP BY brand
ORDER BY avgPrice DESC;
store: true
outputs:
- id: finalResult
value: "{{ outputs.sqlQuery.uri }}"
triggers:
- id: dailyAt9am
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 9 * * *"
```
Both conventions are valid — choose the one that best matches your team’s coding standards.
---
# Managing and Purging Flow Outputs in Kestra
URL: https://kestra.io/docs/best-practices/outputs
> Best practices for managing flow outputs in Kestra, including purging large files and handling conditional outputs efficiently.
Best practices for handling flow outputs, including purging large outputs and conditionally returning outputs.
## Handle flow outputs safely
When a flow can return different outputs depending on certain conditions, you can use an expression in the `outputs` section. This allows you to conditionally return the output of task `A` if it wasn’t skipped, or the output of task `B` otherwise.
```yaml
id: conditionallyReturnOutputs
namespace: company.team
inputs:
- id: runTask
type: BOOLEAN
defaults: true
tasks:
- id: taskA
runIf: "{{ inputs.runTask }}"
type: io.kestra.plugin.core.debug.Return
format: Hello World!
- id: taskB
type: io.kestra.plugin.core.debug.Return
format: Fallback output
outputs:
- id: flowOutput
type: STRING
value: "{{ tasks.taskA.state != 'SKIPPED' ? outputs.taskA.value : outputs.taskB.value }}"
```
## Purging large output files
If a flow generates large output files that are not needed after execution, you can use the `io.kestra.plugin.core.storage.PurgeExecutionFiles` task to delete those files from internal storage.
In the example below, the flow downloads a large file from an HTTP API and uploads it to an S3 bucket. Once the file is uploaded, it’s no longer needed locally, so the `PurgeExecutionFiles` task is used to remove it from internal storage.
```yaml
id: extractLoadPurge
namespace: company.team
tasks:
- id: extract
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: load
type: io.kestra.plugin.aws.s3.Upload
from: "{{ outputs.extract.uri }}"
bucket: myBucket
key: largeFiles/orders.csv
- id: purge
type: io.kestra.plugin.core.storage.PurgeCurrentExecutionFiles
```
---
# Data Retention and Purging in Kestra: Choose a Strategy
URL: https://kestra.io/docs/best-practices/purging-data
> Learn how to choose the right purge strategy in Kestra for executions, logs, key-value pairs, Namespace files, assets, and other retained data.
How to choose the right purge strategy for operational data, retained artifacts, and mutable state in Kestra.
## Choose the right purge strategy
Kestra stores different kinds of data for different reasons. Some data supports execution history and troubleshooting, some supports runtime state, and some supports retained files or assets.
Because of that, purging data in Kestra is not a single decision. The right approach depends on:
- what kind of data you want to remove
- why the data exists
- how long you need to keep it
- whether you want hard deletion or to hide it from the UI
## Quick recommendation
Use this rule of thumb:
- purge **executions** and **logs** to control operational storage growth
- purge **KV pairs** only when they represent expired runtime state
- purge **Namespace files** only when you need version retention on file history
- purge **assets and lineage** only when you are enforcing a retention policy for asset metadata
- do not rely on UI deletion when your goal is storage reclamation or permanent deletion
## Comparison table
| If you want to remove... | Prefer | Why |
| --- | --- | --- |
| Old execution records | [`PurgeExecutions`](/plugins/core/tasks/io.kestra.plugin.core.execution.purgeexecutions) | It permanently deletes execution metadata and related execution data |
| Old execution logs, trigger logs, or both | [`PurgeLogs`](/plugins/core/log/io.kestra.plugin.core.log.purgelogs) | Use `purgeExecutionLogs` and `purgeNonExecutionLogs` to target each type independently, or leave both `true` (default) to purge all logs |
| Expired runtime state in the KV Store | [`PurgeKV`](/plugins/core/kv/io.kestra.plugin.core.kv.purgekv) or automatic KV expiration purge | It removes stale KV entries without treating them as static configuration |
| Old Namespace file versions | [`PurgeFiles`](/plugins/core/namespace/io.kestra.plugin.core.namespace.purgefiles) | It applies retention rules to Namespace files and their versions |
| Old asset records, usages, or lineage data | [`PurgeAssets`](../../10.administrator-guide/purge/index.md#purge-assets-and-lineage-retention) | It applies retention to asset-related records without touching executions or logs |
| A flow, namespace, or other object only in the UI | UI deletion | It hides records, but does not perform the same hard deletion as purge tasks |
## Use purge tasks for retention, not for routine cleanup by hand
If data should be deleted on a recurring basis, treat it as a retention policy rather than a manual maintenance task.
Best practice:
- define retention periods explicitly
- schedule purge flows
- keep those flows in a central administrative or `system` namespace
- test purge behavior in non-production environments first
This makes retention predictable and easier to review.
## When to purge executions and logs
Use [`PurgeExecutions`](/plugins/core/execution/io.kestra.plugin.core.execution.purgeexecutions) and [`PurgeLogs`](/plugins/core/log/io.kestra.plugin.core.log.purgelogs) when your main goal is to reduce the footprint of historical operational data.
This is usually the right choice when:
- you no longer need old execution history for troubleshooting
- old logs are consuming storage
- you already have another system for long-term observability or audit retention
Best practice:
- set separate retention periods for executions and logs if your teams use them differently
- use `purgeExecutionLogs: false` to retain execution logs for failed workflow debugging while still purging trigger logs, or `purgeNonExecutionLogs: false` to do the reverse
- set `batchSize` on `PurgeLogs` when purging large volumes of logs to limit the number of rows deleted per transaction
- avoid deleting recent data that is still useful for troubleshooting failed workflows
- run purge flows on a schedule instead of waiting for storage pressure
Relevant blueprints:
- [Purge execution data including logs, metrics and outputs on a schedule](https://kestra.io/blueprints/purge)
- [Purge disk space interactively](https://kestra.io/blueprints/purge-disk-space-interactively)
## When to purge KV pairs
Use [`PurgeKV`](/plugins/core/kv/io.kestra.plugin.core.kv.purgekv) only for runtime state that has expired or is no longer valid.
The KV Store is best used for mutable state such as:
- cursors
- offsets
- checkpoints
- last processed timestamps
Best practice:
- set TTLs where possible
- enable automatic purging of expired KV pairs when that matches your operational model
- avoid using KV purge as a substitute for redesigning unclear state lifecycles
Relevant blueprint:
- [Purge old KV pairs from the KV Store](https://kestra.io/blueprints/kv-store-purge)
If a value is actually configuration or a secret, it probably does not belong in the KV Store in the first place.
## When to purge Namespace files
Use [`PurgeFiles`](/plugins/core/namespace/io.kestra.plugin.core.namespace.purgefiles) when your goal is to manage retention of Namespace file versions rather than execution data.
This is useful when:
- teams frequently update scripts or SQL files stored as Namespace files
- you want to keep only the most recent versions
- you want to remove versions older than a given date
Best practice:
- scope purge rules with `filePattern` so you do not delete unrelated files
- define whether you want date-based or version-based retention
- be careful when applying purge rules across parent and child namespaces
## When to purge assets and lineage
Use [`PurgeAssets`](../../10.administrator-guide/purge/index.md#purge-assets-and-lineage-retention) when you need retention for asset metadata, asset usage records, or lineage data.
This is different from purging executions or logs. Asset retention is its own concern and should be managed separately.
Best practice:
- filter by namespace, asset type, or metadata when possible
- define retention based on operational or compliance requirements
- purge only the records you intend to remove, especially if you want to keep lineage or usage data longer than the assets themselves
[`PurgeAssets`](../../10.administrator-guide/purge/index.md#purge-assets-and-lineage-retention) is an Enterprise Edition and Cloud feature.
## Purge tasks vs. UI deletion
Do not treat purge tasks and UI deletion as equivalent.
- purge tasks perform hard deletion and reclaim storage
- UI deletion is a soft deletion and preserves underlying history
Use purge tasks when you need permanent removal or storage reduction. Use UI deletion when you only want to remove an item from the visible working set without changing retention at the storage level.
## What purge tasks do not cover
Purge tasks are not a universal retention mechanism for every internal component.
In particular:
- purge tasks do not manage internal queue retention
- queue retention is configured separately depending on your backend
Avoid assuming that a purge flow alone covers all retained system data.
## Recommended patterns
### Pattern 1: Scheduled operational retention
Create a scheduled purge flow for executions and logs with a clearly defined retention period.
This is the most common pattern for controlling storage growth.
### Pattern 2: State lifecycle management for KV pairs
Use KV TTLs and expired-key cleanup for temporary runtime state rather than accumulating state indefinitely.
### Pattern 3: Version retention for Namespace files
Apply [`PurgeFiles`](/plugins/core/namespace/io.kestra.plugin.core.namespace.purgefiles) with explicit namespace scope and file patterns when teams manage reusable scripts or SQL as Namespace files.
### Pattern 4: Separate retention policies by data type
Use different purge flows or schedules for executions, logs, KV pairs, Namespace files, and assets.
This keeps retention aligned with how each type of data is actually used.
## Anti-patterns
Avoid these patterns:
- using one retention period for every type of data without considering how the data is used
- relying on manual cleanup only after storage becomes a problem
- using UI deletion when you actually need hard deletion
- purging KV pairs that are really standing in for missing configuration or poor state design
- running broad Namespace file purges without a `filePattern` or namespace scope
## Decision guide
Ask these questions:
1. Is the data operational history, mutable state, retained files, or asset metadata?
Choose the purge task that matches that data type.
2. Do you need hard deletion or only to remove an item from the UI?
If you need permanent deletion, use a purge task.
3. Should the data expire automatically based on age or lifecycle?
If yes, define a retention policy and schedule it.
4. Is the data still needed for troubleshooting, auditability, or compliance?
If yes, shorten retention carefully rather than purging broadly.
5. Are you trying to solve a storage problem or a modeling problem?
If the data should never have been long-lived state, fix the design instead of only adding a purge flow.
## Summary
- Use purge tasks based on the type of data you want to remove.
- Treat retention as a deliberate operational policy, not an afterthought.
- Use hard-deletion purge tasks for permanent cleanup and storage reclamation.
- Keep separate retention strategies for executions, logs, KV pairs, Namespace files, and assets.
For the underlying purge tasks and configuration options, see [Purge old data in Kestra](../../10.administrator-guide/purge/index.md).
---
# Secrets Management in Kestra: Avoid Accidental Exposure
URL: https://kestra.io/docs/best-practices/secrets-management
> Best practices for securely managing and using secrets in Kestra workflows to prevent accidental exposure.
A quick guide to [secrets](../../07.enterprise/02.governance/secrets-manager/index.md) management best practices in Kestra.
## Manage secrets securely in Kestra
Kestra provides a built-in [secret manager](../../07.enterprise/02.governance/secrets-manager/index.md) with obfuscation capabilities, but it’s important to understand its limitations and follow best practices to minimize the risk of secret exposure.
## Secret obfuscation in logs is best effort
Kestra attempts to mask secrets in logs and during expression evaluation, but masking is not foolproof.
Current log obfuscation replaces full secret matches with `****`. However, if a secret is modified — for example, through substring extraction, concatenation, encoding, or interpolation — it may bypass obfuscation and appear in logs. Refer to the [Filter Reference](../../expressions/03.filters/index.mdx) for a list of possible transformations.
For example, the following flow uses `jq()` in a log message to return a partial value associated with a secret:
```yaml
id: secret_test
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: "You can see my secret token {{ secret('jsonSecret') | jq('.token') }}"
```
In the logs, the token name `SUPER_SECRET` is exposed:

**Best practices:**
- Never log secrets intentionally.
- Avoid passing secrets into string manipulation expressions that could expose partial values.
- Treat all secrets as sensitive, even in debug or test workflows.
## Understand expression evaluation limits
When using the **Debug Expression** tool in the **Outputs** tab of an execution, Kestra forbids direct calls to `secret()` to prevent leaks. Logs are more permissive because tasks can emit any property, but this also increases the risk of accidental exposure.
Avoid using secrets in Log tasks. If they are required, ensure you understand the risks and limitations before doing so.
## Avoid root-level secret placement
Secrets defined at the root namespace are inherited by all sub-namespaces. This can unintentionally broaden accessibility and increase exposure risk.
**Best practices:**
- Store secrets at the lowest namespace level necessary for their use.
- Use granular RBAC permissions to control who can access secrets and which workflows can use them.
## Design workflows to limit exposure
Follow these practices when designing workflows:
- Pass secrets only to tasks that require them.
- Avoid exposing secrets in user-facing outputs or debug messages.
- Where possible, design tasks to reference secrets rather than embedding raw values directly.
---
# Kestra Brand Assets: Logos and Visual Identity
URL: https://kestra.io/docs/brand-assets
> Download official Kestra brand assets including logos, color palettes, and usage guidelines for presentations, blog posts, partner pages, and marketing.
import DownloadLogoPack from '~/components/content/DownloadLogoPack.vue'
import CardLogos from '~/components/content/CardLogos.vue'

## Our Story

Kestra strives to be a simple yet powerful orchestration platform, enabling our clients to manage complex workflows with the same agility as a conductor who leads a symphony. This is how our logo was born, which embodies Kestra’s ability to orchestrate a wide variety of workflows anywhere at any scale.
## Logos
Click on the link below to download the logo pack in PNG and SVG:
---
# Core Concepts in Kestra: Architecture and Templating
URL: https://kestra.io/docs/concepts
> Core concepts of Kestra orchestration. Reference guide for architecture, data handling, templating, and key terminology used throughout the platform.
import ChildCard from "~/components/docs/ChildCard.astro"
This section lists key concepts and templating expressions. You can treat this section as a lookup reference anytime you need more details about a specific concept or expression.
## Explore Key Concepts
---
# Backfill in Kestra: Replay Missed Schedules
URL: https://kestra.io/docs/concepts/backfill
> Replay missed schedule intervals with Kestra Backfills. Rerun historical executions between a start and end date to reprocess data or recover from gaps.
Backfills are replays of missed schedule intervals between a defined start and end date.
Consider a flow that runs every 30 minutes:
```yaml
id: scheduled_flow
namespace: company.team
tasks:
- id: label
type: io.kestra.plugin.core.execution.Labels
labels: # label to track scheduled date
scheduledDate: "{{ trigger.date ?? execution.startDate }}"
- id: external_system_export
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- echo "processing data for {{ trigger.date ?? execution.startDate }}"
- sleep $((RANDOM % 5 + 1))
triggers:
- id: schedule
type: io.kestra.plugin.core.trigger.Schedule
cron: "*/30 * * * *"
```
If the source system had a 5-hour outage, this flow would miss 10 executions. A backfill replays all schedule intervals in the specified time window — including any that succeeded — so set the start and end dates precisely. To replay specific executions rather than a full time window, use [Replay](../10.replay/index.md) instead.
:::alert{type="info"}
**All missed schedules are automatically recovered by default** if the Kestra server is down. The missed schedules will be executed as soon as Kestra is back up because of the `recoverMissedSchedules: ALL` property default. If you have configured this differently in your global Kestra configuration or specifically on a trigger, a Backfill achieves the same behavior. Read more about `recoverMissedSchedules` in the [dedicated documentation](../../05.workflow-components/07.triggers/01.schedule-trigger/index.md#recover-missed-schedules).
:::
To backfill the missed executions, use **Backfill executions** on the **Triggers** tab of the flow's detail page.

Select the start and end date for the backfill and optionally add custom labels to the executions for tracking.
You can pause and resume the backfill at any time. Click **Details** to see progress and execution status:

:::alert{type="info"}
Backfill executions will not be processed if the associated trigger is disabled.
:::
## Delete a backfill
Delete a backfill from **Tenant → Triggers**. Select the trigger and remove the backfill to stop pending replays.

Deleting a backfill only cancels the scheduled catch-up executions. For example, if you defined a `* * * * *` schedule and backfilled the last five minutes, removing that backfill prevents those five replayed runs from being emitted. This is different from **Delete trigger**, which clears the trigger state itself — effectively recreating the trigger so it starts evaluating from the current time. Use **Delete backfill** to stop pending replays, and **Delete trigger** when you need to reset a stuck trigger or start it fresh.
## Trigger backfill via an API call
### Using cURL
```sh
curl -X PUT http://localhost:8080/api/v1/main/triggers \
-H "Authorization: Bearer $KESTRA_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"namespace": "company.team",
"flowId": "myflow",
"triggerId": "schedule",
"backfill": {
"start": "2025-04-29T11:30:00Z",
"end": null,
"labels": [
{
"key": "reason",
"value": "outage"
}
]
}
}'
```
`start` is required; `end` defaults to the current time if omitted. Use `inputs` to pass flow inputs and `labels` to tag the resulting executions for tracking. See the [API Reference](../../api-reference/02.open-source/index.mdx) for all available backfill operations.
### Using a service account
:::badge{version=">=0.15" editions="EE,Cloud"}
:::
Use a [Service Account](../../07.enterprise/03.auth/service-accounts/index.md) token instead of a user token, and include the tenant in the request header and body:
```sh
curl -X PUT http://localhost:8080/api/v1/main/triggers \
-H "Authorization: Bearer $KESTRA_API_TOKEN" \
-H "X-Kestra-Tenant: production" \
-H "Content-Type: application/json" \
-d '{
"namespace": "company.team",
"flowId": "myflow",
"triggerId": "schedule",
"tenantId": "production",
"backfill": {
"start": "2025-04-29T11:30:00Z",
"end": null,
"labels": [
{
"key": "reason",
"value": "outage"
}
]
}
}'
```
### Using Python requests
```python
import requests
import json
url = 'http://localhost:8080/api/v1/main/triggers'
headers = {
'Content-Type': 'application/json'
}
data = {
"backfill": {
"start": "2025-06-03T06:30:00.000Z",
"end": None,
"inputs": None,
"labels": [
{
"key": "reason",
"value": "outage"
}
]
},
"flowId": "myflow",
"namespace": "company.team",
"triggerId": "schedule"
}
response = requests.put(url, headers=headers, data=json.dumps(data))
print(response.status_code)
print(response.text)
```
When `end` is `None`, the backfill runs up to the current time.
---
# Blueprints in Kestra: Reusable Workflow Templates
URL: https://kestra.io/docs/concepts/blueprints
> Explore Kestra Blueprints — ready-to-use workflow templates that help you get started faster. Browse community and Enterprise blueprints for any automation.
Blueprints are a searchable catalog of validated, documented flow examples. Each blueprint combines code and documentation with tags for discoverability. Click **Use** to copy any blueprint into your editor and customize it from there.
Browse the full catalog at the [Blueprints library](/blueprints).
:::alert{type="info"}
The [Kestra MCP server](../../ai-tools/02.kestra-mcp-resources/index.md) exposes the blueprints library directly to AI coding agents like Claude Code and Cursor. Ask your agent to find a blueprint by use case and it will retrieve the full flow YAML for you.
:::

## Community blueprints
Community blueprints are available in the open-source product and reflect common workflow patterns across the Kestra user base. All blueprints are verified by the Kestra team. To contribute a new blueprint or suggest improvements, use the [GitHub issue template](https://github.com/kestra-io/kestra/issues/new?assignees=&labels=blueprint&projects=&template=blueprint.yml).
### Where to find blueprints
Blueprints are available from the **Blueprints** item in the left sidebar. Each blueprint shows its full YAML and topology before you commit — click **Open in Edit Flow** to load it directly into the editor.

### How to find the right blueprint
From the Blueprints page, **search** by use case or integration (Snowflake, DuckDB, Slack, dbt, Docker, etc.) or **filter** by tag to narrow results.
## Custom blueprints
:::alert{type="info"}
This feature requires the [Enterprise Edition](../../07.enterprise/index.mdx).
:::
Custom Blueprints are private blueprints available only to your organization. Beyond static templates, you can create **Templated Blueprints** — form-driven blueprints that generate complete flows from user inputs using Pebble-style templating, without requiring users to edit YAML directly. Custom Blueprints can also be version-controlled with Git using the `PushBlueprints` and `SyncBlueprints` tasks. See the [Custom Blueprints](../../07.enterprise/02.governance/custom-blueprints/index.md) documentation for details.
---
# Caching in Kestra: Speed Up Repeated Tasks
URL: https://kestra.io/docs/concepts/caching
> Speed up repeated tasks with file caching in Kestra. Use the WorkingDirectory task to cache dependencies and skip redundant downloads across flow executions.
Kestra supports two complementary caching strategies: file caching via `WorkingDirectory` and output caching via `taskCache`.
- **File caching** stores files (dependencies, build artifacts) in internal storage and restores them at the start of the next run.
- **Output caching** stores a task's status and outputs in the database and skips re-execution entirely when inputs have not changed. See [Task Cache](../../05.workflow-components/task-cache/index.md).
## Cache files in a `WorkingDirectory` task
Add a `cache` block to a `WorkingDirectory` task to persist files across executions.
```yaml
id: caching_files
namespace: company.team
tasks:
- id: working_dir
type: io.kestra.plugin.core.flow.WorkingDirectory
cache:
patterns:
- some_directory/**
ttl: PT1H
```
The `cache` property accepts a list of glob `patterns` and a `ttl` duration after which the cached files are invalidated.
### How caching works
Kestra packages the matched files and stores them in internal storage at the end of each run. On the next run, those files are restored before any task executes.
### Runner compatibility
File caching works when the script runs in a `PROCESS` runner. If you use a `DOCKER` runner, Kestra cannot cache files that exist only inside the container — attempting to do so [throws an error](https://github.com/kestra-io/kestra/issues/2233): `Unable to execute WorkingDirectory post actions`.
To cache pip packages with a Docker runner, install into a subdirectory of the working directory using `--cache-dir` and cache that directory instead:
```yaml
id: python_cached_pip
namespace: company.team
tasks:
- id: working_dir
type: io.kestra.plugin.core.flow.WorkingDirectory
cache:
patterns:
- cache/pip/**
ttl: PT24H
tasks:
- id: python_script
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
beforeCommands:
- pip install --cache-dir cache/pip pandas
script: |
import pandas as pd
print(pd.__version__)
```
Because `cache/pip` lives inside the working directory (not inside the container), Kestra can read and restore it between runs.
### Node.js example
This flow installs the `colors` package and caches `node_modules` for one hour. Use a `PROCESS` runner when caching `node_modules` directly.
```yaml
id: node_cached_dependencies
namespace: company.team
tasks:
- id: working_dir
type: io.kestra.plugin.core.flow.WorkingDirectory
cache:
patterns:
- node_modules/**
ttl: PT1H
tasks:
- id: node_script
type: io.kestra.plugin.scripts.node.Script
beforeCommands:
- npm install colors
script: |
const colors = require("colors");
console.log(colors.red("Hello"));
```
### Python example (Process runner)
This flow installs `pandas` into a `deps` folder and caches it for one day.
```yaml
id: python_cached_dependencies
namespace: company.team
tasks:
- id: working_dir
type: io.kestra.plugin.core.flow.WorkingDirectory
tasks:
- id: python_script
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.core.runner.Process
beforeCommands:
- pip install --target=./deps pandas
env:
PYTHONPATH: "./deps"
script: |
import pandas as pd
print(pd.__version__)
cache:
patterns:
- deps/**
ttl: PT24H
```
### How to invalidate the cache
- After the first run, files are cached.
- On subsequent runs, if the `ttl` has not elapsed, the cached files are restored. If it has elapsed, the cache is cleared and `beforeCommands` (e.g. `npm install`) runs in full.
- Changing the `ttl` takes effect on the next run.
The `ttl` is evaluated at runtime against the last task execution date.
---
# File Access in Kestra: Local and Namespace Files
URL: https://kestra.io/docs/concepts/file-access
> Access local and namespace files in Kestra using the universal file protocol. Learn how to read, write, and share files between tasks and namespaces.
Kestra supports a universal file protocol for referencing local and [namespace files](../02.namespace-files/index.md) in your flows using consistent URI schemes.
You can reference files inline in YAML, or use `nsfile:///` and `file:///` URIs to point to namespace files or files on the host machine. The flow below demonstrates all three approaches:
```yaml
id: protocol
namespace: company.team
tasks:
- id: inline_file
type: io.kestra.plugin.scripts.python.Commands
inputFiles:
hello.py: |
x = "Hello world!"
print(x)
- id: local_file
type: io.kestra.plugin.scripts.python.Commands
inputFiles:
hello.py: file:///scripts/hello.py
- id: namespace_file_from_the_same_namespace
type: io.kestra.plugin.scripts.python.Commands
inputFiles:
hello.py: nsfile:///scripts/hello.py
- id: namespace_file_from_other_namespace
type: io.kestra.plugin.scripts.python.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- python hello.py
inputFiles:
hello.py: nsfile://company/scripts/hello.py
```
## Allowed paths
To use the `file:///` scheme, bind-mount the host directory into the Kestra container and set the `kestra.local-files.allowed-paths` configuration property. For example, to allow access to a `scripts` folder:
```yaml
kestra:
image: kestra/kestra:latest
volumes:
- /Users/yourdir/scripts:/scripts # Bind-mount the host directory
...
environment: # Allow access to the /scripts directory in Kestra container
KESTRA_CONFIGURATION: |
kestra:
local-files:
allowed-paths:
- /scripts
```
If you see the following error:
```plaintext
java.lang.SecurityException: The path /scripts/hello.py is not authorized. Only files inside the working directory are allowed by default, other paths must be allowed either globally inside the Kestra configuration using the `kestra.local-files.allowed-paths` property, or by plugin using the `allowed-paths` plugin configuration.`.
```
This means the allowed paths are not configured correctly. Confirm that the host directory is bind-mounted into the container and that `kestra.local-files.allowed-paths` includes that path.
## Protocol reference
| Scheme | Purpose |
|--------|---------|
| `file:///path/to/file.txt` | Local file on the host machine from an explicitly allowed path |
| `nsfile:///path/to/file.txt` | File in the current namespace (three slashes; no namespace inheritance) |
| `nsfile://other.namespace/path/to/file.txt` | File in another namespace (two slashes + namespace name); requires that namespace to be allowed |
| `kestra:///path/to/file.txt` | File in Kestra's internal storage |
`nsfile:///` does not inherit from parent namespaces. If `nsfile:///scripts/hello.py` is not found in `company.team`, Kestra will not fall back to `company`. To reference a parent namespace explicitly, use `nsfile://company/scripts/hello.py`.
## Usage with `read()`
Use `read()` in tasks that expect file content rather than a path — for example, to load a SQL query from a namespace file:
```yaml
id: query
namespace: demo
tasks:
- id: duckdb
type: io.kestra.plugin.jdbc.duckdb.Query
sql: "{{ read('nsfile:///query.sql') }}"
```
For local files on the host:
```yaml
id: query
namespace: demo
tasks:
- id: duckdb
type: io.kestra.plugin.jdbc.duckdb.Query
sql: "{{ read('file:///query.sql') }}"
```
## Namespace files as default FILE-type inputs
You can reference a namespace file as the default value for a `FILE`-type input. This flow reads `hello.txt` from the `demo` namespace and logs its content:
```yaml
id: file_input
namespace: demo
inputs:
- id: myfile
type: FILE
defaults: nsfile:///hello.txt
tasks:
- id: print_file_content
type: io.kestra.plugin.core.log.Log
message: "{{ read(inputs.myfile) }}"
```
---
# KV Store in Kestra: Persist Shared State
URL: https://kestra.io/docs/concepts/kv-store
> Build stateful workflows with the Kestra KV Store. Persist and share key-value pairs across flows and executions for dynamic configuration and shared state.
The KV Store lets you persist and share key-value data across executions and flows — beyond what task outputs alone can carry.
Kestra's execution model is stateless by design — task runs are isolated and data moves between tasks via explicit outputs. The KV Store extends this with a namespace-scoped key-value layer for cases where you need to persist state across executions or share values between flows. Values are stored in Kestra's internal storage (your private cloud bucket); the database holds only metadata such as the key, TTL, and timestamps.
## Keys and values
Keys are arbitrary strings containing uppercase or lowercase letters and standard ASCII characters.
Values are stored as ION files in Kestra's internal storage and are strongly typed. Supported types:
- string
- number
- boolean
- datetime
- date
- duration
- JSON
Each KV pair can have a Time to Live (TTL) to automatically expire data that is only relevant for a limited period.
## Namespace binding
KV pairs are scoped to a namespace. Access them under **Namespaces → [namespace] → KV Store** or under **Tenant → KV Store** in the sidebar for a tenant-wide view.
You can create and read KV pairs across namespaces as long as those namespaces are [allowed](../../07.enterprise/02.governance/07.namespace-management/index.md#allowed-namespaces).
## Managing KV pairs
KV pairs can be managed from the UI, in flows via tasks, through the REST API, via Terraform, or from the command line:
1. **Kestra UI**: create, edit, and delete KV pairs directly (see [Namespace binding](#namespace-binding) for navigation paths).
2. **Tasks in a flow**: use `io.kestra.plugin.core.kv.Set`, `Get`, and `Delete` tasks.
3. **REST API**: create, read, and delete KV pairs via the HTTP API.
4. **Terraform**: use the `kestra_kv` resource.
5. **Pebble function**: use `kv()` to retrieve a value inline in a flow expression.
6. **GitHub Actions**: manage KV pairs in your CI/CD pipeline.
7. **kestractl**: use `kestractl kv` to list, set, get, and delete from the command line. See the [kestractl docs](../../kestra-cli/kestractl/index.md) for setup.
### Create a KV pair from the UI
Open the **KV Store** tab and click **New Key-Value**. Enter a key name, select a type (string, number, boolean, datetime, date, duration, or JSON), and enter the value. Optionally set a TTL — choose a standard duration from the dropdown or select **Custom duration** to enter an ISO 8601 duration string. Click **Save**.
### Update, delete, and copy KV pairs
Edit, delete, or copy any KV pair using the action buttons on the right. The copy option copies the [Pebble expression for the KV pair](#read-kv-pairs-with-pebble) (`{{ kv('YOUR_KEY') }}`) ready to paste into a flow.
## KV tasks in flows
### Create a KV pair with the `Set` task
Use `io.kestra.plugin.core.kv.Set` to create or update a KV pair from a flow:
```yaml
id: add_kv_pair
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: set_kv
type: io.kestra.plugin.core.kv.Set
key: my_key
value: "{{ outputs.download.uri }}"
namespace: company.team # the current namespace of the flow is used by default
overwrite: true # whether to overwrite or fail if a value for that key already exists; default true
ttl: P30D # optional Time to Live (TTL) for the KV pair
- id: set_simple_kv
type: io.kestra.plugin.core.kv.Set
key: simple_string
value: hello from Kestra
- id: set_json_kv
type: io.kestra.plugin.core.kv.Set
key: json_kv
value: |
{
"author": "Rick Astley",
"song": "Never Gonna Give You Up"
}
- id: get_kv
type: io.kestra.plugin.core.output.OutputValues
values:
my_key: "{{ kv('my_key') }}"
simple_string: "{{ kv('simple_string') }}"
favorite_song: "{{ fromJson(kv('json_kv')).song }}"
```
Set `overwrite: false` to fail instead of silently replacing an existing value. The default is `true`.
### Read KV pairs with Pebble
Use `{{ kv('YOUR_KEY') }}` to retrieve a value inline. The full signature is:
```
{{ kv(key='your_key_name', namespace='your_namespace_name', errorOnMissing=false) }}
```
When reading from the same namespace as the flow, the namespace argument is optional:
```yaml
id: read_kv_pair
namespace: company.team
tasks:
- id: log_key
type: io.kestra.plugin.core.log.Log
message: "{{ kv('my_key') }}"
```
To read from another namespace, pass it as the second argument:
```yaml
id: read_kv_pair_from_another_namespace
namespace: company.team
tasks:
- id: log_key_from_another_namespace
type: io.kestra.plugin.core.log.Log
message: "{{ kv('my_key', 'kestra.engineering.myproject') }}"
```
By default, referencing a missing key causes the task to fail. Set `errorOnMissing=false` to return `null` instead:
```yaml
id: read_non_existing_kv_pair
namespace: company.team
tasks:
- id: log_key_from_another_namespace
type: io.kestra.plugin.core.debug.Return
format: "{{ kv('non_existing_key', errorOnMissing=false) }}"
```
### Read KV pairs with the `Get` task
The `Get` task produces a `value` output you can reference in downstream tasks — useful when you need to pass the same KV value to multiple steps:
```yaml
id: get_kv_pair
namespace: company.team
tasks:
- id: get
type: io.kestra.plugin.core.kv.Get
key: my_key
namespace: company.team
errorOnMissing: false
- id: log_key_get
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.get.value }}"
```
### Read and parse JSON-type values from KV pairs
To parse JSON values in Kestra's templated expressions, wrap the `kv()` call in the `fromJson()` function: `"{{ fromJson(kv('your_json_key')).json_property }}"`.
This example sets a JSON KV pair and reads individual fields using `fromJson()`:
```yaml
id: kv_json_flow
namespace: company.team
tasks:
- id: set_json_kv
type: io.kestra.plugin.core.kv.Set
key: favorite_song
value: |
{
"author": "Rick Astley",
"song": "Never Gonna Give You Up",
"album": {
"name": "Whenever You Need Somebody",
"release_date": "1987-11-16"
}
}
- id: parse_json_kv
type: io.kestra.plugin.core.log.Log
message:
- "Author: {{ fromJson(kv('favorite_song')).author }}"
- "Song: {{ fromJson(kv('favorite_song')).song }}"
- "Album name: {{ fromJson(kv('favorite_song')).album.name }}"
- "Album release date: {{ fromJson(kv('favorite_song')).album.release_date }}"
- id: get
type: io.kestra.plugin.core.kv.Get
key: favorite_song
- id: parse_json_from_kv
type: io.kestra.plugin.core.log.Log
message: "Album name: {{ fromJson(outputs.get.value).album.name }}"
```
### Read keys by prefix with the `GetKeys` task
Search for keys matching a prefix with `GetKeys`:
```yaml
id: get_keys_by_prefix
namespace: company.team
tasks:
- id: get
type: io.kestra.plugin.core.kv.GetKeys
prefix: "test_"
namespace: company.team
- id: log_key_prefix
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.get.keys }}"
```
The output is a list of matching keys, or an empty list if none are found.
### Delete a KV pair with the `Delete` task
`Delete` produces a boolean `deleted` output confirming whether the key existed and was removed:
```yaml
id: delete_kv_pair
namespace: company.team
tasks:
- id: kv
type: io.kestra.plugin.core.kv.Delete
key: my_key
namespace: company.team
errorOnMissing: false
- id: check_if_deleted
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.kv.deleted }}"
```
---
## REST API
### Create a KV pair
Use a `PUT` request to set a KV pair:
```bash
curl -X PUT -H "Content-Type: application/json" http://localhost:8080/api/v1/main/namespaces/company.team/kv/my_key -d '"Hello World"'
```
The API returns no response body on success.
### Read all keys in a namespace
```bash
curl -G "http://localhost:8080/api/v1/main/kv" \
--data-urlencode "filters[namespace][EQUALS]=company.team" \
-H "Authorization: Bearer "
```
The response is a JSON array of keys with their creation and update timestamps:
```json
[
{"key":"my_key","creationDate":"2024-07-27T06:10:33.422Z","updateDate":"2024-07-27T06:11:08.911Z"},
{"key":"test_key","creationDate":"2024-07-27T04:37:18.196Z","updateDate":"2024-07-27T04:37:18.196Z"}
]
```
:::alert{type="info"}
Your Kestra instance exposes an interactive API reference at `https:///api` listing all available endpoints.
:::
### Delete a KV pair
```bash
curl -X DELETE -H "Content-Type: application/json" http://localhost:8080/api/v1/main/namespaces/company.team/kv/my_key
```
Returns `true` if the key existed and was deleted, `false` if it did not exist.
---
## Terraform
Use the `kestra_kv` resource to create or update a KV pair:
```hcl
resource "kestra_kv" "my_key" {
namespace = "company.team"
key = "my_key"
value = "Hello World"
type = "STRING"
}
```
Use the `kestra_kv` data source to read a KV pair:
```hcl
data "kestra_kv" "new" {
namespace = "company.team"
key = "my_key"
}
```
Run `terraform apply` to create, update, or delete KV pairs from your Terraform state.
---
# Namespace Files in Kestra: Manage Project Assets
URL: https://kestra.io/docs/concepts/namespace-files
> Manage Namespace Files in Kestra and use them in your flows. Store scripts, configs, and assets at the namespace level for centralized file management.
Namespace Files are files tied to a namespace — scripts, queries, configs, and other assets you can reference in any flow within that namespace.
Namespace Files are files tied to a given namespace. You can think of Namespace Files as the equivalent of a project in your local IDE or a copy of your Git repository.
Namespace Files can hold Python files, R or Node.js scripts, SQL queries, dbt or Terraform projects, and much more.
You can synchronize your Git repository with a specific namespace to orchestrate dbt, Terraform or Ansible, or any other project that contains code and configuration files.
Once you add any file to a namespace, you can reference it inside your flows using the `read()` function in EVERY task or trigger from the same namespace.
For instance, if you add a SQL query called `my_query.sql` to the `queries` directory in the `company.team` namespace, you can reference it in any `Query` task or any JDBC Trigger like so: `{{ read('queries/my_query.sql') }}`.
Here is an example showing how you can use the `read()` function in a [ClickHouse Trigger](/plugins/plugin-jdbc-clickhouse/io.kestra.plugin.jdbc.clickhouse.trigger) to read a SQL query stored as a Namespace File:
```yaml
id: jdbc_trigger
namespace: company.team
tasks:
- id: for_each_row
type: io.kestra.plugin.core.flow.Loop
values: "{{ trigger.rows }}"
tasks:
- id: return
type: io.kestra.plugin.core.debug.Return
format: "{{ fromJson(item.value) }}"
triggers:
- id: query_trigger
type: io.kestra.plugin.jdbc.clickhouse.Trigger
interval: "PT5M"
url: jdbc:clickhouse://127.0.0.1:56982/
username: "{{ secret('CLICKHOUSE_USERNAME') }}"
password: "{{ secret('CLICKHOUSE_PASSWORD') }}"
sql: "{{ read('queries/my_query.sql') }}"
fetchType: FETCH
```
:::alert{type="info"}
The `namespaceFiles.enabled: true` property is not required here — it is only needed to inject an entire directory of namespace files into the working directory of a script task. If you only need to read a file’s contents, use `read()` without mounting; mounting is for when the task needs files on disk.
:::
## Why use Namespace Files
Namespace Files let you store scripts, queries, and configs directly in Kestra rather than cloning a Git repository at runtime. Files live in Kestra's internal storage backend and are shared across all flows in the namespace, so you maintain one copy instead of duplicating code between flows.
Common use cases:
- Centralize SQL queries, Python scripts, or config files used by multiple flows
- Sync a full Git project (dbt, Terraform, Ansible) to a namespace and orchestrate it without per-flow cloning
- Share code across teams whose files live in different repositories or Git providers
## How to add Namespace Files
### Embedded code editor
Access Namespace Files from the **Files** tab while creating or editing a flow. From there you can write, import, or paste scripts, queries, and configuration files directly.
To try it, create a folder named `scripts` and a file called `hello.py`:
```python
print("Hello from the Editor!")
```
Once you added a file, you can use it in your flow:
```yaml
id: editor
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
commands:
- python scripts/hello.py
```
The **Execute** button allows you to run your flow directly from the Code Editor. Click **Execute** to run the flow. The **Logs** tab shows a friendly message ``Hello from the Editor!``.
### Namespace Files Revision History
Namespace Files include revision history just like flows, so you can inspect or roll back earlier uploads without leaving the Editor.
- First upload of a path is stored as `queries/my_query.sql` and treated as version 0 for backward compatibility.
- Each subsequent upload keeps `queries/my_query.sql` as the latest version while adding suffixed revisions such as `queries/my_query.sql.v1`, `queries/my_query.sql.v2`, and so on.
- Older revisions remain available under their suffixed filenames, letting you compare and restore as needed.
To access a file's revision history, right-click on the file.

From the history, view, compare, and restore prior versions.

From the **Revisions** list, you can delete a given revision or all revisions older than the selected one. You will be prompted to confirm this choice, as there is no possible way to restore a revision once is has been deleted.
To keep your version history clean, you can purge "N" number of Namespace File revisions or revisions older than a certain date. Refer to the [Purge documentation](../../10.administrator-guide/purge/index.md#purge-namespace-files).
### PushNamespaceFiles and SyncNamespaceFiles tasks
There are two tasks to help you automatically manage your namespace files with Git. This allows you to sync the latest changes from a Git repository.
This example pushes Namespace Files you already have in Kestra to a Git repository for you:
```yaml
id: push_to_git
namespace: system
tasks:
- id: commit_and_push
type: io.kestra.plugin.git.PushNamespaceFiles
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/git_username/scripts
branch: dev
namespace: company.team
files:
- "example.py"
gitDirectory: _files
commitMessage: "add namespace files"
dryRun: true
```
This example syncs Namespace Files inside of a Git repository to your Kestra instance:
```yaml
id: sync_files_from_git
namespace: system
tasks:
- id: sync_files
type: io.kestra.plugin.git.SyncNamespaceFiles
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/git_username/scripts
branch: main
namespace: git
gitDirectory: _files
dryRun: true
```
Check out the dedicated guides for more information:
- [PushNamespaceFiles](../../15.how-to-guides/pushnamespacefiles/index.md)
- [SyncNamespaceFiles](../../15.how-to-guides/syncnamespacefiles/index.md)
### GitHub Actions CI/CD
Use the official Kestra [GitHub Actions](../../version-control-cicd/cicd/01.github-action/index.md) to upload namespace files directly from your repository. This is ideal for promoting configuration, scripts, or other assets that live alongside your code.
Example workflow deploying the `scripts/` folder to the `prod` namespace using the `deploy-namespace-files` action:
```yaml
name: Kestra Namespace Files
on: [push]
jobs:
upload-namespace-files:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Upload scripts folder to prod
uses: kestra-io/github-actions/deploy-namespace-files@main
with:
localPath: ./scripts # folder in the repo
namespacePath: scripts # destination path in the namespace
namespace: prod
server: ${{ secrets.KESTRA_HOSTNAME }}
# Choose one auth method:
# apiToken: ${{ secrets.KESTRA_API_TOKEN }} # Enterprise Edition
user: ${{ secrets.KESTRA_USERNAME }} # Basic auth
password: ${{ secrets.KESTRA_PASSWORD }}
```
:::alert{type="info"}
- Store credentials as GitHub Secrets. Provide `tenant` when targeting multi-tenant Enterprise environments.
- Ensure the service account role grants namespace file permissions (and `FLOWS` when deploying flows) to your target namespace.
:::
### Terraform provider
You can use the `kestra_namespace_file` resource from the official [Kestra Terraform Provider](https://registry.terraform.io/providers/kestra-io/kestra/latest/docs) to deploy all your custom script files from a specific directory to a given Kestra namespace.
Below is a simple example showing how you can synchronize an entire directory of scripts from the directory `src` with the `company.team` namespace using Terraform:
```hcl
resource "kestra_namespace_file" "prod_scripts" {
for_each = fileset(path.module, "src/**")
namespace = "company.team"
filename = each.value # or "/${each.value}"
content = file(each.value)
}
```
### Deploy namespace files via kestractl
You can upload namespace files from the command line using [kestractl](../../kestra-cli/kestractl/index.md). The following example synchronizes an entire local directory with the `prod` namespace:
```bash
kestractl nsfiles upload prod ./scripts --override
```
To upload to a specific path within the namespace rather than the root:
```bash
kestractl nsfiles upload prod ./assets --path resources --override --fail-fast
```
The `--override` flag replaces existing files; `--fail-fast` stops on the first error rather than continuing.
`kestractl nsfiles` also supports `list`, `get`, and `delete` for inspecting and removing individual files. Run `kestractl nsfiles --help` for the full reference.
## How to use Namespace Files in your flows
There are multiple ways to use Namespace Files in your flows. You can use the `read()` function to read the content of a file as a string, point to the file path in the supported tasks, or use a dedicated task to retrieve it as an output.
:::alert{type="info"}
Kestra 0.24 introduced a universal file protocol that simplifies accessing files — local or namespace — in your flow. For more details, refer to the [File Access documentation page](../file-access/index.md).
:::
Usually, pointing to a file location, rather than reading the file's content, is required when you want to use a file as an input to a CLI command (e.g., in a `Commands` task such as `io.kestra.plugin.scripts.python.Commands` or `io.kestra.plugin.scripts.node.Commands`). In all other cases, the `read()` function can be used to read the content of a file as a string (e.g., in `Query` or `Script` tasks).
You can also use the `io.kestra.plugin.core.flow.WorkingDirectory` task to read namespace files and then use them in child tasks that require a file path in CLI commands, for example: `python scripts/hello.py`.
### The `read()` function
`read()` returns the **contents** of a namespace file as a string. Use it in tasks that accept string input — `io.kestra.plugin.scripts.python.Script`, `io.kestra.plugin.scripts.node.Script`, SQL query properties, and similar — not in `Commands` tasks that expect a file path on disk. The path must point to a file in the same namespace as the flow.
This example logs the contents of `example.txt`:
```yaml
id: files
namespace: company.team
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "{{ read('example.txt') }}"
```
### `namespaceFiles.enabled` on supported tasks
With supported tasks, such as the `io.kestra.plugin.scripts` group, we can access files using their path and enabling the task to read namespace files.
Below is a simple `weather.py` script that reads a secret to talk to a Weather Data API:
```python
import requests
api_key = '{{ secret("WEATHER_DATA_API_KEY") }}'
url = f"https://api.openweathermap.org/data/2.5/weather?q=Paris&APPID={api_key}"
weather_data = requests.get(url)
print(weather_data.json())
```
Next, is a flow that uses the script:
```yaml
id: weather_data
namespace: company.team
tasks:
- id: get_weather
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
include:
- scripts/weather.py
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/pydata:latest
commands:
- python scripts/weather.py
```
#### `namespaceFiles` property
The example above uses the `include` field to only allow the `scripts/weather.py` file to be accessible by the task.
We can control what namespace files are available to our flow with the `namespaceFiles` property.
`namespaceFiles` has several configurable attributes:
- `enabled`: when set to `true`, makes all files in the namespace visible to the task.
- `include`: restricts which files are accessible — only the listed paths are mounted.
- `exclude`: mounts all namespace files except those listed.
- `namespaces`: a list of additional namespaces to load files from.
- `ifExists`: controls what happens when a namespace file conflicts with an existing file in the working directory.
- `folderPerNamespace`: when `true`, mounts each namespace's files into a separate subdirectory instead of the working directory root.
The `namespaces` attribute can be used like in the following example:
```yaml
id: namespace_files_example
namespace: dev.test
tasks:
- id: namespace
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
namespaces:
- "dev.test"
- "company"
commands:
- python test.py
- id: namespace2
type: io.kestra.plugin.scripts.python.Script
namespaceFiles:
enabled: true
script: "{{ read('test.py') }}"
```
Namespaces are loaded in list order, but when the same file exists in multiple namespaces, the last listed namespace wins. In the first task, `dev.test` is listed first and `company` is listed second, so `company`'s `test.py` takes precedence.
For the second task, the `test.py` file in the `dev.test` namespace will be used because no namespace has been defined in the `read()` function. If you want to fetch the `test.py` script from a different namespace, you need to explicitly define it as follows: `"{{ read('test.py', namespace='company.team') }}"`.
The `ifExists` attribute has four possible options for behavior when tasks invoke a Namespace file that already exists in the working directory:
- `OVERWRITE`: set by default, adds a debug log that the file was overwritten
- `FAIL`: logs and ERROR and fails the task
- `WARN`: logs a WARNING but continues running the execution
- `IGNORE`: doesn't overwrite the file or log any warnings
For example, in the following task the second instance of `sample_python.py` will overwrite the first:
```yaml
id: test_workdir_issue
namespace: prod
tasks:
- id: git_wdir
type: io.kestra.plugin.core.flow.WorkingDirectory
tasks:
- id: clone
type: io.kestra.plugin.git.Clone
branch: main
url: https://github.com/kestra-io/examples
- id: python_command_1
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
commands:
- python scripts/sample_python.py
- id: python_command_2
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
ifExists: OVERWRITE
commands:
- python scripts/sample_python.py
```
### Namespace tasks
Use the Namespace Tasks to upload, download, and delete files in Kestra.
In the example below, we have a namespace file called `example.ion` that we want to convert to a `.csv` file. We can use the `DownloadFiles` task to generate an output that contains the file so we can easily pass it dynamically to the `IonToCsv` task.
```yaml
id: files
namespace: company.team
tasks:
- id: namespace
type: io.kestra.plugin.core.namespace.DownloadFiles
namespace: company.team
files:
- example.ion
- id: ion_to_csv
type: io.kestra.plugin.serdes.csv.IonToCsv
from: "{{ outputs.namespace.files['/example.ion'] }}"
```
Read more about the tasks below:
- [UploadFiles](/plugins/core/namespace/io.kestra.plugin.core.namespace.uploadfiles)
- [DownloadFiles](/plugins/core/namespace/io.kestra.plugin.core.namespace.downloadfiles)
- [DeleteFiles](/plugins/core/namespace/io.kestra.plugin.core.namespace.deletefiles)
## Include / exclude namespace files
You can selectively include or exclude namespace files.
Let's say that you have multiple namespace files present: file1.txt, file2.txt, file3.json, file4.yml. You can selectively include multiple files using the `include` attribute under `namespaceFiles` as shown below:
```yaml
id: include_namespace_files
namespace: company.team
tasks:
- id: include_files
type: io.kestra.plugin.scripts.shell.Commands
namespaceFiles:
enabled: true
include:
- file1.txt
- file3.json
commands:
- ls
```
The `include_files` task lists all the included files. In the example above, these are `file1.txt` and `file3.json` as only those were included from the namespace through `include`.
The `exclude` attribute, alternatively, includes all the namespace files except those specified under `exclude`.
```yaml
id: exclude_namespace_files
namespace: company.team
tasks:
- id: exclude_files
type: io.kestra.plugin.scripts.shell.Commands
namespaceFiles:
enabled: true
exclude:
- file1.txt
- file3.json
commands:
- ls
```
The `exclude_files` task from the above flow lists `file2.txt` and `file4.yml`, all the namespace files except those that were excluded using `exclude`.
### Pattern matching rules for `include` / `exclude`
- Patterns that do **not** start with `/` are automatically prefixed with `**/`, so they match recursively (e.g., `file1.txt` becomes `**/file1.txt`).
- Patterns that start with `/` match from the namespace root only (e.g., `/config/settings.json`).
- You can force explicit types with `glob:` or `regex:`:
- `glob:/src/**/*.py`
- `regex:^src/.*\\.py$`
Examples:
```yaml
namespaceFiles:
enabled: true
include:
# Root-only matches
- /file1.txt
- /config/settings.json
# Recursive matches (auto **/ prefix)
- file1.txt # becomes **/file1.txt
- src/** # becomes **/src/**
# Explicit glob
- glob:/src/**/*.py
- glob:config/*.json
# Regex
- regex:^src/.*\.py$
- regex:.*test.*\.json
```
:::alert{type="warning"}
Patterns without a leading `/` are automatically prefixed with `**/`, which makes them recursive. Use a leading `/` or explicit `glob:`/`regex:` to restrict matching to the namespace root. Patterns that already contain `**` (e.g. `my_dir/**`) are still prefixed, producing `**/my_dir/**`; use `/my_dir/**` or `glob:/my_dir/**` to avoid the double prefix.
:::
---
# Pebble Templating in Kestra: Dynamic Variables
URL: https://kestra.io/docs/concepts/pebble
> Dynamically render variables, inputs, and outputs in Kestra using Pebble templating. Use expressions to build flexible, data-driven workflows.
Pebble is a Java templating engine inspired by [Twig](https://twig.symfony.com/) and similar to [Jinja](https://palletsprojects.com/p/jinja/). Kestra uses it to dynamically render variables, inputs, and outputs within the execution context. For the full list of available variables, filters, and functions, see the [Expressions](../../expressions/index.md) reference.
## Reading inputs
Access input values in tasks using the `inputs` variable:
```yaml
id: input_string
namespace: company.team
inputs:
- id: name
type: STRING
tasks:
- id: say_hello
type: io.kestra.plugin.core.log.Log
message: "Hello 👋, my name is {{ inputs.name }}"
```
## Reading task outputs
Most tasks expose output values accessible as `outputs..`. The available outputs for each task are listed in its plugin documentation.
This example reads the `value` output of the `Return` task in a downstream `Log` task:
```yaml
id: input_string
namespace: company.team
inputs:
- id: name
type: STRING
tasks:
- id: say_hello
type: io.kestra.plugin.core.debug.Return
format: "Hello 👋, my name is {{ inputs.name }}"
- id: can_you_repeat
type: io.kestra.plugin.core.log.Log
message: '{{ outputs.say_hello.value }}'
```
## Dynamically render a task with `TemplatedTask`
`TemplatedTask` lets you fully template all task properties using Pebble — including properties that are not natively dynamic. This example uses [TemplatedTask](/plugins/core/templating/io.kestra.plugin.core.templating.templatedtask) to create a Databricks job with inputs controlling the cluster, task key, and wait time:
```yaml
id: templated_databricks_job
namespace: company.team
inputs:
- id: host
type: STRING
- id: clusterId
type: STRING
- id: taskKey
type: STRING
- id: pythonFile
type: STRING
- id: sparkPythonTaskSource
type: ENUM
defaults: WORKSPACE
values:
- GIT
- WORKSPACE
- id: maxWaitTime
type: STRING
defaults: "PT30M"
tasks:
- id: templated_spark_job
type: io.kestra.plugin.core.templating.TemplatedTask
spec: |
type: io.kestra.plugin.databricks.job.CreateJob
authentication:
token: "{{ secret('DATABRICKS_API_TOKEN') }}"
host: "{{ inputs.host }}"
jobTasks:
- existingClusterId: "{{ inputs.clusterId }}"
taskKey: "{{ inputs.taskKey }}"
sparkPythonTask:
pythonFile: "{{ inputs.pythonFile }}"
sparkPythonTaskSource: "{{ inputs.sparkPythonTaskSource }}"
waitForCompletion: "{{ inputs.maxWaitTime }}"
```
`waitForCompletion` and `sparkPythonTaskSource` are not natively dynamic properties — `TemplatedTask` makes it possible to drive them from inputs.
---
## Date formatting
Use the `date` filter to format date values inline: `'{{ inputs.my_date | date("yyyyMMdd") }}'`
## Coalesce operator to conditionally use trigger or execution date
Scheduled flows can use `trigger.date` to get the trigger’s date, but that variable is not set on manual executions. Use the coalesce operator `??` to fall back to `execution.startDate` when the trigger date is unavailable:
```yaml
id: pebble_date_trigger
namespace: company.team
tasks:
- id: return_date
type: io.kestra.plugin.core.debug.Return
format: '{{ trigger.date ?? execution.startDate | date("yyyy-MM-dd") }}'
triggers:
- id: schedule
type: io.kestra.plugin.core.trigger.Schedule
cron: "* * * * *"
```
## Parsing objects and lists using jq
Use the `jq` filter to slice, filter, and transform nested objects or lists returned by task outputs — similar to how `sed`, `awk`, and `grep` work on strings.
```yaml
id: object_example
namespace: company.team
inputs:
- id: data
type: JSON
defaults: '{"value": [1, 2, 3]}'
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: "{{ inputs.data }}"
```
The expression `{{ inputs.data.value }}` returns the list `[1, 2, 3]`
The expression `{{ inputs.data.value | jq(".[1]") | first }}` returns `2`.
`jq(".[1]")` accesses the second value of the list and returns an array with one element. We then use `first` to access the value itself.
`{{ inputs | jq(".data.value[1]") | first }}` also works — jq can parse any object in the Kestra context.
Use the **Debug Expression** button in the **Outputs** tab of an execution to troubleshoot complex expressions and validate how objects will be parsed.
## Using conditions in Pebble
Tasks like `If` and `Switch` accept Pebble expressions as conditions, letting you branch on inputs or previous task outputs:
```yaml
id: test-object
namespace: company.team
inputs:
- id: data
type: JSON
defaults: '{"value": [1, 2, 3]}'
tasks:
- id: if
type: io.kestra.plugin.core.flow.If
condition: '{{ inputs.data.value | jq(".[2]") | first == 3 }}'
then:
- id: when_true
type: io.kestra.plugin.core.log.Log
message: 'Condition was true'
else:
- id: when_false
type: io.kestra.plugin.core.log.Log
message: 'Condition was false'
```
---
# Replay Executions in Kestra: Rerun from Any Task
URL: https://kestra.io/docs/concepts/replay
> Replay Kestra workflow executions from any chosen task run. Re-trigger failed or incomplete executions without starting from scratch for faster recovery.
Replay re-runs a workflow execution from any chosen task — skipping tasks that already completed successfully. Use it to recover from failures without reprocessing upstream work, or to iterate on a specific task without re-running the full flow.
To replay from a specific task, open the **Gantt** or **Logs** tab of any execution and use the three-dot menu on the task run. You can also replay a single execution or bulk-replay from the **Executions** page, with the option to use the latest flow revision.
## Example: fixing a failed task and replaying
Consider a flow that downloads a file and then validates its schema:
```yaml
id: replay_demo
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: validate
type: io.kestra.plugin.core.execution.Assert
errorMessage: "Schema validation failed — unexpected number of columns"
conditions:
- "{{ 8 >= 5 }}"
- id: notify
type: io.kestra.plugin.core.log.Log
message: Validation passed, data is ready for processing.
```
Run the flow once with `8 == 5` as the condition — `download` succeeds, then `validate` fails because the assertion is always false.
Open the failed execution and go to the **Gantt** tab. Use the three-dot menu on `validate` to select **Fix with AI** or correct the condition yourself — change `8 == 5` to `8 >= 5` and save as a new revision. Then select **Replay** from the same menu.

In the confirmation dialog, select **Latest flow revision** to use the revision containing your fix.

The `download` task is skipped — Kestra reuses its output from the original execution. Only `validate` and `notify` run again. The **Attempt 2/2** label on `validate` confirms this is the replayed run.

The **Overview** tab shows the attempt number, the revision used, a `system.replay: true` label marking this as a replay, and a `system.correlationId` label linking back to the original execution.

---
# Flow Revisions in Kestra: Versioning and Rollbacks
URL: https://kestra.io/docs/concepts/revision
> Track and manage flow versions in Kestra with built-in revision history. Roll back to any previous version to undo changes and maintain reliability.
Every time you save a flow, Kestra creates a new revision. The **Revisions** tab lists all revisions for a flow — compare any two side-by-side or line-by-line, and roll back to a previous version at any time.

## Draft revisions
When you click **Save as draft** in the flow editor, Kestra saves your changes as a draft revision rather than publishing them immediately.
:::alert{type="warning"}
Executions do not run against a draft revision. If the latest revision of a flow is a draft, any execution — whether triggered manually, by a schedule, or by an event — will run against the last published revision instead. A warning banner in the run panel makes this explicit.
:::
To make your latest changes active, open the run panel — it displays a draft warning banner with a **Publish** button that promotes the draft to a published revision.
Use **Save as draft** when you want to stage changes without affecting running executions — for example, while iterating on a flow that is already in production.
:::alert{type="info"}
On Kestra Enterprise, you can promote a published revision to another environment (staging, production) directly from the flow's **Deploy** tab, without copy-pasting YAML or setting up a CI/CD pipeline. See [Promote flows across environments](../../version-control-cicd/06.promote/index.md).
:::
---
# Secrets in Kestra: Store Sensitive Values Securely
URL: https://kestra.io/docs/concepts/secret
> Store and access sensitive information securely in Kestra. Use Secrets to protect API keys, passwords, and credentials without exposing plain-text values.
Secrets let you store sensitive values (API keys, passwords, certificates) outside your flow definitions and inject them at runtime via the `secret()` function.
How secrets are stored depends on your edition. Enterprise Edition connects to a dedicated Secrets Manager backed by AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, or Kestra's own store. Open-Source has no secret store — `secret()` reads a base64-encoded environment variable instead.
## Enterprise Edition
Secrets are the right choice for static sensitive values: API keys, passwords, webhook URLs, certificates, and long-lived tokens. For reusable server-to-server authentication — where Kestra needs to mint or refresh short-lived tokens at runtime — use [Credentials](../../07.enterprise/03.auth/credentials/index.md) instead. Credentials can reference secrets for sensitive inputs such as client secrets and private keys.
Secrets are available under **Namespaces → [namespace] → Secrets** or under **Tenant → Secrets** in the sidebar. Click **New secret**, set a key name such as `MY_SECRET`, and optionally add a description and tags. From the same tab you can edit, delete, or copy a secret as a Pebble expression — for example, `"{{ secret('API_TOKEN') }}"` — ready to paste into a flow.
For available backends, see the [Secrets Manager](../../07.enterprise/02.governance/secrets-manager/index.md) page. For best practices, see [Secrets management](../../14.best-practices/9.secrets-management/index.md) and [Choosing where to store sensitive and shared values](../../14.best-practices/10.credentials-vs-secrets-vs-kv-store/index.md).
### Reading secrets from another namespace
By default, `secret()` reads from the flow's own namespace. You can pass a `namespace` argument to read a secret stored in a different namespace:
```yaml
tasks:
- id: use_shared_secret
type: io.kestra.plugin.core.log.Log
message: "{{ secret('SHARED_TOKEN', namespace='shared.secrets') }}"
```
The secret resolves using the target namespace's own backend, so a flow can read a value from a namespace backed by a different secrets manager. Cross-namespace reads stay within the same tenant. Access is allowed by default; restrict it by configuring `allowedNamespaces` on the target namespace.
## Secrets in Open-Source
Open-Source has no dedicated secret store. Kestra reads base64-encoded environment variables prefixed with `SECRET_` and exposes them via the `secret()` function. This keeps sensitive values out of flow YAML, but offers no encryption at rest, no audit trail, and no access control beyond what your host environment provides.
See [Configure secrets in Kestra (OSS)](../../15.how-to-guides/secrets/index.md) for step-by-step instructions on encoding values and wiring them into your Docker Compose file.
---
# Data Storage in Kestra: How Task Data Is Managed
URL: https://kestra.io/docs/concepts/storage
> Understand how Kestra stores and processes task data. Learn about internal storage, file handling, and how outputs are passed between tasks in your workflows.
Kestra stores task data in the execution context, in internal storage, or in the KV store — each suited to different data sizes and lifetimes.
Depending on the task, data can be stored inside the execution context or inside Kestra's internal storage. You can also store data in the KV store using [dedicated tasks](/plugins/core/kv/io.kestra.plugin.core.kv.set).
Some tasks give you the choice of where you want to store the data, usually using a `fetchType` property or the three `fetch`/`fetchOne`/`store` properties.
For example, using the DynamoDB Query task:
```yaml
id: query
type: io.kestra.plugin.aws.dynamodb.Query
tableName: persons
keyConditionExpression: id = :id
expressionAttributeValues:
:id: "1"
fetchType: FETCH
```
The `fetchType` property can have four values:
- `FETCH_ONE`: fetches the first row and set it in a task output attribute (the `row` attribute for DynamoDB); the data is stored inside the execution context.
- `FETCH`: fetches all rows and set them in a task output attribute (the `rows` attribute for DynamoDB); the data is stored inside the execution context.
- `STORE`: stores all rows inside Kestra's internal storage. The internal storage returns a URI usually set in the task output attribute `uri` and that can be used to retrieve the file from the internal storage.
- `NONE`: does nothing.
The three `fetch`/`fetchOne`/`store` properties do the same but using three different task properties instead of a single one.
## Storing data
### Storing data inside the flow execution context
Data can be stored as variables inside the flow execution context. This can be convenient for sharing data between tasks.
To do so, tasks store data as [output attributes](../../05.workflow-components/06.outputs/index.md) that are then available inside the flow via Pebble expressions like `{{ outputs.taskName.attributeName }}`.
Large values stored in the execution context increase its size, which can slow execution and grow Kestra's repository storage.
:::alert{type="warning"}
Depending on the Kestra internal queue and repository implementation, there can be a hard limit on the size of the flow execution context as it is stored as a single row/message. Usually, this limit is around 1MB, so this is important to avoid storing large amounts of data inside the flow execution context.
:::
### Storing data inside the internal storage
Kestra has an internal storage that can store data of any size. By default, the internal storage uses the host filesystem, but plugins exist to use other implementations like Amazon S3, Google Cloud Storage, or Microsoft Azure Blobs storage. See [Runtime and Storage](../../configuration/02.runtime-and-storage/index.md).
When using the internal storage, data is, by default, stored using [Amazon Ion](https://amazon-ion.github.io/ion-docs/) format.
Tasks that can store data inside the internal storage usually have an output attribute named `uri` that can be used to access this file in following tasks.
The following example uses the [DynamoDB Query](/plugins/plugin-aws/aws-dynamodb/io.kestra.plugin.aws.dynamodb.query) task to query a table and the [FTP Upload](/plugins/plugin-fs/ftp-file-transfer-protocol/io.kestra.plugin.fs.ftp.upload) task to send the retrieved rows to an external FTP server.
```yaml
tasks:
- id: query
type: io.kestra.plugin.aws.dynamodb.Query
tableName: persons
keyConditionExpression: id = :id
expressionAttributeValues:
:id: "1"
fetchType: STORE
- id: upload
type: io.kestra.plugin.fs.ftp.Upload
host: localhost
port: 80
from: "{{ outputs.query.uri }}"
to: "/upload/file.ion"
```
If you need to access data from the internal storage, you can use the `read()` function to read the file's content as a string.
Dedicated tasks allow managing the files stored inside the internal storage:
- [Concat](/plugins/core/storage/io.kestra.plugin.core.storage.concat): concat multiple files.
- [Delete](/plugins/core/storage/io.kestra.plugin.core.storage.delete): delete a file.
- [Size](/plugins/core/storage/io.kestra.plugin.core.storage.size): get the size of a file.
- [Split](/plugins/core/storage/io.kestra.plugin.core.storage.split): split a file into multiple files depending on the size of the file or the number of rows.
:::alert{type="warning"}
Use internal storage for large data. If an [HTTP Request](/plugins/core/http/io.kestra.plugin.core.http.request) returns a heavy payload, use [HTTP Download](/plugins/core/http/io.kestra.plugin.core.http.download) with a [Serdes](/plugins/plugin-serdes) task instead of carrying the raw data in the [execution context](#storing-data-inside-the-flow-execution-context).
:::
### Storing data inside the KV store
Dedicated tasks can store data inside Kestra's KV store. The KV store transparently uses Kestra's internal storage as its backend store.
The KV store allows storing data that will be shared by all executions of the same namespace. You can think of it as a key/value store dedicated to a namespace.
The following tasks are available:
- [Set](/plugins/core/kv/io.kestra.plugin.core.kv.set): set data in key/value pair.
- [Get](/plugins/core/kv/io.kestra.plugin.core.kv.get): get data from key/value pair.
- [Delete](/plugins/core/kv/io.kestra.plugin.core.kv.delete): delete a key/value pair.
Example:
```yaml
tasks:
- id: set_data
type: io.kestra.plugin.core.kv.Set
key: name
value: John Doe
- id: get_data
type: io.kestra.plugin.core.kv.Get
key: name
```
In the next example, the flow uses `Set`, `Get` and `Delete` on the data:
:::collapse{title="Example Flow"}
```yaml
id: kv_store_example
namespace: company.team
tasks:
- id: set_data
type: io.kestra.plugin.core.kv.Set
key: user_name
value: John Doe
- id: get_data
type: io.kestra.plugin.core.kv.Get
key: user_name
- id: log_state
type: io.kestra.plugin.core.log.Log
message: "{{ kv('user_name') }}"
- id: set_new_data
type: io.kestra.plugin.core.kv.Set
key: user_name
value: Bob Smith
- id: get_new_data
type: io.kestra.plugin.core.kv.Get
key: user_name
- id: log_new_data
type: io.kestra.plugin.core.log.Log
message: "{{ kv('user_name') }}"
- id: delete_data
type: io.kestra.plugin.core.kv.Delete
key: user_name
- id: get_deleted_data
type: io.kestra.plugin.core.kv.Get
description: You will not get any data as the corresponding key is deleted in the earlier task.
key: user_name
```
When we `Set` a new value for `user_name`, we have to use another `Get` task to retrieve the most up-to-date value, and then reference that `Get` task `id` in the log below to show the latest value. The same applies to the `Delete` task. To show that the value has been deleted, we try to retrieve data from the key deleted in the `delete_data` task.
:::
## Processing data
For basic data processing, you can leverage Kestra's [Pebble templating engine](../../expressions/index.mdx).
For more complex data transformations, Kestra offers various data processing plugins including transform tasks or custom scripts.
### Converting files
Files from the internal storage can be converted from/to the Ion format to/from another format using the [Serdes](/plugins/plugin-serdes) plugin.
The following formats are currently available: Avro, JSON, XML, and Parquet.
Each format offers a **reader** to read an Ion serialized data file and write it in the target format and a **writer** to read a file in a specific format and write it as an Ion serialized data file.
For example, to convert an Ion file to CSV, then back to Ion:
```yaml
tasks:
- id: query
type: io.kestra.plugin.aws.dynamodb.Query
tableName: persons
keyConditionExpression: id = :id
expressionAttributeValues:
:id: "1"
fetchType: STORE
- id: convertToCsv
type: io.kestra.plugin.serdes.csv.IonToCsv
from: "{{ outputs.query.uri }}"
- id: convertBackToIon
type: io.kestra.plugin.serdes.csv.CsvToIon
from: "{{ outputs.convertToCsv.uri }}"
```
### Processing data using scripts
Kestra can run Python, R, Node.js, Shell, PowerShell, and Go scripts in a local process or inside a Docker container, depending on the configured `runner`.
These script tasks are available in the [Scripts Plugin](https://github.com/kestra-io/plugin-scripts):
- The [Python](/plugins/plugin-script-python/io.kestra.plugin.scripts.python.script) task runs a Python script in a Docker container or in a local process.
- The [Node](/plugins/plugin-script-node/io.kestra.plugin.scripts.node.script) task runs a Node.js script in a Docker container or in a local process.
- The [R](/plugins/plugin-script-r/io.kestra.plugin.scripts.r.script) task runs an R script in a Docker container or in a local process.
- The [Shell](/plugins/plugin-script-shell/io.kestra.plugin.scripts.shell.script) task executes a single Shell command, or a list of commands that you provide.
- The [PowerShell](/plugins/plugin-script-powershell/io.kestra.plugin.scripts.powershell.script) task executes a single PowerShell command, or a list of commands that you provide.
- The [Go (Script)](/plugins/plugin-script-go/io.kestra.plugin.scripts.go.script) task executes a single multi-line script, while the [Go (Commands)](/plugins/plugin-script-go/io.kestra.plugin.scripts.go.commands) task executes a list of commands that you provide.
The following example queries the BigQuery public dataset with Wikipedia page views to find the top 10 pages, convert it to CSV, and use the CSV file inside a Python task for further transformations using Pandas.
```yaml
id: wikipedia-top-ten-python-panda
namespace: company.team
description: analyze top 10 Wikipedia pages
tasks:
- id: query
type: io.kestra.plugin.gcp.bigquery.Query
sql: |
SELECT DATETIME(datehour) as date, title, views FROM `bigquery-public-data.wikipedia.pageviews_2023`
WHERE DATE(datehour) = current_date() and wiki = 'en'
ORDER BY datehour desc, views desc
LIMIT 10
store: true
projectId: geller
serviceAccount: "{{ envs.gcp_creds }}"
- id: write-csv
type: io.kestra.plugin.serdes.csv.IonToCsv
from: "{{ outputs.query.uri }}"
- id: wdir
type: io.kestra.plugin.core.flow.WorkingDirectory
inputFiles:
data.csv: "{{ outputs['write-csv'].uri }}"
tasks:
- id: pandas
type: io.kestra.plugin.scripts.python.Script
containerImage: ghcr.io/kestra-io/pydata:latest
script: |
import pandas as pd
from kestra import Kestra
df = pd.read_csv("data.csv")
views = df['views'].sum()
Kestra.outputs({'views': int(views)})
```
Kestra offers several plugins for ingesting and transforming data — check [the Plugin list](/plugins) for more details.
Make sure to also check:
1. The [Script documentation](../../16.scripts/index.mdx) for a detailed overview of how to work with Python, R, Node.js, Shell and Powershell scripts, and how to integrate them with Git and Docker.
2. The [Blueprints](/blueprints) catalog — simply search for the relevant language (e.g., Python, R, Rust) or use case (*ETL, Git, dbt, etc.*) to find the relevant examples.
### Processing data using file transform
Kestra can process data **row by row** using file transform tasks. The transformation is done with a small script written in Python, JavaScript, or Groovy.
- The [GraalVM Python FileTransform](/plugins/plugin-graalvm/python-graalvm/io.kestra.plugin.graalvm.python.filetransform) task allows transforming rows with Python.
- The [GraalVM JavaScript FileTransform](/plugins/plugin-graalvm/js-graalvm/io.kestra.plugin.graalvm.js.filetransform) task allows transforming rows with JavaScript.
- The [Groovy Script](/plugins/plugin-script-groovy/io.kestra.plugin.scripts.groovy.script) task allows running scripts with Groovy.
The following example queries the BigQuery public dataset for Wikipedia pages, transforms each row with the GraalVM Python FileTransform, and writes the result to a CSV file.
```yaml
id: wikipedia-top-ten-file-transform
namespace: company.team
description: A flow that loads wikipedia top 10 EN pages
tasks:
- id: query-top-ten
type: io.kestra.plugin.gcp.bigquery.Query
sql: |
SELECT DATETIME(datehour) as date, title, views FROM `bigquery-public-data.wikipedia.pageviews_2023`
WHERE DATE(datehour) = current_date() and wiki = 'en'
ORDER BY datehour desc, views desc
LIMIT 10
store: true
- id: file-transform
type: io.kestra.plugin.graalvm.python.FileTransform
from: "{{ outputs['query-top-ten'].uri }}"
script: |
logger.info('row: {}', row)
if (row['title'] === 'Main_Page' || row['title'] === 'Special:Search' || row['title'] === '-') {
// remove un-needed row
row = null
} else {
// add a 'time' column
row['time'] = String(row['date']).substring(11)
// modify the 'date' column to only keep the date part
row['date'] = String(row['date']).substring(0, 10)
}
- id: write-csv
type: io.kestra.plugin.serdes.csv.IonToCsv
from: "{{ outputs['file-transform'].uri }}"
```
:::alert{type="info"}
The script can access a logger to log messages. Each row is available in a `row` variable where each column is accessible using the dictionary notation `row['columnName']`.
:::
## Purging data
The PurgeExecution task can purge all the files stored inside the internal context by a flow execution.
It can be used at the end of a flow to purge all its generated files.
```yaml
tasks:
- id: purge-execution
type: io.kestra.plugin.core.storage.PurgeExecution
```
The execution context itself is not available after the end of the execution and is automatically deleted from Kestra's repository after a retention period (seven days by default) that can be changed; see [Runtime and Storage](../../configuration/02.runtime-and-storage/index.md).
Also, the [Purge](/plugins/core) task can be used to purge storages, logs, and executions of previous execution. For example, this flow purges all of these every day:
```yaml
id: purge
namespace: company.team
tasks:
- id: purge
type: io.kestra.plugin.core.storage.Purge
endDate: "{{ now() | dateAdd(-1, 'MONTHS') }}"
triggers:
- id: schedule
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 0 * * *"
```
## FAQ
### Internal storage FAQ
#### How to read a file from internal storage as a string
The `read()` function expects a `path` argument that points to a namespace file or an internal storage URI. Note that when using inputs, outputs, or trigger variables, you don't need any extra quotation marks. Here is how you can use those variables with the `read()` function:
- `{{ read(inputs.file) }}` for a FILE-type input variable named `file`
- `{{ read(outputs.mytaskid.uri) }}` for an output `uri` from a task named `mytaskid`
- `{{ read(trigger.uri) }}` for a `uri` of many triggers incl. Kafka, AWS SQS, GCP PubSub, etc.
- `{{ read(trigger.objects | jq('.[].uri')) }}` for a `uri` of a trigger that returns a list of detected objects, e.g. AWS S3, GCP GCS, etc.
Note that the read function can only read files within the same execution. If you try to read a file from a previous execution, you will get an Unauthorized error.
:::collapse{title="Example using a FILE-type inputs variable"}
```yaml
id: read_file_as_string
namespace: company.team
inputs:
- id: file
type: FILE
tasks:
- id: log_internal_storage_uri
type: io.kestra.plugin.core.log.Log
message: "{{ inputs.file }}"
- id: log_file_content
type: io.kestra.plugin.core.log.Log
message: "{{ read(inputs.file) }}"
```
:::
:::collapse{title="Example with the Loop task passing row data to a subflow"}
When using the `Loop` task with a storage URI, each iteration receives one row from the file as `item.value`. You can pass that value directly as an input to a subflow.
Below is a simple subflow example that uses a string input:
```yaml
id: subflow_row_input
namespace: company.team
inputs:
- id: row
type: STRING
defaults: "{}"
tasks:
- id: log
type: io.kestra.plugin.core.debug.Return
format: "{{ inputs.row }}"
```
The parent flow queries a dataset, stores the result as a URI, then loops over each row and passes it to the subflow:
```yaml
id: parent_flow
namespace: company.team
tasks:
- id: extract
type: io.kestra.plugin.jdbc.duckdb.Queries
sql: |
INSTALL httpfs;
LOAD httpfs;
SELECT *
FROM read_csv_auto('https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv', header=True);
store: true
- id: loop
type: io.kestra.plugin.core.flow.Loop
values: "{{ outputs.extract.outputs[0].uri }}"
tasks:
- id: process
type: io.kestra.plugin.core.flow.Subflow
namespace: company.team
flowId: subflow_row_input
wait: true
inputs:
row: "{{ item.value | toJson }}"
```
:::
#### How to read a Namespace File as a string?
So far, you've seen how to read a file from the internal storage as a string. However, you can use the same `read()` function to read a Namespace File as a string. This is especially useful when you want to execute a Python script or a long SQL query stored in a dedicated SQL file.
The `read()` function takes the absolute path to the file you want to read. The path must point to a file stored in the **same namespace** as the flow you are executing.
Below is a simple example showing how you can read a file named `hello.py` stored in the `scripts` directory of the `company.team` namespace:
```yaml
id: hello
namespace: company.team
tasks:
- id: my_python_script
type: io.kestra.plugin.scripts.python.Script
script: "{{ read('scripts/hello.py') }}"
```
The same syntax applies to SQL queries, custom scripts, and many more. Check the [Namespace Files](../../06.concepts/02.namespace-files/index.md) documentation for more details.
#### How to read a file from the internal storage as a JSON object?
You can use the Pebble function `{{ fromJson(myvar) }}` and a `{{ myvar | toJson }}` filter to process JSON data.
:::collapse{title="The fromJson() function"}
`fromJson()` converts a JSON string to an object. For example, this expression returns `42`:
```yaml
{{ fromJson('{"foo": [42, 43, 44]}').foo[0] }}
```
Combine `read()` with `fromJson()` to load a file from internal storage and parse its contents:
```yaml
id: extract_json
namespace: company.team
tasks:
- id: extract
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/json/app_events.json
- id: read_as_string
type: io.kestra.plugin.core.log.Log
message: "{{ read(outputs.extract.uri) }}"
- id: read_as_json
type: io.kestra.plugin.core.log.Log
message: "{{ fromJson(read(outputs.extract.uri)) }}"
- id: parse_json_elements
type: io.kestra.plugin.core.log.Log
message: "{{ fromJson(read(outputs.extract.uri)) | jq('map(.detail | fromjson | .message)') | first }}"
```
:::
:::collapse{title="The toJson filter"}
The `| toJson` filter converts any variable to a JSON string — the reverse of `fromJson()`.
```yaml
{{ [1, 2, 3] | toJson }}
```
:::alert{type="info"}
To read a file's content as a JSON object, use `read()` combined with `fromJson()`, not `| toJson`.
:::
---
# System Flows in Kestra: Automate Maintenance
URL: https://kestra.io/docs/concepts/system-flows
> Automate platform maintenance with System Flows in Kestra. Schedule cleanup, monitoring, and admin tasks that run on a fixed cadence automatically.
System Flows are flows in the `system` namespace that automate recurring platform maintenance — running on a fixed cadence and kept separate from your user-facing executions.
Common uses include:
1. Sending [alert notifications](/blueprints/failure-alert-slack)
2. Creating automated support tickets when critical workflows fail
3. [Purging logs](/blueprints/purge) and removing old executions or internal storage files to save space
4. Syncing code from Git or pushing code to Git
5. Automatically [releasing flows](/blueprints/copy-flows-to-new-tenant) from development to QA and staging environments
By default, System Flows are only visible within the `system` namespace and to users with appropriate access. To use a different namespace name, set the following in your [Plugins and Execution configuration](../../configuration/04.plugins-and-execution/index.md):
```yaml
kestra:
systemFlows:
namespace: system
```
System Flows live in the `system` namespace, pinned at the top of **Namespaces**.

The **Blueprints** tab inside the system namespace provides ready-to-use templates you can customize for your organization.

:::alert{type="info"}
System Flows are not restricted to System Blueprints — any valid Kestra flow added to the `system` namespace becomes a System Flow.
:::
System flow executions appear across the Dashboard, Flows, and Executions pages. Use the **Scope** filter to view user and system executions separately or together.

The `system` namespace is open by default. Using namespace-level RBAC in the Enterprise Edition, you can restrict it to admins while assigning `company.*` namespaces to your general user base.
---
# System Labels in Kestra: Reserved Admin Metadata
URL: https://kestra.io/docs/concepts/system-labels
> Use system and hidden labels in Kestra for admin metadata. Understand how internal labels differ from user labels and how they affect filtering.
System Labels and Hidden Labels are reserved metadata labels used to manage and monitor Kestra. They are hidden in the UI by default. To find executions tagged with a specific hidden label, use the **Labels** filter with the key and value explicitly — for example, `system.correlationId: o8014I6GBOxqcQg8guKRP`.

After applying the filter, the executions table shows only the matching execution.
## Hidden labels
Hidden Labels are labels excluded from the UI by default. You can configure which prefixes should be hidden via the `kestra.hidden-labels.prefixes` configuration. For example, to hide labels starting with `admin.`, `internal.`, and `system.`, you can use the following configuration in your `application.yaml`:
```yaml
kestra:
hidden-labels:
prefixes:
- system.
- internal.
- admin.
```
By default, System Labels (prefixed with `system.`) are hidden. To display them, remove `system.` from the list of hidden prefixes.
## System labels
System Labels are labels prefixed with `system.` that serve specific purposes. The labels below are set automatically by Kestra. For a step-by-step guide on using `system.correlationId` as an idempotency key, see [Idempotency with correlation IDs](../../15.how-to-guides/idempotency/index.md).
### `system.correlationId`
- Automatically set for every execution and propagated to downstream executions created by `Subflow` or `Loop` tasks
- Represents the ID of the first execution in a chain of executions, enabling tracking of execution lineage
- Can also be set to a stable business key and used as an idempotency key for flows that must not process the same event twice
- Use this label to filter all executions originating from a specific parent execution or business event.
For example, if a parent flow triggers multiple subflows, filtering by the parent's `system.correlationId` displays all related executions.
:::alert{type="info"}
The Execution API supports setting this label at execution creation but not modification.
:::
### `system.username`
- Automatically set for every execution and contains the username of the user who triggered the execution
- Useful for auditing and identifying who initiated specific executions
### `system.readOnly`
- Used to mark a flow as read-only, disabling the flow editor in the UI
- Helps prevent modifications to critical workflows, such as production flows managed through CI/CD pipelines
**Example:**
```yaml
id: read_only_flow
namespace: company.team
labels:
system.readOnly: true
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: Hello from a read-only flow!
```

:::alert{type="info"}
In the Enterprise Edition, updating a read-only flow server-side is restricted to service accounts or API keys.
:::
### `system.from`
- Automatically set on every execution to indicate how it was triggered
- Common values include `ui` (triggered from the Kestra UI) and `mcp` (triggered by a Kestra MCP server)
- Use this label to filter executions by their trigger source
### `system.mcpServerId`
- Automatically set on every execution created by a Kestra MCP server
- Value is the `id` of the MCP server that invoked the tool
- Use this label together with `system.from: mcp` to identify which server triggered a specific execution
### `system.mcpSessionId`
- Automatically set on every execution created by a Kestra MCP server
- Value is the session ID of the MCP client connection that triggered the execution
- Use this label to correlate multiple executions that originated from the same agent session
---
# Configure Kestra: Settings, Environments & Defaults
URL: https://kestra.io/docs/configuration
> Learn where Kestra configuration lives and how to edit it. Covers runtime, storage, observability, security, plugins, and enterprise-only settings.
import ChildCard from "~/components/docs/ChildCard.astro"
Use this page when you need to know where Kestra configuration is defined, how it is overridden, and which part of the config tree controls a given capability.
## Where Kestra configuration lives
Kestra configuration is YAML-based, but the place you edit it depends on how you run Kestra.
| Deployment style | Where you usually edit configuration |
| --- | --- |
| Local Docker or Docker Compose | The `KESTRA_CONFIGURATION` environment variable or your `docker-compose.yml` |
| Kubernetes | Your Helm values or rendered manifests |
| VM or standalone server | A YAML config file plus environment variables such as `JAVA_OPTS` or `KESTRA_*` |
| Managed or shared environments | A checked-in deployment config, then environment-specific overrides through environment variables or platform secrets |
Environment variables override file-based configuration, so many teams keep a shared base config in YAML and inject deployment-specific values at runtime.
## Start here
- Use [Configuration Basics](./01.configuration-basics/index.md) to understand config sources, overrides, environment variable conversion, and the minimal setup required to boot Kestra.
- Use [Runtime and Storage](./02.runtime-and-storage/index.md) for repository, queue, datasource, storage, server, and JVM settings.
- Use [Observability and Networking](./03.observability-and-networking/index.md) for telemetry, logs, metrics, Micronaut, endpoints, and webserver behavior.
- Use [Plugins and Execution](./04.plugins-and-execution/index.md) for plugin installation, plugin defaults, retries, tasks, system flows, templates, and file access.
- Use [Security and Secrets](./05.security-and-secrets/index.md) for encryption, secret backends, auth-related security settings, and server hardening.
- Use [Enterprise and Advanced Features](./06.enterprise-and-advanced/index.md) for EE license settings, Elasticsearch, Kafka, indexer, AI Copilot, and air-gapped deployments.
## Common tasks
| If you need to... | Start here |
| --- | --- |
| Find the file or deployment surface where config is edited | [Configuration Basics](./01.configuration-basics/index.md) |
| Configure queue, repository, datasource, storage, or server settings | [Runtime and Storage](./02.runtime-and-storage/index.md) |
| Tune logs, metrics, telemetry, endpoints, CORS, SSL, or UI web settings | [Observability and Networking](./03.observability-and-networking/index.md) |
| Configure plugin defaults, task temp storage, retries, or local flow sync | [Plugins and Execution](./04.plugins-and-execution/index.md) |
| Set up encryption, secret backends, RBAC-adjacent security, or server auth | [Security and Secrets](./05.security-and-secrets/index.md) |
| Configure Enterprise-only platform services such as Kafka, Elasticsearch, AI Copilot, or air-gapped operation | [Enterprise and Advanced Features](./06.enterprise-and-advanced/index.md) |
## Browse configuration docs
---
# Configuration Basics in Kestra: YAML & Env Overrides
URL: https://kestra.io/docs/configuration/configuration-basics
> Learn where Kestra configuration is defined, how YAML and environment variables interact, and what minimal settings are needed to start a Kestra instance.
Use this page first if you are not sure where Kestra configuration is actually edited in your environment.
## Configuration sources
Kestra reads configuration from YAML. In practice, teams usually provide it in one of these ways:
- a YAML file mounted or bundled with the Kestra process
- the `KESTRA_CONFIGURATION` environment variable
- inline YAML inside Docker Compose
- Helm values or Kubernetes manifests
Environment variables override file-based configuration, so many teams keep a shared YAML base config in version control and inject deployment-specific values at runtime.
## Minimal boot configuration
Most deployments need to decide at least these three things:
1. repository type
2. queue type
3. internal storage type
Example:
```yaml
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driver-class-name: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
repository:
type: postgres
queue:
type: postgres
storage:
type: local
url: "http://localhost:8080/"
```
These three choices drive the rest of the deployment:
- `kestra.repository.type` controls the persistence backend for core metadata.
- `kestra.queue.type` must be compatible with the repository type.
- `kestra.storage.type` controls where Kestra stores internal files and task artifacts.
## Environment variable conversion
Convert YAML keys to environment variables like this:
- replace dots (`.`) with underscores
- replace hyphens (`-`) with underscores
- convert camelCase boundaries to underscores
- uppercase everything
- prefix Kestra-specific keys with `KESTRA_`
Examples:
| Configuration value | Resulting properties |
| --- | --- |
| `MYAPP_MYSTUFF` | `myapp.mystuff`, `myapp-mystuff` |
| `MY_APP_MY_STUFF` | `my.app.my.stuff`, `my.app.my-stuff`, `my-app.my.stuff`, `my-app.my-stuff`, and similar variants |
File-based configuration:
```yaml
datasources:
postgres:
username: kestra
```
becomes:
```bash
DATASOURCES_POSTGRES_USERNAME=kestra
```
And:
```yaml
kestra:
storage:
s3:
accessKey: myKey
```
or:
```yaml
kestra:
storage:
s3:
access-key: myKey
```
becomes:
```bash
KESTRA_STORAGE_S3_ACCESS_KEY=myKey
```
Common patterns:
```bash
MICRONAUT_SERVER_PORT=8080
DATASOURCES_POSTGRES_USERNAME=kestra
KESTRA_STORAGE_TYPE=s3
KESTRA_URL=https://kestra.example.com
```
:::alert{type="warning"}
For storage backend properties such as `kestra.storage.s3.access-key` and `kestra.storage.gcs.project-id`, prefer either the YAML form or the double-underscore env-var form. A single underscore between the last two segments can be read as a path separator by Micronaut and produce a nested object that the storage plugin's Jackson mapper rejects with `UnrecognizedPropertyException`. The two forms below are equivalent and both avoid that issue:
```bash
# double underscore preserves the literal kebab/snake separator
KESTRA_STORAGE_S3_ACCESS__KEY=
KESTRA_STORAGE_S3_SECRET__KEY=
```
```yaml
# or keep credentials in YAML and reference an env var
kestra:
storage:
s3:
access-key: "${S3_ACCESS_KEY}"
secret-key: "${S3_SECRET_KEY}"
```
The same applies to `kestra.storage.gcs.project-id`, `kestra.storage.gcs.service-account`, and other multi-word storage properties.
:::
## SDK default authentication
SDK-based plugins resolve authentication in this order:
1. Namespace-level default service account
2. Tenant-level default service account
3. Global SDK defaults (`kestra.tasks.sdk.authentication`)
**In OSS,** when `kestra.server.basic-auth` is configured, Kestra automatically derives the global SDK credentials from it — no additional configuration is needed. SDK-based tasks using `DEFAULT` or `AUTO` authentication work without further setup.
If you need to use different credentials from those in `kestra.server.basic-auth`, or to authenticate with an API token, override the global default explicitly:
```yaml
kestra:
tasks:
sdk:
authentication:
username: my-user # overrides basic-auth username
password: my-password # overrides basic-auth password
# api-token: ${KESTRA_API_TOKEN} # use an API token instead
```
:::alert{type="warning"}
If only one of `username` or `password` is set — either explicitly or resolved from `basic-auth` — Kestra throws an error when a task first attempts to use the SDK. Both must be provided together or neither.
:::
If no credential is available at any level, SDK-based tasks using `DEFAULT` or `AUTO` authentication fail. This applies to OSS instances without basic auth configured, and to EE/Cloud instances without a namespace or tenant-level service account.
## What belongs on the other configuration pages
- Use [Runtime and Storage](../02.runtime-and-storage/index.md) for datasources, queue, repository, internal storage, JVM, environment metadata, and global variables.
- Use [Observability and Networking](../03.observability-and-networking/index.md) for logs, metrics, Micronaut, endpoints, access logs, SSL, and CORS.
- Use [Plugins and Execution](../04.plugins-and-execution/index.md) for plugin installation, plugin defaults, retries, local flow sync, templates, and execution behavior.
- Use [Security and Secrets](../05.security-and-secrets/index.md) for encryption, secret backends, auth hardening, and liveness settings.
- Use [Enterprise and Advanced](../06.enterprise-and-advanced/index.md) for EE license, Kafka, Elasticsearch, indexer, AI Copilot, and air-gapped deployments.
## Next steps
- Need repository, datasource, storage, JVM, or server settings: [Runtime and Storage](../02.runtime-and-storage/index.md)
- Need logs, metrics, or SSL settings: [Observability and Networking](../03.observability-and-networking/index.md)
- Need secret backends or advanced EE infrastructure: [Security and Secrets](../05.security-and-secrets/index.md) and [Enterprise and Advanced](../06.enterprise-and-advanced/index.md)
---
# Enterprise & Advanced Configuration in Kestra
URL: https://kestra.io/docs/configuration/enterprise-and-advanced
> Configure Enterprise-only Kestra settings. Manage licenses, Elasticsearch, Kafka, indexer behavior, UI custom links, AI Copilot, and air-gapped deployments.
Use this page for configuration areas that are either Enterprise-specific or advanced platform concerns.
## Enterprise platform settings
This page groups together settings that are important but not part of a normal OSS-style runtime setup. If the instance is not using EE features, you can ignore most of this page.
This area includes:
- Enterprise license configuration
- Enterprise Java security
- gRPC TLS/mTLS for worker ↔ controller communication
- UI sidebar customization
- historical multi-tenancy and default tenant settings
- custom links in the UI
EE license configuration:
```yaml
kestra:
ee:
license:
id:
fingerprint:
key: |
```
Kestra validates the license on startup. The `fingerprint` is also required for versioned plugins.
EE Java security lets you restrict filesystem access and thread creation. Three controls are available:
- `forbidden-paths` — disallows read/write on listed filesystem paths
- `authorized-class-prefix` — limits which classes are allowed to create threads
- `forbidden-class-prefix` — blocks specific classes from creating threads
```yaml
kestra:
ee:
java-security:
enabled: true
forbidden-paths:
- /etc/
authorized-class-prefix:
- io.kestra.plugin.core
- io.kestra.plugin.gcp
```
Use `forbidden-class-prefix` when you want to block a specific plugin family from spawning threads rather than maintaining an allowlist:
```yaml
kestra:
ee:
java-security:
enabled: true
forbidden-class-prefix:
- io.kestra.plugin.scripts
```
Use EE Java security carefully. It is a platform hardening feature, so the goal is to narrow what plugin code is allowed to touch, not to tune routine runtime behavior.
UI customization examples:
```yaml
kestra:
ee:
right-sidebar:
custom-links:
internal-docs:
title: "Internal Docs"
url: "https://kestra.io/docs/"
```
```yaml
kestra:
ee:
left-sidebar:
disabled-menus:
- "Blueprints/Flow Blueprints"
```
The old multi-tenancy and default-tenant configuration is no longer supported.
## Worker topology bootstrap (kestra.ee.setup)
`kestra.ee.setup` lets you declare worker queues, worker groups, subscriptions, and registration tokens in configuration so the full topology is provisioned at startup without any runtime API calls.
```yaml
kestra:
ee:
setup:
enabled: true
worker-queues:
- id: gpu
tags: [gpu, linux]
allowed-tenants: [acme] # optional; empty = unrestricted
- id: etl
tags: [etl]
worker-groups:
- id: gpu-workers
name: GPU workers
registration-tokens:
- name: bootstrap
token-file: /var/run/secrets/kestra/gpu-workers-token
subscriptions:
- worker-queue-id: gpu
reserved-percent: 70
- worker-queue-id: etl
```
Only the webserver and standalone server roles apply this configuration at startup. Worker processes never apply it.
Each declared entity is created only if it does not already exist in the database. Existing entities are skipped as a whole — no subscriptions are changed and no tokens are added or revoked. The database remains the source of truth once an entity exists.
**Token secret handling.** Each registration token entry requires exactly one of:
| Field | Description |
|---|---|
| `token-file` | Path to a mounted secret file containing the token. Preferred in Kubernetes environments. |
| `token` | Environment variable placeholder resolved at startup, e.g. `"${MY_TOKEN}"`. |
Plaintext tokens in committed configuration files are not recommended.
**Configuration reference**
| Property | Required | Description |
|---|---|---|
| `kestra.ee.setup.enabled` | No | Set to `true` to activate declarative bootstrap. Defaults to `false`. |
| `worker-queues[].id` | Yes | RFC 1123 label. `default` and `system` are reserved and cannot be used. |
| `worker-queues[].tags` | Yes | Non-empty list of routing tags. Must be unique across queues. |
| `worker-queues[].allowed-tenants` | No | Tenant ids permitted to route through this queue. Empty = unrestricted. |
| `worker-groups[].id` | Yes | RFC 1123 label. Use `default` to seed the default group. |
| `worker-groups[].name` | No | Display name. Defaults to the id when omitted. |
| `worker-groups[].registration-tokens[].name` | No | Token display name. Defaults to `bootstrap`. |
| `worker-groups[].registration-tokens[].token-file` | One of | Path to a file containing a pre-generated registration token. Surrounding whitespace is trimmed. |
| `worker-groups[].registration-tokens[].token` | One of | Environment variable placeholder, e.g. `"${MY_TOKEN}"`. Surrounding whitespace is trimmed. |
| `worker-groups[].subscriptions[].worker-queue-id` | Yes | Id of a queue declared under `worker-queues` or already in the database. |
| `worker-groups[].subscriptions[].reserved-percent` | No | Per-worker capacity floor, 1–100. Sum across subscriptions must not exceed 100. |
| `worker-groups[].subscriptions[].mode` | No | `STRICT` or `ELASTIC`. See [Capacity reservation](../../07.enterprise/04.scalability/worker-group/index.md#capacity-reservation). Defaults to `STRICT`. |
See [Declarative configuration](../../07.enterprise/04.scalability/worker-group/index.md#declarative-configuration) for full semantics, validation behavior, and a Kubernetes deployment example.
## gRPC TLS/mTLS (EE only)
Use this section when running Kestra in a distributed topology where the Worker Controller and Workers communicate over gRPC and you need to encrypt that channel. By default, gRPC traffic is plaintext. Enabling TLS here encrypts the controller ↔ worker channel; enabling mTLS additionally requires workers to present a certificate the controller trusts.
This feature is active on any component with server type `CONTROLLER`, `WORKER`, or `STANDALONE`.
### One-way TLS
The controller presents a certificate; workers verify it against a truststore. Configure the controller (server side) with a keystore and the workers (client side) with a matching truststore:
**Controller:**
```yaml
kestra:
grpc:
tls:
enabled: true
key-store:
path: /etc/kestra/tls/controller-keystore.p12
type: PKCS12
password: ""
```
**Worker:**
```yaml
kestra:
grpc:
tls:
enabled: true
trust-store:
path: /etc/kestra/tls/ca-truststore.p12
type: PKCS12
password: ""
```
If no truststore is provided on the worker side, the JVM default trust store is used. This is appropriate when the controller certificate is signed by a well-known CA.
### Mutual TLS (mTLS)
Set `client-auth: REQUIRE` on the controller to enforce that workers present a certificate. Both sides need a keystore and a truststore:
**Controller:**
```yaml
kestra:
grpc:
tls:
enabled: true
client-auth: REQUIRE
key-store:
path: /etc/kestra/tls/controller-keystore.p12
type: PKCS12
password: ""
trust-store:
path: /etc/kestra/tls/ca-truststore.p12
type: PKCS12
password: ""
```
**Worker:**
```yaml
kestra:
grpc:
tls:
enabled: true
key-store:
path: /etc/kestra/tls/worker-keystore.p12
type: PKCS12
password: ""
trust-store:
path: /etc/kestra/tls/ca-truststore.p12
type: PKCS12
password: ""
```
`client-auth` also accepts `OPTIONAL`, which requests a client certificate but does not require one.
### Authority override for static discovery
When using static discovery, the gRPC channel authority is the synthetic value `controllers` rather than a real hostname. If the controller certificate's Subject Alternative Names (SANs) do not include `controllers`, TLS verification will fail. Set `authority-override` on the worker to a hostname that is present in the certificate's SANs:
```yaml
kestra:
grpc:
tls:
enabled: true
authority-override: kestra-controller
trust-store:
path: /etc/kestra/tls/ca-truststore.p12
type: PKCS12
password: ""
```
This is not needed with DNS-based discovery, where the authority is derived from the actual hostname.
### JKS keystores
PKCS12 is the recommended format. For JKS keystores, set `type: JKS`. JKS also supports a separate key password (used when the private key entry password differs from the store password):
```yaml
kestra:
grpc:
tls:
enabled: true
key-store:
path: /etc/kestra/tls/keystore.jks
type: JKS
password: ""
key-password: ""
```
### Development: skip certificate verification
:::alert{type="warning"}
`insecure-trust-all-certificates: true` disables CA verification entirely. Use only in local development or CI environments where certificates are self-signed and not managed. Never enable this in production.
:::
```yaml
kestra:
grpc:
tls:
enabled: true
insecure-trust-all-certificates: true
```
### Configuration reference
| Property | Default | Description |
| --- | --- | --- |
| `kestra.grpc.tls.enabled` | `false` | Enable TLS for gRPC communication |
| `kestra.grpc.tls.key-store.path` | — | Path to keystore file |
| `kestra.grpc.tls.key-store.type` | `PKCS12` | Keystore format (`PKCS12` or `JKS`) |
| `kestra.grpc.tls.key-store.password` | — | Keystore password |
| `kestra.grpc.tls.key-store.key-password` | — | Private key entry password (JKS only) |
| `kestra.grpc.tls.trust-store.path` | — | Path to truststore file |
| `kestra.grpc.tls.trust-store.type` | `PKCS12` | Truststore format |
| `kestra.grpc.tls.trust-store.password` | — | Truststore password |
| `kestra.grpc.tls.client-auth` | `NONE` | Client auth mode: `NONE`, `OPTIONAL`, or `REQUIRE` |
| `kestra.grpc.tls.insecure-trust-all-certificates` | `false` | Skip CA verification (development only) |
| `kestra.grpc.tls.authority-override` | — | Override TLS authority for static discovery |
## Elasticsearch, Kafka, and indexing
This section is really about one architectural choice: running Kestra on the Kafka plus Elasticsearch stack instead of the simpler JDBC-backed setup. If you are on PostgreSQL or MySQL only, much of this page will not apply.
These settings cover the advanced repository and queue stack used in Enterprise deployments:
- Elasticsearch repository settings
- Kafka client and topic settings
- Kafka message protection
- indexer behavior
Use this section when you are running the Kafka plus Elasticsearch architecture instead of a JDBC-only deployment.
Minimal Elasticsearch repository configuration:
```yaml
kestra:
elasticsearch:
client:
http-hosts: "http://localhost:9200"
repository:
type: elasticsearch
```
Start by proving the minimal connection first. After that, add auth, SSL handling, index prefixes, or rotation only when the deployment model requires them.
With authentication:
```yaml
kestra:
elasticsearch:
client:
http-hosts:
- "http://node-1:9200"
- "http://node-2:9200"
basic-auth:
username: ""
password: ""
repository:
type: elasticsearch
```
Related advanced Elasticsearch settings include:
- `trust-all-ssl` for self-signed development clusters
- custom index prefixes
- daily, weekly, monthly, or yearly index rotation
Minimal Kafka queue configuration:
```yaml
kestra:
kafka:
client:
properties:
bootstrap.servers: "localhost:9092"
queue:
type: kafka
```
Kafka tuning is usually about cluster shape rather than syntax. Partition count limits how much component-level concurrency you can achieve, while replication settings should match your broker topology and HA expectations.
This page also covers:
- SSL-secured Kafka clients
- default topic partition and replication settings
- consumer, producer, and stream defaults
- custom topic names and topic properties
- consumer and topic prefixes for shared clusters
- Kafka Streams local state directory
- message protection for oversized Kafka messages
Representative advanced Kafka settings:
```yaml
kestra:
kafka:
client:
properties:
bootstrap.servers: "localhost:9092"
security.protocol: SSL
defaults:
topic:
partitions: 3
replication-factor: 3
topics:
executions:
properties:
retention.ms: 604800000
```
Use client properties for transport and auth, `defaults` for cluster-wide topic behavior, and `topics.*.properties` only when one topic needs behavior that differs from the rest.
Full SSL client configuration with keystores:
```yaml
kestra:
kafka:
client:
properties:
bootstrap.servers: "host:port"
security.protocol: "SSL"
ssl.endpoint.identification.algorithm: ""
ssl.key.password: ""
ssl.keystore.location: "/etc/ssl/private/keystore.p12"
ssl.keystore.password: ""
ssl.keystore.type: "PKCS12"
ssl.truststore.location: "/etc/ssl/private/truststore.jks"
ssl.truststore.password: ""
queue:
type: kafka
```
Consumer, producer, and stream defaults:
```yaml
kestra:
kafka:
defaults:
consumer:
properties:
isolation.level: "read_committed"
auto.offset.reset: "earliest"
enable.auto.commit: "false"
producer:
properties:
acks: "all"
compression.type: "lz4"
max.request.size: "10485760"
stream:
properties:
processing.guarantee: "exactly_once"
replication.factor: "${kestra.kafka.defaults.topic.replication-factor}"
acks: "all"
compression.type: "lz4"
max.request.size: "10485760"
state.dir: "/tmp/kafka-streams"
```
Client loggers for debugging message flow:
```yaml
kestra:
kafka:
client:
loggers:
- level: INFO
type: PRODUCER
topic-regexp: "kestra_(executions|workertaskresult)"
key-regexp: .*parallel.*
value-regexp: .*parallel.*
```
:::alert{type="warning"}
Client loggers have a heavy performance impact. Use them only for short-lived debugging sessions.
:::
Shared-cluster deployments often also need prefixes or dedicated topic names to avoid collisions with other tenants or environments.
To reject oversized Kafka messages early:
```yaml
kestra:
kafka:
message-protection:
enabled: true
limit: 1048576
```
Indexer settings control batch indexing from Kafka into Elasticsearch:
```yaml
kestra:
indexer:
batch-size: 500
batch-duration: PT1S
```
If indexing falls behind, tune indexer batch settings before changing flow definitions. Those settings control how aggressively Kafka-backed events are flushed into Elasticsearch.
## MCP server cache
Each webserver node caches MCP server configuration in memory and hot-reloads it when a server is created, updated, or deleted. Two properties control this cache:
| Property | Default | Description |
|---|---|---|
| `kestra.mcp.server-cache-config.maximum-size` | `500` | Maximum number of MCP server entries held in the cache. |
| `kestra.mcp.server-cache-config.expire-after-access` | `PT5M` | Duration after which a cache entry expires if not accessed. |
```yaml
kestra:
mcp:
server-cache-config:
maximum-size: 200
expire-after-access: PT10M
```
Tune these only if you have a large number of MCP servers or tight memory constraints. The defaults are sufficient for most deployments.
## AI and isolated environments
These are the most optional settings on the page. They matter only if you are enabling Copilot integrations or operating Kestra in restricted network environments.
This page also includes:
- AI Copilot provider configuration
- air-gapped instance settings
### AI Copilot
AI Copilot configuration lives under `kestra.ai` and controls which LLM providers are active, how each provider is authenticated and tuned, and how the agent runtime behaves.
#### Enabling and disabling
`kestra.ai.enabled` (default: `true`) controls whether AI Copilot is active. Set it to `false` to disable the feature entirely, including the built-in fallback to `api.kestra.io`.
#### Providers
Enterprise Edition lets you configure multiple providers in a single deployment. Each entry in `kestra.ai.providers` is an independent provider the UI can offer to users.
```yaml
kestra:
ai:
enabled: true
providers:
- id: internal-gemini
display-name: Gemini (internal)
type: gemini
configuration:
api-key: YOUR_GEMINI_API_KEY
model-name: gemini-2.5-flash
- id: openai-gpt
display-name: OpenAI GPT
type: openai
is-default: true
configuration:
api-key: YOUR_OPENAI_API_KEY
model-name: gpt-4o
```
**Provider wrapper fields**
| Field | Required | Description |
|---|---|---|
| `id` | ✅ | Unique identifier for this provider entry. |
| `display-name` | ✅ | Label shown to users in the Copilot UI. |
| `type` | ✅ | Provider type. One of: `openai`, `azure-openai`, `gemini`, `googlevertexai`, `anthropic`, `bedrock`, `deepseek`, `mistralai`, `ollama`, `open-router`. |
| `is-default` | ❌ | When `true`, this provider is selected automatically when no explicit choice is made. |
| `configuration` | ❌ | Provider-specific settings. See the property reference and provider sections below. |
| `system-prompt` | ❌ | (EE only) Override the built-in system prompt per Copilot mode. A non-blank value for a mode fully replaces the built-in prompt for that mode. |
| `system-prompt.ask` | ❌ | Custom system prompt for Ask mode. |
| `system-prompt.plan` | ❌ | Custom system prompt for Plan mode. |
| `system-prompt.edit` | ❌ | Custom system prompt for Edit mode. |
#### Configuration property reference
These properties appear inside the `configuration:` block of a provider entry. Not every property is available on every provider — the per-provider sections below show which apply and which are required.
**Authentication**
| Property | Description |
|---|---|
| `api-key` | API key for the provider. Most providers require this; Anthropic and Google Vertex AI do not (see their sections). |
| `access-key-id` | AWS access key ID. Amazon Bedrock only. |
| `secret-access-key` | AWS secret access key. Amazon Bedrock only. |
| `client-pem` | PEM-encoded client certificate used for mutual TLS (mTLS) when the provider endpoint requires client authentication. |
| `ca-pem` | PEM-encoded CA certificate to add additional TLS trust beyond the system trust store. Not required for standard provider endpoints. |
**Model selection**
| Property | Description |
|---|---|
| `model-name` | The model identifier to use. The accepted values are provider-specific (e.g. `gpt-4o`, `gemini-2.5-flash`, `claude-opus-4-5`). Each provider section lists its default. |
**Generation parameters**
| Property | Description |
|---|---|
| `temperature` | Controls randomness in sampling. Lower values (e.g. `0.2`) produce more deterministic output; higher values (e.g. `1.0`) produce more varied responses. Most providers default to `0.7`; OpenAI and OpenRouter default to `1`. |
| `top-p` | Nucleus sampling: only tokens whose cumulative probability reaches `top-p` are considered. An alternative to `temperature` — set one or the other, not both. |
| `top-k` | Limits sampling to the top K most probable tokens at each step. Supported by Gemini, Google Vertex AI, Anthropic, Amazon Bedrock, and Ollama. |
| `max-output-tokens` | Maximum number of tokens the model may produce in a single response. Defaults to `8000` on providers that support it. Mistral AI does not expose this setting. |
**Extended reasoning**
| Property | Description |
|---|---|
| `thinking-enabled` | When `true`, enables the provider's extended reasoning or thinking mode. Supported by OpenAI, Azure OpenAI, Gemini, Anthropic, and Amazon Bedrock. Anthropic requires `temperature: 1` and disables `top-p` and `top-k` when thinking is on — these constraints are applied automatically. |
| `thinking-effort` | Provider-neutral reasoning effort: `LOW`, `MEDIUM`, or `HIGH`. Supported by OpenAI, Azure OpenAI, and Gemini. Each provider maps this to its own vocabulary (e.g. OpenAI `reasoning_effort`, Gemini `thinkingLevel`). |
| `thinking-budget-tokens` | Token budget for the reasoning process. Supported by Gemini, Anthropic, and Amazon Bedrock. Anthropic requires a minimum of `1024`; when `thinking-enabled` is `true` and no budget is set, `1024` is used automatically. |
**Network and connectivity**
| Property | Description |
|---|---|
| `base-url` | Override the default API endpoint. Use this for self-hosted deployments, proxies, or OpenAI-compatible local servers. Available on OpenAI, Gemini, DeepSeek, Mistral AI, Ollama, and OpenRouter. For Ollama, `base-url` is required (there is no cloud endpoint). |
| `custom-headers` | A flat map of extra HTTP headers sent with every request to the provider. Useful for passing organization IDs, routing headers, or authentication tokens that the provider requires alongside the API key. |
| `timeout` | Maximum duration for a single HTTP request to the provider (e.g. `PT30S`, `PT2M`). Does not apply to Google Vertex AI. |
**Logging**
| Property | Description |
|---|---|
| `log-requests` | When `true`, logs the full request body sent to the provider. Useful for debugging prompt construction. Avoid in production — request bodies may contain sensitive data. |
| `log-responses` | When `true`, logs the full response body received from the provider. Same caveats as `log-requests`. |
| `log-requests-and-responses` | Azure OpenAI equivalent of `log-requests` + `log-responses` combined in a single toggle. |
#### Provider types
Each provider section lists only its required fields and properties unique to that provider. All other properties from the reference above are available unless noted.
##### OpenAI (`type: openai`)
| Property | Required | Default |
|---|---|---|
| `api-key` | ✅ | — |
| `model-name` | ❌ | `gpt-5-nano` |
| `temperature` | ❌ | `1` |
| `max-output-tokens` | ❌ | `8000` |
Supports `thinking-enabled` and `thinking-effort`. Supports `base-url` for OpenAI-compatible self-hosted endpoints.
Does not support `top-k`.
##### Azure OpenAI (`type: azure-openai`)
Supports two authentication methods: API key or Azure Active Directory (AAD).
| Property | Required | Default |
|---|---|---|
| `endpoint` | ✅ | — |
| `model-name` | ✅ | — |
| `api-key` | ❌ (use this or AAD) | — |
| `tenant-id` | ❌ (AAD auth) | — |
| `client-id` | ❌ (AAD auth) | — |
| `client-secret` | ❌ (AAD auth) | — |
| `service-version` | ❌ | — |
| `temperature` | ❌ | `1` |
| `max-output-tokens` | ❌ | `8000` |
Supports `thinking-enabled` and `thinking-effort`.
Uses `log-requests-and-responses` instead of separate `log-requests` / `log-responses` toggles. Does not support `top-k`, `base-url`, `client-pem`, or `ca-pem`.
##### Gemini (`type: gemini`)
| Property | Required | Default |
|---|---|---|
| `api-key` | ✅ | — |
| `model-name` | ❌ | `gemini-2.5-flash` |
| `temperature` | ❌ | `0.7` |
| `max-output-tokens` | ❌ | `8000` |
Supports `thinking-enabled`, `thinking-effort`, and `thinking-budget-tokens`. Supports `base-url` and `top-k`.
##### Google Vertex AI (`type: googlevertexai`)
Authenticates via [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). No `api-key` field — ensure the Kestra runtime environment has ADC configured (e.g. a service account key via `GOOGLE_APPLICATION_CREDENTIALS`, Workload Identity, or `gcloud auth application-default login`).
| Property | Required | Default |
|---|---|---|
| `project` | ✅ | — |
| `location` | ✅ | — |
| `model-name` | ✅ | — |
| `temperature` | ❌ | `0.7` |
Supports `top-k`. Does not support `thinking-enabled`, `thinking-effort`, `thinking-budget-tokens`, `base-url`, `client-pem`, `ca-pem`, `max-output-tokens`, or `timeout`.
##### Anthropic (`type: anthropic`)
No `api-key` configuration field. Set the `ANTHROPIC_API_KEY` environment variable on the Kestra server instead.
| Property | Required | Default |
|---|---|---|
| `model-name` | ✅ | — |
| `temperature` | ❌ | `0.7` |
| `max-output-tokens` | ❌ | `8000` |
Supports `thinking-enabled` and `thinking-budget-tokens` (minimum `1024`; defaults to `1024` when thinking is enabled without an explicit budget). When `thinking-enabled` is `true`, Anthropic requires `temperature: 1` and ignores `top-p` and `top-k` — these constraints are applied automatically regardless of what you configure. Supports `top-k`.
Does not support `thinking-effort`.
##### Amazon Bedrock (`type: bedrock`)
| Property | Required | Default |
|---|---|---|
| `access-key-id` | ✅ | — |
| `secret-access-key` | ✅ | — |
| `model-name` | ✅ | — |
| `temperature` | ❌ | `0.7` |
| `max-output-tokens` | ❌ | `8000` |
Supports `thinking-enabled` and `thinking-budget-tokens`. Supports `top-k`.
Does not support `thinking-effort`, `base-url`, `client-pem`, or `ca-pem`.
##### DeepSeek (`type: deepseek`)
| Property | Required | Default |
|---|---|---|
| `api-key` | ✅ | — |
| `model-name` | ❌ | `deepseek-chat` |
| `temperature` | ❌ | `0.7` |
| `max-output-tokens` | ❌ | `8000` |
Supports `base-url` for self-hosted or compatible endpoints. Does not support `top-k`, `thinking-enabled`, or `thinking-effort`.
##### Mistral AI (`type: mistralai`)
| Property | Required | Default |
|---|---|---|
| `api-key` | ✅ | — |
| `model-name` | ✅ | — |
| `temperature` | ❌ | `0.7` |
Supports `base-url`. Does not support `max-output-tokens`, `top-k`, `thinking-enabled`, or `thinking-effort`.
##### Ollama (`type: ollama`)
Ollama runs locally — there is no cloud API key. `base-url` points to your Ollama server and is required.
| Property | Required | Default |
|---|---|---|
| `base-url` | ✅ | — |
| `model-name` | ✅ | — |
| `temperature` | ❌ | `0.7` |
| `max-output-tokens` | ❌ | `8000` |
Supports `top-k`. Does not support `api-key`, `thinking-enabled`, or `thinking-effort`.
##### OpenRouter (`type: open-router`)
| Property | Required | Default |
|---|---|---|
| `api-key` | ✅ | — |
| `model-name` | ❌ | `gpt-5-nano` |
| `temperature` | ❌ | `1` |
| `max-output-tokens` | ❌ | `8000` |
Supports `base-url`. Does not support `top-k`, `thinking-enabled`, or `thinking-effort`.
#### Agent runtime settings
`kestra.ai.agent` controls the Copilot agent runtime. The defaults suit most deployments — tune only when hitting provider rate limits, memory pressure, or needing to adjust conversation scope.
| Property | Default | Description |
|---|---|---|
| `model-call-timeout` | `PT5M` | Maximum duration of a single streaming model call. If a provider call hangs beyond this threshold, the turn is failed rather than leaving a thread pinned indefinitely. |
| `docs-mcp-url` | `https://api.kestra.io/v1/mcp` | Kestra docs MCP endpoint used for context grounding in Ask mode. Override this in air-gapped deployments that run a local docs MCP server. |
| `max-sequential-tools-invocations` | `25` | Maximum number of sequential tool-calling round-trips within a single turn. Bounds runaway reasoning loops — each round-trip is a paid model call. |
| `max-turns-per-thread` | `50` | Maximum number of user turns in a single conversation thread before new turns are refused. |
| `max-concurrent-turns` | `32` | Per-node ceiling on simultaneously running agent turns. New turns receive a 429 response when the ceiling is reached rather than queuing. Bounds concurrent provider load — agent turns run on virtual threads so the thread count itself is not a concern. |
| `max-context-turns` | `10` | How many of the most recent turns are replayed into the model context per turn. Older turns remain stored for history but are windowed out of the prompt. Windowing operates on whole turns so tool-call and result pairs are never split. |
| `in-memory-conversation-ttl` | `PT1H` | In-memory store only: how long a conversation is retained after its last activity before eviction. Ignored when a durable backend is configured. |
| `max-in-memory-conversations` | `50` | In-memory store only: hard cap on retained conversations. The least-recently-active conversation is evicted when the cap is exceeded. Ignored when a durable backend is configured. |
```yaml
kestra:
ai:
agent:
model-call-timeout: PT5M
max-sequential-tools-invocations: 25
max-concurrent-turns: 32
max-context-turns: 10
```
### Air-gapped mode
Use air-gapped mode when the UI and blueprint experience must avoid external dependencies:
```yaml
kestra:
ee:
airgapped: true
```
When enabled, the UI hides or adapts features that normally depend on external services, such as hosted fonts, external blueprint sources, or embedded internet content.
### Mail service
Invitation and password-reset emails rely on the EE mail service:
```yaml
kestra:
ee:
mail-service:
host: host.smtp.io
port: 587
username: user
password: password
from: configurable@mail.com
from-name: Kestra
auth: true
starttls-enable: true
```
Use this page when an instance needs non-default enterprise infrastructure, custom UI platform behavior, or advanced deployment constraints rather than routine runtime configuration.
## When to use this page
- Need secure runtime or secret backend settings: [Security and Secrets](../05.security-and-secrets/index.md)
- Need queue, repository, storage, or JVM setup: [Runtime and Storage](../02.runtime-and-storage/index.md)
---
# Observability & Networking Configuration in Kestra
URL: https://kestra.io/docs/configuration/observability-and-networking
> Configure telemetry, logs, metrics, Micronaut settings, endpoints, SSL, CORS, and webserver behavior in Kestra.
Use this page for operational visibility and network-facing configuration.
## Observability
Use this section when you need to understand what Kestra emits about itself, not when you are changing task behavior. The settings here are mostly for platform operators and anyone integrating Kestra with monitoring or logging systems.
Configuration areas in this group include:
- anonymous telemetry
- logger settings
- access logs and log formatting
- metrics and label-based metrics
- Micronaut HTTP settings
These settings are useful when you need to tune visibility, log volume, request handling, or integration with monitoring platforms.
Anonymous usage reporting is enabled by default. Disable or tune it with:
```yaml
kestra:
anonymous-usage-report:
enabled: false
```
```yaml
kestra:
anonymous-usage-report:
initial-delay: 5m
fixed-delay: 1h
```
UI usage reporting is configured separately:
```yaml
kestra:
ui-anonymous-usage-report:
enabled: false
```
## Logs and access logging
There are two different concerns here: application logs and HTTP access logs. Reach for `logger.levels` when you want to change verbosity inside Kestra, and Micronaut access logging when you want request-by-request HTTP visibility.
Use `logger.levels` to adjust server log verbosity:
```yaml
logger:
levels:
io.kestra.core.runners: TRACE
org.apache.kafka: DEBUG
```
You can also suppress execution-scoped logs globally:
```yaml
logger:
levels:
execution: 'OFF'
task: 'OFF'
trigger: 'OFF'
```
Or scope suppression to a specific flow, task, or trigger by appending the flow ID and optionally the task or trigger ID:
```yaml
logger:
levels:
execution.hello-world: 'OFF'
task.hello-world: 'OFF'
trigger.hello-world: 'OFF'
task.hello-world.log: 'OFF'
trigger.hello-world.schedule: 'OFF'
```
Micronaut access logging is configured separately:
```yaml
micronaut:
server:
netty:
access-logger:
enabled: true
logger-name: io.kestra.webserver.access
log-format: "[Date: {}] [Duration: {} ms] [Method: {}] [Url: {}] [Status: {}] [Length: {}] [Ip: {}] [Port: {}]"
exclusions:
- /ui/.+
- /health
- /prometheus
```
Kestra uses [Logback](https://logback.qos.ch/) for logging. To use a custom `logback.xml`, pass it via `JAVA_OPTS`:
```shell
export JAVA_OPTS="-Dlogback.configurationFile=file:/path/to/logback.xml"
```
GCP structured logging:
```xml
```
ECS format:
```xml
```
## Metrics and telemetry exports
These settings are usually enabled with restraint. Metrics are broadly useful, but label-based metrics should stay limited to a small set of low-cardinality dimensions or they become expensive to store and query.
Set a metrics prefix:
```yaml
kestra:
metrics:
prefix: kestra
```
Add low-cardinality labels as metric tags:
```yaml
kestra:
metrics:
labels:
- country
- environment
```
This creates a tag named `label_` for each configured label. When an execution does not have a configured label key, the tag value is set to `__none__`, which keeps the set of tag keys stable and avoids metric series fragmentation.
For example, with `country` and `environment` configured, an execution that has `country=Germany` but no `environment` label produces:
```plaintext
kestra_executions_total{flow_id="my-flow",namespace_id="default",state="SUCCESS",label_country="Germany",label_environment="__none__"} 1
```
To collect metrics from other service instances and re-expose them on the webserver's monitoring endpoint, use `sharedServiceInstanceMetrics`. Each key is a service type (`EXECUTOR`, `INDEXER`, `SCHEDULER`, `WEBSERVER`, `WORKER`) and each value is a list of fully-qualified metric names:
```yaml
kestra:
metrics:
sharedServiceInstanceMetrics:
WORKER:
- kestra.worker.job.pending
- kestra.worker.job.thread
- kestra.worker.job.running
```
See [Service Instance Metrics](../../10.administrator-guide/service-instance-metrics/index.md) for details.
For traces, metrics, and logs exported through OpenTelemetry, use the dedicated [OpenTelemetry guide](../../10.administrator-guide/open-telemetry/index.md).
## Network and HTTP settings
This section matters when Kestra is exposed behind a load balancer, reverse proxy, ingress, or private network boundary. If requests are not arriving with the expected URL, protocol, size limit, or auth behavior, the fix is often here.
Micronaut-backed settings cover:
- server port
- SSL
- timeouts
- upload size
- base path
- host resolution
- CORS
- management endpoints
Common examples:
```yaml
micronaut:
server:
port: 8086
```
```yaml
micronaut:
server:
max-request-size: 10GB
multipart:
max-file-size: 10GB
disk: true
read-idle-timeout: 60m
write-idle-timeout: 60m
idle-timeout: 60m
netty:
max-chunk-size: 10MB
```
Reverse proxy support:
```yaml
micronaut:
server:
context-path: "kestra-prd"
host-resolution:
host-header: Host
protocol-header: X-Forwarded-Proto
```
Enable CORS:
```yaml
micronaut:
server:
cors:
enabled: true
```
Secure or move management endpoints:
```yaml
endpoints:
all:
basic-auth:
username: your-user
password: your-password
port: 8084
```
SSL example:
```yaml
micronaut:
security:
x509:
enabled: true
ssl:
enabled: true
server:
ssl:
client-authentication: need
key-store:
path: classpath:ssl/keystore.p12
password: ${KEYSTORE_PASSWORD}
type: PKCS12
trust-store:
path: classpath:ssl/truststore.jks
password: ${TRUSTSTORE_PASSWORD}
type: JKS
```
## UI and webserver settings
These settings are lighter-weight than the Micronaut server settings above. Use them when you are customizing the user-facing web experience rather than transport-level HTTP behavior.
The webserver-related configuration also includes:
- disabling the UI to run the webserver API-only
- Google Analytics ID
- additional HTML tags
- mail server settings
- security response headers
Examples:
```yaml
kestra:
webserver:
google-analytics-id: G-XXXXXXXXXX
```
```yaml
kestra:
webserver:
html-head:
- ""
```
Mail server settings are useful when you need platform emails for invitations and notifications.
### Security response headers
Kestra adds browser security response headers to every HTTP response, including error responses (401, 403) and static file responses.
Three headers are enabled by default:
| Header | Default value |
|---|---|
| `X-Frame-Options` | `SAMEORIGIN` |
| `X-Content-Type-Options` | `nosniff` |
| `Referrer-Policy` | `strict-origin-when-cross-origin` |
Two additional headers are disabled by default:
| Header | When emitted |
|---|---|
| `Content-Security-Policy` | When `content-security-policy` is set to a non-blank value |
| `Strict-Transport-Security` | When `strict-transport-security` is set and the request arrives at Kestra over HTTPS directly |
Each header is added only if not already present in the response.
Configuration reference:
| Property | Type | Default | Description |
|---|---|---|---|
| `kestra.webserver.security-headers.enabled` | boolean | `true` | Master toggle. When `false`, the filter is not created and no headers are added. |
| `kestra.webserver.security-headers.frame-options` | string | `SAMEORIGIN` | Value for `X-Frame-Options`. Set to `""` to disable this header only. |
| `kestra.webserver.security-headers.content-type-options` | string | `nosniff` | Value for `X-Content-Type-Options`. Set to `""` to disable this header only. |
| `kestra.webserver.security-headers.referrer-policy` | string | `strict-origin-when-cross-origin` | Value for `Referrer-Policy`. Set to `""` to disable this header only. |
| `kestra.webserver.security-headers.content-security-policy` | string | (absent) | Value for `Content-Security-Policy`. Not set unless configured. |
| `kestra.webserver.security-headers.content-security-policy-report-only` | boolean | `false` | When `true`, the CSP value is sent under `Content-Security-Policy-Report-Only` instead of `Content-Security-Policy`. Use this to test a policy before enforcing it. |
| `kestra.webserver.security-headers.strict-transport-security` | string | (absent) | Value for `Strict-Transport-Security`. Not set unless configured, and only emitted when Kestra is serving the request over HTTPS directly (see note below). |
Enable CSP in report-only mode — report-only sends the policy without blocking any resources, so you can test that your CSP does not break any Kestra UI functionality before switching to enforcement:
```yaml
kestra:
webserver:
security-headers:
content-security-policy: "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; font-src 'self' https://fonts.gstatic.com; img-src 'self' data:; connect-src 'self' wss:; frame-ancestors 'none'"
content-security-policy-report-only: true
```
Remove `content-security-policy-report-only` or set it to `false` to switch from reporting to enforcement.
Disable an individual header by setting its value to an empty string:
```yaml
kestra:
webserver:
security-headers:
frame-options: ""
```
Disable all security headers:
```yaml
kestra:
webserver:
security-headers:
enabled: false
```
:::alert{type="warning"}
**HSTS and reverse proxies**: `Strict-Transport-Security` is only emitted when Kestra itself receives the request over HTTPS (i.e., TLS is terminated on the Kestra server). In the common deployment pattern where a reverse proxy (nginx, Traefik, AWS ALB, Cloudflare) terminates TLS and forwards plain HTTP to Kestra, the request arrives at Kestra as plain HTTP and HSTS is never sent — even if `strict-transport-security` is configured. In this case, configure HSTS at the reverse proxy instead.
:::
### Disabling the UI (API-only mode)
The bundled web UI is enabled by default. Set `kestra.webserver.ui.enabled` to `false` to run the webserver as an API-only service — useful when Kestra is driven entirely through the REST API or fronted by your own application, and you don't want the UI exposed.
```yaml
kestra:
webserver:
ui:
enabled: false
```
When the UI is disabled:
- requests to `/ui/**` return `404`, and `/` no longer redirects to the UI;
- the REST API (`/api/v1/**`) and health endpoints keep working as usual.
The setting can also be provided through the `KESTRA_WEBSERVER_UI_ENABLED` environment variable.
## Typical use cases
Use this section when you need to:
- expose Kestra behind a reverse proxy
- enable HTTPS
- adjust access log format for GCP or ECS
- configure Prometheus-style metrics ingestion
- change management endpoint behavior
- configure security response headers (CSP, HSTS, framing)
---
# Plugins & Execution Configuration in Kestra
URL: https://kestra.io/docs/configuration/plugins-and-execution
> Configure plugin installation, plugin defaults, feature flags, retries, task settings, system flows, templates, and execution-related behavior in Kestra.
Use this page when configuring how tasks, plugins, and execution-time behaviors work across your Kestra instance.
## Plugins
This section is about how Kestra discovers and distributes plugin code. If a task type is missing, a plugin version needs to be pinned, or your organization uses a private artifact source, start here.
This area includes:
- installing plugins
- custom Maven repositories
- Enterprise plugin repositories
- plugin defaults
- forced plugin defaults
- plugin security and allowed plugins
- plugin management settings
For many teams, this is the most important section after runtime setup because it centralizes behavior shared across many flows.
Install a plugin from Maven repositories with:
```bash
kestra plugins install io.kestra.plugin:plugin-script-python:LATEST
```
Add custom repositories:
```yaml
kestra:
plugins:
repositories:
central:
url: https://repo.maven.apache.org/maven2/
google-artifact-registry:
url: https://${GCP_REGISTRY_LOCATION}-maven.pkg.dev/${GCP_PROJECT_ID}/${GCP_REPOSITORY}
basic-auth:
username: oauth2accesstoken
password: ${GCP_OAUTH_ACCESS_TOKEN}
```
Install EE plugins from the Kestra registry:
```yaml
kestra:
plugins:
repositories:
kestra-io:
url: https://registry.kestra.io/maven
basic-auth:
username: ${kestra.ee.license.id:}
password: ${kestra.ee.license.fingerprint:}
```
Most teams only need custom repositories if they publish private plugins or mirror public artifacts through an internal registry.
## Static policies (global defaults and enforcement)
In Kestra 2.0, global plugin defaults are replaced by static policies declared under `kestra.policies` in server configuration. Static policies form the outermost scope, apply across all tenants, and cannot be overridden through the API.
:::alert{type="warning"}
`kestra.plugins.defaults` is removed in Kestra 2.0. See the [pluginDefaults Removed migration guide](../../11.migration-guide/v2.0.0/plugin-defaults-removed/index.md) to convert existing defaults to policies.
:::
Apply an installation-wide default that flows can still override:
```yaml
kestra:
policies:
- id: global-log-level
description: "Default log level for all Log tasks."
rules:
- type: io.kestra.plugin.ee.rules.Add
on: PLUGIN
where:
- field: type
operator: EQUAL_TO
value: io.kestra.plugin.core.log.Log
values:
level: ERROR
```
Enforce a value that tasks cannot override (`override: true`):
```yaml
kestra:
policies:
- id: enforce-docker-isolation
description: "Force Docker task runner for all shell script tasks."
rules:
- type: io.kestra.plugin.ee.rules.Add
on: PLUGIN
override: true
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.scripts.shell
values:
containerImage: ubuntu:latest
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
```
Static policies are evaluated by the Executor and propagated to all components, so every server should have the same `kestra.policies` configuration. A malformed static policy prevents server startup (fail-closed) — validate in a staging environment first.
Precedence works as follows:
- Static policies (`kestra.policies`) form the outermost scope
- Tenant-level and namespace-level Policies apply inside that
- Task properties fill any remaining unset values
- `override: true` on a policy rule always wins over the author's value
Use `override: false` (the default) for convenience defaults and `override: true` when the platform must enforce a value such as a specific task runner.
Enable or preconfigure plugin features globally:
```yaml
kestra:
plugins:
configurations:
- type: io.kestra.plugin.core.flow.Subflow
values:
outputs:
enabled: true
- type: io.kestra.plugin.core.trigger.Schedule
values:
recoverMissedSchedules: NONE
```
You can also enable Docker task-runner volume mounting:
```yaml
kestra:
plugins:
configurations:
- type: io.kestra.plugin.scripts.runner.docker.Docker
values:
volumeEnabled: true
```
The examples in this section do different jobs: `defaults` applies reusable task values, while `configurations` enables or tunes plugin features that are not always expressed directly in a flow.
## Plugin security and management
This section becomes relevant once you want governance over what can be installed or when plugin lifecycle is managed centrally instead of manually on each instance.
In Enterprise Edition, you can restrict which plugins are allowed:
```yaml
kestra:
plugins:
security:
includes:
- io.kestra.*
excludes:
- io.kestra.plugin.core.debug.Echo
```
Managed plugins are configured under `kestra.plugins.management`:
```yaml
kestra:
plugins:
management:
enabled: true
remote-storage-enabled: true
custom-plugins-enabled: true
local-repository-path: /tmp/kestra/plugins-repository
auto-reload-enabled: true
auto-reload-interval: 60s
default-version: LATEST
```
- `remote-storage-enabled`: store managed plugins in internal storage rather than on local disk
- `auto-reload-enabled` / `auto-reload-interval`: check for updated plugins on a fixed interval
- `default-version`: controls which plugin version is selected when no explicit version is pinned; accepts `LATEST`, `CURRENT`, `OLDEST`, `NONE`, or a specific version string
## Execution behavior
These settings affect how the platform behaves around tasks and executions globally. Use them for platform-wide operational defaults, not for flow-specific logic.
This part of the configuration also includes:
- retries
- temporary task storage
- HTTP task URL filtering
- tutorial flows
- system flows
- local flow synchronization
- enabling templates
Global retries for internal storage and secret-manager calls:
```yaml
kestra:
retries:
attempts: 5
delay: 1s
max-delay: ~
multiplier: 2.0
```
`max-delay` caps the maximum backoff interval. It is undefined by default, which means the delay grows without bound according to the multiplier.
:::alert{type="warning"}
These retries do not apply to tasks. For task-level retries across many plugins, use plugin defaults.
:::
Example task-level retry default:
```yaml
- type: io.kestra
retry:
type: constant
interval: PT5M
maxDuration: PT1H
maxAttempts: 3
warningOnRetry: true
```
That distinction matters: `kestra.retries` protects platform integrations such as storage and secret backends, while task retry behavior should be managed through plugin defaults or the flow itself.
Use `kestra.tasks.tmp-dir` when task runners need a predictable working directory on the host or inside a mounted volume:
```yaml
kestra:
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
```
Ensure your container or VM volume mounts align with that path:
```yaml
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /home/kestra:/home/kestra
```
### HTTP task URL filtering
Use `kestra.tasks.http` to restrict which URLs HTTP plugin tasks can call. Configure an allow-list, a deny-list, or both:
```yaml
kestra:
tasks:
http:
allowed-list:
- https://api.example.com
denied-list:
- http://169.254.169.254
- http://localhost
```
| Key | Default | Description |
|---|---|---|
| `kestra.tasks.http.allowed-list` | `[]` | When non-empty, a request URI must start with at least one entry or the task fails. |
| `kestra.tasks.http.denied-list` | `[]` | A request URI that starts with any entry causes the task to fail. Evaluated after the allowed-list. |
For security guidance and matching behavior, see [HTTP task URL filtering](../../10.administrator-guide/security-hardening/index.md#http-task-url-filtering).
Reserve `system` for background workflows, or rename it if your organization already uses that namespace for something else:
```yaml
kestra:
system-flows:
namespace: system
```
Disable tutorial flows outside trial or demo environments:
```yaml
kestra:
tutorial-flows:
enabled: false
```
Templates are deprecated and disabled by default, but can still be re-enabled for migration work:
```yaml
kestra:
templates:
enabled: true
```
```yaml
micronaut:
io:
watch:
enabled: true
paths:
- /path/to/your/flows
```
Use Micronaut file watching only when you want local flow synchronization from disk into Kestra.
## Variables and rendering
These settings influence expression rendering across the whole instance. They are linked here because they affect execution-time behavior, but they are documented in more depth on the runtime page.
Relevant runtime-wide settings include:
- environment variable prefixes
- global variables
- recursive rendering
- template cache
Those settings are documented in more detail on [Runtime and Storage](../02.runtime-and-storage/index.md), since they affect the whole instance and not just plugin behavior.
### Subflow function configuration
The `subflow()` Pebble function, used to populate `SELECT` and `MULTISELECT` input dropdowns at form render time, has three configurable limits. All three accept ISO 8601 duration strings or integers.
```yaml
kestra:
pebble:
subflow-function:
default-timeout: PT1M # timeout when the caller omits the timeout argument
max-timeout: PT5M # hard cap — larger values are rejected at runtime
max-depth: 3 # maximum nesting depth of subflow() calls on one render thread
```
| Key | Default | Description |
|---|---|---|
| `kestra.pebble.subflow-function.default-timeout` | `PT1M` | Applied when the `timeout` argument is not passed. Keep this short — the call blocks the Execute form render. |
| `kestra.pebble.subflow-function.max-timeout` | `PT5M` | Hard cap. A `timeout` argument larger than this value is rejected at runtime with an error. |
| `kestra.pebble.subflow-function.max-depth` | `3` | Guards against runaway recursion. A subflow whose own inputs also call `subflow()` counts against this limit. |
Increase `max-timeout` only if your data-fetching subflows genuinely need longer — long form renders degrade user experience. Increase `max-depth` only if you have intentionally nested multi-level dependent dropdowns.
## Related docs
- Installation-wide configuration rules: [Policies](../../07.enterprise/02.governance/policies/index.md) (Enterprise Edition)
- Universal file access: [File Access](../../06.concepts/file-access/index.md)
- Storage backends, JVM, and global variables: [Runtime and Storage](../02.runtime-and-storage/index.md)
- Execution data isolation and enterprise-only runtime features: [Enterprise and Advanced](../06.enterprise-and-advanced/index.md)
---
# Runtime & Storage Configuration in Kestra
URL: https://kestra.io/docs/configuration/runtime-and-storage
> Configure Kestra's repository, queue, datasource, internal storage, server runtime, JVM behavior, environment settings, and variables.
Use this page when configuring the core runtime services that make Kestra run.
## Core setup decisions
Every Kestra deployment must define:
- repository type
- queue type
- internal storage type
Optionally, in Kestra 2.0 and later, you can also configure a separate log store:
- log data store (defaults to the repository backend if not set)
The common production path is PostgreSQL for queue and repository, plus an object store or durable internal storage backend.
Queues and repositories must stay compatible:
- in-memory queue with in-memory repository for local testing only
- JDBC queue with H2, MySQL, or PostgreSQL repository
- Kafka queue with Elasticsearch repository in Enterprise Edition
## Allocated CPU cores
Kestra sizes several internal thread pools based on the number of CPU cores available to the process. By default, it uses the number of CPU cores reported by the runtime environment.
If you want Kestra to size those pools using a different value, set `kestra.allocated-cpu-cores`:
```yaml
kestra:
allocated-cpu-cores: 2
```
This is useful when you want to limit how aggressively Kestra allocates worker, scheduler, and queue-related threads without changing container limits or host-level CPU settings.
## Database and datasources
Start here if you are choosing the queue and repository backend for a new Kestra instance or moving from a local setup to a durable environment. In most teams, this is the first configuration page they revisit after initial installation.
Use `kestra.queue.type` and `kestra.repository.type` to select your backend:
```yaml
kestra:
queue:
type: postgres
repository:
type: postgres
```
Then define the datasource:
```yaml
datasources:
postgres:
url: jdbc:postgresql://localhost:5432/kestra
driver-class-name: org.postgresql.Driver
username: kestra
password: k3str4
```
The examples below are intentionally minimal. Use them to confirm the backend choice and basic connection shape first, then add pooling and operational settings afterward.
Minimal datasource examples:
:::collapse{title="PostgreSQL"}
```yaml
kestra:
queue:
type: postgres
repository:
type: postgres
datasources:
postgres:
url: jdbc:postgresql://localhost:5432/kestra
driver-class-name: org.postgresql.Driver
username: kestra
password: k3str4
```
:::
:::collapse{title="MySQL"}
```yaml
kestra:
queue:
type: mysql
repository:
type: mysql
datasources:
mysql:
url: jdbc:mysql://localhost:3306/kestra
driver-class-name: com.mysql.cj.jdbc.Driver
username: kestra
password: k3str4
dialect: MYSQL
```
:::
:::collapse{title="H2"}
```yaml
kestra:
queue:
type: h2
repository:
type: h2
datasources:
h2:
url: jdbc:h2:mem:public;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
username: sa
password: ""
driver-class-name: org.h2.Driver
```
:::
Use H2 for local development. For production, prefer PostgreSQL, or MySQL if PostgreSQL is not an option.
:::alert{type="info"}
For PostgreSQL performance issues, consider `random_page_cost=1.1` and `kestra.queue.postgres.disable-seq-scan=true` if queue polling is choosing poor query plans.
:::
## Log data store
By default, execution logs are stored in the same database as flows and executions. In Kestra 2.0+, you can route logs to a separate store by setting `kestra.logs.type`. If this key is not set, logs continue to use the repository backend — no migration needed.
```yaml
kestra:
logs:
type: postgres # or h2, mysql, elasticsearch
postgres:
url: jdbc:postgresql://logs-db:5432/kestra_logs
username: kestra
password: k3str4
```
The External Log Data Store is an Enterprise Edition feature. Both JDBC (H2, Postgres, MySQL) and Elasticsearch backends require EE. For full configuration examples and the capability reference, see the [External Log Data Store](../../10.administrator-guide/log-data-store/index.md) guide.
## Connection pooling and JDBC queue tuning
Most users can keep the defaults here until they see either connection pressure or queue latency. This section matters most for larger deployments, split-component topologies, or databases that are already under load.
Kestra uses HikariCP for datasource pooling. Common options include:
| Property | Purpose | Default |
| --- | --- | --- |
| `maximum-pool-size` | Maximum number of open connections | `10` |
| `minimum-idle` | Minimum number of idle connections | `10` |
| `connection-timeout` | Max wait for a connection (ms) | `30000` |
| `idle-timeout` | Max idle time (ms) | `600000` |
| `max-lifetime` | Max connection lifetime (ms) | `1800000` |
:::collapse{title="Full HikariCP property reference"}
| Property | Type | Description | Default |
| --- | --- | --- | --- |
| `url` | String | JDBC connection string | — |
| `username` | String | Database username | — |
| `password` | String | Database password | — |
| `catalog` | String | Default catalog | driver default |
| `schema` | String | Default schema | driver default |
| `transaction-isolation` | String | Default transaction isolation level | driver default |
| `pool-name` | String | Pool name | `HikariPool-` |
| `maximum-pool-size` | Int | Maximum number of open connections | `10` |
| `minimum-idle` | Long | Minimum number of idle connections | `10` |
| `connection-timeout` | Long | Max time to wait for a connection (ms) | `30000` |
| `idle-timeout` | Long | Max time a connection can be idle (ms) | `600000` |
| `max-lifetime` | Long | Max connection lifetime (ms) | `1800000` |
| `validation-timeout` | Long | Max time to validate a connection (ms) | `5000` |
| `initialization-fail-timeout` | Long | Timeout for pool initialization failure (ms) | `1` |
| `leak-detection-threshold` | Long | Threshold before a connection leak is reported (ms) | `0` |
| `connection-init-sql` | String | SQL executed on each new connection | `null` |
| `connection-test-query` | String | Query used to validate connections | `null` |
:::
Example:
```yaml
datasources:
postgres:
url: jdbc:postgresql://localhost:5432/kestra
driver-class-name: org.postgresql.Driver
username: kestra
password: k3str4
maximum-pool-size: 20
minimum-idle: 10
```
Rough connection planning:
- standalone server: about 10 connections
- split components: about 40 connections
- split components with 3 replicas: about 120 connections
JDBC queues long-poll the `queues` table. Lower intervals reduce latency but increase database load:
```yaml
kestra:
jdbc:
queues:
poll-size: 100
min-poll-interval: 25ms
max-poll-interval: 1000ms
poll-switch-interval: 5s
```
The JDBC cleaner removes old queue rows:
```yaml
kestra:
jdbc:
cleaner:
initial-delay: 1h
fixed-delay: 1h
retention: 7d
```
To reject oversized JDBC messages before they create memory pressure:
```yaml
kestra:
jdbc:
queues:
message-protection:
enabled: true
limit: 1048576
```
If you are not troubleshooting queue throughput or database pressure, you can usually leave the JDBC queue settings alone and return to them only when scaling.
## Internal storage
`kestra.storage.type` controls where Kestra stores internal files such as task outputs, namespace files, and execution artifacts. Choose the backend based on durability and whether all Kestra components can reach the same storage.
Common options include:
- `local` for local testing
- `s3`
- `s3files`
- `gcs`
- `azure`
- `minio`
- other object-storage-compatible backends
The default local storage is fine for local testing but not for every production topology. The important distinction is whether every Kestra component can see the same files.
### Local storage deployment guidance
Local storage works well for standalone deployments with a persistent volume. In distributed deployments, it only works safely when all components share the same filesystem through a `ReadWriteMany` volume or an equivalent shared storage layer.
If that shared filesystem does not exist, move to object storage instead of trying to share host paths between services.
### Storage isolation
Storage isolation restricts which Kestra services are permitted to access internal storage files directly:
```yaml
kestra:
storage:
type: gcs
isolation:
enabled: true
denied-services:
- EXECUTOR
```
| Property | Description |
|---|---|
| `isolation.enabled` | Enable service isolation (default `false`). |
| `isolation.denied-services` | List of Kestra service names that must not access storage (e.g. `EXECUTOR`, `INDEXER`, `SCHEDULER`). |
---
### Local
```yaml
kestra:
storage:
type: local
local:
base-path: /app/storage
```
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `base-path` | string | **Yes** | — | Filesystem path where Kestra stores internal files. |
Local storage works for standalone deployments with a persistent volume. In distributed deployments, it only works safely when all components share the same filesystem through a `ReadWriteMany` volume or equivalent shared storage. If that is not the case, use object storage instead.
---
### AWS S3
#### Minimum configuration
```yaml
kestra:
storage:
type: s3
s3:
bucket: my-kestra-bucket
region: us-east-1
```
With explicit credentials:
```yaml
kestra:
storage:
type: s3
s3:
bucket: my-kestra-bucket
region: us-east-1
access-key: YOUR_ACCESS_KEY
secret-key: YOUR_SECRET_KEY
```
#### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `bucket` | string | **Yes** | — | S3 bucket for internal storage. |
| `access-key` | string | No | — | AWS access key ID. Falls back to `DefaultCredentialsProvider` if not set. |
| `secret-key` | string | No | — | AWS secret access key. |
| `region` | string | No | — | AWS region (e.g. `us-east-1`). |
| `endpoint` | string | No | — | Custom endpoint URL for S3-compatible services or VPC PrivateLink. |
| `force-path-style` | boolean | No | `false` | Force path-style addressing instead of virtual-hosted style. |
| `path` | string | No | — | Object key prefix within the bucket (e.g. `kestra/`). |
| `sts-role-arn` | string | No | — | IAM role ARN to assume via STS before accessing the bucket. |
| `sts-role-external-id` | string | No | — | External ID for STS AssumeRole. |
| `sts-role-session-name` | string | No | — | Session name for the assumed role. |
| `sts-role-session-duration` | duration | No | `PT15M` | Duration of the assumed-role session. |
| `sts-endpoint-override` | string | No | — | Override the STS endpoint URL. |
| `s3-files-compatible` | boolean | No | `false` | Enable S3 bucket versioning when the bucket is first initialized. Set to `true` when the same bucket is shared with the `S3FilesStorage` backend (`type: s3-files`), which mounts S3 Files as a local NFS filesystem and requires versioning to be enabled. |
#### S3-compatible endpoints
The `s3` backend targets Amazon S3 by default. Set `endpoint` to point that same backend at another S3-compatible object store, for example Backblaze B2, Cloudflare R2, or MinIO. Some endpoints also require `force-path-style: true`; MinIO deployments can alternatively use the dedicated [`type: minio`](#minio--s3-compatible) backend.
```yaml
kestra:
storage:
type: s3
s3:
bucket: my-kestra-bucket
endpoint: https://your-s3-endpoint.example.com
force-path-style: true
access-key: YOUR_ACCESS_KEY
secret-key: YOUR_SECRET_KEY
```
#### Credential resolution order
1. `access-key` / `secret-key` in config.
2. `sts-role-arn` — role assumption chained on top of any resolved identity.
3. `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` environment variables.
4. AWS CLI profile (`~/.aws/credentials`).
5. EKS Pod Identity or IRSA (IAM Roles for Service Accounts).
6. EC2/ECS instance metadata profile.
:::alert{type="warning"}
If you inject S3 credentials through Helm `extraEnvVars`, use double underscores on multi-word property names. `KESTRA_STORAGE_S3_ACCESS_KEY` is parsed as `kestra.storage.s3.access.key` (nested) and rejected at startup. Use `KESTRA_STORAGE_S3_ACCESS__KEY` instead. See [Environment variable conversion](../01.configuration-basics/index.md#environment-variable-conversion).
:::
#### Permissions
Grant the IAM identity Kestra uses (user, role, or instance profile) the following permissions on the bucket:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:ListBucket", "s3:GetBucketLocation"],
"Resource": "arn:aws:s3:::YOUR_BUCKET"
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
"Resource": "arn:aws:s3:::YOUR_BUCKET/*"
}
]
}
```
If `s3-files-compatible: true` is set, also include `s3:PutBucketVersioning` in the bucket-level statement. If `sts-role-arn` is set, the calling identity additionally needs `sts:AssumeRole` on that role ARN.
---
### Google Cloud Storage
#### Minimum configuration
```yaml
kestra:
storage:
type: gcs
gcs:
bucket: my-kestra-bucket
project-id: my-gcp-project
```
With an explicit service account key:
```yaml
kestra:
storage:
type: gcs
gcs:
bucket: my-kestra-bucket
project-id: my-gcp-project
service-account: |
{ "type": "service_account", ... }
```
#### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `bucket` | string | **Yes** | — | GCS bucket for internal storage. |
| `project-id` | string | No | — | GCP project ID. |
| `service-account` | string | No | — | Service account JSON key. Falls back to `GOOGLE_APPLICATION_CREDENTIALS` or ambient credentials (GKE Workload Identity, GCE metadata) if omitted. |
| `path` | string | No | — | Object prefix within the bucket (e.g. `kestra/`). |
#### Credential resolution order
1. `service-account` JSON key in config.
2. `GOOGLE_APPLICATION_CREDENTIALS` environment variable pointing to a key file.
3. Google default application credentials (Workload Identity on GKE, GCE metadata server, gcloud CLI).
:::alert{type="warning"}
`KESTRA_STORAGE_GCS_PROJECT_ID` is parsed as `kestra.storage.gcs.project.id` (nested) and rejected at startup. Use `KESTRA_STORAGE_GCS_PROJECT__ID` with double underscores, or keep the value in YAML. See [Environment variable conversion](../01.configuration-basics/index.md#environment-variable-conversion).
:::
#### Permissions
Assign the `roles/storage.objectAdmin` predefined role on the bucket (not the project) to the service account or workload identity principal. Fine-grained equivalent:
| Permission | Purpose |
|---|---|
| `storage.objects.create` | Upload files to internal storage |
| `storage.objects.delete` | Remove files from internal storage |
| `storage.objects.get` | Download files from internal storage |
| `storage.objects.list` | List objects in the bucket |
| `storage.buckets.get` | Read bucket metadata |
---
### S3 Files
Use `s3files` when Kestra runs on a host where an [S3 Files](https://aws.amazon.com/blogs/aws/launching-s3-files-making-s3-buckets-accessible-as-file-systems/) NFS filesystem is already mounted locally. This backend reads and writes directly through the local filesystem — no S3 SDK or AWS credentials are required.
Mount the NFS filesystem on every host that runs a Kestra component before configuring this backend. All components must share the same mount path.
```yaml
kestra:
storage:
type: s3files
s3files:
mount-path: "/mnt/s3files"
```
`mount-path` must point to a directory that exists and is readable and writable by the Kestra process. Kestra will not create the directory on startup.
Object metadata is stored in `.meta` sidecar files alongside each object on the filesystem. Custom S3 object metadata is not exposed through this backend.
If you prefer to keep the S3 SDK path (for example, because not every host has the NFS mount), use the standard `s3` backend with `s3-files-compatible: true` instead.
---
### Azure Blob Storage
:::alert{type="info"}
Disable hierarchical namespace on the target container. That Azure feature is not supported by the storage backend.
:::
#### Minimum configuration
```yaml
kestra:
storage:
type: azure
azure:
endpoint: https://myaccount.blob.core.windows.net
container: kestra-storage
```
#### Configuration reference
**Connection**
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `endpoint` | string | **Yes** | — | Azure Blob service endpoint (e.g. `https://.blob.core.windows.net`). |
| `container` | string | **Yes** | — | Container name used for internal storage. |
**Connection string auth**
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `connection-string` | string | No | — | Full Azure Storage connection string. Highest-priority auth — if set, all other auth properties are ignored. |
**Shared key auth**
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `shared-key-account-name` | string | No | — | Storage account name. Used together with `shared-key-account-access-key`. |
| `shared-key-account-access-key` | string | No | — | Storage account access key. |
**SAS token auth**
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `sas-token` | string | No | — | Shared Access Signature token. |
**Managed identity auth**
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `managed-identity-client-id` | string | No | — | Client ID of a user-assigned managed identity. Omit for system-assigned. |
| `managed-identity-resource-id` | string | No | — | Resource ID of a user-assigned managed identity. Alternative to `managed-identity-client-id`. |
**Workload identity auth**
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `workload-identity-client-id` | string | No | — | Client ID for Azure Workload Identity (AKS federated credentials). |
**All auth modes**
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `additionally-allowed-tenants` | string[] | No | — | Additional tenant IDs the credential may acquire tokens for. Use `"*"` to allow any tenant. |
#### Credential resolution order
1. `connection-string` — if set, all other auth properties are skipped.
2. `shared-key-account-name` + `shared-key-account-access-key`.
3. `sas-token`.
4. `DefaultAzureCredential` chain — environment variables (`AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `AZURE_TENANT_ID`), then managed identity (honoring `managed-identity-client-id` / `managed-identity-resource-id`), then workload identity (honoring `workload-identity-client-id`), then Azure CLI.
#### Permissions
Assign the **Storage Blob Data Contributor** built-in role to the identity (service principal, managed identity, or workload identity principal) on the container or storage account. This role covers all data-plane read, write, and delete operations on blobs.
Connection string and shared key auth carry full data-plane access inherently and do not require a role assignment.
---
### MinIO / S3-compatible
Use `type: minio` for MinIO and any S3-compatible object storage — including Ceph, SeaweedFS S3 API, Garage, and Outscale OOS.
#### Minimum configuration
```yaml
kestra:
storage:
type: minio
minio:
endpoint: my.minio.domain.com
port: 9000
secure: false
access-key: ${AWS_ACCESS_KEY_ID}
secret-key: ${AWS_SECRET_ACCESS_KEY}
region: "default"
bucket: my-bucket
part-size: 5MB
# httpConnectTimeout: PT10S # optional; omit to use OkHttp default (10 s)
# httpReadTimeout: PT10S # optional; omit to use OkHttp default (10 s)
# httpWriteTimeout: PT10S # optional; omit to use OkHttp default (10 s)
```
#### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `endpoint` | string | No | — | MinIO or S3-compatible server hostname or URL. |
| `port` | integer | No | `0` | Server port. Default `0` uses the standard port for the chosen scheme. |
| `secure` | boolean | No | `false` | Use TLS for the connection. |
| `access-key` | string | No | — | Access key ID. |
| `secret-key` | string | No | — | Secret access key. |
| `region` | string | No | — | Region to include in request signing. |
| `bucket` | string | No | — | Bucket name. |
| `part-size` | size | No | `5MB` | Multipart upload part size. Minimum 5 MiB. |
| `vhost` | boolean | No | `false` | Enable virtual-hosted style bucket URLs. Set to `true` when MinIO uses `MINIO_DOMAIN`. |
| `ca-pem` | string | No | — | CA certificate PEM for custom TLS trust. |
| `client-pem` | string | No | — | Client certificate PEM for mutual TLS. |
| `http-connect-timeout` | duration | No | — | HTTP connection timeout. |
| `http-read-timeout` | duration | No | — | HTTP read timeout. |
| `http-write-timeout` | duration | No | — | HTTP write timeout. |
| `http-connection-keep-alive` | duration | No | — | HTTP keep-alive duration. |
| `proxy-configuration` | object | No | — | HTTP proxy settings (host, port, type, username, password). |
| `ssl-options` | object | No | — | Advanced SSL/TLS options (protocols, cipher suites, trust manager). |
If MinIO uses `MINIO_DOMAIN`, set `vhost: true` and point `endpoint` at the base domain rather than `bucket.domain`.
#### Permissions
Create a MinIO policy that grants the access key read/write access to the bucket, then attach it via `mc admin policy attach`:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:ListBucket", "s3:GetBucketLocation"],
"Resource": "arn:aws:s3:::YOUR_BUCKET"
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
"Resource": "arn:aws:s3:::YOUR_BUCKET/*"
}
]
}
```
For Ceph RGW, SeaweedFS, or other S3-compatible backends, apply the equivalent bucket ACL or user policy through that system's admin interface.
---
If large bucket operations such as `deleteByPrefix()` produce `SocketException: Socket closed` errors on heavily loaded or large buckets, increase `httpReadTimeout` or set it to `PT0S` to disable the timeout entirely.
### SeaweedFS
```yaml
kestra:
storage:
type: seaweedfs
seaweedfs:
filer-host: localhost
filer-port: 18888
```
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `filer-host` | string | No | — | Hostname or IP of the SeaweedFS filer server. |
| `filer-port` | integer | No | `18888` | gRPC port of the SeaweedFS filer server. |
| `prefix` | string | No | `""` | Root prefix path for all storage operations. |
| `replication` | string | No | `"000"` | Replication factor (`000` = no replication, `001` = 1 copy). |
SeaweedFS has no built-in authentication. Control access at the network level by restricting which hosts can reach the filer gRPC port.
---
### Cloudflare R2
```yaml
kestra:
storage:
type: cloudflare
cloudflare:
account-id: YOUR_CLOUDFLARE_ACCOUNT_ID
bucket: kestra-storage
access-key-id: YOUR_R2_ACCESS_KEY_ID
secret-access-key: YOUR_R2_SECRET_ACCESS_KEY
```
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `account-id` | string | **Yes** | — | Cloudflare account ID, used to build the R2 endpoint. |
| `bucket` | string | **Yes** | — | R2 bucket name. |
| `access-key-id` | string | No | — | R2 API token access key ID. |
| `secret-access-key` | string | No | — | R2 API token secret access key. |
| `path` | string | No | — | Object key prefix within the bucket. |
| `endpoint-override` | string | No | — | Override the R2 endpoint URL (used for testing with S3-compatible services). |
#### Permissions
Create an R2 API token with **Object Read & Write** permissions. Scope the token to the specific bucket rather than all buckets. R2 does not use IAM-style policies — permissions are set directly on the API token in the Cloudflare dashboard under **R2 → Manage R2 API Tokens**.
---
### Huawei OBS
Kestra supports Huawei Cloud Object Storage Service (OBS) as an internal storage backend.
#### Minimum configuration
```yaml
kestra:
storage:
type: obs
obs:
access-key: YOUR_OBS_ACCESS_KEY
secret-key: YOUR_OBS_SECRET_KEY
bucket: kestra-storage
region: cn-north-4
```
#### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `access-key` | string | **Yes** | — | OBS access key (AK). |
| `secret-key` | string | **Yes** | — | OBS secret key (SK). |
| `bucket` | string | **Yes** | — | OBS bucket for internal storage. |
| `region` | string | No | — | Huawei Cloud region (e.g. `cn-north-4`). Resolves to `https://obs..myhuaweicloud.com`. Ignored when `endpoint` is set. |
| `endpoint` | string | No | — | Explicit OBS or S3-compatible endpoint URL. Takes precedence over `region`. |
| `security-token` | string | No | — | Security token for temporary AK/SK credentials. |
| `path` | string | No | — | Object key prefix within the bucket. |
| `path-style-access` | boolean | No | `false` | Use path-style bucket addressing. Required for MinIO and most S3-compatible endpoints. |
#### Permissions
Attach a bucket policy or Huawei IAM policy granting the AK/SK identity the following actions on the bucket and its objects:
| Action | Scope |
|---|---|
| `obs:bucket:ListAllMyBuckets`, `obs:bucket:ListBucket`, `obs:bucket:GetBucketLocation` | Bucket |
| `obs:object:GetObject`, `obs:object:PutObject`, `obs:object:DeleteObject` | Objects (`/*`) |
Alternatively, assign the predefined **OBSFullAccess** Huawei IAM policy scoped to the bucket if a custom bucket policy is not required.
## Server, environment, and JVM settings
These settings shape how the instance presents itself and how the Java process behaves at runtime. They are less about feature enablement and more about making the deployment fit its environment.
Common runtime areas include:
- `kestra.server.*` for basic auth and liveness
- `kestra.url` for the instance URL
- `kestra.environment.*` for environment display metadata
- `JAVA_OPTS` for JVM tuning such as timezone and heap settings
- `kestra.variables.*` for global variables and recursive rendering behavior
Environment metadata shown in the UI:
```yaml
kestra:
environment:
name: Production
color: "#FCB37C"
```
JVM settings are usually passed through `JAVA_OPTS`:
```bash
export JAVA_OPTS="-Duser.timezone=Europe/Paris -Xmx1g"
```
Common uses include:
- setting `user.timezone` to control scheduling and log display
- setting a fixed heap with `-Xmx`
- configuring Java proxy settings for outbound access
Global variables and rendering behavior also live here:
```yaml
kestra:
variables:
env-vars-prefix: ENV_
globals:
region: eu-west-1
recursive-rendering: true
cache-enabled: true
```
`env-vars-prefix` controls which environment variables become available in expressions under `envs.*`. For example, `ENV_MY_VARIABLE` becomes `{{ envs.my_variable }}`.
Use `globals` for values that need to be available in every flow, `recursive-rendering` only when you intentionally want pre-0.14 recursive behavior, and `cache-enabled` when you need to trade CPU for correctness while debugging template changes. Set `cache-size` to limit the number of cached templates (default `1000`):
```yaml
kestra:
variables:
cache-size: 1000
```
## Optional runtime features
These settings are not part of the core queue or repository setup, but they do matter in real deployments.
Some notifications and generated links depend on `kestra.url` being set to the public base URL without `/ui` or `/api`:
```yaml
kestra:
url: https://www.my-host.com/kestra/
```
The web UI can also be customized at runtime:
```yaml
kestra:
webserver:
google-analytics: UA-12345678-1
html-head: |
```
Use `html-head` sparingly for environment banners, extra CSS, or internal scripts that must load with the app shell.
### Local file access
To allow universal file access from host-mounted paths, both mount the directory and add it to the allowlist:
```yaml
kestra:
local-files:
allowed-paths:
- /scripts
enable-preview: false
```
Without the allowlist, file-access URIs pointing at local host paths will be rejected even if the path is mounted into the container.
The `io.kestra.plugin.fs.local.Upload` and `io.kestra.plugin.fs.local.Uploads` tasks enforce their own `allowed-paths` check, independent of `kestra.local-files.allowed-paths`. Configure permitted directories under `plugins.configurations`:
```yaml
kestra:
plugins:
configurations:
- type: io.kestra.plugin.fs.local.Uploads
values:
allowed-paths:
- /data/uploads
- type: io.kestra.plugin.fs.local.Upload
values:
allowed-paths:
- /data/uploads
```
## When to use this page
- Need logs, telemetry, metrics, endpoints, CORS, or SSL: [Observability and Networking](../03.observability-and-networking/index.md)
- Need plugin defaults, retries, task temp storage, templates, or system flows: [Plugins and Execution](../04.plugins-and-execution/index.md)
- Need secret backends or server hardening: [Security and Secrets](../05.security-and-secrets/index.md)
---
# Security & Secrets Configuration in Kestra
URL: https://kestra.io/docs/configuration/security-and-secrets
> Configure encryption, secret backends, auth-related security settings, RBAC-adjacent platform security, and secure server behavior in Kestra.
Use this page when you need to protect sensitive values or harden a Kestra deployment.
## Encryption
This is the minimum security configuration most self-managed instances should think about early, because it determines whether sensitive flow values can be stored safely at rest.
Kestra supports encryption of sensitive inputs and outputs at rest through `kestra.encryption.secret-key`.
Example:
```yaml
kestra:
encryption:
secret-key: BASE64_ENCODED_STRING_OF_32_CHARACTERS
```
Generate a key with:
```bash
openssl rand -base64 32
```
Without `kestra.encryption.secret-key`, `SECRET` inputs and outputs fail at runtime because Kestra cannot encrypt the value at rest.
Example flow using `SECRET` types:
```yaml
id: my_secret_flow
namespace: company.team
inputs:
- id: secret
type: SECRET
tasks:
- id: mytask
type: io.kestra.plugin.core.log.Log
message: task that needs the secret to connect to an external system
outputs:
- id: secret_output
type: SECRET
value: "{{ inputs.secret }}"
```
## Secret backends
Choose the backend based on where your organization already stores secrets. In practice, most teams want Kestra to consume an existing cloud or Vault-based secret system rather than create a separate one just for workflows.
Kestra can be configured to use a secrets backend through `kestra.secret.*`.
:::alert{type="info"}
For the full backend reference — including all supported backends, complete property tables, permissions, credential resolution order, and read-only mode — see [Secrets Manager](../../07.enterprise/02.governance/secrets-manager/index.md).
:::
Base structure:
```yaml
kestra:
secret:
type: azure-key-vault
azure-key-vault:
client-secret:
tenant-id: "id"
client-id: "id"
client-secret: "secret"
isolation:
enabled: true
denied-services:
- EXECUTOR
```
`isolation` is the key control to understand here: it limits which Kestra services are allowed to resolve secrets, which is useful when you want workers or executors to have narrower access than the whole platform.
The Azure service principal referenced in the base structure above must have the following Key Vault access policy permissions: `Get`, `List`, `Set`, `Delete`, `Recover`, `Backup`, `Restore`, `Purge`.
Representative backend examples:
AWS Secrets Manager requires the following IAM permissions: `CreateSecret`, `DeleteSecret`, `DescribeSecret`, `GetSecretValue`, `ListSecrets`, `PutSecretValue`, `RestoreSecret`, `TagResource`, `UpdateSecret`.
```yaml
kestra:
secret:
type: aws-secret-manager
aws-secret-manager:
access-key-id: mysuperaccesskey
secret-key-id: mysupersecret-key
session-token: mysupersessiontoken
region: us-east-1
```
Google Secret Manager requires the `roles/secretmanager.admin` role. Omit `service-account` to fall back to `GOOGLE_APPLICATION_CREDENTIALS` or the environment's default credentials:
```yaml
kestra:
secret:
type: google-secret-manager
google-secret-manager:
project: gcp-project-id
service-account: |
```
Elasticsearch secrets are additionally encrypted with AES. The key must be at least 32 characters:
```yaml
kestra:
secret:
type: elasticsearch
elasticsearch:
secret: "a-secure-32-character-minimum-key"
```
HashiCorp Vault (KV v2) supports Userpass, Token, and AppRole authentication.
Userpass:
```yaml
kestra:
secret:
type: vault
vault:
address: "http://localhost:8200"
password:
user: john
password: foo
```
Token:
```yaml
kestra:
secret:
type: vault
vault:
address: "http://localhost:8200"
token:
token: your-secret-token
```
AppRole:
```yaml
kestra:
secret:
type: vault
vault:
address: "http://localhost:8200"
app-role:
path: approle
role-id: your-role-id
secret-id: your-secret-id
```
JDBC-backed secrets, secret tags, and secret caching are covered below.
### JDBC secret backend
Use JDBC-backed secrets only when secrets must stay inside the same database boundary as Kestra and you do not already have a dedicated secret manager:
```yaml
kestra:
secret:
type: jdbc
jdbc:
secret: "your-secret-key"
```
### Secret tags
Some backends let you scope lookups with tags so the same secret manager can serve multiple environments:
```yaml
kestra:
secret:
:
tags:
application: kestra-production
```
Tags are useful when secrets are selected by metadata rather than by one fixed path convention.
### Secret cache
Caching reduces repeated secret-manager calls for frequently used values:
```yaml
kestra:
secret:
cache:
enabled: true
maximum-size: 1000
expire-after-write: 60s
```
Use a cache when executions hit the same secret names repeatedly, but keep TTLs conservative if secret rotation happens often.

## Security settings
This section is about hardening the running platform rather than managing secret values. Reach for it when you are locking down access, controlling invitations and roles, or tuning how the server reacts to component health issues.
This group includes:
- instance owner behavior
- default roles
- invitation expiration
- password rules
- server basic auth
- deletion of configuration files
Server and endpoint hardening examples:
```yaml
kestra:
server:
basic-auth:
username: admin@kestra.io
password: change-me
open-urls:
- "/api/v1/main/executions/webhook/"
```
```yaml
endpoints:
all:
basic-auth:
username: your-user
password: your-password
```
### Instance Owner
The instance owner account has the highest level of platform access and should be reserved for break-glass administration:
```yaml
kestra:
security:
instance-owner:
username: your_username
password: ${KESTRA_INSTANCE_OWNER_PASSWORD}
tenant-admin-access:
-
```
:::alert{type="info"}
`kestra.security.super-admin` is a deprecated alias for `kestra.security.instance-owner` and still works.
:::
:::alert{type="warning"}
Never store clear-text passwords in config. Use environment variables or your platform secret mechanism.
:::
### Default role
Assign a default role to newly created users when you want them to land with a predictable permission set:
```yaml
kestra:
security:
default-role:
name: default
description: "Default role"
permissions:
FLOW: ["VIEW", "LIST", "CREATE", "UPDATE", "DELETE", "EXECUTE"]
```
In multi-tenant environments, scope that role to one tenant:
```yaml
kestra:
security:
default-role:
name: default
description: "Default role"
permissions:
FLOW: ["VIEW", "LIST", "CREATE", "UPDATE", "DELETE", "EXECUTE"]
tenant-id: staging
```
:::alert{type="info"}
Place `default-role` under `kestra.security`, not `micronaut.security`.
:::
### Invitation expiration and password rules
Invitation links expire after seven days by default. Extend them if user onboarding happens through slower approval processes:
```yaml
kestra:
security:
invitations:
expire-after: P30D
```
For username/password auth, configure password complexity explicitly:
```yaml
kestra:
security:
basic-auth:
password-min-length: 8
password-require-special: true
password-min-digits: 1
password-min-lower-case: 1
password-min-upper-case: 1
password-allowed-special-characters: "!@#$%^&*"
```
These rules apply anywhere Kestra asks a user to set or reset a password, including the initial setup flow, invitation acceptance, and user management screens.
### Delete configuration files after startup
If the runtime reads secrets from configuration files, delete them after startup so tasks cannot read them later from disk:
```yaml
kestra:
configurations:
delete-files-on-start: true
```
Liveness and heartbeat settings also belong here. The parameter constraints below affect cluster stability:
- `timeout` — must match across **all Executors**
- `initial-delay` — must match across **all Executors**
- `heartbeat-interval` — must be strictly less than `timeout`
Recommended settings for JDBC-backed (OSS) deployments:
```yaml
kestra:
server:
liveness:
enabled: true
interval: 5s
timeout: 45s
initial-delay: 45s
heartbeat-interval: 3s
```
Recommended settings for Kafka-based (EE) deployments:
```yaml
kestra:
server:
liveness:
timeout: 1m
initial-delay: 1m
```
:::alert{type="warning"}
Worker liveness in Kafka mode is handled by Kafka's protocol guarantees, so you only need to set `timeout` and `initial-delay` for the EE stack.
:::
Heartbeat and restart behavior also belong here:
- `kestra.heartbeat.frequency` controls how often workers emit heartbeats. Default: `10s`.
- `kestra.heartbeat.heartbeat-missed` controls how many missed heartbeats mark a worker as dead. Default: `3`.
- `kestra.server.worker-task-restart-strategy` accepts `NEVER`, `IMMEDIATELY`, or `AFTER_TERMINATION_GRACE_PERIOD` and determines what happens to running worker tasks during shutdown.
Set the termination grace period long enough for tasks to exit cleanly:
```yaml
kestra:
server:
termination-grace-period: 5m
```
If the deployment regularly creates empty server instances, adjust how often purge runs:
```yaml
kestra:
server:
service:
purge:
retention: 7d
```
:::alert{type="warning"}
Keep the external process manager timeout longer than Kestra's own termination grace period. Otherwise Kubernetes, Docker, or systemd can kill the process before graceful shutdown finishes.
:::
## Regex timeout
Kestra protects worker threads from ReDoS (catastrophic backtracking) by enforcing a timeout on all regex operations. This applies to [Pebble expression filters](../../expressions/index.mdx) (`regexMatch`, `regexReplace`, `regexExtract`, `replace` with `regexp=true`) and to `validator` patterns on `STRING` and `SECRET` inputs. When a pattern exceeds the limit, the task fails immediately with a timeout error rather than hanging indefinitely.
The default timeout is **10 seconds**. To change it, set `kestra.regex.timeout` in your configuration:
```yaml
kestra:
regex:
timeout: 30s
```
Accepts ISO 8601 duration strings (e.g., `PT30S`, `PT1M`) or Micronaut shorthand (e.g., `5s`, `1m`).
:::alert{type="info"}
The timeout is set once at startup and cannot be changed at runtime without restarting the server.
:::
## Related docs
- Secrets manager concepts: [External Secrets Manager](../../07.enterprise/02.governance/secrets-manager/index.md)
- Enterprise auth and RBAC: [Authentication and Users](../../07.enterprise/03.auth/index.mdx)
- EE platform settings and advanced backends: [Enterprise and Advanced](../06.enterprise-and-advanced/index.md)
---
# Contribute to Kestra: Code, Docs, and Community
URL: https://kestra.io/docs/contribute-to-kestra
> Join the Kestra open-source community. Discover how to contribute to the codebase, improve documentation, and engage with other developers.
import ChildCard from "~/components/docs/ChildCard.astro"
Contribute to the Kestra open-source project.
## How to contribute to Kestra
---
# Community Guidelines: How to Participate in Kestra
URL: https://kestra.io/docs/contribute-to-kestra/community-guidelines
> Read the Kestra Community Guidelines. Learn how to participate respectfully, ask for help effectively, and foster a welcoming and inclusive environment.
The Kestra community is a welcoming and inclusive place for everyone.
## Participate in the Kestra Community
1. **Be respectful to the Kestra community**
1. Be respectful toward other members of the Slack community. Harassment will not be tolerated.
2. Assume positive intent.
2. **Make it easy to help you**
1. Share relevant flows (YAML), logs, and stack traces [formatted](https://slack.com/intl/en-gb/help/articles/202288908-Format-your-messages) in code blocks (Avoid using screenshots).
2. Share how you deployed Kestra:
1. Deployment method (Standalone, Docker, Kubernetes, etc.)
2. Kestra version
3. Operating system and version
3. **Use relevant channels**
1. Avoid posting the same question in multiple channels.
4. **Don’t spam** — while we’ll do our best to help you, there is no guaranteed timeline to answer your question. If you need support with SLA guarantees, [reach out to us](/demo).
If you have questions, feel free to ask in our [Slack community](/slack)
---
# Contribute to the Kestra Codebase: Issues and PRs
URL: https://kestra.io/docs/contribute-to-kestra/contributing
> Guide to contributing to the Kestra codebase. Learn how to report bugs, request features, build plugins, and submit pull requests to help improve the platform.
Contribute to the Kestra open-source project.
## Contribute to the Kestra codebase
You can contribute to Kestra in many ways, depending on your skills and interests. The issues with the label `good first issue` are a great place to start and get familiar with the codebase. Check out the current list of [good first issues](https://github.com/search?q=org%3Akestra-io+label%3A%22good+first+issue%22+is%3Aopen&type=issues) and start contributing.
## Build a plugin
Check out our [Plugin Developer Guide](../../plugin-developer-guide/index.mdx) for instructions on how to build a new plugin.
## Contribute to the documentation
To contribute to the documentation, fork the [docs repository](https://github.com/kestra-io/docs/fork) and create a pull request with your changes.
Check out the [Contribute to Kestra Documentation page](../04.docs-contributor-guide/index.mdx) for more information about building the documentation site locally, how we write the documentation, and contributing to the product and plugin documentation.
## Write a blog post
You can contribute an article about how you use Kestra to our [blog](/blogs). Email [hello@kestra.io](mailto:hello@kestra.io) to start the collaboration. If you wrote a post mentioning Kestra on your personal blog, we'd be happy to feature it in our community section.
## Other ways to show support
- Star Kestra on [GitHub](https://github.com/kestra-io/kestra).
- Follow us on [X](https://twitter.com/kestra_io) and [LinkedIn](https://www.linkedin.com/company/kestra).
- Join the [Slack](/slack) community.
## Build Kestra locally
### Requirements
The following dependencies are required to build Kestra locally:
- JDK 25 (runtime) with source/target set to Java 21
- Node 14+ and npm
- Docker & Docker Compose
- an IDE (Intellij IDEA, Eclipse or VS Code)
To start contributing:
- [Fork](https://github.com/kestra-io/kestra/fork) the repository
- Clone the fork on your workstation:
```shell
git clone git@github.com:{YOUR_USERNAME}/kestra.git
cd kestra
```
### Backend development
The backend is built using [Micronaut](https://micronaut.io).
Open the cloned repository in your favorite IDE. In many IDEs, Gradle build will be detected and all dependencies will be downloaded.
You can also build it from a terminal using `./gradlew build`. The Gradle wrapper will automatically download the correct Gradle version to use.
- Set your IDE language level to **Java 21** while using the **JDK 25** toolchain; builds are compiled with `--release 21`.
- You may need to enable Java annotation processors since we use it a lot.
- The main class is `io.kestra.cli.App` from module `kestra.cli.main`.
- Pass as program arguments the server you want to develop, for example `server standalone` starts a standalone Kestra server.
- The Intellij Idea configuration can be found in screenshot below:

- `MICRONAUT_ENVIRONMENTS`: can be set as any string and will load a custom configuration file in `cli/src/main/resources/application-{env}.yml`
- `KESTRA_PLUGINS_PATH`: is the path where you save plugins as Jar and is loaded during the startup process
- If you encounter **JavaScript memory heap out** error during startup, configure `NODE_OPTIONS` environment variable with some large value.
- Example `NODE_OPTIONS: --max-old-space-size=4096` or `NODE_OPTIONS: --max-old-space-size=8192` 
- You can also use the gradle task `./gradlew runLocal` that runs a standalone server with `MICRONAUT_ENVIRONMENTS=override` and plugins path `local/plugins`
- The server start by default on port 8080 and is reachable on `http://localhost:8080`.
If you want to launch all tests, you need Python and some packages installed on your machine. On Ubuntu, you can install them with the following command:
```shell
sudo apt install python3 pip python3-venv
python3 -m pip install virtualenv
```
### Frontend development
All frontend code is located in the `/ui` folder.
The front-end uses [Vue.js](https://vuejs.org/). Deep knowledge of Vue.js is not required to contribute.
To run Kestra's frontend in development mode, you will need Node.js version `22.12.0`.
The repository has a `.nvmrc` file.
#### Initial setup
```shell
npm install
```
#### Run the frontend
```shell
npm run dev
```
This will start a local server on port `5173`.
You will need to open the Kestra UI in a browser at http://localhost:5173
#### Open Storybook
You can also run the [Storybook](https://storybook.js.org/) to view the components in isolation.
```shell
npm run storybook
```
This will start a local server on port `6006` and open the Storybook in your default browser at http://localhost:6006.
You can also run all tests in the command line without opening a browser:
```shell
npm run test:unit
```
Even better, you can run one `test` file or `stories` file in isolation by specifying part of its name or path in the command
```shell
npm run test:unit BarChart
```
### Set up the configuration to connect to the backend
Now that you can run the frontend, if opened, you will see a loading screen running forever.
It waits for a backend to answer.
To set it up:
- To avoid CORS restrictions when using the local development npm server, you need to configure the backend to allow the http://localhost:5173 origin in `cli/src/main/resources/application-override.yml` using the following addition to your [Observability and Networking configuration](../../configuration/03.observability-and-networking/index.md) YAML definition:
```yaml
micronaut:
server:
cors:
enabled: true
configurations:
all:
allowedOrigins:
- http://localhost:5173
```
Then, you can run the backend by running the gradle task.
```shell
MICRONAUT_ENVIRONMENTS=override ./gradlew runLocal server standalone
```
This will start a local server on port 8080, accessible at `http://localhost:8080`.
### Set up Kestra frontend without building the backend from the source code
If you want to work on the frontend without having to install Java and everything to run the Kestra Application, you can start a Kestra [Docker container](https://docs.docker.com/engine/install/) and connect the frontend to it.
To do so, you can first use the following [Docker Compose file](https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml).
Save it as `docker-compose.yml` in a separate directory from the Git repository and run the following command in this new directory:
```shell
docker compose up
```
This starts Kestra running with PostgreSQL as the database. You can change the port or other configurations by updating the `docker-compose.yml` file.
Finally, install the dependencies with `npm install`, and serve the UI with hot reload at http://localhost:5173 using the command: `npm run dev`.
## Kestra devcontainer
Thanks to the Kestra community, if you are using VSCode, you can start development on either the frontend or backend with a bootstrapped Docker container without the need to manually set up the environment.
Check out the [README](https://github.com/kestra-io/kestra/tree/develop/.devcontainer) for set-up instructions and the associated [Dockerfile](https://github.com/kestra-io/kestra/blob/develop/.devcontainer/Dockerfile) in the repository to get started.
## Code of conduct
This project and everyone participating in it is governed by the
[Kestra Code of Conduct](https://github.com/kestra-io/kestra/blob/develop/.github/CODE_OF_CONDUCT.md).
By participating, you are expected to uphold this code. Please report unacceptable behavior to [hello@kestra.io](mailto:hello@kestra.io).
### Legal notice
> When contributing to this project, you must agree that you have authored 100% of the content, that you have the necessary rights to the content and that the content you contribute may be provided under the project license.
### Submit issues
To submit feature requests or report bugs, please open an [issue](https://github.com/kestra-io/kestra/issues) on GitHub.
### Reporting bugs
Bug reports make Kestra better for everyone. We provide a preconfigured template for bugs to make it very clear what information we need.
Before reporting a bug, please search for your issue in our [already reported bugs](https://github.com/kestra-io/kestra/issues?q=is%3Aissue+is%3Aopen+label%3Abug) to avoid raising a duplicate.
### Reporting security issues
Please do not create a public GitHub issue. If you've found a security issue, please email us directly at [security@kestra.io](mailto:security@kestra.io) instead of raising an issue.
### Requesting new features
Use our issue templates when opening new issues. It contains a few essential questions that help us understand the problem you are looking to solve.
To see what has already been proposed by the community, you can refer to our [current issues board](https://github.com/kestra-io/kestra/issues?q=is%3Aissue+is%3Aopen+label%3Aenhancement).
---
# Kestra Docs Contributor Guide: Writer's Reference
URL: https://kestra.io/docs/contribute-to-kestra/docs-contributor-guide
> Help improve Kestra's documentation. This writer's guide covers local build setup, front matter conventions, and best practices for contributing to docs.
import ChildCard from "~/components/docs/ChildCard.astro"
Contribute to the Kestra Documentation.
To contribute to the documentation, fork the [docs repository](https://github.com/kestra-io/docs/fork) and create a pull request with your changes.
## Build the documentation locally
The following dependencies are required to build Kestra docs locally:
- Node 26+ and npm
- An IDE (such as VS Code, IntelliJ, etc.)
To start contributing:
- [Fork](https://github.com/kestra-io/docs/fork) the repository
- Clone the fork on your workstation:
```shell
git clone git@github.com:{YOUR_USERNAME}/docs.git
cd docs
```
Use the following commands to serve the docs locally:
```shell
## install dependencies
npm install
## serve with hot reload at localhost:3001
npm run dev
## to generate static pages
npm run generate
## making a production build
npm run build
```
In addition to contributing content and understanding the overall structure of the documentation, it's important to become familiar with the custom Markdown components and patterns used throughout the Kestra Docs. For those contributing to the Kestra Plugin documentation, a basic understanding of plugin structure and Java syntax for doc strings is required.
This guide is designed to help external contributors get up to speed with the tools, conventions, and components you'll encounter when contributing to Kestra's documentation.
The documentation is structured on multiple levels. The top level is an index page such as "Getting Started," "Workflow Components," and "Cloud & Enterprise Edition." This acts as a landing page for all content that falls under those high-level categories. To serve a visitor everything within that topic, we use a `ChildCard` component on the index page. This component is built from the `ChildCard.vue` file in the `components/content` directory.
The index file's markdown looks like this:
```markdown
---
title: "Install Kestra: Docker, Kubernetes, VM, and JAR"
sidebarTitle: Installation Guide
icon: /src/contents/docs/icons/installation.svg
---
import ChildCard from "~/components/docs/ChildCard.astro"
```
And the page displays the following with all child topics listed as cards:

Note that when writing a standalone documentation page, the first sentence appears in the ChildCard view to introduce the topic. In the above example for the [Kubernetes installation page](../../02.installation/index.mdx) this sentence is visible:
```markdown
Deploy Kestra on Kubernetes using the official Helm chart, scalable for production with PostgreSQL and object storage.
```
Ideally, keep this first sentence as clear and concise as possible to not clutter the view on the card.
### Front matter
Each documentation page is expected to include several key front matter properties. We briefly mentioned one of them, **icon**, in the last section. For example, take our [Apps](../../07.enterprise/04.scalability/apps/index.md) page. This is the front matter specified on the markdown page:
```markdown
---
title: Apps in Kestra Enterprise – Build Frontends for Flows
description: Build custom Apps with Kestra. Create user-facing interfaces for workflows, enabling forms, approvals, and interactive data applications.
sidebarTitle: Apps
icon: /src/contents/docs/icons/admin.svg
editions: ["EE", "Cloud"]
version: ">= 0.20.0"
docId: apps
---
```
And this is the resulting view:

Each property is described below.
#### title
`title` is the SEO title of the page. It appears in browser tabs, search results, and social previews. Write it to be descriptive and keyword-rich. For feature or concept pages this is typically the feature name; for how-to guides, make it clear about both the purpose and the tool involved (e.g., [Access Files on your Local Machine in Kestra](../../15.how-to-guides/access-local-files/index.md)).
Use Title Case for `title` and `sidebarTitle`.
#### h1
`h1` controls the heading displayed at the top of the page. It is separate from `title` so the on-page heading can be written for readability while `title` is optimised for search. If `h1` is omitted, `title` is used as the heading. Most pages define both:
```markdown
---
title: "Handle Errors in Kestra: Retries and Alerts"
h1: Build Resilient Workflows with Retries, Alerts & Failure Handling
---
```
#### description
`description` is the meta description shown in search results and link previews. Write one to two sentences (roughly 150–160 characters) summarising the page's content. Every page should include one.
#### sidebarTitle
`sidebarTitle` controls the label shown in the left-hand navigation. Keep it short — typically just the feature or topic name. Use Title Case.
#### icon
Icons are SVG files that are used to identify a certain tool being used or a general concept. They appear at the top of all documentation pages and in the ChildCard of the page. For example, this [Neon with Kestra guide](../../15.how-to-guides/neon/index.md) has the following properties:
```markdown
---
title: Connect Neon Database to Kestra
icon: /src/contents/docs/icons/neon.svg
stage: Intermediate
topics:
- Integrations
---
```
And appears on the site as follows:

The icon lives in the `public/docs/icons` folder path and is specified as [Neon](https://neon.tech/home), so the correct logo shows for the tool. General icons, such as `api.svg` or `installation.svg`, are also available in the folder. If you contribute a guide incorporating a tool without an existing icon, place the appropriate SVG file in this folder and reference it in the front matter.
#### topics & stage
Our **How-To Guides** require a couple of extra front-matter properties to provide clarity to the site visitor about the guide's topic and level: `topics` and `stage`. Using the same example as above, you can see that the properties are set as `stage: Intermediate` and `topics: Integrations`.
```markdown
---
title: Connect Neon Database to Kestra
icon: /src/contents/docs/icons/neon.svg
stage: Intermediate
topics:
- Integrations
---
```
These properties are `const` variables set in the `GuidesChildCard.vue` file of the repository. They have a set list to choose from when classifying a guide. For example, `stage` can be "Getting Started," "Intermediate," or "Advanced." `topics` can be a multitude of different concepts such as "Scripting," "Kestra Concepts," "Best Practices," and more. If your guide doesn’t fit into any of the existing topics, feel free to suggest a new one in a pull request.
#### editions
Kestra has three editions: Open Source, Enterprise, and Cloud. A feature or guide may be relevant only to one, two, or all editions, so we have a front-matter property to specify that right at the top of a page for the reader. For example, depending on the Kestra edition, there are different pages relevant to handling secrets. We have a [Kubernetes Secrets How-to Guide](../../15.how-to-guides/kubernetes-secrets/index.md) where we set the edition as `OSS` in the front matter:
```markdown
---
title: Set Up Secrets from a Helm Chart
icon: /src/contents/docs/icons/helm.svg
stage: Getting Started
topics:
- Kestra Concepts
- DevOps
editions: ["OSS"]
---
```
And we have a page for [Secrets](../../07.enterprise/02.governance/secrets-manager/index.md) that is specifically for **Enterprise & Cloud** users.
```markdown
---
title: Secrets
icon: /src/contents/docs/icons/admin.svg
editions: ["EE", "Cloud"]
docId: secrets
---
```
#### version
Like `editions`, some Kestra features are only available in specific Kestra versions and onwards. We use the `version` property in the front matter to identify this in the documentation. For example, [Worker Groups](../../07.enterprise/04.scalability/worker-group/index.md) are only available starting in Kestra version 0.10.0. This is specified as follows:
```markdown
---
title: Worker Group
icon: /src/contents/docs/icons/admin.svg
editions: ["EE"]
version: ">= 0.10.0"
---
```
#### docId
One of Kestra's major benefits is its in-app contextual docs. This means that when constructing flows in the platform, you can access the documentation in the same interface without having to navigate to the browser to check against our documentation. This is done through the `docId` front matter.
Kestra knows that you are working with Apps, and it can show you the relevant documentation without a task switch.

The same is true for all the main components of Kestra (e.g., Namespace, Flow, Blueprints, Plugins, etc.).
#### release
`release` is a front matter property only relevant for our [Migration Guides](../../11.migration-guide/index.mdx). These guides outline the need-to-know information for upgrading from one version of Kestra to another. This includes renaming a feature or "Before and After" examples of an action in Kestra. Example configuration looks like this:
```markdown
---
title: Restarting parent flow
icon: /src/contents/docs/icons/migration-guide.svg
release: 0.21.0
editions: ["OSS", "EE"]
---
```
### Writing style
For the complete editorial reference — page types, capitalization rules, procedures structure, anti-patterns, and a full review checklist — see the [Kestra Docs Style Guide](../../docs-style-guide.md).
The most common conventions to keep in mind:
#### Headings
Use sentence case for all body headings — capitalise only the first word and proper nouns. Use Title Case for `title` and `sidebarTitle` in front matter.
Avoid restating the page title as the first H2. The first heading should introduce the first distinct section, not repeat what the title already says.
#### Voice and tone
- Address the reader as "you" (second person).
- Use active voice and present tense for facts and product behavior.
- Be direct. Avoid filler phrases like "In this guide," "It's worth noting that," or "Simply."
- Do not use first-person plural ("we," "our"). Kestra docs address the reader, not the writing team.
### Customized text
We use several components to add customized text presentations in the documentation. To differentiate important information from average text, we use three different levels of alert types: "info," "success," and "warning."
:::alert{type="info"}
This is important to note.
:::
:::alert{type="success"}
Yippee, it worked.
:::
:::alert{type="warning"}
This is a warning, but it's fine.

:::
Use alerts sparingly. Reserve them for content that would cause failure or confusion if missed — a required prerequisite, a destructive side effect, or a non-obvious constraint. Avoid using them for general information that works just as well as a sentence in the body text.
Another helpful component we use is `:::collapse`. This tag keeps the documentation space-efficient and hides long examples or other information that does not need to be seen when scrolling the page. Still, the reader can open it up to reveal its content. This is particularly useful for flows that could otherwise take up a lot of space on a page or FAQ Answers that may not be relevant to every reader and can be selected as needed.
Use the following syntax with whatever should be collapsed within the colons and the title inline with `:::collapse`:
```markdown
:::collapse{title="Introduction to whatever is collapsed"}
Here is where the collapsed text goes.
:::
```
Here is a full example using a flow and subflow with a Loop task:
:::collapse{title="Full Flow Example"}
Subflow:
```yaml
id: subflow
namespace: company.team
inputs:
- id: item
type: STRING
tasks:
- id: process
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- echo "Processing {{ inputs.item }}"
- id: log
type: io.kestra.plugin.core.log.Log
message: "Processed: {{ inputs.item }}"
```
Below is a flow that uses the `Loop` task to iterate over a list of items and run the `subflow` for each one:
```yaml
id: each_parent
namespace: company.team
tasks:
- id: loop
type: io.kestra.plugin.core.flow.Loop
values:
- orders_jan.csv
- orders_feb.csv
- orders_mar.csv
tasks:
- id: process_file
type: io.kestra.plugin.core.flow.Subflow
namespace: company.team
flowId: subflow
wait: true
transmitFailed: true
inputs:
item: "{{ item.value }}"
```
:::
While a feature may be available after a specific Kestra version and indicated in the front matter, an additional function may be added in later versions that don't match the front matter. We use the ':::badge' component to indicate this in the documentation for only a particular page section. This component can be used at any point on the page rather than solely at the top.
The component has the following syntax, able to include both `version` and `editions` like the Front Matter:
```markdown
:::badge{version=">=0.15" editions="OSS,EE,Cloud"}
:::
```
:::badge{version=">=0.15" editions="OSS,EE,Cloud"}
:::
### Video container
In the documentation, we try to always have an accompanying video for the discussed feature. To ensure the YouTube video is embedded and displayed correctly and consistently on every page, we use a custom `video-container` div class. Add the div after the page's introductory sentence and before the main content.
```html
```
The `video-container` is maintained in the repository's `docs.scss` file. Refer back to the top of this page or check out the [Contributing Guide](../03.contributing/index.md) an embedded video.
### Code blocks
When including code blocks in the documentation, specify which language the example is written in. Typically, in the Kestra documentation, example flows are included to demonstrate a feature; they are defined as a `yaml` code block. For example, see the following flow in markdown:
The supported languages for code blocks are fully listed in the `useShiki.ts` file in the repository, and if you need something new added, you can make an addition there.
### How to use images
Images are a key part of the documentation. We couple images used on a page within the page directory. The Apps documentation markdown page and its associated images are contained in the same folder to keep assets together and to make them easy to find and add. Taking this guide as an example, an image we used earlier in the guide appears in the markdown as follows:
```markdown

```
The image has a clear title and is located in the Apps folder. For this guide, all images are placed in this folder path so the organization is clear and easily worked with by another contributor. This same practice is used for our blog and other parts of the website that are kept in the repository.
## Contribute to Kestra Plugin Documentation
Kestra Plugins each have their own documentation page on the website in [Plugins](/plugins). Each plugin also has in-app contextual documentation so that task and property definitions are easily usable while building flows. Plugin docs are maintained in separate repositories rather than the product documentation.
For example, if you want to contribute to the [OpenAI Plugin](/plugins/plugin-openai), you can find the documentation in the [OpenAI Plugin Repository](https://github.com/kestra-io/plugin-openai).
:::alert{type="info"}
All plugin repos are searchable from the central [Kestra GitHub](https://github.com/kestra-io). The name of the repository is in the URL of the plugin documentation page. For example, the OpenAI repo is called `plugin-openai`, which is in the URL path `https://kestra.io/plugins/plugin-openai/io.kestra.plugin.openai.chatcompletion`. Searching the tool's name should suffice, but this always works as a fallback.
:::
To contribute to a plugin's documentation, fork the repository. Once cloned, contributions are welcome to four key components of plugin task documentation: title, description, examples, and properties.
Continuing with [OpenAI](/plugins/plugin-openai), the tasks include [ChatCompletion](/plugins/plugin-openai/io.kestra.plugin.openai.chatcompletion) and [CreateImage](/plugins/plugin-openai/io.kestra.plugin.openai.createimage).

Each task is in the path `src/main/java/io/kestra/plugin/openai`. This will be similar to all other plugins (i.e., `src/main/java/io/kestra/plugin/`). To improve or add to the documentation, open the Java file for the task and edit the `@Schema`, `@Plugin`, and `@Example` doc strings.
:::alert{type="info"}
You do not need to be well-versed in Java to contribute to the plugin documentation. The doc strings are organized so that they are easy to work with, and we will review any contributions anyway, so have no fear. You can read more about how we instruct developers to document their plugins in the [Plugin Developer Guide](../../plugin-developer-guide/07.document/index.md).
:::
The plugin documentation will generally look like the following:
```java
@Schema(
title = "Given a prompt, get a response from an LLM using the OpenAI’s Chat Completions API.",
description = "For more information, refer to the [Chat Completions API docs](https://platform.openai.com/docs/guides/gpt/chat-completions-api)."
)
@Plugin(
examples = {
@Example(
full = true,
title = "Based on a prompt input, generate a completion response and pass it to a downstream task.",
code = """
id: openai
namespace: company.team
inputs:
- id: prompt
type: STRING
defaults: What is data orchestration?
tasks:
- id: completion
type: io.kestra.plugin.openai.ChatCompletion
apiKey: "yourOpenAIapiKey"
model: gpt-4o
prompt: "{{ inputs.prompt }}"
- id: response
type: io.kestra.plugin.core.debug.Return
format: {{ outputs.completion.choices[0].message.content }}"
"""
),
```
The key properties to consider are:
- `title`: A concise single sentence describing the task's objective that is displayed in the Kestra in-app contextual docs.
- `description`: Additional information such as links to the external tool's documentation or best practices for using the task.
- `examples`: Flow examples that demonstrate the task in use. Best if it is a logical use case utilizing multiple Kestra features (e.g., [Triggers](../../05.workflow-components/07.triggers/index.mdx), [Inputs](../../05.workflow-components/05.inputs/index.md), [Outputs](../../05.workflow-components/06.outputs/index.md), etc.).
Similarly to the main plugin attributes, the properties are documented with a `title` and a `description`. For example, the [OpenAI ChatCompletion properties](/plugins/plugin-openai/io.kestra.plugin.openai.chatcompletion#properties-body):
```java
public class ChatCompletion extends AbstractTask implements RunnableTask {
@Schema(
title = "A list of messages comprising the conversation so far.",
description = "Required if prompt is not set."
)
private Property> messages;
@Schema(
title = "The function call(s) the API can use when generating completions."
)
private Property> functions;
@Schema(
title = "The name of the function OpenAI should generate a call for.",
description = "Enter a specific function name, or 'auto' to let the model decide. The default is auto."
)
private Property functionCall;
@Schema(
title = "The prompt(s) to generate completions for. By default, this prompt will be sent as a `user` role.",
description = "If not provided, make sure to set the `messages` property."
)
```
To improve or add titles, descriptions, or examples, create a pull request or issue on the specific plugin repository.
## Contribute to Kestra Blueprints
The official Kestra Blueprints library can be found under [kestra.io/blueprints](/blueprints).
Blueprints are a curated, organized, and searchable catalog of ready-to-use examples designed to help you kickstart your workflow.
Each Blueprint combines code and documentation and can be assigned several tags for organization and discoverability.
To contribute a Blueprint or modify an existing one, clone the [Blueprints repository](https://github.com/kestra-io/blueprints). Within the repository, there are blueprints for [Apps](https://github.com/kestra-io/blueprints/tree/main/apps), [Dashboards](https://github.com/kestra-io/blueprints/tree/main/dashboards), and [Flows](https://github.com/kestra-io/blueprints/tree/main/flows).
All Blueprints are `yaml` files composed of the example Flow, App, or Dashboard and an `extend` property that specifies attributes such as `title` and `description` to propagate onto the website. For example, this [Getting Started with Kestra – a Data Engineering Pipeline](/blueprints/data-engineering-pipeline) has the following `extend` property:
```yaml
extend:
title: Getting started with Kestra — a Data Engineering Pipeline example
description: |
This flow is a simple example of a Kestra flow used for a data engineering
use case. It downloads a JSON file, filters the data, and calculates the
average price per brand.
The flow has three tasks:
1. The first task downloads a JSON file.
2. The second task filters the data and writes it to a new JSON file.
3. The third task reads the filtered data, calculates the average price per
brand using DuckDB, and stores the result as a Kestra output which can be
previewed and downloaded from the UI.
tags:
- Getting Started
- API
- Python
- SQL
ee: false
demo: true
metaTitle: Getting Started - Data Engineering Pipeline
metaDescription: This flow represents a data engineering use case. It downloads
a JSON file, filters the data in Python, and calculates the KPIs in SQL
using DuckDB.
```
Check out the [full file](https://github.com/kestra-io/blueprints/blob/main/flows/data-engineering-pipeline.yaml) to see the Flow's YAML.
For the Blueprint to be easily searchable, it is essential to include the appropriate `tags`. A complete list of tags is available on the [Blueprints homepage](/blueprints).
With the proper YAML and extension, the Flow's topology will display interactively on the Blueprint page along with a **Copy source code** button and task icons.

To suggest new blueprints or improve existing ones, create a pull request or issue on the [Blueprints repository](http://github.com/kestra-io/blueprints).
---
# Kestra Docs Style Guide
URL: https://kestra.io/docs/docs-style-guide
> Writing and editorial standards for Kestra documentation contributors.
Writing and editorial standards for the Kestra docs. Use it when writing new pages, reviewing pull requests, or editing existing content.
## Core principles
- **Be clear before being clever.** Optimize for task completion, not elegance.
- **Explain the why, not just the how.** Readers who understand the reason make better decisions when the instructions don't perfectly match their situation.
- **Write for a global technical audience.** Avoid idioms, culturally specific references, and assumed familiarity.
- **Prefer consistency over novelty.** Match existing Kestra terminology and page patterns before introducing new phrasing.
- **Keep the page focused.** One dominant page type per page. Mixed-type pages are harder to write and harder to use.
## Page types
Every docs page should have a single dominant type. Mixed-type pages are harder to write and harder to use.
| Type | Purpose | Examples |
|---|---|---|
| **Landing page** | Orient readers and route them to sub-pages | Section index pages |
| **Concept** | Explain what something is and why it matters | Architecture, Multi-tenancy |
| **How-to guide** | Drive one specific task to completion | How to set up webhooks |
| **Reference** | Help readers look up facts quickly | API reference, CLI commands |
| **Migration guide** | Explain what changed between versions and how to adapt | pluginDefaults Removed |
If you find yourself writing a concept explanation inside a how-to, extract it. If you find a how-to inside a reference page, move it.
## Page openings
The first sentence is the most important on the page. It appears in ChildCards, search results, and link previews.
**Requirements:**
- Explains what the page covers in plain language
- Stands alone without context from the title or URL
- Stays under 30 words
- Uses no filler: avoid "In this page," "This document will," "Welcome to," or "This guide covers"
**Before:**
> In this guide, we will walk through how you can configure webhooks to trigger Kestra flows.
**After:**
> Configure a Webhook trigger to start a flow in response to an HTTP POST request.
## Voice and tone
- **Second person** — address the reader as "you." Never use "we," "our," or first-person plural; Kestra docs address the reader, not the writing team.
- **Active voice** — prefer "Kestra executes the task" over "the task is executed by Kestra."
- **Present tense** — describe product behavior as it is now, not as it will be or was.
- **Direct** — cut filler phrases. "Note that," "It's worth mentioning," "Simply," and "Just" add no meaning.
- **Calm and technical** — avoid enthusiasm markers ("powerful," "amazing," "easy"), anthropomorphism ("Kestra knows," "the task understands"), and marketing language.
## Headings
- Use **sentence case** for all body headings — capitalise only the first word and proper nouns.
- Use **Title Case** for `title` and `sidebarTitle` in frontmatter.
- Do not restate the page title as the first H2. The first heading should introduce the first distinct section.
- Keep heading depth shallow. Prefer H2 and H3; use H4 only when genuinely necessary.
- Make headings descriptive. "Configuration" is vague; "Configure internal storage" is specific.
**Before:**
> ## Introduction to webhooks
> ### What is a webhook?
> #### Example of webhook usage
**After:**
> ## How webhooks work
> ### Configure a webhook trigger
## Capitalization
Kestra has specific capitalization conventions. The key principle: capitalize a term when referring to the named product entity; use lowercase for the generic concept.
| Term | Generic (lowercase) | Product entity (capitalized) |
|---|---|---|
| flow | "orchestrate your flows" | "open the Flow editor" |
| task | "add a task to the flow" | — (always generic) |
| namespace | "organize flows by namespace" | — (always generic) |
| execution | "view past executions" | "open the Executions tab" |
| trigger | "add a trigger" | "a Webhook trigger," "a Flow trigger" |
**Always match visible UI labels exactly**, including capitalization and punctuation. If the UI says "Worker Queues," write "Worker Queues," not "worker queues."
Kestra feature names that are always capitalized:
- Policies, Worker Queues, Apps, Blueprints, AI Copilot
- Internal Storage (when referring to the storage layer)
- RBAC (always acronym)
## Procedures
A procedure drives one task to completion. Structure it as:
1. **Goal sentence** — one line describing what the reader will accomplish.
2. **Prerequisites** — what the reader needs before starting. Use a dedicated prerequisites section, not buried alerts or body text.
3. **Numbered steps** — each step is one action, written as an imperative verb ("Click **Create**," "Set the `region` property").
4. **Verification** — how the reader confirms it worked.
5. **Next steps** — where to go from here, if relevant.
Avoid collapsing multiple actions into one step. "Click Create, fill in the form, and save" is three steps.
## Code and examples
- Introduce every code block. Tell the reader what they are about to see before they see it.
- Specify the language on every fenced code block: ` ```yaml `, ` ```bash `, ` ```python `.
- Prefer complete, runnable examples in how-to guides. If an example is intentionally incomplete, say so explicitly.
- Explain the result or the key decision after the block when it isn't obvious.
- Use `company.team` as the namespace in all flow examples.
**Before:**
> Here is an example:
> ```yaml
> kestra:
> storage:
> type: s3
> ```
**After:**
> Configure internal storage to use AWS S3:
> ```yaml
> kestra:
> storage:
> type: s3
> s3:
> bucket: "kestra-internal-storage"
> region: "us-east-1"
> ```
> Replace `bucket` and `region` with your own values. See [internal storage configuration](./configuration/02.runtime-and-storage/index.md) for all available options.
## Alerts
Use alerts sparingly. Reserve them for content that would cause failure or confusion if missed.
| Alert type | When to use |
|---|---|
| `warning` | Destructive, irreversible, or data-loss actions |
| `info` | Prerequisites, edition requirements, or non-obvious constraints that affect correctness |
| `success` | Rarely needed; only for outcomes that are genuinely non-obvious |
**Do not use alerts for:**
- General information that works equally well as a sentence in body text
- Tips or suggestions that are optional
- Two alerts in a row — consolidate or convert one to prose
## Links and media
**Links:**
- Use descriptive link text. "See [Policies](./policies)" not "click [here](./policies)."
- Link to the source of truth rather than duplicating content across pages.
- Use relative links between docs pages.
**Screenshots:**
- Include a screenshot only when it materially helps — a complex UI interaction, a before/after comparison, or a result that is hard to describe in words.
- Write descriptive alt text that describes what is shown, not just what the image is called.
- Keep screenshots current. An outdated screenshot is worse than no screenshot.
**Videos:**
- Embed videos after the page's opening sentence, before the main content, using the `video-container` div class.
- Do not use videos as a substitute for written content — readers on slow connections or screen readers cannot access them.
## What to remove
Cut the following on sight:
- **Promotional language** — "powerful," "flexible," "seamless," "robust," "best-in-class"
- **Vague claims** — "easy to use," "quick and simple," "at any scale" without qualification
- **Throat-clearing** — "In this guide we will," "Before we get started," "It is important to note that"
- **Filler adverbs** — "simply," "just," "easily," "quickly"
- **Decorative screenshots** — images that show the UI but add no information the text doesn't already convey
- **Unexplained code blocks** — a block with no introduction or follow-up explanation
- **Redundant alerts** — information already in the body text repeated in an alert box
## Version references
Docs describe the current supported behavior by default. Avoid phrases like "as of version X.Y," "starting in version X.Y," or "currently" in normal docs pages — these go stale and create confusion about what applies to the reader's version.
**Exceptions:**
- Migration guides — always version-scoped by design
- A `version` frontmatter property on a page, which renders a version badge automatically
- An inline `:::badge` component when a single section of a page applies only to certain versions
Outside these cases, if version scoping is necessary, prefer structured metadata over prose mentions.
## Review checklist
Before submitting or approving a docs change, verify:
1. The page type is clear and consistent throughout.
2. The first sentence stands alone as a summary.
3. All body headings use sentence case.
4. Capitalization matches Kestra feature names and visible UI labels.
5. Prose is second person, active voice, and present tense.
6. No filler phrases, hype, or anthropomorphism.
7. Terminology is consistent within the page and matches the rest of the docs.
8. Every code block is introduced and (where non-obvious) followed by an explanation.
9. Links use descriptive text and point to the source of truth.
10. Screenshots are current and earn their place.
11. Alerts are used only for genuinely warning- or constraint-worthy content.
12. The reader can verify success or find the next relevant path.
---
# Cloud & Enterprise Edition: Features and Setup
URL: https://kestra.io/docs/enterprise
> Kestra Enterprise & Cloud. Explore advanced features like SSO, RBAC, Multi-tenancy, and High Availability for enterprise-grade orchestration.
import ChildCard from "~/components/docs/ChildCard.astro"
How to configure Kestra Enterprise Edition and Kestra Cloud.
## Enterprise and Cloud Overview
[Enterprise Edition](/enterprise) is a self-hosted version of Kestra deployed to your private infrastructure. It offers security and governance features including Multi-tenancy, Authentication, SSO, RBAC, Namespace-level management, distributed Worker Groups, Worker isolation, Secrets Manager integrations, Audit Logs, and more.
[Kestra Cloud](/cloud) is a fully managed version of Kestra Enterprise Edition, hosted and maintained by the Kestra team. It provides most of the features of the Enterprise Edition, plus the additional benefits of automatic updates, backups, and infrastructure monitoring. New to Kestra Cloud? See the [Cloud getting started guide](./07.cloud-getting-started/index.md).
## Key differences between Kestra Enterprise and Kestra Cloud
While Kestra Cloud is fully managed, it differs from Kestra Enterprise in several important ways, primarily around infrastructure control, customization, and direct access to backend components.
| Feature / Area | Kestra Cloud | Kestra Enterprise Edition |
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| **Infrastructure Control** | Fully managed by Kestra for simplicity and reliability | Full control and customization |
| **Backend Technology** | PostgreSQL JDBC only | Customizable: PostgreSQL, MySQL, Kafka, Redis, AMQP, GCP Pub/Sub, H2 (testing) |
| **Workers** | Managed worker pools sized for stability | Remote [Worker Groups](/docs/enterprise/scalability/worker-group), autoscaling |
| **Custom Internal Storage & Secrets** | No instance-level control, **tenant/namespace-level only** backends | Fully customizable backends |
| **Network Configuration** | Secure access over public Internet | Private networking (self-hosted, VPC peering, etc.) |
| **Backup Access** | Automatic backups handled by Kestra | Customer-controlled backups |
| **Plugins** | Curated plugin environment | Full plugin customization |
| **Identity Providers (IdP)** | Built-in Google, Microsoft, or Basic Authentication | Custom SSO/SCIM supported |
| **Log Retention** | Automatic retention protocol managed by Kestra | Unlimited (based on customer setup); route logs to a separate JDBC or Elasticsearch store with the [External Log Data Store](../10.administrator-guide/log-data-store/index.md) |
| **Deployment Regions** | US & EU (Belgium) on GCP | Any cloud, any region |
| **Task Runners** | Compatible with most, [Process Task Runner](/docs/task-runners/types/process-task-runner) excluded | Compatible with all task runners |
This section describes those features in detail and explains how to configure them.
If you're interested to learn more, check the [Open-Source and Enterprise Edition comparison](/docs/oss-vs-paid), explore our [Pricing](/pricing), and [get in touch](/demo) to discuss your requirements.
---
# Auth & Users in Kestra Enterprise: RBAC, SSO
URL: https://kestra.io/docs/enterprise/auth
> Manage Authentication and Users in Kestra Enterprise. Overview of RBAC, SSO, API tokens, and service accounts for secure access control.
import ChildCard from "~/components/docs/ChildCard.astro"
Features for managing Authentication, Role-based Access Control, Users, and more in Kestra.
## Authentication and users – RBAC and access
---
# Enterprise API in Kestra: Endpoints and Auth
URL: https://kestra.io/docs/enterprise/auth/api
> Interact with the Kestra Enterprise API. Learn about available endpoints, authentication methods, and how to programmatically manage your Kestra instance.
The Kestra Enterprise API exposes endpoints for managing executions, flows, tenants, and more — all authenticated with API tokens.
## Authentication
Authenticate with an [API token](../api-tokens/index.md). Pass the token in the `Authorization` header:
```bash
curl -X POST http://localhost:8080/api/v1/executions/company.team/hello_world \
-H "Authorization: Bearer YOUR_API_TOKEN"
```
## Browse the API reference
For a full list of available endpoints, see the [Enterprise Edition API Reference](../../../api-reference/01.enterprise/index.mdx).
---
# API Tokens in Kestra: Manage Programmatic Access
URL: https://kestra.io/docs/enterprise/auth/api-tokens
> Manage programmatic access with API Tokens in Kestra. Create and control tokens for users and service accounts to securely interact with the Kestra API.
API tokens grant programmatic access to the Kestra API for users and [service accounts](../service-accounts/index.md).
## Where you can use API tokens
Pass a token as a `Bearer` header to authenticate any Kestra API call — from a CI/CD pipeline, a custom application, or any of the following:
- [GitHub Actions](https://github.com/kestra-io/github-actions)
- [Terraform Provider](https://registry.terraform.io/providers/kestra-io/kestra/latest/docs)
- [Kestra Server CLI](../../../kestra-cli/kestra-server/index.md)
- [kestractl](../../../kestra-cli/kestractl/index.md)
- [Kestra API](../api/index.md)
## How to create a user API token
Two ways to reach the token creation form:
- Click your user avatar at the bottom-left and select **Create API Token**.
- Go to **Settings → API Tokens** and click **+ Create API Token**.
Fill in a **Name**, optional **Description**, and **Max age** (leave blank for a non-expiring token). Enable **Extended** to automatically reset the expiry each time the token is used. Click **Generate**, then copy the token immediately — it is shown only once.
## How to create a service account API token
Go to **IAM → Service Accounts**, open the service account, click the **API Tokens** tab, and click **Create**. The form fields and expiry options are the same as for user tokens.
## How to use an API token in an API request
To authenticate your custom API calls, pass a `Bearer` token to the request's `Authorization` header. The following example triggers a flow execution via the Kestra API:
```bash
curl -X POST http://localhost:8080/api/v1/executions/dev/hello-world \
-H "Authorization: Bearer YOUR_API_TOKEN"
```
---
# Authentication in Kestra Enterprise: OIDC Setup
URL: https://kestra.io/docs/enterprise/auth/authentication
> Configure Authentication in Kestra. Set up Basic Auth and OpenID Connect (OIDC) for secure user login and access management.
Kestra supports two authentication methods: Basic Auth (enabled by default) and OpenID Connect (OIDC).
Kestra uses the default encryption key for JWT sessions. Generate a secret of at least 256 bits and add it to your [Kestra Security and Secrets configuration](../../../configuration/05.security-and-secrets/index.md):
```yaml
kestra:
encryption:
secret-key: your-256-bits-secret
```
This secret must be the same across all your webserver instances and will be used to sign the JWT cookie and encode the refresh token.
To use separate keys for the signature and refresh token:
```yaml
micronaut:
security:
token:
jwt:
generator:
refresh-token:
secret: refresh-token-256-bits-secret
signatures:
secret:
generator:
secret: signature-256-bits-secret
```
:::alert{type="info"}
You can change the JWT cookie behavior using [Micronaut Cookie Token Reader](https://micronaut-projects.github.io/micronaut-security/latest/guide/#cookieToken) configuration. For example, define the cookie's maximum lifetime with `micronaut.security.token.cookie.cookie-max-age: P2D`.
:::
## Basic authentication
The default installation comes with no users defined. To create an administrator account, use the following CLI command:
```bash
./kestra auths users create --admin --username= --password= --tenant=
```
Without multi-tenancy, omit the `--tenant` parameter.
:::alert{type="info"}
Multi-tenancy is enabled by default. Include the `--tenant` parameter.
:::
## Single sign-on (SSO)
Single Sign-On (SSO) is an authentication process that allows users to access multiple applications with one set of login credentials (e.g., Sign in with Google). Kestra supports SSO using the OpenID Connect (OIDC) protocol, which is a simple identity layer built on top of the OAuth 2.0 protocol.
Enable OIDC in your Micronaut configuration:
```yaml
micronaut:
security:
oauth2:
enabled: true
clients:
google:
client-id: "{{ clientId }}"
client-secret: "{{ clientSecret }}"
openid:
issuer: "{{ issuerUrl }}"
```
See the [Micronaut OIDC configuration guide](https://micronaut-projects.github.io/micronaut-security/latest/guide/#openid-configuration) for full details. See [Single Sign-On](../sso/index.md) to configure SSO with Google, Microsoft, and other providers.
---
# Credentials in Kestra: Authenticate External Systems
URL: https://kestra.io/docs/enterprise/auth/credentials
> Authenticate to external systems securely with Kestra Credentials. Store and manage server-to-server auth tokens for use across flows and namespaces.
Credentials let you configure server-to-server authentication once and reuse it across tasks. Instead of embedding token minting and refresh logic in each plugin, Kestra handles this at runtime and you reference the current token with a simple expression.
Many APIs are moving away from long-lived static API keys toward **short-lived tokens** (e.g. OAuth 2.0), which improves security and simplifies rotation and revocation.
For simple static values (API keys, usernames/passwords), use [Secrets](../../../06.concepts/04.secret/index.md) directly.
Sensitive material used by a credential (client secrets, private keys, certificates) is referenced via [Secrets](../../../06.concepts/04.secret/index.md) so it can be stored in an external secret manager and never appears in plain text in the credential config.
## Availability and scope
Credentials can be accessed and created at:
- **Tenant level** (reusable across namespaces in that tenant)
- **Namespace level** (scoped to a single namespace)
During setup, Kestra lets you **test token retrieval** from the UI to ensure your configuration is correct.
## Use a credential in a flow
Use the `credential()` Pebble function to retrieve the **current access token** for a credential key.
```yaml
id: api_call
namespace: company.team
tasks:
- id: request
type: io.kestra.plugin.core.http.Request
uri: https://api.example.com/v1/ping
method: GET
auth:
type: BEARER
token: "{{ credential('my_oauth') }}"
```
`credential()` returns the access token only.
For non-sensitive configuration (e.g., hostnames, table names, feature flags), prefer [Variables](../../../05.workflow-components/04.variables/index.md).
## Credential types
Credentials cover common server-to-server authentication patterns, including:
- OAuth2 `client_credentials` (generic)
- OAuth2 JWT Bearer extension grant (`jwt_bearer`, RFC 7523)
- OAuth2 `private_key_jwt` (client authentication)
- GitHub App
Credentials can reference sensitive inputs via existing [Secrets](../../../06.concepts/04.secret/index.md) (e.g., client secrets, private keys, certificates), including secrets stored in an external or [read-only secrets manager](../../02.governance/secrets-manager/index.md).
## Example: Google service account with JWT Bearer
The following example shows how to use a Google Cloud service account with an OAuth2 JWT Bearer credential in Kestra.
### 1. Create a service account in Google Cloud
In Google Cloud:
1. Go to **IAM & Admin → Service Accounts**.
2. Create a new service account and grant it only the roles required for your use case.
3. Open the service account, go to **Keys**, and create a new **JSON** key.
4. Download the JSON key file.
From that JSON file, you will use:
- `client_email`
- `private_key`
- `private_key_id`
- `token_uri`
For more information, see the [Google service account guide](https://cloud.google.com/iam/docs/service-account-overview).
### 2. Create a secret for the private key
Store the private key from the downloaded JSON in a Kestra secret rather than embedding it directly in the credential.
For example, create a secret named `GCP_PRIVATE_KEY` with the value of the `private_key` field from the JSON file.
You can manage that secret from the Kestra UI or by using an external [Secrets Manager](../../02.governance/secrets-manager/index.md).
### 3. Create the credential in Kestra
In the Credentials UI, create a new credential with the following values:
- **Credential Type:** `OAUTH2`
- **Auth Config Type:** `JWT_BEARER`
- **Token Endpoint:** `https://oauth2.googleapis.com/token`
- **Issuer:** the `client_email` value from the JSON key
- **Subject:** use the service account email for a standard service account flow; for Google Workspace domain-wide delegation, use the delegated user instead
- **Private Key:** reference the `GCP_PRIVATE_KEY` secret
- **Key ID:** the `private_key_id` value from the JSON key
- **Algorithm:** `RS256`
- **Additional Claims:** add a `scope` claim containing the Google OAuth scopes required by the API, for example `https://www.googleapis.com/auth/cloud-platform.read-only`
For Google service accounts, the scope must be included in the JWT claims. If you need multiple scopes, provide them as a single space-delimited string in the `scope` claim, for example:
```plaintext
https://www.googleapis.com/auth/cloud-platform.read-only https://www.googleapis.com/auth/bigquery.readonly
```
:::alert{type="info"}
Use the **Test connection** action in the Credentials UI to confirm that Kestra can mint an access token before using the credential in a flow.
:::
### 4. Use the credential in a flow
Once the credential is saved, you can use it in a flow with the `credential()` Pebble function.
The example below calls the Google Cloud Resource Manager API and sends the access token as a Bearer token in the request header:
```yaml
id: google_api_with_credential
namespace: company.team
inputs:
- id: project_id
type: STRING
tasks:
- id: request
type: io.kestra.plugin.core.http.Request
method: GET
uri: "https://cloudresourcemanager.googleapis.com/v1/projects/{{ inputs.project_id }}"
options:
auth:
type: BEARER
token: "{{ credential('gcp-service-account') }}"
- id: log_result
type: io.kestra.plugin.core.log.Log
message: |
code={{ outputs.request.code }}
body={{ outputs.request.body }}
```
If the service account has the required permissions on the target project, the request should return `200` and the project metadata in the response body.
## Token lifecycle and caching
- Tokens are **not persisted**.
- The token cache is **in-memory only** (when enabled).
- Tokens are retrieved during **task execution** and refreshed based on the **Refresh before expiry** setting configured on the credential.
- Token caching can be **enabled or disabled** per credential.
:::alert{type="warning"}
Avoid storing long-lived secrets directly in flow YAML. Prefer credentials + secrets so Kestra can handle token minting/refresh and reduce exposure risk.
:::
## Credential hygiene
- Scope credentials to the smallest set of permissions required.
- Prefer short-lived tokens where possible; rotate long-lived keys.
- Don’t print tokens or derived values to logs — see [Best Practices for Secrets](../../../14.best-practices/9.secrets-management/index.md).
---
# Invitations in Kestra Enterprise: Onboard Users
URL: https://kestra.io/docs/enterprise/auth/invitations
> Onboard users easily with Invitations in Kestra. Manage user access by sending email invitations to join specific tenants or the entire instance.
Add users to a tenant or instance by invitation.
Administrators can invite users with pre-configured RBAC permissions. If an [email server is configured](../../../configuration/03.observability-and-networking/index.md), Kestra sends the invitation link by email. Otherwise, you can copy and share the link manually.
## How to invite users
1. Go to **IAM** in the sidebar and open the **Users** tab.
2. Click **+ Add**.
3. Enter the user's email address and select a group (via the **Groups** tab) or assign a role directly (via the **Roles** tab).
4. Click **+ Add** to send the invitation email or display the shareable link.
:::alert{type="info"}
Check **Create user directly (skip invitation)** to bypass the email flow. Use this only with third-party authentication such as SSO or LDAP.
:::
## Accepting invitations
When a user receives an invitation, they can click the link in the email to accept it. The user is redirected to the Kestra login page, where they create a password or log in with SSO if enabled.
If password-based login is enabled, the password they choose must satisfy the instance password policy configured under `kestra.security.basic-auth`. See [Security and Secrets configuration](../../../configuration/05.security-and-secrets/index.md) for the available password policy settings.
## Invite expiration time
Users have 7 days to accept the invitation. After this period, the invitation expires and must be reissued.
To change the default expiration, set `expireAfter` in `kestra.security.invitations`. For example, to set 30 days:
```yaml
kestra:
security:
invitations:
expireAfter: P30D
```
---
# RBAC in Kestra Enterprise: Roles and Permissions
URL: https://kestra.io/docs/enterprise/auth/rbac
> Implement Role-Based Access Control (RBAC) in Kestra. Define granular permissions for users, groups, and service accounts to secure your platform.
Kestra Enterprise supports Role-Based Access Control (RBAC) to manage access to tenants, namespaces, flows, and resources.
Kestra has three types of entities:
- Users: Represent a **person**. To add users to your Kestra instance, you can do one of the following:
- [Invite users](../invitations/index.md) to your instance or tenant from the UI
- Sync users from an external identity provider using [SCIM](../scim/index.mdx)
- Create users directly using [Terraform](../../../13.terraform/index.mdx)
- Automate user onboarding, offboarding, and group management from inside a flow using [IAM tasks](../../../15.how-to-guides/iam-automation/index.md)
- Groups: Represent a collection of **Users** and **Service Accounts**. Groups are a useful mechanism for providing the same roles to multiple users or service accounts at once by binding a role to a group.
- Service Accounts: Represent an **application**. They are treated as users when binding role assignments.
All these entities can be assigned to a role, which defines what resources the user, group, or service account can access. These entities do not belong to namespaces, but their permissions can be limited to specific namespaces via bindings on the **IAM** page.
The image below shows the relationship between Users, Groups, Service Accounts, Roles, and Bindings:

## Roles and Bindings
A role is a collection of permissions that can be assigned to users, service accounts, or groups. Each permission is a combination of a **resource** (e.g., `FLOW`, `EXECUTION`, `SECRET`) and one or more **actions** (e.g., `EXECUTE`, `VIEW`, `DELETE`). A role alone grants nothing — it must be attached to a user, service account, or group via a **binding** on the **IAM** page.
Users, service accounts, and groups can hold any number of roles simultaneously. Bindings can be scoped to one or more namespaces — scoped access automatically extends to all child namespaces (for example, binding to `prod` also grants access to `prod.engineering`). You can [configure a default role](../../../configuration/05.security-and-secrets/index.md) to assign it automatically to new users joining via [SSO](../sso/index.md). Use [Impersonate](#impersonate) to verify a user's effective permissions after assigning roles.
## Impersonate
After assigning permissions to a user, Instance Owners can impersonate users to verify their access is correct. Impersonation switches your view to that user's perspective and can be closed back to the Instance Owner view at any time.


### Resources
A resource is a category of product entity or capability that can be controlled through RBAC. Each resource has its own set of allowed actions.
**Core resources** (namespace-scoped — bindings can restrict access to specific namespaces):
| Resource | Description |
|---|---|
| `FLOW` | Flows, their revisions, graphs, and dependencies |
| `EXECUTION` | Executions, their state, logs, outputs, and files |
| `TRIGGER` | Triggers attached to flows |
| `NAMESPACE` | Namespaces and their files |
| `KVSTORE` | Key-value store entries |
| `SECRET` | Secrets stored in the namespace |
| `CREDENTIAL` | Credentials for external integrations (namespace-level and tenant-level) |
**Apps and features** (tenant-scoped):
| Resource | Description |
|---|---|
| `DASHBOARD` | Custom dashboards |
| `BLUEPRINT` | Custom blueprints |
| `APP` | Apps and their executions |
| `TESTSUITE` | Unit tests |
| `ASSET` | Data assets and lineage |
| `MCP_SERVER` | MCP servers exposing flows as AI tools |
| `COPILOT` | AI Copilot flow generation |
**Administration** (tenant-scoped):
| Resource | Description |
|---|---|
| `USER` | Users in the tenant |
| `GROUP` | Groups and their members |
| `ROLE` | RBAC roles |
| `BINDING` | Role-to-entity bindings |
| `SERVICE_ACCOUNT` | Service accounts |
| `INVITATION` | User invitations |
| `AUDITLOG` | Audit log entries |
| `POLICY` | Governance policies controlling flow and task behavior (namespace-scope and tenant-scope) |
| `SYSTEM_SETTINGS` | Instance-level settings |
| `TENANT_SETTINGS` | Tenant-level settings |
### Actions
Each resource defines its own set of allowed actions. Not every action applies to every resource.
**Common actions** (available on most resources):
| Action | Meaning |
|---|---|
| `VIEW` | Read a single item's details |
| `LIST` | Search or browse items |
| `CREATE` | Create a new item |
| `UPDATE` | Modify an existing item |
| `DELETE` | Remove an item |
**Resource-specific actions:**
| Resource | Additional actions |
|---|---|
| `FLOW` | `EXECUTE` (trigger an execution), `DISABLE`, `ENABLE`, `VALIDATE`, `EXPORT`, `IMPORT` |
| `EXECUTION` | `RESTART`, `KILL`, `REPLAY`, `PAUSE`, `RESUME`, `CHANGE_LABELS`, `ACCESS_LOGS`, `ACCESS_OUTPUTS`, `ACCESS_FILES`, `FOLLOW` (live SSE stream), `EXPORT`, `UNQUEUE`, `FORCE_RUN` |
| `TRIGGER` | `UNLOCK`, `RESTART`, `DISABLE`, `ENABLE`, `EXPORT`, `BACKFILL` |
| `NAMESPACE` | `MANAGE_FILES` (all namespace file operations) |
| `POLICY` | `EXECUTE` (dry-run evaluate a policy against its scope), `EXPORT`, `IMPORT` |
| `APP` | `EXECUTE`, `ACCESS_FILES`, `ACCESS_LOGS` |
| `TESTSUITE` | `EXECUTE` |
| `AUDITLOG` | `EXPORT` |
| `USER` | `MANAGE_GROUP_MEMBERSHIP`, `IMPERSONATE` |
| `GROUP` | `MANAGE_MEMBERS` |
| `COPILOT` | `USE` (only action) |
| `SYSTEM_SETTINGS` | — (`VIEW` and `UPDATE` only; no `CREATE`, `DELETE`, or `LIST`) |
| `TENANT_SETTINGS` | — (`VIEW` and `UPDATE` only; no `CREATE`, `DELETE`, or `LIST`) |
:::alert{type="info"}
For a complete resource-to-endpoint mapping, see the [Permissions reference](./permissions-reference/index.md).
If you are upgrading from Kestra 1.x, see the [RBAC action model migration guide](../../../11.migration-guide/v2.0.0/rbac-action-model/index.md) for how old CRUD permissions map to the new actions and what was dropped.
:::
### MCP server permissions
`MCP_SERVER` is a first-class RBAC resource that controls access to [Kestra MCP servers](../../../ai-tools/03.mcp-server/index.md). Supported actions are `VIEW`, `LIST`, `CREATE`, `UPDATE`, and `DELETE`.
Default role assignments:
| Role | Actions granted |
|---|---|
| Admin | All (`VIEW`, `LIST`, `CREATE`, `UPDATE`, `DELETE`) |
| Developer / Editor | All (`VIEW`, `LIST`, `CREATE`, `UPDATE`, `DELETE`) |
| Launcher | — (not included) |
| Viewer | `VIEW`, `LIST` |
In addition to these permissions, access to a **private** MCP server is also flow-scoped: a user can connect to a private server only if they have `FLOW: EXECUTE` on at least one namespace that contains a flow with an `McpToolTrigger` pointing at that server.
### Managed roles
Kestra ships five managed roles. Each role's full permission set is visible under **IAM → Roles**. Instance Owners can create additional custom roles on top of these. Users can hold multiple roles.
| Role | Description |
|---|---|
| **Admin** | All actions on all resources. |
| **Developer** | Everything Editor has, plus: full namespace management (including file management and plugin default import), secrets, credentials, and full blueprint CRUD. For engineers who also need platform-level access. |
| **Editor** | Full flow and execution management (create, update, delete, execute, restart, kill, etc.), triggers, KV, dashboards, apps, test suites, assets, MCP servers, settings, and Copilot. No namespace file management, no secrets or credentials, blueprint read-only. No IAM resources. |
| **Launcher** | Execute flows and monitor executions (`EXECUTE`, `REPLAY`, `RESTART`, `CHANGE_LABELS`, `ACCESS_LOGS`, `ACCESS_OUTPUTS`, `ACCESS_FILES`, `FOLLOW`, `EXPORT`). Read-only on triggers, KV, dashboards, and assets. No flow write access, no namespace management. |
| **Viewer** | `VIEW`, `LIST`, and `EXPORT` on flows, executions, triggers, and namespaces. Can access execution logs, outputs, files, and live-follow executions. No execution state changes (no restart, kill, replay, etc.). No write access anywhere. |
## Instance Owner and Admin
Kestra provides two roles for managing your instance: Instance Owner and Admin.
- Instance Owner is a user type with elevated privileges for global control.
- Admin is a customizable role that grants full access to all resources (scoped to a tenant if multi-tenancy is enabled).
:::collapse{title="Summary"}
Key differences between Admin and Instance Owner:
| Feature | Admin (scoped to a tenant if enabled) | Instance Owner |
|-------------------------------------|----------------------------------------------------|------------------------------------------------------|
| Access Level | By default as all permissions, depends on the Role | Manages tenants and IAM across all tenants |
| Tenant Management | No | View, create, update, delete tenants across all tenants |
| User/Role/Group/Bindings Management | Has the permission by default | View, create, update, delete across all tenants |
| Flow/Execution Management | Has the permission by default | No |
| Set Instance Owner privilege | No | Yes |
:::
## Instance Owner
Instance Owner is a powerful user type with instance-wide privileges. Use it sparingly — only for tasks that require it, such as creating tenants, troubleshooting, or helping a user.
Unlike tenant-scoped roles, Instance Owner operates across all tenants and does not require any Role or Binding. Instance Owners access instance-wide controls through the [Instance Owner console](../../05.instance/00.instance-owner/index.md), which covers tenant management, instance IAM, infrastructure, and governance.
For how to create Instance Owner users and manage the privilege, see [Instance Owner](../../05.instance/00.instance-owner/index.md).
## Admin
Kestra has no Admin user type; Admin is a role with full permissions.
This role can be assigned to any user, service account, or group. This allows you to have different types of admins, grant admin permissions to a whole group, and revoke those permissions at any time without deleting any group or user.
When using multi-tenancy, Kestra assigns the Admin role to the user who created the tenant by default.
:::alert{type="info"}
If you see an error when creating a new User or Service Account, it might be caused by a limit on your license. In that case, [reach out to us](/contact-us) to validate and optionally upgrade your license.
:::
## Creating a user with an Admin role
### Through the UI
When launching Kestra for the first time with no prior CLI setup, you are prompted to set up Kestra through the [Setup Page](../../01.overview/02.setup/index.md).
This creates the first user, automatically assigns the Admin role, and binds it.
Later, create a new user or select an existing user and assign the Admin role from the **Access** tab in IAM.
### Through the CLI
To create a user with the Admin role from the CLI, use the `--admin` option:
```bash
kestra auths users create prod.admin@kestra.io TopSecret42 --admin
## schema:
kestra auths users create --admin
```
## User lockout
Use the following configuration to change the lockout behavior after too many failed login attempts. By default, Kestra locks the user for the `lock-duration` period after a `threshold` number of failed attempts within the `monitoring-window` duration. The snippet below lists the default values — adjust them based on your preferences:
```yaml
kestra:
security:
login:
failed-attempts:
threshold: 10
monitoring-window: PT5M
lock-duration: PT30M
```
The key attributes are:
- `threshold`: Sets the number of allowed failed attempts before a user is locked out.
- `monitoring-window`: Defines the period during which failed login attempts are counted before triggering a lock.
- `lock-duration`: Defines how long the account remains locked.
With the configuration above, a user gets 10 failed login attempts in a 5-minute window before lockout. They must wait 30 minutes, be unlocked by an Admin, or reset their password using the Forgot Password link. An Instance Owner can also unlock a user manually from the user's detail page.
## Change password
Users can change their password from their profile, accessible via the user avatar in the bottom-left corner of the UI. "Forgot Password" settings can be configured in your Kestra configuration under `basic-auth.password-reset`. Settings to consider are the cooldown time between reset requests and how many requests can be made in a given time window.
```yaml
kestra:
security:
basic-auth:
password-reset:
cooldown: PT5M # Minimum time required between two password reset emails for the same user
rate-limit:
max-requests: 10 # Maximum number of password reset requests allowed per client within the time window
window: PT1H # Time window during which password reset requests are counted for rate limiting
```
### Reset a password as an Instance Owner
Users can reset their password via the Forgot Password link on the login page. An Instance Owner can also reset a user's password from the User Edit page at **Instance Owner → Instance IAM → Users**.


## RBAC FAQ
:::collapse{title="Why is Admin a Role rather than User type?"}
The Admin role is a collection of permissions that can be assigned to users, service accounts, or groups. This lets you grant admin permissions to multiple users or groups, and revoke them at any time without deleting anything.
A user can start as Admin and later have that permission narrowed or removed. Multiple users can share the Admin role, or the same user can hold multiple roles with different scopes.
Users represent authentication (who you are); roles represent authorization (what you can do). Decoupling the two lets you grant permissions to multiple users or groups at once by attaching a single role to a group.
:::
:::collapse{title="Why can't I edit an existing Binding?"}
A binding is immutable. If a binding no longer reflects the desired permissions, delete it and create a new one for the same user, service account, or group with different roles and/or namespaces.
This prevents accidental changes to existing permissions.
:::
:::collapse{title="What happens if you delete a Group?"}
All users and service accounts in that group lose the permissions granted by bindings attached to it. The users and service accounts themselves still exist.
:::
---
# RBAC Permissions Reference for Kestra Enterprise
URL: https://kestra.io/docs/enterprise/auth/rbac/permissions-reference
> Reference for Kestra RBAC resources and actions mapped to API endpoints. Use this to design least-privilege roles and troubleshoot authorization errors.
This reference maps each RBAC resource and action to the API endpoints that enforce it. Use it to design least-privilege roles and troubleshoot authorization errors.
## How to read this page
- Endpoints are grouped by resource and action.
- Each resource section gates all its endpoints: a user must have at least one valid action on the resource to reach any endpoint within it. Endpoints that do not check a specific action beyond this gate are marked "any action".
- Some endpoints require permissions on more than one resource; notes call these out.
- Namespace-scoped resources respect namespace bindings: access is evaluated against the namespace of the target item, and child namespaces inherit access.
:::collapse{title="FLOW"}
**Scope:** Namespace
**Actions and their meaning**
- `VIEW`: read a single flow, its graph, revisions, tasks, dependencies, and expressions context.
- `LIST`: search or browse flows.
- `CREATE`: create a flow. Creating a flow in a namespace that does not yet exist also requires `NAMESPACE: CREATE`.
- `UPDATE`: modify a flow's source, tasks, or graph.
- `DELETE`: delete a flow or its revisions.
- `EXECUTE`: trigger an execution of a flow.
- `DISABLE`: disable a flow.
- `ENABLE`: enable a disabled flow.
- `VALIDATE`: validate flow YAML. (Accessible with any `FLOW` action — no dedicated check.)
- `EXPORT`: export flows as a ZIP archive.
- `IMPORT`: import flows from an archive.
**Endpoints**
VIEW
- `GET /api/v1/{tenant}/flows/{namespace}/{id}`
- `GET /api/v1/{tenant}/flows/{namespace}/{id}/graph`
- `GET /api/v1/{tenant}/flows/{namespace}/{id}/revisions`
- `GET /api/v1/{tenant}/flows/{namespace}/{id}/tasks/{taskId}`
- `GET /api/v1/{tenant}/flows/{namespace}/{id}/dependencies`
- `GET /api/v1/{tenant}/namespaces/{namespace}/dependencies`
- `POST /api/v1/{tenant}/flows/expressions`
LIST
- `GET /api/v1/{tenant}/flows/search`
- `GET /api/v1/{tenant}/flows/{namespace}`
- `GET /api/v1/{tenant}/flows/source`
- `GET /api/v1/{tenant}/flows/distinct-namespaces` (any `FLOW` action)
- `GET /api/v1/{tenant}/flows/deprecated`
CREATE
- `POST /api/v1/{tenant}/flows` (single flow)
- `POST /api/v1/{tenant}/flows/{namespace}` (bulk upsert; also requires `UPDATE` and `DELETE`)
UPDATE
- `PUT /api/v1/{tenant}/flows/{namespace}/{id}`
- `PATCH /api/v1/{tenant}/flows/{namespace}/{id}/{taskId}`
- `POST /api/v1/{tenant}/flows/bulk` (also requires `CREATE` and `DELETE`)
- `POST /api/v1/{tenant}/executions/{executionId}/eval/{taskRunId}`
- `POST /api/v1/{tenant}/executions/{executionId}/{taskRunId}/eval`
DELETE
- `DELETE /api/v1/{tenant}/flows/{namespace}/{id}`
- `DELETE /api/v1/{tenant}/flows/{namespace}/{id}/revisions` (specific revisions)
- `DELETE /api/v1/{tenant}/flows/delete/by-query`
- `DELETE /api/v1/{tenant}/flows/delete/by-ids`
EXECUTE
- `POST /api/v1/{tenant}/executions/{namespace}/{id}` (create execution)
- `POST /api/v1/{tenant}/executions/{namespace}/{id}/validate`
- `GET /api/v1/{tenant}/executions/namespaces/{namespace}/flows`
DISABLE
- `POST /api/v1/{tenant}/flows/disable/by-query`
- `POST /api/v1/{tenant}/flows/disable/by-ids`
ENABLE
- `POST /api/v1/{tenant}/flows/enable/by-query`
- `POST /api/v1/{tenant}/flows/enable/by-ids`
EXPORT
- `GET /api/v1/{tenant}/flows/export/by-query`
- `POST /api/v1/{tenant}/flows/export/by-ids`
- `GET /api/v1/{tenant}/flows/export/by-query/csv` (streaming)
IMPORT
- `POST /api/v1/{tenant}/flows/import`
VALIDATE (any `FLOW` action — no dedicated check)
- `POST /api/v1/{tenant}/flows/validate`
- `POST /api/v1/{tenant}/flows/validate/task`
- `POST /api/v1/{tenant}/flows/validate/trigger`
Notes
- Webhook execution endpoints (`/executions/webhook/{namespace}/{id}/{key}`) are authorized by webhook key, not RBAC.
- `POST /api/v1/{tenant}/flows/graph` has no action check.
:::
---
:::collapse{title="EXECUTION"}
**Scope:** Namespace
**Actions and their meaning**
- `VIEW`: read a single execution's details, graph, or associated flow.
- `LIST`: search or browse executions.
- `DELETE`: delete executions.
- `RESTART`: restart one or more executions from the beginning or a specific task.
- `KILL`: kill a running execution.
- `REPLAY`: replay an execution (creates a new execution from a previous one).
- `PAUSE`: pause a running execution.
- `RESUME`: resume a paused execution.
- `CHANGE_LABELS`: update labels on a terminated execution.
- `ACCESS_LOGS`: read or stream execution logs.
- `ACCESS_OUTPUTS`: read task run outputs.
- `ACCESS_FILES`: download or preview execution output files.
- `FOLLOW`: stream live execution status events via SSE.
- `EXPORT`: export execution data as CSV.
- `UNQUEUE`: move a queued execution out of the queue.
- `FORCE_RUN`: force a paused or queued execution to run immediately.
- `UPDATE`: change the state of a task run within an execution.
**Endpoints**
VIEW
- `GET /api/v1/{tenant}/executions/{executionId}`
- `GET /api/v1/{tenant}/executions/{executionId}/graph`
- `GET /api/v1/{tenant}/executions/{executionId}/flow`
- `GET /api/v1/{tenant}/executions/flows/{namespace}/{flowId}`
LIST
- `GET /api/v1/{tenant}/executions/search`
- `GET /api/v1/{tenant}/executions`
- `GET /api/v1/{tenant}/executions/flows/{namespace}/{flowId}` (search by flow)
- `GET /api/v1/{tenant}/executions/namespaces` (distinct namespaces with executions)
- `POST /api/v1/{tenant}/executions/latest` (any `EXECUTION` action)
DELETE
- `DELETE /api/v1/{tenant}/executions/{executionId}`
- `DELETE /api/v1/{tenant}/executions/by-ids`
- `DELETE /api/v1/{tenant}/executions/by-query`
- `DELETE /api/v1/{tenant}/logs/{executionId}` (delete logs for an execution)
- `DELETE /api/v1/{tenant}/logs/{namespace}/{flowId}` (delete logs for a flow)
RESTART
- `POST /api/v1/{tenant}/executions/{executionId}/restart`
- `POST /api/v1/{tenant}/executions/restart/by-ids`
- `POST /api/v1/{tenant}/executions/restart/by-query`
KILL
- `DELETE /api/v1/{tenant}/executions/{executionId}/kill`
- `DELETE /api/v1/{tenant}/executions/kill/by-ids`
- `DELETE /api/v1/{tenant}/executions/kill/by-query`
REPLAY
- `POST /api/v1/{tenant}/executions/{executionId}/replay`
- `POST /api/v1/{tenant}/executions/{executionId}/replay-with-inputs`
- `POST /api/v1/{tenant}/executions/replay/by-ids`
- `POST /api/v1/{tenant}/executions/replay/by-query`
PAUSE
- `POST /api/v1/{tenant}/executions/{executionId}/pause`
- `POST /api/v1/{tenant}/executions/pause/by-ids`
- `POST /api/v1/{tenant}/executions/pause/by-query`
RESUME
- `POST /api/v1/{tenant}/executions/{executionId}/resume`
- `POST /api/v1/{tenant}/executions/{executionId}/resume-from-breakpoint`
- `POST /api/v1/{tenant}/executions/resume/by-ids`
- `POST /api/v1/{tenant}/executions/resume/by-query`
CHANGE_LABELS
- `POST /api/v1/{tenant}/executions/{executionId}/labels`
- `POST /api/v1/{tenant}/executions/labels/by-ids`
- `POST /api/v1/{tenant}/executions/labels/by-query`
ACCESS_LOGS
- `GET /api/v1/{tenant}/logs/search`
- `GET /api/v1/{tenant}/logs/{executionId}`
- `GET /api/v1/{tenant}/logs/{executionId}/download`
- `GET /api/v1/{tenant}/logs/{executionId}/follow` (SSE log stream)
ACCESS_OUTPUTS
- `GET /api/v1/{tenant}/outputs/{executionId}/{taskRunId}`
- `GET /api/v1/{tenant}/outputs/{executionId}`
ACCESS_FILES
- `GET /api/v1/{tenant}/executions/{executionId}/file`
- `GET /api/v1/{tenant}/executions/{executionId}/file/metas`
- `GET /api/v1/{tenant}/executions/{executionId}/file/preview`
FOLLOW
- `GET /api/v1/{tenant}/executions/{executionId}/follow` (SSE execution state stream)
- `GET /api/v1/{tenant}/executions/{executionId}/follow-dependencies`
EXPORT
- `GET /api/v1/{tenant}/executions/export/by-query/csv` (streaming)
UNQUEUE
- `POST /api/v1/{tenant}/executions/{executionId}/unqueue`
- `POST /api/v1/{tenant}/executions/unqueue/by-ids`
- `POST /api/v1/{tenant}/executions/unqueue/by-query`
FORCE_RUN
- `POST /api/v1/{tenant}/executions/{executionId}/force-run`
- `POST /api/v1/{tenant}/executions/force-run/by-ids`
- `POST /api/v1/{tenant}/executions/force-run/by-query`
UPDATE
- `POST /api/v1/{tenant}/executions/{executionId}/state`
- `POST /api/v1/{tenant}/executions/change-status/by-ids`
- `POST /api/v1/{tenant}/executions/change-status/by-query`
- `POST /api/v1/{tenant}/executions/{executionId}/{taskRunId}/state`
Notes
- Webhook execution endpoints are authorized by webhook key only; RBAC is not checked.
- Execution creation (`POST /executions/{namespace}/{id}`) checks `FLOW: EXECUTE`, not an `EXECUTION` action.
:::
---
:::collapse{title="TRIGGER"}
**Scope:** Namespace
**Actions and their meaning**
- `LIST`: search or browse triggers.
- `UNLOCK`: unlock a locked trigger.
- `RESTART`: restart a trigger.
- `DISABLE` / `ENABLE`: disable or enable triggers (both operations check the `DISABLE` action).
- `DELETE`: delete triggers.
- `EXPORT`: export trigger configuration.
- `BACKFILL`: create, pause, unpause, or delete a backfill on a schedule trigger.
**Endpoints**
LIST
- `GET /api/v1/{tenant}/triggers/search`
- `GET /api/v1/{tenant}/triggers/{namespace}/{flowId}`
UNLOCK
- `POST /api/v1/{tenant}/triggers/{namespace}/{flowId}/{triggerId}/unlock`
- `POST /api/v1/{tenant}/triggers/unlock/by-ids`
RESTART
- `POST /api/v1/{tenant}/triggers/{namespace}/{flowId}/{triggerId}/restart`
DISABLE / ENABLE
- `POST /api/v1/{tenant}/triggers/disable/by-ids` (also used for re-enabling)
DELETE
- `DELETE /api/v1/{tenant}/triggers/{namespace}/{flowId}/{triggerId}`
- `DELETE /api/v1/{tenant}/triggers/by-ids`
- `DELETE /api/v1/{tenant}/triggers/by-query`
EXPORT
- `GET /api/v1/{tenant}/triggers/export/by-query` (streaming)
BACKFILL
- `POST /api/v1/{tenant}/triggers/backfills` (create)
- `POST /api/v1/{tenant}/triggers/backfills/pause`
- `POST /api/v1/{tenant}/triggers/backfills/pause/by-ids`
- `POST /api/v1/{tenant}/triggers/backfills/unpause`
- `POST /api/v1/{tenant}/triggers/backfills/unpause/by-ids`
- `DELETE /api/v1/{tenant}/triggers/backfills`
- `DELETE /api/v1/{tenant}/triggers/backfills/by-ids`
:::
---
:::collapse{title="NAMESPACE"}
**Scope:** Namespace
**Actions and their meaning**
- `VIEW`: read a namespace's details and inherited variables.
- `LIST`: search or browse namespaces.
- `CREATE`: create a namespace.
- `UPDATE`: update namespace configuration.
- `DELETE`: delete a namespace.
- `MANAGE_FILES`: all namespace file operations (search, read, create, move, delete, export).
**Endpoints**
VIEW
- `GET /api/v1/{tenant}/namespaces/{id}`
- `GET /api/v1/{tenant}/namespaces/{id}/inherited-variables`
LIST
- `GET /api/v1/{tenant}/namespaces/search`
- `POST /api/v1/{tenant}/namespaces/autocomplete`
CREATE
- `POST /api/v1/{tenant}/namespaces`
UPDATE
- `PUT /api/v1/{tenant}/namespaces/{id}`
DELETE
- `DELETE /api/v1/{tenant}/namespaces/{id}`
MANAGE_FILES (all namespace file operations)
- `GET /api/v1/{tenant}/namespaces/{namespace}/files/search`
- `GET /api/v1/{tenant}/namespaces/{namespace}/files`
- `GET /api/v1/{tenant}/namespaces/{namespace}/files/stats`
- `GET /api/v1/{tenant}/namespaces/{namespace}/files/revisions`
- `GET /api/v1/{tenant}/namespaces/{namespace}/files/directory`
- `GET /api/v1/{tenant}/namespaces/{namespace}/files/export`
- `POST /api/v1/{tenant}/namespaces/{namespace}/files/directory` (create directory)
- `POST /api/v1/{tenant}/namespaces/{namespace}/files` (create file)
- `PUT /api/v1/{tenant}/namespaces/{namespace}/files` (move file or directory)
- `DELETE /api/v1/{tenant}/namespaces/{namespace}/files` (delete file or directory)
:::
---
:::collapse{title="KVSTORE"}
**Scope:** Namespace
**Actions and their meaning**
- `VIEW`: read a KV entry.
- `LIST`: list or browse KV entries, including inherited entries.
- `CREATE` / `UPDATE`: set a KV value (the same endpoint creates or updates depending on whether the key exists).
- `DELETE`: delete KV entries.
**Endpoints**
VIEW
- `GET /api/v1/{tenant}/namespaces/{namespace}/kv/{key}`
- `GET /api/v1/{tenant}/namespaces/{namespace}/kv/{key}/detail`
LIST
- `GET /api/v1/{tenant}/namespaces/{namespace}/kv`
- `GET /api/v1/{tenant}/namespaces/{namespace}/kv/inheritance`
CREATE / UPDATE
- `PUT /api/v1/{tenant}/namespaces/{namespace}/kv/{key}` (creates if key does not exist, updates if it does)
DELETE
- `DELETE /api/v1/{tenant}/namespaces/{namespace}/kv/{key}`
- `DELETE /api/v1/{tenant}/namespaces/{namespace}/kv` (bulk delete)
:::
---
:::collapse{title="DASHBOARD"}
**Scope:** Tenant
**Actions and their meaning**
- `VIEW` / `LIST`: read dashboards and their charts.
- `CREATE`: create dashboards.
- `UPDATE`: update dashboard configuration.
- `DELETE`: delete dashboards.
**Endpoints**
VIEW / LIST (any `DASHBOARD` action — no dedicated per-action check at controller level)
- `GET /api/v1/{tenant}/dashboards`
- `GET /api/v1/{tenant}/dashboards/{id}`
- `POST /api/v1/{tenant}/dashboards/{id}/charts/{chartId}`
- `POST /api/v1/{tenant}/dashboards/charts/preview`
- `POST /api/v1/{tenant}/dashboards/validate`
- `POST /api/v1/{tenant}/dashboards/validate/chart`
- `POST /api/v1/{tenant}/dashboards/{id}/charts/{chartId}/export/to-csv`
- `POST /api/v1/{tenant}/dashboards/charts/export/to-csv`
CREATE
- `POST /api/v1/{tenant}/dashboards`
UPDATE
- `PUT /api/v1/{tenant}/dashboards/{id}`
DELETE
- `DELETE /api/v1/{tenant}/dashboards/{id}`
:::
---
:::collapse{title="SECRET"}
**Scope:** Namespace
**Actions and their meaning**
- `VIEW` / `LIST`: list secrets and view their metadata. Secret values are never returned by the API.
- `UPDATE`: create or update a secret (creation is enforced via this action).
- `DELETE`: delete a secret.
**Endpoints**
VIEW / LIST (any `SECRET` action)
- `GET /api/v1/{tenant}/namespaces/{namespace}/secrets`
- `GET /api/v1/{tenant}/namespaces/{namespace}/inherited-secrets`
UPDATE
- `PUT /api/v1/{tenant}/namespaces/{namespace}/secrets`
- `PATCH /api/v1/{tenant}/namespaces/{namespace}/secrets/{key}`
DELETE
- `DELETE /api/v1/{tenant}/namespaces/{namespace}/secrets/{key}`
Notes
- `SECRET` has no `CREATE` action; creation uses `UPDATE`.
:::
---
:::collapse{title="CREDENTIAL"}
**Scope:** Namespace or tenant (depending on whether the credential is namespace-level or tenant-level)
**Actions and their meaning**
- `VIEW` / `LIST`: list and view credentials.
- `CREATE`: create credentials.
- `UPDATE`: update credentials or test a connection.
- `DELETE`: delete credentials.
**Endpoints**
VIEW / LIST
- `GET /api/v1/{tenant}/credentials`
- `GET /api/v1/{tenant}/credentials/{id}`
- `GET /api/v1/{tenant}/namespaces/{namespace}/credentials`
- `GET /api/v1/{tenant}/namespaces/{namespace}/credentials/{name}`
- `GET /api/v1/{tenant}/namespaces/{namespace}/credentials/inherited`
CREATE
- `POST /api/v1/{tenant}/credentials`
- `POST /api/v1/{tenant}/namespaces/{namespace}/credentials`
UPDATE
- `PUT /api/v1/{tenant}/credentials/{id}`
- `POST /api/v1/{tenant}/credentials/{id}/test`
- `PUT /api/v1/{tenant}/namespaces/{namespace}/credentials/{name}`
- `POST /api/v1/{tenant}/namespaces/{namespace}/credentials/{name}/test`
DELETE
- `DELETE /api/v1/{tenant}/credentials/{id}`
- `DELETE /api/v1/{tenant}/namespaces/{namespace}/credentials/{name}`
:::
---
:::collapse{title="BLUEPRINT"}
**Scope:** Tenant
**Actions and their meaning**
- `VIEW` / `LIST`: view or browse custom blueprints.
- `CREATE`: create a blueprint.
- `UPDATE`: update a blueprint.
- `DELETE`: delete a blueprint.
**Endpoints**
VIEW / LIST
- `GET /api/v1/{tenant}/blueprints/custom`
- `GET /api/v1/{tenant}/blueprints/custom/{id}`
- `GET /api/v1/{tenant}/blueprints/custom/{id}/source`
- `GET /api/v1/{tenant}/blueprints/custom/tags`
- `GET /api/v1/{tenant}/blueprints/flows/{id}`
- `POST /api/v1/{tenant}/blueprints/flows/{id}/use-template`
CREATE
- `POST /api/v1/{tenant}/blueprints/flows`
UPDATE
- `PUT /api/v1/{tenant}/blueprints/flows/{id}`
DELETE
- `DELETE /api/v1/{tenant}/blueprints/flows/{id}`
Notes
- Community blueprint endpoints (`/blueprints/community/...`) do not require `BLUEPRINT` permission.
:::
---
:::collapse{title="APP"}
**Scope:** Tenant (with namespace checks when the app definition references a namespace)
**Actions and their meaning**
- `VIEW` / `LIST`: view app source, search, and catalog.
- `CREATE`: create or import apps.
- `UPDATE`: update apps, enable, or disable them.
- `DELETE`: delete apps.
- `EXECUTE`: dispatch actions through an app (run the app).
- `ACCESS_FILES`: download or preview files generated by an app execution.
- `ACCESS_LOGS`: view app execution logs.
**Endpoints**
VIEW / LIST
- `GET /api/v1/{tenant}/apps/search`
- `GET /api/v1/{tenant}/apps/catalog`
- `GET /api/v1/{tenant}/apps/tags`
- `GET /api/v1/{tenant}/apps/{uid}`
- `GET /api/v1/{tenant}/apps/view/{uid}` (PRIVATE apps require `APP: EXECUTE` or equivalent access level)
CREATE
- `POST /api/v1/{tenant}/apps`
- `POST /api/v1/{tenant}/apps/import`
- `POST /api/v1/{tenant}/apps/preview`
- `POST /api/v1/{tenant}/apps/export`
UPDATE
- `PUT /api/v1/{tenant}/apps/{uid}`
- `POST /api/v1/{tenant}/apps/{uid}/enable`
- `POST /api/v1/{tenant}/apps/{uid}/disable`
- `POST /api/v1/{tenant}/apps/enable`
- `POST /api/v1/{tenant}/apps/disable`
DELETE
- `DELETE /api/v1/{tenant}/apps/{uid}`
- `DELETE /api/v1/{tenant}/apps`
EXECUTE
- `POST /api/v1/{tenant}/apps/view/{id}/dispatch/{dispatch}`
- `GET /api/v1/{tenant}/apps/view/{id}/streams/{stream}`
ACCESS_FILES
- `GET /api/v1/{tenant}/apps/view/{id}/file/preview`
- `GET /api/v1/{tenant}/apps/view/{id}/file/meta`
- `GET /api/v1/{tenant}/apps/view/{id}/file/download`
ACCESS_LOGS
- `GET /api/v1/{tenant}/apps/view/{uid}/logs/download`
Notes
- PUBLIC apps are accessible without authentication for view and dispatch. PRIVATE apps require appropriate `APP` actions.
:::
---
:::collapse{title="TESTSUITE"}
**Scope:** Tenant
**Actions and their meaning**
- `VIEW` / `LIST`: view tests and test results.
- `CREATE`: create a test.
- `UPDATE`: update or enable/disable tests.
- `DELETE`: delete tests.
- `EXECUTE`: run tests.
**Endpoints**
VIEW / LIST
- `GET /api/v1/{tenant}/tests/{namespace}/{id}`
- `GET /api/v1/{tenant}/tests/search`
- `POST /api/v1/{tenant}/tests/validate`
- `GET /api/v1/{tenant}/tests/results/{id}`
- `POST /api/v1/{tenant}/tests/results/search/last`
- `GET /api/v1/{tenant}/tests/results/search`
CREATE
- `POST /api/v1/{tenant}/tests`
UPDATE
- `PUT /api/v1/{tenant}/tests/{namespace}/{id}`
- `POST /api/v1/{tenant}/tests/disable/by-ids`
- `POST /api/v1/{tenant}/tests/enable/by-ids`
DELETE
- `DELETE /api/v1/{tenant}/tests/{namespace}/{id}`
- `DELETE /api/v1/{tenant}/tests/by-ids`
EXECUTE
- `POST /api/v1/{tenant}/tests/{namespace}/{id}/run`
- `POST /api/v1/{tenant}/tests/run`
:::
---
:::collapse{title="ASSET"}
**Scope:** Tenant (with namespace checks when the asset has a namespace)
**Actions and their meaning**
- `VIEW` / `LIST`: view assets and their dependency or usage graphs.
- `CREATE`: create assets.
- `DELETE`: delete assets.
- `LOCK`: acquire a write lock on an asset (from a flow task or the UI).
- `UNLOCK`: release a write lock on an asset (from a flow task or the UI). UI unlock is forced regardless of owner.
**Endpoints**
VIEW / LIST
- `GET /api/v1/{tenant}/assets/{id}`
- `GET /api/v1/{tenant}/assets/{id}/dependencies`
- `GET /api/v1/{tenant}/assets/search`
- `GET /api/v1/{tenant}/assets/usages/search`
CREATE
- `POST /api/v1/{tenant}/assets`
DELETE
- `DELETE /api/v1/{tenant}/assets/{id}`
- `DELETE /api/v1/{tenant}/assets/by-ids`
- `DELETE /api/v1/{tenant}/assets/by-query`
LOCK
- `POST /api/v1/{tenant}/assets/{id}/lock`
UNLOCK
- `DELETE /api/v1/{tenant}/assets/{id}/lock`
:::
---
:::collapse{title="MCP_SERVER"}
**Scope:** Tenant
**Actions and their meaning**
- `VIEW` / `LIST`: view MCP server configuration and registered tools.
- `CREATE`: create an MCP server.
- `UPDATE`: update an MCP server.
- `DELETE`: delete an MCP server.
Notes
- Connecting an AI agent to a PRIVATE server also requires `FLOW: EXECUTE` on at least one namespace with a flow that has an `McpToolTrigger` pointing at that server.
:::
---
:::collapse{title="COPILOT"}
**Scope:** Tenant
**Actions and their meaning**
- `USE`: access AI flow generation and Copilot features.
**Endpoints**
USE (any `COPILOT` action)
- `POST /api/v1/{tenant}/ai/generate/flow`
:::
---
:::collapse{title="AUDITLOG"}
**Scope:** Tenant
**Actions and their meaning**
- `VIEW` / `LIST`: search audit logs and view diffs.
- `EXPORT`: export audit logs.
**Endpoints**
VIEW / LIST
- `GET /api/v1/{tenant}/auditlogs/search`
- `POST /api/v1/{tenant}/auditlogs/find`
- `GET /api/v1/{tenant}/auditlogs/history/{detailId}`
- `GET /api/v1/{tenant}/auditlogs/{id}/diff`
EXPORT
- `GET /api/v1/{tenant}/auditlogs/export`
Notes
- Cross-tenant audit log endpoints under `/api/v1/auditlogs/...` (no tenant segment) are instance-owner-only and are not controlled by `AUDITLOG` permissions.
:::
---
:::collapse{title="SYSTEM_SETTINGS"}
**Scope:** Tenant
**Actions and their meaning**
- `VIEW`: read instance-level settings.
- `UPDATE`: modify instance-level settings.
:::
---
:::collapse{title="TENANT_SETTINGS"}
**Scope:** Tenant
**Actions and their meaning**
- `VIEW`: read tenant-level settings.
- `UPDATE`: modify tenant-level settings.
:::
---
:::collapse{title="USER"}
**Scope:** Tenant
**Actions and their meaning**
- `VIEW` / `LIST`: view user details and group membership.
- `CREATE` / `UPDATE` / `DELETE`: manage users.
- `MANAGE_GROUP_MEMBERSHIP`: update a user's group assignments.
- `IMPERSONATE`: impersonate a user to test their access.
Notes
- `USER` is not included in any of the standard managed roles (Viewer, Launcher, Editor, Developer). Only Admin includes it. Custom roles with `USER` actions are intended for platform administrators.
- IAM user management endpoints under `/api/v1/users` (no tenant segment) are instance-owner-only and do not require `USER` permissions.
:::
---
:::collapse{title="GROUP"}
**Scope:** Tenant
**Actions and their meaning**
- `VIEW` / `LIST`: view groups and their members.
- `CREATE`: create groups.
- `UPDATE`: update group metadata.
- `DELETE`: delete groups.
- `MANAGE_MEMBERS`: add or remove members from a group.
**Endpoints**
VIEW / LIST
- `GET /api/v1/{tenant}/groups/{id}`
- `GET /api/v1/{tenant}/groups/search`
- `POST /api/v1/{tenant}/groups/autocomplete`
- `POST /api/v1/{tenant}/groups/ids`
- `GET /api/v1/{tenant}/groups/{id}/members`
CREATE
- `POST /api/v1/{tenant}/groups`
UPDATE
- `PUT /api/v1/{tenant}/groups/{id}`
DELETE
- `DELETE /api/v1/{tenant}/groups/{id}`
MANAGE_MEMBERS
- `PUT /api/v1/{tenant}/groups/{id}/members/{userId}`
- `PUT /api/v1/{tenant}/groups/{id}/members/membership/{userId}`
- `DELETE /api/v1/{tenant}/groups/{id}/members/{userId}`
- `PUT /api/v1/{tenant}/users/{id}/groups`
Notes
- SCIM group endpoints (`/integrations/{integration}/scim/v2/Groups`) also use `GROUP` permissions.
:::
---
:::collapse{title="ROLE"}
**Scope:** Tenant
**Actions and their meaning**
- `VIEW` / `LIST`: view roles and their permissions.
- `CREATE`: create roles.
- `UPDATE`: update roles.
- `DELETE`: delete roles.
**Endpoints**
VIEW / LIST
- `GET /api/v1/{tenant}/roles/{id}`
- `GET /api/v1/{tenant}/roles/search`
- `POST /api/v1/{tenant}/roles/autocomplete`
- `POST /api/v1/{tenant}/roles/ids`
- `GET /api/v1/{tenant}/acls/permissions` (any `ROLE` action)
- `GET /api/v1/{tenant}/acls/actions` (any `ROLE` action)
CREATE
- `POST /api/v1/{tenant}/roles`
UPDATE
- `PUT /api/v1/{tenant}/roles/{id}`
DELETE
- `DELETE /api/v1/{tenant}/roles/{id}`
:::
---
:::collapse{title="BINDING"}
**Scope:** Tenant
**Actions and their meaning**
- `VIEW` / `LIST`: view bindings.
- `CREATE`: create a binding.
- `DELETE`: delete a binding.
**Endpoints**
VIEW / LIST
- `GET /api/v1/{tenant}/bindings/{id}`
- `GET /api/v1/{tenant}/bindings/search`
CREATE
- `POST /api/v1/{tenant}/bindings`
- `POST /api/v1/{tenant}/bindings/bulk`
DELETE
- `DELETE /api/v1/{tenant}/bindings/{id}`
Notes
- Bindings are immutable. To change a binding's scope or roles, delete it and create a new one.
:::
---
:::collapse{title="SERVICE_ACCOUNT"}
**Scope:** Tenant
**Actions and their meaning**
- `VIEW` / `LIST`: view service accounts and their API tokens.
- `CREATE`: create service accounts.
- `UPDATE`: update service accounts and issue API tokens.
- `DELETE`: delete service accounts or revoke API tokens.
**Endpoints**
VIEW / LIST
- `GET /api/v1/{tenant}/service-accounts/{id}`
- `GET /api/v1/{tenant}/service-accounts/{id}/api-tokens`
CREATE
- `POST /api/v1/{tenant}/service-accounts`
UPDATE
- `PUT /api/v1/{tenant}/service-accounts/{id}`
- `POST /api/v1/{tenant}/service-accounts/{id}/api-tokens`
DELETE
- `DELETE /api/v1/{tenant}/service-accounts/{id}`
- `DELETE /api/v1/{tenant}/service-accounts/{id}/api-tokens/{tokenId}`
Notes
- Instance-owner-only endpoints under `/api/v1/service-accounts` (no tenant segment) do not use `SERVICE_ACCOUNT` permissions.
:::
---
:::collapse{title="INVITATION"}
**Scope:** Tenant
**Actions and their meaning**
- `VIEW` / `LIST`: view invitations.
- `CREATE`: send an invitation.
- `DELETE`: revoke an invitation.
**Endpoints**
VIEW / LIST
- `GET /api/v1/{tenant}/invitations/search`
- `GET /api/v1/{tenant}/invitations/email/{email}`
- `GET /api/v1/{tenant}/invitations/{id}`
CREATE
- `POST /api/v1/{tenant}/invitations`
DELETE
- `DELETE /api/v1/{tenant}/invitations/{id}`
:::
:::collapse{title="POLICY"}
**Scope:** Namespace or Tenant (a policy targets either a namespace or the full tenant, and is authorized accordingly)
**Actions and their meaning**
- `VIEW`: read a single policy.
- `LIST`: search or browse policies.
- `CREATE`: create a policy.
- `UPDATE`: update a policy.
- `DELETE`: delete a policy.
- `EXECUTE`: dry-run evaluate a policy against every flow in its scope without persisting any changes.
- `EXPORT`: export policies as YAML.
- `IMPORT`: import policies from a YAML file. Each document is authorized against its own scope before anything is written — a single denial rejects the whole import.
**Endpoints**
VIEW
- `GET /api/v1/{tenant}/policies/{id}` (tenant-scope)
- `GET /api/v1/{tenant}/namespaces/{namespace}/policies/{id}` (namespace-scope)
LIST
- `GET /api/v1/{tenant}/policies/search` (tenant-scope)
- `GET /api/v1/{tenant}/namespaces/{namespace}/policies/search` (namespace-scope)
CREATE
- `POST /api/v1/{tenant}/policies` (tenant-scope)
- `POST /api/v1/{tenant}/namespaces/{namespace}/policies` (namespace-scope)
UPDATE
- `PUT /api/v1/{tenant}/policies/{id}` (tenant-scope)
- `PUT /api/v1/{tenant}/namespaces/{namespace}/policies/{id}` (namespace-scope)
DELETE
- `DELETE /api/v1/{tenant}/policies/{id}` (tenant-scope)
- `DELETE /api/v1/{tenant}/policies/delete/by-ids` (tenant-scope)
- `DELETE /api/v1/{tenant}/namespaces/{namespace}/policies/{id}` (namespace-scope)
- `DELETE /api/v1/{tenant}/namespaces/{namespace}/policies/delete/by-ids` (namespace-scope)
EXECUTE (dry-run evaluate)
- `GET /api/v1/{tenant}/policies/{id}/evaluate` (tenant-scope)
- `GET /api/v1/{tenant}/namespaces/{namespace}/policies/{id}/evaluate` (namespace-scope)
EXPORT
- `POST /api/v1/{tenant}/policies/export` (tenant-scope, all policies)
- `POST /api/v1/{tenant}/policies/export/by-ids` (tenant-scope, by IDs)
- `POST /api/v1/{tenant}/namespaces/{namespace}/policies/export` (namespace-scope, all policies)
- `POST /api/v1/{tenant}/namespaces/{namespace}/policies/export/by-ids` (namespace-scope, by IDs)
IMPORT
- `POST /api/v1/{tenant}/policies/import`
- `POST /api/v1/{tenant}/namespaces/{namespace}/policies/import`
VALIDATE (any `POLICY` VIEW action — no dedicated check)
- `POST /api/v1/{tenant}/policies/validate`
- `POST /api/v1/{tenant}/namespaces/{namespace}/policies/validate`
Notes
- Instance-scope policy endpoints (`/api/v1/instance/policies/...`) manage read-only static policies declared in server configuration and are instance-owner-only; they are not governed by `POLICY` RBAC permissions.
:::
---
## Related
- [RBAC overview](../index.md) — resources, actions, and managed roles explained
- [RBAC action model migration guide](../../../../11.migration-guide/v2.0.0/rbac-action-model/index.md) — how old CRUD permissions map to the new actions when upgrading from 1.x
- [kestractl roles](../../../../kestra-cli/kestractl/index.md#roles) — create and manage roles from the CLI
---
# SCIM Directory Sync in Kestra Enterprise
URL: https://kestra.io/docs/enterprise/auth/scim
> Automate user provisioning with SCIM Directory Sync. Synchronize users and groups from IdPs like Okta, Azure AD, and Keycloak to Kestra Enterprise.
import ChildCard from "~/components/docs/ChildCard.astro"
Sync users and groups from your Identity Provider (IdP) to Kestra using SCIM.
SCIM (System for Cross-domain Identity Management) is an open-standard protocol that automates user provisioning, de-provisioning, and group synchronization between identity providers (IdPs) such as Microsoft Entra ID or Okta and service providers such as Kestra. Kestra uses the SCIM 2.0 protocol.

## Benefits of directory sync with SCIM
1. **Automated provisioning and de-provisioning**: SCIM automates the provisioning and de-provisioning of users, creating a single source of truth (SSOT) for user identity data. Instead of manually creating and managing users in Kestra, you can synchronize them from your IdP.
2. **Consistency and compliance**: With SCIM, you can ensure consistency of identity information across systems and stay compliant with security and regulatory requirements.
3. **Governance at scale**: Managing users at scale across many applications can be difficult without a standardized method for identity synchronization. SCIM provides a scalable solution for managing user identities.
## Supported identity providers
For setup guides by provider, see the pages below.
---
# authentik SCIM Provisioning in Kestra
URL: https://kestra.io/docs/enterprise/auth/scim/authentik
> Configure SCIM provisioning with authentik. Learn how to automatically sync users and groups from authentik to your Kestra Enterprise instance.
Sync users and groups from authentik to Kestra using SCIM.
## Prerequisites
- **authentik Account**: An account with administrative privileges to configure SCIM provisioning.
- **Enable multi-tenancy in Kestra**: Tenants must be enabled in Kestra to support SCIM provisioning. You can enable tenants by setting the `kestra.ee.tenants.enabled` configuration property to `true`:
```yaml
kestra:
ee:
tenants:
enabled: true
```
:::alert{type="info"}
Tenants are enabled by default. Please refer to the [Migration Guide](../../../../11.migration-guide/v0.23.0/tenant-migration-ee/index.md) to assist with upgrading.
:::
## Kestra SCIM setup: create a new provisioning integration
1. Go to **Instance Owner**, select your tenant from the sidebar, open **IAM**, and click the **SCIM Provisioning** tab.
2. Click **+ Create**.
3. Fill in the following fields:
- **Name**: Enter a name for the provisioning integration.
- **Description**: Provide a brief description of the integration.
- **Provisioning Type**: Only SCIM 2.0 is supported — leave the default selection and click **Save**.
These steps generate a SCIM endpoint URL and a Secret Token. Save both — you will need them in the next steps.
The endpoint should look as follows:
```plaintext
https:///api/v1//integrations/integration_id/scim/v2
```
The Secret Token will be a long string (approximately 200 characters) used to authenticate requests from authentik to Kestra.
### Enable or disable SCIM integration
You can disable or remove the SCIM integration at any time. When disabled, all incoming requests to that endpoint are rejected.
:::alert{type="info"}
You can disable the integration while configuring authentik, then enable it once setup is complete.
:::
### IAM role and service account
When creating a new Provisioning Integration, Kestra will automatically create two additional objects:
1. Role `SCIMProvisioner` with the following permissions:
- `GROUPS`: `CREATE`, `READ` `UPDATE`, `DELETE`
- `USERS`: `CREATE`, `READ`, `UPDATE`
- `BINDINGS`: `CREATE`, `READ`, `UPDATE`, `DELETE`

2. Service Account with an API Token which was previously displayed as a Secret Token for the integration:

:::alert{type="info"}
Why the `SCIMProvisioner` role doesn't have the `DELETE` permission for `USERS`? This is because you cannot delete a user through our SCIM implementation. Users are global and SCIM provisioning is per tenant. When we receive a `DELETE` query for a user, we remove their tenant access but the user itself remains in the system.
:::
## authentik SCIM 2.0 setup
Configuring SCIM 2.0 follows a process similar to SSO — you'll need to create a new `Application`. Then, in the second step, select `SCIM` as the Provider Type.

In the `Protocol settings` section, enter the `URL` and `Secret Token` obtained from Kestra.
:::alert{type="info"}
If you are running authentik on a Mac machine with [docker-compose installer](https://docs.goauthentik.io/docs/installation/docker-compose), make sure to replace `localhost` in your Kestra's SCIM endpoint with `host.docker.internal` since otherwise the sync won't work. Your URL should look as follows: `http://host.docker.internal:8080/api/v1/dev/integrations/zIRjRAMGvkammpeLVuyJl/scim/v2`.
:::

## Test both SSO and SCIM by adding users and groups
First, create `Users` and `Groups` in the `Directory` settings.

Then assign your user(s) to an existing group.

You can set a password for each authentik user to allow them to log in directly to Kestra with their username/email and password.

Once groups and users are created, they are visible in the Kestra UI under **IAM → Users** and **Groups**. Log in as the default admin user and attach the desired role to each group to set the necessary permissions.

Then, to verify access, log in as one of those new authentik users in a separate browser or incognito mode and verify that the user has the permissions you expect.
## Additional resources
- [SCIM for authentik Documentation](https://docs.goauthentik.io/docs/providers/scim/)
- [Manage applications in authentik Documentation](https://docs.goauthentik.io/docs/applications/manage_apps)
---
# Keycloak SCIM Provisioning in Kestra
URL: https://kestra.io/docs/enterprise/auth/scim/keycloak
> Configure SCIM provisioning with Keycloak. Synchronize users and groups from Keycloak to Kestra Enterprise for centralized identity management.
Sync users and groups from Keycloak to Kestra using SCIM.
## Prerequisites
- **Keycloak Account**: An account with administrative privileges is required to configure SCIM provisioning.
- **Enable multi-tenancy in Kestra**: Tenants must be enabled in Kestra to support SCIM provisioning. You can enable tenants by setting the `kestra.ee.tenants.enabled` configuration property to `true`:
```yaml
kestra:
ee:
tenants:
enabled: true
```
:::alert{type="info"}
Tenants are enabled by default. Please refer to the [Migration Guide](../../../../11.migration-guide/v0.23.0/tenant-migration-ee/index.md) to assist with upgrading.
:::
## Kestra SCIM setup: create a new provisioning integration
1. Go to **Instance Owner**, select your tenant from the sidebar, open **IAM**, and click the **SCIM Provisioning** tab.
2. Click **+ Create**.
3. Fill in the following fields:
- **Name**: Enter a name for the provisioning integration.
- **Description**: Provide a brief description of the integration.
- **Provisioning Type**: Only SCIM 2.0 is supported — leave the default selection and click **Save**.
These steps generate a SCIM endpoint URL and a Secret Token. Save both — you will need them in the next steps.

The endpoint should look as follows:
```plaintext
https:///api/v1//integrations/integration_id/scim/v2
```
The Secret Token is a long string (approx. 200 characters) used to authenticate requests from Keycloak to Kestra.
### Enable or disable SCIM integration
You can disable or remove the SCIM integration at any time. When disabled, all incoming requests to that endpoint are rejected.
:::alert{type="info"}
You can disable the integration while configuring Keycloak, then enable it once setup is complete.
:::
### IAM role and service account
When creating a new Provisioning Integration, Kestra will automatically create two additional objects:
1. Role `SCIMProvisioner` with the following permissions:
- `GROUPS`: `CREATE`, `READ` `UPDATE`, `DELETE`
- `USERS`: `CREATE`, `READ`, `UPDATE`
- `BINDINGS`: `CREATE`, `READ`, `UPDATE`, `DELETE`

2. Service Account with an API Token which was previously displayed as the Secret Token for the integration:

:::alert{type="info"}
Why the `SCIMProvisioner` role doesn't have the `DELETE` permission for `USERS`? This is because you cannot delete a user using our SCIM implementation. Users are global and SCIM provisioning is per tenant. When we receive a `DELETE` query for a user, we remove their tenant access but the user itself remains in the system.
:::
## Keycloak SCIM setup
Keycloak [does not provide](https://github.com/keycloak/keycloak/issues/13484) any built-in support for SCIM v2.0. Some [open-source solutions](https://github.com/mitodl/keycloak-scim/) support groups synchronization but not users and membership synchronization.
However, there are paid solutions such as [SCIM for Keycloak](https://scim-for-keycloak.de/) that allow you to extend Keycloak with SCIM. The setup shown below was validated with Kestra 0.18.0 and Keycloak 25.0.2 — best if you use the same or higher versions.
1. Create an account at https://scim-for-keycloak.de/ and purchase a free license (no VAT number or credit card required).

2. Download the plugin JAR file from the **Downloads** section (e.g. `scim-for-keycloak-kc-25-2.2.1-free.jar`).

Place the JAR in the `./providers` directory of your Keycloak installation. See [SCIM for Keycloak Installation](https://scim-for-keycloak.de/documentation/installation/install) for details.
3. **Deploy Keycloak**:
- Create a simple `docker-compose.yaml` file:
```yaml
services:
keycloak:
container_name: keycloak
image: quay.io/keycloak/keycloak:25.0.2
ports:
- 8085:8085
environment:
KEYCLOAK_ADMIN: admin
KEYCLOAK_ADMIN_PASSWORD: admin
KC_SPI_THEME_WELCOME_THEME: scim
KC_SPI_REALM_RESTAPI_EXTENSION_SCIM_LICENSE_KEY:
command:
["start-dev", "--http-port=8085"]
volumes:
- ./providers:/opt/keycloak/providers
network_mode: "host" # Optional: for accessing external Kestra
```
- Run `docker compose up` to start Keycloak.
4. **Configure the SCIM for Keycloak**:
- Connect to the `SCIM Administration Console` to synchronize users and groups from Keycloak to Kestra.

- Enable SCIM for the realm.

- Disable `Bulk` and `Password synchronization` — these operations are not supported by Kestra.
5. **Create a SCIM Client**:
- Navigate to the `Remote SCIM Provider` section
- Fill the `Base URL` field with your Kestra `SCIM Endpoint`:

- Fill the `Authentication` with your Kestra `Secret Token`:

6. Toggle **Enabled** in the Kestra Provisioning Integration to start syncing users and groups from Keycloak to Kestra.
## Additional resources
- [SCIM for Keycloak Documentation](https://scim-for-keycloak.de/documentation/administration/scim-client)
---
# Microsoft Entra ID SCIM Provisioning in Kestra
URL: https://kestra.io/docs/enterprise/auth/scim/microsoft-entra-id
> Set up SCIM provisioning with Microsoft Entra ID. Automatically sync users and groups from Entra ID to Kestra for streamlined user management.
Sync users and groups from Microsoft Entra ID to Kestra using SCIM.
## Prerequisites
- **Microsoft Entra ID Account**: An account with administrative privileges is required to configure SCIM provisioning.
- **Enable multi-tenancy in Kestra**: Tenants must be enabled in Kestra to support SCIM provisioning. You can enable tenants by setting the `kestra.ee.tenants.enabled` configuration property to `true`:
```yaml
kestra:
ee:
tenants:
enabled: true
```
## Kestra SCIM setup: create a new provisioning integration
1. Go to **Instance Owner**, select your tenant from the sidebar, open **IAM**, and click the **SCIM Provisioning** tab.
2. Click **+ Create**.
3. Fill in the following fields:
- **Name**: Enter a name for the provisioning integration.
- **Description**: Provide a brief description of the integration.
- **Provisioning Type**: Only SCIM 2.0 is supported — leave the default selection and click **Save**.
These steps generate a SCIM endpoint URL and a Secret Token. Save both — you will need them in the next steps.

The endpoint should look as follows:
```plaintext
https:///api/v1//integrations/integration_id/scim/v2
```
The Secret Token is a long string (approx. 200 characters) used to authenticate requests from Microsoft Entra ID to Kestra.
### Enable or disable SCIM integration
You can disable or remove the SCIM integration at any time. When disabled, all incoming requests to that endpoint are rejected.
:::alert{type="info"}
You can disable the integration while configuring Entra ID, then enable it once setup is complete.
:::
### IAM role and service account
When creating a new Provisioning Integration, Kestra will automatically create two additional objects:
1. Role `SCIMProvisioner` with the following permissions:
- `GROUPS`: `CREATE`, `READ` `UPDATE`, `DELETE`
- `USERS`: `CREATE`, `READ`, `UPDATE`
- `BINDINGS`: `CREATE`, `READ`, `UPDATE`, `DELETE`

2. Service Account with an API Token which was previously displayed as the Secret Token for the integration:

:::alert{type="info"}
Why the `SCIMProvisioner` role doesn't have the `DELETE` permission for `USERS`? This is because you cannot delete a user through our SCIM implementation. Users are global and SCIM provisioning is per tenant. When we receive a `DELETE` query for a user, we remove their tenant access but the user itself remains in the system.
:::
## Microsoft Entra ID SCIM setup
### 1. Register Kestra as an enterprise application
- Navigate to **Microsoft Entra ID → Enterprise Applications**.
- Click **+ New application** to create a custom application. Name it "KestraSCIM" or any relevant name.

### 2. Configure SCIM provisioning
- Go to the newly created Kestra application.
- Select **Provisioning** and set the Provisioning Mode to **Automatic**.
- Enter the SCIM endpoint URL and Secret Token from Kestra: paste the SCIM endpoint into the **Tenant URL** field and the token into the **Secret Token** field.
- Click **Test Connection**, then **Save**.

### 3. Map user and group attributes
After entering and saving the **Admin Credentials** for the SCIM provisioning connection in Microsoft Entra ID — the Tenant URL and Secret Token — Azure will **enable the `Mappings` section** under the Provisioning settings.
The **Mappings** section allows you to define how user and group attributes should flow between Microsoft Entra ID and Kestra.
#### SCIM schema support in Kestra
Kestra adheres to the [SCIM 2.0 specification (RFC 7643)](https://datatracker.ietf.org/doc/html/rfc7643#section-4), specifically supporting the following resource types:
- **User Resource**:
- Example attributes: `userName`, `name.givenName`, `name.familyName`, `emails`, `active`
- **Group Resource**:
- Example attributes: `displayName`, `members`
#### Retrieve supported schemas
Kestra exposes SCIM resource schemas via its `/Schemas` endpoint at the SCIM URL. This allows Microsoft Entra ID to discover the required attributes automatically.
```plaintext
GET /api/v1//integrations//scim/v2/Schemas
```
:::alert{type="info"}
Replace `` with your actual tenant, and `` with your actual Kestra SCIM integration ID.
:::
This endpoint returns a list of supported schemas and their attributes. Use it as a reference when configuring attribute mappings in Entra ID.
#### Configure user and group mappings
To configure mappings:
1. Go to:
**Microsoft Entra Admin Center** → **Enterprise Applications** → *Your Kestra App* → **Provisioning** → **Mappings**
2. Configure attribute mappings:
- **For Users**:
- Map source attributes such as `userPrincipalName`, `mail` to their SCIM equivalents.
- **For Groups**:
- Map attributes such as `displayName`
- Ensure group `members` are synchronized properly.
3. Refer to the `/Schemas` endpoint response from Kestra to guide accurate mapping.
4. Use attribute expressions or transformations in Entra ID if needed (e.g., to format names or emails).
:::alert{type="info"}
By default, Azure will pre-populate the mapping with many Microsoft Entra ID attributes. You may need to **remove or simplify** some of these mappings if synchronization issues occur with users or groups in Kestra.
:::
#### Test the configuration
After mappings are configured:
- Trigger a **manual provisioning cycle** from the **Provisioning** tab.
- Verify that **users and groups** are correctly created or updated in Kestra.
- Review **provisioning logs** in Entra ID for any errors or warnings.
### 4. Enable provisioning
Enable the provisioning integration toggle in the Kestra UI to start syncing users and groups from Microsoft Entra ID.
## Additional resources
- [Microsoft Entra ID SCIM Documentation](https://docs.microsoft.com/en-us/azure/active-directory/app-provisioning/)
---
# Okta SCIM Provisioning in Kestra Enterprise
URL: https://kestra.io/docs/enterprise/auth/scim/okta
> Enable SCIM provisioning with Okta. Learn how to automatically synchronize Okta users and groups with your Kestra Enterprise instance.
Sync users and groups from Okta to Kestra using SCIM.
## Prerequisites
- **Okta Account**: An account with administrative privileges is required to configure SCIM provisioning.
- **Enable multi-tenancy in Kestra**: Tenants must be enabled in Kestra to support SCIM provisioning. You can enable tenants by setting the `kestra.ee.tenants.enabled` configuration property to `true`:
```yaml
kestra:
ee:
tenants:
enabled: true
```
:::alert{type="info"}
Tenants are enabled by default. Please refer to the [Migration Guide](../../../../11.migration-guide/v0.23.0/tenant-migration-ee/index.md) to assist with upgrading.
:::
## Kestra SCIM setup: create a new provisioning integration
1. Go to **Instance Owner**, select your tenant from the sidebar, open **IAM**, and click the **SCIM Provisioning** tab.
2. Click **+ Create**.
3. Fill in the following fields:
- **Name**: Enter a name for the provisioning integration.
- **Description**: Provide a brief description of the integration.
- **Provisioning Type**: Only SCIM 2.0 is supported — leave the default selection and click **Save**.
These steps generate a SCIM endpoint URL and a Secret Token. Save both — you will need them in the next steps.

The endpoint should look as follows:
```plaintext
https:///api/v1//integrations/integration_id/scim/v2
```
The Secret Token is a long string (approx. 200 characters) used to authenticate requests from Okta to Kestra.
### Enable or disable SCIM integration
You can disable or remove the SCIM integration at any time. When disabled, all incoming requests to that endpoint are rejected.
:::alert{type="info"}
You can disable the integration while configuring Okta, then enable it once setup is complete.
:::
### IAM role and service account
When creating a new Provisioning Integration, Kestra will automatically create two additional objects:
1. Role `SCIMProvisioner` with the following permissions:
- `GROUPS`: `CREATE`, `READ` `UPDATE`, `DELETE`
- `USERS`: `CREATE`, `READ`, `UPDATE`
- `BINDINGS`: `CREATE`, `READ`, `UPDATE`, `DELETE`

2. Service Account with an API Token which was previously displayed as the Secret Token for the integration:

:::alert{type="info"}
Why the `SCIMProvisioner` role doesn't have the `DELETE` permission for `USERS`? This is because you cannot delete a user through our SCIM implementation. Users are global and SCIM provisioning is per tenant. When we receive a `DELETE` query for a user, we remove their tenant access but the user itself remains in the system.
:::
## Okta SCIM setup
1. Navigate to Okta Admin Console → Applications → Applications and click **Create App Integration**:
- Sign-in Method: **OIDC - OpenID Connect**
- Application Type: Web Application
- Name: `Kestra`
- Grant Type: Client Credentials
- Sign-in redirect URIs: `http:///oauth/callback/okta`
- Sign-out redirect URIs: `http:///logout`
Note the client ID and client secret once the application is created.

2. Add the client credentials to your Kestra Micronaut configuration, and enter the SCIM endpoint URL and token provided by Kestra:
```yaml
micronaut:
security:
oauth2:
enabled: true
clients:
okta:
client-id: “CLIENT_ID”
client-secret: “CLIENT-SECRET”
openid:
issuer: “https://{okta-account}.okta.com/”
```
3. In Okta, navigate to **Applications → Applications → Browse App Catalog**, search for **SCIM 2.0**, and select **SCIM 2.0 Test App (OAuth Bearer Token)**. In Sign-in options, select **Secure Web Authentication**, then click **Done**. Select the new integration, open the **Provisioning** tab, and enter the Kestra SCIM endpoint URL in the Base URL field and the Secret Token in the **OAuth Bearer Token** field. Click **Test API Credentials** to verify.

4. Select **Push Groups** and choose the groups to sync to Kestra. Run a test to verify the mappings.
5. Enable the provisioning integration toggle in the Kestra UI to begin automatic synchronization.
## Additional resources
- [Okta SCIM Documentation](https://developer.okta.com/docs/reference/scim/)
---
# Service Accounts in Kestra Enterprise: CI/CD Auth
URL: https://kestra.io/docs/enterprise/auth/service-accounts
> Create and manage Service Accounts in Kestra. Securely authenticate external applications and CI/CD pipelines with programmatic access tokens.
Service accounts represent applications or CI/CD systems that access Kestra programmatically, without a password or UI access. Each service account has a name, an optional description, an optional group assignment, and a list of roles that grant it permissions to specific resources.
## Service accounts vs. users
Service accounts have no password and no access to the Kestra UI — they authenticate exclusively via API token. Users, by contrast, can interact with both the UI and the API using a password or an API token.
## Creating a service account
To create a new service account, go to **IAM** in the sidebar, open the **Service Accounts** tab, and click **Create**. Fill in the name and optional description, then click **Save**.
Once the service account is created, switch to the **Access** tab, click **Add**, and select the role to assign.
To generate an API token, click **Create API Token** in the service account details. You can configure the token to expire after a set period or never expire. The **Extended** toggle (disabled by default) automatically resets the expiry each time the token is used. Click **Generate**, then copy the token immediately — it is shown only once.
## Users, service accounts, and API tokens
You can create an API token for a regular user as well. While service accounts are recommended for programmatic access from CI/CD or external applications, user API tokens are useful when you want programmatic actions to be tracked and audited against a specific person.
The key difference: a service account has no password or personal information and is authenticated exclusively with an API token. A user can interact with both the UI and the API, using either a password or an API token.
## Allocating service accounts to groups
Each service account can be attached to one or more groups, such as a “Bots” group that centrally governs programmatic access for CI/CD across multiple projects with a single role. This simplifies managing Terraform, GitHub Actions, or other external application access in one place.
## CLI authentication
When using the Kestra CLI, you can authenticate with either an API token or a username and password:
1. Use `--api-token` to authenticate with a service account token:
```bash
./kestra namespace files update prod scripts . \
--server=https://demo.kestra.io --api-token yourtoken
```
2. Use `--user` to authenticate with Basic Auth credentials:
```bash
./kestra namespace files update prod scripts . \
--server=https://demo.kestra.io --user=rick.astely@kestra.io:password42
```
## Service account name convention
Follow the DNS naming convention when naming service accounts. The `name` property must:
- contain at most 63 characters
- contain only lowercase alphanumeric characters or hyphens (`-`)
- start with an alphanumeric character
- end with an alphanumeric character.
Examples:
- ✅ `my-service-account` is a valid name
- ✅ `my-service-account-1` is a valid name
- ❌ `MY_SERVICE_ACCOUNT` is not a valid name because it contains uppercase characters and underscores
- ❌ `myServiceAccount` is not a valid name because it contains uppercase characters and camel case
- ❌ `my-service-account-` is not a valid name because it ends with a hyphen.
Kestra uses the DNS-style naming convention so that service account names remain valid across contexts where they may be forwarded, such as Kubernetes pod labels.
---
# Single Sign-On in Kestra: Providers and Setup
URL: https://kestra.io/docs/enterprise/auth/sso
> Enable Single Sign-On (SSO) in Kestra Enterprise. Configure OIDC authentication with providers like Google, Microsoft, Okta, and Keycloak.
Single Sign-On (SSO) lets users authenticate to Kestra using an external identity provider such as Google, Microsoft, Okta, or Keycloak.
## Configure OIDC
Kestra supports SSO using the OpenID Connect (OIDC) protocol, a simple identity layer built on top of OAuth 2.0. Enable OIDC in your configuration file:
```yaml
micronaut:
security:
oauth2:
enabled: true
clients:
oidc-provider:
client-id: "{{ clientId }}"
client-secret: "{{ clientSecret }}"
openid:
issuer: "{{ issuerUrl }}"
```
Replace `oidc-provider` with your chosen provider's name, `{{ clientId }}` with your client ID, `{{ clientSecret }}` with your client secret, and `{{ issuerUrl }}` with your issuer URL.
For more configuration details, refer to the [Micronaut OIDC configuration guide](https://micronaut-projects.github.io/micronaut-security/latest/guide/#openid-configuration).
## Provider guides
Provider-specific setup guides:
- [Google](/docs/enterprise/auth/sso/google-oidc)
- [Microsoft](/docs/enterprise/auth/sso/microsoft-oidc)
- [Keycloak](/docs/enterprise/auth/sso/keycloak)
- [Okta](/docs/enterprise/auth/sso/okta)
- [authentik](/docs/enterprise/auth/sso/authentik)
- [LDAP](/docs/enterprise/auth/sso/ldap)
---
# Set Up authentik SSO in Kestra
URL: https://kestra.io/docs/enterprise/auth/sso/authentik
> Configure authentik SSO for Kestra. Enable seamless user authentication using authentik as your OpenID Connect provider.
Set up authentik as an OIDC provider for Kestra authentication. In conjunction with SSO, see the [authentik SCIM provisioning guide](../../scim/authentik/index.md).
## Install authentik
authentik provides a simple docker-compose installer for testing purposes. Follow [the instructions](https://docs.goauthentik.io/docs/installation/docker-compose) and click on the [initial setup URL](http://docker.for.mac.localhost:9000/if/flow/initial-setup/) to create your first user.

## Create application and SSO provider in authentik
On the left-hand side, select **Applications → Applications**. For simplicity, we’ll use the **Create with Wizard** button, as this will create both an application and a provider.

On the **Application Details** screen, fill in the application **Name** and **Slug**. Set both to `kestra` and click **Next**.

On the **Provider Type** screen, select **OAuth2/OIDC** and click **Next**.

On the **Provider Configuration** screen:
1. In the **Authentication flow** field, select “default-authentication-flow (Welcome to authentik!)”.
2. In the **Authorization flow** field, select “default-provider-authorization-explicit-consent (Authorize Application)”.

3. Keep the Client type as **Confidential**. Under the **Redirect URIs/Origins (RegEx)**, enter your Kestra host's `/oauth/callback/authentik` endpoint in the format `http://:/oauth/callback/authentik` (e.g., http://localhost:8080/oauth/callback/authentik) and click **Submit**.

Note the `Client ID` and `Client Secret` for the next step.
## Configure authentik SSO in Kestra
Add the following to your `micronaut` configuration:
```yaml
micronaut:
security:
oauth2:
enabled: true
clients:
authentik:
clientId: "CLIENT_ID"
clientSecret: "CLIENT_SECRET"
openid:
issuer: "http://localhost:9000/application/o/kestra/"
```
You may need to adjust the above `issuer` URL if you named your application something other than `kestra`. Update that URL to match your application name: `http://localhost:9000/application/o//`.
## Configure a default role for SSO users
SSO users need a default role for initial access in Kestra. Add the following to `kestra.security`:
```yaml
kestra:
security:
defaultRole:
name: default_admin_role
description: "Default Admin Role"
permissions:
FLOW:
- VIEW
- LIST
- CREATE
- UPDATE
- DELETE
- EXECUTE
- DISABLE
- ENABLE
- VALIDATE
- EXPORT
- IMPORT
EXECUTION:
- VIEW
- LIST
- UPDATE
- DELETE
- RESTART
- KILL
- REPLAY
- PAUSE
- RESUME
- CHANGE_LABELS
- ACCESS_LOGS
- ACCESS_OUTPUTS
- ACCESS_FILES
- EXPORT
- UNQUEUE
- FORCE_RUN
- FOLLOW
NAMESPACE:
- VIEW
- LIST
- CREATE
- UPDATE
- DELETE
- MANAGE_FILES
- EXPORT_PLUGIN_DEFAULTS
- IMPORT_PLUGIN_DEFAULTS
SECRET: ["VIEW", "LIST", "UPDATE", "DELETE"]
KVSTORE: ["VIEW", "LIST", "CREATE", "UPDATE", "DELETE"]
BLUEPRINT: ["VIEW", "LIST", "CREATE", "UPDATE", "DELETE"]
ROLE: ["VIEW", "LIST", "CREATE", "UPDATE", "DELETE"]
GROUP: ["VIEW", "LIST", "CREATE", "UPDATE", "DELETE", "MANAGE_MEMBERS"]
USER: ["VIEW", "LIST", "CREATE", "UPDATE", "DELETE", "MANAGE_GROUP_MEMBERSHIP"]
BINDING: ["VIEW", "LIST", "CREATE", "DELETE"]
AUDITLOG: ["VIEW", "LIST", "EXPORT"]
ee:
tenants:
enabled: true
defaultTenant: false
```
:::alert{type="info"}
Place `defaultRole` under `kestra.security`, not under `micronaut.security`. The example above grants broad access — adjust the action lists to match the permissions your users actually need in production.
:::
---
# Set Up Google OIDC SSO in Kestra
URL: https://kestra.io/docs/enterprise/auth/sso/google-oidc
> Set up Google OIDC SSO for Kestra. Authenticate users with their Google accounts using OpenID Connect for secure and easy access.
Authenticate Kestra users with their Google accounts using Google Identity Platform and OIDC.
## Prerequisites
- A Google Cloud project with billing enabled.
- Sufficient permissions to configure Identity Platform and manage identity providers.
See the [Google OIDC setup documentation](https://cloud.google.com/identity-platform/docs/web/oidc) for reference.
## Step 1: Enable Identity Platform in Google Cloud
1. Go to the [Identity Platform page](https://console.cloud.google.com/identity) in the Google Cloud Console.
2. Confirm the correct project is selected.
## Step 2: Add an OIDC Provider in Google Cloud
1. In the Identity Platform menu, select **Providers**.
2. Click **Add a Provider** and choose **OpenID Connect**.

3. Configure the OIDC Provider:
- **Grant type**: Select the Code Flow grant type.
- **Provider Name**: Enter a display name for the OIDC provider.
- **Client ID**: Enter the **Client ID** obtained from Google.
- **Client Secret**: Enter the **Client Secret** associated with the Client ID.
- **Issuer URL**: Provide the **Issuer URL** (e.g., `https://accounts.google.com`).
- **Scopes**: Specify any additional scopes required by your application.

4. Click **Save** to add the provider.
## Step 3: Configure Kestra
Add the following to your [Kestra Security and Secrets configuration](../../../../configuration/05.security-and-secrets/index.md):
```yaml
micronaut:
security:
oauth2:
enabled: true
clients:
google:
client-id: "{{ clientId }}"
client-secret: "{{ clientSecret }}"
openid:
issuer: 'https://accounts.google.com'
```
Replace `clientId` and `clientSecret` with the values from the Google Identity Platform, then restart Kestra.
## Additional resources
- [Managing SAML and OIDC Providers Programmatically](https://cloud.google.com/identity-platform/docs/managing-providers-programmatically)
- [Identity Platform Documentation](https://cloud.google.com/identity-platform/docs)
---
# Set Up Keycloak SSO in Kestra
URL: https://kestra.io/docs/enterprise/auth/sso/keycloak
> Integrate Keycloak SSO with Kestra. Configure OpenID Connect authentication to manage user access via your Keycloak identity provider.
Set up Keycloak as an OIDC provider for Kestra authentication. In conjunction with SSO, see the [Keycloak SCIM provisioning guide](../../scim/keycloak/index.md).
## Start a Keycloak service
If you don't have a Keycloak server already running, you can use a managed service like [Cloud IAM](https://app.cloud-iam.com).
You can follow the steps described in the [Keycloak tutorial documentation](https://documentation.cloud-iam.com/get-started/complete-tutorial.html) to deploy a managed Keycloak cluster for free.
## Configure Keycloak client
Once in Keycloak, create a new client:


Set `https://{{ yourKestraInstanceURL }}/oauth/callback/keycloak` as the valid redirect URI and `https://{{ yourKestraInstanceURL }}/logout` as the valid post-logout redirect URI.

## Kestra Configuration
```yaml
micronaut:
security:
oauth2:
enabled: true
clients:
keycloak:
client-id: "{{clientId}}"
client-secret: "{{clientSecret}}"
openid:
issuer: "https://{{keyCloakServer}}/realms/{{yourRealm}}"
endpoints:
logout:
get-allowed: true
```
You can retrieve the `clientId` and `clientSecret` via the Keycloak user interface.


Set a default role in your [Kestra Security and Secrets configuration](../../../../configuration/05.security-and-secrets/index.md) to assign initial permissions to new SSO users.
```yaml
kestra:
security:
defaultRole:
name: Editor
description: Default Editor role
permissions:
FLOW:
- VIEW
- LIST
- CREATE
- UPDATE
- DELETE
- EXECUTE
- DISABLE
- ENABLE
- VALIDATE
- EXPORT
- IMPORT
EXECUTION:
- VIEW
- LIST
- UPDATE
- DELETE
- RESTART
- KILL
- REPLAY
- PAUSE
- RESUME
- CHANGE_LABELS
- ACCESS_LOGS
- ACCESS_OUTPUTS
- ACCESS_FILES
- EXPORT
- UNQUEUE
- FORCE_RUN
- FOLLOW
```
:::alert{type="info"}
Depending on your Keycloak configuration, you may need to adjust the issuer URL.
:::
For more configuration details, refer to the [Keycloak OIDC configuration guide](https://guides.micronaut.io/latest/micronaut-oauth2-keycloak-gradle-java.html).
## Logout and session termination
By default, clicking logout in Kestra clears the Kestra session but does not terminate the underlying Keycloak SSO session. The next login will skip the username and password prompt and automatically re-authenticate the user via the still-active Keycloak session.
To fully terminate the Keycloak session on logout, add `end-session` to your configuration:
```yaml
micronaut:
security:
oauth2:
enabled: true
clients:
keycloak:
client-id: "{{clientId}}"
client-secret: "{{clientSecret}}"
openid:
issuer: "https://{{keyCloakServer}}/realms/{{yourRealm}}"
end-session:
enabled: true
url: "https://{{keyCloakServer}}/realms/{{yourRealm}}/protocol/openid-connect/logout"
endpoints:
logout:
get-allowed: true
```
If you also want to force users to re-enter their credentials on every Kestra login — even when an active Keycloak SSO session exists — add `?prompt=login` to the authorization URL:
```yaml
micronaut:
security:
oauth2:
clients:
keycloak:
openid:
authorization:
url: "https://{{keyCloakServer}}/realms/{{yourRealm}}/protocol/openid-connect/auth?prompt=login"
```
These two settings are independent: `end-session` controls what happens on logout, while `prompt=login` controls what happens on the next login attempt.
## Manage groups via OIDC claims
If you are unable to use [SCIM with Keycloak](../../scim/keycloak/index.md), you can configure Kestra to source user groups from OIDC claims. In this setup, Keycloak acts as the single source of truth for user group membership. This method requires creating a `groups` client scope that exposes group membership via a claim in the ID Token.
### Create a groups client scope
In Keycloak, go to **Client Scopes** and click **Create Client Scope**. Name it `groups`, set Type to **Default**, and keep Protocol as **OpenID Connect**.

### Add a group membership mapper
In the newly created `groups` scope, go to the **Mappers** tab and click **Configure a new mapper**.

Select **Group Membership** from the list of available mapper types.

Configure the mapper with the following settings:
- **Name**: `groups`
- **Token Claim Name**: `groups`
- **Full group path**: Off
- **Add to ID token**: On

### Add the client scope to your client
Go to **Clients**, select your Kestra client, and add the `groups` client scope.

### Configure Kestra
Update your Micronaut configuration to include `groups` in the scopes:
```yaml
micronaut:
security:
oauth2:
enabled: true
clients:
keycloak:
client-id: "{{clientId}}"
client-secret: "{{clientSecret}}"
openid:
issuer: "https://{{keyCloakServer}}/realms/{{yourRealm}}"
scopes: ["openid", "profile", "email", "groups"]
endpoints:
logout:
get-allowed: true
```
Then configure Kestra to synchronize groups from the `groups` claim:
```yaml
kestra:
security:
oidc:
groups-claim-path: "groups"
```
Once configured, Kestra will source user groups from the `groups` claim in the ID Token, with Keycloak as the single source of truth.
---
# LDAP Authentication in Kestra: Directory Login and Group Sync
URL: https://kestra.io/docs/enterprise/auth/sso/ldap
> Enable LDAP authentication in Kestra. Use your LDAP directory for user login, group synchronization, or both — including alongside an existing SSO provider.
Enable LDAP authentication to authenticate users against your existing directory, sync group memberships, or both. You can also use LDAP solely for group sync while keeping an existing SSO provider for login.
## Configure LDAP authentication
:::alert{type="warning"}
LDAP is a licensed feature. If `micronaut.security.ldap.default` is configured but your license does not include LDAP, Kestra will refuse to start with the error: `LDAP is not supported by your license`. Contact your Kestra account team to enable it.
:::
## What is LDAP
Lightweight directory access protocol (LDAP) allows applications to quickly query user information. Organizations use directories to store usernames, passwords, email addresses, and other static data. LDAP is an open, vendor-neutral protocol for accessing and managing that data.
With Kestra, you can use an existing LDAP directory to authenticate users and sync them to groups with specific access permissions.
## Configuration
LDAP is configured under the security context of your [Kestra Security and Secrets configuration](../../../../configuration/05.security-and-secrets/index.md) file.
[LDAP with Micronaut](https://micronaut-projects.github.io/micronaut-security/4.11.3/guide/#ldap) supports `context`, `search`, and `groups` as core configuration properties. These define the connection context, user attribute mapping, and group filtering needed to synchronize users and their group memberships with Kestra.
The `user-attributes` section maps LDAP attributes such as `givenName`, `sn`, and `mail` to Kestra user properties (first name, last name, and email).
The `mode` property controls how Kestra uses the LDAP connection:
| Mode | Description |
|---|---|
| `AUTHENTICATION` | LDAP handles user login only. No group sync. **This is the default.** |
| `AUTHENTICATION_AND_GROUP_SYNC` | LDAP handles both user login and group membership sync. |
| `GROUP_SYNC_ONLY` | LDAP is used only to resolve group memberships. Users log in via an existing SSO provider. |
The examples below extend the base Micronaut LDAP configuration with these Kestra-specific mappings.
### Unix configuration
```yaml
micronaut:
security:
ldap:
default:
mode: AUTHENTICATION_AND_GROUP_SYNC # or AUTHENTICATION to skip group sync
user-attributes:
firstName: givenName
lastName: sn
email: mail
context:
server: "ldap://localhost:389"
manager-dn: "cn=admin,dc=example,dc=org"
manager-password: "LDAP_ADMIN_PASSWORD"
search:
base: "ou=users,dc=example,dc=org"
filter: "(mail={0})"
attributes:
- "uid"
- "givenName"
- "sn"
- "mail"
groups:
enabled: true
base: "ou=groups,dc=example,dc=org"
filter: "{&(objectClass=posixGroup)(memberUid={0})}"
filter-attribute: uid
attribute: cn
```
### Windows configuration
```yaml
micronaut:
security:
ldap:
default:
enabled: true
mode: AUTHENTICATION_AND_GROUP_SYNC # or AUTHENTICATION to skip group sync
user-attributes:
firstName: givenName
lastName: sn
email: userPrincipalName
context:
server: "ldaps://:636" # ldap://:389 for non-TLS
manager-dn: "CN=********,CN=Users,DC=domain,DC=local"
manager-password: "********"
search:
base: "DC=domain,DC=local"
filter: "(userPrincipalName={0})"
attributes:
- "sAMAccountName"
- "givenName"
- "sn"
- "userPrincipalName"
groups:
enabled: true
base: "DC=domain,DC=local"
filter: "(&(objectClass=group)(member={0}))"
filter-attribute: dn
attribute: cn
```
Key points for Windows Active Directory:
- **Login format**: the `userPrincipalName` filter requires users to log in with their full UPN, e.g. `john@domain.local`. If your users expect to log in with just their short username (e.g. `john`), change the filter to `(sAMAccountName={0})` and update the `email` attribute mapping accordingly.
- **Search base**: setting `search.base` and `groups.base` to the root domain (`DC=domain,DC=local`) covers users and groups across all OUs. Narrow these to a specific OU (e.g. `OU=Engineering,DC=domain,DC=local`) if you want to restrict access to a subset of your directory.
- **Group filter attribute**: AD `member` attributes store full DNs, so `filter-attribute: dn` is required. Without it, Micronaut defaults to `cn` and group membership lookups will silently return no results.
- **TLS**: use `ldaps://` on port 636 in production. Plain `ldap://` on port 389 sends credentials in cleartext. If your AD uses a self-signed certificate, you must add it to the JVM truststore or configure certificate trust in your Kestra deployment.
#### Finding Windows Active Directory values
Use the following PowerShell commands on your Windows domain controller to look up the values needed for the configuration above.
**LDAP server hostname** (`context.server`)
```powershell
(Get-ADDomainController).HostName
```
Use the returned hostname as `ldaps://:636` for TLS or `ldap://:389` for non-TLS.
**Manager DN** (`context.manager-dn`)
```powershell
([adsisearcher]"(sAMAccountName=Administrator)").FindOne().Properties.distinguishedname
```
Replace `Administrator` with the service account you intend to use as the bind user. The returned distinguished name (DN) is the value for `manager-dn`.
**User distinguished name**
To look up the DN of a specific user (useful for verifying your `search.base`):
```powershell
Get-ADUser -Identity "JohnDoe" | Select-Object Name, DistinguishedName
```
**Groups for a user**
To list the groups a user belongs to (useful for planning your `groups.base` and `groups.filter`):
```powershell
Get-ADPrincipalGroupMembership -Identity "JohnDoe" | Select-Object Name, DistinguishedName
```
**Members of a group**
To verify the members of a specific group:
```powershell
Get-ADGroupMember -Identity "CN=Auto,OU=Distro,OU=Groups,DC=kestra,DC=local" | Select-Object sAMAccountName, Name
```
Replace the identity string with the DN of your target group.
### Group sync with SSO (GROUP_SYNC_ONLY)
If your users already authenticate via SSO, Basic auth, or Passwordless, you can use LDAP solely to resolve group memberships without changing how users log in. Set `mode: GROUP_SYNC_ONLY` and configure the `groups` block. No `user-attributes` mapping is required.
```yaml
micronaut:
security:
ldap:
default:
mode: GROUP_SYNC_ONLY
context:
server: "ldap://localhost:389"
manager-dn: "cn=admin,dc=kestra,dc=io"
manager-password: "LDAP_ADMIN_PASSWORD"
search:
base: "ou=users,dc=kestra,dc=io"
filter: "(mail={0})"
groups:
enabled: true
base: "ou=groups,dc=kestra,dc=io"
filter: "(member={0})"
attribute: cn
```
With this configuration:
- Users log in using their SSO provider. LDAP credentials are never checked.
- At each login, Kestra queries the LDAP directory for the user's group memberships and merges them with any groups sourced from OIDC claims.
- Groups found in LDAP are synced to Kestra using the same rules as standard LDAP group sync — new groups are created automatically, and membership is updated on login.
Two `groups` properties control how Kestra reads group entries from the directory:
- `filter`: the LDAP search filter used to find groups for a user. `{0}` is replaced with the user's distinguished name (DN).
- `attribute`: the attribute on the group entry whose value becomes the Kestra group name. Defaults to `cn`.
- `filter-attribute`: the user entry attribute substituted into `{0}` in the group filter. Use `dn` for directories that store full DNs in group membership attributes (common in Active Directory). Use `uid` for POSIX-style directories.
:::alert{type="info"}
`GROUP_SYNC_ONLY` mode requires that the user already exists in Kestra (created on first login). LDAP group sync fires on every subsequent login.
:::
:::alert{type="warning"}
If the LDAP server is unreachable or misconfigured, group sync fails silently — the user logs in successfully but receives no LDAP-sourced groups. Check server connectivity and `groups` configuration if group assignments are not appearing after login.
:::
## LDAP users in Kestra
On first login with LDAP, Kestra validates the user's credentials against the directory and creates a corresponding user account. If an account already exists, the user authenticates with their LDAP credentials.
If the user belongs to any groups in the directory, those groups are created in Kestra if they don't exist, and the user is added to each. Group assignments sync only at login — if a user is added to a group in LDAP after their first login, they must log out and back in to pick up the new membership. Any user authenticated via LDAP shows `LDAP` in the **Login & API Tokens** column on the **IAM → Users** page.

Any updates to a user and their group access on the LDAP server will update in Kestra at the next synchronization (typically at the next login).
:::alert{type="info"}
Users who log in via SSO with `GROUP_SYNC_ONLY` mode show their SSO provider in the **Login & API Tokens** column on the **IAM → Users** page, not `LDAP`. The LDAP connection is used only to resolve group memberships in the background.
:::
:::alert{type="warning"}
If a user is deleted from the LDAP server, they will lose access to Kestra at the next synchronization or login attempt.
:::
---
# Set Up Microsoft OIDC SSO in Kestra
URL: https://kestra.io/docs/enterprise/auth/sso/microsoft-oidc
> Configure Microsoft OIDC SSO for Kestra. Enable users to sign in with their Microsoft Entra ID (Azure AD) credentials using OpenID Connect.
Authenticate Kestra users with their Microsoft Entra ID credentials using OIDC.
## Configure Microsoft OIDC
To authenticate with Microsoft, add the following to your Micronaut configuration:
```yaml
micronaut:
security:
oauth2:
enabled: true
clients:
microsoft:
client-id: "{{ clientId }}"
client-secret: "{{ clientSecret }}"
openid:
issuer: 'https://login.microsoftonline.com/common/v2.0/'
```
To get your `client-id` and `client-secret`, refer to the [Microsoft Documentation](https://learn.microsoft.com/en-us/entra/identity-platform/v2-protocols-oidc).
## Using Microsoft Entra ID as an OIDC provider
### Create an Enterprise Application
1. Visit the [Azure portal](https://portal.azure.com/).
2. Select **Microsoft Entra ID**.
3. Navigate to **App registrations**.
4. Click on **New registration** and provide the necessary details:
- Enter a name for your application.
- Set **Supported account types** (e.g., "Default Directory only - Single tenant").
- Under **Redirect URI**, select *Web* and enter `https://{{ url }}/oauth/callback/microsoft`. Use `https` and your actual webserver URL.
### Generate client secret
1. Go to **Certificates & secrets**.
2. Under **Client secrets**, click on **New client secret**.
3. Copy the generated secret and use it in the `{{ clientSecret }}` field in your [Security and Secrets configuration](../../../../configuration/05.security-and-secrets/index.md).
### Kestra configuration
- Copy the **Application (client) ID** from the **Overview** section and use it as your `{{ clientId }}`.
- In the **Endpoints** section, locate the **OpenID Connect metadata document** URL. Remove the `.well-known/openid-configuration` suffix, and use the remaining base URL as your `{{ issuerUrl }}`.
The final URL should look like `https://login.microsoftonline.com/{{ directory }}/v2.0/`.
Example configuration:
```yaml
micronaut:
security:
oauth2:
enabled: true
clients:
microsoft:
client-id: "{{ clientId }}"
client-secret: "{{ clientSecret }}"
openid:
issuer: '{{ issuerUrl }}'
```
Replace all placeholders with the values obtained from Entra ID.
---
# Set Up Okta OIDC SSO in Kestra
URL: https://kestra.io/docs/enterprise/auth/sso/okta
> Set up Okta OIDC SSO for Kestra. Securely authenticate users via Okta OpenID Connect for centralized access management.
## Prerequisites
- An Okta Developer Account or Organization with administrative access.
For more detail, refer to the [Okta OIDC setup documentation](https://help.okta.com/oie/en-us/content/topics/apps/apps_app_integration_wizard_oidc.htm).
## Step 1: Create an App Integration
Log in to your Okta account and select **Applications** from the left side menu.

Next, select **Create App Integration**, select **OIDC - OpenID Connect** as the sign-in method and **Web Application** as the application type. Select **Next** to configure the general settings of the new web app integration.

## Step 2: Configure the web app integration
In the General Settings, give your App integration a name and set your grant type. For this example, we are using Authorization Code. You can open **Advanced Settings** to configure more sensitive grants. Okta has several direct-auth API grants, such as OTP, OOB, MFA OTP, and MFA OOB that you can select only if necessary.

Here, you also set the **Sign-in redirect URIs** and **Sign-out redirect URIs** for your App integration. For this example connecting to Kestra, we set a Sign-in redirect URI as `http://localhost:8080/oauth/callback/okta` and sign-out as `http://localhost:8080/logout`, but you can customize this to your environment.
Further down the page, you can configure optional **Trusted Origins**, and then choose the **Assignments** and the access settings for the App integration.
We'll set the access to everyone in the organization, but you can set stricter access to only certain selected groups or skip for now. Lastly, we uncheck the setting to enable immediate access with Federation Broker Mode because we will give manual app access for this basic example. Click **Save**.

## Step 3: Add test user to Okta app integration
To create a test user in your Okta Directory to test your app integration, in your Okta Admin Dashboard, navigate to **Directory → People**. Select **Add Person**.

Enter user test details, including a password, and save the test user.
In the **Directory**, select the new user, and navigate to the **Applications** tab for the user and choose **Assign Applications**.

Select the Kestra application, enter the required details for the user, and click **Save**.
## Step 4: Connect to Kestra
After saving, Okta redirects you to your integration, where you can find your **Client ID** and **Client Secret**.

After copying your **Client ID** and **Client Secret**, switch to the **Sign On** tab. Under **OpenID Connect ID Token**, change the issuer from Dynamic to your Okta URL. Click **Save** and copy the URL for use in your [Kestra Security and Secrets configuration](../../../../configuration/05.security-and-secrets/index.md).

Add the following configuration to enable Okta as an OIDC provider:
```yaml
micronaut:
security:
oauth2:
enabled: true
clients:
okta:
client-id: "{{ clientId }}"
client-secret: "{{ clientSecret }}"
openid:
issuer: 'https://.okta.com'
```
- Replace `clientId` and `clientSecret` with the values copied from the Okta App integration.
- Replace `issuer` with your issuer URL from the application's sign-on settings.
- Restart Kestra to apply the changes and log in.
On restart, Okta appears as an available login method.

After logging in, go to **IAM → Users** to confirm the user shows both login methods in the **Login & API Tokens** column.

---
# Kestra Cloud: Getting Started
URL: https://kestra.io/docs/enterprise/cloud-getting-started
> Set up your Kestra Cloud instance, run your first flow, add secrets, invite your team, and connect to production.
Kestra Cloud is a fully managed Kestra instance run by the Kestra team. You build the pipelines; Kestra handles hosting, scaling, upgrades, SSO, and secrets.
## Two places you'll work
Kestra Cloud consists of two platforms:
- **[Console](https://console.kestra.io/ui/login):** where you create, monitor, and manage your instances and billing.
- **Your Kestra instance:** where you and your team build and run flows.
| | Console | Your Kestra instance |
|---|---|---|
| For | Managing instances and billing | Building and running flows |
| What | View instances, usage, and invoices; set up SSO; add Console users | Write flows, run executions, add secrets, invite teammates, configure IAM/RBAC, build Apps |
:::alert{type="info"}
Console users and instance users are separate. Adding someone in Console does not give them access to the instance, and vice versa. Invite people where they'll actually work (usually the instance). SSO is configured in Console per instance.
:::
## Step 1: Log in
Go to your instance URL, or sign in to **[Console](https://console.kestra.io/ui/login)** to find it. Sign-in is passwordless: enter the login code emailed to you.
If the code doesn't arrive, your email security filter is likely blocking it. Allowlist **noreply@kestra.io**, check spam, then request a new code.
Whoever creates the instance becomes its **Instance Owner**: the role that manages instances, users, and IAM across the whole instance.
:::alert{type="info"}
Instance Owner does not grant access to namespaces or data by default. Working with flows also requires a tenant role (see [Step 5](#step-5-invite-your-team)).
:::
## Step 2: Run your first flow from a blueprint
Three getting-started blueprints run successfully on the first execution with public data and no secrets required. Pick one based on your use case:
| Blueprint | Use for | What it does |
|---|---|---|
| [ELT pipeline](https://kestra.io/blueprints/getting-started-elt-pipeline) | Data pipelines | Public CSV → aggregate in DuckDB → log top products |
| [AI enrichment](https://kestra.io/blueprints/getting-started-ai-enrichment) | AI / LLM work | Fetch records → log a preview; add a key to summarize and classify |
| [Automation](https://kestra.io/blueprints/getting-started-automation) | Alerts / ops | Call an API → log the response; add a Slack alert and a trigger |
In your instance: go to **Flows → Create → Blueprints**, pick one, **Save**, then **Execute**. Open the run and review the **Logs**, the **Gantt** timeline, and each task's **Outputs**. Edit, execute, review the run: that's the core loop.
Hundreds more are available in the [blueprints library](https://kestra.io/blueprints). The [AI Copilot](../../ai-tools/01.ai-copilot/index.md) can also write a flow from a plain-English description.
Building with AI tools? The [Kestra MCP server](../../ai-tools/03.mcp-server/index.md) gives Claude, Cursor, and other tools live access to Kestra docs, plugins, and blueprints. Add [kestractl](https://github.com/kestra-io/kestractl) to generate and deploy flows from the CLI.
Prefer your editor? The [VS Code extension](../../version-control-cicd/05.vscode/index.md) edits, validates, and runs flows locally with instance-aware autocomplete and a topology preview.
## Step 3: Namespaces
Every flow belongs to a [namespace](../../05.workflow-components/02.namespace/index.md) (for example, `mycompany`): a folder that holds flows, secrets, files, and variables. Three rules to follow:
- **Create your own** namespace (for example, `dev`) and build there. Leave `system` alone; it holds Kestra's own maintenance flows.
- **Nest with `.` to share config.** `marketing.dev` and `marketing.prod` are two environments under `marketing`. Both inherit the parent's secrets, variables, and files, so shared config lives in one place.
- **Split `dev` from `prod`.** When flows matter, use separate namespaces and promote between them (see [Step 6](#step-6-go-to-production)) so only reviewed work reaches production.
## Step 4: Add your first secret
Store credentials as [secrets](../../06.concepts/04.secret/index.md) and reference them with `{{ secret('NAME') }}`. Never paste credentials directly into a flow.
To add a Slack webhook (used by all three getting-started blueprints):
1. In your instance: go to **Namespaces → your namespace → Secrets → Add**. Name the key `SLACK_WEBHOOK`.
2. In the blueprint's `notify` task, remove `disabled: true` and run it. You'll receive a Slack message.
Use the same pattern for a `DB_PASSWORD` to point the ELT blueprint at your own warehouse.
Secrets are namespace-scoped but readable across namespaces in the same tenant, so shared tokens can live in one place:
```yaml
message: "{{ secret('SHARED_TOKEN', namespace='shared') }}"
```
Cross-namespace reads are allowed by default. Restrict them with `allowedNamespaces` on the owning namespace.
[Policies](../02.governance/policies/index.md) set plugin values (host, credentials, region) once per namespace and apply them to every matching flow. Use them to avoid repeating auth config on every task.
## Step 5: Invite your team
Go to **Tenant → IAM → Invitations → Create**: enter their email and assign a role. For people building flows, **Developer** is the best fit (**Editor** and **Admin** also work). Avoid Instance Owner — that role manages the instance, not the data.
Leave the namespace field empty to apply the role to all namespaces, or name specific namespaces to limit access.
Related:
- [Invite colleagues](../03.auth/invitations/index.md)
- [Set up SSO](../03.auth/sso/index.md) (configured in [Console](https://console.kestra.io/ui/login))
## Step 6: Go to production
Once flows matter, stop editing production by hand. Build in `dev`, commit to Git, and let Git be the source of truth for `prod`. Common patterns — combinable:
| Method | Best for |
|---|---|
| [Git Sync](../../version-control-cicd/04.git/index.md) | Flows and namespace files: Kestra pulls `main` on a schedule or push |
| [Terraform](../../13.terraform/index.mdx) | Static config: secrets, variables, namespaces, roles |
| GitHub Actions / CI/CD | Validate on PR, deploy on merge: [validate flows](https://github.com/kestra-io/validate-flows-action-v2), [deploy flows](https://github.com/kestra-io/deploy-flows-action-v2), [deploy namespace files](https://github.com/kestra-io/deploy-namespace-files-action) |
No Git? [Promote](../02.governance/promote/index.md) copies a flow from one instance to another directly from the UI with a diff review before it lands. It moves the flow YAML only, so configure secrets and files per environment separately. See the [dev-to-prod guide](../../14.best-practices/1.from-dev-to-prod/index.md) for the full picture.
## What's next
In the order most active teams adopt them:
1. **Put it on a [schedule or trigger](../../05.workflow-components/07.triggers/index.mdx)** so it runs unattended.
2. **Wire a Slack alert** (Step 4) so you know when something breaks.
3. **Connect a second system**: your warehouse, dbt, an API.
4. **Sync from Git** (Step 6).
5. **Create a Kestra App.** An [App](../04.scalability/apps/index.md) gives stakeholders a dedicated URL to submit or download data and approve runs without touching flows directly.
Common first-week blockers:
- Credentials not reaching a task: [Secrets](../../06.concepts/04.secret/index.md)
- A worker that can't reach your network: [Worker Groups](../04.scalability/worker-group/index.md)
- Cloud auth or OIDC trust issues: [Credentials](../03.auth/credentials/index.md)
For help, reach the team at support@kestra.io, in the [Slack community](https://kestra.io/slack), or by replying to your onboarding email.
---
# Cloud & Enterprise FAQ: Licensing and Configuration
URL: https://kestra.io/docs/enterprise/ee-faq
> FAQ for Kestra Cloud and Enterprise. Find answers to common questions about licensing, configuration, session management, and enterprise features.
Frequently asked questions about the Cloud and Enterprise Edition of Kestra.
## My session expires too quickly. Is there a way to change the session expiration time?
Add the following Micronaut setting to your [Observability and Networking configuration](../../configuration/03.observability-and-networking/index.md) to set the session expiration time to 10 hours:
```yaml
environment:
KESTRA_CONFIGURATION: |
micronaut:
security:
token:
generator:
access-token:
expiration: 36000
cookie:
cookie-max-age: 10h
```
On Kestra Cloud, contact support to change this setting.
## How do I configure Kestra with my license details?
Kestra Enterprise Edition requires a valid license configured under `kestra.ee.license`. The license is unique to your organization. To get a license, contact the Sales team at [sales@kestra.io](mailto:sales@kestra.io).
The license uses three configuration properties: `id`, `fingerprint`, and `key`.
- `kestra.ee.license.id`: license identifier.
- `kestra.ee.license.fingerprint`: license authentication.
- `kestra.ee.license.key`: license key.
```yaml
kestra:
ee:
license:
id:
fingerprint:
key: |
```
Kestra validates the license on startup and logs the validation result.
## When should I use Secrets vs Credentials?
Use [Secrets](../../06.concepts/04.secret/index.md) when you need to store and reference sensitive values such as API keys, passwords, webhook URLs, or tokens in your flows and configuration. Secrets are the right choice when you want to inject a protected value with the `secret()` function or manage sensitive data centrally.
Use [Credentials](../03.auth/credentials/index.md) when a supported integration or plugin expects a reusable authentication object managed through the UI. Credentials are better suited to connection-level authentication that you want to define once and reuse across multiple flows.
In short: use **Secrets** for protected values, and use **Credentials** for managed authentication objects supported by Kestra integrations.
---
# Governance in Kestra Enterprise: Security and Control
URL: https://kestra.io/docs/enterprise/governance
> Give your team secured, isolated environments and control over workflows with tenants, audit logs, secrets and more.
import ChildCard from "~/components/docs/ChildCard.astro"
Governance features let you control what runs, who can see it, and where sensitive data lives — across tenants, namespaces, and external systems.
---
# Allowed & Restricted Plugins in Kestra Enterprise
URL: https://kestra.io/docs/enterprise/governance/allowed-plugins
> Control plugin usage in Kestra Enterprise. Configure allowed and restricted plugins to enforce security policies and compliance standards.
Configure Kestra to allow or restrict specific plugins.
Kestra comes with the full library of official plugins by default. However, in some cases you may want to restrict which plugins are available to specific teams or users. For example, you might allow a team to use only BigQuery tasks while blocking script execution. Kestra enables this by letting you define allowlists (`includes`) and blocklists (`excludes`) in your [Plugins and Execution configuration](../../../configuration/04.plugins-and-execution/index.md).
## Matching syntax
Each entry in `includes` or `excludes` supports three formats:
- **Trailing wildcard** (`io.kestra.*`) — matches all plugins whose class name starts with the given prefix. The trailing `*` is stripped and the remainder is used as a prefix.
- **Regex** (`regex:`) — matches using a full Java regular expression. Example: `regex:^io\.kestra\.plugin\.core\.flow\.Parallel$`.
- **Plain value** (`io.kestra.plugin.core.flow.Parallel`) — prefix match for backward compatibility. Behaves the same as adding a trailing `*`. Use explicit trailing wildcard or `regex:` for clarity.
## Allowed plugins
To allow specific plugins, add the `includes` attribute and list the approved plugins. The following example allows all plugins from the `io.kestra` package:
```yaml
kestra:
plugins:
security:
includes:
- io.kestra.*
```
## Restricted plugins
To restrict certain plugins, add the `excludes` attribute. The following example allows all `io.kestra` plugins while blocking `io.kestra.plugin.core.debug.Echo`:
```yaml
kestra:
plugins:
security:
includes:
- io.kestra.*
excludes:
- io.kestra.plugin.core.debug.Echo
```
Use the `regex:` prefix for more precise pattern matching, such as excluding a single plugin without prefix side-effects:
```yaml
kestra:
plugins:
security:
excludes:
- regex:^io\.kestra\.plugin\.core\.flow\.Parallel$
```
---
# Assets in Kestra: Track Lineage and Metadata
URL: https://kestra.io/docs/enterprise/governance/assets
> Use Assets in Kestra Enterprise to track workflow lineage and metadata. Manage resources like tables, files, and datasets across your data stack.
Track and manage the resources your workflows create and use.
An **asset** is any named resource a workflow reads from or writes to — a database table, a file, a virtual machine. Declaring assets on tasks builds a lineage graph: which workflows touch which resources, in what order, and how they depend on each other. Kestra can ship that graph to external lineage platforms such as DataHub, Marquez, or Atlan via OpenLineage, so orchestration lineage appears alongside warehouse and pipeline lineage in one place.
:::alert{type="info"}
For an end-to-end architecture walkthrough with diagrams, see [Assets for infrastructure automation](https://kestra.io/blogs/assets-for-infra-automation).
:::
## Declare and capture assets
Assets are captured automatically when tasks declare `assets.inputs` or `assets.outputs`; you can also add them manually from the **Assets** tab. Once created, you can view asset details, check which workflow runs created or modified them, and see how assets connect to each other across your workflows.
Assets enable:
- Shipping metadata to lineage providers (e.g., OpenLineage).
- Populating dropdowns or Pebble inputs with live assets (e.g., available VMs).
- Monitoring assets and their state.
## Asset definition
Define assets directly on any task using the `assets` property. Each task can declare `inputs` assets (resources it reads) and `outputs` assets (resources it creates or modifies).
Every asset includes these fields:
| Field | Description |
| --- | --- |
| `id` | unique within a tenant |
| `namespace` | each asset can be associated with a namespace for filtering and RBAC management |
| `type` | use predefined Kestra types like `io.kestra.plugin.ee.assets.Table` or any custom string value |
| `displayName` | optional human-readable name |
| `description` | markdown-supported documentation |
| `metadata` | map of key-value for adding custom metadata to the given asset |
## Asset identifier
An asset is uniquely identified by its `id` and the tenant (`tenantId`) where you create it - the `id` must be unique per tenant. Neither the namespace nor the type is part of that identity: two assets with the same `id` and different namespaces or types cannot exist in the same tenant. Creating an asset with an `id` that is already taken is rejected.
You can attach a namespace to an asset to improve filtering and to restrict visibility so only users or groups with the appropriate RBAC can access the asset.
## Asset type
Asset types fall into two categories:
- **Kestra-defined asset types**: These predefined types use the `io.kestra.core.models.assets` model and provide structured metadata fields specific to each asset type. Plugins that support auto-generation populate these fields automatically during task execution — for example, a JDBC plugin creates a `Table` asset with `system`, `database`, and `schema` filled in from the connection details.
Kestra provides these built-in asset types:
- `io.kestra.plugin.ee.assets.Dataset`
- Represents a dataset asset managed by Kestra.
- Metadata: `system`, `location`, `format`
- `io.kestra.plugin.ee.assets.File`
- Represents a file asset, such as documents, logs, or other file-based outputs.
- Metadata: `system`, `path`
- `io.kestra.plugin.ee.assets.Table`
- Represents a database table asset with schema and data location metadata.
- Metadata: `system`, `database`, `schema`
- `io.kestra.plugin.ee.assets.VM`
- Represents a virtual machine asset, including attributes like IP address and provider.
- Metadata: `provider`, `region`, `state`
- `io.kestra.core.models.assets.External`
- Represents an external asset that exists outside of Kestra's managed resources.
- This type is automatically assigned when you reference an asset in `assets.inputs` that doesn't already exist in Kestra. You don't need to explicitly set the type — Kestra will create the asset with the `External` type automatically.
- This is useful for tracking dependencies on resources managed outside your workflows, such as external database tables, third-party APIs, or manually provisioned infrastructure.
- **Free-form asset types**: You can define asset types using any custom string value to represent asset categories that fit your organization's needs. This lets you create and manage your own asset taxonomies, giving you flexibility to describe resources that are not covered by Kestra's standard models. These assets require manual definition and will not be auto-generated by plugins.
## Quick start: minimal asset flow
A small example that registers one output asset and logs its ID:
```yaml
id: hello_assets
namespace: company.team
tasks:
- id: write_file
type: io.kestra.plugin.core.log.Log
message: "Created report.csv"
assets:
outputs:
- id: report.csv
type: io.kestra.plugin.ee.assets.File
metadata:
path: s3://company/reports/report.csv
- id: confirm
type: io.kestra.plugin.core.log.Log
message: "Asset recorded: {{ assets() | jq('.[] | {id: .id, type: .type, metadata: .metadata}') }}"
```
## Auto-generated assets
Some plugins support automatic asset generation when `assets.enableAuto: true` is set on a task. This removes the need to manually declare `assets.inputs` and `assets.outputs` — the plugin inspects its execution context and emits assets automatically:
- **JDBC Query**: detects `CREATE TABLE` statements and emits a single `io.kestra.plugin.ee.assets.Table` output; JDBC URL populates `system` and `database`.
- **Ansible CLI**: parses `inventory` hosts as `inputs` of type `io.kestra.core.models.assets.External`, marking the infrastructure targets the playbook runs against.
- **dbt CLI**: parses `manifest.json` to emit each model as an `io.kestra.plugin.ee.assets.Table` output with `database`, `schema`, `name`, and lineage edges based on `depends_on`.
:::collapse{title="JDBC Query auto-generated assets"}
```yaml
id: jdbc_create_trips
namespace: company.team
tasks:
- id: create_trips_table
type: io.kestra.plugin.jdbc.sqlite.Query
url: jdbc:sqlite:myfile.db
outputDbFile: true
sql: |
CREATE TABLE IF NOT EXISTS trips (
VendorID INTEGER,
passenger_count INTEGER,
trip_distance REAL
);
assets:
enableAuto: true
```
:::
:::collapse{title="Ansible CLI auto-generated assets"}
```yaml
id: ansible_playbook
namespace: company.team
tasks:
- id: ansible_task
type: io.kestra.plugin.ansible.cli.AnsibleCLI
inputFiles:
inventory.ini: |
localhost ansible_connection=local
myplaybook.yml: |
---
- hosts: localhost
tasks:
- name: Print Hello World
debug:
msg: "Hello, World!"
assets:
enableAuto: true
commands:
- ansible-playbook -i inventory.ini myplaybook.yml
```
:::
:::collapse{title="dbt CLI auto-generated assets"}
```yaml
id: dbt_build_duckdb
namespace: company.team
tasks:
- id: dbt
type: io.kestra.plugin.core.flow.WorkingDirectory
tasks:
- id: clone_repository
type: io.kestra.plugin.git.Clone
url: https://github.com/kestra-io/dbt-example
branch: main
- id: dbt_build
type: io.kestra.plugin.dbt.cli.DbtCLI
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
commands:
- dbt deps
- dbt build
- dbt run
profiles: |
my_dbt_project:
outputs:
dev:
type: duckdb
path: ":memory:"
fixed_retries: 1
threads: 16
timeout_seconds: 300
target: dev
assets:
enableAuto: true
```
:::
## Operational automation
Assets also support lifecycle management, event-driven triggers, and freshness monitoring directly from flows:
- Lifecycle tasks to create, update, list, and delete assets (`Set`, `List`, `Delete`).
- Event-based triggers with `EventTrigger` that react to asset lifecycle events (`CREATED`, `UPDATED`, `DELETED`, `USED`).
- Freshness monitoring with `FreshnessTrigger` to detect stale assets and launch flows automatically.
- Scope triggers by asset ID, namespace, type, and metadata filters.
- Trigger context variables (`event`, `eventTime`, `lastUpdated`, `staleDuration`, `checkTime`) available for routing, alerting, and recovery logic.
### Trigger use mapping
| Trigger | Primary use |
| --- | --- |
| `EventTrigger` | React instantly to asset lifecycle events (`CREATED`, `UPDATED`, `DELETED`, `USED`). |
| `FreshnessTrigger` | Poll assets on an interval to detect staleness and launch remediation. |
:::collapse{title="Advanced: event-driven automation"}
```yaml
id: asset_event_driven_pipeline
namespace: company.data
tasks:
- id: transform_to_mart
type: io.kestra.plugin.core.flow.Subflow
namespace: company.data
flowId: create_mart_tables
inputs:
source_asset_id: "{{ trigger.asset.id }}"
source_event: "{{ trigger.asset.event }}"
event_time: "{{ trigger.asset.eventTime }}"
triggers:
- id: staging_table_event
type: io.kestra.plugin.ee.assets.EventTrigger
namespace: company.data
assetType: io.kestra.plugin.ee.assets.Table
events:
- CREATED
- UPDATED
metadataQuery:
- field: model_layer
type: EQUAL_TO
value: staging
```
:::
:::collapse{title="Advanced: audit deletions"}
```yaml
id: audit_asset_deletions
namespace: company.security
tasks:
- id: log_deletion
type: io.kestra.plugin.jdbc.postgresql.Query
sql: |
INSERT INTO audit_log (asset_id, asset_type, namespace, event, event_time)
VALUES (
'{{ trigger.asset.id }}',
'{{ trigger.asset.type }}',
'{{ trigger.asset.namespace }}',
'{{ trigger.asset.event }}',
'{{ trigger.asset.eventTime }}'
)
triggers:
- id: asset_deletion_event
type: io.kestra.plugin.ee.assets.EventTrigger
events:
- DELETED
```
:::
:::collapse{title="Advanced: freshness monitoring"}
```yaml
id: stale_assets_monitor
namespace: company.monitoring
tasks:
- id: log_stale
type: io.kestra.plugin.core.log.Log
message: >
Found {{ trigger.assets | length }} stale assets.
First asset: {{ trigger.assets[0].id ?? 'n/a' }}.
Stale for: {{ trigger.assets[0].staleDuration ?? 'n/a' }}.
triggers:
- id: stale_assets
type: io.kestra.plugin.ee.assets.FreshnessTrigger
maxStaleness: PT24H
interval: PT1H
```
:::
:::collapse{title="Advanced: scoped freshness checks"}
```yaml
id: prod_assets_freshness
namespace: company.monitoring
tasks:
- id: trigger_remediation
type: io.kestra.plugin.core.flow.Subflow
namespace: company.data
flowId: refresh_marts
inputs:
asset_id: "{{ trigger.assets[0].id }}"
last_updated: "{{ trigger.assets[0].lastUpdated }}"
stale_duration: "{{ trigger.assets[0].staleDuration }}"
triggers:
- id: stale_prod_marts
type: io.kestra.plugin.ee.assets.FreshnessTrigger
namespace: company.data
assetType: TABLE
maxStaleness: PT6H
interval: PT30M
metadataQuery:
- field: environment
type: EQUAL_TO
value: prod
- field: model_layer
type: EQUAL_TO
value: mart
```
:::
:::collapse{title="Advanced: lifecycle tasks"}
```yaml
id: asset_lifecycle_ops
namespace: company.data
tasks:
- id: upsert_asset
type: io.kestra.plugin.ee.assets.Set
namespace: assets.data
assetId: customers_by_country
assetType: TABLE
displayName: Customers by Country
assetDescription: Customer distribution by country
metadata:
owner: data-team
environment: prod
- id: list_assets
type: io.kestra.plugin.ee.assets.List
namespace: assets.data
types:
- TABLE
metadataQuery:
- field: owner
type: EQUAL_TO
value: data-team
fetchType: FETCH
- id: delete_asset
type: io.kestra.plugin.ee.assets.Delete
assetId: customers_by_country
```
:::
## Locking assets
A lock prevents concurrent writes to a shared asset while a flow operates on it. Locks are TTL-bounded: they expire automatically when their duration elapses and can also be released explicitly. Reads are always open — only writes (edit, delete) are blocked while a lock is held.
Two owner types exist:
| Owner type | Acquired by | Behavior while held |
|---|---|---|
| `EXECUTION` | `Acquire` task | Blocks other executions' writes. The lock-holding execution can still write to the asset, and each write extends the lease. |
| `USER` | UI or REST API | Blocks all execution writes. Use for manual maintenance windows. |
### Locking from a flow
The `Acquire` and `Release` tasks wrap the work that needs exclusive write access. Both require the `LOCK` permission on the `ASSET` resource (`UNLOCK` for `Release`).
If another execution already holds the lock when `Acquire` runs, the task fails with a 423 error. Add a `Retry` to the `Acquire` task to wait for the lock to become available.
```yaml
id: update_customer_asset
namespace: company.team
tasks:
- id: acquire
type: io.kestra.plugin.kestra.ee.locks.Acquire
assetId: customers_by_country
ttl: PT1H
- id: write
type: io.kestra.plugin.core.log.Log
message: Writing to the locked asset
- id: release
type: io.kestra.plugin.kestra.ee.locks.Release
assetId: customers_by_country
```
**`Acquire`** properties:
| Property | Required | Description |
|---|---|---|
| `assetId` | Yes | ID of the asset to lock. |
| `ttl` | No | How long to hold the lock before it expires automatically. ISO-8601 duration (e.g. `PT1H`). Defaults to 5 minutes when unset. |
**`Acquire`** outputs:
| Output | Description |
|---|---|
| `lockedUntil` | When the lock expires. |
| `ownerType` | Always `EXECUTION` for a task-acquired lock. |
| `executionId` | ID of the execution holding the lock. |
`Release` is owner-checked: it removes the lock only if the current execution holds it. If the lock has already expired or belongs to a different owner, `Release` is a no-op — safe to call unconditionally.
### Locking from the UI
From any asset's detail page, users with the `LOCK` permission can lock the asset manually. Choose from preset durations (5 minutes to 24 hours) or enter a custom ISO-8601 duration. The page shows who holds the lock and when it expires. Users with the `UNLOCK` permission can release any lock regardless of owner.
When an asset is locked, the detail page shows a banner: *You might be seeing outdated metadata as this asset is currently locked for writing.*
The asset list supports filtering by lock status.
## Data pipeline use cases
:::collapse{title="Advanced: data pipeline examples"}
Assets are essential for tracking data lineage in analytics and data engineering workflows. The following examples demonstrate how to use assets for simple table creation and complex multi-layer data pipelines.
### Example 1: Simple table creation
**Scenario**: You're creating a new database table from scratch. This is a foundational asset with no upstream dependencies.
```yaml
id: pipeline_with_assets
namespace: company.team
tasks:
- id: create_trips_table
type: io.kestra.plugin.jdbc.sqlite.Queries
url: jdbc:sqlite:myfile.db
outputDbFile: true
sql: |
CREATE TABLE IF NOT EXISTS trips (
VendorID INTEGER,
passenger_count INTEGER,
trip_distance REAL
);
INSERT INTO trips (VendorID, passenger_count, trip_distance) VALUES
(1, 1, 1.5),
(1, 2, 2.3),
(2, 1, 0.8),
(2, 3, 3.1);
assets:
outputs:
- id: trips
namespace: "{{ flow.namespace }}"
type: io.kestra.plugin.ee.assets.Table
metadata:
database: sqlite
table: trips
```
**Key points**:
- There are no `inputs` assets as this is a source table with no dependencies
- The `trips` table is registered as an output asset that downstream workflows can reference
- Metadata captures the database type and table name for easier discovery
### Example 2: Multi-layer data pipeline
**Scenario**: You're building a modern data stack with staging and mart layers. The staging layer reads from an external source, and the mart layer creates aggregated analytics tables.
```yaml
id: data_pipeline_assets
namespace: kestra.company.data
tasks:
- id: create_staging_layer_asset
type: io.kestra.plugin.jdbc.duckdb.Query
url: "jdbc:duckdb:md:my_db?motherduck_token={{ secret('MOTHERDUCK_TOKEN') }}"
fetchType: STORE
sql: |
CREATE TABLE IF NOT EXISTS trips AS
select VendorID, passenger_count, trip_distance from sample_data.nyc.taxi limit 10;
assets:
inputs:
- id: sample_data.nyc.taxi
outputs:
- id: trips
namespace: "{{flow.namespace}}"
type: io.kestra.plugin.ee.assets.Table
metadata:
model_layer: staging
- id: for_each
type: io.kestra.plugin.core.flow.Loop
values:
- passenger_count
- trip_distance
tasks:
- id: create_mart_layer_asset
type: io.kestra.plugin.jdbc.duckdb.Query
url: "jdbc:duckdb:md:my_db?motherduck_token={{ secret('MOTHERDUCK_TOKEN') }}"
fetchType: STORE
sql: SELECT AVG({{item.value}}) AS avg_{{item.value}} FROM trips;
assets:
inputs:
- id: trips
outputs:
- id: avg_{{item.value}}
type: io.kestra.plugin.ee.assets.Table
namespace: "{{flow.namespace}}"
metadata:
model_layer: mart
```
**What's happening in this pipeline**:
1. **External Source Tracking**: The `create_staging_layer_asset` task references `sample_data.nyc.taxi` as an input asset, even though it's managed outside this workflow. This establishes lineage to external data sources.
2. **Staging Layer**: The `trips` table is created and registered with `model_layer: staging` metadata. This becomes an intermediate asset that mart layers will consume.
3. **Dynamic Mart Creation**: The `Loop` task generates two mart tables:
- `avg_passenger_count`
- `avg_trip_distance`
Both declare `trips` as an input, creating a clear dependency chain.
4. **Complete Lineage Graph**: Kestra automatically builds the dependency graph.
**Benefits of this approach**:
- **Impact Analysis**: If `sample_data.nyc.taxi` changes, you can instantly see that it affects 3 downstream assets
- **Layer Organization**: Filter assets by `model_layer` to view only staging or mart tables
- **Dependency Tracking**: Know exactly which tables depend on others before making schema changes
- **Audit Trail**: Track which workflows created each table and when
:::
## Infrastructure use case: team bucket provisioning
:::collapse{title="Advanced: infrastructure provisioning"}
Assets are particularly valuable for infrastructure management scenarios. This example demonstrates how a DevOps team can provision cloud resources and track their usage across different teams.
**Scenario**: Your DevOps team needs to create dedicated S3 buckets for multiple teams (Business, Data, Finance, Product). By registering these buckets as assets during provisioning, you establish a clear lineage of which workflows and executions interact with each infrastructure component.
The following flow creates S3 buckets for selected teams and registers them as assets:
```yaml
id: infra_assets
namespace: kestra.company.infra
inputs:
- id: teams
type: MULTISELECT
values:
- Business
- Data
- Finance
- Product
tasks:
- id: for_each
type: io.kestra.plugin.core.flow.Loop
values: "{{ inputs.teams }}"
tasks:
- id: create_bucket
type: io.kestra.plugin.aws.cli.AwsCLI
accessKeyId: "{{ secret('AWS_ACCESS_KEY') }}"
secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
region: "{{ secret('AWS_REGION') }}"
allowFailure: true
commands:
- aws s3 mb s3://kestra-{{ item.value | slugify }}-bucket
assets:
outputs:
- id: kestra-{{ item.value | slugify }}-bucket
type: AWS_BUCKET
metadata:
provider: s3
address: s3://kestra-{{ item.value | slugify }}-bucket
```
This flow dynamically creates buckets (e.g., `kestra-data-bucket`, `kestra-finance-bucket`) and registers each as an `AWS_BUCKET` asset with relevant metadata.
Once the infrastructure is provisioned, teams can reference these assets in their workflows. Here's how the Data team uses their bucket:
```yaml
id: upload_file
namespace: kestra.company.data
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/jaffle-csv/raw_customers.csv
- id: aws_upload
type: io.kestra.plugin.aws.s3.Upload
accessKeyId: "{{ secret('AWS_ACCESS_KEY') }}"
secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
region: "{{ secret('AWS_REGION') }}"
bucket: kestra-data-bucket
from: '{{ outputs.download.uri }}'
key: raw_customer.csv
assets:
inputs:
- id: kestra-data-bucket
outputs:
- id: raw_customer
type: io.kestra.plugin.ee.assets.File
metadata:
owner: data
```
In this workflow:
- The `aws_upload` task declares `kestra-data-bucket` as an **input asset**, linking it to the infrastructure provisioned earlier
- It also creates an **output asset** (`raw_customer`) representing the uploaded file
- This establishes a complete lineage chain: infrastructure creation → data upload → file asset
**Benefits**: With this approach, you can easily answer questions like:
- Which teams are using which buckets?
- What files have been uploaded to each bucket?
- Which workflows and executions have interacted with a specific infrastructure component?
- When was this infrastructure resource created and by which flow?
:::
## Populate dropdowns and app inputs
Use the `assets()` Pebble function to query assets at runtime — for example, to populate dropdown inputs or select resources based on type, namespace, or metadata.
### Function signature
```plaintext
assets(id: string, type: string, namespace: string, metadata: map)
```
### Parameters
| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| `id` | string | No | Filter by asset ID. Because IDs are unique per tenant, this returns at most one result. |
| `type` | string | No | Filter by asset type (e.g., `"io.kestra.plugin.ee.assets.Table"`). If omitted, returns all types. |
| `namespace` | string | No | Filter by namespace. Defaults to the flow's namespace. |
| `metadata` | map | No | Filter by metadata key-value pairs (e.g., `{"model_layer": "mart"}`). |
### Return value
Returns an array of asset objects. Each object contains:
| Field | Type | Description |
| --- | --- | --- |
| `id` | string | Asset identifier |
| `namespace` | string | Namespace the asset belongs to |
| `type` | string | Asset type |
| `metadata` | map | Custom metadata key-value pairs |
| `tenantId` | string | Tenant ID where the asset was created |
| `created` | string | ISO 8601 timestamp of creation |
| `updated` | string | ISO 8601 timestamp of last update |
| `deleted` | boolean | Whether the asset has been deleted |
### Examples
**Fetch a specific asset by ID:**
```yaml
id: check_asset
namespace: company.team
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "{{ assets(id='report.csv') | jq('.[0].metadata.path') }}"
```
**Populate a multiselect dropdown with table assets:**
```yaml
id: select_assets
namespace: company.team
inputs:
- id: assets
type: MULTISELECT
expression: '{{ assets(type="io.kestra.core.models.assets.Table") | jq(".[].id") }}'
tasks:
- id: for_each
type: io.kestra.plugin.core.flow.Loop
values: "{{inputs.assets}}"
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "{{item.value}}"
```
**Filter assets by namespace:**
```yaml
inputs:
- id: staging_tables
type: MULTISELECT
expression: '{{ assets(type="io.kestra.core.models.assets.Table", namespace="company.team") | jq(".[].id") }}'
```
**Filter assets by metadata:**
```yaml
inputs:
- id: mart_tables
type: MULTISELECT
expression: '{{ assets(metadata={"model_layer": "mart"}) | jq(".[].id") }}'
```
**Get all assets and extract metadata:**
```yaml
id: list_assets_metadata
namespace: company.team
tasks:
- id: list_all_assets
type: io.kestra.plugin.core.log.Log
message: "{{ assets() | jq('.[] | {id: .id, type: .type, metadata: .metadata}') }}"
```
## Export assets with AssetShipper
Use the `AssetShipper` task to export asset metadata to external systems for lineage tracking, monitoring, or integration with data catalogs. Supported destinations include files and OpenLineage-compatible providers.
### Export assets to file
Export asset metadata to a file in either ION or JSON format. This is useful for archiving, auditing, or importing into other systems.
```yaml
id: ship_asset_to_file
namespace: kestra.company.data
tasks:
- id: export_assets
type: io.kestra.plugin.ee.assets.AssetShipper
assetExporters:
- id: file_exporter
type: io.kestra.plugin.ee.assets.FileAssetExporter
format: ION
```
You can change the `format` property to `JSON` if you prefer a more widely-compatible format.
### Export assets to OpenLineage
Ship asset metadata to an OpenLineage-compatible lineage provider. This requires mapping Kestra asset fields to OpenLineage conventions.
```yaml
id: ship_asset_to_openlineage
namespace: kestra.company.data
tasks:
- id: export_to_lineage
type: io.kestra.plugin.ee.assets.AssetShipper
assetExporters:
- id: openlineage_exporter
type: io.kestra.plugin.ee.openlineage.OpenLineageAssetExporter
uri: http://host.docker.internal:5000
mappings:
io.kestra.plugin.ee.assets.Table:
namespace: namespace
```
The `mappings` property defines how Kestra asset metadata fields map to OpenLineage dataset facets. Each asset type can have its own mapping configuration. For more information about OpenLineage dataset facets and available fields, see the [OpenLineage Dataset Facets documentation](https://openlineage.io/docs/spec/facets/dataset-facets/).
## Purge assets and lineage data
Use the `io.kestra.plugin.ee.assets.PurgeAssets` task to enforce asset retention without touching executions or logs. By default, this task purges assets, asset usage events (execution view), and asset lineage events (for asset exporters) matching the filters. You can configure it to only purge specific types of records.
**Filters:**
| Property | Description |
| --- | --- |
| `namespace` | Filter by namespace. Supports prefix matching (e.g., `company.data` matches `company.data.staging`). |
| `assetId` | Filter by a specific asset ID. |
| `assetType` | Filter by one or more asset types (e.g., `io.kestra.plugin.ee.assets.Table`). |
| `metadataQuery` | Filter by metadata key-value pairs. |
| `endDate` | **(required)** Purge records created or updated before this date (ISO 8601). |
**Purge scope:**
| Property | Default | Description |
| --- | --- | --- |
| `purgeAssets` | `true` | Whether to purge the asset records themselves. |
| `purgeAssetUsages` | `true` | Whether to purge asset usage events (execution view). |
| `purgeAssetLineages` | `true` | Whether to purge asset lineage events. |
**Outputs:** `purgedAssetsCount`, `purgedAssetUsagesCount`, `purgedAssetLineagesCount`.
Example: purge old VM assets on a monthly schedule.
```yaml
id: asset_retention_policy
namespace: company.infra
triggers:
- id: monthly_cleanup
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 0 1 * *"
tasks:
- id: purge_old_vms
type: io.kestra.plugin.ee.assets.PurgeAssets
assetType:
- io.kestra.plugin.ee.assets.VM
endDate: "{{ now() | dateAdd(-180, 'DAYS') }}"
```
## Visualizing assets in dashboards
Use the `io.kestra.plugin.ee.dashboard.data.Assets` data source to build charts over your asset inventory directly in a custom dashboard. Asset charts are not filtered by the dashboard time range — they always reflect the current state of your inventory.
See [Assets (EE and Cloud only)](../../../09.ui/00.dashboard/index.md#assets-ee-and-cloud-only) in the Dashboards documentation for available fields, chart type compatibility, and configuration examples.
---
# Audit Logs in Kestra: Governance and Compliance
URL: https://kestra.io/docs/enterprise/governance/audit-logs
> Ensure compliance with Kestra Audit Logs. Track and monitor all user activities, flow executions, and system changes for security and governance.
Audit Logs record every action taken in your Kestra instance by users and service accounts.
## Audit logs
By reviewing Audit Logs, system administrators can track user activity, and security teams can investigate incidents and ensure compliance with regulatory requirements.
## Why audit logs matter
Audit logs are a historical record that developers and system administrators can use to track changes, monitor system usage, and verify system activity. They track the sequence of activities, ensuring accountability and providing data for troubleshooting and analysis. Because audit logs are immutable, they can also be used to detect and investigate security incidents. If you use the Elasticsearch backend, you can use Kibana to search and visualize your logs.
## How to access Audit Logs
You can access Audit Logs from the **Tenant** section in the UI. That UI page provides a detailed table of recorded events, capturing the actions taken within the system:

Each row in the table represents a distinct event with several columns providing specific details:
- **Resource Type** column categorizes the resource that the event is associated with, such as editing a flow (FLOW) or executing it (EXECUTION).
- **Action** indicates whether a given resource has been created, updated, or deleted.
- **Actor** identifies who performed the action. The user can be a human, a system, or a service account.
- **Details** section offers an in-depth description of the event, including identifiers such as the `id`, `namespace`, `flowId`, `executionId`, revision, etc. — those fields depend on the type of resource the event is associated with.
- **Date** represents the timestamp of when the event occurred.
- **Changes** shows two buttons: one to view the revision and a second to link you directly to the resource that created the log.
## How to see a full diff of a specific event
To see a full diff of a specific event, click the icon in the **Changes** column. The expanded view shows the full diff of the event side-by-side, including the `before` and `after` states of a given resource:

## How to filter audit logs
Click **Add filters** to open the Advanced filter dialog. You can combine multiple conditions — for example, filter by Interval (Last 7 days) and Resource type (NAMESPACE) to narrow the table:

To filter for a specific event, click any tag in the **Details** column to add it as a filter condition.
## How to purge audit logs
The Enterprise Edition of Kestra generates an audit log for _every action_ taken on the platform. While these logs are essential for tracking changes and ensuring compliance, they can accumulate over time and take up a significant amount of space in the database.
The `PurgeAuditLogs` task removes old audit logs that are no longer needed. You can set a date range for the logs you want to delete, choose a specific `namespace`, and even filter by `resources` or `actions` (`CREATE`, `READ`, `UPDATE`, `DELETE`).
:::alert{type="info"}
Additional types of **Purge tasks** are described in the [dedicated section](../../../10.administrator-guide/purge/index.md).
:::
Here is the recommended way to implement the audit logs retention policy that purges audit logs older than one month:
```yaml
id: audit_log_cleanup
namespace: system
tasks:
- id: purge_audit_logs
type: io.kestra.plugin.ee.core.log.PurgeAuditLogs
description: Purge audit logs older than 1 month
endDate: "{{ now() | dateAdd(-1, 'MONTHS') }}"
```
Note how the above flow is added to the `system` namespace, which is the default namespace for System Flows. This ensures that this maintenance flow and its executions are hidden from the main UI, making them only visible within the `system` namespace that can be managed by platform administrators.
Combining the [System Flows](../../../06.concepts/system-flows/index.md) functionality with the `PurgeAuditLogs` task provides a simple way to manage your audit logs as code and from the UI, ensuring you keep them as long as you need to stay compliant while keeping your database clean and performant.
## Export audit logs
Audit logs can be forwarded to an external monitoring system such as Datadog, AWS CloudWatch, Google Operational Suite, and more with the [Audit Log Shipper task](../logshipper/index.md#audit-log-shipper).
---
# Cases in Kestra: Incident Management for Executions
URL: https://kestra.io/docs/enterprise/governance/cases
> Use Cases in Kestra Enterprise to track and resolve incidents directly in Kestra. Create cases automatically from failed executions, deduplicate alerts, and manage SLAs.
Track and resolve incidents directly in Kestra, next to the executions that caused them.
When an execution fails, it is often an incident. Cases lets you track and resolve those incidents directly in Kestra, without a separate tool.
Cases are not limited to failures; you can open one for executions that need approvals, or for successful executions that produced unexpected outputs.
The `CreateCase` task lets you open cases automatically from your flow YAML:
- in the `errors`, `finally`, or `afterExecution` blocks, to auto-create custom cases when the execution moves to a terminal state like Failed, Killed, Warning, or Success
- anywhere in the regular `tasks` block, to create cases based on custom conditions within the flow, for example if a given output is bad (error status code on some API request).
Not every execution needs to be treated as an independent case, though. Often a single production issue (e.g. an external API that is temporarily unreachable) can easily create 10 to 100 failed executions within an hour. Sending Slack alerts for each failed execution can quickly get so noisy that users mute the channel, making the alerting useless. Cases solve this with **deduplication**: with `linkMatchingExecutions: true`, new failing executions are attached to the already-open case for the same flow and task instead of opening a new one.
:::alert{type="warning"}
Cases require the [Kestra plugin](https://kestra.io/plugins/plugin-kestra) to be installed on your Kestra EE instance, because the `CreateCase` task used under the hood by Cases comes from this plugin. The default Docker image with plugins included already contains it. If you manage plugins yourself, install the Kestra plugin through [Versioned Plugins](../../05.instance/versioned-plugins/index.md) to use Cases reliably.
:::
## What a case contains
- Title and description: the description supports Markdown
- Severity: `Critical`, `High`, `Medium`, or `Low`; defaults to `Medium`
- Status: `Open`, `Acknowledged`, `Investigating`, `Resolved`, or `Cancelled`; `Resolved` and `Cancelled` are terminal
- Namespace: every case belongs to a namespace, and all permission checks are scoped to it
- Assignees and watchers: both accept users and groups; assignees are responsible for the case, watchers only receive notifications. The same user or group cannot be both at once — assigning them as an assignee removes them from the watchers
- Custom fields: typed fields (Text, Select, Multi-select) defined per case or inherited from a template; cases use custom fields instead of labels
- SLAs: optional acknowledgement and resolution targets
- Linked executions and assets: the executions and [assets](../01.assets/index.md) affected by the incident
- Case actions: flows attached to the case as one-click remediation buttons
- Origin: the flow, task, and execution that created the case (set only when created by the `CreateCase` task); this is the deduplication key
- Template reference: the case template the case was created from, if any
## Case lifecycle
There is no restrictive state machine: any status can be changed to any other, including reopening a resolved or cancelled case. A few rules apply:
1. Resolving requires a **resolution reason**; the note is optional. When the case was created from a template, the template can restrict the allowed reasons and make the note mandatory. Without a template, the built-in reasons are offered: Fixed, Workaround applied, Configuration change, Duplicate, No action needed, Won't fix.
2. Cancelling accepts an optional reason and note.
3. The first time the case moves from `Open` to any other status, the acknowledgement time is recorded (this satisfies the acknowledgement SLA and is kept even if the case is later reopened).
4. Reopening a terminal case clears the resolution, and the case's auto-attach flow (if any) is recreated.
## SLAs
Each case can have two optional SLA targets, set directly or inherited from a template:
1. Acknowledgement: met the first time the case moves from `Open` to any other status
2. Resolution: met when the case is resolved
Both clocks start at case creation. Each of the two SLAs can be in one of six states: `Not started`, `Running`, `Overdue`, `Met`, `Missed`, or `Voided` (cancelling a case voids its pending SLAs). States are computed at read time and never persisted. The UI shows a live countdown ("Due in 2h", "Due 30m ago") on the case detail page, board cards, and the list's Resolution SLA column.
When a target is missed, a background check records an `SLA_ACKNOWLEDGEMENT_BREACHED` or `SLA_RESOLUTION_BREACHED` event in the case's timeline and sends an in-app notification to the case's assignees and watchers. The check runs every 5 minutes by default and can be tuned or disabled through the `kestra.cases.sla-breach-check` configuration; each of the two SLAs is notified at most once per case.
## The CreateCase task
The task type is `io.kestra.plugin.kestra.ee.cases.CreateCase`. It is an SDK-based task that calls the Kestra API, so it can be placed in any task block: `tasks`, `errors`, `finally`, or `afterExecution`.
```yaml
id: orders_sync
namespace: company.team
tasks:
- id: sync
type: io.kestra.plugin.core.http.Request
uri: https://api.example.com/orders
errors:
- id: open_case
type: io.kestra.plugin.kestra.ee.cases.CreateCase
kestraUrl: http://localhost:8080
auth:
apiToken: "{{ secret('KESTRA_API_TOKEN') }}"
title: "Orders sync failed: {{ flow.id }}"
severity: HIGH
linkMatchingExecutions: true
sla:
acknowledgement: PT1H
resolution: PT8H
```
The task calls the Kestra API, so it needs an endpoint and credentials. `kestraUrl` defaults to the URL of the current instance, and `auth` accepts either an `apiToken` or a `username` and `password` pair. Both can also be omitted: an administrator can set **Default authentication credentials** when editing a namespace or tenant, and those credentials are then used automatically (the instance configuration `kestra.tasks.sdk.authentication` works as a global fallback). Without `auth` on the task and without such defaults, the task fails at runtime.
For condition-based cases inside the regular `tasks` block, combine it with `runIf`:
```yaml
tasks:
- id: check
type: io.kestra.plugin.core.http.Request
uri: https://api.example.com/health
allowFailed: true
- id: open_case
type: io.kestra.plugin.kestra.ee.cases.CreateCase
kestraUrl: http://localhost:8080
auth:
apiToken: "{{ secret('KESTRA_API_TOKEN') }}"
runIf: "{{ outputs.check.code != 200 }}"
title: "Health check returned {{ outputs.check.code }}"
severity: CRITICAL
```
### Properties
| Property | Description |
|---|---|
| `title` | Case title; required unless `caseId` is set. Supports expressions. |
| `caseDescription` | Markdown description (named this way because `description` is reserved for the task itself). |
| `severity` | `CRITICAL`, `HIGH`, `MEDIUM`, or `LOW`; defaults to `MEDIUM`. |
| `status` | Initial status; defaults to `OPEN`. |
| `namespace` | Namespace the case belongs to; defaults to the flow's namespace. |
| `sla` | `acknowledgement` and `resolution` durations (ISO-8601, e.g. `PT1H`). |
| `linkMatchingExecutions` | When `true`, the execution is attached to an already-open case with the same origin instead of creating a new one; defaults to `false`. |
| `caseId` | Attach the execution to this exact case instead of creating one (used by generated auto-attach flows). |
| `executionId` | Execution to link; defaults to the current execution. |
| `assignees` / `watchers` | User emails and group names, resolved to stable IDs on the server. |
| `labels` | Key/value pairs, stored as Text custom fields on the case. |
| `assetIds` | Assets to link to the case. |
| `actions` | Case actions to attach, each with `label`, `namespace`, and `flowId`. |
Connection properties (`kestraUrl`, `auth`, `tenantId`) are shared by all tasks from the Kestra plugin; `tenantId` defaults to the execution's tenant. Authentication is resolved as described above: `auth` set on the task takes precedence, then the namespace's default authentication credentials, then the tenant's, then the instance configuration.
### Outputs
The task returns `caseId` and `created`. `created: false` means the execution was attached to an existing case (via deduplication or `caseId`) rather than opening a new one.
Note that the task cannot reference a case template. Case templates are mainly helpers to make it easier to configure cases from the UI; when creating cases from code, all properties such as severity or SLAs are declared explicitly on the task, following Kestra's declarative nature.
## Deduplication
With `linkMatchingExecutions: true`, the server first looks for an active case with the same **origin**: the combination of flow namespace, flow ID, and task ID (within the tenant). The title is not part of the key. A case counts as active when its status is `Open`, `Acknowledged`, or `Investigating`.
- On a match: the triggering execution is appended to the existing case's linked executions (visible in the timeline as an `EXECUTION_LINKED` event), no new case is created, and the task returns `created: false`. Fields such as title or severity are not merged, and no new "created" notification is sent.
- On no match: a new case is created with the origin recorded, and the execution is linked to it.
The same behavior can also be enabled from the UI on an existing case with [auto-attach](#auto-attach).
One caveat: the deduplication check is not atomic. The server first checks for an existing case and then creates one, so two executions of the same task failing at exactly the same moment can each create their own case.
## Auto-attach
Auto-attach allows you to avoid getting a new case and a new notification for every single failed execution. A case is created for the first failure, and all following executions matching the same flow and state are automatically attached to that same case until the case is resolved or cancelled.
Deduplication with `linkMatchingExecutions` achieves the same from the flow YAML; auto-attach is its UI counterpart, configured on an existing case. It can be enabled with the "Auto-link matching executions" toggle when attaching executions to a case, from the case detail page or in the Create Case modal. A banner then lists exactly what will be matched, for example "New executions matching company.team/orders_sync (FAILED) will automatically attach to this case." Several executions can be selected at once, even across different flows: the selection is grouped into one rule per flow, and the states of that flow's selected executions are combined.
Under the hood, enabling it generates a flow named `attach_executions_` in the system namespace, with one Flow trigger per (namespace, flow, states) rule and a single `CreateCase` task that passes the `caseId`. Multiple rules accumulate on the same generated flow. The flow is deleted when the case reaches a terminal status or is deleted, and recreated on reopen. These generated flows are visible on the [System Flows](../../../06.concepts/system-flows/index.md) page.
Enabling auto-attach is gated by RBAC: it requires the `UPDATE` action on the `CASE` permission in the case's namespace, plus the `EXECUTE` action on the `FLOW` permission in the namespace of the flow whose executions will be attached.
## Creating cases from the UI
Besides the task, cases can be created in four places:
1. Cases page: the Create button opens a modal with template selection, title, severity, initial status, namespace, assignees and watchers, both SLA targets (presets from 1 hour to 72 hours, or a custom duration), description, custom fields, and executions to link.
2. Executions page (bulk): select executions and use "Create case from selection" or "Add to existing case". Both also work with "select all matching filter", so a case can be created from everything matching the current query.
3. Execution's Overview tab: a "Linked cases" panel lists the cases the execution belongs to and offers a Create case button pre-linked to that execution.
4. Asset page: the same panel exists on the asset overview; a case created there is linked to the asset automatically.
## Cases in the UI
Cases sit in the left menu between Executions and System Flows.
- Board view (default): a kanban board grouped by status, severity, or assignee. Dragging a card between columns updates the case; dropping onto Resolved opens the resolve modal, because a reason is mandatory. Cards show the case ID, severity, title, assignees, and a live SLA countdown.
- List view: columns for Case, Title, Severity, Status, Resolution SLA, and Assignee, with an inline next-step button per row (Acknowledge, Investigating, Resolve) and bulk Acknowledge/Delete.
- Toolbar: full-text search; filters on namespace, status, severity, assignee, and time range; an "Assigned to me" toggle; and one chip per status with live counts.

The case detail page shows the editable header (title, severity, status, quick transitions, Resolve/Reopen), the Markdown description, custom fields, a resolution card once resolved, SLA countdowns, assignees and watchers (with a Watch/Unwatch toggle and "Assign with note"), case actions, linked executions, linked assets, and the activity timeline.

## Comments and activity timeline
Every change to a case is recorded as a timeline event: creation, field updates, status and severity changes, assignments (with optional note), execution and asset links, action runs, auto-attach changes, and SLA breaches. Comments support **Markdown** and up to 5 file attachments of 10 MB each (drag-and-drop and clipboard paste work).
## Case actions
A case action is a flow attached to the case as a one-click button, intended for remediation or diagnostics (e.g. "Restart service"). Clicking an action opens the Execute flow modal so you can review and fill the flow's inputs, and optionally set labels, a schedule date, or breakpoints before running. The resulting execution is automatically linked back to the case and labeled `system.caseId: ` and `system.from: case`, and the run is recorded in the timeline. Attaching or running an action requires the `EXECUTE` action on the `FLOW` permission in the target flow's namespace.
## Linked executions and assets
- Executions can be linked by the task, by auto-attach, manually from the case detail page, or in bulk by ID or by filter query. The linked-executions card shows each execution's live state and keeps a row (marked as no longer existing) even if the execution was purged.
- Assets come in two kinds: explicitly attached ones, and derived ones (assets used by the case's linked executions, shown with an "Auto-detected" tag). Derived assets cannot be unlinked; they disappear when the executions that reference them are unlinked.
## Case templates
Templates standardize how cases are created. A template can define a default severity, default assignees and watchers, a title pattern (supporting the same expressions as the task title), a description, both SLA targets, custom field definitions, default case actions, allowed resolution reasons, and whether a resolution note is required. One template per tenant can be marked as the default; it is preselected in the Create Case modal.
Templates are managed in **Cases → Settings** and require the `TEMPLATE` action on the `CASE` permission.
 A template can be scoped to a namespace or left tenant-wide (tenant-wide templates and the default flag require a global grant).
Every tenant is seeded with a built-in "Execution failure incident" template: severity High, title pattern `{{ execution.id }} failed for {{ flow.id }}`, 1 hour acknowledgement and 8 hour resolution SLA, and the built-in resolution reasons.
Template defaults apply when creating cases via the UI or the API; they cannot be used by the `CreateCase` task, where all properties are declared explicitly in the flow YAML.
## Notifications
Case events are wired into the in-app notification system (the bell icon). Seven events send notifications: case created, assigned, status changed, severity changed, commented, acknowledgement SLA breached, and resolution SLA breached. Recipients are the case's assignees and watchers, with groups expanded to their members; assignment notifications go only to the users whose assignment actually changed, with the assignment note included. Each notification contains a link to the case.
## Permissions and audit
- RBAC: a new `CASE` permission resource with actions `VIEW`, `LIST`, `CREATE`, `UPDATE`, `DELETE`, `FOLLOW`, and `TEMPLATE`, optionally scoped to a namespace (_creating a case requires `CREATE` on its namespace, transitions and linking require `UPDATE`, commenting only requires `VIEW`_).
- Audit: create, update, and delete of cases and case templates are recorded in the [audit log](../06.audit-logs/index.md) (resource types `CASE` and `CASE_TEMPLATE`). Timeline events and comments are not separately audited.
- Backup: cases, templates, timeline events, and execution links are included in backup and restore.
## API
All case operations are available via REST under `/api/v1/{tenant}/cases`. The calls below cover the most common integration patterns. For the full endpoint list, request shapes, and filter DSL, see the [Enterprise API reference](../../../api-reference/01.enterprise/index.mdx).
**Search cases**
```bash
curl -X GET "https://{host}/api/v1/{tenant}/cases/search?page=1&size=25" \
-H "Authorization: Bearer {token}"
```
Accepts `filters` (a list of structured query filter objects), `sort`, and `dateFilter` as query parameters. Use `GET /cases/counts` to get a `{status: count}` breakdown without pagination.
**Create a case**
```bash
curl -X POST "https://{host}/api/v1/{tenant}/cases" \
-H "Authorization: Bearer {token}" \
-H "Content-Type: application/json" \
-d '{
"namespace": "company.team",
"title": "Orders sync failed",
"severity": "HIGH",
"sla": { "acknowledgement": "PT1H", "resolution": "PT8H" }
}'
```
Returns the full case object including the generated `id`.
**Acknowledge and resolve**
```bash
# Acknowledge
curl -X POST "https://{host}/api/v1/{tenant}/cases/{id}/acknowledge" \
-H "Authorization: Bearer {token}"
# Resolve
curl -X POST "https://{host}/api/v1/{tenant}/cases/{id}/resolve" \
-H "Authorization: Bearer {token}" \
-H "Content-Type: application/json" \
-d '{ "reason": "Fixed", "note": "Deployed hotfix v1.2.3" }'
```
A `reason` is required to resolve. The reason is a free-form string; case templates can configure a `resolutionReasons` list to restrict which values are accepted.
**Link executions**
```bash
curl -X POST "https://{host}/api/v1/{tenant}/cases/{id}/executions" \
-H "Authorization: Bearer {token}" \
-H "Content-Type: application/json" \
-d '["execution-id-1", "execution-id-2"]'
```
Use `POST /cases/{id}/executions/by-query` to link everything matching a filter query (capped at 1,000).
**Look up cases by execution**
```bash
curl -X POST "https://{host}/api/v1/{tenant}/cases/by-executions" \
-H "Authorization: Bearer {token}" \
-H "Content-Type: application/json" \
-d '["execution-id-1", "execution-id-2"]'
```
Returns `{ "executionId": [caseSummary, ...] }`, useful for annotating an execution list with its linked cases.
**Add a comment**
```bash
curl -X POST "https://{host}/api/v1/{tenant}/cases/{id}/comments" \
-H "Authorization: Bearer {token}" \
-F 'comment={"message":"Restarted the service, monitoring now."};type=application/json'
```
Supports up to 5 file attachments (10 MB each) as additional `-F file=@path` parts.
## Current limitations
- Notifications are in-app only: case events appear in the notification bell, but no Slack message or email is sent by Cases. External alerting can be added with standard notification tasks alongside `CreateCase`, or as a case action.
- Deduplication is not atomic: concurrent failures of the same task can occasionally create duplicate cases.
---
# Custom Blueprints in Kestra Enterprise: Templates
URL: https://kestra.io/docs/enterprise/governance/custom-blueprints
> Create Custom Blueprints in Kestra Enterprise. Standardize workflows with private templates, promoting reuse and best practices across your organization.
Custom Blueprints are private, reusable workflow templates that extend the publicly available [Community Blueprints](../../../06.concepts/07.blueprints/index.md).
Custom Blueprints help centralize orchestration patterns, document best practices, and streamline collaboration across your organization.
### How to create a new custom blueprint
From the left navigation menu, go to **Blueprints**, select the **Custom Blueprints** tab, and click **Create**. Add a title, description, and the flow YAML, then click **Create**.
You can edit Blueprints at any time, for example, to add new tasks or expand the documentation.
## Templated Blueprints
Templated Blueprints allow you to create reusable, configurable workflows that users can instantiate without editing YAML. Instead of copying and modifying Blueprints, users fill in guided inputs and Kestra generates the complete flow automatically.
Platform teams build templates once; business users instantiate them by filling in a form rather than editing YAML.
**How it works:** Templated Blueprints use [Pebble templating](../../../06.concepts/06.pebble/index.md), with custom delimiters to avoid conflicts with Kestra expressions.
### Define Template Arguments
Template arguments define the inputs users must provide. To add them to your Blueprint, use the `extend` key with a `templateArguments` section:
```yaml
extend:
templateArguments:
- id: values
displayName: An array of values
type: MULTISELECT
values:
- value1
- value2
- value3
```
All Kestra [input types](../../../05.workflow-components/05.inputs/index.md) and their validation rules are supported. These arguments automatically generate a UI form when the blueprint is instantiated.
### Use Template Arguments
Templated blueprints use the Pebble templating engine. To avoid conflicts with Kestra expressions (`{{ }}`), template arguments use custom delimiters: `<<` and `>>`.
Template arguments are accessed using the `arg` prefix. For example, if you have a template argument with `id: my_custom_field`, you can use it in your flow as follows:
```yaml
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: Hello << arg.my_custom_field >>
```
### Loops and Conditions
You can dynamically generate multiple tasks, inputs, variables, or triggers through for-loops and if/else conditions using the `<% %>` syntax.
For example, the following loop creates one log task for each value in an array input.
```yaml
extend:
templateArguments:
- id: values
displayName: An array of values
type: MULTISELECT
values:
- value1
- value2
- value3
id: myflow
namespace: company.team
tasks:
<% for value in arg.values %>
- id: log_<< value >>
type: io.kestra.plugin.core.log.Log
message: Hello << value >>
<% endfor %>
```
This allows you to dynamically generate tasks or include them conditionally.
### Example: Data Ingestion Template
Here's an example showing a Templated Blueprint that generates data ingestion workflows based on user selections:
:::collapse{title="Template Definition"}
```yaml
id: data-ingest
namespace: kestra.data
extend:
templateArguments:
- id: domains
displayName: Domains
type: MULTISELECT
values:
- Online Shop
- Manufacture
- HR
- Finance
- id: target
type: SELECT
values:
- Postgres
- Oracle
- id: env
type: SELECT
values:
- dev
- staging
- prod
tasks:
- id: parallel_<< arg.env >>
type: io.kestra.plugin.core.flow.Parallel
tasks:
<% for domain in arg.domains %>
- id: sequential_<< domain | slugify >>
type: io.kestra.plugin.core.flow.Sequential
tasks:
- id: << domain | slugify >>-download
type: io.kestra.plugin.jdbc.postgresql.CopyOut
url: jdbc:postgresql://sample_postgres:5432/<>
username: '{{ secret("POSTGRES_USERNAME") }}'
password: '{{ secret("POSTGRES_PASSWORD") }}'
format: CSV
sql: SELECT * FROM public.<< domain | slugify >>
- id: << domain | slugify >>-ingest
<% if arg.target == 'Oracle' %>
type: io.kestra.plugin.jdbc.oracle.Batch
url: jdbc:oracle:thin:@<< arg.env >>:49161:XE
username: '{{ secret("ORACLE_USERNAME") }}'
password: '{{ secret("ORACLE_USERNAME") }}'
from: "{{ << domain | slugify >>-download.uri }}"
table: public.< domain | slugify >>
<% elseif arg.target == 'Postgres' %>
type: io.kestra.plugin.jdbc.postgresql.CopyIn
url: jdbc:postgres://sample_<< arg.target | lower>>:5432/<>
username: '{{ secret("POSTGRES_USERNAME") }}'
password: '{{ secret("POSTGRES_PASSWORD") }}'
format: CSV
from: "{{ outputs.<< domain | slugify >>-download.uri }}"
table: public.< domain | slugify >>
<% endif %>
<% endfor %>
```
:::
:::collapse{title="Generated Flow (after template rendering)"}
After selecting `env: dev`, `domains: [HR, Manufacture]`, and `target: Oracle`, the template generates this complete workflow:
```yaml
id: data-ingest
namespace: kestra.data
tasks:
- id: parallel_dev
type: io.kestra.plugin.core.flow.Parallel
tasks:
- id: sequential_hr
type: io.kestra.plugin.core.flow.Sequential
tasks:
- id: hr-download
type: io.kestra.plugin.jdbc.postgresql.CopyOut
url: jdbc:postgresql://sample_postgres:5432/dev
username: '{{ secret("POSTGRES_USERNAME") }}'
password: '{{ secret("POSTGRES_PASSWORD") }}'
format: CSV
sql: SELECT * FROM public.hr
- id: hr-ingest
type: io.kestra.plugin.jdbc.oracle.Batch
url: jdbc:oracle:thin:@dev:49161:XE
username: '{{ secret("ORACLE_USERNAME") }}'
password: '{{ secret("ORACLE_USERNAME") }}'
from: "{{ hr-download.uri }}"
table: public.< domain | slugify >>
- id: sequential_manufacture
type: io.kestra.plugin.core.flow.Sequential
tasks:
- id: manufacture-download
type: io.kestra.plugin.jdbc.postgresql.CopyOut
url: jdbc:postgresql://sample_postgres:5432/dev
username: '{{ secret("POSTGRES_USERNAME") }}'
password: '{{ secret("POSTGRES_PASSWORD") }}'
format: CSV
sql: SELECT * FROM public.manufacture
- id: manufacture-ingest
type: io.kestra.plugin.jdbc.oracle.Batch
url: jdbc:oracle:thin:@dev:49161:XE
username: '{{ secret("ORACLE_USERNAME") }}'
password: '{{ secret("ORACLE_USERNAME") }}'
from: "{{ manufacture-download.uri }}"
table: public.< domain | slugify >>
```
:::
## Version control for Custom Blueprints
Custom Blueprints can be version-controlled with Git using two dedicated tasks from the `plugin-ee-git` plugin:
- [PushBlueprints](/plugins/plugin-ee-git/io.kestra.plugin.ee.git.PushBlueprints) commits and pushes blueprints from Kestra to a Git repository.
- [SyncBlueprints](/plugins/plugin-ee-git/io.kestra.plugin.ee.git.SyncBlueprints) syncs blueprints from a Git repository into Kestra, treating Git as the single source of truth.
These tasks mirror the [PushFlows and SyncFlows patterns](../../../version-control-cicd/04.git/index.md) used for flows, applied to Custom Blueprints.
### Push blueprints to Git
Use `PushBlueprints` to export your blueprints from Kestra into a Git repository. This is useful for creating backups, reviewing changes via pull requests, or promoting blueprints across environments.
Each blueprint is written as a YAML file to the target `gitDirectory` (default: `_blueprints`). Use the `blueprints` property with glob patterns to push only a subset of blueprints.
```yaml
id: push_blueprints
namespace: system
tasks:
- id: commit_and_push
type: io.kestra.plugin.ee.git.PushBlueprints
url: https://github.com/your-org/blueprints-repo
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
branch: main
commitMessage: "push blueprints from {{ flow.namespace ~ '.' ~ flow.id }}"
triggers:
- id: schedule
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 * * * *"
```
The task outputs a `commitId`, a `commitURL`, and a `blueprints` URI pointing to a diff report that lists the number of lines added, deleted, and changed per file.
### Sync blueprints from Git
Use `SyncBlueprints` to pull blueprints from Git into Kestra. This is the recommended pattern when Git is your single source of truth, for example when platform teams manage approved blueprint libraries centrally and deploy them across multiple Kestra instances.
By default, `SyncBlueprints` only adds and updates blueprints. Set `delete: true` to also remove any blueprints present in Kestra but absent in Git.
```yaml
id: sync_blueprints_from_git
namespace: system
tasks:
- id: git
type: io.kestra.plugin.ee.git.SyncBlueprints
url: https://github.com/your-org/blueprints-repo
branch: main
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
delete: true
dryRun: true
triggers:
- id: every_full_hour
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 * * * *"
```
Set `dryRun: true` to preview what would change without applying it. The `blueprints` output URI contains a row-per-blueprint report showing each blueprint's `syncState`: `ADDED`, `UPDATED`, `UNCHANGED`, or `DELETED`.
Use caution with `delete: true`: it removes all blueprints not present in Git, not just those that differ.
### Blueprint YAML file format
Both tasks read and write blueprints as YAML files. Each file represents one blueprint:
```yaml
id: my-blueprint-id
title: My Blueprint Title
description: Optional description of what this blueprint does
tags:
- tag1
- tag2
flow: |
id: my-flow
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Hello World
```
The `id` field controls how blueprints are matched on sync: if a blueprint with that ID already exists in Kestra, it is updated; if not, it is created with that ID. If `id` is omitted, a new blueprint is created with an auto-generated ID.
---
# Log Shipper in Kestra Enterprise: Centralize Logs
URL: https://kestra.io/docs/enterprise/governance/logshipper
> Centralize monitoring with Kestra Log Shipper. Export workflow and audit logs to Datadog, Splunk, Elastic, AWS S3, and other observability platforms.
Log Shipper exports Kestra execution and audit logs to external observability platforms and SIEMs.
:::alert{type="info"}
**Log Shipper vs External Log Data Store** — these solve different problems. Log Shipper copies logs to third-party observability platforms (Datadog, Splunk, CloudWatch, etc.) for alerting and search. The [External Log Data Store](../../../10.administrator-guide/log-data-store/index.md) routes Kestra's primary log storage to a separate database or Elasticsearch, reducing main database size without any flow or sidecar. Both can be active at the same time.
:::
## Log shipper — centralize logs
Log Shipper can distribute Kestra logs from across your instance to an external logging platform. Log synchronization fetches logs and batches them into optimized chunks automatically. The batch process is done intelligently through defined synchronization points. Once batched, the Log Shipper delivers consistent and reliable data to your monitoring platform.
Log Shipper is built on top of [Kestra plugins](/plugins), ensuring it can integrate with popular logging platforms and expand as more plugins are developed. Supported observability platforms include ElasticSearch, Datadog, New Relic, Azure Monitor, Google Operational Suite, AWS Cloudwatch, Splunk, OpenSearch, Huawei Cloud LTS, OpenTelemetry, Graylog, Dash0, and Syslog (CEF).
## Log shipper properties
The Log Shipper plugin has several key properties to define where the logs should be sent and how they are batched. Below is a list of the definable properties and their purpose:
- `logExporters` - This property is required, and it specifies the platform where the logs will be exported. It supports a list of entries, allowing you to export logs to different platforms at once
- `logLevelFilter` - Specifies the minimum log level to send with the default being `INFO`. With `INFO`, all log levels `INFO` and above (`WARNING` and `ERROR`) are batched. If you only want logs that are warnings or errors, then you can set this property to `WARNING` and so on.
- `lookbackPeriod` - Determines the fetch period for logs to be sent. For example, with a default value of `P1D`, all logs generated between now and one day ago are batched.
- `namespace` - Sets the task to only gather logs from a specific Kestra [Namespace](../../../05.workflow-components/02.namespace/index.md). If not specified, all instance logs are fetched.
- `offsetKey` - Specifies the prefix of the [Key Value (KV) store](../../../06.concepts/05.kv-store/index.md) key that contains the last execution's end fetched date. By default this is set as `LogShipper-state`. You can change this key store name to reset the last fetched date if, for example, you want to export previously exported logs.
- `delete` - By default this property is set to `false`. Boolean property that when set to `true` deletes the batched logs as a part of the task run
## How log shipper works
The following flow exports logs to AWS CloudWatch, Google Operational Suite, and Azure Monitor simultaneously.
```yaml
id: logShipper
namespace: system
tasks:
- id: shipLogs
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: logShipperOffset
logExporters:
- id: awsCloudWatch
type: io.kestra.plugin.ee.aws.cloudwatch.LogExporter
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: us-east-1
logGroupName: kestra
logStreamName: production
chunk: 5000
- id: googleOperationalSuite
type: io.kestra.plugin.ee.gcp.gcs.LogExporter
projectId: my-gcp-project
chunk: 2000
- id: azureMonitor
type: io.kestra.plugin.ee.azure.monitor.LogExporter
endpoint: https://endpoint-host.ingest.monitor.azure.com
tenantId: "{{ secret('AZURE_TENANT_ID') }}"
clientId: "{{ secret('AZURE_CLIENT_ID') }}"
clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
ruleId: dcr-69f0b123041d4d6e9f2bf72aad0b62cf
streamName: kestraLogs
chunk: 1000
```
The plugin starts by identifying the starting timestamp and checking if the last processed log exists. If it does, the plugin uses the `offsetKey` to fetch logs from the database. If the last processed log does not exist, the plugin uses the current time minus the `lookbackPeriod` to fetch logs from the database.
The logs are then distributed to the exporters in chunks of 5000, 2000, and 1000 for AWS CloudWatch, Google Suite, and Azure Monitor, respectively. Once the logs are distributed, the offset key in the Key Value store is updated.

## Log shipper examples
### Kestra `FileLogExporter`
The following example uses Kestra's core `FileLogExporter` plugin to synchronize the logs of the `company.team` namespace. The `synchronize_logs` task outputs a file, and the log file `uri` is passed as an expression in the `upload` task to then upload the logs to an S3 bucket.
```yaml
id: log_shipper_file
namespace: system
tasks:
- id: synchronize_logs
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: LogShipper-local-demo
delete: false
namespace: company.team
logExporters:
- id: file
type: io.kestra.plugin.ee.core.log.FileLogExporter
format: JSON # default ION
maxLinesPerFile: 100
- id: upload
type: io.kestra.plugin.aws.s3.Upload
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
from: "{{ outputs.synchronize_logs.outputs.file.uri }}"
key: logs/kestra.txt
bucket: kestra-log-demo-bucket
region: eu-west-2
```
### Datadog
This example exports logs to [Datadog](https://www.datadoghq.com/) using the default property settings:
```yaml
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
delete: false
logExporters:
- id: DatadogLogExporter
type: io.kestra.plugin.ee.datadog.LogExporter
basePath: '{{ secret("DATADOG_INSTANCE_URL") }}'
apiKey: '{{ secret("DATADOG_APIK_KEY") }}'
```
The batched logs directly populate your Datadog instance like in the following screenshot:

### AWS CloudWatch
This example exports logs to [AWS CloudWatch](https://aws.amazon.com/cloudwatch/):
```yaml
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: log_shipper_aws_cloudwatch_state
delete: false
logExporters:
- id: aws_cloudwatch
type: io.kestra.plugin.ee.aws.cloudwatch.LogExporter
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "{{ vars.region }}"
logGroupName: kestra
logStreamName: kestra-log-stream
```
The logs are viewable in the interface of the specified Log Group and can be examined like in the following screenshot:

### AWS S3
This example exports logs to [AWS S3](https://aws.amazon.com/s3/):
```yaml
id: log_shipper
namespace: system
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
logExporters:
- id: S3LogExporter
type: io.kestra.plugin.ee.aws.s3.LogExporter
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "{{ vars.region }}"
format: JSON
bucket: logbucket
logFilePrefix: kestra-log-file
maxLinesPerFile: 1000000
```
### Google Operational Suite
This example exports logs to [Google Cloud Observability](https://cloud.google.com/products/observability):
```yaml
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: shipLogs
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: logShipperOffset
delete: false
logExporters:
- id: googleOperationalSuite
type: io.kestra.plugin.ee.gcp.operationalsuite.LogExporter
projectId: my-gcp-project
```
This example exports logs to [Google Cloud Storage](https://cloud.google.com/storage?hl=en):
```yaml
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
logExporters:
- id: GCPLogExporter
type: io.kestra.plugin.ee.gcp.gcs.LogExporter
projectId: myProjectId
format: JSON
maxLinesPerFile: 10000
bucket: my-bucket
logFilePrefix: kestra-log-file
```
### Azure Monitor
This example exports logs to [Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/overview):
```yaml
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: shipLogs
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: logShipperOffset
delete: false
logExporters:
- id: azureMonitor
type: io.kestra.plugin.ee.azure.monitor.LogExporter
endpoint: https://endpoint-host.ingest.monitor.azure.com
tenantId: "{{ secret('AZURE_TENANT_ID') }}"
clientId: "{{ secret('AZURE_CLIENT_ID') }}"
clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
ruleId: dcr-69f0b123041d4d6e9f2bf72aad0b62cf
streamName: kestraLogs
```
### Azure Blob Storage
This example exports logs to [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/):
```yaml
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
logExporters:
- id: AzureLogExporter
type: io.kestra.plugin.ee.azure.storage.LogExporter
endpoint: https://myblob.blob.core.windows.net/
tenantId: tenant_id
clientId: client_id
clientSecret: client_secret
containerName: logs
format: JSON
logFilePrefix: kestra-log-file
maxLinesPerFile: 1000000
```
### Elasticsearch
This example exports logs to [Elasticsearch](https://www.elastic.co/observability):
```yaml
id: logShipper
namespace: system
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: shipLogs
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: logShipperOffset
delete: false
logExporters:
- id: elasticsearch
type: io.kestra.plugin.elasticsearch.LogExporter
indexName: kestra-logs
connection:
basicAuth:
password: "{{ secret('ES_PASSWORD') }}"
username: kestra_user
hosts:
- https://elastic.example.com:9200
```
### New Relic
This example exports logs to [New Relic](https://newrelic.com/platform):
```yaml
id: logShipper
namespace: system
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: shipLogs
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: logShipperOffset
delete: false
logExporters:
- id: newRelic
type: io.kestra.plugin.ee.newrelic.LogExporter
basePath: https://log-api.newrelic.com
apiKey: "{{ secret('NEWRELIC_API_KEY') }}"
```
### Splunk
This example exports logs to [Splunk Observability Cloud](https://www.splunk.com/en_us/products/observability-cloud.html):
```yaml
id: log_shipper
namespace: system
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: logShipperOffset
delete: false
logExporters:
- id: SplunkLogExporter
type: io.kestra.plugin.ee.splunk.LogExporter
host: https://example.splunkcloud.com:8088
token: "{{ secret('SPLUNK_API_KEY') }}"
```
### OpenSearch
This example exports logs to the [OpenSearch observability platform](https://opensearch.org/platform/observability/index.html):
```yaml
id: log_shipper
namespace: system
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: logSync
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: logShipperOffset
delete: false
logExporters:
- id: OpensearchLogExporter
type: io.kestra.plugin.ee.opensearch.LogExporter
connection:
hosts:
- "http://localhost:9200/"
indexName: "logs"
```
### OpenTelemetry
This example exports logs to an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/):
```yaml
id: logShipper
namespace: system
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: shipLogs
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: logShipperOffset
delete: false
logExporters:
- id: openTelemetry
type: io.kestra.plugin.ee.opentelemetry.LogExporter
otlpEndpoint: http://otel-collector:4318/v1/logs
authorizationHeaderName: Authorization
authorizationHeaderValue: "Bearer {{ secret('OTEL_TOKEN') }}"
```
### Graylog
This example exports logs to [Graylog](https://graylog.org/) using a GELF HTTP input. See the [Graylog Plugin Documentation](/plugins/plugin-ee-graylog) for the full property reference.
```yaml
id: log_shipper
namespace: system
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
logExporters:
- id: GraylogExporter
type: io.kestra.plugin.ee.graylog.LogExporter
endpoint: "http://localhost:12201/gelf"
graylogHost: "Kestra"
chunk: 1000
```
### Huawei Cloud LTS
This example exports logs to [Huawei Cloud Log Tank Service (LTS)](https://www.huaweicloud.com/intl/en-us/product/lts.html). See the [Huawei EE Plugin Documentation](/plugins/plugin-ee-huawei) for the full property reference.
```yaml
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: logShipperOffset
delete: false
logExporters:
- id: huawei_lts
type: io.kestra.plugin.ee.huawei.lts.LogExporter
region: eu-west-101
projectId: "{{ secret('HUAWEI_PROJECT_ID') }}"
accessKeyId: "{{ secret('HUAWEI_ACCESS_KEY_ID') }}"
secretAccessKey: "{{ secret('HUAWEI_SECRET_ACCESS_KEY') }}"
logGroupId: "{{ secret('HUAWEI_LOG_GROUP_ID') }}"
logStreamId: "{{ secret('HUAWEI_LOG_STREAM_ID') }}"
chunk: 1000
```
The Huawei Cloud LTS exporter authenticates with Huawei Cloud using either static AK/SK credentials (`accessKeyId` and `secretAccessKey`), a pre-obtained `securityToken`, or inline IAM STS credential exchange via `temporaryCredentials`. The `logGroupId` and `logStreamId` are UUIDs that you can copy from the LTS console (they are not the human-readable group and stream names).
### Syslog (CEF)
This example exports logs to a syslog-compatible SIEM such as ArcSight, QRadar, or Trellix using [Common Event Format (CEF)](https://www.microfocus.com/documentation/arcsight/arcsight-smartconnectors-8.4/cef-implementation-standard/) over TCP. Refer to the [Syslog Plugin Documentation](/plugins/plugin-ee-syslog) for the full property reference.
```yaml
id: log_shipper
namespace: system
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: syslog-log-offset
logExporters:
- id: syslog
type: io.kestra.plugin.ee.syslog.LogExporter
host: "{{ secret('SIEM_HOST') }}"
port: 601
protocol: TCP
```
For TLS transport — recommended for production — use port 6514 and configure the truststore:
```yaml
id: log_shipper_tls
namespace: system
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
logExporters:
- id: syslog
type: io.kestra.plugin.ee.syslog.LogExporter
host: "{{ secret('SIEM_HOST') }}"
port: 6514
protocol: TLS
trustStore: "{{ secret('SIEM_TRUSTSTORE_BASE64') }}"
trustStorePassword: "{{ secret('SIEM_TRUSTSTORE_PASSWORD') }}"
```
:::alert{type="warning"}
`skipCertVerification: true` disables certificate validation. Use it only in lab environments with self-signed certificates, never in production.
:::
To ship [Audit Logs](../06.audit-logs/index.md) to your SIEM, use the same exporter with `AuditLogShipper` — replace `logLevelFilter` with `resources`.
### Dash0
This example exports logs to [Dash0](https://www.dash0.com/) via OTLP/HTTP. Set `endpoint` to the ingestion URL for your Dash0 region. Set `dataset` to route logs to a named dataset, or omit it to use the Dash0 `default` dataset.
```yaml
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
logExporters:
- id: dash0LogExporter
type: io.kestra.plugin.ee.dash0.LogExporter
endpoint: https://ingress.eu-west-1.aws.dash0.com/v1/logs
authToken: "{{ secret('DASH0_AUTH_TOKEN') }}"
dataset: my-dataset
```
## Audit log shipper
Use [Audit Log Shipper](../06.audit-logs/index.md) to ship audit logs from the Kestra backend to external destinations including Datadog, Elasticsearch, New Relic, OpenTelemetry, AWS CloudWatch, Google Operational Suite, Azure Monitor, and Syslog (CEF).
The Audit Log Shipper uses the following properties similar to the execution Log Shipper, except that the `resources` property replaces the `logLevelFilter` property.
- `logExporters` - This property is required, and it specifies the platform where the audit logs will be exported. It supports a list of entries, allowing you to export logs to different platforms at once
- `resources` - Specifies from which Kestra resource to ship audit logs for (e.g., FLOW, EXECUTION, USER, KV STORE, etc.)
- `lookbackPeriod` - Determines the fetch period for audit logs to be sent. For example, with a default value of `P1D`, all audit logs generated between now and one day ago are batched.
- `offsetKey` - Specifies the [key](../../../06.concepts/05.kv-store/index.md) that contains the last fetched date. By default, Kestra uses the key `LogShipper-state`. You can change the value of that KV pair if you want to export previously fetched logs again.
- `delete` - Boolean property that, when set to `true`, deletes the logs from Kestra’s database immediately after successful export, helping optimize storage by removing logs that no longer need to reside in Kestra’s metadata store. By default, this property is set to `false`.
The following workflow ships audit logs to multiple destinations simultaneously.
```yaml
id: Audit-logShipper
namespace: system
tasks:
- id: shipLogs
type: io.kestra.plugin.ee.core.log.AuditLogShipper
resources:
- FLOW
- EXECUTION
lookbackPeriod: P1D
offsetKey: logShipperOffset
logExporters:
- id: file
type: io.kestra.plugin.ee.core.log.FileLogExporter
- id: awsCloudWatch
type: io.kestra.plugin.ee.aws.cloudwatch.LogExporter
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: us-east-1
logGroupName: kestra
logStreamName: production
- id: googleOperationalSuite
type: io.kestra.plugin.ee.gcp.operationalsuite.LogExporter
projectId: my-gcp-project
- id: azureMonitor
type: io.kestra.plugin.ee.azure.monitor.LogExporter
endpoint: https://endpoint-host.ingest.monitor.azure.com
tenantId: "{{ secret('AZURE_TENANT_ID') }}"
clientId: "{{ secret('AZURE_CLIENT_ID') }}"
clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
ruleId: dcr-69f0b123041d4d6e9f2bf72aad0b62cf
streamName: kestraLogs
- id: datadog
type: io.kestra.plugin.ee.datadog.LogExporter
basePath: https://http-intake.logs.datadoghq.eu
apiKey: "{{ secret('DATADOG_API_KEY') }}"
- id: elasticsearch
type: io.kestra.plugin.elasticsearch.LogExporter
indexName: kestra-logs
connection:
basicAuth:
password: "{{ secret('ES_PASSWORD') }}"
username: kestra_user
hosts:
- https://elastic.example.com:9200
- id: newRelic
type: io.kestra.plugin.ee.newrelic.LogExporter
basePath: https://log-api.newrelic.com
apiKey: "{{ secret('NEWRELIC_API_KEY') }}"
- id: openTelemetry
type: io.kestra.plugin.ee.opentelemetry.LogExporter
otlpEndpoint: http://otel-collector:4318/v1/logs
authorizationHeaderName: Authorization
authorizationHeaderValue: "Bearer {{ secret('OTEL_TOKEN') }}"
triggers:
- id: dailySchedule
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 0 * * *"
disabled: true
```
---
# Namespace Management in Kestra Enterprise: Isolation
URL: https://kestra.io/docs/enterprise/governance/namespace-management
> Secure your Kestra instance with Namespace Management. Configure isolated environments, manage secrets, and set Namespace-level plugin defaults.
Namespaces provide an additional layer of isolation for secrets, variables, and plugin defaults within a tenant.
## Namespace management — secure configuration
Kestra is a [multi-tenant](../../02.governance/tenants/index.md) platform. Each tenant can have multiple Namespaces, and each Namespace provides additional isolation and security.
Namespaces provide:
- Logical isolation of resources on top of instance- or tenant-level isolation
- Fine-grained access control for secrets, variables, and task configurations
Namespaces are particularly useful in environments with many users, teams, projects, and applications.
## Namespace-level features
The Namespace page allows you to configure secrets, plugin defaults, and variables that can be used within any flow in that Namespace.
It allows your organization to centrally manage your secrets, variables, and task configuration while providing fine-grained access-control to those resources.
Since Kestra supports [everything as code and from the UI](https://youtu.be/dU3p6Jf5fMw?si=bqNWS1e3_if-mePS), you can manage Namespaces from the UI or programmatically (e.g., via our [Terraform provider](https://registry.terraform.io/providers/kestra-io/kestra/latest/docs)).
### Secrets
On the namespace page, go to the **Secrets** tab, click **Add a secret**, enter a key and value, and save.
The secret key now appears on the **Secrets** tab. Edit or delete it using the action buttons on the right. Reference the secret in flows using its key, for example, `"{{ secret('MYSQL_PASSWORD') }}"`.
For APIs that issue short-lived access tokens (e.g., OAuth2), create a [Credential](../../03.auth/credentials/index.md) that relies on these secrets and fetch the token in flows with `{{ credential('your_credential_key') }}`.
Here is how you can use it in a flow:
```yaml
id: query-mysql
namespace: company.team
tasks:
- id: query
type: io.kestra.plugin.jdbc.mysql.Query
url: jdbc:mysql://localhost:3306/test
username: root
password: "{{ secret('MYSQL_PASSWORD') }}"
sql: select * from employees
fetchOne: true
```
:::alert{type="info"}
Make sure to only use the secret in flows defined in the same Namespace (or child Namespace) as your secret.
:::
When building new flows in a Namespace, Namespace secrets are accessible from the **Secrets** tab. Open the tab to view all available Namespace secret key names.
### Policies
[Policies](../policies/index.md) can be defined at the Namespace level to inject, restrict, or validate configuration for all flows in the Namespace. On the namespace page, open the **Policies** tab to create and manage Policies.
Policies can reference secrets and variables defined in the same Namespace.
For example, a namespace-scoped Policy can inject database credentials into every MySQL task so flows don't need to declare them individually:
```yaml
id: mysql-credentials
description: "Inject MySQL credentials for all MySQL tasks in this namespace."
enforcement: ACTIVE
rules:
- type: io.kestra.plugin.ee.rules.Add
on: PLUGIN
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.jdbc.mysql
values:
url: jdbc:mysql://localhost:3306/test
username: root
password: "{{ secret('MYSQL_PASSWORD') }}"
```
With this Policy applied, flows in the Namespace need no credentials on the task:
```yaml
id: query-mysql
namespace: company.team
tasks:
- id: query
type: io.kestra.plugin.jdbc.mysql.Query
sql: select * from employees
fetchOne: true
```
Namespace-level Policies are inherited by child Namespaces. A Policy created in a parent Namespace applies to all flows in the parent and every child Namespace under it. See [Policies](../policies/index.md) for the full DSL reference, enforcement modes, and inheritance behavior.
### Default service account for SDK plugins
Namespaces can now provide **default authentication credentials** that [SDK-based plugins](/plugins/plugin-kestra) use to run tasks such as [List all Namespaces](/plugins/plugin-kestra/kestra-namespaces/io.kestra.plugin.kestra.namespaces.list). This allows tasks relying on the [Kestra SDK](../../../api-reference/kestra-sdk/index.mdx) to call the API without hard-coding credentials inside the flow.
On the Namespace **Edit** page, open the **Default authentication** section and choose either:
- **API token** (recommended), or
- **Basic auth** (username/password)
### Variables
Variables defined at the Namespace level can be used in any flow defined under the same Namespace using the syntax: `{{ namespace.variable_name }}`.
On the namespace page, go to the **Variables** tab, define the variables, and save.
Here is an example flow where the Namespace variable is used:
```yaml
id: query-mysql
namespace: company.team
tasks:
- id: query
type: io.kestra.plugin.jdbc.mysql.Query
url: jdbc:mysql://localhost:3306/test
username: "{{ namespace.mysql_user }}"
sql: select * from employees
fetchOne: true
```
When building new flows in a Namespace, Namespace variables are accessible from the **Variables** tab.
## Creating Namespaces
### From the UI
The video below shows how to create a namespace and add:
- several new secrets
- a nested Namespace variable that references one of these secrets
- a list of plugin defaults helping to use those pre-configured secrets and variables in all the tasks from the AWS and Git plugins.
### From Terraform
The following example reproduces those steps in Terraform.
To create a Namespace from Terraform, use the [kestra_namespace](https://registry.terraform.io/providers/kestra-io/kestra/latest/docs) resource.
First, configure your Terraform backend and add Kestra as a required provider:
```hcl
terraform {
backend "s3" {
bucket = "kestraio"
key = "terraform.tfstate"
region = "us-east-1"
}
required_providers {
kestra = {
source = "kestra-io/kestra"
version = "~>0.14"
}
}
}
provider "kestra" {
url = var.kestra_host
username = var.kestra_user
password = var.kestra_password
tenant_id = var.kestra_tenant_id # only if you are using multi-tenancy
}
```
You can add a file `main.tf` to your Terraform project with the following content:
```hcl
resource "kestra_namespace" "marketing" {
namespace_id = "marketing"
description = "Namespace for the marketing team"
}
```
The only required property is the `namespace_id`, which is the name of the Namespace. The `description` and all other properties are optional.
#### Adding variables and plugin defaults to a Namespace Terraform resource
You can add variables and plugin defaults directly to the Namespace resource by pointing to the YAML configuration files.
First, create the `variables_marketing.yml` file:
```yaml
github:
token: "{{ secret('GITHUB_TOKEN') }}"
```
Then, create another file for `plugin_defaults_marketing.yml`:
```yaml
- type: io.kestra.plugin.aws
values:
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
region: us-east-1
secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
- type: io.kestra.plugin.git
values:
password: "{{ render(namespace.github.token) }}"
username: your-github-username
```
Finally, reference those files in your Namespace resource definition:
```hcl
resource "kestra_namespace" "marketing" {
namespace_id = "marketing"
description = "Namespace for the marketing team"
variables = file("variables_marketing.yml")
plugin_defaults = file("plugin_defaults_marketing.yml")
}
```
#### Adding secrets to a Namespace using Terraform
To programmatically add secrets to your Namespace via [Terraform](https://registry.terraform.io/providers/kestra-io/kestra/latest/docs), you can use the [kestra_namespace_secret](../../../13.terraform/resources/namespace_secret/index.md) resource. Here is an example of adding multiple secrets to the `marketing` Namespace:
```hcl
resource "kestra_namespace_secret" "github_token" {
namespace = "marketing"
secret_key = "GITHUB_TOKEN"
secret_value = var.github_token
}
resource "kestra_namespace_secret" "aws_access_key_id" {
namespace = "marketing"
secret_key = "AWS_ACCESS_KEY_ID"
secret_value = var.aws_access_key_id
}
resource "kestra_namespace_secret" "aws_secret_access_key" {
namespace = "marketing"
secret_key = "AWS_SECRET_ACCESS_KEY"
secret_value = var.aws_secret_access_key
}
```
Before referencing variables in your Terraform configuration, make sure to define them in your `variables.tf` file:
```hcl
variable "github_token" {
type = string
sensitive = true
}
variable "aws_access_key_id" {
type = string
sensitive = true
}
variable "aws_secret_access_key" {
type = string
sensitive = true
}
variable "kestra_user" {
type = string
sensitive = true
}
variable "kestra_password" {
type = string
sensitive = true
}
variable "kestra_host" {
type = string
sensitive = false
default = "http://your_kestra_host:8080" # Change this to your Kestra host URL
}
variable "kestra_tenant_id" {
type = string
sensitive = false
default = "kestra-tech"
}
```
And add your secrets to the `terraform.tfvars` file:
```hcl
github_token = "your-github-token"
aws_access_key_id = "your-aws-access-key-id"
aws_secret_access_key = "your-aws-secret-access-key"
kestra_user = "your-kestra-user"
kestra_password = "your-kestra-password"
```
## Allowed Namespaces
On the **Edit** tab of any namespace, configure which namespaces are allowed to access its flows and resources.
By default, **all Namespaces** are allowed. To restrict access, **select specific Namespaces** — access automatically extends to each selected namespace's children.
---
# Policies
URL: https://kestra.io/docs/enterprise/governance/policies
> Use Policies to inject, validate, and enforce plugin and flow configuration across namespaces and tenants. Policies give platform administrators centralized, observable control over how flows run.
Policies let platform administrators inject, validate, and enforce configuration rules across namespaces and tenants.
A Policy bundles rules that inject property values, strip properties, restrict allowed values, or block non-compliant flows from saving or running.
## Why Policies
Without Policies, ensuring that flows across many namespaces use approved configurations requires manual coordination; authors must set values correctly on every task, and administrators have no way to enforce or verify compliance.
Policies address this with a model that supports:
- **Mutate rules** — inject or remove property values before execution, on flows and plugins alike
- **Validate rules** — block or warn when flows violate configuration requirements
- **Enforcement modes** — control whether a Policy actively blocks, only reports, or is opt-in
- **Inheritance** — rules from parent namespaces apply to children, and children can only tighten them
## Policy structure
A Policy has an `id`, an optional `description`, an optional `displayName` shown in the Governance UI, an `enforcement` mode, an optional `target` that narrows which tenants or namespaces the policy applies to within its scope, and a list of `rules`.
```yaml
id: prod-cost-controls
displayName: "Prod Cost Controls"
description: "Cost guardrails for prod flows."
enforcement: ACTIVE
rules:
- type: io.kestra.plugin.ee.rules.Add
on: FLOW
values:
concurrency:
limit: 5
- type: io.kestra.plugin.ee.rules.Restrict
on: FLOW
property: concurrency.limit
max: 10
errorMessage: "concurrency.limit cannot exceed 10 in prod."
```
A Policy has no `type` field of its own — each rule has a `type` that selects one of the five rule types. A single policy can mix mutate and validate rules, which lets a coherent intent (such as "all prod flows: inject a default concurrency, then block values above 10") live in one place.
## Enforcement modes
| Mode | Behavior |
|------|----------|
| `ACTIVE` | Rules are enforced on save and before execution. A `block` violation on a caller-initiated route (API, UI, webhook) returns `403` with the policy reason and creates no execution. System-initiated routes (schedule, flow trigger) skip the execution silently and log the block. Default. |
| `EVALUATE` | Validate rule violations are reported in the Governance UI without blocking. Mutate rules (`Add`, `Delete`) are suppressed — no values are injected or removed, and the Governance UI shows no violations for them. Use this mode to audit validate rule compliance before enabling enforcement; it is not useful for auditing mutate rule coverage. |
| `DISABLED` | The Policy is inactive. Rules are not checked. |
| `REFERENCE` | The Policy is opt-in. It only applies to flows or tasks that explicitly list it via `policyRefs:`. Use reference policies for opt-in configuration injection (`Add` rules); validate rules are not enforced. |
Start new validate policies in `EVALUATE` mode to see which flows already violate the rule before blocking saves. Once violation counts are acceptable, switch to `ACTIVE`.
```yaml
id: label-audit
enforcement: EVALUATE
rules:
- type: io.kestra.plugin.ee.rules.Require
on: FLOW
properties:
- labels.team
errorMessage: "Every flow must declare labels.team."
```
## Rule targeting
Every rule has two targeting fields: `on` selects whether the rule applies to the flow itself or to plugin instances within it, and `where` filters which plugin instances match.
### `on` field
Each rule declares `on: FLOW` to target the flow's own properties, or `on: PLUGIN` to target every plugin instance in the flow — tasks including nested ones, triggers, and task runners.
There is no flow-level `where:` filter. To target flows, use scope placement: place the Policy at the namespace that owns the subtree, and inheritance carries it down. A tenant-scoped policy applies to every flow in the tenant.
### `where` clause
The `where` clause narrows which plugin instances a rule applies to. It is only valid on `on: PLUGIN` rules. Each condition specifies a `field`, an `operator`, and a `value`. Multiple conditions combine with AND.
| Operator | Behavior |
|----------|----------|
| `EQUAL_TO` | Exact match |
| `NOT_EQUAL_TO` | Not an exact match |
| `STARTS_WITH` | Prefix match |
| `ENDS_WITH` | Suffix match |
| `CONTAINS` | Substring match |
| `IN` | Value is in a list |
| `NOT_IN` | Value is not in a list |
| `REGEX` | Full regular expression match |
| `IS_NULL` | Property is null or absent |
| `IS_NOT_NULL` | Property is set and non-null |
```yaml
# Prefix match — all Python script plugins
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.scripts.python
# Match a specific set of plugin types
where:
- field: type
operator: IN
value:
- io.kestra.plugin.gcp.bigquery.Query
- io.kestra.plugin.gcp.bigquery.Load
- io.kestra.plugin.gcp.bigquery.ExtractToGcs
```
`field` is a property path on the matched target — `type` is the most common, matching on the plugin's class name.
## Rule types
Rules come in two families: mutate rules change the resolved configuration before execution, and validate rules check it at save time and before execution.
### Mutate rules
Mutate rules change the resolved configuration before execution. They do not alter the stored flow YAML — the injection or deletion happens at resolution time. Every mutation is visible to the author in the flow editor's merged preview.
**`io.kestra.plugin.ee.rules.Add`**: inject values into flows or plugin instances. With `override: false` (the default), the author's explicit value wins and the policy injects only when the property is absent. With `override: true`, the policy value always wins.
Add rules can target flow-level properties (like `retry`, `concurrency`, or `labels`) with `on: FLOW`, or individual plugin instances with `on: PLUGIN`.
```yaml
# Inject flow-level defaults — retry, concurrency, and a team label
- type: io.kestra.plugin.ee.rules.Add
on: FLOW
values:
retry:
type: constant
maxAttempts: 3
interval: PT1M
concurrency:
limit: 5
labels:
team: data-platform
# Inject credentials into every AWS plugin instance
- type: io.kestra.plugin.ee.rules.Add
on: PLUGIN
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.aws
values:
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
# Force a region value — override any author-supplied value
- type: io.kestra.plugin.ee.rules.Add
on: PLUGIN
override: true
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.aws
values:
region: eu-west-1
```
Injected `values` may contain Pebble expressions such as `{{ secret('…') }}`. These are evaluated at runtime as plugin properties.
:::alert{type="info"}
When injecting a nested object, keys from the flow and the policy are merged. If both supply a list at the same path, the policy's list replaces the flow's list entirely — lists are not combined.
:::
**`io.kestra.plugin.ee.rules.Delete`**: remove specific properties from matching flows or plugin instances at resolution time. Delete always wins over an author-supplied value, and is always visible in the merged preview.
```yaml
# Strip hardcoded credentials — a central injection policy supplies them
- type: io.kestra.plugin.ee.rules.Delete
on: PLUGIN
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.aws
properties:
- accessKeyId
- secretKeyId
```
:::alert{type="warning"}
If an `Add` and a `Delete` rule both target the same property in the same scope chain, Kestra raises a conflict error at save time. Express delete-then-re-inject as a single `Add` with `override: true` instead.
:::
### Validate rules
Validate rules check flow or plugin properties at save time and before execution. Each validate rule has an `action` of `block` (default) or `warn`.
:::alert{type="info"}
Validation runs after all `Add` and `Delete` rules have been applied. An `Add` rule can satisfy a `Require` rule, and a `Restrict` rule catches a bad value regardless of whether the author or a Policy supplied it.
:::
**`io.kestra.plugin.ee.rules.Deny`**: reject a plugin type wholesale. Requires `on: PLUGIN`. Matches every instance of the plugin wherever it appears — tasks, error handlers, triggers, and task runners.
```yaml
- type: io.kestra.plugin.ee.rules.Deny
on: PLUGIN
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.gcp.bigquery
errorMessage: "BigQuery plugins are disabled — contact the data platform team."
```
**`io.kestra.plugin.ee.rules.Restrict`**: constrain a property to an allowed range or set of values. Requires at least one of: `min`, `max`, `enum`, `regex`, `equals`. If the property is absent, no violation is raised — pair with a `Require` or `Add` rule for guaranteed presence.
```yaml
# Cap concurrency on flow-level properties
- type: io.kestra.plugin.ee.rules.Restrict
on: FLOW
property: concurrency.limit
max: 10
errorMessage: "concurrency.limit cannot exceed 10 in prod."
# Allow only approved task runners
- type: io.kestra.plugin.ee.rules.Restrict
on: PLUGIN
property: taskRunner.type
enum:
- io.kestra.plugin.scripts.runner.docker.Docker
- io.kestra.plugin.scripts.runner.kubernetes.Kubernetes
errorMessage: "Only Docker and Kubernetes task runners are allowed in prod."
# Enforce a naming convention — warn only
- type: io.kestra.plugin.ee.rules.Restrict
on: FLOW
property: id
regex: "^[a-z][a-z0-9]*(-[a-z0-9]+)*$"
action: warn
errorMessage: "Flow ids should be kebab-case (e.g. daily-orders-sync)."
```
**`io.kestra.plugin.ee.rules.Require`**: mandate that one or more properties are set and non-empty. Empty string, whitespace-only, `[]`, and `{}` do not satisfy the rule; `0` and `false` do.
```yaml
# Every flow must carry a team label
- type: io.kestra.plugin.ee.rules.Require
on: FLOW
properties:
- labels.team
errorMessage: "Every flow must declare labels.team."
# Every script task must declare an explicit task runner
- type: io.kestra.plugin.ee.rules.Require
on: PLUGIN
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.scripts
properties:
- taskRunner
errorMessage: "Every script task must declare an explicit taskRunner."
```
#### Combining Add and Require
Use `Add` with `Require` when you want tasks that don't declare a property to receive a sensible default, while still blocking tasks that explicitly omit it. Validation runs after all mutate rules, so an `Add` rule can satisfy a `Require` rule.
```yaml
id: log-level-default
enforcement: ACTIVE
rules:
# Inject a default log level when the author hasn't set one
- type: io.kestra.plugin.ee.rules.Add
on: PLUGIN
where:
- field: type
operator: EQUAL_TO
value: io.kestra.plugin.core.log.Log
values:
level: WARN
# Require level to be set — satisfied by the Add rule above for tasks that omit it
- type: io.kestra.plugin.ee.rules.Require
on: PLUGIN
where:
- field: type
operator: EQUAL_TO
value: io.kestra.plugin.core.log.Log
properties:
- level
errorMessage: "Log tasks must declare an explicit level."
```
Log tasks without a `level` receive `WARN` from the `Add` rule, satisfying the `Require`. Tasks that explicitly set `level` keep their value.
## Override behavior
For `Add` rules:
- `override: false` (default) — if the author sets a value, it is used. The policy injects only when the property is absent.
- `override: true` — the policy value always wins. The replacement is annotated in the merged preview so the author can see it.
When multiple policies with `override: true` apply to the same property, the outermost scope wins: a tenant-level policy with `override: true` cannot be overridden by a namespace-level policy.
### Choosing between `override: true` and `Restrict(equals:)`
Both approaches pin a property to a specific value. The difference is posture:
| | `Add(override: true)` | `Add` + `Restrict(equals:)` |
|-|----------------------|----------------------------|
| Author sets a different value | Silently corrected at runtime; visible in merged preview | Rejected at save; author is told to fix it |
| Requires | One rule | Two rules in the same policy |
| Personas | Platform teams — "just make it right" | Compliance teams — "make the author fix it" |
Both can live in the same policy.
## Reference policies
A Policy with `enforcement: REFERENCE` is not applied automatically. It applies only to flows or tasks that explicitly list it using `policyRefs:`. This lets teams share opt-in configuration bundles — named runner profiles, database connections, or compliance defaults that flows can adopt voluntarily.
```yaml
# Policy: db-analytics
id: db-analytics
description: "Analytics warehouse connection."
enforcement: REFERENCE
rules:
- type: io.kestra.plugin.ee.rules.Add
on: PLUGIN
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.jdbc.postgresql
values:
url: jdbc:postgresql://analytics:5432/dw
username: "{{ secret('DW_USER') }}"
password: "{{ secret('DW_PASSWORD') }}"
```
```yaml
# Policy: db-orders
id: db-orders
description: "Orders OLTP connection."
enforcement: REFERENCE
rules:
- type: io.kestra.plugin.ee.rules.Add
on: PLUGIN
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.jdbc.postgresql
values:
url: jdbc:postgresql://orders:5432/oltp
username: "{{ secret('ORDERS_USER') }}"
password: "{{ secret('ORDERS_PASSWORD') }}"
```
```yaml
# Attach the bundle per task — each task opts into the policy it needs
id: orders-to-warehouse
namespace: acme.data
tasks:
- id: extract
type: io.kestra.plugin.jdbc.postgresql.Query
policyRefs:
- db-orders
- id: load
type: io.kestra.plugin.jdbc.postgresql.Query
policyRefs:
- db-analytics
```
`policyRefs` is valid at the flow level, on individual tasks, and on triggers. The attachment level determines which rules in the bundle activate:
- **Flow-level `policyRefs`** — activates all rules in the bundle: both `on: FLOW` and `on: PLUGIN` rules apply across the entire flow.
- **Task- or trigger-level `policyRefs`** — activates only `on: PLUGIN` rules, scoped to that plugin and its nested configuration. `on: FLOW` rules in the same bundle are filtered out at rule-scoping time and are never enforced.
Flow-level `policyRefs` applies the bundle to all tasks in the flow:
```yaml
# Flow-level policyRefs — applies the bundle to all tasks
id: daily-report
namespace: company.team
policyRefs:
- pydata-defaults
tasks:
- id: transform
type: io.kestra.plugin.scripts.python.Script
script: |
print("done")
```
Triggers also support `policyRefs`:
```yaml
id: event-pipeline
namespace: acme.data
tasks:
- id: process
type: io.kestra.plugin.jdbc.postgresql.Query
sql: SELECT 1
triggers:
- id: on-file
type: io.kestra.plugin.aws.s3.Trigger
policyRefs:
- aws-s3-credentials
bucket: my-bucket
prefix: "data/"
```
:::alert{type="warning"}
If a referenced bundle contains `on: FLOW` rules and `policyRefs` is on a task or trigger, those rules are silently dropped. Place `policyRefs` at the flow level to activate them.
:::
Reference policies do not propagate through namespace inheritance — flows and tasks must opt in explicitly using `policyRefs:`. Reference policies are designed for opt-in configuration injection: use `Add` rules to supply credentials, defaults, or runner configuration that teams can adopt voluntarily. Validate rules (`Deny`, `Restrict`, `Require`) are not enforced in `REFERENCE` mode.
## Visibility
The Policy overview page (accessible from the tenant **Policies** menu or **Namespaces → [namespace] → Policies**) lists every flow violating the policy's validate rules, with a finding count per flow and a **Fix** button.
The Fix dialog shows:
- The policy and flow names
- The rule action (`block` or `warn`) and your `errorMessage`
- The rule type and the property path that triggered the violation
- A diff preview of the suggested change
`block` and `warn` findings require manual corrections — use **Open flow** to go directly to the flow editor. The **Apply** button applies auto-fixable suggestions where available.
When saving a flow violates an `ACTIVE` Policy, Kestra rejects the save and shows the policy name, which rule was violated, and the admin-authored `errorMessage`. The flow is never auto-disabled; it must be corrected before it can be saved. Warnings from `action: warn` rules are surfaced separately without blocking the save.
When a flow is open in the editor, the Policies panel shows a **Mutation preview** toggle that surfaces injected values from `Add` rules, `override: true` replacements, and `Delete` removals, each annotated with the source policy that supplied them.
## Policy scope and inheritance
Policies apply along a scope chain, from outermost to innermost:
| Scope | Where defined | Who can edit |
|---|---|---|
| `STATIC` | `kestra.policies` in server configuration | Infrastructure / platform team |
| `INSTANCE` | Tenant root (no specific namespace) | Instance Owner |
| `TENANT` | Tenant-level via API or UI | Tenant admin |
| `NAMESPACE` | Namespace-level via API or UI | Namespace admin with `POLICY` permission |
An optional `target` field narrows a policy's reach within its scope. Which sub-field is valid depends on the scope:
| Scope | Valid `target` sub-field | Effect |
|---|---|---|
| `STATIC`, `INSTANCE` | `tenants` | Apply only to the listed tenants; absent = every tenant |
| `TENANT` | `namespaces` | Apply only to the listed namespace subtrees; absent = whole tenant |
| `NAMESPACE` | — | `target` is not valid and will be rejected |
**`target.namespaces` uses ancestor-chain matching.** Listing `analytics` covers `analytics` itself and every descendant (`analytics.finance`, `analytics.finance.reports`, and so on). Listing `analytics.finance` covers only that subtree. You do not need to list child namespaces explicitly.
```yaml
# TENANT policy scoped to the `analytics` subtree and the `ml` namespace
id: data-team-policy
enforcement: ACTIVE
target:
namespaces:
- analytics
- ml
rules:
- type: io.kestra.plugin.ee.rules.Require
on: FLOW
properties:
- labels.team
errorMessage: "Every flow must declare labels.team."
```
```yaml
# INSTANCE policy scoped to specific tenants
id: prod-only-controls
enforcement: ACTIVE
target:
tenants:
- prod
- staging
rules:
- type: io.kestra.plugin.ee.rules.Restrict
on: FLOW
property: concurrency.limit
max: 20
errorMessage: "concurrency.limit cannot exceed 20."
```
A `target` must carry at least one entry — an empty `tenants: []` or `namespaces: []` is invalid.
Policies from parent namespaces automatically apply to all child namespaces. Children can add stricter validate rules but cannot relax rules inherited from a parent.
For `override: false` `Add` rules, the innermost scope wins — a namespace policy fills a property that a tenant policy left unset. For `override: true` `Add` rules, the outermost scope wins — a static or tenant policy with `override: true` cannot be overridden by a namespace policy.
## Static policies
Static policies are declared in server configuration under `kestra.policies`. They form the outermost scope, are cross-tenant, and are read-only through the API. Use them for installation-wide governance that no namespace or tenant can override.
```yaml
kestra:
policies:
instance-defaults:
description: "Global task runner and cost controls."
rules:
- type: io.kestra.plugin.ee.rules.Add
on: PLUGIN
override: true
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.aws
values:
region: eu-west-1
- type: io.kestra.plugin.ee.rules.Restrict
on: FLOW
property: concurrency.limit
max: 20
errorMessage: "concurrency.limit cannot exceed 20."
```
- `kestra.policies` is a map. The key (`instance-defaults` above) is the policy identity — do not add an `id` field inside the body.
- A malformed static policy prevents server startup (fail-closed). Validate in a staging environment before deploying.
Static policies are the replacement for the removed `kestra.plugins.defaults` server configuration key. See the [pluginDefaults Removed migration guide](../../../11.migration-guide/v2.0.0/plugin-defaults-removed/index.md) for the full conversion.
## Creating and managing policies
Create and manage Policies from the Kestra UI in two places:
- **Tenant-level**: navigate to the tenant administration menu and open **Policies** to manage tenant-scoped policies.
- **Namespace-level**: open **Namespaces → [your namespace] → Policies** to manage policies scoped to that namespace.
You can also create and manage Policies via the API:
```
POST /api/v1/{tenant}/policies # tenant-scoped
POST /api/v1/{tenant}/namespaces/{namespace}/policies # namespace-scoped
```
Policies can also be exported as YAML and imported into other namespaces or environments.
## Verifying policy behavior
### Preview the effective policy chain
Before saving a flow, preview what policies will inject and which rules will flag violations:
```
POST /api/v1/{tenant}/flows/policies/preview
```
Send the flow source YAML in the request body. The response returns the mutated source with per-property attribution — each injected value is annotated with the policy ID and scope that supplied it.
### Dry-run a policy against saved flows
Evaluate a policy against flows already in a namespace without enforcing it:
```
GET /api/v1/{tenant}/policies/{id}/evaluate
GET /api/v1/{tenant}/namespaces/{namespace}/policies/{id}/evaluate
```
The response lists every flow that would violate the policy's validate rules, grouped by rule. Use this to audit your compliance baseline before switching a policy from `EVALUATE` to `ACTIVE`.
## RBAC
Managing Policies requires the `POLICY` permission on the target resource. This permission is separate from namespace edit rights — a user who can edit flows in a namespace does not automatically have permission to manage its Policies.
Permissions: `VIEW`, `CREATE`, `UPDATE`, `DELETE`.
---
# Promote Flows Across Environments in Kestra
URL: https://kestra.io/docs/enterprise/governance/promote
> Move flows from dev to staging to production directly from the Kestra UI, with a diff review, confirmation gate, and full promotion history.
Promote copies a flow from one Kestra instance to another, with a source-to-target diff review and an optional confirmation gate before anything lands in production.
## When to use Promote
Use Promote when your team authors flows in the Kestra UI, runs separate instances per environment (dev, staging, production), and does not want to build or maintain a Git pipeline to move flows between them.
If you already treat Git as the source of truth and deploy flows on merge via CI/CD, continue with that path. Promote and [Git-based deployment](../../../version-control-cicd/04.git/index.md) solve the same problem with different tradeoffs: Promote is UI-first and requires no pipeline; Git-based deployment is automated and auditable at the repository level. See [Version Control & CI/CD](../../../version-control-cicd/index.mdx) for a comparison of all deployment paths.
---
## Promotion targets
A promotion target is a remote Kestra instance that flows are promoted into. Each target has a name, a base URL, an optional target tenant (for multi-tenant instances), and a connection mode. Targets are managed at the tenant level and are available to all users with the appropriate permissions.

### Connection modes
Each target uses one of two connection modes that control how the promote action reaches the remote instance.
**SERVER mode**: the Kestra backend holds an encrypted API token for the remote instance. When a user promotes a flow, the source Kestra server makes the API call to the target on their behalf. Users never see or handle the token.
Use SERVER mode when you want to centralize credential management and prevent users from needing direct access to the target instance.
**CLIENT mode**: no token is stored on the source instance. When a user promotes a flow, they supply their own API token for the target at promote time. The browser calls the target instance directly, then reports the result back to the source for history and audit purposes. Tokens are stored in the user's browser for convenience and are never sent to the source server.
Use CLIENT mode when users already have personal API tokens on the target, or when you want each promotion to be attributable to the individual user's identity on the target.

### Confirmation gate
Each target can optionally require explicit confirmation before any promotion runs. When **Require a confirmation gate before promoting** is enabled on a target, the UI presents the diff and requires the user to acknowledge before the flow is copied to the target. If a promote request is submitted to a gated target without confirmation, no promotion is attempted and nothing is recorded. The user must confirm and resubmit.
Use gated targets for production environments where you want a deliberate review step before deployment.

### Disabling a target
Targets can be disabled without being deleted. A disabled target is hidden from the promote UI (users cannot select it when promoting flows) but its configuration is preserved.
---
## Promote a flow
### From the Deploy tab
Each flow has a **Deploy** tab alongside the editor. From the Deploy tab, select a target, review the diff between the local revision and what is currently running on the target, and confirm. If the target is gated, an explicit confirmation is required before the promotion proceeds.

Only the flow's latest published revision can be promoted. Draft revisions are not eligible.
### From the flow editor
You can also promote a flow by clicking the **Promote** tab directly from the flow editor, without navigating away. The same diff review and gate confirmation apply.
### Bulk promote
From the flows list, select up to 100 flows and choose **Promote** from the action bar. The bulk promote dialog shows a drift summary per target (how many are out of sync, how many have never been promoted) and requires gate confirmation if any selected target has a gate enabled. Results are reported per flow; a failure on one flow does not block the others.

---
## Drift detection
The flows list includes a **Deploy** column that shows the sync state of each flow relative to a selected target. The active target for drift comparison is selected in the flows list and persisted per user session.
| State | Meaning |
|---|---|
| `IN_SYNC` | The target is running the same revision as the local instance |
| `OUT_OF_SYNC` | The target has the flow but is running a different revision |
| `NOT_PROMOTED` | The flow has never been promoted to this target |
| `UNREACHABLE` | The target could not be reached to compare hashes |
| `NEEDS_AUTH` | The target is CLIENT mode and no token has been provided for it |
Drift is computed by comparing each flow's source YAML between the local instance and the target. Only the latest published revision is included.

---
## Promotion history
Every promotion (including who initiated it, which revision was promoted, which target it went to, and whether a gate was confirmed) is recorded as an auditable event. The history for a specific flow is visible on its Deploy tab. For tenant-wide audit purposes, promotions are also available in [Audit Logs](../06.audit-logs/index.md), filtered by resource type `FLOW` and action `PROMOTE`.
From the history, you can recompute the diff of any past promotion to see exactly what changed at the time it was deployed.

---
## What is and is not promoted
Promote copies the flow's YAML definition from the selected revision to the target instance. It does not copy namespace-level resources. If the flow depends on KV pairs, secrets, variables, or namespace files that differ between environments, those must be managed separately on the target. See [Namespace Management](../07.namespace-management/index.md) and [Secrets Manager](../secrets-manager/index.md) for managing these resources per environment.
---
## Access control
Promote uses two distinct RBAC resources.
**Promoting flows** requires the `FLOW: PROMOTE` permission on the source flow's namespace. This permission is namespace-scoped: a user with `FLOW: PROMOTE` on `company.dev` can promote flows in that namespace, but not flows in other namespaces unless they also have the permission there.
**Managing promotion targets** (creating, editing, deleting, viewing targets) requires `PROMOTION_TARGET` permissions. Unlike flow permissions, `PROMOTION_TARGET` is not namespace-scoped; it applies at the tenant level.
| Action | Required permission |
|---|---|
| Promote a flow | `FLOW: PROMOTE` on the flow's namespace |
| View available targets when promoting | `PROMOTION_TARGET: VIEW` |
| Create or edit targets | `PROMOTION_TARGET: CREATE` / `UPDATE` |
| Delete targets | `PROMOTION_TARGET: DELETE` |
| List all targets | `PROMOTION_TARGET: LIST` |
See [RBAC](../../03.auth/rbac/index.md) for how to assign permissions to roles and groups.
---
## Configuration
### Allow insecure target URLs
By default, promotion target URLs must use HTTPS. To allow HTTP URLs, for example when targets are on a private network without TLS, set the following in your Kestra configuration:
```yaml
kestra:
ee:
promote:
allow-insecure-url: true
```
This setting applies instance-wide. Target URL validation is enforced at create and update time.
---
# External Secrets Manager in Kestra
URL: https://kestra.io/docs/enterprise/governance/secrets-manager
> Secure sensitive data in Kestra with External Secrets Managers. Integrate with AWS, Azure, Google Cloud, Vault, and more for robust secret management.
Configure Kestra to store and retrieve secrets from an external backend such as AWS Secrets Manager, Azure Key Vault, Google Secret Manager, HashiCorp Vault, and others.
:::alert{type="info"}
For short-lived OAuth-style access tokens, use a [Credential](../../03.auth/credentials/index.md) that mints and refreshes tokens from secrets stored in your external manager, then reference it with `credential()` in flows.
:::
## Secrets manager modes
Kestra supports two modes when integrating an external secrets backend. They differ fundamentally in **who is the source of truth** for secret metadata.
| | Managed mode (default) | Read-only mode |
|---|---|---|
| **Source of truth** | Kestra | External system |
| **Secret metadata** | Kestra tags secrets automatically | Tags must be set manually in the external backend |
| **Create / edit / delete** | From Kestra UI or Terraform | Not possible from Kestra — external system only |
| **Tag changes externally** | Not reflected in Kestra | Reflected immediately |
| **Value changes externally** | Picked up at next read | Picked up at next read |
### Managed mode (default)
Kestra is the source of truth. You create, edit, and delete secrets from the Kestra UI or via Terraform. When you create a secret, Kestra stores the encrypted value in the external backend and attaches metadata tags (`namespace`, `tenant`, `key`, `prefix`) automatically.
- Kestra reads the value from the external backend at runtime.
- Tag changes made directly in the external backend are **not** reflected in Kestra — Kestra ignores external metadata.
- The external backend is encrypted storage only; Kestra controls the index.
:::alert{type="warning"}
Secrets created directly in the external backend (without going through Kestra) will **not** appear in Kestra in managed mode. Kestra only discovers secrets it created itself. To make externally-managed secrets visible in Kestra, use [read-only mode](#read-only-mode) instead.
:::
### Read-only mode
The external system is the source of truth. You populate secrets and their metadata tags in the external backend yourself, and Kestra reads them. Kestra cannot create, edit, or delete secrets — the UI shows a lock icon and hides the **Create New Secret** button.
Read-only mode is supported for all external secrets backends. For backend-specific requirements such as secret format constraints, see the read-only subsection under the relevant backend below.
**Enable globally** — add `read-only: true` to your `kestra.secret` configuration:
```yaml
kestra:
secret:
type: aws-secret-manager
read-only: true
aws-secret-manager:
region: us-east-1
```
**Enable per tenant** — toggle the setting in the **Dedicated secrets manager** configuration under the tenant settings.
**Enable per namespace** — toggle the setting in the **Dedicated secrets manager** section of the namespace **Edit** tab.
Once enabled, secrets display a lock icon and the **Create New Secret** button is hidden.
Required tags that must be set externally in read-only mode:
- `namespace` — the namespace the secret belongs to.
- `key` — name used to reference the secret in flows via `secret('KEY')`.
- `prefix` — defaults to `kestra` when a secret is created inside Kestra; must be set manually in read-only mode.
## Configure a secrets manager
Set the backend globally in your Kestra configuration file using `kestra.secret.type`, or scope it to a specific [tenant](../tenants/index.md) or [namespace](../../../05.workflow-components/02.namespace/index.md) via the **Dedicated secrets manager** setting in the UI. Each backend uses its own sub-key matching the type name.
:::alert{type="warning"}
**Property naming differs between global config and the UI.** In your `kestra.yml` configuration file, use kebab-case (e.g., `safe-name`, `auth-method`). When configuring a dedicated secrets manager for a namespace or tenant via the UI, use camelCase (e.g., `safeName`, `authMethod`).
:::
**Supported backends:** [AWS Secrets Manager](#aws-secrets-manager) · [AWS SSM Parameter Store](#aws-ssm-parameter-store) · [Azure Key Vault](#azure-key-vault) · [Google Secret Manager](#google-secret-manager) · [HashiCorp Vault](#hashicorp-vault) · [CyberArk](#cyberark) · [Doppler](#doppler) · [1Password](#1password) · [BeyondTrust](#beyondtrust) · [Delinea Secret Server](#delinea-secret-server) · [Bitwarden](#bitwarden) · [JDBC](#jdbc-postgresql-h2-mysql) · [Elasticsearch](#elasticsearch)
---
## AWS Secrets Manager
### Permissions
**Managed mode** — the IAM identity needs: `CreateSecret`, `DeleteSecret`, `DescribeSecret`, `GetSecretValue`, `ListSecrets`, `PutSecretValue`, `RestoreSecret`, `TagResource`, `UpdateSecret`.
**Read-only mode** — only `GetSecretValue`, `ListSecrets`, and `DescribeSecret` are needed.
### Minimum configuration
```yaml
kestra:
secret:
type: aws-secret-manager
aws-secret-manager:
region: us-east-1
```
With explicit credentials:
```yaml
kestra:
secret:
type: aws-secret-manager
aws-secret-manager:
access-key-id: YOUR_ACCESS_KEY_ID
secret-key-id: YOUR_SECRET_KEY_ID
region: us-east-1
```
### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `access-key-id` | string | No | — | AWS access key ID. |
| `secret-key-id` | string | No | — | AWS secret access key. |
| `session-token` | string | No | — | Temporary session token. |
| `region` | string | No | — | AWS region (e.g. `us-east-1`). |
| `prefix` | string | No | `kestra` | Prefix applied to all secret names. Use to share one backend across multiple Kestra instances. |
| `endpoint-override` | string | No | — | Replace the default AWS endpoint with a compatible service (e.g. LocalStack). |
| `sts-role-arn` | string | No | — | ARN of an IAM role to assume via STS before accessing secrets. |
| `sts-role-external-id` | string | No | — | External ID passed with the STS AssumeRole call. |
| `sts-role-session-name` | string | No | — | Session name for the STS assumed role. |
| `sts-role-session-duration` | duration | No | `PT15M` | Duration of the assumed-role session. |
| `sts-endpoint-override` | string | No | — | Override the STS endpoint URL. |
| `tags` | map | No | — | Default tags added to every new or updated secret. |
| `filter-on-tags` | map | No | — | Read-only mode: filter visible secrets by matching these tags. |
| `filter-on-prefix` | object | No | — | Read-only mode: filter by secret name prefix. |
| `excluded-tags` | map | No | — | Secrets with these tags are excluded from Kestra's view. |
### Credential resolution order
1. `access-key-id` / `secret-key-id` (+ optional `session-token`) in config.
2. STS role assumption using `sts-role-arn` (chained on top of any resolved identity).
3. `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_DEFAULT_REGION` environment variables.
4. AWS CLI profile (`~/.aws/credentials`).
5. EKS Pod Identity or IRSA (IAM Roles for Service Accounts).
6. EC2/ECS instance metadata profile.
### Read-only mode
Secrets must be stored as **plain text** in AWS Secrets Manager, not as key-value pairs.
```yaml
kestra:
secret:
type: aws-secret-manager
read-only: true
aws-secret-manager:
access-key-id: YOUR_ACCESS_KEY_ID
secret-key-id: YOUR_SECRET_KEY_ID
region: us-east-1
```
Each secret must have the following tags set in AWS:
- `namespace` — the Kestra namespace this secret belongs to.
- `key` — the name used in `secret('KEY')` expressions.
- `prefix` — typically `kestra`.
:::alert{type="info"}
The secret name in AWS will not display in Kestra. Use the `key` tag to control what name Kestra uses.
:::
**Filter by tags** — limit visible secrets to those matching specific tags:
```yaml
kestra:
secret:
type: aws-secret-manager
read-only: true
aws-secret-manager:
filter-on-tags:
tags:
application: kestra-production
```
**Filter by prefix** — limit visible secrets to those whose names start with a given prefix:
```yaml
kestra:
secret:
type: aws-secret-manager
read-only: true
aws-secret-manager:
filter-on-prefix:
prefix: prod_
keep-prefix: true # set false to strip the prefix from the Kestra key name
```
---
## AWS SSM Parameter Store
Secrets are stored as `SecureString` parameters. The parameter name is `-`; namespace, tenant, and key are stored as parameter tags. In read-only mode, Kestra reads existing parameters by their name, so a Parameter Store already populated outside Kestra can be used directly.
### Permissions
**Managed mode** — the IAM identity needs: `ssm:PutParameter`, `ssm:GetParameter`, `ssm:DeleteParameter`, `ssm:DescribeParameters`, `ssm:AddTagsToResource`, `ssm:RemoveTagsFromResource`, `ssm:ListTagsForResource`, plus `kms:Encrypt` and `kms:Decrypt` on the KMS key used for `SecureString` encryption.
**Read-only mode** — only `ssm:GetParameter`, `ssm:DescribeParameters`, and `ssm:ListTagsForResource` are needed.
### Minimum configuration
```yaml
kestra:
secret:
type: aws-ssm-parameter-store
aws-ssm-parameter-store:
region: us-east-1
```
### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `access-key-id` | string | No | — | AWS access key ID. |
| `secret-key-id` | string | No | — | AWS secret access key. |
| `session-token` | string | No | — | Temporary session token. |
| `region` | string | No | — | AWS region. |
| `kms-key-id` | string | No | — | Customer-managed KMS key ID for `SecureString` encryption. Defaults to `alias/aws/ssm`. |
| `prefix` | string | No | `kestra` | Prefix applied to all parameter names. |
| `endpoint-override` | string | No | — | Replace the default SSM endpoint. |
| `sts-role-arn` | string | No | — | IAM role to assume via STS. |
| `sts-role-external-id` | string | No | — | External ID for STS AssumeRole. |
| `sts-role-session-name` | string | No | — | Session name for the assumed role. |
| `sts-role-session-duration` | duration | No | `PT15M` | Duration of the assumed-role session. |
| `sts-endpoint-override` | string | No | — | Override the STS endpoint URL. |
| `tags` | map | No | — | Default tags added to every new or updated parameter. |
| `filter-on-tags` | map | No | — | Read-only mode: filter visible parameters by matching tags. |
| `filter-on-prefix` | object | No | — | Read-only mode: filter by parameter name prefix. |
| `excluded-tags` | map | No | — | Parameters with these tags are excluded from Kestra's view. |
Credential resolution order is identical to [AWS Secrets Manager](#credential-resolution-order).
### Read-only mode
```yaml
kestra:
secret:
type: aws-ssm-parameter-store
read-only: true
aws-ssm-parameter-store:
region: us-east-1
```
Tag and prefix filtering work the same as for [AWS Secrets Manager](#read-only-mode).
---
## Azure Key Vault
### Permissions
**Managed mode** — the service principal or managed identity needs: `Get`, `List`, `Set`, `Delete`, `Recover`, `Backup`, `Restore`, `Purge`.
**Read-only mode** — only `Get` and `List` are needed.
### Minimum configuration
```yaml
kestra:
secret:
type: azure-key-vault
azure-key-vault:
vault-name: my-vault-name
client-secret:
tenant-id: YOUR_TENANT_ID
client-id: YOUR_CLIENT_ID
client-secret: YOUR_CLIENT_SECRET
```
### Configuration reference
#### Connection
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `vault-name` | string | No | — | Azure Key Vault name. Required unless `key-vault-uri` is set. |
| `key-vault-uri` | string | No | — | Full vault URI (e.g. `https://my-vault.vault.azure.net`). Overrides `vault-name`. |
#### Service principal auth
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `client-secret` | object | No | — | Service principal credentials block. Sub-fields: `tenant-id`, `client-id`, `client-secret`. |
#### Managed identity auth
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `managed-identity-client-id` | string | No | — | Client ID of a user-assigned managed identity. Omit for system-assigned. |
| `managed-identity-resource-id` | string | No | — | Resource ID of a user-assigned managed identity. Alternative to `managed-identity-client-id`. |
#### Workload identity auth
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `workload-identity-client-id` | string | No | — | Client ID for Azure Workload Identity (AKS federated credentials). |
#### All auth modes
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `additionally-allowed-tenants` | string[] | No | — | Additional tenant IDs the credential may acquire tokens for. Use `"*"` to allow any tenant. |
| `prefix` | string | No | — | Prefix applied to all secret names. |
| `tags` | map | No | — | Default tags added to every new or updated secret. |
| `filter-on-tags` | map | No | — | Read-only mode: filter visible secrets by matching tags. |
| `filter-on-prefix` | object | No | — | Read-only mode: filter by secret name prefix. Fields: `prefix` (string), `keep-prefix` (boolean). |
| `excluded-tags` | map | No | — | Secrets with these tags are excluded from Kestra's view. |
### Credential resolution order
1. `client-secret` block (service principal with client secret).
2. `managed-identity-client-id` or `managed-identity-resource-id` (managed identity).
3. `workload-identity-client-id` (workload identity / federated credentials).
4. Azure default credential chain — environment variables (`AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `AZURE_TENANT_ID`), then Azure CLI, then managed identity.
### Read-only mode
```yaml
kestra:
secret:
type: azure-key-vault
read-only: true
azure-key-vault:
client-secret:
tenant-id: YOUR_TENANT_ID
client-id: YOUR_CLIENT_ID
client-secret: YOUR_CLIENT_SECRET
```
**Filter by tags:**
```yaml
kestra:
secret:
type: azure-key-vault
read-only: true
azure-key-vault:
filter-on-tags:
tags:
application: kestra-production
```
**Filter by prefix:**
```yaml
kestra:
secret:
type: azure-key-vault
read-only: true
azure-key-vault:
filter-on-prefix:
prefix: prod_
keep-prefix: true # set false to strip the prefix from the Kestra key name
```
---
## Google Secret Manager
### Permissions
**Managed mode** — the service account needs `roles/secretmanager.admin`.
**Read-only mode** — `roles/secretmanager.secretAccessor` is sufficient.
### Minimum configuration
```yaml
kestra:
secret:
type: google-secret-manager
google-secret-manager:
project: my-gcp-project-id
```
With an explicit service account key:
```yaml
kestra:
secret:
type: google-secret-manager
google-secret-manager:
project: my-gcp-project-id
service-account: |
{
"type": "service_account",
"project_id": "my-gcp-project-id",
"private_key_id": "...",
"private_key": "...",
...
}
```
### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `project` | string | No | — | GCP project ID. |
| `service-account` | string | No | — | Service account JSON key file contents. |
| `prefix` | string | No | `kestra` | Prefix applied to all secret names. |
| `tags` | map | No | — | Default labels added to every new or updated secret. |
| `filter-on-tags` | map | No | — | Read-only mode: filter visible secrets by matching labels. |
| `filter-on-prefix` | object | No | — | Read-only mode: filter by secret name prefix. |
| `excluded-tags` | map | No | — | Secrets with these labels are excluded from Kestra's view. |
### Credential resolution order
1. `service-account` JSON key in config.
2. `GOOGLE_APPLICATION_CREDENTIALS` environment variable pointing to a key file.
3. Google default application credentials (Workload Identity, GCE metadata server, gcloud CLI).
When configuring via the Kestra UI, only `project` and `service-account` fields are shown.
### Read-only mode
```yaml
kestra:
secret:
type: google-secret-manager
read-only: true
google-secret-manager:
project: my-gcp-project-id
service-account: |
Paste the contents of the service account JSON key file here.
```
**Filter by labels:**
```yaml
kestra:
secret:
type: google-secret-manager
read-only: true
google-secret-manager:
filter-on-tags:
tags:
application: kestra-production
```
---
## HashiCorp Vault
Kestra supports the [KV Secrets Engine v2](https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2). Dynamic secrets engines and the Vault Secrets Operator on Kubernetes are not directly supported (the Vault Secrets Operator creates a Kubernetes Secret that Kestra can use with additional steps — [contact us](/demo) for guidance).
### Permissions
The token or AppRole must have a Vault policy granting the following capabilities on your secrets engine path. Replace `secret` with your `root-engine` value if different.
**Managed mode:**
```hcl
path "secret/data/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
path "secret/metadata/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
```
**Read-only mode:**
```hcl
path "secret/data/*" {
capabilities = ["read", "list"]
}
path "secret/metadata/*" {
capabilities = ["read", "list"]
}
```
### Auth methods
Choose one of the following auth methods:
**Userpass**
```yaml
kestra:
secret:
type: vault
vault:
address: "http://localhost:8200"
password:
user: john
password: foo
```
**Token**
```yaml
kestra:
secret:
type: vault
vault:
address: "http://localhost:8200"
token:
token: YOUR_VAULT_TOKEN
```
**AppRole**
```yaml
kestra:
secret:
type: vault
vault:
address: "http://localhost:8200"
app-role:
path: approle
role-id: YOUR_ROLE_ID
secret-id: YOUR_SECRET_ID
```
**Kubernetes**
```yaml
kestra:
secret:
type: vault
vault:
address: "http://localhost:8200"
kubernetes:
path: "kubernetes"
role: "kestra"
```
### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `address` | string | No | — | Fully qualified Vault address with scheme and port. |
| `password` | object | No | — | Userpass auth. Sub-fields: `user`, `password`. |
| `token` | object | No | — | Token auth. Sub-field: `token`. |
| `app-role` | object | No | — | AppRole auth. Sub-fields: `path`, `role-id`, `secret-id`. |
| `kubernetes` | object | No | — | Kubernetes auth. Sub-fields: `path`, `role`. |
| `namespace` | string | No | — | Vault Enterprise namespace. |
| `engine-version` | integer | No | `2` | KV secrets engine version. |
| `root-engine` | string | No | `secret` | KV secrets engine path in Vault. |
| `prefix` | string | No | `""` | Prefix applied to all secret paths. |
| `secret-path-prefix` | string | No | `""` | Path prefix within the engine to scope Kestra's access. |
| `role` | string | No | — | Vault role name (used with some auth methods). |
| `read-only` | boolean | No | `false` | Enable read-only mode for this backend. |
| `connect-timeout` | duration | No | `PT15S` | Vault API connection timeout. |
| `read-timeout` | duration | No | `PT1M` | Vault API read timeout. |
| `tags` | map | No | — | Default tags added to every new or updated secret. |
| `filter-on-tags` | map | No | — | Read-only mode: filter visible secrets by matching tags. |
| `excluded-tags` | map | No | — | Secrets with these tags are excluded from Kestra's view. |
In Vault, secrets created through Kestra are stored under `TENANT_ID/NAMESPACE_PARENT/.../NAMESPACE_CHILD/SECRET_NAME`. For a tenant `internal` and namespace `company.team`, the path would be `internal/company/team/SECRET_NAME` under the configured `root-engine`.
### Read-only mode
Set `read-only: true` inside the `vault` config block (unlike other backends which use `kestra.secret.read-only`):
```yaml
kestra:
secret:
type: vault
vault:
address: https://my-vault:8200/
root-engine: secret
secret-path-prefix: app1
token:
token: my-vault-access-token
read-only: true
```
Use `secret-path-prefix` to scope Kestra's view to a specific path within the engine. In the example above, only secrets under the `app1` path are visible. Reference a subkey in a flow with:
```twig
{{ secret('my-app', subkey='MY_SUBKEY') }}
```
For a full end-to-end walkthrough with screenshots, see [Use HashiCorp Vault as a Read-Only Secrets Backend](../../../15.how-to-guides/vault-read-only-secrets/index.md).
---
## CyberArk
Kestra integrates with CyberArk Privilege Cloud as a secrets backend using one of two authentication methods:
- **PVWA** (default) — username and password logon via the CyberArk REST API. Supports read and write.
- **CCP** — Central Credential Provider (AIM web service), authenticated by IP allowlisting and an Application ID. Read-only. No credentials stored in Kestra configuration.
### Permissions
#### PVWA
**Managed mode** — the account needs: `GetAccounts`, `AddAccounts`, `UpdateAccountContent`, `UpdateAccountProperties`, `DeleteAccounts`, `RetrieveAccounts` on the target Safe.
**Read-only mode** — only `GetAccounts` and `RetrieveAccounts` are needed.
#### CCP
CCP uses IP-based authentication — the Kestra server's IP must be registered in the CyberArk Application ID's allowed machines list. No CyberArk user account or password is required.
### Minimum configuration
#### PVWA
```yaml
kestra:
secret:
type: cyberark
cyberark:
address: https://your-cyberark-host
username: YOUR_USERNAME
password: YOUR_PASSWORD
platform-id: YOUR_PLATFORM_ID
safe-name: YOUR_SAFE_NAME
```
#### CCP
```yaml
kestra:
secret:
type: cyberark
read-only: true
cyberark:
auth-method: CCP
address: https://your-cyberark-host
application-id: YOUR_APP_ID
safe-name: YOUR_SAFE_NAME
```
### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `address` | string | **Yes** | — | CyberArk API base URL. |
| `auth-method` | enum | No | `PVWA` | Authentication method: `PVWA` or `CCP`. |
| `username` | string | PVWA only | — | Username for PVWA authentication. |
| `password` | string | PVWA only | — | Password for PVWA authentication. |
| `application-id` | string | CCP only | — | Application ID registered in CyberArk for CCP authentication. |
| `safe-name` | string | **Yes** | — | Name of the CyberArk Safe where secrets are stored. |
| `platform-id` | string | PVWA managed | — | CyberArk platform ID used when creating secrets. Not required in read-only mode or for CCP. |
| `web-service-id` | string | No | `AIMWebService` | CCP web service name. Only used when `auth-method` is `CCP`. |
| `client-certificate` | string | No | — | PEM-encoded client certificate for mutual TLS. Applies to both auth methods. |
| `client-certificate-key` | string | No | — | PEM-encoded PKCS#8 private key for the client certificate. Required if `client-certificate` is set. |
| `reason` | string | No | — | Reason recorded in the CyberArk audit log on CCP credential retrieval. |
| `query-format` | enum | No | `Exact` | CCP object query matching: `Exact` or `Regexp`. Only used when `auth-method` is `CCP`. |
| `object-property` | string | No | — | Account property to return as the secret value. Defaults to the password field. Only used when `auth-method` is `CCP`. |
| `use-proxy` | boolean | No | — | Route API calls through the configured system proxy. |
| `validate-certs` | boolean | No | `true` | Validate TLS certificates on the CyberArk endpoint. |
| `secret-path-prefix` | string | No | `""` | Path prefix within the Safe. |
| `tags` | map | No | — | Default tags added to every new or updated secret. |
| `filter-on-tags` | map | No | — | Filter visible secrets by matching tags. |
| `excluded-tags` | map | No | — | Secrets with these tags are excluded from Kestra's view. |
### Read-only mode
Set `read-only: true` to prevent Kestra from creating or modifying secrets.
**PVWA read-only** — `platform-id` is not required in this mode.
```yaml
kestra:
secret:
type: cyberark
read-only: true
cyberark:
address: https://your-cyberark-host
username: YOUR_USERNAME
password: YOUR_PASSWORD
safe-name: YOUR_SAFE_NAME
```
**CCP** is always read-only and does not support write operations.
---
## Doppler
Kestra integrates with Doppler as a secrets backend. Secrets are accessed at runtime via the Doppler API and kept only in memory.
### Minimum configuration
```yaml
kestra:
secret:
type: doppler
doppler:
token: YOUR_SERVICE_TOKEN
project: my-project
config: production
```
### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `token` | string | No | — | Doppler service token with access to the target project and config. |
| `project` | string | No | — | Doppler project containing the secrets. |
| `config` | string | No | — | Doppler config (environment) to read from. |
| `secret-name-prefix` | string | No | — | Prefix added to secret keys to avoid collisions when sharing a backend. |
| `connect-timeout` | duration | No | `PT15S` | HTTP connection timeout for Doppler API calls. |
| `read-timeout` | duration | No | `PT60S` | HTTP read timeout for Doppler API calls. |
| `tags` | map | No | — | Default tags for new or updated secrets. |
| `filter-on-tags` | map | No | — | Filter visible secrets by tags. |
| `excluded-tags` | map | No | — | Secrets with these tags are excluded. |
### Read-only mode
Use a service token with `read` access only. The `project` and `config` values scope Kestra's view to a single Doppler environment.
```yaml
kestra:
secret:
type: doppler
read-only: true
doppler:
token: YOUR_SERVICE_TOKEN
project: my-project
config: production
```
---
## 1Password
Kestra integrates with 1Password via the [1Password Connect API](https://developer.1password.com/docs/connect/api-reference/). A running 1Password Connect server is required.
### Minimum configuration
```yaml
kestra:
secret:
type: 1password
1password:
address: http://localhost:18080
token: YOUR_CONNECT_TOKEN
vault-id: YOUR_VAULT_ID
```
### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `address` | string | **Yes** | — | Base URL of the 1Password Connect server. |
| `token` | string | **Yes** | — | 1Password Connect API token. |
| `vault-id` | string | **Yes** | — | ID of the vault containing the secrets. |
| `validate-certs` | boolean | No | — | Validate TLS certificates on the Connect server. |
| `secret-path-prefix` | string | No | `""` | Path prefix for secrets in the vault. |
| `tags` | map | No | — | Default tags for new or updated secrets. |
| `filter-on-tags` | map | No | — | Filter visible secrets by tags. |
| `excluded-tags` | map | No | — | Secrets with these tags are excluded. |
### Read-only mode
The Connect API token needs only read access to the target vault (`item:read` scope). No special format requirements apply to secrets.
```yaml
kestra:
secret:
type: 1password
read-only: true
1password:
address: http://localhost:18080
token: YOUR_CONNECT_TOKEN
vault-id: YOUR_VAULT_ID
```
---
## BeyondTrust
Kestra integrates with BeyondTrust Password Safe (Secrets Safe) via the [Secret Safe API](https://docs.beyondtrust.com/bips/v24.3/docs/secrets-safe-api).
### Permissions
**Managed mode** — the API key must have Secrets Safe permissions to retrieve, create, update, and delete secrets in the target folder.
**Read-only mode** — only Retrieve permissions are needed.
### Minimum configuration
```yaml
kestra:
secret:
type: beyondtrust
beyondtrust:
address: https://beyondtrust.example.com
api-key: YOUR_API_KEY
run-as: domain\\service-account
```
### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `address` | string | **Yes** | — | Base URL of the BeyondTrust Password Safe instance. |
| `api-key` | string | **Yes** | — | API key for authentication. |
| `run-as` | string | **Yes** | — | User context for API calls (e.g. `domain\\username`). |
| `folder-id` | string | No | — | Secrets Safe folder ID where Kestra secrets are stored. |
| `owner-id` | integer | No | — | Owner ID for new secrets. |
| `owner-type` | string | No | `User` | Owner type: `User` or `Group`. |
| `api-version` | string | No | `V3_0` | API version to use: `V3_0` or `V3_1`. |
| `connect-timeout` | duration | No | `PT15S` | HTTP connection timeout. |
| `read-timeout` | duration | No | `PT1M` | HTTP read timeout. |
| `secret-path-prefix` | string | No | `""` | Path prefix for secrets. |
| `tags` | map | No | — | Default tags for new or updated secrets. |
| `filter-on-tags` | map | No | — | Filter visible secrets by tags. |
| `excluded-tags` | map | No | — | Secrets with these tags are excluded. |
### Read-only mode
In read-only mode, `folder-id`, `owner-id`, and `owner-type` are not needed — they apply only when Kestra creates new secrets. The API key needs only Retrieve (`read`) permissions on the target Secrets Safe.
```yaml
kestra:
secret:
type: beyondtrust
read-only: true
beyondtrust:
address: https://beyondtrust.example.com
api-key: YOUR_API_KEY
run-as: domain\\service-account
```
---
## Delinea Secret Server
Kestra integrates with [Delinea Secret Server](https://delinea.com/products/secret-server) as a secrets backend.
### Minimum configuration
```yaml
kestra:
secret:
type: delinea
delinea:
address: https://your-delinea-instance.secretservercloud.com
username: YOUR_USERNAME
password: YOUR_PASSWORD
```
### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `address` | string | **Yes** | — | Base URL of the Delinea Secret Server instance. |
| `username` | string | **Yes** | — | Username for authentication. |
| `password` | string | **Yes** | — | Password for authentication. |
| `domain` | string | No | — | Active Directory domain for on-premise deployments using domain accounts. |
| `folder-id` | integer | No | — | Folder ID in Delinea where Kestra secrets are stored. Required for write operations. |
| `secret-template-id` | integer | No | — | Secret template ID for creating new secrets. Required for write operations. |
| `site-id` | integer | No | — | Site ID for distributed engine deployments. |
| `connect-timeout` | duration | No | `PT15S` | HTTP connection timeout. |
| `read-timeout` | duration | No | `PT1M` | HTTP read timeout. |
| `secret-path-prefix` | string | No | `""` | Path prefix for secrets. |
| `tags` | map | No | — | Default tags for new or updated secrets. |
| `filter-on-tags` | map | No | — | Filter visible secrets by tags. |
| `excluded-tags` | map | No | — | Secrets with these tags are excluded. |
### Reading multi-field Delinea secrets
Delinea secrets contain structured credentials — for example, an Active Directory template has a password, username, and domain as separate fields.
By default, `secret()` returns the password field:
```twig
{{ secret('AD_CREDS') }}
```
Pass `full=true` to retrieve all fields. The `value` key holds the password; `metadata` holds all other non-password, non-notes template fields keyed by their Delinea item slug:
```twig
{% set creds = secret('AD_CREDS', full=true) %}
{{ creds.value }} {# password #}
{{ creds.metadata.username }}
{{ creds.metadata.domain }}
```
The keys available under `creds.metadata` depend on the fields defined in your Delinea secret template. The `notes` field is always excluded.
### Read-only mode
In read-only mode, `folder-id` and `secret-template-id` are not required — they are only used when Kestra creates new secrets. The Delinea account needs only view access to the target secrets.
```yaml
kestra:
secret:
type: delinea
read-only: true
delinea:
address: https://your-delinea-instance.secretservercloud.com
username: YOUR_USERNAME
password: YOUR_PASSWORD
```
---
## Bitwarden
Kestra integrates with [Bitwarden Secrets Manager](https://bitwarden.com/products/secrets-manager/). Secrets are decrypted client-side by Kestra workers at runtime and kept only in memory. No Bitwarden CLI or native SDK is required.
### Minimum configuration
```yaml
kestra:
secret:
type: bitwarden
bitwarden:
access-token: YOUR_ACCESS_TOKEN
organization-id: YOUR_ORGANIZATION_ID
```
### Configuration reference
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
| `access-token` | string | No | — | Machine account access token for authentication and client-side decryption. |
| `organization-id` | string | No | — | Bitwarden organization the machine account belongs to. |
| `api-url` | string | No | `https://api.bitwarden.com` | Bitwarden API URL. Use `https://api.bitwarden.eu` for EU cloud or your instance URL for self-hosted. |
| `identity-url` | string | No | `https://identity.bitwarden.com` | Bitwarden identity URL. Use `https://identity.bitwarden.eu` for EU cloud. |
| `project-id` | string | No | — | Restrict resolution to a single Bitwarden project. |
| `secret-name-prefix` | string | No | `""` | Prefix applied to all secret names. |
| `connect-timeout` | duration | No | `PT15S` | HTTP connection timeout. |
| `read-timeout` | duration | No | `PT1M` | HTTP read timeout. |
| `tags` | map | No | — | Default tags for new or updated secrets. |
| `filter-on-tags` | map | No | — | Filter visible secrets by tags. |
| `excluded-tags` | map | No | — | Secrets with these tags are excluded. |
### Read-only mode
The machine account access token must have read access to the Bitwarden project containing the secrets. Read-only mode prevents Kestra from creating or modifying secrets.
```yaml
kestra:
secret:
type: bitwarden
read-only: true
bitwarden:
access-token: YOUR_ACCESS_TOKEN
organization-id: YOUR_ORGANIZATION_ID
```
---
## JDBC (PostgreSQL, H2, MySQL)
The JDBC backend stores encrypted secrets in Kestra's own database. This is the default for open-source deployments.
```yaml
kestra:
secret:
type: jdbc
jdbc:
secret: YOUR_ENCRYPTION_KEY
```
The encryption key must be at least 32 characters. See the [Security and Secrets configuration guide](../../../configuration/05.security-and-secrets/index.md) for key generation details.
---
## Elasticsearch
The Elasticsearch backend stores secrets with AES encryption in the same Elasticsearch cluster used by Kestra (Kafka/Elastic backend deployments).
```yaml
kestra:
secret:
type: elasticsearch
elasticsearch:
secret: "a-secure-32-character-minimum-key"
```
See the [Security and Secrets configuration guide](../../../configuration/05.security-and-secrets/index.md) for key generation details.
---
## Shared configuration
### Default tags
Each secret backend accepts a `tags` map. Tags are added to every secret created or updated through Kestra and can be used as read-only mode filters.
```yaml
kestra:
secret:
aws-secret-manager:
tags:
application: kestra-production
environment: prod
```
### Filter secrets by tags (read-only mode)
When integrating an external secrets manager in read-only mode, you can filter which secrets are visible in Kestra by matching tags. Set `read-only: true` and configure `filter-on-tags` with the key/value pairs to match.
:::alert{type="info"}
AWS Secrets Manager, Azure Key Vault, and Google Secret Manager use a nested `tags` sub-key under `filter-on-tags`. All other providers accept `filter-on-tags` as a flat map of key/value pairs.
:::
```yaml
kestra:
secret:
type: aws-secret-manager
read-only: true
aws-secret-manager:
filter-on-tags:
tags:
application: kestra-production
```
```yaml
kestra:
secret:
type: azure-key-vault
read-only: true
azure-key-vault:
filter-on-tags:
tags:
application: kestra-production
```
```yaml
kestra:
secret:
type: google-secret-manager
read-only: true
google-secret-manager:
filter-on-tags:
tags:
application: kestra-production
```
```yaml
kestra:
secret:
type: vault
read-only: true
vault:
filter-on-tags:
application: kestra-production
```
```yaml
kestra:
secret:
type: cyberark
read-only: true
cyberark:
filter-on-tags:
application: kestra-production
```
```yaml
kestra:
secret:
type: doppler
read-only: true
doppler:
filter-on-tags:
application: kestra-production
```
```yaml
kestra:
secret:
type: 1password
read-only: true
1password:
filter-on-tags:
application: kestra-production
```
```yaml
kestra:
secret:
type: beyondtrust
read-only: true
beyondtrust:
filter-on-tags:
application: kestra-production
```
```yaml
kestra:
secret:
type: delinea
read-only: true
delinea:
filter-on-tags:
application: kestra-production
```
### Exclude secrets by tags (read-only mode)
Use `excluded-tags` to hide secrets from Kestra based on their tags. Any secret whose tags match at least one key-value pair in `excluded-tags` is excluded from Kestra's view, even if it would otherwise be included by `filter-on-tags`. This filter applies only when `read-only: true` is set.
When both `filter-on-tags` and `excluded-tags` are configured, a secret must match all entries in `filter-on-tags` and must not match any entry in `excluded-tags`.
The following examples exclude secrets tagged `hidden: "true"` for each supported provider:
```yaml
kestra:
secret:
type: aws-secret-manager
read-only: true
aws-secret-manager:
excluded-tags:
hidden: "true"
```
```yaml
kestra:
secret:
type: azure-key-vault
read-only: true
azure-key-vault:
excluded-tags:
hidden: "true"
```
```yaml
kestra:
secret:
type: google-secret-manager
read-only: true
google-secret-manager:
excluded-tags:
hidden: "true"
```
```yaml
kestra:
secret:
type: vault
read-only: true
vault:
excluded-tags:
hidden: "true"
```
```yaml
kestra:
secret:
type: cyberark
read-only: true
cyberark:
excluded-tags:
hidden: "true"
```
```yaml
kestra:
secret:
type: doppler
read-only: true
doppler:
excluded-tags:
hidden: "true"
```
```yaml
kestra:
secret:
type: 1password
read-only: true
1password:
excluded-tags:
hidden: "true"
```
```yaml
kestra:
secret:
type: beyondtrust
read-only: true
beyondtrust:
excluded-tags:
hidden: "true"
```
```yaml
kestra:
secret:
type: delinea
read-only: true
delinea:
excluded-tags:
hidden: "true"
```
:::alert{type="info"}
AWS Secrets Manager does not support negative tag filtering in its `ListSecrets` API. Kestra evaluates `excluded-tags` client-side after fetching the secret list from AWS. For CyberArk, Doppler, 1Password, Vault, BeyondTrust, and Delinea, both `filter-on-tags` and `excluded-tags` are also evaluated client-side.
:::
### Filter secrets by prefix (AWS, read-only mode)
For AWS Secrets Manager, you can filter secrets by a name prefix when using read-only mode. Use `filter-on-prefix.prefix` to select secrets whose names start with the given prefix and `filter-on-prefix.keep-prefix` to control whether the prefix is kept in the Kestra secret key.
```yaml
kestra:
secret:
type: aws-secret-manager
read-only: true
aws-secret-manager:
filter-on-prefix:
prefix: prod_
keep-prefix: true
```
### Secret caching
For cloud-backed secrets managers, enable caching to reduce the number of API calls per execution.
```yaml
kestra:
secret:
cache:
enabled: true
maximum-size: 1000
expire-after-write: 60s
```
| Property | Description |
|---|---|
| `cache.enabled` | Enable caching (default `false`). |
| `cache.maximum-size` | Maximum number of cached secret entries. |
| `cache.expire-after-write` | TTL for each cache entry after creation. |
### Secret isolation
Restrict which Kestra services are permitted to resolve secrets. Use this to prevent components such as the Executor from accessing secret values directly, limiting resolution to Workers only.
```yaml
kestra:
secret:
isolation:
enabled: true
denied-services:
- EXECUTOR
```
| Property | Description |
|---|---|
| `isolation.enabled` | Enable service isolation (default `false`). |
| `isolation.denied-services` | List of Kestra service names that must not resolve secrets (e.g. `EXECUTOR`, `INDEXER`, `SCHEDULER`). |
---
## Known limitations
### Pre-existing secrets are not migrated when switching backends
When a namespace or tenant switches from Kestra's internal secret backend (JDBC or Elasticsearch) to an external secrets manager, existing secrets stored in the internal backend are not automatically migrated.
Kestra will attempt to resolve those secret keys against the new external backend. Because the external backend does not hold them, resolution fails silently — the Kestra UI shows no error, but the `secret()` function fails at runtime when a flow uses one of those keys.
**Before switching backends:**
1. Export all existing secrets from the current backend.
2. Re-create them in the new external backend with the correct tags (`namespace`, `key`, `prefix`).
3. Switch the backend configuration in Kestra.
This also applies when adding a dedicated external backend to a namespace that previously used the instance-level JDBC or Elasticsearch backend.
---
# Multi-Tenancy in Kestra: Configure Tenants
URL: https://kestra.io/docs/enterprise/governance/tenants
> Enable Multi-Tenancy in Kestra Enterprise. Isolate resources, flows, and users across different teams or projects within a single Kestra instance.
Multi-tenancy lets you run isolated environments for different teams, projects, or customers within a single Kestra instance.
## Multi-tenancy — configure and manage tenants
A tenant represents an **isolated environment within a single Kestra instance**.
Each tenant functions as a separate entity with its own resources, such as flows, triggers, or executions. Multi-tenancy enables different teams, projects, or customers to operate independently within the same Kestra instance, ensuring data privacy, security, and separation of resources between business units, teams, or customers. For example, you can have a `dev` tenant for development, a `staging` tenant for testing, and a `prod` tenant for production.
:::alert{type="info"}
You can think of multi-tenancy as running multiple virtual instances in a single physical instance of [Kestra Cloud](/cloud) or [Kestra Enterprise Edition](../../01.overview/01.enterprise-edition/index.md).
:::
All resources (such as [flows](../../../05.workflow-components/01.flow/index.md), [triggers](../../../05.workflow-components/07.triggers/index.mdx), [executions](../../../05.workflow-components/03.execution/index.md), [RBAC](../../03.auth/rbac/index.md), and more) are isolated by the tenant. This means that you can have a flow with the same identifier and the same namespace in multiple tenants at the same time.
Data stored inside the internal storage is also separated by tenants.
Users switch between tenants using the tenant dropdown in the bottom-left corner of the UI. The dropdown shows every tenant the user has access to; the active tenant is indicated with a checkmark. Each UI page includes the tenant ID in the URL (e.g., `https://demo.kestra.io/ui/yourTenantId/executions/namespace/flow/executionId`).

Most [API](../../../api-reference/index.mdx) endpoints also include the tenant identifier. The exception to that is instance-level endpoints such as `/configs`, `/license-info` or `/banners` that require Instance Owner access.
For example, the URL of the API operation to list flows of the `products` namespace is `/api/v1/{your_tenant_id}/flows/products`. You can check the [Enterprise Edition API Guide](../../../api-reference/01.enterprise/index.mdx) for more information.
Tenants must be created upfront, and a user needs to be granted access to use a specific tenant.
## Key benefits of multi-tenancy
1. **Data Isolation**: each tenant's data, configuration, and code is isolated and inaccessible to other tenants.
2. **Resource Isolation**: each tenant's resources are isolated from other tenants — including flows, triggers, executions, logs, audit logs, secrets, etc.
3. **Simple Configuration**: create new tenants at any time, each providing a fresh, fully isolated workspace accessible from your existing Kestra instance.
4. **Intuitive UI Navigation**: the UI provides a dropdown as well as tenant identifiers included in the URL to make switching between tenants seamless.
## Creating and managing tenants
Tenants are created and managed through the **Instance Owner console** — only users with the Instance Owner privilege can create, edit, or delete tenants. The console is accessible from **Instance Owner → Tenants** in the UI. Tenants can also be managed via the CLI, API, or Terraform.
### Creating a tenant from the UI
Go to **Instance Owner → Tenants**, click **Create**, fill in the form, and click **Save**.
The user who creates a tenant is automatically granted the Admin Role for that tenant. You may need to refresh the UI to see updated Roles.
### Creating a tenant from the CLI
Kestra provides CLI commands for tenant creation. The following command creates a tenant with the identifier `stage` and the name `Staging`:
```bash
kestra tenants create --tenant stage --name "Staging"
```
Running `kestra tenants create --help` shows you all available properties:
```bash
$ kestra tenants create --help
Usage: kestra tenants create [-hVv] [--internal-log]
[--admin-username=] [-c=]
[-l=] [--name=]
[-p=] [--tenant=]
create a tenant and assign admin roles to an existing admin user
--admin-username=
Username of an existing admin user that will be
admin of this tenant
-c, --config= Path to a configuration file, default: /Users/anna/.
kestra/config.yml)
-h, --help Show this help message and exit.
--internal-log Change also log level for internal log, default:
false)
-l, --log-level=
Change log level (values: TRACE, DEBUG, INFO, WARN,
ERROR; default: INFO)
--name= tenant description
-p, --plugins=
Path to plugins directory , default:
/Users/anna/dev/plugins)
--tenant= tenant identifier
-v, --verbose Change log level. Multiple -v options increase the
verbosity.
-V, --version Print version information and exit.
```
### Creating a tenant from the API
Tenants can be managed programmatically via Kestra's [API](../../../api-reference/01.enterprise/index.mdx#post-/api/v1/tenants). Here is an example of an API call for creating a tenant:
```bash
curl -X POST "https://demo.kestra.io/api/v1/tenants" \
-H "accept: application/json" \
-H "Content-Type: application/json" \
-d "{ \"id\": \"stage\", \"name\": \"staging\", \"deleted\": false}"
```
### Creating a tenant from Terraform
Tenants can be managed via Infrastructure as Code using [Kestra's Terraform provider](../../../13.terraform/resources/tenant/index.md).
:::alert{type="info"}
This example assumes you have already configured the [Kestra Terraform Provider](../../../13.terraform/index.mdx)
:::
Here is an example of a Terraform configuration for creating a tenant:
```hcl
resource "kestra_tenant" "stage" {
tenant_id = "stage"
name = "staging"
}
```
### Deleting a tenant
Deleting a tenant removes all associated resources including flows, namespaces, apps, dashboards, and roles. Execution data, logs, metrics, and audit logs are retained in the database, and they can be purged if needed with their corresponding [Purge tasks](../../../10.administrator-guide/purge/index.md).
:::alert{type="warning"}
Deleting a tenant is irreversible. All resources under the tenant will be permanently removed, except for logs and execution history stored in the database.
:::
Key-value pairs and namespace files will not be deleted as they are persisted in internal storage.
### Admin role assignment
Regardless of which of the above methods you use to create a tenant, the User who creates the tenant automatically gets the Admin Role assigned. That role grants admin rights to that user on that tenant.
Note that there is an exception to this rule if a tenant is created by an Instance Owner. In that case, the Instance Owner has to explicitly assign the Admin Role for that tenant to themselves or any other User, Service Account, or Group.
### Dedicated storage and secrets backend per tenant
By default, each tenant uses the same [runtime and storage configuration](../../../configuration/02.runtime-and-storage/index.md) and [secrets backend](../secrets-manager/index.md) configured for your Kestra instance. If you need more isolation, you can configure a dedicated storage and secrets backend per tenant. This can be useful if each of your tenants serves different customers and you need to ensure complete data isolation between them.
To configure a dedicated storage and secrets backend per tenant, open the tenant's **Settings** page and scroll to the **Dedicated internal storage** and **Dedicated secrets manager** sections. Each section has a storage type selector and a YAML configuration editor.
For storage configuration examples, refer to [Runtime and Storage](../../../configuration/02.runtime-and-storage/index.md). For secrets backend options, refer to the [Secret Managers documentation](../secrets-manager/index.md).
:::alert{type="warning"}
Make sure to use `camelCase` notation. For example, if you want to use the `GCS` storage backend, you should use `projectId` as the value rather than `project-id`.
:::
### Isolate Kestra services
When using [Dedicated Storage or Secret backends](../tenants/index.md#dedicated-storage-and-secrets-backend-per-tenant), you can isolate specific [Kestra services](../../../08.architecture/02.server-components/index.md) to prevent them from accessing the storage or secret backend. For example, you may not want the [Webserver](../../../08.architecture/02.server-components/index.md#webserver) to be able to access the dedicated internal storage. This isolation is intended for Kestra instances where multiple teams or organizations share access, but storage or secret data access must be limited to specific segments.
The configuration utilizes the `deniedServices` property with a list of the services to isolate. Take the following as an example using `storage` (this can be replaced with `secret` for a dedicated secret backend), where the Executor and Webserver must be isolated:
```yaml
kestra:
storage: # or secret
isolation:
enabled: true
deniedServices: [EXECUTOR, WEBSERVER]
```
For additional configuration details, refer to dedicated [Security and Secrets](../../../configuration/05.security-and-secrets/index.md) and [Runtime and Storage](../../../configuration/02.runtime-and-storage/index.md) pages in the configuration guide.
:::alert{type="info"}
If this feature is enabled some UI or flow execution capabilities may not work as expected. If unsure, contact support.
:::
### Default service account for SDK plugins
Each tenant can define **default authentication credentials** used by [SDK-based plugins](/plugins/plugin-kestra). Configure this in the Tenant settings (API token or basic auth). [Namespaces](../07.namespace-management/index.md#default-service-account-for-sdk-plugins) can override it with their own default service account; otherwise the tenant-level default is used. If neither is set, SDK plugins require the properties to be defined in the tasks.
---
# Unit Tests in Kestra Enterprise: Validate Flows
URL: https://kestra.io/docs/enterprise/governance/unit-tests
> Validate workflows with Unit Tests in Kestra Enterprise. Create test suites, mock tasks, and assert flow behavior to ensure reliability before production.
Unit tests verify that flows behave as expected without running every task or producing real side effects — for example, mocking a notification task to confirm the configuration is correct without sending dummy alerts, or isolating a single changed task without re-executing the whole flow.
## Flow unit tests
Each test runs a single flow and checks its outcomes against your **assertions**, helping you avoid regressions when you change the flow later. Each **test case** creates a new transient execution, making it easy to run multiple tests in parallel, and each test case will not affect the others. Use **fixtures** to mock specific tasks or inputs by returning predefined outputs and states without executing the tasks.
Unit tests are configured for and connected to their respective flows. To create a new Unit Test, access them either through the **Tests** tab on the left-hand side panel of the Kestra UI or via the **Tests** tab of a flow. When creating tests, you can open the YAML for both the test and its flow side by side.
Once tests are created, they can all be viewed from the **Tests** tab with their respective Id, Namespace, Tested Flow, and current State listed. Additionally, tests can be run from this view with expandable results.
The following diagram illustrates the structure of flows and unit tests together in Kestra:

## Configuration
Unit tests are written in YAML like flows. A test is made up of `testCases`, and each test case is made up of `fixtures`, `assertions`, and an optional `expectedState`. Fixtures can target **files**, **inputs**, **tasks**, or **triggers** depending on what you need to mock or override. Like flows, you can write unit tests as code, in No Code, or with the [AI Copilot](../../../ai-tools/01.ai-copilot/index.md).
- A **fixture** refers to the setup required before a test runs, such as initializing objects or configuring environments, to ensure the test has a consistent starting state.
- An **assertion** is a statement that checks if a specific condition is true during the test. If the condition is false, the test fails, indicating an issue with the code being tested, while true indicates the expectation is met.
- **expectedState** sets the terminal state the flow must reach for the test to pass. It defaults to `SUCCESS`; set it to `FAILED`, `WARNING`, `KILLED`, or any other valid state to test intentional failure paths. Note that `assertions` is required on every test case — a case with only `expectedState` and no `assertions` is rejected.
Common fixture types:
- **files**: provide inline files or namespace file URIs the flow can read.
- **inputs**: set flow input values without changing the flow definition.
- **tasks**: skip or mock task execution, override outputs, or force a state.
- **triggers**: simulate an incoming event (e.g., webhook payload) that starts the flow.
:::alert{type="warning"}
If you don't specify any fixtures, the test will run the entire flow as in production, executing all tasks and producing outputs as usual.
:::
The following flow:
1. sends a Slack message when it starts
2. extracts data from an API
3. transforms the returned data
4. loads the transformed data into BigQuery
```yaml
id: etl_daily_products_bigquery
namespace: company.team
tasks:
- id: send_slack_message_started
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
url: "https://kestra.io/api/mock" # To use this example, replace the url with your own Slack webhook
payload: |
{
"text": "{{ flow.namespace }}.{{ flow.id }}: Daily products flow has started"
}
- id: extract
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/json/orders.json
- id: transform_to_products_name
type: io.kestra.plugin.core.debug.Return
format: "{{ fromJson(read(outputs.extract.uri)) | jq('.Account.Order[].Product[].\"Product Name\"') }}"
- id: transform_to_uppercase
type: io.kestra.plugin.core.debug.Return
format: "{{ fromJson(outputs.transform_to_products_name.value) | upper }}"
- id: load
type: io.kestra.plugin.gcp.bigquery.Load
from: "{{ outputs.transform_to_uppercase.value }}"
destinationTable: "my_project.my_dataset.my_table"
format: JSON
```
This example test suite shows two common patterns: letting one transformation run normally while mocking side effects, and fully mocking an upstream task to isolate a downstream transformation.
```yaml
id: etl_daily_products_bigquery_testsuite
namespace: company.team
flowId: etl_daily_products_bigquery
testCases:
- id: extract_should_return_data
type: io.kestra.core.tests.flow.UnitTest
fixtures:
tasks:
- id: send_slack_message_started
description: "don't send Slack message"
- id: load
description: "don't load data into BigQuery"
assertions:
- value: "{{outputs.transform_to_uppercase.value}}"
isNotNull: true
- id: extract_should_transform_product_names_to_uppercase_mocked
type: io.kestra.core.tests.flow.UnitTest
fixtures:
tasks:
- id: send_slack_message_started
description: "don't send Slack message"
- id: load
description: "don't load data into BigQuery"
- id: extract
description: "don't fetch data from API"
- id: transform_to_products_name
outputs:
value: |
[
"my-product-1"
]
assertions:
- value: "{{outputs.transform_to_uppercase.value}}"
contains: "MY-PRODUCT-1"
```
The `id` is unique to the test suite, and the `namespace` and `flowId` must match the intended flow. When you create a test from a flow, those values are filled in automatically. The `testCases` property contains the `fixtures` and `assertions` for each test case.
In the first test case, `extract_should_return_data`, the `fixtures` include tasks to replace the Slack alert and BigQuery data load so as to not clutter a Slack channel with test alert messages or a BigQuery table with test data but still test the overall design of the flow.
The `assertions` property defines the conditions for success or failure. In the example, the test aims to ensure that the outputs from the `transform_to_uppercase` task are not null.
Because we wrote the test suite with two test cases, both execute during the run. For more isolation, you could separate test cases into multiple tests of the flow as needed. While we know from the first test that the uppercase transformation was successful, you may not want to extract actual data during testing, as it could add load to an external service or send unnecessary alerts. To mitigate this, the second test case `extract_should_transform_product_names_to_uppercase_mocked` mocks `extract` to prevent the API call and mocks `transform_to_products_name` to return `my-product-1` in lowercase — then asserts that `transform_to_uppercase` produced `MY-PRODUCT-1`.
After running, both assertions pass with their actual and expected values visible in the expanded results:

Execution details are not stored in the Executions page like normally run flows to avoid cluttering that space with unnecessary execution details. To view an execution made from a test, open the test case and click the ExecutionId link.
## Unit test with a namespace file
You can also simulate flows with namespace files that contain scripts, test data, or any other file content. In the previous example, you can add a namespace file that contains sample data from the production API endpoint so you do not need to make any API calls during testing. This avoids extra cost and unnecessary calls to external services.
Use the following flow:
```yaml
id: etl_download_file
namespace: company.team
tasks:
- id: extract
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/json/orders.json
method: GET
- id: transform_to_products_name
type: io.kestra.plugin.core.debug.Return
format: "{{ fromJson(read(outputs.extract.uri)) | jq('.Account.Order[].Product[].\"Product Name\"') }}"
- id: transform_to_uppercase
type: io.kestra.plugin.core.debug.Return
format: "{{ fromJson(outputs.transform_to_products_name.value) | upper }}"
- id: load_result_to_outgoing_api
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.transform_to_uppercase.this_task_should_not_be_run }}"
```
Then add a namespace file in the `company.team` namespace that mimics the API response format.
For example, add `my-namespace-file-with-products.json` to the `company.team` namespace:
```json
{
"Account": {
"Account Name": "Firefly",
"Order": [
{
"OrderID": "order103",
"Product": [
{
"Product Name": "Bowler Hat",
"ProductID": 858383,
"SKU": "0406654608",
"Description": {
"Colour": "Purple",
"Width": 300,
"Height": 200,
"Depth": 210,
"Weight": 0.75
},
"Price": 34.45,
"Quantity": 2
},
{
"Product Name": "Trilby hat",
"ProductID": 858236,
"SKU": "0406634348",
"Description": {
"Colour": "Orange",
"Width": 300,
"Height": 200,
"Depth": 210,
"Weight": 0.6
},
"Price": 21.67,
"Quantity": 1
}
]
}
]
}
}
```
This test uses the namespace file as mocked task output so the transformation runs against sample data instead of making the API request:
```yaml
id: etl_mockfile_from_ns
namespace: company.team
flowId: etl_download_file
testCases:
- id: extract_should_transform_productNames_to_uppercase_with_mocked_file
type: io.kestra.core.tests.flow.UnitTest
fixtures:
tasks:
- id: extract
description: "mock extract data file"
outputs:
uri: "{{ fileURI('my-namespace-file-with-products.json') }}" # this file is a namespace file in the same namespace
- id: load_result_to_outgoing_api
description: "don't send end output"
assertions:
- value: "{{outputs.transform_to_uppercase.value}}"
equalTo: "[BOWLER HAT, TRILBY HAT]"
```
With a combination of namespace files and tests, you can target specific components of your flow for correct functionality without using up any external resources or unnecessarily communicating with external hosts for scripts or files.
## Inline file fixture
If you prefer not to use a namespace file for the file fixture in the test, you can also write the file contents inline with the `files` property to achieve the same result:
```yaml
id: etl_mockfile_from_ns
namespace: company.team
flowId: etl_download_file
testCases:
- id: extract_should_transform_product_names_to_uppercase_with_mocked_file
type: io.kestra.core.tests.flow.UnitTest
fixtures:
files:
products.json: |
{
"Account": {
"Account Name": "Firefly",
"Order": [
{
"OrderID": "order103",
"Product": [
{
"Product Name": "Bowler Hat",
"ProductID": 858383,
"SKU": "0406654608",
"Description": {
"Colour": "Purple",
"Width": 300,
"Height": 200,
"Depth": 210,
"Weight": 0.75
},
"Price": 34.45,
"Quantity": 2
},
{
"Product Name": "Trilby hat",
"ProductID": 858236,
"SKU": "0406634348",
"Description": {
"Colour": "Orange",
"Width": 300,
"Height": 200,
"Depth": 210,
"Weight": 0.6
},
"Price": 21.67,
"Quantity": 1
}
]
}
]
}
}
tasks:
- id: extract
description: "mock extract data file"
outputs:
# this file is a namespace file in the same namespace, the fileURI() function will return its URI.
uri: "{{files['products.json']}}"
```
## Trigger fixture example
When your flow is kicked off by a trigger, you can mock the trigger payload directly in the test so you don't have to hit the real endpoint. The example below stubs a webhook trigger carrying an order event and asserts the flow formats the message correctly.
Example flow:
```yaml
id: return-flow-webhook
namespace: io.kestra.tests
triggers:
- id: webhook
type: io.kestra.plugin.core.trigger.Webhook
key: webhook
tasks:
- id: return_summary
type: io.kestra.plugin.core.output.OutputValues
values:
body: "{{ trigger.body }}"
```
Example unit test:
```yaml
id: simple-webhook-test-suite-1-id
namespace: io.kestra.tests
description: assert flow is returning the input value as output
flowId: return-flow-webhook
testCases:
- id: test_case_1
type: io.kestra.core.tests.flow.UnitTest
fixtures:
trigger:
id: webhook
type: io.kestra.plugin.core.trigger.Webhook
variables:
body: webhook
assertions:
- value: "{{ trigger.body }}"
equalTo: "webhook"
```
What this test does: it mocks the webhook trigger payload (`body: webhook`), skips any real HTTP call, runs the flow once, and asserts that the flow receives the mocked payload via `trigger.body`. Because fixtures create a transient execution, the test is fast, isolated, and leaves no execution history clutter.
## Mock task output files
When testing flows that include script tasks (such as Shell, Python, or other scripts) that generate output files, you can mock these output files in your test fixtures. This is particularly useful when:
- You want to test downstream tasks that parse or process output files without running the actual script
- The script is expensive or time-consuming to execute
- You want to test specific edge cases by providing controlled output file content
For example, consider a flow where a shell script generates an output file that is later processed by another task:
```yaml
id: shell_output
namespace: company.team
tasks:
- id: generate_output_file
type: io.kestra.plugin.scripts.shell.Script
taskRunner:
type: io.kestra.plugin.core.runner.Process
outputFiles:
- out.txt
script: |
echo "Processing data..." > out.txt
echo "Result: SUCCESS" >> out.txt
- id: parse_output
type: io.kestra.plugin.core.log.Log
message: "Output file content: {{ read(outputs.generate_output_file.outputFiles['out.txt']) }}"
```
You can create a unit test that mocks the output file content without executing the shell script:
```yaml
id: test_shell_output
flowId: shell_output
namespace: company.team
testCases:
- id: mock_shell_output
type: io.kestra.core.tests.flow.UnitTest
description: Mock shell script output file to test downstream processing
fixtures:
files:
mocked_output.txt: |
Processing data...
Result: SUCCESS
tasks:
- id: generate_output_file
state: SUCCESS
description: "don't run the shell script, mock its output"
outputs:
outputFiles:
out.txt: "{{files['mocked_output.txt']}}"
assertions:
- value: "{{ outputs.generate_output_file.outputFiles['out.txt'] }}"
isNotNull: true
```
In this example:
1. The `files` property defines inline file content (`mocked_output.txt`) that will be used as the mocked output
2. The task fixture for `generate_output_file` specifies `state: SUCCESS` to mark the task as successful without execution
3. The `outputs.outputFiles` property maps the expected output file name (`out.txt`) to the mocked file content using the `files` reference
4. Downstream tasks can read the mocked output file as if the script had actually run
This approach allows you to test the complete flow logic while avoiding the overhead and complexity of executing actual scripts during testing.
## Assert expected failure state
Some flows are designed to fail when conditions are not met — for example, a validation guard that uses `io.kestra.plugin.core.execution.Fail` to reject invalid inputs. The `expectedState` property on a test case lets you assert that a flow ends in a specific terminal state. It defaults to `SUCCESS`; set it to `FAILED`, `WARNING`, `KILLED`, or any other valid state.
The following flow fails when the supplied quantity is not positive:
```yaml
id: order_validation
namespace: company.team
inputs:
- id: quantity
type: INT
tasks:
- id: validate_quantity
type: io.kestra.plugin.core.execution.Fail
condition: "{{ inputs.quantity <= 0 }}"
errorMessage: "Order quantity must be greater than zero"
```
The test asserts that passing a negative value causes the expected failure:
```yaml
id: order_validation_tests
namespace: company.team
flowId: order_validation
testCases:
- id: invalid_quantity_should_fail
type: io.kestra.core.tests.flow.UnitTest
expectedState: FAILED
fixtures:
inputs:
quantity: -1
assertions:
- value: "{{ inputs.quantity }}"
lessThan: 1
```
When `expectedState` is set, the test passes only if the execution ends in exactly that state. If it ends in a different state, the test fails and reports both the expected and actual states.
## Available assertion operators
While the above example uses `isNotNull` and `contains` as assertion operators, there are many more that can be used when designing unit tests for your flows. The complete list is as follows:
| **Operator** | **Description of the assertion operator** |
| -------------------- | ------------------------------------------------------------------------------------------------- |
| isNotNull | Asserts the value is not null, e.g. `isNotNull: true` |
| isNull | Asserts the value is null, e.g. `isNull: true` |
| equalTo | Asserts the value is equal to the expected value, e.g. `equalTo: 200` |
| notEqualTo | Asserts the value is not equal to the specified value, e.g. `notEqualTo: 200` |
| endsWith | Asserts the value ends with the specified suffix, e.g. `endsWith: .json` |
| startsWith | Asserts the value starts with the specified prefix, e.g. `startsWith: prod-` |
| contains | Asserts the value contains the specified substring, e.g. `contains: success` |
| greaterThan | Asserts the value is greater than the specified value, e.g. `greaterThan: 10` |
| greaterThanOrEqualTo | Asserts the value is greater than or equal to the specified value, e.g. `greaterThanOrEqualTo: 5` |
| lessThan | Asserts the value is less than the specified value, e.g. `lessThan: 100` |
| lessThanOrEqualTo | Asserts the value is less than or equal to the specified value, e.g. `lessThanOrEqualTo: 20` |
| in | Asserts the value is in the specified list of values, e.g. `in: [200, 201, 202]` |
| notIn | Asserts the value is not in the specified list of values, e.g. `notIn: [404, 500]` |
## Assert on execution outputs
Rather than assert with an operator and a fixed value, you can use execution outputs in your tests. To assert on execution outputs, use the `{{ execution.outputs.your_output_id }}` syntax in your test assertions. This allows you to verify that task outputs match the expected values.
The following example assumes there is a flow that outputs a value:
```yaml
id: flow_outputs_demo
namespace: demo
tasks:
- id: mytask
type: io.kestra.plugin.core.output.OutputValues
values:
myvalue: kestra
outputs:
- id: myvalue
type: STRING
value: "{{ outputs.mytask.values.myvalue }}"
```
Then, create a unit test for this flow that asserts the output value as follows:
```yaml
id: test_flow_outputs_demo
flowId: flow_outputs_demo
namespace: demo
testCases:
- id: flow_output
type: io.kestra.core.tests.flow.UnitTest
assertions:
- value: "{{ execution.outputs.myvalue }}"
equalTo: kestra
```
When you run this test, Kestra will execute the flow and verify that the output value matches the expected value. If the assertion fails, the test will be marked as failed, and you can inspect the execution logs to see what went wrong.
---
# Worker Isolation in Kestra Enterprise: Separation
URL: https://kestra.io/docs/enterprise/governance/worker-isolation
> Enforce security with Worker Isolation in Kestra. Isolate execution environments, file systems, and processes for secure multi-tenant operations.
Configure worker isolation to separate file systems, restrict thread creation, and enforce script task isolation in multi-tenant Kestra instances.
## Worker isolation — enforce separation
When dealing with multiple teams, you can add extra security measures to your Kestra instance to isolate access so that there is no shared file system, only certain plugins can create worker threads, and script tasks are isolated.
## Java security
By default, Kestra uses a shared worker to handle workloads. This is fine for most use cases. However, when using a shared Kestra instance between multiple teams, this can allow people to access temporary files created by Kestra with powerful tasks like [Groovy](/plugins/plugin-script-groovy), [GraalVM Python](/plugins/plugin-graalvm/python-graalvm), and more. This is because the worker shares the same file system.
You can use the following to opt in to real isolation of file systems using advanced Kestra EE Java security:
```yaml
kestra:
ee:
javaSecurity:
enabled: true
forbiddenPaths:
- /etc/
authorizedClassPrefix:
- io.kestra.plugin.core
- io.kestra.plugin.gcp
```
To only limit access to certain plugins on a Worker without requiring file path protection, you can also consider configuring Kestra with [Allowed & Restricted plugins](../allowed-plugins/index.md).
### `kestra.ee.java-security.forbidden-paths`
This is a list of paths on the file system that the Kestra Worker will be forbidden to read or write to. This can help to protect [Kestra Security and Secrets configuration](../../../configuration/05.security-and-secrets/index.md) files and ensure security for audits and compliance. With this property configured, you can reduce the amount of directories that a Worker can access such as protecting access to the folders where global Kestra configuration or `~/.aws/credentials` are stored.
### `kestra.ee.java-security.authorized-class-prefix`
This is a list of classes that can create threads. Here you can set a list of prefixes (namespace) classes that will be allowed. All others will be refused.
For example, [GCP plugins](/plugins/plugin-gcp) will need to create a thread in order to reach the GCP API. Because this whole plugin is deemed safe, you can authorize it.
### `kestra.ee.java-security.forbidden-class-prefix`
This is a list of classes that can't create any threads. Other plugins will be authorized.
```yaml
kestra:
ee:
javaSecurity:
enabled: true
forbiddenClassPrefix:
- io.kestra.plugin.scripts
```
:::alert{type="warning"}
Currently, all the official Kestra plugins are safe to be authorized **except** [all scripts plugins](../../../16.scripts/00.languages/index.md) because they allow custom code to be created that can be read and written on the file system. Do not add these to the `forbidden-class-prefix`.
:::
## Scripting isolation
Use a [Policy](../policies/index.md) to enforce Docker isolation for script tasks. For installation-wide enforcement, declare a static policy in server configuration (Enterprise Edition):
```yaml
kestra:
policies:
- id: enforce-docker-isolation
description: "Force Docker isolation for all shell script tasks."
rules:
- type: io.kestra.plugin.ee.rules.Add
on: PLUGIN
override: true
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.scripts.shell
values:
containerImage: ubuntu:latest
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
```
Static policies apply across all tenants and cannot be overridden by namespace-level policies, making them suitable for cluster-wide security requirements.
:::alert{type="warning"}
Add rules for all script plugin types (Python, Node, and others) to ensure no tasks can bypass Docker isolation.
:::
---
# Instance Management in Kestra Enterprise: Health
URL: https://kestra.io/docs/enterprise/instance
> Manage your Kestra Instance. Monitor service health, handle upgrades, configure maintenance mode, and manage global settings from a centralized dashboard.
import ChildCard from "~/components/docs/ChildCard.astro"
The [Instance Owner console](./00.instance-owner/index.md) gives you a centralized view of your Kestra deployment's health along with controls for upgrades, maintenance, and user notifications.
## System overview
Inside the Instance Owner console, the **Instance** section covers service health, kill switches, announcements, maintenance mode, and plugin versioning. The **System Overview** tab gives a high-level snapshot of your instance’s operational status, including license validity, usage statistics, and installed secrets and storage plugins.
## Services
Kestra tracks the health of critical components, including:
- **Workers**: Execute tasks.
- **Schedulers**: Trigger workflows.
- **Executors**: Manage task execution.
- **Webservers**: Host the UI and API.
Each service displays its state, hostname, server type, version, start date, and last health check date. Click a service to see per-instance details for debugging: session timeout, heartbeat interval, last heartbeat, and termination grace period.
### Service states
- **RUNNING** — service is active and healthy.
- **MAINTENANCE** — intentionally paused; executions are blocked until resumed.
- **DISCONNECTED** — lost a required dependency (e.g., DB/queue) and may recover or shut down.
- **TERMINATING** — shutting down; tries to drain work before stopping.
- **TERMINATED_GRACEFULLY / TERMINATED_FORCED** — shutdown completed (clean or forced), moves to `NOT_RUNNING`.
- **NOT_RUNNING / INACTIVE** — post-shutdown, final states reported in the UI.
Additional tabs include **Configuration** for port details, **Metrics** such as CPU usage and executor thread count, and an **Events Timeline** showing the service’s full lifecycle.
## Announcements
[Announcements](./announcements/index.md) let you broadcast in-app banners to notify users about planned maintenance, incidents, or other events. Each announcement has a message, a severity type (`INFO`, `WARNING`, or `ERROR`), and a start and end date.
## Maintenance mode
[Maintenance mode](./maintenance-mode/index.md) pauses the platform broadly: new executions are queued and running ones are allowed to finish before services stop. Combine it with an Announcement so users see a banner while the platform drains.
## Versioned Plugins
[Versioned Plugins](./versioned-plugins/index.md) lists all plugins installed on the instance. Upgrade, install, or uninstall plugins without restarting Kestra.
---
# Announcements in Kestra Enterprise: In-App Banners
URL: https://kestra.io/docs/enterprise/instance/announcements
> Broadcast messages with Kestra Announcements. Create in-app banners to notify users about maintenance, updates, or important system information.
Communicate planned maintenance or incidents with in-app banners
## How to create an announcement
To add a custom in-app banner, go to **Instance Owner → Instance → Announcements**.
As an Instance Owner, you can configure the following within each announcement:
- **Message**: the text to display in the banner
- **Type**: the severity of the banner — `INFO`, `WARNING`, or `ERROR`
- **Start** and **End** date during which the announcement is displayed.

---
# Instance Owner Console in Kestra Enterprise
URL: https://kestra.io/docs/enterprise/instance/instance-owner
> Use the Instance Owner console in Kestra Enterprise to manage tenants, instance-wide IAM, infrastructure, and governance from a single interface.
The Instance Owner console provides instance-wide administration for tenants, IAM, infrastructure, and governance — separate from the tenant workspace you work in day to day.
:::alert{type="warning"}
Instance Owner operations are instance-wide. Changes to tenants, users, worker groups, and instance-level policies affect the entire Kestra instance, not just the tenant you are currently logged into.
:::
## Entering Instance Owner
In the sidebar, click **Instance Owner**. The entry is only visible to users with the Instance Owner privilege.
The UI switches to the Instance Owner console and shows a banner confirming you are administering the whole instance. Your regular tenant workspace is unaffected — you can return to it at any time.
## The Instance Owner console
The console organizes instance-wide administration into five sections:

| Section | What you can manage |
| :--- | :--- |
| **Instance** | Overview, System Overview, Services, Kill Switch, Announcements |
| **Tenants** | Create, edit, and delete tenants; configure dedicated storage and secrets backends per tenant |
| **Instance IAM** | Users and Service Accounts that exist at the instance level, independently of any tenant |
| **Infrastructure** | Worker Groups, Worker Queues, Versioned Plugins, MCP Servers |
| **Governance** | Instance-level Policies and Audit Logs across all tenants |
## Exiting Instance Owner
Click **Exit Instance Owner** at the top of the left sidebar to return to your tenant workspace.
## Who can access Instance Owner
Only users with the Instance Owner privilege can enter the Instance Owner console.
## Creating an Instance Owner user
### Through the setup wizard
When you launch Kestra for the first time, the [setup wizard](../../01.overview/02.setup/index.md) invites you to create the first user, which is automatically assigned the Instance Owner privilege.
### Through the CLI
To create a new user with the Instance Owner privilege:
```bash
kestra auths users create admin@kestra.io TopSecret42 --instance-owner
# with tenant scoping:
kestra auths users create \
--tenant= --instance-owner
```
:::alert{type="info"}
`--superadmin` is a deprecated alias for `--instance-owner` and still works.
:::
### Through configuration
An Instance Owner can also be defined in the configuration file:
```yaml
kestra:
security:
instance-owner:
username:
password:
tenant-admin-access:
-
```
:::alert{type="info"}
`kestra.security.super-admin` is a deprecated alias for `kestra.security.instance-owner` and still works.
:::
For the full list of security configuration options, see [Security and Secrets configuration](../../../configuration/05.security-and-secrets/index.md).
## Granting and revoking Instance Owner access
You must be an Instance Owner yourself to grant or revoke the privilege.
### Through the UI
Open the user's detail page and toggle the Instance Owner switch:

### Through the CLI
```bash
kestra auths users set-instance-owner admin@kestra.io true # grant
kestra auths users set-instance-owner admin@kestra.io false # revoke
```
:::alert{type="info"}
`set-superadmin` is a deprecated alias for `set-instance-owner` and still works.
:::
---
# Kill Switch in Kestra Enterprise: Stop Executions
URL: https://kestra.io/docs/enterprise/instance/kill-switch
> Use Kill Switch in Kestra Enterprise to immediately kill, cancel, or ignore executions by scope, with scheduling, audit logs, and in-app banners.
Kill Switch is an operational safety lever that lets administrators stop misbehaving executions directly from the UI.
## Why a Kill Switch exists
A runaway flow, a bad deployment, or a tenant-specific incident can flood workers with problematic executions. The Kill Switch lets administrators halt or quarantine those executions instantly, without pausing the entire platform or touching infrastructure.
Use it when you need to:
- Contain impact quickly while you ship a fix or rollback.
- Target only the affected tenant/namespace/flow/execution instead of stopping everything.
- Keep an auditable record of who intervened, when, and why.
- Surface a visible banner so impacted users know what happened.
Kill Switch replaces the CLI-only `--skip-executions` and `--skip-flows` commands with a scoped, auditable administration interface.
## Configure a Kill Switch
To configure a Kill Switch, go to **Instance Owner → Instance → Kill Switch**. Name the switch (e.g., `Kill Switch – Payments Namespace Outage (TEMP)`) and configure its specifications.
### Kill Switch types
| Type | Behavior |
|------|----------|
| **KILL** | Kills running executions after the current task completes; any remaining tasks in the execution will not run. New executions are transitioned to `KILLED` state instantly. |
| **CANCEL** | Blocks new executions; lets current task runs finish before marking the execution `CANCELLED`. |
| **IGNORE** | Ignores all messages for matching executions—use as a last resort when an execution cannot be killed or cancelled. |
For **KILL** and **CANCEL**, executions receive a [system label](../../../06.concepts/system-labels/index.md) identifying which Kill Switch applied.
### Scope
Scope sets the reach of the Kill Switch with **Tenant** being the most inclusive and **Execution** the most specific. A **Namespace** scope requires a **Tenant**, and a **Flow** scope requires a **Tenant** and **Namespace**. The UI automatically adjusts to show only the relevant scope requirements depending on your first selection.
All possible scopes are listed below:
- **Tenant**
- **Namespace**
- **Flow**
- **Execution**
### Scheduling
The Kill Switch requires a **Start Date** and can be kept open ended if needed.
- Mandatory **Start Date** (default: now)
- Optional **End Date**
- Enable/disable from the **Kill Switch** tab at any time
### Description
Admins can optionally include a free-text reason stored with the Kill Switch and surfaced in banners to document the incident or change request.
## Lifecycle and audit
Creation and updates are written to [**Audit Logs**](../../02.governance/06.audit-logs/index.md), and every state change—create, enable, disable, or archive—is recorded. Deleting a Kill Switch performs a soft delete, so the archived entry remains visible for traceability.
## Announcement banner
Kill Switches raise contextual banners to alert affected users. A namespace-scoped Kill Switch shows the banner only to users working in that namespace, while a tenant-scoped one surfaces the banner across the UI for all users in the tenant.
## CLI compatibility
The CLI remains for open-source parity, with renamed flags to match the behavior:
```bash
# Old
--skip-executions / --skip-flows
# New
--ignore-executions / --ignore-flows
```
## Relationship to maintenance mode
[Maintenance Mode](../maintenance-mode/index.md) pauses the platform broadly (queues new executions, lets running ones finish). Kill Switch keeps services up and targets specific tenants/namespaces/flows/executions to stop or ignore problematic runs—an operational tool rather than a platform pause.
---
# Maintenance Mode in Kestra Enterprise: Safe Upgrades
URL: https://kestra.io/docs/enterprise/instance/maintenance-mode
> Safely upgrade with Kestra Maintenance Mode. Pause new executions while allowing running tasks to complete for seamless system updates.
Prepare your Kestra instance for maintenance or migration.
## How maintenance mode works
When activated:
- The [executor](../../../08.architecture/02.server-components/index.md#executor) stops processing new executions and queues them automatically.
- [Workers](../../../08.architecture/02.server-components/index.md#worker) finish their current tasks without picking up new ones.
- The [web server](../../../08.architecture/02.server-components/index.md#webserver) and [scheduler](../../../08.architecture/02.server-components/index.md#scheduler) remain active — no requests are lost.
## Enter maintenance mode
Go to **Instance Owner → Instance → Services** and click **Enter maintenance mode** in the top-right corner. A confirmation prompt explains what will happen before the transition begins. To exit, click the same button and confirm you want to return to a live state.
---
# Versioned Plugins in Kestra Enterprise: Install and Test Any Version
URL: https://kestra.io/docs/enterprise/instance/versioned-plugins
> Run multiple versions of any plugin side-by-side across your Kestra instance. Build, test, and ship plugin updates from a jar file without disrupting existing flows — and upgrade on your own timeline.
Install multiple versions of any plugin and run them simultaneously across all your flows — assigned at the task, flow, namespace, or instance level.
## Use cases
Versioned plugins give you fine-grained control over which plugin version runs in each flow, namespace, or across your entire instance. Common use cases include:
- **Build and iterate on plugins**: Upload a custom plugin directly from a `.jar` file and test it immediately in any flow — no instance restart required.
- **Ship fixes to production fast**: When Kestra ships a new or patched plugin, install it via the UI or API and point specific flows to the new version within minutes.
- **Upgrade on your own timeline**: Run the latest plugin version in new flows while existing flows stay pinned to a stable version until you're ready to migrate.
## Configuration
Versioned plugins support several properties that can be modified in your Kestra configuration:
- `remoteStorageEnabled`: Specifies whether remote storage is enabled (i.e., plugins are stored on the internal storage).
- `localRepositoryPath`: The local path where managed plugins will be synced.
- `autoReloadEnabled`: Whether the server should periodically rescan repositories for new or removed plugins.
- `autoReloadInterval`: How often to rescan (duration, e.g., `60s`).
- `defaultVersion`: The version to use when none is specified in a flow. Accepted values: `LATEST`, `CURRENT`, `OLDEST`, `NONE`, or an explicit version (e.g., `0.20.0`).
```yaml
kestra:
plugins:
management:
enabled: true # setting to false will make Versioned plugin tab disappear + API will return an error
remoteStorageEnabled: true
customPluginsEnabled: true # setting to false will disable installing or uploading custom plugins
localRepositoryPath: /tmp/kestra/plugins-repository
autoReloadEnabled: true
autoReloadInterval: 60s
defaultVersion: LATEST
```
### Allow-list URLs
The following URLs must be reachable from your Kestra instance:
- https://repo.maven.apache.org/maven2/
- https://registry.kestra.io/maven/
- https://api.kestra.io/
```yaml
kestra:
plugins:
repositories:
central:
url: https://repo.maven.apache.org/maven2/
kestra:
url: https://registry.kestra.io/maven
```
Refer to the [Plugins and Execution](../../../configuration/04.plugins-and-execution/index.md) page in the Configuration guide for custom Maven repositories.
With remote storage enabled, installed plugins are stored in a plugins repository in the `_plugins/repository` path. For example, the below paths show the storage for 0.19.0 and 0.20.0 versions of the Shell script plugin:
```bash
_plugins/repository/io_kestra_plugin__plugin-script-shell__0_19_0
_plugins/repository/io_kestra_plugin__plugin-script-shell__0_19_0.jar
_plugins/repository/io_kestra_plugin__plugin-script-shell__0_20_0
_plugins/repository/io_kestra_plugin__plugin-script-shell__0_20_0.jar
```
Artifact files are renamed using the format: `____` to be easily parseable (dots `.` are replaced with `_` for `groupId` and `version`).
For locally stored plugins configured by the `localRepositoryPath` attribute, the file path looks like `/tmp/kestra/plugins-repository`. For example, the following plugins are stored locally, where the local repository contains a JSON `plugins.meta` file that contains metadata about remote plugins. This file is used for synchronization, where only plugins with detected changes are synchronized.
```bash
├── io_kestra_plugin__plugin-kafka__0_20_0.jar
├── io_kestra_plugin__plugin-script-shell__0_20_0.jar
├── io_kestra_plugin__plugin-terraform__0_20_0.jar
├── io_kestra_plugin__plugin-transform-grok__0_20_0.jar
└── plugins.meta
```
## Configuration for EE-specific plugins
Some plugins are available only in the Enterprise Edition (EE) of Kestra. To install EE-specific plugins, you need to make sure that your [Enterprise and Advanced configuration](../../../configuration/06.enterprise-and-advanced/index.md) has the `kestra.ee.license.fingerprint` property set (apart from the `kestra.ee.license.id` and `kestra.ee.license.key` properties). The `kestra.ee.license.fingerprint` property is used to verify that the EE license is valid and allows you to use EE-specific plugins.
## Install versioned plugins
Versioned plugins can be installed from the Kestra UI as well as programmatically.
### From the UI
Below is a video demonstration walking through each step from installation to application in a flow.
Both official and custom plugins can be installed from the UI. Go to **Instance Owner → Infrastructure → Versioned Plugins** and click **+ Install** to browse the full plugin library. Search for the plugin, select the version, and confirm.
After installing, the full list of versioned plugins is displayed. Kestra alerts you when a newer version is available. Upgrading installs the new version separately — the previous version stays active in existing flows.
For a custom plugin, click **+ Install**, switch to **Custom plugin**, and provide:
- **Group ID**: The group identifier of the plugin.
- **Artifact ID**: The artifact identifier of the plugin.
To upload a custom build, click **Upload** and select a valid `.jar` file.
### From the API
Only Instance Owner users can install versioned plugins with the API. To install a versioned plugin, you can use the API POST request with your username and password with `-u` or an [API token](../../03.auth/api-tokens/index.md).
With Kestra username and password:
```bash
curl -X POST http://0.0.0.0:8080/api/v1/cluster/versioned-plugins/install \
-u 'admin@kestra.io:kestra' \
-H "Content-Type: application/json" \
-d '{"plugins":["io.kestra.plugin:plugin-airbyte:0.21.0"]}'
```
With API Token:
```bash
curl -X POST http://0.0.0.0:8080/api/v1/cluster/versioned-plugins/install \
-H "Authorization: Bearer YOUR-API-TOKEN" \
-H "Content-Type: application/json" \
-d '{"plugins":["io.kestra.plugin:plugin-airbyte:0.21.0"]}'
```
To uninstall a versioned plugin, use the following DELETE request:
```bash
curl -X DELETE http://0.0.0.0:8080/api/v1/cluster/versioned-plugins/uninstall \
-u 'admin@kestra.io:kestra' \
-H "Content-Type: application/json" \
-d '{"plugins":["io.kestra.plugin:plugin-airbyte:0.21.0"]}'
```
To check for all available versions of a plugin, you can use the following API request to resolve:
```bash
curl -X POST http://0.0.0.0:8080/api/v1/cluster/versioned-plugins/resolve \
-u 'admin@kestra.io:kestra' \
-H "Content-Type: application/json" \
-d '{"plugins":["io.kestra.plugin:plugin-airbyte:0.21.0"]}'
```
If you want to install a newer plugin version, use the install request with the specified version or use `LATEST` instead of the version number. This creates a second, separate installation of the plugin, so you can keep using an old version in production flows and test using the newer version in development.
```bash
curl -X POST http://0.0.0.0:8080/api/v1/cluster/versioned-plugins/install \
-u 'admin@kestra.io:kestra' \
-H "Content-Type: application/json" \
-d '{"plugins":["io.kestra.plugin:plugin-airbyte:LATEST"]}'
```
### From the CLI
To install versioned plugins from the [Kestra CLI](../../../kestra-cli/kestra-server/index.md), you can use the following command:
```bash
./kestra plugins install --locally=false io.kestra.plugin:plugin-jdbc-duckdb:0.21.2
```
The `--locally` flag specifies whether the plugin should be installed locally or according to your Kestra configuration, where remote storage can be enabled.
- `--locally=true` installs the plugin locally.
- `--locally=false` checks if `remoteStorageEnabled` is enabled and then plugins are downloaded and pushed to the [configured runtime and storage backend](../../../configuration/02.runtime-and-storage/index.md) directly.
## `version` property in a Flow
Specify the plugin version on any task or trigger using the `version` property. The following flow pins the Shell Script plugin to `0.21.0` while a newer version is also installed on the instance:
```yaml
id: shell_script_example
namespace: company.team
tasks:
- id: http_download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: shell_script_task
type: io.kestra.plugin.scripts.shell.Script
version: "0.21.0"
outputFiles:
- first.txt
script: |
echo "The current execution is : {{ execution.id }}"
echo "1" >> first.txt
cat {{ outputs.http_download.uri }}
```
The `version` property also accepts specific, non-case-sensitive values like in the configuration file:
- `LATEST` (or `latest`): To use the latest available version of a Kestra plugin.
- `OLDEST` (or `oldest`): To use the oldest available version of a Kestra plugin.
When there are multiple versions of a plugin available, Kestra resolves the version of a plugin by following this priority order:
1. **Task-Level**: Using the version specified in the `version` property.
2. **Flow-Level**: Using the plugin’s default version.
3. **Namespace-Level**: Using the plugin’s default version for the namespace.
4. **Instance-Level**: Using the value set in `kestra.plugins.management.defaultVersion` (default: `LATEST`).
- This property can be configured to `NONE` to enforce that a version is always explicitly defined.
By default, Kestra uses `LATEST` for core plugins when no version can be resolved. For other plugins, an unresolvable version renders the flow invalid.
:::alert{type="info"}
The version is resolved both at flow creation time and execution time to ensure the correct plugin version is used during both stages. This means that a Task/Trigger can only be deserialized after ensuring that all default versions are properly resolved.
:::
---
# Enterprise Edition in Kestra: Architecture and Setup
URL: https://kestra.io/docs/enterprise/overview
> Overview of the Enterprise Edition with an introduction to our enterprise-level features and initial setup guide.
import ChildCard from "~/components/docs/ChildCard.astro"
Kestra Enterprise Edition builds on the open-source version with granular access control, data isolation, high-availability architecture, and enterprise support.
Explore the sections below or follow the setup guide to get started.
---
# Enterprise Features in Kestra: High-Availability
URL: https://kestra.io/docs/enterprise/overview/enterprise-edition
> Learn about the Enterprise Edition and how it can help you run Kestra securely and reliably at scale.
Kestra Enterprise Edition is built for production workloads with high security and compliance requirements, deployable wherever you need.
## Key features
Kestra Enterprise is an additive overlay on the [Open Source Edition](https://github.com/kestra-io/kestra) — it shares one codebase and preserves all open-source behavior while adding enterprise capabilities on top.
**High availability**: Designed to be highly available and fault-tolerant. Supports **Kafka**, **Redis**, **AMQP**, or **GCP Pub/Sub** as the queue backend, and **Elasticsearch** for the search and read model — eliminating single points of failure and enabling horizontal scaling for large workloads.
**Multi-tenancy**: Separate environments for different teams or projects. Each tenant is fully isolated, with its own access control policies, and can optionally run with Worker Isolation and dedicated worker groups to prevent cross-tenant contention.
**Security and access control**: SSO and RBAC let you integrate with your existing identity provider and manage user access to workflows and resources. Enforce plugin allow-lists, apply read-only secrets for least privilege, and use audit logs for full traceability.
**Enterprise features**: Audit Logs, Custom Blueprints, namespace-level secrets, variables and plugin defaults, Assets packaging, declarative Unit Tests for flows, Versioned Plugins for safe upgrades, operational safeguards like the Kill Switch and in-product Announcements, and [Promote](../../../version-control-cicd/06.promote/index.md) for moving flows across environments from the UI with a review gate.
**Secrets management**: Securely stores and manages secrets. Supports read-only secrets for sensitive values and integrates with AWS Secrets Manager, Azure Key Vault, Elasticsearch, Google Secret Manager, HashiCorp Vault, Doppler, 1Password, and more.
**Support**: Guaranteed SLAs and priority support, with onboarding and training to ensure a fast and confident start.
[Get in touch](/demo) to learn more.
:::alert{type="info"}
**Kestra Cloud:** If you’re unable to host Kestra Enterprise yourself, you can try Kestra Cloud — a fully managed SaaS solution hosted by the Kestra team. Kestra Cloud is currently in early access. If you are interested in trying it out, [sign up here](/cloud).
:::
---
# Migrate from OSS to Kestra Enterprise Edition
URL: https://kestra.io/docs/enterprise/overview/migrate-from-oss
> Migrate your Kestra OSS instance to Enterprise Edition. Learn how to export flows, data, and settings before importing them into Kestra Enterprise.
When you move to Kestra Enterprise Edition, you can bring your existing flows from the open-source version. This guide covers how to export and import flows, and what to migrate separately.
## Export and import flows
Kestra provides a built-in export/import mechanism for flows:
1. In the **Open Source** UI, go to **Settings**.
2. Click **Export All Flows** to download a single `.zip` file containing all your flows.
3. In the **Enterprise Edition** UI, go to **Flows**.
4. Click **Import** and select the `.zip` file you downloaded.
This will import all flows into your Enterprise Edition instance.
## Namespace files, KV store, and other resources
**Namespace Files** and the **Key-Value Store** data are not included in the flow export. If you rely on these, you will need to migrate them manually.
For **Namespace Files**, re-upload the files through the Enterprise Edition UI or use the [API](../../../api-reference/index.mdx). For the **KV Store**, recreate the entries in your new instance.
## What's next
Once your flows are imported, you can start using enterprise features such as [RBAC](../../03.auth/rbac/index.md), [Secrets Management](../../02.governance/secrets-manager/index.md), [Worker Groups](../../04.scalability/worker-group/index.md), and more.
---
# Set Up Kestra Enterprise: License and First Tenant
URL: https://kestra.io/docs/enterprise/overview/setup
> Configure your Kestra Enterprise instance. Activate your license, create the first tenant, and complete the initial setup to start using Enterprise features.
When you launch Kestra Enterprise Edition for the first time, Kestra walks you through configuring your instance: setting up your first tenant, creating your first user, and starting the Kestra UI.
## Prerequisites
Kestra Enterprise Edition requires a valid license under `kestra.ee.license`. Contact [sales@kestra.io](mailto:sales@kestra.io) to obtain one.
The license is set up using three configuration properties: `id`, `fingerprint`, and `key`.
- `kestra.ee.license.id`: license identifier.
- `kestra.ee.license.fingerprint`: license authentication. This is required to use [Versioned Plugins](../../05.instance/versioned-plugins/index.md).
- `kestra.ee.license.key`: license key.
```yaml
kestra:
ee:
license:
id:
fingerprint:
key: |
```
When you launch Kestra Enterprise Edition, it will check the license and display the validation step in the log.
## Step 1: Validate configuration
The first screen shows the main configuration of your instance. It displays:
- whether `multitenancy` is enabled
- whether `default tenant` is enabled — if yes, you can skip Step 2 allowing you to create your first tenant
- which `database` backend is configured (e.g., PostgreSQL or Elasticsearch)
- which `queue` backend is configured (e.g., PostgreSQL or Kafka)
- which `internal storage` backend is configured (e.g., S3, GCS, Azure Blob Storage, MinIO, or local storage)
- which `secret` backend is configured (e.g., Vault, AWS Secrets Manager, Elasticsearch, or not set up yet)

This step asks you to confirm whether your configuration is valid. If not, you can correct the configuration, restart the instance, and start the setup from scratch.
## Step 2: Create your first tenant
If `multitenancy` is enabled, Kestra will prompt you to create your first tenant. Enter a Tenant ID and Tenant Name, for example:
- tenant id: `stage`
- tenant name: `Staging Environment`
If you enabled a default tenant, you can skip this step.

## Step 3: Create your first user
Create the first user. This user receives the [Instance Owner](../../03.auth/rbac/index.md#instance-owner) privilege and can manage tenants, users, and roles across the instance.

## Step 4: Start Kestra UI
Once your tenant and user are configured, Kestra will launch the UI and log you into your new tenant as the first user.

---
# Install Kestra Enterprise from Standalone JAR
URL: https://kestra.io/docs/enterprise/overview/standalone-server-installation
> Install Kestra Enterprise on a standalone server using an executable JAR file. Run the platform without Docker where containerization is unavailable.
To deploy Kestra without Docker, use the standalone JAR — a single executable that runs in any environment with JVM 21+.
## Standalone JAR
Download the latest version of the Kestra EE JAR from:
[http://registry.kestra.io/exe/latest](http://registry.kestra.io/exe/latest)
**Credentials:**
- **Username**: `license-id`
- **Password**: `fingerprint`
:::alert{type="info"}
Make sure to store your credentials in an `application.yaml` file.
:::
This provides a single JAR file that can be used to start Kestra. Store the file in your execution environment as `kestra` (make it executable).
Make the file executable (Linux or macOS):
```bash
chmod +x kestra-ee-VERSION # Replace VERSION with your version
```
Or with a file path:
```bash
mv kestra-ee-VERSION /usr/local/bin/kestra # Replace with your version and execution environment file path
chmod +x /usr/local/bin/kestra
```
Then run:
```bash
./kestra-ee-VERSION server standalone # Replace VERSION with your version
```
:::alert{type="info"}
You need to provide a configuration with a connection to a database.
:::
For Windows users:
```powershell
java -jar kestra-ee-VERSION # Replace VERSION with your version
```
Or with a file path assuming execution from the current directory:
```powershell
java -jar kestra-ee-VERSION server standalone -c ./application.yaml -p ./plugins --port=8080 # Replace VERSION with your version
```
## Plugins
In standalone JAR deployments, all plugins must be downloaded separately.
Kestra EE provides a command to install all available plugins:
```shell
## Install all available plugins
kestra plugins install --all
```
This installs task plugins in the `plugins` directory. To install them elsewhere, specify a path with the `-p` argument.
Additional Enterprise Edition plugins that are not task-related may also be required — such as secrets or storage plugins.
## Secret plugins
Secret plugins must be downloaded from the Kestra registry using the same credentials, and placed in your `plugins` directory.
| Secret Service | Download Link |
| :------------- | :------------- |
| Vault | https://registry.kestra.io/maven/io/kestra/ee/secret/secret-vault/0.24.0/secret-vault-0.24.0.jar |
| AWS | https://registry.kestra.io/maven/io/kestra/ee/secret/secret-aws/0.24.0/secret-aws-0.24.0.jar |
| GCP | https://registry.kestra.io/maven/io/kestra/ee/secret/secret-gcp/0.24.0/secret-gcp-0.24.0.jar |
| Azure | https://registry.kestra.io/maven/io/kestra/ee/secret/secret-azure/0.24.0/secret-azure-0.24.0.jar |
## S3-compatible internal storage
To use S3-compatible object storage (AWS S3, MinIO, Ceph, SeaweedFS, Garage), install the storage plugin:
```shell
kestra plugins install io.kestra.storage:storage-minio:LATEST
```
## Enterprise deployment configuration
For the full list of configuration options, refer to the [Configuration Reference](https://kestra.io/docs/configuration).
To enable Kestra Enterprise features, configure the following parameters:
| Configuration Parameter | Required | Documentation Link | Description |
| :---------------------- | :------- |:-----------------------------------------------------------------------------------------------------------------------------------| :---------- |
| Enterprise License | Yes | [Enterprise and Advanced Features](../../../configuration/06.enterprise-and-advanced/index.md) | License information for the Kestra instance |
| Multi-tenancy | Yes | [Enterprise and Advanced Features](../../../configuration/06.enterprise-and-advanced/index.md) | Enables/disables multi-tenancy (required for SCIM) |
| Secret Manager | Yes | [Security and Secrets](../../../configuration/05.security-and-secrets/index.md) | Configure a secret manager in RW or RO mode |
| Encryption Key | Yes | [Security and Secrets](../../../configuration/05.security-and-secrets/index.md) | Key to encrypt inputs/outputs in flows |
| Security | No | [Security and Secrets](../../../configuration/05.security-and-secrets/index.md) | Configure Instance Owner (also settable in UI on startup) |
| User Invitations | No | [Runtime and Storage](../../../configuration/02.runtime-and-storage/index.md), [Observability and Networking](../../../configuration/03.observability-and-networking/index.md) | Required for email invitations (not needed with LDAP/SCIM) |
| SSO | No | [SSO](../../03.auth/sso/index.md) | Configure OIDC provider |
| LDAP | No | [LDAP](../../03.auth/sso/ldap/index.md) | Connect to an existing LDAP provider |
| SCIM | No | [SCIM](../../03.auth/scim/index.mdx) | Sync user/group membership with SCIM 2.0 |
## Starting Kestra
Kestra can be started in **standalone mode** or in a **distributed setup** for production.
Ensure a database is configured and your Enterprise credentials are stored in `application.yaml`.
## Standalone server
```shell
kestra server standalone -c ./application.yaml -p ./plugins --port=8080
```
This starts Kestra as a standalone service on port `8080`.
## Distributed mode
For production, run Kestra in distributed mode for scalability and high availability.
Each component can run independently across servers, with shared access to the same database (no TCP communication is required between components).
Example with all components on one server:
```shell
kestra server webserver -c ./application.yaml -p ./plugins --port=8080
kestra server scheduler -c ./application.yaml -p ./plugins --port=8081
kestra server worker -c ./application.yaml -p ./plugins --port=8082
kestra server executor -c ./application.yaml -p ./plugins --port=8083
```
---
# Scale Kestra Enterprise: Worker Groups and Apps
URL: https://kestra.io/docs/enterprise/scalability
> Scale Kestra Enterprise with advanced features. Explore Worker Groups, Task Runners, and Apps to enhance performance, isolation, and productivity.
import ChildCard from "~/components/docs/ChildCard.astro"
Use Worker Groups and Apps to scale your Kestra Enterprise deployment and give non-technical users a way to interact with your workflows. For offloading compute-intensive scripting tasks to remote environments, see [Task Runners](../../task-runners/index.mdx).
---
# Apps in Kestra Enterprise: Frontends for Flows
URL: https://kestra.io/docs/enterprise/scalability/apps
> Build custom Apps with Kestra. Create user-facing interfaces for workflows, enabling forms, approvals, and interactive data applications.
Build custom UIs to interact with Kestra from the outside world.
## What are Apps
Apps let you wrap any Kestra flow in a custom UI — forms for data entry, output displays, approval buttons, progress indicators, and markdown blocks — without writing any frontend code. Building and serving a frontend, connecting it to Kestra's API, validating user inputs, handling responses, managing workflow outputs, and handling authentication and authorization — Apps take care of all of that.
Flows act as the backend, processing data and executing tasks. Apps act as the frontend, giving anyone — including business users without Kestra access — a clean interface to trigger executions, submit approvals, and view results.
Apps work well for external-facing forms and for internal workflows such as approvals, requests, and guided operations.
---
## Common App patterns
Start with one of these patterns when designing your own App:
- **FTP upload portal**: give users a simple upload form while Kestra handles the backend credentials and transfer logic. See the [business user Apps blog example](../../../../blogs/use-case-apps/index.md#requests--review).
- **Self-serve analytics request**: let users choose a dimension and time range, run a query and chart generation flow, and return the generated output on `SUCCESS`. See the [dynamic self-serve example](../../../../blogs/use-case-apps/index.md#dynamic-self-serve).
- **AI-assisted intake or user research assistant**: collect free-form context from a sales, product, or support team member, run an LLM-backed flow, and display the suggested answer or categorization back in the App. See the [everyday automation example](../../../../blogs/use-case-apps/index.md#simple-interfaces-for-everyday-automation).
- **VM or infrastructure request**: collect the requested environment, size, region, and justification on `OPEN`, show validation progress on `RUNNING`, pause for approval on `PAUSE`, then display the created VM details on `SUCCESS`. This pattern also fits the infrastructure workflows described in the [infrastructure automation blog](../../../../blogs/infra-automation/index.md).
- **Human-in-the-loop review**: display task outputs, logs, or model results, then let an approver accept or reject the execution from the same screen.
Start by mapping the user journey:
1. What should the user submit?
2. What should they see while the flow is running?
3. Does the flow need approval or review?
4. What is the final outcome you want to show back in the App?
Those answers determine which blocks to choose for each stage.
For more patterns, browse the Apps-focused posts in the [blog section](../../../../blogs/introducing-apps/index.md) and [solutions content](../../../../blogs/use-case-apps/index.md).
---
## How App stages map to execution progress
The stage-based model lets a single app guide users through the full request lifecycle — from submission through approval to delivery — by rendering different blocks based on the current execution state.
| App stage | What the user sees | What usually happens in the flow |
|-----------|--------------------|----------------------------------|
| `OPEN` | The initial form or landing page | No execution exists yet. The user is about to submit a request. |
| `CREATED` | Optional confirmation that the request was accepted | Kestra created the execution and is about to start processing it. |
| `RUNNING` | Progress text, logs, loading indicators, or intermediate outputs | Tasks are actively running. |
| `PAUSE` | Approval or review screen | The flow is waiting on a paused task or a manual decision. |
| `RESUME` | Post-approval confirmation and follow-up details | The paused execution was resumed and continues running. |
| `SUCCESS` | Final outputs, download links, or next-step buttons | The execution completed successfully. |
| `FAILURE`, `ERROR`, `FALLBACK` | Error messages, logs, retry guidance, escalation links | The execution did not complete as expected. |
For example, a VM request app might start with an `OPEN` form, move to `RUNNING` while Kestra validates the request, switch to `PAUSE` while a platform engineer reviews the requested size and environment, then show `SUCCESS` once the VM has been provisioned.
This stage-based layout is what makes Apps easier for non-technical users: they don't need to understand workflow internals, only the current step of their request.
---
## Creating Apps in code
If you already have a flow, building an App is a single YAML file: point it at the flow's `namespace` and `flowId`, define what users see at each execution stage with layout blocks, and save. No additional backend code is required — your flow is already the backend.
To create a new app, go to **Apps** and click **+ Create**. Add your app configuration as YAML and click **Save**. Like flows, apps have multiple editor views — you can configure the app while viewing documentation, previewing the layout, or searching the blueprint repository.
Set `disabled: true` to stage an app before release. A disabled app does not appear in the catalog and cannot be opened via its URL until you enable it.
For the full list of available blocks and their properties, see [App layout blocks](#app-layout-blocks) below.

### App to run a Hello World flow
Every app is backed by a flow. The following flow accepts a name input and logs a message when triggered:
```yaml
id: myflow
namespace: company.team
inputs:
- id: user
type: STRING
defaults: World
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Hello {{ inputs.user }}
```
Then add an app that triggers that flow:
```yaml
id: hello_world_form
type: io.kestra.plugin.ee.apps.Execution
displayName: Hello World Form
namespace: company.team
flowId: myflow
access:
type: PUBLIC
layout:
- on: OPEN
blocks:
- type: io.kestra.plugin.ee.apps.core.blocks.Markdown
content: |
## Say hello
Enter a name and submit the form.
- type: io.kestra.plugin.ee.apps.execution.blocks.CreateExecutionForm
- type: io.kestra.plugin.ee.apps.execution.blocks.CreateExecutionButton
text: Submit
- on: SUCCESS
blocks:
- type: io.kestra.plugin.ee.apps.core.blocks.Alert
style: SUCCESS
showIcon: true
content: Your request completed successfully.
- type: io.kestra.plugin.ee.apps.execution.blocks.Logs
```
You can find a related example in the [enterprise-edition-examples repository](https://github.com/kestra-io/enterprise-edition-examples/blob/main/apps/06_hello_world_app.yaml).
This app is `PUBLIC`, so anyone with the URL can access it without logging in. Set `access.type` to `PRIVATE` to restrict it to authenticated users.
### App to request and download data
A common pattern is to let business users request a filtered dataset and download it as a file — no Kestra access required. The flow fetches data from an external API, filters and sorts it based on user inputs, writes a CSV, and surfaces a summary string and the file as typed flow outputs. The app wraps that flow in a form, shows a loading indicator while it runs, and presents the summary and a download button on success.
**Flow:**
```yaml
id: product-report
namespace: company.ops
inputs:
- id: requested_by
type: STRING
displayName: Requested By
- id: category
type: SELECT
displayName: Product Category
values:
- smartphones
- laptops
- fragrances
- skincare
- groceries
- furniture
- id: min_rating
type: FLOAT
displayName: Minimum Rating
defaults: 4.0
- id: sort_by
type: SELECT
displayName: Sort By
values:
- rating
- price
- stock
tasks:
- id: fetch_products
type: io.kestra.plugin.core.http.Download
uri: "https://dummyjson.com/products/category/{{ inputs.category }}?limit=100"
- id: build_report
type: io.kestra.plugin.scripts.python.Script
containerImage: python:3.12-slim
inputFiles:
data.json: "{{ outputs.fetch_products.uri }}"
script: |
import csv
import json
with open("data.json") as f:
products = json.load(f)["products"]
filtered = [p for p in products if p["rating"] >= {{ inputs.min_rating }}]
filtered.sort(key=lambda p: p["{{ inputs.sort_by }}"], reverse=True)
fields = ["title", "brand", "category", "price", "rating", "stock", "discountPercentage"]
with open("report.csv", "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
writer.writeheader()
writer.writerows(filtered)
avg_price = sum(p["price"] for p in filtered) / len(filtered) if filtered else 0
avg_rating = sum(p["rating"] for p in filtered) / len(filtered) if filtered else 0
summary = (
f"Report for {{ inputs.requested_by }}: "
f"{len(filtered)} {{'{{ inputs.category }}'}} products rated ≥ {{ inputs.min_rating }}. "
f"Avg price ${avg_price:.2f} · Avg rating {avg_rating:.2f}/5 · "
f"Sorted by {{ inputs.sort_by }}."
)
print("::" + json.dumps({"outputs": {"summary": summary, "count": len(filtered)}}) + "::")
outputFiles:
- report.csv
outputs:
- id: report_summary
type: STRING
value: "{{ outputs.build_report.vars.summary }}"
- id: report_file
type: FILE
value: "{{ outputs.build_report.outputFiles['report.csv'] }}"
```
**App:**
```yaml
id: product_report
type: io.kestra.plugin.ee.apps.Execution
namespace: company.ops
displayName: Product Report
flowId: product-report
access:
type: PRIVATE
catalog: true
layout:
- on: OPEN
blocks:
- type: io.kestra.plugin.ee.apps.core.blocks.Markdown
content: "## Submit a request"
- type: io.kestra.plugin.ee.apps.execution.blocks.CreateExecutionForm
- type: io.kestra.plugin.ee.apps.execution.blocks.CreateExecutionButton
text: Submit
- on: RUNNING
blocks:
- type: io.kestra.plugin.ee.apps.core.blocks.Markdown
content: "## Keep this page open to see the results"
- type: io.kestra.plugin.ee.apps.core.blocks.Loading
- type: io.kestra.plugin.ee.apps.execution.blocks.Logs
- type: io.kestra.plugin.ee.apps.execution.blocks.CancelExecutionButton
text: Cancel request
style: WARNING
- on: SUCCESS
blocks:
- type: io.kestra.plugin.ee.apps.core.blocks.Alert
style: SUCCESS
showIcon: true
content: Your request has completed.
- type: io.kestra.plugin.ee.apps.execution.blocks.Outputs
- type: io.kestra.plugin.ee.apps.execution.blocks.Logs
- type: io.kestra.plugin.ee.apps.core.blocks.Button
text: Submit a new request
url: "{{ app.url }}"
style: DEFAULT
- on: FAILURE
blocks:
- type: io.kestra.plugin.ee.apps.core.blocks.Alert
style: ERROR
showIcon: true
content: The request failed. The logs below show what went wrong.
- type: io.kestra.plugin.ee.apps.execution.blocks.Logs
filter:
logLevel: ERROR
```
#### Interactive demo
This pattern works for any reporting or analytics use case where users need to request and download data without accessing Kestra directly.
### App to request a VM and get it approved
Self-service infrastructure requests are one of the most common approval-based App patterns. A requester fills out a form with the VM size, environment, region, and business justification. The flow logs and validates the request, then pauses for review. A platform engineer opens the same app in its `PAUSE` state, sees the original request details, adds reviewer notes, and either approves (resumes the execution) or rejects (cancels it). On approval, the flow continues to provisioning.
This example demonstrates two distinct user roles interacting with the same app at different stages.
**Flow:**
```yaml
id: vm-request
namespace: company.ops
inputs:
- id: requested_by
type: STRING
displayName: Requested By
- id: vm_size
type: SELECT
displayName: VM Size
values:
- "small (2 vCPU / 4 GB)"
- "medium (4 vCPU / 8 GB)"
- "large (8 vCPU / 16 GB)"
- id: environment
type: SELECT
displayName: Environment
values:
- dev
- staging
- production
- id: region
type: SELECT
displayName: Region
values:
- us-east-1
- eu-west-1
- ap-southeast-1
- id: justification
type: STRING
displayName: Business Justification
tasks:
- id: validate_request
type: io.kestra.plugin.core.log.Log
message: |
New VM request from {{ inputs.requested_by }}.
Size: {{ inputs.vm_size }}, Environment: {{ inputs.environment }}, Region: {{ inputs.region }}.
Justification: {{ inputs.justification }}
- id: await_approval
type: io.kestra.plugin.core.flow.Pause
onResume:
- id: notes
type: STRING
displayName: Reviewer Notes
required: false
- id: provision_vm
type: io.kestra.plugin.core.log.Log
message: |
Provisioning {{ inputs.vm_size }} in {{ inputs.environment }}/{{ inputs.region }} for {{ inputs.requested_by }}.
Reviewer notes: {{ resume.notes }}
```
The `Pause` task's `onResume` inputs define what the reviewer submits when approving — in this case, optional notes. Those values are available to downstream tasks as `{{ resume. }}`.
**App:**
```yaml
id: vm_request
type: io.kestra.plugin.ee.apps.Execution
displayName: VM Request
namespace: company.ops
flowId: vm-request
access:
type: PRIVATE
catalog: true
layout:
- on: OPEN
blocks:
- type: io.kestra.plugin.ee.apps.core.blocks.Markdown
content: |
## Request a Virtual Machine
Fill in the details below. Your request will be validated and sent for approval before provisioning begins.
- type: io.kestra.plugin.ee.apps.execution.blocks.CreateExecutionForm
- type: io.kestra.plugin.ee.apps.execution.blocks.CreateExecutionButton
text: Submit Request
style: SUCCESS
- on: RUNNING
blocks:
- type: io.kestra.plugin.ee.apps.core.blocks.Markdown
content: "## Validating your request…"
- type: io.kestra.plugin.ee.apps.core.blocks.Loading
- type: io.kestra.plugin.ee.apps.execution.blocks.Logs
- on: PAUSE
blocks:
- type: io.kestra.plugin.ee.apps.core.blocks.Markdown
content: |
## Review this VM request
Inspect the request details below, then approve or reject.
- type: io.kestra.plugin.ee.apps.execution.blocks.Inputs
- type: io.kestra.plugin.ee.apps.execution.blocks.ResumeExecutionForm
- type: io.kestra.plugin.ee.apps.execution.blocks.ResumeExecutionButton
text: Approve
style: SUCCESS
- type: io.kestra.plugin.ee.apps.execution.blocks.CancelExecutionButton
text: Reject
style: DANGER
- on: SUCCESS
blocks:
- type: io.kestra.plugin.ee.apps.core.blocks.Alert
style: SUCCESS
showIcon: true
content: Your VM has been provisioned.
- type: io.kestra.plugin.ee.apps.execution.blocks.Logs
- type: io.kestra.plugin.ee.apps.core.blocks.Button
text: Submit a new request
url: "{{ app.url }}"
style: DEFAULT
- on: FAILURE
blocks:
- type: io.kestra.plugin.ee.apps.core.blocks.Alert
style: ERROR
showIcon: true
content: The request could not be processed. Review the logs for details.
- type: io.kestra.plugin.ee.apps.execution.blocks.Logs
filter:
logLevel: ERROR
```
The `PAUSE` stage shows the original request inputs (`ExecutionInputs`), the `onResume` form for reviewer notes (`ResumeExecutionForm`), and the approve/reject buttons. Clicking **Approve** resumes the execution; clicking **Reject** cancels it.
This pattern applies directly to database access requests, sandbox environment creation, firewall rule approvals, SaaS account provisioning, and any other workflow that needs a human gate before it continues.
---
## Creating Apps without code
Apps can also be built using the no-code editor. Every block, property, and configuration option available in YAML is fully supported visually. Changes made in the no-code editor are immediately reflected in the code view, and vice versa — teams can switch between approaches at any time without losing the underlying YAML definition.
---
## App catalog
The App Catalog lists all available apps in a tenant. Filter by name, type, namespace, or tags. From this page you can also create new apps, edit existing ones, enable or disable individual apps, or delete them.

Kestra provides a direct access URL to the Apps Catalog in the format `http://your_host/ui/your_tenant/apps/catalog`. Any Kestra user with at least `APP: VIEW` permission in that tenant can reach this URL.
The catalog page requires authentication, so it is never publicly accessible. Users see only the apps they are permitted to see based on their RBAC permissions. You can limit visibility to specific groups by setting the `groups` property in the `access` block:
```yaml
access:
catalog: true
type: PRIVATE
groups:
- Admins
```
### Hiding an app from the catalog
Setting `catalog: false` removes the app from the browseable catalog while keeping its direct URL fully functional. Use this when you want to share an app with a specific audience via URL without surfacing it to everyone who can browse the catalog.
```yaml
access:
catalog: false
type: PRIVATE
```
### Managing apps in bulk
From the Apps Catalog, you can select multiple apps and enable, disable, or delete them in a single operation. Bulk operations report partial failures individually so you can see which apps were affected and which were not.
You can also export a selection of apps as a ZIP archive (`kestra-{tenant}-apps.zip`) and import that archive — or a multi-document YAML file — into another tenant or environment. The export produces one `{namespace}-{id}.yaml` file per app. On import, each app is validated independently; errors are reported per file so a single bad app does not block the rest.
### Customize the Apps Catalog
Customize the catalog's title, colors, and banner image to match your organization's look and feel. Go to **Instance Owner**, then in the left sidebar under your tenant, click **Apps Catalog**.

The configuration panel lets you set a **Title**, **Title Color**, **Primary Color** (used for buttons), **Background Color**, **Tile Color**, **Button Text Color**, and a **Banner** image displayed at the top of the catalog page.
:::alert{type="info"}
The banner image must be an `.svg` file.
:::
Once saved, open the **Apps Catalog** to see your branding:

Access these settings any time from the **gear icon** in the catalog.
### App tags
Add custom tags to organize and filter apps in the catalog. For example: `DevOps`, `data-team`, `project-x`.
---
## App URL
Each app has a unique URL — share it with team members, customers, or partners, or embed it in internal tools. Anyone with the link (and appropriate access) can open the app directly.
The URL format is: `https://yourHost/ui/tenantId/apps/appUid`, for example `http://localhost:8080/ui/release/apps/5CS8qsm7YTif4PWuAUWHQ5`.
Copy the URL from the **Apps Catalog**.
:::alert{type="info"}
App URL generation relies on the `kestra.url` server configuration property. If this property is not set, generated links may be broken or missing. Set it to the externally reachable base URL of your Kestra instance, for example `kestra.url: https://kestra.example.com`.
:::
### App expressions
From within flows, you can generate app URLs using the Enterprise-only `appLink` expression. See [Workflow Functions](../../../expressions/04.functions/04.workflow/index.mdx) for parameters and examples.
---
## App access and RBAC permissions
Each app has an `access` block that controls who can open and submit it.
### Public access
When an app is set to `PUBLIC`, anyone with the URL can open the form and submit requests without logging in. This is suitable for public-facing forms, surveys, or intake pages you share via email or embed on a website.
:::alert{type="info"}
For `PUBLIC` apps, execution IDs exposed through file download or log links are encrypted so that anonymous users cannot reference executions outside the app.
:::
### Private access for using apps
When an app is set to `PRIVATE`, only users with `APP: EXECUTE` permission on the app's namespace can open or submit it. `PRIVATE` apps are also only visible in the App Catalog to users with `APP: EXECUTE` — they do not appear for users who only hold `APP: LIST` or `APP: VIEW`. You can further narrow access to specific IAM groups using the `groups` field:
```yaml
access:
type: PRIVATE
groups:
- DataOps
- Finance
```
Group membership is checked at runtime on every request. Users who belong to at least one listed group are granted access; users outside those groups are denied even if they have `APP: EXECUTE` permission on the namespace. If `groups` is omitted, any authenticated user with `APP: EXECUTE` permission on the namespace can use the app.
`APP: EXECUTE` is namespace-scoped. A user with `APP: EXECUTE` on `company.team` cannot dispatch an app in `company.other`, even if both apps appear in the same catalog view.
This makes the `PRIVATE` + `groups` combination useful when you want to allow a specific group of business stakeholders or external partners to use an app without giving them access to the broader Kestra UI.
### Private access for building apps
The `APP` resource controls who can create, view, update, or delete apps within a tenant. It can be scoped to specific namespaces. `APP: EXECUTE`, `APP: ACCESS_FILES`, and `APP: ACCESS_LOGS` govern the ability to submit requests through an app and access its artifacts; the remaining `APP` actions govern the ability to build and manage apps.
---
## App executions
Each time a user creates an execution by submitting a form in the app, a new execution is generated with the system label `system.app` and a value of `yourAppId`. For example, to filter all executions created by the `computeResourcesForm` app, you can search for `system.app:computeResourcesForm` in the label filter.
For every execution, you can track the user inputs, see the current state, view logs, and check the outputs — all from the Kestra UI. This lets you observe, troubleshoot, and manage app executions just as you would any other workflow execution.
---
## App expiration
You can limit an app to a specific time window using the `expiration` property. Once the window closes, the app is filtered out of the catalog and blocks new submissions — existing executions are unaffected.
```yaml
id: survey_form
type: io.kestra.plugin.ee.apps.Execution
displayName: Q2 Survey
namespace: company.team
flowId: survey_processor
access:
type: PUBLIC
expiration:
startDate: "2025-06-01T00:00:00Z"
endDate: "2025-06-30T23:59:59Z"
layout:
- on: OPEN
blocks:
- type: io.kestra.plugin.ee.apps.core.blocks.Markdown
content: "## Please complete the survey before the end of June."
- type: io.kestra.plugin.ee.apps.execution.blocks.CreateExecutionForm
- type: io.kestra.plugin.ee.apps.execution.blocks.CreateExecutionButton
text: Submit
```
Both fields are optional:
- Omit `startDate` and the app is available immediately.
- Omit `endDate` and the app never expires.
- Omit `expiration` entirely and the app stays active indefinitely.
Expiration is evaluated against the server clock at the moment a user opens or submits the app.
---
## App thumbnails
Add a thumbnail to give catalog users a visual indicator of what the app does. Upload an image file as a [namespace file](../../../06.concepts/02.namespace-files/index.md) to the same namespace as the app's connected flow. Supported formats include `.svg`, `.jpg`, and `.png`. The example below adds `kestra-icon.svg`.

In your app code, add the `thumbnail` string property and point it toward the correct namespace file using `nsfiles:///`. For example:
```yaml
id: request_data_form
type: io.kestra.plugin.ee.apps.Execution
displayName: Form to request and download data
namespace: company.team
flowId: get_data
thumbnail: "nsfiles:///kestra-icon.svg" # Point this property to the correct namespace file.
access:
type: PRIVATE
tags:
- Reporting
- Analytics
```
Once added, open the **Apps Catalog** — a new thumbnail displays on the connected app:

---
## App layout blocks
Each app layout is a list of stage-specific blocks. Each block renders when the execution reaches the matching stage (`OPEN`, `RUNNING`, `SUCCESS`, `FAILURE`, `PAUSE`, `RESUME`). Combine blocks to guide users through the full lifecycle of a request — form on `OPEN`, progress indicator on `RUNNING`, download link and logs on `SUCCESS`.
| Block type | Available on | Properties | Example |
|--------------------------|--------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `Markdown` | OPEN, CREATED, RUNNING, PAUSE, RESUME, SUCCESS, FAILURE, FALLBACK | - `content` (Pebble template) | `- type: io.kestra.plugin.ee.apps.core.blocks.Markdown` `content: "## Please validate the request. Inspect the logs and outputs below. Then, approve or reject the request."` |
| `RedirectTo` | OPEN, CREATED, RUNNING, PAUSE, RESUME, SUCCESS, FAILURE, ERROR, FALLBACK | - `url`: redirect URL - `delay`: delay in seconds | `- type: io.kestra.plugin.ee.apps.core.blocks.RedirectTo` `url: "https://kestra.io/docs"` `delay: "PT60S"` |
| `CreateExecutionForm` | OPEN | None | `- type: io.kestra.plugin.ee.apps.execution.blocks.CreateExecutionForm` |
| `ResumeExecutionForm` | PAUSE | None | `- type: io.kestra.plugin.ee.apps.execution.blocks.ResumeExecutionForm` |
| `CreateExecutionButton` | OPEN | - `text` - `style`: DEFAULT, SUCCESS, DANGER, INFO - `size`: SMALL, MEDIUM, LARGE | `- type: io.kestra.plugin.ee.apps.execution.blocks.CreateExecutionButton` `text: "Submit"` `style: "SUCCESS"` `size: "MEDIUM"` |
| `CancelExecutionButton` | CREATED, RUNNING, PAUSE | - `text` - `style`: DEFAULT, SUCCESS, DANGER, INFO - `size`: SMALL, MEDIUM, LARGE | `- type: io.kestra.plugin.ee.apps.execution.blocks.CancelExecutionButton` `text: "Reject"` `style: "DANGER"` `size: "SMALL"` |
| `ResumeExecutionButton` | PAUSE | - `text` - `style`: DEFAULT, SUCCESS, DANGER, INFO - `size`: SMALL, MEDIUM, LARGE | `- type: io.kestra.plugin.ee.apps.execution.blocks.ResumeExecutionButton` `text: "Approve"` `style: "SUCCESS"` `size: "LARGE"` |
| `ExecutionInputs` | PAUSE, RESUME, SUCCESS, FAILURE | - `filter`: include, exclude | `- type: io.kestra.plugin.ee.apps.execution.blocks.Inputs` `filter:` `include: []` `exclude: []` |
| `ExecutionOutputs` | PAUSE, RESUME, SUCCESS, FAILURE | - `filter`: include, exclude | `- type: io.kestra.plugin.ee.apps.execution.blocks.Outputs` `filter:` `include: []` `exclude: []` |
| `ExecutionLogs` | PAUSE, RESUME, SUCCESS, FAILURE, FALLBACK | - `filter`: logLevel, taskIds | `- type: io.kestra.plugin.ee.apps.execution.blocks.Logs` `filter:` `logLevel: "INFO"` `taskIds: []` |
| `Loading` | RUNNING | None | `- type: io.kestra.plugin.ee.apps.core.blocks.Loading` |
| `Alert` | OPEN, CREATED, RUNNING, PAUSE, RESUME, SUCCESS, FAILURE, FALLBACK | - `style`: SUCCESS, WARNING, ERROR, INFO - `showIcon`: true, false | `- type: io.kestra.plugin.ee.apps.core.blocks.Alert` `style: "WARNING"` `showIcon: true` `content: "An error occurred!"` |
| `Button` | SUCCESS, FAILURE | - `text` - `url` - `style`: DEFAULT, SUCCESS, DANGER, INFO | `- type: io.kestra.plugin.ee.apps.core.blocks.Button` `text: "More examples"` `url: "https://github.com/kestra-io/examples"` `style: "INFO"` |
| `TaskOutputs` | RUNNING, PAUSE, RESUME, SUCCESS | - `outputs`: list of outputs with `displayName`, `value`, and `type` | `- type: io.kestra.plugin.ee.apps.execution.blocks.TaskOutputs` `outputs:` `- displayName: My Task Output` `value: "{{ outputs.test.value }}"` `type: FILE` |
### Dynamic content in the OPEN state
The `Markdown` block's `content` property is a Pebble template. When the layout includes a `CreateExecutionForm`, `{{ inputs. }}` is available in the render context and updates live as the user interacts with the form — useful for context-sensitive descriptions that react to dropdown changes.
For example, given a flow with a `product` SELECT input, the app below renders the current selection inline:
```yaml
id: software_request_form
type: io.kestra.plugin.ee.apps.Execution
displayName: Software Request Form
namespace: company.team
flowId: software_request
access:
type: PRIVATE
layout:
- on: OPEN
blocks:
- type: io.kestra.plugin.ee.apps.execution.blocks.CreateExecutionForm
- type: io.kestra.plugin.ee.apps.core.blocks.Markdown
content: "You are requesting access to: **{{ inputs.product }}**"
- type: io.kestra.plugin.ee.apps.execution.blocks.CreateExecutionButton
text: Submit
- on: SUCCESS
blocks:
- type: io.kestra.plugin.ee.apps.core.blocks.Alert
style: SUCCESS
content: Your request has been submitted.
```
Before the user interacts with the form, `{{ inputs.* }}` resolves to each input's default value. In other states (RUNNING, SUCCESS, FAILURE), `{{ inputs.* }}` references the execution's submitted values.
:::alert{type="info"}
When the flow uses [`FORM` inputs](../../../05.workflow-components/05.inputs/index.md#form-inputs), `CreateExecutionForm` renders a multi-step Next/Back wizard — one step per FORM group, a step for ungrouped inputs, then a recap. No additional App configuration is required; the wizard is driven entirely by the flow's input definition.
:::
### File preview and download
The `Outputs` and `TaskOutputs` blocks can render file download links for outputs stored in Kestra's internal storage. File preview, metadata, and download are only available when:
- The app type is `io.kestra.plugin.ee.apps.Execution`.
- The layout includes an `Outputs` or `TaskOutputs` block.
- The storage path belongs to an execution that the app has access to.
By default, file preview shows the first 100 rows. You can change this server-side with `kestra.server.preview.initial-rows` (default `100`) and cap it with `kestra.server.preview.max-rows` (default `5000`).
### Log download
The `ExecutionLogs` block renders an inline log viewer. When a `Logs` block is present in the layout, users can also download the full log file directly from the app. Log download is only available for `Execution`-type apps that include a `Logs` block in their layout.
---
# Worker Groups in Kestra Enterprise: Target Workers
URL: https://kestra.io/docs/enterprise/scalability/worker-group
> Manage workloads with Kestra Worker Groups. Target specific workers for tasks based on hardware, region, or security requirements for optimized execution.
Worker Groups route tasks to the right machines in your fleet. A Worker Group is a named, token-authenticated pool of workers that subscribes to one or more Worker Queues — tag-based routing lanes that tasks declare requirements against. The result is flexible many-to-many routing: GPU machines and spot instances can serve the same queue, or a single group can cover multiple queues with per-queue capacity guarantees.
Worker Groups are an Enterprise Edition feature. In the open-source edition, all work runs in a single implicit default pool.
## Getting started
To set up Worker Groups end-to-end:
1. [Create a Worker Queue](#worker-queues) — define a routing lane with tags
2. [Create a Worker Group](#creating-and-managing-worker-groups) — create a pool and subscribe it to queues
3. [Generate a registration token](#generating-a-registration-token) — authenticate workers to the group
4. [Start a worker](#starting-a-worker-for-a-group) — connect with the token and controller endpoint
5. [Route tasks](#using-workerselector-in-tasks) — add `workerSelector` to any task
For IaC and Helm deployments, see [Declarative configuration](#declarative-configuration) to provision the full topology at startup without runtime API calls.
## How Worker Groups work
Three building blocks define the routing model:
| Building block | Role |
|---|---|
| **Worker** | A process that runs tasks; joins a group by presenting a registration token and locally enforces its capacity allocation |
| **Worker Group** | A named pool of workers that subscribes to queues and holds capacity reservation settings |
| **Worker Queue** | A routing lane identified by a tag set; tasks declare `workerSelector.tags` to target a queue |
The routing path flows from task requirements down to infrastructure:
1. A task declares `workerSelector.tags: [gpu, eu]`
2. Kestra finds the Worker Queue whose tags match
3. Kestra checks which Worker Groups subscribe to that queue
4. A worker from one of those groups picks up the task
**Developer perspective**: declare what a task needs using tags. No machine names, no group names.
**Operator perspective**: create queues with meaningful tags, subscribe groups to those queues, and set capacity guarantees per subscription.
## Using workerSelector in tasks
Add `workerSelector` to any task to route it to a matching Worker Queue. The `workerSelector` object has three properties:
| Property | Description | Default |
|---|---|---|
| `tags` | List of RFC 1123 labels (max 20) identifying the required Worker Queue | — |
| `match` | `ALL`: queue tags must include all selector tags. `ANY`: queue tags must include at least one selector tag | `ALL` |
| `fallback` | Behavior when no worker is available for the matched queue: `FAIL`, `WAIT`, `CANCEL`, or `IGNORE` | `FAIL` |
:::alert{type="warning"}
The default `fallback` in 2.0 is `FAIL`. If you upgraded from an earlier version where tasks waited by default, set `fallback: WAIT` directly on each task, or use a [Policy](../../02.governance/policies/index.md) with an `Add` rule to apply it across a namespace.
:::
```yaml
id: process_sensitive_data
namespace: company.team
tasks:
- id: process
type: io.kestra.plugin.scripts.python.Commands
workerSelector:
tags: [sensitive, eu]
fallback: WAIT
commands:
- python process.py
```
Kestra routes the task to a Worker Queue whose tag set includes all declared tags (or any, when `match: ANY`). Any Worker Group subscribed to that queue may execute the task.
If `workerSelector` is absent or all tags resolve to null, the task routes to the default queue.
### Fallback options
| Value | Behavior |
|---|---|
| `FAIL` | Fail the task run immediately if no worker is available (default) |
| `WAIT` | Hold the task in `CREATED` state until a worker becomes available |
| `CANCEL` | Cancel the task gracefully; the execution is marked `KILLED` |
| `IGNORE` | Drop the tag requirement and route to the default Worker Queue instead |
`IGNORE` is useful when the target infrastructure is optional — the task proceeds on any available worker rather than failing when the specialized pool is unavailable.
`fallback` can only be set when `tags` is non-empty.
:::alert{type="warning"}
`fallback` only applies when a matching Worker Queue exists but has no live worker right now. If no queue matches the tags at all, the task fails immediately regardless of the `fallback` value — that is a configuration error, not a capacity gap.
:::
### Dynamic routing
Use Pebble expressions to set tags at runtime:
```yaml
inputs:
- id: region
type: STRING
defaults: eu
tasks:
- id: process
type: io.kestra.plugin.scripts.python.Commands
workerSelector:
tags:
- "{{ inputs.region }}"
- sensitive
fallback: WAIT
commands:
- python process.py
```
When an expression resolves to null or a blank string, that tag is omitted from the selector. If all tags resolve to null, the task routes to the default queue.
### Namespace and tenant-level routing defaults
Instead of adding `workerSelector` to every task, set a default selector at the namespace or tenant level. Kestra resolves the selector most-specific-first — task, then flow, then the nearest namespace ancestor, then the tenant — and stops at the first level that declares one.
Set a namespace-level default in the namespace settings:
```yaml
workerSelector:
tags: [eu]
fallback: WAIT
```
Every task in that namespace (and its child namespaces, unless overridden closer) inherits this selector automatically. Any selector closer to the task — on the task itself or the flow — wins over the namespace or tenant default.
This is the recommended approach when an entire namespace or team should always run on a specific fleet — it keeps flow YAML clean and makes routing changes a single admin update rather than a find-and-replace across all flows.
### Applying workerSelector with Policies
Use a [Policy](../../02.governance/policies/index.md) to route all tasks of a given plugin type to a specific Worker Queue without modifying each task individually:
```yaml
id: gpu-worker-routing
description: "Route all Python tasks to GPU workers."
enforcement: ACTIVE
rules:
- type: io.kestra.plugin.ee.rules.Add
on: PLUGIN
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.scripts.python
values:
workerSelector:
tags: [gpu]
fallback: WAIT
```
With this Policy applied to the namespace, flows need no per-task configuration:
```yaml
id: ml_pipeline
namespace: company.team
tasks:
- id: train
type: io.kestra.plugin.scripts.python.Commands
commands:
- python train.py
- id: evaluate
type: io.kestra.plugin.scripts.python.Commands
commands:
- python eval.py
```
## Worker Queues
A Worker Queue is a routing lane with a stable id and a set of tags. Multiple Worker Groups may subscribe to the same queue. Removing a group's subscription never deletes the queue — queues exist independently.
Two ids are reserved and never created manually:
- `default` — the global default queue; receives tasks with no `workerSelector`
- `system` — the in-process system worker
Worker Queue ids must follow RFC 1123 label format: lowercase alphanumerics and hyphens, starting and ending with an alphanumeric character, max 63 characters.
### Creating Worker Queues
Navigate to **Instance Owner → Infrastructure → Worker Queues** and click **Create**. You can also create Worker Queues via the API or Terraform.
**Tenant scoping**: a Worker Queue can restrict which tenants may route tasks through it. An empty tenant list means unrestricted.
## Creating and managing Worker Groups
A Worker Group is identified by a stable id (RFC 1123 label), has a display name, and holds a list of queue subscriptions and registration tokens.
### Creating a Worker Group
Navigate to **Instance Owner → Infrastructure → Worker Groups** and click **Add Worker Group**. Set an id, display name, and optional description. You can add queue subscriptions and generate registration tokens immediately, or configure them after creation.
Worker Group ids must follow RFC 1123 label format.
### The default group
One group always exists and cannot be deleted: the `default` group. It subscribes to the `default` queue and receives all tasks that have no `workerSelector`. Workers that start without a registration token join the default group automatically.
:::alert{type="warning"}
Keep at least one worker running in the default group to ensure tasks without a `workerSelector` always have somewhere to execute.
:::
### Queue subscriptions
A subscription connects a Worker Group to a Worker Queue. Each subscription specifies:
- **Target queue id** — which Worker Queue this group's workers will serve
- **Reserved capacity percentage** (optional) — a per-worker floor guarantee, 1–100
- **Interaction mode** — `STRICT` or `ELASTIC` (see [Capacity reservation](#capacity-reservation))
A group may subscribe to multiple queues. The sum of reserved percentages across a worker's subscriptions must not exceed 100.
Manage subscriptions through the UI or the subscriptions API:
| Operation | Endpoint |
|---|---|
| Add subscription | `POST /api/v1/instance/worker-groups/{id}/subscriptions` |
| Update reservation | `PATCH /api/v1/instance/worker-groups/{id}/subscriptions/{workerQueueId}` |
| Remove subscription | `DELETE /api/v1/instance/worker-groups/{id}/subscriptions/{workerQueueId}` |
## Capacity reservation
Reserved capacity is a per-worker floor guarantee, not a fleet-wide quota. Remaining slots beyond reserved percentages form a shared pool available to all of that worker's subscriptions.
**Example**: a worker with 16 slots subscribing to two queues at 50% and 25% reserves 8 slots for queue A and 4 slots for queue B, with 4 slots in the shared pool.
### Interaction modes
- **STRICT** — idle reserved slots remain exclusive to this subscription and are never lent to other work
- **ELASTIC** — idle reserved slots may be lent to other `ELASTIC` subscriptions on the same worker when the subscription has spare capacity
In both modes, tasks also draw from the shared pool once reserved slots are busy. Lent slots are not preempted — a busy lender may temporarily dip below its floor until borrowed work completes.
Capacity reservations are live-configurable: updating a subscription's reserved percentage takes effect within seconds with no worker restarts required.
## Worker authentication
:::alert{type="info"}
Worker authentication must be enabled server-side before registration tokens have any effect. A worker configured with a token but connecting to an instance where auth is disabled will join the default group instead.
:::
Workers join a group by presenting a registration token generated for that group. The token is stored as a hash and shown only once at creation — copy it immediately.
On first connect, the worker exchanges the registration token for a short-lived access token and a rotating refresh token. The access token is refreshed automatically before it expires. Revoking or deleting a token immediately invalidates credentials for any workers that registered with it; those workers fail closed once their current access token expires.
### Generating a registration token
In the Worker Groups UI, select a group and generate a token from the **Tokens** tab. Alternatively, use the API:
| Operation | Endpoint |
|---|---|
| Generate token | `POST /api/v1/instance/worker-groups/{id}/tokens` |
| Revoke token | `POST /api/v1/instance/worker-groups/{id}/tokens/{tokenId}/revoke` |
| Delete token | `DELETE /api/v1/instance/worker-groups/{id}/tokens/{tokenId}` |
### Server-side configuration
Enable worker authentication on your webserver or standalone Kestra instance:
```yaml
kestra:
ee:
worker:
auth:
enabled: true
jwt-signing-key: "{{ a strong shared secret, >= 32 bytes }}"
access-token-lifetime: PT5M # optional, default PT5M
refresh-token-lifetime: P7D # optional, default P7D
```
### Worker-side configuration
Each worker needs two things to join a group: a registration token that identifies the group, and a controller endpoint that tells the worker where to connect. Both are required — a worker started with only the token will try `localhost` and fail.
```yaml
kestra:
worker:
name: gpu-pool-1 # optional display name
auth:
registration-token: "{{ token generated for the target group }}"
credentials-path: /var/kestra/worker/.auth/worker-credentials.json # default: /tmp/kestra/worker/.auth/...
refresh-buffer: PT60S # how early to refresh the access token before it expires
controllers:
type: STATIC
static:
endpoints:
- host: kestra-controller.internal
port: 50051
```
:::alert{type="warning"}
The endpoint `host` and `port` must be separate YAML keys. A single `host:port` string fails with `Static configuration requires at least one endpoint`.
:::
#### Controller discovery strategies
`type: STATIC` is the default and suitable for most bare-metal and Docker deployments. Two other strategies are available:
| Type | When to use |
|---|---|
| `STATIC` | Fixed controller addresses — explicit `host`/`port` list |
| `DNS` | Kubernetes or any environment where controllers are reachable by a stable DNS name; resolves SRV or A records on an interval |
| `STORAGE` | Dynamic, cross-cloud deployments; controllers self-register in internal storage and workers list the registry |
For Kubernetes, use `type: DNS` with a service hostname:
```yaml
kestra:
worker:
controllers:
type: DNS
dns:
hostname: kestra-controller.kestra.svc.cluster.local
record-type: SRV # or A if no SRV records
default-port: 50051 # used with A records only
refresh-interval: PT30S
```
For Helm deployments, controller discovery is preconfigured — see the [Helm gRPC and Worker-Controller migration guide](../../../11.migration-guide/v2.0.0/helm-grpc-worker-controller/index.md). For bare-metal or Docker with components on separate hosts, see [running Kestra with separated server components](../../../kestra-cli/kestra-server/index.md#kestra-with-server-components-in-different-services).
### Starting a worker for a group
With both the registration token and controller endpoint configured, start the worker normally:
```bash
kestra server worker
```
No additional CLI flags are needed. The registration token in `kestra.worker.auth.registration-token` identifies which group the worker joins at connection time.
:::alert{type="info"}
Workers do not connect to the database. Any `datasources` or `kestra.repository.type` keys in the config are ignored; a startup warning lists which keys were skipped.
:::
## Declarative configuration
You can declare the entire worker topology — queues, groups, subscriptions, and registration tokens — in `application.yml` under `kestra.ee.setup`. Kestra applies this configuration at startup, which enables a fully automated single-pass deployment with no runtime API calls.
```yaml
kestra:
ee:
setup:
enabled: true
worker-queues:
- id: gpu
tags: [gpu, linux]
allowed-tenants: [acme] # optional; empty = unrestricted
- id: etl
tags: [etl]
worker-groups:
- id: gpu-workers
name: GPU workers
registration-tokens:
- name: bootstrap
token-file: /var/run/secrets/kestra/gpu-workers-token
subscriptions:
- worker-queue-id: gpu
reserved-percent: 70
- worker-queue-id: etl
```
Workers already retry registration until their token is known to the controller, so all services can start concurrently. Workers converge as soon as the webserver has applied the configuration.
### Secret handling
Registration tokens must not appear as plaintext in a committed configuration file. Two options are available per token entry:
- **`token-file`** — path to a file containing the pre-generated token. Preferred in Kubernetes environments where Secrets mount as files. The file must exist and be non-empty at startup.
- **`token: "${ENV_VAR}"`** — environment variable placeholder resolved at startup. Simpler outside Kubernetes, but environment variables are readable from `/proc//environ` and may appear in crash dumps.
Exactly one of the two is required. Use `kestra workers registration-tokens generate` to mint a token offline before deployment.
### The default group
The default group can be declared under its reserved id `default`:
```yaml
worker-groups:
- id: default
name: Shared workers
registration-tokens:
- name: bootstrap
token-file: /var/run/secrets/kestra/default-workers-token
```
The default group always subscribes to the default queue — Kestra adds that subscription automatically even when `subscriptions` is omitted or does not include the default queue. Any subscriptions you declare are added alongside it.
### Semantics
`kestra.ee.setup` is a seed, not a desired state:
- Each declared entity is created only when no entity with the same id already exists in the database.
- An existing entity is skipped as a whole — no subscriptions are changed, no tokens are added or revoked.
- Re-applying a changed configuration against an existing entity is a no-op. The database remains the source of truth once an entity exists; editing a live topology stays an API or UI operation.
A rogue instance cannot self-authorize by changing configuration: declarative setup can only add what is absent, never replace or revoke what the authenticated API created.
### Which server roles apply it
Only the webserver and standalone server roles apply `kestra.ee.setup` at startup. Worker processes never apply it — a worker must not be able to create the group or the token it authenticates against.
### Validation
The entire declaration is validated before anything is written. An invalid configuration fails startup with an actionable message identifying the offending path:
```
The subscription declared at 'kestra.ee.setup.worker-groups[0].subscriptions[1]'
references the unknown Worker Queue 'etl'. Declare it under
'kestra.ee.setup.worker-queues' or create it first.
```
Validation rejects: missing or duplicate ids, non-RFC-1123 ids, the reserved worker queue ids `default` and `system`, empty tag sets, tag collisions with existing queues, unknown `worker-queue-id` references, `reserved-percent` out of range or summing above 100, unreadable or empty token files, tokens already registered on another group, and malformed registration tokens.
Validation is all-or-nothing, but writes are not atomic. A failure mid-apply leaves already-created entities in place. The next startup resumes from where it stopped because existing entities are skipped.
### Observability
Kestra logs one line per created entity and one line per skipped entity. A summary line follows after the setup phase completes. Each created entity also produces a regular audit log entry. Token values are never logged.
## Transport security (TLS)
By default, gRPC traffic between workers and the controller is unencrypted. For production deployments, enable TLS on both sides.
### Server-side TLS
Add TLS config to the controller (or standalone) instance:
```yaml
kestra:
grpc:
tls:
enabled: true
key-store:
path: /etc/kestra/tls/controller-keystore.p12
password: "{{ secret('TLS_KEYSTORE_PASSWORD') }}"
# Required when client-auth is OPTIONAL or REQUIRE
trust-store:
path: /etc/kestra/tls/ca-truststore.p12
password: "{{ secret('TLS_TRUSTSTORE_PASSWORD') }}"
client-auth: NONE # NONE | OPTIONAL | REQUIRE (mTLS)
```
### Worker-side TLS
Add matching TLS config to each worker:
```yaml
kestra:
grpc:
tls:
enabled: true
# Required only for mTLS (client-auth: REQUIRE on the server)
key-store:
path: /etc/kestra/tls/worker-keystore.p12
password: "{{ secret('TLS_KEYSTORE_PASSWORD') }}"
# Optional — falls back to the system trust store
trust-store:
path: /etc/kestra/tls/ca-truststore.p12
password: "{{ secret('TLS_TRUSTSTORE_PASSWORD') }}"
```
:::alert{type="info"}
When using `type: STATIC` discovery with TLS, the synthetic gRPC authority derived from the endpoint list may not match the certificate's SANs. Add `authority-override` to the worker config to specify the hostname the certificate was issued for:
```yaml
kestra:
grpc:
tls:
enabled: true
authority-override: kestra-controller
```
Under `type: DNS` discovery, the authority is derived from the DNS hostname automatically and no override is needed.
:::
## Use cases
### Hardware affinity
Dedicate workers with GPUs, high-memory configurations, or OS-specific environments to tasks that need them. Developers declare the requirement via tags; operators manage the physical mapping independently.
```yaml
workerSelector:
tags: [gpu, cuda-12]
```
### Multi-tenant isolation
Give each tenant a dedicated Worker Queue with a reserved capacity percentage to prevent noisy-neighbor effects. An additional ELASTIC subscription to a shared burst queue lets idle capacity absorb traffic spikes while the per-tenant floor stays guaranteed.
### Regulated and air-gapped environments
Workers in restricted networks connect outbound-only, presenting a registration token to authenticate. No inbound firewall rules are required. Revoking a token immediately stops those workers from receiving new work, giving operators a fast, clean isolation path.
### Spiky workloads
Use a fixed worker pool with STRICT reservations to handle baseline load, and a spot pool with ELASTIC subscriptions that claims shared-pool capacity during spikes. The ELASTIC pool scales out and in without changing the baseline pool's guarantees.
### Priority lanes
Split capacity across multiple queues with reserved percentages to guarantee throughput for high-priority work:
```yaml
# Three priority queues — critical: 50%, standard: 25%, batch: 25%
workerSelector:
tags: [critical] # or [standard], or [batch]
```
Critical work always has guaranteed slots regardless of the volume of batch jobs in the queue.
### Day/night capacity shifting
Reserved percentages are live-configurable via the API. Changes propagate to all workers within seconds, with no restarts required. Shift capacity toward batch workloads during off-peak hours and back to interactive workloads during business hours without touching any worker process.
### Zero-downtime worker upgrades
Run two Worker Groups subscribed to the same queues simultaneously. Reduce the old group's reservation to 0% to drain it of new work, bring up the new group, verify it is healthy, then delete the old group. At no point does the queue go unserved.
For guidance on when to use Worker Groups versus Task Runners for compute-intensive scripting workloads, see [Task Runners vs Worker Groups](../../../task-runners/03.task-runners-vs-worker-groups/index.md).
## Worker shutdown and task continuity
When a worker process stops — whether from a deployment, a crash, or a manual restart — any tasks it was running may be interrupted. The `worker-task-restart-strategy` setting controls what happens to those tasks cluster-wide:
| Strategy | Behavior |
|---|---|
| `AFTER_TERMINATION_GRACE_PERIOD` | The worker stops accepting new work and waits up to the grace period for in-flight tasks to finish; any tasks still running at that point are re-dispatched to another worker (default) |
| `IMMEDIATELY` | Interrupted tasks are re-dispatched immediately to another worker without waiting |
| `NEVER` | Interrupted tasks fail permanently and are not re-dispatched |
Configure these in `application.yml` on each worker:
```yaml
kestra:
server:
termination-grace-period: 5m
worker-task-restart-strategy: AFTER_TERMINATION_GRACE_PERIOD
```
During the grace period, the worker stops accepting new jobs but lets running tasks finish. If the grace period elapses before all tasks complete, the worker force-terminates and the restart strategy decides the outcome for the remaining tasks.
:::alert{type="info"}
`AFTER_TERMINATION_GRACE_PERIOD` is the recommended setting for production deployments. It gives tasks time to finish cleanly while still guaranteeing that a stalled worker does not block the cluster indefinitely.
:::
## Monitoring
Metrics scoped to a group carry a `worker_group` tag; metrics scoped to a queue also carry a `worker_queue` tag. The configurable metrics prefix (default `kestra`) is prepended before export.
### Controller metrics
Published by the controller process — the server-side view of fleet capacity and dispatch activity:
| Metric | Type | Tags | Description |
|---|---|---|---|
| `controller.worker.active` | gauge | `worker_group`, `worker_queue` | Workers currently subscribed to a queue |
| `controller.worker.active.all` | gauge | — | Total workers connected to this controller |
| `controller.permits.available` | gauge | `worker_group`, `worker_queue` | Remaining advertised capacity across subscribed workers |
| `controller.permits.available.all` | gauge | — | Remaining capacity across all connected workers |
| `controller.job.inflight` | gauge | `worker_queue` | In-flight jobs for a queue |
| `controller.worker.group.job.inflight` | gauge | `worker_group` | In-flight jobs across workers in a group |
| `controller.capacity.subscription.allocated` | gauge | `worker_group`, `worker_queue` | Reserved slots allocated to a queue subscription |
| `controller.capacity.subscription.used` | gauge | `worker_group`, `worker_queue` | Reserved slots currently in use |
| `controller.capacity.shared.allocated` | gauge | `worker_group` | Shared (unreserved) slots allocated |
| `controller.capacity.shared.used` | gauge | `worker_group` | Shared slots currently in use |
| `controller.job.dispatched.total` | counter | `worker_queue` | Total jobs dispatched to workers |
| `controller.job.requeued.total` | counter | `worker_queue` | Jobs re-queued because no worker had capacity |
| `controller.job.killed.total` | counter | `worker_queue` | Jobs short-circuited by the pre-dispatch kill check |
| `controller.job.dispatch.failed.total` | counter | `worker_queue` | Dispatch attempts that failed on send |
| `controller.worker.registered.total` | counter | — | Worker-queue subscription registrations |
| `controller.worker.unregistered.total` | counter | — | Worker-queue subscription removals |
| `controller.subscription.paused.total` | counter | — | Queue subscription pause transitions |
| `controller.subscription.resumed.total` | counter | — | Queue subscription resume transitions |
### Worker metrics
Published by each worker process — the worker-side view of capacity and throughput:
| Metric | Type | Description |
|---|---|---|
| `worker.job.thread` | gauge | Configured thread count (maximum concurrent jobs) |
| `worker.max.concurrency` | gauge | Maximum in-flight capacity: threads + buffered jobs |
| `worker.running.count` | gauge | Tasks currently executing |
| `worker.pending.count` | gauge | Tasks waiting for a free thread slot |
| `worker.queue.size` | gauge | Items currently held in a buffer (job, result, log, or metric) |
| `worker.queue.remaining.capacity` | gauge | Free slots in the inbound job buffer — equals the worker's advertised permit count |
| `worker.queued.duration` | timer | Time a task spent waiting before a thread was available |
| `worker.started.count` | counter | Total tasks started |
| `worker.ended.count` | counter | Total tasks completed (any terminal state) |
| `worker.ended.duration` | timer | Task run duration as measured by the worker |
| `worker.timeout.count` | counter | Tasks that exceeded their configured timeout |
| `worker.killed.count` | counter | Kill events received from the controller |
| `worker.queue.enqueued` | counter | Total items put into a buffer |
| `worker.queue.dequeued` | counter | Total items drained from a buffer |
| `worker.trigger.running.count` | gauge | Trigger evaluations currently in progress |
| `worker.trigger.started.count` | counter | Total trigger evaluations started |
| `worker.trigger.ended.count` | counter | Total trigger evaluations completed |
| `worker.trigger.error.count` | counter | Trigger evaluations that failed |
| `worker.trigger.execution.count` | counter | Executions produced by triggers on this worker |
| `worker.trigger.duration` | timer | Trigger evaluation duration |
When `worker.running.count` consistently equals `worker.job.thread` and `worker.pending.count` is non-zero, that worker is fully saturated — scale by adding more workers to the group or increasing the thread count. When `worker.queue.remaining.capacity` on the `job` buffer approaches zero, the worker's local inbound buffer is full.
The live capacity snapshot is also available via the API:
```
GET /api/v1/instance/worker-groups/{id}/capacity
GET /api/v1/instance/worker-groups/{id}/workers
```
## Migrating from earlier versions
In Kestra 2.0, the task-level routing property changed from targeting a group by name to declaring requirements via tags:
| Before 2.0 | 2.0+ |
|---|---|
| `workerGroup.key: gpu` | `workerSelector.tags: [gpu]` |
| Routes directly to a named group | Routes to a Worker Queue by tags; any subscribed group may serve the task |
| `workerGroup.fallback` (defaults to `WAIT`) | `workerSelector.fallback` (defaults to `FAIL`) |
| No `match` strategy | `workerSelector.match: ALL` or `ANY` |
| No capacity control per queue | Reserved percentage per subscription, STRICT or ELASTIC mode |
| No worker authentication | Registration token-based authentication with rotating credentials |
`workerGroup` is not recognized in 2.0. Flows using it will fail validation and cannot be saved. Update your flows to replace `workerGroup.key` with `workerSelector.tags`. The group name in the old property corresponds to a tag on a Worker Queue in the new model.
:::alert{type="warning"}
The fallback default changed from `WAIT` to `FAIL`. Tasks that previously waited for an unavailable worker will now fail immediately unless you explicitly set `workerSelector.fallback: WAIT`.
:::
---
# Expressions in Kestra: Pebble Syntax and Variables
URL: https://kestra.io/docs/expressions
> Learn how to work with Kestra expressions using the execution context, Pebble syntax, filters, functions, and operators.
import ChildCard from "~/components/docs/ChildCard.astro"
Use expressions to dynamically set values in flows using `{{ ... }}` syntax backed by the Pebble templating engine.
## Common tasks
| If you need to... | Start here |
| --- | --- |
| Access `inputs`, `outputs`, `vars`, `trigger`, or `namespace` values | [Execution Context](./01.context/index.mdx) |
| Access secrets or credentials at runtime | [Data Access Functions](./04.functions/02.data-access/index.mdx) |
| Format dates, parse JSON, or transform strings | [Filter Reference](./03.filters/index.mdx) |
| Render nested expressions or inspect the full context | [Rendering Functions](./04.functions/01.rendering/index.mdx) |
| Write loops, conditions, fallbacks, and comparisons | [Pebble Syntax](./02.syntax/index.mdx) |
| Build or debug a multiline or nested expression | [Pebble Syntax](./02.syntax/index.mdx#multiline-json-bodies) and [render()](./04.functions/01.rendering/index.mdx#render) |
---
# Kestra Expression Context: Inputs, Outputs & Variables
URL: https://kestra.io/docs/expressions/context
> Reference for all variables available inside Kestra expressions at runtime — flow metadata, inputs, outputs, trigger values, secrets, and namespace variables.
Use this page to find out what data is available inside `{{ ... }}` at runtime — including flow metadata, inputs, outputs, trigger values, secrets, and namespace variables.
## Understand the execution context
Kestra expressions combine the [Pebble templating engine](/docs/concepts/pebble) with the execution context to dynamically render flow properties.
The execution context usually includes:
- `flow`
- `execution`
- `inputs`
- `outputs`
- `labels`
- `tasks`
- `trigger` when the flow was started by a trigger
- `vars` when the flow defines variables
- `namespace` in Enterprise Edition when namespace variables are configured
- `envs` for environment variables
- `globals` for global configuration values
- `item` inside a [Loop](../../05.workflow-components/01.tasks/00.flowable-tasks/index.md#loop) task iteration
:::alert{type="info"}
To inspect the full runtime context, use `{{ fetchContext() }}` in the Debug Expression console.
:::
The Debug Expression console is available in the Kestra UI under **Executions → Logs → Debug Expression**. Enter any expression and evaluate it against the live execution context without modifying the flow.
## Default execution context variables
| Parameter | Description |
| --- | --- |
| `{{ flow.id }}` | Identifier of the flow |
| `{{ flow.namespace }}` | Namespace of the flow |
| `{{ flow.tenantId }}` | Tenant identifier in Enterprise Edition |
| `{{ flow.revision }}` | Flow revision number |
| `{{ execution.id }}` | Unique execution identifier |
| `{{ execution.startDate }}` | Start date of the execution |
| `{{ execution.state }}` | Current execution state |
| `{{ execution.originalId }}` | Original execution ID preserved across replays |
| `{{ task.id }}` | Current task identifier |
| `{{ task.type }}` | Fully qualified class name of the current task |
| `{{ taskrun.id }}` | Current task run identifier |
| `{{ taskrun.startDate }}` | Start date of the current task run |
| `{{ taskrun.attemptsCount }}` | Retry and restart attempt count |
| `{{ taskrun.parentId }}` | Parent task run identifier for nested tasks |
| `{{ parent.outputs }}` | Outputs of the nearest parent task run |
| `{{ parents }}` | List of parent task runs |
| `{{ labels }}` | Execution labels accessible by key |
| `{{ trace.parent }}` | W3C `traceparent` header for the current execution; only populated when [OpenTelemetry tracing is enabled](../../10.administrator-guide/open-telemetry/index.md#traces) |
Example:
```yaml
id: expressions
namespace: company.team
tasks:
- id: debug_expressions
type: io.kestra.plugin.core.debug.Return
format: |
taskId: {{ task.id }}
date: {{ execution.startDate | date("yyyy-MM-dd HH:mm:ss.SSSSSS") }}
```
## Trigger variables
When the execution is started by a `Schedule` trigger:
| Parameter | Description |
| --- | --- |
| `{{ trigger.date }}` | Date of the current schedule |
| `{{ trigger.next }}` | Date of the next schedule |
| `{{ trigger.previous }}` | Date of the previous schedule |
When the execution is started by a `Flow` trigger:
| Parameter | Description |
| --- | --- |
| `{{ trigger.executionId }}` | ID of the triggering execution |
| `{{ trigger.namespace }}` | Namespace of the triggering flow |
| `{{ trigger.flowId }}` | ID of the triggering flow |
| `{{ trigger.flowRevision }}` | Revision of the triggering flow |
## Loop iteration context
Inside a [Loop](../../05.workflow-components/01.tasks/00.flowable-tasks/index.md#loop) task, each iteration runs as an isolated sub-execution. The `item` variable is available to all tasks within that sub-execution.
| Expression | Description |
|---|---|
| `{{ item.index }}` | Zero-based index of the current iteration |
| `{{ item.value }}` | Value of the current iteration |
| `{{ item.key }}` | Map key of the current iteration; only set when `values` is a map |
| `{{ item.parent.index }}` | Index of the nearest enclosing loop (nested loops only) |
| `{{ item.parent.value }}` | Value of the nearest enclosing loop (nested loops only) |
| `{{ item.parents[n].value }}` | Value of the nth ancestor loop, counting from innermost (`[0]` = immediate parent) |
Because `item` is bound to the loop execution rather than individual task runs, flowable tasks nested inside a `Loop` (such as `If` or `Parallel`) can access `item` directly without any `parent.` prefix.
```yaml
tasks:
- id: loop
type: io.kestra.plugin.core.flow.Loop
values: ["value 1", "value 2", "value 3"]
tasks:
- id: check
type: io.kestra.plugin.core.flow.If
condition: '{{ item.value == "value 2" }}'
then:
- id: log
type: io.kestra.plugin.core.log.Log
message: "Matched at index {{ item.index }}: {{ item.value }}"
```
## Environment and global variables
Kestra provides access to environment variables prefixed with `ENV_` by default, unless configured otherwise in the [runtime and storage configuration](/docs/configuration/runtime-and-storage).
- reference `ENV_FOO` as `{{ envs.foo }}`
- reference the configured environment name as `{{ kestra.environment }}`
- reference the configured Kestra URL as `{{ kestra.url }}`
- reference global variables from configuration as `{{ globals.foo }}`
## Flow variables and inputs
Use flow-level variables with `vars.*`:
```yaml
id: flow_variables
namespace: company.team
variables:
my_variable: "my_value"
tasks:
- id: print_variable
type: io.kestra.plugin.core.debug.Return
format: "{{ vars.my_variable }}"
```
Use inputs with `inputs.*`:
```yaml
id: render_inputs
namespace: company.team
inputs:
- id: myInput
type: STRING
tasks:
- id: myTask
type: io.kestra.plugin.core.debug.Return
format: "{{ inputs.myInput }}"
```
## Secrets, credentials, namespace variables, and outputs
Use `secret()` to inject secret values at runtime:
```yaml
tasks:
- id: myTask
type: io.kestra.plugin.core.debug.Return
format: "{{ secret('MY_SECRET') }}"
```
Use `credential()` in Enterprise Edition to inject a short-lived token from a managed [Credential](/docs/enterprise/auth/credentials):
```yaml
tasks:
- id: request
type: io.kestra.plugin.core.http.Request
method: GET
uri: https://api.example.com/v1/ping
auth:
type: BEARER
token: "{{ credential('my_oauth') }}"
```
`credential()` returns the short-lived token only. The credential itself is managed in the Kestra UI.
Use namespace variables in Enterprise Edition with `namespace.*`. To set them up:
1. Open the Kestra UI and navigate to **Namespaces**.
2. Select the namespace where the flow runs.
3. Open the **Variables** tab.
4. Add a key-value pair such as `github.token` with the desired value.
Reference namespace variables in expressions using dot notation:
```yaml
format: "{{ namespace.github.token }}"
```
If a namespace variable itself contains Pebble, evaluate it with `render()`:
```yaml
format: "{{ render(namespace.github.token) }}"
```
Use outputs with `outputs.taskId.attribute`:
```yaml
message: |
First: {{ outputs.first.value }}
Second: {{ outputs['second-task'].value }}
```
:::alert{type="info"}
If a task ID or output key contains a hyphen, use bracket notation such as `outputs['second-task']`. To avoid that, prefer `camelCase` or `snake_case`.
:::
---
# Kestra Filter Reference: Transform Expression Values
URL: https://kestra.io/docs/expressions/filters
> Complete reference for Kestra Pebble filters — JSON, collections, strings, dates, and YAML. Use filters to transform values with the pipe syntax.
import ChildCard from "~/components/docs/ChildCard.astro"
Use filters when you need to transform a value with the pipe syntax: `{{ value | filterName(...) }}`.
## Filter categories
- [JSON and structured data](./01.json/index.mdx) — `toJson`, `toIon`, `jq`
- [Numbers and collections](./02.collections/index.mdx) — `abs`, `number`, `first`, `last`, `sort`, `chunk`, `distinct`, and more
- [Strings](./03.strings/index.mdx) — `lower`, `upper`, `replace`, `slugify`, `base64encode`, `regexMatch`, `regexReplace`, `regexExtract`, and more
- [Dates](./04.dates/index.mdx) — `date`, `dateAdd`, `timestamp`, `timestampMilli`, and precision variants
- [YAML](./05.yaml/index.mdx) — `yaml`, `indent`, `nindent`
## Choosing the right filter quickly
| If you need to... | Use |
| --- | --- |
| Parse or transform JSON payloads | `toJson`, `jq`, `first` |
| Provide a fallback string or value | `default` |
| Format a date | `date` |
| Offset a date | `dateAdd` |
| Split or join text | `split`, `join` |
| Normalize casing | `lower`, `upper`, `title`, `capitalize` |
| Convert a value to a string | `string` |
| Sort a collection | `sort`, `rsort` |
| Count items in a collection | `length` |
| Get unique values | `distinct` |
| Encode or decode Base64 | `base64encode`, `base64decode` |
| Hash a string | `sha1`, `sha512`, `md5` |
| Convert to a number | `number` |
| Render YAML in a templated task | `yaml`, `indent`, `nindent` |
---
# Number and Collection Filters in Kestra
URL: https://kestra.io/docs/expressions/filters/collections
> Reference for Kestra's number and collection filters — abs, number, first, last, sort, chunk, distinct, slice, merge, flatten, keys, values, and more.
These filters are the everyday cleanup tools for expression values. Use them when you already have the right data but need to reformat it, count it, sort it, or coerce it into the type another task expects.
## `abs`
Returns the absolute value of a number:
```twig
{{ -7 | abs }}
{# output: 7 #}
```
## `number`
Parses a string into a numeric type. Supports `INT`, `FLOAT`, `LONG`, `DOUBLE`, `BIGDECIMAL`, and `BIGINTEGER`. When no type is specified, the type is inferred:
```twig
{{ "12.3" | number | className }}
{# output: java.lang.Float #}
{{ "9223372036854775807" | number('BIGDECIMAL') | className }}
{# output: java.math.BigDecimal #}
```
Use `BIGDECIMAL` or `BIGINTEGER` when values exceed standard long or double precision.
## `className`
Returns the Java class name of an object. Useful for debugging type inference when combined with `number`:
```twig
{{ "12.3" | number | className }}
{# output: java.lang.Float #}
```
## `numberFormat`
Formats a number using a Java `DecimalFormat` pattern:
```twig
{{ 3.141592653 | numberFormat("#.##") }}
{# output: 3.14 #}
```
## `first` and `last`
Returns the first or last element of a collection, or the first or last character of a string:
```twig
{{ ['apple', 'banana', 'cherry'] | first }}
{# output: apple #}
{{ ['apple', 'banana', 'cherry'] | last }}
{# output: cherry #}
{{ 'Kestra' | first }}
{# output: K #}
{{ 'Kestra' | last }}
{# output: a #}
```
## `length`
Returns the number of elements in a collection, or the number of characters in a string:
```twig
{{ ['apple', 'banana'] | length }}
{# output: 2 #}
{{ 'Kestra' | length }}
{# output: 6 #}
```
## `join`
Concatenates a collection into a single string with an optional delimiter:
```twig
{{ ['apple', 'banana', 'cherry'] | join(', ') }}
{# output: apple, banana, cherry #}
```
## `split`
Splits a string into a list using a delimiter. The delimiter is a regex, so escape special characters:
```twig
{{ 'apple,banana,cherry' | split(',') }}
{# output: ['apple', 'banana', 'cherry'] #}
{{ 'a.b.c' | split('\\.') }}
```
The optional `limit` argument controls how many splits are performed:
- **Positive**: limits the array size; the last entry contains the remaining content
- **Zero**: no limit; trailing empty strings are discarded
- **Negative**: no limit; trailing empty strings are included
```twig
{{ 'apple,banana,cherry,grape' | split(',', 2) }}
{# output: ['apple', 'banana,cherry,grape'] #}
```
## `sort` and `rsort`
Sort a collection in ascending or descending order:
```twig
{{ [3, 1, 2] | sort }}
{# output: [1, 2, 3] #}
{{ [3, 1, 2] | rsort }}
{# output: [3, 2, 1] #}
```
## `reverse`
Reverses the order of a collection:
```twig
{{ [1, 2, 3] | reverse }}
{# output: [3, 2, 1] #}
```
## `chunk`
Splits a collection into groups of a specified size:
```twig
{{ [1, 2, 3, 4, 5] | chunk(2) }}
{# output: [[1, 2], [3, 4], [5]] #}
```
## `distinct`
Returns only unique values from a collection:
```twig
{{ [1, 2, 2, 3, 1] | distinct }}
{# output: [1, 2, 3] #}
```
## `slice`
Extracts a portion of a collection or string using `fromIndex` (inclusive) and `toIndex` (exclusive):
```twig
{{ ['apple', 'banana', 'cherry'] | slice(1, 2) }}
{# output: [banana] #}
{{ 'Kestra' | slice(1, 3) }}
{# output: es #}
```
## `merge`
Merges two collections into one:
```twig
{{ [1, 2] | merge([3, 4]) }}
{# output: [1, 2, 3, 4] #}
```
## `flatten`
Removes one level of nesting from a collection:
```twig
{{ [[1, 2], [3, 4], [5]] | flatten }}
{# output: [1, 2, 3, 4, 5] #}
```
## `keys` and `values`
Return the keys or values of a map:
```twig
{{ {'foo': 'bar', 'baz': 'qux'} | keys }}
{# output: [foo, baz] #}
{{ {'foo': 'bar', 'baz': 'qux'} | values }}
{# output: [bar, qux] #}
```
---
# Date and Time Filters in Kestra Expressions
URL: https://kestra.io/docs/expressions/filters/dates
> Reference for Kestra's date and time filters — date, dateAdd, timestamp, timestampMilli, timestampMicro, and timestampNano — for formatting dates and converting to Unix timestamps.
These are the most common filters in scheduled flows and integrations. Reach for them whenever a downstream system expects a specific date format or timestamp precision rather than Kestra's native datetime value.
## `date`
```twig
{{ execution.startDate | date("yyyy-MM-dd") }}
```
You can also provide existing and target formats with named arguments:
```twig
{{ stringDate | date(existingFormat="yyyy-MMMM-d", format="yyyy/MMMM/d") }}
```
When you are formatting an already parsed datetime, only `format` is usually needed. Use `existingFormat` when the source is still a plain string.
### Time zones
Specify a target time zone when downstream systems require a local representation rather than UTC:
```twig
{{ now() | date("yyyy-MM-dd'T'HH:mm:ssX", timeZone="UTC") }}
```
Supported arguments include:
- `format`
- `existingFormat`
- `timeZone`
- `locale`
## `dateAdd`
Adds or subtracts time from a date. Arguments:
- `amount`: integer specifying how much to add or subtract
- `unit`: time unit such as `DAYS`, `HOURS`, `MONTHS`, or `YEARS`
```twig
{{ now() | dateAdd(-1, 'DAYS') }}
```
## Timestamp helpers
Convert a date to a Unix timestamp at a specific precision:
- `timestamp` — seconds
- `timestampMilli` — milliseconds
- `timestampMicro` — microseconds
- `timestampNano` — nanoseconds
:::alert{type="warning"}
`timestampMicro` previously returned a nanosecond-precision value due to a bug. If you are migrating an older flow, verify the precision your downstream system expects.
:::
All timestamp filters accept the same arguments as the `date` filter: `existingFormat` and `timeZone`.
```twig
{{ now() | timestamp(timeZone="Europe/Paris") }}
{{ now() | timestampMilli(timeZone="Asia/Kolkata") }}
```
Supported date formats include standard Java `DateTimeFormatter` patterns and shortcuts such as `iso`, `sql`, `iso_date_time`, and `iso_zoned_date_time`.
## Worked example
```yaml
id: temporal_dates
namespace: company.team
tasks:
- id: print_status
type: io.kestra.plugin.core.log.Log
message:
- "Present timestamp: {{ now() }}"
- "Formatted timestamp: {{ now() | date('yyyy-MM-dd') }}"
- "Previous day: {{ now() | dateAdd(-1, 'DAYS') }}"
- "Next day: {{ now() | dateAdd(1, 'DAYS') }}"
- "Timezone (seconds): {{ now() | timestamp(timeZone='Asia/Kolkata') }}"
- "Timezone (microseconds): {{ now() | timestampMicro(timeZone='Asia/Kolkata') }}"
- "Timezone (milliseconds): {{ now() | timestampMilli(timeZone='Asia/Kolkata') }}"
- "Timezone (nanoseconds): {{ now() | timestampNano(timeZone='Asia/Kolkata') }}"
```
This kind of example is a good sanity check when you are validating timestamp precision before sending values to an external API.
---
# JSON and Structured Data Filters in Kestra
URL: https://kestra.io/docs/expressions/filters/json
> Reference for Kestra's JSON and structured data filters — toJson, toIon, and jq — for serializing, reshaping, and extracting fields from task outputs and API responses.
Use these filters when the value you already have is structured and you need to reshape it, serialize it, or extract one field from a larger payload. They are especially common when working with task outputs and API responses.
## `toJson`
Convert an object into JSON:
```twig
{{ [1, 2, 3] | toJson }}
{{ true | toJson }}
{{ "foo" | toJson }}
```
## `toIon`
Convert an object into Ion:
```twig
{{ myObject | toIon }}
```
## `jq`
Apply a JQ expression to a value. The result is always an array, so combine it with `first` when appropriate:
```twig
{{ outputs | jq('.task1.value') | first }}
```
Examples:
```twig
{{ [1, 2, 3] | jq('.') }}
{{ [1, 2, 3] | jq('.[0]') | first }}
```
Example flow using `jq` inside a `Loop`:
```yaml
id: jq_with_loop
namespace: company.team
tasks:
- id: generate
type: io.kestra.plugin.core.debug.Return
format: |
[
{"name": "alpha", "value": 1},
{"name": "bravo", "value": 2}
]
- id: loop
type: io.kestra.plugin.core.flow.Loop
values: "{{ fromJson(outputs.generate.value) }}"
tasks:
- id: log_filtered
type: io.kestra.plugin.core.log.Log
message: |
Name: {{ fromJson(item.value).name }}
Doubled value: {{ fromJson(item.value) | jq('.value * 2') | first }}
```
The practical rule with `jq` is that it is great for extracting or transforming a small part of a larger payload, but it is usually overkill when plain dot access already gets you the value you need.
## Worked JSON payload example
This larger example is useful when you need to mix accessors, math, collection helpers, and JSON-aware filters in one expression flow:
```yaml
id: json_payload_example
namespace: company.team
inputs:
- id: payload
type: JSON
defaults: |-
{
"name": "John Doe",
"score": {
"English": 72,
"Maths": 88,
"French": 95,
"Spanish": 85,
"Science": 91
},
"address": {
"city": "Paris",
"country": "France"
},
"graduation_years": [2020, 2021, 2022, 2023]
}
tasks:
- id: print_status
type: io.kestra.plugin.core.log.Log
message:
- "Student name: {{ inputs.payload.name }}"
- "Score in languages: {{ inputs.payload.score.English + inputs.payload.score.French + inputs.payload.score.Spanish }}"
- "Total subjects: {{ inputs.payload.score | length }}"
- "Total score: {{ inputs.payload.score | values | jq('reduce .[] as $num (0; .+$num)') | first }}"
- "Complete address: {{ inputs.payload.address.city }}, {{ inputs.payload.address.country | upper }}"
- "Started college in: {{ inputs.payload.graduation_years | first }}"
- "Completed college in: {{ inputs.payload.graduation_years | last }}"
```
Use a pattern like this when the payload already arrives as JSON input and you want to keep the manipulation inside expressions instead of adding a preprocessing task.
---
# String Filters in Kestra Expressions
URL: https://kestra.io/docs/expressions/filters/strings
> Reference for Kestra's string filters — casing, trimming, encoding, hashing, regex, and substring extraction. Use them for display formatting, filename shaping, and API-compatible encodings.
String filters are where most small presentation fixes happen. They are usually the right tool for display formatting, filename shaping, templated messages, and API-compatible encodings.
## Case and whitespace
`lower`, `upper`, `title`, and `capitalize` normalize casing. `trim` removes leading and trailing whitespace.
```twig
{{ "LOUD TEXT" | lower }} {# loud text #}
{{ "quiet text" | upper }} {# QUIET TEXT #}
{{ "article title" | title }} {# Article Title #}
{{ "hello world" | capitalize }} {# Hello world #}
{{ " padded " | trim }} {# padded #}
```
## `abbreviate`
Truncates a string to a maximum length and appends an ellipsis. The length argument includes the ellipsis:
```twig
{{ "this is a long sentence." | abbreviate(7) }} {# this... #}
```
Useful when you need to keep log messages or notification subjects within a character limit.
## `replace`
Substitutes one or more substrings using a map. Pass `regexp=true` to use regex patterns in the keys:
```twig
{{ "I like %this% and %that%." | replace({'%this%': foo, '%that%': "bar"}) }}
```
## `substringBefore`, `substringAfter`, and their `Last` variants
Extract the portion of a string before or after a delimiter. The `Last` variants match the final occurrence:
```twig
{{ "a.b.c" | substringBefore(".") }} {# a #}
{{ "a.b.c" | substringAfter(".") }} {# b.c #}
{{ "a.b.c" | substringBeforeLast(".") }} {# a.b #}
{{ "a.b.c" | substringAfterLast(".") }} {# c #}
```
These are particularly useful for extracting file extensions, path segments, or identifier prefixes from task output values.
## `slugify`
Converts a string into a URL-safe slug:
```twig
{{ "Hello World!" | slugify }} {# hello-world #}
```
## `default`
Returns a fallback value when the expression is null or empty:
```twig
{{ user.phoneNumber | default("No phone number") }}
```
## `startsWith`
Returns `true` if the string begins with the given prefix:
```twig
{{ "kestra://file.csv" | startsWith("kestra://") }} {# true #}
```
## `endsWith`
Returns `true` if the string ends with the given suffix:
```twig
{{ "report.csv" | endsWith(".csv") }} {# true #}
```
## Encoding and hashing
`base64encode` and `base64decode` handle Base64 encoding. `urlencode` and `urldecode` percent-encode strings for use in URLs. `sha1`, `sha512`, and `md5` produce hex-encoded hashes of the corresponding algorithms.
```twig
{{ "test" | base64encode }}
{# output: dGVzdA== #}
{{ "dGVzdA==" | base64decode }}
{# output: test #}
{{ "The string ü@foo-bar" | urlencode }}
{# output: The+string+%C3%BC%40foo-bar #}
{{ "The+string+%C3%BC%40foo-bar" | urldecode }}
{# output: The string ü@foo-bar #}
{{ "test" | sha1 }}
{{ "test" | sha512 }}
{{ "test" | md5 }}
```
## `string`
Coerces any value to its string representation:
```twig
{{ 42 | string }}
```
Use this when chaining filters that expect string input on a value that may arrive as a number or boolean.
## `escapeChar`
Escapes special characters in a string. The `type` argument controls which style of escaping is applied: `single`, `double`, or `shell`:
```twig
{{ "Can't be here" | escapeChar('single') }}
{# output: Can\'t be here #}
```
## Regex filters
Three filters cover the most common regex operations:
- `regexMatch(regex)` — returns `true` if the input contains a substring matching the pattern, `false` otherwise.
- `regexReplace(regex, replacement)` — replaces all non-overlapping matches. Use `$1`, `$2`, … to reference capture groups in the replacement.
- `regexExtract(regex, group)` — returns the first match or a specific capture group. `group` defaults to `0` (the whole match); returns `null` if there is no match.
```twig
{{ "hello world" | regexMatch("w[a-z]+") }}
{# output: true #}
{{ "2024-01-15" | regexReplace("(\\d{4})-(\\d{2})-(\\d{2})", "$3/$2/$1") }}
{# output: 15/01/2024 #}
{{ "order-12345-done" | regexExtract("\\d+") }}
{# output: 12345 #}
{{ "2024-01-15" | regexExtract("(\\d{4})-(\\d{2})-(\\d{2})", 1) }}
{# output: 2024 #}
```
:::alert{type="warning"}
Regex filter operations are subject to a **10-second timeout** to prevent ReDoS (catastrophic backtracking). If a pattern takes longer than the limit, the task fails with a timeout error.
Patterns with nested quantifiers such as `(a+)+` applied to large inputs are most likely to trigger this. Use anchored, non-ambiguous patterns to avoid it. The timeout can be adjusted with [`kestra.regex.timeout`](../../../configuration/05.security-and-secrets/index.md#regex-timeout) in your Kestra configuration.
:::
## Worked string filter example
This flow builds a sanitized filename and a display-safe summary from a raw input title:
```yaml
id: string_filter_example
namespace: company.team
inputs:
- id: title
type: STRING
defaults: " Quarterly Report: Q1 2025 (FINAL) "
tasks:
- id: format_output
type: io.kestra.plugin.core.log.Log
message:
- "Trimmed: {{ inputs.title | trim }}"
- "Normalized: {{ inputs.title | trim | lower }}"
- "Slug (for filename): {{ inputs.title | trim | slugify }}"
- "Abbreviated (for subject line): {{ inputs.title | trim | abbreviate(30) }}"
- "Prefix check: {{ inputs.title | trim | startsWith('Quarterly') }}"
- "After colon: {{ inputs.title | trim | substringAfter(':') | trim }}"
```
---
# YAML Filters in Kestra Expressions
URL: https://kestra.io/docs/expressions/filters/yaml
> Reference for Kestra's YAML filters — yaml, indent, and nindent — for parsing and formatting YAML in templated tasks, Kubernetes manifests, and config-management patterns.
Use YAML filters when you are generating configuration or manifest-style text inside a task. They are less common in simple flows, but very useful in templated Kubernetes, Docker, or config-management patterns.
## `yaml`
Parse YAML into an object:
```twig
{{ "foo: bar" | yaml }}
```
This is especially useful in templated tasks where the source data starts as text but later expressions need object-style access.
### Example: using `yaml` in a templated task
```yaml
id: yaml_filter_example
namespace: company.team
tasks:
- id: yaml_filter
type: io.kestra.plugin.core.log.Log
message: |
{{ "foo: bar" | yaml }}
{{ {"key": "value"} | yaml }}
```
## `indent` and `nindent`
Useful when generating templated YAML or embedding structured content:
```twig
{{ labels | yaml | indent(4) }}
{{ variables.yaml_data | yaml | nindent(4) }}
```
### Example with `indent` and `nindent`
```yaml
id: templated_task_example
namespace: company.team
labels:
example: test
variables:
yaml_data: |
key1: value1
key2: value2
tasks:
- id: yaml_with_indent
type: io.kestra.plugin.core.templating.TemplatedTask
spec: |
id: example-task
type: io.kestra.plugin.core.log.Log
message: |
Metadata:
{{ labels | yaml | indent(4) }}
Variables:
{{ variables.yaml_data | yaml | nindent(4) }}
```
Use `indent` when the first line is already in place and only following lines need alignment. Use `nindent` when you need to start a fresh indented block on the next line.
---
# Kestra Function Reference: Generate and Retrieve Values
URL: https://kestra.io/docs/expressions/functions
> Complete reference for Kestra Pebble functions — rendering, data access, parsing, workflow helpers, utilities, and date/calendar functions.
import ChildCard from "~/components/docs/ChildCard.astro"
Use functions when you need to generate or retrieve a value dynamically with syntax such as `{{ functionName(...) }}`.
Functions are best thought of as helpers that either fetch something, compute something, or force evaluation behavior that plain variables and filters cannot provide on their own.
## Function groups
- [Rendering and debugging](./01.rendering/index.mdx) — `render()`, `renderOnce()`, `fetchContext()`, template inheritance helpers
- [Data access](./02.data-access/index.mdx) — `secret()`, `env()`, `credential()`, `read()`, `fileURI()`, `kv()`, `encrypt()`, `decrypt()`
- [Data parsing](./03.parsing/index.mdx) — `fromJson()`, `fromIon()`, `yaml()`
- [Workflow helpers](./04.workflow/index.mdx) — `errorLogs()`, `tasksWithState()`, `loopOutputs()`, `appLink()`
- [Utilities](./05.utilities/index.mdx) — `now()`, `uuid()`, `randomInt()`, `http()`, `fileSize()`, `fileExists()`, and more
- [Date and calendar](./06.dates/index.mdx) — `isWeekend()`, `isPublicHoliday()`, `dayOfWeek()`, `monthOfYear()`, and more
## Worked example
This flow uses several runtime functions together: `now()` for a timestamp, `uuid()` for a unique run identifier, `secret()` for a credential, and `render()` to evaluate a namespace variable containing Pebble:
```yaml
id: function_reference_example
namespace: company.team
tasks:
- id: log_context
type: io.kestra.plugin.core.log.Log
message:
- "Run ID: {{ uuid() }}"
- "Started at: {{ now() | date('yyyy-MM-dd HH:mm:ss') }}"
- "API key: {{ secret('MY_API_KEY') }}"
- "Config value: {{ render(namespace.my_config) }}"
```
---
# Data Access Functions in Kestra Expressions
URL: https://kestra.io/docs/expressions/functions/data-access
> Reference for Kestra's data access functions — secret(), env(), credential(), read(), fileURI(), kv(), encrypt(), and decrypt() — for resolving secrets, environment variables, files, and stored values at runtime.
These functions bridge expressions to external or stored data. Use them when the value is not already present in the execution context and must be resolved at runtime.
## `secret()`
Use `secret()` for sensitive values that should not appear in the flow definition. The namespace defaults to the flow's namespace; pass a `namespace` to read a secret stored in another namespace, and a `subkey` to extract a single field from a JSON secret:
```twig
{{ secret('API_KEY') }}
{{ secret('GITHUB_ACCESS_TOKEN') }}
{{ secret('SHARED_SECRET', namespace='other.namespace') }}
{{ secret('DB_CREDENTIALS', subkey='password') }}
```
Arguments:
- `key` — the secret key
- `namespace` — defaults to the flow's namespace; the secret is resolved using that namespace's secret backend, with values inherited from parent namespaces
- `subkey` — optional field to extract when the secret holds a JSON object
- `full` — when `true`, returns a map with two fields: `value` (the primary secret) and `metadata` (any additional fields exposed by the secret manager). Cannot be combined with `subkey`.
Cross-namespace reads stay within the same tenant. In the Enterprise Edition, a flow may read another namespace's secrets by default; restrict this by configuring `allowedNamespaces` on the target namespace.
### Retrieving multi-field secrets with `full=true`
Some secret managers store credentials as structured objects — for example, a database credential with a password, username, and domain. Pass `full=true` to retrieve all fields at once:
```twig
{% set cred = secret('prod-mssql', full=true) %}
{{ cred.value }} {# primary secret, same as secret('prod-mssql') #}
{{ cred.metadata.username }} {# additional field from the secret manager #}
{{ cred.metadata.domain }} {# additional field from the secret manager #}
```
`cred.value` is always present and holds the primary secret. `cred.metadata` is only present when the secret manager exposes additional fields — for single-value backends (AWS, Azure, GCP, Doppler, internal), the `metadata` key is absent and only `cred.value` is accessible.
| Backend | `cred.metadata` |
| --- | --- |
| Delinea | credential fields from the secret template |
| CyberArk | account fields (username, address, etc.) |
| BeyondTrust | credential fields |
| 1Password | item fields |
| AWS, Azure, GCP, Doppler, internal | absent — use `cred.value` only |
## `env()`
Reads an environment variable from the execution context by name. Use `env()` when the variable name is dynamic — composed at runtime from inputs or outputs — something not possible with the static `envs.varname` dot notation.
```twig
{{ env('API_HOST') }}
{{ env('API_HOST', 'localhost') }}
{{ env('api_url_' ~ inputs.environment) }}
```
Arguments:
- `name` — the environment variable name, after normalization (lowercase, prefix stripped)
- `default` — optional value returned when the variable is missing or empty
Environment variables are exposed via the `envs` context map. By default, only variables prefixed with `ENV_` are available; the prefix is stripped and the name lowercased: `ENV_API_URL_PROD` is accessible as `env('api_url_prod')`. The prefix is configurable via `kestra.variables.env-vars-prefix` in the [runtime configuration](../../../configuration/02.runtime-and-storage/index.md).
The key difference from `{{ envs.api_url_prod }}` is that the name argument accepts any expression, allowing the key to be composed dynamically:
```yaml
inputs:
- id: environment
type: STRING
defaults: prod
tasks:
- id: log_url
type: io.kestra.plugin.core.log.Log
message: "{{ env('api_url_' ~ inputs.environment) }}"
```
With `ENV_API_URL_PROD` and `ENV_API_URL_DEV` set in the host environment, running the flow with `environment` set to `prod` resolves to the production URL and `dev` to the development URL — without any `if`/`switch` logic in the flow.
## `credential()`
In Enterprise Edition, use `credential()` to inject a short-lived token from a managed credential:
```twig
{{ credential('my_oauth') }}
```
`credential()` returns the token only, while the credential definition itself is managed in the Kestra UI:
```yaml
tasks:
- id: request
type: io.kestra.plugin.core.http.Request
method: GET
uri: https://api.example.com/v1/ping
auth:
type: BEARER
token: "{{ credential('my_oauth') }}"
```
## `read()`
`read()` is the simplest way to turn a file URI back into inline content for a later expression:
```twig
{{ read(outputs.someTask.uri) }}
{{ read('subdir/file.txt') }}
```
`read()` accepts both namespace files and internal-storage URIs, which makes it useful after download or transformation tasks that write files as outputs.
For files in CSV, JSON, XML, YAML, or any other non-ION format, `read()` returns a `String`. For **binary ION files** — the format used by ION task outputs — `read()` returns `byte[]` to preserve binary fidelity. Pass the result directly to `fromIon()` to deserialize it:
```twig
{{ fromIon(read(outputs.transform.uri)) }}
{{ fromIon(read(outputs.transform.uri), allRows=true) }}
```
Do not perform string operations such as `contains` directly on `read()` when the URI points to an ION task output — use `fromIon()` first.
## `fileURI()`
Returns the internal URI of a namespace file without reading its contents. Use `fileURI()` when a task parameter expects a URI rather than inline content:
```twig
{{ fileURI('my_file.txt') }}
```
Use `read()` instead when you need to embed the file contents inline in a later expression.
## `kv()`
Reads a value from the KV store by key. The namespace defaults to the flow's namespace; set `errorOnMissing` to `false` to return `null` instead of throwing when the key is absent:
```twig
{{ kv('MY_KEY') }}
{{ kv('MY_KEY', 'other.namespace') }}
{{ kv('OPTIONAL_KEY', namespace, false) }}
```
Arguments:
- `key` — the KV store key
- `namespace` — defaults to the flow's namespace
- `errorOnMissing` — defaults to `true`
## `encrypt()` and `decrypt()`
Encrypt and decrypt string values using Kestra's encryption service. Both require a `key` argument that identifies which encryption key to use:
```twig
{{ encrypt('MY_ENCRYPTION_KEY', inputs.sensitiveValue) }}
{{ decrypt('MY_ENCRYPTION_KEY', outputs.encryptTask.value) }}
```
---
# Date and Calendar Functions in Kestra Expressions
URL: https://kestra.io/docs/expressions/functions/dates
> Reference for Kestra's date and calendar functions — isWeekend(), isPublicHoliday(), isDayWeekInMonth(), dayOfWeek(), dayOfMonth(), monthOfYear(), and hourOfDay() — for scheduling and routing logic.
Use these functions when you need to make scheduling or routing decisions based on the calendar — for example, skipping runs on weekends or public holidays.
## `isWeekend()`
Returns `true` if the date falls on Saturday or Sunday:
```twig
{{ isWeekend(trigger.date) }}
```
## `isPublicHoliday()`
Checks against a country's public holiday calendar. `countryCode` is an ISO 3166-1 alpha-2 code; `subDivision` is optional and accepts ISO 3166-2 codes:
```twig
{{ isPublicHoliday(trigger.date, 'US') }}
{{ isPublicHoliday(trigger.date, 'DE', 'DE-BY') }}
```
## `isDayWeekInMonth()`
Returns `true` if the date is the Nth occurrence of the given weekday in its month. `position` accepts `FIRST`, `SECOND`, `THIRD`, `FOURTH`, or `LAST`:
```twig
{{ isDayWeekInMonth(trigger.date, 'MONDAY', 'FIRST') }}
```
## `isLastWorkingDay()`
Returns `true` if the date is the last working day of its month. Working days default to Monday–Friday. An optional second argument overrides which days count as working days using a comma-separated list of uppercase day names:
```twig
{{ isLastWorkingDay(trigger.date) }}
{{ isLastWorkingDay(trigger.date, 'MONDAY,TUESDAY,WEDNESDAY,THURSDAY') }}
```
The `date` argument accepts any ISO 8601 date or datetime string. Combine with `isPublicHoliday()` if you also need to exclude public holidays.
## `dayOfWeek()`
Returns the uppercase day name such as `MONDAY`:
```twig
{{ dayOfWeek(trigger.date) }}
```
## `dayOfMonth()`
Returns the day of the month as an integer (1–31):
```twig
{{ dayOfMonth(trigger.date) }}
```
## `monthOfYear()`
Returns the month as an integer (1–12):
```twig
{{ monthOfYear(trigger.date) }}
```
## `hourOfDay()`
Returns the hour as an integer (0–23):
```twig
{{ hourOfDay(execution.startDate) }}
```
---
# Data Parsing Functions in Kestra Expressions
URL: https://kestra.io/docs/expressions/functions/parsing
> Reference for Kestra's data parsing functions — fromJson(), fromIon(), and yaml() — for deserializing task outputs and working with structured data in expressions.
These helpers are most useful when a task output is still a serialized string and you want to treat it like structured data in later expressions.
## `fromJson()`
Parses a JSON string into an object so you can access its fields with dot or bracket notation:
```twig
{{ fromJson(outputs.myTask.value).name }}
{{ fromJson('[1, 2, 3]')[0] }}
```
Use `fromJson()` when a task output arrives as a serialized JSON string rather than a structured object. To go the other direction, use the [`toJson` filter](../../03.filters/01.json/index.mdx#tojson).
## `fromIon()`
Use `fromIon()` when a previous task or serializer produces ION rather than JSON. `fromIon()` accepts both text ION strings and binary ION byte arrays, so it works correctly with `read()` regardless of the storage format.
Access a field from the first row:
```twig
{{ fromIon(read(outputs.serialize.uri)).someField }}
```
Return all rows as a list with `allRows=true`:
```twig
{{ fromIon(read(outputs.transform.uri), allRows=true) }}
```
Count rows:
```twig
{{ fromIon(read(outputs.query.uri), allRows=true) | length }}
```
## `yaml()`
Parses a YAML string into an object so you can access its fields with dot or array notation:
```twig
{{ yaml('foo: [666, 1, 2]').foo[0] }}
```
`yaml()` is available both as a function and as a filter (`{{ value | yaml }}`). Use the function form when you are working with a raw YAML string literal or a variable containing YAML text. See the [`yaml` filter](../../03.filters/05.yaml/index.mdx) for additional options including `indent` and `nindent` for template formatting.
---
# Rendering and Debugging Functions in Kestra
URL: https://kestra.io/docs/expressions/functions/rendering
> Reference for Kestra's rendering and debugging functions — render(), renderOnce(), fetchContext(), block(), and parent() — for evaluating nested expressions and inspecting execution context.
This group matters when expressions stop behaving the way you expect. `render()` and `fetchContext()` are often the quickest way to understand whether a value is missing, nested, or still just a string.
## `render()`
Use `render()` when a variable itself contains Pebble and must be evaluated:
```twig
{{ render(namespace.github.token) }}
{{ render("{{ trigger.date ?? execution.startDate | date('yyyy-MM-dd') }}") }}
```
Without `render()`, namespace or flow variables that contain Pebble are treated as plain strings.
This pattern is especially useful with namespace variables, composed flow variables, and fallback logic based on trigger context:
```yaml
variables:
trigger_or_yesterday: "{{ trigger.date ?? (execution.startDate | dateAdd(-1, 'DAYS')) }}"
tasks:
- id: yesterday
type: io.kestra.plugin.core.log.Log
message: "{{ render(vars.trigger_or_yesterday) }}"
```
## `renderOnce()`
Equivalent to `render(expression, recursive=false)`. Use `renderOnce()` when you need one extra evaluation pass but do not want recursive expansion to keep walking nested Pebble content:
```twig
{{ renderOnce(namespace.github.token) }}
```
## `fetchContext()`
Outputs the full execution context as a string. Use it in the Debug Expression console to inspect every variable available at that point in the execution:
```twig
{{ fetchContext() }}
```
This is the fastest way to discover the exact key names and structure of `inputs`, `outputs`, `trigger`, and other context variables when an expression is not resolving as expected.
## Template inheritance helpers
These are less common than runtime-oriented helpers, but they matter when you are using Pebble blocks and template inheritance directly.
### `block()`
`block()` renders the contents of a named block multiple times. It is different from the Pebble `block` tag, which declares the block:
```twig
{% block "post" %}content{% endblock %}
{{ block("post") }}
```
### `parent()`
Use `parent()` inside an overriding block to include the original block content from the parent template:
```twig
{% extends "parent.peb" %}
{% block "content" %}
child content
{{ parent() }}
{% endblock %}
```
---
# Utility Functions in Kestra Expressions
URL: https://kestra.io/docs/expressions/functions/utilities
> Reference for Kestra's utility functions — now(), uuid(), randomInt(), range(), http(), fileSize(), fileExists(), isFileEmpty(), and more — for generating values and inspecting files at runtime.
## `now()`
Returns the current datetime. Accepts a `timeZone` argument:
```twig
{{ now() }}
{{ now(timeZone="Europe/Paris") }}
```
## `max()` and `min()`
Returns the largest or smallest of its arguments:
```twig
{{ max(5, 10, 15) }}
{# output: 15 #}
{{ min(5, 10, 15) }}
{# output: 5 #}
```
## `range()`
Generates a list of integers up to and including `end`. The step defaults to 1:
```twig
{{ range(0, 3) }}
{# output: [0, 1, 2, 3] #}
{{ range(0, 6, 2) }}
{# output: [0, 2, 4, 6] #}
```
## `uuid()`
Generates a UUID in URL-safe base62 encoding:
```twig
{{ uuid() }}
```
## `id()`
Generates a short unique ID using Kestra's internal ID utility:
```twig
{{ id() }}
```
## `ksuid()`
Generates a K-Sortable Unique Identifier (timestamp-prefixed, base62-encoded). Useful when sort order by creation time matters:
```twig
{{ ksuid() }}
```
## `nanoId()`
Generates a NanoID. `length` defaults to 21 and `alphabet` defaults to alphanumeric plus `-_`:
```twig
{{ nanoId() }}
{{ nanoId(length=10) }}
```
## `randomInt()`
Generates a random integer. The upper bound is **excluded**:
```twig
{{ randomInt(1, 10) }}
{# generates a random integer from 1 to 9 (10 is excluded) #}
```
## `randomPort()`
Picks an available local port. Useful in test or dev container flows:
```twig
{{ randomPort() }}
```
## `http()`
Fetches a remote payload directly from an expression:
```twig
{{ http(uri = 'https://dummyjson.com/products/categories') | jq('.[].slug') }}
```
Use it sparingly. It is convenient for dynamic dropdowns and lightweight lookups, but task-level HTTP calls are usually easier to observe and retry.
## `fileSize()`
Returns the size in bytes of a file from internal storage:
```twig
{{ fileSize(outputs.download.uri) }}
```
## `fileExists()`
Returns `true` if the file exists:
```twig
{{ fileExists(outputs.download.uri) }}
```
## `isFileEmpty()`
Returns `true` if the file has no content:
```twig
{{ isFileEmpty(outputs.download.uri) }}
```
---
# Workflow Helper Functions in Kestra Expressions
URL: https://kestra.io/docs/expressions/functions/workflow
> Reference for Kestra's workflow and execution helper functions — loopOutputs(), errorLogs(), tasksWithState(), and appLink().
This group is more situational, but it becomes valuable in complex flows where you need to inspect sibling results, build links back into Kestra, or summarize failures.
## `loopOutputs()`
Extracts a named output from every iteration of a [Loop](../../../05.workflow-components/01.tasks/00.flowable-tasks/index.md#loop) task and returns the values as an ordered list.
**Prerequisite**: the Loop task must declare an `outputs:` block. Without it, `outputs.loop.outputs` does not exist and this function will fail.
```twig
{{ loopOutputs(outputs.myLoop.outputs, 'result') }}
```
The first argument must be the loop task's `.outputs` list. The second argument is the output ID — it must match an output ID declared in the Loop task's `outputs:` block. The function returns one value per iteration in order, with `null` for iterations where that key is missing.
For a `Loop` over `["a", "b", "c"]` with a declared output `result`:
```twig
{{ loopOutputs(outputs.loop.outputs, 'result') }}
{# → ["processed a", "processed b", "processed c"] #}
```
To access a single iteration directly, use list index notation:
```twig
{{ outputs.loop.outputs[0].outputs.result }} {# first iteration's declared output #}
{{ outputs.loop.outputs[0].item.value }} {# first iteration's input value #}
{{ outputs.loop.outputs[0].item.iteration }} {# first iteration's zero-based index #}
```
## `errorLogs()`
Prints all error logs from the current execution:
```twig
{{ errorLogs() }}
```
It is most useful in `errors` blocks, where you need a compact summary of what failed without manually traversing task state objects.
## `tasksWithState()`
Returns a list of task run objects matching the given state. Use it in error handlers or notifications to report which tasks failed:
```twig
{{ tasksWithState('FAILED') }}
```
Useful for building conditional logic or failure summaries based on task outcomes.
## `subflow()`
Synchronously runs a subflow and returns its terminal execution result, so you can read the subflow's outputs, state, or labels from within an expression.
```twig
{{ subflow(namespace='company.team', id='my_subflow', inputs={'key': 'value'}).outputs.my_output }}
```
**Arguments:**
| Argument | Required | Description |
|---|---|---|
| `namespace` | Yes | Namespace of the subflow to run |
| `id` | Yes | Flow ID of the subflow to run |
| `inputs` | No | Map of inputs to pass to the subflow |
| `revision` | No | Specific revision to run; defaults to latest |
| `labels` | No | Labels to attach to the subflow execution |
| `timeout` | No | ISO 8601 duration; defaults to `PT1M`, hard cap `PT5M` |
**Return value** — an object with four fields:
| Field | Type | Description |
|---|---|---|
| `.id` | string | Execution ID of the subflow run |
| `.state` | string | Terminal state name, e.g. `SUCCESS`, `FAILED` |
| `.outputs.` | any | Flow-level outputs declared in the subflow's `outputs:` block |
| `.labels.` | string | Execution labels as a key → value map |
**Primary use case** — populating a `SELECT` or `MULTISELECT` input's `expression:` at form render time:
```yaml
inputs:
- id: datacenter
type: SELECT
expression: "{{ subflow(namespace='company.ops', id='fetch_datacenters').outputs.datacenter_list }}"
```
When a user opens the Execute form, Kestra runs the subflow synchronously, reads its output, and populates the dropdown before the main flow begins.
**Important constraints:**
- Only valid in an input `expression:` context. Using `subflow()` inside a task or trigger property throws an error, because blocking a worker thread while waiting for a child execution can deadlock a worker under load.
- Flows referenced by `subflow()` in a `SELECT` or `MULTISELECT` input expression appear in the parent flow's **Dependencies** graph, the same way a `Subflow` task reference does.
- Only available on `WEBSERVER` and `STANDALONE` server types. The function is not registered on other server types.
- The default timeout is `PT1M`. The hard cap is `PT5M` — passing a larger `timeout` value is rejected at runtime. Both limits are configurable; see [configuration reference](../../../configuration/04.plugins-and-execution/index.md#subflow-function-configuration).
- Subflow recursion depth is capped at 3. A subflow whose own inputs call `subflow()` counts against this limit.
- Executions triggered by `subflow()` carry a `system.from: subflow` label automatically.
## `appLink()`
Enterprise Edition's `appLink()` builds links back to Kestra Apps:
```twig
{{ appLink(appId='com.example.my-app') }}
{{ appLink(baseUrl=true) }}
```
Use it in notifications when you want recipients to jump directly into the related app rather than the generic flow UI.
---
# Pebble Syntax in Kestra: Tags, Operators & Control Flow
URL: https://kestra.io/docs/expressions/syntax
> Complete reference for writing Kestra expressions — delimiters, attribute access, nested rendering, control flow, comparisons, logic operators, and Pebble type tests.
Use this page when you need help writing expressions — delimiters, attribute access, nested rendering, control flow, fallback patterns, comparisons, logic operators, and type tests.
## Pebble basics
Pebble templates use two primary delimiters:
- `{{ ... }}` to output the result of an expression
- `{% ... %}` to control template flow with tags such as `if`, `for`, or `set`
Examples:
```twig
{{ flow.id }}
{% if inputs.region == "eu" %}Europe{% endif %}
```
To escape Pebble syntax literally, use the `raw` tag described in [Tags](#raw).
## Accessing values
Use dot notation for standard property access:
```twig
{{ foo.bar }}
```
Use bracket notation for special characters or indexed access:
```twig
{{ foo['foo-bar'] }}
{{ items[0] }}
```
:::alert{type="warning"}
If a task ID, output key, or attribute contains a hyphen, use bracket notation. To avoid that, prefer `camelCase` or `snake_case`.
:::
## Parsing nested expressions
Kestra renders expressions once by default. If a variable contains Pebble that should be evaluated later, use `render()`:
```yaml
variables:
trigger_or_yesterday: "{{ trigger.date ?? (execution.startDate | dateAdd(-1, 'DAYS')) }}"
input_or_yesterday: "{{ inputs.mydate ?? (execution.startDate | dateAdd(-1, 'DAYS')) }}"
tasks:
- id: yesterday
type: io.kestra.plugin.core.log.Log
message: "{{ render(vars.trigger_or_yesterday) }}"
- id: input_or_yesterday
type: io.kestra.plugin.core.log.Log
message: "{{ render(vars.input_or_yesterday) }}"
```
This pattern is especially useful with namespace variables, composed flow variables, and fallback logic based on trigger context.
### Multiline JSON bodies
When an HTTP request body contains multiline user input, avoid partial string interpolation. Instead, build the whole payload as a single Pebble expression so JSON escaping happens correctly.
```yaml
id: multiline_input_passed_to_json_body
namespace: company.team
inputs:
- id: title
type: STRING
defaults: This is my title
- id: message
type: STRING
defaults: |-
This is my long
multiline message.
- id: priority
type: INT
defaults: 5
tasks:
- id: hello
type: io.kestra.plugin.core.http.Request
uri: https://kestra.io/api/mock
method: POST
body: |
{{ {
"title": inputs.title,
"message": inputs.message,
"priority": inputs.priority
} | toJson }}
```
## Common syntax patterns
### Comments
Use Pebble comments with `{# ... #}`:
```twig
{# This is a comment #}
{{ "Visible content" }}
```
In YAML, continue to use `#` for comments outside the expression itself.
### Literals and collections
Pebble supports:
- strings: `"Hello World"`
- numbers such as `100 + 10l * 2.5`
- booleans: `true`, `false`
- null: `null`
- lists: `["apple", "banana"]`
- maps: `{"apple":"red", "banana":"yellow"}`
### Named arguments
Filters, functions, and macros can accept named arguments:
```twig
{{ stringDate | date(existingFormat="yyyy-MMMM-d", format="yyyy/MMMM/d") }}
```
## Control flow and fallbacks
Common patterns:
- `if` and `elseif` for branching
- `for` for iteration
- `??` for fallback values
- `? :` for ternary expressions
Examples:
```twig
{{ inputs.mydate ?? (execution.startDate | dateAdd(-1, 'DAYS')) }}
```
```twig
{% for article in articles %}
{{ article.title }}
{% else %}
No articles available.
{% endfor %}
```
Inside a `for` loop, Pebble provides a `loop` object with properties such as `loop.index`, `loop.first`, `loop.last`, and `loop.length`. For the full table and examples, see [for](#for).
```twig
{% if category == "news" %}
{{ news }}
{% elseif category == "sports" %}
{{ sports }}
{% else %}
Select a category
{% endif %}
```
## Operators
### Comparisons
Supported comparison operators:
- `==`
- `!=`
- `<`
- `>`
- `<=`
- `>=`
```twig
{% if execution.state == "SUCCESS" %}
Flow completed successfully.
{% endif %}
{% if taskrun.attemptsCount >= 3 %}
Max retries reached.
{% endif %}
```
### Logic and boolean checks
Use:
- `and`
- `or`
- `not`
- `is`
- `contains`
Use parentheses to group expressions and make precedence explicit:
```twig
{% if 2 is even and 3 is odd %}
...
{% endif %}
{% if (3 is not even) and (2 is odd or 3 is even) %}
...
{% endif %}
```
### `contains`
Checks whether an item exists within a list, string, map, or array:
```twig
{% if ["apple", "pear", "banana"] contains "apple" %}
...
{% endif %}
```
For maps, `contains` checks for a matching key:
```twig
{% if {"apple": "red", "banana": "yellow"} contains "banana" %}
...
{% endif %}
```
To check for multiple items at once, pass a list on the right-hand side:
```twig
{% if ["apple", "pear", "banana", "peach"] contains ["apple", "peach"] %}
...
{% endif %}
```
`contains` also works inline in output expressions:
```twig
{{ inputs.mainString contains inputs.subString }}
```
### `isIn`
Use `isIn` to test whether a value matches any item in a list. It reads more clearly than chaining multiple equality checks in `runIf`, SLAs, or alert conditions:
```twig
{{ execution.state isIn ['SUCCESS', 'KILLED', 'CANCELLED'] }}
```
### Math and concatenation
Use:
- `+`, `-`, `*`, `/`, `%`
- `~` for string concatenation
Example:
```twig
{{ "apple" ~ "pear" ~ "banana" }}
{{ 2 + 2 / (10 % 3) * (8 - 1) }}
```
### Fallbacks and conditionals
Use:
- `??` for null-coalescing: returns the first non-null value
- `???` for undefined-coalescing: returns the right-hand side only when the left is undefined (not just null)
- `? :` for ternary expressions
Examples:
```twig
{{ foo ?? bar ?? "default" }} {# first non-null value #}
{{ foo ??? "default" }} {# only if foo is undefined #}
{{ foo == null ? bar : baz }}
{{ foo ?? bar ?? raise }} {# raises an exception if all are undefined #}
```
For detailed null vs undefined behavior, see the [Handling null and undefined values](/docs/how-to-guides/null-values) guide.
### Operator precedence
Pebble operators are evaluated in this order:
1. `.`
2. `|`
3. `%`, `/`, `*`
4. `-`, `+`
5. `==`, `!=`, `>`, `<`, `>=`, `<=`
6. `is`, `is not`
7. `and`
8. `or`
## Tags
Pebble tags are enclosed in `{% %}` and control template flow.
### `set`
Defines a variable in the template context:
```twig
{% set header = "Welcome Page" %}
{{ header }}
{# output: Welcome Page #}
```
### `if`
Evaluates conditional logic. Use `elseif` and `else` for multiple branches:
```twig
{% if users is empty %}
No users available.
{% elseif users.length == 1 %}
One user found.
{% else %}
Multiple users found.
{% endif %}
```
### `for`
Iterates over arrays, maps, or any `java.lang.Iterable`.
**Iterating over a list:**
```twig
{% for user in users %}
{{ user.name }} lives in {{ user.city }}.
{% else %}
No users found.
{% endfor %}
```
The `else` block runs when the collection is empty.
**Iterating over a map:**
```twig
{% for entry in map %}
{{ entry.key }}: {{ entry.value }}
{% endfor %}
```
**Loop special variables:**
Inside any `for` loop, Pebble provides a `loop` object with these properties:
| Variable | Description |
| --- | --- |
| `loop.index` | Zero-based index of the current iteration |
| `loop.length` | Total number of items in the iterable |
| `loop.first` | `true` on the first iteration |
| `loop.last` | `true` on the last iteration |
| `loop.revindex` | Number of iterations remaining |
Example:
```twig
{% for user in users %}
{{ loop.index }}: {{ user.name }}{% if loop.last %} (last){% endif %}
{% endfor %}
```
### `filter`
Applies a filter to a block of content. Filters can be chained:
```twig
{% filter upper %}
hello
{% endfilter %}
{# output: HELLO #}
{% filter lower | title %}
hello world
{% endfilter %}
{# output: Hello World #}
```
### `raw`
Prevents Pebble from parsing its content — useful when you need to output literal `{{ }}` syntax:
```twig
{% raw %}{{ user.name }}{% endraw %}
{# output: {{ user.name }} #}
```
### `macro`
Defines a reusable template snippet. Macros only have access to their own arguments by default:
```twig
{% macro input(type="text", name, value="") %}
type: "{{ type }}", name: "{{ name }}", value: "{{ value }}"
{% endmacro %}
{{ input(name="country") }}
{# output: type: "text", name: "country", value: "" #}
```
To access variables from the outer template context, pass `_context` explicitly:
```twig
{% set foo = "bar" %}
{% macro display(_context) %}
{{ _context.foo }}
{% endmacro %}
{{ display(_context) }}
{# output: bar #}
```
### `block`
Defines a named, reusable template block. Use the `block()` function to render the block elsewhere:
```twig
{% block "header" %}
Introduction
{% endblock %}
{{ block("header") }}
```
## Tests
Tests are used with `is` and `is not` to perform type and value checks.
### `defined`
Checks whether a variable exists in the context (regardless of its value):
```twig
{% if missing is not defined %}
Variable is not defined.
{% endif %}
```
### `empty`
Returns `true` when a variable is null, an empty string, an empty collection, or an empty map:
```twig
{% if user.email is empty %}
No email on record.
{% endif %}
```
### `null`
Checks whether a variable is null:
```twig
{% if user.email is null %}
...
{% endif %}
{% if name is not null %}
...
{% endif %}
```
### `even` and `odd`
Check whether an integer is even or odd:
```twig
{% if 2 is even %}
...
{% endif %}
{% if 3 is odd %}
...
{% endif %}
```
### `iterable`
Returns `true` when a variable implements `java.lang.Iterable`. Use this to guard a `for` loop when the collection may not always be present:
```twig
{% if users is iterable %}
{% for user in users %}
{{ user.name }}
{% endfor %}
{% endif %}
```
### `json`
Returns `true` when a variable is a valid JSON string:
```twig
{% if '{"test": 1}' is json %}
...
{% endif %}
```
### `map`
Returns `true` when a variable is a map:
```twig
{% if {"apple": "red", "banana": "yellow"} is map %}
...
{% endif %}
```
---
# Kestra Glossary: Terms and Definitions
URL: https://kestra.io/docs/glossary
> Glossary of Kestra and declarative orchestration terms. Definitions for flows, tasks, triggers, namespaces, and key concepts used across the platform.
A list of terms useful for understanding Kestra and declarative orchestration.
## A
- [Apps](#apps) - custom user interfaces (UIs) or frontends for workflows, allowing your users to interact with Kestra from the outside world. Apps can trigger workflows or enable human-in-the-loop workflows. Available on [Enterprise Edition](../07.enterprise/04.scalability/apps/index.md).
- [Approval Apps](#approval-apps) - Apps that enable forms for approving or rejecting paused workflows.
- [Form Apps](#form-apps) - Apps that allow you to create forms that can trigger workflows with input parameters.
- [Assets & Lineage](#assets-lineage) - [Assets](../07.enterprise/02.governance/01.assets/index.md) keep a live inventory of the resources your workflows interact with, such as database tables, files, or datasets. Lineage tracks how those assets connect to each other across workflow runs — which execution created or modified each one — and can be shipped to providers like OpenLineage.
## B
- [Backfill](#backfill) - replays of missed schedule intervals between a defined start and end date. All missed schedules are automatically recovered by default if the Kestra server is down. Learn how to manage and configure [backfills](../06.concepts/08.backfill/index.md).
- [Blueprints](#blueprints) - ready-to-use examples with code and documentation designed to kickstart your worflow. [Blueprints](../06.concepts/07.blueprints/index.md) typically include multiple plugins.
## C
- [Concurrency](#concurrency) - a flow-level property that limits the number of executions of a specific flow that can run simultaneously. Learn when to use [concurrency](../05.workflow-components/14.concurrency/index.md).
- [Connector sprawl](#connector-sprawl) - the uncontrolled proliferation of integrations, or connectors, in an organization. [Connector sprawl](https://kestra.io/docs/tutorial/outputs#pass-outputs-between-tasks) can create security, operational, and maintenance issues. Kestra's architecture around outputs and internal storage works to prevent these risks.
- [Context](#context) - typically referred to as "execution context" or a collection of variables and metadata that allows for dynamic rendering of flow properties during a workflow's execution.
- [Control plane](#control-plane) - the centralized layer that orchestrates and manages workflows: the [server components](../08.architecture/02.server-components/index.md) responsible for scheduling, coordinating, and exposing the API and UI (Scheduler, Executor, Webserver). It governs *what* runs and *when*, while [Workers](#workers) execute the actual tasks and handle your data.
- [Cron](#cron) - a standard syntax for expressing recurring schedules as a string of time fields (e.g. `0 9 * * *` for every day at 9am). Kestra uses cron expressions in the [Schedule trigger](../05.workflow-components/07.triggers/01.schedule-trigger/index.md) to run flows at precise times.
## D
- [Declarative](#declarative) - An approach where you describe _what_ a workflow should accomplish rather than _how_ to achieve it or expressesing logic without describing control flow.
- [Declarative orchestration](#declarative-orchestration) - A declarative orchestrator is a system that allows you to define and manage complex workflows using a high-level, descriptive language. Instead of specifying the exact steps and sequences to achieve a specific outcome, a declarative orchestrator lets you define the desired end state and the system figures out how to reach it.
## E
- [Events](#events) - in orchestration, an event is something that happens, internal or external to the system, to start a flow.
- [Internal Events](#internal-events) - internal events happen internal, or inside of the Kestra platform, like scheduled CRON triggers, to start a flow.
- [External Events](#external-events) - external events happen external, or outside of the Kestra platform to start a flow.
- [Execution](#execution) - a single run of a flow, existing in a specific state.
- [Execution context](#execution-context) - a collection of variables and metadata that allows for dynamic rendering of flow properties during a workflow's execution.
- [Expressions](#expressions) - accessing and using variables in flows, combining the Pebble templating engine with the execution context to dynamically render flow properties. [Expressions](../expressions/index.mdx) allow you to dynamically set values within your workflows. Expression syntax uses curly braces, e.g., `{{ your_expression }}`.
## F
- [Flowable Tasks](#flowable-tasks) - [Flowable tasks](../05.workflow-components/01.tasks/00.flowable-tasks/index.md) control orchestration logic — running tasks or subflows in parallel, creating loops, and handling conditional branching. They do not run heavy operations.
- [Flows](#flows) - Flows act as a backend, processing data and executing tasks. Flows are versioned by default. [Flows](../05.workflow-components/01.flow/index.md) and workflows are often used interchangeable.
## I
- [Inputs](#inputs) - dynamic values passed to the flow at runtime. Flow inputs are stored in the execution context and accessed with `{{ inputs.parameter_name }}`.Learn more about [inputs](../05.workflow-components/05.inputs/index.md).
- [Instance](#instance) - a single deployment of Kestra, whether self-hosted or running on Kestra Cloud. On Enterprise Edition, the [Instance](../07.enterprise/05.instance/index.mdx) menu provides a centralized view of the deployment's health, upgrades, maintenance mode, and global settings.
## K
- [KV Store](#kv-store) - also known as [Key Value Store](../06.concepts/05.kv-store/index.md), allows you to store any data in a key-value format. These values can be shared acrss executions and different workflows to provide persistent data.
## N
- [Namespace](#namespace) - separates projects, teams, and environments to logically group things and provide structure. Working with languages like Java, you may have encountered the concept of [namespaces](../05.workflow-components/02.namespace/index.md) implemented as packages.
- [Namespace File](#namespace-file) - files tied to a specific namespace, serving as project assets. They are analogous to a project in a local IDE or a copy of a Git repository. Learn more about [namespace files](../06.concepts/02.namespace-files/index.md).
## O
- [Orchestration](#orchestration) - a process or a tool that automates, manages, and coordinates various workflows and tasks across different services, systems, or applications. It functions like a conductor of an orchestra, ensuring all components perform in harmony, following a predefined sequence or set of rules.
- [Outputs](#outputs) - a mechanism to pass data between tasks and flows. They can be accessed by all downstream tasks and flows using dynamic properties (e.g., `{{ outputs.task_id.attribute_name }}`). Learn more about [outputs](../05.workflow-components/06.outputs/index.md).
## P
- [Pebble Templating Engine](#pebble-templating-engine) - inspired by the Java templating engine, use `.` notation to access nested properties. [Pebble](../06.concepts/06.pebble/index.md) is used to dynamically render variables, inputs, and outputs withint the execution context.
- [Plugin](#plugin) - the building blocks of tasks in Kestra that offer integerations to different systems and functionality. [Plugins](../05.workflow-components/02.plugins/index.md) power every task and trigger in Kestra.
## R
- [Replay](#replay) - re-run a workflow execution from any chosen task, useful for iterative developer and reprocessing data. Learn more about [replay](../06.concepts/10.replay/index.md).
- [Revision](#revision) - any changes to a flow create a new version of that flow, otherwise known as a [revision](../06.concepts/03.revision/index.md).
- [Runnable Tasks](#runnable-tasks) - [Runnable tasks](../05.workflow-components/01.tasks/01.runnable-tasks/index.md) handle data processing, such as file system operations, API calls, and database queries. They can be compute-intensive and are executed by workers. Most tasks are runnable.
## S
- [SCIM](#scim) - System for Cross-domain Identity Management, an open standard for automating the provisioning of users and groups. Kestra uses [SCIM directory sync](../07.enterprise/03.auth/scim/index.mdx) to synchronize users and groups from identity providers like Okta, Azure AD, and Keycloak.
- [Secrets](#secrets) - sensitive information stored securely. [Secrets](../06.concepts/04.secret/index.md) can be retrieved and used within Kestra flows using the `secret()` function (e.g., `{{ secret('API_TOKEN') }}`).
- [Sibling task](#sibling-task) - A sibling task is a task that shares a common parent task with other tasks, like in the `tasks` list inside a loop.
- [Subflow](#subflow) - Subflows let you build modular and reusable workflow components. They work like function calls: executing a [subflow](../05.workflow-components/10.subflows/index.md) creates a new flow run from within another flow.
- [System flows](#system-flows) - System flows automate maintenance workflows. Any valid Kestra flow can become a [System Flow](../06.concepts/system-flows/index.md) if it’s added to the `system` namespace.
## T
- [Task runner](#task-runner) - extensible, pluggable system within Kestra capable of executing your tasks in arbitrary remote environments, to offload computationally intensive tasks. Learn more about [task runners](../task-runners/01.overview/index.md).
- [Tasks](#tasks) - atomic actions in a flow. [Tasks](../05.workflow-components/01.tasks/index.mdx) are a required element in a flow and can be [Flowable Tasks](#flowable-tasks) or [Runnable Tasks](#runnable-tasks).
- [Time To Live (TTL)](#ttl) - the expiration or duration something like a token, secret, or key-value pair is available.
- [Triggers](#triggers) - a mechanism that automatically starts the execution of a flow. There are five core trigger types: schedule, flow, webhook, polling, realtime. [Triggers](../05.workflow-components/07.triggers/index.mdx) are scheduled or event-based.
## W
- [Worker group](#worker-group) - offload computer-intensive tasks to dedicated workers, but at a broader scope than task runners. Available in [Enterprise Edition](../07.enterprise/04.scalability/worker-group/index.md).
- [Workers](#workers) - a Kestra server component responsible for executing all runnable tasks and polling triggers.
## Y
- [YAML](#yaml) - a human-readable data serialization language used to define Kestra flows. Its declarative, indentation-based syntax describes the tasks, triggers, and properties of a workflow without requiring code. Learn more at [yaml.org](https://yaml.org).
---
# Kestra How-to Guides: Hands-On Workflow Tutorials
URL: https://kestra.io/docs/how-to-guides
> Explore our collection of hands-on guides to learn how to integrate tools, manage workflows, and master Kestra's features.
import GuidesChildCard from "~/components/docs/GuidesChildCard.astro"
Learn Kestra with our hands-on guides.
---
# Access Local Files in Kestra: Bind Mounts Guide
URL: https://kestra.io/docs/how-to-guides/access-local-files
> Access files stored on your local machine within Kestra workflows using bind mounts, and batch-upload files to the local filesystem using the local.Uploads task.
Access locally stored files on your machine inside Kestra workflows.
In Kestra, you can access files stored on your local machine from within your flows.
This is useful when you have a directory of files to process or scripts to execute without needing to copy them into Kestra.
## Setting up Kestra with Docker
If you're running Kestra with [Docker](../../02.installation/02.docker/index.md), you’ll need to create a bind mount to a local directory on your machine so that Kestra can access those files inside the container.
In your [Docker Compose](../../02.installation/03.docker-compose/index.md) file, add the absolute path of the local directory and define its mount point inside the container.
In this example, the local path `/Users/username/Documents/files` is mounted to `/files` inside the container using `- /Users/username/Documents/files:/files`.
Add this under the `volumes` section of your Docker Compose file:
```yaml
...
kestra:
image: kestra/kestra:latest
pull_policy: always
user: "root"
command: server standalone
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/kestra-wd:/tmp/kestra-wd
- /Users/username/Documents/files:/files
...
```
You can now access any files or directories within `/Users/username/Documents/files` from inside Kestra under the `/files` path.
## Accessing files inside Script tasks
By default, a Script task runs inside a [Docker Task Runner](../../task-runners/04.types/02.docker-task-runner/index.md).
To access local files, change the Task Runner type to [Process](../../task-runners/04.types/01.process-task-runner/index.md), so it runs as a subprocess on your Kestra instance:
```yaml
id: process
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- cat /files/myfile.txt
```
## Batch-uploading files with `local.Uploads`
[`io.kestra.plugin.fs.local.Uploads`](/plugins/plugin-fs/local/io.kestra.plugin.fs.local.uploads) writes multiple Kestra internal storage files to a directory on the local filesystem in a single task. It mirrors the `Uploads` task available on the FTP, FTPS, SFTP, and SMB backends.
### Configure allowed paths
Both [`local.Upload`](/plugins/plugin-fs/local/io.kestra.plugin.fs.local.upload) (single file) and `local.Uploads` (batch) require the destination directory to be listed in the plugin's `allowed-paths` configuration. Add the following to your `kestra.yml`:
```yaml
kestra:
plugins:
configurations:
- type: io.kestra.plugin.fs.local.Uploads
values:
allowed-paths:
- /data/uploads
- type: io.kestra.plugin.fs.local.Upload
values:
allowed-paths:
- /data/uploads
```
Without this, any write to `/data/uploads` is rejected with a `SecurityException` even if the path is bind-mounted into the container.
### Upload a list of files
Pass a list of Kestra internal storage URIs to `from`. Each file is written to the `to` directory using its original filename.
The flow below runs a data ingestion job that produces run logs and SQL migration scripts, then archives the logs to a local directory:
```yaml
id: archive_pipeline_logs
namespace: company.team
tasks:
- id: run_pipeline
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
outputFiles:
- "*.log"
- "*.sql"
commands:
- echo "ingested 1024 rows" > ingest.log
- echo "0 errors" > errors.log
- echo "ALTER TABLE orders ADD COLUMN status TEXT;" > schema.sql
- echo "INSERT INTO orders VALUES (1, 'pending');" > seed.sql
- id: upload_logs
type: io.kestra.plugin.fs.local.Uploads
from:
- "{{ outputs.run_pipeline.outputFiles['ingest.log'] }}"
- "{{ outputs.run_pipeline.outputFiles['errors.log'] }}"
to: /data/uploads/logs
```
### Upload with custom destination filenames
To rename files at the destination, pass a map of `destinationFilename: sourceURI` pairs instead of a list. This is useful for versioning — for example, tagging migration scripts with a version prefix before archiving them.
In the flow above, replace the `upload_logs` task with:
```yaml
- id: upload_migrations
type: io.kestra.plugin.fs.local.Uploads
from:
v1_schema.sql: "{{ outputs.run_pipeline.outputFiles['schema.sql'] }}"
v1_seed.sql: "{{ outputs.run_pipeline.outputFiles['seed.sql'] }}"
to: /data/uploads/migrations
```
### Filter by regular expression
Use `regExp` to upload only files whose internal storage URI matches a pattern. Files that do not match are skipped.
When a task produces a mixed set of outputs, `regExp` lets you route file types to separate destinations without splitting the upstream task. In the flow above, replace the `upload_logs` task with:
```yaml
- id: upload_sql_only
type: io.kestra.plugin.fs.local.Uploads
from:
- "{{ outputs.run_pipeline.outputFiles['ingest.log'] }}"
- "{{ outputs.run_pipeline.outputFiles['errors.log'] }}"
- "{{ outputs.run_pipeline.outputFiles['schema.sql'] }}"
- "{{ outputs.run_pipeline.outputFiles['seed.sql'] }}"
regExp: ".*\\.sql$"
to: /data/uploads/migrations
```
### Additional properties
| Property | Default | Description |
|---|---|---|
| `maxFiles` | `25` | Upper bound on how many files are written. Excess files are dropped with a warning. |
| `overwrite` | `true` | When `false`, the task fails if a destination file already exists. |
---
# Configure Alerts in Kestra
URL: https://kestra.io/docs/how-to-guides/alerting
> Configure alerts in Kestra to notify you of workflow failures via Slack, PagerDuty, or other platforms using subflows and flow triggers.
Configure alerts that fire whenever a workflow fails.
Alerting is essential to keeping production systems reliable. Kestra makes it easy with multiple ways to attach alerts to workflows so you always know what’s happening.
Kestra’s Notifications plugin group ships tasks for popular platforms such as Slack, Teams, and PagerDuty, making it straightforward to configure alerts directly inside workflows.

Each platform exposes two task types:
- **Execution task** – sends execution metadata directly in the notification, including a link to the run, ID, namespace, flow name, start time, duration, and final status.
- **Send task** – sends a custom message, useful when you want to describe the state of a specific task or output.
For this walkthrough we’ll use the `SlackExecution` task to send a detailed execution summary.
## `errors` property
If we add the task directly to a workflow, it runs every time — which isn’t useful. Instead, place it in the `errors` block so it only fires when the execution fails.
Just like the `tasks` block, define `SlackExecution` under `errors`:
```yaml
errors:
- id: alert
type: io.kestra.plugin.slack.notifications.SlackExecution
channel: "#general"
url: "{{ secret('SLACK_WEBHOOK') }}"
```
When executed, it looks like this in Slack:

Use `errors` when you only want failure alerts. If you need different notifications for different final states such as `SUCCESS`, `FAILED`, or `WARNING`, use [`afterExecution`](../../05.workflow-components/20.afterexecution/index.md) instead.
## Subflows
Copying that snippet into every flow is repetitive and hard to maintain. Instead, move the alerting logic into a subflow and reference it from any workflow that needs alerts.
Move the `errors` tasks into their own subflow so the `errors` block only calls that subflow. Update the alert logic once and every consumer benefits.
Subflow containing the alert logic:
```yaml
id: slack_alert
namespace: system
tasks:
- id: alert
type: io.kestra.plugin.slack.notifications.SlackExecution
channel: "#general"
url: "{{ secret('SLACK_WEBHOOK') }}"
```
Parent flow that calls the subflow only when an error occurs:
```yaml
errors:
- id: alert
type: io.kestra.plugin.core.flow.Subflow
flowId: slack_alert
namespace: system
```
## Flow trigger
Subflows cut down on duplication, but you still need the `errors` block in every flow. For a fully centralized approach, use a **Flow trigger** that reacts to execution status. The `when` expression lets you target specific states, such as `FAILED` or `WARNING`, and you can define separate triggers per status if needed.
```yaml
id: failure_alert_slack
namespace: system
tasks:
- id: send_alert
type: io.kestra.plugin.slack.notifications.SlackExecution
url: "{{ secret('SLACK_WEBHOOK') }}"
channel: "#general"
executionId: "{{ trigger.executionId }}"
triggers:
- id: on_failure
type: io.kestra.plugin.core.trigger.Flow
dependsOn:
- states: [FAILED, WARNING]
```
With multiple options for automatic alerting in Kestra, you can choose the level of centralization and customization that fits each use case.
---
# Audit Machines and Tool Versions with Ansible in Kestra
URL: https://kestra.io/docs/how-to-guides/ansible
> Use Ansible playbooks orchestrated by Kestra to audit machine resources, check tool versions, and automate infrastructure updates.
Run Ansible playbooks from Kestra and coordinate downstream infrastructure tasks.
Ansible is an agentless automation tool that uses YAML playbooks to describe desired state and apply it over SSH or APIs. Teams rely on it to install software, manage configs, update systems, and provision cloud infrastructure.
## System report playbook (cross-platform)
This playbook audits a host without assuming the OS, captures diagnostics, and upgrades `python3` when needed using the appropriate package manager (`apt`, `yum`, or Homebrew). It writes a JSON report to `./system_info.json`. You can extend the same pattern to a real-world fleet by adding an inventory of servers or laptops, running over SSH instead of `localhost`, and inserting more version/presence checks for tools your team depends on (e.g., `node`, `aws`, `kubectl`). In multi-machine mode, facts and JSON outputs can be aggregated centrally to spot drift and trigger remediations.
:::collapse{title="View the playbook"}
```yaml
---
- name: Collect and report system information (system agnostic)
hosts: localhost
connection: local
gather_facts: true
vars:
system_info_output: "./system_info.json"
python3_min_version: "3.11.0"
tasks:
- name: Show basic system summary
ansible.builtin.debug:
msg:
- "Hostname: {{ ansible_facts['hostname'] | default('unknown') }}"
- "OS family: {{ ansible_facts['os_family'] | default('unknown') }}"
- "Distribution: {{ ansible_facts['distribution'] | default('') }} {{ ansible_facts['distribution_version'] | default('') }}"
- "Kernel: {{ ansible_facts['kernel'] | default('unknown') }}"
- "Architecture: {{ ansible_facts['architecture'] | default('unknown') }}"
- "CPU(s): {{ ansible_facts['processor_vcpus'] | default('unknown') }}"
- "Total RAM (MB): {{ ansible_facts['memtotal_mb'] | default('unknown') }}"
- "Primary IP: {{ ansible_facts['default_ipv4']['address'] | default('unknown') }}"
# -----------------------------
# Extra checks / diagnostics
# -----------------------------
- name: Check overall disk usage (df -h)
ansible.builtin.command: df -h
register: disk_usage
changed_when: false
failed_when: false # in case df is not available
- name: Check load average and uptime
ansible.builtin.command: uptime
register: uptime_cmd
changed_when: false
failed_when: false
- name: Show top 5 memory-hungry processes
ansible.builtin.shell: |
ps aux | head -n 1
ps aux | sort -nrk 4 | head -n 5
register: top_mem_processes
changed_when: false
failed_when: false
# -----------------------------
# Python3 detection & version
# -----------------------------
- name: Check if python3 is installed
ansible.builtin.command: python3 --version
register: python3_check
failed_when: false
changed_when: false
- name: Parse python3 version
ansible.builtin.set_fact:
python3_installed: "{{ python3_check.rc == 0 }}"
python3_version: >-
{{
(python3_check.stdout.split()[1])
if (python3_check.rc == 0 and (python3_check.stdout | length > 0))
else 'unknown'
}}
- name: Debug python3 detection
ansible.builtin.debug:
msg:
- "python3 installed: {{ python3_installed }}"
- "python3 version: {{ python3_version }}"
# -----------------------------
# OS family convenience flags
# -----------------------------
- name: Set OS family flags
ansible.builtin.set_fact:
os_family: "{{ ansible_facts['os_family'] | default('Unknown') }}"
is_debian: "{{ ansible_facts['os_family'] == 'Debian' }}"
is_redhat: "{{ ansible_facts['os_family'] == 'RedHat' }}"
is_darwin: "{{ ansible_facts['os_family'] == 'Darwin' }}"
# -----------------------------
# Decide if python3 upgrade is needed
# -----------------------------
- name: Decide if python3 upgrade is needed
ansible.builtin.set_fact:
python3_needs_upgrade: >-
{{
python3_installed
and python3_version != 'unknown'
and (python3_version is version(python3_min_version, '<'))
}}
- name: Debug python3 upgrade decision
ansible.builtin.debug:
msg:
- "Minimum required python3 version: {{ python3_min_version }}"
- "Current python3 version: {{ python3_version }}"
- "Needs upgrade: {{ python3_needs_upgrade }}"
- name: Initialize python3 upgrade result
ansible.builtin.set_fact:
python3_upgrade_result:
manager: "none"
attempted: false
note: "No upgrade attempted yet."
# -----------------------------
# Debian / Ubuntu path (apt)
# -----------------------------
- name: Upgrade python3 via apt if needed (Debian family)
ansible.builtin.apt:
name: python3
state: latest
update_cache: yes
when:
- is_debian
- python3_needs_upgrade
register: python3_upgrade_apt
- name: Record python3 upgrade result for Debian family
ansible.builtin.set_fact:
python3_upgrade_result: >-
{{
python3_upgrade_result | combine(
{
'manager': 'apt',
'attempted': python3_needs_upgrade,
'note': (
python3_needs_upgrade
| ternary(
'python3 upgrade handled by apt on Debian-based system (see play output).',
'python3 already meets minimum version; apt upgrade not required.'
)
)
},
recursive=True
)
}}
when: is_debian
# -----------------------------
# RedHat / CentOS / Fedora path (yum)
# -----------------------------
- name: Upgrade python3 via yum if needed (RedHat family)
ansible.builtin.yum:
name: python3
state: latest
when:
- is_redhat
- python3_needs_upgrade
register: python3_upgrade_yum
- name: Record python3 upgrade result for RedHat family
ansible.builtin.set_fact:
python3_upgrade_result: >-
{{
python3_upgrade_result | combine(
{
'manager': 'yum',
'attempted': python3_needs_upgrade,
'note': (
python3_needs_upgrade
| ternary(
'python3 upgrade handled by yum on RedHat-based system (see play output).',
'python3 already meets minimum version; yum upgrade not required.'
)
)
},
recursive=True
)
}}
when: is_redhat
# -----------------------------
# macOS path (Homebrew)
# -----------------------------
- name: Check if Homebrew is installed (macOS)
ansible.builtin.command: brew --version
register: brew_check
failed_when: false
changed_when: false
when: is_darwin
- name: Upgrade python via Homebrew if needed (macOS)
ansible.builtin.command: brew upgrade python
when:
- is_darwin
- python3_needs_upgrade
- brew_check.rc == 0
register: python3_upgrade_brew
changed_when: true
- name: Record python3 upgrade result for macOS
ansible.builtin.set_fact:
python3_upgrade_result: >-
{{
python3_upgrade_result | combine(
{
'manager': (brew_check.rc == 0) | ternary('brew', 'none'),
'attempted': (python3_needs_upgrade and brew_check.rc == 0),
'note': (
(not python3_needs_upgrade)
| ternary(
'python3 already meets minimum version; brew upgrade not required.',
(
brew_check.rc == 0
| ternary(
'python upgrade handled by Homebrew on macOS (see play output).',
'Homebrew not available; cannot upgrade python on macOS.'
)
)
)
)
},
recursive=True
)
}}
when: is_darwin
# -----------------------------
# Build & write combined report
# -----------------------------
- name: Build combined system info structure
ansible.builtin.set_fact:
full_system_info:
collected_at: "{{ ansible_facts['date_time']['iso8601'] | default('') }}"
hostname: "{{ ansible_facts['hostname'] | default('') }}"
os:
family: "{{ ansible_facts['os_family'] | default('') }}"
distribution: "{{ ansible_facts['distribution'] | default('') }}"
version: "{{ ansible_facts['distribution_version'] | default('') }}"
release: "{{ ansible_facts['distribution_release'] | default('') }}"
kernel: "{{ ansible_facts['kernel'] | default('') }}"
hardware:
architecture: "{{ ansible_facts['architecture'] | default('') }}"
cpu_model: "{{ ansible_facts['processor'][1] | default('') if ansible_facts.get('processor') else '' }}"
vcpus: "{{ ansible_facts['processor_vcpus'] | default(0) }}"
memtotal_mb: "{{ ansible_facts['memtotal_mb'] | default(0) }}"
network:
default_ipv4: "{{ ansible_facts['default_ipv4'] | default({}) }}"
all_ipv4: "{{ ansible_facts['all_ipv4_addresses'] | default([]) }}"
interfaces: "{{ ansible_facts['interfaces'] | default([]) }}"
storage:
mounts: "{{ ansible_facts['mounts'] | default([]) }}"
virtualization:
type: "{{ ansible_facts['virtualization_type'] | default('') }}"
role: "{{ ansible_facts['virtualization_role'] | default('') }}"
diagnostics:
disk_usage: "{{ disk_usage.stdout | default('') }}"
uptime: "{{ uptime_cmd.stdout | default('') }}"
top_mem_processes: "{{ top_mem_processes.stdout | default('') }}"
python3:
installed: "{{ python3_installed }}"
version: "{{ python3_version }}"
minimum_required: "{{ python3_min_version }}"
needs_upgrade: "{{ python3_needs_upgrade }}"
upgrade: "{{ python3_upgrade_result }}"
ansible_facts: "{{ ansible_facts }}"
- name: Write full system info to JSON file
ansible.builtin.copy:
dest: "{{ system_info_output }}"
content: "{{ full_system_info | to_nice_json }}"
mode: "0600"
- name: Print location of saved system info
ansible.builtin.debug:
msg:
- "Full system information written to: {{ system_info_output }}"
- "You can inspect it with: jq '.' {{ system_info_output }} (if jq is installed)"
```
:::
### What this playbook covers
It gathers the usual suspects (OS family, distro, kernel, CPU, RAM, IP), then pulls quick diagnostics like disk usage, uptime, and top memory processes. It checks `python3` and, if it's older than `3.11.0`, upgrades it with the right package manager depending on the OS of the machine (`apt`, `yum`, or Homebrew).
Each play in the playbook generates a log and output, for example, there is a log for each diagnostic metric check, a log for Python3 detection and version, and a log for building and writing the combined report to name a few.
The image below shows an example output targeting a local machine where `python3` is installed (`python3_installed`), but the Python version is `"3.10.4"`.

Ansible also reports that `"python3_needs_upgrade": true` and depending on the detected OS of the machine, upgrades accordingly.

Everything from this Python upgrade to other machine diagnostics are aggregated in `system_info.json` with mode `0600` so you have a tidy, readable report. This playbook can of course be adapted for other checks and in principle demonstrates the possibilities when you combine Ansible with Kestra.
### Run it locally
Ensure Ansible is installed and save the YAML as `system_info.yml`, run it against localhost, and inspect the output:
- `ansible-playbook -i localhost, -c local system_info.yml`
- Optionally inspect the JSON: `jq '.' system_info.json`
The diagnostics report captured looks like the following (macOS):
```json
{
"diagnostics": {
"disk_usage": [
"Filesystem Size Used Avail Capacity iused ifree %iused Mounted on",
"/dev/disk3s1 466Gi 128Gi 318Gi 29% 1453290 4882459910 0% /"
],
"uptime": "18:42 up 5 days, 7:31, 4 users, load averages: 2.34 2.11 1.98",
"top_mem_processes": [
"USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND",
"jdoe 4287 23.5 9.8 9876544 823456 ?? R 9:12PM 0:21.43 /Applications/Chrome",
"jdoe 1562 7.3 5.4 6453320 455121 ?? S 7:58AM 12:11.01 /usr/bin/python3 myscript.py",
"_windowser 991 3.8 3.8 5432100 315789 ?? S Fri11AM 5:45.22 WindowServer",
"root 72 1.2 2.2 4321000 190233 ?? S Sun09AM 3:12.90 /usr/libexec/trustd",
"jdoe 2178 0.9 1.6 3876543 131442 ?? S Sat08PM 1:03.07 Slack"
]
}
}
```
And the machine information outputs the follwing for local macOS machine:
```plaintext
TASK [Show basic system summary] *************************************************************************************************************************************
ok: [localhost] => {
"msg": [
"Hostname: Mac",
"OS: Darwin MacOSX 15.6.1",
"Kernel: 24.6.0",
"Architecture: arm64",
"CPU(s): 10",
"Total RAM (MB): 24576",
"Primary IP: 10.0.0.42"
]
}
```
### Run it from Kestra
Embed the playbook in your flow's YAML inline, and collect the report with a single [Ansible CLI task](/plugins/plugin-ansible/io.kestra.plugin.ansible.cli.ansiblecli):
```yaml
id: system_report
namespace: company.team
tasks:
- id: system_info
type: io.kestra.plugin.ansible.cli.AnsibleCLI
inputFiles:
playbook.yml: |
# paste the playbook above
inventory.ini: |
localhost ansible_connection=local
outputFiles:
- system_info.json
containerImage: cytopia/ansible:latest-tools
commands:
- ansible-playbook -i inventory.ini playbook.yml
```
Or, keep the playbook as a [Namespace File](../../06.concepts/02.namespace-files/index.md) and reference it directly with the same [Ansible CLI task](/plugins/plugin-ansible/io.kestra.plugin.ansible.cli.ansiblecli).

Also add the `inventory.ini` file to the Namespace (`localhost ansible_connection=local`). For simplicity, this guide checks the local machine, but of course this example can be expanded to utilize Ansible's capability to SSH into multiple servers and perform operations:
```yaml
id: system_report
namespace: company.team
tasks:
- id: system_info
type: io.kestra.plugin.ansible.cli.AnsibleCLI
namespaceFiles:
enabled: true
outputFiles:
- system_info.json
containerImage: cytopia/ansible:latest-tools
commands:
- ansible-playbook -i inventory.ini system_info.yml
```
After the run, the `outputFiles` property allows you to preview or download `system_info.json` from the task outputs and feed it into downstream checks or dashboards.

### Upload the report to S3
Extend the Namespace File flow with an [S3 Upload task](/plugins/plugin-aws/aws-s3/io.kestra.plugin.aws.s3.upload) and store credentials in [secrets](../../06.concepts/04.secret/index.md):
```yaml
id: system_report_to_s3
namespace: company.team
tasks:
- id: system_info
type: io.kestra.plugin.ansible.cli.AnsibleCLI
namespaceFiles:
enabled: true
outputFiles:
- system_info.json
containerImage: cytopia/ansible:latest-tools
commands:
- ansible-playbook -i inventory.ini system_info.yml
- id: upload_output_to_s3
type: io.kestra.plugin.aws.s3.Upload
region: "{{ secret('AWS_DEFAULT_REGION') }}"
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
bucket: "{{ secret('S3_BUCKET_NAME') }}"
key: "system_reports/{{ execution.id }}/system_info.json"
from: "{{ outputs.system_info.outputFiles['system_info.json'] }}"
```

The `upload_output_to_s3` task pushes the generated JSON to S3 using secrets for credentials and bucket name; reuse `outputFiles` expressions anywhere you need the file.
### Add a Slack notification
To include a separate notification to the relevant channels, add the [Slack Incoming Webhook task](/plugins/plugin-slack/slack-notifications/io.kestra.plugin.slack.notifications.slackincomingwebhook) after the upload with a message alerting that "Machine X" had outdated software and patched an upgrade. You can swap Slack for any other notifier in the Plugin catalog or chain multiple notifications if needed:
```yaml
- id: slack_notification
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
url: "{{ secret('SLACK_WEBHOOK_URL') }}"
messageText: "Machine `{{ flow.id }}` had outdated Python and an upgrade took place during execution `{{ execution.id }}`. Report available at S3: `{{ outputs.upload_output_to_s3.key }}`"
```

### Trigger it (scheduled or event-driven)
Lastly, add a trigger so the flow runs automatically — either on a schedule ([Schedule trigger](../../05.workflow-components/07.triggers/01.schedule-trigger/index.md)) or from an external event ([Webhook trigger](../../05.workflow-components/07.triggers/03.webhook-trigger/index.md)):
```yaml
triggers:
- id: nightly_audit
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 2 * * *" # every night at 2 AM
# Or, event-driven example (e.g., HTTP webhook from your MDM/ITSM):
# - id: mdm_webhook
# type: io.kestra.plugin.core.http.Webhook
```
A trigger allows you to build a historical log of machine health in S3 and Slack without manual runs.
### Wrap up
Ansible handles host-level automation — collecting facts, checking software package versions, remediating with the right package manager, and so much more. Kestra now orchestrates the run, stores secrets, uploads the JSON report to S3, and notifies Slack (or your preferred channel) so teams see when upgrades occur. Together they scale this cross-platform playbook from one laptop to a fleet, with repeatable runs and downstream integrations ready to consume the results.
If your playbooks handle sensitive data such as credentials or connection strings, see [Declare Explicit Outputs in Ansible Playbooks](../ansible-explicit-outputs/index.md) to control exactly what appears in `vars.outputs`.
---
# Detect Ansible Config Drift with Kestra
URL: https://kestra.io/docs/how-to-guides/ansible-config-drift
> Detect configuration drift across your infrastructure using Ansible and Kestra, and get alerted via Slack on changes.
Keeps configs consistent and surfaces drift without manual checks with Ansible and Kestra.
Use Ansible to enforce a required environment variable across multiple hosts and have Kestra alert you in Slack when a change occurs.
## Files to store as Namespace Files
Ansible expects two file types: an `inventory.ini` and a `playbook.yml`. To use with Kestra, they can either be stored as [Namespace Files](../../06.concepts/02.namespace-files/index.md) or written in-line in the flow code. The example continues using Namespace Files.
- `inventory.ini` (replace with your hosts and users; keys shown as placeholders):
```ini
[servers]
server1.example.test ansible_user=admin ansible_ssh_private_key_file=~/.ssh/id_rsa
server2.example.test ansible_user=admin ansible_ssh_private_key_file=~/.ssh/id_rsa
server3.example.test ansible_user=admin ansible_ssh_private_key_file=~/.ssh/id_rsa
```
- `myplaybook.yml` (enforce `MY_APP_MODE` and refresh the shell):
```yaml
---
- name: Ensure environment variable is set correctly
hosts: servers
become: true
tasks:
- name: Ensure MY_APP_MODE is set
lineinfile:
path: /home/{{ ansible_user }}/.bashrc
regexp: '^MY_APP_MODE='
line: 'MY_APP_MODE=production'
state: present
notify: Refresh environment
handlers:
- name: Refresh environment
shell: . /home/{{ ansible_user }}/.bashrc
changed_when: false
```
## Flow: run Ansible and alert on drift
This flow runs the playbook with the [Ansible CLI task](/plugins/plugin-ansible/io.kestra.plugin.ansible.cli.ansiblecli), inspects each host result in a [`Loop`](/plugins/core/flow/io.kestra.plugin.core.flow.loop), and posts a Slack alert only when a host was changed using the [Slack Incoming Webhook task](/plugins/plugin-slack/slack-notifications/io.kestra.plugin.slack.notifications.slackincomingwebhook). The schedule trigger is disabled by default — enable it to run nightly.
```yaml
id: ansible_config_drift
namespace: company.team
tasks:
- id: set_up_env
type: io.kestra.plugin.ansible.cli.AnsibleCLI
namespaceFiles:
enabled: true
taskRunner:
type: io.kestra.plugin.core.runner.Process
ansibleConfig: |
[defaults]
interpreter_python = auto_silent
log_path={{ workingDir }}/log
callback_plugins = ./callback_plugins
stdout_callback = kestra_logger
commands:
- ansible-playbook -i inventory.ini myplaybook.yml
- id: loop_hosts
type: io.kestra.plugin.core.flow.Loop
values: "{{ outputs.set_up_env.vars.outputs }}"
tasks:
- id: check_drift
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
runIf: "{{ item.value | jq('.changed') | first == true }}"
url: "{{ secret('SLACK_WEBHOOK') }}"
payload: |
{
"text": "Configuration updated - {{ item.value | jq('.msg') | first ?? Null }}"
}
triggers:
- id: check_nightly
type: io.kestra.plugin.core.trigger.Schedule
cron: 0 3 * * *
disabled: true
```
The execution generates logs for every play in the playbook for clear results and monitoring:

In the execution outputs, you can examine results and debug expressions to use in potential downstream tasks or subflows:

## Why this matters
This pattern enforces a critical env var across a fleet to catch drift quickly, streams Ansible output in structured form via `stdout_callback = kestra_logger`, and alerts only on changed hosts to keep Slack noise low. Keeping the playbook and inventory as Namespace Files means you can version and reuse them across flows without hardcoding paths in each run.
You can expand this pattern to check multiple config files, package versions, or CIS controls per host, while letting Kestra handle scheduling, secrets, notifications, and downstream tasks (tickets, S3 archiving, SIEM) so Ansible remediation and orchestration stay tightly linked.
---
# Declare Explicit Outputs in Ansible Playbooks with Kestra
URL: https://kestra.io/docs/how-to-guides/ansible-explicit-outputs
> Use the bundled kestra Ansible module and outputsMode EXPLICIT to declare only the outputs you need, keeping sensitive per-host data out of Kestra's output store.
By default, the AnsibleCLI task captures the full result payload from every Ansible module on every host, including `module_args` and raw return values. If your playbooks handle credentials, connection strings, or host-specific secrets, that raw output lands in Kestra's internal storage where it is visible to anyone with execution access.
Setting `outputsMode: EXPLICIT` changes this behavior. Only the values you declare with the bundled `kestra` Ansible module appear in `vars.outputs`. Everything else in `vars.playbooks` is reduced to a status flag (`changed: true/false`), preserving play-level observability without leaking host data.
## How it works
When `outputsMode: EXPLICIT` is set, the AnsibleCLI task passes `KESTRA_OUTPUTS_MODE=explicit` to the `kestra_logger` callback plugin. The callback then:
1. Collects outputs only from `kestra` module calls — `kestra: outputs: { key: value }` in your playbook.
2. Merges all declared outputs into a single flat dict (last write wins across hosts and module calls).
3. Replaces each host's full result payload in `vars.playbooks` with `{"changed": }`. Failures and unreachable hosts also retain `msg`.
4. Preserves task status (ok, failed, skipped, unreachable) so you can still inspect play health.
The `kestra` module is bundled with the plugin and injected automatically — no extra installation is needed.
## Enable EXPLICIT mode
Add `outputsMode: EXPLICIT` to your `AnsibleCLI` task:
```yaml
id: deploy
namespace: company.infra
tasks:
- id: run_playbook
type: io.kestra.plugin.ansible.cli.AnsibleCLI
outputsMode: EXPLICIT
containerImage: cytopia/ansible:latest-tools
inputFiles:
hosts: |
[webservers]
web-01
web-02
playbook.yml: |
- hosts: webservers
tasks:
- name: Gather app version
shell: cat /opt/app/VERSION
register: version_output
- name: Declare output
kestra:
outputs:
app_version: "{{ version_output.stdout }}"
commands:
- ansible-playbook -i hosts playbook.yml
```
:::alert{type="info"}
Playbooks are passed via `inputFiles`. If your playbook contains Ansible Jinja expressions like `{{ variable }}`, store the playbook content in `variables:` instead to prevent Kestra from evaluating those expressions before Ansible sees them. See [Avoiding expression conflicts](#avoiding-expression-conflicts).
:::
## Declare outputs with the kestra module
Use `kestra: outputs: { key: value }` anywhere in your playbook tasks. The module name is `kestra` (also resolvable as `ansible.legacy.kestra`).
```yaml
- name: Declare database connection info
kestra:
outputs:
db_host: "{{ hostvars[inventory_hostname]['db_host'] }}"
schema_version: "{{ schema_version }}"
```
The module always sets `changed: false`. It is safe to run in check mode.
### Avoid using loop
Do not call the `kestra` module inside an Ansible `loop:`. Ansible aggregates loop results under a `results` key, which the callback cannot parse for output collection. A warning is emitted and declared outputs are silently skipped for that task.
```yaml
# WRONG — outputs will not be collected
- name: Declare per-item outputs
kestra:
outputs:
item: "{{ item }}"
loop: [a, b, c]
# CORRECT — call the module once with all values
- name: Declare outputs
kestra:
outputs:
items: [a, b, c]
```
## Access outputs downstream
Declared outputs are available under `vars.outputs` exactly as in `ALL` mode:
```yaml
- id: next_task
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.run_playbook.vars.outputs.app_version }}"
```
Where `run_playbook` is the `id` of your `AnsibleCLI` task and `app_version` is the key you declared.
## Output structure: ALL vs EXPLICIT
The shape of `vars.outputs` differs between the two modes.
**ALL mode** (default): a list of per-host result dicts — each entry contains the full Ansible result object for that host.
```json
[
{
"host": "web-01",
"ok": {
"changed": true,
"stdout": "1.4.2",
"module_args": { ... }
}
}
]
```
**EXPLICIT mode**: a flat dict of the keys you declared — merged across all hosts and `kestra` module calls.
```json
{
"app_version": "1.4.2",
"db_host": "pg-prod.internal"
}
```
:::alert{type="warning"}
Switching between `ALL` and `EXPLICIT` changes the type of `vars.outputs` from a list to a dict. Update any downstream expressions that iterate over the list if you migrate an existing flow.
:::
## Multi-host output merging
When multiple hosts call `kestra: outputs:` with the same key, the last write wins. If hosts run in parallel, which host writes last is non-deterministic. Use one of these patterns for predictable results:
**Run the declaration on one host only:**
```yaml
- name: Declare run-level output
kestra:
outputs:
cluster_leader: "{{ inventory_hostname }}"
run_once: true
delegate_to: localhost
```
**Key by hostname for per-host values:**
```yaml
- name: Declare per-host version
kestra:
outputs:
"version_{{ inventory_hostname }}": "{{ app_version }}"
```
## Avoiding expression conflicts
Ansible uses `{{ variable }}` for Jinja2 expressions, and so does Kestra's Pebble templating engine. When a playbook is defined inline in a Kestra flow (for example, in `inputFiles` or as a flow variable), Kestra evaluates its own `{{ }}` expressions before passing the file to Ansible — which breaks Ansible's expressions.
Store playbooks that contain Ansible Jinja expressions in `variables:` to prevent Kestra from evaluating their `{{ }}` expressions:
```yaml
id: deploy
namespace: company.infra
variables:
playbook: |
- hosts: webservers
tasks:
- name: Gather version
shell: cat /opt/app/VERSION
register: result
- name: Declare output
kestra:
outputs:
version: "{{ result.stdout }}"
tasks:
- id: run_playbook
type: io.kestra.plugin.ansible.cli.AnsibleCLI
outputsMode: EXPLICIT
containerImage: cytopia/ansible:latest-tools
inputFiles:
hosts: |
[webservers]
web-01
playbook.yml: "{{ vars.playbook }}"
commands:
- ansible-playbook -i hosts playbook.yml
```
The `vars.playbook` reference is resolved by Kestra, returning the raw playbook string. Ansible then evaluates `{{ result.stdout }}` as expected.
## Custom ansibleConfig
If you supply a custom `ansibleConfig`, you must manually include the `library` path so Ansible can find the bundled `kestra` module. Without it, `kestra: outputs:` tasks will fail with a module-not-found error.
```yaml
tasks:
- id: run_playbook
type: io.kestra.plugin.ansible.cli.AnsibleCLI
outputsMode: EXPLICIT
ansibleConfig: |
[defaults]
library = ./library
stdout_callback = kestra_logger
callbacks_enabled = kestra_logger
commands:
- ansible-playbook -i hosts playbook.yml
```
:::alert{type="warning"}
The exact `ansibleConfig` lines required alongside `library = ./library` depend on how the plugin configures the callback. Confirm the required fields in a test run before deploying to production. A missing callback line will silently fall back to `ALL` mode behavior.
:::
## What EXPLICIT mode does not protect
`outputsMode: EXPLICIT` controls the `vars.outputs` payload. It does not affect:
- **Ansible verbose logging** (`-v`, `-vv`, etc.) — verbose output still appears in task logs.
- **`log_path`** in `ansible.cfg` — if set, Ansible writes its full log to that path.
- **Kestra task logs** — stdout from shell commands and module output may appear in execution logs.
If your playbooks handle highly sensitive data, use `no_log: true` on individual tasks to suppress their output from Ansible's own logging, and avoid enabling verbose mode.
---
# Extend Kestra with the API
URL: https://kestra.io/docs/how-to-guides/api
> Discover how to extend Kestra by using its API to create flows, trigger executions, manage KV store entries, and handle namespace files.
Extend Kestra by using the API.
Kestra is API-first, so it’s straightforward to connect external systems to your flows or call the platform directly. This guide focuses on the Kestra API itself and how you can extend or integrate Kestra from other services.
## Using the API Reference
The docs include references for both the [Open Source](../../api-reference/02.open-source/index.mdx) and [Cloud & Enterprise](../../api-reference/01.enterprise/index.mdx) APIs so you know exactly what endpoints are available. Opening the [Open Source reference](../../api-reference/02.open-source/index.mdx) shows a structured layout that’s easy to scan:

## Making Requests with Authentication
If you have [Basic Auth enabled](../../configuration/05.security-and-secrets/index.md) or you’re using the [Enterprise Edition](/enterprise), authenticate each request. With cURL you can pass credentials via `-u username:password`. The example below uses the defaults from the [Kestra Docker Compose](../../02.installation/03.docker-compose/index.md):
```bash
curl -X POST -u 'admin@kestra.io:kestra' http://localhost:8080/api/v1/executions/company.team/hello_world
```
Enterprise users can generate [API tokens](../../07.enterprise/03.auth/api-tokens/index.md) and send them as Bearer headers:
```bash
curl -X POST http://localhost:8080/api/v1/executions/company.team/hello_world \
-H "Authorization: Bearer YOUR_API_TOKEN"
```
The remaining examples assume authentication is disabled.
## Create a Flow
To create a flow via API, open the **Flows** section and look for the `/api/v1/main/flows` [POST endpoint](https://kestra.io/docs/api-reference/open-source#post-/api/v1/flows). It expects a YAML payload containing the flow definition.
The request body uses Content-Type `application/x-yaml`:
```yaml
id: created_by_api
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Hello World! 🚀
```
Send the request with [cURL](https://en.wikipedia.org/wiki/CURL):
```bash
curl -X POST http://localhost:8080/api/v1/main/flows -H "Content-Type:application/x-yaml" -d "id: created_by_api
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Hello World! 🚀"
```
The response looks like this:
```json
{
"id": "created_by_api",
"namespace": "company.team",
"revision": 1,
"disabled": false,
"deleted": false,
"tasks":
[
{
"id": "hello",
"type": "io.kestra.plugin.core.log.Log",
"message": "Hello World! \uD83D\uDE80"
}
],
"source": "id: created_by_api\nnamespace: company.team\n\ntasks:\n - id: hello\n type: io.kestra.plugin.core.log.Log\n message: Hello World! \uD83D\uDE80"
}
```
## Execute a Flow
To execute a flow, provide the namespace and flow ID. The sample flow below (`hello_world`) lives in the `company.team` namespace and accepts a string input:
```yaml
id: hello_world
namespace: company.team
inputs:
- id: greeting
type: STRING
defaults: hey
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: "{{ inputs.greeting }}"
```
Because the input has a default, we can call the [POST endpoint](https://kestra.io/docs/api-reference/open-source#post-/api/v1/executions/-namespace-/-id-) `/api/v1/main/executions/{namespace}/{id}` without providing additional data:
```bash
curl -X POST \
http://localhost:8080/api/v1/main/executions/company.team/hello_world
```
To override inputs, send them as form data with `-F`:
```bash
curl -X POST \
http://localhost:8080/api/v1/main/executions/company.team/hello_world \
-F greeting="hey there"
```
The response includes execution metadata and a link to the UI:
```json
{
"id": "MYkTmLrI36s10iVXHwRbR",
"namespace": "company.team",
"flowId": "hello_world",
"flowRevision": 10,
"inputs": {
"greeting": "hey"
},
"labels": [
{
"key": "system.correlationId",
"value": "MYkTmLrI36s10iVXHwRbR"
}
],
"state": {
"current": "CREATED",
"histories": [
{
"state": "CREATED",
"date": "2024-11-21T16:31:27.943162175Z"
}
],
"duration": 0.044177500,
"startDate": "2024-11-21T16:31:27.943162175Z"
},
"originalId": "MYkTmLrI36s10iVXHwRbR",
"deleted": false,
"metadata": {
"attemptNumber": 1,
"originalCreatedDate": "2024-11-21T16:31:27.943194342Z"
},
"url": "http://localhost:8080//ui/executions/company.team/hello_world/MYkTmLrI36s10iVXHwRbR"
}
```
For end-to-end idempotency using a stable business key, set `system.correlationId` when you create the execution and add a guard as shown in [Idempotency with correlation IDs](../idempotency/index.md).
See the [Executions documentation](../../05.workflow-components/03.execution/index.md#execute-a-flow-via-an-api-call) for additional examples.
## Get Information from an Execution
The execution response returns the execution ID, which you can use to fetch additional details once the run completes. Using `MYkTmLrI36s10iVXHwRbR` from the earlier example, call the [GET endpoint](https://kestra.io/docs/api-reference/open-source#get-/api/v1/executions/-executionId-) `/api/v1/main/executions/{executionId}`:
```bash
curl -X GET http://localhost:8080/api/v1/main/executions/MYkTmLrI36s10iVXHwRbR
```
The response includes state transitions, durations, and outputs:
:::collapse{title="Response Body"}
```json
{
"id": "MYkTmLrI36s10iVXHwRbR",
"namespace": "company.team",
"flowId": "hello_world",
"flowRevision": 10,
"taskRunList": [
{
"id": "1ZSXuswTiOeLggIwxT3V98",
"executionId": "MYkTmLrI36s10iVXHwRbR",
"namespace": "company.team",
"flowId": "hello_world",
"taskId": "hello",
"attempts": [
{
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2024-11-21T16:31:29.463Z"
},
{
"state": "RUNNING",
"date": "2024-11-21T16:31:29.463Z"
},
{
"state": "SUCCESS",
"date": "2024-11-21T16:31:29.512Z"
}
],
"duration": 0.049000000,
"startDate": "2024-11-21T16:31:29.463Z",
"endDate": "2024-11-21T16:31:29.512Z"
}
}
],
"outputs": {},
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2024-11-21T16:31:28.455Z"
},
{
"state": "RUNNING",
"date": "2024-11-21T16:31:29.448Z"
},
{
"state": "SUCCESS",
"date": "2024-11-21T16:31:29.512Z"
}
],
"duration": 1.057000000,
"startDate": "2024-11-21T16:31:28.455Z",
"endDate": "2024-11-21T16:31:29.512Z"
}
}
],
"inputs": {
"greeting": "hey"
},
"labels": [
{
"key": "system.correlationId",
"value": "MYkTmLrI36s10iVXHwRbR"
}
],
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2024-11-21T16:31:27.943Z"
},
{
"state": "RUNNING",
"date": "2024-11-21T16:31:28.463Z"
},
{
"state": "SUCCESS",
"date": "2024-11-21T16:31:30.474Z"
}
],
"duration": 2.531000000,
"startDate": "2024-11-21T16:31:27.943Z",
"endDate": "2024-11-21T16:31:30.474Z"
},
"originalId": "MYkTmLrI36s10iVXHwRbR",
"deleted": false,
"metadata": {
"attemptNumber": 1,
"originalCreatedDate": "2024-11-21T16:31:27.943Z"
}
}
```
:::
Modify the flow to emit an output:
```yaml
id: hello_world
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.debug.Return
format: "This is an output"
```
Fetching execution `59uQXHbkMy5YwHEDom72Xv` now shows the output payload:
:::collapse{title="Response Body"}
```json
{
"id": "59uQXHbkMy5YwHEDom72Xv",
"namespace": "company.team",
"flowId": "hello_world",
"flowRevision": 13,
"taskRunList": [
{
"id": "4G8EJhI2IwTdlHYi250h7m",
"executionId": "59uQXHbkMy5YwHEDom72Xv",
"namespace": "company.team",
"flowId": "hello_world",
"taskId": "hello",
"attempts": [
{
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2024-11-21T17:09:42.016Z"
},
{
"state": "RUNNING",
"date": "2024-11-21T17:09:42.016Z"
},
{
"state": "SUCCESS",
"date": "2024-11-21T17:09:42.045Z"
}
],
"duration": 0.029000000,
"startDate": "2024-11-21T17:09:42.016Z",
"endDate": "2024-11-21T17:09:42.045Z"
}
}
],
"outputs": {
"value": "This is an output"
},
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2024-11-21T17:09:40.937Z"
},
{
"state": "RUNNING",
"date": "2024-11-21T17:09:41.967Z"
},
{
"state": "SUCCESS",
"date": "2024-11-21T17:09:42.053Z"
}
],
"duration": 1.116000000,
"startDate": "2024-11-21T17:09:40.937Z",
"endDate": "2024-11-21T17:09:42.053Z"
}
}
],
"labels": [
{
"key": "system.correlationId",
"value": "59uQXHbkMy5YwHEDom72Xv"
}
],
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2024-11-21T17:09:40.204Z"
},
{
"state": "RUNNING",
"date": "2024-11-21T17:09:40.942Z"
},
{
"state": "SUCCESS",
"date": "2024-11-21T17:09:42.994Z"
}
],
"duration": 2.790000000,
"startDate": "2024-11-21T17:09:40.204Z",
"endDate": "2024-11-21T17:09:42.994Z"
},
"originalId": "59uQXHbkMy5YwHEDom72Xv",
"deleted": false,
"metadata": {
"attemptNumber": 1,
"originalCreatedDate": "2024-11-21T17:09:40.204Z"
},
"scheduleDate": "2024-11-21T17:09:40.181Z"
}
```
:::
## Accessing the KV Store
Kestra’s [KV Store](../../06.concepts/05.kv-store/index.md) keeps flows stateful. You can create, update, and delete entries via the API — either from code running inside a flow or from external systems.
Add a key/value pair with the [PUT endpoint](https://kestra.io/docs/api-reference/open-source#put-/api/v1/namespaces/-namespace-/kv/-key-) `/api/v1/main/namespaces/{namespace}/kv/{key}`. The example below writes `"Hello, World"` to `my_key` in the `company.team` namespace:
```bash
curl -X PUT -H "Content-Type: application/json" http://localhost:8080/api/v1/main/namespaces/company.team/kv/my_key -d '"Hello, World"'
```
Verify in Kestra that the entry exists:

Update the value by sending a different body, for example `"This is a modified value"`:
```bash
curl -X PUT -H "Content-Type: application/json" http://localhost:8080/api/v1/main/namespaces/company.team/kv/my_key -d '"This is a modified value"'
```
Kestra shows the key as updated:

Opening the entry reveals the new value.

Fetch the value with the [GET endpoint](https://kestra.io/docs/api-reference/open-source#get-/api/v1/namespaces/-namespace-/kv/-key-) `/api/v1/main/namespaces/{namespace}/kv/{key}`:
```bash
curl -X GET http://localhost:8080/api/v1/main/namespaces/company.team/kv/my_key
```
The response contains the type and value:
```json
{
"type": "STRING",
"value": "This is a modified value"
}
```
See the [KV Store documentation](../../06.concepts/05.kv-store/index.md#api-how-to-create-read-update-and-delete-kv-pairs-via-rest-api) for more operations.
## Get and Upload Namespaces Files
Beyond flows, you can manage namespace files via the API.
Use the [GET endpoint](https://kestra.io/docs/api-reference/open-source#get-/api/v1/namespaces/-namespace-/files/directory) `/api/v1/main/namespaces/{namespace}/files/directory` to list files in a namespace:

For `company.team`:
```bash
curl -X GET http://localhost:8080/api/v1/main/namespaces/company.team/files/directory
```
The response is an array of file metadata:
```json
[
{
"type": "File",
"size": 13,
"fileName": "example.txt",
"lastModifiedTime": 1731430406183,
"creationTime": 1731430400773
},
{
"type": "File",
"size": 27,
"fileName": "example.js",
"lastModifiedTime": 1731415024668,
"creationTime": 1730997234841
},
{
"type": "File",
"size": 19,
"fileName": "example.sh",
"lastModifiedTime": 1731415024667,
"creationTime": 1730997234839
},
{
"type": "File",
"size": 171,
"fileName": "example.ion",
"lastModifiedTime": 1731430044778,
"creationTime": 1731430012804
},
{
"type": "File",
"size": 21,
"fileName": "example.py",
"lastModifiedTime": 1731415024667,
"creationTime": 1729781670534
}
]
```
Use the [GET endpoint](https://kestra.io/docs/api-reference/open-source#get-/api/v1/namespaces/-namespace-/files) `/api/v1/main/namespaces/{namespace}/files` to fetch file contents:
Example request for `example.txt`:
```bash
curl -X GET 'http://localhost:8080/api/v1/main/namespaces/company.team/files?path=example.txt'
```
which returns:
```plaintext
Hello, World!
```
Upload files using the [POST endpoint](https://kestra.io/docs/api-reference/open-source#post-/api/v1/namespaces/-namespace-/files) `/api/v1/main/namespaces/{namespace}/files`. The example below uploads `api_example.py` with the following content:
```python
import requests
r = requests.get("https://kestra.io")
print({r.status_code})
```
Run:
```bash
curl -X POST 'http://localhost:8080/api/v1/main/namespaces/company.team/files?path=api_example.py' -H "Content-Type:multipart/form-data" -F "fileContent=@api_example.py"
```
:::alert{type="info"}
Ensure `fileContent` has the correct path to your file.
:::
After the upload, the file appears in the Namespace editor:

---
# Deploy Kestra with ArgoCD
URL: https://kestra.io/docs/how-to-guides/argocd
> Learn how to deploy Kestra with ArgoCD using a wrapper chart that pulls from helm.kestra.io, so you stay in full control of when upgrades happen.
Deploy Kestra with ArgoCD by creating a wrapper chart in your own infrastructure repository. Nothing deploys until you commit a change — you decide when upgrades happen.
## Prerequisites
- **kubectl** — to interact with your cluster
- **Helm** — to manage chart dependencies
- **ArgoCD** — installed and configured in your cluster
## Source charts from helm.kestra.io, not the git repository
The `kestra` GitHub repository is a development repository, not a Helm distribution channel. The `develop` branch intentionally carries `version: 0.0.0` in `Chart.yaml` — a deliberate signal that it is not a release artifact.
The stable, versioned Helm chart is published at **[helm.kestra.io](https://helm.kestra.io/)**, which Kestra keeps stable and immutable across versions.
:::alert{type="warning"}
Do not point ArgoCD directly at `github.com/kestra-io/kestra` to source the Helm chart. The develop branch does not reflect released chart versions and its structure may change at any time.
:::
## Recommended pattern: wrapper chart
Create a **wrapper chart** in your own infrastructure repository. The wrapper chart declares kestra as a dependency pinned to a specific version from `helm.kestra.io`. Your `values.yaml` overrides sit alongside it. ArgoCD watches only your repository, so nothing deploys until you push a commit.
### Directory structure
```
your-infra-repo/
└── kestra/
├── Chart.yaml # wrapper chart with pinned kestra dependency
├── Chart.lock # generated; locks the resolved dependency
├── values.yaml # your overrides (secrets, database, storage)
└── charts/
└── kestra-x.y.z.tgz # vendored chart; committed for reproducibility
```
### 1. Create the wrapper Chart.yaml
```yaml
# kestra/Chart.yaml
apiVersion: v2
name: my-kestra
version: 0.1.0
dependencies:
- name: kestra
repository: https://helm.kestra.io
version: "1.3.16" # pin to the version you want to run
```
To find available versions:
```bash
helm repo add kestra https://helm.kestra.io/
helm repo update
helm search repo kestra/kestra --versions
```
### 2. Add your values.yaml
Place your environment-specific configuration alongside the wrapper chart. Prefix all keys with `kestra.` to scope them to the dependency:
```yaml
# kestra/values.yaml
kestra:
# All keys here match the Kestra Helm chart values.
# See https://github.com/kestra-io/kestra/blob/develop/charts/kestra/values.yaml
configurations:
application:
kestra:
storage:
type: s3
s3:
bucket: your-kestra-bucket
region: us-east-1
datasources:
postgres:
url: jdbc:postgresql://your-db-host:5432/kestra
driverClassName: org.postgresql.Driver
username: kestra
password: ""
```
For credentials, use Kubernetes Secrets rather than plain values. See [Pass Kubernetes Secrets and Env Vars via Helm Chart](../kubernetes-secrets/index.md) for the recommended pattern.
### 3. Vendor the chart
Fetch the pinned chart from `helm.kestra.io` and commit it to your repository:
```bash
cd kestra/
helm dependency update
```
This creates `Chart.lock` and downloads `charts/kestra-x.y.z.tgz`. Commit all three files (`Chart.yaml`, `Chart.lock`, and `charts/kestra-x.y.z.tgz`) so the chart content is fully reproducible from your repository without any external network dependency at deploy time.
```bash
git add Chart.yaml Chart.lock charts/
git commit -m "chore: add kestra helm chart dependency v1.3.16"
```
### 4. Configure an ArgoCD Application
Point ArgoCD at the `kestra/` directory in your repository:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: kestra
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/your-org/your-infra-repo
targetRevision: main
path: kestra
destination:
server: https://kubernetes.default.svc
namespace: kestra
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
```
ArgoCD now watches your repository. Because the chart is vendored, syncs do not require network access to `helm.kestra.io`.
## Upgrade to a new chart version
1. Update the version pin in `kestra/Chart.yaml`
2. Re-vendor the chart:
```bash
cd kestra/
helm dependency update
```
3. Commit the updated `Chart.yaml`, `Chart.lock`, and `charts/kestra-x.y.z.tgz`
4. Push to your repository — ArgoCD detects the change and syncs
## Trigger ArgoCD syncs from Kestra flows
If you use Kestra to orchestrate infrastructure operations, the [ArgoCD plugin](/plugins/plugin-argocd/io.kestra.plugin.argocd.apps.sync) lets you trigger and monitor ArgoCD application syncs directly from a flow:
```yaml
id: deploy_kestra_upgrade
namespace: company.infra
tasks:
- id: sync
type: io.kestra.plugin.argocd.apps.Sync
server: "{{ secret('ARGOCD_SERVER') }}"
token: "{{ secret('ARGOCD_TOKEN') }}"
application: kestra
- id: status
type: io.kestra.plugin.argocd.apps.Status
server: "{{ secret('ARGOCD_SERVER') }}"
token: "{{ secret('ARGOCD_TOKEN') }}"
application: kestra
```
Use this to chain a chart upgrade commit with an automated sync and status check as part of a larger deployment pipeline.
---
# Use Azure Managed Workload Identity with Kestra
URL: https://kestra.io/docs/how-to-guides/azure-workload-id
> Configure Azure Workload Identity on Kestra Enterprise to securely access Azure resources like Key Vault without managing secrets.
How to use Azure Workload identity to provide access to resources such as Azure Key Vault in Kestra
:::alert{type="info"}
This page is only relevant for the Enterprise Edition of Kestra. For Cloud-based secret manager integrations, contact us at sales@kestra.io or chat with us in our [Slack community](https://kestra.io/slack).
:::
## Pre-Requisites
To follow this guide you will need the following
1. [Kestra Enterprise Edition](https://kestra.io/docs/enterprise)
2. Account with Azure
3. [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/) installed
4. Kubernetes tools (kubectl & helm)
5. Permissions to provision the following:
- [AKS Cluster](https://azure.microsoft.com/en-us/products/kubernetes-service/)
- [Azure Key Vault](https://learn.microsoft.com/en-us/azure/key-vault/general/)
- [User-assigned managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview)
This guide is based on the official Azure documentation on Workload Identity — it's best to read [this Azure guide](https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster) first for full context. Here, we'll focus on enabling this feature in Kestra.
## Variables
Define the following variables and update them to match your environment.
```shell
## Managed User Identity Name
ID_NAME="kestra-managed-user"
## Azure Resource Group
RESOURCE_GROUP="demo"
## Physical location you wish to provision resources
LOCATION="eastus"
## The name of your Azure Kubernetes Cluster
AKS_NAME="demo-cluster"
## The name of your Azure Key Vault
KEYVAULT_NAME="my-demo-vault"
## The name you wish to provide to the Kubernetes Service Account linked to the managed identity
SERVICE_ACCOUNT_NAME="kestra-sa"
## The namespace to deploy the service account. Use the same location as your Kestra deployment
SERVICE_ACCOUNT_NAMESPACE="default"
## The Federated ID credential for linking the OIDC issuer to the service account
FEDERATED_IDENTITY_CREDENTIAL_NAME="kestra-fed-cred"
```
## Create the resources
First, create the following main resources:
1. The Key Vault
2. The Managed Identity
3. The AKS cluster.
Once these have been provisioned, there are several identifiers we must capture for later use.
### Azure Key Vault
This creates an Azure Key Vault. By default this will be created with RBAC (role-based access control) enabled which is the recommended configuration.
```shell
az keyvault create \
--name $KEYVAULT_NAME \
--resource-group $RESOURCE_GROUP \
--location $LOCATION
```
### Managed Identity
This creates the user-assigned managed identity used to provision access to resources within the Kubernetes cluster.
```shell
az identity create --name $ID_NAME \
--resource-group $RESOURCE_GROUP
```
### AKS Cluster
```shell
az aks create \
--resource-group $RESOURCE_GROUP \
--name $AKS_NAME \
--enable-oidc-issuer \
--enable-workload-identity \
--node-count 1 \
--generate-ssh-keys
```
### Setting identifiers from new resources
Once all the above have been created, capture the following information in variables for use later on:
```shell
OBJECT_ID=$(az identity show --name $ID_NAME --resource-group $RESOURCE_GROUP --query 'principalId' --output tsv)
MANAGED_CLIENT_ID=$(az identity show --name $ID_NAME --resource-group $RESOURCE_GROUP --query clientId --output tsv)
AKS_OIDC_ISSUER="$(az aks show --name "${AKS_NAME}" --resource-group "${RESOURCE_GROUP}" --query "oidcIssuerProfile.issuerUrl" --output tsv)"
```
## Link Identity Resources
One of the more challenging aspects of this setup is correctly linking together the various resources. This section covers how to tie the managed identities to the resources to allow access by the Kestra application.
### Create role assignment for created user
This is one of the most critical steps as it sets the permission the resource has on the Key Vault. As Kestra needs to read and write secrets to the vault, the "Key Vault Secrets Officer" provides least priviledged access for this operation. Further details on this role can be found [in Azure's RBAC guide](https://learn.microsoft.com/en-us/azure/key-vault/general/rbac-guide?tabs=azure-cli#azure-built-in-roles-for-key-vault-data-plane-operations).
```shell
az role assignment create \
--assignee-object-id $OBJECT_ID \
--role "Key Vault Secrets Officer" \
--scope $(az keyvault show --name $KEYVAULT_NAME --query id -o tsv)
```
### Create the service account in the AKS Cluster
First, we must switch context to the newly created AKS cluster:
```shell
az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_NAME
```
Next, create a service account in the same namespace where you deploy Kestra.
```shell
cat < Use Kestra flows to deploy applications, provision add-ons, forward logs, and manage organisation members on Clever Cloud.
Orchestrate Clever Cloud infrastructure from Kestra: deploy applications, provision managed databases, route logs to external platforms, and manage organisation members. For the full task and trigger reference, see the [plugin page](https://kestra.io/plugins/plugin-clevercloud).
## Prerequisites
- A [Clever Cloud](https://www.clever-cloud.com/) account
- A Clever Cloud API token — generate one in the console under **Profile > API tokens**
- An organisation ID if managing organisation-owned resources (find it in the console URL: `console.clever-cloud.com/organisations/orga_xxx`)
## Authentication
All Clever Cloud tasks share a single `apiToken` property. Store it as a [Kestra secret](../../06.concepts/04.secret/index.md) and reference it with `{{ secret('CC_API_TOKEN') }}`.
Most tasks also accept an optional `organisationId`. When omitted, the plugin targets your personal account via the `/self` API endpoint. Two task groups always require `organisationId`: all `logs` tasks (the APIv4 log endpoints have no `/self` shortcut) and the organisation member tasks (`ListMembers`, `AddMember`, `RemoveMember`).
### Inject credentials with a Policy (EE)
On Enterprise Edition, a [Policy](../../07.enterprise/02.governance/policies/index.md) `Add` rule injects `apiToken` into every Clever Cloud task in a namespace without touching any flow YAML:
```yaml
id: clever-cloud-auth
namespace: company.team
enforcement: ACTIVE
rules:
- type: io.kestra.plugin.ee.rules.Add
on: PLUGIN
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.clevercloud
properties:
apiToken: "{{ secret('CC_API_TOKEN') }}"
```
## Deploy an application and wait for it
Trigger a redeployment and block the flow until the deployment reaches a terminal state.
```yaml
id: redeploy_and_wait
namespace: company.team
inputs:
- id: app_id
type: STRING
tasks:
- id: redeploy
type: io.kestra.plugin.clevercloud.applications.Redeploy
apiToken: "{{ secret('CC_API_TOKEN') }}"
applicationId: "{{ inputs.app_id }}"
- id: get_deployment
type: io.kestra.plugin.clevercloud.deployments.List
apiToken: "{{ secret('CC_API_TOKEN') }}"
applicationId: "{{ inputs.app_id }}"
limit: 1
fetchType: FETCH_ONE
- id: wait
type: io.kestra.plugin.clevercloud.deployments.WaitForState
apiToken: "{{ secret('CC_API_TOKEN') }}"
applicationId: "{{ inputs.app_id }}"
deploymentId: "{{ outputs.get_deployment.deployment.uuid }}"
targetState: OK
failOnUnreached: true
timeout: PT20M
```
Deployment states: `WIP` (in progress), `OK` (success), `FAIL` (error), `CANCELLED`. Set `failOnUnreached: true` to fail the flow if the deployment reaches `FAIL` or `CANCELLED` instead of `OK`. Use `deployments.Trigger` to fire a flow automatically when a deployment reaches a target state.
:::alert{type="warning"}
`Redeploy` returns no output. The `get_deployment` step fetches the most recent deployment immediately after — if the new deployment has not yet registered in the Clever Cloud API, it may return the previous one. Add a short `Wait` task between `redeploy` and `get_deployment` if you observe this in practice.
:::
## Provision a database add-on and read its credentials
Provision a managed database and retrieve its connection credentials.
```yaml
id: provision_postgres
namespace: company.team
tasks:
- id: create_addon
type: io.kestra.plugin.clevercloud.addons.Create
apiToken: "{{ secret('CC_API_TOKEN') }}"
organisationId: "{{ secret('CC_ORG_ID') }}"
providerId: postgresql-addon
plan: dev
region: par
name: my-app-db
- id: get_credentials
type: io.kestra.plugin.clevercloud.addons.GetEnv
apiToken: "{{ secret('CC_API_TOKEN') }}"
organisationId: "{{ secret('CC_ORG_ID') }}"
addonId: "{{ outputs.create_addon.id }}"
- id: link_to_app
type: io.kestra.plugin.clevercloud.addons.LinkToApplication
apiToken: "{{ secret('CC_API_TOKEN') }}"
organisationId: "{{ secret('CC_ORG_ID') }}"
applicationId: "{{ secret('CC_APP_ID') }}"
addonId: "{{ outputs.create_addon.id }}"
```
`addons.GetEnv` returns connection credentials in plain text via `{{ outputs.get_credentials.variables }}`. Avoid logging or persisting this output directly — pass values to downstream tasks using Kestra's secret store or output references.
Use `addons.AddonProvisionedTrigger` to fire a flow automatically when a new add-on appears in your account.
## Forward application logs to an external platform
Create a log drain to continuously forward an application's logs to New Relic, Datadog, Elasticsearch, or a syslog endpoint.
```yaml
id: setup_log_drain
namespace: company.team
tasks:
- id: create_drain
type: io.kestra.plugin.clevercloud.logs.CreateDrain
apiToken: "{{ secret('CC_API_TOKEN') }}"
organisationId: "{{ secret('CC_ORG_ID') }}"
applicationId: "{{ secret('CC_APP_ID') }}"
drainType: NEWRELIC
url: https://log-api.newrelic.com/log/v1
newRelicApiKey: "{{ secret('CC_NEWRELIC_API_KEY') }}"
kind: LOG
```
Supported `drainType` values:
- `NEWRELIC` — requires `url` and `newRelicApiKey`; use `https://log-api.eu.newrelic.com/log/v1` for EU accounts
- `DATADOG` — requires `url`
- `ELASTICSEARCH` — requires `url`
- `RAW_HTTP`, `SYSLOG_TCP`, `SYSLOG_UDP` — generic syslog or HTTP; use these for OVHcloud (no dedicated drain type)
Use `logs.DeleteDrain` to remove a drain and `logs.ListDrains` to audit active forwarding.
## Trigger a flow when a log line matches a pattern
`LogPatternTrigger` polls application logs at each interval and fires when a line matches a regex. Use it to react to application errors, OOM kills, or deployment signals without polling from a separate flow.
```yaml
id: alert_on_error
namespace: company.team
triggers:
- id: on_oom
type: io.kestra.plugin.clevercloud.logs.LogPatternTrigger
apiToken: "{{ secret('CC_API_TOKEN') }}"
organisationId: "{{ secret('CC_ORG_ID') }}"
applicationId: "{{ secret('CC_APP_ID') }}"
pattern: "OutOfMemoryError|OOM killed"
interval: PT2M
limit: 500
tasks:
- id: notify
type: io.kestra.plugin.core.log.Log
message: "OOM detected at {{ trigger.matchedAt }}: {{ trigger.matchedLine }}"
```
Available trigger outputs:
| Output | Description |
|--------|-------------|
| `trigger.matchedLine` | Full text of the matched log line |
| `trigger.matchedAt` | Timestamp of the matched log line |
| `trigger.severity` | Log severity level |
| `trigger.service` | Service that emitted the log |
When multiple lines match in the same poll, only the most recent fires the trigger.
Replace the `notify` task with your preferred alerting method — a Slack message, a PagerDuty call, or an email via `io.kestra.plugin.notifications.slack.SlackIncomingWebhook`.
## Manage organisation members
Add a member and react when the organisation's membership changes.
```yaml
id: onboard_member
namespace: company.team
inputs:
- id: email
type: STRING
- id: role
type: STRING
defaults: DEVELOPER
tasks:
- id: add_member
type: io.kestra.plugin.clevercloud.organisations.AddMember
apiToken: "{{ secret('CC_API_TOKEN') }}"
organisationId: "{{ secret('CC_ORG_ID') }}"
email: "{{ inputs.email }}"
role: "{{ inputs.role }}"
```
Valid roles: `ADMIN`, `MANAGER`, `DEVELOPER`, `ACCOUNTING`, `READ_ONLY`.
To offboard, call `organisations.RemoveMember` with a `userId` retrieved from `organisations.ListMembers`.
`MemberChangeTrigger` fires when any member is added or removed. It uses the KV store to track membership between polls — the first evaluation always establishes the baseline without firing.
---
# Add Conditional Branching in Kestra
URL: https://kestra.io/docs/how-to-guides/conditional-branching
> Master conditional branching in Kestra workflows using the Switch task to direct execution paths based on dynamic input values.
How to use the Switch task to branch the flow based on a value.
Conditional branching is a process in which the execution of a task is directed along different paths based on specific values. In a flow, it allows for decision-making, where different tasks are executed depending on the value provided.
This guide shows how to use Kestra's `Switch` task to control your flow based on a value. Depending on the value passed, the flow branches to different task `cases`. If there is no matching value, Kestra uses the `defaults` branch.
## Prerequisites
Before you begin:
- Deploy [Kestra](../../02.installation/index.mdx) in your preferred development environment.
- Ensure you have a [basic understanding of how to run Kestra flows.](../../03.tutorial/index.mdx)
## Example 1: Conditional Branching with Input Strings
This flow template serves as an introductory example to understand how the `Switch` task works within Kestra. The flow dynamically branches to different tasks depending on the input string. To see the flow in action, define the `switch` flow as shown below:
```yaml
id: switch
namespace: company.team
inputs:
- id: string
type: STRING
tasks:
- id: parent-seq
type: io.kestra.plugin.core.flow.Switch
value: "{{inputs.string}}"
cases:
FIRST:
- id: first
type: io.kestra.plugin.core.debug.Return
format: "{{task.id}} > {{taskrun.startDate}}"
SECOND:
- id: second1
type: io.kestra.plugin.core.debug.Return
format: "{{task.id}} > {{taskrun.startDate}}"
- id: second2
type: io.kestra.plugin.core.debug.Return
format: "{{task.id}} > {{taskrun.startDate}}"
THIRD:
- id: third1
type: io.kestra.plugin.core.flow.Sequential
tasks:
- id: failed
type: io.kestra.plugin.core.execution.Fail
errors:
- id: error1
type: io.kestra.plugin.core.debug.Return
format: "Error Trigger ! {{task.id}}"
defaults:
- id: default
type: io.kestra.plugin.core.debug.Return
format: "{{task.id}} > {{taskrun.startDate}}"
outputs:
- id: extracted
type: STRING
value: "{{ outputs.first ?? outputs.second1 ?? outputs.third1 ?? outputs.default }}"
```
Save and execute the `switch` flow.
You can input `FIRST`, `SECOND`, `THIRD`, or any other input string to see the flow in action. The above flow when executed checks a sequence of tasks based on the input string you provide.
Within the flow:
- `inputs`: Takes a string input to determine which case to execute.
- `tasks`: Handles the input string with the following values:
- `id: parent-seq`: Uses the `Switch` task to evaluate the input string and execute a case that matches the input string with the following cases:
- `FIRST`: Executes task `first`, returning its ID and start time.
- `SECOND`: Executes two tasks (`second1` and `second2`), both returning their task ID and start time.
- `THIRD`: Runs a sequence of tasks where the `failed` task triggers an error and executes the `error1` task, which logs an error message.
- `defaults`: If the input doesn't match any cases, it runs the `default` task and logs the task ID and start time.
- `outputs`: Extracts and logs the output from one of the executed tasks (`first`, `second1`, `third1`, or `default`).
## Example 2: Conditional branching with Kestra’s website status
To see the `Switch` task in action without an input string, we’ll create a flow to make a decision based on the status of an HTTP request to Kestra’s website. To follow along and implement this, define the `kestra-http-switch` flow as described below:
```yaml
id: kestra-http-switch
namespace: company.team
tasks:
- id: check_kestra_site
type: io.kestra.plugin.scripts.python.Script
outputFiles:
- site_status.txt
script: |
import requests
response = requests.head("https://kestra.io")
with open('site_status.txt', 'w') as f:
f.write(str(response.status_code))
- id: decide_site_status
type: io.kestra.plugin.core.flow.Switch
value: "{{ read(outputs.check_kestra_site.outputFiles['site_status.txt']) }}"
cases:
"200":
- id: log-site-up
type: io.kestra.plugin.core.log.Log
message: "Kestra website is up and running. Status: 200"
"404":
- id: log-site-down
type: io.kestra.plugin.core.log.Log
message: "Kestra website not found. Status: 404"
defaults:
- id: unknown-status
type: io.kestra.plugin.core.log.Log
message: "Received unexpected status code: {{ read(outputs.check_kestra_site.outputFiles['site_status.txt']) }}"
outputs:
- id: status-output
type: STRING
value: "{{ outputs.log-site-up ?? outputs.log-site-down ?? outputs.unknown-status }}"
```
Save and execute the `kestra-http-switch` flow.
The above flow when executed checks the status of Kestra’s website and logs a message depending on the response code returned.
Within the flow:
- `tasks`: Handles the status check of the Kestra website with the following tasks:
- `id: check_kestra_site`: Executes a Python script to send a HEAD request to the Kestra website and writes the HTTP status code to a `site_status.txt` file.
- `id: decide_site_status`: Utilizes the `Switch` task to evaluate the HTTP status code from the `check_kestra_site` task:
- If the status code is `"200"`, it logs a message indicating the site is up.
- If the status code is `"404"`, it logs a message indicating the site is not found.
- If an unexpected status code is received, it falls back to the `defaults` branch with a message indicating unknown status.
- `outputs`: Extracts and logs the output status message based on the logs generated from the `Switch` task.
## Next Steps
You have implemented conditional branching with the `Switch` task using the `switch` flow to check your input strings and `kestra-http-switch` flow to check Kestra’s website status. The `Switch` task can further be implemented in various use cases to support your flows. Further resources about the `Switch` task:
- [Kestra’s official Switch task plugin documentation](/plugins/core/flow/io.kestra.plugin.core.flow.switch)
- [Kestra’s Blueprint Switch task use cases](/blueprints/switch)
---
# Build a Custom Plugin for Kestra
URL: https://kestra.io/docs/how-to-guides/custom-plugin
> Learn how to build, package, and test a custom Kestra plugin in Java to extend Kestra's capabilities for your specific needs.
Build your own Custom Plugin for Kestra.
This tutorial walks through building a custom plugin for Kestra.
## Use-case for Custom Plugin
We will be building a plugin that fetches the data for a given pokemon. We will use the API provided by [PokeAPI.co](https://pokeapi.co/) to fetch the pokemon's details: `https://pokeapi.co/api/v2/pokemon/{pokemon_name}`.
The API provides detailed information about any pokemon. We will fetch a few fields like the ability names, base experience, height and move names, and showcase it in the output of our plugin. The plugin task will accept the pokemon name, and return the selected fields in the output. This is how the task should look:
```yaml
id: fetch_details
type: io.kestra.plugin.pokemon.Fetch
pokemon: pikachu
```
## Requirements
You will need the following installed on your machine before proceeding:
* [Java](https://java.com) 21 or later.
* [IntelliJ IDEA](https://www.jetbrains.com/idea/) (or any other Java IDE, we provide only help for IntelliJ IDEA).
* [Gradle](https://gradle.org/) (included most of the time with the IDE).
#### Create a new plugin
Here are the steps:
1. Go on the [plugin-template](https://github.com/kestra-io/plugin-template) repository.
2. Click on *Use this template*.
3. Choose the GitHub account you want to link and the repository name for the new plugin.
4. Clone the new repository: `git clone git@github.com:{{user}}/{{name}}.git`.
5. Open the cloned directory in IntelliJ IDEA.
6. Enable [annotations processors](https://www.jetbrains.com/help/idea/annotation-processors-support.html).
7. If you are using an IntelliJ IDEA < 2020.03, install the [lombok plugins](https://plugins.jetbrains.com/plugin/6317-lombok) (if not, it's included by default).
Once you completed the steps above, you should see a similar directory structure:

As you can see, there is one generated plugin: the `Example` class representing the `Example` plugin (a task).
A project typically hosts multiple plugins. We call a project a group of plugins, and you can have multiple sub-groups inside a project by splitting plugins into different packages. Each package that has a plugin class is a sub-group of plugins.
## Gradle Configuration
We use [Gradle](https://gradle.org/) as a build tool.
### Mandatory configuration
The first thing we need to configure is the plugin name and the class package.
1. Change in `settings.gradle` the `rootProject.name = 'plugin-template'` with the plugin name `rootProject.name = 'plugin-pokemon'`.
2. Change the class package: by default, the template provides a package `io.kestra.plugin.templates`, just rename the `templates` folder in `src/main/java` & `src/test/java` to `pokemon`. And change the first line in `Example.java`, `ExampleRunnerTest.java` and `ExampleTest.java` to `package io.kestra.plugin.pokemon;`.
3. In `build.gradle`:
a. replace `description 'Plugin template for Kestra'` to the package name `description 'Plugin pokemon for Kestra'`.
b. In the `dependencies` section, add a dependency which we will be using in our plugin task:
`implementation group: 'com.googlecode.json-simple', name: 'json-simple', version: '1.1.1'`
c. Change the `jar` section to the following:
```groovy
jar {
manifest {
attributes(
"X-Kestra-Name": project.name,
"X-Kestra-Title": "Pokemon",
"X-Kestra-Group": project.group + ".pokemon",
"X-Kestra-Description": project.description,
"X-Kestra-Version": project.version
)
}
}
```
## Develop Fetch Task
### Create Pokemon class
In `src/main/java/io/kestra/plugin/pokemon`, we will create a new class `Pokemon.java`. This will be used to map the JSON output of the pokemon API to the Java class. We only need to add the fields that we are interested in, and ignore the rest.
:::collapse{title="Here is how the Pokemon.java file should look"}
```java
package io.kestra.plugin.pokemon;
import java.util.*;
import lombok.*;
@Data
public class Pokemon {
List abilities;
long base_experience;
long height;
List moves;
}
@Data
class DetailedAbility {
Ability ability;
}
@Data
class Ability {
String name;
}
@Data
class DetailedMove {
Move move;
}
@Data
class Move {
String name;
}
```
:::
### Runnable Task
We will refactor the java file from `Example.java` to `Fetch.java`.
In this file, we will put in the appropriate schema for the plugin, including the inputs and output of the plugin. This will help us generate documentation for the plugin too. Also, we will include a few examples to help users understand how to use the plugin.
The class should extend `Task` and implement `RunnableTask` for it to be considered as a plugin task. The `RunnableTask` has a generic representing the output class. The output class should implement `io.kestra.core.models.tasks.Output`.
The actual crux of the task logic resides in the `run` method. This is an override method from the `Task` class, and takes `RunContext` as an argument, and return the instance of the `Output` class.
In the `run` method, we use the name of the pokemon, and make a call to the pokemon API. The fetched response is then mapped to the Pokemon class using the `ObjectMapper`. The resulting `Pokemon` object is then transformed into the `Fetch.Output` class, and returned.
:::collapse{title="Here is a pokemon Fetch task that will fetch the details of a given pokemon"}
```java
package io.kestra.plugin.pokemon;
import io.kestra.core.models.annotations.Plugin;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.*;
import lombok.experimental.SuperBuilder;
import org.apache.commons.lang3.StringUtils;
import io.kestra.core.models.annotations.PluginProperty;
import io.kestra.core.models.tasks.RunnableTask;
import io.kestra.core.models.tasks.Task;
import io.kestra.core.runners.RunContext;
import org.slf4j.Logger;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.DeserializationFeature;
import java.io.*;
import java.net.*;
import java.util.*;
import io.kestra.plugin.pokemon.Pokemon;
@SuperBuilder
@ToString
@EqualsAndHashCode
@Getter
@NoArgsConstructor
@Schema(
title = "Fetch the pokemon details.",
description = "Fetches all the details about the given pokemon."
)
@Plugin(
examples = {
@io.kestra.core.models.annotations.Example(
title = "Fetching the details for pikachu",
code = {
"pokemon: gengar"
}
)
}
)
public class Fetch extends Task implements RunnableTask {
@Schema(
title = "Name of the pokemon.",
description = "Name of the pokemon for which details need to be fetched."
)
@PluginProperty(dynamic = true) // If the variables will be rendered with template {{ }}
@Builder.Default
private String pokemon = "pikachu";
@Override
public Fetch.Output run(RunContext runContext) throws Exception {
Logger logger = runContext.logger();
ObjectMapper om = new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
String inputPokemon = runContext.render(pokemon);
StringBuilder result = new StringBuilder();
URL url = new URL("https://pokeapi.co/api/v2/pokemon/" + inputPokemon);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
try (BufferedReader reader = new BufferedReader(
new InputStreamReader(conn.getInputStream()))) {
for (String line; (line = reader.readLine()) != null; ) {
result.append(line);
}
}
Pokemon pokemonObject = om.readValue(result.toString(), Pokemon.class);
List abilities = new ArrayList();
for(DetailedAbility detailedAbility: pokemonObject.abilities) {
abilities.add(detailedAbility.ability.name);
}
List moves = new ArrayList();
for(DetailedMove detailedMove: pokemonObject.moves) {
moves.add(detailedMove.move.name);
}
return Output.builder()
.abilities(abilities)
.baseExperience(pokemonObject.base_experience)
.height(pokemonObject.height)
.moves(moves)
.build();
}
/**
* Input or Output can be nested as you need
*/
@Builder
@Getter
public static class Output implements io.kestra.core.models.tasks.Output {
@Schema(
title = "Abilities of the pokemon."
)
private final List abilities;
@Schema(
title = "Whether the ability is hidden."
)
private final long baseExperience;
@Schema(
title = "Slot corresponding to the ability."
)
private final long height;
@Schema(
title = "Slot corresponding to the ability."
)
private final List moves;
}
}
```
:::
### Compile the plugin
Now that the plugin is developed, package and test it on a Kestra instance. Use the included Gradle task to build the plugin.
To build your plugin, execute the `./gradlew shadowJar` command from the plugin directory.
The resulting JAR file will be generated in the `build/libs` directory.
To use this plugin in your Kestra instance, add this JAR to the [Kestra plugins path](../../kestra-cli/kestra-server/index.md#plugin-commands).
## Writing unit tests
Refactor the file names from `ExampleRunnerTest.java` and `ExampleTest.java` to `FetchRunnerTest.java` and `FetchTest.java` respectively. Under the `tests/resources/flows` folder, rename `example.yaml` to `pokemonFetch.yaml`.
Use the following flow in `pokemonFetch.yaml`:
:::collapse{title="Contents of pokemonFetch.yaml"}
```yaml
id: pokemonFetch
namespace: company.team
tasks:
- id: fetch-pikachu
type: io.kestra.plugin.pokemon.Fetch
pokemon: "pikachu"
- id: fetch-gengar
type: io.kestra.plugin.pokemon.Fetch
pokemon: "gengar"
```
:::
Update `FetchRunnerTest.java` to load `pokemonFetch.yaml` and run the flow, then assert that all tasks were executed.
:::collapse{title="Contents of FetchRunnerTest.java"}
```java
package io.kestra.plugin.pokemon;
import io.kestra.core.junit.annotations.KestraTest;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import io.kestra.core.models.executions.Execution;
import io.kestra.core.repositories.LocalFlowRepositoryLoader;
import io.kestra.core.runners.RunnerUtils;
import io.kestra.core.runners.StandAloneRunner;
import jakarta.inject.Inject;
import java.io.IOException;
import java.net.URISyntaxException;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.TimeoutException;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.hasSize;
import static org.hamcrest.Matchers.is;
/**
* This test will load all flow located in `src/test/resources/flows/`
* and will run an in-memory runner to be able to test a full flow. There is also a
* configuration file in `src/test/resources/application.yml` that is only for the full runner
* test to configure in-memory runner.
*/
@KestraTest
class FetchRunnerTest {
@Inject
protected StandAloneRunner runner;
@Inject
protected RunnerUtils runnerUtils;
@Inject
protected LocalFlowRepositoryLoader repositoryLoader;
@BeforeEach
protected void init() throws IOException, URISyntaxException {
repositoryLoader.load(Objects.requireNonNull(FetchRunnerTest.class.getClassLoader().getResource("flows")));
this.runner.run();
}
@SuppressWarnings("unchecked")
@Test
void flow() throws TimeoutException {
Execution execution = runnerUtils.runOne(null, "io.kestra.plugin", "pokemonFetch");
assertThat(execution.getTaskRunList(), hasSize(2));
}
}
```
:::
Test the plugin logic in `FetchTest.java` by creating the input, invoking the task, and verifying the output.
:::collapse{title="Contents of FetchTest.java"}
```java
package io.kestra.plugin.pokemon;
import com.google.common.collect.ImmutableMap;
import io.micronaut.test.extensions.junit5.annotation.MicronautTest;
import org.apache.commons.lang3.StringUtils;
import org.junit.jupiter.api.Test;
import io.kestra.core.runners.RunContext;
import io.kestra.core.runners.RunContextFactory;
import jakarta.inject.Inject;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
/**
* This test will only test the main task, this allow you to send any input
* parameters to your task and test the returning behavior easily.
*/
@MicronautTest
class FetchTest {
@Inject
private RunContextFactory runContextFactory;
@Test
void run() throws Exception {
RunContext runContext = runContextFactory.of(ImmutableMap.of("variable", "gengar"));
Fetch task = Fetch.builder()
.pokemon("{{ variable }}")
.build();
Fetch.Output runOutput = task.run(runContext);
assertThat(runOutput.getBaseExperience(), is(250L));
assertThat(runOutput.getHeight(), is(15L));
assertThat(runOutput.getAbilities().size(), is(1));
assertThat(runOutput.getMoves().size(), is(123));
}
}
```
:::
### Running the tests
You can run the tests from Intellij IDE, or from the terminal using the command:
```bash
./gradlew test
```
## Plugin in Action
Now that the plugin is developed and tested, its time to see the plugin in action.
### Use a custom docker image with your plugin
Add this `Dockerfile` to the root of your plugin project:
```dockerfile
FROM kestra/kestra:develop
COPY build/libs/* /app/plugins/
```
You can build and run the image with the following command assuming you're in the root directory of your plugin:
`./gradlew shadowJar && docker build -t kestra-custom . && docker run --rm -p 8080:8080 kestra-custom server local`
You can now navigate to http://localhost:8080 and start using your custom plugin.
### Execute the plugin and check the Output
Create a new flow, and use this newly-built plugin's task in the flow.
Here is a sample flow:
```yaml
id: pokemonFetch
namespace: company.team
tasks:
- id: fetch-pikachu
type: io.kestra.plugin.pokemon.Fetch
pokemon: "pikachu"
```
On executing the flow, navigate to the `Outputs` tab to view the output.

You are now all set to build more plugins and explore Kestra to its fullest!
---
# Build a KPI Success Ratio Chart in Kestra Dashboards
URL: https://kestra.io/docs/how-to-guides/dashboard-kpi-chart
> Build a KPI success ratio chart in Kestra using the No Code dashboard editor, then extend it to track failure rates.
Build a KPI chart that displays the percentage of successful executions using Kestra's No Code dashboard editor.
## Prerequisites
- A Kestra instance with at least one completed execution.
- Access to the **Dashboards** section in the UI.
## Steps
### 1. Open the dashboard editor
From **Dashboards**, select **Default Dashboard** and click **+ Create Dashboard**.

In the editor, select the **No Code** tab. It appears alongside the YAML editor so you can view both as you work.

### 2. Set the dashboard properties
Give your dashboard an ID, title, description, and time window. Changes in the No Code form immediately reflect in the YAML editor.

### 3. Add a KPI chart
Click **+ Add** in the **charts** block. Choose **KPI Chart** as the chart type. Open the **Documentation** tab at any time to view chart-specific guidance without leaving the editor.

Give the chart an ID and set the data type to **Executions**. Set `field` to `ID` and `agg` to `COUNT` to count all executions.

### 4. Add a numerator filter
Click **+ Add** under the numerator section. Set `type` to `IN`, add `SUCCESS` as a value, and set `field` to `STATE`. This scopes the numerator to successful executions — the denominator remains all executions.

### 5. Set display options
Return to the `charts` No Code tab and open **Optional Properties**. Set `displayName`, change `numberType` to `PERCENTAGE`, and set `width` to `3`.

### 6. Preview and save
Open the **Preview** tab to review the chart. Click **Save** when satisfied.

## Extend: add a failure ratio chart
To add a failure ratio chart alongside the success ratio, copy the generated YAML for the KPI chart, paste it into the YAML editor as a second chart entry, and replace `SUCCESS` with `FAILED`. The two charts will sit side by side on the dashboard.
## Best practices
**Organize by purpose.** Group related charts into dashboards with a clear goal — for example, separate dashboards for system health, execution performance, and user activity.
**Use consistent naming.** A pattern like `team_metric_type` (e.g., `dataops_executions_latency`) makes dashboards easier to find, version, and export.
**Use YAML for reuse.** When charts share the same structure with small differences in filters or fields, copy-paste the YAML and modify — faster than rebuilding forms.
**Preview before saving.** Catch mismatched fields and aggregation errors early before they make it into a published dashboard.
## Next steps
See the [Dashboards reference](../../09.ui/00.dashboard/index.md) for the full list of chart types, data source fields, and filter options.
---
# Use Dataform in Kestra
URL: https://kestra.io/docs/how-to-guides/dataform
> Orchestrate DataForm transformations in Kestra. Schedule and run DataForm jobs in your data pipeline for reliable, version-controlled SQL-based data modeling.
Run transformations on BigQuery using Dataform in Kestra
Dataform is modern data pipeline tool based on Extract-Load-Transform (ELT). It has been acquired by Google Cloud and has been integrated within the BigQuery.
Similar to other ELT tools, Dataform handles the transformation on diffferent warehouses. Some of the data stores that are supported by Dataform include BigQuery, Snowflake, Redshift, etc. One of the advantages of using Dataform is that you can put together the transformation in the form of SQL, thus empowering multiple roles like Data Analysts and Data Scientists to perform the transformations. Being based on SQL makes it easier for anyone to onboard onto Dataform.
## Using Dataform with Kestra
There are two ways in which you can create a Dataform project while running with Kestra:
1. Create the Dataform project in GitHub, clone the GitHub project in Kestra and then run it using the [DataformCLI](/plugins/plugin-dataform/io.kestra.plugin.dataform.cli.dataformcli) task.
2. Create the Dataform project in Kestra using [Namespace Files](../../../docs/06.concepts/02.namespace-files/index.md), and then run it using the [DataformCLI](/plugins/plugin-dataform/io.kestra.plugin.dataform.cli.dataformcli) task. You can later choose to push the Namespace Files into GitHub repository using [PushNamespaceFiles](/plugins/plugin-git/io.kestra.plugin.git.pushnamespacefiles) task.
This guide covers both methods for transforming data using Dataform in Kestra for BigQuery.
### Using GitHub repository
Here is how you can pull an existing project from a GitHub repository and run it with DataformCLI task:
```yaml
id: dataform
namespace: company.team
tasks:
- id: wdir
type: io.kestra.plugin.core.flow.WorkingDirectory
tasks:
- id: clone_repo
type: io.kestra.plugin.git.Clone
url: https://github.com/dataform-co/dataform-example-project-bigquery
- id: transform
type: io.kestra.plugin.dataform.cli.DataformCLI
beforeCommands:
- npm install @dataform/core
- dataform compile
env:
GOOGLE_APPLICATION_CREDENTIALS: "sa.json"
inputFiles:
sa.json: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
.df-credentials.json: |
{
"projectId": "",
"location": "us"
}
commands:
- dataform run --dry-run
```
The `clone_repo` task pulls the repository with the Dataform project, and the `transform` task executes the Dataform project.
### Using Dataform project creation in Kestra
First, create and save the Kestra flow. The flow contains the following tasks:
1. HTTP Download task that downloads the `orders.csv` file using HTTP URL.
2. BigQuery CreateTable task that creates the `orders` table in the `ecommerce` dataset.
3. BigQuery Load task that loads the `orders.csv` contents into the BigQuery `orders` table.
4. DataformCLI task that runs the Dataform project, created later using Namespace Files. The project creates the `stg_orders` BigQuery view based on the `orders` BigQuery table.
```yaml
id: dataform_project
namespace: company.team
tasks:
- id: orders_http_download
type: io.kestra.plugin.core.http.Download
description: Download orders.csv using HTTP Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/#main/csv/orders.csv
- id: create_orders_table
type: io.kestra.plugin.gcp.bigquery.CreateTable
description: Create orders table in BigQuery
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
projectId:
dataset: ecommerce
table: orders
tableDefinition:
type: TABLE
schema:
fields:
- name: order_id
type: INT64
- name: customer_name
type: STRING
- name: customer_email
type: STRING
- name: product_id
type: INT64
- name: price
type: FLOAT64
- name: quantity
type: INT64
- name: total
type: FLOAT64
- id: load_orders_table
type: io.kestra.plugin.gcp.bigquery.Load
description: Load orders table with data from orders.csv
from: "{{ outputs.orders_http_download.uri }}"
projectId:
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
destinationTable: ".ecommerce.orders"
format: CSV
csvOptions:
fieldDelimiter: ","
skipLeadingRows: 1
- id: dataform_cli
type: io.kestra.plugin.dataform.cli.DataformCLI
beforeCommands:
- npm install @dataform/core
- dataform compile
namespaceFiles:
enabled: true
env:
GOOGLE_APPLICATION_CREDENTIALS: "sa.json"
inputFiles:
sa.json: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
.df-credentials.json: |
{
"projectId": "",
"location": "us"
}
commands:
- dataform run
```
Once the flow is saved, navigate to the Editor, and create a file `package.json` with the contents:
```json
{
"dependencies": {
"@dataform/core": "2.3.0"
}
}
```
This file is not required for Kestra execution, as the dependency is installed via `beforeCommands`. It is required if you push the namespace files to a GitHub repository so you can run the project in other ways.
Next, create `dataform.json`.
```json
{
"warehouse": "bigquery",
"defaultSchema": "ecommerce",
"defaultDatabase": "",
"defaultLocation": "us"
}
```
Most often, the `database` is same as the GCP project ID.
Create a `definitions` folder. Inside it, create a file `orders.sqlx` to define the `orders` table as the source table:
```javascript
config {
type: "declaration",
database: "",
schema: "ecommerce",
name: "orders",
description: "raw orders table"
}
```
Next, create `stg_orders.sqlx` under the `definitions` folder to define the `stg_orders` view:
```javascript
config {
type: "view", // Specify whether this model will create a table or a view
schema: "ecommerce",
database: ""
}
select
order_id,
customer_name,
customer_email,
product_id,
price,
quantity,
total
from
${ref("orders")}
```
That's it! We are now ready to run the flow.
Once the flow runs successfully, you can go to the BigQuery console, and ensure that the view `stg_orders` has been created.
This is how we can run Dataform for BigQuery in Kestra. These instructions can also help you integrate the DataformCLI task with other data stores like Snowflake, Redshift, Postgres and more.
---
# Manage dbt Projects with Kestra's Code Editor
URL: https://kestra.io/docs/how-to-guides/dbt
> Clone dbt projects from Git, edit models in Kestra's Code Editor, run tests, and push changes back to Git for seamless dbt project management.
Edit dbt code from Kestra's Code Editor
Kestra's built-in Code Editor allows you to manage dbt projects by cloning the Git repository with the dbt code and uploading it to your Kestra namespace. You can make changes to the dbt models directly from the Kestra UI, test them as part of an end-to-end workflow, and push the changes to the desired Git branch when you are ready.
## Clone a dbt project from Git
This flow pulls a dbt project from Git and uploads it to Kestra as Namespace Files:
```yaml
id: upload_dbt_project
namespace: company.datateam.dbt
tasks:
- id: wdir
type: io.kestra.plugin.core.flow.WorkingDirectory
tasks:
- id: git_clone
type: io.kestra.plugin.git.Clone
url: https://github.com/kestra-io/dbt-example
branch: master
- id: upload
type: io.kestra.plugin.core.namespace.UploadFiles
namespace: "{{ flow.namespace }}"
files:
- "glob:**/dbt/**"
```
You can use this flow as an initial setup:
1. Add this flow within Kestra UI
2. Save it
3. Execute that flow
4. Click on the `Files` sidebar in the code editor to view the uploaded dbt files.

## Run dbt CLI commands
Create a flow that runs dbt CLI commands:
```yaml
id: dbt_build
namespace: company.datateam.dbt
inputs:
- id: dbt_command
type: SELECT
allowCustomValue: true
defaults: dbt build --project-dir dbt --profiles-dir dbt --no-partial-parse --target prod
values:
- dbt build --project-dir dbt --profiles-dir dbt --no-partial-parse --target prod
- dbt build --project-dir dbt --profiles-dir dbt --no-partial-parse --target prod --select state:modified+ --defer --state ./target --target-path ./dev
tasks:
- id: dbt
type: io.kestra.plugin.dbt.cli.DbtCLI
namespaceFiles:
enabled: true
containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
projectDir: dbt
commands:
- "{{ inputs.dbt_command }}"
loadManifest:
key: manifest.json
namespace: "{{ flow.namespace }}"
storeManifest:
key: manifest.json
namespace: "{{ flow.namespace }}"
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
```
The `namespaceFiles` property lets you run dbt commands on the files uploaded to the namespace. This allows you to test the dbt models without having to build the entire project every time.
Execute the flow using the default value for the `dbt_command` input.
## Edit dbt files
You can now open the dbt files in the Code Editor and make changes as needed. For example, add a new model `my_third_dbt_model.sql`:
```sql
select *
from {{ ref('my_first_dbt_model') }}
where id = 2
```

When you now run the flow using the second dropdown value for the `dbt_command` input, only the new model will be built. This allows you to test the changes quickly and iterate faster.
## Push changes to Git
Once you are satisfied with the changes, you can push them to the same Git repository to your desired Git branch using the [PushNamespaceFiles](../pushnamespacefiles/index.md).
```yaml
id: push_dbt_to_git
namespace: company.datateam.dbt
inputs:
- id: commit_message
type: STRING
defaults: "Changes to dbt from Kestra"
tasks:
- id: commit_and_push
type: io.kestra.plugin.git.PushNamespaceFiles
namespace: "{{ flow.namespace }}"
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/git_username/scripts
branch: dev
gitDirectory: dbt
commitMessage: "{{ inputs.commit_message }}"
```
Adjust the `url`, `branch`, and `gitDirectory` properties to match your dbt Git repository structure. If the branch does not exist, it will be created. If you want to test this step more incrementally, you can set the `dryRun` property to `true` to validate the changes before committing them to Git.
---
# Use Debezium Tasks and Triggers in Kestra
URL: https://kestra.io/docs/how-to-guides/debezium
> Enable Change Data Capture (CDC) in your databases to use Debezium tasks and triggers in Kestra for real-time data ingestion.
To use Debezium tasks and triggers, enable Change Data Capture (CDC) on your database. Each connector has different prerequisites — follow the links to the Debezium documentation for complete setup steps.
## MySQL
Before using the Debezium MySQL connector, you must:
- Create a MySQL user with `SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT` privileges
- Enable binary logging with `binlog_format = ROW` and `binlog_row_image = FULL`
- Optionally enable GTIDs for simpler replication management
See the [Debezium MySQL connector documentation](https://debezium.io/documentation/reference/3.0/connectors/mysql.html) for complete setup instructions, including hosted options such as Amazon RDS and Aurora.
### Kestra flow
```yaml
id: debezium_mysql
namespace: company.team
tasks:
- id: send_data
type: io.kestra.plugin.core.log.Log
message: "{{ trigger.data }}"
triggers:
- id: realtime
type: io.kestra.plugin.debezium.mysql.RealtimeTrigger
serverId: 123456789
hostname: 127.0.0.1
port: 63306
username: mysql_user
password: mysql_passwd
```
The `RealtimeTrigger` collects records from change data capture as they occur. The flow processes each record when it arrives.
## PostgreSQL
Before using the Debezium PostgreSQL connector, enable Write-Ahead Logging (WAL) on your PostgreSQL server. The `pgoutput` logical decoding plugin is available natively since PostgreSQL 10 and is the recommended choice — no additional installation required.
See the [Debezium PostgreSQL connector documentation](https://debezium.io/documentation/reference/3.0/connectors/postgresql.html) for complete setup instructions, including cloud-hosted options such as Amazon RDS, Azure Database for PostgreSQL, and CrunchyBridge.
### Kestra flow
```yaml
id: debezium_postgres
namespace: company.team
tasks:
- id: send_data
type: io.kestra.plugin.core.log.Log
message: "{{ trigger.data }}"
triggers:
- id: realtime
type: io.kestra.plugin.debezium.postgres.RealtimeTrigger
database: postgres
hostname: 127.0.0.1
port: 65432
username: postgres
password: pg_passwd
```
## Microsoft SQL Server
Before using the Debezium SQL Server connector, a database administrator must enable CDC on the SQL Server database and on each table to capture. The SQL Server Agent must be running.
See the [Debezium SQL Server connector documentation](https://debezium.io/documentation/reference/3.0/connectors/sqlserver.html) for complete setup instructions, including SQL Server on Azure and Always On read-only replicas.
### Kestra flow
```yaml
id: debezium_sqlserver
namespace: company.team
tasks:
- id: send_data
type: io.kestra.plugin.core.log.Log
message: "{{ trigger.data }}"
triggers:
- id: realtime
type: io.kestra.plugin.debezium.sqlserver.RealtimeTrigger
hostname: 127.0.0.1
port: 61433
username: sa
password: password
database: deb
```
## MongoDB
Before using the Debezium MongoDB connector, MongoDB must be configured as a replica set or sharded cluster. Create a MongoDB user with the permissions to read the `admin` database (for oplog access) and `listDatabases`, `find`, and `changeStream` cluster-wide privilege actions.
See the [Debezium MongoDB connector documentation](https://debezium.io/documentation/reference/3.0/connectors/mongodb.html) for complete setup instructions, including MongoDB Atlas (SSL required).
### Kestra flow
```yaml
id: debezium_mongodb
namespace: company.team
tasks:
- id: send_data
type: io.kestra.plugin.core.log.Log
message: "{{ trigger.data }}"
triggers:
- id: realtime
type: io.kestra.plugin.debezium.mongodb.RealtimeTrigger
snapshotMode: INITIAL
connectionString: mongodb://mongo_user:mongo_passwd@mongos0.example.com:27017,mongos1.example.com:27017/
```
---
# Run C# Inside Your Flows
URL: https://kestra.io/docs/how-to-guides/dotnet
> Run C# scripts in Kestra using dotnet-script. Automate .NET workflows, reference NuGet packages inline, and capture outputs for downstream tasks.
Run C# code directly in your flows using [dotnet-script](https://github.com/dotnet-script/dotnet-script).
Write inline `.csx` scripts with NuGet package references, or run arbitrary `dotnet` CLI commands inside a .NET SDK container.
## Scripts
Use `io.kestra.plugin.scripts.dotnet.Script` to write C# code inline in your flow. The script is written to a temporary `.csx` file and executed with `dotnet-script`, which is installed automatically before each run. The default container image is `mcr.microsoft.com/dotnet/sdk:10.0`; set `containerImage` to override it, for example to pin an older SDK version.
```yaml
id: dotnet_script
namespace: company.team
tasks:
- id: hello_csharp
type: io.kestra.plugin.scripts.dotnet.Script
script: |
Console.WriteLine("Hello from Kestra and C#");
```
### NuGet package references
Reference NuGet packages inline with `#r "nuget:PackageName,Version"` directives at the top of your script. The first run with a new package triggers a NuGet restore that may take 30–60 seconds; subsequent runs use the cached result.
```yaml
id: dotnet_script_nuget
namespace: company.team
tasks:
- id: parse_json
type: io.kestra.plugin.scripts.dotnet.Script
script: |
#r "nuget:Newtonsoft.Json,13.0.3"
using Newtonsoft.Json;
var data = new { message = "Hello from Kestra", timestamp = DateTime.UtcNow };
Console.WriteLine(JsonConvert.SerializeObject(data));
```
For the full property list, see the [Script plugin documentation](/plugins/plugin-script-dotnet/io.kestra.plugin.scripts.dotnet.script).
## Commands
Use `io.kestra.plugin.scripts.dotnet.Commands` when your C# code lives in namespace files or when you need direct `dotnet` CLI access.
`dotnet-script` is not pre-installed in the default image. Add it to `beforeCommands` when running `.csx` files:
```yaml
id: dotnet_commands
namespace: company.team
tasks:
- id: run_script_file
type: io.kestra.plugin.scripts.dotnet.Commands
namespaceFiles:
enabled: true
beforeCommands:
- dotnet tool install -g dotnet-script --ignore-failed-sources || true
- export PATH="$PATH:$HOME/.dotnet/tools"
commands:
- dotnet-script main.csx
```
Add `main.csx` to your namespace using the Editor or by [syncing from Git](../../version-control-cicd/04.git/index.md). The file can contain:
```csharp
Console.WriteLine("Hello from Kestra");
```
You can also pass a script inline using `inputFiles`:
```yaml
id: dotnet_commands_inline
namespace: company.team
tasks:
- id: run_inline
type: io.kestra.plugin.scripts.dotnet.Commands
inputFiles:
main.csx: |
Console.WriteLine("Hello from Kestra");
beforeCommands:
- dotnet tool install -g dotnet-script --ignore-failed-sources || true
- export PATH="$PATH:$HOME/.dotnet/tools"
commands:
- dotnet-script main.csx
```
For the full property list, see the [Commands plugin documentation](/plugins/plugin-script-dotnet/io.kestra.plugin.scripts.dotnet.commands).
## Handling outputs
### Variable output
Emit key-value outputs from your script using the `::{}::` pattern. Kestra captures any line matching this format and makes the values available to downstream tasks.
```yaml
id: dotnet_outputs
namespace: company.team
tasks:
- id: emit_output
type: io.kestra.plugin.scripts.dotnet.Script
script: |
Console.WriteLine("::{\"outputs\":{\"test\":\"value\",\"int\":2,\"bool\":true,\"float\":3.65}}::");
- id: use_output
type: io.kestra.plugin.core.debug.Return
format: "{{ outputs.emit_output.vars.test }}"
```
All output variables are visible in the **Outputs** tab of the execution.
### File output
Write a file inside your script and declare it in `outputFiles`. Kestra captures the file into internal storage and makes it available for download or use in downstream tasks.
```yaml
id: dotnet_output_file
namespace: company.team
tasks:
- id: write_file
type: io.kestra.plugin.scripts.dotnet.Script
outputFiles:
- output.txt
script: |
File.WriteAllText("output.txt", "Hello from C#");
- id: log_file
type: io.kestra.plugin.core.log.Log
message: "{{ read(outputs.write_file.outputFiles['output.txt']) }}"
```
## Handling metrics
Emit metrics using the same `::{}::` pattern as outputs. Kestra supports `counter` and `timer` metric types.
```yaml
id: dotnet_metrics
namespace: company.team
tasks:
- id: emit_metrics
type: io.kestra.plugin.scripts.dotnet.Script
script: |
Console.WriteLine("::{\"metrics\":[{\"name\":\"recordCount\",\"type\":\"counter\",\"value\":100}]}::");
Console.WriteLine("::{\"metrics\":[{\"name\":\"processingTime\",\"type\":\"timer\",\"value\":1.23}]}::");
```
Metrics appear under the **Metrics** tab of the execution after it completes.
---
# Build Dynamic Flows in Kestra
URL: https://kestra.io/docs/how-to-guides/dynamic-flows
> Create dynamic Kestra flows at runtime using inputs and Pebble expressions. Generate flow configurations on the fly for data-driven, adaptive orchestration.
Implement dynamic flows in Kestra.
## Dynamic Flows using Inputs
In this method, we will create a flow as a template, and the dynamic values in the template can then be filled using Kestra inputs to generate the desired flow. Let us see this with the help of an example.
Here, we will create a sample flow that downloads CSV data using the HTTP Download task and then loads the data to a PostgreSQL table. Such a dynamic flow can be helpful when you have new HTTP URLs getting generated on a regular cadence, and you need to pull in the latest data from the new HTTP URL to upload to a table.
The flow will take the HTTP URL and the PostgreSQL database connection details as inputs. This leads to a dynamic flow, as the same flow can then be utilized with different HTTP URLs and different PostgreSQL databases and tables.
```yaml
id: dynamic_flow
namespace: company.team
inputs:
- id: http_url
type: STRING
defaults: "https://huggingface.co/datasets/kestra/datasets/raw/main/csv/products.csv"
- id: postgres_host
type: STRING
defaults: "localhost"
- id: postgres_port
type: STRING
defaults: "5432"
- id: postgres_db
type: STRING
defaults: "postgres"
- id: postgres_table
type: STRING
- id: postgres_username
type: STRING
- id: postgres_password
type: STRING
tasks:
- id: http_download
type: io.kestra.plugin.core.http.Download
uri: "{{ inputs.http_url }}"
- id: copyin
type: io.kestra.plugin.jdbc.postgresql.CopyIn
url: "jdbc:postgresql://{{ inputs.postgres_host }}:{{ inputs.postgres_port }}/{{ inputs.postgres_db }}"
username: "{{ inputs.postgres_username }}"
password: "{{ inputs.postgres_password }}"
format: CSV
from: "{{ outputs.http_download.uri }}"
table: "{{ inputs.postgres_table }}"
header: true
```
As can be seen from the above flow, it is dynamic as all its important parameters are controlled via inputs.
## Dynamic Flow using Code
We can write code in any language to generate the dynamic flow, and then upload the flow to Kestra. Let us understand this with the help of an example.
We will create a dynamic flow using python which downloads a CSV file using the HTTP Download task and upload the contents into PostgreSQL table. Say, we want to extract the data from multiple HTTP URLs and upload the data to corresponding a PostgreSQL table. We can, in parallel, start the process of downloading the data from HTTP URL and uploading it to PostgreSQL table. For two items, products and orders, this is how our flow should look like:
```yaml
id: dynamic_flow
namespace: company.team
tasks:
- id: parallel
type: io.kestra.plugin.core.flow.Parallel
tasks:
- id: sequential_task_0
type: io.kestra.plugin.core.flow.Sequential
tasks:
- id: http_download_0
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/products.csv
- id: postgres_upload_0
type: io.kestra.plugin.jdbc.postgresql.CopyIn
url: jdbc:postgresql://{{ kv('postgres_host', 'company.infra') }}:{{ kv('postgres_port',
'company.infra') }}/{{ kv('postgres_db', 'company.infra') }}
username: "{{ secret('POSTGRES_USERNAME') }}"
password: "{{ secret('POSTGRES_PASSWORD') }}"
format: CSV
from: '{{ outputs.http_download_0.uri }}'
table: products
header: true
- id: sequential_task_1
type: io.kestra.plugin.core.flow.Sequential
tasks:
- id: http_download_1
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: postgres_upload_1
type: io.kestra.plugin.jdbc.postgresql.CopyIn
url: jdbc:postgresql://{{ kv('postgres_host', 'company.infra') }}:{{ kv('postgres_port',
'company.infra') }}/{{ kv('postgres_db', 'company.infra') }}
username: "{{ secret('POSTGRES_USERNAME') }}"
password: "{{ secret('POSTGRES_PASSWORD') }}"
format: CSV
from: '{{ outputs.http_download_1.uri }}'
table: orders
header: true
```
To generate this flow dynamically for any number of items, use the following Python code.
```python
import os
from ruamel.yaml import YAML
## Get the items from the environment variable and split them by commas
items = os.getenv('EXTRACT_ITEMS', "products,orders").split(",")
def http_download_task(idx, item):
"""Create HTTP Download task based on the item. The task id and uri will get dynamically generated based on `idx` and `item` respectively.
"""
return {
"id": f"http_download_{str(idx)}",
"type": "io.kestra.plugin.core.http.Download",
"uri": f"https://huggingface.co/datasets/kestra/datasets/raw/main/csv/{item}.csv"
}
def postgres_upload_task(idx, item):
"""Create postgres CopyIn task to upload the data from CSV to the corresponding postgres table.
"""
return {
"id": f"postgres_upload_{str(idx)}",
"type": "io.kestra.plugin.jdbc.postgresql.CopyIn",
"url": "jdbc:postgresql://" + "{{ kv('postgres_host', 'company.infra') }}:{{ kv('postgres_port', 'company.infra') }}/{{ kv('postgres_db', 'company.infra') }}",
"username": "{{ secret('POSTGRES_USERNAME') }}",
"password": "{{ secret('POSTGRES_PASSWORD') }}",
"format": "CSV",
"from": "{{ outputs.http_download_" + str(idx) + ".uri }}",
"table": f"{item}",
"header": True
}
def create_sequential_task(idx, task_list):
"""Create Sequential task for every item which will have two tasks:
1. Download the CSV data using HTTP Download task
2. Upload the CSV file into the corresponding postgres table using CopyIn task
"""
return {
"id": f"sequential_task_{str(idx)}",
"type": "io.kestra.plugin.core.flow.Sequential",
"tasks": task_list
}
tasks_per_item = []
## Iterate over the items and generate Sequential task for each item, and append it to `tasks_per_item`
for idx, item in enumerate(items):
sequential_tasks = []
sequential_tasks.append(http_download_task(idx, item))
sequential_tasks.append(postgres_upload_task(idx, item))
sequential_task = create_sequential_task(idx, sequential_tasks)
tasks_per_item.append(sequential_task)
## Generate the dynamic flow
kestra_flow = {
"id": os.getenv('FLOW_ID', "postgres_upload_flow"),
"namespace": os.getenv('FLOW_NAMESPACE', "company.team"),
"tasks": [
{
"id": "parallel",
"type": "io.kestra.plugin.core.flow.Parallel",
"tasks": tasks_per_item
}
]
}
yaml = YAML()
yaml.indent(mapping=2, sequence=4, offset=2)
yaml.preserve_quotes = True
## Write the generated dynamic flow in yaml format in `kestra_flow.yaml` file
output_path = "kestra_flow.yaml"
with open(output_path, "w") as f:
yaml.dump(kestra_flow, f)
```
The above python code will generate a dynamic flow with multiple Sequential tasks that will download the data from an HTTP URL and upload the CSV into the corresponding PostgreSQL table. You can write the above code in a namespace file, say `dynamic_flow.py`.
Next, we will write a Kestra flow that will invoke the `dynamic_flow.py` python script and load the generated dynamic flow into Kestra.
```yaml
id: generate_dynamic_flow
namespace: company.team
inputs:
- id: flow_id
type: STRING
description: Name for the dynamic flow to be created
defaults: dynamic_flow
- id: flow_namespace
type: STRING
description: Namespace in which the dynamic flow is to be created
defaults: company.team
- id: extract_items
type: STRING
description: Comma separated list of items to be extracted
defaults: products,orders
- id: kestra_host
type: STRING
description: Your Kestra hostname
defaults: "localhost:8080"
tasks:
- id: generate_kestra_flow
type: io.kestra.plugin.scripts.python.Commands
env:
FLOW_ID: "{{ inputs.flow_id }}"
FLOW_NAMESPACE: "{{ inputs.flow_namespace }}"
EXTRACT_ITEMS: "{{ inputs.extract_items }}"
beforeCommands:
- pip install -q ruamel.yaml
namespaceFiles:
enabled: true
inputFiles:
script.py: "{{ read('dynamic_flow.py') }}"
commands:
- python script.py
outputFiles:
- "*.yaml"
- id: create_flow
type: io.kestra.plugin.scripts.shell.Commands
inputFiles:
flow.yaml: "{{ outputs.generate_kestra_flow.outputFiles['kestra_flow.yaml'] }}"
beforeCommands:
- apt-get update
- apt-get -y install curl
commands:
- curl -X POST http://{{inputs.kestra_host}}/api/v1/main/flows/import -F fileUpload=@flow.yaml
- echo "Executing the flow from http://{{inputs.kestra_host}}/ui/flows/edit/{{ inputs.flow_namespace }}/{{ inputs.flow_id }}"
- id: subflow
type: io.kestra.plugin.core.flow.Subflow
namespace: "{{ inputs.flow_namespace }}"
flowId: "{{ inputs.flow_id }}"
wait: true
transmitFailed: true
```
The flow has the following tasks:
1. generate_kestra_flow: This task invokes the `dynamic_flow.py` and generates the dynamic flow by running the Python code. The environment variables like `FLOW_ID`, `FLOW_NAMESPACE`, and `EXTRACT_ITEMS` are provided in the task, which are then used by the Python script to dynamically generate the flow.
2. create_flow: This task creates the flow in Kestra by uploading the yaml file containing the dynamically generated flow.
3. subflow: This task triggers the newly created dynamic flow.
Thus, you can generate the dynamic flow by generating it in the language of your choice and then loading it into Kestra.
## Dynamic Flow using Terraform
Yet another way of generating dynamic flows in Kestra is using Terraform templates. Check out the detailed guide on implementing dynamic flows using [Terraform templating](../../15.how-to-guides/terraform-templating/index.md).
---
# Create a Dynamic Dropdown for Inputs
URL: https://kestra.io/docs/how-to-guides/dynamic-inputs
> Create dynamic dropdown menus for flow inputs that populate from external sources like databases or APIs using the KV store or HTTP functions.
Support dynamic dropdown for inputs based on data from external source.
In this guide, we show how you can create a dynamic dropdown list for inputs. The dropdown retrieves the values from an external source. It is possible to do so by storing the values in the [KV store](../../06.concepts/05.kv-store/index.md), and also to directly integrate the external source with the HTTP Pebble function, `http()`.
## Update KV store on schedule
To get started, we create a flow that fetches the data from the external source and set the value in the KV store. The value will be in the form of a list of strings.
In this example, the flow fetches data from a PostgreSQL table on an hourly schedule. You can change the `cron` property to run at a different frequency depending on how frequently you expect the data at the source to change. If the external source is in a database that supports change data capture, as in this case where we use PostgreSQL table, you can also use the [debezium trigger](/plugins/plugin-debezium-postgres/io.kestra.plugin.debezium.postgres.trigger) and immediately update the KV store.
```yaml
id: update_kv_store
namespace: company.team
tasks:
- id: fetch_departments
type: io.kestra.plugin.jdbc.postgresql.Query
url: "jdbc:postgresql://{{ secret('POSTGRES_HOST') }}:5432/postgres"
username: "{{ secret('POSTGRES_USERNAME') }}"
password: "{{ secret('POSTGRES_PASSWORD') }}"
sql: select department_name from departments
fetchType: FETCH
- id: department_key
type: io.kestra.plugin.core.kv.Set
key: "{{ task.id }}"
kvType: JSON
value: "{{ outputs.fetch_departments.rows | jq('.[].department_name') }}"
triggers:
- id: schedule
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 */1 * * *"
```
This is how the KV store will look post execution of the above flow.

## Flow supporting Dynamic Inputs
Let us now create the flow that supports dynamic dropdown for inputs powered by the KV store key.
```yaml
id: dynamic_input_flow
namespace: company.team
inputs:
- id: department
displayName: Department Name
type: SELECT
expression: "{{ kv('department_key') }}"
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: "The selected department is {{ inputs.department }}"
```
When you execute this flow, the `department` input will have a dropdown that contains the values fetched from the `department_key` key in the KV store.

## Dynamic Inputs with HTTP function
With the `http()` function, you can make `SELECT` and `MULTISELECT` inputs dynamic by fetching options from an external API. This proves valuable when your data used in dropdowns changes frequently or when you already have an API serving that data for existing applications.
The example below demonstrates how to create a flow with two dynamic dropdowns: one for selecting a product category and another for selecting a product from that category. The first dropdown fetches the product categories from an external HTTP API. The second dropdown makes another HTTP call to dynamically retrieve products matching the selected category.
```yaml
id: dynamic_dropdowns
namespace: company.team
inputs:
- id: category
type: SELECT
expression: "{{ http(uri = 'https://dummyjson.com/products/categories') | jq('.[].slug') }}"
- id: product
type: SELECT
dependsOn:
inputs:
- category
expression: "{{ http(uri = 'https://dummyjson.com/products/category/' + inputs.category) | jq('.products[].title') }}"
tasks:
- id: display_selection
type: io.kestra.plugin.core.log.Log
message: |
You selected Category: {{ inputs.category }}
And Product: {{ inputs.product }}
```
---
Dynamic inputs are useful for flows using authenticated API requests like the following:
```yaml
id: approversFlow
namespace: company.team
inputs:
- id: executionIdsToBeApproved
type: MULTISELECT
expression: >-
{{
http(
uri = 'http://localhost:8080/api/v1/internal/executions/search?state=PAUSED',
method = 'GET',
contentType = 'application/json',
headers={
'User-Agent': 'kestra',
'Connection': 'keep-alive',
'Authorization': 'Bearer ' ~ secret("bearerToken")
}
) | jq('.results[] | "ExecutionId: \(.id), FlowId: \(.flowId), RequestedBy: \(.labels[] | select(.key == "system.username").value) InputParams: \( .inputs | to_entries | map("\(.key):\(.value)") | join(" ") )"') }}
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Hello World! 🚀
```
:::alert{type="info"}
When using `http()` inside an `expression` with secrets in headers (e.g., an authenticated API request), use named arguments and string concatenation ([Pebble Literals](https://pebbletemplates.io/wiki/guide/basic-usage/#literals)). The key to the syntax is to use string interpolation with `~`.
:::
## Populate a dropdown from a subflow
When `kv()` and `http()` are not enough — for example, when you need to run a script task, call a CLI command (`aws ec2 describe-instances`, `gcloud projects list`), or execute complex multi-step logic — use the `subflow()` Pebble function.
`subflow()` runs a subflow synchronously at form render time and exposes its flow-level outputs as the dropdown values. The main flow does not start until the subflow finishes and the form is submitted.
**Step 1 — Create the data-fetching subflow.** This flow queries your infrastructure and returns a list as a flow-level output:
```yaml
id: fetch_aws_regions
namespace: company.ops
tasks:
- id: get_regions
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- |
regions=$(aws ec2 describe-regions --query 'Regions[].RegionName' --output json)
echo "::$(printf '{"outputs":{"regions":%s}}' "$regions")::"
outputs:
- id: regions
type: JSON
value: "{{ outputs.get_regions.vars.regions }}"
```
The `::{"outputs":{"key":"value"}}::` line is Kestra's [script output format](../../16.scripts/06.outputs-metrics/index.md) — it's how `shell.Commands` tasks publish named values that downstream expressions can reference via `outputs..vars.`.
**Step 2 — Reference it from a SELECT input in your main flow:**
```yaml
id: deploy_to_region
namespace: company.ops
inputs:
- id: region
type: SELECT
displayName: AWS Region
expression: "{{ subflow(namespace='company.ops', id='fetch_aws_regions').outputs.regions }}"
tasks:
- id: deploy
type: io.kestra.plugin.core.log.Log
message: "Deploying to {{ inputs.region }}"
```
When a user opens the Execute form, Kestra runs `fetch_aws_regions` synchronously and populates the dropdown from its output.
### Chaining dropdowns with `dependsOn`
You can chain dropdowns so the second list depends on the first selection:
```yaml
inputs:
- id: environment
type: SELECT
expression: "{{ subflow(namespace='company.ops', id='fetch_environments').outputs.envs }}"
- id: cluster
type: SELECT
dependsOn:
inputs:
- environment
expression: "{{ subflow(namespace='company.ops', id='fetch_clusters', inputs={'env': inputs.environment}).outputs.clusters }}"
```
**Constraints to be aware of:**
- `subflow()` is only valid in the `expression:` property of a `SELECT` or `MULTISELECT` input. It throws if used in a task or trigger property.
- The subflow must complete within the timeout (default `PT1M`, max `PT5M`). Keep data-fetching subflows fast.
- Recursion is capped at depth 3.
- Each subflow referenced in a `SELECT` or `MULTISELECT` expression appears in the parent flow's **Dependencies** graph automatically.
## Conditional inputs
Use `dependsOn` and `condition` to show an input only when a previous input matches a value. The following flow shows different inputs depending on which resource type the user selects:
```yaml
id: request_resources
namespace: company.team
inputs:
- id: resource_type
displayName: Resource type
type: SELECT
values:
- Access permissions
- SaaS application
- Cloud VM
- id: access_permissions
displayName: Access permissions
type: SELECT
expression: "{{ kv('access_permissions') }}"
dependsOn:
inputs:
- resource_type
condition: "{{ inputs.resource_type == 'Access permissions' }}"
- id: saas_applications
displayName: SaaS application
type: MULTISELECT
expression: "{{ kv('saas_applications') }}"
dependsOn:
inputs:
- resource_type
condition: "{{ inputs.resource_type == 'SaaS application' }}"
- id: cloud_provider
displayName: Cloud provider
type: SELECT
values:
- AWS
- GCP
- Azure
dependsOn:
inputs:
- resource_type
condition: "{{ inputs.resource_type == 'Cloud VM' }}"
- id: cloud_vm
displayName: Cloud VM
type: SELECT
expression: "{{ kv('cloud_vms')[inputs.cloud_provider] }}"
dependsOn:
inputs:
- resource_type
- cloud_provider
condition: "{{ inputs.resource_type == 'Cloud VM' }}"
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "Resource type: {{ inputs.resource_type }}"
```
`dependsOn.inputs` lists the inputs that must be provided first. `dependsOn.condition` is a Pebble expression that controls visibility — the dependent input only appears in the Execute modal when the condition is `true`. An input can depend on multiple parents; all listed inputs must be provided before the condition is evaluated.
Populate the KV store keys before running the flow:
:::collapse{title="Flow to add key-value pairs"}
```yaml
id: add_kv_pairs
namespace: company.team
tasks:
- id: access_permissions
type: io.kestra.plugin.core.kv.Set
key: "{{ task.id }}"
kvType: JSON
value: |
["Admin", "Developer", "Editor", "Launcher", "Viewer"]
- id: saas_applications
type: io.kestra.plugin.core.kv.Set
key: "{{ task.id }}"
kvType: JSON
value: |
["Slack", "Notion", "HubSpot", "GitHub", "Jira"]
- id: cloud_vms
type: io.kestra.plugin.core.kv.Set
key: "{{ task.id }}"
kvType: JSON
value: |
{
"AWS": ["t2.micro", "t2.small", "t2.medium", "t2.large"],
"GCP": ["f1-micro", "g1-small", "n1-standard-1", "n1-standard-2"],
"Azure": ["Standard_B1s", "Standard_B1ms", "Standard_B2s", "Standard_B2ms"]
}
```
:::
### dependsOn inside FORM inputs
To make one child input inside a FORM depend on another, use the full dotted path in `dependsOn.inputs`:
```yaml
inputs:
- id: cloud
type: FORM
displayName: Cloud configuration
inputs:
- id: provider
type: SELECT
values: [AWS, GCP, Azure]
- id: region
type: SELECT
dependsOn:
inputs:
- cloud.provider
condition: "{{ inputs.cloud.provider == 'AWS' }}"
values:
- us-east-1
- eu-west-1
```
## Label/value pairs for decoupled dropdowns
When your API returns structured data, use a `{label, value}` jq projection so the dropdown shows a human-readable label while `{{ inputs.x }}` resolves to the underlying technical identifier:
```yaml
id: dynamic_account_selector
namespace: company.team
inputs:
- id: aws_account
type: SELECT
displayName: AWS Account
expression: "{{ http(uri = 'https://api.example.com/accounts') | jq('.accounts[] | {label: .name, value: .id}') }}"
tasks:
- id: log_account
type: io.kestra.plugin.core.log.Log
message: "Selected account ID: {{ inputs.aws_account }}"
```
The dropdown displays account names; `{{ inputs.aws_account }}` resolves to the account ID. The same pattern works with static `values` lists — see [Label/value pairs in SELECT and MULTISELECT inputs](../../05.workflow-components/05.inputs/index.md#labelvalue-pairs-in-select-and-multiselect-inputs).
---
# Build ETL Pipelines in Kestra
URL: https://kestra.io/docs/how-to-guides/etl-pipelines
> Build end-to-end ETL pipelines in Kestra. Extract data from any source, transform it, and load to your target data warehouse with full observability.
Build ETL pipelines in Kestra using DuckDB, Python and Task Runners.
This tutorial demonstrates building different ETL pipelines in Kestra.
:::alert{type="info"}
We have used AWS access key and secret key in the example workflows below. To know more about these keys and how to get one, you can refer to the [AWS guide on secret access keys](https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/). Once we have these, we can store them in the [KV Store](../../06.concepts/05.kv-store/index.md) or as [Secrets](../../06.concepts/04.secret/index.md).
:::
## Using DuckDB
DuckDB transforms data directly using SQL queries.
In the example below, we fetch CSV files, perform the join transformation using DuckDB Query task, store the result, upload the detailed orders onto S3, perform another transformation on the stored result, and finally upload the file as CSV onto S3.
```yaml
id: etl_using_duckdb
namespace: company.team
tasks:
- id: download_orders_csv
type: io.kestra.plugin.core.http.Download
description: Download orders.csv file
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: download_products_csv
type: io.kestra.plugin.core.http.Download
description: Download products.csv file
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/products.csv
- id: get_detailed_orders
type: io.kestra.plugin.jdbc.duckdb.Query
description: Perform JOIN transformation using DuckDB
inputFiles:
orders.csv: "{{ outputs.download_orders_csv.uri }}"
products.csv: "{{ outputs.download_products_csv.uri }}"
sql: |
SELECT
o.order_id,
o.customer_name,
o.customer_email,
o.product_id,
o.price,
o.quantity,
o.total,
p.product_name,
p.product_category,
p.brand
FROM read_csv_auto('{{ workingDir }}/orders.csv', header=True) o
JOIN read_csv_auto('{{ workingDir }}/products.csv', header=True) p
ON o.product_id = p.product_id
ORDER BY order_id ASC;
store: true
- id: ion_to_csv
type: io.kestra.plugin.serdes.csv.IonToCsv
description: Convert the result into CSV
from: "{{ outputs.get_detailed_orders.uri }}"
- id: upload_detailed_orders_to_s3
type: io.kestra.plugin.aws.s3.Upload
description: Upload the resulting CSV file onto S3
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "eu-central-1"
from: "{{ outputs.ion_to_csv.uri }}"
bucket: "my_bucket"
key: "orders/detailed_orders"
- id: get_orders_per_product
type: io.kestra.plugin.jdbc.duckdb.Query
description: Perform aggregation using DuckDB
inputFiles:
detailed_orders.csv: "{{ outputs.ion_to_csv.uri }}"
sql: |
SELECT product_id,
COUNT(product_id) as order_count,
SUM(quantity) as product_count,
CAST(SUM(total) AS DECIMAL(10,2)) AS order_total
FROM read_csv_auto('{{ workingDir }}/detailed_orders.csv', header=True)
GROUP BY product_id
ORDER BY product_id ASC
store: true
- id: get_orders_per_product_csv
type: io.kestra.plugin.serdes.csv.IonToCsv
description: Convert the result into CSV
from: "{{ outputs.get_orders_per_product.uri }}"
- id: upload_orders_per_product_to_s3
type: io.kestra.plugin.aws.s3.Upload
description: Upload the resulting CSV file onto S3
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "eu-central-1"
from: "{{ outputs.get_orders_per_product_csv.uri }}"
bucket: "my_bucket"
key: "orders/orders_per_product"
```
Similar Query tasks can be performed on different databases like Snowflake, Postgres, etc.
## Using Python
You can choose to perform ETL using python (pandas) and then run it as a Python script.
The ETL performed using [DuckDB](#using-duckdb) above can be performed using Python as shown in the example flow below.
```yaml
id: python_etl
namespace: company.team
tasks:
- id: etl
type: io.kestra.plugin.scripts.python.Script
description: Python ETL Script
beforeCommands:
- pip install requests pandas
script: |
import io
import requests
import pandas as pd
def _extract(url):
csv_data = requests.get(url).content
return pd.read_csv(io.StringIO(csv_data.decode('utf-8')), header=0)
def run_etl():
orders_data = _extract("https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv")
products_data = _extract("https://huggingface.co/datasets/kestra/datasets/raw/main/csv/products.csv")
# perform join transformation
detailed_orders = orders_data.merge(products_data, how='left', left_on='product_id', right_on='product_id')
detailed_orders.to_csv("detailed_orders.csv")
# perform aggregation
orders_per_product = detailed_orders.groupby('product_id').agg(order_count= ('product_id', 'count'), product_count=('quantity', 'sum'), order_total=('total', 'sum')).sort_values('product_id')
orders_per_product['order_total'] = orders_per_product['order_total'].apply(lambda x: float("{:.2f}".format(x)))
orders_per_product.to_csv("orders_per_product.csv")
if __name__ == "__main__":
run_etl()
outputFiles:
- detailed_orders.csv
- orders_per_product.csv
- id: upload_detailed_orders_to_s3
type: io.kestra.plugin.aws.s3.Upload
description: Upload the resulting CSV file onto S3
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "eu-central-1"
from: "{{ outputs.python_etl.outputFiles('detailed_orders.csv') }}"
bucket: "my_bucket"
key: "orders/detailed_orders"
- id: upload_orders_per_product_to_s3
type: io.kestra.plugin.aws.s3.Upload
description: Upload the resulting CSV file onto S3
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "eu-central-1"
from: "{{ outputs.python_etl.outputFiles('orders_per_product.csv') }}"
bucket: "my_bucket"
key: "orders/orders_per_product"
```
## Using Batch Task Runners
When the python scripts get more compute-intesive or memory-intensive, it is advised to run them on remote batch compute resources using Batch Task Runners.
Kestra provides a variety of [Batch Task Runners](../../07.enterprise/04.scalability/task-runners/index.md#task-runner-types). Here is an example of how the ETL python script can be run on a AWS Batch Task Runner.
```yaml
id: aws_batch_task_runner_etl
namespace: company.team
tasks:
- id: python_etl_on_aws_task_runner
type: io.kestra.plugin.scripts.python.Script
description: Run python ETL script on Docker Task Runner
containerImage: python:3.11-slim
taskRunner:
type: io.kestra.plugin.ee.aws.runner.Batch
region: eu-central-1
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
computeEnvironmentArn: "arn:aws:batch:eu-central-1:01234567890:compute-environment/kestraFargateEnvironment"
jobQueueArn: "arn:aws:batch:eu-central-1:01234567890:job-queue/kestraJobQueue"
executionRoleArn: "arn:aws:iam::01234567890:role/kestraEcsTaskExecutionRole"
taskRoleArn: arn:aws:iam::01234567890:role/ecsTaskRole
bucket: kestra-product-de
beforeCommands:
- pip install requests pandas
script: |
import io
import requests
import pandas as pd
def _extract(url):
csv_data = requests.get(url).content
return pd.read_csv(io.StringIO(csv_data.decode('utf-8')), header=0)
def run_etl():
orders_data = _extract("https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv")
products_data = _extract("https://huggingface.co/datasets/kestra/datasets/raw/main/csv/products.csv")
# perform join transformation
detailed_orders = orders_data.merge(products_data, how='left', left_on='product_id', right_on='product_id')
detailed_orders.to_csv("detailed_orders.csv")
# perform aggregation
orders_per_product = detailed_orders.groupby('product_id').agg(order_count= ('product_id', 'count'), product_count=('quantity', 'sum'), order_total=('total', 'sum')).sort_values('product_id')
orders_per_product['order_total'] = orders_per_product['order_total'].apply(lambda x: float("{:.2f}".format(x)))
orders_per_product.to_csv("orders_per_product.csv")
if __name__ == "__main__":
run_etl()
outputFiles:
- detailed_orders.csv
- orders_per_product.csv
- id: upload_detailed_orders_to_s3
type: io.kestra.plugin.aws.s3.Upload
description: Upload the resulting CSV file onto S3
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "eu-central-1"
from: "{{ outputs.python_etl.outputFiles('detailed_orders.csv') }}"
bucket: "my_bucket"
key: "orders/detailed_orders"
- id: upload_orders_per_product_to_s3
type: io.kestra.plugin.aws.s3.Upload
description: Upload the resulting CSV file onto S3
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "eu-central-1"
from: "{{ outputs.python_etl.outputFiles('orders_per_product.csv') }}"
bucket: "my_bucket"
key: "orders/orders_per_product"
```
## Using dbt
You can create a similar pipeline based on an ELT model using dbt via Kestra, using namespace files for the dbt models.
This example uses dbt + BigQuery to perform the ELT process: it loads data from an HTTP request to Hugging Face into BigQuery tables, performs join and aggregate transformations using dbt, and then queries the resulting tables.
```yaml
id: dbt_transformations
namespace: kestra.engineering.bigquery.dbt
tasks:
- id: orders_http_download
type: io.kestra.plugin.core.http.Download
description: Download orders.csv using HTTP Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: products_http_download
type: io.kestra.plugin.core.http.Download
description: Download products.csv using HTTP Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/products.csv
- id: create_orders_table
type: io.kestra.plugin.gcp.bigquery.CreateTable
description: Create orders table in BigQuery
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
projectId:
dataset: ecommerce
table: orders
tableDefinition:
type: TABLE
schema:
fields:
- name: order_id
type: INT64
- name: customer_name
type: STRING
- name: customer_email
type: STRING
- name: product_id
type: INT64
- name: price
type: FLOAT64
- name: quantity
type: INT64
- name: total
type: FLOAT64
- id: create_products_table
type: io.kestra.plugin.gcp.bigquery.CreateTable
description: Create products table in BigQuery.
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
projectId:
dataset: ecommerce
table: products
tableDefinition:
type: TABLE
schema:
fields:
- name: product_id
type: INT64
- name: product_name
type: STRING
- name: product_category
type: STRING
- name: brand
type: STRING
- id: load_orders_table
type: io.kestra.plugin.gcp.bigquery.Load
description: Load orders table with data from orders.csv
from: "{{ outputs.orders_http_download.uri }}"
projectId:
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
destinationTable: ".ecommerce.orders"
format: CSV
csvOptions:
fieldDelimiter: ","
skipLeadingRows: 1
- id: load_products_table
type: io.kestra.plugin.gcp.bigquery.Load
description: Load products table with data from products.csv
from: "{{ outputs.products_http_download.uri }}"
projectId:
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
destinationTable: ".ecommerce.products"
format: CSV
csvOptions:
fieldDelimiter: ","
skipLeadingRows: 1
- id: dbt
type: io.kestra.plugin.dbt.cli.DbtCLI
description: Use dbt build to perform the dbt transformations
inputFiles:
sa.json: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/dbt-bigquery:latest
namespaceFiles:
enabled : true
profiles: |
bq_dbt_project:
outputs:
dev:
type: bigquery
method: service-account
dataset: ecommerce
project:
keyfile: sa.json
location: US
priority: interactive
threads: 16
timeout_seconds: 300
fixed_retries: 1
target: dev
commands:
- dbt deps
- dbt build
- id: query_detailed_orders
type: io.kestra.plugin.gcp.bigquery.Query
description: Query the newly generated detailed_orders BigQuery table
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
projectId:
sql: |
SELECT * FROM .ecommerce.detailed_orders
store: true
- id: query_orders_per_product
type: io.kestra.plugin.gcp.bigquery.Query
description: Query the newly generated orders_per_product BigQuery table
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
projectId:
sql: |
SELECT * FROM .ecommerce.orders_per_product
store: true
```
Here are the files that you should create in the Kestra editor.
Firstly, create `dbt_project.yml` file, and put the following contents into it.
```yaml
name: 'bq_dbt_project'
version: '1.0.0'
config-version: 2
profile: 'bq_dbt_project'
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
clean-targets:
- "target"
- "dbt_packages"
models:
bq_dbt_project:
example:
+materialized: view
```
Next, create `models` folder. All the upcoming files will be created under the `models` folder.
Create `sources.yml`, which defines the source tables referenced in other models.
```yaml
version: 2
sources:
- name: ecommerce
database:
schema: ecommerce
tables:
- name: orders
- name: products
```
Next, create two files — `stg_orders.sql` and `stg_products.sql` — which materialize as views on top of the source tables:
**stg_orders.sql**
```sql
{{ config(materialized="view") }}
select order_id,
customer_name,
customer_email,
product_id,
price,
quantity,
total
from {{ source('ecommerce', 'orders') }}
```
**stg_products.sql**
```sql
{{ config(materialized="view") }}
select
product_id,
product_name,
product_category,
brand
from {{ source('ecommerce', 'products') }}
```
Next, create `detailed_orders.sql`, which creates the `detailed_orders` table by joining the `stg_orders` and `stg_products` views on `product_id`:
```sql
{{ config(materialized="table") }}
select
o.order_id,
o.customer_name,
o.customer_email,
o.product_id,
p.product_name,
p.product_category,
p.brand,
o.price,
o.quantity,
o.total
from {{ ref('stg_orders') }} o join {{ ref('stg_products') }} p
on o.product_id = p.product_id
```
Next, create `order_per_product.sql`, which creates the `order_per_product` table by aggregating the `detailed_orders` table:
```sql
{{ config(materialized="table") }}
select
product_id,
COUNT(product_id) as order_count,
SUM(quantity) as product_count,
SUM(total) AS order_total
from {{ ref('detailed_orders') }}
group by product_id
order by product_id asc
```
With this, we are ready with all the dbt models. You can now execute the flow. The flow will generate the `detailed_orders` and `orders_per_product` tables. You can view the content of this table by going to the Outputs of the last two tasks.
## Using Spark
We can perform the same ETL process using Spark.
The flow for performing the same transformation using Spark will look as follows:
```yaml
id: spark_python_submit
namespace: kestra.engineering.spark
tasks:
- id: python_submit
type: io.kestra.plugin.spark.PythonSubmit
runner: DOCKER
docker:
networkMode: host
user: root
master: spark://localhost:7077
args:
- "10"
mainScript: |
from pyspark.sql import SparkSession
from pyspark import SparkFiles
orders_url = "https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv"
products_url = "https://huggingface.co/datasets/kestra/datasets/raw/main/csv/products.csv"
spark.sparkContext.addFile(orders_url)
spark.sparkContext.addFile(products_url)
if __name__ == "__main__":
spark = SparkSession.builder.appName("EcommerceApp").getOrCreate()
#Create orders dataframe based on orders.csv
orders_df = spark.read.csv("file://" + SparkFiles.get("orders.csv"), inferSchema=True, header=True)
#Create products dataframe based on orders.csv
products_df = spark.read.csv("file://" + SparkFiles.get("products.csv"), inferSchema=True, header=True)
#Create detailed_orders by joining orders_df and products_df
detailed_orders_df = orders_df.join(products_df, orders_df.product_id == products_df.product_id, "left")
# Print the contents of detailed_orders_df
detailed_orders_df.show(10)
spark.stop()
```
---
# Generate and Pseudonymize Test Data in Kestra
URL: https://kestra.io/docs/how-to-guides/generate-and-pseudonymize-data
> Use the plugin-datagen plugin to generate realistic fake data from scratch or replace PII fields in existing CSV, JSON, or ION files before moving data across environments.
Using real production data in dev or staging environments risks exposing customer PII and violating data protection regulations like GDPR and CCPA.
The `plugin-datagen` plugin solves this with two tasks — choose based on what you already have:
| Situation | Task | What it does |
|---|---|---|
| You have a production export | `Pseudonymize` | Replaces PII fields in the file with realistic fakes; all other fields pass through unchanged |
| You have no production data | `Generate` | Creates a synthetic dataset from scratch using [Datafaker](https://www.datafaker.net/documentation/expressions/) expressions |
Both tasks write output to Kestra internal storage.
## Prerequisites
- A running Kestra instance
- `plugin-datagen` installed — add it from **Administration → Plugins** or via the CLI:
```bash
kestra plugins install io.kestra.plugin:plugin-datagen:LATEST
```
---
## Path A: Replace PII in an existing export
Use `Pseudonymize` when you have a real production file and need to replace sensitive fields before it enters a dev or staging environment. The task reads the file, replaces only the fields you specify using Datafaker expressions, and writes the result back to internal storage.
- `from` — URI of the input file in Kestra internal storage (required)
- `fields` — map of field names to Datafaker expressions; unlisted fields pass through unchanged
- `contentType` — `CSV`, `JSON`, or `ION`; auto-detected from the file extension when not set
- `locale` — optional Faker locale list such as `["en", "US"]` or `["fr", "FR"]`
Outputs: `uri` (the pseudonymized file) and `count` (records processed).
### CSV export
`fields` keys match column headers exactly. The header row is preserved verbatim.
```yaml
id: pseudonymize_customer_export
namespace: company.team
tasks:
- id: pseudonymize
type: io.kestra.plugin.datagen.core.Pseudonymize
from: "{{ inputs.file }}"
contentType: CSV
locale: ["en", "US"]
fields:
first_name: "#{name.firstName}"
last_name: "#{name.lastName}"
email: "#{internet.emailAddress}"
phone: "#{phoneNumber.cellPhone}"
national_id: "#{idNumber.ssnValid}"
- id: log
type: io.kestra.plugin.core.log.Log
message: "Pseudonymized {{ outputs.pseudonymize.count }} records → {{ outputs.pseudonymize.uri }}"
```
### JSON export with nested fields
Use dot-notation to address nested fields (e.g. `user.profile.email`). A path that does not exist in a given record is silently skipped — the record is written as-is.
Replace `export_task` with the ID of your upstream task that produces the JSON file.
```yaml
id: pseudonymize_json_export
namespace: company.team
tasks:
- id: pseudonymize
type: io.kestra.plugin.datagen.core.Pseudonymize
from: "{{ outputs.export_task.uri }}"
contentType: JSON
locale: ["en", "US"]
fields:
"user.profile.fullName": "#{name.fullName}"
"user.profile.email": "#{internet.emailAddress}"
"user.address.city": "#{address.city}"
"user.address.zipCode": "#{address.zipCode}"
- id: log
type: io.kestra.plugin.core.log.Log
message: "Pseudonymized {{ outputs.pseudonymize.count }} records → {{ outputs.pseudonymize.uri }}"
```
### Schedule a nightly refresh
Wrap `Pseudonymize` in a scheduled flow to automatically clean a daily production export before it reaches staging.
Set `vars.daily_export_uri` in the flow's **Variables** tab to the Kestra internal storage URI of your daily export file.
```yaml
id: nightly_pseudonymize_export
namespace: company.team
triggers:
- id: nightly
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 2 * * *"
tasks:
- id: pseudonymize
type: io.kestra.plugin.datagen.core.Pseudonymize
from: "{{ vars.daily_export_uri }}"
contentType: CSV
locale: ["en", "US"]
fields:
customer_name: "#{name.fullName}"
customer_email: "#{internet.emailAddress}"
customer_phone: "#{phoneNumber.cellPhone}"
national_id: "#{idNumber.ssnValid}"
- id: load_to_staging
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.pseudonymize.count }} records ready at {{ outputs.pseudonymize.uri }}"
```
---
## Path B: Generate a synthetic dataset from scratch
Use `Generate` with `JsonObjectGenerator` when you have no production data to start from. Define a template map and use Datafaker expressions (`#{...}`) for any field that should vary per record. Set `store: true` and `batchSize` to write a full batch to internal storage.
Outputs: `uri` (the generated ION file) and `count` (records written).
```yaml
id: generate_staging_data
namespace: company.team
tasks:
- id: generate
type: io.kestra.plugin.datagen.core.Generate
store: true
batchSize: 1000
generator:
type: io.kestra.plugin.datagen.generators.JsonObjectGenerator
locale: ["en", "US"]
value:
customer_id: "#{number.numberBetween '10000','99999'}"
name: "#{name.fullName}"
email: "#{internet.emailAddress}"
phone: "#{phoneNumber.cellPhone}"
tier: "#{options.option 'free','pro','enterprise'}"
address:
city: "#{address.city}"
zip: "#{address.zipCode}"
- id: load_to_staging
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.generate.count }} synthetic records ready at {{ outputs.generate.uri }}"
```
:::alert{type="info"}
`Generate` always writes ION format when `store: true`. Pass `{{ outputs.generate.uri }}` to any downstream task that reads from Kestra internal storage.
:::
---
## Datafaker expressions
Both tasks use [Datafaker](https://www.datafaker.net/documentation/expressions/) expressions in the format `#{category.method}`.
| Expression | Example output |
|---|---|
| `#{name.fullName}` | `Jane Doe` |
| `#{name.firstName}` | `Jane` |
| `#{internet.emailAddress}` | `jane.doe@example.com` |
| `#{phoneNumber.cellPhone}` | `(555) 867-5309` |
| `#{address.city}` | `Springfield` |
| `#{address.zipCode}` | `90210` |
| `#{address.fullAddress}` | `123 Main St, Springfield, IL 90210` |
| `#{idNumber.ssnValid}` | `123-45-6789` |
| `#{commerce.department}` | `Electronics` |
| `#{number.numberBetween '1','100'}` | `42` |
| `#{options.option 'free','pro','enterprise'}` | `pro` |
For the full provider list, see the [Datafaker documentation](https://www.datafaker.net/documentation/expressions/).
If an expression cannot be evaluated, the raw expression string is written to the output field rather than throwing an error.
Both tasks accept a `locale` list to control name, address, and phone formatting — for example, `["fr", "FR"]` for French output or `["de", "DE"]` for German. Omit `locale` to use Faker's default (English).
---
# Validate and Deploy Flows with GitHub Actions
URL: https://kestra.io/docs/how-to-guides/github-actions
> Automate the validation and deployment of your Kestra flows using GitHub Actions for a robust CI/CD pipeline.
How to use GitHub Actions to automatically validate and deploy your flows to Kestra.
If you're version controlling your flows in a Git repository, it can be useful to automatically validate that they're in the correct format before merging into your `main` branch. On top of that, you can automatically deploy your flows in your `main` branch to your Kestra instance.
There are three GitHub Actions available:
- [Validate Flows](https://github.com/kestra-io/github-actions/tree/main/validate-flows) - Validate your flows before deploying anything.
- [Deploy Flows](https://github.com/kestra-io/github-actions/tree/main/deploy-flows) - Deploy your flows to your Kestra server.
- [Deploy Namespace Files](https://github.com/kestra-io/github-actions/tree/main/deploy-namespace-files) - Deploy namespace files to your Kestra server.
## Validate Your Flows
The Validate Flows Action sets up a workflow to check all flows in the specified `directory` when a commit is pushed to `main` or a Pull Request is opened for the `main` branch. For the full list of inputs, see the [GitHub Actions reference](../../version-control-cicd/cicd/01.github-action/index.md#validate-flows-action-inputs).
In the example below:
1. Triggers when a commit is pushed to `main` or when a PR is opened for the `main` branch.
2. Checks out the repository so we can access the files in later steps.
3. Uses the Validate Flows Action to check all the flows in the `./kestra/flows` directory.
```yaml
name: Kestra CI/CD
on:
push:
branches: [ "main" ]
pull_request:
branches: [ "main" ]
jobs:
validate:
runs-on: ubuntu-latest
name: Kestra validate
steps:
- name: Checkout repo content
uses: actions/checkout@v4
- name: Validate all flows
uses: kestra-io/github-actions/validate-flows@main
with:
directory: ./kestra/flows
server: https://server-url.com
```
## Deploy Your Flows
The Deploy Flows Action sets up a workflow to deploy when new commits are pushed to the `main` branch. Specify a `directory` containing your flows and optionally a `namespace` to deploy them to. For the full list of inputs, see the [GitHub Actions reference](../../version-control-cicd/cicd/01.github-action/index.md#deploy-flows-action-inputs).
If you want to deploy flows to multiple namespaces, you can add multiple steps using the Deploy Flows Action, each with a different `namespace` and `directory`.
In the example below:
1. Triggers when commits are pushed to `main`.
2. Checks out the repository so we can access the files in later steps.
3. Deploys flows from `kestra/flows` to the `company.team` namespace in the Kestra instance.
```yaml
name: Kestra CI/CD
on:
push:
branches: [ "main" ]
jobs:
deploy:
runs-on: ubuntu-latest
name: Kestra deploy
steps:
- name: Checkout repo content
uses: actions/checkout@v4
- name: Deploy flows
uses: kestra-io/github-actions/deploy-flows@main
with:
namespace: company.team
directory: ./kestra/flows
server: https://server-url.com
```
## Deploy Namespace Files
Using the Deploy Namespace Files Action, you can deploy configuration files or other resources to a namespace. This is useful for managing shared files across your flows.
In the example below:
1. Triggers when commits are pushed to `main`.
2. Checks out the repository so we can access the files in later steps.
3. Deploys a configuration file to the `company.team` namespace.
```yaml
name: Kestra CI/CD
on:
push:
branches: [ "main" ]
jobs:
deploy-nsfiles:
runs-on: ubuntu-latest
name: Kestra deploy namespace files
steps:
- name: Checkout repo content
uses: actions/checkout@v4
- name: Deploy namespace files
uses: kestra-io/github-actions/deploy-namespace-files@main
with:
namespace: company.team
localPath: ./config/app.yaml
namespacePath: config/app.yaml
server: https://server-url.com
```
## Authentication
If you have [authentication](../../configuration/05.security-and-secrets/index.md) enabled in your Kestra instance, you will need to add additional properties so your action can authenticate with your instance.
### Basic Authentication
If you have basic authentication enabled with a username and password (e.g. on the Open Source Edition), you can add the `user` and `password` properties to your action using [GitHub Secrets](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions):
```yaml
name: Kestra CI/CD
on:
push:
branches: [ "main" ]
jobs:
deploy:
runs-on: ubuntu-latest
name: Kestra deploy
steps:
- name: Checkout repo content
uses: actions/checkout@v4
- name: Deploy flows
uses: kestra-io/github-actions/deploy-flows@main
with:
namespace: company.team
directory: ./kestra/flows
server: https://server-url.com
user: ${{ secrets.KESTRA_USERNAME }}
password: ${{ secrets.KESTRA_PASSWORD }}
```
As you can see, the `user` and `password` are added as secrets with the expression syntax `${{ secrets.name }}` to prevent you from committing these to your repository.
### API Token Authentication
If you're using the [Enterprise Edition](../../oss-vs-paid/index.md), you can use an [API Token](../../07.enterprise/03.auth/api-tokens/index.md) instead:
```yaml
name: Kestra CI/CD
on:
push:
branches: [ "main" ]
jobs:
deploy:
runs-on: ubuntu-latest
name: Kestra deploy
steps:
- name: Checkout repo content
uses: actions/checkout@v4
- name: Deploy flows
uses: kestra-io/github-actions/deploy-flows@main
with:
namespace: company.team
directory: ./kestra/flows
server: https://server-url.com
apiToken: ${{ secrets.KESTRA_API_TOKEN }}
```
## Set Up a Branch Ruleset
If you're working in a team, it can be useful to set up a [Ruleset](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/about-rulesets) on your `main` branch to prevent broken flows from being deployed accidentally to your production instance.
To enable this, go to the **Settings** of your repository on GitHub and go to **Rules** then **Rulesets**. In here, we can create a new branch ruleset.
The goal of this ruleset is to protect the `main` branch as our GitHub Action will automatically deploy any flows in this branch to our Kestra instance. To achieve this, we can set the specific Branch rules:
- Require a pull request before merging - No commits can be made directly to the `main` branch
- Require status checks to pass - Requires our Validate Flows Action to pass before we can merge our Pull Requests

With these enabled, we are required to make a Pull Request before our flows end up in production. This enables us to run our validate check and require that to pass before we can merge any pull requests.

In the example above, the flow had an incorrect indentation so it failed the validate check. As a result of this, the Pull Request is unable to be merged until it is fixed.
---
# Back Up GitHub Repos with Kestra Playground
URL: https://kestra.io/docs/how-to-guides/github-repo-backup
> Automate GitHub repository backups with Kestra. Schedule periodic exports of your repos to cloud storage with built-in error handling and audit logging.
Clone every repository in the `kestra-io` GitHub organization, zip each repo, and upload the archives to Google Cloud Storage (GCS) for safekeeping.
---
## Why run this backup?
Organizations often mirror source control data outside GitHub to satisfy compliance, enable disaster recovery drills, or seed analytics and search workloads. This flow collects every repository, produces portable zip artifacts, and stores them in GCS so you have an off-platform copy you can restore or inspect independently of GitHub.
This flow has potentially long running operations, so to optimize testing certain tasks, we use the [Playground](../../09.ui/10.playground/index.md) feature to ensure each component works before a production execution.
---
## Prerequisites
- GitHub personal access token stored as `GITHUB_TOKEN`.
- GCP service account JSON key stored as `GCP_SERVICE_ACCOUNT`.
- A target bucket such as `gs://your_gcs_bucket/kestra-backups/`.
- The [Google Cloud Storage plugin](/plugins/plugin-gcp/google-cloud-storage-gcs/io.kestra.plugin.gcp.gcs.upload) available to your workers.
---
## Choosing a fetch mode
The `repositories.Search` task exposes a `fetchType` property that controls how search results reach downstream tasks:
| `fetchType` | Output field | Best for |
|---|---|---|
| `FETCH` | `rows` — a list of result objects directly in the task output | Moderate result sets where you want to use Pebble expressions immediately |
| `FETCH_ONE` | `row` — only the first result | Lookups where you expect a single match |
| `STORE` (default) | `uri` — an Ion file written to Kestra internal storage | Large result sets, auditing, or when you need to persist the raw data |
| `NONE` | _(empty)_ | Triggering a side-effect without needing results |
Use `FETCH` when you want to feed results directly into a Loop task with a simple Pebble expression. Use `STORE` when the result set may be large, when you want the raw file persisted in internal storage for inspection or reuse, or when downstream tasks need to read the data multiple times.
---
## Flow Definition — FETCH mode
`fetchType: FETCH` places results directly in `outputs.search_kestra_repos.rows` as a list of objects. The Loop `values` expression reads from that list without any file I/O step.
```yaml
id: github_repo_backup
namespace: company.team
description: Clones Kestra GitHub repositories and backs them up to Google Cloud Storage.
tasks:
- id: search_kestra_repos
type: io.kestra.plugin.github.repositories.Search
description: Search for all repositories under the 'kestra-io' GitHub organization.
query: "user:kestra-io"
fetchType: FETCH
oauthToken: "{{ secret('GITHUB_TOKEN') }}"
- id: for_each_repo
type: io.kestra.plugin.core.flow.Loop
description: Iterate over each found repository.
values: "{{ outputs.search_kestra_repos.rows | jq('.[].clone_url') }}"
tasks:
- id: working_dir
type: io.kestra.plugin.core.flow.WorkingDirectory
description: Create a temporary working directory for cloning and zipping each repository.
tasks:
- id: clone_repo
type: io.kestra.plugin.git.Clone
description: Clone the current repository from GitHub.
url: "{{ item.value }}"
directory: "{{ item.value | split('/') | last | split('.') | first }}"
- id: zip_repo
type: io.kestra.plugin.scripts.shell.Commands
description: Zip the cloned repository's contents.
beforeCommands:
- apk add zip > /dev/null 2>&1 || true
commands:
- |
REPO_DIR="{{ outputs.clone_repo.directory }}"
REPO_NAME="{{ REPO_DIR | split('/') | last }}"
cd "${REPO_DIR}"
zip -r "../${REPO_NAME}.zip" .
outputFiles:
- "{{ outputs.clone_repo.directory | split('/') | last }}.zip"
- id: upload_to_gcs
type: io.kestra.plugin.gcp.gcs.Upload
description: Upload the zipped repository to Google Cloud Storage.
from: "{{ outputs.zip_repo.outputFiles['' ~ (outputs.clone_repo.directory | split('/') | last) ~ '.zip'] }}"
to: "gs://your_gcs_bucket/kestra-backups/{{ outputs.clone_repo.directory | split('/') | last }}.zip"
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT') }}"
```
---
## Flow Definition — STORE mode
`fetchType: STORE` (the default) writes results to an Ion file in Kestra internal storage and returns a `uri`. Use this variant when the result set is large, when you want to retain the raw file for auditing or reuse across multiple tasks, or when you are integrating with tasks that consume a storage URI directly.
```yaml
id: github_repo_backup
namespace: company.team
description: Clones Kestra GitHub repositories and backs them up to Google Cloud Storage.
tasks:
- id: search_kestra_repos
type: io.kestra.plugin.github.repositories.Search
description: Search for all repositories under the 'kestra-io' GitHub organization.
query: "user:kestra-io"
fetchType: STORE
oauthToken: "{{ secret('GITHUB_TOKEN') }}"
- id: for_each_repo
type: io.kestra.plugin.core.flow.Loop
description: Iterate over each found repository.
values: "{{ outputs.search_kestra_repos.uri | internalStorage.get() | jq('.[].clone_url') }}"
tasks:
- id: working_dir
type: io.kestra.plugin.core.flow.WorkingDirectory
description: Create a temporary working directory for cloning and zipping each repository.
tasks:
- id: clone_repo
type: io.kestra.plugin.git.Clone
description: Clone the current repository from GitHub.
url: "{{ item.value }}"
directory: "{{ item.value | split('/') | last | split('.') | first }}"
- id: zip_repo
type: io.kestra.plugin.scripts.shell.Commands
description: Zip the cloned repository's contents.
beforeCommands:
- apk add zip > /dev/null 2>&1 || true
commands:
- |
REPO_DIR="{{ outputs.clone_repo.directory }}"
REPO_NAME="{{ REPO_DIR | split('/') | last }}"
cd "${REPO_DIR}"
zip -r "../${REPO_NAME}.zip" .
outputFiles:
- "{{ outputs.clone_repo.directory | split('/') | last }}.zip"
- id: upload_to_gcs
type: io.kestra.plugin.gcp.gcs.Upload
description: Upload the zipped repository to Google Cloud Storage.
from: "{{ outputs.zip_repo.outputFiles['' ~ (outputs.clone_repo.directory | split('/') | last) ~ '.zip'] }}"
to: "gs://your_gcs_bucket/kestra-backups/{{ outputs.clone_repo.directory | split('/') | last }}.zip"
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT') }}"
```
---
## How It Works
Both variants share the same structure. The only difference is how the search results move from the `search_kestra_repos` task to the `for_each_repo` loop.
With `FETCH`, results live in `outputs.search_kestra_repos.rows` as a native list — no file read needed. With `STORE`, results are written to an Ion file and the Loop `values` expression reads the file via `internalStorage.get()` before applying the `jq` filter.
In both cases:
1. `search_kestra_repos` fetches all repositories in the `kestra-io` organization.
2. `for_each_repo` loops over each `clone_url` extracted from the results.
3. `working_dir` isolates each iteration, keeping cloned data and archives scoped to a temporary folder.
4. `clone_repo` clones the current repository URL.
5. `zip_repo` compresses the cloned repository and exposes the zip file through `outputFiles` so the next task can read it.
6. `upload_to_gcs` uploads each archive to the chosen bucket path using the GCP service account key.
Secrets supply tokens to GitHub and GCP at runtime without embedding credentials in the flow definition.
For a smaller dry run, narrow the search query (for example, add `topic:cli`) or slice the list — `jq('.[0:2].clone_url')` with FETCH or `jq('.[0:2].clone_url')` with STORE — to process only a few repositories.
---
## Use Playground to Test Safely
Playground mode helps you validate expensive steps incrementally. Start with the search task to confirm authentication and inspect the results before any cloning. When refining the zip or upload steps, slice the list to a single repository so you can replay those tasks without hitting GitHub or GCS repeatedly. Because Playground keeps prior task outputs, you can iterate on shell commands and storage paths while reusing the same search result and clone, keeping feedback fast and low-risk.
Playground mode lets you validate one task at a time without running the whole backup loop. Follow [the Playground guide](../../09.ui/10.playground/index.md) and use this flow as follows:
1. Toggle Playground mode in the editor.
2. Run only `search_kestra_repos` to confirm your GitHub token works and inspect the search output. In the example below, the task fails in Playground due to a misconfigured secret. This lets you catch the issue before any attempted executions are made. Update the secret, then run it in Playground again to verify that it's correct.
3. Temporarily limit the `values` expression to a single repository while you iterate — with FETCH, use `jq('.[0:1].clone_url')`; with STORE, use `jq('.[0:1].clone_url')` after `internalStorage.get()`.
4. Play `zip_repo` and `upload_to_gcs` individually inside Playground; Kestra reuses outputs from previous played tasks, so you avoid recloning every repository.
5. When satisfied, revert any temporary limits and use **Run all tasks** for a full backup execution.
This approach prevents unnecessary GitHub calls and GCS writes while you refine the flow logic.
---
You now have a reusable flow that continuously backs up the `kestra-io` GitHub organization to GCS with secrets-managed authentication and a safe Playground workflow for testing.
---
# Configure a Google Service Account in Kestra
URL: https://kestra.io/docs/how-to-guides/google-credentials
> Securely configure Google Service Accounts in Kestra to authenticate and access Google Cloud resources and Workspace apps.
Set Up a Google Service Account in Kestra.
When you're using Google Cloud (and for some Google Workspace apps), you're going to need to authenticate in Kestra. The best way to do this is by using a Service Account. However, there's a few ways you can set this up. This guide will walk you through the best way to get your service account working correctly in Kestra.
## Create Service Account in Google Cloud
Inside of Google Cloud, head to `IAM` and then `Service Accounts`. In here you can add the specific roles to the service account before creating it (this will depend on your use case).
Once you've done that, you can go to `Keys` and click on `Add Key`. From the dropdown, select `Create New Key`. Select the Key type as `JSON` and click on `Create`. Download this as we'll need this in a second.
For more information on Google Cloud Service Accounts, see the [documentation](https://cloud.google.com/iam/docs/service-account-overview).
## Configuring a task with a Service Account
Inside of Kestra, you can paste the service account JSON directly to the task property. This is useful for testing purposes:
```yaml
- id: upload
type: io.kestra.plugin.googleworkspace.drive.Upload
from: "{{ inputs.file }}"
parents:
- "1HuxzpLt1b0111MuKMgy8wAv-m9Myc1E_"
name: "My awesome CSV"
contentType: "text/csv"
mimeType: "application/vnd.google-apps.spreadsheet"
serviceAccount: |
{
"type": "service_account",
"project_id": "...",
"private_key_id": "...",
"private_key": "...",
"client_email": "...",
"client_id": "...",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "...",
"universe_domain": "googleapis.com"
}
```
:::alert{type="warning"}
This is not recommended as you might expose your key. We'd recommend using [secrets](#add-service-account-as-a-secret) to store your Service Account JSON.
:::
## Add Service Account as a Secret
Add the Service Account with the `serviceAccount` property to any Google Cloud or Workspaces task. To do this, add it as a secret to Kestra. There are several ways to add secrets; this guide uses environment variables linked to the Docker Compose file. For more information on how secrets work, see the [secrets page](../../06.concepts/04.secret/index.md).
Once you have the service account file downloaded, you can rename it to `service-account.json`. Then we'll encode the service account JSON and store it inside a file named `.env_encoded` which will hold all of our encoded secrets:
```bash
echo SECRET_GCP_SERVICE_ACCOUNT=$(cat service-account.json | base64 -w 0) >> .env_encoded
```
If you already have an existing `.env` file, you can use the following bash script:
```bash
#!/bin/bash
ENV_FILENAME=.env_encoded
while IFS='=' read -r key value; do
echo "SECRET_$key=$(echo -n "$value" | base64)";
done < .env > $ENV_FILENAME
## Encodes the service account file without line wrapping to make sure the whole JSON value is intact.
echo "SECRET_GCP_SERVICE_ACCOUNT=$(cat service-account.json | base64 -w 0)" >> $ENV_FILENAME
```
You can then set the `.env_encoded` file in your `docker-compose.yml`:
```yaml
kestra:
env_file: .env_encoded
```
## Access Service Account in Kestra
You can now access this in Kestra with the following pebble expression:
```yaml
"{{ secret('GCP_SERVICE_ACCOUNT') }}"
```
With this, we can add this to the `serviceAccount` property like so:
```yaml
- id: upload
type: io.kestra.plugin.googleworkspace.drive.Upload
from: "{{ inputs.file }}"
parents:
- "1HuxzpLt1b0111MuKMgy8wAv-m9Myc1E_"
name: "My awesome CSV"
contentType: "text/csv"
mimeType: "application/vnd.google-apps.spreadsheet"
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT') }}"
```
```yaml
- id: fetch
type: io.kestra.plugin.gcp.bigquery.Query
fetch: true
sql: |
SELECT 1 as id, "John" as name
UNION ALL
SELECT 2 as id, "Doe" as name
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT') }}"
```
## Centralize the service account with a Policy (Enterprise Edition)
If you use multiple tasks that require the same service account, create a [Policy](../../07.enterprise/02.governance/policies/index.md) at the namespace level to inject it automatically:
```yaml
id: gcp-service-account
description: "Inject GCP service account into all Google Workspace Drive tasks."
enforcement: ACTIVE
rules:
- type: io.kestra.plugin.ee.rules.Add
on: PLUGIN
where:
- field: type
operator: STARTS_WITH
value: io.kestra.plugin.googleworkspace.drive
values:
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT') }}"
```
With this Policy applied, tasks in the namespace need no `serviceAccount` property:
```yaml
tasks:
- id: upload
type: io.kestra.plugin.googleworkspace.drive.Upload
from: "{{ inputs.file }}"
parents:
- "1HuxzpLt1b0111MuKMgy8wAv-m9Myc1E_"
name: "My awesome CSV"
contentType: "text/csv"
mimeType: "application/vnd.google-apps.spreadsheet"
```
## Configuring Secrets in the Enterprise Edition
In Kestra Enterprise Edition, secrets are managed directly from the UI — no base64 encoding required. Navigate to **Namespaces**, open the namespace where your flow runs, and go to the **Secrets** tab. Click **New secret**, set the key to `GCP_SERVICE_ACCOUNT`, and paste the service account JSON directly as the value.
Reference it in your tasks with `{{ secret('GCP_SERVICE_ACCOUNT') }}`, exactly as shown in the examples above.
To learn more about secret backends and enterprise secret managers, see the [secrets page](../../06.concepts/04.secret/index.md#enterprise-edition).
## `GOOGLE_APPLICATION_CREDENTIALS`
By setting the `GOOGLE_APPLICATION_CREDENTIALS` environment variable on the nodes running Kestra. It must point to an application credentials file. Warning: it must be the same on all worker nodes and can cause some security concerns.
While you can use the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, this is not advised as you'll need to mount the JSON file to Docker which isn't always possible depending on how you've setup Kestra.
To set it up, ensure Kestra has access to the JSON file containing the service account details. If you're using Docker, you'll need to create a bind mount like the example below:
```yaml
kestra:
image: kestra/kestra:latest
pull_policy: always
user: "root"
command: server standalone
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/kestra-wd:/tmp/kestra-wd
- ~/.gcp/workflow-orchestration-credentials.json:/.gcp/credentials.json
...
```
The example uses a file at `~/.gcp/workflow-orchestration-credentials.json`. Update this path to the location of your JSON file. It maps it to `/.gcp/credentials.json` inside the container, which we'll need to reference in the environment variable.
After that, add an environment variable under `environment` called `GOOGLE_APPLICATION_CREDENTIALS`
```yaml
environment:
GOOGLE_APPLICATION_CREDENTIALS: '/.gcp/credentials.json'
KESTRA_CONFIGURATION: |
...
```
:::collapse{title="Full Docker Compose with GOOGLE_APPLICATION_CREDENTIALS"}
Here is a full Docker Compose that you can use to add a service account using the environment variable `GOOGLE_APPLICATION_CREDENTIALS`:
```yaml
volumes:
postgres-data:
driver: local
kestra-data:
driver: local
services:
postgres:
image: postgres:18
volumes:
- postgres-data:/var/lib/postgresql
environment:
POSTGRES_DB: kestra
POSTGRES_USER: kestra
POSTGRES_PASSWORD: k3str4
healthcheck:
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
interval: 30s
retries: 10
kestra:
image: kestra/kestra:latest
pull_policy: always
user: "root"
command: server standalone
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/kestra-wd:/tmp/kestra-wd
- ~/.gcp/workflow-orchestration-credentials.json:/.gcp/credentials.json
environment:
GOOGLE_APPLICATION_CREDENTIALS: '/.gcp/credentials.json'
KESTRA_CONFIGURATION: |
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driver-class-name: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
server:
basic-auth:
enabled: false
username: "admin@kestra.io" # it must be a valid email address
password: kestra
repository:
type: postgres
storage:
type: local
local:
base-path: "/app/storage"
tutorial-flows:
enabled: false
queue:
type: postgres
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
url: http://localhost:8080/
ports:
- "8080:8080"
- "8081:8081"
depends_on:
postgres:
condition: service_started
```
:::
## Google App Passwords
For some Google applications, such as Gmail, you won't use a service account for authentication. Instead, you'll use a normal username and password associated with a Google account. However, this doesn't work if your account has two-factor authentication enabled. In that case, you'll need to generate an **App Password**. You can do this by going to **Manage your Google Account**, then **Security**. Select the **App Passwords** option, and you'll be able to generate a new one. This can be used wherever you would normally enter your password to connect it to Kestra.
:::alert{type="info"}
If your account is associated with Google Workspaces, you might need your Administrator to enable App Passwords in the Admin Console.
:::
---
# Connect Google Sheets to Kestra
URL: https://kestra.io/docs/how-to-guides/google-sheets
> Integrate Google Sheets with Kestra workflows. Read spreadsheet data, write pipeline outputs, and trigger flows automatically from sheet updates.
Learn step-by-step how to read data from a Google Sheet in a Kestra flow.
You can use any Google Sheet for this tutorial. In case you do not have Google Sheet, you can:
1. Download the [orders CSV dataset](https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv) and save it locally as `orders.csv` file.
2. Create a new Google Sheet.
3. Navigate to the `File` menu on the top, and select `Import` option.
4. Navigate to the `Upload` tab, and click on the `Browse` button.
5. Select the recently created `orders.csv` file, and click on `Open` button at the bottom of the popup.
6. On the `Import file` popup, choose the import location as `Replace spreadsheet` and separator type as `Detect automatically`. In this case, it does not matter whether you check or uncheck the box `Convert text to numbers, dates, and formulas`. Click on `Import data` button.
7. The contents of the file will be populated in the spreadsheet.
8. You can put an appropriate title to the spreadsheet, and name the sheet containing the order records as `orders`.




Now that we have the spreadsheet ready, let us proceed to assign appropriate authorization for the spreadsheet. For this:
1. Go to the GCP console, navigate to the [IAM service](https://console.cloud.google.com/iam-admin/iam).
2. Select `Service accounts` from the left navigation menu.
3. Click on `Create Service Account` on the top. You may choose to use an existing service account in which case you can skip the next step.
4. Put in appropriate service account name, service account id (the auto-populated value should be good to start with) and service account description, and click on `Done`.


The new service account has been created. Add a key to the service account.
1. Click on the corresponding service account from the Service Accounts page.
2. Navigate to `Keys` tab, and click on `Add Key` -> `Create new key`.
3. On the `Create private key` popup, select `JSON` option, and click on `Create`.
4. This will download the service account JSON file on your local computer.
5. Provide this JSON file's content as the secret.
a. With Kestra EE, provide the secret key `GCP_SERVICE_ACCOUNT_JSON` and the file contents as the value.
b. For docker-based Kestra instance, convert the JSON file's contents into base64 encoded format using `cat .json | base64` and then provide the secret value as part of the environment file to the docker instance: `SECRET_GCP_SERVICE_ACCOUNT_JSON=`.
Detailed instructions on creating service account can also be found in the [Google credentials guide](../google-credentials/index.md).

We will now provide access to the spreadsheet for the service account.
1. Copy the email corresponding to the service account from the Service Accounts page.
2. Go to the spreadsheet, and click on the `Share` button on the top right.
3. Add the service account email in the `Add people, groups, and calendar events` text box.
4. You can give the `Viewer` access to the service account.
5. Click on `Done`.
Let us now enable the Google Sheets API in the GCP console.
1. On the GCP console, search for `Google Sheets API` service, or directly navigate to the [Google Sheets API page](https://console.cloud.google.com/marketplace/product/google/sheets.googleapis.com).
2. Check whether the Google Sheets API is already enabled. If not, you will see an `Enable` button on the page. Click on the `Enable` button.

With this, we are all set to access the Google Spreadsheet from Kestra flow. Here is an example of how the Kestra flow might look like:
```yaml
id: gsheet
namespace: company.team
tasks:
- id: read_gsheet
type: io.kestra.plugin.googleworkspace.sheets.Read
description: Read data from Google Spreadsheet
spreadsheetId: "1U4AoiUrqiVaSIVcm_TwDc9RoKOdCULNGWxuC1vmDT_A"
store: true
valueRender: FORMATTED_VALUE
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
header: true
```
The `spreadsheetId` in the flow is the ID that is present in the spreadsheet URL. For the URL `https://docs.google.com/spreadsheets/d/1U4AoiUrqiVaSIVcm_TwDc9RoKOdCULNGWxuC1vmDT_A/edit`, the `spreadsheetId` will be `1U4AoiUrqiVaSIVcm_TwDc9RoKOdCULNGWxuC1vmDT_A`.
The `store : true` implies that the values read from the spreadsheet will be stored as a file in the Kestra's internal storage. In case, you only want to fetch the result, and not store them as a file in the Kestra's internal storage, you can use `fetch: true` instead.
The `serviceAccount` value is fetched from the secret store, and its value is the service account key's JSON file contents.
The `header: true` implies that the first line of the input contains the column headers.
In case you only want a few selected sheets to be read, you can provide the array of sheets as part of the attribute `selectedSheetsTitle` as follows:
```yaml
selectedSheetsTitle:
- orders
- products
```
Here is the output of executing the above Kestra flow:

This is how Kestra's Google Workspace plugin can be used to read the spreadsheet with its Sheet's [`Read`](/plugins/plugin-googleworkspace/google-sheets/io.kestra.plugin.googleworkspace.sheets.read) task.
---
# Make HTTP Requests Inside Your Flows
URL: https://kestra.io/docs/how-to-guides/http-request
> Make HTTP requests inside Kestra workflows. Call REST APIs, fetch remote data, and chain API responses as inputs to downstream tasks.
Make HTTP requests to fetch data and generate outputs.
You can make HTTP requests directly inside a flow and use the responses as outputs in downstream tasks. This guide covers what HTTP requests are and how to use the most common request methods in Kestra.
## What is an HTTP request?
Hypertext Transfer Protocol (better known as HTTP) [requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages#http_requests) are messages sent between a client and server to request something.
Requests can send or request data, with common methods known as GET, POST, PUT and DELETE requests. You can use these directly in Kestra to interact with third-party systems.
| Request Method | Description |
| - | - |
| GET | Used to retrieve data from a server |
| POST | Used to create new data on a server |
| PUT | Used to replace data on a server |
| DELETE | Used to delete data on a server |
There are many other request methods too, which you can read more about on the [MDN docs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods).
When you make a request, you will receive a [response](https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages#http_responses) from the server with the answer. This answer can drive Kestra automations. First, here is what makes up a request.
### Status code
When you make a request, you'll receive a response with a status code that indicates whether the request succeeded. The format follows:
| Status Codes | Description |
| - | - |
| 100 - 199 | Informational |
| 200 - 299 | Successful |
| 300 - 399 | Redirection |
| 400 - 499 | Client error |
| 500 - 599 | Server error |
A few common ones you might have seen include:
- 200: OK - Request was successful.
- 404: Not Found - Request reached the server but the resource wasn't found. A common one you see when you go to a page on a website that doesn't exist.
- 500: Internal Server Error - Request reached the server but the server was unable to process it. Usually means the server has thrown an error.
You can read the full list of status codes on the [MDN docs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status).
### Headers
Each request also has a set of request headers that provide additional information, such as the client type and the content type being sent. You can read more about HTTP headers on the [MDN docs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/). Responses include headers in the same structure.
### Body
Requests can also have a request body that contains data to send. For example, if you want to add a user to a system, you would include their name and email in the body. Request bodies are fundamental for POST and PUT requests, but methods like GET don't have them. You can read more about the request body on the [MDN docs](https://developer.mozilla.org/en-US/docs/Web/API/Request/body).
When you receive a [response](https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages#body_2), it may also have a body — for example, a GET request might return JSON data that you can pass to downstream tasks in your workflow.
## How can I make HTTP requests?
You can make requests by putting a URL directly into your browser, especially for GET requests, but it can be challenging to specify the body and headers for other methods, such as POST and PUT requests. There's a variety of tools that can make this easier such as [Postman](https://www.postman.com/) and [cURL](https://curl.se/).
The example below uses Postman to make a POST request to [dummyjson.com](https://dummyjson.com). Use the `/products/add` route to add a new product by providing a body like this:
```json
{
"title": "Kestra Pen"
}
```
In Postman, add the URL `https://dummyjson.com/products/add`, set the request type to `POST`, add the body above as a `raw` option, and change the type to JSON. Then press send:

You can do the same with cURL:
```bash
curl -X POST https://dummyjson.com/products/add \
-H 'Content-Type: application/json' \
-d '{ "title": "Kestra Pen" }'
```
The arguments used are:
- `-X {method} {url}` — specifies the HTTP method and URL
- `-H {header-type}` — sets request headers
- `-d {body}` — provides the request body
The response is the same as in Postman:
```json
{
"id": 101,
"title": "Kestra Pen"
}
```
These tools work well for one-off API testing but require extra effort to automate or integrate with other systems.
## Making HTTP requests in Kestra
Kestra's HTTP task lets you automate requests alongside other tasks. Below, you'll find examples for `GET`, `POST`, `PUT`, and `DELETE` requests in your flow.
To make a request, use the task type `io.kestra.plugin.core.http.Request`. For more information on the task type, see the [dedicated documentation](/plugins/core/http/io.kestra.plugin.core.http.request).
### GET request
Use a GET request to fetch data from a server and pass it to downstream tasks.
This flow sends a GET request to collect a list of products and print the output to the logs:
```yaml
id: http_get_request_example
namespace: company.team
description: Make a HTTP Request and Handle the Output
tasks:
- id: send_data
type: io.kestra.plugin.core.http.Request
uri: https://dummyjson.com/products
method: GET
contentType: application/json
- id: print_status
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.send_data.body }}"
```
The Logs page shows the response:

To view task outputs without Log tasks, use the Outputs page in the UI:

The [Debug Expression](../../05.workflow-components/06.outputs/index.md#using-debug-expression) option lets you inspect specific outputs using an expression after the flow executes — useful when debugging tasks to see what was generated.
### POST request
Build on the GET request example above by changing these properties:
- `uri` → `https://dummyjson.com/products/add`
- `method` → `POST`
- `body` → add the data to send to the server
```yaml
id: http_post_request_example
namespace: company.team
description: Make a HTTP Request and Handle the Output
inputs:
- id: payload
type: JSON
defaults: |
{ "title": "Kestra Pen" }
tasks:
- id: send_data
type: io.kestra.plugin.core.http.Request
uri: https://dummyjson.com/products/add
method: POST
contentType: application/json
body: "{{ inputs.payload }}"
- id: print_status
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.send_data.body }}"
```
Defining the request body as an input makes it easy to change at execution time or reuse across multiple requests.
:::alert{type="info"}
If your body message input is multiple lines, the best practice is to use a pebble expression to convert it to JSON and avoid escape function issues. For more details, check out this [multiline JSON example with pebble](../../expressions/02.syntax/index.mdx#multiline-json-bodies).
:::
Executing this as a POST request returns the following response, visible in the Outputs page via the Debug Expression option:

This produces the same output as the earlier example, with the added ability to pass the response to downstream tasks.
### PUT request
Similar to the `POST` request, change the `method` property to `PUT`. Since the `PUT` request replaces content, adjust the body with the data to update. From the `GET` request, `id` 1 is an `iPhone 9` — change it to an `iPhone 10`:
```yaml
id: http_put_request_example
namespace: company.team
description: Make a HTTP Request and Handle the Output
inputs:
- id: payload
type: JSON
defaults: |
{"title": "iPhone 10"}
tasks:
- id: send_data
type: io.kestra.plugin.core.http.Request
uri: https://dummyjson.com/products/1
method: PUT
contentType: application/json
body: "{{ inputs.payload }}"
- id: print_status
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.send_data.body }}"
```
The response body confirms the updated title field.

### DELETE request
Use a DELETE request to remove a resource. Unlike POST and PUT, you don't need to provide a body — just specify the resource ID as an input.
```yaml
id: http_delete_request_example
namespace: company.team
description: Make a HTTP Request and Handle the Output
inputs:
- id: product_id
type: INT
tasks:
- id: send_data
type: io.kestra.plugin.core.http.Request
uri: "https://dummyjson.com/products/{{ inputs.product_id }}"
method: DELETE
contentType: application/json
- id: print_status
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.send_data.body }}"
```
Adding an input lets you specify which product to remove by providing the `id` at execution.

The response confirms the deletion:

## SSL and self-signed certificates
If the endpoint you are calling uses a self-signed or internally issued certificate, the task fails with an SSL verification error.
### Development and testing
For non-production environments, you can disable certificate verification with `options.ssl.insecureTrustAllCertificates`:
```yaml
id: http_insecure_ssl_example
namespace: company.team
tasks:
- id: send_data
type: io.kestra.plugin.core.http.Request
uri: https://internal-service.example.com/api
method: GET
options:
ssl:
insecureTrustAllCertificates: true
```
:::alert{type="warning"}
`insecureTrustAllCertificates: true` disables all certificate verification. Never use this in production — it exposes connections to man-in-the-middle attacks.
:::
### Production
For production, import the CA certificate into the JVM trust store on the Kestra worker. Once imported, the worker trusts any endpoint signed by that CA and tasks connect without disabling verification.
- **Kubernetes**: see [Trusting a custom CA for outbound connections on Kubernetes](../../10.administrator-guide/custom-ca-kubernetes/index.md) for keytool steps and Helm configuration.
- **Standalone or Docker**: set `JAVA_OPTS` in your Kestra environment:
```bash
JAVA_OPTS="-Djavax.net.ssl.trustStore=/path/to/truststore.p12 \
-Djavax.net.ssl.trustStorePassword=changeit \
-Djavax.net.ssl.trustStoreType=PKCS12"
```
---
# Automate IAM Management with Tasks
URL: https://kestra.io/docs/how-to-guides/iam-automation
> Use Kestra's IAM tasks to automate user onboarding, offboarding, group management, and service account provisioning directly from flows.
Automate Identity and Access Management (IAM) operations from inside a Kestra flow using the `io.kestra.plugin.kestra.ee.iam` task family.
These tasks let you manage users, groups, roles, bindings, service accounts, invitations, and tenant access programmatically — enabling event-driven onboarding from HR webhooks, scheduled access reviews, and automated service account provisioning.
:::alert{type="info"}
IAM tasks require Kestra Enterprise Edition or Kestra Cloud. The executing service account must hold the appropriate IAM permissions for each operation (see [RBAC permissions](../../07.enterprise/03.auth/rbac/index.md#resources)).
:::
## Authentication
All IAM tasks inherit `kestraUrl` and `auth` from `AbstractKestraTask`.
- `kestraUrl` defaults to `{{ kestra.url }}`, which resolves to the current Kestra instance. Omit it when targeting the same instance.
- `auth` is required. Use an API token stored as a secret:
```yaml
auth:
apiToken: "{{ secret('KESTRA_API_TOKEN') }}"
```
The API token must belong to a user or service account with the appropriate IAM permissions for the operations the flow performs.
## Onboard a user
This flow is triggered by a webhook — for example, from an HRIS system when a new employee is added. It invites the user to the tenant and assigns them to an existing group in a single task.
```yaml
id: onboard_user
namespace: company.iam
editions: ["EE", "Cloud"]
inputs:
- id: email
type: STRING
description: New user's email address
- id: group_id
type: STRING
description: ID of the group to assign the user to
tasks:
- id: invite
type: io.kestra.plugin.kestra.ee.iam.invitations.Create
auth:
apiToken: "{{ secret('KESTRA_API_TOKEN') }}"
email: "{{ inputs.email }}"
groupIds:
- "{{ inputs.group_id }}"
triggers:
- id: from_hris
type: io.kestra.plugin.core.trigger.Webhook
key: "{{ secret('ONBOARD_WEBHOOK_KEY') }}"
```
The task outputs `invitationId`, which you can chain into downstream tasks if needed. If an email server is configured in Kestra, the user receives an invitation email automatically. Otherwise, retrieve the invitation link from the IAM page to share manually.
## Offboard a user
This flow removes a user from a group and revokes their tenant access. Both steps require the user's internal `userId` — retrieve it from the IAM Users page or via your IdP integration.
```yaml
id: offboard_user
namespace: company.iam
inputs:
- id: user_id
type: STRING
description: Internal Kestra user ID
- id: group_id
type: STRING
description: ID of the group to remove the user from
tasks:
- id: remove_from_group
type: io.kestra.plugin.kestra.ee.iam.groups.RemoveMember
auth:
apiToken: "{{ secret('KESTRA_API_TOKEN') }}"
groupId: "{{ inputs.group_id }}"
userId: "{{ inputs.user_id }}"
- id: revoke_access
type: io.kestra.plugin.kestra.ee.iam.tenantAccess.Delete
auth:
apiToken: "{{ secret('KESTRA_API_TOKEN') }}"
userId: "{{ inputs.user_id }}"
triggers:
- id: from_hris
type: io.kestra.plugin.core.trigger.Webhook
key: "{{ secret('OFFBOARD_WEBHOOK_KEY') }}"
```
## Provision a service account for CI/CD
This flow creates a service account and attaches a role to it. It outputs the service account ID so you can use it in downstream automation — for example, to generate an API token or configure a CI/CD secret.
```yaml
id: provision_service_account
namespace: company.iam
inputs:
- id: name
type: STRING
description: Service account name
- id: role_id
type: STRING
description: Role ID to assign
tasks:
- id: create_sa
type: io.kestra.plugin.kestra.ee.iam.serviceAccounts.Set
auth:
apiToken: "{{ secret('KESTRA_API_TOKEN') }}"
name: "{{ inputs.name }}"
- id: bind_role
type: io.kestra.plugin.kestra.ee.iam.bindings.Set
auth:
apiToken: "{{ secret('KESTRA_API_TOKEN') }}"
subjectType: USER
externalId: "{{ outputs.create_sa.id }}"
roleId: "{{ inputs.role_id }}"
outputs:
- id: service_account_id
type: STRING
value: "{{ outputs.create_sa.id }}"
```
`serviceAccounts.Set` upserts by name — running the flow again with the same name updates the existing service account rather than creating a duplicate.
## Create and manage groups
Use `groups.Set` to create or update a group, then manage membership with `groups.AddMember` and `groups.RemoveMember`.
```yaml
tasks:
- id: create_group
type: io.kestra.plugin.kestra.ee.iam.groups.Set
auth:
apiToken: "{{ secret('KESTRA_API_TOKEN') }}"
name: data-engineering
groupDescription: "Data Engineering team"
- id: add_member
type: io.kestra.plugin.kestra.ee.iam.groups.AddMember
auth:
apiToken: "{{ secret('KESTRA_API_TOKEN') }}"
groupId: "{{ outputs.create_group.id }}"
userId: "{{ inputs.user_id }}"
```
Like `serviceAccounts.Set`, `groups.Set` upserts by name and outputs the group `id` for chaining.
## Available IAM tasks
| Task | Purpose |
| --- | --- |
| `io.kestra.plugin.kestra.ee.iam.invitations.Create` | Invite a user; optionally assign to groups |
| `io.kestra.plugin.kestra.ee.iam.invitations.List` | List pending invitations |
| `io.kestra.plugin.kestra.ee.iam.invitations.Delete` | Cancel a pending invitation |
| `io.kestra.plugin.kestra.ee.iam.groups.Set` | Create or update a group (upsert by name) |
| `io.kestra.plugin.kestra.ee.iam.groups.List` | List groups |
| `io.kestra.plugin.kestra.ee.iam.groups.Delete` | Delete a group by ID |
| `io.kestra.plugin.kestra.ee.iam.groups.AddMember` | Add a user to a group |
| `io.kestra.plugin.kestra.ee.iam.groups.RemoveMember` | Remove a user from a group |
| `io.kestra.plugin.kestra.ee.iam.roles.Set` | Create or update a role (upsert by name) |
| `io.kestra.plugin.kestra.ee.iam.roles.List` | List roles |
| `io.kestra.plugin.kestra.ee.iam.roles.Delete` | Delete a role by ID |
| `io.kestra.plugin.kestra.ee.iam.bindings.Set` | Attach a role to a user or group (use `USER` for service accounts) |
| `io.kestra.plugin.kestra.ee.iam.serviceAccounts.Set` | Create or update a service account (upsert by name) |
| `io.kestra.plugin.kestra.ee.iam.serviceAccounts.List` | List service accounts |
| `io.kestra.plugin.kestra.ee.iam.serviceAccounts.Delete` | Delete a service account by ID |
| `io.kestra.plugin.kestra.ee.iam.tenantAccess.Set` | Grant a user access to the tenant by email |
| `io.kestra.plugin.kestra.ee.iam.tenantAccess.Delete` | Revoke a user's tenant access by user ID |
For full property reference, see the [plugin-kestra documentation](/plugins/plugin-kestra).
---
# Prevent Duplicate Executions with Correlation IDs
URL: https://kestra.io/docs/how-to-guides/idempotency
Use `system.correlationId` as an idempotency key to group related executions, trace execution lineage across subflows, and skip duplicate processing.
This guide covers two patterns: setting the correlation ID at execution creation for API-triggered flows, and using a custom label for webhook-triggered flows where the key only becomes available after the execution starts.
:::alert{type="info"}
This guide applies to Kestra Enterprise. API token authentication and tenant-scoped endpoints are Enterprise features.
:::
## How `system.correlationId` works
- A built-in system label present on every execution.
- Defaults to the execution's own ID and propagates to subflows automatically — all executions in a lineage share the root execution's `system.correlationId`.
- Can be set to any stable business identifier at execution creation time, such as a payment intent, message UUID, or event key.
- Immutable once the execution is created — it cannot be changed mid-flow.
## When to use it
Use `system.correlationId` when the same business event might arrive more than once and you need to process it only once:
- payment processing triggered via API
- event-driven pipelines consuming messages from a queue
- any flow where the caller controls the trigger and holds the stable key at creation time
For **webhook-triggered flows**, where the idempotency key arrives in the request headers after the execution is already created, use a custom label instead. See [Webhook-triggered flows](#webhook-triggered-flows).
## Set the correlation ID at execution creation
Pass `system.correlationId` as a `labels` query parameter when creating the execution. Labels use `key:value` format.
Replace `{your-tenant}` with your tenant ID (visible in **Administration → Tenants**).
```bash
curl -X POST "http://localhost:8080/api/v1/{your-tenant}/executions/company.team/payments?labels=system.correlationId:payment-ORD-123" \
-H "Authorization: Bearer {your-api-token}"
```
Use this approach when the caller already holds the stable business key.
## Short-circuit duplicates early
After setting the correlation ID at creation, check whether a successful execution with that key already exists. If one does, skip the current run.
Pass the idempotency key as both the `labels` query parameter and an input so the flow can reference it in the duplicate check:
```bash
curl -X POST "http://localhost:8080/api/v1/{your-tenant}/executions/company.team/payment_flow_guarded?labels=system.correlationId:payment-ORD-123" \
-H "Authorization: Bearer {your-api-token}" \
-F "idempotencyKey=payment-ORD-123"
```
```yaml
id: payment_flow_guarded
namespace: company.team
inputs:
- id: idempotencyKey
type: STRING
tasks:
- id: check_existing
type: io.kestra.plugin.core.http.Request
uri: "http://localhost:8080/api/v1/{{ kv('KESTRA_TENANT') }}/executions/search?filters[labels][EQUALS][system.correlationId]={{ inputs.idempotencyKey }}&filters[namespace][EQUALS]=company.team&filters[flowId][EQUALS]=payment_flow_guarded&filters[state][IN]=SUCCESS&size=1"
method: GET
headers:
Authorization: "Bearer {{ secret('KESTRA_API_TOKEN') }}"
- id: maybe_skip
type: io.kestra.plugin.core.flow.If
condition: "{{ not (outputs.check_existing.body contains '\"total\":0') }}"
then:
- id: skip
type: io.kestra.plugin.core.log.Log
message: "Duplicate {{ inputs.idempotencyKey }} skipped; already succeeded"
else:
- id: continue
type: io.kestra.plugin.core.log.Log
message: "First time for {{ inputs.idempotencyKey }}, proceed"
```
Store your tenant ID and API token as a [KV pair](../../06.concepts/05.kv-store/index.md) and [Secret](../../06.concepts/04.secret/index.md) respectively.
## Webhook-triggered flows
`system.correlationId` is assigned automatically when the execution is created and cannot be changed afterwards. For webhook-triggered flows, the provider's idempotency key is only available once the execution has started. Store it in a **custom label** using the [Labels task](/plugins/core/execution/io.kestra.plugin.core.execution.labels), then use that label for the duplicate check.
```yaml
id: payment_webhook
namespace: company.team
variables:
idem_key: "{{ trigger.headers['Idempotency-Key'] | first }}"
tasks:
- id: set_idempotency_key
type: io.kestra.plugin.core.execution.Labels
labels:
idempotency.key: "{{ vars.idem_key }}"
- id: check_existing
type: io.kestra.plugin.core.http.Request
uri: "http://localhost:8080/api/v1/{{ kv('KESTRA_TENANT') }}/executions/search?filters[labels][EQUALS][idempotency.key]={{ vars.idem_key }}&filters[namespace][EQUALS]=company.team&filters[flowId][EQUALS]=payment_webhook&filters[state][IN]=SUCCESS&size=1"
method: GET
headers:
Authorization: "Bearer {{ secret('KESTRA_API_TOKEN') }}"
- id: maybe_skip
type: io.kestra.plugin.core.flow.If
condition: "{{ not (outputs.check_existing.body contains '\"total\":0') }}"
then:
- id: skip
type: io.kestra.plugin.core.log.Log
message: "Duplicate {{ vars.idem_key }} skipped"
else:
- id: process_payment
type: io.kestra.plugin.core.log.Log
message: "Charge payment for {{ vars.idem_key }}"
triggers:
- id: webhook
type: io.kestra.plugin.core.trigger.Webhook
key: payment-events
```
## Operate with correlation IDs
- **UI filtering:** In Executions, add the label filter `system.correlationId:your-key` to view the entire lineage.
- **API search:** Use `filters[labels][EQUALS][system.correlationId]={value}` and `filters[state][IN]=SUCCESS` as query parameters in the Executions search API to audit or detect duplicates programmatically.
- **Subflows:** The value propagates automatically, so downstream executions share the same `system.correlationId` without additional configuration.
:::alert{type="warning"}
`system.correlationId` identifies and groups executions for the same business event, but it does not prevent duplicate processing on its own. Pair it with an explicit duplicate check as shown in the examples in this guide.
The duplicate check is not atomic. If two executions with the same key start simultaneously, both may pass the SUCCESS check before either completes — neither will be in SUCCESS state yet. For strict once-only guarantees under concurrent load, enforce uniqueness at the system that triggers the execution (message broker deduplication, database unique constraint, or API gateway idempotency).
:::
## Quick checklist
- [ ] Pick a stable business key (payment intent, message ID, event key).
- [ ] For API-triggered flows: set `system.correlationId` at execution creation via the `labels` query parameter.
- [ ] For webhook-triggered flows: store the provider's idempotency key as a custom label via the Labels task.
- [ ] Add an early guard to skip if a successful execution with the same key already exists.
- [ ] Filter by `system.correlationId` in the UI or API for audit and lineage troubleshooting.
---
# Run JavaScript Inside Your Flows
URL: https://kestra.io/docs/how-to-guides/javascript
> Run JavaScript and Node.js scripts in Kestra. Install npm packages at runtime and pass outputs between tasks using inputs and variables.
Run Node.js code directly in your flows and generate outputs.
You can execute NodeJS code in a flow by either writing your NodeJS inline or by executing a `.js` file. You can also get outputs and metrics from your NodeJS code too.
In this example, the flow will install the required npm packages, make an API request to fetch data and use the NodeJS Kestra Library to generate outputs and metrics using this data.
## Scripts
If you want to write a short amount of NodeJS code to perform a task, you can use the `io.kestra.plugin.scripts.node.Script` type to write it directly inside your flow. This allows you to keep everything in one place.
```yaml
id: js_scripts
namespace: company.team
description: This flow will install the npm package in a Docker container, and use kestra's NodeJS Script task to run the script.
tasks:
- id: run_nodejs_script
type: io.kestra.plugin.scripts.node.Script
beforeCommands:
- npm install requestify
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: node:slim
script: |
const requestify = require('requestify');
function GetDockerImageDownloads(imageName){
// Queries the Docker Hub API to get the number of downloads for a specific Docker image.
var url = `https://hub.docker.com/v2/repositories/${imageName}/`
console.log(url)
requestify.get(url)
.then(function(response) {
result = JSON.parse(response.body);
console.log(result['pull_count']);
return result['pull_count'];
})
.catch(function(error) {
console.log(error);
})
}
GetDockerImageDownloads("kestra/kestra")
```
You can read more about the Scripts type in the [Plugin documentation](/plugins/plugin-script-node/io.kestra.plugin.scripts.node.script)
## Commands
If you would prefer to put your NodeJS code in a `.js` file (e.g. your code is much longer or spread across multiple files), you can run the previous example using the `io.kestra.plugin.scripts.node.Commands` type:
```yaml
id: js_commands
namespace: company.team
description: This flow will install the npm package in a Docker container, and use kestra's NodeJS Commands task to run the script.
tasks:
- id: run_nodejs_commands
type: io.kestra.plugin.scripts.node.Commands
namespaceFiles:
enabled: true
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: node:slim
beforeCommands:
- npm install requestify
commands:
- node docker_image_downloads.js
```
You'll need to add your JavaScript code using the Editor or [sync it using Git](../../version-control-cicd/04.git/index.md) so Kestra can see it. You'll also need to set the `enabled` flag for the `namespaceFiles` property to `true` so Kestra can access the file.
You can read more about the Commands type in the [Plugin documentation](/plugins/plugin-script-node/io.kestra.plugin.scripts.node.commands).
## Handling Outputs
If you want to get a variable or file from your JavaScript code, you can use an [output](../../05.workflow-components/06.outputs/index.md).
To pass your variables to Kestra, install the [`@kestra-io/libs` npm package](https://npm.io/package/@kestra-io/libs).
```bash
npm install @kestra-io/libs
```
### Variable Output
You'll need to use the `Kestra` class to pass your variables to Kestra as outputs. Using the `outputs` method, you can pass a dictionary of variables where the `key` is the name of the output you'll reference in Kestra.
Using the same example as above, we can pass the number of downloads as an output.
```javascript
const requestify = require('requestify');
const Kestra = require('@kestra-io/libs');
function GetDockerImageDownloads(imageName){
// Queries the Docker Hub API to get the number of downloads for a specific Docker image.
var url = `https://hub.docker.com/v2/repositories/${imageName}/`
console.log(url)
requestify.get(url)
.then(function(response) {
result = JSON.parse(response.body);
Kestra.outputs({"pull_count": result['pull_count']})
return result['pull_count'];
})
.catch(function(error) {
console.log(error);
})
}
GetDockerImageDownloads("kestra/kestra")
```
Once your NodeJS file has executed, you'll be able to access the outputs in later tasks as seen below:
```yaml
id: outputs_nodejs
namespace: company.team
description: This flow will install the npm package in a Docker container, and use kestra's NodeJS Commands task to run the script.
tasks:
- id: run_nodejs_commands
type: io.kestra.plugin.scripts.node.Commands
namespaceFiles:
enabled: true
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: node:slim
beforeCommands:
- npm install requestify
- npm install @kestra-io/libs
commands:
- node outputs_nodejs.js
- id: log_downloads
type: io.kestra.plugin.core.log.Log
message: "Number of downloads: {{ outputs.run_nodejs_commands.vars.pull_count }}"
```
_This example works for both `io.kestra.plugin.scripts.node.Script` and `io.kestra.plugin.scripts.node.Commands`._
### File Output
Inside of your JavaScript code, write a file to the system. You'll need to add `outputFiles` property to your flow and list the file you're trying to access. In this case, we want to access `downloads.txt`. More information on the formats you can use for this property can be found in [Script Output Metrics](../../16.scripts/06.outputs-metrics/index.md).
The example below write a `.txt` file containing the number of downloads, similar the output we used earlier. We can then read the content of the file using the syntax `{{ outputs.{task_id}.outputFiles['{filename}'] }}`
```yaml
id: js_outputs_files_scripts
namespace: company.team
description: This flow will install the npm package in a Docker container, and use kestra's NodeJS library to generate outputs (number of downloads of the Kestra Docker image).
tasks:
- id: nodejs_outputs
type: io.kestra.plugin.scripts.node.Script
beforeCommands:
- npm install requestify
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: node:slim
outputFiles:
- downloads.txt
script: |
const requestify = require('requestify');
const fs = require('fs');
function GetDockerImageDownloads(imageName){
// Queries the Docker Hub API to get the number of downloads for a specific Docker image.
var url = `https://hub.docker.com/v2/repositories/${imageName}/`
console.log(url)
requestify.get(url)
.then(function(response) {
result = JSON.parse(response.body);
fs.writeFile("downloads.txt", result['pull_count'].toString(), (err) => {
if (err) throw err;
})
return result['pull_count'];
})
.catch(function(error) {
console.log(error);
})
}
GetDockerImageDownloads("kestra/kestra");
```
We can also preview our file directly in the Outputs tab as well.

_This example works for both `io.kestra.plugin.scripts.node.Script` and `io.kestra.plugin.scripts.node.Commands`._
## Handling Metrics
You can also get [metrics](../../16.scripts/06.outputs-metrics/index.md#outputs-and-metrics-in-script-and-commands-tasks) from your NodeJS code. In this example, we can use the `Date` class to time the execution time of the function and then pass this to Kestra so it can be viewed in the Metrics tab. No flow changes are needed.
```javascript
const Kestra = require('@kestra-io/libs');
const requestify = require('requestify');
function GetDockerImageDownloads(imageName){
// Queries the Docker Hub API to get the number of downloads for a specific Docker image.
var url = `https://hub.docker.com/v2/repositories/${imageName}/`
console.log(url)
requestify.get(url)
.then(function(response) {
result = JSON.parse(response.body);
Kestra.outputs({"pull_count": result['pull_count']})
return result['pull_count'];
})
.catch(function(error) {
console.log(error);
})
}
start = new Date().getTime();
GetDockerImageDownloads("kestra/kestra")
end = new Date().getTime();
duration = (end - start) / 1000
Kestra.timer('duration', end - start);
```
Once this has executed, `duration` will be viewable under **Metrics**.

## Automate JavaScript with triggers
You can also use JavaScript itself as polling logic by using `ScriptTrigger` or `CommandsTrigger`. These trigger types run Node.js code on an interval and start a flow execution only when the `exitCondition` matches.
Use `ScriptTrigger` for inline Node.js code:
```yaml
id: node_script_trigger
namespace: company.team
triggers:
- id: script_failure
type: io.kestra.plugin.scripts.node.ScriptTrigger
interval: PT10S
exitCondition: "exit 1"
edge: true
script: |
throw new Error("boom");
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "Triggered with exitCode={{ trigger.exitCode }} (condition={{ trigger.condition }})"
```
Use `CommandsTrigger` when you want to run Node.js commands instead:
```yaml
id: node_commands_trigger
namespace: company.team
triggers:
- id: on_fail
type: io.kestra.plugin.scripts.node.CommandsTrigger
interval: PT5S
exitCondition: "exit 1"
edge: true
commands:
- node -e "throw new Error('boom')"
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "Triggered with exitCode={{ trigger.exitCode }} (condition={{ trigger.condition }})"
```
These trigger types support:
- `interval` to control how often the script or commands run
- `exitCondition` to match an exit code such as `exit 1`, or a regex or substring matched against emitted vars and failure logs
- `edge` to emit only on a transition from not matching to matching
## Execute GraalVM Task
Kestra also supports GraalVM integration, allowing you to execute JavaScript code directly on the JVM, with the potential for performance improvements. There are currently two tasks:
- [Eval](/plugins/plugin-graalvm/js-graalvm/io.kestra.plugin.graalvm.js.eval)
- [FileTransform](/plugins/plugin-graalvm/js-graalvm/io.kestra.plugin.graalvm.js.filetransform)
In this example, the `Eval` task is used to manipulate data from a previous task. As GraalVM can polyfill from Java, we can use the `int()` function to convert the string into an integer. Additionally, using the `outputs` property simplifies the process of fetching variables from JavaScript and accessing them inside Kestra. It is useful if you want to manipulate data and pass the new format to another task.
```yaml
id: parse_json_data
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: http://xkcd.com/info.0.json
- id: graal
type: io.kestra.plugin.graalvm.js.Eval
outputs:
- data
script: |
data = {{ read(outputs.download.uri) }}
data["next_month"] = int(data["month"]) + 1
```
---
# Connect Web Apps to Kestra via Webhooks
URL: https://kestra.io/docs/how-to-guides/js-webhook
> Integrate your web applications with Kestra using Webhook triggers to start workflows from your frontend or backend code.
Integrate Kestra into your JavaScript App using Webhooks.
With Kestra's API First Design, you can build web applications to integrate with Kestra acting as a backend server.
This can be useful if you want a request from your website to be made and start a workflow execution to process orders. For example, you have an online shop where orders are made and you want Kestra to receive these orders and start processing them.
In this guide, we'll walk through how you can set up Kestra to receive webhooks as well as build a basic JavaScript application with React.js that can make requests.
## Configuring CORS
To allow requests to Kestra from a JavaScript application running locally, enable CORS in your Kestra configuration:
```yaml
micronaut:
server:
cors:
enabled: true
```
More information can be found in the [Observability and Networking configuration](../../configuration/03.observability-and-networking/index.md).
## Building a Workflow with a Webhook Trigger
The JavaScript application needs a workflow with a [Webhook Trigger](../../05.workflow-components/07.triggers/03.webhook-trigger/index.md) to receive requests and start executions.
Once we've added it, we can add any tasks to run. In this example, we have a log message that will log the request body field `dataField` from the webhook:
```yaml
id: webhook_example
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: "{{ trigger.body.dataField ?? 'null' }}"
triggers:
- id: webhook
type: io.kestra.plugin.core.trigger.Webhook
key: abcdefg
```
## Building our JavaScript application with React.js
In this example, I am using React.js to interact with Kestra but this will work with any web framework that can make requests.
Create the application using `create-react-app`:
```bash
npx create-react-app example
```
Start it with:
```bash
npm start
```
Navigate to the application at `localhost:3000`.
With the application running, modify `App.js` to make a request to Kestra.
First, install axios to make the POST request:
```bash
npm install axios
```
Once we've done that, we can add a `useState` hook to help us make our request and handle the response, specifically handle the request body state:
```js
function App() {
const [formData, setFormData] = useState({});
const handleSubmit = async (e) => {
e.preventDefault();
try {
await axios.post('http://localhost:8080/api/v1/main/executions/webhook/company.team/webhook_example/abcdefg',
formData).then(response => {
console.log(response.data)
});
} catch (error) {
console.error('Error:', error);
}
};
return (
Kestra Webhook Example
);
}
```
Get the Webhook URL by navigating to **Triggers** at the top of the flow in Kestra and hovering over the webhook icon on the right:

This example makes a request with data from a form (added later) using the `useState` hook, storing state in `formData` and updating it using `setFormData`.
Add UI elements to set `formData`, make the request, and display the response.
Modify the JSX in the return statement to include a form that handles the request:
```js
function App() {
...
return (
Kestra Webhook Example
Send a message to Kestra
);
}
```
The form uses `onSubmit` and `onChange` to call functions. We use `onSubmit` on the button to call our newly added `handleSubmit` function.
However, we don't have a function to handle automatically adding our text to our state variable `formData`. We can add a new function called `handleChange` which will use our state updater function `setFormData` to update `formData` everytime new text is added to the input. This means that when we press **Submit**, the text is ready to be sent in a request body.
```js
function App() {
const [formData, setFormData] = useState({});
const handleSubmit = async (e) => {
e.preventDefault();
try {
await axios.post('http://localhost:8080/api/v1/main/executions/webhook/company.team/webhook_example/abcdefg',
formData).then(response => {
console.log(response.data)
});
} catch (error) {
console.error('Error:', error);
}
};
const handleChange = (e) => {
setFormData({ ...formData, [e.target.name]: e.target.value });
};
return (
Kestra Webhook Example
Send a message to Kestra
);
}
```
Now our example will collect the data in the `input` field as `dataField` and send it in our request as a key value pair: `dataField: {the input value}`. For example, if I type "Hello" and press **Submit**, it will send the body `{dataField: "Hello"}`.

The last thing to add now is to display the response back to the user.
We can add another state variable called `responseData` to handle the response from the request. We can add it into our JSX to only display if we have a response:
```js
{responseData.id &&
Execution ID: {responseData.id}
}
```
In this case, `id` is the Execution ID of the Execution that was started because of the webhook request.
Now we've added the response, our full `App.js` should look like this now:
```js
import React, { useState } from 'react';
import axios from 'axios';
import './App.css';
function App() {
const [formData, setFormData] = useState({});
const [responseData, setResponseData] = useState({});
const handleSubmit = async (e) => {
e.preventDefault();
try {
await axios.post('http://localhost:8080/api/v1/main/executions/webhook/company.team/webhook_example/abcdefg',
formData).then(response => {
setResponseData(response.data)
});
} catch (error) {
console.error('Error:', error);
}
};
const handleChange = (e) => {
setFormData({ ...formData, [e.target.name]: e.target.value });
};
return (
Kestra Webhook Example
Send a message to Kestra
{responseData.id &&
Execution ID: {responseData.id}
}
);
}
export default App;
```

This will:
1. Display a Form with an input and a button.
2. Make a request to Kestra with the input data as our request body.
3. Receive the response and display it to the user.
When we type in a value and press **Submit**, we can see a new execution is created in Kestra and our request body was received and used in our Log task:

:::collapse{title="CSS Styling"}
These are the CSS styles used in the example:
```css
.App {
text-align: center;
}
.App-header {
background-color: #4b0aaa;
min-height: 100vh;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
font-size: calc(10px + 2vmin);
color: white;
}
.App h1 {
font-size: 34px;
}
.App h2 {
font-size: 24px;
}
.App p {
font-size: 16px;
}
```
:::
---
# Work with JSON in Kestra
URL: https://kestra.io/docs/how-to-guides/json
> Learn how to interact with JSON data in Kestra workflows, including parsing, accessing nested fields, and using jq expressions.
Interact with JSONs using expressions.
APIs often use JSON bodies to send data. Being able to interact with them in your workflows is crucial to any API related orchestration.
## Making a Request Inside of Your Workflow
The API `https://kestra.io/api/mock` will return a JSON body that looks like the following:
```json
{
"title":"Success",
"method":"GET",
"params":{},
"code":200,
"createdAt":"2025-07-04T15:42:29.545Z",
"body":"Request processed successfully"
}
```
Kestra can make a request to this API using the `Request` task. This will give us an output called `body` containing our JSON body. To access this in later tasks, we can use an [expression](../../expressions/index.mdx) like `{{ outputs.request.body }}`. This will return the full body:
```yaml
id: json_demo
namespace: company.team
tasks:
- id: request
type: io.kestra.plugin.core.http.Request
uri: https://kestra.io/api/mock
- id: log
type: io.kestra.plugin.core.log.Log
message: "My response: {{ outputs.request.body }}"
```
The log message returns:
```json
My response: {"title":"Success","method":"GET","params":{},"code":200,"createdAt":"2025-07-04T16:36:44.193Z","body":"Request processed successfully"}
```
## Accessing Part of the Body
However, if the body is large, we may only want to access a certain part of it. To do this, `jq` is required as the expression returns a string, not a JSON. Using `jq`, the JSON can be parsed and accessed:
```yaml
{{ outputs.request.body | jq('.title') | first }}
```
This will access the key `title` from the JSON. `jq` will return the result in an array when used within an expression. To access the value, add the function `first` to the end of the expression to remove it from the array.
We can put that into the example:
```yaml
id: json_demo
namespace: company.team
tasks:
- id: request
type: io.kestra.plugin.core.http.Request
uri: https://kestra.io/api/mock
- id: log
type: io.kestra.plugin.core.log.Log
message: "My response: {{ outputs.request.body | jq('.title') | first }}"
```
The log message says `My response: Success`.
## Nested JSON
If the JSON you're working with has multiple levels, you can extend the `jq` expression.
In this example, the API `https://kestra.io/api/mock?example=test` has additional parameters which return the following body with nesting:
```json
{
"title": "Success",
"method": "GET",
"params": {
"example": "test"
},
"code": 200,
"createdAt": "2025-07-04T16:39:02.871Z",
"body": "Request processed successfully"
}
```
The `jq` expression can be extended as follows to access `example`:
```yaml
{{ outputs.request.body | jq('.params.example') | first }}
```
It looks like this when added to the workflow:
```yaml
id: json_demo
namespace: company.team
tasks:
- id: request
type: io.kestra.plugin.core.http.Request
uri: https://kestra.io/api/mock?example=test
- id: log
type: io.kestra.plugin.core.log.Log
message: "My response: {{ outputs.request.body | jq('.params.example') | first }}"
```
The log message returns `My response: test`.
## Debugging Expressions
You can use [Debug Expression](../../05.workflow-components/06.outputs/index.md#using-debug-expression) to test expressions without running your workflow. This is useful for inspecting different parts of a JSON structure.

---
# Set Up Secrets from a Helm Chart
URL: https://kestra.io/docs/how-to-guides/kubernetes-secrets
> Learn how to pass secrets to your Kestra deployment via Helm Chart using environment variables or Kubernetes Secrets.
How to add Kestra Secrets to your Helm Chart deployment.
:::alert{type="info"}
This page is only relevant for the Open-Source edition of Kestra. For the Enterprise Edition, you can use the built-in [Secrets](../../07.enterprise/02.governance/secrets-manager/index.md) functionality allowing you to securely store your secrets in an [external secrets manager](../../07.enterprise/02.governance/secrets-manager/index.md) of your choice.
:::
## Pass environment variables directly
The simplest way to pass secrets to Kestra is to use environment variables referenced using the `common.extraEnv` property. Each environment variable's key must start with `SECRET_`.
To add two secrets to your Helm Chart:
1. `DB_USERNAME` with the value `admin`
2. `DB_PASSWORD` with the value `password`
You can set them directly in your Helm Chart `values.yaml` as follows:
```yaml
deployments:
standalone:
enabled: true
common:
extraEnv:
- name: SECRET_DB_USERNAME
value: "admin"
- name: SECRET_DB_PASSWORD
value: "password"
```
:::alert{type="info"}
Note how each environment variable's key starts with `SECRET_`. This is important for Kestra to recognize them as secrets.
:::
Now, install or upgrade your Helm Chart:
```shell
helm repo add kestra https://helm.kestra.io/
helm install kestra kestra/kestra -f values.yaml
## or if you already have Kestra installed:
helm upgrade kestra kestra/kestra -f values.yaml
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=kestra,app.kubernetes.io/instance=kestra,app.kubernetes.io/component=standalone" -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward $POD_NAME 8080:8080
```
To test that the secrets have been correctly set, go to the UI e.g. http://localhost:8080 and create a new flow:
```yaml
id: secret_test
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.output.OutputValues
values:
username: "{{ secret('DB_USERNAME') }}"
password: "{{ secret('DB_PASSWORD') }}"
```
Execute the flow and check the output values in the Outputs tab in the UI. You should see the values `admin` and `password`.
---
## Pass environment variables from a Kubernetes Secret
If you want to define your secrets in a Kubernetes Secret, you can use the `common.extraEnvFrom` property in your Helm Chart. This property allows you to reference an existing Kubernetes Secret and pass its values as environment variables to Kestra.
Here is an example of a Kubernetes Secret definition:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: db-creds
type: Opaque
stringData:
SECRET_DB_USERNAME: admin
SECRET_DB_PASSWORD: password
```
First, create the Secret in your Kubernetes cluster:
```shell
kubectl apply -f secret.yaml
```
Then, reference this secret in your Helm Chart `values.yaml`:
```yaml
deployments:
standalone:
enabled: true
common:
extraEnvFrom:
- secretRef:
name: db-creds
```
Redeploy your Helm Chart:
```shell
helm upgrade kestra kestra/kestra -f values.yaml
```
And test the secrets in a flow as described in the previous section.
In this method, the Kubernetes Secret's keys must start with `SECRET_` to be recognized as Kestra Secrets.
---
## Use Kubernetes Secrets as Kestra Secrets with `configurations.secrets`
An alternative is to mount an entire Kubernetes Secret as a [Kestra configuration](../../configuration/01.configuration-basics/index.md) file using the `configurations.secrets` property.
For example, in `values.yaml`:
```yaml
configurations:
secrets:
- name: db-creds
key: db.yml
```
And in your Helm chart, define the secret in `extraManifests`:
```yaml
extraManifests:
- apiVersion: v1
kind: Secret
metadata:
name: db-creds
stringData:
db.yml: |
kestra:
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
username: admin
password: password
```
This method avoids the need for encoding and allows you to configure secrets in YAML format directly.
---
## Summary
- Use `common.extraEnv` for simple inline secrets.
- Use `common.extraEnvFrom` to load secrets from existing Kubernetes Secret objects.
- Use `configurations.secrets` when you want to mount YAML-based secrets as part of Kestra's configuration.
Choose the method that best fits your security and deployment requirements.
---
# Synchronize Local Flows in Kestra
URL: https://kestra.io/docs/how-to-guides/local-flow-sync
> Synchronize Kestra flows from a local directory to your development instance for a seamless developer experience using file watching.
Sync Flows from a local directory.
How to synchronize flows from a local directory on a local development instance.
## Configure your instance
:::alert{type="warning"}
This feature is only for local development, that is why you can not connect to a distant Kestra instance.
:::
When developing on a local Kestra instance, it can be more convenient to have your flows in a local directory, maybe synchronize with a GitHub repository on your local machine, and have Kestra automatically load them.
Below is the minimal configuration to enable local flow synchronization:
```yaml
micronaut:
io:
watch:
enabled: true
paths:
- /path/to/your/flows
```
Multiple paths can be provided, and nested files will also be watched.
Files have to end with `.yml` or `.yaml` to be considered as a flow. And only valid flows will be loaded, invalid flows will be ignored.
File created locally should use `_..yml` or `__.yml` syntax to be loaded properly. For open-source users, `tenantID` is always `main`.
Flow created inside the UI will be created at the root of the first path supplied in the configuration.
:::alert{type="info"}
If you are using the docker-compose installation, you will need to mount a volume so Kestra container can access your local folder.
```yaml
volumes:
# ... other volumes
- ./local_folder:/docker_folder
environment:
KESTRA_CONFIGURATION: |
micronaut:
io:
watch:
enabled: true
paths:
- /docker_folder
```
:::
## Details
At startup, every file in the watched directory will be loaded into the database. Then every flow not existing in the watched directory will be created in the first path supplied in the configuration.
When a file is created, updated, or deleted in the watched directory, Kestra will automatically load the flow into the database or remove it if the file is deleted.
If a flow is created, updated or deleted in the UI, the file will be created, updated or deleted in the watched directory.
In the Kestra UI, you cannot change an ID nor a namespace, but in a file you can, in this case, the previous flow will be deleted, and a new one created.
---
# Long-Running Tasks on Kubernetes in Kestra
URL: https://kestra.io/docs/how-to-guides/long-running-intensive-tasks
> Execute long-running and resource-intensive tasks on Kubernetes using Kestra's podCreate task or Kubernetes Task Runners.
Schedule long running and intensive processes with Kestra on Kubernetes.
Long running tasks hold strong importance in the world of automation.
They can range from data processing, machine learning, and data analytics to batch processing, ETL, and more.
While these tasks are essential for business operations, they can be resource-intensive and time-consuming while requiring specific hardware. To execute these tasks efficiently, you need a robust and scalable infrastructure that can handle the workload effectively.
Kestra offers various task execution solutions such as Docker, local processes, and Kubernetes. See [Task Runners](../../07.enterprise/04.scalability/task-runners/index.md) for more details.
This guide focuses on executing long-running tasks on **Kubernetes** using Kestra.
Kubernetes pods are a great fit due to the control and flexibility they provide. With Kubernetes, you can precisely define resource requirements, permissions, namespace, handle workload identity, and ensure proper networking for your tasks. Pods can also access other Kubernetes services hosted on the cluster such as databases, storage, and applications.
As an example, this guide uses a [dbt job](https://docs.getdbt.com/docs/running-a-dbt-project/run-your-dbt-projects) to demonstrate how Kestra executes complex tasks on Kubernetes with resource requirements.
## Schedule task in a Kubernetes pod using podCreate
Kestra's [podCreate](/plugins/plugin-kubernetes/kubernetes-core/io.kestra.plugin.kubernetes.core.podcreate) task allows you to launch a Kubernetes pod directly by providing the complete Kubernetes YAML configuration as an input. This gives you full control over the pod’s specifications, such as CPU, memory, image, or node selector.
Here is an example of a dbt job that runs on Kubernetes using Kestra:
```yaml
id: my-dbt-job
namespace: dev
tasks:
- id: dbt-command
type: io.kestra.plugin.kubernetes.PodCreate
# Retry the task if it fails
retry:
behavior: RETRY_FAILED_TASK
maxAttempts: 2
type: constant
interval: PT5M
warningOnRetry: true
namespace: kestra
# Define the commands to run
inputFiles:
dbt-commands.sh: |
#!/bin/bash
# Exit on error
set -eo pipefail
# Clone the dbt example repository
git clone --depth 1 https://github.com/dbt-labs/jaffle_shop_duckdb.git --branch duckdb --single-branch
# Copy the dbt example repository to the working directory
cp -a jaffle_shop_duckdb/. .
# dbt commands to run
dbt deps
dbt build
# Define the pod specification using the Kubernetes YAML syntax
spec:
restartPolicy: Never
containers:
- name: dbt-duckdb
image: ghcr.io/kestra-io/dbt-duckdb:latest
# Specify resource requirements
resources:
request:
cpu: "300m"
memory: "500Mi"
# Run the script in the container
command:
- "/bin/bash"
- "{{workingDir}}/dbt-commands.sh"
# Node selector to run the pod on a specific node
nodeSelector: {}
```
This flow will:
- create a Kubernetes pod in the `kestra` namespace with the specified resource requirements: 300m CPU and 500Mi memory
- clone the dbt example repository inside the pod
- run the dbt seed and build commands

At the end of the execution, the pod is deleted, and the logs remain available in the Kestra UI.

## Embrace Kestra versatility with Kubernetes Task Runners
While podCreate provides deep control, it takes aways all the benefits of Kestra's rich plugins ecosystem, [dbt plugin](/plugins/plugin-dbt/dbt-cli/io.kestra.plugin.dbt.cli.dbtcli) in this case. Also it can be cumbersome to manage complex Kubernetes pod YAML specification for each task, especially when you have multiple commands to run.
Kestra’s Task Runners let you define workflows that benefit from the plugin system, using familiar plugins while still taking advantage of Kubernetes to secure and scale tasks effectively.
Task Runners also let you test a task locally using Docker or Process before deploying it on Kubernetes.
The same example would look like this using a Task Runner:
```yaml
id: dbt-task-runner
namespace: dev
tasks:
- id: dbt_build
type: io.kestra.plugin.dbt.cli.DbtCLI
taskRunner:
type: io.kestra.plugin.ee.kubernetes.runner.Kubernetes
namespace: kestra
resources:
request:
cpu: "300m"
memory: "500Mi"
containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
commands:
- git clone --depth 1 https://github.com/dbt-labs/jaffle_shop_duckdb.git --branch duckdb --single-branch
- cp -a jaffle_shop_duckdb/. .
- dbt deps
- dbt build
```

To test this flow on a local Kestra instance, change the `taskRunner` type to `io.kestra.plugin.scripts.runner.docker.Docker`:
```yaml
id: dbt-task-runner
namespace: dev
tasks:
- id: dbt_build
type: io.kestra.plugin.dbt.cli.DbtCLI
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
commands:
- git clone --depth 1 https://github.com/dbt-labs/jaffle_shop_duckdb.git --branch duckdb --single-branch
- cp -a jaffle_shop_duckdb/. .
- dbt deps
- dbt build
```
This flexibility allows you to test your tasks locally before deploying them on Kubernetes.
## Conclusion
Kestra provides a flexible way to execute long-running and intensive tasks on Kubernetes. With Kestra’s Task Runners, you can define workflows that use the plugin system while taking advantage of Kubernetes to secure and scale tasks effectively.
If needed, you can also use the `podCreate` task to launch a Kubernetes pod directly by providing the complete Kubernetes YAML configuration as input.
---
# Loop Over a List of Values
URL: https://kestra.io/docs/how-to-guides/loop
> Learn how to iterate over a list of values in Kestra workflows using the Loop task, access iteration context, collect outputs, and run iterations in parallel.
Use the `Loop` task to iterate over a list of values and run tasks for each item. Each iteration runs as an isolated sub-execution with access to the current value via `item.value` and the zero-based index via `item.index`.
## Prerequisites
Before you begin:
- Deploy [Kestra](../../02.installation/index.mdx) in your preferred development environment.
- Ensure you have a [basic understanding of how to run Kestra flows.](../../03.tutorial/index.mdx)
## Basic iteration
The simplest use of `Loop` iterates over a static list and runs child tasks for each item. The example below makes an API call for each author in the list.
```yaml
id: loop_basic
namespace: company.team
tasks:
- id: loop
type: io.kestra.plugin.core.flow.Loop
values: ["pynchon", "dostoyevsky", "hedayat"]
tasks:
- id: api
type: io.kestra.plugin.core.http.Request
uri: "https://openlibrary.org/search.json?author={{ item.value }}&sort=new"
```
Inside each iteration:
- `{{ item.value }}` — the current value from the list
- `{{ item.index }}` — the zero-based position (0, 1, 2, …)
After execution, the Gantt view shows a separate task group for each author.
When `values` contains objects, each `item.value` is a JSON string. Use `fromJson(item.value).field` to access fields — `item.value.field` does not work.
## Nested loops
To iterate over multiple dimensions, nest `Loop` tasks. The inner loop accesses the outer loop's value with `{{ item.parent.value }}`. For three or more levels, `{{ item.parents[1].value }}` is the grandparent — `item.parents[0]` is the same as `item.parent`.
```yaml
id: loop_nested
namespace: company.team
tasks:
- id: outer
type: io.kestra.plugin.core.flow.Loop
values: ["bucket1", "bucket2"]
tasks:
- id: inner
type: io.kestra.plugin.core.flow.Loop
values: [2025, 2026]
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "bucket={{ item.parent.value }} year={{ item.value }}"
```
## Collect outputs across iterations
By default, outputs produced inside a loop are not visible to tasks that run after it. Declare an `outputs:` block on the Loop task to surface values explicitly. After the loop, `outputs.loop.outputs` is a list of per-iteration results. Use `loopOutputs()` to extract one field across all iterations as a flat list.
```yaml
id: loop_outputs
namespace: company.team
tasks:
- id: loop
type: io.kestra.plugin.core.flow.Loop
values: ["alpha", "beta", "gamma"]
fetchType: AUTO
outputs:
- id: label
type: STRING
value: "{{ outputs.process.value }}"
tasks:
- id: process
type: io.kestra.plugin.core.debug.Return
format: "processed {{ item.value }}"
- id: read_outputs
type: io.kestra.plugin.core.log.Log
message: "All results: {{ loopOutputs(outputs.loop.outputs, 'label') }}"
```
## Run iterations in parallel
Set `concurrencyLimit` to a positive integer to cap how many iterations run at once. Setting it to `0` removes the cap entirely — only do this for small datasets where you understand the resource implications.
```yaml
id: loop_parallel
namespace: company.team
tasks:
- id: loop
type: io.kestra.plugin.core.flow.Loop
values: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
concurrencyLimit: 0
tasks:
- id: parallel
type: io.kestra.plugin.core.flow.Parallel
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "Processing {{ item.value }}"
- id: shell
type: io.kestra.plugin.scripts.shell.Commands
commands:
- "echo done {{ item.value }}"
```
## Fan out with subflows
Use `Loop` with `Subflow` to launch an isolated child execution per iteration. Each subflow gets its own retry policy, logs, and failure state — useful when you want per-item isolation rather than running everything inside a single parent execution.
The subflow to call per iteration:
```yaml
id: process_item
namespace: company.team
inputs:
- id: item
type: STRING
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "Processing: {{ inputs.item }}"
```
The parent flow that queries a dataset and fans out one subflow per row:
```yaml
id: fan_out
namespace: company.team
tasks:
- id: extract
type: io.kestra.plugin.jdbc.duckdb.Query
sql: |
INSTALL httpfs;
LOAD httpfs;
SELECT * FROM read_csv_auto('https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv', header=True);
store: true
- id: loop
type: io.kestra.plugin.core.flow.Loop
values: "{{ outputs.extract.uri }}"
tasks:
- id: process
type: io.kestra.plugin.core.flow.Subflow
namespace: company.team
flowId: process_item
wait: true
transmitFailed: true
inputs:
item: "{{ item.value }}"
```
Set `wait: true` so the parent tracks each child's outcome. Set `transmitFailed: true` to fail the loop if any subflow fails. Combine with `concurrencyLimit` on the Loop task to cap how many subflows run simultaneously.
## Next steps
- For the full Loop property reference, see the [Loop task documentation](/plugins/core/flow/io.kestra.plugin.core.flow.loop).
- For output collection patterns, error handling, and map-reduce examples, see the [Flowable Tasks](../../05.workflow-components/01.tasks/00.flowable-tasks/index.md#loop) reference.
- For Loop best practices, see the [Loop best practices guide](../../14.best-practices/11.loop/index.md).
---
# Safeguard Microservices with Unit Tests
URL: https://kestra.io/docs/how-to-guides/microservices-unit-tests
> Write unit tests for Kestra Enterprise workflows. Create test suites, mock task dependencies, and assert flow behavior before deploying to production.
Build an automated guardrail that pings a microservice endpoint, alerts Slack when it fails, and runs only when its unit tests pass.
Modern microservices and API backends often expose health endpoints. With Kestra you can monitor those endpoints, write unit tests to validate the monitoring flow, and gate downstream automations on the test results. This guide walks through:
- Creating a flow that checks an HTTP endpoint and notifies Slack when it is down
- Writing Enterprise Edition unit tests that cover both success and failure paths
- Triggering a downstream flow only when the test suite passes
## Prerequisites
- Kestra Enterprise Edition (required for [Unit Tests](../../07.enterprise/02.governance/unit-tests/index.md) and the `RunTest` task)
- A Slack Incoming Webhook URL (or another channel supported by the Notifications plugin)
- A Kestra API token stored as `KESTRA_API_TOKEN` (used by the test runner flow)
## Step 1: Monitor the API endpoint
Create the following flow in the your namespace to send an alert when the target server is unreachable:
```yaml
id: microservices-and-apis
namespace: tutorial
description: Microservices and APIs
inputs:
- id: server_uri
type: URI
defaults: https://kestra.io
- id: slack_webhook_uri
type: URI
defaults: https://kestra.io/api/mock
tasks:
- id: http_request
type: io.kestra.plugin.core.http.Request
uri: "{{ inputs.server_uri }}"
options:
allowFailed: true
- id: check_status
type: io.kestra.plugin.core.flow.If
condition: "{{ outputs.http_request.code != 200 }}"
then:
- id: server_unreachable_alert
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
url: "{{ inputs.slack_webhook_uri }}"
payload: |
{
"channel": "#alerts",
"text": "The server {{ inputs.server_uri }} is down!"
}
else:
- id: healthy
type: io.kestra.plugin.core.log.Log
message: Everything is fine!
```

This flow issues an HTTP request, lets it fail gracefully (`allowFailed: true`), then either sends a Slack alert or logs a healthy status.
Breakdown of the components:
- **Inputs**
- `server_uri`: parameterizes the target so you can reuse the flow for staging, production, or any other health endpoint.
- `slack_webhook_uri`: stores the Slack webhook that receives alerts without hardcoding secrets in the flow body. Instead of an input, you can also use the [KV Store](../../06.concepts/05.kv-store/index.md) or a [secret](../../06.concepts/04.secret/index.md) in the `url` property.
- **`http_request` task**: performs the status check and captures the HTTP code; `allowFailed` ensures the flow continues even if the request fails.
- **`check_status` conditional**: branches on the HTTP response, triggering the Slack alert when the service is down or logging “Everything is fine!” when the endpoint returns 200.
## Step 2: Add unit tests
Next, define unit tests to cover both outcomes. Save the snippet below as a test resource in the same namespace.
```yaml
id: test_microservices_and_apis
flowId: microservices-and-apis
namespace: tutorial
testCases:
- id: server_should_be_reachable
type: io.kestra.core.tests.flow.UnitTest
fixtures:
inputs:
server_uri: https://kestra.io
assertions:
- value: "{{ outputs.http_request.code }}"
equalTo: 200
- id: server_should_be_unreachable
type: io.kestra.core.tests.flow.UnitTest
fixtures:
inputs:
server_uri: https://kestra.io/bad-url
tasks:
- id: server_unreachable_alert
description: no Slack message from tests
assertions:
- value: "{{ outputs.http_request.code }}"
notEqualTo: 200
```

Each test case supplies fixtures (inputs and optional task overrides) and assertions. The second test disables the Slack call while still confirming that the alert path runs when the endpoint fails.
Breakdown:
- **Test definition**: `id`, `flowId`, and `namespace` tie this test suite to the flow created in Step 1.
- **`server_should_be_reachable` case**: feeds a valid `server_uri` and asserts the HTTP response code is 200.
- **`server_should_be_unreachable` case**: points to a bad URL, stubs the Slack task so no message is sent during testing (reduce channel noise or spamming test messages), and asserts the HTTP code differs from 200.
## Step 3: Run downstream logic only when tests pass
Finally, create a control flow that executes the test suite and gates additional work on the result. The `RunTest` task returns a boolean in `outputs.run_test.result.state`.
```yaml
id: run_if_tests_pass
namespace: tutorial
tasks:
- id: run_test
type: io.kestra.plugin.kestra.ee.tests.RunTest
auth:
apiToken: "{{ secret('KESTRA_API_TOKEN') }}"
namespace: tutorial
testId: test_microservices_and_apis
- id: run_if_tests_pass
type: io.kestra.plugin.core.flow.If
condition: "{{ outputs.run_test.result.state }}"
then:
- id: log
type: io.kestra.plugin.core.log.Log
message: hello
```

Replace the final `log` task with deployments, escalations, or other automations that should run only after the tests succeed.
Breakdown:
- **`run_test` task**: invokes the Enterprise Edition `RunTest` plugin with an API token, namespace, and test ID; the result includes a `state` boolean.
- **`run_if_tests_pass` conditional**: checks that boolean before proceeding, ensuring downstream work executes only when all test cases pass.
## Step 4: Execute the tests
Run the unit tests from the Kestra UI or CLI to verify both assertions pass. A successful run confirms the monitor behaves correctly without sending Slack noise during testing.

## Next steps
- Expand the monitoring flow to cover multiple endpoints by looping over inputs or using a namespace file.
- Send alerts to PagerDuty, Teams, or email by swapping the Slack task for a different Notifications plugin.
- Wire the gated flow into your CI/CD pipeline so every deployment validates critical monitors before rollout.
---
# Configure Local MinIO Storage for Kestra
URL: https://kestra.io/docs/how-to-guides/minio
> Configure MinIO as a local object storage backend for Kestra using Docker and the MinIO client for development and testing.
Set up and verify a local [MinIO](https://min.io/) storage backend for Kestra using the `mc` CLI and Docker.
---
:::alert{type="warning"}
This guide is intended for **local development and testing only**.
MinIO is configured in gateway mode and exposed on `localhost`, without TLS or public access.
**Do not use this setup in production** without additional security measures (e.g., HTTPS, access controls, and network isolation).
:::
## Install and Configure `mc` (MinIO Client)
Download and install the MinIO Client (`mc`) tool using the following command:
```sh
curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o $HOME/minio-binaries/mc && \
chmod +x $HOME/minio-binaries/mc && \
export PATH=$PATH:$HOME/minio-binaries/
```
### Remove and Recreate Local Alias
Remove any existing local alias:
```sh
mc alias remove local
```
Recreate the alias with your MinIO access credentials:
```sh
mc alias set local http://localhost:9000 YOUR_ACCESS_KEY YOUR_SECRET_KEY
```
### Create a Local Bucket
Create the bucket where outputs will be stored:
```sh
mc mb local/your-bucket
```
## Start MinIO Server
Run the MinIO Docker container using the dedicated CI Compose file (e.g., from [kestra-io/storage-minio](https://github.com/kestra-io/storage-minio/)):
```sh
docker compose -f docker-compose-ci.yml up
```
## Configure Kestra for MinIO Storage
Update your `application-psql.yml` (or other relevant configuration file) under the `kestra:` section:
```yaml
kestra:
storage:
type: minio
minio:
endpoint: localhost
port: 9000
bucket: your-bucket
access-key: YOUR_ACCESS_KEY
secret-key: YOUR_SECRET_KEY
```
## Launch Kestra
Start Kestra as usual. Ensure the updated configuration file is correctly mounted or included.
## Test with a Flow that Produces Outputs
Here is a sample flow that generates output files and logs intermediate data:
```yaml
id: alligator_743987
namespace: company.team
tasks:
- id: pass_output
type: io.kestra.plugin.core.debug.Return
format: hello
- id: py_outputs
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/pydata:latest
outputFiles:
- myoutput.json
script: |
import json
from kestra import Kestra
my_kv_pair = {'mykey': 'from Kestra'}
Kestra.outputs(my_kv_pair)
with open('myoutput.json', 'w') as f:
json.dump(my_kv_pair, f)
- id: take_inputs
type: io.kestra.plugin.core.log.Log
message: |
data from previous tasks: {{ outputs.pass_output.value }} and {{ outputs.py_outputs.vars.mykey }}
- id: check_output_file
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- cat {{ outputs.py_outputs.outputFiles['myoutput.json'] }}
```
## Verify Output in MinIO Bucket
You can now validate that the output file is stored in the MinIO bucket:
```sh
mc cat local/your-bucket/main/company/team/alligator-743987/executions/23z9cJWEa23kNAxu6sm0CT/tasks/py-outputs/5kxYRM7UqUurvnpVNvHca7/1noPFEiCFGPf2hcqjVzywu-myoutput.json
```
Replace the following placeholders with your own values:
- the bucket name (here `your-bucket`)
- the path (namespace) (here `company/team`)
- the flow id (here `alligator-743987`)
- the execution id (here `23z9cJWEa23kNAxu6sm0CT`)
- the task id (here `py-outputs`)
- and finally the output file name (here `1noPFEiCFGPf2hcqjVzywu-myoutput.json`)
The result should look like:
```json
{"mykey": "from Kestra"}
```
You have successfully configured and validated MinIO as a local storage backend for Kestra.
---
# Configure Monitoring with Grafana and Prometheus
URL: https://kestra.io/docs/how-to-guides/monitoring
> Set up comprehensive monitoring for Kestra using Prometheus for metrics scraping and Grafana for visualization and dashboards.
Set up Prometheus and Grafana for monitoring Kestra.
Kestra exposes [Prometheus](https://prometheus.io/) metrics at port 8081 on the endpoint `/prometheus`. This endpoint can be used by any compatible monitoring system.
Use the [docker-compose.yml](https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml) file and start Kestra using the command:
```sh
docker compose up
```
Once Kestra is up and running, view the available metrics at `http://localhost:8081/prometheus` in your browser. The metrics should appear as below:

Create a few flows and execute them to generate some metrics for visualization. You can also add triggers to the flows to check the metrics corresponding to executions happening on a regular basis.
## Setting up Prometheus
With metrics available from Kestra, set up a Prometheus server to scrape them and store them in a time-series DB.
Create a `prometheus.yml` file for scraping the metrics:
```yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "prometheus"
metrics_path: /prometheus
static_configs:
- targets: [":8081"]
```
Be sure to put the appropriate `` in the last line, e.g. `localhost:8081` or `host.docker.internal:8081`.
:::alert{type="info"}
If you're running everything in Docker on the same machine, you will need to change your host address to `host.docker.internal` rather than localhost.
:::
We can start the Prometheus server using the following docker command in the same directory as `prometheus.yml`:
```sh
docker run -d -p 9090:9090 -v ./prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
```
Note, in this last command you may have to add `--add-host=host.docker.internal:host-gateway` to ensure your Prometheus endpoint is shown as `UP` (you can check it in the [targets](http://localhost:9090/targets)).
You can now go to `http://localhost:9090/graph` and try out visualizing some metrics using the PromQL. Here is one of the graphs for `kestra_executor_execution_started_count_total` metric:

## Scraping Kestra on Kubernetes
The setup above uses `static_configs` with a single hardcoded target, which suits a single Kestra instance running locally. It does not translate to [Kubernetes](../../02.installation/03.kubernetes/index.md): pod IP addresses change whenever a pod restarts or is rescheduled, and a distributed deployment runs many pods to scrape rather than one.
Use Kubernetes service discovery instead. Prometheus resolves its targets from the Kubernetes API and updates them as pods come and go, so pod churn is handled for you.
You do not need to create a Service to scrape Kestra. Prometheus discovers and scrapes the pods directly.
:::alert{type="warning"}
Avoid pointing Prometheus at a Service `ClusterIP`. A `ClusterIP` load-balances, so each scrape reaches one arbitrary pod while reporting under a single target identity. Counters then appear to jump backwards between scrapes, and no series can be attributed to a specific pod.
:::
### Enable the metrics endpoint
The Helm chart renders `configurations.application` into a single ConfigMap that every component mounts, so enabling the endpoint once covers the webserver, executor, scheduler, indexer, worker, and every worker group:
```yaml
configurations:
application:
endpoints:
metrics:
enabled: true
prometheus:
enabled: true
sensitive: false
micronaut:
metrics:
enabled: true
export:
prometheus:
enabled: true
step: PT1M
```
Every Kestra pod already declares port 8081 as a named container port called `management`, so no additional port configuration is required.
:::alert{type="info"}
Metrics are served on port 8081, not the 8080 UI port. Requesting `/prometheus` on port 8080 returns a redirect.
:::
### Discover pods with annotations
If you run Prometheus yourself with a `kubernetes_sd_configs` scrape job, annotate the pods so your existing annotation-based job picks them up. Setting this under `common` applies it to every component, including all worker groups:
```yaml
common:
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8081"
prometheus.io/path: "/prometheus"
```
### Discover pods with a PodMonitor
If you use the Prometheus Operator, a single `PodMonitor` covers every Kestra component. Add it through the chart's `extraManifests`:
```yaml
extraManifests:
- |
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: {{ include "kestra.fullname" . }}
labels:
{{- include "kestra.labels" . | nindent 4 }}
spec:
selector:
matchLabels:
{{- include "kestra.selectorLabels" . | nindent 6 }}
podMetricsEndpoints:
- port: management
path: /prometheus
interval: 30s
podTargetLabels:
- app.kubernetes.io/component
- kestra.io/worker-group
```
`podMetricsEndpoints.port` refers to the named `management` container port, so it stays correct regardless of how the port is configured. `podTargetLabels` promotes the listed pod labels onto the scraped metrics, which is what lets you filter by component and by worker group in PromQL.
:::alert{type="info"}
Entries in `extraManifests` are rendered as Helm templates. When writing one as a YAML block scalar, the `nindent` value must match the indentation of the surrounding literal keys after YAML strips the block indent. A mismatch fails with a `did not find expected key` error.
:::
### Distinguishing worker groups
Kestra's metrics payload does not include a worker group tag, so worker group identity has to come from Kubernetes pod labels. The Helm chart does not add a per-group label automatically, so declare one for the worker deployment and for each worker group:
```yaml
deployments:
worker:
enabled: true
podLabels:
kestra.io/worker-group: default
workerGroups:
gpu:
enabled: true
podLabels:
kestra.io/worker-group: gpu
```
Together with `podTargetLabels` in the `PodMonitor` above, this makes each worker group separately queryable. Without it, every worker pod carries the same labels and its metrics cannot be attributed to a group.
:::alert{type="info"}
`podLabels` and `podAnnotations` are supported per component and per worker group, not only under `common`.
:::
## Setting up Grafana
Let us now move on to setting up Grafana. You start by installing Grafana using docker via the following command:
```sh
docker run -d -p 3000:3000 --name=grafana grafana/grafana-enterprise
```
You can open the Grafana server at `http://localhost:3000`. The default credentials are `admin` as both username and password. Once logged into Grafana, click on the hamburger menu on the top left and go to **Connections -> Data Sources**.

### Add Data Source
Click on **Add new Data Source** button present on the top right, and select **Prometheus** from the time series databases list. In the **Prometheus server URL** text box, put in the following URL: `http://:9090`. All the other configuration can be left as default. You can click on **Save and Test** button at the bottom, and confirm that the connection to Prometheus database is successful.
## Add Dashboard
We are now all set to create the Grafana dashboard. For this, click on the **+** button on the top of the page to add a **New Dashboard** to Grafana. Save the dashboard with an appropriate name. Then, click on **Add visualization**, and select **prometheus** as the data source.
We will create a Gauge that shows the number of tasks that are presently running. For this, select **Gauge** as the Visualization in the top right corner. In the PromQL metrics explorer text box, you can write `sum(kestra_worker_running_count)`. Click on **Run queries** button to ensure the Gauge shows up the number.
Head back to Kestra and create a number of tasks that will execute for a long time. The example below will sleep for 60 seconds:
```yaml
id: sleep
namespace: company.team
tasks:
- id: sleep_task
type: io.kestra.plugin.scripts.shell.Commands
commands:
- sleep 60
```
Now we have some long-running tasks in progress, we can check that the Gauge correctly reflects the count. You can now put an appropriate title in the Panel options that says **Tasks running**.
This is how your Grafana should look like:

Click on **Save** and **Apply** to add this gauge to the dashboard.
Similarly, you can now keep on adding more graphs to your dashboard. Here is one of the example dashboards for Kestra metrics.

The [Alerting & Monitoring](../../10.administrator-guide/03.monitoring/index.md#grafana-and-kibana) section includes an import-ready Grafana dashboard definition.
---
# MultipleCondition Listener in Kestra: How It Works
URL: https://kestra.io/docs/how-to-guides/multiplecondition-listener
> Configure MultipleCondition triggers in Kestra to start flows only when multiple conditions are met for precise event-driven orchestration.
How to set up a flow that only triggers when multiple upstream flows have all succeeded.
In this guide, we’ll create a flow that only executes if two other flows, `multiplecondition_flow_a` and `multiplecondition_flow_b`, have each completed successfully within the last 24 hours. This pattern uses the `dependsOn` property on the Flow trigger.
## When to use this pattern
Use multiple upstream dependencies when a downstream process should only run after several independent upstream flows all succeed. For example, if you have separate ingestion flows for different data sources and want to run a transformation only after all sources have completed, `dependsOn` with a time window is the right tool.
## How it works
1. Both `multiplecondition_flow_a` and `multiplecondition_flow_b` must complete successfully.
2. Both must complete within the same 24-hour window (`window.every: P1D`).
3. Once both conditions are satisfied, the listener flow triggers.
4. The window resets each day, so both flows must succeed again within the next window to re-trigger the listener.
## First Flow: `multiplecondition_flow_a`
This is the first flow that the listener will check for success.
```yaml
id: multiplecondition_flow_a
namespace: company.team
description: |
This flow will start `multiplecondition_listener` if `MultipleCondition` is validated
tasks:
- id: only
type: io.kestra.plugin.core.log.Log
message: "from parents: {{ execution.id }}"
```
This flow is a simple one that returns the execution ID as output. The listener checks whether this flow has executed successfully within the past 24 hours.
## Second Flow: multiplecondition_flow_b
This is the second flow that the listener will check for success.
```yaml
id: multiplecondition_flow_b
namespace: company.team
description: |
This flow will start `multiplecondition_listener` if `MultipleCondition` is validated
tasks:
- id: only
type: io.kestra.plugin.core.log.Log
message: "from parents: {{ execution.id }}"
```
Just like `multiplecondition_flow_a`, this flow also returns its execution ID. The listener will wait for both this and the first flow to succeed before activating the final flow.
## Final Flow with Trigger: `multiplecondition_listener`
The final flow is where we define the trigger that listens to both `multiplecondition_flow_a` and `multiplecondition_flow_b`.
```yaml
id: multiplecondition_listener
namespace: company.team
description: |
This flow starts only if `multiplecondition_flow_a` and `multiplecondition_flow_b` both succeed within the same 24-hour window.
tasks:
- id: only_listener
type: io.kestra.plugin.core.log.Log
message: "children"
triggers:
- id: multiple_listen_flow
type: io.kestra.plugin.core.trigger.Flow
dependsOn:
- flowId: multiplecondition_flow_a
namespace: company.team
states: [SUCCESS]
- flowId: multiplecondition_flow_b
namespace: company.team
states: [SUCCESS]
window:
every: P1D
```
## Explanation of the flow
1. **Tasks** — `only_listener` outputs a static value when the trigger fires. Replace this with whatever downstream logic you need.
2. **`dependsOn`** — declares two upstream flow dependencies. Both entries must be satisfied before the trigger fires. `states: [SUCCESS]` means only successful executions count.
3. **`window.every: P1D`** — defines a 24-hour evaluation window. Kestra accumulates upstream executions within this window and fires the trigger once all `dependsOn` entries are satisfied within the same window period.
## Expected Output
When both multiplecondition_flow_a and multiplecondition_flow_b succeed within 24 hours, the listener will trigger multiplecondition_listener, and you will see output similar to this:
`only listener > children`
## Common Pitfalls and Troubleshooting
1. **Conditions Not Met**:
If the flow doesn't trigger, ensure both `multiplecondition_flow_a` and `multiplecondition_flow_b` have completed successfully within the time window.
2. **Incorrect Output Reference**:
Verify the flow IDs and namespaces to ensure the trigger is referencing the correct flows.
## Conclusion
This guide demonstrated how to use `dependsOn` with a time window to trigger a flow only when multiple upstream flows all succeed within the same period. Use this pattern whenever a downstream process must wait on several independent upstream flows before running.
---
# Namespace Variables vs. KV Store in Kestra
URL: https://kestra.io/docs/how-to-guides/namespace-variables-vs-kvstore
> Understand the differences between Namespace Variables and the KV Store in Kestra to choose the right storage for your data.
When to store key-value pairs as namespace-level Variables vs. KV store
When navigating to a namespace in the Kestra UI, you can see two tabs: Variables and KV Store. Both allow you to store key-value pairs, but there are some significant differences in how those are handled and stored, and when you should use one over the other.
## Variables: use when you need to inherit values from parent namespaces
Variables are typically intended for slowly changing values. Think of the database hostname, the bucket name in a cloud storage service, or the name of a shared queue in a message broker. These values are typically set once and then used across multiple flows and tasks.
To add those, navigate to the Variables tab in the namespace and paste your key-value pairs as shown below:
```yaml
POSTGRES_HOSTNAME: my-postgres-prod-hostname
DATALAKE_S3_BUCKET_NAME: my-datalake-s3-bucket-name
RABBITMQ_QUEUE_NAME: my-rabbitmq-queue-name
GCP_PROJECT_ID: my-gcp-project-id
GITHUB_REPO_URL: https://github.com/kestra-io/kestra
```
The additional benefit of using Variables is that they can be grouped to simplify some configurations. For example, you can group all database-related connection variables under a `postgres` prefix and access them using e.g. `{{ namespace.postgres.hostname }}` in your flows and tasks.
```yaml
postgres:
hostname: my-postgres-prod-hostname
port: 5432
username: my-postgres-prod-username
dataLake:
s3BucketName: my-datalake-s3-bucket-name
region: us-east-1
```
You may notice that the Variables can be defined using the uppercased `SNAKE_CASE` convention, as well as `camelCase` or any other convention you prefer.
Storing those values as Variables in a namespace allows you to:
1. Set them once by a DevOps engineer or a system administrator.
2. Centrally govern them in a single place (e.g. to update a database host or port, bucket names, regions, etc.).
3. Inherit them from parent namespace (e.g. `company` namespace) to all child namespaces (e.g. `company.myteam`, `company.myteam.myproject`).
4. Group them to simplify configurations of database connections, cloud storage services, message brokers, etc.
This means that if you have a variable `POSTGRES_HOSTNAME` set in a parent namespace `company`, you can use `{{namespace.POSTGRES_HOSTNAME}}` in a child namespace `company.myteam` and `company.myteam.myproject` (and all other infinitely nested namespaces) without having to worry where in the namespace hierarchy that value is managed.
## KV Store: use when you need to store ephemeral or dynamic values
Trying to use KV Store for the above use case would also work, but you would always need to remember to include the pointer to the namespace under which that key-value pair is stored (unless using the same one as the flow). This is because KV Store is not inherited from parent to child namespaces. Example:
```yaml
{{ kv('POSTGRES_HOSTNAME', 'company') }}
```
The KV Store is more suited for storing ephemeral or dynamic values. Think of the last scraped timestamp, the offset of a Kafka consumer group, or the most recently processed file name. These values are typically set and updated by the workflow itself. Using KV Store for those use cases is better than Variables because KV pairs can be set and updated at runtime, while Variables are typically set once, centrally governed by Kestra Admins, and inherited from parent namespaces to reuse centrally governed configuration across multiple flows and tasks.
## Recap
- **Variables**: use for slowly changing configuration values that are set once, updated fairly infrequently and inherited from parent namespaces by indefinitely nested child namespaces (e.g. `company`, `company.myteam`, `company.myteam.myproject`).
- **KV Store**: use for ephemeral or dynamic values that are set and/or updated at runtime.
Here are some examples to consolidate your understanding:
- **Variables**: `POSTGRES_HOSTNAME`, `DATALAKE_S3_BUCKET_NAME`, `RABBITMQ_QUEUE_NAME`, `GCP_PROJECT_ID`, `GITHUB_REPO_URL`
- **KV Store**: `last_scraped_timestamp`, `kafka_consumer_group_offset`, `last_processed_file_name`.
---
# Connect a Neon Database to Kestra
URL: https://kestra.io/docs/how-to-guides/neon
> Connect your serverless Neon PostgreSQL database to Kestra workflows to query, ingest, and manage data seamlessly.
Connect your Neon serverless database to your workflows using the PostgreSQL plugin.
Neon is an open-source database company whose mission is to take everything that developers love about Postgres — reliability, performance, extensibility — and deliver it as a serverless product.
Before you begin, ensure you have a [Neon account](https://neon.tech/home) set up and a [Kestra installation](../../02.installation/index.mdx) running.
## Setting up a Database in Neon
Once you've logged into Neon, you'll need to set up a project where you'll give it a name, select your desired PostgreSQL version, and select your cloud provider and region.

Once your project is created, you'll arrive at the Project Dashboard page. From here, you can connect to your database, import data, get sample data, view database content, and much more.

## Connecting Neon to Kestra
To have Kestra supply the data, connect to your database. Leave the Branch, Compute, Database, and Role as their defaults, or adjust as needed. Click on the **Connection string** dropdown list and select Java. This is the connection string used in Kestra to connect to the Neon database. Make note of the password and save it for later steps.

With a database set up in Neon, create a table for the incoming data. Click on **Tables** on the left sidebar.

Next, click on the '+' icon to add a table, name it, and create it. You can leave just the default `id` column or add in the columns of your data set now. Kestra will alter the table, so leave it empty for now.

With the setup in Neon done, we can go Kestra to set up our connection. While there's no official Neon plugin, we can connect using the [PostgreSQL plugin](/plugins/plugin-jdbc-postgres), which supports a number of tasks such as `Query`, `CopyIn`, and `CopyOut`.
To connect, copy the URL provided from before. Store the password as a [secret](../../06.concepts/04.secret/index.md) and reference it in the URL with `{{ secret('NEON_PASSWORD') }}`. Add the `url` property directly to each PostgreSQL task:
```yaml
url: "jdbc:postgresql://ep-gentle-tree-a25pyhxb-pooler.eu-central-1.aws.neon.tech/neondb?user=neondb_owner&password={{ secret('NEON_PASSWORD') }}&sslmode=require"
```
:::alert{type="info"}
You can also split the connection string into separate `url`, `username`, and `password` properties:
```yaml
url: "jdbc:postgresql://ep-gentle-tree-a25pyhxb-pooler.eu-central-1.aws.neon.tech/neondb"
username: "neondb_owner"
password: "{{ secret('NEON_PASSWORD') }}"
```
In Enterprise Edition, you can centralize connection properties across flows using a [Policy](../../07.enterprise/02.governance/policies/index.md) with an `Add` rule targeting `io.kestra.plugin.jdbc.postgresql`.
:::
## Copying a CSV File into Neon in a Flow
Using this [example CSV](https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv), we can copy the data into our table directly from Kestra. You can either set up the columns directly in Neon in the earlier steps or add a task in Kestra to add them automatically like this:
```yaml
id: neon_db_add_columns
namespace: company.team
tasks:
- id: create_columns
type: io.kestra.plugin.jdbc.postgresql.Queries
url: "jdbc:postgresql://ep-gentle-tree-a25pyhxb-pooler.eu-central-1.aws.neon.tech/neondb?user=neondb_owner&password={{ secret('NEON_PASSWORD') }}&sslmode=require"
sql: |
ALTER TABLE kestra_example
ADD COLUMN order_id int,
ADD COLUMN customer_name text,
ADD COLUMN customer_email text,
ADD COLUMN product_id int,
ADD COLUMN price double precision,
ADD COLUMN quantity int,
ADD COLUMN total double precision;
```
Once your columns are configured, use the [CopyIn](/plugins/plugin-jdbc-postgres/io.kestra.plugin.jdbc.postgresql.copyin) task combined with the [HTTP Download](/plugins/core/http/io.kestra.plugin.core.http.download) task to download the CSV file and copy it directly into the table.
```yaml
id: neon_db_copyin
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: copy_in
type: io.kestra.plugin.jdbc.postgresql.CopyIn
url: "jdbc:postgresql://ep-gentle-tree-a25pyhxb-pooler.eu-central-1.aws.neon.tech/neondb?user=neondb_owner&password={{ secret('NEON_PASSWORD') }}&sslmode=require"
table: "kestra_example"
from: "{{ outputs.download.uri }}"
header: true
columns: [order_id,customer_name,customer_email,product_id,price,quantity,total]
delimiter: ","
```
Once this flow completes, we can view the contents of our database in Neon:

---
# Integrate Notion Webhooks with Kestra
URL: https://kestra.io/docs/how-to-guides/notion-webhook
> Automate Notion database updates and send Slack notifications by triggering Kestra flows via Notion webhooks.
Use Notion webhooks to trigger Kestra flows when pages or databases are updated in your Notion workspace.
This guide shows you how to create a workflow that responds to Notion database changes, retrieves page details, and sends notifications to Slack when new tasks are assigned.
## Prerequisites
Before you begin, you need:
- A Notion workspace with a database
- A [Notion integration](https://www.notion.so/my-integrations) with access to your database
- A Slack workspace with webhook capabilities ([Slack Webhook Documentation](https://api.slack.com/messaging/webhooks))
- Access to your Notion API token and Slack webhook URL
## Create a Notion integration
1. Go to [Notion's My Integrations page](https://www.notion.so/my-integrations)
2. Click **"New integration"**
3. Give your integration a name and select your workspace
4. Copy the **Internal Integration Token** - you'll need this for the `NOTION_API_KEY` secret
## Share your database with the integration
1. Open your Notion database
2. Click the **"..."** menu in the top right
3. Select **"Add connections"**
4. Find and select your integration
5. Click **"Confirm"** to grant access
## Set up secrets in Kestra
Store your sensitive credentials as [secrets](../../06.concepts/04.secret/index.md) or [key-value](../../06.concepts/05.kv-store/index.md) pairs:
1. Navigate to your namespace in the Kestra UI
2. Go to the **Secrets** tab (Alternatively go to the **KV Store** tab and do the same)
3. Create these secrets:
- `NOTION_API_KEY`: Your Notion integration token
- `SLACK_WEBHOOK_URL`: Your Slack incoming webhook URL
## Create the webhook flow
Create a flow that listens for Notion webhook events and processes them:
```yaml
id: notion-webhook
namespace: company.team
tasks:
- id: get_notion_page_details
type: io.kestra.plugin.notion.page.Read
apiToken: "{{ secret('NOTION_API_KEY') }}"
pageId: "{{ trigger.body.entity.id }}"
- id: send_slack_alert
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
url: "{{ secret('SLACK_WEBHOOK_URL') }}"
messageText: "New task titled {{ outputs.get_notion_page_details | jq('.properties.Button.title[0].text.content') | first }} assigned to {{ outputs.get_notion_page_details | jq('.properties.Assignee.multi_select[0].name') | first }} on the Product team Notion board! Link: {{ outputs.get_notion_page_details.url }}"
triggers:
- id: notion_new_task_webhook
type: io.kestra.plugin.core.trigger.Webhook
key: my-notion-product-alert-key # Replace with a secure key
```
:::alert{type="warning"}
Replace `my-notion-product-alert-key` with a secure, randomly generated key. Consider storing this as a [secret](../../06.concepts/04.secret/index.md) or [key-value pair](../../06.concepts/05.kv-store/index.md) for better security.
:::
## Configure Notion webhooks
Set up webhooks directly in your Notion integration:
1. Go to your [Notion integration settings](https://www.notion.so/my-integrations)
2. Select your integration
3. Navigate to the **"Webhooks"** section
4. Click **"Add webhook"**
5. Enter your Kestra webhook URL (see format below)
6. Select the events you want to listen for:
- `page.property_values.updated` - When page properties change
- `page.created` - When new pages are created
- `database.created` - When new databases are created
7. Click **"Create"** to save the webhook

For more details, see the [Notion Webhooks API documentation](https://developers.notion.com/reference/webhooks).
## Webhook URL format
Your Kestra webhook URL follows this pattern:
```plaintext
http://your-kestra-host:8080/api/v1/main/executions/webhook/{namespace}/{flow_id}/{key}
```
For this example:
- **Namespace**: `company.team`
- **Flow ID**: `notion-webhook`
- **Key**: `my-notion-product-alert-key`
Complete URL:
```plaintext
http://your-kestra-host:8080/api/v1/main/executions/webhook/company.team/notion-webhook/my-notion-product-alert-key
```
You can copy your webhook URL directly from the Kestra UI from the **Triggers** tab and paste it in Notion:

## Testing the integration
Test your webhook flow manually:
```bash
curl -X POST \
http://your-kestra-host:8080/api/v1/main/executions/webhook/company.team/notion-webhook/my-notion-product-alert-key \
-H "Content-Type: application/json" \
-d '{"entity": {"id": "your-notion-page-id"}}'
```
Replace `your-notion-page-id` with an actual page ID from your Notion database.
## Understanding the flow
The flow performs these steps:
1. **Webhook trigger**: Listens for incoming webhook requests from Notion on the specified endpoint
2. **Get page details**: Uses the [Notion plugin](/plugins/plugin-notion) to fetch complete page information from Notion
3. **Send notification**: Extracts the task title and assignee information, then sends a formatted message to Slack
## Customizing the flow
### Different Notion properties
Modify the Slack message to use different Notion properties. Common property types include:
```yaml
## For title properties
title: "{{ outputs.get_notion_page_details | jq('.properties.Title.title[0].text.content') | first }}"
## For select properties
status: "{{ outputs.get_notion_page_details | jq('.properties.Status.select.name') | first }}"
## For date properties
due_date: "{{ outputs.get_notion_page_details | jq('.properties.DueDate.date.start') | first }}"
## For people properties
assignee: "{{ outputs.get_notion_page_details | jq('.properties.Assignee.people[0].name') | first }}"
```
### Adding conditional logic
Add conditions to process only specific types of changes:
```yaml
tasks:
- id: check_status
type: io.kestra.plugin.core.flow.If
condition: "{{ outputs.get_notion_page_details | jq('.properties.Status.select.name') | first == 'In Progress' }}"
then:
- id: send_slack_alert
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
url: "{{ secret('SLACK_WEBHOOK_URL') }}"
messageText: "Task moved to In Progress: {{ outputs.get_notion_page_details | jq('.properties.Title.title[0].text.content') | first }}"
```
### Multiple notification channels
Send notifications to different channels based on the assignee or project:
```yaml
tasks:
- id: send_to_team_channel
type: io.kestra.plugin.core.flow.If
condition: "{{ outputs.get_notion_page_details | jq('.properties.Project.select.name') | first == 'Product' }}"
then:
- id: product_team_notification
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
url: "{{ secret('PRODUCT_SLACK_WEBHOOK_URL') }}"
messageText: "New product task assigned!"
else:
- id: general_notification
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
url: "{{ secret('GENERAL_SLACK_WEBHOOK_URL') }}"
messageText: "New task assigned!"
```
:::alert{type="info"}
Keep in mind that the above examples are additional tasks to add to the flow and not standalone flows. You need to add `id` and `namespace` properties to execute them standalone.
:::
## Security considerations
- Use strong, randomly generated webhook keys
- Store all sensitive tokens as [secrets](../../06.concepts/04.secret/index.md) or [key-value pairs](../../06.concepts/05.kv-store/index.md)
- Consider implementing request validation in your webhook handler
- Regularly rotate your API tokens and webhook URLs
## Related resources
- [Webhook triggers](../../05.workflow-components/07.triggers/03.webhook-trigger/index.md)
- [Notion plugin documentation](https://kestra.io/plugins/plugin-notion)
- [Slack notifications](../../15.how-to-guides/slack-webhook/index.md)
- [Secrets management](../../06.concepts/04.secret/index.md)
- [Expression language guide](../../06.concepts/06.pebble/index.md)
---
# Handle Null and Undefined Values in Kestra
URL: https://kestra.io/docs/how-to-guides/null-values
> Handle null and missing values in Kestra flows. Check for null inputs, set defaults, and use conditional logic to manage undefined task outputs.
How to use the null coalescing operator to handle null and undefined values.
The null coalescing operator is a binary operator that returns its left-hand value if it's not null; otherwise, it returns its right-hand value. You can think of it as a way to provide a default value when the left-hand value is null.
```yaml
"{{ null ?? now() | date('yyyy-MM-dd') }}"
```
In this example, since the left-hand side of the `??` operator is `null`, the right-hand side will be returned. The `now()` function will be called, and the result will be formatted as a date string in the `yyyy-MM-dd` format.
## Processing date values
Imagine that you have a flow that processes data between two dates. You want to provide default values for the start and end dates if they're not provided as inputs. You can use the null coalescing operator to set default values for the start and end dates. In this example, the start date is set to one month ago, and the end date is set to today, effectively processing data for the last month by default.
```yaml
id: process_data_between_dates
namespace: company.team
inputs:
- id: start_date
type: DATE
required: false
description: Start date to fetch data from
- id: end_date
type: DATE
required: false
description: End date to fetch data from
variables:
start_date: "{{ inputs.start_date ?? now() | dateAdd(-1, 'MONTHS') | date('yyyy-MM-dd') }}"
end_date: "{{ inputs.end_date ?? now() | date('yyyy-MM-dd') }}"
tasks:
- id: process_data_between_dates
type: io.kestra.plugin.core.log.Log
message: processing data from {{ render(vars.start_date) }} to {{ render(vars.end_date) }}
```
Use the `render` function to recursively render variables containing Pebble expressions.
## Providing default values for optional and undefined inputs
The null-coalescing operator `??` will return the right-hand value if the left-hand value is null or undefined (e.g. an `input` or a `variable` that have not been defined). This behavior is useful when you want to provide default values for optional inputs and for dynamic properties that may not be defined.
If you want to return the right-hand side only if the left-hand side is undefined, you can use the `???` operator instead of `??`.
The example below shows how to use both `??` and `???` operators to set defaults for optional or undefined values.
```yaml
id: provide_default_values
namespace: company.team
inputs:
- id: optional_input
type: STRING
required: false
description: An optional input
tasks:
- id: coalesce_optional_input
type: io.kestra.plugin.core.log.Log
message: |
Expression: inputs.optional_input ?? 'mydefault'
Left-hand value: null
Right-hand value: 'mydefault'
Operator used: '??'
This expression "{{ inputs.optional_input ?? 'mydefault' }}" will return 'mydefault'
because the coalesce-operator '??' returns the right-hand value
if the left-hand value is null or undefined.
Only if you provide a value at runtime, that value will be used instead of 'mydefault'.
- id: coalesce_undefined_input
type: io.kestra.plugin.core.log.Log
message: |
Expression: inputs.undefined_input ?? 'mydefault'
Left-hand value: undefined
Right-hand value: 'mydefault'
Operator used: '??'
The expression "{{ inputs.undefined_input ?? 'mydefault' }}" will return 'mydefault'
because the coalesce-operator '??' returns the right-hand value
if the left-hand value is null or undefined.
- id: coalesce_only_undefined_input_1
type: io.kestra.plugin.core.log.Log
message: |
Expression: inputs.undefined_input ??? 'mydefault'
Left-hand value: undefined
Right-hand value: 'mydefault'
Operator used: '???'
The expression "{{ inputs.undefined_input ??? 'mydefault' }}" will return 'mydefault'
because he undefined-coalesce-operator '???' returns the right-hand value
if the left-hand value is undefined.
- id: coalesce_only_undefined_input_2
type: io.kestra.plugin.core.log.Log
message: |
Expression: inputs.optional_input ??? 'mydefault'
Left-hand value: null
Right-hand value: 'mydefault'
Operator used: '???'
The expression "{{ inputs.optional_input ??? 'mydefault' }}" will return "" i.e. no value aka null value
because optional_input is defined and the undefined-coalesce-operator '???' only returns the right-hand value
if the left-hand value is undefined.
- id: both_operators_combined
type: io.kestra.plugin.core.log.Log
message: |
Expression: (inputs.optional_input ??? 'mydefault') ?? 'other_default'
Left-hand value: null
Right-hand value: 'other_default'
Operator used: '??'
The expression "{{ (inputs.optional_input ??? 'mydefault') ?? 'other_default' }}" will return 'other_default'
because the first expression using the undefined-coalesce-operator '???' will return null,
and the coalesce-operator '??' will return the default value 'other_default'.
```
## Processing Trigger values
When using a Trigger, you can use the `{{ trigger }}` expression in your flow. However, this expression is undefined if you execute your flow manually.
Here's an example of a Webhook trigger that might receive a body of data. You can use the null coalescing operator to handle when the body of data is different to what's expected:
```yaml
id: webhook_example
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: "{{ trigger.body.dataField ?? 'data' }}"
triggers:
- id: webhook
type: io.kestra.plugin.core.trigger.Webhook
key: abcdefg
```
You can also use this for Schedule triggers. Here's an example that uses the date when the Schedule trigger starts an execution but in combination with the null coalescing operator to use an input value if an execution is started manually.
```yaml
id: scheduling
namespace: company.team
inputs:
- id: country
type: STRING
defaults: US
- id: date
type: DATETIME
required: false
defaults: 2023-12-24T14:00:00.000Z
tasks:
- id: check_if_business_date
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
commands:
- python schedule.py "{{ trigger.date ?? inputs.date }}" {{ inputs.country }}
beforeCommands:
- pip install workalendar
taskRunner:
type: io.kestra.plugin.core.runner.Process
- id: log
type: io.kestra.plugin.core.log.Log
message: business day - continuing the flow...
triggers:
- id: schedule
type: io.kestra.plugin.core.trigger.Schedule
cron: 0 14 25 12 *
```
---
# Deploy to OpenShift with Kestra
URL: https://kestra.io/docs/how-to-guides/openshift
> Use the Kubernetes plugin's kubectl.Apply task to deploy a containerized application to a Red Hat OpenShift cluster, exposing it externally via a TLS-terminated Route.
The [`kubectl.Apply`](/plugins/plugin-kubernetes/kubernetes-api-kubectl/io.kestra.plugin.kubernetes.kubectl.apply) task in the Kubernetes plugin supports OpenShift's OAuth token authentication and OpenShift-specific resource types, so you can deploy to OpenShift the same way you would any standard cluster.
## Prerequisites
- A running OpenShift cluster
- The `oc` CLI installed and authenticated (`oc login`)
- The [Kubernetes plugin](/plugins/plugin-kubernetes) version 1.9.4 or later installed in your Kestra instance:
```bash
kestra plugins install io.kestra.plugin:plugin-kubernetes:1.9.4
```
- Two Kestra secrets:
| Secret | How to obtain |
|---|---|
| `OPENSHIFT_API_URL` | `oc whoami --show-server` — format: `https://api.:6443` |
| `OPENSHIFT_TOKEN` | `oc whoami -t` after `oc login`, or a long-lived token from a `ServiceAccount` secret for CI use |
## The flow
The flow below applies three resources to OpenShift in a single `kubectl.Apply` task using multi-document YAML (`---` separators): a `Deployment`, a `Service`, and a `Route`.
The image reference is a flow input, so the same flow can deploy any image without editing the manifest.
```yaml
id: openshift_deploy_python_app
namespace: company.devops
inputs:
- id: image
type: STRING
description: "Full image reference to deploy, e.g. my-registry.example.com/company/python-app:1.0.0"
tasks:
- id: deploy_to_openshift
type: io.kestra.plugin.kubernetes.kubectl.Apply
connection:
masterUrl: "{{ secret('OPENSHIFT_API_URL') }}"
oauthToken: "{{ secret('OPENSHIFT_TOKEN') }}"
namespace: my-project
spec: |-
apiVersion: apps/v1
kind: Deployment
metadata:
name: python-app
labels:
app: python-app
spec:
selector:
matchLabels:
app: python-app
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: python-app
spec:
containers:
- name: python-app
image: "{{ inputs.image }}"
imagePullPolicy: Always
ports:
- name: http
containerPort: 8080
protocol: TCP
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "256Mi"
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 30
periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
name: python-app-service
labels:
app: python-app
spec:
selector:
app: python-app
ports:
- name: http
port: 8080
targetPort: http
type: ClusterIP
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: python-app-route
labels:
app: python-app
spec:
to:
kind: Service
name: python-app-service
weight: 100
port:
targetPort: http
tls:
termination: edge
insecureEdgeTerminationPolicy: Redirect
```
## What to adapt before use
- Replace the `image` input default with the full registry path for your image.
- Set `namespace: my-project` to the target OpenShift project name.
- Adjust the readiness and liveness probe `path` if your application does not expose a `/health` endpoint on port 8080.
- Adjust resource requests and limits to match your workload.
## Design notes
### Route instead of Ingress
OpenShift uses `route.openshift.io/v1 Route` for external access rather than a standard Kubernetes `Ingress`. The Route references the `Service` by name and port. Using the named port `http` consistently across the `Deployment`, `Service`, and `Route` means the port number only needs to change in one place.
### TLS edge termination
`termination: edge` terminates TLS at the OpenShift router, so traffic inside the cluster is plain HTTP. `insecureEdgeTerminationPolicy: Redirect` ensures plain HTTP requests to the Route are redirected to HTTPS — without it, edge-terminated Routes accept both.
### `imagePullPolicy: Always`
When the image tag is dynamic (a git SHA or build number passed from an upstream task), `IfNotPresent` can silently run a stale cached layer if the tag has been reused. `Always` ensures the declared image is what actually runs.
### `spec.replicas` and the Developer Sandbox
The OpenShift Developer Sandbox runs a `member-operator` that takes server-side apply ownership of `spec.replicas` on all Deployments for quota enforcement. Declaring `replicas` in your manifest causes a 409 field manager conflict. Omit it entirely — the Sandbox defaults to 1 replica.
On a production OpenShift cluster without `member-operator`, you can safely declare `replicas`.
### ImageStream
This example pulls the image directly from the registry rather than through an OpenShift `ImageStream`. If you need OpenShift-native image tracking and promotion, create an `ImageStream` with `spec.tags` importing the external image, set `lookupPolicy.local: true`, and reference the short name in the `Deployment` instead of the full registry path.
---
# Parallel vs. Sequential Tasks in Kestra
URL: https://kestra.io/docs/how-to-guides/parallel-vs-sequential
> Choose between parallel and sequential task execution in Kestra. Understand trade-offs and dependency management to design efficient workflow patterns.
When to use parallel tasks and when to use sequential tasks in Kestra.
## Parallel Tasks
The following flow has 6 tasks wrapped in the `Parallel` task. Since the `concurrent` property is set to 3, Kestra will run 3 tasks in parallel. As soon as any of the three tasks completes, the next task will start.
The addition of the Parallel task with the `concurrent` property set to 3 ensures that Kestra will run up to 3 tasks concurrently. The `last` task will run after all the tasks in the `Parallel` task group have completed.
```yaml
id: parallel
namespace: company.team
tasks:
- id: parent
type: io.kestra.plugin.core.flow.Parallel
concurrent: 3
tasks:
- id: t1
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- 'echo "running {{task.id}}"'
- 'sleep 1'
- id: t2
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- 'echo "running {{task.id}}"'
- 'sleep 1'
- id: t3
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- 'echo "running {{task.id}}"'
- 'sleep 1'
- id: t4
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- 'echo "running {{task.id}}"'
- 'sleep 1'
- id: t5
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- 'echo "running {{task.id}}"'
- 'sleep 1'
- id: t6
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- 'echo "running {{task.id}}"'
- 'sleep 1'
- id: last
type: io.kestra.plugin.core.log.Log
message: "{{task.id}} > {{taskrun.startDate}}"
```
## Sequential Tasks
This flow will start two sequential task groups in parallel. The addition of the `Sequential` task ensures that the tasks within each group will run one after the other. The `last` task will run after all the tasks in the `Sequential` task group have completed
```yaml
id: sequential
namespace: company.team
description: |
This flow will start the 2 sequential tasks in parallel and those will launch tasks one after the other.
tasks:
- id: parent
type: io.kestra.plugin.core.flow.Parallel
tasks:
- id: seq1
type: io.kestra.plugin.core.flow.Sequential
tasks:
- id: t1
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- 'echo "running {{task.id}}"'
- 'sleep 1'
- id: t2
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- 'echo "running {{task.id}}"'
- 'sleep 1'
- id: t3
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- 'echo "running {{task.id}}"'
- 'sleep 1'
- id: seq2
type: io.kestra.plugin.core.flow.Sequential
tasks:
- id: t4
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- 'echo "running {{task.id}}"'
- 'sleep 1'
- id: t5
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- 'echo "running {{task.id}}"'
- 'sleep 1'
- id: t6
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- 'echo "running {{task.id}}"'
- 'sleep 1'
- id: last
type: io.kestra.plugin.core.log.Log
message: "{{task.id}} > {{taskrun.startDate}}"
```
---
# Pause and Resume Flows in Kestra
URL: https://kestra.io/docs/how-to-guides/pause-resume
> Pause and resume Kestra workflow executions on demand. Use manual triggers, scheduled waits, and approval gates to control flow progression at runtime.
How to Pause and Resume your flows.
Here are common scenarios where the Pause and Resume feature is particularly useful:
1. **Output Validation**: you can pause a workflow to check the logs and view the generated outputs before processing downstream tasks.
2. **Manual Approval**: the execution can wait for manual approval, e.g. after validating that a file has been correctly uploaded to an external system.
3. **Human-in-the-loop**: you can pause a workflow execution to perform a human task before resuming the execution, e.g. to validate a trained machine learning model before deploying it to production.
## How to pause and resume a workflow
```yaml
id: pause_resume
namespace: company.team
tasks:
- id: pause
type: io.kestra.plugin.core.flow.Pause
- id: after_pause
type: io.kestra.plugin.core.log.Log
message: Execution has been resumed!
```
The `Pause` task will pause the execution and the `Log` task will run only once the workflow has been resumed.
## Pause and resume a workflow from the UI
You can either use the Pause task or manually pause from the Execution overview page. Once the execution is paused, inspect the logs and outputs, then resume using the **Resume** button in the **Overview** tab:

## Bulk-resuming paused workflows
You can bulk-resume paused workflows from the `Executions` page by selecting the workflows you want to resume and clicking on the `Resume` button:

This feature is useful when you have multiple paused workflows and want to resume them all at once.
:::alert{type="warning"}
Select only workflows in the `PAUSED` state, as the `Resume` button will not work if you select workflows in other states.
:::
### Manual Approval Process
Below, you can see an example of a workflow that sends a Slack message requesting approval for a vacation request to a manager. The workflow execution is paused until the manager resumes it with custom input values. Those input values indicate whether the request was approved and the reason for the decision.
```yaml
id: vacation_approval_process
namespace: company.team
inputs:
- id: request.name
type: STRING
defaults: Rick Astley
- id: request.start_date
type: DATE
defaults: 2024-07-01
- id: request.end_date
type: DATE
defaults: 2024-07-07
- id: slack_webhook_uri
type: URI
defaults: https://kestra.io/api/mock
tasks:
- id: sendApprovalRequest
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
url: "{{ inputs.slack_webhook_uri }}"
payload: |
{
"channel": "#vacation",
"text": "Validate holiday request for {{ inputs.request.name }}. To approve the request, click on the `Resume` button here http://localhost:28080/ui/executions/{{flow.namespace}}/{{flow.id}}/{{execution.id}}"
}
- id: waitForApproval
type: io.kestra.plugin.core.flow.Pause
onResume:
- id: approved
description: Approve the request?
type: BOOLEAN
defaults: true
- id: reason
description: Reason for approval or rejection?
type: STRING
defaults: Approved
- id: approve
type: io.kestra.plugin.core.http.Request
uri: https://kestra.io/api/mock
method: POST
contentType: application/json
body: "{{ inputs.request }}"
- id: log
type: io.kestra.plugin.core.log.Log
message: Status is {{ outputs.waitForApproval.onResume.reason }}. Process finished with {{ outputs.approve.body }}
```
When you click on the `Resume` button in the UI, you will be prompted to provide the approval status and the reason for their decision. The workflow will then continue with the provided input values.

After the Execution has been resumed, any downstream task can access the `onResume` inputs using the `outputs` of the `Pause` task:

---
# Populate Your Instance with Sample Data
URL: https://kestra.io/docs/how-to-guides/populate-demo-data
> Populate your Kestra instance with demo data. Use sample flows and datasets to explore features, test integrations, and validate your setup.
Quickly populate your Kestra instance with realistic demo flows and executions using a single SQL script.
This is useful for demos, testing dashboards, taking screenshots, or exploring Kestra's UI with meaningful data.
## Prerequisites
- Kestra running via [Docker Compose](../../02.installation/03.docker-compose/index.md) with a **PostgreSQL** backend
- Access to the `postgres` container via `docker compose exec`
## What Gets Inserted
The script creates a fully populated instance with:
| Category | Details |
|------------|-------------------------------------------------------------------------|
| **Flows** | 10 flows across 6 namespaces (`acme`, `acme.sales`, `acme.company.data`, `acme.operations`, `acme.marketing`, `acme.finance`) |
| **Executions** | ~224 executions spread over the past 7 days |
| **States** | Realistic distribution: ~70% SUCCESS, ~10% FAILED, ~8% WARNING, ~5% RUNNING, ~4% CANCELLED, ~3% RETRIED |
| **Timing** | Weighted toward business hours (8 AM–6 PM), with some evening and night runs |
## How to Run
Download the SQL script and pipe it into the PostgreSQL container:
```bash
cat seed_demo_data.sql | docker compose exec -T postgres psql -U kestra
```
:::alert{type="info"}
The `-T` flag disables pseudo-TTY allocation, which is required when piping input to `docker compose exec`.
:::
## Key Properties
- **Idempotent** — uses `ON CONFLICT DO NOTHING`, so it's safe to run multiple times without duplicating data.
- **Multi-tenant aware** — all records are created under the `main` tenant ID.
- **Deterministic IDs** — execution IDs are generated with `md5(flow_id + day_offset + index)`, ensuring consistent results across runs.
## Full SQL Script
:::collapse{title="View the full SQL script"}
```sql
-- =============================================================================
-- Kestra Demo Seed Data
-- Inserts ~10 flows and ~180 executions for the past 7 days
-- Idempotent: safe to re-run (ON CONFLICT DO NOTHING)
-- =============================================================================
BEGIN;
-- ============================================================
-- FLOWS
-- ============================================================
-- 1. hello-world (acme)
INSERT INTO flows (key, value, source_code) VALUES (
'acme_hello-world_1',
'{"id":"hello-world","namespace":"acme","tenantId":"main","revision":1,"deleted":false,"description":"Hello World","tasks":[{"id":"first_task","type":"io.kestra.plugin.core.debug.Return","format":"thrilled"},{"id":"second_task","type":"io.kestra.plugin.scripts.shell.Commands","commands":["sleep 0.42","echo ''::{ \"outputs\":{\"returned_data\":\"mydata\"}}::''"]},{"id":"hello_world","type":"io.kestra.plugin.core.log.Log","message":"Welcome to Acme, {{ inputs.user }}!\nWe are {{ outputs.first_task.value }} to have you here!"}],"inputs":[{"id":"user","type":"STRING","defaults":"Rick Astley"}],"triggers":[{"id":"daily","type":"io.kestra.plugin.core.trigger.Schedule","cron":"0 9 * * *","disabled":true}]}'::jsonb,
'id: hello-world
namespace: acme
description: Hello World
inputs:
- id: user
type: STRING
defaults: Rick Astley
tasks:
- id: first_task
type: io.kestra.plugin.core.debug.Return
format: thrilled
- id: second_task
type: io.kestra.plugin.scripts.shell.Commands
commands:
- sleep 0.42
- echo ''::{"outputs":{"returned_data":"mydata"}}::''
- id: hello_world
type: io.kestra.plugin.core.log.Log
message: |
Welcome to Acme, {{ inputs.user }}!
We are {{ outputs.first_task.value }} to have you here!
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: 0 9 * * *
disabled: true'
) ON CONFLICT (key) DO NOTHING;
-- 2. customer_onboarding (acme.sales)
INSERT INTO flows (key, value, source_code) VALUES (
'acme.sales_customer_onboarding_1',
'{"id":"customer_onboarding","namespace":"acme.sales","tenantId":"main","revision":1,"deleted":false,"description":"Automated customer onboarding workflow","tasks":[{"id":"welcome_message","type":"io.kestra.plugin.core.log.Log","message":"Starting onboarding process"},{"id":"generate_customer_id","type":"io.kestra.plugin.core.output.OutputValues","values":{"customer_id":"ACME-20260317"}},{"id":"send_welcome_email","type":"io.kestra.plugin.core.log.Log","message":"Welcome email sent"},{"id":"complete","type":"io.kestra.plugin.core.log.Log","message":"Customer onboarding completed successfully!"}],"inputs":[{"id":"customer_name","type":"STRING","required":true},{"id":"customer_email","type":"STRING","required":true}]}'::jsonb,
'id: customer_onboarding
namespace: acme.sales
description: Automated customer onboarding workflow
inputs:
- id: customer_name
type: STRING
required: true
- id: customer_email
type: STRING
required: true
tasks:
- id: welcome_message
type: io.kestra.plugin.core.log.Log
message: Starting onboarding process
- id: generate_customer_id
type: io.kestra.plugin.core.output.OutputValues
values:
customer_id: ACME-20260317
- id: send_welcome_email
type: io.kestra.plugin.core.log.Log
message: Welcome email sent
- id: complete
type: io.kestra.plugin.core.log.Log
message: Customer onboarding completed successfully!'
) ON CONFLICT (key) DO NOTHING;
-- 3. monthly_sales_report (acme.sales)
INSERT INTO flows (key, value, source_code) VALUES (
'acme.sales_monthly_sales_report_1',
'{"id":"monthly_sales_report","namespace":"acme.sales","tenantId":"main","revision":1,"deleted":false,"description":"Generate monthly sales performance report","tasks":[{"id":"fetch_sales_data","type":"io.kestra.plugin.core.log.Log","message":"Fetching sales data"},{"id":"calculate_metrics","type":"io.kestra.plugin.core.output.OutputValues","values":{"total_sales":"85000","num_customers":"342","avg_deal_size":"2480"}},{"id":"generate_report","type":"io.kestra.plugin.core.log.Log","message":"Monthly Sales Report generated"}],"triggers":[{"id":"monthly_schedule","type":"io.kestra.plugin.core.trigger.Schedule","cron":"0 9 1 * *","disabled":true}]}'::jsonb,
'id: monthly_sales_report
namespace: acme.sales
description: Generate monthly sales performance report
tasks:
- id: fetch_sales_data
type: io.kestra.plugin.core.log.Log
message: Fetching sales data
- id: calculate_metrics
type: io.kestra.plugin.core.output.OutputValues
values:
total_sales: "85000"
num_customers: "342"
avg_deal_size: "2480"
- id: generate_report
type: io.kestra.plugin.core.log.Log
message: Monthly Sales Report generated
triggers:
- id: monthly_schedule
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 9 1 * *"
disabled: true'
) ON CONFLICT (key) DO NOTHING;
-- 4. data_pipeline_assets (acme.company.data)
INSERT INTO flows (key, value, source_code) VALUES (
'acme.company.data_data_pipeline_assets_1',
'{"id":"data_pipeline_assets","namespace":"acme.company.data","tenantId":"main","revision":1,"deleted":false,"tasks":[{"id":"create_staging_layer_asset","type":"io.kestra.plugin.jdbc.duckdb.Query","sql":"CREATE TABLE IF NOT EXISTS trips AS select VendorID, passenger_count, trip_distance from sample_data.nyc.taxi limit 10;"},{"id":"for_each","type":"io.kestra.plugin.core.flow.Loop","values":["passenger_count","trip_distance"],"tasks":[{"id":"create_mart_layer_asset","type":"io.kestra.plugin.jdbc.duckdb.Query","sql":"SELECT AVG({{item.value}}) AS avg_{{item.value}} FROM trips;"}]}]}'::jsonb,
'id: data_pipeline_assets
namespace: acme.company.data
tasks:
- id: create_staging_layer_asset
type: io.kestra.plugin.jdbc.duckdb.Query
sql: |
CREATE TABLE IF NOT EXISTS trips AS
select VendorID, passenger_count, trip_distance from sample_data.nyc.taxi limit 10;
- id: for_each
type: io.kestra.plugin.core.flow.Loop
values:
- passenger_count
- trip_distance
tasks:
- id: create_mart_layer_asset
type: io.kestra.plugin.jdbc.duckdb.Query
sql: SELECT AVG({{item.value}}) AS avg_{{item.value}} FROM trips;'
) ON CONFLICT (key) DO NOTHING;
-- 5. system_health_check (acme.operations)
INSERT INTO flows (key, value, source_code) VALUES (
'acme.operations_system_health_check_1',
'{"id":"system_health_check","namespace":"acme.operations","tenantId":"main","revision":1,"deleted":false,"description":"Monitor system health and performance","tasks":[{"id":"check_api_endpoints","type":"io.kestra.plugin.core.log.Log","message":"Checking API endpoint availability..."},{"id":"check_database","type":"io.kestra.plugin.core.log.Log","message":"Checking database connections..."},{"id":"check_services","type":"io.kestra.plugin.core.log.Log","message":"Checking microservices status..."},{"id":"calculate_uptime","type":"io.kestra.plugin.core.output.OutputValues","values":{"api_uptime":"99.95","db_response_time":"12","services_healthy":"10"}},{"id":"generate_report","type":"io.kestra.plugin.core.log.Log","message":"System Health Report generated"}],"triggers":[{"id":"hourly_check","type":"io.kestra.plugin.core.trigger.Schedule","cron":"0 * * * *","disabled":true}]}'::jsonb,
'id: system_health_check
namespace: acme.operations
description: Monitor system health and performance
tasks:
- id: check_api_endpoints
type: io.kestra.plugin.core.log.Log
message: Checking API endpoint availability...
- id: check_database
type: io.kestra.plugin.core.log.Log
message: Checking database connections...
- id: check_services
type: io.kestra.plugin.core.log.Log
message: Checking microservices status...
- id: calculate_uptime
type: io.kestra.plugin.core.output.OutputValues
values:
api_uptime: "99.95"
db_response_time: "12"
services_healthy: "10"
- id: generate_report
type: io.kestra.plugin.core.log.Log
message: System Health Report generated
triggers:
- id: hourly_check
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 * * * *"
disabled: true'
) ON CONFLICT (key) DO NOTHING;
-- 6. inventory_check (acme.operations)
INSERT INTO flows (key, value, source_code) VALUES (
'acme.operations_inventory_check_1',
'{"id":"inventory_check","namespace":"acme.operations","tenantId":"main","revision":1,"deleted":false,"description":"Daily inventory level monitoring","tasks":[{"id":"scan_inventory","type":"io.kestra.plugin.core.log.Log","message":"Scanning inventory levels across all warehouses..."},{"id":"check_levels","type":"io.kestra.plugin.core.output.OutputValues","values":{"total_items":"8542","low_stock_items":"23","out_of_stock":"2"}},{"id":"generate_alerts","type":"io.kestra.plugin.core.log.Log","message":"Inventory Status Report generated"},{"id":"notify_purchasing","type":"io.kestra.plugin.core.log.Log","message":"Reorder notifications sent to purchasing team"}],"triggers":[{"id":"daily_check","type":"io.kestra.plugin.core.trigger.Schedule","cron":"0 7 * * *","disabled":true}]}'::jsonb,
'id: inventory_check
namespace: acme.operations
description: Daily inventory level monitoring
tasks:
- id: scan_inventory
type: io.kestra.plugin.core.log.Log
message: Scanning inventory levels across all warehouses...
- id: check_levels
type: io.kestra.plugin.core.output.OutputValues
values:
total_items: "8542"
low_stock_items: "23"
out_of_stock: "2"
- id: generate_alerts
type: io.kestra.plugin.core.log.Log
message: Inventory Status Report generated
- id: notify_purchasing
type: io.kestra.plugin.core.log.Log
message: Reorder notifications sent to purchasing team
triggers:
- id: daily_check
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 7 * * *"
disabled: true'
) ON CONFLICT (key) DO NOTHING;
-- 7. email_campaign_trigger (acme.marketing)
INSERT INTO flows (key, value, source_code) VALUES (
'acme.marketing_email_campaign_trigger_1',
'{"id":"email_campaign_trigger","namespace":"acme.marketing","tenantId":"main","revision":1,"deleted":false,"description":"Trigger email marketing campaigns","tasks":[{"id":"validate_campaign","type":"io.kestra.plugin.core.log.Log","message":"Validating campaign"},{"id":"calculate_audience","type":"io.kestra.plugin.core.output.OutputValues","values":{"audience_size":"4500"}},{"id":"send_campaign","type":"io.kestra.plugin.core.log.Log","message":"Sending campaign"},{"id":"track_metrics","type":"io.kestra.plugin.core.log.Log","message":"Campaign sent successfully. Tracking metrics..."}],"inputs":[{"id":"campaign_name","type":"STRING","defaults":"Monthly Newsletter"},{"id":"target_segment","type":"SELECT","values":["All Customers","Premium Customers","Trial Users","Inactive Users"],"defaults":"All Customers"}]}'::jsonb,
'id: email_campaign_trigger
namespace: acme.marketing
description: Trigger email marketing campaigns
inputs:
- id: campaign_name
type: STRING
defaults: "Monthly Newsletter"
- id: target_segment
type: SELECT
values:
- All Customers
- Premium Customers
- Trial Users
- Inactive Users
defaults: "All Customers"
tasks:
- id: validate_campaign
type: io.kestra.plugin.core.log.Log
message: Validating campaign
- id: calculate_audience
type: io.kestra.plugin.core.output.OutputValues
values:
audience_size: "4500"
- id: send_campaign
type: io.kestra.plugin.core.log.Log
message: Sending campaign
- id: track_metrics
type: io.kestra.plugin.core.log.Log
message: Campaign sent successfully. Tracking metrics...'
) ON CONFLICT (key) DO NOTHING;
-- 8. social_media_analytics (acme.marketing)
INSERT INTO flows (key, value, source_code) VALUES (
'acme.marketing_social_media_analytics_1',
'{"id":"social_media_analytics","namespace":"acme.marketing","tenantId":"main","revision":1,"deleted":false,"description":"Aggregate social media performance metrics","tasks":[{"id":"fetch_twitter_metrics","type":"io.kestra.plugin.core.log.Log","message":"Fetching Twitter/X metrics..."},{"id":"fetch_linkedin_metrics","type":"io.kestra.plugin.core.log.Log","message":"Fetching LinkedIn metrics..."},{"id":"aggregate_data","type":"io.kestra.plugin.core.output.OutputValues","values":{"total_impressions":"32000","total_engagement":"1250","follower_growth":"87"}},{"id":"generate_insights","type":"io.kestra.plugin.core.log.Log","message":"Social Media Weekly Report generated"}],"triggers":[{"id":"weekly_report","type":"io.kestra.plugin.core.trigger.Schedule","cron":"0 10 * * 1","disabled":true}]}'::jsonb,
'id: social_media_analytics
namespace: acme.marketing
description: Aggregate social media performance metrics
tasks:
- id: fetch_twitter_metrics
type: io.kestra.plugin.core.log.Log
message: Fetching Twitter/X metrics...
- id: fetch_linkedin_metrics
type: io.kestra.plugin.core.log.Log
message: Fetching LinkedIn metrics...
- id: aggregate_data
type: io.kestra.plugin.core.output.OutputValues
values:
total_impressions: "32000"
total_engagement: "1250"
follower_growth: "87"
- id: generate_insights
type: io.kestra.plugin.core.log.Log
message: Social Media Weekly Report generated
triggers:
- id: weekly_report
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 10 * * 1"
disabled: true'
) ON CONFLICT (key) DO NOTHING;
-- 9. invoice_processing (acme.finance)
INSERT INTO flows (key, value, source_code) VALUES (
'acme.finance_invoice_processing_1',
'{"id":"invoice_processing","namespace":"acme.finance","tenantId":"main","revision":1,"deleted":false,"description":"Process and validate invoices","tasks":[{"id":"validate_invoice","type":"io.kestra.plugin.core.log.Log","message":"Validating invoice"},{"id":"check_approval_needed","type":"io.kestra.plugin.core.output.OutputValues","values":{"needs_approval":"true","approver":"CFO"}},{"id":"process_payment","type":"io.kestra.plugin.core.log.Log","message":"Processing payment"},{"id":"send_confirmation","type":"io.kestra.plugin.core.log.Log","message":"Payment confirmation sent"}],"inputs":[{"id":"invoice_number","type":"STRING","required":true},{"id":"amount","type":"FLOAT","required":true},{"id":"vendor_name","type":"STRING","required":true}]}'::jsonb,
'id: invoice_processing
namespace: acme.finance
description: Process and validate invoices
inputs:
- id: invoice_number
type: STRING
required: true
- id: amount
type: FLOAT
required: true
- id: vendor_name
type: STRING
required: true
tasks:
- id: validate_invoice
type: io.kestra.plugin.core.log.Log
message: Validating invoice
- id: check_approval_needed
type: io.kestra.plugin.core.output.OutputValues
values:
needs_approval: "true"
approver: CFO
- id: process_payment
type: io.kestra.plugin.core.log.Log
message: Processing payment
- id: send_confirmation
type: io.kestra.plugin.core.log.Log
message: Payment confirmation sent'
) ON CONFLICT (key) DO NOTHING;
-- 10. quarterly_financial_report (acme.finance)
INSERT INTO flows (key, value, source_code) VALUES (
'acme.finance_quarterly_financial_report_1',
'{"id":"quarterly_financial_report","namespace":"acme.finance","tenantId":"main","revision":1,"deleted":false,"description":"Generate quarterly financial statements","tasks":[{"id":"gather_financial_data","type":"io.kestra.plugin.core.log.Log","message":"Gathering financial data"},{"id":"calculate_financials","type":"io.kestra.plugin.core.output.OutputValues","values":{"revenue":"842000","expenses":"510000","profit_margin":"24.5"}},{"id":"generate_report","type":"io.kestra.plugin.core.log.Log","message":"Quarterly Financial Report generated"},{"id":"distribute_report","type":"io.kestra.plugin.core.log.Log","message":"Report distributed to executive team"}],"triggers":[{"id":"quarterly_schedule","type":"io.kestra.plugin.core.trigger.Schedule","cron":"0 8 1 1,4,7,10 *","disabled":true}]}'::jsonb,
'id: quarterly_financial_report
namespace: acme.finance
description: Generate quarterly financial statements
tasks:
- id: gather_financial_data
type: io.kestra.plugin.core.log.Log
message: Gathering financial data
- id: calculate_financials
type: io.kestra.plugin.core.output.OutputValues
values:
revenue: "842000"
expenses: "510000"
profit_margin: "24.5"
- id: generate_report
type: io.kestra.plugin.core.log.Log
message: Quarterly Financial Report generated
- id: distribute_report
type: io.kestra.plugin.core.log.Log
message: Report distributed to executive team
triggers:
- id: quarterly_schedule
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 8 1 1,4,7,10 *"
disabled: true'
) ON CONFLICT (key) DO NOTHING;
-- ============================================================
-- EXECUTIONS
-- Generated via PL/pgSQL to create ~180 executions over 7 days
-- ============================================================
DO $$
DECLARE
-- Flow definitions: flow_id, namespace, avg_duration_seconds, daily_frequency
flow_configs TEXT[][] := ARRAY[
ARRAY['hello-world', 'acme', '8', '5'],
ARRAY['customer_onboarding', 'acme.sales', '15', '4'],
ARRAY['monthly_sales_report', 'acme.sales', '120', '2'],
ARRAY['data_pipeline_assets', 'acme.company.data', '300', '3'],
ARRAY['system_health_check', 'acme.operations', '25', '8'],
ARRAY['inventory_check', 'acme.operations', '45', '3'],
ARRAY['email_campaign_trigger', 'acme.marketing', '90', '2'],
ARRAY['social_media_analytics', 'acme.marketing', '180', '1'],
ARRAY['invoice_processing', 'acme.finance', '30', '3'],
ARRAY['quarterly_financial_report','acme.finance', '600', '1']
];
-- State distribution weights (cumulative out of 100):
-- SUCCESS=70, FAILED=80, WARNING=88, RUNNING=93, CANCELLED=97, RETRIED=100
state_thresholds INT[] := ARRAY[70, 80, 88, 93, 97, 100];
state_names TEXT[] := ARRAY['SUCCESS', 'FAILED', 'WARNING', 'RUNNING', 'CANCELLED', 'RETRIED'];
base_date TIMESTAMP;
exec_id TEXT;
flow_id TEXT;
flow_ns TEXT;
avg_dur INT;
daily_freq INT;
exec_start TIMESTAMP;
exec_end TIMESTAMP;
duration_secs INT;
duration_iso TEXT;
state TEXT;
rand_val INT;
day_offset INT;
hour_val INT;
minute_val INT;
histories_json TEXT;
exec_json TEXT;
hour_weight FLOAT;
i INT;
j INT;
BEGIN
-- Base date = 7 days ago at midnight UTC
base_date := date_trunc('day', NOW() - INTERVAL '7 days');
FOR i IN 1..array_length(flow_configs, 1) LOOP
flow_id := flow_configs[i][1];
flow_ns := flow_configs[i][2];
avg_dur := flow_configs[i][3]::INT;
daily_freq := flow_configs[i][4]::INT;
FOR day_offset IN 0..6 LOOP
-- Fixed daily count for idempotency (deterministic loop bounds)
FOR j IN 1..daily_freq LOOP
-- Generate hour weighted toward business hours (8-18)
hour_weight := random();
IF hour_weight < 0.7 THEN
-- 70% during business hours 8-18
hour_val := 8 + (random() * 10)::INT;
ELSIF hour_weight < 0.9 THEN
-- 20% during evening 18-23
hour_val := 18 + (random() * 5)::INT;
ELSE
-- 10% during night 0-7
hour_val := (random() * 7)::INT;
END IF;
minute_val := (random() * 59)::INT;
exec_start := base_date + (day_offset || ' days')::INTERVAL
+ (hour_val || ' hours')::INTERVAL
+ (minute_val || ' minutes')::INTERVAL
+ ((random() * 59)::INT || ' seconds')::INTERVAL;
-- Duration: vary between 30% and 250% of avg
duration_secs := greatest(2, (avg_dur * (0.3 + random() * 2.2))::INT);
-- Pick state based on distribution
rand_val := (random() * 99)::INT + 1;
state := 'SUCCESS';
FOR k IN 1..array_length(state_thresholds, 1) LOOP
IF rand_val <= state_thresholds[k] THEN
state := state_names[k];
EXIT;
END IF;
END LOOP;
-- RUNNING executions have no end date
IF state = 'RUNNING' THEN
exec_end := NULL;
-- Make start_date recent (within last 30 min)
exec_start := NOW() - (random() * 30 || ' minutes')::INTERVAL;
duration_iso := 'PT' || (EXTRACT(EPOCH FROM (NOW() - exec_start))::INT) || 'S';
ELSE
exec_end := exec_start + (duration_secs || ' seconds')::INTERVAL;
-- Build ISO 8601 duration
IF duration_secs >= 3600 THEN
duration_iso := 'PT' || (duration_secs / 3600) || 'H'
|| ((duration_secs % 3600) / 60) || 'M'
|| (duration_secs % 60) || 'S';
ELSIF duration_secs >= 60 THEN
duration_iso := 'PT' || (duration_secs / 60) || 'M'
|| (duration_secs % 60) || 'S';
ELSE
duration_iso := 'PT' || duration_secs || 'S';
END IF;
END IF;
-- Deterministic ID based on flow + day + index for idempotency
exec_id := md5(flow_id || '_' || day_offset::TEXT || '_' || j::TEXT);
-- Build state histories JSON
IF state = 'RUNNING' THEN
histories_json := '[{"state":"CREATED","date":"' || to_char(exec_start, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"},'
|| '{"state":"RUNNING","date":"' || to_char(exec_start + INTERVAL '100 milliseconds', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"}]';
ELSIF state = 'RETRIED' THEN
histories_json := '[{"state":"CREATED","date":"' || to_char(exec_start, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"},'
|| '{"state":"RUNNING","date":"' || to_char(exec_start + INTERVAL '100 milliseconds', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"},'
|| '{"state":"FAILED","date":"' || to_char(exec_start + (duration_secs / 2 || ' seconds')::INTERVAL, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"},'
|| '{"state":"RETRYING","date":"' || to_char(exec_start + (duration_secs / 2 || ' seconds')::INTERVAL + INTERVAL '1 second', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"},'
|| '{"state":"RUNNING","date":"' || to_char(exec_start + (duration_secs / 2 || ' seconds')::INTERVAL + INTERVAL '2 seconds', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"},'
|| '{"state":"RETRIED","date":"' || to_char(exec_end, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"}]';
ELSIF state = 'CANCELLED' THEN
histories_json := '[{"state":"CREATED","date":"' || to_char(exec_start, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"},'
|| '{"state":"RUNNING","date":"' || to_char(exec_start + INTERVAL '100 milliseconds', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"},'
|| '{"state":"KILLING","date":"' || to_char(exec_end - INTERVAL '1 second', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"},'
|| '{"state":"CANCELLED","date":"' || to_char(exec_end, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"}]';
ELSE
-- SUCCESS, FAILED, WARNING
histories_json := '[{"state":"CREATED","date":"' || to_char(exec_start, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"},'
|| '{"state":"RUNNING","date":"' || to_char(exec_start + INTERVAL '100 milliseconds', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"},'
|| '{"state":"' || state || '","date":"' || to_char(exec_end, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '"}]';
END IF;
-- Build full execution JSON
exec_json := '{"id":"' || exec_id || '",'
|| '"namespace":"' || flow_ns || '",'
|| '"tenantId":"main",'
|| '"flowId":"' || flow_id || '",'
|| '"flowRevision":1,'
|| '"deleted":false,'
|| '"state":{'
|| '"current":"' || state || '",'
|| '"startDate":"' || to_char(exec_start, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '",'
|| CASE WHEN exec_end IS NOT NULL
THEN '"endDate":"' || to_char(exec_end, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') || '",'
ELSE ''
END
|| '"duration":"' || duration_iso || '",'
|| '"histories":' || histories_json
|| '},'
|| '"taskRunList":[]}';
INSERT INTO executions (key, value)
VALUES (exec_id, exec_json::jsonb)
ON CONFLICT (key) DO NOTHING;
END LOOP; -- j (executions per day)
END LOOP; -- day_offset
END LOOP; -- i (flows)
END $$;
COMMIT;
```
:::
---
# Run PowerShell Inside Your Flows
URL: https://kestra.io/docs/how-to-guides/powershell
> Run PowerShell scripts in Kestra. Automate Windows administration, call Azure APIs, and integrate Microsoft services into your automation pipelines.
Run PowerShell code in your flow.
PowerShell is commonly used for automating the management of systems and resources. With Kestra, you can effortlessly automate builds and tests for production systems, as well as manage cloud configurations and resources. Kestra's robust orchestration capabilities ensure that your PowerShell scripts run smoothly and efficiently, streamlining your infrastructure.
This guide is going to walk you through how to get PowerShell running in a workflow, how to manage input and output files, and how you can pass outputs and metrics back to Kestra to use in later tasks.
You can execute PowerShell code in a flow by either writing your PowerShell code inline or by executing a `.ps1` file. You can get outputs and metrics from your PowerShell code too.
## Scripts
If you want to write a short amount of PowerShell code to perform a task, you can use the `io.kestra.plugin.scripts.powershell.Script` type to write it directly inside your flow. This allows you to keep everything in one place.
```yaml
id: powershell_script
namespace: company.team
description: This flow runs the PowerShell script.
tasks:
- id: http_download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: powershell_script_task
type: io.kestra.plugin.scripts.powershell.Script
script: |
Write-Output "The current execution is {{ execution.id }}"
# Read the file downloaded in `http_download` task
$content = Get-Content "{{ outputs.http_download.uri }}"
$content
```
You can read more about the Scripts type in the [Plugin documentation](/plugins/plugin-script-powershell/io.kestra.plugin.scripts.powershell.script)
## Commands
If you would prefer to put your PowerShell code in a `.ps1` file (e.g. your code is much longer or spread across multiple files), you can run the previous example using the `io.kestra.plugin.scripts.powershell.Commands` type:
```yaml
id: powershell_commands
namespace: company.team
tasks:
- id: run_powershell
type: io.kestra.plugin.scripts.powershell.Commands
namespaceFiles:
enabled: true
commands:
- ./main.ps1
```
The contents of the `main.ps1` file can be:
```powershell
Write-Output "Hello World"
```
You'll need to add your PowerShell code using the Editor or [sync it using Git](../../version-control-cicd/04.git/index.md) so Kestra can see it. You'll also need to set the `enabled` flag for the `namespaceFiles` property to `true` so Kestra can access the file.
You can also have the PowerShell code written inline.
```yaml
id: powershell_commands
namespace: company.team
tasks:
- id: http_download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: run_powershell
type: io.kestra.plugin.scripts.powershell.Commands
inputFiles:
orders.csv: "{{ read(outputs.http_download.uri) }}"
main.ps1: |
Write-Output "The current execution is {{ execution.id }}"
# Read the file
$content = Get-Content "orders.csv"
$content
commands:
- ./main.ps1
```
You can read more about the Commands type in the [Plugin documentation](/plugins/plugin-script-powershell/io.kestra.plugin.scripts.powershell.commands).
## Handling Outputs
If you want to get a variable or file from your PowerShell script, you can use an [output](../../05.workflow-components/06.outputs/index.md).
### Variable Output
You can put out the JSON outputs from the PowerShell commands / script using the `::{}::` pattern. Here is an example:
```yaml
id: powershell_outputs
namespace: company.team
description: This flow runs the PowerShell script, and outputs the variable.
tasks:
- id: powershell_outputs_task
type: io.kestra.plugin.scripts.powershell.Script
script: |
Write-Output '::{"outputs":{"test":"value","int":2,"bool":true,"float":3.65}}::'
```
All the output variables can be viewed in the Outputs tab of the execution.

You can refer to the outputs in another task as shown in the example below:
```yaml
id: powershell_outputs
namespace: company.team
description: This flow runs the PowerShell script, and outputs the variable.
tasks:
- id: powershell_outputs_task
type: io.kestra.plugin.scripts.powershell.Script
script: |
Write-Output '::{"outputs":{"test":"value","int":2,"bool":true,"float":3.65}}::'
- id: return
type: io.kestra.plugin.core.debug.Return
format: '{{ outputs.powershell_outputs_task.vars.test }}'
```
_This example works for both `io.kestra.plugin.scripts.powershell.Script` and `io.kestra.plugin.scripts.powershell.Commands`._
### File Output
Inside of your PowerShell script, write a file to the system. You'll need to add the `outputFiles` property to your flow and list the files you're trying to put out. In this case, we want to output `output.txt`. More information on the formats you can use for this property can be found in [Script Ouput Metrics](../../16.scripts/06.outputs-metrics/index.md).
The example below writes a `output.txt` file containing the "Hello World" text. We can then refer the file using the syntax `{{ outputs.{task_id}.outputFiles[''] }}`, and read the contents of the file using the `read()` function.
```yaml
id: powershell_output_file
namespace: company.team
description: This flow runs the PowerShell script to output a file.
tasks:
- id: powershell_outputs_task
type: io.kestra.plugin.scripts.powershell.Script
outputFiles:
- output.txt
script: |
Set-Content -Path "output.txt" -Value "Hello World"
- id: log_output
type: io.kestra.plugin.core.log.Log
message: "{{ read(outputs.powershell_outputs_task.outputFiles['output.txt']) }}"
```
_This example works for both `io.kestra.plugin.scripts.powershell.Script` and `io.kestra.plugin.scripts.powershell.Commands`._
## Handling Metrics
You can also get [metrics](../../16.scripts/06.outputs-metrics/index.md#outputs-and-metrics-in-script-and-commands-tasks) from your PowerShell script. Metrics use the same `::{}::` pattern as outputs. This example demonstrates both the counter and timer metrics.
```yaml
id: powershell_metrics
namespace: company.team
description: This flow runs the PowerShell script, and puts out the metrics.
tasks:
- id: powershell_metrics_task
type: io.kestra.plugin.scripts.powershell.Script
script: |
Write-Output 'There are 20 products in the cart'
Write-Output '::{"outputs":{"productCount":20}}::'
Write-Output '::{"metrics":[{"name":"productCount","type":"counter","value":20}]}::'
Write-Output '::{"metrics":[{"name":"purchaseTime","type":"timer","value":32.44}]}::'
```
Once this has executed, both the metrics can be viewed under **Metrics**.

---
# Trigger a Flow on a Prometheus Alert
URL: https://kestra.io/docs/how-to-guides/prometheus-alert-trigger
> Connect Prometheus alerts to Kestra to automatically trigger flows via webhooks when specific metrics thresholds are breached.
Connect Prometheus alerts to Kestra to trigger flows.
## Monitoring with Prometheus and Triggering Flows in Kestra
This guide explains how to connect Prometheus to Kestra and configure a workflow that is triggered by Prometheus alerts. This guide covers the basics and is intended as a starting off point for any production workflows. You will:
1. Integrate Prometheus with Kestra
2. Configure Prometheus Alertmanager to send alerts via webhook
3. Use a Webhook Trigger in a Kestra flow
---
## Connect Prometheus to Kestra
Kestra natively supports integration with Prometheus for metric scraping and visualization. Kestra exposes [Prometheus](https://prometheus.io/) metrics at port 8081 on the endpoint `/prometheus`. This endpoint can be used by any compatible monitoring system.
Follow these steps based on the [Kestra Monitoring Guide](../monitoring/index.md).
Once Kestra is up and running, view the available metrics at `http://localhost:8081/prometheus` in your browser.
## Configure Prometheus to Scrape Kestra
Add the Kestra metrics endpoint to your Prometheus configuration (`prometheus.yml`):
```yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "prometheus"
metrics_path: /prometheus
static_configs:
- targets: [":8081"]
```
Be sure to put the appropriate in the last line, e.g., `localhost:8081` or `host.docker.internal:8081`. Restart Prometheus for the changes to take effect.
:::alert{type="info"}
If you're running everything with Docker on the same machine, you will need to change your host address to `host.docker.internal` rather than localhost, or the name of the container.
:::
---
## Create a Prometheus Alert and Webhook Receiver
To trigger a Kestra flow on a Prometheus alert, configure [Prometheus Alertmanager](https://github.com/prometheus/alertmanager) to send a webhook to Kestra. You can [download Alertmanager and Prometheus](https://prometheus.io/download/) from the official site as well as run the services in a Docker Compose file, refer below for an example:
```yaml
services:
prometheus:
image: prom/prometheus
privileged: true
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- ./alertmanager/alert.rules:/alertmanager/alert.rules
command:
- '--config.file=/etc/prometheus/prometheus.yml'
ports:
- '9090:9090'
node-exporter:
image: prom/node-exporter
ports:
- '9100:9100'
alertmanager:
image: prom/alertmanager
privileged: true
volumes:
- ./alertmanager/alertmanager.yml:/alertmanager.yml
command:
- '--config.file=/alertmanager.yml'
ports:
- '9093:9093'
```
You can verify Prometheus is up and running by going to `http://localhost:9090/graph` and visualizing some metrics using the PromQL. Below is one of the graphs for `kestra_executor_execution_started_count_total` metric:

Similarly, you can go to `http://localhost:9093/status` and see that the Alertmanager is ready.

### Step 1: Define a Prometheus Alert
In your `prometheus.yml` file, you must add Alertmanager and some rules files. For example, the `prometheus.yml` configuration now looks as follows:
```yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "prometheus"
metrics_path: /prometheus
static_configs:
- targets: [":8081"]
## Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- 'localhost: 9093' # Replace with your host name (i.e., host.docker.internal)
## Load rules once and periodically evaluate them according to global 'evaluation_interval'.
rule_files:
- "/alertmanager/alert.rules"
```
Create a simple rule to alert on high CPU usage or another metric exposed by Kestra. Refer to our full list at [Kestra Prometheus Metrics](../../10.administrator-guide/prometheus-metrics/index.md):
```yaml
groups:
- name: alert.rules
rules:
- alert: HighCPUUsage
expr: system_cpu_usage == 1.0
for: 1m
labels:
severity: "critical"
annotations:
summary: "High CPU usage on {{ $labels.instance }}"
```
You can also use a simple, generic instance down alert such as:
```yaml
groups:
- name: alert.rules
rules:
# Alert for any instance that is unreachable for >5 minutes.
- alert: InstanceDown
expr: up == 0
for: 5m
```
Test different metrics and statuses for what fits your use case. Save these rules in a file such as `alert.rules.yml` and configure Prometheus to load it like in the above `prometheus.yml`:
```yaml
rule_files:
- "/alertmanager/alert.rules"
```
:::alert{type="info"}
Ensure that `groups.name` and `rule_files` have the same name so that Alertmanager correctly connects the alerts.
:::
To check that your rules are picked up by Prometheus, go to `http://localhost:9090/rules`.

From there, you can see a list of the rules set in the `alert.rules.yml` file:

### Step 2: Configure Alertmanager to Use a Webhook
Now that all the services are connected, edit `alertmanager.yml` to send alerts to a Kestra webhook:
```yaml
receivers:
- name: 'kestra-webhook'
webhook_configs:
- url: 'https:///api/v1/triggers/webhook'
send_resolved: true
route:
receiver: 'kestra-webhook'
```
Ensure your Alertmanager is restarted and using this configuration.
---
## 3. Create a Kestra Webhook Triggered Flow
Now create a Kestra flow that is triggered by a Prometheus alert via [webhook](../../05.workflow-components/07.triggers/03.webhook-trigger/index.md) from the rule definitions specified in the `alert.rules.yml` file.
### Example Flow Definition
```yaml
id: prometheus-alert
namespace: system
triggers:
- id: from-prometheus
type: io.kestra.plugin.core.trigger.Webhook
key: prometheus
tasks:
- id: log-alert
type: io.kestra.plugin.core.log.Log
message: "Received alert: {{ trigger.body }}"
```
Once the flow is written, you can verify the trigger is active from the **Flows -> Triggers** tab in the UI.

### How It Works
* The `Webhook` trigger listens for HTTP POST requests to:
```text
https:///api/v1/triggers/webhook/prometheus
```
- Prometheus Alertmanager sends alerts to this endpoint.
- The flow is executed with the alert content available as `{{ trigger.body }}`.
---
For more on the Webhook trigger, see the [Kestra Webhook Trigger Docs](https://kestra.io/docs/workflow-components/triggers/webhook-trigger). Again, see the [full list of metrics Kestra exposes to Prometheus](../../10.administrator-guide/prometheus-metrics/index.md).
---
# Push Flows to a Git Repository
URL: https://kestra.io/docs/how-to-guides/pushflows
> Use the PushFlows task to push your Kestra flows to a Git repository directly from the UI, keeping your development and production in sync.
Push your Flows to a Git Repository with the PushFlows Task.
## How it works
The [PushFlows](/plugins/plugin-git/io.kestra.plugin.git.pushflows) task is a powerful integration that allows you to **push your code to Git from the UI while still managing this process entirely in code**! Kestra unifies the development experience between the UI and code so you can combine the best of both worlds without sacrificing the benefits of version control.
You can **build your flows** in a development namespace using all **productivity features of the Kestra UI** (_such as the built-in code editor, autocompletion, syntax validation, documentation, blueprint examples, live-updating topology view, output previews, replays, execution and revision history_) and then **push them to Git** after you have tested and validated them.
The task pushes one or more flows from a given namespace (and optionally also child namespaces) to any Git-based Version Control System.
Additionally, the `dryRun` property will help you see what files will be added, modified, or deleted without overwriting the files on Git yet.
The following examples cover common patterns for the `PushFlows` task.
## Before you begin
Before you start using the `PushFlows` task, ensure the following prerequisites are in place:
1. A Git repository where you want to push your flows.
2. A Personal Access Token (PAT) for Git authentication.
3. A running Kestra instance in a version 0.17.0 or later with the PAT stored as a [secret](../../06.concepts/04.secret/index.md) within the Kestra instance.
## Using the `dryRun` property
Start by creating a single `hello_world` flow in the `dev` namespace and pushing it to a Git repository. Initially set the `dryRun` property to `true` to validate the changes before committing them to Git.
```yaml
id: hello_world
namespace: company.team
inputs:
- id: greeting
type: STRING
defaults: kestra
tasks:
- id: welcome
type: io.kestra.plugin.core.log.Log
message: welcome to {{ inputs.greeting }}
```
Here is a flow that will push the `hello_world` flow to a Git repository:
```yaml
id: push_to_git
namespace: company.ops
tasks:
- id: commit_and_push
type: io.kestra.plugin.git.PushFlows
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/git_username/flows
branch: develop
flows:
- hello_world
sourceNamespace: company.team
targetNamespace: prod
gitDirectory: flows
commitMessage: "changes to kestra flows"
dryRun: true
```
Given that the `dryRun` property is set to `true`, the task will only output modifications without pushing any flows to Git yet:

## Pushing a single flow to Git
Set the `dryRun` property to `false` and push the `hello_world` flow to Git:
```yaml
id: push_to_git
namespace: company.ops
tasks:
- id: commit_and_push
type: io.kestra.plugin.git.PushFlows
...
dryRun: false
```
You should see the following log message:

And here is what you should see in the Outputs tab:

When you click on the commit URL from the logs or from the Outputs tab, you'll be redirected to the commit page on GitHub:

Now, you can create a pull request and merge the changes to the main branch.

## Pushing all flows from a single namespace to Git
To push all flows from a given namespace to Git, create two more flows in the `dev` namespace:
```yaml
id: flow1
namespace: company.team
tasks:
- id: test
type: io.kestra.plugin.core.log.Log
message: this is too easy
```
The `flow2` flow is just a copy of the `flow1` with a different flow ID and message:
```yaml
id: flow2
namespace: company.team
tasks:
- id: test
type: io.kestra.plugin.core.log.Log
message: the simplest dev-to-prod workflow ever
```

Adjust the flow to push all flows from the `dev` namespace to the `develop` branch:
```yaml
id: push_to_git
namespace: company.ops
tasks:
- id: commit_and_push
type: io.kestra.plugin.git.PushFlows
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/git_username/flows
branch: develop
sourcenamespace: company.team
targetNamespace: prod
gitDirectory: flows
commitMessage: "push all development flows to Git and create a PR"
dryRun: true
```
Setting `dryRun` to `true` shows what files will be added, modified, or deleted based on the Git version without overwriting the files in Git yet:

Now if you change the `dryRun` property to `false` and run the flow again, you should see all three flows being pushed to the `flows` directory on the `develop` branch with the exact commit messages we have specified in the `commitMessage` property:

## Pushing all flows including child namespaces
Finally, we get to the fun part of pushing all flows from the `dev` namespace **including all child namespaces**. Kestra will automatically create a subfolder for each child namespace and push the flows there to keep everything organized.
Create two more flows in the `dev.tutorial` namespace:
1. `hello_world_1` flow:
```yaml
id: hello_world_1
namespace: company.team.tutorial
inputs:
- id: greeting
type: STRING
defaults: hey
tasks:
- id: print_status
type: io.kestra.plugin.core.log.Log
message: hello on {{ inputs.greeting }}
```
2. `hello_world_2` flow:
```yaml
id: hello_world_2
namespace: company.team.tutorial
inputs:
- id: greeting
type: STRING
defaults: hey
tasks:
- id: print_status
type: io.kestra.plugin.core.log.Log
message: hello on {{ inputs.greeting }}
```
To include all child namespaces in our Git commit, we only need to add the `includeChildNamespaces` property, set to `true`:
```yaml
id: push_to_git
namespace: company.ops
tasks:
- id: commit_and_push
type: io.kestra.plugin.git.PushFlows
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/git_username/flows
branch: develop
sourcenamespace: company.team
targetNamespace: prod
gitDirectory: flows
commitMessage: "push all flows"
includeChildNamespaces: true
```
When you run this flow, you should see the following output:

And here is a confirmation that all flows from the `dev` namespace and its child namespaces have been pushed to the Git repository:

Here is a simple table to illustrate how flows are mapped to files in the Git repository:
| Flow | Source namespace | Git directory path |
|---------------|-----------------------------------|----------------------------------|
| hello_world | dev | flows/hello_world.yml |
| flow1 | dev | flows/flow1.yml |
| flow2 | dev | flows/flow2.yml |
| hello_world_1 | dev.tutorial | flows/tutorial/hello_world_1.yml |
| hello_world_2 | dev.tutorial | flows/tutorial/hello_world_2.yml |
You can see that each child namespace is represented as a subfolder in the Git repository, and all flows are neatly organized in their respective directories.
## Extra notes
- The `flows` property allows you to specify a list of Regex strings that declare which flows should be included in the Git commit. By default, all flows from the specified `sourceNamespace` will be pushed (and optionally adjusted to match the `targetNamespace` before pushing to Git). If you want to push only the current flow, you can use the `{{flow.id}}` expression or specify the flow ID explicitly, e.g. `myflow`. You can also use this property to push only specific flows — you have full flexibility to customize this task to your preferred deployment strategy.
- The `branch` property allows you to specify the branch to which files should be committed and pushed. If the branch doesn’t exist yet, it will be created.
- The `commitMessage` property allows you to specify the Git commit message. You can use templating to include dynamic values in your commit message.
- The `gitDirectory` property allows you to specify the directory to which flows should be pushed. If not set, flows will be pushed to a Git directory named `_flows` and will optionally also include subdirectories named after the child namespaces. If you prefer, you can specify an arbitrary path, e.g. `kestra/flows`, allowing you to push flows to that specific Git directory.
- If you omit the `targetNamespace`, the `sourceNamespace` will be used as the `targetNamespace` by default. The `targetNamespace` is an optional mechanism to help you prepare your development flows to be merged into the production branch/namespace. If you set the `targetNamespace`, the `sourceNamespace` in the source code will be overwritten by the `targetNamespace` so that you can sync the flows to production.
- If you try to add the Personal Access Token (PAT) directly in your source code in the `password` property, you will get an error message. This is a safety mechanism to prevent you and your users from accidentally exposing your PAT in the source code. You should store the PAT as a Kestra Secret, environment variable, namespace variable or as a SECRET-type input in your flow.
- Git does not guarantee the order of push operations to a remote repository, which can lead to potential conflicts when multiple users or flows attempt to push changes simultaneously. To minimize the risk of data loss and merge conflicts, it is strongly recommended to use sequential workflows or push changes to separate branches.

---
# Push Namespace Files to a Git Repository
URL: https://kestra.io/docs/how-to-guides/pushnamespacefiles
> Push your namespace files, such as scripts and configuration, from Kestra to a Git repository to maintain version control.
Push files in your namespace to a Git Repository with the PushNamespaceFiles Task.
## How it works
The [PushNamespaceFiles](/plugins/plugin-git/io.kestra.plugin.git.pushnamespacefiles) task is a powerful integration that allows you to **push your namespace files to Git from the UI while still managing this process entirely in code**! Kestra unifies the development experience between the UI and code so you can combine the best of both worlds without sacrificing the benefits of version control.
The process is simple: you can **build your flows and files** in a development namespace using all **productivity features of the Kestra UI** (_such as the built-in code editor, autocompletion, syntax validation, documentation, blueprint examples, live-updating topology view, output previews, replays, execution and revision history_) and then **push them to Git** after you have tested and validated them.
The task pushes one or more files from a given namespace (and optionally also child namespaces) to any Git-based Version Control System.
Additionally, the `dryRun` property will help you see what files will be added, modified, or deleted without overwriting the files on Git yet.
The following examples cover common patterns for the `PushNamespaceFiles` task.
## Before you begin
Before you start using the `PushNamespaceFiles` task, ensure the following prerequisites are in place:
1. A Git repository where you want to push your files.
2. A Personal Access Token (PAT) for Git authentication.
3. A running Kestra instance in a version 0.17.0 or later with the PAT stored as a [secret](../../06.concepts/04.secret/index.md) within the Kestra instance.
## Using the `dryRun` property
Start by creating a single `example.py` file in the `dev` namespace and pushing it to a Git repository. Initially set the `dryRun` property to `true` to validate changes before committing to Git. You'll need a flow already in the `dev` namespace to create a new file.
```python
print("Hello, World")
```
Here is a flow that will push the `example.py` file to a Git repository:
```yaml
id: push_to_git
namespace: company.ops
tasks:
- id: commit_and_push
type: io.kestra.plugin.git.PushNamespaceFiles
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/git_username/scripts
branch: dev
namespace: company.team
files:
- "example.py"
gitDirectory: _files
commitMessage: "add namespace files"
dryRun: true
```
Given that the `dryRun` property is set to `true`, the task will only output modifications without pushing any files to Git yet:

## Pushing a single file to Git
Set the `dryRun` property to `false` and push the `example.py` file to Git:
```yaml
id: push_to_git
namespace: company.ops
tasks:
- id: commit_and_push
type: io.kestra.plugin.git.PushNamespaceFiles
...
dryRun: false
```
You should see the following log message:

And here is what you should see in the Outputs tab:

When you click on the commit URL from the logs or from the Outputs tab, you'll be redirected to the commit page on GitHub:

Now, you can create a pull request and merge the changes to the main branch.

## Pushing all files from a single namespace to Git
To push all files from a given namespace to Git, create two more files in the `dev` namespace:
`example.sh` file:
```sh
echo "Hello, World"
```
`example.js` file:
```js
console.log("Hello, World")
```

Adjust the flow to push all files from the `dev` namespace to the `dev` branch:
```yaml
id: push_to_git
namespace: company.ops
tasks:
- id: commit_and_push
type: io.kestra.plugin.git.PushNamespaceFiles
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/git_username/scripts
branch: dev
namespace: company.team
gitDirectory: _files
commitMessage: "push all namespace files and create a PR"
dryRun: true
```
Again, we can set the `dryRun` property to `true` to see what files will be added, modified, or deleted based on the Git version without overwriting the files in Git yet:

Now if you change the `dryRun` property to `false` and run the flow again, you should see all three files being pushed to the `_files` directory on the `develop` branch with the exact commit messages we have specified in the `commitMessage` property:

## Extra notes
- Git does not guarantee the order of push operations to a remote repository, which can lead to potential conflicts when multiple users or flows attempt to push changes simultaneously. To minimize the risk of data loss and merge conflicts, it is strongly recommended to use sequential workflows or push changes to separate branches.
---
# Run Python Inside Your Flows
URL: https://kestra.io/docs/how-to-guides/python
> Run Python scripts in Kestra. Install pip packages at runtime, execute code in Docker containers, and pass data between tasks using inputs and outputs.
Run Python code directly in your flows and generate outputs.
You can execute Python code in a flow by either writing your Python inline or by executing a `.py` file. You can also get outputs and metrics from your Python code too.
In this example, the flow will install the required pip packages, make an API request to fetch data and use the Python Kestra library to generate outputs and metrics using this data.
## Managing Dependencies
Managing Python Dependencies can be frustrating. There's 3 ways you can manage your dependencies in Kestra:
- Install with pip using `beforeCommands`
- Set Container Image with Docker Task Runner
- Build Docker Image and set it with Docker Task Runner
For more information, see the [dedicated guide](../python-dependencies/index.md).
## Script Task
If you want to write a short amount of Python to perform a task, you can use the `io.kestra.plugin.scripts.python.Script` type to write it directly in your flow configuration. This allows you to keep everything in one place.
```yaml
id: python_scripts
namespace: company.team
description: This flow will install the pip package in a Docker container, and use kestra's Python library to generate outputs (number of downloads of the Kestra Docker image) and metrics (duration of the script).
tasks:
- id: outputs_metrics
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: python:slim
dependencies:
- requests
script: |
import requests
def get_docker_image_downloads(image_name: str = "kestra/kestra"):
"""Queries the Docker Hub API to get the number of downloads for a specific Docker image."""
url = f"https://hub.docker.com/v2/repositories/{image_name}/"
response = requests.get(url)
data = response.json()
downloads = data.get('pull_count', 'Not available')
return downloads
downloads = get_docker_image_downloads()
```
You can also include expressions directly in your Python code. In this example, an input is used in the Python method:
```yaml
id: python_scripts_expression_input
namespace: company.team
description: This flow will install the pip package in a Docker container, and use kestra's Python library to generate outputs (number of downloads of the Kestra Docker image) and metrics (duration of the script).
inputs:
- id: image_name
type: STRING
defaults: kestra/kestra
tasks:
- id: outputs_metrics
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: python:slim
dependencies:
- requests
script: |
import requests
def get_docker_image_downloads():
"""Queries the Docker Hub API to get the number of downloads for a specific Docker image."""
url = f"https://hub.docker.com/v2/repositories/{{ inputs.image_name }}/"
response = requests.get(url)
data = response.json()
downloads = data.get('pull_count', 'Not available')
return downloads
downloads = get_docker_image_downloads()
```
## Commands Task
If you would prefer to put your Python code in a `.py` file (e.g. your code is much longer or spread across multiple files), you can run the previous example using the `io.kestra.plugin.scripts.python.Commands` type:
```yaml
id: python_commands
namespace: company.team
description: This flow will install the pip package in a Docker container, and use kestra's Python library to generate outputs (number of downloads of the Kestra Docker image) and metrics (duration of the script).
tasks:
- id: outputs_metrics
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: python:slim
dependencies:
- requests
commands:
- python outputs_metrics.py
```
You'll need to add your Python code using the Editor or [sync it using Git](../../version-control-cicd/04.git/index.md) so Kestra can see it. You'll also need to set the `enabled` flag for the `namespaceFiles` property to `true` so Kestra can access the file.
You can read more about the Commands type in the [Plugin documentation](/plugins/plugin-script-python/io.kestra.plugin.scripts.python.commands).
## Handling Outputs
If you want to get a variable or file from your Python code, you can use an [output](../../05.workflow-components/06.outputs/index.md).
Install the [`kestra` python module](https://pypi.org/project/kestra/) to pass your variables to Kestra. This Kestra Python client provides functionality to interact with the Kestra server for sending metrics, outputs, and logs and executing/polling flows. For example, The Kestra ION extra (`kestra[ion]`) provides a method to read files and convert them to a list of dictionaries, which can be converted into a dataframe in Python (using any Python library supporting dataframes, e.g., Pandas or Polars).
See the [README](https://github.com/kestra-io/libs/blob/main/python/README.md) for more details and examples.
```bash
pip install kestra
```
### Variable Output
You'll need to use the `Kestra` class to pass your variables to Kestra as outputs. Using the `outputs` method, you can pass a dictionary of variables where the `key` is the name of the output you'll reference in Kestra.
Using the same example as above, we can pass the number of downloads as an output.
```python
from kestra import Kestra
import requests
def get_docker_image_downloads(image_name: str = "kestra/kestra"):
"""Queries the Docker Hub API to get the number of downloads for a specific Docker image."""
url = f"https://hub.docker.com/v2/repositories/{image_name}/"
response = requests.get(url)
data = response.json()
downloads = data.get('pull_count', 'Not available')
return downloads
downloads = get_docker_image_downloads()
outputs = {
'downloads': downloads
}
Kestra.outputs(outputs)
```
Once your Python file has executed, you'll be able to access the outputs in later tasks as seen below:
```yaml
id: python_outputs
namespace: company.team
tasks:
- id: outputs_metrics
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: python:slim
dependencies:
- requests
- kestra
commands:
- python outputs_metrics.py
- id: log_downloads
type: io.kestra.plugin.core.log.Log
message: "Number of downloads: {{ outputs.outputs_metrics.vars.downloads }}"
```
_This example works for both `io.kestra.plugin.scripts.python.Script` and `io.kestra.plugin.scripts.python.Commands`._
### File Output
Inside of your Python code, write a file to the system. You'll need to add the `outputFiles` property to your flow and list the file you're trying to access. In this case, we want to access `downloads.txt`. More information on the formats you can use for this property can be found in [Script Output Metrics](../../16.scripts/06.outputs-metrics/index.md).
The example below write a `.txt` file containing the number of downloads, similar the output we used earlier. We can then read the content of the file using the syntax `{{ outputs.{task_id}.outputFiles['{filename}'] }}`
```yaml
id: python_output_files
namespace: company.team
tasks:
- id: outputs_metrics
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: python:slim
dependencies:
- requests
outputFiles:
- downloads.txt
script: |
import requests
def get_docker_image_downloads(image_name: str = "kestra/kestra"):
"""Queries the Docker Hub API to get the number of downloads for a specific Docker image."""
url = f"https://hub.docker.com/v2/repositories/{image_name}/"
response = requests.get(url)
data = response.json()
downloads = data.get('pull_count', 'Not available')
return downloads
downloads = get_docker_image_downloads()
# Generate a file with the output
f = open("downloads.txt", "a")
f.write(str(downloads))
f.close()
- id: log_downloads
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- cat {{ outputs.outputs_metrics.outputFiles['downloads.txt'] }}
```
_This example works for both `io.kestra.plugin.scripts.python.Script` and `io.kestra.plugin.scripts.python.Commands`._
## Capture Logs
If your Python code needs to log something to the console, use the `Kestra.logger()` method from the [Kestra pip package](https://github.com/kestra-io/libs) to instantiate a `logger` object — this logger is configured to correctly capture all Python log levels and send them to the Kestra backend.
```yaml
id: python_logs
namespace: company.team
tasks:
- id: python_logger
type: io.kestra.plugin.scripts.python.Script
allowFailure: true
script: |
import time
from kestra import Kestra
logger = Kestra.logger()
logger.debug("DEBUG is used for diagnostic info.")
time.sleep(0.5)
logger.info("INFO confirms normal operation.")
time.sleep(0.5)
logger.warning("WARNING signals something unexpected.")
time.sleep(0.5)
logger.error("ERROR indicates a serious issue.")
time.sleep(0.5)
logger.critical("CRITICAL means a severe failure.")
```
When we execute the above example, we can see Kestra correctly captures the log levels in the Logs view:

You can read more about the Python Script task in the [Plugin documentation](/plugins/plugin-script-python/io.kestra.plugin.scripts.python.script)
## Handling Metrics
You can also get [metrics](../../16.scripts/06.outputs-metrics/index.md#outputs-and-metrics-in-script-and-commands-tasks) from your Python code. In this example, we can use the `time` module to time the execution time of the function and then pass this to Kestra so it can be viewed in the Metrics tab. You don't need to modify your flow in order for this to work.
```python
from kestra import Kestra
import requests
import time
start = time.perf_counter()
def get_docker_image_downloads(image_name: str = "kestra/kestra"):
"""Queries the Docker Hub API to get the number of downloads for a specific Docker image."""
url = f"https://hub.docker.com/v2/repositories/{image_name}/"
response = requests.get(url)
data = response.json()
downloads = data.get('pull_count', 'Not available')
return downloads
downloads = get_docker_image_downloads()
end = time.perf_counter()
outputs = {
'downloads': downloads
}
Kestra.outputs(outputs)
Kestra.timer('duration', end - start)
```
Once this has executed, `duration` will be viewable under **Metrics**.

## Execute Flows in Python
Inside of your Python code, you can execute flows. This is useful if you want to manage your orchestration directly in Python rather than using the Kestra flow editor. However, using [Subflows](../../05.workflow-components/10.subflows/index.md) to execute flows from other flows provides a more integrated experience.
You can trigger a flow execution by calling the `execute()` method. Here is an example for the same `python_scripts` flow in the namespace `example` as above:
```python
from kestra import Flow
os.environ["KESTRA_HOSTNAME"] = "http://host.docker.internal:8080" # Set this when executing this Python code inside Kestra
flow = Flow()
flow.execute('example', 'python_scripts', {'greeting': 'hello from Python'})
```
Read more about it on the [execution page](../../05.workflow-components/03.execution/index.md).
## Automate Python with triggers
You can combine your Python code with a trigger to automatically execute your code. There's a few key ways you can automate it:
- Run on a schedule
- Run when a webhook is called
- Run when a file is available in a data lake or storage bucket
- Run Python code on a polling interval and emit only when a condition matches
### Run on a schedule
You can use the [Schedule Trigger](../../05.workflow-components/07.triggers/01.schedule-trigger/index.md) to run your flow on a routine. You can pass the date that the trigger executed to your code through an expression. This is useful when using backfills as it allows you to pass the date of when the execution was suppose to run from the schedule directly to your code, rather than the current time, for example useful when fetching a daily report from a specific date in the past:
```yaml
id: schedule_code
namespace: company.team
tasks:
- id: python
type: io.kestra.plugin.scripts.python.Script
outputFiles:
- "*.txt"
script: |
date = f"{{ trigger.date | date("yyyy-MM-dd") }}"
report_content = f"Daily Report - {date}\nSales: $5000\nUsers: 200"
with open(f"daily_report_{date}.txt", "w") as file:
file.write(report_content)
triggers:
- id: schedule
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 8 * * *"
```
### Run when a webhook is called
You can use the [Webhook Trigger](../../05.workflow-components/07.triggers/03.webhook-trigger/index.md) to run your flow when a webhook is called. You can also call the webhook with a body, which can be passed to your code through an expression or environment variable:
```yaml
id: python_webhook
namespace: company.team
tasks:
- id: python
type: io.kestra.plugin.scripts.python.Script
script: |
response = {{ trigger.body ?? '' }}
print(f"{response['first_name']} {response['last_name']}")
triggers:
- id: webhook
type: io.kestra.plugin.core.trigger.Webhook
key: abcdefg
```
### Run when a file is available in a data lake or storage bucket
You can use a [Polling Trigger](../../05.workflow-components/07.triggers/04.polling-trigger/index.md), such as the [S3 Trigger](/plugins/plugin-aws/aws-s3/io.kestra.plugin.aws.s3.trigger) to run your flow when a new file arrives in an S3 bucket. This is useful if you have a data pipeline that can start once the data is available and begin transforming it with Python:
```yaml
id: s3_trigger
namespace: company.team
tasks:
- id: process_data
type: io.kestra.plugin.scripts.python.Script
containerImage: python:slim
dependencies:
- pandas
- kestra
inputFiles:
input.csv: "{{ read(trigger.objects[0].uri) }}"
outputFiles:
- data.csv
script: |
import os
import pandas as pd
from kestra import Kestra
df = pd.read_csv('input.csv')
df['discounted_total'] = df['total'] * 0.9
df.to_csv('data.csv')
triggers:
- id: watch
type: io.kestra.plugin.aws.s3.Trigger
interval: "PT1S"
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "eu-west-2"
bucket: "kestra-python-s3"
action: DELETE
filter: FILES
maxKeys: 1
```
### Run Python code as a polling trigger
You can also use Python itself as polling logic by using `ScriptTrigger` or `CommandsTrigger`. These trigger types run Python code on an interval and start a flow execution only when the `exitCondition` matches.
Use `ScriptTrigger` for inline Python code:
```yaml
id: python_script_trigger
namespace: company.team
triggers:
- id: script_failure
type: io.kestra.plugin.scripts.python.ScriptTrigger
interval: PT10S
exitCondition: "exit 1"
edge: true
script: |
raise Exception("boom")
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "Triggered with exitCode={{ trigger.exitCode }} (condition={{ trigger.condition }})"
```
Use `CommandsTrigger` when you want to run Python commands instead:
```yaml
id: python_commands_trigger
namespace: company.team
triggers:
- id: on_fail
type: io.kestra.plugin.scripts.python.CommandsTrigger
interval: PT10S
exitCondition: "exit 1"
edge: true
containerImage: python:3.13-slim
commands:
- python3 -c "raise Exception('boom')"
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "Triggered with exitCode={{ trigger.exitCode }} (condition={{ trigger.condition }})"
```
These triggers support:
- `interval` to control how often the Python code runs
- `exitCondition` to match an exit code such as `exit 1`, or a regex or substring matched against emitted vars and failure logs
- `edge` to emit only when the condition changes from not matching to matching
Use these trigger types when you want Python itself to decide whether a polling condition has been met, rather than relying on a separate external-system trigger.
## Execute GraalVM Task
Kestra also supports GraalVM integration, allowing you to execute Python code directly on the JVM, with the potential for performance improvements. There are currently two tasks:
- [Eval](/plugins/plugin-graalvm/python-graalvm/io.kestra.plugin.graalvm.python.eval)
- [FileTransform](/plugins/plugin-graalvm/python-graalvm/io.kestra.plugin.graalvm.python.filetransform)
In this example, the `Eval` task is used to manipulate data from a previous task. GraalVM makes it easy to generate outputs from variables in Python using the `outputs` property. This is useful if you want to manipulate data and pass the new format to another task.
```yaml
id: parse_json_data
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: http://xkcd.com/info.0.json
- id: graal
type: io.kestra.plugin.graalvm.python.Eval
outputs:
- data
script: |
data = {{ read(outputs.download.uri) }}
data["next_month"] = int(data["month"]) + 1
```
---
# Manage Python Dependencies in Kestra
URL: https://kestra.io/docs/how-to-guides/python-dependencies
> Learn various ways to manage Python dependencies in Kestra, including using pip, virtual environments, caching, and custom Docker images.
Manage your Python dependencies in Kestra.
Managing Python Dependencies can be frustrating. There's several ways you can manage your dependencies in Kestra.
## Install with pip using `beforeCommands`
Before your `Script` and `Commands` tasks, you can add a list of commands under the `beforeCommands` property. This works well for installing packages with `pip` or setting up a virtual environment:
```yaml
id: beforecommands
namespace: company.team
tasks:
- id: code
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.core.runner.Process
beforeCommands:
- python3 -m venv .venv
- . .venv/bin/activate
- pip install pandas kestra
script: |
import pandas as pd
from kestra import Kestra
df = pd.read_csv('https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv')
total_revenue = df['total'].sum()
Kestra.outputs({"total": total_revenue})
```
By using a [Process Task Runner](../../task-runners/04.types/01.process-task-runner/index.md), we can speed up the execution time so that our task isn't pulling a container image to run the task in a container.
## Cache dependencies
:::badge{version=">=0.23" editions="OSS,EE"}
:::
Since Kestra 0.23, you can also use the `dependencies` property allowing you to cache Python dependencies across multiple executions.
With this feature, Python dependencies are cached and reused across executions of different flows. The [uv package manager](../python-uv/index.md) installs the dependencies on the [worker](../../08.architecture/02.server-components/index.md#worker) under the hood. These cached dependencies will be available for subsequent executions, leading to performance improvements. This method is recommended for smaller tasks that require only a few dependencies, which you don't want to add each time. For more complex workflows, you can continue to use `beforeCommands`.
The added properties are `dependencies`, which lists the dependencies (e.g., pandas), and `dependencyCacheEnabled`, which, when set to true, enables caching of dependencies across tasks. An example flow is as follows: the first execution installs the dependencies, but each subsequent execution of this flow, or any other flow relying on these packages, will show improved performance.
```yaml
id: python_dependencies
namespace: company-team
tasks:
- id: python
type: io.kestra.plugin.scripts.python.Script
containerImage: python:3.13-slim
dependencies:
- pandas
- kestra
script: |
from kestra import Kestra
import pandas as pd
data = {
'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [25, 30, 35]
}
df = pd.DataFrame(data)
print(df)
print("Average age:", df['Age'].mean())
Kestra.outputs({"average_age": df['Age'].mean()})
```
## Set Container Image with Docker Task Runner
If we would prefer to run our task in a container, we can set our Task Runner to Docker and specify a container image with the appropriate dependencies bundled in. Our previous example used `pandas` which is bundled into the `ghcr.io/kestra-io/pydata:latest` available as one of the ready to go images on our [GitHub](https://github.com/orgs/kestra-io/packages?repo_name=examples).
```yaml
id: container_image
namespace: company.team
tasks:
- id: code
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/pydata:latest
script: |
import pandas as pd
from kestra import Kestra
df = pd.read_csv('https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv')
total_revenue = df['total'].sum()
Kestra.outputs({"total": total_revenue})
```
## Build Docker Image and set it with Docker Task Runner
If an image with the required dependencies isn't available, build your own using the `docker.Build` task.
Specify a Dockerfile that uses a `python:3.10` image as the base and installs the required dependencies on top.
The example below uses `pip install` to install both `kestra` and `pandas`. Once the image is built, reference it in an expression in the Python task:
```yaml
id: container_image_build
namespace: company.team
tasks:
- id: build
type: io.kestra.plugin.docker.Build
dockerfile: |
FROM python:3.10
RUN pip install --upgrade pip
RUN pip install --no-cache-dir kestra pandas
tags:
- python_image
- id: code
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
pullPolicy: NEVER
containerImage: "{{ outputs.build.imageId }}"
script: |
import pandas as pd
from kestra import Kestra
df = pd.read_csv('https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv')
total_revenue = df['total'].sum()
Kestra.outputs({"total": total_revenue})
```
## Build Custom Packages
You can also build packages directly in Kestra and then use that package between different flows in the same namespace. This works for zip files and wheels.
Here's an example that generates a `.tar.gz` package:
```yaml
id: build_tar_gz
namespace: company
tasks:
- id: sync_code_to_kestra
type: io.kestra.plugin.git.SyncNamespaceFiles
disabled: true # already synced files
namespace: "{{ flow.namespace }}"
gitDirectory: .
url: https://github.com/anna-geller/python-in-kestra
branch: main
username: anna-geller
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
- id: build
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
beforeCommands:
- pip install build
commands:
- python -m build
outputFiles:
- "**/*.tar.gz"
- id: upload
type: io.kestra.plugin.core.namespace.UploadFiles
namespace: company.sales
filesMap:
"etl-0.1.0.tar.gz": "{{ outputs.build.outputFiles['dist/etl-0.1.0.tar.gz']}}"
```
The package can be used in a separate workflow:
```yaml
id: install_from_zip
namespace: company.sales
inputs:
- id: date
type: STRING
defaults: 12/24/2024
displayName: Delivery Date
tasks:
- id: python
type: io.kestra.plugin.scripts.python.Script
namespaceFiles:
enabled: true
beforeCommands:
- pip install etl-0.1.0.tar.gz
script: |
import etl.utils as etl
out = etl.standardize_date_format("{{ inputs.date }}")
print(out)
```
---
# Manage Python Dependencies with uv in Kestra
URL: https://kestra.io/docs/how-to-guides/python-uv
> Use uv in Kestra to manage Python dependencies and virtual environments for faster and more reliable script execution.
Manage Python dependencies in Kestra using `uv`.
:::alert{type="info"}
For most use cases, the native `dependencies` property on Python `Script` and `Commands` tasks is the simplest way to install packages — Kestra handles installation automatically without needing `beforeCommands`. Use `uv` when you need faster resolution, virtual environments with the Process runner, or a custom Docker image with `uv` pre-installed.
:::
`uv` is a fast Python package and project manager written in Rust. It combines tools like `virtualenv`, `poetry`, and `pip` into one unified interface and is pre-installed in Kestra's default Python image `kestrapy`.
`uv` is useful in Kestra for managing virtual environments with the [Process Task Runner](../../task-runners/04.types/01.process-task-runner/index.md) or when you need explicit control over dependency resolution speed.
## Install Dependencies
By default, Kestra has `uv` installed to our default Python image `kestrapy`, so anytime you use a Python `Commands` or `Script` task with the [Docker Task Runner](../../task-runners/04.types/02.docker-task-runner/index.md), it will be preinstalled.
If you're using a different image or you'd prefer to use the [Process Task Runner](../../task-runners/04.types/01.process-task-runner/index.md), you can also install `uv` using `beforeCommands` with `pip install uv`.
```yaml
id: python_example
namespace: company.team
tasks:
- id: code_process
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.core.runner.Process
beforeCommands:
- pip install uv 2> /dev/null
script: |
print("Hello, World!")
```
By default, `uv` will look for a virtual environment to install dependencies into, but this is not required when using the [Docker Task Runner](../../task-runners/04.types/02.docker-task-runner/index.md) as that provides the isolation we would get from a virtual environment. To override this, we can add the `--system` flag to our install command.
```yaml
id: python_example
namespace: company.team
tasks:
- id: code
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
beforeCommands:
- uv pip install pandas --system 2> /dev/null
script: |
import pandas as pd
from kestra import Kestra
df = pd.read_csv('https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv')
total_revenue = df['total'].sum()
Kestra.outputs({"total": total_revenue})
```
If you're using the [Process Task Runner](../../task-runners/04.types/01.process-task-runner/index.md), you can use `uv` to create a virtual environment with `uv venv`. Once this has completed, you can run `uv pip install`, and it will automatically install these dependencies to this virtual environment without needing to activate the virtual environment.
```yaml
id: python_example
namespace: company.team
tasks:
- id: code_process
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.core.runner.Process
beforeCommands:
- pip install uv 2> /dev/null
- uv venv 2> /dev/null
- uv pip install pandas kestra 2> /dev/null
- . .venv/bin/activate
script: |
import pandas as pd
from kestra import Kestra
df = pd.read_csv('https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv')
total_revenue = df['total'].sum()
Kestra.outputs({"total": total_revenue})
```
## Install with a custom Docker image
If you have multiple workflows using `uv`, you can install it on the Kestra server by creating a custom Docker image for Kestra. Here's an example of a Dockerfile which is based off the Kestra image but installs `uv` on top of it.
```dockerfile
FROM kestra/kestra:latest
USER root
RUN pip install uv
CMD ["server", "standalone"]
```
Learn more about [installing pip package dependencies at server startup](../../14.best-practices/4.managing-pip-dependencies/index.md#installing-pip-package-dependencies-at-server-startup).
---
# Realtime Triggers in Kestra: Kafka, SQS, Pub/Sub
URL: https://kestra.io/docs/how-to-guides/realtime-triggers
> React to events instantly with Kestra's Realtime Triggers for Kafka, Pulsar, AWS SQS, GCP Pub/Sub, and Azure Event Hubs.
How to React to events as they happen with millisecond latency.
As soon as you add a Realtime Trigger to your workflow, Kestra starts an always-on thread that listens to the external system for new events. When a new event occurs, Kestra starts a workflow execution to process the event.
The following examples show how to implement Realtime Triggers for common messaging systems.
## Apache Kafka
To setup Apache Kafka locally, follow the instructions mentioned in the [official documentation](https://kafka.apache.org/quickstart). Once Apache Kafka is installed, you can create the `logs` topic, and start producing data into the topic using the following commands:
```bash
## Create topic
$ bin/kafka-topics.sh --create --topic logs --bootstrap-server localhost:9092
## Produce data into Kafka topic
$ bin/kafka-console-producer.sh --topic logs --bootstrap-server localhost:9092
> Hello World
```
You can use the Apache Kafka [RealtimeTrigger](/plugins/plugin-kafka/io.kestra.plugin.kafka.realtimetrigger) in the Kestra flow as follows:
```yaml
id: kafka
namespace: company.team
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "{{ trigger.value }}"
triggers:
- id: realtime_trigger
type: io.kestra.plugin.kafka.RealtimeTrigger
topic: logs
properties:
bootstrap.servers: localhost:9092
serdeProperties:
valueDeserializer: STRING
groupId: kestraConsumerGroup
```
When any message is pushed into the `logs` Kafka topic, this flow will get triggered immediately.
## Apache Pulsar
To setup Apache Pulsar locally, you can install the [standalone cluster](https://pulsar.apache.org/docs/next/getting-started-standalone/) or [docker cluster](https://pulsar.apache.org/docs/next/getting-started-docker/) for Apache Pulsar. For docker cluster, you can run the `pulsar-admin` commands from the Apache Pulsar docker container.
You can run the following commands to create the topic, and produce data to the topic:
1) Setup a tenant
`bin/pulsar-admin tenants create apache`
2) Create a namespace
`bin/pulsar-admin namespaces create apache/pulsar`
3) Create a topic
`bin/pulsar-admin topics create-partitioned-topic apache/pulsar/logs -p 4`
4) Produce data to topic
`bin/pulsar-client produce apache/pulsar/logs -m '--Hello World--' -n 1`
You can use the Apache Pulsar [RealtimeTrigger](/plugins/plugin-pulsar/io.kestra.plugin.pulsar.realtimetrigger) in the Kestra flow as follows:
```yaml
id: pulsar
namespace: company.team
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "{{ trigger.value }}"
triggers:
- id: realtime_trigger
type: io.kestra.plugin.pulsar.RealtimeTrigger
topic: apache/pulsar/logs
uri: pulsar://localhost:26650
subscriptionName: kestra_trigger_sub
```
When any message is pushed into the `apache/pulsar/logs` Pulsar topic, this flow will get triggered immediately.
## AWS SQS
We will first create the SQS queue from the AWS Console. You can also AWS CLI for this purpose. This is how you can create the AWS SQS from the console:

You only need to put in the Queue name. Rest all the configuration can be kept as is, and click on "Create Queue" at the bottom of the page.
You can now send messages to this queue by clicking on "Send and receive messages" button on the top of the page.

On the Send and Receive messages page, you can put the Message body under the Send message section, and click on the "Send message" button to send the message to the queue.

You can use the AWS SQS [RealtimeTrigger](/plugins/plugin-aws/aws-sqs/io.kestra.plugin.aws.sqs.realtimetrigger) in the Kestra flow as follows:
```yaml
id: aws-sqs
namespace: company.team
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "{{ trigger.data }}"
triggers:
- id: "realtime_trigger"
type: io.kestra.plugin.aws.sqs.RealtimeTrigger
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "eu-central-1"
queueUrl: "https://sqs.eu-central-1.amazonaws.com/000000000000/logs"
```
When any message is pushed into the `logs` SQS queue, this flow will get triggered immediately.
## GCP Pub/Sub
We will first create the Pub/Sub topic from the GCP console. For this, click on "Create topic" button on the GCP Pub/Sub console. On the Create topic page, put the topic name `logs` in the Topic ID text box, and leave the rest of the settings as default. Ensure the "Add a default subscription" checkbox is ticked. Click on "CREATE" button at the bottom. This will create the `logs` Pub/Sub topic with the default subscription `logs-sub`.


Navigate to the "MESSAGES" tab. On this tab, click on the "PUBLISH MESSAGE" button.

On the Publish message popup, put the message you would like to publish to the topic, and click on the "PUBLISH" button on the bottom of the page. This would publish the message to the Pub/Sub topic.

You can use the GCP Pub/Sub [RealtimeTrigger](/plugins/plugin-gcp/google-cloud-pubsub/io.kestra.plugin.gcp.pubsub.realtimetrigger) in the Kestra flow as follows:
```yaml
id: gcp-pubsub
namespace: company.team
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "{{ trigger.data }}"
triggers:
- id: trigger
type: io.kestra.plugin.gcp.pubsub.RealtimeTrigger
projectId: test-project-id
topic: logs
subscription: logs-sub
```
When any message is published into the `logs` Pub/Sub topic, this flow will get triggered immediately.
## Azure Event Hubs
Create an Event Hub and a container for checkpoint storage:
1. Go to [Event Hubs](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.EventHub%2Fnamespaces) in the Azure portal.
2. Click on "Create" to create an Event Hubs namespace.
3. On the Create Namespace page, choose an appropriate Subscription and Resource Group.
4. Put an appropriate Namespace name, Location, Pricing tier and Throughput units.
5. Click on "Review + Create". Once the validation is successful, click on "Create".
6. Once the Event Hub namespace is created, click on the namespace.
7. Once on that particular namespace's page, click on "+ Event Hub" button to create an Event Hub.
8. Put an appropriate Name for the Event Hub. You can change the remaining settings as per your requirements.
9. Click on "Review + Create". Once the validation is successful, click on "Create".
10. On the particular Event Hub namespace page, you can now see the newly created Event Hub.
11. On the namespace page, click on "Shared access policies" from the left menu bar.
12. Click on the "RootManageSharedAccessKey".
13. In the popup page that appears, you can copy the "Connection string – primary key" to be used later in the Kestra flow. With this, the Event Hub is created.










14. Create the container. Go to [Storage accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts).
15. Click on "Create storage account".
16. On the "Create storage account" page, choose an appropriate Subscription and Resource Group.
17. Put an appropriate Storage account name, Region, Performance, and Redundancy.
18. Click on "Review + Create". Once the validation is successful, click on "Create".
19. Once the storage account is created, click on the storage account name.
20. On the storage account page, navigate from the left menu bar to the "Data storage", and then to "Containers".
21. Click on the "+ Container" button to create a container.
22. Put an appropriate name for the container, and click "Create".
23. Once the container is created, navigate to "Access keys" under "Security + networking" from the left menu bar.
24. For the key, click on the "Show" button for the connection string and note it down to be used later in the Kestra flow.








Now that all the setup is ready in Azure, start the Kestra cluster with the environment variables "SECRET_EVENTHUBS_CONNECTION" and "SECRET_BLOB_CONNECTION" containing the base64-encoded value for the Event Hubs connection string and Blob connection string, respectively.
The Kestra flow with the Azure Event Hub Realtime Trigger will look as follows:
```yaml
id: TriggerFromAzureEventHubs
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Hello there! I received {{ trigger.body }} from Azure EventHubs!
triggers:
- id: readFromEventHubs
type: io.kestra.plugin.azure.eventhubs.RealtimeTrigger
eventHubName: kestra
namespace: kestra-namespace
connectionString: "{{ secret('EVENTHUBS_CONNECTION') }}"
bodyDeserializer: JSON
consumerGroup: "$Default"
checkpointStoreProperties:
containerName: kestralogs
connectionString: "{{ secret('BLOB_CONNECTION') }}"
```
On the particular Event Hubs page, you can click on "Generate Data" under "Features" from the left menu bar. Choose an appropriate Content-Type from the drop-down, and put the payload you want to push to the Event Hub. When you click on the "Send" button on the bottom, the payload will be pushed to the Event Hub. Also, the flow will be triggered immediately, and you can see the corresponding execution in Kestra.


Realtime triggers let you react to events in real time to orchestrate business-critical processes.
---
# Build SecOps Workflows with Kestra
URL: https://kestra.io/docs/how-to-guides/secops-with-kestra
> Automate security operations with Kestra. Build SecOps workflows for incident response, vulnerability scanning, and compliance automation.
Operationalize SecOps benchmarks with Kestra.
This how-to shows how to operationalize SecOps benchmarks with Kestra. You will download a CIS benchmark, store control recommendations as settings, and orchestrate compliance scans and automated remediation across multiple controls and teams.
## Prerequisites
- Access to the CIS benchmark for your target operating system (Ubuntu 24.04 LTS in this example)
- A Kestra namespace strategy for SecOps (for example `company.security.cis.linux.ubuntu.22-04-lts.devops`)
- SSH access (public key) to the target VMs you plan to scan/remediate
- Appropriate secrets configured in Kestra for usernames, private keys, and webhook triggers
## Step 1: Download the Benchmark
1. Go to [https://downloads.cisecurity.org/#/](https://downloads.cisecurity.org/#/) and download the **CIS_Ubuntu_Linux_24.04_LTS_Benchmark_v1.0.0** (or the benchmark that matches your OS).
2. Review the controls you plan to enforce and note the recommended settings.

## Step 2: Define the Namespace and Settings Structure
1. Decide how to segment namespaces per team or environment. Examples:
- `company.security.cis.linux.ubuntu.22-04-lts.devops`
- `company.security.cis.linux.ubuntu.22-04-lts.dataeng`
2. Create settings ([KV pairs](../../06.concepts/05.kv-store/index.md)) for every control you want to validate. For instance, controls under section **1.6**:

And they can be stored by following this hierarchy:
```plaintext
1
├── 1.1
│ ├── 1.1.1
│ │ ├── 1.1.1.1
│ │ └── 1.1.1.2
└── 1.6
└── 1.6.4
```
3. Use consistent KV naming so any flow can dynamically fetch a control setting. Example naming convention: `control-1-1_6-1_6_4` for control **1.6.4**.
4. Store the recommended permission string or configuration snippet for each control. Control 1.6.4, for example, ensures `/etc/motd` permissions follow security guidance.

Repeat this process for every control you intend to enforce. The walkthrough below focuses on 1.6.4, 1.6.5, and 1.6.6.
## Step 3: Store Secrets for VM Access
1. Add [secrets](../../06.concepts/04.secret/index.md) for the SSH username (`vmUser`) and private key (`vmKey`) used to connect to the VM.
2. Store any additional secrets (for example, webhook secrets) you will reference in flows and triggers.


## Step 4: Model the Parent Flow
Design a flow that evaluates each control, remediates if required, and proceeds to the next control. At a high level the logic looks like this:
```plaintext
Start → Execute Control 1.6.4 → Assess Compliance
→ If compliant → Move to next control
→ If not compliant → Remediate → Re-assess → Next control
```

## Step 5: Create Reusable Control Subflows
Create a subflow per control so you can reuse the same logic across namespaces. The example below implements control **1.6.5**. Note how periods in the control number are converted to underscores for IDs (for example, `1_6_5`).
```yaml
id: control-1-1_6-1_6_5
namespace: company.security.cis.linux.ubuntu.22-04-lts.devops
inputs:
- id: remediateControls
description: Toggle ON to auto-remediate non-compliant controls.
displayName: Auto Remediate
type: BOOL
defaults: true
- id: ipAddress
type: STRING
defaults: localhost
variables:
COMPLIANT: Compliant
NOT_COMPLIANT: Not Compliant
tasks:
# Retrieve the recommended configuration from the KV store
- id: getConfiguration
type: io.kestra.plugin.core.kv.Get
key: "{{ render(flow.id) }}"
# Assess the current VM state
- id: assess-1_6_5
type: io.kestra.plugin.fs.ssh.Command
host: "{{ inputs.ipAddress }}"
authMethod: PUBLIC_KEY
username: "{{ secret('vmUser') }}"
privateKey: "{{ secret('vmKey') }}"
commands:
- 'echo $(stat -Lc "Access: (%#a/%A) Uid: ( %u/ %U) Gid: { %g/ %G)" /etc/issue) > output.log'
- echo '::{"outputs":{"result":"'$(cat output.log)'"}}::'
- id: status-1_6_5
type: io.kestra.plugin.core.flow.If
condition: "{{ outputs['assess-1_6_5']['vars']['result'] == outputs.getConfiguration.value }}"
then:
- id: compliant-1_6_5
type: io.kestra.plugin.core.debug.Return
format: "{{ vars.COMPLIANT }}"
else:
- id: doRemediate
type: io.kestra.plugin.core.flow.If
condition: "{{ inputs.remediateControls == true }}"
then:
- id: remediate-1_6_5
type: io.kestra.plugin.fs.ssh.Command
host: "{{ inputs.ipAddress }}"
username: "{{ secret('vmUser') }}"
privateKey: "{{ secret('vmKey') }}"
authMethod: PUBLIC_KEY
commands:
- sudo chown root:root $(readlink -e /etc/issue)
- sudo chmod u-x,go-wx $(readlink -e /etc/issue)
- id: remediateResult-1_6_5
type: io.kestra.plugin.core.debug.Return
format: "{{ vars.COMPLIANT }}"
else:
- id: not-compliant-1_6_5
type: io.kestra.plugin.core.debug.Return
format: "{{ vars.NOT_COMPLIANT }}"
## Return output for the parent flow
outputs:
- id: complianceStatus-1_6_5
type: STRING
value: "{{ outputs['compliant-1_6_5']['value'] ?? outputs['remediateResult-1_6_5']['value'] ?? outputs['not-compliant-1_6_5']['value'] ?? 'Error' }}"
```
Repeat the same pattern for controls **1.6.4** and **1.6.6**.
## Step 6: Assemble the Parent Flow
Use the subflows inside a parent orchestration that evaluates each control sequentially within a `Parallel` task (with concurrency set to 1). This lets you retrigger individual control branches without re-running the entire benchmark.
```yaml
id: csrRevamped
namespace: company.security.cis.linux.ubuntu.22-04-lts.devops
inputs:
- id: remediateControls
description: Toggle ON to auto-remediate non-compliant controls.
displayName: Auto Remediate
type: BOOL
defaults: true
- id: ipAddress
displayName: IP Address
description: Host on which the scan must run.
type: STRING
defaults: localhost
tasks:
- id: section-1-1_6
type: io.kestra.plugin.core.flow.Parallel
# Tasks run in parallel but concurrency is limited to 1 so each control
# can be retriggered independently without re-running downstream steps.
concurrent: 1
tasks:
- id: trigger-1-1_6-1_6_4
type: io.kestra.plugin.core.flow.Sequential
tasks:
- id: control-1-1_6-1_6_4
type: io.kestra.plugin.core.flow.Subflow
namespace: "{{ flow.namespace }}"
flowId: control-1-1_6-1_6_4
inputs:
ipAddress: "{{ inputs.ipAddress }}"
remediateControls: "{{ inputs.remediateControls }}"
wait: true
transmitFailed: true
- id: logStatus-1-1_6-1_6_4
type: io.kestra.plugin.core.log.Log
message: "{{ outputs['control-1-1_6-1_6_4'].outputs['complianceStatus-1_6_4'] }}"
- id: trigger-1-1_6-1_6_5
type: io.kestra.plugin.core.flow.Sequential
tasks:
- id: control-1-1_6-1_6_5
type: io.kestra.plugin.core.flow.Subflow
namespace: "{{ flow.namespace }}"
flowId: control-1-1_6-1_6_5
inputs:
ipAddress: "{{ inputs.ipAddress }}"
remediateControls: "{{ inputs.remediateControls }}"
wait: true
transmitFailed: true
- id: logStatus-1-1_6-1_6_5
type: io.kestra.plugin.core.log.Log
message: "{{ outputs['control-1-1_6-1_6_5'].outputs['complianceStatus-1_6_5'] }}"
## These triggers will be demonstrated in the VM creation and ServiceNow tutorial
triggers:
- id: vmCreateFromServiceNow
type: io.kestra.plugin.core.trigger.Webhook
key: "{{ secret('webHookTriggerSecret') }}"
- id: postVMCreation
type: io.kestra.plugin.core.trigger.Flow
inputs:
ipAddress: "{{ trigger.outputs.createVMRevamped.externalIPAddress }}"
dependsOn:
- namespace: company.ops.it
flowId: createVMRevamped
states: [SUCCESS, WARNING]
```
## Step 7: Review the Topology
Each control runs in parallel but only one at a time because `concurrent: 1`. This makes it easy to rerun non-compliant controls individually without re-running the entire benchmark.

## Demo
1. **Execute the flow.** Observe the initial compliance check.

2. **Check the results.** Review the compliance summary.

3. **Inspect the subflow.** Confirm whether the VM was already compliant.

4. **Force a drift.** Change the VM setting for control `1_6_5` (for example, from `644` to `664`).

5. **Retrigger only control `1_6_5`.**

6. **Review the logs.** Verify that remediation executed for `1_6_5`.

7. **Validate the VM permissions.** Confirm they returned to `644`.

## Result
You have enforced CIS benchmark controls through Kestra, combined compliance assessment with optional remediation, and validated that individual controls can be retriggered independently. Replace the placeholder images with real screenshots from your environment to complete the documentation.
---
# Configure Secrets in Kestra
URL: https://kestra.io/docs/how-to-guides/secrets
> Learn how to securely configure and use secrets in Kestra to protect sensitive information like passwords and API keys in your flows.
Configure sensitive values as base64-encoded environment variables so flows can reference them without exposing plain text in flow definitions or logs.
For a detailed overview of how secrets work across editions, see the [Secrets concept page](../../06.concepts/04.secret/index.md). For a UI-based secrets manager with access controls and audit logs, see the [Enterprise Edition](../../oss-vs-paid/index.md).
---
## Using secrets in Kestra
### Step 1: Create a `.env` file
Start by defining your secrets in a standard environment file:
```bash
POSTGRES_PASSWORD=actual_postgres_password
OPENAI_KEY=actual_openai_key
AWS_ACCESS_KEY=actual_aws_access_key
AWS_SECRET_KEY=actual_aws_secret_key
```
### Step 2: Encode and prefix your secrets
Kestra expects all secret keys to be **prefixed with `SECRET_`** and their values **base64-encoded**.
The resulting `.env_encoded` file should look like this:
```bash
SECRET_POSTGRES_PASSWORD=base64_encoded_postgres_password
SECRET_OPENAI_KEY=base64_encoded_openai_key
SECRET_AWS_ACCESS_KEY=base64_encoded_aws_access_key
SECRET_AWS_SECRET_KEY=base64_encoded_aws_secret_key
```
To generate this file automatically, use the following Bash script:
```bash
while IFS='=' read -r key value; do
echo "SECRET_$key=$(echo -n "$value" | base64)";
done < .env > .env_encoded
```
This script:
1. Base64-encodes all values.
2. Adds the `SECRET_` prefix to all variable names.
3. Saves the result as `.env_encoded`.
You can verify the output by opening `.env_encoded` — it should look like the example above.
Alternatively, you can manually write the file using macros to encode secrets dynamically:
```bash
SECRET_POSTGRES_PASSWORD={{ "actual_postgres_password" | base64encode }}
SECRET_OPENAI_KEY={{ "actual_openai_key" | base64encode }}
SECRET_AWS_ACCESS_KEY={{ "actual_aws_access_key" | base64encode }}
SECRET_AWS_SECRET_KEY={{ "actual_aws_secret_key" | base64encode }}
```
---
### Step 3: Point Docker to the encoded file
Update your `docker-compose.yaml` to use the `.env_encoded` file:
```yaml
kestra:
image: kestra/kestra:latest
env_file:
- .env_encoded
```
This ensures your secrets are loaded when Kestra starts.
---
### Step 4: Use secrets in a flow
Once your secrets are loaded, reference them in your flows using the `secret()` function — **without** including the `SECRET_` prefix.
For example, this flow connects to PostgreSQL using `SECRET_POSTGRES_PASSWORD` and uploads query results to AWS S3 using `SECRET_AWS_ACCESS_KEY` and `SECRET_AWS_SECRET_KEY`.
```yaml
id: postgres_to_s3
namespace: company.team
tasks:
- id: fetch
type: io.kestra.plugin.jdbc.postgresql.Query
url: jdbc:postgresql://127.0.0.1:56982/
username: pg_user
password: "{{ secret('POSTGRES_PASSWORD') }}"
sql: select id, first_name, last_name, city from users
fetchType: STORE
- id: write_to_s3
type: io.kestra.plugin.aws.s3.Upload
accessKeyId: "{{ secret('AWS_ACCESS_KEY') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY') }}"
region: "eu-central-1"
from: "{{ outputs.fetch.uri }}"
bucket: "kestra-bucket"
key: "data/users.csv"
```
---
### How secrets are resolved
When you reference a secret using `{{ secret('POSTGRES_PASSWORD') }}`, Kestra finds the corresponding `SECRET_POSTGRES_PASSWORD` environment variable, base64-decodes its value, and injects it into the execution context. The decoded value does not appear in flow definitions or logs, but base64 encoding is not encryption — secure your host environment accordingly.
---
# Install Only Selected Plugins in Kestra OSS
URL: https://kestra.io/docs/how-to-guides/selected-plugin-installation
> Learn how to install specific Kestra plugins in the open-source version for a lightweight build and faster startup using the -slim Docker image.
Install a selection of Kestra plugins in the open-source version.
Pick and choose Kestra plugins to create lightweight builds and achieve a faster startup. This guide explains how to:
- Install specific plugins when using the `-slim` Docker image (formerly `-no-plugins`)
- Understand plugin versioning across Open Source and [Enterprise](../../07.enterprise/01.overview/01.enterprise-edition/index.md)
- Automate plugin installation using Docker Compose
- Link to plugin documentation and versioning support
See also: [Versioned Plugins in Kestra Enterprise](../../07.enterprise/05.instance/versioned-plugins/index.md).
To download plugins for a standalone worker or local development environment, use [`kestractl plugins download`](../../kestra-cli/kestractl/index.md#plugin-management).
## Plugin basics in Kestra Open Source
Kestra plugins are distributed as individual JAR files and loaded at runtime. Plugins are not embedded by default in `-slim` Docker images (formerly published as `-no-plugins`). You can:
- Download specific [plugin JARs](https://repo.maven.apache.org/maven2/io/kestra/plugin/) manually or via `kestra plugins install`.
- Mount them into `/app/plugins/` in your [Docker Compose](../../02.installation/03.docker-compose/index.md) setup.
## Install plugins via `kestra plugins install`
You can install any plugin using:
```bash
kestra plugins install io.kestra.plugin:plugin-dbt:LATEST
```
This will download the [plugin JAR from Maven Central](https://repo.maven.apache.org/maven2/io/kestra/plugin/) into `/app/plugins`. Just replace `plugin-dbt` with whichever plugin you'd like to download (e.g., `plugin-script-python`, `plugin-aws`, etc.)
You can run this inside a container (interactively or as part of Dockerfile) to build custom plugin bundles.
## Automate plugin selection with Docker Compose
If you're using the `kestra/kestra:*-slim` image and want to add only selected plugins:
### Option 1: Use `kestra plugins install` inside the container
```yaml
services:
kestra:
image: kestra/kestra:latest-slim
entrypoint: /bin/sh -c "
kestra plugins install io.kestra.plugin:plugin-dbt:LATEST && \
kestra plugins install io.kestra.plugin:plugin-scripts:LATEST && \
kestra server standalone"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./storage:/app/storage
```
### Option 2: Preload plugin JARs locally
You can copy only the JARs you need from a full Kestra image:
```bash
docker run --rm -d --name kestra-temp kestra/kestra:latest
docker cp kestra-temp:/app/plugins/. ./local-plugins
docker rm -f kestra-temp
```
Then remove unwanted plugins:
```bash
rm ./local-plugins/*unwanted-plugin*.jar
```
And mount your plugin folder:
```yaml
volumes:
- ./local-plugins:/app/plugins
```
You may also use a scripted alias to automate this process. Below is an example for reference:
```bash
alias dl="rm -rf ./jar-plugins/* && docker run -d kestra/kestra:develop server local \
| xargs -I {} sh -c 'docker cp {}:/app/plugins ./jar-plugins && docker rm -f {}'"
```
## Plugin versioning in Enterprise
In Kestra Open Source, plugins must be installed at the latest compatible version. In Kestra Enterprise, you can:
- Pin specific plugin versions
- Upload custom plugin binaries per tenant
- Enable version-aware workflows
Learn more about versioned plugins in Enterprise:
[Versioned Plugins](../../07.enterprise/05.instance/versioned-plugins/index.md)
## Best practices
| Use Case | Recommendation |
| -------------------------- | ---------------------------------------------------- |
| Minimal runtime image | Use `kestra/kestra:*-slim` with mounted JARs |
| Dynamic plugin setup | Use `kestra plugins install` in entrypoint |
| Controlled plugin versions | Use Enterprise with versioned plugins |
| Custom plugin development | Build and copy plugins into `/app/plugins/` manually |
---
# Trigger Kestra Flows from ServiceNow
URL: https://kestra.io/docs/how-to-guides/servicenow-trigger
> Integrate ServiceNow with Kestra by triggering flows via webhooks from ServiceNow Service Catalog items for automated fulfillment.
Execute Kestra flows with a ServiceNow webhook trigger.
ServiceNow often acts as the front door for enterprise automation. This guide shows how to let analysts request an on-demand compliance scan from a ServiceNow catalog item while Kestra executes the workflow behind the scenes through a webhook trigger.
:::alert{type="info"}
This guide assumes the existence of a flow like in our [SecOps with Kestra guide](../secops-with-kestra/index.md).
:::
## Prerequisites
- A ServiceNow instance with Flow Designer access
- A Kestra tenant with a flow exposed through a webhook trigger
- The webhook URL, namespace, and token for the Kestra flow
## What You Will Build
- A Service Catalog item (`complianceScanAndRemediate`) that collects the host IP and remediation preferences
- Catalog variables that persist the user input
- A reusable ServiceNow Action that calls the Kestra webhook
- A Flow Designer flow that ties the catalog submission to the Action
## Step 1: Create the Catalog Item
1. Sign in to ServiceNow as an administrator and navigate to **Service Catalog → Catalog Definitions → Maintain Items**.

2. Select **New** and provide the basic metadata:
- **Name**: `complianceScanAndRemediate`
- **Catalogs**: *Service Catalog*
- **Category**: *Services*
- **Fulfillment automation level**: *Fully automated*
3. Fill in the **Short description** and **Description**, adjust any Portal settings you do not need, and click **Save**.

## Step 2: Add Catalog Variables
1. In the Variables related list, choose **New** and create the primary inputs:
- **Type**: *Single Line Text*
- **Question**: *IP Address*
- **Name**: `ipAddress`
- **Mandatory**: enabled

2. Create an additional variable for remediation control, for example:
- **Type**: *Single Line Text* (or *Boolean* if you prefer a toggle)
- **Question**: *Auto remediate* (Name `autoRemediate`)

3. (Optional) Add a multi-choice variable if you want to offer canned scan profiles. Define the choices under the **Choices** related list once the variable has been saved.

4. Click **Update** to persist the catalog item changes.
## Step 3: Build the Script Action
Navigate to the **Workflow Studio**:

1. Open **Flow Designer → Action** and create a new Action named `triggerKestraWebhook` in the **Service Catalog** category.
2. Add two Action inputs: `ipAddress` and `remediateControls`.

3. Insert a **Script** step, expose the same inputs to that step, and paste the following code, updating the endpoint with your Kestra domain, tenant, namespace, flow ID, and webhook token. Store any secrets (such as the token) in ServiceNow Credential or Connection records rather than hardcoding them.
```javascript
(function execute(inputs, outputs) {
outputs.error = "";
try {
var request = new sn_ws.RESTMessageV2();
request.setHttpMethod("post");
request.setEndpoint("https://{YOUR.KESTRA.DOMAIN}/api/v1/{TENANT}/executions/webhook/{NAMESPACE}/{FLOW_ID}/{WEBHOOK_TOKEN}");
request.setRequestHeader("Content-Type", "application/json");
request.setRequestHeader("Accept", "application/json");
var body = {
ipAddress: inputs.ipAddress,
remediateControls: inputs.remediateControls
};
request.setRequestBody(JSON.stringify(body));
var response = request.execute();
var httpStatus = response.getStatusCode();
var responseBody = response.getBody();
gs.info("Kestra webhook response status: " + httpStatus);
gs.info("Kestra webhook body: " + responseBody);
outputs.responseBody = responseBody;
outputs.statusCode = httpStatus;
} catch (error) {
gs.error("Kestra webhook failed: " + error.message);
outputs.error = error.message;
}
})(inputs, outputs);
```
4. Define Script outputs for `responseBody`, `statusCode`, and `error`, then map them to Action outputs so downstream flows can inspect the response.

5. Publish the Action.

## Step 4: Create the ServiceNow Flow
1. In Flow Designer, create a flow named `catalogSubmissionFlow`.
2. Select the **Service Catalog** trigger so the flow runs whenever the catalog item is submitted.
3. Add the **Get Catalog Variables** action and configure it to:
- Use the **Requested Item record** from the trigger as the submitted request
- Limit the template to the `complianceScanAndRemediate` catalog item
- Return all of the variables you created earlier
4. Add the `triggerKestraWebhook` Action to the flow and map each Action input to the corresponding variable output from the previous step.
5. Activate the flow.
## Step 5: Connect the Catalog Item to the Flow
In the **Workflow Editor**, click on **New -> Flow**:

1. Name the flow `catalogSubmissionFlow` and give a description

2. Set the Trigger as Service Catalog
3. In Actions, get the Catalog Variables

4. Set Action Inputs

5. Set Template Catalog Items: Click on the magnifying glass and select `complianceScanAndRemediate`.

6. Set the Catalog Variables and **Save**.

7. Add an **Action** and search for `triggerKestraWebhook`:

8. Under **Action Inputs**, for `ipAddress` and click on the wand icon to select **Get Catalog Variables → `ipAddress`** and repeat for Auto Remediate.

## Validate the End-to-End Run
1. Open your **Catalog Item Catalog → Catalog Definitions -> Maintain Items→ `complianceScanAndRemediate`**
2. Go to **Process Engine**, and under **Flow** select `catalogSubmissionFlow`

3. Click on **Update**, then try the workflow

4. Submit the request and navigate **System Log -> All**:

The webhook will be triggered:

5. Navigate to Kestra, and view the **Flow Executions** tab:

## Conclusion
By fronting Kestra with a ServiceNow catalog item, you let users stay inside their familiar ITSM portal while still benefiting from Kestra's orchestration capabilities. The same pattern works for any flow that exposes a webhook trigger — swap in different inputs, reuse the Action, and tailor the downstream automation without changing the ServiceNow experience.
---
# Run Shell Scripts Inside Your Flows
URL: https://kestra.io/docs/how-to-guides/shell
> Run Bash and shell scripts in Kestra workflows. Execute multi-step commands, chain scripts with pipes, and handle errors in automated shell tasks.
Run Shell scripts directly in your flows and generate outputs.
You can execute bash script in a flow by either writing your Shell commands inline or by executing a `.sh` file. You can get outputs and metrics from your Shell script too.
## Scripts
If you want to write a series of commands together to form a small script, and run that script as a task in the flow, you can use the `io.kestra.plugin.scripts.shell.Script`.
```yaml
id: shell_script
namespace: company.team
description: This flow runs the shell script.
tasks:
- id: shell_script_task
type: io.kestra.plugin.scripts.shell.Script
containerImage: badouralix/curl-jq
script: |
# invoke a GET call on an API and extract information from the JSON response
downloads=$(curl https://hub.docker.com/v2/repositories/kestra/kestra/ | jq -r '.pull_count')
echo "Downloads: ${downloads}"
```
You can read more about the Scripts type in the [Plugin documentation](/plugins/plugin-script-shell/io.kestra.plugin.scripts.shell.script).
## Commands
You could also choose to provide the series of Shell commands in the task, and get the same result. Here is an example of how you can run the previous example using the `io.kestra.plugin.scripts.shell.Commands` type:
```yaml
id: shell_commands
namespace: company.team
description: This flow runs the shell commands.
tasks:
- id: http_download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: shell_commands_task
type: io.kestra.plugin.scripts.shell.Commands
commands:
- echo "The current execution is {{ execution.id }}"
- cat {{ outputs.http_download.uri }}
```
You can also put a Shell script in a separate `.sh` file, and invoke the script as a command. For example, we have a script file called `hello.sh` that contains:
```bash
echo "Hi there! This is an example of executing a Shell script file."
sleep 2
echo "I am back from sleep"
```
You can now invoke this script as one of the commands in the `io.kestra.plugin.scripts.shell.Commands` task. Note that we have set the `enabled` flag for the `namespaceFiles` property to `true` so Kestra can access the file.
```yaml
id: shell_invoke_file
namespace: company.team
description: This flow runs the shell script file.
tasks:
- id: shell_invoke_file_task
type: io.kestra.plugin.scripts.shell.Commands
namespaceFiles:
enabled: true
commands:
- sh hello.sh
```
You can read more about the Commands type in the [Plugin documentation](/plugins/plugin-script-shell/io.kestra.plugin.scripts.shell.commands).
## Handling Outputs
If you want to get a variable or file from your Shell script, you can use an [output](../../05.workflow-components/06.outputs/index.md).
### Variable Output
You can get the JSON outputs from the Shell commands / script using the `::{}::` pattern. Here is an example:
```yaml
id: shell_outputs
namespace: company.team
description: This flow runs the shell command, and outputs the variable.
tasks:
- id: shell_outputs_task
type: io.kestra.plugin.scripts.shell.Commands
commands:
- echo '::{"outputs":{"test":"value","int":2,"bool":true,"float":3.65}}::'
```
All the output variables can be viewed in the Outputs tab of the execution.

You can refer to the outputs in another task as shown in the example below:
```yaml
id: shell_outputs_usage
namespace: company.team
description: This flow runs the shell command, and outputs the variable.
tasks:
- id: shell_outputs_task
type: io.kestra.plugin.scripts.shell.Commands
commands:
- echo '::{"outputs":{"test":"value","int":2,"bool":true,"float":3.65}}::'
- id: log_vars
type: io.kestra.plugin.core.log.Log
message: '{{ outputs.shell_outputs_task.vars.test }}'
```
_This example works for both `io.kestra.plugin.scripts.shell.Script` and `io.kestra.plugin.scripts.shell.Commands`._
### File Output
Inside of your Shell script, write a file to the system. You'll need to add the `outputFiles` property to your flow and list the files you're trying to put out. In this case, we want to output `output.txt`. More information on the formats you can use for this property can be found in [Script Output Metrics](../../16.scripts/06.outputs-metrics/index.md).
The example below writes a `output.txt` file containing the "Hello world" text, similar the output we used earlier. We can then refer the file using the syntax `{{ outputs.{task_id}.outputFiles[''] }}`, and read the contents of the file using the `read()` function.
```yaml
id: shell_output_file
namespace: company.team
description: This flow runs the shell command to output a file.
tasks:
- id: shell_outputs_task
type: io.kestra.plugin.scripts.shell.Commands
outputFiles:
- output.txt
commands:
- echo 'Hello world' > output.txt
- id: log_output
type: io.kestra.plugin.core.log.Log
message: "{{ read(outputs.shell_outputs_task.outputFiles['output.txt']) }}"
```
_This example works for both `io.kestra.plugin.scripts.shell.Script` and `io.kestra.plugin.scripts.shell.Commands`._
## Handling Metrics
You can also get [metrics](../../16.scripts/06.outputs-metrics/index.md#outputs-and-metrics-in-script-and-commands-tasks) from your Shell script. Metrics use the same `::{}::` pattern as outputs. This example demonstrates both the counter and timer metrics.
```yaml
id: shell_metrics
namespace: company.team
description: This flow runs the shell command, and puts out the metrics.
tasks:
- id: shell_outputs_task
type: io.kestra.plugin.scripts.shell.Commands
commands:
- echo 'There are 20 products in the cart'
- echo '::{"outputs":{"productCount":20}}::'
- echo '::{"metrics":[{"name":"productCount","type":"counter","value":20}]}::'
- echo '::{"metrics":[{"name":"purchaseTime","type":"timer","value":32.44}]}::'
```
Once this has executed, both the metrics can be viewed under **Metrics**.

## Automate Shell with triggers
You can also use shell code as polling logic by using `ScriptTrigger` or `CommandsTrigger`. These trigger types run shell code on an interval and start a flow execution only when the `exitCondition` matches.
Use `ScriptTrigger` for inline shell code:
```yaml
id: script_trigger
namespace: company.team
triggers:
- id: script_failure
type: io.kestra.plugin.scripts.shell.ScriptTrigger
interval: PT10S
exitCondition: "exit 1"
edge: true
containerImage: ubuntu
script: |
cat /path/that/does/not/exist
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "Triggered with exitCode={{ trigger.exitCode }} (condition={{ trigger.condition }})"
```
Use `CommandsTrigger` when you want to run shell commands instead:
```yaml
id: commands_trigger
namespace: company.team
triggers:
- id: commands_failure
type: io.kestra.plugin.scripts.shell.CommandsTrigger
interval: PT10S
exitCondition: "exit 1"
edge: true
containerImage: ubuntu
commands:
- cat /path/that/does/not/exist
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "Triggered with exitCode={{ trigger.exitCode }} (condition={{ trigger.condition }})"
```
These trigger types support:
- `interval` to control how often the script or commands run
- `exitCondition` to match an exit code such as `exit 1`, or a regex or substring matched against emitted vars and failure logs
- `edge` to emit only on a transition from not matching to matching
---
# Send HTTP-downloaded Data as a Slack or Email Attachment
URL: https://kestra.io/docs/how-to-guides/slack-email-attachments
> Download a file over HTTP and deliver it as a Slack file attachment or a Gmail email attachment in a single Kestra flow.
Extract a file from any HTTP source and deliver it straight to your team, either as a Slack attachment or as an email attachment.
A common pattern is to download a report or export from an external system and forward it to stakeholders without storing it anywhere in between. This guide shows both delivery options starting from the same [`Download`](https://kestra.io/plugins/tasks/http/io.kestra.plugin.core.http.download) task.
## Download the file over HTTP
The [`io.kestra.plugin.core.http.Download`](https://kestra.io/plugins/tasks/http/io.kestra.plugin.core.http.download) task fetches the file and stores it in Kestra's internal storage. Every following task references it via `{{ outputs.download.uri }}`.
```yaml
id: download_orders_csv
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
```
## Option 1: send the file as a Slack attachment
:::alert{type="warning"}
`SlackIncomingWebhook` **cannot** send file attachments — that's a Slack API limitation, incoming webhooks only accept a JSON `payload` and don't support the `files.upload` endpoint. To attach a file, you need a Slack App with a bot token instead.
:::
### Create a Slack App with a bot token
1. Go to the [Slack API website](https://api.slack.com/apps) and create a new app "From scratch".
2. Under **OAuth & Permissions**, add the `files:write` Bot Token Scope (add `chat:write` too if you also want to post a message).
3. Install the app to your workspace and copy the **Bot User OAuth Token** (starts with `xoxb-`).
4. Invite the bot to the target channel: `/invite @your-app-name`.
5. Store the token as a Kestra [Secret](../../06.concepts/04.secret/index.md) called `SLACK_TOKEN`.
### Upload the file to a channel
Use the [`io.kestra.plugin.slack.app.files.Upload`](https://kestra.io/plugins/plugin-slack/io.kestra.plugin.slack.app.files.upload) task, with `initialComment` to attach a message to the file in the same call:
```yaml
id: send_report_to_slack
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: upload_to_slack
type: io.kestra.plugin.slack.app.files.Upload
token: "{{ secret('SLACK_TOKEN') }}"
channels: ["#reports"]
from: "{{ outputs.download.uri }}"
filename: "orders.csv"
title: "Daily orders export"
initialComment: "Here is today's orders export :point_down:"
```
:::alert{type="info"}
`initialComment` requires a recent version of the Slack plugin. On older versions, post a message with [`io.kestra.plugin.slack.app.chats.Post`](https://kestra.io/plugins/plugin-slack/io.kestra.plugin.slack.app.chats.post) to the same channel before running `Upload` instead.
:::
## Option 2: send the file as an email attachment via Gmail
Gmail's SMTP server accepts an [App Password](https://support.google.com/accounts/answer/185833) once 2-Step Verification is enabled on the account. Use it with the [`io.kestra.plugin.email.MailSend`](https://kestra.io/plugins/plugin-email/io.kestra.plugin.email.mailsend) task.
### Create a Gmail App Password
1. Enable **2-Step Verification** on the Google account, if not already enabled.
2. Go to [myaccount.google.com/apppasswords](https://myaccount.google.com/apppasswords), create a new app password (e.g. named "Kestra").
3. Store the generated 16-character password as a Kestra [Secret](../../06.concepts/04.secret/index.md) called `GMAIL_APP_PASSWORD`.
### Send the file as an attachment
```yaml
id: send_report_by_email
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: send_email
type: io.kestra.plugin.email.MailSend
host: smtp.gmail.com
port: 587
transportStrategy: SMTP_TLS
username: "{{ secret('GMAIL_ADDRESS') }}"
password: "{{ secret('GMAIL_APP_PASSWORD') }}"
from: "{{ secret('GMAIL_ADDRESS') }}"
to: team@company.com
subject: "Daily orders export"
htmlTextContent: "Please find today's orders export attached."
attachments:
- name: orders.csv
uri: "{{ outputs.download.uri }}"
contentType: text/csv
```
## Combine both in a single flow
Both delivery paths read from the same `download` output, so nothing stops you from running them side by side in one flow:
```yaml
id: distribute_orders_export
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: upload_to_slack
type: io.kestra.plugin.slack.app.files.Upload
token: "{{ secret('SLACK_TOKEN') }}"
channels: ["#reports"]
from: "{{ outputs.download.uri }}"
filename: "orders.csv"
title: "Daily orders export"
initialComment: "Here is today's orders export :point_down:"
- id: send_email
type: io.kestra.plugin.email.MailSend
host: smtp.gmail.com
port: 587
transportStrategy: SMTP_TLS
username: "{{ secret('GMAIL_ADDRESS') }}"
password: "{{ secret('GMAIL_APP_PASSWORD') }}"
from: "{{ secret('GMAIL_ADDRESS') }}"
to: team@company.com
subject: "Daily orders export"
htmlTextContent: "Please find today's orders export attached."
attachments:
- name: orders.csv
uri: "{{ outputs.download.uri }}"
contentType: text/csv
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 8 * * *"
```
---
# Slack Events API with Kestra: Trigger Flows
URL: https://kestra.io/docs/how-to-guides/slack-webhook
> Trigger Kestra flows based on Slack events using the Slack Events API and Webhook triggers to automate responses and interactions.
Trigger Kestra flows based on Slack events.
The Slack Events API allows you to build apps that respond to events from Slack. For example, you can trigger a custom action anytime a user joins a channel or when someone reacts to a message with a specific emoji.
## Create a Slack App
To use the Slack Events API, you'll need to create a Slack app. You can do this from the [Slack API website](https://api.slack.com/apps).
First, click on the "Create New App" button:

Choose the option "From scratch":

Then, give your app a name and select the workspace where you want to install it:

Now, you need to enable the "Event Subscriptions" feature:

In the "Subscribe to bot events" section, you can add events you want to listen to.

For example, you can listen to the `app_mentions` and `reaction_added` events:

## Create a flow with a Webhook trigger
You can now create a Kestra flow that will listen to the events you've subscribed to:
```yaml
id: slack_events
namespace: prod
tasks:
- id: process_slack_event
type: io.kestra.plugin.core.log.Log
message: "{{ trigger.body }}"
triggers:
- id: slack_event
type: io.kestra.plugin.core.trigger.Webhook
key: superStrongSecretKey42
```
:::alert{type="warning"}
The **webhook key** cannot contain any **special characters** — only letters and digits. Also, consider it as a secret that you should keep safe. You can use Kestra's [Secrets](../../06.concepts/04.secret/index.md) to store it securely.
:::
Now, the only part left is to create a simple app that will listen to Slack events and will forward them to your Kestra flow via the Webhook trigger.
We'll look at how to do this using Python and FastAPI. For deployments, we'll show two options:
1. Using Modal for easy deployment
2. Using ngrok to expose our local FastAPI server to the internet. You can replace ngrok for any other deployment method you prefer.
## Deploy a Slack app with Modal
First, sign up for a free account on [Modal](https://modal.com/). Then, go to your Settings:

And create a new API token:

You will see a similar command:
```bash
modal token set --token-id ak-zzzzzzzzz --token-secret as-zzzzzzzzz
```
Now, create the following flow in Kestra and replace the token ID and token secret with the ones you got from Modal. You can use Kestra's [Secrets](../../06.concepts/04.secret/index.md) to store those securely. Also, replace `your_kestra_host` with your Kestra host URL in the `slack.py` file.
```yaml
id: slack_app
namespace: prod
tasks:
- id: modal_slack_app
type: io.kestra.plugin.modal.cli.ModalCLI
commands:
- modal deploy slack.py
env:
MODAL_TOKEN_ID: "{{ secret('MODAL_TOKEN_ID') }}"
MODAL_TOKEN_SECRET: "{{ secret('MODAL_TOKEN_SECRET') }}"
inputFiles:
slack.py: |
import logging
from fastapi import FastAPI, Request, BackgroundTasks
from fastapi.responses import JSONResponse
from modal import Image, App, asgi_app
import requests
web_app = FastAPI()
app = App("slack_app")
image = Image.debian_slim().pip_install("requests")
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def process_event(event):
# Replace with your Kestra Webhook URL
url = "http://your_kestra_host:8080/api/v1/main/executions/webhook/prod/slack_events/superStrongSecretKey42"
headers = {"Content-Type": "application/json"}
response = requests.post(url, headers=headers, json=event)
logger.info(f"Forwarding event response: {response.status_code} - {response.text}")
@web_app.post("/slack/events")
async def slack_events(request: Request, background_tasks: BackgroundTasks):
json_data = await request.json()
if "challenge" in json_data:
logger.info("Received Slack challenge event")
return JSONResponse(content={"challenge": json_data["challenge"]})
logger.info(f"Received event: {json_data}")
# Process the event asynchronously
background_tasks.add_task(process_event, json_data)
# Respond immediately to Slack
logger.info("Responding immediately to Slack")
return JSONResponse(content={"status": "ok"})
@app.function(image=image)
@asgi_app()
def fastapi_app():
return web_app
```
:::alert{type="info"}
If you don't like adding the Python script inline in the YAML file, you can enable `namespaceFiles` and add the Python code in the embedded Code Editor in a separate file e.g. called `slack.py` and reference it in the flow as shown below:
```yaml
id: slack_app
namespace: prod
tasks:
- id: modal_slack_app
type: io.kestra.plugin.modal.cli.ModalCLI
namespaceFiles:
enabled: true
commands:
- modal deploy slack.py
env:
MODAL_TOKEN_ID: "{{ secret('MODAL_TOKEN_ID') }}"
MODAL_TOKEN_SECRET: "{{ secret('MODAL_TOKEN_SECRET') }}"
```

:::
Once you execute that flow, you will see the endpoint to your app in the logs:

Go back to Slack and add the URL to the "Request URL" field in the "Event Subscriptions" section. Add `slack/events` at the end of the URL, e.g.:
```bash
https://anna-geller--slack-app-fastapi-app.modal.run/slack/events
```
You should see the `Verified` message. Hit `Save Changes` and you're all set!

## Install the Slack app to a Workspace and test it
First, we need to install the app to the workspace. Go to "Install App" and click on "Install to Workspace":


Now you can test the integration by mentioning your app in a channel. For example, you can write a hello message `hello @kestra`:

Confirm to invite the app to the channel and congratulate yourself with the "Nicely done!" emoji 🙌:

You should see that both events (`app mention` and `reaction added`) have triggered an execution of your Kestra flow:


Now it's up to you to automate your daily operations with Slack and Kestra!
## Example automation: AI Chatbot
You can extend the `slack_events` flow to automate your daily business operations.
To do something more useful than just logging the Slack event, you can create a flow that listens to the `app_mention` event and responds to that message with a GPT-4 chatbot. First, create an incoming webhook in your Slack app:


Copy the webhook URL:

...and paste it into the `url` field of the `SlackIncomingWebhook` task in the flow below:
```yaml
id: slack_events
namespace: prod
tasks:
- id: if_app_mention
type: io.kestra.plugin.core.flow.If
condition: "{{ trigger.body.event.type == 'app_mention' }}"
then:
- id: gpt
type: io.kestra.plugin.openai.ChatCompletion
apiKey: "{{ secret('OPENAI_API_KEY') }}"
model: gpt-4-0125-preview
messages:
- role: system
content: The user will refer to you as <@{{ trigger.body.authorizations[0].user_id }}>. You get a prompt from a user and provide a concise answer.
prompt: "{{ trigger.body.event.text ?? null }}"
- id: slack
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
url: "{{ secret('SLACK_WEBHOOK_URL') }}"
payload: |
{"channel":"{{ trigger.body.event.channel }}","text":"{{ outputs.gpt.choices[0].message.content }}"}
else:
- id: other_events
type: io.kestra.plugin.core.log.Log
message: "{{ trigger.body }}"
triggers:
- id: slack_event
type: io.kestra.plugin.core.trigger.Webhook
key: superStrongSecretKey42
```
:::alert{type="info"}
The `SlackIncomingWebhook` task also has the `messageText` property that can be used instead of the `payload` property, depending on the task's requirements.
:::
And here is the result:


---
## Local testing with ngrok
If you don't want to host your app on Modal, you can use ngrok to expose your local server to the internet.
First, install ngrok:
```bash
brew install ngrok/ngrok/ngrok
```
Then, [sign up](https://ngrok.com/) for a free account and then authenticate your terminal, as explained in the [Getting Started guide](https://dashboard.ngrok.com/get-started/setup/):
```bash
ngrok config add-authtoken long_token_string
```
Create a FastAPI app in a file called `main.py`:
```python
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import requests
app = FastAPI()
@app.post("/slack/events")
async def slack_events(request: Request):
json_data = await request.json()
# Slack URL Verification Challenge
if "challenge" in json_data:
return JSONResponse(content={"challenge": json_data["challenge"]})
print("Received an event")
print(json_data)
# URL of your Kestra flow webhook
url = "http://your_kestra_host:8080/api/v1/main/executions/webhook/prod/slack_events/superStrongSecretKey42"
headers = {
"Content-Type": "application/json",
}
response = requests.post(url, headers=headers, json=json_data)
print(response.text)
return JSONResponse(
content={"status": response.status_code, "response": response.text}
)
```
Then, set up your FastAPI server:
```bash
pip install fastapi uvicorn requests
uvicorn main:app --reload --port 3000
ngrok http http://localhost:3000
```
This will expose your local server to the internet. You should see a similar URL:
```bash
https://0913-31-18-152-123.ngrok-free.app
```
Go back to your Slack app and add the URL to the "Request URL" field in the "Event Subscriptions" section. Add `slack/events` at the end of the URL, e.g.:
```bash
https://0913-31-18-152-123.ngrok-free.app/slack/events
```
The rest of the process is the same as with Modal. You can now adjust the flow `slack_events` referenced in the FastAPI code and start automationg various processes based on Slack events.
## Other deployment options
You can deploy that Slack app in many other ways including:
- an on-prem VM
- a serverless approach with [AWS Lambda](https://www.youtube.com/watch?v=rpVLOVeky6A), Google Cloud Functions, or Azure Functions
- a containerized approach with AWS Fargate, Google Cloud Run, or Azure Container Instances
- a Kubernetes deployment.
And of course, you can use any other programming language and framework to build your Slack app. The only requirement is to forward the Slack events to your Kestra flow via the Webhook trigger.
---
# Trigger Kestra Flows from Splunk Alerts
URL: https://kestra.io/docs/how-to-guides/splunk-alert-trigger
> Install the Kestra Add-on for Splunk and configure the Trigger Kestra Flow alert action to start Kestra flows from Splunk saved-search alerts using a Bearer API token.
The Kestra Add-on for Splunk adds a **Trigger Kestra Flow** alert action that starts Kestra flows when a saved search fires.
When an alert fires, the add-on calls the Kestra Executions API with a Bearer token, forwarding alert metadata and search results as flow inputs. No webhook URL required — the token is stored encrypted in Splunk's credential store.
This guide covers the Kestra-side setup: creating an API token, designing a receiving flow, and wiring up the alert action fields. For Splunk-side installation, see the [TA-kestra README](https://github.com/kestra-io/splunk-addon).
## Prerequisites
- Kestra Enterprise Edition with at least one namespace and a service account or user API token with `EXECUTION:CREATE` permission on the target namespace.
- Splunk Enterprise or Splunk Cloud with the TA-kestra add-on installed. See [Installation](https://github.com/kestra-io/splunk-addon#installation) in the add-on README.
- A saved search with a trigger condition. Any search that produces results can be used as a trigger.
## Step 1: Create a Kestra API token
The add-on authenticates with Kestra using a Bearer token stored encrypted in Splunk's credential store. Create a dedicated token for the integration.
In the Kestra UI, open **Administration → API Tokens** and create a new token scoped to the service account or user that has `EXECUTION:CREATE` on the target namespace.
Copy the full token value now — it is shown only once.
See [API Tokens](../../07.enterprise/03.auth/api-tokens/index.md) for token creation and rotation details.
## Step 2: Configure a Kestra connection in the add-on
The add-on stores Kestra credentials as named connections. Each connection holds a base URL and an API token. Alert actions reference a connection by name, so you configure the URL and token once per Kestra instance.
In Splunk Web, go to **Apps → Kestra Add-on for Splunk → Configuration → Kestra Connections → Add** and fill in:
| Field | Value |
|-------|-------|
| Name | A unique identifier for this connection, e.g. `prod_kestra`. |
| Kestra URL | Base URL of your instance, e.g. `https://kestra.corp.example.com` (no trailing slash). |
| API token | The token from Step 1. Splunk stores it encrypted in `storage/passwords` — it is never written in plaintext to a `.conf` file or logged. |
## Step 3: Design the receiving flow
Create a flow in the target namespace with inputs that match what the alert action will send.
All flow inputs from the alert action are optional at the Kestra API level, so declare them with `required: false` unless you want Kestra to reject executions that omit them.
```yaml
id: splunk-alert-handler
namespace: company.ops
inputs:
# From "Input mapping" — map any result field or alert token
- id: severity
type: STRING
required: false
- id: src_host
type: STRING
required: false
# From "Send alert metadata" (enabled by default)
- id: splunk_search_name
type: STRING
required: false
- id: splunk_sid
type: STRING
required: false
- id: splunk_owner
type: STRING
required: false
- id: splunk_app
type: STRING
required: false
- id: splunk_results_link
type: STRING
required: false
- id: splunk_trigger_time
type: STRING
required: false
# From "Attach results file" (enabled by default)
- id: results
type: FILE
required: false
tasks:
- id: triage
type: io.kestra.plugin.core.log.Log
message: |
Alert '{{ inputs.splunk_search_name }}' fired at {{ inputs.splunk_trigger_time }}.
Severity: {{ inputs.severity }} — host: {{ inputs.src_host }}
```
### What the add-on sends
The alert action has three input sources, all controlled by the alert configuration:
**Alert metadata** (sent when **Send alert metadata** is on, which is the default):
| Input name | Splunk source |
|------------|---------------|
| `splunk_search_name` | Saved search name |
| `splunk_sid` | Search ID |
| `splunk_app` | Splunk app context |
| `splunk_owner` | Alert owner |
| `splunk_results_link` | Link to search results in Splunk |
| `splunk_trigger_time` | Alert trigger timestamp |
**Input mapping** (free-form, one `flow_input=$token$` per line):
Map any Splunk alert token or search result field to a named flow input. Splunk resolves the tokens before passing them to the handler.
```
severity=$result.severity$
src_host=$result.src_host$
search_name=$name$
```
`$result.fieldname$` picks the value from the first matching result row. `$name$` is the saved search name (same as `splunk_search_name`, useful when you want a shorter input name).
**Results file** (sent when **Attach results file** is on, which is the default):
The full search results CSV is attached as a `FILE` input named `results`. Declare it as `type: FILE` in your flow if you want to process it.
## Step 4: Attach the alert action
Open the saved search in Splunk, go to **Edit → Edit Alert**, and under **Trigger Actions** click **+ Add Actions → Trigger Kestra Flow**.
Fill in the action fields:
| Field | Description |
|-------|-------------|
| Kestra connection | The connection from Step 2 (URL + API token). |
| Tenant | Kestra tenant ID. Use `main` for single-tenant instances. |
| Namespace | Namespace of the flow, e.g. `company.ops`. |
| Flow ID | ID of the flow, e.g. `splunk-alert-handler`. |
| Attach results file | Attaches the search results CSV as a `FILE` input named `results`. On by default. |
| Send alert metadata | Sends alert metadata as `splunk_*` inputs. On by default. |
| Input mapping | One `flow_input=$token$` per line. Leave blank if you only need metadata and the results file. |
## Step 5: Verify the integration
When the alert fires, the add-on logs the created execution ID and a link to it in the alert action log:
```
Kestra execution created: id= url=https://kestra.corp.example.com/ui/main/executions/company.ops/splunk-alert-handler/
```
Open the execution in Kestra to confirm the inputs arrived and the flow completed.
If the execution is not created, check the modalert log in Splunk at `$SPLUNK_HOME/var/log/splunk/trigger_kestra_flow_modalert.log`. Common errors and their fixes:
| Error in log | Fix |
|---|---|
| `connection 'x' does not exist` | The connection name in the alert action does not match any connection on the Configuration page. |
| `missing a URL or API token` | Edit the connection on the Configuration page and re-enter the API token. |
| `HTTP 401` | The API token is expired or revoked. Rotate it in Kestra and update the Splunk connection. |
| `HTTP 404` | The namespace or flow ID does not exist. Check the tenant, namespace, and flow ID fields. |
| `HTTP 422` | The flow rejected a required input or an input type mismatch. Check the flow's input declarations. |
The add-on makes up to three attempts on 5xx responses and transient network errors, with backoff between retries. The API token is never written to logs.
## Related
- [TA-kestra add-on source and installation](https://github.com/kestra-io/splunk-addon)
- [Kestra API Tokens](../../07.enterprise/03.auth/api-tokens/index.md)
- [Trigger flows via the Kestra API](../api/index.md)
- [Log Shipper](../../07.enterprise/02.governance/logshipper/index.md) — the reverse direction: ship Kestra logs into Splunk
---
# Use SQLMesh to Run dbt Projects
URL: https://kestra.io/docs/how-to-guides/sqlmesh
> Orchestrate SQLMesh transformations in Kestra. Run and schedule SQLMesh plans as part of your data pipeline for version-controlled, SQL-first modeling.
Using SQLMesh to run dbt project with Kestra.
SQLMesh is an open source python data transformation and modelling framework. It automates everything needed to run a scalable data transformation platform. SQLMesh works with a variety of [engines and orchestrators](https://sqlmesh.readthedocs.io/en/stable/integrations/overview/).
SQLMesh enables data teams to efficiently run and deploy data transformations written in SQL or Python.
This guide shows how to run dbt projects on BigQuery using SQLMesh with Kestra.
## Example
Our Flow will do the following steps:
1. Download `orders.csv` using HTTP download task.
2. Create the table in BigQuery.
3. Upload the data from the csv file into the BigQuery table.
4. Create a dbt project which will create the BigQuery view from the BigQuery table.
5. Create SQLMeshCLI task that will run the dbt project.
SQLMesh supports integration with a variety of tools like Airflow, dbt, dlt, etc. One of the common use-cases of SQLMesh is to run dbt projects.
You can choose to pull your dbt project from a Git repository as mentioned in the [How-to guide on dbt](../dbt/index.md) or create [namespace files](../../06.concepts/02.namespace-files/index.md) for the project. This guide creates the complete project using namespace files built up step by step. You can later choose to push all the namespace files to a GitHub repository using [PushNamespaceFiles](../pushnamespacefiles/index.md).
### Creating the flow with the SQLMeshCLI task
Create tasks for each step:
```yaml
id: sqlmesh_transform
namespace: company.team
tasks:
- id: orders_http_download
type: io.kestra.plugin.core.http.Download
description: Download orders.csv using HTTP Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/#main/csv/orders.csv
- id: create_orders_table
type: io.kestra.plugin.gcp.bigquery.CreateTable
description: Create orders table in BigQuery
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
projectId:
dataset: ecommerce
table: orders
tableDefinition:
type: TABLE
schema:
fields:
- name: order_id
type: INT64
- name: customer_name
type: STRING
- name: customer_email
type: STRING
- name: product_id
type: INT64
- name: price
type: FLOAT64
- name: quantity
type: INT64
- name: total
type: FLOAT64
- id: load_orders_table
type: io.kestra.plugin.gcp.bigquery.Load
description: Load orders table with data from orders.csv
from: "{{ outputs.orders_http_download.uri }}"
projectId:
serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
destinationTable: ".ecommerce.orders"
format: CSV
csvOptions:
fieldDelimiter: ","
skipLeadingRows: 1
- id: sqlmesh_transform
type: io.kestra.plugin.sqlmesh.cli.SQLMeshCLI
description: Use SQLMesh to run the dbt project
inputFiles:
sa.json: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
namespaceFiles:
enabled : true
beforeCommands:
- pip install "sqlmesh[bigquery]"
- pip install dbt-bigquery
commands:
- sqlmesh init -t dbt
- sqlmesh plan --auto-apply
```
It's important that we have the following properties configured:
- `namespaceFiles` property has `enabled` set to `true` to ensure that the task has access to your namespace files.
- Provide the GCP service account JSON file so that the task can connect to your GCP account to access BigQuery. See the [dedicated guide](../google-credentials/index.md) on how to add it. This file is referenced in the dbt project file.
- Install the `sqlmesh[bigquery]` and `dbt-bigquery` depenedencies with `beforeCommands`. These allow SQLMesh and dbt to perform operations on BigQuery.
Once the task is created and configured correctly, save the flow.
### Creating dbt project
Now go to the Editor, create a new file called `profiles.yml` with the following content:
```yaml
bq_dbt_project:
outputs:
dev:
type: bigquery
method: service-account
dataset: ecommerce
project:
keyfile: sa.json
location: US
priority: interactive
threads: 16
timeout_seconds: 300
fixed_retries: 1
target: dev
```
Next, we will create `dbt_project.yml` with the following content:
```yaml
name: 'bq_dbt_project'
version: '1.0.0'
config-version: 2
profile: 'bq_dbt_project'
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
clean-targets:
- "target"
- "dbt_packages"
models:
bq_dbt_project:
example:
+materialized: view
+start: Nov 10 2024
```
:::alert{type="info"}
`models` require a start date for backfilling data through use of the `start` configuration parameter.
:::
Now create a folder called `models` in the namespace. In the `models` folder, create `sources.yml` to define the source models:
```yaml
version: 2
sources:
- name: ecommerce
database:
schema: ecommerce
tables:
- name: orders
```
Lastly, create `stg_orders.sql` to materialize the `stg_orders` view for the `orders` table.
```sql
{{ config(materialized="view") }}
select order_id,
customer_name,
customer_email,
product_id,
price,
quantity,
total
from {{ source('ecommerce', 'orders') }}
```
Thats it! We are now ready to run the flow.
Once the flow runs successfully, you can go to BigQuery console, and ensure that the view `stg_orders` has been created.
This is how we can run SQLMeshCLI for the dbt project. These instructions can also help you integrate the SQLMeshCLI task with other SQLMesh [integrations and execution engines](https://sqlmesh.readthedocs.io/en/stable/integrations/dbt/).
---
# Subflow Retries, Restarts, and Replays in Kestra
URL: https://kestra.io/docs/how-to-guides/subflow-executions
> Best practices for configuring retries, restarts, and replays in subflow executions to ensure efficient error handling and resumption.
How to configure your flows so that failed subflow executions resume correctly without rerunning successful tasks.
---
When working with subflows, it’s important to understand the difference between retries at the **Subflow task level** and retries at the **flow level** within the subflow. This guide explains how to manage retries, restarts, and replays in subflow executions to avoid unnecessary re-execution of completed tasks.
## Flow-level vs. Subflow-level retries
### Subflow task-level retry
When you define a retry on the `Subflow` task, it controls how the **Subflow task** itself is retried within the parent flow. For example:
```yaml
id: parent_flow
namespace: company.team
tasks:
- id: subflow
type: io.kestra.plugin.core.flow.Subflow
namespace: company.team
flowId: my_subflow
wait: true
retry:
type: constant
maxAttempts: 3
interval: PT1S
````
In this case, the retry applies to the `Subflow` task in the parent flow. When the task fails, the entire subflow execution is retried from the beginning. This means all subflow tasks will re-run within each retried execution, including tasks that already succeeded.
### Flow-level retry inside the subflow
To retry the execution from **failed task** within a subflow (without rerunning tasks that already succeeded), configure the `retry` property **on the subflow flow definition**, not on the Subflow task. This allows the subflow execution to restart from the failed task rather than from the start.
Example:
```yaml
id: my_subflow
namespace: company.team
retry:
maxAttempts: 3
behavior: RETRY_FAILED_TASK
type: constant
interval: PT1S
tasks:
- id: start
type: io.kestra.plugin.core.log.Log
message: This task will succeed and won't be retried
- id: fail
type: io.kestra.plugin.core.execution.Fail
runIf: "{{ randomInt(lower=0, upper=2) == 1 }}"
errorMessage: Bad value returned!
- id: end
type: io.kestra.plugin.core.log.Log
message: This task will only run if the fail task succeeds
```
When this flow fails, only the failed task (`fail` in this example) will be retried. Tasks that already succeeded (`start`) will not run again.
## Recommended configuration
In most cases, you should:
* Define **`retry`** at the **flow level inside the subflow** (not on the Subflow task)
* Use **`behavior: RETRY_FAILED_TASK`** to resume from the failed task (recommended to avoid rerunning tasks that already succeeded)
* Use **`behavior: CREATE_NEW_EXECUTION`** ONLY if you want to always restart the subflow execution from the beginning.
### Example: Parent flow calling a subflow
```yaml
id: my_parent_flow
namespace: company.team
tasks:
- id: subflow
type: io.kestra.plugin.core.flow.Subflow
namespace: company.team
flowId: my_subflow
wait: true
- id: hello
type: io.kestra.plugin.core.log.Log
message: Success
```
When `my_subflow` is configured with `behavior: RETRY_FAILED_TASK`, it automatically restarts from the failed task during retries. The parent flow does not need additional configuration.
## Replays and restarts
* **Replay**: You can replay the parent execution from the failed subflow task if the subflow defines `behavior: RETRY_FAILED_TASK`.
* **Restart**: When you restart the parent execution where a subflow task failed from the UI or API, this will restart the entire child execution from the beginning (regardless of the subflow task definition), unless the subflow YAML defines `behavior: RETRY_FAILED_TASK` as flow-level retry configuration.
## Summary
* Use **flow-level retry** inside the subflow for fine-grained restart control.
* Use **`behavior: RETRY_FAILED_TASK`** to continue from the failed task.
* Avoid configuring conflicting retry behaviors between parent and subflow.
* Use subflow task retries only if you want to **create an entirely new subflow execution** in each retry attempt - when `retry` is defined on the `Subflow`-task level, it will **always** create an entirely new execution from start rather than restarting the existing child execution from failed task, regardless of the `behavior` configured on the flow-level in the subflow.
---
# Connect a Supabase Database to Kestra
URL: https://kestra.io/docs/how-to-guides/supabase-db
> Learn how to connect your Supabase Database to Kestra workflows using the PostgreSQL plugin to query, copy, and manage your data.
Use the native [Supabase plugin](/plugins/plugin-supabase) for new integrations. This guide documents the PostgreSQL plugin approach for direct database access.
Supabase is an open-source Backend-as-a-service (BaaS) platform that provides hosted PostgreSQL databases you can query directly from Kestra flows.
Before you begin, ensure you have a [Supabase account](https://supabase.com/) set up and a [Kestra installation](../../02.installation/index.mdx) running.
## Setting up a Database in Supabase
Once you've logged into Supabase, you'll need to set up an organization where you will create projects to access resources such as a database.

Once your organization is created, you'll be prompted to create a new project. Set a password for this project to use later for authenticating with the database in Kestra.

Once your project is created, you will now be able to access resources in Supabase. Head to the menu on the left side and select **Database**. You will be prompted to create a new table in your database, as well as configure any columns you want to use. Leave the columns blank for now and modify them later once you know what data to copy into the database.

## Connecting Supabase to Kestra
Now that we have a database set up in Supabase, we can move into Kestra to set up our connection. While there's no official Supabase plugin, we can connect using the [PostgreSQL plugin](/plugins/plugin-jdbc-postgres), which supports a number of tasks such as `Query`, `CopyIn`, and `CopyOut`.
Inside of Supabase, select the **Connect** button at the top to get information about our databases connection. Select **Type** and change this JDBC. This will give us 3 ways of connecting with a Connection String. As we're only connecting to the database when our workflow runs, the Transaction pooler is a good option to use.

Copy the URL provided for the Transaction pooler and replace `[YOUR-PASSWORD]` with the password set earlier. Store the password as a [secret](../../06.concepts/04.secret/index.md) and add the `url` property directly to each PostgreSQL task:
```yaml
url: "jdbc:postgresql://aws-0-eu-west-2.pooler.supabase.com:6543/postgres?user=postgres.nqxaafovehwkjapsqqlk&password={{ secret('SUPABASE_PASSWORD') }}"
```
:::alert{type="info"}
You can also split the connection string into separate properties:
```yaml
url: "jdbc:postgresql://aws-0-eu-west-2.pooler.supabase.com:6543/postgres"
username: "postgres.nqxaafovehwkjapsqqlk"
password: "{{ secret('SUPABASE_PASSWORD') }}"
```
In Enterprise Edition, you can centralize connection properties across flows using a [Policy](../../07.enterprise/02.governance/policies/index.md) with an `Add` rule targeting `io.kestra.plugin.jdbc.postgresql`.
:::
## Copying a CSV File into Supabase DB in a Flow
Using this [example CSV](https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv), we can copy the data into our table directly from Kestra. You can either set up the columns directly in Supabase or add a task in Kestra to add them automatically like this:
```yaml
id: supabase_db_add_columns
namespace: company.team
tasks:
- id: create_columns
type: io.kestra.plugin.jdbc.postgresql.Queries
url: "jdbc:postgresql://aws-0-eu-west-2.pooler.supabase.com:6543/postgres?user=postgres.nqxaafovehwkjapsqqlk&password={{ secret('SUPABASE_PASSWORD') }}"
sql: |
ALTER TABLE kestra_example
ADD COLUMN order_id int,
ADD COLUMN customer_name text,
ADD COLUMN customer_email text,
ADD COLUMN product_id int,
ADD COLUMN price double precision,
ADD COLUMN quantity int,
ADD COLUMN total double precision;
```
Once your columns are configured, use the [CopyIn](/plugins/plugin-jdbc-postgres/io.kestra.plugin.jdbc.postgresql.copyin) task combined with the [HTTP Download](/plugins/core/http/io.kestra.plugin.core.http.download) task to download the CSV file and copy it directly into the database.
```yaml
id: supabase_db_copyin
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: copy_in
type: io.kestra.plugin.jdbc.postgresql.CopyIn
url: "jdbc:postgresql://aws-0-eu-west-2.pooler.supabase.com:6543/postgres?user=postgres.nqxaafovehwkjapsqqlk&password={{ secret('SUPABASE_PASSWORD') }}"
table: "kestra_example"
from: "{{ outputs.download.uri }}"
header: true
columns: [order_id,customer_name,customer_email,product_id,price,quantity,total]
delimiter: ","
```
Once this flow completes, we can view the contents of our database in Supabase:

---
# Sync Flows from a Git Repository
URL: https://kestra.io/docs/how-to-guides/syncflows
> Automatically sync your flows from a Git repository to Kestra using the SyncFlows task, enabling GitOps and version control for your workflows.
Sync flows from a Git Repository to Kestra with the SyncFlows Task.
The [SyncFlows](/plugins/plugin-git/io.kestra.plugin.git.syncflows) task is a powerful integration that allows you to **sync your code with Git from the UI while still managing this process entirely in code**! Kestra unifies the development experience between the UI and code so you can combine the best of both worlds without sacrificing the benefits of version control.
The task syncs one or more flows from a Git repository on a schedule or anytime you push a change to a given Git branch.
## Before you begin
Before you start using the `SyncFlows` task, ensure the following prerequisites are in place:
1. A Git repository where you want to sync your flows. If you haven't pushed any flows yet, see the [guide using the PushFlows task](../pushflows/index.md).
2. A Personal Access Token (PAT) for Git authentication.
3. A running Kestra instance in a version 0.17.0 or later with the PAT stored as a [secret](../../06.concepts/04.secret/index.md) within the Kestra instance.
## Using the `dryRun` property
Here is a sync flow that will sync the `git` namespace with flows from the repository in the `flows` directory.
```yaml
id: sync_flows_from_git
namespace: company.ops
tasks:
- id: sync_flows
type: io.kestra.plugin.git.SyncFlows
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/git_username/flows
branch: main
targetNamespace: git
gitDirectory: flows
dryRun: true
```
Given that the `dryRun` property is set to `true`, the task will only output changes from the Git repository without syncing any flows to Kestra yet:

The files listed are the same ones we added in the [PushFlows guide](../pushflows/index.md).
## Sync all flows to a single namespace from Git
Set the `dryRun` property to `false` and sync the repository with Kestra:
```yaml
id: sync_flows_from_git
namespace: company.ops
tasks:
- id: sync_flows
type: io.kestra.plugin.git.SyncFlows
...
dryRun: false
```
You should see the same flows from the earlier log now in Kestra:

A full list is also available in the Outputs tab:

## Sync all flows including child namespaces
You can also sync all flows in child namespaces. In the repository, there is a sub-folder called `tutorial` with more flows. Sync those as well by adding the `includeChildNamespaces` property and setting it to `true`.
```yaml
id: sync_flows_from_git
namespace: company.ops
tasks:
- id: sync_flows
type: io.kestra.plugin.git.SyncFlows
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/kestra-io/flows
branch: main
targetNamespace: git
gitDirectory: flows
includeChildNamespaces: true
```
After executing, all flows — including those from the `tutorial` child namespace — are synced into Kestra:

The Outputs tab shows the same result:

## Set up a schedule
A common use case for this task is to set up a routine schedule to keep Kestra in sync with the Git repository. Add a [Schedule trigger](../../05.workflow-components/07.triggers/01.schedule-trigger/index.md). This example has a cron expression to execute once every hour:
```yaml
id: sync_flows_from_git
namespace: company.ops
tasks:
- id: sync_flows
type: io.kestra.plugin.git.SyncFlows
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/git_username/flows
branch: main
targetNamespace: git
gitDirectory: flows
triggers:
- id: every_full_hour
type: io.kestra.plugin.core.trigger.Schedule
cron: "* 0 * * *"
```
## Automatically sync when a change is pushed to Git
You can also automate the syncing process by adding a [Webhook trigger](../../05.workflow-components/07.triggers/03.webhook-trigger/index.md) and creating a Webhook on your GitHub repository to trigger the flow every time something is pushed to the repository. This is useful for keeping Kestra always in sync with the repository.
```yaml
id: sync_flows_from_git
namespace: company.ops
tasks:
- id: sync_flows
type: io.kestra.plugin.git.SyncFlows
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/kestra-io/flows
targetNamespace: git
gitDirectory: flows
triggers:
- id: gh_webhook
type: io.kestra.plugin.core.trigger.Webhook
key: abcdefg
```
To setup this webhook, go to the Settings for your GitHub repository and head to Webhooks and create a new Webhook:

For the Payload URL, your URL will follow the following format:
```plaintext
https://{your_hostname}/api/v1/main/executions/webhook/system/sync_flows_from_git/abcdefg
```
This will require your host name to be publicly accessible. If you want to test this without having to deploy Kestra first, you can use a tool like [ngrok](https://ngrok.com/) to tunnel Kestra so GitHub can see it. As we're putting the secret in the URL, we can leave the Secret field blank.
Save and test by committing something to the Git repository.

The most recent execution was triggered by the Webhook, keeping Kestra in sync with the Git repository automatically.
If you also want to sync your files, see the [guide on syncing namespace files](../syncnamespacefiles/index.md).
## Extra notes
- The `branch` property allows you to specify the branch to which files should be synced from.
- The `gitDirectory` property allows you to specify the directory to which flows should be synced from. If not set, flows will be synced from the Git directory named `_flows` and will optionally also include subdirectories named after the child namespaces. If you prefer, you can specify an arbitrary path, e.g. `kestra/flows`, allowing you to sync flows to that specific Git directory.
- If you try to add the Personal Access Token (PAT) directly in your source code in the `password` property, you will get an error message. This is a safety mechanism to prevent you and your users from accidentally exposing your PAT in the source code. You should store the PAT as a Kestra Secret, environment variable, namespace variable or as a SECRET-type input in your flow.
---
# Synchronous Executions API in Kestra
URL: https://kestra.io/docs/how-to-guides/synchronous-executions-api
> Trigger Kestra workflow executions synchronously via the REST API. Wait for completion and retrieve outputs in a single API call for real-time integrations.
Manage the Executions API Synchronously.
There are many use cases where you may want to trigger the flow and get the flow's output in the API's response. In other words, you want the Executions API to behave synchronously.
## Executions API
Executions API is capable of creating a parametrized flow execution. Say you have the following flow:
```yaml
id: myflow
namespace: company.team
tasks:
- id: mytask
type: io.kestra.plugin.core.debug.Return
format: hello from kestra
outputs:
- id: mydata
type: STRING
value: "{{ outputs.mytask.value }}"
description: return some data
```
You invoke this flow using the Executions API as follows:
```bash
curl -X POST http://localhost:8080/api/v1/main/executions/company.team/myflow
```
By default, the Executions API is asynchronous. It will invoke the execution of the flow, and return immediately with a response that includes the Execution ID and the time at which the execution was created:
```json
{
"id": "1KWLxLeaXEXNDaXWP7YSKA",
"namespace": "company.team",
"flowId": "myflow",
"flowRevision": 1,
"state": {
"current": "CREATED",
"histories": [
{
"state": "CREATED",
"date": "2024-07-12T05:07:28.447110427Z"
}
],
"duration": "PT0.002939292S",
"startDate": "2024-07-12T05:07:28.447110427Z"
},
"originalId": "1KWLxLeaXEXNDaXWP7YSKA",
"metadata": {
"attemptNumber": 1,
"originalCreatedDate": "2024-07-12T05:07:28.447113302Z"
}
}
```
## Synchronous Executions API
To wait for an execution to finish and return the flow outputs in the response, call the Executions API with the `wait=true` query parameter. This would make the API call synchronous, and you will receive all outputs in the response that are explicitly exposed in the flow. You can invoke the Executions API in a synchronous fashion as follows:
```bash
curl -X POST 'http://localhost:8080/api/v1/main/executions/company.team/myflow?wait=true'
```
Here is the output of this API invocation:
```json
{
"id": "24znmto07B2ZGrI9IQoSSH",
"namespace": "company.team",
"flowId": "myflow",
"flowRevision": 1,
"taskRunList": [
{
"id": "4536yghIDGwqeRWZEE7AEE",
"taskId": "mytask",
"attempts": [
{
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2024-07-12T05:13:42.140Z"
},
{
"state": "RUNNING",
"date": "2024-07-12T05:13:42.140Z"
},
{
"state": "SUCCESS",
"date": "2024-07-12T05:13:42.142Z"
}
],
"duration": "PT0.002S",
"endDate": "2024-07-12T05:13:42.142Z",
"startDate": "2024-07-12T05:13:42.140Z"
}
}
],
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2024-07-12T05:13:42.011Z"
},
{
"state": "RUNNING",
"date": "2024-07-12T05:13:42.140Z"
},
{
"state": "SUCCESS",
"date": "2024-07-12T05:13:42.144Z"
}
],
"duration": "PT0.133S",
"endDate": "2024-07-12T05:13:42.144Z",
"startDate": "2024-07-12T05:13:42.011Z"
}
}
],
"outputs": {
"mydata": "hello from kestra" # ✅ this is the data that we returned in the flow
},
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2024-07-12T05:13:41.789Z"
},
{
"state": "RUNNING",
"date": "2024-07-12T05:13:42.012Z"
},
{
"state": "SUCCESS",
"date": "2024-07-12T05:13:42.335Z"
}
],
"duration": "PT0.546S",
"endDate": "2024-07-12T05:13:42.335Z",
"startDate": "2024-07-12T05:13:41.789Z"
},
"originalId": "24znmto07B2ZGrI9IQoSSH",
"metadata": {
"attemptNumber": 1,
"originalCreatedDate": "2024-07-12T05:13:41.789Z"
}
}
```
The API response includes the flow-level `outputs` (the values explicitly declared in the flow's `outputs:` block) and the execution state. Task run outputs are not included in the execution response — use `GET /api/v1/{tenant}/outputs/{executionId}/{taskRunId}` to retrieve per-task outputs.
## Authentication
You can use the same authentication mechanism for this API call as applicable to the other Kestra's APIs.
### Basic Authentication
First, base64-encode your username and password. You can do this using the following command:
```bash
echo -n "username:password" | base64
```
Then, you can use the encoded string in the `Authorization` header:
```bash
curl -X POST 'http://localhost:8080/api/v1/main/executions/company.team/myflow?wait=true' -H 'Authorization: Basic '
```
### API Token
If you're on the Enterprise Edition, you can use the API token for authentication. You can use the API token in the `Authorization` header as follows:
```bash
curl -X POST 'http://localhost:8080/api/v1/main/executions/company.team/myflow?wait=true' -H 'Authorization: Bearer YOUR_API_TOKEN'
```
Usually, you would need to include your tenant ID in the URL. Here is an example:
```bash
curl -X POST 'http://localhost:8080/api/v1/{tenant_id}/executions/company.team/myflow?wait=true' -H 'Authorization: Bearer YOUR_API_TOKEN'
```
---
# Sync Namespace Files from a Git Repository
URL: https://kestra.io/docs/how-to-guides/syncnamespacefiles
> Sync your namespace files, such as scripts and configuration, from a Git repository to Kestra using the SyncNamespaceFiles task.
Sync files from a Git Repository to Kestra with SyncNamespaceFiles Task.
The [SyncNamespaceFiles](/plugins/plugin-git/io.kestra.plugin.git.syncnamespacefiles) task is a powerful integration that allows you to **sync your namespace files with Git from the UI while still managing this process entirely in code**! Kestra unifies the development experience between the UI and code so you can combine the best of both worlds without sacrificing the benefits of version control.
The task syncs one or more namespace files from a Git repository on a schedule or anytime you push a change to a given Git branch.
## Before you begin
Before you start using the `SyncNamespaceFiles` task, ensure the following prerequisites are in place:
1. A Git repository where you want to sync your files. If you haven't pushed any files yet, see the [guide using the PushNamespaceFiles task](../pushnamespacefiles/index.md).
2. A Personal Access Token (PAT) for Git authentication.
3. A running Kestra instance in a version 0.17.0 or later with the PAT stored as a [secret](../../06.concepts/04.secret/index.md) within the Kestra instance.
## Using the `dryRun` property
Here is a sync flow that will sync the `git` namespace with files from the repository in the `_files` directory.
```yaml
id: sync_files_from_git
namespace: company.ops
tasks:
- id: sync_files
type: io.kestra.plugin.git.SyncNamespaceFiles
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/git_username/scripts
branch: main
namespace: git
gitDirectory: _files
dryRun: true
```
Given that the `dryRun` property is set to `true`, the task will only output changes from the Git repository without syncing any files to Kestra yet:

The files listed are the same ones we added in the [PushNamespaceFiles guide](../pushnamespacefiles/index.md).
## Sync all files to a single namespace from Git
Set the `dryRun` property to `false` and sync the repository with Kestra:
```yaml
id: sync_files_from_git
namespace: company.ops
tasks:
- id: sync_files
type: io.kestra.plugin.git.SyncNamespaceFiles
...
dryRun: false
```
You should see the same files from the earlier log now in Kestra:

A full list is also available in the Outputs tab:

## Set up a schedule
A common use case for this task is to set up a routine schedule to keep Kestra in sync with the Git repository. Add a [Schedule trigger](../../05.workflow-components/07.triggers/01.schedule-trigger/index.md). This example has a cron expression to execute once every hour:
```yaml
id: sync_files_from_git
namespace: company.ops
tasks:
- id: sync_files
type: io.kestra.plugin.git.SyncNamespaceFiles
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/git_username/scripts
branch: main
namespace: git
gitDirectory: _files
triggers:
- id: every_full_hour
type: io.kestra.plugin.core.trigger.Schedule
cron: "* 0 * * *"
```
## Automatically sync when a change is pushed to Git
You can also automate the syncing process by adding a [Webhook trigger](../../05.workflow-components/07.triggers/03.webhook-trigger/index.md) and creating a Webhook on your GitHub repository to trigger the flow every time something is pushed to the repository. This is useful for keeping Kestra always in sync with the repository.
```yaml
id: sync_files_from_git
namespace: company.ops
tasks:
- id: sync_files
type: io.kestra.plugin.git.SyncNamespaceFiles
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
url: https://github.com/git_username/scripts
branch: main
namespace: git
gitDirectory: _files
triggers:
- id: gh_webhook
type: io.kestra.plugin.core.trigger.Webhook
key: abcdefg
```
To setup this webhook, go to the Settings for your GitHub repository and head to Webhooks and create a new Webhook:

For the Payload URL, your URL will follow the following format:
```plaintext
https://{your_hostname}/api/v1/main/executions/webhook/company.ops/sync_files_from_git/abcdefg
```
This will require your host name to be publicly accessible. If you want to test this without having to deploy Kestra first, you can use a tool like [ngrok](https://ngrok.com/) to tunnel Kestra so GitHub can see it. As we're putting the secret in the URL, we can leave the Secret field blank.
Save and test by committing something to the Git repository.

The most recent execution was triggered by the Webhook, keeping Kestra in sync with the Git repository automatically.
If you also want to sync your flows, see the [guide on syncing flows](../syncflows/index.md).
---
# Modularize Triggers and Schedules with Terraform
URL: https://kestra.io/docs/how-to-guides/terraform-modules-for-triggers
> Scale your Kestra codebase by modularizing triggers and schedules using Terraform templates for reusable and consistent definitions.
Scale your codebase using Terraform to template and make scheduling a breeze
As shown in the [terraform templating](../terraform-templating/index.md) guide, you can use Terraform to template and define flows.
Managing triggers and schedules can be a **tedious task**, especially when you have a lot of flows generating **peak hours** due to reuse of same trigger schedules.
This guide will show you how to use Terraform to define triggers and schedules for your flows with modularity.
Note: we created the repo [kestra-flows-template](https://github.com/kestra-io/kestra-flows-template) for you to directly start from a very scalable codebase.
## Code structure
```plaintext
.
└── environment/
├── development
├── production/ # Contains subfolders defining Kestra flows resources
│ ├── airbyte/
│ └── ...
├── modules/ # Terraform modules to be used in environments
│ ├── trigger_cron/
│ ├── trigger_cron_hourly_random/
│ ├── trigger_flow/
│ ├── trigger_webhook/
│ └── ...
```
Use `null_resource` to create reusable resources to DRY (Do not Repeat Yourself) your trigger definitions.
With Terraform version >= 1.4, you can use `terraform_data` instead. For backward compatibility, this guide uses `null_resource`.
## Example of Cron schedule implementation
Below an example of implementation for a Terraform module that defines a cron schedule trigger.
`triggers.yml`
```yaml
triggers:
- id: ${cron-name}
type: io.kestra.plugin.core.trigger.Schedule
cron: "${cron-expression}"
lateMaximumDelay: "${late-maximum-delay}"
```
`main.tf`
```hcl
resource "null_resource" "trigger_cron" {
triggers = {
value = templatefile("${path.module}/triggers.yml", {
cron-name = var.cron_name
cron-expression = var.cron_expression
late-maximum-delay = var.late_maximum_delay
})
}
}
```
`variables.tf`
```hcl
variable "cron_expression" {
type = string
description = "Cron expression or supported expression like : @hourly"
default = null
}
variable "cron_name" {
type = string
description = "Provide a description of your Cron expression for simplicity"
default = null
}
variable "late_maximum_delay" {
type = string
description = "Allow to disable auto-backfill : if the schedule didn't start after this delay, the execution will be skipped."
}
```
`outputs.tf`
```hcl
output "trigger_content" {
value = null_resource.trigger_cron.triggers.value
}
```
Usage of this module would look like :
```hcl
module "trigger_purge" {
source = "../../../../modules/trigger_cron"
cron_expression = "0 0 * * 0"
cron_name = "weekly_kestra_purge"
late_maximum_delay = "PT1H"
}
module "my_flow_module" {
source = "../../../../modules/my_flow_module"
trigger = module.trigger_purge.trigger_content
}
```
Module `my_flow_module` will use the trigger defined in `trigger_purge` module.
## Scaling your codebase with Terraform trigger modules
You can check here an example [using cron trigger using fully terraform](https://github.com/kestra-io/kestra-flows-template/blob/b6937f9d95970a4e909687eb64936f5ea3f02c1c/environment/production/dbt/jaffle_shop_classic.tf#L28).
---
# Terraform for Flow Modularity in Kestra
URL: https://kestra.io/docs/how-to-guides/terraform-templating
> Combine Kestra with Terraform for IaC workflows. Template and provision cloud resources automatically as part of your CI/CD automation pipelines.
Scale your codebase using Terraform to template and define flows
This guide shows how to use Terraform's HCL (Hashicorp Configuration Language) templating features in a Kestra codebase.
To make your codebase accessible to users unfamiliar with Kestra syntax, encapsulate most of the logic and DSL (Domain-specific programming language) into [Terraform modules](https://developer.hashicorp.com/terraform/language/modules).
This quick tutorial, will show you how templating capabilities brought by Terraform can help you :
- DRY (Do Not Repeat Yourself) your codebase
- Facilitate onboarding on Kestra
- Incorporate extra modularity
- Implement complex pipelines while keeping syntax clear
You can check the [kestra-flows-template](https://github.com/kestra-io/kestra-flows-template) repo which contains a set of modules and subflows to help you get started with Terraform.
This guide covers creating a Terraform module and a subflow, and how to use them in your codebase.
## Code structure
```plaintext
.
└── environment/
├── development
├── production/ # Contains subfolders defining Kestra flows resources
│ ├── airbyte/
│ ├── dbt/
│ ├── triggers/
│ ├── main.tf # Instantiate each folder (airbyte, dbt ...)
│ └── ...
├── modules/ # Terraform modules to be used in environments
│ ├── airbyte_sync/
│ ├── trigger_cron/
│ └── ...
└── subflows/ # Kestra subflows
├── main.tf
├── sub_cloud_sql_airbyte_query.yml
└── ...
```
Modules are folders under `modules` folder and can be instantiated either in `development` or `production` environments.
They only expose variables that are meant to be changed for usage purpose.
Inside a module, you can define a `main.tf` file that will define the resources to be created.
## Creating a module, example with Airbyte
Create a module that defines a Kestra flow to sync data from Airbyte.
## Tree structure of a Terraform module
```plaintext
.
└── airbyte_sync/
├── main.tf
├── tasks.yml
└── variables.tf
```
### `main.tf` contains the kestra_flow terraform resource, which will define the flow using a templated YAML file
```hcl
resource "kestra_flow" "airbyte_sync" {
keep_original_source = true
flow_id = var.flow_id
namespace = var.namespace
content = join("", [
yamlencode({
id = var.flow_id
namespace = var.namespace
labels = var.priority != null ? merge(var.labels, { priority = var.priority }) : var.labels
description = var.description
}),
templatefile("${path.module}/tasks.yml", {
description = var.description
airbyte-url = var.airbyte_url
airbyte-connections = var.airbyte_connections
max-duration = var.max_sync_duration
late-maximum-delay = var.late_maximum_delay
cron-expression = var.cron_expression
}),
var.trigger,
])
}
```
## `variables.tf` will contain all the variables that can be passed to the module with appropriate validation and description
```hcl
variable "airbyte_connections" {
description = "List of Airbyte connections to trigger : id (can be found in URL), name is whatever makes sense"
type = list(object({
name = string
id = string
}))
validation {
condition = length(var.airbyte_connections) > 0 && length([
for o in var.airbyte_connections : true
if length(regexall("^[A-Za-z_]+$", o.name)) > 0
]) == length(var.airbyte_connections)
error_message = "At least one connection should be provided, and connection names should not contain hyphens."
}
}
variable "flow_id" {
type = string
}
variable "description" {
type = string
}
variable "namespace" {
type = string
default = "blueprint"
}
variable "airbyte_url" {
type = string
}
variable "trigger" {
type = string
description = "String containing triggers sections of the flow"
default = ""
}
variable "max_sync_duration" {
type = string
description = "Tell Kestra to wait logs for this max duration"
default = ""
}
variable "labels" {
type = map(string)
default = null
description = "Labels to apply to the flow"
}
variable "priority" {
type = string
default = null
description = "Priority tag to apply to the flow"
}
variable "cron_expression" {
type = string
description = "Cron expression or supported expression like : @hourly"
default = null
}
variable "late_maximum_delay" {
type = string
description = "Allow to disable auto-backfill : if the schedule didn't start after this delay, the execution will be skip."
}
```
## `tasks.yml`: flow definition with Terraform templatefile jinja syntax
```yaml
tasks:
## Here we leverage the Terraform templating capabilities to generate the tasks
## Using jinja-like syntax, we can loop over the list of connections and generate tasks for each of them
%{ for connection in airbyte-connections ~}
- id: "trigger_${connection.name}"
type: io.kestra.plugin.airbyte.connections.Sync
connectionId: ${connection.id}
url: "${airbyte-url}"
httpTimeout: "PT1M"
wait: false
- id: "check_${connection.name}"
type: io.kestra.plugin.airbyte.connections.CheckStatus
url: "${airbyte-url}"
jobId: "{{ outputs.trigger_${connection.name}.jobId }}"
pollFrequency: "PT1M"
httpTimeout: "PT1M"
retry:
type: constant
interval: PT1M
maxAttempts: 5
%{ if length(max-duration) > 0}
maxDuration: "${max-duration}"
%{ endif }
%{ endfor ~}
triggers:
- id: cron_trigger
type: io.kestra.plugin.core.trigger.Schedule
cron: "${cron-expression}"
lateMaximumDelay: "${late-maximum-delay}"
```
## Using the module in a Terraform environment
Using the module will look like this :
```hcl
module "stripe_events_incremental" {
source = "../../../modules/airbyte_sync"
flow_id = "stripe_events"
priority = "high"
namespace = local.namespace
description = "Stripe Events"
airbyte_connections = [
{
name = "stripe_events_incremental"
id = module.airbyte_connection_stripe_offical.connection_id
}
]
max_sync_duration = "PT30M"
airbyte_url = var.airbyte_url
cron_expression = "@hourly"
late_maximum_delay = "PT1H"
}
```
It is now easy to instantiate the module in your `main.tf` file, and to expose only the variables that are meant to be changed:
- `flow_id`: the flow id
- `namespace`: the namespace to save the flow in
- `description`: the description
- `airbyte_connections`: the list of Airbyte connections to trigger in a linear order
- `max_sync_duration`: the maximum duration to wait for logs
- `airbyte_url`: the Airbyte URL of the instance
- `cron_expression`: the cron expression to trigger the flow
- `late_maximum_delay`: the maximum delay to wait for the flow to start, in case of missed schedules (backfill)
In case of changes in the way you want to implement the underlying tasks, you can modify the Terraform module without changing the interface (variables).
## Subflow example: query and display results for a given Postgres database
Subflows are a way to encapsulate logic and make it reusable across your codebase.
Here is an example of a subflow that will query a Cloud SQL instance:
```yaml
id: query_my_postgres_database
namespace: company.team
description: "Query Postgres database and display results in logs"
inputs:
- id: sqlQuery
type: STRING
defaults: "SELECT * FROM public.jobs ORDER BY created_at desc limit 1" # SQL query example
tasks:
- id: query_data
type: io.kestra.plugin.jdbc.postgresql.Query
url: jdbc:postgresql://MY_HOST/MY_DATABASE
username: MY_USER
password: "{{ secret('my-postgres-password') }}"
sql: "{{ inputs.sqlQuery }}"
fetchType: FETCH
- id: show_result
type: io.kestra.plugin.core.log.Log
message: |
{% for row in outputs.query_data.rows %}
{%- for key in row.keySet() -%}
{{key}} : {{row.get(key)}} |
{%- endfor -%}
\n
{% endfor %}"
## To make it easier to use the results in another flow
## we expose the query result by using `outputs`
outputs:
- id: query_result
value: "{{ outputs.query_data.rows }}"
type: JSON
```
You can either execute this subflow as is, or use it in another flow to avoid repeating the same logic.
Executing the subflow will prompt you to enter the SQL query you want to execute :

## Using the subflow in a flow
```yaml
- id: query_last_job
type: io.kestra.plugin.core.flow.Subflow
namespace: company.team
flowId: query_my_postgres_database
inputs:
sqlQuery: "SELECT * FROM public.jobs ORDER BY created_at desc limit 1"
wait: true
transmitFailed: true
- id: use_result
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.query_last_job.outputs.query_result }}"
```
1. Connection details are stored in the subflow, and only the SQL query is exposed to the user.
1. Subflow natively displays results in logs for easy debugging.
1. Outputs of the subflow can be used in the parent flow by using `outputs.query_data.rows` in the `show_result` task.
> Note: `wait: true` will wait for the subflow to finish before continuing the flow execution. `transmitFailed: true` will transmit the failed status of the subflow to the parent flow.
Parent flow logs will display tasks from subflow directly:

## Subflows vs Terraform templating
Subflows hide unnecessary details to their users, abstracting connection details, logging and such for a given set of tasks.
Terraform modules allow you to define complex flows in a modular way. Also it supports passing outputs from one Terraform resource to another across systems (Airbyte terraform resource output to Kestra module input variable) and strongly validate inputs which is not possible with subflows.
## Conclusion
Terraform templating is a powerful way to define flows in a modular way, and to expose only the variables that are meant to be changed.
It is a great way to make your codebase more maintainable and to facilitate onboarding for users unfamiliar with Kestra syntax.
---
# Kestra with Pulumi's Terraform Provider
URL: https://kestra.io/docs/how-to-guides/using-pulumis-terraform-provider
> Integrate Kestra infrastructure management into your Pulumi projects using Pulumi's Terraform Provider bridge.
Utilize Pulumi's Terraform Provider to manage Kestra infrastructure.
This post outlines the process of leveraging Pulumi's terraform-provider package to seamlessly integrate the Kestra Terraform provider into your Pulumi projects. This approach allows you to manage Kestra resources using the familiar Pulumi infrastructure-as-code workflow, even if the provider isn't officially published in the main Pulumi Registry.
## About the example repository
The [pulumi-kestra-example](https://github.com/japerry911/pulumi-kestra-example) repository is a hands-on example that shows how to provision and manage Kestra resources with Pulumi using a Python-based provider and SDK generated locally. The repo includes:
- a Pulumi project YAML
- a complete example flow and namespace
- an app - demonstrating a real-world use case: uploading a file to Google Cloud Storage via a Kestra flow and app
Note the flow.yaml does not perform an actual GCS File Upload, and that portion is commented out. This blog post is meant to prioritize understanding of the Pulumi Terraform-Provider.
## Step-by-step process
Follow these steps to set up your environment and begin managing Kestra resources with Pulumi:
1. Clone/Fork the Example Repository
- Start by cloning or forking the example repository, which provides a foundational structure for your project: https://github.com/japerry911/pulumi-kestra-example
2. Download and Install Pulumi
- If you haven't already, install the Pulumi CLI on your system:
- Mac: `brew install pulumi/tap/pulumi`
- Linux: `curl -fsSL https://get.pulumi.com | sh`
- Windows: Refer to the official Pulumi documentation for installation instructions: [Pulumi Docs](https://www.pulumi.com/docs/get-started/download-install/)
3. Create a Pulumi Python Project
- For this example, we'll be using Python. You can create a new Pulumi Python project with the following command: `pulumi new python`
- During the project creation process, you will be prompted to:
- Log in with your browser or an access token.
- Fill in project-specific options such as project name, stack, and preferred package management tool.
4. Add the Terraform Provider
- Pulumi's `pulumi package add terraform-provider ` command is a powerful feature that utilizes local packages. This command instantly generates a language-specific SDK for any existing Terraform or OpenTofu provider directly within your project. This means you can use providers in your Pulumi code even if they are not officially published in the main Pulumi Registry.
- Execute the following command in your terminal to add the Kestra Terraform provider: `pulumi package add terraform-provider kestra-io/kestra`
- This command downloads the specified provider (kestra-io/kestra) and creates all the necessary wrapper code in a local directory (e.g., ./sdks/), enabling you to immediately manage that provider's resources as part of your Pulumi infrastructure.
5. Install the Local SDK
- Now that you have a local SDK in your project's `sdks` folder, you need to install it into your local Python virtual environment. (If you're using a different language project, you'll need to follow the equivalent installation steps for that language.)
- Add the SDK path to your requirements.txt file: `echo sdks/kestra >> requirements.txt`
- Install the project dependencies: `pulumi install`
6. Create and Fill .env File
- Create a .env file based on a .env.local template. This file will hold your Kestra secrets and provider URL.
```bash
## Kestra secrets
## API Token is required (Enterprise-only),
## or Username AND Password are required
KESTRA_API_TOKEN=
KESTRA_USERNAME=
KESTRA_PASSWORD=
## Kestra Provider URL for Provider declaration
KESTRA_PROVIDER_URL=
```
- Fill in the appropriate values for `KESTRA_API_TOKEN`, `KESTRA_USERNAME`, `KESTRA_PASSWORD`, and `KESTRA_PROVIDER_URL` based on your Kestra instance edition.
7. Prepare for Resource Building
- With the local Pulumi SDK for the Kestra Terraform provider set up and installed, install some additional Python packages before defining your resources:
- Activate your Python environment: `source venv/bin/activate`
- Install python-dotenv and PyYAML: `pip install python dotenv PyYaml`
8. Build and Deploy
- You are now ready to build and deploy your Kestra resources using Pulumi!
- Execute the following command: `pulumi up`
- This will initiate the deployment process, and Pulumi will provision your Kestra resources as defined in your project.
## What did we provision?
## Conclusion
By following these steps, you can effectively integrate the Kestra Terraform provider into your Pulumi workflows, allowing for robust and consistent management of your Kestra infrastructure.
Thank you for reading, Happy Coding!
---
# Access Values Between Flows
URL: https://kestra.io/docs/how-to-guides/values-between-flows
> Share data across Kestra flows using Subflows, KV Store, and Namespace Variables. Learn best patterns for passing values between different workflows.
How to access values across different flows.
Sometimes it's useful to store values so they can be used across multiple flows. Whether that's configuration or state generated by similar flows, accessing values between flows has many benefits. There are multiple ways to do that in Kestra, each with different advantages depending on the use case.
There are three different ways you can access values across different flows:
1. Subflows
2. KV Store
3. Namespace Variables
## Subflows
Using [Subflows](../../05.workflow-components/10.subflows/index.md), you can execute one flow from another flow. As part of that, you can pass inputs from the parent flow to the subflow and retrieve outputs generated from it. This is useful if you want multiple flows to execute together and interact directly with one another. However, this doesn't work if you want one flow to generate a value and another flow to use it later when it executes.
In this example, our parent flow is passing the [variable](../../05.workflow-components/04.variables/index.md) `debug` into the subflow as an [input](../../05.workflow-components/05.inputs/index.md). On top of that, the subflow returns an [output](../../05.workflow-components/06.outputs/index.md) `subflow_output` too.
```yaml
id: parent_flow
namespace: company.team
variables:
debug: true
tasks:
- id: subflow
type: io.kestra.plugin.core.flow.Subflow
flowId: subflow
namespace: company.team
inputs:
debug: "{{ vars.debug }}"
- id: log
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.subflow.outputs.subflow_output }}"
```
In the subflow, the output is explicitly defined to make it accessible in the parent flow. This example uses the input to generate the output sent to the parent flow.
```yaml
id: subflow
namespace: company.team
inputs:
- id: debug
type: BOOLEAN
tasks:
- id: return
type: io.kestra.plugin.core.debug.Return
format: "Subflow: {{ inputs.debug }}"
outputs:
- id: subflow_output
type: STRING
value: "{{ outputs.return.value }}"
```
## KV Store
Using the [KV Store](../../06.concepts/05.kv-store/index.md), you can set and get values across different flows. This is good if you want to be able to store values without flows directly interacting with one another, like they do with Subflows. Flows can use the Get and Set tasks to make themselves stateful, allowing one flow to store the state, and another to access it when it wants. However, this approach isn't ideal if you don't want these values to be modified by the flows directly.
For example, you can use `io.kestra.plugin.core.kv.Set` task as well as use the UI interface to manage the values in the KV Store. To access them, you can use the `io.kestra.plugin.core.kv.Get` task which will return them as an output.
```yaml
id: kv_store
namespace: company.team
variables:
debug: true
tasks:
- id: set
type: io.kestra.plugin.core.kv.Set
key: debug
value: "{{ vars.debug }}"
namespace: "{{ flow.namespace }}"
- id: get
type: io.kestra.plugin.core.kv.Get
key: debug
- id: log
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.get.value }}"
```
## Namespace Variables
:::alert{type="info"}
This is an [Enterprise Edition](../../07.enterprise/index.mdx) feature.
:::
Using [Namespace Variables](../../07.enterprise/02.governance/07.namespace-management/index.md), you can define values that can be accessed between flows in a namespace, similar to the KV Store. However, these can only be set in the [Namespace page](../../07.enterprise/02.governance/07.namespace-management/index.md). This is useful if you want to access values across flows but do not want to update them dynamically inside your flows at the same time.
For example, we can define our variables as a key-value pair in our Namespace:
```yaml
debug: true
state: failed
hello: world
```
We can access them using the `{{ namespace.var_key }}` expression where `var_key` is the key of our key-value pair.
```yaml
id: global_variables
namespace: company.team
variables:
debug: true
tasks:
- id: debug_1
type: io.kestra.plugin.core.log.Log
message: "Namespace: {{ namespace.state }}"
- id: debug_2
type: io.kestra.plugin.core.log.Log
message: "Local: {{ vars.debug }}"
```
---
# Use HashiCorp Vault as a Read-Only Secrets Backend in Kestra
URL: https://kestra.io/docs/how-to-guides/vault-read-only-secrets
> Configure HashiCorp Vault as a read-only secrets backend in Kestra Enterprise so workflows read from Vault without Kestra managing the secret lifecycle.
Configure HashiCorp Vault as a read-only secrets backend so Kestra reads secrets from an existing Vault instance without managing their lifecycle.
## Prerequisites
- A running Vault instance with the [KV Secrets Engine v2](https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2) enabled.
- A Kestra Enterprise namespace configured to use a dedicated secrets manager.
- A Vault token with read access to the relevant secret paths.
For background on how read-only mode differs from managed mode, see [Secrets manager modes](../../07.enterprise/02.governance/secrets-manager/index.md#secrets-manager-modes).
## Vault secret structure
Kestra reads secrets from Vault using a path-based structure. A Vault secret engine can host multiple secret paths, and each path contains key-value pairs (subkeys).
```plaintext
secret/
├── app1/
│ ├── db/ ← secret path (visible as a secret name in Kestra)
│ │ ├── DATABASE_USERNAME ← subkey
│ │ └── DATABASE_PASSWORD ← subkey
│ └── api/
│ └── API_TOKEN
└── app2/
└── config
```
- `secret` — the KV engine name (`root-engine` in Kestra config).
- `app1`, `app2` — path prefixes (set via `secret-path-prefix`).
- `db`, `api`, `config` — secret names visible in Kestra UI.
- `DATABASE_USERNAME`, `API_TOKEN` — subkey names referenced in flows.
## Configure Vault as a read-only backend
In Kestra, navigate to the namespace you want to connect to Vault and open the **Edit** tab. Under **Dedicated secrets manager**, enter the Vault configuration:

The equivalent YAML configuration is:
```yaml
kestra:
secret:
type: vault
vault:
address: https://my-vault:8200/
root-engine: secret
secret-path-prefix: app1
token:
token: my-vault-access-token
read-only: true
```
`secret-path-prefix: app1` limits Kestra's view to secrets under the `app1` path — in this example, `db` and `api`.
After saving, the **Secrets** tab shows the available secret paths with a lock icon, confirming read-only mode is active. No new secrets can be created and existing ones cannot be edited from Kestra:

## Read a Vault secret in a flow
Use the `secret()` function with a `subkey` parameter to access a specific key within a Vault secret path.
In this example, `my-app` is the Vault secret (path visible in Kestra) and `NEON_PASSWORD` is the subkey:

Reference it in a flow:
```yaml
{{ secret('my-app', subkey='NEON_PASSWORD') }}
```
Here is a complete flow example that uses the Vault-backed secret to connect to a [Neon](../neon/index.md) PostgreSQL database:
:::collapse{title="Expand for full flow YAML"}
```yaml
id: neon-db
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: create_columns
type: io.kestra.plugin.jdbc.postgresql.Queries
sql: |
ALTER TABLE kestra_example_secret
ADD COLUMN order_id int,
ADD COLUMN customer_name text,
ADD COLUMN customer_email text,
ADD COLUMN product_id int,
ADD COLUMN price double precision,
ADD COLUMN quantity int,
ADD COLUMN total double precision;
- id: copy_in
type: io.kestra.plugin.jdbc.postgresql.CopyIn
url: "jdbc:postgresql://ep-ancient-flower-a2e73um1-pooler.eu-central-1.aws.neon.tech/neondb?user=neondb_owner&password={{ secret('my-app', subkey='NEON_PASSWORD') }}"
table: "kestra_example_secret"
from: "{{ outputs.download.uri }}"
header: true
columns: [order_id,customer_name,customer_email,product_id,price,quantity,total]
delimiter: ","
```
:::
After execution, Kestra confirms it read `NEON_PASSWORD` from Vault and inserted 100 rows:


## What happened in Vault
Vault held the secret at the path `business-unit/my-app` (in the `admin` namespace, if using Vault Enterprise):

Kestra resolved `NEON_PASSWORD` from that path at runtime. The value is not persisted; it is cached in memory (~60s, configurable).
## Related
- [Secrets manager configuration reference](../../07.enterprise/02.governance/secrets-manager/index.md)
- [Secrets manager modes](../../07.enterprise/02.governance/secrets-manager/index.md#secrets-manager-modes)
---
# Set Up Webhooks to Trigger Flows
URL: https://kestra.io/docs/how-to-guides/webhooks
> Trigger Kestra workflows via webhooks. Configure webhook listeners to start flows in response to GitHub events, Slack commands, or any HTTP POST request.
Execute flows using the Webhooks Trigger.
Webhooks are HTTP requests that are triggered by an event. These are useful for being able to tell another application to do something, such as starting the execution of a Flow in Kestra.
If your provider sends an idempotency key header (e.g., `Idempotency-Key`), map it to `system.correlationId` and add a duplicate guard as shown in [Idempotency with correlation IDs](../idempotency/index.md) to prevent double-processing.
## Using Webhooks in Kestra
You can use webhooks to trigger an execution of your flow in Kestra. To do this, we can make a [trigger](../../05.workflow-components/07.triggers/03.webhook-trigger/index.md) with the type `io.kestra.plugin.core.trigger.Webhook`.
Once we've done this, we can add a `key` property, which can be random as this will be used to trigger the webhook. In the example, the `key` is set to `1KERKzRQZSMtLdMdNI7Nkr` which is what we put at the end of our webhook URL to trigger it.
```yaml
id: webhook_example
namespace: company.team
description: |
Example flow for a webhook trigger.
This endpoint doesn't need any login / password and is secured by `key` that is different for every flow
tasks:
- id: out
type: io.kestra.plugin.core.debug.Return
format: "{{ trigger | toJson }}"
triggers:
- id: webhook_trigger
type: io.kestra.plugin.core.trigger.Webhook
# the required key to start this flow - might be passed as a secret
key: 1KERKzRQZSMtLdMdNI7Nkr
```
The format of the Webhook URL follows:
`https://{your_hostname}/api/v1/main/executions/webhook/{namespace}/{flow_id}/{key}`
where:
- `your_hostname` is the domain or IP of your server, e.g. example.com
- `namespace` is `io.kestra.demo.flows`
- `flow_id` is `webhook_example`
- `key` is `1KERKzRQZSMtLdMdNI7Nkr`
With this information, you can test your flow by running the following command in the terminal to trigger the flow:
```bash
curl http://localhost:8080/api/v1/main/executions/webhook/company.team/webhook_example/1KERKzRQZSMtLdMdNI7Nkr
```
You can also copy the formed Webhook URL from the **Triggers** tab.
## Webhooks in Kestra EE
Use Kestra Secrets to store the webhook key. From the left navigation menu on the Kestra UI, navigate to `Namespaces`. Click on the namespace under which you want to create the flow with the webhook trigger. We will use `company.team` namespace for this example. On the corresponding namespace page, navigate to the `Secrets` tab. Click on the `New secret` button at the top, and create a new secret with `Key` as `WEBHOOK_KEY` (you may choose any appropriate name) and `Secret` as the webhook key value. Let us use `1KERKzRQZSMtLdMdNI7Nkr` for this example. Once you've done that, save the secret.


Create the flow in the same namespace where you defined the `WEBHOOK_KEY` secret. The flow will use the webhook trigger, like this:
```yaml
id: webhook_ee_example
namespace: company.team
description: |
Example flow for a webhook trigger in Kestra EE.
This endpoint doesn't need any login / password and is secured by `key` that is different for every flow
tasks:
- id: out
type: io.kestra.plugin.core.debug.Return
format: "{{ trigger | toJson }}"
triggers:
- id: webhook_trigger
type: io.kestra.plugin.core.trigger.Webhook
# the required key to start this flow - might be passed as a secret
key: "{{ secret('WEBHOOK_KEY') }}"
```
In the `triggers` section of the flow, the secret is referenced in the `key` as `{{ secret('WEBHOOK_KEY') }}` rather than hardcoding the webhook key directly.
The format of the Webhook URL follows:
`https://{your_hostname}/api/v1/{tenant_id}/executions/webhook/{namespace}/{flow_id}/{key}`
where:
- `your_hostname` is the domain or IP of your server, e.g. example.com
- `tenant_id` is the tenant ID belonging to your Kestra EE account
- `namespace` is `company.team``
- `flow_id` is `webhook_ee_example`
- `key` is `1KERKzRQZSMtLdMdNI7Nkr`
With this information, you can test your flow by running the following command in the terminal to trigger the flow:
```bash
curl http://localhost:8080/api/v1/my_tenant/executions/webhook/company.team/webhook_ee_example/1KERKzRQZSMtLdMdNI7Nkr
```
---
# Install Kestra: Docker, Kubernetes, VM, and JAR
URL: https://kestra.io/docs/installation
> Overview of Kestra installation methods, including Docker, Kubernetes, Virtual Machines, and Standalone JAR.
import ChildCard from "~/components/docs/ChildCard.astro"
Install Kestra using the method that fits your environment.
## Which installation method should I choose?
Not sure where to start?
- **New to Kestra or evaluating it?** Start with [Docker](./02.docker/index.md) — it runs Kestra locally on a laptop with a single command.
- **Want persistent local storage with a PostgreSQL database?** Use [Docker Compose](./03.docker-compose/index.md).
- **Running in production?** Choose [Kubernetes](./03.kubernetes/index.md) (or a managed equivalent on [AWS EKS](./04.kubernetes-aws-eks/index.md), [GCP GKE](./05.kubernetes-gcp-gke/index.md), or [Azure AKS](./06.kubernetes-azure-aks/index.md)) for a scalable, distributed deployment.
- **Prefer a single VM or on-prem server?** Deploy on [AWS EC2](./08.aws-ec2/index.md), [GCP VM](./09.gcp-vm/index.md), [Azure VM](./10.azure-vm/index.md), or a [DigitalOcean Droplet](./11.digitalocean-droplet/index.md).
:::alert{type="info"}
**Minimum system requirements:** Kestra requires at least **2 vCPUs and 4 GiB of memory** to run correctly. Local Docker installs run comfortably on a typical laptop; size production deployments above this baseline based on your workload.
:::
You can deploy Kestra from a laptop or on-prem server to a distributed cluster in a public cloud. Some plugins such as the [Script plugin](../16.scripts/index.mdx) require Docker-in-Docker (DinD), which is not supported in some environments like AWS Fargate. For production, use Kubernetes or a virtual machine.
If you prefer a fully managed platform without the overhead of infrastructure maintenance, [Kestra Cloud](/cloud) is available now. See the [Cloud getting started guide](../07.enterprise/07.cloud-getting-started/index.md) to set up your first instance.
## Choose how to install Kestra for your environment
---
# Deploy Kestra on AWS EC2 – RDS and S3 Backend
URL: https://kestra.io/docs/installation/aws-ec2
> Install Kestra on AWS EC2 using Amazon RDS for the database and S3 for internal storage backend.
Install Kestra on AWS EC2 with PostgreSQL RDS database and S3 internal storage backend.
:::alert{type="info"}
Prefer a one-click option? You can launch Kestra directly from the [AWS Marketplace listing](https://aws.amazon.com/marketplace/pp/prodview-uilmngucs45cg).
:::
## Prerequisites
- Basic knowledge about using a command line interface
- Basic knowledge about EC2, S3, and PostgreSQL.
You can find the corresponding [full Terraform configuration in this repository](https://github.com/kestra-io/deployment-templates/tree/main/aws).
## Step 1: Create an EC2 instance & install Docker
First, create an EC2 instance. To do so, [go to the AWS console and choose EC2](https://eu-north-1.console.aws.amazon.com/ec2/home).
1. Give a name to your instance.
2. Choose Ubuntu as your OS.
3. Instance type: Kestra requires at least 4GiB of memory and 2 vCPUs to run correctly. Choosing t3-medium is a good starting point.
4. Create a key-pair to securely connect to your instance. This key is needed to connect through SSH in the following steps.
5. Create a security group that allows SSH traffic from your IP and also allow HTTPS traffic.



You can now click on **Launch instance** and wait a few seconds for the instance to be up and running.
Once running, open a terminal on your laptop and connect to your instance through SSH: `ssh -i ubuntu@`
Kestra can be run directly from the `.jar` binary or using Docker. We use Docker here for quicker setup:
1. Install Docker on the EC2 instance. [You can find the last updated instruction on the Docker website](https://docs.docker.com/engine/install/ubuntu/).
2. [Install Docker Compose](https://docs.docker.com/compose/install/).
To check your installation, run `docker version` and `docker compose version`. You're now ready to download and launch the Kestra server.
## Step 2: Download and run Kestra
Download the official Docker-Compose file:
```bash
curl -o docker-compose.yml \
https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml
```
Use an editor such as Vim to modify the `docker-compose.yml` and set basic authentication to `true` and configure your basic authentication credentials to secure your Kestra instance. Make sure to add a valid email address too.
```yaml
kestra:
server:
basic-auth:
enabled: true
username: admin@kestra.io # it must be a valid email address
password: kestra
```
Next, use the following command to start the Kestra server:
```bash
docker compose up -d
```
## Step 3: Allow external traffic
Kestra is now running and the Kestra server is exposing traffic on the `8080` port. To connect through your web browser, update the inbound traffic rules in the EC2 security group.
Go to the EC2 console and select Security Group. Choose the security group attached to your EC2 instance and add a new inbound rule to open access to the `8080` port. If you did not select an existing security group when creating the instance, the security group will be prefixed with "launch-wizard-".
If you want to only allow traffic coming from your IP address, set the source to your own IP. If you want to make it open to the entire Internet, leave it at `0.0.0.0`.
:::alert{type="warning"}
Note that if you haven't set up basic authentication in the previous step, your Kestra instance will be publicly available to anyone without any access restriction.
:::

You can now access your Kestra instance and create, edit, and run Flows.
## Step 4: Use AWS RDS PostgreSQL as a database backend
This first installation relies on a PostgreSQL database running alongside the Kestra server on the EC2 instance (see the PostgreSQL service running in Docker Compose).
For a simple proof of concept (PoC), you can keep the PostgreSQL database running in Docker.
However, for a production-grade installation, we recommend a managed database service such as [AWS RDS](https://aws.amazon.com/rds/).
**Create an AWS RDS database**
1. Go to the [RDS console](https://eu-north-1.console.aws.amazon.com/rds/home).
2. Create a database and choose PostgreSQL (Kestra also supports MySQL, but PostgreSQL is recommended)
3. Set a username and password.
4. On the connectivity configuration choose “Connect to an EC2 compute resource” and choose your EC2 instance.
5. Also select the existing DB subnet group and existing VPC security group and choose the one attached to your EC2 instance.
6. Fine-tune instance class and storage type to avoid incurring AWS costs. For a first step, a small PostgreSQL instance is enough.
6. Click create and wait for completion



**Create Kestra database**
Before attaching your Kestra server to the new database backend, initialize the database with a base schema as follows:
1. Connect to your EC2 instance with ssh.
2. Install a PostgreSQL client: `sudo apt-get install postgresql-client`.
3. Create the Kestra database: `createdb -h -U -p 5432 kestra`.
**Update Kestra configuration**
In the Docker compose configuration, edit the `datasources` property of the Kestra service in the following way:
```yaml
datasources:
postgres:
url: jdbc:postgresql://:5432/kestra
driver-class-name: org.postgresql.Driver
username:
password:
```
Because you now use the RDS service, you do not need the PostgreSQL service anymore. Remove it from the `docker-compose.yml` file.
For the changes to take effect, restart the Docker services with `docker compose restart` or `docker compose up -d`.
## Step 5: Use AWS S3 for storage
By default, internal storage is implemented using the local file system. This section guides you how to change the storage backend to S3 to ensure a more reliable, durable, and scalable storage.
1. Go to the S3 console and create a bucket.
2. Go to IAM and create a new User Group with AWS S3 full access.
3. Create a new user and attach it to the user group.
4. For the new user, go to **Security Credentials** and create an access key. Choose “Application running on an AWS compute service” and retrieve the access and secret keys.
5. Edit the Kestra storage configuration.
```yaml
kestra:
storage:
type: s3
s3:
access-key: ""
secret-key: ""
region: ""
bucket: ""
```
6. Restart docker services.

For more information on S3 storage configuration, check out the [Runtime and Storage configuration guide](../../configuration/02.runtime-and-storage/index.md).
## Next steps
This setup provides the easiest starting point for running Kestra in production on a single machine. For a deployment to a distributed cluster, check the [Kubernetes deployment guide](../03.kubernetes/index.md).
Reach out via [Slack](/slack) if you encounter any issues, or if you have any questions regarding deploying Kestra to production.
Make sure to also check the [CI/CD guide](../../version-control-cicd/cicd/index.md) to automate your workflow deployments based on changes in Git.
---
# Deploy Kestra on Azure VM – Azure Database Backend
URL: https://kestra.io/docs/installation/azure-vm
> Deploy Kestra on an Azure Virtual Machine using Azure Database for PostgreSQL and Azure Blob Storage.
Install Kestra on an Azure VM with Azure Database for PostgreSQL as the database backend and Azure Blob Storage as the internal storage backend.
:::alert{type="info"}
Prefer an Azure-native option? You can deploy Kestra directly from the [Azure Marketplace listing](https://marketplace.microsoft.com/en-us/product/AzureApplication/kestra_technologies.kestra-open-source-official).
:::
## Deploy Kestra on an Azure VM with Azure Database
Prerequisites:
- Basic command-line interface (CLI) skills.
- Familiarity with Azure and PostgreSQL.
## Create an Azure VM
First, create a virtual machine using Azure Virtual Machines. To do so, go to the Azure portal and choose [Virtual Machines](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Compute%2FVirtualMachines).
1. Click **Create** and select **Azure Virtual Machine**.
2. Choose an appropriate **Subscription** and **Resource Group**.
3. Give a name for your VM, and choose a **Region** where it should be launched.
4. For **Availability options**, choose **Availability zone**, and keep the default availability zone.
5. For **Image**, choose **Ubuntu Server 22.04 LTS - x64 Gen2**, and **x64** as the VM architecture.
6. Kestra requires at least 4GiB of memory and 2 vCPUs to run correctly. Choosing the **Size** as **Standard_D2s_v3** is a good starting point.
7. Select **SSH public key** as the **Authentication type**.
8. You can keep the default `azureuser` as the **Username**.
9. For **SSH public key source**, you can select **Generate new key pair** and provide an appropriate name for the key pair.
10. For **Public inbound ports**, choose **Allow selected ports** and from the **Select inbound ports** dropdown, select **HTTPS** and **SSH**.
11. Click **Review + Create**.
12. You can now review the configurations and click on **Create**. On the **Generate new key pair** popup, click **Download private key and create** resource.



Wait until the virtual machine is up and running.

## Install Docker
In your terminal, run the following commands to SSH into the virtual machine:
```shell
chmod 400
ssh -i azureuser@
```
Kestra can be started using a `.jar` binary or Docker. In this guide, we’ll use Docker for a quick setup:
1. Install Docker on the Azure VM instance. You can find the last updated [instruction on the Docker website](https://docs.docker.com/engine/install/ubuntu/).
2. [Install Docker Compose](https://docs.docker.com/compose/install/).
To check your installation, run `sudo docker version` and `sudo docker compose version`. You're now ready to download and launch the Kestra server.
## Install Kestra
Download the official Docker-Compose file:
```bash
curl -o docker-compose.yml \
https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml
```
Use an editor such as Vim to modify the `docker-compose.yml`, set basic authentication to `true`, and configure your basic authentication credentials to secure your Kestra instance.
```yaml
kestra:
server:
basic-auth:
username: admin@kestra.io # it must be a valid email address
password: kestra
```
Next, use the following command to start the Kestra server:
```bash
sudo docker compose up -d
```
## Allow external traffic
Kestra is now running and the Kestra server exposes traffic on the `8080` port. To connect through your web browser, update the inbound traffic rules in the Azure security group.
1. Go to the Virtual Machines console and select the recently created virtual machine.
2. On the left-side navigation menu, click **Networking**.
3. Under **Inbound port rules** tab, click the **Add inbound port rule** button.
4. In the **Add inbound security rule** page, put **Destination port ranges** as `8080`. You can keep the default values for the remaining properties. Finally, click **Add** at the bottom of the page.
If you want to only allow traffic coming from your local machine, set the **Source** to your own IP address. To open the instance to the entire Internet, leave it as **Any**.


:::alert{type="warning"}
If you haven’t set up basic authentication, your Kestra instance will be publicly accessible to anyone without authentication.
:::
You can now access your Kestra instance and start developing flows.
## Launch Azure Database
This first installation relies on a PostgreSQL database running alongside the Kestra server on the VM instance (see the PostgreSQL service running in Docker Compose).
For a simple proof of concept (PoC), you can keep the PostgreSQL database running in Docker.
However, for a production-grade installation, we recommend a managed database service such as Azure Database for PostgreSQL servers.
**Launch a database using Azure Database for PostgreSQL servers**
1. Go to the [Azure Database for PostgreSQL servers](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.DBforPostgreSQL%2Fservers).
2. Click on **Create Azure Database for PostgreSQL server** (Kestra also supports MySQL, but PostgreSQL is recommended).
3. Choose an appropriate **Subscription** and **Resource Group**.
4. Put an appropriate **Server name** and select the preferred **Region**.
5. Choose the latest **PostgreSQL version**. We recommend version 17.
6. Select the **Workload type** as per your requirement.
7. Choose **Authentication method** as **PostgreSQL authentication only**.
8. Provide an appropriate **Admin username** and **Password** and re-write the password in **Confirm password**.
9. Click **Next: Networking**.
10. Check the select box for **Allow public access from any Azure service within Azure to this server**.
11. Click **Review + Create**. Review the configurations and click **Create**.
12. Wait for the database to be provisioned.



**Create a Kestra database**
1. Go to the database overview page and click **Databases** from the left-side navigation menu.
2. Click **Add**.
3. Put an appropriate database name and click **Save** at the top.
**Update Kestra configuration**
In the `docker-compose.yml` file, edit the `datasources` property of the Kestra service to point Kestra to your Azure database:
```yaml
datasources:
postgres:
url: jdbc:postgresql://:5432/
driver-class-name: org.postgresql.Driver
username:
password:
```
Because you now use the "Azure Database for PostgreSQL servers" service, you don't need the PostgreSQL Docker service anymore. Remove it from the `docker-compose.yml` file.
For the changes to take effect, restart the Docker services with `sudo docker compose restart` or `sudo docker compose up -d`.
## Configure Azure Blob Storage
By default, internal storage is implemented using the local file system. This section guides you how to change the storage backend to Blob Storage to ensure more reliable, durable, and scalable storage.
1. Go to the [Storage Accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts).
2. Click **Create**.
3. Choose an appropriate **Subscription** and **Resource Group**.
4. Put an appropriate **Storage account name** and select the preferred **Region**.
5. Select **Performance** and **Redundancy** as per your requirement.
6. Click **Review** and post reviewing the configurations, click **Create**.
7. Click on the newly created storage account.
8. On the storage account overview page, click **Containers** from the left-side navigation menu.
9. Click the **Create** button at the top to create a new container.
10. Put an appropriate name for the container and click **Create**. A new container will be created.
11. Now, click **Access keys** from the left-side navigation menu.
12. For one of the keys, either key1 or key2, click **Show** for the **Connection string** and click the **Copy to clipboard** button.
13. Make a note of the connection string for later use. We will require this for configuring the storage backend.
14. Edit the Kestra storage configuration in the `docker-compose.yml` file.
```yaml
kestra:
storage:
type: azure
azure:
container: ""
endpoint: "https://.blob.core.windows.net/"
connection-string: ""
```
For the changes to take effect, restart the Docker services with `sudo docker compose restart` or `sudo docker compose up -d`.
For more information on Azure Blob storage configuration, check out the [Runtime and Storage configuration guide](../../configuration/02.runtime-and-storage/index.md).
## Next steps
This setup provides a simple starting point for running Kestra in production on a single machine. For a deployment to a distributed Kubernetes cluster, check the [Azure AKS deployment guide](../06.kubernetes-azure-aks/index.md).
Reach out via [Slack](/slack) if you encounter any issues or have any questions regarding deploying Kestra to production.
Also, check the [CI/CD guide](../../version-control-cicd/cicd/index.md) to automate your workflow deployments based on changes in Git.
---
# Deploy Kestra on DigitalOcean – Managed DB Setup
URL: https://kestra.io/docs/installation/digitalocean-droplet
> Install Kestra on a DigitalOcean Droplet with Managed Database and Spaces Object Storage for a cloud-native setup.
Install Kestra on a DigitalOcean Droplet with Managed Database as the database backend.
## Prerequisites
- Basic knowledge about using a command line interface
- Basic knowledge about DigitalOcean and PostgreSQL
## Create a DigitalOcean Droplet
Go to the DigitalOcean portal and choose [Droplets](https://www.digitalocean.com/products/droplets) from the left navigation bar.
1. On the Droplets page, click **Create Droplet**.
2. Choose an appropriate region.
3. Choose `Ubuntu` as the OS image with the latest version.
4. Kestra requires at least 4 GiB of memory and 2 vCPUs. The `Basic` plan with `Regular` CPU and 4 GiB / 2 vCPU is a good starting point.
5. You can choose an appropriate authentication method: SSH Key or Password based.
6. Provide an appropriate hostname and click on the `Create Droplet` button at the bottom.




Wait until the virtual machine is up and running. From the Droplets page, you can navigate to the recently created Droplet. From here, you can open the machine's console by clicking on the `Console` button at the top.

## Install Docker
Once in the console terminal, you can run the commands to install Kestra.
Kestra can be started directly from a `.jar` binary or using Docker. We use Docker here for a quick setup:
1. Install Docker on the Droplet. [You can find the last updated instruction on the Docker website](https://docs.docker.com/engine/install/ubuntu/).
2. [Install docker compose](https://docs.docker.com/compose/install/).
To check your installation, run `sudo docker version` and `sudo docker compose version`. You're now ready to download and launch the Kestra server.
## Install Kestra
Download the official Docker-Compose file:
```bash
curl -o docker-compose.yml https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml
```
Use an editor such as Vim to modify the `docker-compose.yml`, set basic authentication to `true`, and configure your basic authentication credentials to secure your Kestra instance.
```yaml
kestra:
server:
basic-auth:
enabled: true
username: admin@kestra.io # it must be a valid email address
password: kestra
```
Next, use the following command to start the Kestra server:
```bash
sudo docker compose up -d
```
You can now access the Kestra UI at `http://:8080` and start developing flows.
## Launch DigitalOcean Database
This first installation relies on a PostgreSQL database running alongside the Kestra server - on the VM instance (see the PostgreSQL service running thanks to the docker-compose).
For a simple proof of concept (PoC), you can keep the PostgreSQL database running in Docker.
However, for a production-grade installation, we recommend a managed database service such as DigitalOcean Database.
**Launch a PostgreSQL database using [DigitalOcean Database](https://www.digitalocean.com/products/managed-databases-postgresql)**
1. Go to the [DigitalOcean Databases](https://cloud.digitalocean.com/databases).
2. Click on `Create Database` button on the top.
3. Choose an appropriate region and select `PostgreSQL` as the database engine (Kestra also supports MySQL, but PostgreSQL is recommended).
4. Choose the database configuration as per your requirement.
5. Provide an appropriate database cluster name.
6. Click on the `Create Database Cluster` button at the bottom of the page.
7. Wait for the database to be provisioned. Generally, this takes around 5 minutes.



8. Once the database is ready, you can click on the `Get Started` button.
9. In the `Add trusted sources` dropbox, you can select your computer (in case you want to connect to this database from the PostgreSQL client running on your computer) and the `kestra-host` droplet created in the earlier section.
10. Click on `Allow these inbound sources only`.
11. On this page, ensure `Public network` is selected on the top. Take a note of the Connection Details that appear, and click on `Continue`.
12. On the next page, click on `Great, I'm done` button.




**Create a Kestra database**
1. Go to the database overview page and navigate to `Users & Databases` tab.
2. Under `Databases`, type an appropriate database name and click on `Save`.


**Update Kestra configuration**
In the docker-compose configuration, edit the `datasources` property of the Kestra service to point Kestra to your DigitalOcean database:
```yaml
datasources:
postgres:
url: jdbc:postgresql://:25060/
driver-class-name: org.postgresql.Driver
username: doadmin
password:
```
Because you now use the database powered by "DigitalOcean Database", you don't need the PostgreSQL Docker service anymore. Remove it from the `docker-compose.yml` file.
You'll also need to delete the `depends_on` section at the end of the YAML file:
```yaml
depends_on:
postgres:
condition: service_started
```
To apply the changes, restart the docker services with `sudo docker compose restart` or `sudo docker compose up -d`.
## Configure Spaces Object Storage
By default, internal storage is implemented using the local file system. This section guides you how to change the storage backend to Spaces Object Storage to ensure more reliable, durable, and scalable storage.
First, we create the access key and secret key that can be used to connect to Spaces Object Storage.
1. Navigate to the `API` from the left navigation menu.
2. Go to the `Spaces Keys` tab.
3. Click on `Generate New Key` button.
4. Provide an appropriate name for the spaces access key and click on `Create Access Key`.
5. A new access key with the given name will be generated. Take a note of the secret key as you will not be able to retrieve it later.



Let's create a bucket in the Spaces Object Storage.
1. Go to the [Spaces Object Storage](https://cloud.digitalocean.com/spaces). You can also navigate to the Spaces Object Storage from the left navigation menu.
2. Click on `Create Spaces Bucket` button.
3. Choose an appropriate data center region.
4. Put an appropriate unique Spaces Bucket name and select the corresponding project in which the Spaces Bucket needs to be created.
5. Click on `Create a Spaces Bucket` at the bottom to create the Spaces Bucket.
6. Once the bucket is created, you can go to the bucket's page and note down the `Original Endpoint`.
7. Edit the Kestra storage configuration in the `docker-compose.yml` file.
```yaml
kestra:
storage:
type: minio
minio:
endpoint: ""
port: "443"
secure: true
access-key: ""
secret-key: ""
region: "FRA1"
bucket: ""
```
To apply the changes, restart the docker services with `sudo docker compose restart` or `sudo docker compose up -d`.




## Next steps
This setup provides a simple starting point for running Kestra in production on a single machine.
Reach out via [Slack](/slack) if you encounter any issues or if you have any questions regarding deploying Kestra to production.
Make sure to also check the [CI/CD guide](../../version-control-cicd/cicd/index.md) to automate your workflow deployments based on changes in Git.
To manage DigitalOcean resources (Droplets, Kubernetes clusters, managed databases, load balancers, volumes, domains, and firewalls) directly from Kestra flows, see the [DigitalOcean plugin](/plugins/plugin-digitalocean).
---
# Run Kestra with Docker – Single-Container Setup
URL: https://kestra.io/docs/installation/docker
> Run Kestra in a single Docker container for quick testing and development, with options for custom configuration.
Start Kestra using a single Docker container.
## Install Kestra with a single Docker Container
Once you have Docker running, you can start Kestra in a single command (*if you're running on Windows, make sure to use [WSL](https://docs.docker.com/desktop/wsl/)*):
```bash
docker run --pull=always --rm -it -p 8080:8080 --user=root \
--name kestra \
-v kestra_data:/app/storage \
-v kestra_db:/app/data \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp:/tmp \
-e KESTRA_PLUGINS_AUTO_INSTALL_ENABLED=true \
kestra/kestra:latest-slim server local
```
Open http://localhost:8080 in your browser to launch the UI and start building your first flows.
:::alert{type="info"}
The `kestra/kestra:latest-slim` image ships without any plugins to keep the download small. With `KESTRA_PLUGINS_AUTO_INSTALL_ENABLED=true`, Kestra installs any plugin automatically the first time a flow needs it, so you don't need to pre-install anything. If you prefer an image with all plugins bundled, use `kestra/kestra:latest` instead.
:::
:::alert{type="info"}
The above command starts Kestra with an embedded H2 database. Storage files are stored on the `kestra_data` Docker volume, and the H2 database is persisted on the `kestra_db` Docker volume. For production-ready persistence with a PostgreSQL database and more configurability, follow the [Docker Compose installation](../03.docker-compose/index.md).
:::
:::alert{type="info"}
**Enterprise Edition images** — log in to the private registry with your license credentials before pulling images:
```bash
docker login registry.kestra.io --username $LICENSEID --password $FINGERPRINT
```
Use `registry.kestra.io/docker/kestra-ee:latest` for the newest image, or pin a specific version such as `registry.kestra.io/docker/kestra-ee:v1.0`. Review the [Enterprise documentation](../../07.enterprise/index.mdx) and [configuration requirements](../../07.enterprise/05.instance/index.mdx) for additional setup guidance.
Compare editions in [Open Source vs Enterprise](../../oss-vs-paid/index.md) if you are deciding between versions.
:::
## Configuration
### Using a configuration file
You can adjust Kestra's configuration using a file mounted to the Docker container as a bind volume.
First, create a configuration `.yml` file like the example below:
```yaml
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driver-class-name: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
server:
basic-auth:
enabled: false
username: "admin@kestra.io" # It must be a valid email address
password: kestra
repository:
type: postgres
storage:
type: local
local:
base-path: "/app/storage"
queue:
type: postgres
tasks:
tmp-dir:
path: "/tmp/kestra-wd/tmp"
url: "http://localhost:8080/"
```
:::alert{type="info"}
This configuration is taken from the official [docker-compose.yaml](https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml#L33) file and uses a PostgreSQL database; you may want to retrieve it there to be sure it is accurate.
:::
After creating the configuration file, update the command to mount the file to the container and start Kestra. We also adjust the Kestra command to start a standalone version, as we now have a PostgreSQL database as a backend.
```bash
docker run --pull=always --rm -it -p 8080:8080 --user=root \
-v $PWD/application.yaml:/etc/config/application.yaml \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp:/tmp kestra/kestra:latest server standalone --config /etc/config/application.yaml
```
### Using the `KESTRA_CONFIGURATION` environment variable
You can adjust the [Kestra configuration](../../configuration/01.configuration-basics/index.md) by passing the `KESTRA_CONFIGURATION` variable to the Docker container via the `-e` option.
This environment variable must be a valid YAML string.
Managing a large configuration via a single YAML string can be tedious. To simplify this, consider using a configuration file instead.
First, define an environment variable:
```bash
export KESTRA_CONFIGURATION=$'
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driver-class-name: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
server:
basic-auth:
enabled: false
username: "admin@kestra.io" # it must be a valid email address
password: kestra
repository:
type: postgres
storage:
type: local
local:
base-path: "/app/storage"
queue:
type: postgres
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
url: http://localhost:8080/
```
:::alert{type="info"}
This configuration is taken from the official [docker-compose.yaml](https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml#L33) file and uses a PostgreSQL database; you may want to retrieve it there to be sure it is accurate.
:::
Once configured, pass the `KESTRA_CONFIGURATION` environment variable in a Docker command and adjust the Kestra command to run the standalone server:
```bash
docker run --pull=always --rm -it -p 8080:8080 --user=root \
-e KESTRA_CONFIGURATION="$KESTRA_CONFIGURATION" \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp:/tmp kestra/kestra:latest server standalone
```
## Official Docker images
The official Kestra Docker images are available on [DockerHub](https://hub.docker.com/r/kestra/kestra) for both `linux/amd64` and `linux/arm64` platforms.
Two image variants are available:
- `kestra/kestra:*`
- `kestra/kestra:*-slim`
Both variants are based on the [`eclipse-temurin:21-jre`](https://hub.docker.com/_/eclipse-temurin) Docker image.
The `kestra/kestra:*` images include all Kestra [plugins](/plugins) in their **latest versions**. The `kestra/kestra:*-slim` images do not bundle any plugins, which keeps them much smaller to download. When plugin auto-install is enabled (`KESTRA_PLUGINS_AUTO_INSTALL_ENABLED=true`), Kestra downloads and installs any missing plugin automatically the first time a flow uses it — so the `-slim` image stays fully usable.
:::alert{type="info"}
The `*-slim` images were previously published under the `*-no-plugins` suffix. The `-no-plugins` tags are deprecated aliases of `-slim` and will be removed in a future release.
:::
## Docker image tags
The following tags are available for each Docker image (append `-slim` to any image to exclude all but Kestra core plugins):
- `latest`: The most recent stable release (rolling tag). Intended for trying new features; not an LTS. Support ends when the next stable release (~ 2 months) becomes available.
- `latest-lts`: The current Long-Term Support (rolling tag). Tracks the active LTS line (updates roughly every 6 months to the new LTS) and receives fixes for ~1 year.
- `v`: Minor-series floating tag (e.g., `v1.0`) that always points to the latest patch of that series (e.g., `v1.0.5`). Use `v` when you want automatic patch updates but to stay on a minor line.
- `v`: Immutable tag for an exact version (e.g., `v1.0.1`). Never changes; **best for locked-down production.**
- `develop`: Nightly/continuous build from the `develop` branch. Unstable and not recommended for production, only for testing.
The **default Kestra image** `kestra/kestra:latest` already includes **all plugins**. To use a lightweight version of Kestra without bundled plugins, add the suffix `-slim`. These images ship without plugins on purpose — they are a fraction of the size of the full image, and combined with plugin auto-install (`KESTRA_PLUGINS_AUTO_INSTALL_ENABLED=true`), missing plugins are fetched on demand when a flow needs them.
### Recommended images for production
For production deployments, choose one of the following:
**Latest stable version** for staying most up to date while also stable (make note that this is a rolling tag that changes quite frequently):
- `kestra/kestra:latest` — latest stable with all plugins
- `kestra/kestra:latest-slim` — latest stable without plugins
**Pinned versions** for maximum stability:
- `kestra/kestra:v` — all plugins included
- `kestra/kestra:v-slim` — no bundled plugins, only core to Kestra
**LTS rolling tag** if you want automatic updates within the LTS line:
- `kestra/kestra:latest-lts`
- `kestra/kestra:latest-lts-slim`
### Recommended images for development
For development or testing new features:
- `kestra/kestra:latest` — latest stable with all plugins
- `kestra/kestra:latest-slim` — latest stable without plugins
- `kestra/kestra:develop` / `kestra/kestra:develop-slim` — daily builds with unreleased features, unstable
## Build a custom Docker image
If the base or full image doesn't contain package dependencies you need, you can build a custom image by using the Kestra base image and adding the required binaries and dependencies.
### Add custom binaries
The following `Dockerfile` creates a new image from the Kestra base image and adds the `golang` binary along with Python packages:
```dockerfile
ARG IMAGE_TAG=latest
FROM kestra/kestra:$IMAGE_TAG
RUN mkdir -p /app/plugins && \
apt-get update -y && \
apt-get install -y --no-install-recommends golang && \
apt-get install -y pip && \
pip install pandas==2.0.3 requests==2.31.0 && \
apt-get clean && rm -rf /var/lib/apt/lists/* /var/tmp/*
```
### Add plugins to a Docker image
By default, the base Docker image `kestra/kestra:latest` contains all plugins (unless you use the `kestra/kestra:latest-slim` version). You can add specific plugins to the base image and build a custom image.
The following `Dockerfile` creates an image from the base image and adds the `plugin-aws`, `storage-gcs` and `plugin-gcp` binaries using the command `kestra plugins install`:
```dockerfile
ARG IMAGE_TAG=latest-slim
FROM kestra/kestra:$IMAGE_TAG
RUN /app/kestra plugins install \
io.kestra.plugin:plugin-aws:LATEST \
io.kestra.storage:storage-gcs:LATEST \
io.kestra.plugin:plugin-gcp:LATEST
```
### Add custom plugins to a Docker image
The above `Dockerfile` installs plugins that have already been published to [Maven Central](https://central.sonatype.com/). If you are developing a custom plugin, make sure to build it following our [plugin developer guide](../../plugin-developer-guide/index.mdx). Once the `shadowJar` is built, add it to the plugins directory:
```dockerfile
ARG IMAGE_TAG=latest
FROM kestra/kestra:$IMAGE_TAG
RUN mkdir -p /app/plugins
COPY /build/libs/*.jar /app/plugins
```
### Add custom plugins from a Git repository
If you would like to build custom plugins from a specific Git repository, you can use the following approach:
```dockerfile
FROM openjdk:17-slim as stage-build
WORKDIR /
USER root
RUN apt-get update -y
RUN apt-get install git -y && \
git clone https://github.com/kestra-io/plugin-aws.git
RUN cd plugin-aws && ./gradlew :shadowJar
FROM kestra/kestra:latest
## https://github.com/WASdev/ci.docker/issues/194#issuecomment-433519379
USER root
RUN mkdir -p /app/plugins && \
apt-get update -y && \
apt-get install -y --no-install-recommends golang && \
apt-get install -y pip && \
pip install pandas==2.0.3 requests==2.31.0 && \
apt-get clean && rm -rf /var/lib/apt/lists/* /var/tmp/*
RUN rm -rf /app/plugins/plugin-aws-*.jar
COPY --from=stage-build /plugin-aws/build/libs/plugin-aws-*.jar /app/plugins
```
This multi-stage Docker build allows you to override a plugin that has already been installed. In this example, the AWS plugin is by default already included in the `kestra/kestra:latest` image. However, it's overridden by a plugin built in the first Docker build stage.
---
# Deploy Kestra with Docker Compose – PostgreSQL
URL: https://kestra.io/docs/installation/docker-compose
> Get started with Kestra quickly using Docker Compose with a PostgreSQL backend for a robust local or server deployment.
Start Kestra with a PostgreSQL database backend by using a Docker Compose file.
## Prerequisites
- Install [Docker](https://docs.docker.com/compose/install/) before you begin.
- Make sure Docker Compose is available in your Docker installation.
## Download the Docker Compose file
Download the Docker Compose file using the following command on Linux and macOS:
```bash
curl -o docker-compose.yml \
https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml
```
On Windows, use the following command:
```powershell
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml" -OutFile "docker-compose.yml"
```
You can also download the [Docker Compose file](https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml) manually and save it as `docker-compose.yml`.
## Launch Kestra
Use the following command to start the Kestra server:
```bash
docker compose up -d
```
Open the URL `http://localhost:8080` in your browser to launch the UI.
:::alert{type="info"}
**Enterprise Edition images** — log in to the private registry with your license credentials before pulling images:
```bash
docker login registry.kestra.io --username $LICENSEID --password $FINGERPRINT
```
Use `registry.kestra.io/docker/kestra-ee:latest` for the newest image, or pin a specific version such as `registry.kestra.io/docker/kestra-ee:v1.0`. See the [Enterprise documentation](../../07.enterprise/index.mdx) and [configuration requirements](../../07.enterprise/05.instance/index.mdx) for deployment prerequisites.
Compare editions in [Open Source vs Enterprise](../../oss-vs-paid/index.md) if you are deciding between versions.
:::
### Adjusting the configuration
The command from the previous section starts a standalone server, with all architectural components running in one JVM.
The [configuration](../../configuration/01.configuration-basics/index.md) lives in the `KESTRA_CONFIGURATION` environment variable of the Kestra container. You can update that environment variable inside the Docker Compose file or pass it as a Docker CLI argument.
:::alert{type="info"}
If you want to extend your Docker Compose file, modify container networking, or if you have any other issues using this Docker Compose file, check the [Troubleshooting Guide](../../10.administrator-guide/16.troubleshooting/index.md).
For running Kestra in Docker Compose with each server component as a separate service, see the [multi-component Docker Compose example](../../kestra-cli/kestra-server/index.md#kestra-with-server-components-in-different-services).
:::
### Use a configuration file
If you want to use a configuration file instead of the `KESTRA_CONFIGURATION` environment variable, update the default `docker-compose.yml`.
First, create a configuration file containing the `KESTRA_CONFIGURATION` environment variable defined in the `docker-compose.yml` file. You can name it `application.yaml`.
Next, update the `kestra` service in the `docker-compose.yml` file to mount this file into the container and start up Kestra using the `--config` option:
```yaml
## [...]
kestra:
image: kestra/kestra:latest
pull_policy: always
# Note that this is meant for development only. Refer to the documentation for production deployments of Kestra which runs without a root user.
user: "root"
command: server standalone --worker-thread=128 --config /etc/config/application.yaml
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/kestra-wd:/tmp/kestra-wd
- $PWD/application.yaml:/etc/config/application.yaml
ports:
- "8080:8080"
# The management port (8081) is not exposed by default. It is unauthenticated
# and intended for internal use only. Only expose it within a trusted network.
# - "8081:8081"
depends_on:
postgres:
condition: service_started
```
:::alert{type="info"}
Check out all of our available [Docker image tags](./../02.docker/index.md#docker-image-tags) to see which one is best for your use case.
:::
### Configure networking in Docker Compose
The [default Docker Compose file](https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml) does not configure networking for the Kestra containers. This means you cannot access services exposed via `localhost` on your local machine, such as another Docker container with a mapped port. Your machine and the Docker container operate on different networks. To use a locally exposed service from the Kestra container, use the `host.docker.internal` hostname or `172.17.0.1`. The `host.docker.internal` address lets you reach your host machine's services from the container.
Alternatively, you can use a Docker network. By default, your Kestra container is placed in a `default` network. You can add your custom services to the `docker-compose.yml` file provided by Kestra and use the service aliases, which are the keys in `services`, to reach them.
A better approach may be to create a new network such as `kestra_net` and add your services to it. Then add that network to the `networks` section of the `kestra` service. With this configuration, you can access your exposed ports through `localhost`.
The example below shows how you can add `iceberg-rest`, `minio`, and `mc` (i.e., MinIO client) to your Kestra Docker Compose file.
:::collapse{title="Example"}
```yaml
volumes:
postgres-data:
driver: local
kestra-data:
driver: local
networks:
kestra_net:
services:
postgres:
image: postgres
volumes:
- postgres-data:/var/lib/postgresql
environment:
POSTGRES_DB: kestra
POSTGRES_USER: kestra
POSTGRES_PASSWORD: k3str4
healthcheck:
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
interval: 30s
timeout: 10s
retries: 10
networks:
kestra_net:
iceberg-rest:
image: tabulario/iceberg-rest
ports:
- 8181:8181
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
- AWS_REGION=us-east-1
- CATALOG_WAREHOUSE=s3://warehouse/
- CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
- CATALOG_S3_ENDPOINT=http://minio:9000
networks:
kestra_net:
minio:
image: minio/minio
container_name: minio
environment:
- MINIO_ROOT_USER=admin
- MINIO_ROOT_PASSWORD=password
- MINIO_DOMAIN=minio
networks:
kestra_net:
aliases:
- warehouse.minio
ports:
- 9001:9001
- 9000:9000
command: ["server", "/data", "--console-address", ":9001"]
mc:
depends_on:
- minio
image: minio/mc
container_name: mc
networks:
kestra_net:
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
- AWS_REGION=us-east-1
entrypoint: >
/bin/sh -c "
until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
/usr/bin/mc rm -r --force minio/warehouse;
/usr/bin/mc mb minio/warehouse;
/usr/bin/mc policy set public minio/warehouse;
tail -f /dev/null
"
kestra:
image: kestra/kestra:latest
pull_policy: always
entrypoint: /bin/bash
# Note that this is meant for development only. Refer to the documentation for production deployments of Kestra which runs without a root user.
user: "root"
command:
- -c
- /app/kestra server standalone --worker-thread=128
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/kestra-wd:/tmp/kestra-wd
environment:
KESTRA_CONFIGURATION: |
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driver-class-name: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
server:
basic-auth:
username: admin
password: kestra
repository:
type: postgres
storage:
type: minio
minio:
endpoint: http://minio
port: 9000
access-key: admin
secret-key: password
region: us-east-1
bucket: warehouse
queue:
type: postgres
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
url: http://localhost:8080/
ports:
- "8080:8080"
# The management port (8081) is not exposed by default. It is unauthenticated
# and intended for internal use only. Only expose it within a trusted network.
# - "8081:8081"
depends_on:
postgres:
condition: service_started
networks:
kestra_net:
```
:::
Finally, you can use `host` network mode for the `kestra` service. This makes the container use your host network, so it can reach all exposed ports. In that case, change `services.kestra.environment.KESTRA_CONFIGURATION.datasources.postgres.url` to `jdbc:postgresql://localhost:5432/kestra`. This is the easiest way to reach all ports, but it can be a security risk.
See the example below using `network_mode: host`.
:::collapse{title="Example"}
```yaml
volumes:
kestra-data:
driver: local
services:
kestra:
image: kestra/kestra:latest
pull_policy: always
entrypoint: /bin/bash
network_mode: host
environment:
JAVA_OPTS: "--add-opens java.base/java.nio=ALL-UNNAMED"
NODE_OPTIONS: "--max-old-space-size=4096"
KESTRA_CONFIGURATION: |
datasources:
postgres:
url: jdbc:postgresql://localhost:5432/kestra
driver-class-name: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
server:
basic-auth:
username: admin
password: kestra
anonymous-usage-report:
enabled: true
repository:
type: postgres
storage:
type: local
local:
base-path: "/app/storage"
queue:
type: postgres
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
scripts:
docker:
volume-enabled: true
# Global plugin defaults removed in Kestra 2.0.
# Use kestra.policies for installation-wide defaults.
# See: https://kestra.io/docs/enterprise/governance/policies
url: http://localhost:8080/
variables:
env-vars-prefix: "" # To avoid requiring KESTRA_ prefix on env vars
```
:::
### PostgreSQL 16 incompatibility error
By default, the Docker Compose template uses the latest image for PostgreSQL. However, if you initialized your Kestra database on an older version of PostgreSQL, you might encounter the following error:
```plaintext
The data directory was initialized by PostgreSQL version 16, which is not compatible with this version 17.0 (Debian 17.0-1.pgdg120+1).
```
To resolve this, you need to specify a specific tag for the PostgreSQL image in your Docker Compose file. In the example below, we specify `16` as the error above was originally initialized by version 16:
```yaml
services:
postgres:
image: postgres:16
```
### SIGILL in Java Runtime Environment on macOS M4 chip
Add the following environment variable to your Kestra container: `-e JAVA_OPTS="-XX:UseSVE=0"`:
```bash
docker run --pull=always --rm -it -p 8080:8080 --user=root -e JAVA_OPTS="-XX:UseSVE=0" --name kestra -v kestra_data:/app/storage -v kestra_db:/app/data -v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp kestra/kestra:latest server local
```
To apply the same setting in a Docker Compose file:
```yaml
services:
kestra:
image: kestra/kestra:latest
environment:
JAVA_OPTS: "-XX:UseSVE=0"
```
## Kestra with server components in different services
Server components can run independently from each other. Each of them communicate through the database. The `kestra server` command starts each server component individually:
- `kestra server executor`
- `kestra server worker`
- `kestra server indexer`
- `kestra server scheduler`
- `kestra server webserver`
For more details on Kestra server commands, check out the [Server CLI documentation](../../kestra-cli/kestra-server/index.md).
Here is an example Docker Compose configuration file running Kestra services with replicas on the Postgres database backend. Each service maps a unique host port range to its internal management port (`8081`) so you can monitor individual replicas. The management port is unauthenticated by default — only expose these mappings within a trusted internal network.
```yaml
volumes:
postgres-data:
driver: local
kestra-data:
driver: local
services:
postgres:
image: postgres
volumes:
- postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_DB: kestra
POSTGRES_USER: kestra
POSTGRES_PASSWORD: k3str4
healthcheck:
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
interval: 30s
timeout: 10s
retries: 10
kestra-scheduler:
image: kestra/kestra:latest
deploy:
replicas: 2
pull_policy: if_not_present
user: "root"
command: server scheduler
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/kestra-wd:/tmp/kestra-wd
environment:
KESTRA_CONFIGURATION: &common_configuration |
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driver-class-name: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
server:
basic-auth:
username: "admin@kestra.io"
password: kestra
repository:
type: postgres
storage:
type: local
local:
base-path: "/app/storage"
queue:
type: postgres
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
ports:
- "8082-8083:8081"
depends_on:
postgres:
condition: service_started
kestra-worker:
image: kestra/kestra:latest
deploy:
replicas: 2
pull_policy: if_not_present
user: "root"
command: server worker
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/kestra-wd:/tmp/kestra-wd
environment:
KESTRA_CONFIGURATION: *common_configuration
ports:
- "8084-8085:8081"
depends_on:
postgres:
condition: service_started
kestra-executor:
image: kestra/kestra:latest
deploy:
replicas: 2
pull_policy: if_not_present
user: "root"
command: server executor
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/kestra-wd:/tmp/kestra-wd
environment:
KESTRA_CONFIGURATION: *common_configuration
ports:
- "8086-8087:8081"
depends_on:
postgres:
condition: service_started
kestra-webserver:
image: kestra/kestra:latest
deploy:
replicas: 1
pull_policy: if_not_present
user: "root"
command: server webserver
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/kestra-wd:/tmp/kestra-wd
environment:
KESTRA_CONFIGURATION: *common_configuration
KESTRA_URL: http://localhost:8080/
ports:
- "8080:8080"
- "8081:8081"
depends_on:
postgres:
condition: service_started
```
---
# Deploy Kestra on GCP VM – Cloud SQL and GCS
URL: https://kestra.io/docs/installation/gcp-vm
> Deploy Kestra on a Google Cloud Platform (GCP) VM with Cloud SQL and Google Cloud Storage (GCS).
Install Kestra on a GCP VM with Cloud SQL as the database backend and Cloud Storage as the internal storage backend.
## Prerequisites
- Basic command-line interface (CLI) skills.
- Familiarity with Compute Engine, Cloud Storage, and PostgreSQL.
## Create a VM instance
First, create a VM instance using the Compute Engine. To do so, [go to the GCP console and choose Compute Engine](https://console.cloud.google.com/compute/instances).
1. Click the **Create Instance** button at the top.
2. Give a name to your instance.
3. Choose an appropriate Region and Zone.
4. Choose the **General Purpose** machine of the **E2** series.
5. Machine type: Kestra requires at least 4GiB of memory and 2 vCPUs to run correctly. Choosing the **Preset** machine type **e2-standard-2** is a good starting point.
6. Click on **Change** in the **Boot Disk** section, as we would like to change the image.
7. Under the "Public Images" tab, choose **Ubuntu** as the operating system and the **Ubuntu 22.04 LTS** version.
8. Continue with the **Allow default access** access scope, and select **Allow HTTPS traffic** in the Firewall section.




You can now click on **Create** and wait a few seconds for the VM instance to be up and running.
## Install Docker
Click on the **SSH** button on the right side of the VM instance details to SSH into the VM instance terminal. Click on the **Authorize** button in the pop-up to authorize the SSH connection into the VM instance.

Kestra can be started directly from a `.jar` binary or using Docker. We use Docker here for a quicker setup.
Install Docker on the GCP VM instance. You can find the last updated [instruction on the Docker website](https://docs.docker.com/engine/install/ubuntu/).
To check your installation, run `sudo docker version` and `sudo docker compose version`. You're now ready to download and launch the Kestra server.
## Install Kestra
Download the official Docker-Compose file:
```bash
curl -o docker-compose.yml \
https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml
```
Use an editor such as Vim to modify the `docker-compose.yml`, set basic authentication to `true`, and configure your basic authentication credentials to secure your Kestra instance.
```yaml
kestra:
server:
basic-auth:
enabled: true
username: admin@kestra.io # it must be a valid email address
password: kestra
```




:::alert{type="warning"}
Note that if you haven't set up basic authentication in the previous step, your Kestra instance will be publicly accessible to anyone without authentication.
:::
You can now access your Kestra instance and start developing flows.
## Launch Cloud SQL
This first installation relies on a PostgreSQL database running alongside the Kestra server on the VM instance (see the PostgreSQL service running in Docker Compose).
For a simple proof of concept (PoC), you can keep the PostgreSQL database running in Docker.
However, for a production-grade installation, we recommend a managed database service such as Cloud SQL.
**Create a Cloud SQL database**
1. Go to the [Cloud SQL console](https://console.cloud.google.com/sql/instances).
2. Click on `Choose PostgreSQL` (Kestra also supports MySQL, but PostgreSQL is recommended).
3. Put an appropriate Instance ID and password for the admin user `postgres`.
4. Select the latest PostgreSQL version from the dropdown.
5. Choose `Enterprise Plus` or `Enterprise` edition based on your requirements.
6. Choose an appropriate preset among `Production`, `Development` or `Sandbox` as per your requirement.
7. Choose the appropriate region and zonal availability.
8. Expand Show `Show Configuration Options` at the bottom of the page.



**Enable VM connection to database**
1. Expand the `Connections` section from the dropdown.
2. Uncheck Public IP and check Private IP. If this is your first time using a Private IP connection, you will be prompted to `Setup Connection`.
3. You will then need to choose `Enable API` on the right hand side pop out.
4. Choose `Use an automatically allocated IP range` and click `Continue`.
5. Click on `Create Connection`.




**Enable Deletion**
If you are just testing or would like to be able to delete your instance and all of its data, then expand the `Data Protection` on the left hand side and make sure `Enable deletion protection` is UNCHECKED.

**Create database user**
1. Go to the database overview page and click **Users** from the left-side navigation menu.
2. Click **Add User Account**.
3. Put an appropriate username and password and click **Add**.


**Create Kestra database**
1. Go to the database overview page and click **Databases** from the left side navigation menu.
2. Click **Create Database**.
3. Put an appropriate database name and click **Create**.
**Update Kestra configuration**
In the Docker compose configuration, edit the `datasources` property of the Kestra service in the following way:
```yaml
datasources:
postgres:
url: jdbc:postgresql://:5432/
driver-class-name: org.postgresql.Driver
username: