Explore Kestra's Repository, Queue, Storage, and Plugin System

Table of Contents

For the complete documentation index, see llms.txt. For a full content snapshot, see llms-full.txt. Append .md to any kestra.io/docs/* URL for plain Markdown.

Kestra relies on the following internal components:

Internal storage: stores flow data such as task outputs and flow inputs.
Queue: enables internal communication between Kestra server components.
Repository: persists flows, executions, logs, and all other internal objects.
Plugins: extend Kestra’s core with additional task and trigger types, storage implementations, and data transformations.

Each component has multiple implementations depending on deployment architecture. Some require additional plugins.

Internal storage

The internal storage is a dedicated system that handles files of any size during flow executions. It manages both inputs and outputs, enabling scalable file sharing between tasks.

Purpose

Internal storage is used to:

Save files generated during a flow’s execution and pass them between tasks via outputs.
Automatically persist flow inputs of type FILE.
Provide download links for stored files in the Outputs tab of an execution.

Files are addressed by stable kestra:// URIs that the engine resolves on demand. A worker on one host and the Webserver on another can both reach the same artifact through the same URI. Files can be retrieved in the execution context using {{ outputs.task_id.output_attribute }} (often the uri property). Kestra fetches the file automatically when referenced.

Execution metadata — including storage file paths — is recorded in the repository.

Storage types

By default, Kestra uses local storage, which stores files on the host filesystem. Local storage is not recommended for production distributed deployments — use cloud object storage or a self-hosted alternative instead.

Local storage behavior differs between standalone and distributed deployments:

✅ Standalone: Local storage with persistent volumes is OK
❌ Distributed on single-writer storage: NOT recommended — storage that only one instance can mount read-write at a time (in Kubernetes, ReadWriteOnce) cannot back a distributed service
✅ Distributed on shared storage: OK when all instances can mount it read-write at once (in Kubernetes, ReadWriteMany), though this is rarely available
❌ Host storage sharing: NOT recommended — difficult to achieve reliably

When shared read-write storage (ReadWriteMany) is unavailable, use cloud storage (S3, GCS, Azure) or self-hosted S3-compatible object storage (Ceph, SeaweedFS, Garage, MinIO).

Scalable alternatives are available as plugins:

Storage MinIO — for AWS S3 and any S3-compatible object storage (Ceph, SeaweedFS, Garage, MinIO).
Storage GCS — for Google Cloud Storage.
Storage Azure — for Azure Blob Storage.
Storage Cloudflare — for Cloudflare R2.
Storage OBS — for Huawei Cloud OBS.

For details, see Runtime and Storage.

Queue

The queue is the internal communication channel between Kestra’s server components. Server roles emit typed messages onto named queues and subscribe to the queues they consume — no role calls another directly. The full queue surface is defined once as an abstract contract, satisfied by one chosen backend:

Database queue (default) — backed by PostgreSQL or MySQL. Available in all editions.
In-memory queue — for testing and ephemeral use only.
Kafka queue — Enterprise Edition. Higher throughput; pairs with the Elasticsearch repository.
Redis queue — Enterprise Edition.
AMQP queue — Enterprise Edition.
GCP Pub/Sub queue — Enterprise Edition.

The queue surface covers four delivery families:

Dispatch — point-to-point; exactly one subscriber processes each message. Used for executions, execution events and commands, worker task results, logs, and metrics.
Keyed dispatch — point-to-point partitioned by a routing key so a subscriber receives only messages for its key. Used for worker job routing: each Worker Queue is a key, and a worker subscribes only to the queues its group covers.
VNode dispatch — sharded across a fixed set of virtual nodes so a scaled component can divide a single logical stream deterministically. Used for trigger evaluation across a Scheduler fleet.
Broadcast — fan-out; every active subscriber receives every message. Used for kill signals, flow and metadata change notifications, follow-execution streams, and cluster-wide events.

Messages above a configurable size limit are rejected before reaching the backend, protecting it from oversized payloads. Terminal execution states are always allowed through regardless of size. Message protection is enabled by default with a 1 MB limit and can be adjusted under kestra.queue.message-protection.

Repository

The repository persists all domain entities, including flows, executions, logs, and triggers. The backend is chosen alongside the queue:

Database repository (default) — backed by PostgreSQL, MySQL, or H2. Available in all editions.
In-memory repository — for testing only.
Elasticsearch repository — Enterprise Edition. Backs the high-volume search and read model; requires the Kafka queue and the Indexer server role to keep it in sync.

Plugins

Kestra’s core only provides basic functionality. A plugin ecosystem extends the platform with:

New task and trigger types.
Alternative implementations of core components (e.g., storage backends).
Integrations with external systems and data transformation utilities.

A wide range of plugins is already available, and the ecosystem continues to expand.

Server componentsServer Components in Kestra Architecture Explained

Was this page helpful?