Route Tasks to Specific Workers by Hardware or Region

Worker Groups route tasks to the right machines in your fleet. A Worker Group is a named, token-authenticated pool of workers that subscribes to one or more Worker Queues — tag-based routing lanes that tasks declare requirements against. The result is flexible many-to-many routing: GPU machines and spot instances can serve the same queue, or a single group can cover multiple queues with per-queue capacity guarantees.

Worker Groups are an Enterprise Edition feature. In the open-source edition, all work runs in a single implicit default pool.

Getting started

To set up Worker Groups end-to-end:

1. Create a Worker Queue — define a routing lane with tags
2. Create a Worker Group — create a pool and subscribe it to queues
3. Generate a registration token — authenticate workers to the group
4. Start a worker — connect with the token and controller endpoint
5. Route tasks — add workerSelector to any task

For IaC and Helm deployments, see Declarative configuration to provision the full topology at startup without runtime API calls.

How Worker Groups work

Three building blocks define the routing model:

The routing path flows from task requirements down to infrastructure:

1. A task declares workerSelector.tags: [gpu, eu]
2. Kestra finds the Worker Queue whose tags match
3. Kestra checks which Worker Groups subscribe to that queue
4. A worker from one of those groups picks up the task

Developer perspective: declare what a task needs using tags. No machine names, no group names.

Operator perspective: create queues with meaningful tags, subscribe groups to those queues, and set capacity guarantees per subscription.

Using workerSelector in tasks

Add workerSelector to any task to route it to a matching Worker Queue. The workerSelector object has three properties:

id: process_sensitive_data
namespace: company.team

tasks:
  - id: process
    type: io.kestra.plugin.scripts.python.Commands
    workerSelector:
      tags: [sensitive, eu]
      fallback: WAIT
    commands:
      - python process.py

Kestra routes the task to a Worker Queue whose tag set includes all declared tags (or any, when match: ANY). Any Worker Group subscribed to that queue may execute the task.

If workerSelector is absent or all tags resolve to null, the task routes to the default queue.

Fallback options

IGNORE is useful when the target infrastructure is optional — the task proceeds on any available worker rather than failing when the specialized pool is unavailable.

fallback can only be set when tags is non-empty.

Dynamic routing

Use Pebble expressions to set tags at runtime:

inputs:
  - id: region
    type: STRING
    defaults: eu

tasks:
  - id: process
    type: io.kestra.plugin.scripts.python.Commands
    workerSelector:
      tags:
        - "{{ inputs.region }}"
        - sensitive
      fallback: WAIT
    commands:
      - python process.py

When an expression resolves to null or a blank string, that tag is omitted from the selector. If all tags resolve to null, the task routes to the default queue.

Namespace and tenant-level routing defaults

Instead of adding workerSelector to every task, set a default selector at the namespace or tenant level. Kestra resolves the selector most-specific-first — task, then flow, then the nearest namespace ancestor, then the tenant — and stops at the first level that declares one.

Set a namespace-level default in the namespace settings:

workerSelector:
  tags: [eu]
  fallback: WAIT

Every task in that namespace (and its child namespaces, unless overridden closer) inherits this selector automatically. Any selector closer to the task — on the task itself or the flow — wins over the namespace or tenant default.

This is the recommended approach when an entire namespace or team should always run on a specific fleet — it keeps flow YAML clean and makes routing changes a single admin update rather than a find-and-replace across all flows.

Applying workerSelector with Policies

Use a Policy to route all tasks of a given plugin type to a specific Worker Queue without modifying each task individually:

id: gpu-worker-routing
description: "Route all Python tasks to GPU workers."
enforcement: ACTIVE
rules:
  - type: io.kestra.plugin.ee.rules.Add
    on: PLUGIN
    where:
      - field: type
        operator: STARTS_WITH
        value: io.kestra.plugin.scripts.python
    values:
      workerSelector:
        tags: [gpu]
        fallback: WAIT

With this Policy applied to the namespace, flows need no per-task configuration:

id: ml_pipeline
namespace: company.team

tasks:
  - id: train
    type: io.kestra.plugin.scripts.python.Commands
    commands:
      - python train.py

  - id: evaluate
    type: io.kestra.plugin.scripts.python.Commands
    commands:
      - python eval.py

Worker Queues

A Worker Queue is a routing lane with a stable id and a set of tags. Multiple Worker Groups may subscribe to the same queue. Removing a group’s subscription never deletes the queue — queues exist independently.

Two ids are reserved and never created manually:

• default — the global default queue; receives tasks with no workerSelector
• system — the in-process system worker

Worker Queue ids must follow RFC 1123 label format: lowercase alphanumerics and hyphens, starting and ending with an alphanumeric character, max 63 characters.

Creating Worker Queues

Navigate to Instance Owner → Infrastructure → Worker Queues and click Create. You can also create Worker Queues via the API or Terraform.

Tenant scoping: a Worker Queue can restrict which tenants may route tasks through it. An empty tenant list means unrestricted.

Creating and managing Worker Groups

A Worker Group is identified by a stable id (RFC 1123 label), has a display name, and holds a list of queue subscriptions and registration tokens.

Creating a Worker Group

Navigate to Instance Owner → Infrastructure → Worker Groups and click Add Worker Group. Set an id, display name, and optional description. You can add queue subscriptions and generate registration tokens immediately, or configure them after creation.

Worker Group ids must follow RFC 1123 label format.

The default group

One group always exists and cannot be deleted: the default group. It subscribes to the default queue and receives all tasks that have no workerSelector. Workers that start without a registration token join the default group automatically.

Queue subscriptions

A subscription connects a Worker Group to a Worker Queue. Each subscription specifies:

• Target queue id — which Worker Queue this group’s workers will serve
• Reserved capacity percentage (optional) — a per-worker floor guarantee, 1–100
• Interaction mode — STRICT or ELASTIC (see Capacity reservation)

A group may subscribe to multiple queues. The sum of reserved percentages across a worker’s subscriptions must not exceed 100.

Manage subscriptions through the UI or the subscriptions API:

Capacity reservation

Reserved capacity is a per-worker floor guarantee, not a fleet-wide quota. Remaining slots beyond reserved percentages form a shared pool available to all of that worker’s subscriptions.

Example: a worker with 16 slots subscribing to two queues at 50% and 25% reserves 8 slots for queue A and 4 slots for queue B, with 4 slots in the shared pool.

Interaction modes

• STRICT — idle reserved slots remain exclusive to this subscription and are never lent to other work
• ELASTIC — idle reserved slots may be lent to other ELASTIC subscriptions on the same worker when the subscription has spare capacity

In both modes, tasks also draw from the shared pool once reserved slots are busy. Lent slots are not preempted — a busy lender may temporarily dip below its floor until borrowed work completes.

Capacity reservations are live-configurable: updating a subscription’s reserved percentage takes effect within seconds with no worker restarts required.

Worker authentication

Workers join a group by presenting a registration token generated for that group. The token is stored as a hash and shown only once at creation — copy it immediately.

On first connect, the worker exchanges the registration token for a short-lived access token and a rotating refresh token. The access token is refreshed automatically before it expires. Revoking or deleting a token immediately invalidates credentials for any workers that registered with it; those workers fail closed once their current access token expires.

Generating a registration token

In the Worker Groups UI, select a group and generate a token from the Tokens tab. Alternatively, use the API:

Server-side configuration

Enable worker authentication on your webserver or standalone Kestra instance:

kestra:
  ee:
    worker:
      auth:
        enabled: true
        jwt-signing-key: "{{ a strong shared secret, >= 32 bytes }}"
        access-token-lifetime: PT5M    # optional, default PT5M
        refresh-token-lifetime: P7D    # optional, default P7D

Worker-side configuration

Each worker needs two things to join a group: a registration token that identifies the group, and a controller endpoint that tells the worker where to connect. Both are required — a worker started with only the token will try localhost and fail.

kestra:
  worker:
    name: gpu-pool-1                       # optional display name
    auth:
      registration-token: "{{ token generated for the target group }}"
      credentials-path: /var/kestra/worker/.auth/worker-credentials.json  # default: /tmp/kestra/worker/.auth/...
      refresh-buffer: PT60S               # how early to refresh the access token before it expires
    controllers:
      type: STATIC
      static:
        endpoints:
          - host: kestra-controller.internal
            port: 50051

Controller discovery strategies

type: STATIC is the default and suitable for most bare-metal and Docker deployments. Two other strategies are available:

For Kubernetes, use type: DNS with a service hostname:

kestra:
  worker:
    controllers:
      type: DNS
      dns:
        hostname: kestra-controller.kestra.svc.cluster.local
        record-type: SRV      # or A if no SRV records
        default-port: 50051   # used with A records only
        refresh-interval: PT30S

For Helm deployments, controller discovery is preconfigured — see the Helm gRPC and Worker-Controller migration guide. For bare-metal or Docker with components on separate hosts, see running Kestra with separated server components.

Starting a worker for a group

With both the registration token and controller endpoint configured, start the worker normally:

kestra server worker

No additional CLI flags are needed. The registration token in kestra.worker.auth.registration-token identifies which group the worker joins at connection time.

Declarative configuration

You can declare the entire worker topology — queues, groups, subscriptions, and registration tokens — in application.yml under kestra.ee.setup. Kestra applies this configuration at startup, which enables a fully automated single-pass deployment with no runtime API calls.

kestra:
  ee:
    setup:
      enabled: true

      worker-queues:
        - id: gpu
          tags: [gpu, linux]
          allowed-tenants: [acme]   # optional; empty = unrestricted
        - id: etl
          tags: [etl]

      worker-groups:
        - id: gpu-workers
          name: GPU workers
          registration-tokens:
            - name: bootstrap
              token-file: /var/run/secrets/kestra/gpu-workers-token
          subscriptions:
            - worker-queue-id: gpu
              reserved-percent: 70
            - worker-queue-id: etl

Workers already retry registration until their token is known to the controller, so all services can start concurrently. Workers converge as soon as the webserver has applied the configuration.

Secret handling

Registration tokens must not appear as plaintext in a committed configuration file. Two options are available per token entry:

• token-file — path to a file containing the pre-generated token. Preferred in Kubernetes environments where Secrets mount as files. The file must exist and be non-empty at startup.
• token: "${ENV_VAR}" — environment variable placeholder resolved at startup. Simpler outside Kubernetes, but environment variables are readable from /proc/<pid>/environ and may appear in crash dumps.

Exactly one of the two is required. Use kestra workers registration-tokens generate to mint a token offline before deployment.

The default group

The default group can be declared under its reserved id default:

worker-groups:
  - id: default
    name: Shared workers
    registration-tokens:
      - name: bootstrap
        token-file: /var/run/secrets/kestra/default-workers-token

The default group always subscribes to the default queue — Kestra adds that subscription automatically even when subscriptions is omitted or does not include the default queue. Any subscriptions you declare are added alongside it.

Semantics

kestra.ee.setup is a seed, not a desired state:

• Each declared entity is created only when no entity with the same id already exists in the database.
• An existing entity is skipped as a whole — no subscriptions are changed, no tokens are added or revoked.
• Re-applying a changed configuration against an existing entity is a no-op. The database remains the source of truth once an entity exists; editing a live topology stays an API or UI operation.

A rogue instance cannot self-authorize by changing configuration: declarative setup can only add what is absent, never replace or revoke what the authenticated API created.

Which server roles apply it

Only the webserver and standalone server roles apply kestra.ee.setup at startup. Worker processes never apply it — a worker must not be able to create the group or the token it authenticates against.

Validation

The entire declaration is validated before anything is written. An invalid configuration fails startup with an actionable message identifying the offending path:

The subscription declared at 'kestra.ee.setup.worker-groups[0].subscriptions[1]'
references the unknown Worker Queue 'etl'. Declare it under
'kestra.ee.setup.worker-queues' or create it first.

Validation rejects: missing or duplicate ids, non-RFC-1123 ids, the reserved worker queue ids default and system, empty tag sets, tag collisions with existing queues, unknown worker-queue-id references, reserved-percent out of range or summing above 100, unreadable or empty token files, tokens already registered on another group, and malformed registration tokens.

Validation is all-or-nothing, but writes are not atomic. A failure mid-apply leaves already-created entities in place. The next startup resumes from where it stopped because existing entities are skipped.

Observability

Kestra logs one line per created entity and one line per skipped entity. A summary line follows after the setup phase completes. Each created entity also produces a regular audit log entry. Token values are never logged.

Transport security (TLS)

By default, gRPC traffic between workers and the controller is unencrypted. For production deployments, enable TLS on both sides.

Server-side TLS

Add TLS config to the controller (or standalone) instance:

kestra:
  grpc:
    tls:
      enabled: true
      key-store:
        path: /etc/kestra/tls/controller-keystore.p12
        password: "{{ secret('TLS_KEYSTORE_PASSWORD') }}"
      # Required when client-auth is OPTIONAL or REQUIRE
      trust-store:
        path: /etc/kestra/tls/ca-truststore.p12
        password: "{{ secret('TLS_TRUSTSTORE_PASSWORD') }}"
      client-auth: NONE    # NONE | OPTIONAL | REQUIRE (mTLS)

Worker-side TLS

Add matching TLS config to each worker:

kestra:
  grpc:
    tls:
      enabled: true
      # Required only for mTLS (client-auth: REQUIRE on the server)
      key-store:
        path: /etc/kestra/tls/worker-keystore.p12
        password: "{{ secret('TLS_KEYSTORE_PASSWORD') }}"
      # Optional — falls back to the system trust store
      trust-store:
        path: /etc/kestra/tls/ca-truststore.p12
        password: "{{ secret('TLS_TRUSTSTORE_PASSWORD') }}"

kestra:
  grpc:
    tls:
      enabled: true
      authority-override: kestra-controller

Use cases

Hardware affinity

Dedicate workers with GPUs, high-memory configurations, or OS-specific environments to tasks that need them. Developers declare the requirement via tags; operators manage the physical mapping independently.

workerSelector:
  tags: [gpu, cuda-12]

Multi-tenant isolation

Give each tenant a dedicated Worker Queue with a reserved capacity percentage to prevent noisy-neighbor effects. An additional ELASTIC subscription to a shared burst queue lets idle capacity absorb traffic spikes while the per-tenant floor stays guaranteed.

Regulated and air-gapped environments

Workers in restricted networks connect outbound-only, presenting a registration token to authenticate. No inbound firewall rules are required. Revoking a token immediately stops those workers from receiving new work, giving operators a fast, clean isolation path.

Spiky workloads

Use a fixed worker pool with STRICT reservations to handle baseline load, and a spot pool with ELASTIC subscriptions that claims shared-pool capacity during spikes. The ELASTIC pool scales out and in without changing the baseline pool’s guarantees.

Priority lanes

Split capacity across multiple queues with reserved percentages to guarantee throughput for high-priority work:

# Three priority queues — critical: 50%, standard: 25%, batch: 25%
workerSelector:
  tags: [critical]    # or [standard], or [batch]

Critical work always has guaranteed slots regardless of the volume of batch jobs in the queue.

Day/night capacity shifting

Reserved percentages are live-configurable via the API. Changes propagate to all workers within seconds, with no restarts required. Shift capacity toward batch workloads during off-peak hours and back to interactive workloads during business hours without touching any worker process.

Zero-downtime worker upgrades

Run two Worker Groups subscribed to the same queues simultaneously. Reduce the old group’s reservation to 0% to drain it of new work, bring up the new group, verify it is healthy, then delete the old group. At no point does the queue go unserved.

For guidance on when to use Worker Groups versus Task Runners for compute-intensive scripting workloads, see Task Runners vs Worker Groups.

Worker shutdown and task continuity

When a worker process stops — whether from a deployment, a crash, or a manual restart — any tasks it was running may be interrupted. The worker-task-restart-strategy setting controls what happens to those tasks cluster-wide:

Configure these in application.yml on each worker:

kestra:
  server:
    termination-grace-period: 5m
    worker-task-restart-strategy: AFTER_TERMINATION_GRACE_PERIOD

During the grace period, the worker stops accepting new jobs but lets running tasks finish. If the grace period elapses before all tasks complete, the worker force-terminates and the restart strategy decides the outcome for the remaining tasks.

Monitoring

Metrics scoped to a group carry a worker_group tag; metrics scoped to a queue also carry a worker_queue tag. The configurable metrics prefix (default kestra) is prepended before export.

Controller metrics

Published by the controller process — the server-side view of fleet capacity and dispatch activity:

Worker metrics

Published by each worker process — the worker-side view of capacity and throughput:

When worker.running.count consistently equals worker.job.thread and worker.pending.count is non-zero, that worker is fully saturated — scale by adding more workers to the group or increasing the thread count. When worker.queue.remaining.capacity on the job buffer approaches zero, the worker’s local inbound buffer is full.

The live capacity snapshot is also available via the API:

GET /api/v1/instance/worker-groups/{id}/capacity
GET /api/v1/instance/worker-groups/{id}/workers

Migrating from earlier versions

In Kestra 2.0, the task-level routing property changed from targeting a group by name to declaring requirements via tags:

workerGroup is not recognized in 2.0. Flows using it will fail validation and cannot be saved. Update your flows to replace workerGroup.key with workerSelector.tags. The group name in the old property corresponds to a tag on a Worker Queue in the new model.