Store Execution Logs Outside Your Main Database

The external log data store routes Kestra execution logs to a dedicated JDBC database or Elasticsearch, separate from the main backend.

By default, Kestra stores execution logs in the same database as flows, executions, and state.

Why move logs out of the main database

Logs are high-volume, write-heavy data. Keeping them in the main database creates three operational problems:

• Database bloat: Log volume grows independently of flows and executions, inflating main database size over time.
• Migration risk: Kestra schema migrations run across every table, including logs. Large log tables extend upgrade downtime.
• Retention mismatch: Flows, executions, and logs typically have different retention requirements but share the same database and backup lifecycle.

How it works

The Indexer writes logs to the configured log store in real time. The log UI and dashboards read from the same store. The main database stores only flows, executions, and state.

Default behavior

If kestra.logs.type is not set, logs stay in the main backend (kestra.repository.type). Existing installations see no change on upgrade.

The full resolution order is:

1. kestra.logs.type — explicit log store selection
2. kestra.repository.type — fallback: logs stay in the main backend (the default)
3. If neither is configured, Kestra fails fast at startup with a clear error

Migration note

Configuring an external log store applies to new executions only. Historical logs remain in the main database.

Configure the JDBC log store

Select a JDBC store by setting kestra.logs.type to h2, postgres, or mysql. The store can connect to a dedicated log database or reuse the main datasource.

Config reference

When kestra.logs.<type>.url is not set, the JDBC store reuses the primary datasource and kestra.logs.type must match kestra.repository.type. This is useful for verifying config shape without standing up a second database, but it does not move logs out of the main database.

Example: PostgreSQL backend with a separate PostgreSQL log database

Two independent Postgres databases: one for the main backend, one for logs only.

services:
  postgres:                         # main backend DB
    image: postgres:16
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: k3str4
    volumes:
      - postgres-data:/var/lib/postgresql/data

  postgres-logs:                    # dedicated log DB
    image: postgres:16
    environment:
      POSTGRES_DB: kestra_logs
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: k3str4
    volumes:
      - postgres-logs-data:/var/lib/postgresql/data

  kestra:
    image: kestra/kestra:latest
    command: server standalone
    depends_on: [postgres, postgres-logs]
    ports: ["8080:8080"]
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driverClassName: org.postgresql.Driver
            username: kestra
            password: k3str4
        kestra:
          repository:
            type: postgres
          queue:
            type: postgres
          storage:
            type: local
          logs:
            type: postgres
            postgres:
              url: jdbc:postgresql://postgres-logs:5432/kestra_logs
              username: kestra
              password: k3str4
              table: logs

volumes:
  postgres-data: {}
  postgres-logs-data: {}

The main backend keeps using datasources.postgres. The log store opens its own connection pool against postgres-logs, runs log-table migrations there, and routes all log reads and writes to that database.

Configure the Elasticsearch log store

The Elasticsearch log store reuses the existing kestra.elasticsearch.* connection config; you add a kestra.logs block naming the index.

Config reference

Example: PostgreSQL backend with Elasticsearch log store

The main backend stays Postgres; execution logs go to Elasticsearch.

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: k3str4
    volumes:
      - postgres-data:/var/lib/postgresql/data

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    environment:
      discovery.type: single-node
      xpack.security.enabled: "false"
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
    ports: ["9200:9200"]

  kestra:
    image: kestra/kestra:latest-ee           # EE image required
    command: server standalone
    depends_on: [postgres, elasticsearch]
    ports: ["8080:8080"]
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driverClassName: org.postgresql.Driver
            username: kestra
            password: k3str4
        kestra:
          repository:
            type: postgres
          queue:
            type: postgres
          storage:
            type: local
          logs:
            type: elasticsearch
            elasticsearch:
              index: logs
          elasticsearch:
            client:
              http-hosts:
                - http://elasticsearch:9200
              # basic-auth:
              #   username: elastic
              #   password: changeme
            indices:
              logs:
                cls: io.kestra.core.models.executions.LogEntry
                index: logs

volumes:
  postgres-data: {}

Store capabilities

JDBC and Elasticsearch are complete implementations: they support offset pagination with exact totals, dashboard aggregation, and on-demand purge.

Each store declares its capabilities, and Kestra adapts gracefully:

When a store does not support aggregation, dashboard log charts display “No data” rather than erroring. When a store does not support purge, Kestra’s PurgeLogs operations no-op for logs — the backend’s own retention or TTL policy governs deletion.

Log Shipper vs External Log Data Store

Use the External Log Data Store when you want logs out of the main database. Use the Log Shipper when you want logs in a third-party observability platform as well, regardless of where Kestra stores them internally.

Build a new log store plugin

A new log store is a Kestra plugin that implements LogDataStoreInterface. Two reference implementations are available:

• log-data-store-splunk — offset pagination, full aggregation support, canPurge() == true
• log-data-store-datadog — cursor pagination, canPurge() == false (logs expire via index retention)

1. Implement the interface

Implement io.kestra.core.repositories.LogDataStoreInterface. For a new JDBC dialect, extend AbstractJdbcLogDataStore — it provides pagination, aggregation, and purge. Provide a no-arg constructor (the factory deserializes config onto the instance). If you need runtime Micronaut beans, implement ApplicationContextInitializable and wire dependencies in init(ApplicationContext).

2. Declare the plugin id

@Plugin
@Plugin.Id("mystore")     // the value operators set in kestra.logs.type
public class MyLogDataStore implements LogDataStoreInterface, ApplicationContextInitializable {
    @PluginProperty private String someOption;   // bound from kestra.logs.mystore.someOption
    ...
}

3. Declare capabilities

The defaults are canPurge() == true, canAggregate() == true, and paginationType() == OFFSET. Override only what your store cannot support:

@Override public boolean canPurge()              { return false; }  // backend handles TTL
@Override public boolean canAggregate()          { return false; }  // no server-side aggregation
@Override public PaginationType paginationType() { return PaginationType.CURSOR; }

A store declaring canPurge() == false must no-op all delete/purge methods: return 0 for integer-return methods and provide an empty body for void ones. A store declaring CURSOR must return a CursoredPage with a nextPageable() cursor token from paginated find(...) methods.

When the backend has no further results, emit an end-of-results sentinel as the next cursor token. If a caller sends that sentinel back, return an empty page without hitting the backend. The exact sentinel value is an implementation detail; your store just needs to recognize it on inbound requests and short-circuit.

4. Pass the contract test

Every log store implementation must pass io.kestra.core.repositories.AbstractLogDataStoreTest from the :tests module. The base class exercises every interface method and asserts every capability branch. Extend it with an empty class:

class MyLogDataStoreTest extends AbstractLogDataStoreTest {}

Wire your store as the injected LogDataStoreInterface bean via test config. The suite adapts to whatever your store declares, but your store must genuinely honor its declarations.

Two testing strategies are viable: run the contract suite against a real backend via Testcontainers (the Splunk approach, using the same Docker Compose stack as local QA), or inject a mock client through a package-private test constructor and gate live backend testing in a separate integration test class (the Datadog approach). Testcontainers gives higher confidence in query assembly and wire behavior; mock injection keeps CI fast when a live account is not available.