How to use the GCP EE plugin

Run tasks on Google Cloud Batch, Cloud Run, and Compute Engine, orchestrate existing Cloud Run Jobs, and ship execution logs to Cloud Storage and Cloud Logging from Kestra flows.

Authentication

• serviceAccount: a GCP service account JSON key, to authenticate explicitly.
• impersonatedServiceAccount: a service account email to impersonate for API calls.
• Neither set: application default credentials are used (GOOGLE_APPLICATION_CREDENTIALS or the environment's default service account).
• scopes: optional, defaults to https://www.googleapis.com/auth/cloud-platform.

Set projectId on each task, and keep keys in secrets.

Task runners

All three runners execute from the root directory, not the working directory. Use {{ workingDir }} or the WORKING_DIR environment variable to reach the task's files, and write output files to {{ outputDir }} (OUTPUT_DIR) for automatic capture.

runner.Batch

Runs task scripts on Google Cloud Batch VMs. Requires projectId and region.

• machineType: defaults to e2-medium.
• bucket: GCS bucket for staging inputFiles, outputFiles, and namespaceFiles.
• computeResource: cpu in milliCPU, memory in MiB, and bootDisk.
• reservation: pre-reserved VM resource name, to avoid provisioning delays.
• networkInterfaces: each requires network, optionally subnetwork.
• maxRetryCount: 0 to 10, defaults to 0.
• lifecyclePolicies.

Behaviour: delete (default true), resume (default true), waitUntilCompletion (default PT1H), completionCheckInterval (default PT5S), and logPollInterval (defaults to completionCheckInterval).

completionCheckInterval drives both the job-status poll and the Cloud Logging log poll, whose quotas differ widely: Cloud Logging reads are capped at 60 requests per minute per project. Raise logPollInterval alone to relieve that limit at high concurrency without slowing status detection.

IAM: roles/batch.jobsEditor, roles/logging.viewer, roles/storage.objectAdmin on the staging bucket, and roles/iam.serviceAccountUser.

runner.CloudRun

Runs task scripts as Google Cloud Run jobs. Requires projectId and region.

• bucket: GCS staging bucket.
• resources: cpu such as 1 or 1000m, and memory such as 512Mi or 2Gi.
• runtimeServiceAccount: the identity the Cloud Run job executes as, separate from the API authentication account.
• vpcAccessConnector, vpcEgress (PRIVATE_RANGES_ONLY or ALL_TRAFFIC), or network and subnetwork for Direct VPC Egress.
• volumes: GCS bucket volumes, each with bucket, mountPath, and readOnly.
• maxRetries: defaults to 3.

Behaviour: delete (default true), resume (default true), waitUntilCompletion (default PT1H), completionCheckInterval (default PT5S), and logPollInterval (defaults to completionCheckInterval).

IAM: roles/run.developer, roles/storage.objectAdmin on the staging bucket, roles/iam.serviceAccountUser, and roles/logging.viewer unless useBucketForLog is enabled. In bucket mode the runtime service account (runtimeServiceAccount, or the project's default compute service account) additionally needs write access to the staging bucket, because the container writes its own logs there.

Choosing a log transport

By default Kestra reads the container's output back from Cloud Logging, whose read path allows 60 requests per minute per project and which Google does not raise. Each running task polls for its own logs, and a task's outputs travel as a log line, so that limit caps you at roughly five concurrent Cloud Run tasks per project before log lines and outputs start going missing.

Set useBucketForLog: true to have the container write its own stdout and stderr into the staging bucket instead, as immutable chunk files Kestra streams as they appear. Cloud Storage meters reads per bucket rather than per project, so concurrency is no longer capped by logging.

Bucket mode in detail:

• Requires bucket. roles/logging.viewer is not needed for reads.
• Wraps the container command with kotlp, a small binary Kestra stages into the working directory automatically. No image change is needed beyond a POSIX /bin/sh.
• logFlushInterval (default PT2S) sets how often the current chunk is rotated and polled, so the worst-case delay before a line reaches Kestra is about twice that value.
• kotlp also samples the container's CPU, memory, disk I/O, and open file descriptors at the same cadence, reported as task metrics.
• Cloud Run still ships the container's console output to Cloud Logging as usual, so logs and outputs remain there and incur its ingestion cost.
• It is fail-closed. An incomplete chunk sequence fails the task rather than reporting success with missing output. A container killed outright, by out-of-memory or the Cloud Run task timeout, never writes its completion record, so it fails that way and may also lose its last flush interval of output.
• Chunk files live under the task's own working-directory prefix and are removed with it, unless deleteLogs is false, which keeps them for audit even when delete is true.

Cloud Logging gives no completeness guarantee, at any concurrency. A line can be indexed after the runner stops reading it and is then lost with no error and no rejected poll, so a clean run does not prove every line arrived. Bucket mode is the only transport that can verify it received everything, because Kestra checks the chunk sequence against the record kotlp writes on exit.

Log poller metrics

When logs come from Cloud Logging, each run reports the poller as task metrics tagged with projectId and region:

• cloudlogging.polls.issued
• cloudlogging.polls.rejected
• cloudlogging.entries.received, what Kestra read rather than what the container wrote
• cloudlogging.drain.duration

Rejection begins before data is lost, so sustained cloudlogging.polls.rejected is the signal to alert on. Isolated rejections are recovered by the poller's lookback window and are normal. logPollInterval has no effect in bucket mode, where logs are read at logFlushInterval.

Reusing one job with jobName

Set jobName to reuse a single stable Cloud Run Job across runs instead of creating and deleting one per run. The job is created once if absent, each run submits an execution against it, passing its command and working directory as per-run overrides, and only the execution is deleted afterwards. This removes the 1,000-jobs-per-project cap and roughly halves Admin API writes, from about four per run to two.

• The fixed template (image, resources, volumes, VPC, service account, maxRetries) and the command Kestra bakes in are set at first creation and reused as-is. Later changes are ignored until the job is deleted and recreated.
• useBucketForLog and logFlushInterval are part of that command, so changing them has no effect on an existing job. Changing useBucketForLog is rejected rather than ignored: delete the job, or point jobName at a new name.
• resume does not apply. A worker crash mid-run submits a new execution instead of reattaching, and the crashed run's execution is left behind, so repeated crashes accumulate undeleted executions.
• Unlike cloudrun.Run, which only triggers an existing job, runner.CloudRun creates the job when absent.

Resume after a worker restart

resume reattaches to a still-running execution so a restarted worker continues instead of re-running the task. Once Cloud Run has removed the execution, the transport decides what happens next:

• Bucket mode recovers the run from the bucket instead of re-running it, reported as adopted in the task runner output.
• The default transport cannot recover it, so the task re-runs and any side effect of the completed run happens twice.

The poller keeps no state across a restart, so it re-reads the chunk sequence and already-shown lines appear once more. Outputs and metrics are unaffected, because the replayed records carry the same values.

runner.ComputeEngine

Runs task scripts directly on a Compute Engine VM instance, with no SSH and no IAP tunnel. Requires projectId and zone, plus either instanceConfig (a raw instances.insert JSON object to create an instance) or instanceName (to target an existing one). Optionally set machineType (default e2-medium, only when creating an instance) and bucket for staging files.

The script is injected as the instance's startup-script metadata, and completion is detected through a guest attribute the script writes on exit. The image therefore needs bash, curl, and python3, which the default Debian and Ubuntu images have, plus enable-guest-attributes, which is set automatically.

Targeting an existing instance with instanceName reboots it (instances.reset) to trigger the script again. That is destructive to anything else the instance is doing, so never point two concurrent executions at the same instanceName. This runner never creates or deletes an instance it did not create itself, whatever deleteInstance says.

Behaviour:

• stopInstance (default true), ignored when deleteInstance is also true, or when targeting an existing instance.
• deleteInstance (default true), never deletes an instanceName-targeted instance but still controls whether that run's staged GCS prefix is cleaned up.
• resume (default true), reattaches to a matching Kestra-created instance instead of creating a duplicate when the worker restarts mid-run.
• waitUntilCompletion (default PT1H) and completionCheckInterval (default PT5S).

IAM: roles/compute.instanceAdmin.v1, or equivalent fine-grained permissions to create, start, stop and delete instances, read guest attributes and read the serial console, plus roles/iam.serviceAccountUser if the instance runs under a dedicated service account.

Cloud Run jobs

cloudrun.Run submits a new execution of an already-existing Cloud Run Job. Unlike runner.CloudRun it never creates, updates, or deletes the Job definition, it only triggers run.jobs.run. Requires projectId, region, and jobName, where jobName accepts either the short name or the full projects/{project}/locations/{region}/jobs/{job} path.

Override the Job template for one execution with containerArgs, containerEnvironment, and taskCount. Anything left unset falls back to the Job template's own configuration.

Behaviour: wait (default true, and when false the task returns straight after submission with only executionName set), completionCheckInterval (default PT5S), and logPollInterval (defaults to completionCheckInterval). The standard task-level timeout bounds how long it polls for completion, defaulting to one hour.

Logs come from Cloud Logging, capped at 60 requests per minute per project, so raise logPollInterval when several of these tasks run at once. Each waiting run reports the same four cloudlogging.* metrics described above. This task has no bucket transport, so Cloud Logging's lack of a completeness guarantee applies without an alternative: treat the metrics as an audit trail rather than proof every line arrived.

IAM: roles/run.developer to submit executions, and roles/logging.viewer to stream logs.

Log exporters

gcs.LogExporter ships execution logs to Google Cloud Storage. Requires projectId and bucket. Optionally set format (default JSON, also ION), maxLinesPerFile (default 100000), logFilePrefix (default kestra-log-file), and chunk (logs per request, default 1000). Requires roles/storage.objectCreator on the target bucket.

operationalsuite.LogExporter ships execution logs to Google Cloud Logging. Requires projectId. Optionally set chunk (logs per request, default 1000). Requires roles/logging.logWriter at the project level.