Google Cloud Upload

Google Cloud Upload
Certified
Upload a file to GCS

Reads a file from Kestra internal storage and writes it to a gs:// destination. Supports content metadata settings.

yaml
type: io.kestra.plugin.gcp.gcs.Upload

Examples

Upload a FILE input to GCS

yaml
id: gcp_gcs_upload
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: upload
    type: io.kestra.plugin.gcp.gcs.Upload
    from: "{{ inputs.file }}"
    to: "gs://my_bucket/dir/file.csv"

Download data and upload to Google Cloud Storage

yaml
id: load_to_cloud_storage
namespace: company.team

tasks:
  - id: data
    type: io.kestra.plugin.core.http.Download
    uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv

  - id: cloud_storage
    type: io.kestra.plugin.gcp.gcs.Upload
    from: "{{ outputs.data.uri }}"
    to: gs://kestra-demo/data.csv

Properties

cacheControl string

Cache-Control

contentDisposition string

Content-Disposition

contentEncoding string

Content-Encoding

contentType string

Content-Type

expectedCrc32c string

Expected CRC32C of the source

Optional base64-encoded big-endian CRC32C. When set, the task fails if the source data read from Kestra internal storage does not match this checksum.

expectedMd5 string

Expected MD5 of the source

Optional base64-encoded MD5 digest. When set, the task fails if the source data read from Kestra internal storage does not match this digest. Use this to assert end-to-end integrity against a checksum produced upstream.

from string

Source file URI

Kestra internal storage URI to upload

Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.

impersonatedServiceAccount string

The GCP service account to impersonate

pluginDefaultsRef string

Reference (ref) of the pluginDefaults to apply to this task.

projectId string

The GCP project ID

scopes array

SubTypestring

Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

serviceAccount string

The GCP service account

to string

Destination object URI

gs:// bucket/path for the uploaded object

validateChecksum booleanstring

Defaulttrue

Validate checksum after upload

When true (default), compute MD5 and CRC32C client-side while streaming and compare them against what GCS reports server-side once the upload completes. When false, the implicit transit integrity check is skipped; explicit expectedMd5/expectedCrc32c assertions are still honored when set.

Outputs

crc32c string

Base64-encoded big-endian CRC32C of the uploaded data

md5 string

Base64-encoded MD5 of the uploaded data

uri string

Formaturi

Uploaded object URI

Metrics

file.size counter

Unitbytes

Size of the uploaded file.