Upload

Upload a file(s) to a S3 bucket.

Uploads a single or multiple files to an Amazon S3 bucket.

yaml
type: "io.kestra.plugin.aws.s3.Upload"

Examples

Upload a FILE input to S3.

yaml
id: aws_s3_upload
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

  - id: myfile
    type: FILE

tasks:
  - id: upload
    type: io.kestra.plugin.aws.s3.Upload
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
    from: "{{ inputs.myfile }}"
    bucket: "{{ inputs.bucket }}"
    key: "path/to/file"

Download a file and upload it to S3.

yaml
id: upload_file_to_s3
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

  - id: file_url
    type: STRING
    defaults: https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip

tasks:
  - id: download_file
    type: io.kestra.plugin.core.http.Download
    uri: "{{ inputs.file_url }}"

  - id: upload_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    from: "{{ outputs.download_file.uri }}"
    key: powerplant/global_power_plant_database.zip
    bucket: "{{ inputs.bucket }}"
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"

Upload multiple files to S3 using a JSON array.

yaml
id: upload_multiple_files_from_json_array
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

tasks:
  - id: download_file1
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip"

  - id: download_file2
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/enhancing-adaptation-ambition-supplementary-materials.zip"

  - id: upload_files_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    from: |
      [
        "{{ outputs.download_file1.uri }}",
        "{{ outputs.download_file2.uri }}"
      ]
    key: "path/to/files"
    bucket: "{{ inputs.bucket }}"
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"

Upload multiple files to S3 using a Collection.

yaml
id: upload_multiple_files_to_s3
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

tasks:
  - id: download_file1
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip"

  - id: download_file2
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/enhancing-adaptation-ambition-supplementary-materials.zip"

  - id: upload_multiple_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    from:
      - "{{ outputs.download_file1.uri }}"
      - "{{ outputs.download_file2.uri }}"
    key: "path/to/files"
    bucket: "{{ inputs.bucket }}"
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"

Properties

bucket *string

The S3 bucket name.

from *Liststring

The file(s) to upload

Can be a single file, a list of files or json array

Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.

key *string

The key where to upload the file

A full key (with filename) or the directory path if from is multiple files

accessKeyId string

Access Key Id in order to connect to AWS.

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

acl string

The canned ACL to apply to the object

bucketKeyEnabled booleanstring

Specifies whether Amazon S3 should use an S3 Bucket Key for object encryption with server-side encryption using Key Management Service (KMS) keys (SSE-KMS).

Setting this header to true causes Amazon S3 to use an S3 Bucket Key for object encryption with SSE-KMS.

cacheControl string

Can be used to specify caching behavior along the request/response chain.

checksum string

The checksum data integrity check to verify that the data received is the same data that was originally sent.

Must be used in pair with checksumAlgorithm to defined the expect algorithm of these values

checksumAlgorithm string

Possible Values

CRC32CRC32_CSHA1SHA256CRC64_NVMEUNKNOWN_TO_SDK_VERSION

Indicates the algorithm used to create the checksum for the object when using the SDK

compatibilityMode booleanstring

Default false

This property will use the AWS S3 DefaultAsyncClient instead of the S3CrtAsyncClient, which maximizes compatibility with S3-compatible services but restricts uploads and downloads to 2GB. For some S3 endpoints such as CloudFlare R2, you may need to set this value to true.

contentDisposition string

Specifies presentational information for the object

contentEncoding string

Specifies what content encodings have been applied to the object

And thus, what decoding mechanisms must be applied to obtain the media-type referenced by the Content-Type header field.

contentLanguage string

The language the content is in

contentLength integerstring

The size of the body in bytes

This parameter is useful when the size of the body cannot be determined automatically.

contentType string

A standard MIME type that describes the format of the contents.

endpointOverride string

The endpoint with which the SDK should communicate.

This property allows you to use a different S3 compatible storage backend.

expectedBucketOwner string

The account ID of the expected bucket owner

If the bucket is owned by a different account, the request fails with the HTTP status code 403 Forbidden (access denied).

expires string

The date and time after which the object is no longer cacheable

forcePathStyle booleanstring

Force path style access.

Must only be used when compatibilityMode is enabled.

metadata object

SubType string

A map of metadata to store with the object in S3.

objectLockLegalHoldStatus string

Possible Values

ONOFFUNKNOWN_TO_SDK_VERSION

Specifies whether a legal hold will be applied to this object

objectLockMode string

Possible Values

GOVERNANCECOMPLIANCEUNKNOWN_TO_SDK_VERSION

The Object Lock mode that you want to apply to this object

objectLockRetainUntilDate string

The date and time when you want this object's Object Lock to expire

region string

AWS region with which the SDK should communicate.

requestPayer string

Sets the value of the RequestPayer property for this object.

secretKeyId string

Secret Key Id in order to connect to AWS.

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

serverSideEncryption string

Possible Values

AES256AWS_FSXAWS_KMSAWS_KMS_DSSEUNKNOWN_TO_SDK_VERSION

The server-side encryption algorithm used when storing this object in Amazon S3

For example, AES256, aws: kms, aws: kms: dsse

sessionToken string

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource.

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

storageClass string

Possible Values

STANDARDREDUCED_REDUNDANCYSTANDARD_IAONEZONE_IAINTELLIGENT_TIERINGGLACIERDEEP_ARCHIVEOUTPOSTSGLACIER_IRSNOWEXPRESS_ONEZONEFSX_OPENZFSUNKNOWN_TO_SDK_VERSION

If you don't specify, S3 Standard is the default storage class. Amazon S3 supports other storage classes.

stsEndpointOverride string

The AWS STS endpoint with which the SDKClient should communicate.

stsRoleArn string

AWS STS Role.

The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

stsRoleExternalId string

AWS STS External Id.

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

stsRoleSessionDuration string

Default PT15M

Format duration

AWS STS Session duration.

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

stsRoleSessionName string

AWS STS Session name.

This property is only used when an stsRoleArn is defined.

tagging object

SubType string

The tag-set for the object

Outputs

bucket string

The S3 bucket name

The name of the bucket where the file(s) were uploaded

files object

SubType

Information about uploaded files

A map of file names to their corresponding file information. Returned only for multiple file uploads.

key string

The S3 object key

The key (path) where the file(s) were uploaded in the bucket

versionId string

The version of the object.

Definitions

List keys of an S3 bucket.

bucket *string

The S3 bucket where to download the file.

accessKeyId string

Access Key Id in order to connect to AWS.

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

compatibilityMode booleanstring

Enable compatibility mode.

Use it to connect to S3 bucket with S3 compatible services that don't support the new transport client.

delimiter string

A delimiter is a character you use to group keys.

encodingType string

The EncodingType property for this object.

endpointOverride string

The endpoint with which the SDK should communicate.

This property allows you to use a different S3 compatible storage backend.

expectedBucketOwner string

The account ID of the expected bucket owner.

If the bucket is owned by a different account, the request fails with the HTTP status code 403 Forbidden (access denied).

filter string

Default BOTH

Possible Values

FILESDIRECTORYBOTH

The type of objects to filter: files, directory, or both.

forcePathStyle booleanstring

Force path style access.

Must only be used when compatibilityMode is enabled.

marker string

Marker is where you want Amazon S3 to start listing from.

Amazon S3 starts listing after this specified key. Marker can be any key in the bucket.

maxKeys integerstring

Default 1000

Sets the maximum number of keys returned in the response.

By default, the action returns up to 1,000 key names. The response might contain fewer keys but will never contain more.

prefix string

Limits the response to keys that begin with the specified prefix.

regexp string

A regexp to filter on full key.

ex: regExp: .* to match all files regExp: .*2020-01-0.\\.csv to match files between 01 and 09 of january ending with .csv

region string

AWS region with which the SDK should communicate.

requestPayer string

Sets the value of the RequestPayer property for this object.

secretKeyId string

Secret Key Id in order to connect to AWS.

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

sessionToken string

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource.

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

stsEndpointOverride string

The AWS STS endpoint with which the SDKClient should communicate.

stsRoleArn string

AWS STS Role.

stsRoleExternalId string

AWS STS External Id.

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

stsRoleSessionDuration string

Default PT15M

Format duration

AWS STS Session duration.

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

stsRoleSessionName string

AWS STS Session name.

This property is only used when an stsRoleArn is defined.

type object

io.kestra.plugin.aws.s3.models.FileInfo

contentLength integer

The size of the file in bytes

contentType string

The MIME type of the file

eTag string

An ETag is an opaque identifier assigned by a web server to a specific version of a resource found at a URL.

metadata object

SubType string

The metadata of the file

uri string

Format uri

The URI of the downloaded file in Kestra's storage

versionId string

The version ID of the file

​Upload

Upload