Amazon S3 data sources for ES|QL Data Federation

An s3 data source reads files from Amazon S3. This page compares the authentication models, lists the AWS Identity and Access Management (IAM) permissions Elasticsearch needs to read your data, and explains how to troubleshoot access errors. For step-by-step setup, follow the guide for your authentication model.

Warning

This feature is experimental. It is not intended for production use and there are no guarantees around performance, scale, or stability in this release.

Choose an authentication model

Each authentication model has its own setup guide:

Model	`auth` value	Setup
Static credentials	`static_credentials`	Connect with static credentials
Federated identity	`federated_identity`	Connect with federated identity
Anonymous	`anonymous`	Quickstart
Managed identity	`managed_identity`	Authentication models

For a description of each model and where it's available, refer to authentication models. For every setting an s3 data source accepts, refer to the data source settings reference.

Grant read access

Elasticsearch reads your data as the IAM identity the data source uses: the IAM user that owns the access key, the IAM role that Elasticsearch assumes, or the node's own role. Attach a policy to that identity that allows the following actions:

Action	Resource	When it's required
`s3:GetObject`	The objects the dataset reads, for example `arn:aws:s3:::<bucket-name>/<path>/*`	Always
`s3:ListBucket`	The bucket, for example `arn:aws:s3:::<bucket-name>`	When a dataset's resource is a prefix or a glob pattern rather than a single file
`kms:Decrypt`	The AWS Key Management Service (AWS KMS) key that encrypts the objects	When the objects use server-side encryption with AWS KMS keys (SSE-KMS)

Scope the s3:GetObject resource to the prefixes your datasets use to grant the least access needed. When you add a dataset that reads from a new bucket or prefix, extend the policy to cover it. For a complete policy and AWS CLI commands, refer to the setup guide for your authentication model.

For anonymous data sources, the bucket policy must allow public access to the same actions.

Check access

To confirm the data source can read a dataset, run a query against the dataset. Creating a dataset doesn't contact S3.

The test connection endpoint confirms that the data source's credentials are valid, but not that they can read a particular bucket or path. For s3 data sources it calls ListBuckets, so credentials scoped to specific buckets, return untestable rather than success. A failure result means the request couldn't reach S3 or S3 rejected the credentials, for example because a key was deleted or rotated.

Troubleshoot access errors

Elasticsearch doesn't check permissions when you create a dataset. A missing permission shows up as an error the first time you query the dataset:

Access denied reading [<file>]: The identity can't read an object. Allow s3:GetObject on the object's path. If the message names a kms: action, the object is encrypted with a KMS key the identity can't use. Allow that action on the key.
Access denied listing objects in the configured path: The identity can't list the bucket. Allow s3:ListBucket on the bucket, or change the dataset's resource to an exact file path.
Access denied listing objects in bucket [<bucket>] with prefix [<prefix>]: The identity can't list the bucket. Allow s3:ListBucket on the bucket.