No pipeline, no second copy: Querying Amazon S3 from Elasticsearch with ES|QL

Get hands-on with Elasticsearch: Dive into our sample notebooks in the Elasticsearch Labs repo, start a free cloud trial, or try Elastic on your local machine now.

Elasticsearch Query Language (ES|QL) Data Federation lets you register an Amazon S3 bucket as a data source in Elasticsearch and query the Parquet, CSV, TSV, and NDJSON files inside it with the same FROM syntax that you use for an index. Nothing is copied into the cluster. Because the schema comes from the file itself, there’s no ingest pipeline to build or mapping to define. Filters and column selection push down into the Parquet reader, so a query reads only the columns it needs from S3. Datasets share a namespace with indices, and one query can cover a hot index and a year of cold objects at once. The feature is experimental in Elastic Cloud Hosted (ECH) 9.5 and 9.6, so treat it as exploratory for now. In this walkthrough, we go from an empty bucket to FROM sales | LIMIT 10 in just a few minutes, with a companion repo that sets up the bucket and the identity and access management (IAM) role, along with the sample data for you.

Use cases for querying Amazon S3 with ES|QL

Data federation is a useful tool in many use cases, including:

• Security incident investigation and threat hunting. Query multiyear CloudTrail, Virtual Private Cloud (VPC) Flow Logs, and Amazon Security Lake data in S3, directly alongside active alert indices.

• Compliance and legal audit. Retain long-term evidence in S3 object storage, while preserving on-demand search capabilities for compliance audits.

• Observability retention extension. Expand metric and log troubleshooting beyond hot-tier index retention windows, without doubling cluster sizes.

• AI security operations center (SOC) and agentic workflows. Supply security AI agents with historical baseline context, without ingesting data into Elastic.

Our walkthrough uses a small synthetic sales dataset so you can run it end to end in a few minutes. The same steps apply to CloudTrail, VPC Flow Logs, or Amazon Security Lake data stored as Parquet; just point the dataset at a different prefix.

Why ETL pipelines slow down time to insight

Zero-ETL minimizes or eliminates traditional data pipelines, enabling native queries across disparate storage silos without data movement. Traditional extract, transform, load (ETL) architectures introduce three major friction points:

1. Increased complexity. Every pipeline requires custom error handling, schema mapping, field transformations, and distinct access controls.

2. Escalating costs. Copying data that already lives in S3 into search indices means storing it twice, once in S3 and again in the cluster (plus replicas), and paying for the compute to ingest and index it.

3. Delayed time to insights and AI. Ingestion bottlenecks can delay data availability, rendering historical context stale for real-time threat hunting or AI decision-making.

How ES|QL Data Federation works

ES|QL Data Federation simplifies in-place querying into a straightforward four-step process:

1. Store data in Amazon S3. Parquet, NDJSON, CSV, or TSV files stay in your S3 bucket. The companion GitHub repo uses Parquet. Because Parquet is columnar, queries that select or filter columns read far fewer bytes from S3.

2. Define a data source. Register the connection and authentication model (federated identity or static credentials) in Elasticsearch. Updating credentials at the data source level updates all associated datasets automatically.

3. Define datasets. Point a dataset at specific file paths or wildcards in S3. Datasets reside in the same global namespace as indices, data streams, aliases, and views.

4. Query directly with ES|QL. Run standard FROM <dataset_name> queries. Elasticsearch handles file discovery, schema inference, and compression without mapping templates. It also infers the file format when the resource pattern implies a single format (for example, *.parquet). For extensionless paths or mixed patterns, such as *.{parquet,csv}, set format on the dataset explicitly.

How to set up ES|QL Data Federation with Amazon S3

Prerequisites

• An ECH deployment on version 9.5 or later, with an Enterprise subscription. Federated identity, used in this walkthrough, is available only on ECH. Self-managed, Elastic Cloud Enterprise (ECE), and Elastic Cloud on Kubernetes (ECK) clusters can use static credentials instead (see the repo.

• In 9.5 only, turn on the feature esql.federation.enabled: true (Elasticsearch) and xpack.dataFederation.enabled: true (Kibana). On Elastic Cloud, add these under the deployment's user settings and apply them. This triggers a rolling restart. In 9.6 and later, the feature is on by default.

• An AWS account, with the AWS Command Line Interface (CLI) configured with permission to create S3 buckets and IAM resources.

Setup checklist

1. Deploy Elastic. Use ECH (9.5+).

2. Enable the feature (9.5 only). Set the Elasticsearch and Kibana settings. No change needed in 9.6+.

3. Load the data. Create the S3 bucket, and upload the sample Parquet files.

4. Grant read access. Create a federated identity role with a least-privilege policy (s3:GetObject, s3:ListBucket).

5. Register. Create the data source and the sales dataset, with PUT /_query/data_source/... and PUT /_query/dataset/....

6. Query. Run FROM sales | LIMIT 10 in Discover.

Step 1: Deploy Elasticsearch on Elastic Cloud

• Open Elastic Cloud on AWS Marketplace to view purchase options and start the service.

• Or start Elastic Cloud directly. 

Step 2: Enable ES|QL Data Federation in Elasticsearch and Kibana

In Elasticsearch 9.5, Data Federation is an experimental feature and must be explicitly enabled across your Elasticsearch cluster and Kibana instance.

Note: In Elasticsearch 9.6 and later, the feature is in technical preview and turned on by default, so no configuration is needed. (To turn it off, set esql.federation.enabled: false.)

Elasticsearch configuration (elasticsearch.yml):

esql.federation.enabled: true

Kibana configuration (kibana.yml):

xpack.dataFederation.enabled: true

Step 3: Create the Amazon S3 bucket, and upload the Parquet files

Clone the companion repo, and then create the bucket and upload the four Parquet files to the business-data/ prefix:

git clone https://github.com/Udayel/s3-federation-elasticsearch.git
cd s3-federation-elasticsearch
export BUCKET=my-s3-federation-bucket   # must be globally unique
export REGION=us-west-2                 # use your deployment's region to avoid transfer costs

bash scripts/create-s3-bucket.sh "BUCKET""REGION"
python3 scripts/upload-parquet.py "BUCKET"--region"REGION"

The bucket is created, with versioning and default encryption. Plus, all public access is blocked.

Step 4: Create a least-privilege AWS IAM policy for S3 access

Elasticsearch needs only two permissions: s3:GetObject on the objects it reads, and s3:ListBucket on the bucket (used when a dataset's resource is a prefix or a glob):

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["s3:GetObject"],  "Resource": ["arn:aws:s3:::<bucket>/business-data/*"] },
    { "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::<bucket>"] }
  ]
}

The repo's setup script in the next step creates and attaches this policy for you.

Prefix scoping matters. The policy is scoped to arn:aws:s3:::<bucket>/business-data/*. If you place data under a different prefix (for example, business-data-parquet/), queries return “403 Access Denied” until you widen the policy resource; business-data/* doesn’t match business-data-parquet/.... Keeping the data in business-data/ needs no change.

Step 5: Connect Amazon S3 with federated identity (no AWS keys)

Federated identity avoids static AWS access keys. Elastic issues a signed token (JSON Web Token [JWT]) for your deployment, and AWS exchanges it, through sts:AssumeRoleWithWebIdentity, for temporary credentials on a role that only your deployment can assume.

• Get your identity details. In Kibana, go to Data management → ES|QL Data Federation → Connect data source. Choose Amazon S3, keep Federated Identity as the authentication method, and select the Manual setup method. Copy the issuer URL and the subject (deployment:<deployment-id>). Keep the flyout open.
• Create the AWS resources. Run the repo's script. It creates (or reuses) an IAM OpenID Connect (OIDC) identity provider for the issuer and a role whose trust policy accepts only tokens for your deployment and then attaches the read policy from Step 4:

bash iam/setup-federated-identity.sh "$BUCKET" "<JWT_ISSUER_URL>" "deployment:<deployment-id>"
   ```
   The trust policy it creates looks like this:
   ```json
   {
     "Effect": "Allow",
     "Principal": { "Federated": "arn:aws:iam::<account-id>:oidc-provider/<issuer-host>/<path>" },
     "Action": "sts:AssumeRoleWithWebIdentity",
     "Condition": {
       "StringEquals": {
         "<issuer-host>/<path>:aud": "sts.amazonaws.com",
         "<issuer-host>/<path>:sub": "deployment:<deployment-id>"
       }
     }
   }

The script prints the role Amazon Resource Name (ARN).

• As an alternative, select CloudFormation in the flyout, and click Open template in AWS Console. Elastic's template creates the identity provider and role. Then attach the Step 4 policy to that role.

For details, see Elastic's guide: Connect to Amazon S3 with federated identity.

Step 6: Register the data source and dataset in Elasticsearch

In Kibana → Dev Tools, create the data source with the role ARN from Step 5. Then create a dataset that points at the sales file:

PUT /_query/data_source/s3_federation
{
  "type": "s3",
  "settings": {
    "auth": "federated_identity",
    "role_arn": "arn:aws:iam::<account-id>:role/elastic-esql-federation-role",
    "jwt_audience": "sts.amazonaws.com"
  }
}

PUT /_query/dataset/sales
{
  "data_source": "s3_federation",
  "resource": "s3://<bucket>/business-data/sales.parquet"
}

In 9.5, also add "region": "<bucket-region>" to the data source settings. In 9.6+, that setting is deprecated, and Elasticsearch detects the region automatically.

The repo's elastic/02-datasets.console registers three more datasets (customers, inventory, and crm) from the same bucket.

Register a data source and dataset in Kibana

If you prefer the UI, in Data management → ES|QL Data Federation, the same flyout from Step 5 creates the data source. Paste the role ARN, and click Connect. Then use Add dataset to register sales, as shown in the following screenshots:

OPTIONAL: To script these requests instead of using Dev Tools, create an Elasticsearch API key in Kibana (Stack Management → API keys) and send each request with curl and the header Authorization: ApiKey <key>.

When should you index S3 data instead of querying it in place?

ES|QL Data Federation enables a progressive acceleration strategy; that is, keep raw data in low-cost S3 storage, and explore it on demand. Plus, promote high-value subsets into Elasticsearch indices only when advanced search capabilities are required. 

Regarding cost considerations, federation reads bytes from S3 per query, and nothing is duplicated into the cluster, so you pay S3 GET and data-transfer costs rather than index storage. On Parquet, using KEEP, WHERE, and LIMIT pushes down to the reader and cuts the bytes read, and keeping the bucket and deployment in the same region avoids cross-region transfer.

S3 federated datasets compared with Elasticsearch indices

Most ES|QL commands work the same way on a federated dataset as on an index. The exceptions are the ones that need structures that only an index has:

How to query Amazon S3 data with ES|QL in Discover

With the sales dataset registered, query it in Discover (ES|QL mode) exactly as you would an index:

1. Verify the dataset and inferred schema:

FROM sales
| LIMIT 10

Elasticsearch reads the schema from the Parquet footer, so the columns and types (order_date as a date, total_amount as a double) appear with no mapping.

2. Calculate the monthly revenue trend:

FROM sales
| EVAL ym = DATE_FORMAT("yyyy-MM", order_date)
| STATS orders = COUNT(*), revenue = ROUND(SUM(total_amount), 2) BY ym
| SORT ym

As we can see, the revenue peaked in September 2026 ($2.19 million across 94 orders). Note that October 2026 is a partial month.

3. Filter and project to read less from S3:

FROM sales
| WHERE status == "Completed"
| STATS orders = COUNT(*), revenue = ROUND(SUM(total_amount), 2) BY payment_method
| SORT revenue DESC

Because the filter and column selection are pushed down into the Parquet reader, this query reads only the status, payment_method, and total_amount columns from S3.

Beyond this, the repo registers three more datasets from the same bucket (customers, inventory, and crm), and elastic/03-queries.esql has queries for segmentation, customer relationship management (CRM) pipeline, and reorder alerts. You can also build Kibana dashboards on federated datasets, just as on indices.

Clean up the S3 bucket, IAM role, and datasets

To avoid ongoing charges when you're done, you need to remove the data, as well as IAM and keys for best practices:

# Elasticsearch: remove datasets + data source
DELETE /_query/dataset/customers  (and sales, inventory, crm)
DELETE /_query/data_source/s3_federation
# AWS: remove data, IAM, and keys
aws iam delete-access-key --user-name elasticsearch-s3-federation --access-key-id <KEY_ID>
aws s3 rm s3://my-s3-federation-bucket/business-data/ --recursive
aws s3 rb s3://my-s3-federation-bucket
# detach + delete the policy, role, and user created by setup-iam.sh

Next steps: From a federated dataset to an indexed one

ES|QL Data Federation lets you query Parquet, CSV, TSV, and NDJSON in Amazon S3 directly, providing one query language across hot indices and cold object storage, with no pipeline in between. Start with the repo's quickstart, and point a dataset at your own bucket. Promote to an index only when a workload needs kNN or joins, or if it needs time-series functions.

Try it: Using github.com/Udayel/s3-federation-elasticsearch, create the bucket, run upload-parquet.py, register a dataset, and then run FROM sales | LIMIT 5.