Create and manage datasets with the Elasticsearch API

Warning

This feature is experimental. It is not intended for production use and there are no guarantees around performance, scale, or stability in this release.

Datasets are managed under the /_query/dataset endpoint. All dataset operations require the index manage privilege on the dataset name, or a fine-grained dataset privilege. Refer to manage credentials and privileges for details.

Operation Endpoint API reference
Create or update PUT /_query/dataset/{name} Create or update an ES|QL dataset
Get GET /_query/dataset/{name} Get ES|QL datasets
List all GET /_query/dataset Get ES|QL datasets
Delete DELETE /_query/dataset/{name} Delete ES|QL datasets

PUT /_query/dataset/{name} creates a new dataset or replaces an existing one entirely.

A dataset cannot have the same name as an existing index, data stream, alias, or view, because dataset names share the same namespace. Dataset names must be lowercase and cannot begin with -, _, or +.

The optional description can be at most 1,000 characters long.

For restrictions on S3 bucket names, aliases, and Amazon Resource Names (ARNs), refer to Amazon S3 resource restrictions.

				PUT /_query/dataset/access_logs
					{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.parquet",
  "description": "Production access logs",
  "settings": {
    "partition_detection": "hive"
  }
}
		
curl -X PUT "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
  -H "Authorization: ApiKey ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.parquet",
  "description": "Production access logs",
  "settings": {
    "partition_detection": "hive"
  }
}'

Elasticsearch validates setting values when you register the dataset, rather than waiting until the first query. A value that isn't supported, such as a multi-character delimiter, an unknown encoding, or a segment_size below the minimum, returns a 400 error that identifies the setting.

Note

Datasets registered before this validation was introduced continue to work without being revalidated. Replacing one of these datasets triggers validation, so correct any unsupported values in the replacement request.

After creating a dataset, verify the inferred schema by checking its field mappings.

GET /_query/dataset/{name} retrieves a dataset by name.

				GET /_query/dataset/access_logs
		
curl -X GET "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
  -H "Authorization: ApiKey ${API_KEY}"

GET /_query/dataset returns all registered datasets.

				GET /_query/dataset
		
curl -X GET "${ELASTICSEARCH_URL}/_query/dataset" \
  -H "Authorization: ApiKey ${API_KEY}"

DELETE /_query/dataset/{name} deletes a dataset by name.

				DELETE /_query/dataset/access_logs
		
curl -X DELETE "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
  -H "Authorization: ApiKey ${API_KEY}"

After creating a dataset, query it with ES|QL. To change how the dataset selects or interprets files, return to define a dataset.