Connect and manage data sources for ES|QL Data Federation

You can manage data sources in the Kibana UI or with the Elasticsearch API.

In the Kibana UI, you connect and manage data sources from the Data sources tab under Data management > ES|QL Data Federation.

To use the UI, enable it in Kibana first.

The Data sources tab lists each data source with its type, its description, and the number of datasets that read from it. You can search your data sources and connect new ones. Each row has the following actions:

• Edit ( ): Opens the data source's connection settings.
• Delete ( ): Available only when no datasets read from the data source.

In the following example, ookla_speedtest can't be deleted because a dataset reads from it:

×

To connect a data source:

1. Click Connect data source to open the Connect external data source flyout.
2. Enter a Name and an optional Description.
3. In Preferred method, choose how Elasticsearch authenticates to your storage:
   • Your bucket is public: select Anonymous. No credentials are stored.
   • Your bucket is private and your deployment runs on Elastic Cloud Hosted on AWS, Azure, or Google Cloud: select Federated Identity. It's the recommended method because no keys are stored, and it's the default when it's available. Refer to connect with federated identity.
   • Your bucket is private and federated identity isn't available: select Access and Secret Keys, then enter the key pair. Refer to connect with static credentials.
4. Click Connect.

Connecting a data source doesn't test whether it can read your files. To check the connection, create a dataset that reads from it and query the dataset.

To change a data source:

1. Click Edit ( ) in the data source's row.
2. Update its settings. You can't change the data source type or name.
3. Click Save.

Credential fields that you leave unchanged keep their stored values. When credentials rotate, update them here. Datasets that read from the data source use the new credentials without any changes.

A data source can't be deleted while datasets read from it. To delete a data source:

1. Delete the datasets that read from it.
2. Click Delete ( ) in the data source's row.

To delete several unused data sources at once, select them, then click Delete.

Data sources are managed under the /_query/data_source endpoint. All data source operations require the cluster manage privilege or a global.data_source privilege. Refer to manage credentials and privileges for details.

This section is a quick reference for each operation. For the complete request and response details, follow the API reference links in the table.

PUT creates a new data source or replaces an existing one entirely with one exception. Secrets you omit from the request are carried forward from the existing definition rather than cleared, so you can update non-secret settings without re-sending credentials.

The create request does not validate connectivity to the external system.

				PUT /_query/data_source/prod_s3_logs
					{
  "type": "s3",
  "description": "Production S3 logs bucket",
  "settings": {
    "auth": "static_credentials",
    "access_key": "<AWS_ACCESS_KEY_ID>",
    "secret_key": "<AWS_SECRET_ACCESS_KEY>"
  }
}
		

curl -X PUT "${ELASTICSEARCH_URL}/_query/data_source/prod_s3_logs" \
  -H "Authorization: ApiKey ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
  "type": "s3",
  "description": "Production S3 logs bucket",
  "settings": {
    "auth": "static_credentials",
    "access_key": "<AWS_ACCESS_KEY_ID>",
    "secret_key": "<AWS_SECRET_ACCESS_KEY>"
  }
}'
		

A create or update request is rejected when it breaks one of these rules or limits:

• Name: Follows the same rules as index names. Lowercase only and at most 255 bytes. Can't begin with -, _, or +, and can't contain spaces or the characters \ / * ? " < > |.
• Description: At most 1,000 characters.
• Number of data sources: At most 100 per cluster by default. In Elastic Stack deployments, you can raise the limit with the esql.data_sources.max_count cluster setting.

Retrieves a data source by name. You can pass a comma-separated list of names and use * wildcards. A concrete name that does not exist returns a 404; a wildcard that matches nothing returns an empty list. Credential values are replaced by ::es_redacted:: in the response.

				GET /_query/data_source/prod_s3_logs
		

curl -X GET "${ELASTICSEARCH_URL}/_query/data_source/prod_s3_logs" \
  -H "Authorization: ApiKey ${API_KEY}"
		

Returns all registered data sources.

				GET /_query/data_source
		

curl -X GET "${ELASTICSEARCH_URL}/_query/data_source" \
  -H "Authorization: ApiKey ${API_KEY}"
		

Deletes one or more data sources by name. You can pass a comma-separated list. If any named data source does not exist, the request returns a 404 and nothing is deleted.

				DELETE /_query/data_source/prod_s3_logs
		

curl -X DELETE "${ELASTICSEARCH_URL}/_query/data_source/prod_s3_logs" \
  -H "Authorization: ApiKey ${API_KEY}"
		

Continue with the following data-source tasks:

• Create datasets that point at specific files in your data source, then review file formats and dataset settings.
• Query your datasets with FROM to learn how partition pruning, column selection, and filter pushdown reduce storage reads.
• Manage credentials and privileges to control who can create data sources and read external data.