Loading

Manage credentials and privileges for ES|QL Data Federation

Because federated data lives outside Elasticsearch, connecting to a private store means Elasticsearch holds that data source's credentials and must control who can manage connections and read external data. This page covers credential encryption and masking, and the privilege model for data sources and datasets.

When a data source connects to a private bucket, its encrypted credentials are stored in the cluster state. Elasticsearch protects these credentials with encryption at rest and redaction in API responses.

When a data source includes credentials, Elasticsearch encrypts them using the cluster state encryption key before storing them in the cluster state.

The cluster state encryption key is available automatically in most environments, including Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and Serverless.

By default, if the cluster state encryption key is not available when you create a data source, a PUT /_query/data_source request that includes credentials returns a 503 error.

On upgraded self-managed clusters, the encryption password is not configured automatically. To configure a cluster state encryption key, add cluster.state.encryption.password.<id> and cluster.state.encryption.active_password_id to the keystore on every node, then call POST /_nodes/reload_secure_settings.

Warning

Setting cluster.state.encryption.required: false lets a data source with credentials be created even when no encryption key is available. The credentials are then stored unencrypted in the cluster state, with only a warning in the logs. Keep the default unless you accept that risk.

Learn more about the cluster state encryption key.

All credential values are replaced by ::es_redacted:: in GET responses. Credentials are never returned in API responses.

A data source's credentials are masked when its definition is read back, and at query time the store is accessed using the data source's stored credentials.

Dataset operations are authorized by the standard Elasticsearch index privileges, so a role that already administers or reads the matching index names covers datasets with no additional grant.

Operation Privilege Type
Query a dataset read Index, on the dataset name
Create, read, or delete a dataset manage or all Index, on the dataset name
Dataset administration granted on its own create_dataset, read_dataset_metadata, delete_dataset, manage_dataset Index, on the dataset name
Create or replace a data source global.data_source create / cluster.manage Global (fine-grained) / Cluster
Read a data source definition global.data_source read_metadata / cluster.manage Global (fine-grained) / Cluster
Delete a data source global.data_source delete / cluster.manage Global (fine-grained) / Cluster
Reference a data source from a dataset global.data_source read / cluster.manage Global (fine-grained) / Cluster
All data source operations global.data_source manage / cluster.manage Global (fine-grained) / Cluster

Creating a dataset that references a data source also requires the read data source privilege for that data source. The two are authorized independently.

The read privilege granted on a dataset name must not carry document-level or field-level security. FROM <dataset> is rejected if it does. The same restriction applies to ES|QL views.

superuser has full access to data sources and datasets. Data source management is reached through the cluster manage (or all) privilege, or through a role explicitly granted global.data_source for fine-grained control.

A role configures these privileges as follows. The example grants querying sales and clicks, dataset administration over acme_*, and management of the acme_* data sources:

{
  "indices": [
    { "names": ["sales", "clicks"], "privileges": ["read"] },
    { "names": ["acme_*"], "privileges": ["manage"] }
  ],
  "global": {
    "data_source": [
      { "names": ["acme_*"], "privileges": ["manage"] }
    ]
  }
}