Use ES|QL across clusters

With ES|QL, you can execute a single query across multiple clusters.

• ES|QL cross-cluster search requires an Enterprise subscription on both the local (querying) cluster and every remote cluster.

• Remote clusters must be configured before running ES|QL across clusters.

  The destination cluster must be configured as a remote cluster on the local cluster, and the remote cluster connection must be correctly established. For setup instructions, refer to Set up remote clusters.

• The local node receiving the query must have the remote_cluster_client node role to connect to the remote clusters.

• ES|QL across clusters requires the remote cluster connection to use the API key-based security model.

  To verify which security model is active, run GET _remote/info. When API key authentication is in use, the response includes "cluster_credentials".

• For supported version pairings, see Supported cross-cluster search configurations.

ES|QL cross-cluster search requires some additional permissions beyond a standard Query DSL search. The following example creates a role that can query remote indices using ES|QL. The final remote_cluster privilege is required for remote enrich operations.

				POST /_security/role/remote1
					{
  "cluster": ["cross_cluster_search"],
  "indices": [
    {
      "names" : [""],
      "privileges": ["read"]
    }
  ],
  "remote_indices": [
    {
      "names": [ "logs-*" ],
      "privileges": [ "read","read_cross_cluster" ],
      "clusters" : ["my_remote_cluster"]
    }
  ],
   "remote_cluster": [
        {
            "privileges": [
                "monitor_enrich"
            ],
            "clusters": [
                "my_remote_cluster"
            ]
        }
    ]
}
		

1. The cross_cluster_search cluster privilege is required for the local cluster.
2. ES|QL authorizes the query at the local node before resolving which clusters it will target. A role used only for remote queries must therefore still grant read in the local indices block. Setting names to an empty string satisfies that check while matching no real index, so the user cannot read any local data. This requirement is specific to ES|QL; classic cross-cluster search authorizes per resolved index and does not need a local grant.
3. The indices allowed read access to the remote cluster. The configured cross-cluster API key must also allow this index to be read.
4. The read_cross_cluster privilege is always required when using ES|QL across clusters with the API key-based security model.
5. The remote clusters to which these privileges apply. This remote cluster must be configured with a cross-cluster API key and connected to the local cluster before the remote index can be queried. Verify connection using the Remote cluster info API.
6. Required to allow remote enrichment. Without this, the user cannot read from the .enrich indices on the remote cluster. The remote_cluster security privilege was introduced in version 8.15.0.

You then need a user or API key with the permissions you just created. The following example API call creates a user with the remote1 role.

				POST /_security/user/remote_user
					{
  "password" : "<PASSWORD>",
  "roles" : [ "remote1" ]
}
		

For the full reference of cross-cluster role privileges across all deployment types, refer to Configure privileges for cross-cluster search.

In the examples that follow, cluster_one, cluster_two, and cluster_three represent remote clusters that you've already configured on the local cluster where the query runs. The cluster name in each FROM clause is the alias you assigned during remote cluster setup.

In the FROM command, specify data streams and indices on remote clusters using the format <remote_cluster_name>:<target>. For instance, the following ES|QL request queries the my-index-000001 index on a single remote cluster named cluster_one:

FROM cluster_one:my-index-000001
| LIMIT 10
		

Similarly, this ES|QL request queries the my-index-000001 index from three clusters:

• The local ("querying") cluster
• Two remote clusters, cluster_one and cluster_two

FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001
| LIMIT 10
		

Likewise, this ES|QL request queries the my-index-000001 index from all remote clusters (cluster_one, cluster_two, and cluster_three):

FROM *:my-index-000001
| LIMIT 10
		

Using the "include_ccs_metadata": true option, you can request that ES|QL cross-cluster search responses include metadata about the search on each cluster (when the response format is JSON). Here we show an example using the async search endpoint. Cross-cluster search metadata is also present in the synchronous search endpoint response when requested. If the search returns partial results and there are partial shard or remote cluster failures, _clusters metadata containing the failures is included in the response regardless of the include_ccs_metadata parameter.

				POST /_query/async?format=json
					{
  "query": """
    FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index*
    | STATS COUNT(http.response.status_code) BY user.id
    | LIMIT 2
  """,
  "include_ccs_metadata": true
}
		

Which returns:

{
  "is_running": false,
  "took": 42,
  "is_partial": false,
  "columns" : [
    {
      "name" : "COUNT(http.response.status_code)",
      "type" : "long"
    },
    {
      "name" : "user.id",
      "type" : "keyword"
    }
  ],
  "values" : [
    [4, "elkbee"],
    [1, "kimchy"]
  ],
  "_clusters": {
    "total": 3,
    "successful": 3,
    "running": 0,
    "skipped": 0,
    "partial": 0,
    "failed": 0,
    "details": {
      "(local)": {
        "status": "successful",
        "indices": "blogs",
        "took": 41,
        "_shards": {
          "total": 13,
          "successful": 13,
          "skipped": 0,
          "failed": 0
        }
      },
      "cluster_one": {
        "status": "successful",
        "indices": "cluster_one:my-index-000001",
        "took": 38,
        "_shards": {
          "total": 4,
          "successful": 4,
          "skipped": 0,
          "failed": 0
        }
      },
      "cluster_two": {
        "status": "successful",
        "indices": "cluster_two:my-index*",
        "took": 40,
        "_shards": {
          "total": 18,
          "successful": 18,
          "skipped": 1,
          "failed": 0
        }
      }
    }
  }
}
		

1. How long the entire search (across all clusters) took, in milliseconds.
2. This section of counters shows all possible cluster search states and how many cluster searches are currently in that state. The clusters can have one of the following statuses: running, successful (searches on all shards were successful), skipped (the search failed on a cluster marked with skip_unavailable=true), failed (the search failed on a cluster marked with skip_unavailable=false) or partial (the search was interrupted before finishing or has partially failed).
3. The _clusters/details section shows metadata about the search on each cluster.
4. If you included indices from the local cluster you sent the request to in your cross-cluster search, it is identified as "(local)".
5. How long (in milliseconds) the search took on each cluster. This can be useful to determine which clusters have slower response times than others.
6. The shard details for the search on that cluster, including a count of shards that were skipped due to the can-match phase results. Shards are skipped when they cannot have any matching data and therefore are not included in the full ES|QL query.
7. The is_partial field is set to true if the search has partial results for any reason, for example due to partial shard failures, failures in remote clusters, or if the async query was stopped by calling the async query stop API.

You can use the cross-cluster metadata to determine whether any data came back from a cluster. For instance, in the query below, the wildcard expression for cluster_two did not resolve to a concrete index (or indices). The cluster is, therefore, marked as skipped and the total number of shards searched is set to zero.

				POST /_query/async?format=json
					{
  "query": """
    FROM cluster_one:my-index*,cluster_two:logs*
    | STATS COUNT(http.response.status_code) BY user.id
    | LIMIT 2
  """,
  "include_ccs_metadata": true
}
		

Which returns:

{
  "is_running": false,
  "took": 55,
  "is_partial": true,
  "columns": [
     ...
  ],
  "values": [
     ...
  ],
  "_clusters": {
    "total": 2,
    "successful": 1,
    "running": 0,
    "skipped": 1,
    "partial": 0,
    "failed": 0,
    "details": {
      "cluster_one": {
        "status": "successful",
        "indices": "cluster_one:my-index*",
        "took": 38,
        "_shards": {
          "total": 4,
          "successful": 4,
          "skipped": 0,
          "failed": 0
        }
      },
      "cluster_two": {
        "status": "skipped",
        "indices": "cluster_two:logs*",
        "took": 0,
        "_shards": {
          "total": 0,
          "successful": 0,
          "skipped": 0,
          "failed": 0
        }
      }
    }
  }
}
		

1. This cluster is marked as skipped, since there were no matching indices on that cluster.
2. Indicates that no shards were searched (due to not having any matching indices).
3. Since one of the clusters is skipped, the search result is marked as partial.

For more on partial results and how cluster status is determined when failures occur, see Cross-cluster search failures.

Enrich in ES|QL across clusters operates similarly to local enrich. If the enrich policy and its enrich indices are consistent across all clusters, write the enrich command as you would without remote clusters. In this default mode, ES|QL can execute the enrich command on either the local cluster or the remote clusters, aiming to minimize computation or inter-cluster data transfer. Ensuring that the policy exists with consistent data on both the local cluster and the remote clusters is critical for ES|QL to produce a consistent query result.

In the following example, the enrich with hosts policy can be executed on either the local cluster or the remote cluster cluster_one.

FROM my-index-000001,cluster_one:my-index-000001
| ENRICH hosts ON ip
| LIMIT 10
		

Enrich with an ES|QL query against remote clusters only can also happen on the local cluster. This means the following query requires the hosts enrich policy to exist on the local cluster as well.

FROM cluster_one:my-index-000001,cluster_two:my-index-000001
| LIMIT 10
| ENRICH hosts ON ip
		

ES|QL provides the enrich _coordinator mode to force ES|QL to execute the enrich command on the local cluster. Use this mode when the enrich policy is not available on the remote clusters or maintaining consistency of enrich indices across clusters is challenging.

FROM my-index-000001,cluster_one:my-index-000001
| ENRICH _coordinator:hosts ON ip
| SORT host_name
| LIMIT 10
		

ES|QL also provides the enrich _remote mode to force ES|QL to execute the enrich command independently on each remote cluster where the target indices reside. This mode is useful for managing different enrich data on each cluster, such as detailed information of hosts for each region where the target (main) indices contain log events from these hosts.

In the following example, the hosts enrich policy is required to exist on all remote clusters: the "querying" cluster (as local indices are included), the remote cluster cluster_one, and cluster_two.

FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001
| ENRICH _remote:hosts ON ip
| SORT host_name
| LIMIT 10
		

A _remote enrich cannot be executed after a STATS command. The following example would result in an error:

FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001
| STATS COUNT(*) BY ip
| ENRICH _remote:hosts ON ip
| SORT host_name
| LIMIT 10
		

You can include multiple enrich commands in the same query with different modes. ES|QL attempts to execute them accordingly. For example, this query performs two enrich commands, first with the hosts policy on any cluster and then with the vendors policy on the local cluster.

FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001
| ENRICH hosts ON ip
| ENRICH _coordinator:vendors ON os
| LIMIT 10
		

A _remote enrich command can't be executed after a _coordinator enrich command. The following example would result in an error.

FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001
| ENRICH _coordinator:hosts ON ip
| ENRICH _remote:vendors ON os
| LIMIT 10
		

To exclude an entire cluster, prefix the cluster alias with a minus sign in the FROM command, for example: -my_cluster:*:

FROM my-index-000001,cluster*:my-index-000001,-cluster_three:*
| LIMIT 10
		

To exclude a specific remote index, prefix the index with a minus sign in the FROM command, such as my_cluster:-my_index:

FROM my-index-000001,cluster*:my-index-*,cluster_three:-my-index-000001
| LIMIT 10
		

Cross-cluster search for ES|QL behavior when there are problems connecting to or running a query on remote clusters differs between versions.

FROM cluster_one:missing-index | LIMIT 10
FROM cluster_one:missing-index* | LIMIT 10
FROM cluster_one:missing-index*,cluster_two:missing-index | LIMIT 10
		

For broader context on the skip_unavailable setting, including how it interacts with allow_no_indices and ignore_unavailable, see Optional remote clusters.

You can still search a remote cluster while performing a rolling upgrade on the local cluster. However, the local node receiving the query must have an "upgrade from" and "upgrade to" version that is compatible with the remote cluster's gateway node.

For more information about upgrades, see Upgrading Elasticsearch.

You can use cross-project search (CPS) to query across multiple linked serverless projects. To learn more, refer to Query across Serverless projects.