Data source settings reference for ES|QL Data Federation

Data source settings control how Elasticsearch connects and authenticates to an external storage system. Add settings to the settings object when you create or update a data source, or fill in the matching fields in the Connect data source flyout in Kibana. The available settings depend on the data source type.

Warning

This feature is experimental. It is not intended for production use and there are no guarantees around performance, scale, or stability in this release.

The following settings apply to data sources with type set to s3.

These settings control which endpoint Elasticsearch sends S3 requests to and how it addresses buckets.

endpoint

An Amazon S3 endpoint override, for example https://s3.us-east-1.amazonaws.com.

  • Default: None. The endpoint is resolved from the region.
  • Valid values:
    • An absolute URL naming a supported AWS S3 endpoint. Refer to S3 endpoint requirements.
    • Any endpoint URL. The value isn't validated.
  • Related: addressing_style, region

Omit endpoint to resolve the endpoint from the region, which is the recommended configuration.

S3 endpoint requirements

The endpoint value must be an absolute https URL naming a supported AWS S3 endpoint. Elasticsearch accepts the following endpoint forms. Every AWS partition supports the regional, historical, and VPC interface forms, and only the commercial partition supports the global form:

  • Regional: https://s3.us-east-1.amazonaws.com
  • Historical: https://s3-us-west-2.amazonaws.com
  • VPC interface: https://bucket.vpce-0a1b2c3d.s3.us-east-1.vpce.amazonaws.com
  • Global: https://s3.amazonaws.com

A regional endpoint must name a region that your Elasticsearch version knows about. Elasticsearch rejects a region that AWS added after that release until you upgrade, or until a node permits its host with esql.external.allowed_endpoint_hosts.

Note

https://s3.amazonaws.com has no region. When you set endpoint, the AWS SDK stops following cross-region redirects, so this global endpoint only reaches us-east-1 buckets. Other regions get an error. Omit endpoint to let the AWS SDK resolve the correct regional endpoint from the region setting.

Elasticsearch rejects every other AWS endpoint family, including FIPS endpoints, dual-stack endpoints, transfer acceleration, access points, object lambda, Outposts, the account-level control plane, the legacy s3-external-1 alias, and S3 Express. It also rejects a bucket-qualified endpoint such as https://mybucket.s3.us-east-1.amazonaws.com. Name the regional endpoint instead, and let the bucket come from the dataset. It also rejects plain http, a value without a scheme, and a host that the URL syntax doesn't allow, such as one with an underscore or a non-numeric port.

A node can permit additional hosts with the esql.external.allowed_endpoint_hosts node setting, a list of host:port patterns in elasticsearch.yml that defaults to empty. A host it names is also accepted over plain http for endpoint. The sts_endpoint setting always requires https.

Warning

A data source created before these endpoint restrictions were introduced keeps working for queries, but updating it requires an endpoint that meets these requirements.

addressing_style

The URL addressing style for S3 requests.

  • Default: auto
  • Valid values:
    • auto: Uses path-style when endpoint is set, and the AWS SDK default otherwise.
    • path: Always uses path-style.
    • virtual_hosted: Lets the AWS SDK decide. Bare-IP endpoints fall back to path-style.
  • Related: endpoint

Because auto resolves to path-style whenever endpoint is set, set virtual_hosted if reads through a VPC interface endpoint fail with an addressing error.

region

The AWS region used for the S3 client.

The region setting on a data source is deprecated and has no effect. Set region in the dataset settings instead, or omit it to let Elasticsearch detect the region automatically. When no endpoint is set, the AWS SDK redirects transparently. When one is set, Elasticsearch issues a HeadBucket probe on the first request and caches the discovered region. The cache is cleared after a few minutes without requests, and the next request discovers the region again.

Set region on the data source. Datasets don't accept a region setting.

These settings select the authentication model and supply the values it needs. For the AWS Identity and Access Management (IAM) permissions the configured identity needs, refer to grant read access in Amazon S3.

auth

The authentication model the data source uses.

  • Default: auto
  • Valid values:
    • auto: Infers the model from the other settings. Federated identity settings such as role_arn select federated_identity, and access_key with secret_key selects static_credentials. A data source with both, or with neither, is rejected. auto never selects anonymous or managed_identity, so set those explicitly.
    • anonymous
    • static_credentials
    • managed_identity
    • federated_identity

The Requires line on each of the following settings names the model the setting belongs to. When auth is auto or omitted, it refers to the model Elasticsearch infers.

access_key

The AWS access key ID.

  • Default: None
  • Requires: auth set to static_credentials
  • Related: secret_key, session_token

secret_key

The AWS secret access key.

  • Default: None
  • Requires: auth set to static_credentials
  • Related: access_key, session_token

session_token

The AWS session token for temporary security credentials issued by AWS Security Token Service (STS).

  • Default: None
  • Requires: auth set to static_credentials, with access_key and secret_key from the same temporary credentials
  • Related: access_key, secret_key

Temporary credentials expire. When they do, requests from the data source fail until you update all three values with new credentials.

role_arn

The Amazon Resource Name (ARN) of the IAM role Elasticsearch assumes through STS.

  • Default: None. Required when auth is set to federated_identity.
  • Requires: auth set to federated_identity
  • Related: jwt_audience, role_session_name, sts_endpoint, sts_region

jwt_audience

Overrides the JWT audience claim sent to STS.

  • Default: sts.amazonaws.com
  • Requires: auth set to federated_identity
  • Related: role_arn

role_session_name

A label for the assumed-role session.

  • Default: elasticsearch-esql-datasource
  • Requires: auth set to federated_identity
  • Related: role_arn

sts_endpoint

An STS endpoint override, for example https://sts.us-east-1.amazonaws.com.

  • Default: None
  • Valid values: An absolute https URL that meets the same S3 endpoint requirements as endpoint, but for STS hosts
  • Requires: auth set to federated_identity
  • Related: sts_region

Any host permitted through esql.external.allowed_endpoint_hosts receives the node's OIDC token, so only add hosts on trusted network paths.

sts_region

The AWS region of the STS endpoint.

  • Default:
    • The dataset's region setting, or us-east-1 if the dataset has no region
    • The data source's region setting, or us-east-1 if no region is set
  • Requires: auth set to federated_identity
  • Related: sts_endpoint, region

A region set on the data source no longer applies to STS. If a federated_identity data source relied on it, set sts_region on the data source, or region on each dataset, to keep calling STS in that region.