Create or update an ES|QL dataset Technical preview

PUT /_query/dataset/{name}
Api key auth

Creates or replaces a dataset that references a data source in ES|QL data federation. Dataset names participate in the index namespace and must follow index or alias naming rules. Returns 404 if the referenced data source does not exist.

Required authorization

  • Index privileges: manage
ES|QL Data Federation

Path parameters

  • name string Required

    The dataset name to create or update.

Query parameters

application/json

Body Required

  • data_source string Required

    The name of the referenced data source. The data source must already exist.

  • resource string Required

    The URI that identifies the data to read, resolved against the referenced data source. It can include glob patterns. For example, a recursive pattern can match all Parquet files under the s3://logs-bucket/access prefix.

  • description string

    A free-text description of the dataset.

  • mappings object

    User-declared mapping on the dataset definition

    Hide mappings attributes Show mappings attributes object
    • dynamic string

      The policy for columns that are not declared in properties. true (the default) infers undeclared columns and overlays the declarations; false makes the declaration the entire schema, so undeclared columns are not queryable.

      Values are true or false.

    • properties object

      The per-column declarations, keyed by logical column name.

      Hide properties attribute Show properties attribute object
      • * object Additional properties

        A per-column declaration inside a dataset mapping's properties.

        Hide * attributes Show * attributes object
        • type string Required

          The declared column type.

        • path string

          The underlying file column that this logical column reads from. This is a read-path rename: the physical column becomes this single logical column.

        • format string

          The date-parse pattern for a declared date column, mirroring the index date-field format.

    • _id object

      The _id meta-field configuration, sourcing the document identity from a column.

      Hide _id attribute Show _id attribute object
      • path string Required

        The name of the column that provides the document identity.

  • settings object

    Format and parsing-specific settings that configure how the resource is read. Common keys include format, which explicitly selects a registered format, and partition_detection, which accepts auto, hive, template, or none. Additional keys depend on the format reader. Compression can be inferred from the resource URI.

    Hide settings attribute Show settings attribute object
    • * object Additional properties

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.
PUT /_query/dataset/{name} 
PUT /_query/dataset/access_logs
{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.parquet",
  "description": "Production access logs",
  "settings": {
    "partition_detection": "hive"
  }
}
resp = client.esql.put_dataset(
    name="access_logs",
    data_source="prod_s3_logs",
    resource="s3://logs-bucket/access/**/*.parquet",
    description="Production access logs",
    settings={
        "partition_detection": "hive"
    },
)
const response = await client.esql.putDataset({
  name: "access_logs",
  data_source: "prod_s3_logs",
  resource: "s3://logs-bucket/access/**/*.parquet",
  description: "Production access logs",
  settings: {
    partition_detection: "hive",
  },
});
response = client.esql.put_dataset(
  name: "access_logs",
  body: {
    "data_source": "prod_s3_logs",
    "resource": "s3://logs-bucket/access/**/*.parquet",
    "description": "Production access logs",
    "settings": {
      "partition_detection": "hive"
    }
  }
)
$resp = $client->esql()->putDataset([
    "name" => "access_logs",
    "body" => [
        "data_source" => "prod_s3_logs",
        "resource" => "s3://logs-bucket/access/**/*.parquet",
        "description" => "Production access logs",
        "settings" => [
            "partition_detection" => "hive",
        ],
    ],
]);
curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"data_source":"prod_s3_logs","resource":"s3://logs-bucket/access/**/*.parquet","description":"Production access logs","settings":{"partition_detection":"hive"}}' "$ELASTICSEARCH_URL/_query/dataset/access_logs"
Request example
Create a dataset that reads partitioned Parquet access logs through the prod_s3_logs data source. 
{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.parquet",
  "description": "Production access logs",
  "settings": {
    "partition_detection": "hive"
  }
}
Response examples (200)
The dataset definition was stored successfully. 
{
  "acknowledged": true
}