Resource patterns for ES|QL Data Federation

A dataset's resource selects the objects it reads with a glob pattern, and entries in its file_exclusions setting are patterns in the same language. This page is the reference for that language: what each construct matches, how a resource is split into a listing prefix and a pattern, and which patterns are rejected as invalid.

The language is deliberately compatible with the glob syntax of the ClickHouse s3 table function, so a pattern written for it means the same thing here. The differences are listed in ClickHouse compatibility.

A resource is a storage URI whose path may contain pattern metacharacters: *, ?, [, and {. It is split into two parts:

• The listing prefix: everything up to and including the last / before the first metacharacter. This is the location that gets listed.
• The pattern: the remainder. Every listed object's path, taken relative to the listing prefix, is matched against it. The pattern must match the whole relative path, not a part of it.

For example, with the resource s3://logs/access/year=*/month=*/*.parquet, the object s3://logs/access/year=2024/month=06/part-0.parquet is matched as the relative path year=2024/month=06/part-0.parquet.

Some rules that follow from this model:

• Matching is case-sensitive. *.csv does not match data.CSV.
• Metacharacters are only special in the path. A * or { in the scheme or bucket name is literal text, not a pattern.
• A resource with no metacharacters names exactly one object, which is read directly without any listing.
• A comma separates resources. s3://b/a.csv, s3://b/b.csv is a list of two resources, each resolved independently. A comma inside a brace group belongs to the pattern, so s3://b/x.{csv,tsv} is a single resource. Whitespace around each listed resource is trimmed, and empty entries are ignored.

* matches any run of characters within a single path segment, including the empty run. ? matches exactly one character. Neither ever matches the / separator, so both are contained within one directory or file name.

** matches zero or more whole path segments, and is the way to search a directory tree recursively. It is only special when it stands alone as a complete segment: written next to other characters, a run of stars is the same as a single *, so a** means a* and stays within one segment.

Note what logs/**/events.csv does not match: ** matches whole segments only, so the pattern requires a file named exactly events.csv. It cannot stop partway through a name and treat old_events.csv as a match.

Because ** can match zero segments, a trailing /** also matches its anchor itself: logs/** matches an object named logs as well as everything under logs/.

[...] matches exactly one character from a set. The set can list characters, ranges, or both: [abc], [0-9], [a-cx-z0-9]. A class beginning with ! or ^ is negated and matches one character not in the set.

Like * and ?, a class never matches the / separator, negated or not, and a class containing / is rejected as invalid.

Within a class, a few positions make a metacharacter literal:

• ] as the first member is a literal ]: the class []] matches the character ].
• - first, last, or anywhere it does not sit between two characters is a literal -: [-x] and [x-] both match - or x.

There is no escape character in this language. \ matches a literal backslash and does not change the meaning of the character after it: the pattern a\*b matches a\ followed by anything and then b, because the * is still a wildcard.

To match a *, ?, [, or { that appears literally in an object name, put it in a one-character class:

] and } need no such treatment: outside a class or brace group they are ordinary characters.

A brace group with commas matches any one of its alternatives. Each alternative is itself a small pattern, so it can contain *, ?, and classes. An empty alternative matches the empty string. Brace groups cannot be nested.

A brace group of the form {N..M}, where both endpoints are non-negative integers and the group contains no comma, matches each integer in the range, endpoints included. Descending ranges such as {3..1} work too.

If either endpoint is written with a leading zero and more than one digit, every value is left-padded with zeros to the width of the wider endpoint. Otherwise there is no padding: a bare 0 does not turn it on, so {0..10} matches 0 through 10 unpadded, while {00..10} matches 00 through 10.

A brace body with .. that is not two bare integers is not a range. It falls back to plain alternation, and .. within an alternative is literal text: {a..c} matches only the literal a..c, not b, and {-1..3} matches only -1..3. Likewise, when a comma is present the body is alternation: {1..3,5} matches 1..3 or 5, not 2.

A numeric range can produce at most 1024 values. A wider one, such as {1..100000}, is rejected as invalid rather than silently truncated. The cap is on ranges because a range turns a dozen characters into any number of values; a comma list is limited by how much of it you type, and is not capped.

Malformed patterns are rejected with an error naming the problem, rather than being silently reinterpreted. A malformed resource fails the query that reads the dataset; a malformed file_exclusions entry is rejected when you register the dataset. The error always starts with Invalid glob pattern [<pattern>]: followed by one of:

By contrast, a stray ] or } with no matching opener is an ordinary character, not an error: a]b and a}b match themselves.

The language is compatible with the glob syntax of the ClickHouse s3 table function: *, ?, **, {a,b} alternation, and {N..M} numeric ranges mean the same thing, and \ is a literal there too. A pattern written for ClickHouse selects the same objects here, with two deliberate exceptions:

• Character classes are supported here. ClickHouse treats [ and ] as literal characters. A pattern relying on that, such as one matching a file literally named part[0].csv with bare brackets, must use a one-character class here: part[[]0].csv.
• Malformed patterns are rejected here. ClickHouse treats shapes such as an unclosed [ or { as literal text; here they are errors.

Four smaller differences:

• Here a * inside a brace alternative is a wildcard of that alternative, so {a*,b}.csv matches a.csv, axyz.csv and b.csv. In ClickHouse the braces and comma become literal characters while the * stays a live wildcard, so the same pattern matches names such as {ax,b}.csv.
• Here a run of stars glued to other text, such as a**, stays within one segment like a*. In ClickHouse it can cross directory levels. This follows ClickHouse's stated rule, that ** is special only as a complete path component, rather than its behaviour.
• Zero-padding of a numeric range differs at the edges. Here a range pads when either endpoint is written padded, so {1..05} pads; ClickHouse pads from one endpoint only.
• A numeric range here expands to at most 1024 values. ClickHouse has no such cap.

The partition_path dataset setting, which declares partition columns for paths that do not follow the Hive name=value convention, uses single braces as column placeholders: in partition_path, {year} means "this path segment is the year column". A resource gives the same spelling a different meaning: {year} is a brace group with one alternative, which matches only a directory literally named year.

				PUT /_query/dataset/access_logs
					{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/*/*.parquet",
  "settings": {
    "partition_path": "{year}"
  }
}
		

Here the placeholder belongs in partition_path, and the corresponding resource segment is a plain *. Writing "resource": "s3://logs-bucket/access/{year}/*.parquet" instead would read only a directory named year, which almost certainly does not exist, and the query would report that the pattern matched no files.

An entry in the file_exclusions dataset setting is an ordinary pattern in this language. There is no second dialect: an exclusion entry is matched against the same prefix-relative path as the resource pattern that discovered the object, with the same whole-path rule. An object whose relative path matches any entry is dropped from the listing.

The default is:

["**/_*", "**/.*", "**/_temporary/**", "**/_delta_log/**"]
		

The four entries are two different shapes, on purpose:

• The file-name rules **/_* and **/.* implement the Spark and Hive convention that a name beginning with _ or . is not data, covering markers and sidecars such as _SUCCESS, _metadata, and .part-0.crc at any depth. Because * cannot cross a /, these entries match only the final segment of the path: they exclude files by name and cannot touch a directory. That is what makes them safe for partitioned data, where values live in directory names: _dept=alpha/part-0.parquet and _foo/part-0.parquet are read, whatever the partition detection mode.
• The named-directory rules **/_temporary/** and **/_delta_log/** cover the two well-known directories whose contents look like data but are not: a failed Spark job's leftover part-files, and a Delta Lake transaction log. A wildcard directory rule such as **/_*/** would also swallow partition directories starting with _, so directories are only ever excluded by their exact name. A directory named literally _temporary or _delta_log that holds real data would be excluded by the default; replace the list to keep it.

Exclusions apply to wildcard discovery only. An object the resource names explicitly, whether as a pattern-free resource, a pattern-free member of a comma-separated list, or one of the objects a finite brace pattern names, is always read: naming an object is a request to read it.

Because exclusion entries are patterns, a malformed entry is rejected when the dataset is registered, with the message [file_exclusions] must contain only valid patterns followed by the specific pattern error.