Static ID lists vs. dynamic filtering with ES|QL's WHERE clause

Static ID list filtering

Dynamic filtering with WHERE IN subquery

Run query A to find the IDs you care about

Outer query asks the main question

Copy the values by hand

Subquery builds the filter list from live data

Paste them into a static WHERE IN list

WHERE field IN (subquery) applies the filter live

Run query B with the hardcoded list

Results stay current with the data

Repeat the whole process when the data changes

No copied list to maintain

Figure 1. The old copy-paste loop collapses into a single dynamic filter.

How the WHERE IN subquery replaces static filter lists

Traditional IN filtering is still useful when the list is small and static:

But many real investigations don’t start with a tidy list. They start with a question: Which users are suspicious, which hosts are noisy, which services are failing, or which accounts crossed a threshold?

That’s where the WHERE IN subquery becomes useful. The list is produced by ES|QL rather than typed by hand.

WHERE IN subquery example: filtering logs by suspicious users

Read it like this: Show me log events for users who appear in the list of users with at least 10 failed login attempts.

The outer query asks the main question, and the subquery builds the dynamic filter list, eliminating the need to copy and paste.

The subquery can target a different index or index pattern from the outer query.

Filtering without subqueries: the manual ID copy workflow

Imagine that you want to inspect traffic for the top failing services. First you run:

Then you copy the five service names and paste them into another query:

That’s fine once, but less fine when the top five change every hour.

Dynamic filtering with a WHERE IN subquery

The subquery finds the top failing services from the last two days, and the outer query returns the log events for those services. One query builds the full picture.

Excluding values with NOT IN subqueries in ES|QL

Sometimes the interesting question is about what doesn’t belong:

That pattern is useful for exclusion checks, gap analysis, and workflows that ask for the things outside an approved or expected set.

Nested subquery chains in ES|QL's WHERE clause

An IN subquery replaces the literal value list with a query in parentheses. The inner query runs first and returns a single column, and the outer WHERE filters against it. Because that inner query is a full pipeline, it can contain its own IN subquery, which lets you express a chain of lookups that would otherwise require three separate queries and two rounds of copy-paste.

Read it inside out. The innermost query finds priority regions, and the middle query finds customers in those regions, while the outer query sums revenue for those customers. Each layer is a normal ES|QL pipeline, so each one can filter, aggregate, or sort on its own before handing a clean column up to the layer above.

Combining WHERE IN subqueries with AND and OR conditions

Because an IN subquery is a Boolean condition, it composes with AND and OR like any other predicate. You can require membership in two independent sets or accept membership in either:

The AND combination finds orders placed by VIP customers for products that are being discontinued. Swap AND for OR, and you get orders that match either condition. Each subquery runs its own pipeline, so the two sets are computed independently and then combined by the Boolean operator.

Merging multiple indices into one WHERE IN subquery

The query inside an IN subquery is a full pipeline, so its FROM command can reference more than one subquery. Each branch runs its own pipeline, and the FROM command merges the rows from all branches into one result set. KEEP host_id selects the single column that the outer filter needs. This is useful when the values you want to filter against live in several indices with different schemas. For more details on how subqueries in the FROM command handle indices with different schemas, see Three indices walk into a FROM clause: ES|QL subqueries in Elasticsearch.

The IN subquery combines matching host IDs from three indices, prod, staging, and edge, into one value list. The outer query then counts alerts for any host in that combined set. Adding a fourth source means adding one more branch, with no change to the outer query.

Using an Elasticsearch subquery inside each FROM branch

A FROM subquery gives each index its own branch with its own WHERE, and that WHERE can use an IN subquery. This is how you apply the same dynamic filter across several indices that each have their own schema.

Each branch filters its index down to VIP customers before the two branches combine, so the final aggregation runs over a single normalized set of rows.

When to use ES|QL WHERE IN subqueries

• Investigations that start by finding risky users, hosts, accounts, or services.

• Operational dashboards where the interesting entities change over time.

• Top-N follow-up queries, such as events for the five noisiest services.

• Set comparison workflows, especially with NOT IN.

• Queries that would otherwise need glue code just to pass values from one step to the next.

Requirements and constraints for WHERE IN subqueries

• Return exactly one column from the IN subquery.

• Use KEEP at the end of the subquery so the comparison field is obvious.

• Make sure the outer field and the subquery field have compatible types.

• If the subquery uses SORT, add an explicit LIMIT, as unbounded SORT isn’t supported in ES|QL yet.

• Use this for membership filtering. If you need columns from both sides, a JOIN` may be the better tool.

Why ES|QL dynamic filtering replaces manual query workflows

The WHERE IN subquery turns a manual workflow into a declarative one. You can let one query build the filter for another query directly inside the WHERE command, instead of asking ES|QL for a list, copying it somewhere else, and hoping it stays fresh. 

Your WHERE clause now has a better way to handle Filter this by whatever that query finds.

If you've been exploring ES|QL, you might have noticed that a subquery used to feel like it had exactly one front door: FROM. And it made sense. Most of the time, queries start with a simple directive to go fetch documents from an index. In an earlier post, we taught the WHERE command a new trick: IN subqueries. And before that, subqueries showed up in the FROM command to combine data sources. In both instances, every subquery started the same way, with FROM.

But not every useful query starts with a bulk document fetch. Sometimes you need to evaluate time-series semantics. Other times, you just need to whip up a tiny inline row for testing. Sometimes, the absolute best input to a filter is a dynamic query that builds the list for you on the fly, rather than a static, hard-coded list.

Source command

Reads from

Best for

Requirements

FROM

Indexes, data streams, aliases, views

Stored document lookups, live filter lists

None (works with any index)

TS

Time-series data streams

Metric aggregations with counter-reset handling

TSDS with index.mode: time_series

ROW

Inline literal values

Test cases, seed values, synthetic placeholders

None (no index needed)

The FROM, TS and ROW source commands are generally available (GA) in Elasticsearch 9.5, while the WHERE IN subquery remains in technical preview in 9.5.

How ES|QL subquery source commands work

Think of subqueries as having two specific placements and three different engines.

Where subqueries go: FROM and WHERE placements

• Inside the FROM command: The subquery is an independent result source, contributing its rows to the outer query. Fields that exist in one source but not the other are gracefully filled with null values.

• Inside the WHERE command: The IN subquery contributes dynamic values to be used as a predicate or filter.

Three source commands for starting a subquery

Door #1: FROM (the classic door)

FROM is the familiar workhorse. You use it when the subquery should read stored data from indices, data streams, aliases, or views. One of its best use cases is the "stop-copy-pasting-IDs" pattern. Instead of running one query, manually copying the output values, and pasting them into the filter of another query, the subquery becomes your live filter list.

Door #2: ROW (the tiny door)

ROW is the lightweight option that requires absolutely no index setup. It allows you to build rows completely out of literal, inline values. This makes ROW useful for small seed values, test cases, allow/deny lists, or one-off "what if?" scenarios. In the query below, ROW is the perfect way to staple a synthetic sentinel row or placeholder directly onto real data.

Door #3: TS (the time-series door)

The TS command targets time-series data streams and enables time-series aggregation functions. Why not just use FROM for metrics? TS is uniquely optimized for time-series data, and it natively handles tricky scenarios, like counter resets on process restarts and uneven metric publish intervals. Using TS as a subquery lets you filter your application logs by what your metrics are saying. For example, imagine asking ES|QL: Show me log events only from clusters whose peak metric throughput crossed 800:

In this scenario, the filter is reacting to live metric behavior rather than being hard-coded.

Combining FROM, TS and ROW in one query

Because subquery placements and source commands are independent, you can freely mix and match them. You can throw all three doors into a single FROM union to generate a cohesive table containing real logs, a live metrics summary, and a synthetic row.

ES|QL subquery constraints

A few constraints to keep in mind before using subqueries:

• IN subqueries demand one column: If a subquery feeds an IN operator, it must project exactly one column. Use the KEEP command to make that explicitly clear.

• TS requires a time series data stream (TSDS): The TS command only works on data stored in a TSDS, which uses index.mode: time_series.

The takeaway: when to use each source command

Subqueries in ES|QL provide a structured way to compose queries by using the results of one query as the input to another. Choosing the appropriate source command (FROM, ROW, or TS) lets you combine data and generate inline values. It also lets you filter dynamically without duplicating query logic. For more details and additional examples, see the ES|QL subqueries documentation.

• Subquery in FROM command

• Subquery in WHERE command