A scrolled search takes a snapshot in time. It doesn’t see any changes that are made to the index after the initial search request has been made. It does this by keeping the old data files around, so that it can preserve its “view” on what the index looked like at the time it started.
The costly part of deep pagination is the global sorting of results, but if we
disable sorting, then we can return all documents quite cheaply. To do this, we
_doc. This instructs Elasticsearch just return the next batch of
results from every shard that still has results to return.
To scroll through results, we execute a search request and set the
scroll value to
the length of time we want to keep the scroll window open. The scroll expiry
time is refreshed every time we run a scroll request, so it only needs to be long enough
to process the current batch of results, not all of the documents that match
the query. The timeout is important because keeping the scroll window open
consumes resources and we want to free them as soon as they are no longer needed.
Setting the timeout enables Elasticsearch to automatically free the resources
after a small period of inactivity.
The response to this scroll request includes the next batch of results.
Although we specified a
size of 1,000, we get back many more
When scanning, the
size is applied to each shard, so you will
get back a maximum of
size * number_of_primary_shards documents in each
The scroll request also returns a new
_scroll_id. Every time
we make the next scroll request, we must pass the
_scroll_id returned by the
previous scroll request.
When no more hits are returned, we have processed all matching documents.