Elastic web crawleredit

Looking for the App Search web crawler? See the App Search documentation.

To compare the web crawler with the App Search web crawler, see the reference table on this page.

This feature is not available at all Elastic subscription levels. Refer to the Elastic subscriptions pages for Elastic Cloud and self-managed deployments.

Overviewedit

Use the web crawler to programmatically discover, extract, and index searchable content from websites and knowledge bases. When you ingest data with the web crawler a search-optimized Elasticsearch index is created to hold and sync webpage content.

The web crawler is a native Elasticsearch solution. It reads and writes directly to Elasticsearch indices in a format that enables developers to build intuitive, relevant search experiences using App Search engines and the Search UI library.

Web crawler documentation:

App Search web crawler

Web crawler

Interface

GUI / API

GUI-only in 8.4.0

Binary content extraction

Yes

Yes

Search

App Search

Elasticsearch / App Search using Elasticsearch search API for App Search

Ingest pipelines

Yes

Yes

Monitoring

Yes

Yes

APM

Yes

Yes

Audit logging

Yes

No

Event logging

Yes

Yes

Public REST API

Yes

No