icon

Elastic web crawler

An intelligent, intuitive indexing tool

Content indexing and syncing is a critical, behind-the-scenes function of any premium website search experience. The Elastic web crawler gives you all the power and flexibility you need to take the hassle out of ingesting content.

Start free trial

14-day trial, no credit card required

videoImage

New

A warm welcome to the web crawler — now generally available in 7.15! Enjoy flexible, automated indexing of all your website content.

What exactly is a web crawler?

Web crawling is a speedy way to index all the content on your website. You simply let it know what parts of your website you want it to scan — or "crawl" in web parlance — and it does all the heavy lifting of indexing your web content. Then at intervals you specify, it automatically recrawls and keeps all your content synced. Like magic, your content is ready for search.

Building memorable search experiences just got easier

There's a lot that goes into providing stellar search on your website. The new web crawler is just one of many Elastic tools that smooth the path for faster time to value, clearing hurdles to help you create, maintain, and optimize powerful, flexible search capability for your web properties.

It sweats the crawl stuff (so you don’t have to)

videoImage

It sweats the crawl stuff (so you don’t have to)

With the web crawler automatically handling all the intricacies of indexing, you're free to focus on the important stuff: driving engagement on your website, creating compelling content, and making strategic adjustments to the search experience.

Fire and forget...

Simple

Fire and forget...

Despite its raw indexing power, the web crawler is easy to use — no code needed. You can completely control your crawls right from the intuitive UI.

...Or get programmatical

Flexible

...Or get programmatical

If you prefer getting under the hood, you can take full control of the web crawler via flexible APIs.

You make the rules

Configurable

You make the rules

Easily configure rules to set up crawls your way, including allowing or blocking indexing of certain portions of your site.

Full visibility into crawl activity

Observable

Full visibility into crawl activity

Detailed activity logs and metrics give you full visibility into your crawl activity and history, so you get a clear picture of indexing performance. Plus, Kibana gives you an unlimited toolset for visualizing that data.

Do more with Elastic Enterprise Search

Search experiences for your workplace, websites, and apps, powered by the trusted speed, scale, and relevance of Elastic.

Elastic Workplace Search

Elastic Workplace Search

Unified search across all your content, with pretuned relevance and quick connections to popular apps.

Elastic App Search

Elastic App Search

Powerful, modern, easy-to-implement search experiences for your websites and apps.