Elasticsearch accelerates building AI search apps on serverless

By

Alvin Richards,

Yaru Lin

starry_night_lake.jpg

Today we are announcing the availability of Elasticsearch Serverless in technical preview, which features:

  • A developer-focused experience that simplifies creating AI-driven search with intuitive onboarding and relevant code examples, all as a fully managed service

  • Cloud native, serverless architecture separating compute from data with the Search AI Lake

  • Elasticsearch project creation in just a few clicks to try out new AI search features

Early access customers have used this new self-service option for a range of use cases — from internal analytics to building generative AI applications and conducting machine learning tasks.

Optimized for vectors and generative AI development

In creating Elasticsearch Serverless, our goal has been to streamline the experience for the developer building generative AI applications. We made many API changes to provide smarter defaults and simplify data and language client onboarding. We added developer tools like the Developer Console and AI Playground, as well as ES|QL, a simplified search language with the power of a pipeline query model. All of these changes make the first steps of building AI search apps with Elasticsearch, for developers like you, simpler and easier. Building first class search experiences has never been faster — using open inference APIs, semantic search, and first- and third-party transformer models which work seamlessly with Elasticsearch search functionalities.

1 - Developer Console available everywhere in Kibana — example using built-in ELSER model
Developer Console available everywhere in Kibana — example using built-in ELSER model

We want to get generative AI innovation into developers’ hands faster. Going forward, you will see many new features, integrations, and techniques exposed in serverless before they are released to Elastic Cloud and Elastic’s self-hosted versions. Building prototypes and exploring the value of new features early in your project lifecycle helps you accelerate projects so you can get them into production sooner — all with a pay-as-you-go consumption model.

Finally, we are adding a specific profile for serverless projects that is optimized for vector storage and retrieval. Vectors need careful optimization, including byte quantization, to ensure that more low latency searches can be executed on a higher density of vectors and dimensions in memory. Low latency vector searches are critical for powering real-time end-user search experiences. Search Labs is your source for best practices and announcements of new features — which you can immediately try with Elasticsearch Serverless.

Search AI Lake and architectural changes

Under the covers, we have made some fundamental changes to the Elasticsearch architecture that enables us to bring the innovations in vector search and generative AI to you faster. The Search AI Lake allows your data to be stored in a low cost, reliable layer with full access to search that data for as long as you need. To learn more about the core architectural changes built into Elasticsearch Serverless, read the technical blog.

We have separated the data in the Search AI Lake from the compute nodes that index and search the data. This separation means that both Index and Search can automatically scale up and down independently based on usage — without the Index processing interfering with the Search processing and vice-versa. With autoscaling, the system can better meet the ebb and flow of indexing and search as required — all without intervention from your administrators or developers. Machine Learning can also be used on-demand, with similar autoscaling capabilities. As a further simplification, administrators and developers no longer have to think about nodes, shards, and other artifacts of how Elasticsearch is deployed. 

Serverless also means you no longer have to think about versions and upgrades. When you create a serverless project through the UI or API, Elastic keeps every project up to date — so that you can focus on building AI search experiences. We are launching with AWS and will extend the regions we support along with adding Azure and Google Cloud.

For developers accessing Elasticsearch through one of many language clients, existing code can already connect to serverless and execute searches. Just add API keys for the new serverless endpoint and you are done. We have also added a new collection of language clients just for serverless, which focus on the core APIs available for serverless.

2 - select your client

Pay for what you consume, without any management

Developers want simple endpoints to integrate with so they can build new AI applications — without management or administration tasks. Elasticsearch Serverless is an ideal platform to build generative AI experiences using RAG workflows. The power of Elasticsearch with no overhead lets developers focus on what they care about. The service scales up and down as you apply more or less load, optimizing your spend automatically — you are only invoiced for the consumption used.

3 - elasticsearch

Try it out

  • Sign up for a free Cloud trial from the Elastic Cloud Console on AWS.

  • Create and launch a serverless project for Elasticsearch.

  • Follow this step-by-step guide to select, install, configure, and test your client.

  • Ingest data and build your first search query.

  • Transform and enrich your data.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use. 

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial