Why AutoOps for Elasticsearch needs clearer prioritization

Running Elasticsearch at scale requires administrators to monitor cluster health, performance, capacity, and configuration at the same time. AutoOps now provides a clearer way to distinguish conditions that threaten cluster functionality from significant but less urgent degradation. The redesigned interface also follows familiar Elastic Cloud Console patterns, making active issues easier to find and investigate.

What changed in AutoOps: severity, navigation, configuration, and page design

The monitoring engine remains the same. The redesigned layout, navigation, and workflows now follow familiar Elastic patterns.

A clearer severity model

We added Critical as a new severity level for conditions that pose an immediate threat to cluster functionality and require urgent intervention. Several events previously classified as High are now Critical. Others are now Medium because they represent potential risk rather than active, significant degradation. The reclassified events are:

• Promoted from High to Critical: Disk Watermark Flood Stage, Master Not Discovered, and Status Red.

• Demoted from High to Medium: Disk Watermark Low Threshold, Disk Watermark Low, and Disk Watermark Configuration Incorrect.

Severity

What it means

Critical

Immediate threat to cluster functionality. Urgent intervention required.

High

Significant degradation to usability, performance, or stability.

Medium

Potential risk that can escalate if left unaddressed.

Low

Minor anomalies with minimal operational impact.

Info

Routine operational updates and configuration changes. No action required. (Coming in a near-future update).

Every severity level ships with an updated icon set and color palette. Levels are fixed so teams can build consistent runbooks and notification filters: route Critical and High events to PagerDuty or Slack, keep Medium and Low in the console for periodic review, and when Info arrives, use it for awareness without alert fatigue.

Deployment view: open events and history, side by side

The redesigned deployment view presents the existing Open events and Event history tabs in a clearer layout.

Event flyout: a clearer view of what matters

The event detail flyout is redesigned around action. High-severity events include a notification callout and an interactive badge that shows whether alerts are configured and links directly to setup. Recommendations collapse by default so the core event stays in focus. Settings live in the flyout menu; share is a separate icon in the header. The Dismiss action appears only when your role has the required admin permissions and the event is dismissible.

AutoOps overview: triage active events across your Elasticsearch fleet

The Overview page is reorganized around how operators scan an estate. Elasticsearch context sits directly under the page header, and active events appear as event ribbons below the deployments table. Each ribbon shows the latest active event in your selected time range; if the same event type is open on other deployments, a new badge lets you expand the view without opening each resource individually. Event search moved to the left for quicker filtering.

The “Events over time” chart moved off Overview to keep this page focused on fleet-level triage; open a single deployment when you need that timeline.

Nodes, Shards, and Indices are designed with easier navigation and information hierarchy

Nodes view now uses updated chart components and the Elastic UI color scheme, with clear expansion indicators on accordion sections. Event and instance lists that duplicated deployment-level views were removed to reduce noise.

Shards view improves node selection and groups view controls in the upper-right corner. A horizontal scrollbar supports wider layouts, and the time slider now uses native Elastic UI components. Node selection in Shards view now works across larger clusters and presents up to 100 nodes at a time.

Index view keeps the Indices table experience you already use, including sorting, time-range brushing, and chart zoom behavior tuned for meaningful ranges.

Template Optimizer

The Template Optimizer now provides a searchable list of templates ordered by the most recently identified recommendations. You can open each recommendation directly or expand the JSON panel to inspect the complete template.

Configure notifications and event settings

Notification settings now include connector search, clearer filters, and a simpler connector editing flow. Event settings moved from a popup to a flyout, matching the pattern used across AutoOps. Notification reports retain the same 10-day history window with minor layout updates, and dismiss events use updated confirmation components aligned with Elastic UI.

Navigation and controls

The deployment picker now shows deployment ID and real-time cluster status, with copy actions for deployment name and ID in the dropdown sub-menu. Node selection supports select-all, select-by-tier grouping, and clear master node indication. The date picker follows the same relative-range and custom-range model used in Kibana and other Cloud Console monitoring views.

AutoOps roadmap: API, MCP, CLI, and agentic experience

Looking ahead, we are building toward a headless, agentic AutoOps experience. A forthcoming public AutoOps API will make insights and raw metrics available outside the AutoOps interface. Administrators and agents will be able to query the API directly or store its data in Elasticsearch. The API will also provide the foundation for integrations with MCP, Elastic Agent Builder, the Elastic CLI, Kibana, and native AutoOps chat.

• Hosted MCP server: Make AutoOps insights available to MCP clients such as Claude and Cursor.

• Native Elastic Agent Builder tool: Use AutoOps insights in Elastic Agent Builder.

• Elastic CLI support: Access the AutoOps API through the Elastic CLI.

• AutoOps in Kibana: Surface relevant insights and metrics within Kibana.

• Native AutoOps chat: Investigate cluster issues through an agentic chat experience within AutoOps UI in Elastic Cloud Console.

The application redesign is the foundation; these surfaces will meet operators where automation and AI already live. Read more about what is coming on the Elastic public roadmap.

How to start using the redesigned AutoOps in Elastic Cloud Console

Sign in to Elastic Cloud Console, open a deployment, project, or connected cluster, and select AutoOps from the navigation. Learn more in the AutoOps documentation.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Now, what if we could use real user interactions with our search application to create this training data? Using UBI data lets us do just that. Creating an automatic system that can capture and use our searches, clicks, and other interactions to generate a judgment list. This process can scale and be repeated far more easily than a manual interaction and would tend to yield better results. In this blog, we will explore how we can query UBI data stored in Elasticsearch to calculate meaningful signals to generate a training dataset for an LTR model.

You can find the full experiment here.

Why UBI data can be useful to train your LTR model

UBI data offers several advantages over a manual annotation:

• Volume: Given that UBI data comes from real interactions, we can collect much more data than we can generate manually. This is assuming we have enough traffic to generate this data, of course.

• Real User intent: Traditionally, a manual judgment list comes from an expert evaluation of the available data. On the other hand, UBI data reflects real user behavior. This means we can generate better training data that will improve our search system's accuracy, because it's based on how users actually interact with and find value in your content rather than theoretical assumptions about what should be relevant.

• Continuous updates: Judgment lists need to be refreshed over time. If we create them from UBI data, we can have current data that results in updated judgment lists.

• Cost effectiveness: Without the overhead of manually creating a judgment list, the process can be repeated efficiently any number of times.

• Natural query distribution: UBI data represent real user queries, which can drive deeper changes. For example, do our users use natural language to search in our system? If so, we might want to implement a semantic search or hybrid search approach.

It does come with some warnings, though:

• Bias amplification: Popular content is more likely to receive clicks, just because it gets more exposure. So this might end up amplifying popular items and possibly drowning out better options.

• Incomplete coverage: New content lacks any interactions, so it might be difficult for it to be high in the results. Rare queries can also lack sufficient data points to create meaningful training data.

• Seasonal variations: If you expect user behaviour to change drastically over time, historical data might not tell you much about what is a good result.

• Task ambiguity: A click doesn’t always guarantee that the user found what they were looking for.

Grades calculation

Grades for LTR training

To train LTR models, we need to provide some numerical representation of how relevant a document is for a query. In our implementation, this number is a continuous score going from 0.0 to 5.0+, where higher scores indicate higher relevance.

To show how this grading system works, consider this manually created example:

Query

Document content

Grade

Explanation

"best pizza recipe"

"Authentic Italian Pizza Dough Recipe with Step-by-Step Photos"

4.0

Highly relevant, exactly what the user is looking for

"best pizza recipe"

"History of Pizza in Italy"

1.0

Somewhat in topic, it is about pizza but is not a recipe

"best pizza recipe"

"Quick 15-Minute Pizza Recipe for Beginners"

3.0

Relevant, a good result but it maybe misses the mark on being the “best” recipe.

"best pizza recipe"

"Car Maintenance Guide"

0.0

Not relevant at all, completely unrelated to the query

As we can see here, the grade is a numerical representation of how relevant a document is to our sample query of “best pizza recipe”. With these scores, our LTR model can learn which documents should be presented higher in the results.

How to calculate the grades is the core of our training dataset. There are multiple approaches to do this, each with its own strengths and weaknesses. For example, we could assign a binary score of 1 for relevant 0 for not relevant or we could just count the number of clicks in a resulting document for each query.

In this blog post, we will be using a different approach, taking into account the user behavior as our input and calculating a grade number as the output. We will also be correcting bias that could occur from the fact that higher results tend to be more clicked, regardless of the relevancy of the document.

Calculating the grades - COEC algorithm

The COEC (Clicks over Expected Clicks) algorithm is a methodology for calculating judgment grades from user clicks. As we stated earlier, users tend to click on higher-positioned results even if the document is not the most relevant to the query; this is called Position Bias. The core idea for using the COEC algorithm is that not all clicks are equally significant; a click on a document at position 10 indicates that the document is much more relevant to the query than a click on a document at position 1. To quote the research paper about the COEC algorithm (linked above):

“It is well known that the click-through rate (CTR) of search results or advertisements decreases significantly depending on the position of the results.”

You can further read about position bias here.

To address this with the COEC algorithm, we follow these steps:

1. Establish position baselines: We calculate the click-through rate (CTR) for each search position from 1 to 10. This means we determine what percentage of users typically click on position 1, position 2, and so on. This step captures the users’ natural position bias. We calculate the CTR using:Where:

= Position. From 1 to 10

= Total clicks (on any document) at position p across all queries

= Total impressions: How many times any document appeared at the position p across all queries

Here, we expect higher positions to get more clicks.

2. Calculate Expected Clicks (EC):

This metric establishes how many clicks a document “should” have received based on the positions it appeared in and the CTR for those positions We calculate EC using:Where:

= All queries where the document d appeared

= Position of the document d in the query q results

3. Count actual clicks: We count the actual total clicks a document received across all queries where it appeared, hereafter called A(d).

4. Compute the COEC score: This is the ratio of Actual clicks (A(d)) over the Expected clicks (EC(d)):This metric normalizes for position bias like this:

• A score of 1.0 means the document performed exactly as expected given the positions it appeared in.

• A score above 1.0 means the document performed better than expected by looking at its positions. So this document is more relevant for the query.

• A score under 1.0 means the document performed worse than expected by looking at its positions. So this document is less relevant for the query.

The end result is a grade number that captures what users are looking for, taking into account position-based expectations extracted from real interactions with our search system.

Technical implementation

We will be creating a script to create a judgment list to train an LTR model.

The input for this script is the UBI data indexed in Elastic (queries and events).

The output is a judgment list in a CSV file generated from these UBI documents using the COEC algorithm. This judgment list can be used with Eland to extract relevant features and train an LTR model.

Quick start

To generate a judgment list from the sample data in this blog, you can follow these steps:

1. Clone the repository:

2. Install required libraries

For this script, we need the following libraries:

• pandas: to save the judgment list

• elasticsearch: To get the UBI data from our Elastic deployment

We also need Python 3.11

3. Update the environment variables for your Elastic deployment in a .env file

• ES_HOST

• API_KEY

To add the environment variables, use:

4. Create the ubi_queries, ubi_events indices, and upload the sample data. Run the setup.py file:

5. Run the Python script:

If you follow these steps, you should see a new file called judgment_list.csv that looks like this:

This script calculates the grades applying the COEC algorithm discussed before using the calculate_relevance_grade() function that is shown below.

Data architecture

Ubi queries

Our UBI queries index has information about the queries executed in our search system. This is a sample document:

Here we can see data from the user (client_id), from the results of the query (query_response_object_ids), and the query itself (timestamp, user_query)

Ubi click events

Our ubi_events index has data from each time a user clicked a document in the results. This is a sample document:

Judgment list generation script

General script overview

This script automates the generation of the judgment list using UBI data from Queries and Click events stored in Elasticsearch. It executes these tasks:

• Fetches and processes the UBI data in Elasticsearch.

• Correlates UBI events with its queries.

• Calculates the CTR for each position.

• Calculates the expected clicks (EC) for each document.

• Counts the actual clicks for each document.

• Calculates the COEC score for each query-document pair.

• Generates a judgment list and writes it in a CSV file.

Let’s go over each function:

connect_to_elasticsearch()

This function returns an Elasticsearch client object using the host and api key.

fetch_ubi_data()

This function is the data extraction layer; it connects with Elasticsearch to fetch UBI queries using a match_all query and filters UBI events to get ‘CLICK_THROUGH’ events only.

process_ubi_data()

This function handles the judgment list generation. It starts processing the UBI data by associating UBI events and queries. Then it calls the calculate_relevance_grade() function for each document-query pair to obtain the entries for the judgment list. Finally, it returns the resulting list as a pandas dataframe.

calculate_relevance_grade()

This is the function that implements the COEC algorithm. It calculates the CTR for each position, then it compares the actual clicks for a document-query pair, and finally calculates the actual COEC score for each one.

generate_judgment_statistics()

It generates useful statistics from the judgment list, such as total queries, total unique documents, or the grade distribution. This is purely informational and does not change the resulting judgment list.

Results and impact

If you follow the instructions in the Quick start section, you should see a resulting CSV file containing a judgment list with 320 entries (you can see a sample output in the repo). With these fields:

• qid: unique ID of the query

• docid: unique identifier for a resulting document

• grade: the calculated grade for the query-document pair

• query: The user query

Let’s look at the results for the query “Italian recipes”:

qid

docid

grade

query

q1-italian-recipes

recipe_pasta_basics

0.0

Italian recipes

q1-italian-recipes

recipe_pizza_margherita

3.333333

Italian recipes

q1-italian-recipes

recipe_risotto_guide

10.0

Italian recipes

q1-italian-recipes

recipe_french_croissant

0.0

Italian recipes

q1-italian-recipes

recipe_spanish_paella

0.0

Italian recipes

q1-italian-recipes

recipe_greek_moussaka

1.875

Italian recipes

We can see from the results that for the query “Italian recipes”:

• The risotto recipe is definitely the best result for the query, receiving 10 times more clicks than expected

• Pizza Margherita is a great result too.

• The Greek mousaka (surprisingly) is a good result as well and performs better than its position on the results would suggest. This means a few users looking for Italian recipes got interested in this recipe instead. Maybe these users are interested in Mediterranean dishes in general. At the end, what this tells us is that this could be a good result to be shown under the other two ‘better’ matches we discussed above.

Conclusion

Using UBI data lets us automate the training of LTR models, creating high-quality judgment lists from our own users. UBI data provides a big dataset that reflects how our search system is being used.By using the COEC algorithm to generate the grades, we account for inherent bias while at the same time, it reflects what a user considers a better result. The method outlined here can be applied to real use cases to provide a better search experience that evolves with real usage trends.

GCP Vector Search optimized instances: what you need to know

Elastic Cloud users benefit from having Elastic managed infrastructure across all major cloud providers (GCP, AWS and Azure) along with wide region support for GCP users. For more specific details on the instance configuration for this hardware profile, refer to our documentation for instance type: gcp.es.datahot.n2d.64x8x11

Vector Search, HNSW, and memory

Elasticsearch uses the Hierarchical Navigable Small World graph (HNSW) data structure to implement its Approximate Nearest Neighbor search (ANN). Because of its layered approach, HNSW's hierarchical aspect offers excellent query latency. To be most performant, HNSW requires the vectors to be cached in the node's memory. This caching is done automatically and uses the available RAM not taken up by the Elasticsearch JVM. Because of this, memory optimizations are important steps for scalability.

Consult our vector search tuning guide to determine the right setup for your vector search embeddings and whether you have adequate memory for your deployment.

With this in mind, the Vector Search optimized hardware profile is configured with a smaller than standard Elasticsearch JVM heap setting. This provides more RAM for caching vectors on a node, allowing users to provision fewer nodes for their vector search use cases.

If you’re using compression techniques like scalar quantization, the memory requirement is lowered by a factor of 4. To store quantized embeddings (available in versions Elasticsearch 8.12 and later) simply ensure that you’re storing in the correct element_type: byte. To utilize our automatic quantization of float vectors update your embeddings to use index type: int8_hnsw like in the following mapping example.

In upcoming versions, Elasticsearch will provide this as the default mapping, removing the need for users to adjust their mapping.

Combining this optimized hardware profile with Elasticsearch’s automatic quantization are two examples where Elastic is focused on vector search to be cost-effective while still being extremely performant.

Getting Started with Elastic Cloud vector search optimized profile for GCP

Start a free trial on Elastic Cloud and simply select the new Vector Search optimized profile to get started.

Migrating existing Elastic Cloud deployments

Migrating to this new Vector Search optimized hardware profile is a few clicks away. Simply navigate to your Elastic Cloud management UI, click to manage the specific deployment, and edit the hardware profile. In this example, we are migrating from a ‘Storage optimized’ profile to the new ‘Vector Search’ optimized profile. When choosing to do so, while there is a reduction to available storage and vCPU, what is gained is the ability to store more vectors per memory with vector search.

Migrating to a new hardware profile uses the grow and shrink approach for deployment changes. This approach adds new instances, migrates data from old instances to the new ones, and then shrinks the deployment by removing the old instances. This approach allows for high availability during configuration changes even for single availability zones.

The following image shows a typical architecture for a deployment running in Elastic Cloud, where vector search will be the primary use case.

This example deployment uses our new Vector Search optimized hardware profile, now available in GCP. This setup includes:

• Two data nodes in our hot tier with our vector search profile

• One Kibana node

• One Machine Learning node

• One integration server

• One master tiebreaker

By deploying these two “full-sized” data nodes with the Vector Search optimized hardware profile and while taking advantage of Elastic’s automatic dense vector scalar quantization, you can index roughly 60 million vectors, including one replica (with 768 dimensions).

Conclusion

Vector search is a powerful tool when building modern search applications, be it for semantic document retrieval on its own or integrating with an LLM service provider in a RAG setup. Elasticsearch provides a full-featured vector database natively integrated with a full-featured search platform. Along with improving vector search feature set and usability, Elastic continues to improve scalability. The vector search node type is the latest example, allowing users to scale their search application.

Elastic is committed to providing scalable, price effective infrastructure to support enterprise grade search experiences. Customers can depend on us for reliable and easy to maintain infrastructure and cost levers like vector compression, so you benefit from the lowest possible total cost of ownership for building search experiences powered by AI.