Preventing context rot with agentic AI

The recent explosion of generative AI (GenAI) introduced the public to large language models (LLMs), and by the same token (pun intended), to new ways of searching for information. Today, LLMs power conversational interactions between people and machines in an exchange of prompts and outputs. But the most exciting development may be the ability to curate the most relevant responses using context.

For earlier GenAI models, prompt engineering was used to craft inputs to direct the LLM’s reasoning. However, because general LLMs cannot generate outputs that rely on real-world or proprietary data unless explicitly fed this data, prompt engineering has evolved into context engineering.

Context engineering is context management. By limiting what information the LLM considers during reasoning, context engineering ensures that systems reason on pertinent data. By this merit, it is also the key to unlocking the full potential of agentic AI and enabling reliable agentic workflows.

Benchmark tests, such as the needle in the haystack (NIAH) paradigm, are advancing to improve model performance. NIAH tests search relevance by concealing a piece of data relevant to the query — the needle — in a mass of irrelevant data — the haystack. If the model successfully retrieves the needle during reasoning, it meets the benchmark. 

Search relevance measures how well search results provided by a search engine align with a user's query and intent. Attention mechanisms are instrumental to LLM information recall, the foundation of machine relevance and reasoning. But what does this mean for long-context scenarios? 

Long-context reasoning occurs over multiple sessions and multiple dependent subtasks and relies on heavier data inputs. In other words, in order to complete the query, the model must remember and recall more information throughout several interactions. Inevitably, this introduces added reasoning complexity and affects model performance. The attention mechanism faces increased difficulty in longer contexts, making it harder to retrieve relevant information.

As agentic AI develops and models are required to perform increasingly complex reasoning tasks, this long-context deficiency has earned its own moniker: context rot.

Context rot is an oversaturation of context data. An LLM’s context is its short-term memory. Effective short-term recall depends on retaining relevant information. This is as true for humans as it is for LLMs. Too much information in short-term memory rots a model’s attention budget and derails reasoning — hence context rot.

To better understand context rot, we must first understand the concept of the context window, interchangeably referred to as context length. In technical terms, context is the set of tokens that an LLM requires to reason, effectively the model’s working memory. The context window, or context length, is the number of tokens an LLM can “remember” in a given session. The larger the context window, the harder it is to “remember” and thus, to reason. This presents an important engineering challenge: managing the context window to consistently achieve desired outcomes.

The stakes for successfully managing context and avoiding context rot are high. For AI agents, poor model performance can result in hallucinations or inaccurate responses that have real effects on operations, slowing down teams to troubleshoot, or worse, go unnoticed and cause reputational harm. Down the line, poor model performance carries financial risks.

Context engineering is the answer to these challenges.

Context engineering is used to manage an LLM’s limited attention. Context engineering is like giving a machine a thorough brief: It contains the requests, the deliverables, and importantly, the tools and sources of information required for the completion of the task (or tasks) at hand. In this way, context engineering covers not only AI agents, but also the tools agents communicate with through the MCP protocol, providing tool-specific context sources and systems.

Context engineering relies on six key elements:

• System prompt: The set of main instructions that outline the type of tasks the agent must perform; includes guardrails and expected results
• User prompt: The set of instructions provided by the user outlining their desired result
• Retrieval augmented generation (RAG): A technique that allows teams to inject relevant data into the LLM context in order to ground the model
• Memory: User messages from a current session (short-term memory) and messages exchanged through previous sessions (long-term memory)
• Tools: Defined functions that the LLM can invoke to perform relevant actions, such as retrieve data and perform key operations
• Structured outputs: Defined data schemes used to inform the LLM of the structure generated outputs should take

All of these elements make up the context required in building a capable agent. As with any AI, the results are only as good as the data that goes in. Effective context engineering operates on the same principle. Simple system prompts, the right tools, and periodic context reviews are the foundation of building capable agents.

LLMs have limited attention: every new token eats into the attention budget. Effective context engineering, therefore, means finding the smallest possible set of high-signal tokens to maximize the likelihood and quality of desired outcomes. This requires:

• Simple system prompts: Direct language that strikes the right balance of detail. The optimal amount sits in the vector shared by two failure extremes: fragility and continued required maintenance as a result of complex logic, and inaccurate responses as a result of vague indications and assumed shared context.
• Thoughtful tool selection: Tools define the contract between agents and their information/action space, bridging agents and their context. Engineers must consider how the wrong tools (expressed as tools with overlapping functionality, or tools that aren’t well understood by LLMs) can contribute to context saturation and model performance degradation.
  

In addition to this, good context engineering requires periodically reviewing and trimming context and removing outdated or irrelevant information, effectively culling unnecessary tokens, to successfully manage model attention.

Context engineering encompasses prompt engineering, a narrower feature of instructing LLMs. For agentic AI, prompts remain valuable though incomplete LLM performance management techniques.

As agentic AI continues to develop, prompt engineering is being progressively incorporated into context engineering, which entails the dynamic management of an LLM’s attention. It’s not enough to ask for an answer in one go; users can adjust the instructions, examples, data, and tools required for the completion of a request.

Let’s consider an example. You ask an LLM, “How do I get to the airport?”

Prompt engineering would consist of adding a few elements to shape the LLM’s answer, like your starting point, which airport you’re headed to, and your mode of transportation.

Context engineering would add context such as a starting point, a map, a live list of street closures, or traffic updates to ensure that the agent can reason on its own.

LLMs have changed search by enabling exactly this kind of question. By leveraging natural language processing (NLP), LLMs have introduced conversational interactions between people and machines and given us conversational search.

There was a time when finding information on the internet required inputting specific search terms; search interfaces relied on keyword matching and ranking algorithms to produce results.

The evolution of keyword search to semantic search — or more commonly, conversational search — was made possible by the development of vector databases.

Vectors, or vector embeddings, are numeric representations of terms and weights. As a result, they allow broader machine “thinking”: machines can read words and understand their meaning by reading the relationship between vector points. This unlocks semantic understanding for machines and is key to a conversational search experience informed by semantic intent — the understanding of what the user actually means by the query.

Up until recently, however, vector search was limited to specific domains because models were trained on domain-specific data. LLMs fundamentally changed this paradigm.

Trained on trillions of tokens, LLMs have a wider surface across which to learn patterns and relationships between data points or embeddings. This gives them general knowledge and enables the understanding of concepts or terms within different contexts. Users can ask questions in plain language and in multiple parts and get a plain language response. 

Today, search engines also rely on LLMs to synthesize and summarize search findings, shifting the search experience from traditional discovery to understanding.  

When it comes to LLM-enabled search, we cannot throw caution to the wind. LLMs carry certain limitations and risks, namely:

• Accuracy: Too much context, not the right context, or not the right source material can cause hallucinations — false or inaccurate outputs that often sound plausible.
• Transparency: Even though models are trained on large corpora, they don’t retain a reliable index of “this sentence came from this URL.” Instead, they learn patterns across those texts. They perform predictions, and as a result, lack transparency into their sources.
• Bias: If training or source selection was biased, a model’s outputs will inherently carry that bias. This can have insidious consequences for users — and society at large.

Today, we interact with LLMs primarily through GenAI applications like Claude, ChatGPT, and Gemini. Already, they’ve reshaped how we find, interpret, and interrogate information, turning search into something more conversational, contextual, and immediate.

But this is only the beginning. The next shift is toward agentic AI: systems that don’t just respond to prompts but take initiative by planning, researching, and executing tasks on a user’s behalf. What is still a manual, back-and-forth process today is quickly becoming automated, as LLMs evolve from tools we query into agents that act.

Successful agents require successful context engineering — efficient tools, just the right amount of information and instructions, and relevant data. Successfully retrieving relevant data requires hybrid search with RAG. 

By combining lexical search (or keyword search) and semantic search, hybrid search provides a powerful and flexible means to surfacing relevant context.

The future of search lies in balance. It is achieved by hybrid approaches that combine multiple retrieval methods. By blending keyword precision with semantic understanding, hybrid search strikes a balance between accuracy, relevance, and contextual understanding.

This balance is critical. Hybrid search delivers stronger accuracy and efficiency — key to building reliable retrieval pipelines that power AI agents.

Need to use advanced query syntax to add missing components of subjective intent and filter out irrelevant content that muddies returned contextual information? Hybrid search can do that.

Need to filter responses to only include the fields and values you want? Use hybrid search.

Want the ability to select from individual available syntax options or combine modalities into a single search? Hybrid search is the way to go.

As far as context refinement goes, aggregations are the results of a hybrid lexical and semantic search, which can be especially useful in shaping the content a tool delivers to the context window because they provide numerical-based facts. These are more easily understood by LLMs and therefore more digestible reasoning feed. Aggregations help with managing context window size. Aggregations can also be ranked in order of relevance, which gives LLMs a more nuanced understanding of context.

Hybrid search platforms, such as the Elasticsearch Platform, combine traditional keyword and advanced vector search to refine context retrieval, enabling nuanced, intent-driven interactions. 

By unifying structured and unstructured data, these platforms give AI systems a richer foundation to reason over as they connect signals across logs, documents, metrics, and more. This allows models to move beyond surface-level relevance and retrieve information that reflects true user intent, even when queries are ambiguous or incomplete.

Just as importantly, an AI data platform introduces consistency and control into retrieval. It ensures that results are grounded in trusted, up-to-date data, improving both accuracy and explainability. For agentic systems, this is critical: Better retrieval doesn’t just improve answers; it also enables more reliable decision-making and action.

For agentic AI to be truly trustworthy, its responses must reflect the full spectrum of relevant information, including subjective factors that shape context and intent. Achieving this requires more than just clever algorithms; It depends on how the system retrieves and integrates knowledge.

Grounding AI with deliberate context engineering ensures that each decision or answer is informed by the right sources. Equally important is the performance of the retrieval tools themselves. The quality of what is pulled initially directly determines the accuracy, reliability, and usefulness of the agent’s output.