From retrieval to agents: 5 takeaways on production architecture for AI agents

Agentic AI is evolving quickly: Customers are no longer evaluating platforms based on traditional search benchmarks or metrics. Instead, they’re asking, “Can this solution be the retrieval and context layer my agents can trust?”

While the fundamental challenges of retrieving any data at low latency and at scale aren’t new, what’s changed is who’s consuming that information. In the past, human users were reading search results. Now, autonomous agents operate in reasoning loops, where every retrieval step informs the next.

This evolution is accelerating quickly. Model Context Protocol (MCP) has become the de facto standard for connecting agents to enterprise data and tools, and protocols like Agent2Agent (A2A) are doing the same for A2A collaboration. For many teams, exposing your data through a standard agent interface is now a fundamental architectural decision, much like building an index was a decade ago. Precision, latency, and data freshness are no longer "nice-to-have" optimizations; they're critical infrastructure.

Before enterprises can fully rely on agents to act, they need to trust the answers those agents produce. Right now, that trust gap is wide.

IDC’s research quantified it clearly: Only 12% of companies say they’re always confident in the factual accuracy of answers from their primary discovery tools. And as Amy pointed out during the webinar, this lack of trust is directly slowing agentic AI deployments. Over 50% of companies are still struggling to move from early stage trials to full production, largely because they don’t trust their agents’ outputs.

This trust challenge becomes even more critical when agents operate in reasoning loops. A single retrieval step that returns outdated, incomplete, or irrelevant data doesn’t just result in one bad answer; it compounds. The agent builds decisions on that flawed foundation, and the downstream impact escalates quickly.

I summed it up during the webinar: Your retrieval layer needs to deliver up-to-the-minute data. When it doesn’t, the agent’s decisions degrade fast. This is why real-time data ingestion is no longer optional.

If accuracy is the goal, then context engineering is how you achieve it.

Humans searching for information carry domain knowledge. They understand the context of what they’re looking for, but agents don’t. That context has to be explicitly provided at the right level of granularity and at the right moment in the reasoning loop.

This is where things get complex. As I explained during the webinar, you need to manage everything: the right chunks from a document, the right metadata from various systems, and the right data delivered to the agent at the right time. If the context isn’t precise, the agent starts to hallucinate, and the outputs become unreliable.

This phenomenon now has a name: context rot. Context rot is the degradation of an agent's reasoning performance caused by an oversaturation of irrelevant information, which exhausts the model's attention budget and compromises the accuracy of its outputs. Bigger context windows don't fix this; they just move the failure point. That's why context engineering has effectively succeeded prompt engineering as the core discipline for agent quality.

Context engineering spans retrieval, relevance, reranking, chunking strategies, metadata management, and more. It’s not glamorous work, but it’s where agent quality is won or lost. And it’s where teams run into trouble when they rely on fragmented point solutions.

So, what should leaders focus on when evaluating platforms for agentic AI? Amy and I outlined several key criteria during the webinar:

• Cost efficiency at scale: Token economics make or break agent ROI. Platforms that deliver only the most relevant context to the model rather than forcing you to pay for bloated context windows on every call, resulting from skill and tool sprawl, keep costs predictable as usage grows.

• AI-ready data: Managing structured, unstructured, and metadata at scale is the foundation for building reliable agentic systems.

• Speed from proof of concept to production: Scaling from concept to production is where most teams struggle. Platforms that shorten this gap are critical.

• Trust and security: Audit logs, compliance, and secure enterprise data access are non-negotiable.

• Roadmap alignment: Invest in a partner whose vision aligns with your future needs.

• Developer enablement: Empower developers to leverage a search and retrieval-based context engineering platform to reduce token costs and improve agent accuracy for production use cases.

As I said during the webinar, we’re moving from the aspiration phase of agentic AI to the production architecture phase. Decisions made now will define how fast you can build future use cases and how well those systems scale. If you want to see how Elastic handles retrieval, context engineering, and observability in a unified platform, check out our on-demand webinar.