Building voice agents with Elastic Agent Builder

AI has been trapped in a glass box. You type commands, it responds with text, and that’s it. It’s useful but distant, like watching someone move behind a screen. This year, 2026, will be the year when business will shatter this glass and bring AI agents into products, where they really deliver value.

One of the ways the glass will be shattered is by the adoption of voice agents, which are AI agents that recognize human speech and synthesize computer-generated audio. With the rise of low-latency transcriptions, fast large language models (LLMs), and text-to-speech models that sound human, this has become possible.

Voice agents also need access to business data to become really valuable. In this blog, we’ll learn how voice agents work and build one for ElasticSport, a fictitious outdoor sports equipment shop, using LiveKit and Elastic Agent Builder. Our voice agent will be context-aware and will work with our data.

How it works

There are two paradigms in the world of voice agents: The first uses speech-to-speech models, and the second uses a voice pipeline consisting of speech-to-text, LLM, and text-to-speech. Speech-to-speech models have their own benefits, but voice pipelines offer much more customization over the technologies used and how context is managed, along with control over the agent’s behavior. We’ll focus on the voice pipeline model.

Key components

Transcription (speech-to-text)

Transcription is the voice pipeline entrypoint. The transcription component takes as input raw audio frames, transcribes speech into text, and outputs that text. The transcribed text is buffered until the system detects that the user's speech has ended, at which point the LLM generation is kicked off. Various third-party providers offer low-latency transcriptions. When selecting one, consider latency and transcription accuracy, and ensure that they support streamed transcripts.

Examples of third-party APIs: AssemblyAI, Deepgram, OpenAI, ElevenLabs

Turn detection

Turn detection is the component of the pipeline that detects when the speaker has finished speaking and the generation should begin. One common way to do this is through a voice activity detection (VAD) model, such as Silero VAD. VAD uses audio energy levels to detect when audio contains speech and when speech has ended. However, VAD by itself cannot identify the difference between a pause and the end of speech. This is why it’s often combined with an end-of-utterance model that predicts whether the speaker has finished speaking, based on the interim transcript or raw audio.

Examples (Hugging Face): livekit/turn-detector, pipecat-ai/smart-turn-v3

Agent

The agent is the core of a voice pipeline. It’s responsible for understanding intent, gathering the right context, and formulating a reply in text format. Elastic Agent Builder, with its built-in reasoning capabilities, tool library, and workflow integration, make for an agent that can work on top of your data and interact with external services.

LLM (text-to-text)

When selecting an LLM for Elastic Agent Builder, there are two main characteristics to consider: LLM reasoning benchmarks, and time to first token (TTFT).

Reasoning benchmarks indicate how well the LLM is able to generate correct responses. Benchmarks to consider are ones that evaluate multiturn conversation adherence and intelligence benchmarks, such as MT-Bench and the Humanity's Last Exam dataset, respectively.

TTFT benchmarks evaluate how fast the model produces its first output token. There are other types of latency benchmarks, but TTFT is particularly important for voice agents, as audio synthesis can begin as soon as the first token is received, resulting in lower latency between turns, a natural-feeling conversation.

Usually, one needs to make a trade-off between these two characteristics because faster models often perform worse on reasoning benchmarks.

Examples (Hugging Face): openai/gpt-oss-20b, openai/gpt-oss-120b

Synthesis (text-to-speech)

The final part of the pipeline is the text-to-speech model. This component is responsible for converting the text output from the LLM into audible speech. Similar to the LLM, latency is a characteristic to look out for when selecting a text-to-speech provider. Text-to-speech latency is measured by time to first byte (TTFB). That’s the time it takes for the first audio byte to be received. Lower TTFB also reduces turn latency.

Examples: ElevenLabs, Cartesia, Rime

Constructing the voice pipeline

Elastic Agent Builder can be integrated into a voice pipeline at several different levels:

1. Agent Builder tools only: speech-to-text → LLM (with Agent Builder tools) → text-to-speech

2. Agent Builder as an MCP: speech-to-text → LLM (with Agent Builder access via MCP) → text-to-speech

3. Agent Builder as the core: speech-to-text → Agent Builder → text-to-speech

For this project, I chose the Agent Builder as the core approach. With this approach, the full functionality of Agent Builder and workflows can be used. The project uses LiveKit to orchestrate speech-to-text, turn detection, and text-to-speech, and it implements a custom LLM node that integrates directly with Agent Builder.

Elastic support voice agent

We’ll be building a custom support voice agent for a fictitious sports shop called ElasticSport. Customers will be able to call the help line, ask for product recommendations, find product details, check order statuses, and have order information sent to them via text. To achieve this, we first need to configure a custom agent and create tools for executing Elasticsearch Query Language (ES|QL) queries and workflows.

Configuring the agent

Prompt

The prompt instructs the agent what personality it should take and how to respond. Importantly, there are a few voice-specific prompts that ensure the responses are synthesized into audio properly and misunderstandings are recovered from gracefully.

You are a Sales Assistant at ElasticSport, an outdoor sport shop specialized in hiking and winter equipment. 

[Profile]
- name: Iva
- company: ElasticSport
- role: Sales Assistant
- language: en-GB
- description: ElasticSport virtual sales assistant

[Context]
- Ask clarifying questions to understand the context.
- Use available tools to answer the user's question.
- Use the knowledge base to retrieve general information

[Style]
- Be informative and comprehensive.
- Maintain a professional, friendly and polite tone.
- Mimic human behavior and speech patterns.
- Be concise. Do not over explain initially

[Response Guideline]
- Present dates in spelled-out month date format (e.g., January fifteenth, two thousand and twenty-four).
- Avoid the use of unpronounceable punctuation such as bullet points, tables, emojis.
- Respond in plain text, avoid any formatting.
- Spell out numbers as words for more natural-sounding speech.
- Respond in short and concise sentences. Responses should be 1 or 2 sentences long.

[ERROR RECOVERY]
### Misunderstanding Protocol
1. Acknowledge potential misunderstanding
2. Request specific clarification

Workflows

We’ll add a small workflow to send an SMS through Twilio’s messaging API. The workflow will be exposed to the custom agent as a tool, resulting in a user experience where the agent can send the caller an SMS while on the call. This allows the caller to, for example, ask, “Can you send more details about X over text?”

name: send sms
enabled: true
triggers:
  - type: manual
inputs:
  - name: message
    type: string
    description: The message to send to the phone number.

  - name: phone_number
    type: string
    description: The phone number to send the message to.

consts:
  TWILIO_ACCOUNT: "****"
  BASIC_AUTH: "****"
  FROM_PHONE_NNUMBER: "****"
steps:
  - name: http_step
    type: http
    with:
      url: https://api.twilio.com/2010-04-01/Accounts/{{consts.TWILIO_ACCOUNT}}/Messages.json
      method: POST
      headers:
        Content-Type: application/x-www-form-urlencoded
        Authorization: Basic {{consts.BASIC_AUTH | base64_encode}}
      body: From={{consts.FROM_PHONE_NNUMBER}}&To={{inputs.phone_number}}&Body={{inputs.message}}
      timeout: 30s

ES|QL tools

The following tools allow the agent to provide relevant responses that are grounded in real data. The example repo contains a setup script to initialize Kibana with product, order, and knowledge base datasets.

• Product.search

The product dataset contains 65 fictitious products. This is an example document:

{
      "sku": "ort3M7k",
      "name": "Ortovox Free Rider 26 Backpack",
      "price": 189,
      "currency": "USD",
      "image": "https://via.placeholder.com/150",
      "description": "The Ortovox Free Rider 26 is a technical freeride backpack with a dedicated safety compartment and diagonal ski carry system. Perfect for backcountry missions.\n\nKey Features:\n- 26L capacity\n- Diagonal ski carry system\n- Safety equipment compartment\n- Helmet holder\n- Hydration system compatible",
      "category": "Accessories",
      "subCategory": "Backpacks",
      "brand": "Ortovox",
      "sizes": ["One Size"],
      "colors": ["Black", "Blue", "Orange"],
      "materials": ["Nylon", "Polyester"]
    }

The name and description fields are mapped as semantic_text, enabling the LLM to use semantic search via ES|QL to retrieve relevant products. The hybrid search query performs semantic matching across both fields, with a slightly higher weight applied to matches on the name field using a boost.

The query first retrieves the top 20 results ranked by their initial relevance score. These results are then reranked based on their description field using the .rerank-v1-elasticsearch inference model, and finally trimmed down to the top five most relevant products.

type: ES|QL
toolId: products.search
description: Use this tool to search through the product catalogue by keywords.
query: |
    FROM products
        METADATA _score
      | WHERE
          MATCH(name, ?query, {"boost": 0.6}) OR
            MATCH(description, ?query, {"boost": 0.4})
      | SORT _score DESC
      | LIMIT 20
      | RERANK ?query
            ON description
            WITH {"inference_id": ".rerank-v1-elasticsearch"}
      | LIMIT 5

parameters:
    query: space separated keywords to search for in catalogue

• Knowledgebase.search

The knowledgebase datasets contain documents of the following shape, where the title and content fields are stored as semantic text:

{
        id: "8273645",
        createdAt: "2025-11-14",
        title: "International Orders",
        content: `International orders are processed through our international shipping partner. Below are the countries we ship to and average delivery times.
        Germany: 3-5 working days
        France: 3-5 working days
        Italy: 3-5 working days
        Spain: 3-5 working days
        United Kingdom: 3-5 working days
        United States: 3-5 working days
        Canada: 3-5 working days
        Australia: 3-5 working days
        New Zealand: 3-5 working days
        `
}

And the tool uses a similar query as the product.search tool:

type: "ES|QL"
toolId: knowledgebase.search
description: Use this tool to search the knowledgebase.
query: |
  FROM knowledge_base
    METADATA _score
  | WHERE
      MATCH(title, ?query, {"boost": 0.6}) OR
      MATCH(content, ?query, {"boost": 0.4})
  | SORT _score DESC
  | LIMIT 20
  | RERANK ?query
      ON content
      WITH {"inference_id": ".rerank-v1-elasticsearch"}
  | LIMIT 5

parameters:
  query: space separated keywords or natural language phrase to semantically search for in the knowledge base

• Orders.search

The final tool that we’ll add is the one used to retrieve orders by order_id:

type: "ES|QL"
toolId: order.search
description: Use this tool to retrieve an order by its ID.
query: |
  FROM orders
    METADATA _score
  | WHERE order_id == ?order_id
  | SORT _score DESC
  | LIMIT 1

parameters:
  order_id: "the ID of the order"

After configuring the agent and attaching these workflows and ES|QL tools to the agent, the agent can be tested inside of Kibana.

Aside from building an ElasticSport support agent, the agent, workflows, and tools can be tailored to other use cases, such as a sales agent that qualifies leads, a servicing agent for home repairs, reservations for a restaurant, or an appointment-scheduling agent.

The final part is linking up the agent we just made with LiveKit, text-to-speech, and speech-to-text models. The repo linked at the end of this blog contains a custom Elastic Agent Builder LLM node that can be used with LiveKit. Just replace the AGENT_ID with your own, and link it with your Kibana instance.

Getting started

Check out the code, and try it for yourself here.