This guide explains the key considerations for full-stack monitoring AI web agents, exploring both best practices and practical examples using OpenLit, OpenTelemetry and Elastic. Specifically we'll cover monitoring an example web travel planner located in this example repo.

Why is AI agent observability different?

The aim of traditional monitoring is to detect and alert on failures, performance issues, inefficiencies and resource bottlenecks. Monitoring AI agents still adheres to this common goal, but there are several differences that must be considered:

• AI models are probabilistic, meaning that the same input can lead to different outputs. This makes it hard to define and monitor success based on a single correct answer.
• AI systems can appear to function correctly on the surface, but their outputs may be suspect, incorrect, or biased without a way to immediately detect it. Telemetry must therefore be able to capture hidden capabilities such as tool call executions for SREs to scrutinize.
• The dynamic and evolving nature of LLMs can mean that their behavior can change dramatically between updates and versions due to changes in data, embeddings, or prompts. This means monitoring and pre-production evaluation when upgrading is vitally important for performance continuity.
• Models are black boxes. For this reason it's often difficult to understand why an AI made a particular decision. This makes troubleshooting harder compared to systems with clear, explicit logic.
• Beyond traditional metrics, AI output must be monitored for issues like hallucinations (generating false information), toxicity, and bias, which can damage user trust and lead to reputational harm.
• Contextual performance of an AI system can vary greatly depending on the context, including user interaction. Capturing user prompts and telemetry helps establish a complete picture of system performance.
• From a security perspective, AI agents can be vulnerable to adversarial attacks including data poisoning and obfuscation. Monitoring for unusual behavioral patterns and prompts is crucial to detect and mitigate these threats.

For these reasons the metrics and tracing that SREs capture and investigate will differ.

AI agent monitoring in practice

Let's apply these concepts by instrumenting an actual AI agent and capturing telemetry. Here we shall be using the TypeScript SDK of OpenLit, an open-source library that generates OpenTelemetry signals from LLM interactions in JavaScript applications. Specifically we shall instrument a simple web travel planner agent, available here that uses LLMs to generate travel recommendations based on user prompts and information from various tools. OpenLit works well for this type of project due to its TypeScript SDK and built in capabilities for capturing LLM interactions, tool calls, and generating evaluation and guardrail metrics.

The architecture diagram shows the key components:

The concepts and best practices discussed in this article can be applied to any AI agent regardless of the specific monitoring tools used. Many vendors have AI monitoring capabilities. Alternative open source technologies are also available for agentic monitoring, including LangSmith, OpenLLMetry, or indeed manual instrumentation using OpenTelemetry SDKs and the AI semantic conventions.

Prerequisites

This project requires that the following prerequisites are met:

• Active Elastic cluster (Cloud, Serverless or self-managed)
• OpenAI, Azure OpenAI, or compatible LLM provider API key
• Node.js 18+ with npm or yarn
• OTLP-compatible endpoint (Elastic Managed OTLP endpoint or OTel collector)

The following environment variables should be set:

• OTEL_ENDPOINT: Your Elastic OTLP endpoint or OTel collector URL
• OPENAI_API_KEY: API key for the evaluation/guardrail LLM
• OPENAI_ENDPOINT: Optional custom base URL for OpenAI-compatible providers

Basic instrumentation

Often DevOps engineers and SREs start with automatic instrumentation to obtain basic telemetry. This is possible with the OpenLit Python SDK. However with TypeScript we manually have to add our configuration to the AI entrypoint (here api/chat/route.ts).

First we install the dependency using our favourite package manager:

npm install openlit

Then we add the OpenLit configuration to our entrypoint:

import openlit from "openlit";

// Other imports omitted for brevity 

// Allow streaming responses up to 30 seconds to address typically longer responses from LLMs
export const maxDuration = 30;

openlit.init({
  applicationName: "ai-travel-agent", // akin to OTEL resource name
  environment: "development",
  otlpEndpoint: process.env.OTEL_ENDPOINT, // OTLP compatible endpoint (Elastic ingest or OTel collector)
  disableBatch: true, // batching disabled for demo purposes - not recommended for production use
});

// Post request handler
export async function POST(req: Request) {
   // AI logic omitted for brevity - see full code in repo
}

This instrumentation will automatically generate OpenTelemetry traces for all LLM interactions, including tool calls, and send them to the specified OTLP endpoint. Note that for production rather than demo usage, environment should be set to production and batching should not be disabled to ensure optimal network usage and protect the OTel backend.

Let's discuss the key telemetry signals that are generated in subsequent sections.

Inputs

The first rule of debugging AI agents is simple: if you don't capture the prompt, you can't reproduce the problem. Unlike traditional applications where inputs are predictable request parameters, AI agents consume free-form user prompts that can trigger wildly different behaviors based on subtle phrasing changes. OpenLit automatically captures system prompts and all user messages as structured attributes on your traces, giving you the exact input that caused your agent to hallucinate, loop, or fail.

The full conversation needs to be available to SREs to understand the context of failures and performance issues and identify patterns in the inputs that may be causing issues. However these inputs are also useful for improving agent behavior, and can be used as testing messages to evaluate model performance and test enhancements once sanitized for identifiable attributes such as PII.

Beyond prompts, we still need comprehensive logging, specifically capturing full stack traces emitted by our applications. This is crucial for diagnosing issues that may arise from the underlying infrastructure or codebase, rather than the AI model itself. For example, the below error sent to Elastic shows a simple fetch error. We must not forget that traditional errors can still occur in AI applications, and capturing them is essential.

Tracing

Traces are essential for understanding the flow of requests through your AI agent, especially when it comes to tool calls. Generally traces are a hierarchy of spans, which themselves are a single, timed unit representing a specific operation, such as a database query or an HTTP handler. In AI systems they also represent tool calls made by the LLM, along with the API calls and data retrieval steps performed within the tool execution.

Visualizing tool calling patterns is important in validating pre-production systems as well as monitoring production systems for several reasons:

1. It helps us evaluate the tool calling capabilities of different models. LLMs make the choice of which tools to use based on the user prompt, system instructions and the tool metadata (such as name and description). By visualizing the tool calling patterns we can understand whether the model is correctly interpreting the tool metadata and making appropriate calls based on the prompt.
2. It allows us to identify inefficient or erroneous tool calling patterns. For example, if we see a pattern of repeated calls to the same tool with similar inputs, it may indicate that the model is stuck in a loop or not effectively utilizing the tools. Or if a single tool is being called where we would expect multiple tools to be called, it may indicate that the model is not correctly connecting the prompt or system instructions require said tool(s).
3. Commonly occurring tool-calling patterns can also be identified to optimize the available tools. For example, if the location and weather tools are frequently called together, it may make sense to combine them into a single tool that provides both pieces of information in one call.

With the above configuration, we can see the traces for each tool call, as illustrated in the below example:

Metrics

While tracing is essential for understanding the flow of requests and tool calls, metrics are crucial for monitoring the overall health and performance of your system, agentic or not. When considering metrics many think solely of cost and total token usage. While both are important, they are not the only metrics that matter.

Through the example above OpenLit automatically generates key metrics that can be used to evaluate agent performance, such as request latency, error rates, cost and token usage, which can be visualized in Elastic to identify trends and anomalies. Token usage specifically can be split by input, output and reasoning token counts, helping us identify optimization opportunities at key stages in the generation cycle. For example, an increase in input token counts may indicate a significant increase in context length that can be optimized via prompt and context engineering techniques.

It's also important to make sure that metrics capture traditional performance metrics such as CPU, memory and request counts to caches, traditional databases and vector databases. This helps us identify whether performance issues are being caused by the AI model itself or by underlying infrastructure problems. Alerting for spikes in key measures such as large token usage increases or request volumes would also be considered best practice.

Evaluation

AI evaluation refers to the process of assessing the performance of AI models and the quality of the responses they generate. This involves monitoring various metrics and signals to ensure that the AI system is functioning as intended, providing accurate outputs, and not exhibiting undesirable behaviors such as hallucinations, toxic behavior or bias. While evaluation is considered as a pre-production activity to test and validate an agentic system, it's also important to continue monitoring these signals in production to identify issues over time.

There are several different evaluation methodologies that we can use. OpenLit makes use of AI as a Judge. This involves using an LLM to evaluate the quality of the output generated by another LLM based on a set of criteria. An example of traditional evaluation is depicted below:

When considering evaluation from a monitoring viewpoint, it's important to identify hallucinations, bias, toxicity and potential injection issues in production. Hallucinations, bias and toxic responses expose us to reputational risk and loss of user trust. Out of the box, OpenLit identifies the following issues, calculates a score and provides an explanation of the issue:

These issues can be identified using the below code:

import openlit from "openlit";

// Other imports omitted for brevity

// Allow streaming responses up to 30 seconds to address typically longer responses from LLMs
export const maxDuration = 30;

// Tools and Azure configuration omitted for brevity

openlit.init({
  applicationName: "ai-travel-agent",
  environment: "development",
  otlpEndpoint: process.env.OTEL_ENDPOINT,
  disableBatch: true,
});

// Choose one of the following approaches:
// Option 1: enable all available evaluations
const evalsAll = openlit.evals.All({
  provider: "openai",
  collectMetrics: true, // Ensures evaluations are exported to Elastic
  apiKey: process.env.OPENAI_API_KEY,
  baseUrl: process.env.OPENAI_ENDPOINT
});

// Option 2: enable specific evaluations with custom configuration
const evalsHallucination = openlit.evals.Hallucination({
  provider: "openai",
  collectMetrics: true,
  apiKey: process.env.OPENAI_API_KEY,
  baseUrl: process.env.OPENAI_ENDPOINT
});

// Post request handler
export async function POST(req: Request) {
  const { messages, id } = await req.json();

  try {
    const convertedMessages = await convertToModelMessages(messages);
    const prompt = `You are a helpful assistant that returns travel itineraries...`;

    const result = streamText({
      model: azure("gpt-4o"),
      system: prompt,
      messages: convertedMessages,
      stopWhen: stepCountIs(2),
      tools,
      experimental_telemetry: { isEnabled: true },
      onFinish: async ({ text, steps }) => {
        // Concatenate tool results and content as full evaluation context
        const toolResults = steps.flatMap((step) => {
          return step.content
            .filter((content) => content.type == "tool-result")
            .map((c) => {
              return JSON.stringify(c.output);
            });
        });

        // Measure evaluation
        const evalResults = await evalsAll.measure({
          prompt: prompt,
          contexts: convertedMessages
            .map((m) => {
              return m.content.toString();
            })
            .concat(toolResults),
          text: text,
        });
        console.log(`Evals results: ${evalResults}`);
      },
    });

    // Return data stream to allow the useChat hook to handle the results as they are streamed through for a better user experience
    return result.toUIMessageStreamResponse();
  } catch (e) {
    console.error(e);
    return new NextResponse(
      "Unable to generate a plan. Please try again later!"
    );
  }
}

By using the collectMetrics option, the evaluation results are automatically exported as metrics to Elastic, allowing us to monitor the quality of our AI agent's outputs over time and identify trends or issues that may arise in production. The evaluation results can also be used to trigger alerts or automated responses if certain thresholds or SLOs are breached, such as a high evaluation score sustained for several minutes, increased number of hallucinations detected over time, or triggering of a toxic result.

The advantage of using LLMs to evaluate results is that they help identify issues and inaccuracies quickly compared to leveraging manual quality checks. However, this methodology does have limitations, specifically:

1. Increased cost and latency due to the additional requests to an LLM to evaluate results. This can be mitigated by using a smaller, cheaper model for evaluation, by only evaluating a sample of responses, or using cached responses to reduce the number of LLM calls for similar questions.
2. LLM evaluations are prone to biases. Specifically, Zheng et al. cite in their 2023 paper that LLM evaluations are subject to:

• Positional bias, where an LLM prefers responses where the answer is located in a specific position in the response, and may miss correct answers located elsewhere in the reply.
• Self-enhancement bias, where LLMs show preference for responses they have generated compared to other models. This can be a consideration if you wish to use cheaper, or self-hosted models for evaluation.
• Verbosity bias, where they prefer more expansive responses over succinct replies.

Guardrail monitoring

In addition to assessing the quality of responses, we must also be monitoring for dangerous or irrelevant responses that could be harmful to users. The quality of in-built protections within models are patchy and model dependent. Research from several research papers, including Anthropic in their 2025 agentic misalignment paper, show that in some cases models can resort to malicious behaviors and the model bypassing company policies and moral expectations.

Guardrail detection in monitoring tools allows us to identify risky responses generated by AI agents, such as generating harmful content, engaging in inappropriate interactions, or performing injection actions to try and hack into systems or elicit confidential information. Using OpenLit as our example, we are able to monitor for breaches of the following guardrail types:

These shields can be set up using OpenLit as per the below code:

import openlit from "openlit";

// Other imports omitted for brevity

// Allow streaming responses up to 30 seconds to address typically longer responses from LLMs
export const maxDuration = 30;

// Tools and Azure configuration omitted for brevity

openlit.init({
  applicationName: "ai-travel-agent",
  environment: "development",
  otlpEndpoint: process.env.OTEL_ENDPOINT,
  disableBatch: true,
});

// Choose one of the following approaches:
// Option 1: enable all available guardrails
const guardsAll = openlit.guard.All({
  provider: "openai",
  collectMetrics: true,
  apiKey: process.env.OPENAI_API_KEY,
  baseUrl: process.env.OPENAI_ENDPOINT,
  validTopics: ["travel", "culture"],
  invalidTopics: ["finance", "software engineering"],
});

// Option 2: enables specific guardrail types (for example, prompt injection detection)
const guardsPromptInjection = openlit.guard.PromptInjection({
  provider: "openai",
  collectMetrics: true, // Ensures guardrail breaches are exported to Elastic
  apiKey: process.env.OPENAI_API_KEY,
  baseUrl: process.env.OPENAI_ENDPOINT
});

// Post request handler
export async function POST(req: Request) {
  const { messages, id } = await req.json();

  try {
    const convertedMessages = await convertToModelMessages(messages);
    const prompt = `You are a helpful assistant that returns travel itineraries...`;

    const result = streamText({
      model: azure("gpt-4o"),
      system: prompt,
      messages: convertedMessages,
      stopWhen: stepCountIs(2),
      tools,
      experimental_telemetry: { isEnabled: true },
      onFinish: async ({ text, steps }) => {
        const guardrailResult = await guardsAll.detect(text);
        console.log(`Guardrail results: ${guardrailResult}`);
      },
    });

    // Return data stream to allow the useChat hook to handle the results as they are streamed through for a better user experience
    return result.toUIMessageStreamResponse();
  } catch (e) {
    console.error(e);
    return new NextResponse(
      "Unable to generate a plan. Please try again later!"
    );
  }
}

In the event of a guardrail breach, metrics containing detail of the breach are sent to Elastic, similar to the following:

Of course we can leverage dashboards to visualize trends of guardrail breaches, including metrics such as volumes by category, as shown in the below example (with the corresponding NDJSON available here):

We can also perform action on these breaches to notify relevant teams via alerts triggered via the available alerting tools. These should be triggered based on the severity of the detected issue, as well as the classification, for example mentions of violence, illegal themes, or injection attacks may trigger immediately compared to minor inaccuracies. The guardrail breaches can also be used to trigger automated responses, such as blocking the response from being sent to the user, or providing a warning message to the user that their request has been flagged for review, triggering a human-in-the-loop response for the relevant teams.

Conclusion

AI agents are becoming more autonomous, more powerful, and more unpredictable. For this reason, it's important to introduce monitoring telemetry as early as possible in the development process and in organizational cultures. This article helps you understand how monitoring AI agents is different and how to do it using OpenLit to generate OpenTelemetry signals to send to Elastic. Check out the code here and start monitoring your AI agents in production.

Developer resources:

• Observing AI Agents Example
• OpenLit SDK Documentation
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena | Zheng et al. 2023
• Agentic Misalignment: How LLMs could be insider threats | Anthropic

filebeat.inputs:
  - type: filestream
    id: my-logging-app
    paths:
      - /var/tmp/other.log
      - /var/log/*.log
processors:
  - drop_event:
      when:
        and:
          - equals:
            url.scheme: http
          - equals:
            url.path: /profile

filebeat.inputs:
  - type: filestream
    id: my-logging-app
    paths:
      - /var/tmp/other.log
      - /var/log/*.log
processors:
  - drop_fields:
      when:
        and:
          - equals:
            url.scheme: http
          - equals:
            http.response.status_code: 200
        fields: ["event.message"]
        ignore_missing: false

input {
  file {
    id => "my-logging-app"
    path => [ "/var/tmp/other.log", "/var/log/*.log" ]
  }
}
filter {
  if [url.scheme] == "http" && [url.path] == "/profile" {
    drop {
      percentage => 80
    }
  }
}
output {
  elasticsearch {
        hosts => "https://my-elasticsearch:9200"
        data_stream => "true"
    }
}

# Input configuration omitted
filter {
  if [url.scheme] == "http" && [http.response.status_code] == 200 {
    drop {
      percentage => 80
    }
    mutate {
      remove_field: [ "event.message" ]
    }
  }
}
# Output configuration omitted

PUT _ingest/pipeline/my-logging-app-pipeline
{
  "description": "Event and field dropping for my-logging-app",
  "processors": [
    {
      "drop": {
        "description" : "Drop event",
        "if": "ctx?.url?.scheme == 'http' && ctx?.url?.path == '/profile'",
        "ignore_failure": true
      }
    },
    {
      "remove": {
        "description" : "Drop field",
        "field" : "event.message",
        "if": "ctx?.url?.scheme == 'http' && ctx?.http?.response?.status_code == 200",
        "ignore_failure": false
      }
    }
  ]
}

PUT _ingest/pipeline/my-logging-app-pipeline
{
  "description": "Event and field dropping for my-logging-app with failures",
  "processors": [
    {
      "drop": {
        "description" : "Drop event",
        "if": "ctx?.url?.scheme == 'http' && ctx?.url?.path == '/profile'",
        "ignore_failure": true
      }
    },
    {
      "remove": {
        "description" : "Drop field",
        "field" : "event.message",
        "if": "ctx?.url?.scheme == 'http' && ctx?.http?.response?.status_code == 200",
        "ignore_failure": false
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "description": "Set 'ingest.failure.message'",
        "field": "ingest.failure.message",
        "value": "Ingestion issue"
        }
      }
  ]
}

receivers:
  filelog:
    include: [/var/tmp/other.log, /var/log/*.log]
processors:
  filter/denylist:
    error_mode: ignore
    logs:
      log_record:
        - 'url.scheme == "info"'
        - 'url.path == "/profile"'
        - "http.response.status_code == 200"
  attributes/errors:
    actions:
      - key: error.message
        action: delete
  memory_limiter:
    check_interval: 1s
    limit_mib: 2000
  batch:
exporters:
  # Exporters configuration omitted
service:
  pipelines:
    # Pipelines configuration omitted

Outside of cultural challenges, one of the most common reasons for this disconnect is using different tools to achieve similar goals — case in point, end-to-end (e2e) testing versus synthetic monitoring.

This blog shares an overview of these techniques. Using the example repository carlyrichmond/synthetics-replicator, we’ll also show how Playwright, @elastic/synthetics, and GitHub Actions can combine forces with Elastic Synthetics and the recorder to unite development and SRE teams in validating and monitoring the user experience for a simple web application hosted on a provider such as Netlify.

Elastic recently introduced synthetics monitoring, and as highlighted in our prior blog, it can replace e2e tests altogether. Uniting around a single tool to validate the user workflow early provides a common language to recreate user issues to validate fixes against.

Synthetics Monitoring versus e2e tests

If development and operations tools are at war, it’s difficult to unify their different cultures together. Considering the definitions of these approaches shows that they in fact aim to achieve the same objective.

e2e tests are a suite of tests that recreate the user path, including clicks, user text entry, and navigations. Although many argue it’s about testing the integration of the layers of a software application, it’s the user workflow that e2e tests emulate. Meanwhile, Synthetic Monitoring, specifically a subset known as browser monitoring, is an application performance monitoring practice that emulates the user path through an application.

Both these techniques emulate the user path. If we use tooling that crosses the developer and operational divide, we can work together to build tests that can also provide production monitoring in our web applications.

Creating user journeys

When a new user workflow, or set of features that accomplish a key goal, is under development in our application, developers can use @elastic/synthetics to create user journeys. The initial project scaffolding can be generated using the init utility once installed, as in the below example. Note that Node.js must be installed prior to using this utility.

npm install -g @elastic/synthetics
npx @elastic/synthetics init synthetics-replicator-tests

Before commencing the wizard, make sure you have your Elastic cluster information and the Elastic Synthetics integration set on your cluster. You will need:

1. Monitor Management must be enabled within the Elastic Synthetics app as per the prerequisites in the documentation getting started.
2. The Elastic Cloud cluster Cloud ID if using Elastic Cloud. Alternatively, if you are using on-prem hosting you need to enter your Kibana endpoint.
3. An API key generated from your cluster. There is a shortcut in the Synthetics application Settings to generate this key under the Project API Keys tab, as shown in the documentation.

This wizard will take you through and generate a sample project containing configuration and example monitor journeys, with a structure similar to the below:

For web developers, most of the elements such as the README and package.json and lock files will be familiar. The main configuration for your monitors is available in synthetics.config.tsas shown below. This configuration can be amended to include production and development-specific configuration. This is essential for combining forces and reusing the same monitors for e2e tests and allowing for any journeys to be used as e2e tests and production monitors. Although not in this example, details of private locations can be included if you would prefer to monitor from your own dedicated Elastic instance rather than from Elastic infrastructure.

import type { SyntheticsConfig } from "@elastic/synthetics";

export default (env) => {
  const config: SyntheticsConfig = {
    params: {
      url: "http://localhost:5173",
    },
    playwrightOptions: {
      ignoreHTTPSErrors: false,
    },
    /**
     * Configure global monitor settings
     */
    monitor: {
      schedule: 10,
      locations: ["united_kingdom"],
      privateLocations: [],
    },
    /**
     * Project monitors settings
     */
    project: {
      id: "synthetics-replicator-tests",
      url: "https://elastic-deployment:port",
      space: "default",
    },
  };
  if (env === "production") {
    config.params = { url: "https://synthetics-replicator.netlify.app/" };
  }
  return config;
};

Writing your first journey

Although the above configuration applies to all monitors in the project, it can be overridden for a given test.

import { journey, step, monitor, expect, before } from "@elastic/synthetics";

journey("Replicator Order Journey", ({ page, params }) => {
  // Only relevant for the push command to create
  // monitors in Kibana
  monitor.use({
    id: "synthetics-replicator-monitor",
    schedule: 10,
  });

  // journey steps go here
});

The @elastic/synthetics wrapper exposes many standard test methods such as the before and after constructs that allow for setup and tear down of typical properties in the tests, as well as support for many common assertion helper methods. A full list of supported expect methods are listed in the documentation. The Playwright page object is also exposed, which enables us to perform all the expected activities provided in the API such as locating page elements and simulating user events such as clicks that are depicted in the below example.

import { journey, step, monitor, expect, before } from "@elastic/synthetics";

journey("Replicator Order Journey", ({ page, params }) => {
  // monitor configuration goes here

  before(async () => {
    await page.goto(params.url);
  });

  step("assert home page loads", async () => {
    const header = await page.locator("h1");
    expect(await header.textContent()).toBe("Replicatr");
  });

  step("assert move to order page", async () => {
    const orderButton = await page.locator("data-testid=order-button");
    await orderButton.click();

    const url = page.url();
    expect(url).toContain("/order");

    const menuTiles = await page.locator("data-testid=menu-item-card");
    expect(await menuTiles.count()).toBeGreaterThan(2);
  });

  // other steps go here
});

As you can see in the above example, it also exposes the journey and step constructs. This construct mirrors the behavior-driven development (BDD) practice of showing the user journey through the application in tests.

Developers are able to execute the tests against a locally running application as part of their feature development to see successful and failed steps in the user workflow. In the below example, the local server startup command is outlined in blue at the top. The monitor execution command is presented in red further down.

As you can see from the green ticks next to each journey step, each of our tests pass. Woo!

Gating your CI pipelines

It’s important to use the execution of the monitors within your CI pipeline as a gate for merging code changes and uploading the new version of your monitors. Each of the jobs in our GitHub Actions workflow will be discussed in this and the subsequent section.

The test job spins up a test instance and runs our user journeys to validate our changes, as illustrated below. This step should run for pull requests to validate developer changes, as well as on push.

jobs:
  test:
    env:
      NODE_ENV: development
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 18
      - run: npm install
      - run: npm start &
      - run: "npm install @elastic/synthetics && SYNTHETICS_JUNIT_FILE='junit-synthetics.xml' npx @elastic/synthetics . --reporter=junit"
        working-directory: ./apps/synthetics-replicator-tests/journeys
      - name: Publish Unit Test Results
        uses: EnricoMi/publish-unit-test-result-action@v2
        if: always()
        with:
          junit_files: "**/junit-*.xml"
          check_name: Elastic Synthetics Tests

Note that, unlike the journey execution on our local machine, we make use of the --reporter=junit option when executing npx @elastic/synthetics to provide visibility of our passing, or sadly sometimes failing, journeys to the CI job.

Automatically upload monitors

To ensure the latest monitors are available in Elastic Uptime, it’s advisable to push the monitors programmatically as part of the CI workflow such as the example task below does. Our workflow has a second job push, shown below, which is dependent on the successful execution of our test job that uploads your monitors to your cluster. Note that this job is configured in our workflow to run on push to ensure changes have been validated rather than just raised within a pull request.

jobs:
  test: …
  push:
    env:
      NODE_ENV: production
      SYNTHETICS_API_KEY: ${{ secrets.SYNTHETICS_API_KEY }}
    needs: test
    defaults:
      run:
        working-directory: ./apps/synthetics-replicator-tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 18
      - run: npm install
      - run: npm run push

The @elastic/synthetics init wizard generates a push command for you when you create your project that can be triggered from the project folder. This is shown below through the steps and working_directory configuration. The push command requires the API key from your Elastic cluster, which should be stored as a secret within a trusted vault and referenced via a workflow environment variable. It is also vital that monitors pass ahead of pushing the updated monitor configuration to your Elastic Synthetics instance to prevent breaking your production monitoring. Unlike e2e tests running against a testing environment, broken monitors impact SRE activities and therefore any changes need to be validated. For that reason, applying a dependency to your test step via the needs option is recommended.

Monitoring using Elastic Synthetics

Once monitors have been uploaded, they give a regular checkpoint to SRE teams as to whether the user workflow is functioning as intended — not just because they will run on a regular schedule as configured for the project and individual tests as shown previously, but also due to the ability to check the state of all monitor runs and execute them on demand.

The Monitors Overview tab gives us an immediate view of the status of all configured monitors, as well as the ability to run the monitor manually via the card ellipsis menu.

From the Monitor screen, we can also navigate to an overview of an individual monitor execution to investigate failures.

The other monitoring superpower SREs now have is the integration between these monitors to familiar tools SREs already use in scrutinizing the performance and availability of applications such as APM, metrics, and logs. The aptly named Investigate menu allows easy navigation while SREs are performing investigations into potential failures or bottlenecks.

There is also a balance between finding issues and being notified of potential problems automatically. SREs already familiar with setting rules and thresholds for notification of issues will be happy to know that this is also possible for browser monitors. The editing of an example rule is shown below.

The status of browser monitors can be configured not only to consider if any individual or collective monitors have been down several times, such as in the status check above, but also to gauge the overall availability by looking at the percentage of passed checks within a given time period. SREs are not only interested in reacting to issues in a traditional production management way — they want to improve the availability of applications, too.

Recording user workflows

The limitation of generating e2e tests through the development lifecycle is that sometimes teams miss things, and the prior toolset is geared toward development teams. Despite the best intentions to design an intuitive product using multi-discipline teams, users may use applications in unintended ways. Furthermore, the monitors written by developers will only cover those expected workflows and raise the alarm either when these monitors fail in production or when they start to behave differently if anomaly detection is applied to them.

When user issues arise, it’s useful to recreate that problem in the same format as our monitors. It’s also important to leverage the experience of SREs in generating user journeys, as they will consider failure cases intuitively where developers may struggle and focus on happy cases. However, not all SREs will have the experience or confidence to write these journeys using Playwright and @elastic/synthetics.

Enter the Elastic Synthetics Recorder! The above video gives a walkthrough of how it can be used to record the steps in a user journey and export them to a JavaScript file for inclusion in your monitor project. This is useful for feeding back into the development phase and testing developed fixes to solve the problem. This approach cannot be made unless we all combine forces to use these monitors together.

Try it out!

As of 8.8, @elastic/synthetics and the Elastic Synthetics app are generally available, and the trusty recorder is in beta. Share your experiences of bridging the developer and operations divide with Synthetic Monitoring via the Uptime category in the Community Discuss forums or via Slack.

Happy monitoring!

Originally published February 6, 2023; updated May 23, 2023.

1. Why and how to replace end-to-end tests with synthetic monitors
2. Uptime and Synthetic Monitoring
3. Scripting browser monitors
4. Use the Synthetics Recorder
5. Playwright
6. GitHub Actions

In this blog, we will discuss the current state of client instrumentation for the browser, along with an example showing how to instrument a simple JavaScript frontend using the OpenTelemetry browser instrumentation. Furthermore, we'll also share how the baggage propagators help us build a full picture of what is going on across the entire application by connecting backend traces with frontend signals. If you want to dive straight into the code, check out the repo here.

Application Overview

The application that we use for this blog is called OTel Record Store, a simple web application written with Svelte and JavaScript (albeit our implementation is compatible with other web frameworks), communicating with a Java backend. Both send telemetry signals to an Elastic backend.

Eagle-eyed readers will noticed that signals from our frontend pass through a proxy and collector. The proxy is required to ensure that the appropriate Cross-Origin headers are populated to allow the signals to pass into Elastic, as well as the traditional reasons such as security, privacy and access control:

events {}

http {

  server {

    listen 8123; 

    # Traces endpoint exposed as example, others available in code repo
    location /v1/traces {
      proxy_pass http://host.docker.internal:4318;
      # Apply CORS headers to ALL responses, including POST
      add_header 'Access-Control-Allow-Origin' 'http://localhost:4173' always;
      add_header 'Access-Control-Allow-Methods' 'POST, OPTIONS' always;
      add_header 'Access-Control-Allow-Headers' 'Content-Type' always;
      add_header 'Access-Control-Allow-Credentials' 'true' always;

      # Preflight requests receive a 204 No Content response
      if ($request_method = OPTIONS) {
        return 204;
      }
    }
  }
}

While collectors can also be used to add headers, we have left this example to perform traditional tasks such as routing and processing.

Prerequisites

This example requires an Elastic cluster, run either locally via start-local, via Elastic Cloud or Serverless. Here we use the Managed OLTP endpoint in Elastic Serverless. Any mechanism requires you to specify several key environment variables, listed in the .env-example file:

ELASTIC_ENDPOINT=https://my-elastic-endpoint:443
ELASTIC_API_KEY=my-api-key

Running the application

To run our example, follow the steps in the project README, summarized below:

# Terminal 1: backend service, proxy and collector
docker-compose build
docker-compose up

# Terminal 2: frontend and sample telemetry data
cd records-ui
npm install
npm run generate

Java Backend Instrumentation

We will not cover the specifics of instrumentation of Java services with EDOT as there is already a great guide to get started in the elastic-otel-java README. The example is here purely for showcasing propagation that is important for investigating UI issues. All you need to know is that we make use of automatic instrumentation, sending logs, metrics and traces via OpenTelemetry Protocol, or OTLP using the below environment variables:

OTEL_RESOURCE_ATTRIBUTES=service.version=1,deployment.environment=dev
OTEL_SERVICE_NAME=record-store-server-java
OTEL_EXPORTER_OTLP_ENDPOINT=$ELASTIC_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey ${ELASTIC_API_KEY}"
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp

The instrumentation is then initialized using the -javaagent option:

ENV JAVA_TOOL_OPTIONS="-javaagent:./elastic-otel-javaagent-1.2.1.jar"

Client Instrumentation

Now that we have established our prerequisites, let's dive into the instrumentation code for our simple web application. Although we'll cover the implementation in sections, the full solution is available here in frontend.tracer.ts.

State of OTel Client Instrumentation

At time of writing, the OpenTelemetry JavaScript SDK has stable support for metrics and traces, with logs currently under development and therefore subject to breaking changes as listed in their documentation:

What differs from many other SDKs is the note warning that client instrumentation for the browser is experimental and mostly unspecified. It is subject to breaking change, and many pieces such as plugin support for measuring Google Core Web Vitals are in progress as reflected in the Client Instrumentation SIG project board. In subsequent sections we'll show examples for signal capture, and also browser specific instrumentations including document load, user interaction and Core Web Vitals capture.

Resource Definition

When instrumenting web UIs, we need to establish our UI as an OpenTelemetry Resource. By definition, resources are entites that produce telemetry information. We want to see our UI as an entity in our system that interacts with other entities, which can be specified using the following code:

// Defines a Resource to include metadata like service.name, required by Elastic
import { resourceFromAttributes, detectResources } from '@opentelemetry/resources';

// Experimental detector for browser environment
import { browserDetector } from '@opentelemetry/opentelemetry-browser-detector';

// Provides standard semantic keys for attributes, like service.name
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';

const detectedResources = detectResources({ detectors: [browserDetector] });
let resource = resourceFromAttributes({
	[ATTR_SERVICE_NAME]: 'records-ui-web',
	'service.version': 1,
	'deployment.environment': 'dev'
});
resource = resource.merge(detectedResources);

A unique identifier for the service is required, and is common to all SDKs. What differs from other implementations is the inclusion of the browserDetector which, when merged with our defined resource attributes adds browser attributes such as platform, brands (e.g. Chrome versus Edge) and whether a mobile browser is being used:

Having this information on spans and errors is useful in diagnostic situations in identifying application and dependency compatibility issues with certain browsers (such as Internet Explorer from my time as an engineer 🤦).

Logs

Traditionally, frontend engineers rely on the DevTools console of their favourite browser to examine logs. With UI log messages only being accessible within your browser rather than forwarded to a file somewhere, which is the common pattern with backend services, we lose visibility of this resource when triaging user issues.

OpenTelemetry defines the concept of an exporter that allow us to send signals to a particular destination, such as logs.

// Get logger and severity constant imports
import { logs, SeverityNumber } from '@opentelemetry/api-logs';

// Provider and batch processor for sending logs
import { BatchLogRecordProcessor, LoggerProvider } from '@opentelemetry/sdk-logs';

// Export logs via OTLP
import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-http';

// Configure logging to send to the collector via nginx
const logExporter = new OTLPLogExporter({
	url: 'http://localhost:8123/v1/logs' // nginx proxy
});

const loggerProvider = new LoggerProvider({
	resource: resource, // see resource initialisation above
	processors: [new BatchLogRecordProcessor(logExporter)]
});

logs.setGlobalLoggerProvider(loggerProvider);

Once the provider has been initialized, we need to get a hold of the logger to send our traces to Elastic rather than using good ol' console.log('Help!'):

// Example gets logger and sends a message to Elastic
const logger = logs.getLogger('default', '1.0.0');
logger.emit({
	severityNumber: SeverityNumber.INFO,
	severityText: 'INFO',
	body: 'Logger initialized'
});

They will now be visible in Discover and the Logs views, allowing us to search for relevant outages as part of investigations and incidents:

Traces

The power of traces in diagnosing issues in the UI is in the visibility of not just what is going on within the web application, but seeing the connections and time taken to make calls to the labyrinth of services behind. To instrument a web-based application, we need to make use of the WebTraceProvider using the OTLPTraceExporter in a similar way to how exporters work for logs and metrics:

/* Packages for exporting traces */

// Import the WebTracerProvider, which is the core provider for browser-based tracing
import { WebTracerProvider } from '@opentelemetry/sdk-trace-web';

// BatchSpanProcessor forwards spans to the exporter in batches to prevent flooding
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';

// Import the OTLP HTTP exporter for sending traces to the collector over HTTP
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

// Configure the OTLP exporter to talk to the collector via nginx
const exporter = new OTLPTraceExporter({
	url: 'http://localhost:8123/v1/traces' // nginx proxy
});

// Instantiate the trace provider and inject the resource
const provider = new WebTracerProvider({
	resource: resource,
	spanProcessors: [
		// Send each completed span through the OTLP exporter
		new BatchSpanProcessor(exporter)
	]
});

Next we need to register our provider. One thing that's slightly different in the web world is how we configure propagation. Context propagation in OpenTelemetry refers to the concept of moving context between services and processes which, in our case, allows us to correlate the web signals with those of backend services. Often this is done automatically. As you will see from the below snippet, there are 3 concepts that help us with propagation:

// This context manager ensures span context is maintained across async boundaries in the browser
import { ZoneContextManager } from '@opentelemetry/context-zone';

// Context Propagation across signals
import {
	CompositePropagator,
	W3CBaggagePropagator,
	W3CTraceContextPropagator
} from '@opentelemetry/core';

// Provider instantiation code omitted

// Register the provider with propagation and set up the async context manager for spans
provider.register({
	contextManager: new ZoneContextManager(),
	propagator: new CompositePropagator({
		propagators: [new W3CBaggagePropagator(), new W3CTraceContextPropagator()]
	})
});

The first is the ZoneContextManager which propagates context such as spans and traces across asynchronous operations. Web developers will be familiar with zone.js, the framework used by many JS frameworks to provide an execution context that persists across async tasks.

Additionally, we have combined the W3CBaggagePropagator and W3CTraceContextPropagator using the CompositePropagator to ensure key value pair attributes are passed between signals as per the W3C specification defined here. In the case of the W3CTraceContextPropagator, it allows the propagation of the traceparent and tracestate HTTP headers as per the specification located here.

Auto Instrumentation

The simplest way to start instrumenting a web application is to register the web auto-instrumentations. At time of writing the documentation states that the following instrumentations can be configured via this approach:

1. @opentelemetry/instrumentation-document-load
2. @opentelemetry/instrumentation-fetch
3. @opentelemetry/instrumentation-user-interaction
4. @opentelemetry/instrumentation-xml-http-request

Configuration for each configuration can be passed as configuration to registerInstrumentations as shown in the below example configuring the fetch and XMLHTTPRequest instrumentations:

// Used to auto-register built-in instrumentations
import { registerInstrumentations } from '@opentelemetry/instrumentation';

// Import the auto-instrumentations for web, which includes common libraries, frameworks and document load
import { getWebAutoInstrumentations } from '@opentelemetry/auto-instrumentations-web';

// Enable automatic span generation for document load and user click interactions
registerInstrumentations({
  instrumentations: [
    getWebAutoInstrumentations({
      '@opentelemetry/instrumentation-fetch': {
        propagateTraceHeaderCorsUrls: /.*/,
        clearTimingResources: true
        },
        '@opentelemetry/instrumentation-xml-http-request': {
          propagateTraceHeaderCorsUrls: /.*/
          }
      })
    ]
});

Taking the @opentelemetry/instrumentation-fetch instrumentation as an example, we are able to see traces for HTTP requests, and the propagators also ensure that the spans can connect with our Java backend services to give a full picture of the amount of time taken to process the request at each stage:

While auto-instrumentations is agreat way to get common instrumentations, we can also instantiate instrumentations directly, as we'll see in the remainder of this article.

Document Load Instrumentation

Another consideration unique to web frontend is the time taken to load assets such as images, JavaScript files and even stylesheets. Such assets taking considerable time to load can impact metrics such as First Contentful Paint, and therefore the user experience. The OTel Document Load instrumentation allows for automatic instrumentation of the time taken to load assets when using the @opentelemetry/sdk-trace-web package.

It is simply a case of adding the instrumentation to the instrumentations array we have provided to our provider using registerInstrumentations:

// Used to auto-register built-in instrumentations like page load and user interaction
import { registerInstrumentations } from '@opentelemetry/instrumentation';

// Document Load Instrumentation automatically creates spans for document load events
import { DocumentLoadInstrumentation } from '@opentelemetry/instrumentation-document-load';

// Configuration discussed above omitted

// Enable automatic span generation for document load and user click interactions
registerInstrumentations({
  instrumentations: [
    // Automatically tracks when the document loads
    new DocumentLoadInstrumentation({
      ignoreNetworkEvents: false,
      ignorePerformancePaintEvents: false
      }),
      // Other instrumentations omitted
  ]
});

This configuration will create a new trace conventiently named documentLoad, that will show us the time taken to load resources within the document, similar to the following:

Each span will have metadata attached to help us identify which resources are taking considerable time to load, such as this image example, where the resource takes 837ms to load:

Click Events

You may wonder why we want to capture user interactions with web applications for diagnostic purposes. Being able to see the trigger points for errors can be useful in incidents to establish a timeline of what happened, and determine if users are indeed being impact as is the case for Real Ueer Monitoring tools. But if we also consider the field of Digital Experience Monitoring, or DEM, software teams need details on usage of application features to understand the user journey and how it could possibly being improved in a data-drive way. Capturing user events is required for both.

The OTel UserInteraction instrumentation for web is how we capture these events. Similar to the document load instrumentation it depends on the @opentelemetry/sdk-trace-web package, and when used with zone-js and the ZoneContextManager it also supports async operations.

Like other instrumentations it is added via registerInstrumentations:

// Used to auto-register built-in instrumentations like page load and user interaction
import { registerInstrumentations } from '@opentelemetry/instrumentation';

// Automatically creates spans for user interactions like clicks
import { UserInteractionInstrumentation } from '@opentelemetry/instrumentation-user-interaction';

// Configuration discussed above omitted

// Enable automatic span generation for document load and user click interactions
registerInstrumentations({
  instrumentations: [
    // User events
    new UserInteractionInstrumentation({
      eventNames: ['click', 'input'] // instrument click and input events only
    }),
    // Other instrumentations omitted
  ]
});

It will capture and label spans for the user events we configure, and leveraging the propagators configured previously can connect spans from other resources to the user event, similar to the below example where we see the service call to get records when the user adds a search term to the input box:

Metrics

There are numerous different measurements that are helpful in capturing useful indicators of availability and performace of web applications, such as latency, throughput or the number of 404 errors. Google Core Web Vitals are a set of standard metrics used by web developers to measure real-world user experience of web sites, including loading performance, reactivity to user input and visual stability. Given at time of writing the Core Web Vitals Plugin for OTel Browser is on the backlog, let's try building our own custom instrumentation using the web-vitals JS library to capture these as OTel metrics.

In OpenTelemetry you can create your own custom instrumentation by extending the InstrumentationBase, overriding the constructor to create the MeterProvider, Meter and OTLPMetricExporter that will allow us to send our Core Web Vital measurements to Elastic via our proxy, as presented in web-vitals.instrumentation.ts. Note that below we show only the LCP meter for succinctness, but the full example here measures all web vitals.

/* OpenTelemetry JS packages */
// Instrumentation base to create a custom Instrumentation for our provider
import {
	InstrumentationBase,
	type InstrumentationConfig,
	type InstrumentationModuleDefinition
} from '@opentelemetry/instrumentation';

// Metrics API
import {
	metrics,
	type ObservableGauge,
	type Meter,
	type Attributes,
	type ObservableResult,

} from '@opentelemetry/api';

export class WebVitalsInstrumentation extends InstrumentationBase {

  // Meter captures measurements at runtime
	private cwvMeter: Meter;

	/* Core Web Vitals Measures, LCP provided, others omitted */
	private lcp: ObservableGauge;

	constructor(config: InstrumentationConfig, resource: Resource) {
		super('WebVitalsInstrumentation', '1.0', config);

    // Create metric reader to process metrics and export using OTLP
		const metricReader = new PeriodicExportingMetricReader({
			exporter: new OTLPMetricExporter({
				url: 'http://localhost:8123/v1/metrics' // nginx proxy
			}),
			// Default is 60000ms (60 seconds).
			// Set to 10 seconds for demo purposes only.
			exportIntervalMillis: 10000
		});

    // Creating Meter Provider factory to send metrics
		const myServiceMeterProvider = new MeterProvider({
			resource: resource,
			readers: [metricReader]
		});
		metrics.setGlobalMeterProvider(myServiceMeterProvider);

    // Create web vitals meter
		this.cwvMeter = metrics.getMeter('core-web-vitals', '1.0.0');

		// Initialising CWV metric gauge instruments (LCP given as example, others omitted here)
		this.lcp = this.cwvMeter.createObservableGauge('lcp', { unit: 'ms', description: 'Largest Contentful Paint' });
	}

	protected init(): InstrumentationModuleDefinition | InstrumentationModuleDefinition[] | void {}

  // Other steps discussed later
}

You'll notice in our LCP example we have created an ObservableGauge to capture the value at the time it is read via a callback function. This can be setup when we enable our custom instrumentation, specifying when the LCP event is triggered the value will be sent via result.observe:

/* Web Vitals Frontend package, LCP shown as example*/
import { onLCP, type LCPMetric } from 'web-vitals';

/* OpenTelemetry JS packages */
// Instrumentation base to create a custom Instrumentation for our provider
import {
	InstrumentationBase,
	type InstrumentationConfig,
	type InstrumentationModuleDefinition
} from '@opentelemetry/instrumentation';

// Metrics API
import {
	metrics,
	type ObservableGauge,
	type Meter,
	type Attributes,
	type ObservableResult,

} from '@opentelemetry/api';
 
// Other OTel Metrics imports omitted

// Time calculator via performance component
import { hrTime } from '@opentelemetry/core';

type CWVMetric = LCPMetric | CLSMetric | INPMetric | TTFBMetric | FCPMetric;

export class WebVitalsInstrumentation extends InstrumentationBase {

	/* Core Web Vitals Measures */
	private lcp: ObservableGauge;

	// Constructor and Initialization omitted

	enable() {
		// Capture Largest Contentful Paint, other vitals omitted
		onLCP(
			(metric) => {
				this.lcp.addCallback((result) => {
					this.sendMetric(metric, result);
				});
			},
			{ reportAllChanges: true }
		);
	}

  // Callback utility to add attributes and send captured metric
	private sendMetric(metric: CWVMetric, result: ObservableResult<Attributes>): void {
		const now = hrTime();

		const attributes = {
			startTime: now,
			'web_vital.name': metric.name,
			'web_vital.id': metric.id,
			'web_vital.navigationType': metric.navigationType,
			'web_vital.delta': metric.delta,
			'web_vital.value': metric.value,
			'web_vital.rating': metric.rating,
			// metric specific attributes
			'web_vital.entries': JSON.stringify(metric.entries)
		};

		result.observe(metric.value, attributes);
	}
}

To use our own instrumentation, we need to register our instrumentation just like we did in frontend.tracer.ts for the available web instrumentations to capture document and user event instrumentations:

registerInstrumentations({
  instrumentations: [
    // Other web instrumentations omitted
    // Custom Web Vitals instrumentation
    new WebVitalsInstrumentation({}, resource)
    ]
});

The lcp metric, along with the attributes we specified as part of our sendMetric function will be sent to our Elastic cluster:

These metrics will not feed into the User Experience dashboard due to compatibility, but we can create a dashboard leveraging the values to show the trends of each of our vitals:

Summary

In this blog, we presented the current state of client instrumentation for the browser, along with an example showing how to instrument a simple JavaScript frontend using the OpenTelemetry browser instrumentation. To reflect back on the code, check out the repo here. If you have any questions or want to learn from other developers connect with the Elastic Community.

Developer resources:

• OTel Record Store Application
• JavaScript Browser Instrumentation