OpenTelemetry lookup processor: File and DNS enrichment

Enrichment is one of those tasks that sounds trivial until you try to do it inside a telemetry pipeline. You have a user.id on a log record and you want the user.name. You have a client.ip and you want the hostname behind it. Until now, the OpenTelemetry Collector had no general way to do this kind of lookup.

The closest option today is to hand-code the mapping in the transform processor:

# otel.yml
processors:
  transform:
    log_statements:
      - context: log
        statements:
          - set(attributes["user.name"], "Alice") where attributes["user.id"] == "user001"
          - set(attributes["user.name"], "Bob") where attributes["user.id"] == "user002"
          - set(attributes["user.name"], "Carol") where attributes["user.id"] == "user003"
          # ...and one more line for every user

That works for a handful of entries, but it does not scale beyond 10 to 20 items. Every new mapping means another statement, the lookup data lives in the same file as your pipeline config, and there is no way to point at reference data that already exists. It proves the need is real, but it only covers the simplest, smallest cases.

If you run the Collector and you have been reaching for the transform processor, a sidecar script, or a downstream ingest pipeline just to add reference data, this component is for you.

Why the OpenTelemetry Collector needed a lookup processor

The Collector was already good at a couple of enrichment patterns. The transform processor reshapes and derives data from what is already on a record. The k8sattributes and resourcedetection processors attach system and environment metadata, like Kubernetes pod details or cloud host information.

What it could not do was look up related data by a key you already have. Three patterns in particular had no home:

File-based lookups from static reference data in JSON, YAML, or CSV
HTTP and API-based enrichment from an external service
DNS lookups, such as resolving an IP to a hostname

These are everyday tasks in other data collectors and transformation tools. Mapping an internal service ID to a friendly name, attaching business metadata by customer ID, or resolving an IP all fall into this category. Without a native component, teams built brittle workarounds or pushed the work downstream where it is harder to reuse.

That gap is what the new lookup processor closes. Elastic's Data Processing team proposed it, the community accepted it, and it was built in partnership with Grafana, thanks to Sam DeHaan (GH: dehaansa). The lookup processor takes a value from your telemetry, uses OTTL to build a lookup key, queries a source such as a YAML file or DNS, and writes the result back as new attributes.

How the OpenTelemetry lookup processor is designed

The design grew out of repeated requests from the community for richer enrichment. A few examples that fed into it:

Rather than build a one-off component for each request, the goal was a single processor flexible enough to cover them. Four ideas shaped it:

Multiple lookups per processor, so one instance can enrich several attributes in a single pass.
Caching, so external sources like DNS do not get queried for the same key over and over.
OTTL for key extraction, so you get a full expression language for pulling the lookup key off a record, including converters.
Extensible sources, so you are not limited to the built-in set. You can register a custom source for your own data.

Extensible sources matter most for the processor's long-term value. A source is a small, well-defined interface, so the processor is a foundation for enrichment rather than a fixed list of features. YAML and CSV cover static reference data today, DNS covers dynamic resolution, and the same interface leaves the door open for HTTP APIs, key-value stores, or anything specific to your environment.

How the lookup processor evaluates keys with OTTL

Whatever source you configure, the processor runs the same three steps for every record.

First, it evaluates an OTTL expression to produce a lookup key from the record. Second, it hands that key to the configured source and gets a value back. Third, it writes the value to the attributes you name, on the record or on its parent resource. When a key has no match, the processor writes a configurable default so downstream queries stay predictable.

The next two sections walk through the two sources available today.

File-based lookups with YAML in the OpenTelemetry Collector

The most common case is static reference data. You keep a mapping file next to the Collector and enrich records against it. Here the processor reads a YAML file and adds user.name to each log based on its user.id.

# otel.yml
processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/mappings.yaml
    lookups:
      - key: log.attributes["user.id"]
        attributes:
          - destination: user.name
            default: "Unknown User"

The mapping file is a plain set of key-value pairs:

# /etc/otel/mappings.yaml
user001: "Alice"
user002: "Bob"

The key field is an OTTL value expression, so log.attributes["user.id"] reads the user.id attribute off the log record. The destination is where the looked-up value lands, and default is what gets written when the key is missing from the file.

Given this incoming log record:

{
  "body": "User logged in",
  "attributes": {
    "user.id": "user001",
    "http.method": "POST"
  }
}

The processor looks up user001, finds Alice, and produces:

{
  "body": "User logged in",
  "attributes": {
    "user.id": "user001",
    "user.name": "Alice",
    "http.method": "POST"
  }
}

The original attributes stay intact and the enriched value is added alongside them. The CSV source works the same way for teams that keep reference data in spreadsheets or exports rather than YAML.

DNS lookup enrichment inside the OpenTelemetry Collector

Static files are one thing, but some enrichment requires live, changing data. Resolving an IP address to a hostname is the classic example, and it is the first dynamic source the processor supports. Instead of a file, you point it at a DNS server.

# otel.yml
processors:
  lookup:
    source:
      type: dns
    lookups:
      - key: log.attributes["client.ip"]
        attributes:
          - destination: client.hostname
            default: "Not found"

The shape of the config is identical to the YAML example. Only the source changed. The processor pulls client.ip off the record, asks a DNS resolver to reverse-resolve it, and writes the hostname back.

Given this incoming record:

{
  "body": "User logged in",
  "attributes": {
    "client.ip": "8.8.8.8",
    "http.method": "POST"
  }
}

The DNS source resolves 8.8.8.8 and produces:

{
  "body": "User logged in",
  "attributes": {
    "client.ip": "8.8.8.8",
    "client.hostname": "dns.google",
    "http.method": "POST"
  }
}

Because DNS queries hit an external system, this is where caching earns its place. The processor keeps results in an in-memory cache so a stream of records sharing the same IP does not turn into a stream of identical DNS queries. That keeps latency down and avoids hammering your resolver.

Writing a custom lookup source

The built-in sources cover common cases, but the real design goal was extensibility. A source is a small contract: you implement a lookup function that takes a string key and returns a value. The processor takes care of OTTL key evaluation, caching, defaults, and writing attributes, so a custom source only has to answer the question "what value goes with this key?"

That means if you already run an internal metadata API, a Redis cache, or a custom database, you can wire it in as a source and reuse everything else the processor provides. HTTP-based sources and key-value stores are natural fits, and they are on the roadmap precisely because the interface makes them straightforward to add.

How Elastic plans to use this lookup processor

Elastic builds its Collector distributions on OpenTelemetry Collector Contrib. The plan is to include lookup processor so the enrichment work teams do today with brittle workarounds can run inside the pipeline instead of downstream.

Two patterns stand out. The first is reference-data enrichment: turning an internal service ID into a friendly name, or attaching metadata such as a team or customer by ID, so telemetry arrives already labeled for search and correlation. The second is DNS resolution: turning a client.ip into a hostname before the data lands, which matters for network and security telemetry where you want the name rather than the raw address.

Doing this in the Collector keeps reference data close to where telemetry is processed and avoids duplicating the logic in separate ingest steps. As the source interface grows to cover HTTP APIs and key-value stores, the same processor can back richer enrichment without changing how pipelines are configured.

Lookup processor roadmap and how to contribute

The main implementation of the lookup processor has merged upstream into OpenTelemetry Collector Contrib. The core processor and YAML source landed first, and the DNS source followed as the first dynamic lookup.

There is a healthy backlog of work ahead, and contributions are welcome:

An HTTP lookup source for enrichment from external APIs.
More DNS capabilities, including A and AAAA queries and support for multiple DNS servers.
Component telemetry, so you can observe cache hit and miss rates, lookup latency, and error rates.
Performance improvements as real-world usage grows.

If you want to try it, add the processor to a Collector build that includes Contrib, point a YAML source at a mapping file, and enrich a real record. Then open an issue or pull request on the OpenTelemetry Collector Contrib repository. The component is community-owned, and the more sources and feedback it gets, the more useful it becomes.

To go deeper, read the lookup processor README and the enrichment tracking issue. For more on how Elastic builds on OpenTelemetry, browse the OpenTelemetry articles on Elastic Observability Labs.

You have the IP, you want the hostname: building a lookup processor for OpenTelemetry