NVIDIA GPU OpenTelemetry Assets

NVIDIA Data Center GPU Manager (DCGM) is NVIDIA's telemetry and health management framework for monitoring GPU performance, temperature, memory, power, and hardware errors in data center environments.

This content pack provides dashboards, alert rules, and SLO templates for NVIDIA GPU metrics collected via the OpenTelemetry Collector's Prometheus receiver scraping the DCGM exporter. The assets cover GPU thermal health, compute and memory utilization, power consumption, clock frequencies, framebuffer memory, and XID hardware errors.

The NVIDIA GPU OpenTelemetry assets have been tested with the OpenTelemetry Collector Contrib Prometheus receiver v0.146.1, scraping metrics from the NVIDIA DCGM exporter.

DCGM exporter tested against:

• DCGM exporter v3.3.9

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

The NVIDIA DCGM exporter must be running and exposing a Prometheus /metrics endpoint. In Kubernetes environments, dcgm-exporter is typically deployed as a DaemonSet on GPU nodes.

Verify that the DCGM exporter is accessible:

curl http://<DCGM_EXPORTER_HOST>:9400/metrics
		

You should see Prometheus-formatted metrics with names starting with DCGM_FI_.

Configure the OpenTelemetry Collector (or EDOT Collector) to scrape DCGM metrics and export them to Elasticsearch.

Placeholders used in the configuration below:

• <DCGM_EXPORTER_HOST> — Hostname or IP of the DCGM exporter (e.g. localhost)
• <DCGM_EXPORTER_PORT> — Port of the DCGM exporter (e.g. 9400)
• <ES_ENDPOINT> — Your Elasticsearch endpoint (e.g. https://my-cluster.es.io:443)
• ${env:ES_API_KEY} — Elasticsearch API key, set via the ES_API_KEY environment variable

receivers:
  prometheus/nvidia_gpu:
    config:
      scrape_configs:
        - job_name: nvidia_gpu
          scrape_interval: 10s
          metrics_path: /metrics
          scheme: http
          static_configs:
            - targets:
                - <DCGM_EXPORTER_HOST>:<DCGM_EXPORTER_PORT>

processors:
  resourcedetection/system:
    detectors: [system]
    system:
      hostname_sources: [os]
  resource/nvidia_gpu:
    attributes:
      - key: data_stream.dataset
        value: nvidia_gpu
        action: upsert

exporters:
  elasticsearch/otel:
    endpoint: <ES_ENDPOINT>
    api_key: ${env:ES_API_KEY}
    mapping:
      mode: otel

service:
  pipelines:
    metrics:
      receivers: [prometheus/nvidia_gpu]
      processors: [resourcedetection/system, resource/nvidia_gpu]
      exporters: [elasticsearch/otel]
		

Refer to the NVIDIA DCGM exporter documentation for the full list of available metrics. The key metrics used by the generated assets are:

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.

×