What is telemetry data?
Telemetry data refers to the raw data your systems continuously emit to report on their own behavior, including logs, metrics, traces, and events. It comes from devices such as sensors and network equipment, software such as applications and services, and systems such as servers and cloud infrastructure. These sources transmit it remotely, over a network, to a central system where you store and analyze it. It is crucial for proactive problem-solving and making informed decisions. To achieve this, though, you can't just rely on raw data (unprocessed records such as individual log lines, metric readings, and request traces); you need to have access to real-time insights. This is what analyzing telemetry data gives you.
Telemetry differs from observability. Telemetry is the process of collecting and transmitting raw system data, and observability is the ability to understand the state of your systems from that data.
Types of telemetry data
Telemetry data is commonly organized into four types, known as the MELT framework: metrics, events, logs, and traces.
| Type | What it records | Example | What you use it for |
|---|---|---|---|
| Metrics | Numerical measurements captured over time | CPU usage or error rate | Tracking trends and alerting on thresholds |
| Events | Notable occurrences or state changes, with context | A user login or a purchase | Business analysis, auditing, and automations |
| Logs | Timestamped text records of discrete events | An application error message | Debugging, troubleshooting, and compliance |
| Traces | The path and timing of a request across services | A checkout request moving through several microservices | Finding bottlenecks and root causes |
Metrics
Telemetry metrics are the numerical measurements of a system, software, or application captured at runtime. These metrics make it possible to carry out quantitative analysis of performance over time and include things like CPU usage, request rate, error rate, response time, and memory utilization. Common metric types include counters (values that only increase), gauges (values that can rise or fall), and histograms (how values are distributed). In observability, metrics are stored in a time series database (TSDB): a database purpose-built for high-volume, append-only numeric writes and fast time-range queries. Logs typically go to search-optimized systems and traces to tracing back ends. Elasticsearch stores all three, so metrics monitoring runs on the same data as your log and trace analysis.
In telemetry, these metrics play a big part in generating visibility for things like system health, performance, and behavior. This visibility can provide insights for a variety of tools, from performance monitoring and anomaly detection to things like capacity planning and SLA compliance monitoring.
Events
Events are records of notable occurrences or state changes within your system. They capture significant moments or changes that impact the system or business and differ from logs by often including additional contextual information. These events will include things like user actions like logins and purchases, system state changes, and business-specific events.
Because of the extra contextual data, events are vital for business intelligence, system health monitoring, compliance auditing, and triggering automations. It's also great for user behavior analysis, because you can see exactly what they did and when.
Traces
A trace is essentially the breadcrumb trail a request leaves as it moves through a system. It's a record of the path a request takes, showing the flow and precise timing across multiple services. This includes things like distributed application traces and end-to-end request traces in microservice architectures. Collecting traces across services is called distributed tracing: the practice of tracking a request end-to-end across multiple services, which is essential for root cause analysis in microservice architectures.
As telemetry data, these traces provide valuable data to help with performance optimization, understanding system dependencies, root cause analysis, and identifying bottlenecks.
How telemetry data is collected and sent
Telemetry is the process of collecting, transmitting, and analyzing data from remote sources such as servers, applications, and monitoring devices. This process can be broken down into the following steps:
- Data collection: Sensors, software, and other monitoring tools gather data such as user behavior, performance metrics, and environmental conditions.
- Data transmission: The collected data is sent to a central system.
- Analysis and monitoring: Once the data has been received, the data is then processed, analyzed, and displayed to provide insights into the monitored system's status, performance, and behavior.
In order to achieve this, you have to have the right telemetry data. This is all the raw data that will include things like logs, metrics, events, and any other traces that are created by systems, applications, or devices. Without this raw data, the analysis and monitoring won't be as useful or accurate.
Streaming telemetry is a way to send this data continuously. Instead of a monitoring system polling each device at fixed intervals, a collector subscribes to a stream of data that the device pushes as it's generated.
What is a telemetry pipeline?
A telemetry pipeline is the path your telemetry data follows from the systems that generate it to the platform where you store and analyze it. Along the way, the pipeline collects, processes, and routes the data.
Most telemetry pipelines are built from these parts:
- Instrumentation: Code in your applications and services that emits telemetry data — add it with SDKs, or use zero-code instrumentation that requires no changes to your application's code
- Agents: Software that runs on a host, next to your workloads, and collects telemetry data from applications and infrastructure
- Shippers: Lightweight tools that forward data, such as log files, from where it's created to a central system
- Pipelines: Processing stages that parse, filter, enrich, sample, and redact data, then route it to one or more destinations
In Elastic Observability, Elastic Distributions of OpenTelemetry (EDOT) language SDKs instrument your applications and send data over OTLP to Elastic Agent, which acts as the collection and processing layer and writes the data to Elasticsearch.
How to set up and use telemetry data
Step 1: Define your goals
The first step in setting up your telemetry data is deciding exactly what you're trying to achieve. You need to identify specific objectives for collecting your telemetry data, such as improving system performance, enhancing your user experience, or maybe identifying security threats. A good way to do this is by aligning your telemetry goals with your existing business objectives.
Then you need to determine what key performance indicators (KPIs) and metrics you'll need to track these. For example, if your goal is to improve your system performance, you might want to track things like response times, error rates, and resource utilization. Another important consideration when deciding your telemetry goals is privacy and compliance. You should review relevant regulations (CCPA, GDPR, etc.) and define guidelines for data collection that will meet these regulations and respect user privacy.
Step 2: Configure your tools
The next step is to select and configure the appropriate tools to meet your goals. You should choose telemetry collection tools that both help you do this and integrate easily with your existing tech stack. This might include things like application performance monitoring tools (APM), log aggregators, or custom SDKs.
You need to implement collection mechanisms, integrating logging frameworks or monitoring agents into your applications. OpenTelemetry could be a good vendor-neutral option for this. This will likely involve adding code instrumentation or configuring existing systems to receive the data you need.
To configure your collection parameters, you'll need to set up data collection specifics like sampling rates, types of events to capture, and level of detail for your logs. Sampling means collecting only a share of your data, such as one in every 10 traces, to reduce volume and cost. Then configure data filtering to filter out sensitive information and anonymization to comply with privacy regulations.
Step 3: Send your data
Next you'll need to actually send your telemetry data. To start, you'll need to implement transmission protocols for sending your data. This could be HTTPS (or HTTP) for web apps, MQTT for IoT devices, or even specialized protocols such as OpenTelemetry (OTLP). This includes making sure the transfers are secure with encryption and authentication mechanisms.
To handle network interruptions and ensure reliability, you should set up local data buffering and retry logic. You will also need to balance the volume and frequency of your data transmission to minimize the potential impact of application performance and network bandwidth.
Step 4: Store your data
Telemetry data is stored in type-matched back ends: metrics in a time series database (TSDB), logs in a search-optimized store, and traces in a tracing back end. Elasticsearch consolidates all three. You then need to choose the most appropriate storage solution to receive your telemetry data. You can configure different storage for different data types, or it can be much simpler to take a distributed storage approach using Elasticsearch and its cross-cluster search and replication to create a data mesh.
Next, you'll need to implement retention policies, defining how long different types of data should be stored. To decide this, consider both the analytical and regulatory requirements that apply to that data. Decisions around storage of your telemetry data will often require balancing the need for analytical performance versus the cost of storing it.
Another consideration is partitioning and indexing. Organize your storage to optimize query performance using things like time-based partitioning and indexing. Backup and disaster recovery should also be in place to protect against data loss and make sure continuous access to telemetry data.
Step 5: Analyze your telemetry data
Finally, you need to be able to analyze and visualize your telemetry data. This includes developing dashboards and visualizations to represent the key metrics and trends. You also need to implement alerting systems so you're notified when metrics exceed predefined thresholds or when it detects anomalies.
To get the most out of your telemetry data, set up advanced analytics such as statistical analysis and machine learning to better identify patterns and predict future trends. This works especially well when you combine insights from different data types to get a comprehensive view of system activity and user behavior.
Telemetry data benefits
Telemetry data gives your organization real-time visibility, proactive issue detection and resolution, improved operational efficiency, faster troubleshooting, and enhanced decision-making. A continuous stream of data from your infrastructure, network, services, and IoT devices is the basis of observability: You see how your systems perform, get an alert when events occur, and take action on performance and security issues with greater speed.
Here's what you get from building effective telemetry processes using your data:
- Real-time visibility: By tracking and analyzing critical metrics and logs, you get instant insights into what's going well and where you can make improvements.
- Proactive issue detection and resolution: Your telemetry data makes it possible to identify anomalies and patterns that could indicate potential problems before they escalate.
- Improved operational efficiency: You're able to streamline your workflows and increase productivity by implementing automated monitoring and data collection.
- Faster troubleshooting: Your detailed telemetry data means you can quickly identify the root cause of an issue, reducing mean time to resolution (MTTR).
- Enhanced decision-making: By having all the data you need available, you can make data-driven decisions and informed strategic choices about resource allocation, product development, and UX improvements.
In order to get all the benefits from your telemetry data, it's crucial to use a unified data platform to consolidate your telemetry data. This way you can bring together your data from various sources and reduce swivel chairing. Consolidated data matters most in cloud computing and IT environments, where workloads run across many services, containers, and clouds. Containers and the Kubernetes clusters that orchestrate them report resource usage and health for each pod and node, and serverless platforms report the duration and errors of every function invocation. Autoscaling uses signals such as CPU and memory utilization to add or remove capacity as demand changes, and cloud-provider monitoring services such as Amazon CloudWatch, Azure Monitor, and Google Cloud Monitoring collect metrics and logs from the managed services you run. Unified telemetry also gives machine learning a complete view of your systems: Models learn the normal behavior of your metrics and logs, detect anomalies automatically, and forecast trends.
This consolidation not only gives you the above benefits but also improves scalability and cost efficiency while ensuring consistency across all your data processing. That's also why open standards like OpenTelemetry semantic conventions are so important. They make sure you have interoperability between your different tools and platforms.
Telemetry data challenges
As with any technology, using telemetry data comes with both benefits and challenges. The way to overcome these challenges is to understand them and make a plan on how to overcome them.
- Data security and privacy: The safety and privacy of your user and customer data is vital. Not just because it’s good practice, but also because it can have huge reputational and regulatory repercussions. To combat this, you need to implement robust security measures like encryption, access controls, and comprehensive logging. You also need to protect sensitive information and maintain tight control over your data access.
- Legacy software integration: You need to include support for a range of data formats and sources by implementing flexible data ingestion. This makes it possible to incorporate telemetry data from both modern and legacy systems into a unified platform.
- Cost-effective storage: Tiered storage options and Information Lifecycle Management (ILM) policies mean you can optimize your storage costs while ensuring data accessibility. Hot‑warm‑cold architecture can also help you balance performance and cost-effectiveness.
- Open standards support: To ensure interoperability, use open standards like OpenTelemetry as early as possible. This allows you to future-proof your telemetry infrastructure and easily integrate with other tools and platforms.
Telemetry data, privacy, and security
Telemetry data often contains more than system measurements. Logs, traces, and events can capture personal data such as user IDs, email addresses, and IP addresses, so you need to protect it from collection through to storage.
- Personal data: Under the GDPR, personal data is any information relating to an identified or identifiable person. That includes online identifiers such as IP addresses and cookie identifiers. Review what your instrumentation and logs capture, and collect only what you need.
- Vendor safety: Telemetry data often leaves your environment, so check where each vendor stores it, who can access it, and which independent security certifications the vendor holds. Elastic publishes its audits and certifications in the Elastic Trust Center.
- Governance: Define who owns each type of data, who can read it, and how long you keep it. In Elasticsearch, role-based access control limits access down to individual fields and documents.
- Anonymization: Anonymization removes identifying details so that a person can no longer be identified. Pseudonymization replaces them with a value that additional information can link back to the person, so the GDPR still treats that data as personal data. With Elasticsearch, you can detect and redact personally identifiable information (PII) in logs before you store them.
OpenTelemetry with Elastic
Earlier in the guide we touched on OpenTelemetry, the open source observability framework that unifies data collection and standardizes telemetry data formats. It also gives you a vendor-neutral approach to telemetry that avoids lock-in and is very scalable for high-volume data handling. Elastic's implementation is EDOT, a family of supported OpenTelemetry distributions preconfigured to route telemetry data into Elasticsearch. It includes Elastic Agent (which packages OpenTelemetry Collector capabilities), EDOT language SDKs, and Elastic Serverless Forwarders for cloud environments.
This fits very closely with our commitments to open standards at Elastic, which is why we've contributed our open source specification, Elastic Common Schema (ECS), to OpenTelemetry.
We also include seamless ingestion of OTel-formatted data into Elasticsearch as part of our broader observability function. This means you can use the advanced Elasticsearch analytics and search with OTel data and future-proof your observability platform.
This combination of Elastic and OpenTelemetry addresses a lot of the key challenges you're likely to face:
- Consolidating heterogeneous data sources
- Ensuring data security and privacy
- Managing legacy software integration
- Optimizing storage costs
- Handling diverse data types
- Supporting open standards for interoperability