Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

illustration-dev-sec-ops-cloud-automations-1680x980.png

As an operations engineer (SRE, IT Operations, DevOps), managing technology and data sprawl is an ongoing challenge. Cloud Native Computing Foundation (CNCF) projects are helping minimize sprawl and standardize technology and data, from Kubernetes, OpenTelemetry, Prometheus, Istio, and more. 

Kubernetes and OpenTelemetry are becoming the de facto standard for deploying and monitoring a cloud native application. Regardless of what cloud you use, Kubernetes provides the ability to deploy anywhere and OpenTelemetry gives you the opportunity to use any observability tool of choice, without having to re-instrument.  

As a single platform, Elastic helps SREs unify and correlate limitless telemetry data, including metrics, logs, and traces, into a single datastore, Elasticsearch. By then applying the power of Elastic’s advanced machine learning (ML) and analytics, you can break down silos and turn data into insights. As a full-stack observability solution, everything from infrastructure monitoring to log monitoring and application performance monitoring (APM) can be found in a single, unified experience.

elastic observability achieving a balance

With its open source roots, Elastic supports the most recent and popular open source projects, from Kubernetes, OpenTelemetry, Istio, Prometheus, and more. This support helps eliminate churn and unnecessary tooling to integrate a new capability. It allows developers to use a consistent set of integrations, libraries, and schemas to build, deploy, and observe an application.

Additionally, managing these applications with CNCF projects gives SREs an enormous amount of control to concentrate on operations, efficiencies, and scale. It allows them to focus on business performance rather than tooling, as well as constant interrupt-driven operations. 

With Elastic Observability, you can: 

elastic observability cloud native application kubernetes

In this blog, we’ll provide a summary of the support we highlighted in the above list and image.

Kubernetes observability with Elastic

The day-to-day operations of understanding the status and health of your Kubernetes clusters and applications running on them, through the logs, metrics, and traces they generate, will likely be your biggest challenge. But as an operations engineer, you will need all of that important data to help prevent, predict, and remediate issues. And you certainly don’t want that volume of metrics, logs, and traces spread across multiple tools when you need to visualize and analyze Kubernetes telemetry data for troubleshooting and support.

Elastic Observability helps manage the sprawl of Kubernetes metrics and logs by providing extensive and centralized observability capabilities beyond just the logging that we are known for. Elastic Observability provides you with granular insights and context into the behavior of your Kubernetes clusters along with the applications running on them by unifying all of your metrics, log, and trace data through OpenTelemetry and APM agents. 

Here are a few resources providing an overview of what Elastic’s support is like for Kubernetes:

Regardless of the cluster location (EKS, GKE, AKS, self-managed) or application, Kubernetes monitoring is made simple with Elastic Observability. All of the node, pod, container, application, and infrastructure (AWS, GCP, Azure) metrics, infrastructure, and application logs, along with application traces are available in Elastic Observability. 

  • Elastic Cloud can aggregate and ingest metrics and log data through the Elastic Agent (easily deployed on your cluster as a DaemonSet) to retrieve logs and metrics from the host (system metrics, container stats) along with logs from all services running on top of Kubernetes.
  • Elastic Observability can bring a unified telemetry experience (logs, metrics, traces) across all your Kubernetes cluster components (pods, nodes, services, namespaces, and more).

elastic observability cloud
Elastic Agent with Kubernetes Integration - 1

Additionally, you can visualize Kubernetes cluster metrics and logs with out-of-the-box (OOTB) dashboards, create your own dashboards, analyze logs, and even utilize Elastic’s machine learning against all this telemetry data. Let’s take a sneak peek at Elastic’s helpful dashboards.

elastic observability hipster shop cluster
Hipster Shop cluster metrics on Elastic Kubernetes overview dashboard - 2
elastic observability hipster shop default namespace
Hipster Shop default namespace pod dashboard on Elastic Observability - 3

In addition to the cluster overview dashboard and pod dashboard, Elastic has several useful OOTB dashboards:

  • Kubernetes overview dashboard (see above)
  • Kubernetes pod dashboard (see above)
  • Kubernetes nodes dashboard
  • Kubernetes deployments dashboard
  • Kubernetes DaemonSets dashboard
  • Kubernetes StatefulSets dashboards
  • Kubernetes CronJob & Jobs dashboards
  • Kubernetes services dashboards
  • More being added regularly

Working with logs on Elastic Observability utilizes Elastic’s Discovery capabilities:

elastic observability stream
Kubernetes container logs and Elastic Agent logs

As you can see from the screens above, not only can I get Kubernetes cluster metrics, but also all the Kubernetes logs simply by using the Elastic Agent in my Kubernetes cluster. 

OpenTelemetry support on Elastic

Once you’ve migrated your app to the cloud, how do you manage and monitor production, scale, and availability of the service? OpenTelemetry is quickly becoming the de facto standard for instrumentation and collection of application telemetry data for Kubernetes applications. Here are two blogs and a video that provide a great overview of Elastic’s OpenTelemetry support with Kubernetes:

OpenTelemetry (OTel) is an open source project providing a collection of tools, APIs, and SDKs that can be used to generate, collect, and export telemetry data (metrics, logs, and traces) to understand software performance and behavior. OpenTelemetry recently became a CNCF incubating project and has a significant amount of growing community and vendor support.

While OTel provides a standard way to instrument applications with a standard telemetry format, it doesn’t provide any backend or analytics components. Hence using OTel libraries in applications, infrastructure, and user experience monitoring provides flexibility in choosing the appropriate observability tool of choice. There is no longer any vendor lock-in for application performance monitoring (APM).

elastic observability microservices

Elastic Observability natively supports OpenTelemetry and its OpenTelemetry protocol (OTLP) to ingest traces, metrics, and logs. All of Elastic Observability’s APM capabilities are available with OTel data. Hence the following capabilities (and more) are available for OTel data:

  • Service maps

elastic observability service map
  • Service details (latency, throughput, failed transactions)
elastic observability frontend
  • Dependencies between services
elastic observability dependencies between services
  • Transactions (traces)
elastic observability transactions traces
  • Machine learning correlations (specifically for latency)
elastic observability machine learning correlations

In addition to Elastic’s APM and unified view of the telemetry data, you will now be able to use Elastic’s powerful machine learning capabilities to reduce the analysis and alerting to help reduce MTTR. 

Elastic’s support for other CNCF projects

In addition to Kubernetes and OpenTelemetry, Elastic also provides:

The Elastic Stack can securely ingest operational data from multiple sources with ease. 

Ingesting Fluentd logs is fairly simple with two options available:

For Prometheus, view your metrics across geographically dispersed Prometheus instances, unify your Prometheus metrics with your logs and APM data in Elasticsearch, and correlate them all in Kibana. Read more about the benefits of running Prometheus with Elastic at scale.

ingest prometheus metrics at elastic

When monitoring your interactions between services and the components managing them, such as Istio, Traefik, or Envoy, Elastic provides integrations to help ingest and analyze the data. Here are the links to instructions on how to ingest logs and metrics from:

Elastic Observability truly supports CNCF projects natively

I hope you’ve gotten an appreciation for how Elastic Observability can help you manage Kubernetes clusters along with other CNCF projects you will use to manage and observe your applications and clusters. 

A quick recap of information covered above:

Ready to get started? Register and try out the features and capabilities I’ve outlined above.