Elastic and Optimyze join forces to deliver "always on" continuous profiling of infrastructure, applications, and services

We are excited to announce that Elastic is joining forces with Optimyze, to accelerate our vision for unified, actionable observability and enhance the ability to detect and find root cause faster in complex distributed environments.

With deep expertise in large-scale distributed systems, the Optimyze team envisioned a simpler way to get insights into the entire IT ecosystem and eliminate blind spots with their Prodfiler offering. Leveraging Extended Berkeley Packet Filter (eBPF), Optimyze delivers an innovative way for whole-system continuous profiling of systems and code with low performance overhead.

This acquisition, combined with our recent acquisitions of Cmd and build.security enables us to expand on our vision to both observe and protect their data on one unified platform, the Elastic Search Platform. We intend to integrate Optimyze and Cmd eBPF-related innovations as well as the Open Policy Agent (OPA) capabilities from build.security into the Elastic Agent to deliver a simple deployment process and a unified approach to data collection for observability and security.

Optimyze provides frictionless whole-system continuous profiling, while the Elastic Search Platform delivers analytics and machine learning capabilities with the ability to correlate and contextualize profiling data with metrics, logs, and traces. The ability to unify the three pillars of observability (metrics, logs and traces) with emerging continuous profiling capabilities delivers actionable insights, leading to improvements in service quality and performance while reducing MTTD (mean-time-to-detect) and MTTR (mean-time-to-resolution).

Thomas Dullien, CEO and co-founder of Optimyze, shares his thoughts on the future of continuous profiling and what it means for our customers.

“We are excited to join forces with Elastic. Continuous profiling across systems, applications, and services with no code changes, and little performance overhead is by itself a game changer. The value increases exponentially when this data can be easily combined and cross-referenced with metrics, traces, logs, and other operational data. Using the analytics and machine learning capabilities of the Elastic Stack, we will make it easier for users to effectively solve performance mysteries, debug production incidents quicker, reduce wasteful computation and ultimately deliver services that will be faster, cheaper, and more energy-efficient. We look forward to being part of the Elastic team and making this vision a reality."

Delivering on ubiquitous observability

The rapid evolution of ephemeral infrastructure and distributed microservices based applications has created blind spots for operations teams across the applications and infrastructure they manage. With traditional application performance monitoring, it's not always possible to instrument third party libraries and services that an application might have dependencies on. Other applications running on the same infrastructure may also be competing for resources and starving the application. These limitations reduce visibility for development and operations teams. As a result, when a problem occurs, developers and operations teams often don’t know the root cause of the problem, leaving problems unresolved. The adoption of cloud-native, microservices-based distributed architectures further magnifies this problem.

While profiling is a proven method for improving efficiency and solving performance problems, traditional profiling techniques are usually not suitable for enterprise-wide use on production systems:

• Cost and performance overhead of standard profiling techniques is prohibitive, resulting in only intermittent use.
• There is an inability to instrument third party application, service, or library dependencies.
• Developers need to initialize profilers, which often involves changes to source code.

Optimyze overcomes the limitations of traditional monitoring systems, enabling visibility into the black box and solving performance bottlenecks by delivering support for:

• All major containerization and orchestration frameworks, such as Google Kubernetes Engine (GKE), Azure Kubernetes Service (AKS), or Amazon EKS, as well as non-containerized environments.
• Native C/C++, Rust, and Go code, even without debug symbols, as well as PHP, Python, Java, Scala, Ruby, and Perl, as well as mixed-language stack traces e.g. Python or Java code calling native code and then calling the kernel.
• Natively built stack traces that go from the kernel and userspace native code all the way into code running in higher level runtimes, enabling unprecedented insight into system behaviour at all levels.
• Less than 1% CPU utilization and less than 250MB RAM consumption, supporting 24x7 runtime on most workloads without noticeable impact.

Furthering our vision for Elastic Observability

Elastic Observability is focused on leveraging the power of our search platform and machine learning capabilities to change the operational paradigm and deliver a solution that improves developer productivity, accelerates innovation, and delivers rich customer experience.

For example, consider the use case of a Site Reliability Engineer (SRE) monitoring infrastructure with hosts reporting high CPU utilization. In a typical scenario, the user is able to view CPU utilization metrics on charts with limited insights into the processes or runtime analysis. With Optimyze technology integrated into Elastic, users are able to:

• Perform detailed analysis of the usage including at the level of hosts, containers, pods, and individual threads.
• Drill down into individual stack frames, and analyze where the CPU cycles are being spent. The stack traces span kernel space and user space to see what’s precisely going on.
• Compare performance across dimensions like time, thread name, container name, and deployment name in the case of Kubernetes including historical comparisons.
• Scale across Kubernetes clusters with hundreds and thousands of nodes, with the ability to filter, facet and compare the information by dimensions.

Similarly, for DevOps teams Optimyze offers a deeper view into third party components and processes competing for machine resources, such as your database server or web server. Applying Elastic’s machine learning capabilities to the rich telemetry data set will further accelerate root cause analysis leading to a reduction in MTTD and MTTR.

Elastic continues to make major advances in our cloud-native observability and security capabilities by investing in innovative teams that have built differentiated capabilities leveraging open technologies like eBPF. Elastic users will see these capabilities added over time into the architecture they already deploy today using the power of the Elastic Stack and the Elastic Agent.

Follow us at https://www.elastic.co/observability to continue to watch the evolution of Elastic Observability.

Forward-Looking Statements

This blog post contains forward-looking statements which include but are not limited to statements about future features and functionality. The release and timing of any features or functionality described in this document remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.