Elastic at KubeCon Europe 2020: Orchestration to observability, and beyond! | Elastic Blog

Elastic at KubeCon Europe 2020: Orchestration to observability, and beyond!

KubeCon Europe 2020 is virtual this year, and Elastic is doing our part to help "keep cloud native connected." We would rather be there in person to shake hands, tell stories, and laugh, but the challenges of a virtual conference also provide the opportunity to share great content and materials that we might not be able to at a crowded booth.

So please join us at our virtual booth at Virtual KubeCon 2020 (Amsterdam) to see a virtual demo session, get access to custom content, grab exclusive virtual swag, and chat with real people.

At the booth we’ll give a quick introduction to Elasticsearch (the free and open, scalable, and highly-available search engine and datastore) and show how it powers use cases like enterprise search, observability, and security. We’ll demonstrate how the applications that power Elastic Observability (Logs, Metrics, APM, and Uptime) consolidate the event streams from your Kubernetes cluster, cloud and physical infrastructure, Prometheus, and other CNCF projects and services with your application traces in a fast, scalable, and unified data store. We'll also show you how to spin up an Elastic Stack cluster with Elastic Cloud on Kubernetes (ECK), the official operator for orchestrating Elasticsearch on Kubernetes.

Orchestration with Elastic Cloud on Kubernetes

At KubeCon 2019 in San Diego, the buzz was about the beta of Elastic Cloud on Kubernetes (ECK). ECK is the official operator for orchestration of Elasticsearch and the Elastic Stack — not just for initial day-one deployments, but also for day-two administration and beyond.  

This year, we’re excited to chat with folks who have been using ECK to simplify the orchestration of their Elastic Stack deployments and get feedback around how we can continue to improve their experience. The ECK operator is now generally available (GA) and was recently updated to include the ability to orchestrate the lifecycle for Elastic Enterprise Search as well as Beats. Deploying an Elastic cluster with ECK is as simple as installing the operator:

kubectl apply -f https://download.elastic.co/downloads/eck/1.1.2/all-in-one.yaml

Then applying a configuration, as shown in this example, to create a three-node cluster:

cat <<EOF | kubectl apply -f - 
apiVersion: elasticsearch.k8s.elastic.co/v1 
kind: Elasticsearch 
  name: quickstart 
  version: 7.8.0 
  - name: default 
    count: 3 
      node.master: true 
      node.data: true 
      node.ingest: true 
      node.store.allow_mmap: false 

ECK can streamline the orchestration of your Elastic deployments when running on Kubernetes, but when using the Elastic Stack you have more than one choice. If you'd prefer a SaaS experience, you can sign up for a free trial of the Elasticsearch Service on Elastic Cloud.  

Of course, you can always download Elasticsearch and Kibana, and provision your own clusters, or get the best of both worlds and choose Elastic Cloud Enterprise, which provides the SaaS experience of Elastic Cloud, but on your infrastructure.  

The freedom to run how and where you choose gives you full control of where your data lives and its complete lifecycle.

Elastic Observability

Orchestration is one side of the coin. The other, of course, is observing the health and performance of those deployments. Our focus around Kubernetes and other CNCF projects is not solely based on orchestration of the Elastic Stack on Kubernetes, but digs deep into the observability of these systems and the workloads running on them as well. Automatic discovery of new or scaled containers or pods means that you don't need to update static configurations when your dynamic ecosystem changes, whether it be hour to hour, minute to minute, or second to second.  

Elastic Observability lets you gather logs and metrics from your full application stack. With support for many CNCF projects — sandbox, graduated, and in between — gathering logs and metrics from your CNCF services and sending them to Elasticsearch is easy. Elastic also recently added support for resource quota metricsets, supporting all fields exposed by the kube-state-metrics agent, along with expanded support for ingesting Prometheus metrics by adding remote_write capabilities, bringing long-term, highly-available storage of high-cardinality Prometheus data.

Logs and metrics are a good start, but for full visibility you also need to see what the applications are doing both behind the scenes as well as how they behave from the outside — and that's where application performance monitoring (APM) and service monitoring come in.

Elastic APM provides developers with native language instrumentation agents for many common programming languages and frameworks, along with support for several other open source instrumentation services and frameworks. Elastic APM expanded open instrumentation support by implementing a Jaeger intake, allowing developers to send traces from the CNCF-graduated Jaeger project directly to Elastic APM. In addition, we recently created an OpenTelemetry collector to allow gathering of OpenTelemetry traces, which joins the OpenTracing support that was added in 2018.  

Elastic APM distributed tracing

APM shows what is going on inside your applications and services — where they are spending their time, what they are interacting with, and any errors that may be occurring — and combining APM with real user monitoring (RUM) and service monitoring (Uptime and Heartbeat) allows you to accurately gauge your users' experience. Just because you've sent a response doesn't mean that the transaction is over, and just because it works for one user doesn't mean that it works for all of them.

Service monitoring with Elastic Observability

Many organizations don't run entirely in one ecosystem, but rather across multiple public and private clouds with virtualization and bare metal sprinkled in: some workloads in an on-prem Kubernetes cluster, and others deployed across multiple cloud providers. When faced with a hybrid-cloud environment, many organizations end up with siloed monitoring tools, one per cloud provider. Elastic Observability allows you to "monitor the monitors", providing visibility across your hybrid infrastructure by consolidating logs and metrics from Azure Monitor, Google Cloud Operations Suite (formerly known as StackDriver), and AWS CloudWatch, and interacting with that data alongside your Prometheus metrics, and all of your other logs and metrics, in one single datastore.

All of your logs, metrics, application traces, and availability data are stored in Elasticsearch indices, so you can mix and match, correlate and chart, and analyze and augment across different data streams.

Full observability with Elastic

Teaming up with more use cases

It's not just observability data that you can mix and match. Your observability data can be stored alongside your business KPIs as well — sales metrics, ads served, products shipped, you name it — giving you extra dimensions to slice and dice, and new ways to answer old questions. How much did that last outage cost? How did that last software change to our recommendation engine help sales? Or simple things, like determining the best time of day for an upgrade. With powerful visualizations and tools like Kibana Lens you can answer those types of questions.

Anomaly detection in observability data with Elastic

Observability is just one of the many use cases that the Elastic Stack supports. In the 1.2 release of ECK, support was added for orchestration of the enterprise search solution, along with a workflow for Beats, adding to the previously supported observability and security solutions.

With Elastic Security, the ever-growing list of prebuilt detection rules lets you leverage observability data in your threat hunting efforts, strengthening the partnership between DevOps and SecOps, speeding up investigations without duplication of data and with no extra work or cost.

Signal detection rules in Elastic Security

With built-in integrations to the free and open Kibana Alerting and Actions, you can set your rules and thresholds, then get notified via system log or index (or optionally via third party services like Slack, PagerDuty, or simple webhooks), letting you know when services are running outside of expectations or if you are in danger of breaching your SLAs.

Create alerts from anomalies

With the added advantage of automated anomaly detection that’s powered by machine learning, you can see at a glance which services are having trouble, or detect when something is behaving outside of normal parameters.

Service maps in Elastic APM

Get in touch

If you're attending KubeCon Europe 2020 "swing by" our virtual booth and chat — we'd love to hear from you. Alternatively, feel free to start a conversation on our Discuss forum, or reach us on the Elastic Stack Community on Slack.

You can get started with Elasticsearch on Kubernetes today using Elastic Cloud on Kubernetes (ECK), and you always have access to the latest version of the Elastic Stack when using the Elasticsearch Service. Sign up for a free trial of Elastic Cloud to create your first cluster, or you can download the Elastic Stack today.