Revolutionizing big data management: Unveiling the power of Amazon EMR and Elastic integration

By

Udayasimha Theepireddy (Uday),

Subhrata Kulshrestha

21-cubes.jpeg

In the dynamic realm of data processing, Amazon EMR takes center stage as an AWS-provided big data service, offering a cost-effective conduit for running Apache Spark and a plethora of other open-source applications. While the capabilities of EMR are impressive, the art of vigilant monitoring holds the key to unlocking its full potential. This blog post explains the pivotal role of monitoring Amazon EMR clusters, accentuating the transformative integration with Elastic®.

Elastic can make it easier for organizations to transform data into actionable insights and stop threats quickly with unified visibility across your environment — so mission-critical applications can keep running smoothly no matter what. From a free trial and fast deployment to sending logs to Elastic securely and frictionlessly, all you need to do is point and click to capture, store, and search data from your AWS services.

Monitoring EMR via Elastic Observability

In this article, we will delve into the following key aspects:

  • Enabling EMR cluster metrics for Elastic integration: Learn the intricacies of configuring an EMR cluster to emit metrics that Elastic can effectively extract, paving the way for insightful analysis.
  • Harnessing Kibana® dashboards for EMR workload analysis: Discover the potential of utilizing Kibana dashboards to dissect metrics related to an EMR workload. By gaining a deeper understanding, we open the doors to optimization opportunities.

Key benefits of AWS EMR integration

  • Comprehensive monitoring: Monitor the health and performance of your EMR clusters in real time. Track metrics related to cluster status and utilization, node status, IO, and many others, allowing you to identify bottlenecks and optimize your data processing.
  • Log analysis: Dive deep into EMR logs with ease. Our integration enables you to collect and analyze logs from your clusters, helping you troubleshoot issues and gain valuable insights.
  • Cost optimization: Understand the cost implications of your EMR clusters. By monitoring resource utilization, you can identify opportunities to optimize your cluster configurations and reduce costs.
  • Alerting and notifications: Set up custom alerts based on EMR metrics and logs. Receive notifications when performance thresholds are breached, ensuring that you can take action promptly.
  • Seamless integration: Our integration is designed for ease of use. Getting started is simple, and you can start monitoring your EMR clusters quickly.

Accompanying these discussions is an illustrative solution architecture diagram, providing a visual representation of the intricacies and interactions within the proposed solution.

1

How to get started

Getting started with AWS EMR integration in Observability is easy. Here's a quick overview of the steps:

Prerequisites and configurations

If you intend to follow the steps outlined in this blog post, there are a few prerequisites and configurations that you should have in place beforehand.

  1. You will need an account on Elastic Cloud and a deployed stack and agent. Instructions for deploying a stack on AWS can be found here. This is necessary for AWS EMR logging and analysis.

  2. You will also need an AWS account with the necessary permissions to pull data from AWS. Details on the required permissions can be found in our documentation.

  3. Finally, be sure to turn on EMR monitoring for the EMR cluster when you deploy the cluster.

Step 1: Create an account with Elastic

Create an account on Elastic Cloud by following the steps provided.

Step 2: Add integration

1. Log in to your Elastic Cloud on AWS deployment.

2 free trial

2. Click on Add Integration. You will be navigated to a catalog of supported integrations.

3 welcome home

3. Search and select Amazon EMR.

4 integrations

Step 3: Configure integration

  1. Click on the Add Amazon EMR button and provide the required details.

  2. Provide the required access credentials to connect to your EMR instance.

  3. You can choose to collect EMR metrics, EMR logs via S3, or EMR logs via Cloudwatch.

  4. Click on the Save and continue button at the bottom of the page.

5 amazon emr
6 add amazon emr integration

Step 4: Analyze and monitor

Explore the data using the out-of-the-box dashboards available for the integration. Select Discover from the Elastic Cloud top-level menu.

7 manage deployment

Or, create custom dashboards, set up alerts, and gain actionable insights into your EMR clusters' performance.

This integration streamlines the collection of vital metrics and logs, including Cluster Status, Node Status, IO, and Cluster Capacity. Some metrics gathered include:

  • IsIdle: Indicates that a cluster is no longer performing work, but is still alive and accruing charges
  • ContainerAllocated: The number of resource containers allocated by the ResourceManager
  • ContainerReserved: The number of containers reserved
  • CoreNodesRunning: The number of core nodes working
  • CoreNodesPending: The number of core nodes waiting to be assigned
  • MRActiveNodes: The number of nodes presently running MapReduce tasks or jobs
  • MRLostNodes: The number of nodes allocated to MapReduce that have been marked in a LOST state
  • HDFSUtilization: The percentage of HDFS storage currently used
  • HDFSBytesRead/Written: The number of bytes read/written from HDFS (This metric aggregates MapReduce jobs only, and does not apply for other workloads on Amazon EMR.)
  • TotalUnitsRequested/TotalNodesRequested/TotalVCPURequested: The target total number of units/nodes/vCPUs in a cluster as determined by managed scaling
8 pie graph

Conclusion

Elastic is committed to fulfilling all your observability requirements, offering an effortless experience. Our integrations are designed to simplify the process of ingesting telemetry data, granting you convenient access to critical information for monitoring, analytics, and observability. The native AWS EMR integration underscores our dedication to delivering seamless solutions for your data needs. With this integration, you'll find the confidence to monitor, analyze, and optimize your EMR clusters, opening up exciting opportunities for your data-driven initiatives.

Start a free trial today

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial