Managing the Hidden Complexity of Message-Driven Architectures

Amazon MQ is a managed message broker service for Apache ActiveMQ Classic and RabbitMQ that manages the setup, operation, and maintenance of message brokers. Messaging systems like RabbitMQ, managed by Amazon MQ, are pivotal in modern decoupled, event-driven applications. By serving as an intermediary between services, RabbitMQ facilitates asynchronous communication through message queuing, routing, and reliable delivery, making it an ideal fit for microservices, real-time pipelines, and event-driven architectures. However, this flexibility introduces operational challenges, such as retries, processing delays, consumer failures, and queue backlogs, which can gradually impact downstream performance and system reliability.

With Elastic’s Amazon MQ integration, users gain deep visibility into message flow patterns, queue performance, and consumer health. This integration allows for the proactive detection of bottlenecks, helps optimize system behaviour, and ensures reliable message delivery at scale.

In this blog, we'll dive into the operational challenges of RabbitMQ in modern architectures, while also examining the common gaps and strategies for overcoming them.

Why Observability for RabbitMQ on Amazon MQ Matters?

RabbitMQ brokers are integral to distributed systems, handling tasks ranging from order processing to payment workflows and notification delivery. Any disruption can cascade into significant downstream issues. Observability into RabbitMQ helps answer critical operational questions like:​

• Is CPU and memory utilization increasing over time?
• What are the trends in the message publish rate, message confirmation rate?
• Are consumers failing to acknowledge messages?
• Which queues are experiencing abnormal growth?
• Are there an increasing number of messages being dead-lettered over time?

Enhanced Observability with Amazon MQ Integration

Elastic provides a dedicated Amazon MQ integration for RabbitMQ that utilizes Amazon CloudWatch metrics and logs to deliver comprehensive observability data. This integration enables the ingestion of metrics related to connections, nodes, queues, exchanges, and system logs.

By deploying Elastic Agent with this integration, the users can monitor:​

• Queue performance and Dead-letter queue (DLQ) metrics include total message count (MessageCount.max), messages ready for delivery (MessageReadyCount.max), and unacknowledged messages (MessageUnacknowledgedCount.max). MessageCount.max metric tracks the total number of messages in a queue, including those that have been dead-lettered, and monitoring this over time can help identify trends in message accumulation, which may suggest issues leading to dead-lettering.
• Consumer behaviour through metrics like consumer count (ConsumerCount.max) and acknowledgement rate (AckRate.max), which help identify underperforming consumers or potential backlogs.
• Messaging throughput by tracking publish (PublishRate.max), confirm (ConfirmRate.max), and acknowledgement rates in real time. These are crucial for understanding application messaging patterns and flow.
• Broker and node-level health, including memory usage (RabbitMQMemUsed.max), CPU utilization (SystemCpuUtilization.max), disk availability (RabbitMQDiskFree.min), and file descriptor usage (RabbitMQFdUsed.max). These indicators are essential for diagnosing resource saturation and avoiding service disruption.

Integrating Amazon MQ Metrics into Elastic Observability

Elastic's Amazon MQ integration facilitates the ingestion of CloudWatch metrics and logs into Elastic Observability, delivering near real-time insights into RabbitMQ. The prebuilt Amazon MQ dashboard visualizes this data, providing a centralized view of broker health, messaging activity, and resource usage, helping users quickly detect and resolve issues. Elastic's alerting for Observability enables proactive notifications based on custom conditions, while its SLO capabilities allow users to define and track key performance targets, strengthening system reliability and service commitments. 

Elastic brings together logs and metrics from Amazon MQ alongside data from a wide range of other services and applications, whether running in AWS, on-premises, or across multi-cloud environments, offering unified observability from a single platform.

Prerequisites

To follow along, ensure you have:

• An account on Elastic Cloud and a deployed stack in AWS (see instructions here). Ensure you are using version 8.16.5 or higher. Alternatively, you can use Elastic Cloud Serverless, a fully managed solution that eliminates infrastructure management, automatically scales based on usage, and lets you focus entirely on extracting value from your data.
• An AWS account with permissions to pull the necessary data from AWS. See details in our documentation.

Architecture

Tracing Audit Flows from RabbitMQ to AWS Lambda

Consider a financial audit trail use case, where every user action, such as a funds transfer, is published to RabbitMQ. A Python-based AWS Lambda function consumes these messages, deduplicates them using the id field, and logs structured audit events for downstream analysis.

Sample payload sent through RabbitMQ:

{
  "id": "txn-849302",
  "type": "audit",
  "payload": {
    "user_id": "u-10245",
    "event": "funds.transfer",
    "amount": 1200.75,
    "currency": "USD",
    "timestamp": "T14:20:15Z",
    "ip": "192.168.0.8",
    "location": "New York, USA"
  }
}

You can now correlate message publishing activity from RabbitMQ with AWS Lambda invocation logs, track processing latency, and configure alerts for conditions like drops in consumer throughput or an unexpected surge in RabbitMQ queue depth.

AWS Lambda Function: Processing RabbitMQ Messages

This Python-based AWS Lambda function processes audit events received from RabbitMQ. It deduplicates messages based on the id field and logs structured event data for downstream analysis or compliance. Save the code below in a file named app.py.

import json
import logging
import base64
# Configure logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# In-memory set to track processed message IDs for deduplication
processed_ids = set()
def lambda_handler(event, context):
    logger.info("Lambda triggered by RabbitMQ event")
    if 'rmqMessagesByQueue' not in event:
        logger.warning("Invalid event: missing 'rmqMessagesByQueue'")
        return {'statusCode': 400, 'body': 'Invalid RabbitMQ event'}
    for queue_name, messages in event['rmqMessagesByQueue'].items():
        logger.info(f"Processing queue: {queue_name}, Messages count: {len(messages)}")
        for msg in messages:
            try:
                raw_data = msg['data']
                decoded_json = base64.b64decode(raw_data).decode('utf-8')
                message = json.loads(decoded_json)
                logger.info(f"Decoded message: {json.dumps(message)}")
                message_id = message.get('id')
                if not message_id:
                    logger.warning("Message missing 'id', skipping.")
                    continue
                if message_id in processed_ids:
                    logger.warning(f"Duplicate message detected: {message_id}")
                    continue
                payload = message.get('payload', {})
                logger.info(f"Processing message ID: {message_id}")
                logger.info(f"Event Type: {message.get('type')}")
                logger.info(f"User ID: {payload.get('user_id')}")
                logger.info(f"Event: {payload.get('event')}")
                logger.info(f"Amount: {payload.get('amount')} {payload.get('currency')}")
                logger.info(f"Timestamp: {payload.get('timestamp')}")
                logger.info(f"IP Address: {payload.get('ip')}")
                logger.info(f"Location: {payload.get('location')}")
                processed_ids.add(message_id)
            except Exception as e:
                logger.error(f"Error processing message: {str(e)}")
    return {'statusCode': 200, 'body': 'Messages processed successfully'}

Setting up AWS Secrets Manager

To securely store and manage your RabbitMQ credentials, use AWS Secrets Manager.​

1. Create a New Secret:

   • Navigate to the AWS Secrets Manager console.
   • Choose Store a new secret.
   • Select Other type of secret.
   • Enter the following key-value pairs:
     • username: Your RabbitMQ username
     • password: Your RabbitMQ password

2. Configure the Secret:

   • Provide a meaningful name, such as RabbitMQAccess.
   • Optionally, add tags and set rotation if needed.​

3. Store the Secret:

   • Review the settings and store the secret. Note the ARN of the secret you have created.

Setting up Amazon MQ for RabbitMQ

To get started with RabbitMQ on Amazon MQ, follow these steps to set up your broker.

• Open the Amazon MQ console.
• Create a new broker with the RabbitMQ engine.
• Choose your preferred deployment option—single-instance or clustered
• Use the same username and password that you previously stored in AWS Secrets Manager.
• Under Additional settings, enable CloudWatch Logs for observability.
• Configure access and security settings, ensuring that the broker is accessible to your AWS Lambda function.

• After the broker is created, note the following important details:

  • ARN of the RabbitMQ broker.
  • RabbitMQ web console URL.

• You’ll need the RabbitMQ log group ARN to set up Elastic’s Amazon MQ integration for RabbitMQ. Follow these steps to locate it:

  • Go to the General – Enabled Logs section of the broker. 
  • Copy the CloudWatch log group ARN.

Create a RabbitMQ Queue

Now that the RabbitMQ broker is configured, use the management console to create a queue where messages will be published.

• Access the RabbitMQ management console using the web console URL.
• Create a new queue (example: myQueue) to receive messages.

Build and deploy the AWS Lambda function

In this section, we'll set up the Lambda function using AWS SAM, add the message processing logic, and deploy it to AWS. This Lambda function will be responsible for consuming messages from RabbitMQ and logging audit events.

Before continuing, make sure you have completed the following prerequisites.

• AWS SAM prerequisites

• Install the AWS SAM CLI

Next, follow the steps outlined below to continue with the setup.

1. In your command line, run the command sam init from a directory of your choice.
2. The AWS SAM CLI will walk you through the setup.
   • Select AWS Quick Start Templates.
   • Choose the Hello World Example 
   • Use the Python runtime and zip package type.
   • Proceed with the default options.
   • Name your application as sample-rabbitmq-app.
   • The AWS SAM CLI downloads your starting template and creates the application project directory structure.
3. From your command line, move to the newly created sample-rabbitmq-app directory.
   • Replace the content of the hello_world/app.py file with the lambda function code for rabbitmq message processing.
   • In the template.yaml file, use the values mentioned below to update the file content.

     Resources: SampleRabbitMQApp:   Type: AWS::Serverless::Function   Properties:     CodeUri: hello_world/     Description: A starter AWS Lambda function.     MemorySize: 128     Timeout: 3     Handler: app.lambda_handler     Runtime: python3.10     PackageType: Zip     Policies:       - Statement:           - Effect: Allow             Resource: '*'             Action:               - mq:DescribeBroker               - secretsmanager:GetSecretValue               - ec2:CreateNetworkInterface               - ec2:DescribeNetworkInterfaces               - ec2:DescribeVpcs               - ec2:DeleteNetworkInterface               - ec2:DescribeSubnets               - ec2:DescribeSecurityGroups     Events:       MQEvent:         Type: MQ         Properties:           Broker: <ARN of the Broker>           Queues:             - myQueue           SourceAccessConfigurations:             - Type: BASIC_AUTH               URI: <ARN of the secret>
     

4. Run the command sam deploy --guided and wait for the confirmation message. This deploys all of the resources.

Sending Audit Events to RabbitMQ and Triggering Lambda

To test the end-to-end setup, simulate the flow by publishing audit event data into RabbitMQ using its web UI. Once the message is sent, it triggers the Lambda function. 

1. Navigate to the Amazon MQ console and select your newly created broker.

2. Locate and open the Rabbit web console URL
   

3. Under the Queues and Streams tab, select the target queue (example: myQueue).

4. Enter the message payload, and click Publish message to send it to the queue.
   Here’s a sample payload published via RabbitMQ:

   {
     "id": "txn-849302",
     "type": "audit",
     "payload": {
       "user_id": "u-10245",
       "event": "funds.transfer",
       "amount": 1200.75,
       "currency": "USD",
       "timestamp": "T14:20:15Z",
       "ip": "192.168.0.8",
       "location": "New York, USA"
     }
   }
   

5. Navigate to the AWS Lambda function created earlier.

6. Under the Monitor tab, click View CloudWatch logs.

7. Check the latest log stream to confirm that the Lambda was triggered by Amazon MQ and that the message was processed successfully.

Configuring Amazon MQ integration for Metrics and Logs collection

Elastic’s Amazon MQ integration simplifies the collection of logs and metrics from RabbitMQ brokers managed by Amazon MQ. Logs are ingested via Amazon CloudWatch Logs, while metrics are fetched from the specified AWS region at a defined interval.

Elastic provides a default configuration for metrics collection. You can accept these defaults or adjust settings such as the Collection Period to better fit your needs.

To enable the collection of logs:

1. Navigate to the Amazon MQ console and select the newly created broker.
2. Click the Logs hyperlink under the General – Enabled Logs section to open the detailed log settings page.
3. From this page, copy the CloudWatch log group ARN.
4. In Elastic, set up the Amazon MQ integration and paste the CloudWatch log group ARN.
5. Accept Defaults or Customize Settings – Elastic provides a default configuration for logs collection. You can accept these defaults or adjust settings such as collection intervals to better fit your needs.

Visualizing RabbitMQ Workloads with the Pre-Built Amazon MQ Dashboard

You can access the RabbitMQ dashboard by:

1. Navigate to the Dashboard Menu – Select the Dashboard menu option in Elastic and search for [Amazon MQ] RabbitMQ Overview to open the dashboard.

2. Navigate to the Integrations Menu – Open the Integrations menu in Elastic, select Amazon MQ, go to the Assets tab, and choose [Amazon MQ] RabbitMQ Overview from the dashboard assets

The Amazon MQ RabbitMQ dashboard in the Elastic integration delivers a comprehensive overview of broker health and messaging activity. It provides real-time insights into broker resource utilization, queue and topic performance, connection trends, and messaging throughput. The dashboard helps users track system behaviour, detect performance bottlenecks, and ensure reliable message delivery across distributed applications.

Broker Metrics

This section provides a centralised view of the overall health and performance of the RabbitMQ broker on Amazon MQ. The visualizations highlights the number of configured exchanges and queues, active broker connections, producers, consumers, and total messages in flight. System-level metrics such as CPU utilization, memory consumption, and free disk space help assess whether the broker has sufficient resources to handle current workloads.

Message flow metrics such as publish rate, confirmation rate, and acknowledgement rate are displayed to provide visibility into how messages are processed through the broker. Monitoring trends in these values helps detect message delivery issues, throughput degradation, or potential saturation of the broker under load.

Node Metrics

Node-level visibility helps identify resource imbalances across nodes in clustered RabbitMQ setups. This section includes per-node CPU usage, memory consumption, and available disk space, offering insight into the underlying infrastructure's ability to support broker operations.

Queue Metrics

Queue-specific insights are critical for understanding message delivery patterns and backlog conditions. This section details total messages, ready messages, and unacknowledged messages, segmented by broker, virtual host, and queue.

By observing how these counts change over time, users can identify slow consumers, message build-ups, or delivery issues that may affect application performance or lead to dropped messages under pressure.

Logs

This section displays log level, process ID, and raw message content. These logs provide immediate visibility into events such as connection failures, resource thresholds being hit, or unexpected queue behaviors.

Detecting Queue Backlogs with Alerting Rules

Elastic’s alert framework allows you to define rules that monitor critical RabbitMQ metrics and automatically trigger actions when specific thresholds are breached. 

Alert: Queue Backlog (Message Ready or Unacknowledged Messages)

This alert helps detect queue backlog in Amazon MQ by evaluating two metrics

• MessageUnacknowledgedCount.max and
• MessageReadyCount.max. 

The alert is triggered if either condition persists for more than 10 minutes:

• MessageUnacknowledgedCount.max exceeds 5,000
• MessageReadyCount.max exceeds 7,000

These thresholds should be adjusted based on typical message volume and consumer throughput. Sustained high values can indicate that consumers are not keeping up or message delivery pipelines are congested, potentially causing delays or dropped messages. Sustained high values may result in processing delays or dropped messages if not addressed.

Tracking Resource Utilization to Maintain RabbitMQ Performance

Elastic’s Service-level objectives (SLOs) capabilities allow you to define and monitor performance targets using key indicators like latency, availability, and error rates. Once configured, Elastic continuously evaluates these SLOs in real time, offering intuitive dashboards, alerts for threshold violations, and insights into error budget consumption. This enables teams to stay ahead of issues, ensuring service reliability and consistent performance.

SLO: Node Resource Health (CPU, Memory, Disk)

This SLO focuses on ensuring RabbitMQ brokers and nodes have sufficient resources to process messages without performance degradation. It tracks CPU, memory, and disk usage across RabbitMQ brokers and nodes to prevent resource exhaustion that could lead to service interruptions.

Target thresholds:

• SystemCpuUtilization.max remains below 85% for 99% of the time.
• RabbitMQMemUsed.max remains below 80% of RabbitMQMemLimit.max for 99% of the time.
• RabbitMQDiskFree.min remains above 25% of RabbitMQDiskFreeLimit.max for 99% of the time.

Sustained high values in CPU or memory usage can signal resource contention, which may result in slower message processing or downtime. Low disk availability may cause the broker to stop accepting messages, risking message loss. These thresholds are designed to catch early signs of resource saturation and ensure smooth, uninterrupted message flow across RabbitMQ deployments.

Conclusion

As RabbitMQ-based messaging architectures scale and become more complex, the need for in-depth visibility into system performance and potential issues deepens. Elastic’s Amazon MQ integration brings that visibility front and center—helping you go beyond basic health checks to understand real-time messaging throughput, queue backlog trends, and resource saturation across your brokers and consumers.

By leveraging the prebuilt dashboards, configuring alerts and SLOs, you can proactively detect anomalies, fine-tune consumer performance, and ensure reliable delivery across your event-driven applications.

In an era where APIs serve as the backbone of modern applications, having the means to maintain visibility and control over these vital components is absolutely essential. In this blog post, we dive deep into the comprehensive observability solution offered by Elastic^®, ensuring real-time visibility, advanced analytics, and actionable insights, empowering you to fine-tune your API Gateway for optimal performance.

For application owners and developers, this integration stands as a beacon of empowerment. Elastic's meticulous orchestration of the seamless merging of metrics, logs, and traces, built upon the robust ELK Stack foundation, equips them with potent real-time monitoring and analysis tools. These tools facilitate precise performance optimization and swift issue resolution, all within a secure and dependable environment.

With Elastic's AWS API Gateway integration, application owners and developers unlock the capability to proactively identify and resolve problems, fine-tune resource utilization, and provide extraordinary digital experiences to their users.

Architecture

Why the AWS API Gateway integration matters

API Gateway now serves as the foundation of contemporary application development, simplifying the process of creating and overseeing APIs on a large scale. Yet, monitoring and troubleshooting these API endpoints can be challenging. With the new AWS API Gateway integration introduced by Elastic, you can gain the following:

• Unprecedented visibility: Monitor your API Gateway endpoints' performance, error rates, and usage metrics in real time. Get a comprehensive view of your APIs' health and performance.
• Log analysis: Dive deep into API Gateway logs with ease. Our integration enables you to collect and analyze logs for HTTP, REST, and Websocket API types, helping you troubleshoot issues and gain valuable insights.
• Rapid issue resolution: Identify and resolve issues in your API Gateway workflows faster than ever. Elastic Observability's powerful search and analytics tools help you pinpoint problems with ease.
• Alerting and notifications: Set up custom alerts based on API Gateway metrics and logs. Receive notifications when performance thresholds are breached, ensuring that you can take action promptly.
• Optimized costs: Visualize resource usage and performance metrics for your API Gateway deployments. Use these insights to optimize resource allocation and reduce operational costs.
• Custom dashboards: Create customized dashboards and visualizations tailored to your API Gateway monitoring needs. Stay in control with real-time data and actionable insights.
• Effortless integration: Seamlessly connect your AWS API Gateway to our observability solution. Our intuitive setup process ensures a smooth integration experience.
• Scalability: Whether you have a handful of APIs or a complex API Gateway landscape, our observability solution scales to meet your needs. Grow confidently as your API infrastructure expands.

How to get started

Getting started with the AWS API Gateway integration in Elastic Observability is seamless. Here's a quick overview of the steps:

Prerequisites and configurations

If you intend to follow the steps outlined in this blog post, there are a few prerequisites and configurations that you should have in place beforehand.

1. You will need an account on Elastic Cloud and a deployed stack and agent. Instructions for deploying a stack on AWS can be found here. This is necessary for AWS API Gateway logging and analysis.

2. You will also need an AWS account with the necessary permissions to pull data from AWS. Details on the required permissions can be found in our documentation.

3. You can monitor API execution by using CloudWatch, which collects and processes raw data from API Gateway into readable, near-real-time metrics and logs. Details on the required steps to enable logging can be found here.

Step 1. Create an account with Elastic

Create an account on Elastic Cloud by following the steps provided.

Step 2. Add integration

• Log in to your Elastic Cloud deployment.

• Click on Add integrations. You will be navigated to a catalog of supported integrations.

• Search and select AWS API Gateway.

Step 3. Configure integration

• Click on the Add AWS API Gateway button and provide the required details.
• If this is your first time adding an AWS integration, you’ll need to configure and enroll the Elastic Agent on an AWS instance.

• Then complete the “Configure integration” form, providing all the necessary information required for agents to collect the AWS API Gateway metrics and associated CloudWatch logs. Multiple AWS credential methods are supported, including access keys, temporary security credentials, and IAM role ARN. Please see the IAM security and access documentation for more details. You can choose to collect API Gateway metrics, API Gateway logs via S3, or API Gateway logs via CloudWatch.
• Click on the Save and continue button at the bottom of the page.

Step 4. Analyze and monitor

Explore the data using the out-of-the-box dashboards available for the integration. Select Discover from the Elastic Cloud top-level menu.

Or, create custom dashboards, set up alerts, and gain actionable insights into your API Gateway service performance.

Here are key monitoring metrics collected through this integration across Rest APIs, HTTP APIs, and Websocket APIs:

• 4XXError – The number of client-side errors captured in a given period
• 5XXError – The number of server-side errors captured in a given period
• CacheHitCount – The number of requests served from the API cache in a given period
• CacheMissCount – The number of requests served from the backend in a given period, when API caching is enabled
• Count – The total number of API requests in a given period
• IntegrationLatency – The time between when API Gateway relays a request to the backend and when it receives a response from the backend
• Latency – The time between when API Gateway receives a request from a client and when it returns a response to the client — the latency includes the integration latency and other API Gateway overhead
• DataProcessed – The amount of data processed in bytes
• ConnectCount – The number of messages sent to the $connect route integration
  MessageCount – The number of messages sent to the WebSocket API, either from or to the client

Conclusion

The native integration of AWS API Gateway into Elastic Observability marks a significant advancement in streamlining the monitoring and management of your APIs. With this integration, you gain access to a wealth of insights, real-time visibility, and powerful analytics tools, empowering you to optimize your API performance, enhance security, and troubleshoot with ease. Don't miss out on this opportunity to take your API management to the next level, ensuring your digital assets operate at their best, all while providing a seamless experience for your users. Embrace this integration, and stay at the forefront of API observability in the ever-evolving world of digital technology.

Visit our documentation to learn more about Elastic Observability and the AWS API Gateway integration, or contact our sales team to get started!

Start a free trial today

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

To simplify this process and reduce management overhead, AWS users can now leverage the new Amazon Kinesis Firehose Delivery Stream to ingest logs into Elastic Cloud in AWS in real time and view them in the Elastic Stack alongside other logs for centralized analytics. This eliminates the necessity for time-consuming and expensive procedures such as VM provisioning or data shipper operations.

Elastic Observability unifies logs, metrics, and application performance monitoring (APM) traces for a full contextual view across your hybrid AWS environments alongside their on-premises data sets. Elastic Observability enables you to track and monitor performance across a broad range of AWS services, including AWS Lambda, Amazon Elastic Compute Cloud (EC2), Amazon Elastic Container Service (ECS), Amazon Elastic Kubernetes Service (EKS), Amazon Simple Storage Service (S3), Amazon Cloudtrail, Amazon Network Firewall, and more.

In this blog, we will walk you through how to use the Amazon Kinesis Data Firehose integration — Elastic is listed in the Amazon Kinesis Firehose drop-down list — to simplify your architecture and send logs to Elastic, so you can monitor and safeguard your multi-account AWS environments.

Announcing the Kinesis Firehose method

Elastic currently provides both agent-based and serverless mechanisms, and we are pleased to announce the addition of the Kinesis Firehose method. This new method enables customers to directly ingest logs from AWS into Elastic, supplementing our existing options.

• Elastic Agent pulls metrics and logs from CloudWatch and S3 where logs are generally pushed from a service (for example, EC2, ELB, WAF, Route53) and ingests them into Elastic Cloud.
• Elastic’s Serverless Forwarder (runs Lambda and available in AWS SAR) sends logs from Kinesis Data Stream, Amazon S3, and AWS Cloudwatch log groups into Elastic. To learn more about this topic, please see this blog post.
• Amazon Kinesis Firehose directly ingests logs from AWS into Elastic (specifically, if you are running the Elastic Cloud on AWS).

In this blog, we will cover the last option since we have recently released the Amazon Kinesis Data Firehose integration. Specifically, we'll review:

• A general overview of the Amazon Kinesis Data Firehose integration and how it works with AWS
• Step-by-step instructions to set up the Amazon Kinesis Data Firehose integration on AWS and on Elastic Cloud

By the end of this blog, you'll be equipped with the knowledge and tools to simplify your AWS log management with Elastic Observability and Amazon Kinesis Data Firehose.

Prerequisites and configurations

If you intend to follow the steps outlined in this blog post, there are a few prerequisites and configurations that you should have in place beforehand.

1. You will need an account on Elastic Cloud and a deployed stack on AWS. Instructions for deploying a stack on AWS can be found here. This is necessary for AWS Firehose Log ingestion.
2. You will also need an AWS account with the necessary permissions to pull data from AWS. Details on the required permissions can be found in our documentation.
3. Finally, be sure to turn on VPC Flow Logs for the VPC where your application is deployed and send them to AWS Firehose.

Elastic’s Amazon Kinesis Data Firehose integration

Elastic has collaborated with AWS to offer a seamless integration of Amazon Kinesis Data Firehose with Elastic, enabling direct ingestion of data from Amazon Kinesis Data Firehose into Elastic without the need for Agents or Beats. All you need to do is configure the Amazon Kinesis Data Firehose delivery stream to send its data to Elastic's endpoint. In this configuration, we will demonstrate how to ingest VPC Flow logs and Firewall logs into Elastic. You can follow a similar process to ingest other logs from your AWS environment into Elastic.

There are three distinct configurations available for ingesting VPC Flow and Network firewall logs into Elastic. One configuration involves sending logs through CloudWatch, and another uses S3 and Kinesis Firehose; each has its own unique setup. With Cloudwatch and S3 you can store and forward but with Kinesis Firehose you will have to ingest immediately. However, in this blog post, we will focus on this new configuration that involves sending VPC Flow logs and Network Firewall logs directly to Elastic.

We will guide you through the configuration of the easiest setup, which involves directly sending VPC Flow logs and Firewalls logs to Amazon Kinesis Data Firehose and then into Elastic Cloud.

Note: It's important to note that this setup is only compatible with Elastic Cloud on AWS and cannot be used with self-managed or on-premise or other cloud provider Elastic deployments.

Setting it all up

To begin setting up the integration between Amazon Kinesis Data Firehose and Elastic, let's go through the necessary steps.

Step 0: Get an account on Elastic Cloud

Create an account on Elastic Cloud by following the instructions provided to get started on Elastic Cloud.

Step 1: Deploy Elastic on AWS

You can deploy Elastic on AWS via two different approaches: through the UI or through Terraform. We’ll start first with the UI option.

After logging into Elastic Cloud, create a deployment on Elastic. It's crucial to make sure that the deployment is on Elastic Cloud on AWS since the Amazon Kinesis Data Firehose connects to a specific endpoint that must be on AWS.

After your deployment is created, it's essential to copy the Elasticsearch endpoint to ensure a seamless configuration process.

The Elasticsearch HTTP endpoint should be copied and used for Amazon Firehose destination configuration purposes, as it will be required. Here's an example of what the endpoint should look like:

https://elastic-O11y-log.es.us-east-1.aws.found.io

Alternative approach using Terraform

An alternative approach to deploying Elastic Cloud on AWS is by using Terraform. It's also an effective way to automate and streamline the deployment process.

To begin, simply create a Terraform configuration file that outlines the necessary infrastructure. This file should include resources for your Elastic Cloud deployment and any required IAM roles and policies. By using this approach, you can simplify the deployment process and ensure consistency across environments.

One easy way to create your Elastic Cloud deployment with Terraform is to use this Github repo. This resource lets you specify the region, version, and deployment template for your Elastic Cloud deployment, as well as any additional settings you require.

Step 2: To turn on Elastic's AWS integrations, navigate to the Elastic Integration section in your deployment

To install AWS assets in your deployment's Elastic Integration section, follow these steps:

1. Log in to your Elastic Cloud deployment and open Kibana.
2. To get started, go to the management section of Kibana and click on " Integrations."
3. Navigate to the AWS integration and click on the "Install AWS Assets" button in the settings.This step is important as it installs the necessary assets such as dashboards and ingest pipelines to enable data ingestion from AWS services into Elastic.

Step 3: Set up the Amazon Kinesis Data Firehose delivery stream on the AWS Console

You can set up the Kinesis Data Firehose delivery stream via two different approaches: through the AWS Management Console or through Terraform. We’ll start first with the console option.

To set up the Kinesis Data Firehose delivery stream on AWS, follow these steps:

1. Go to the AWS Management Console and select Amazon Kinesis Data Firehose.

2. Click on Create delivery stream.

3. Choose a delivery stream name and select Direct PUT or other sources as the source.

4. Choose Elastic as the destination.

5. In the Elastic destination section, enter the Elastic endpoint URL that you copied from your Elastic Cloud deployment.

6. Choose the content encoding and retry duration as shown above.

7. Enter the appropriate parameter values for your AWS log type. For example, for VPC Flow logs, you would need to specify the _ es_datastream_name _ and _ logs-aws.vpc flow-default _.

8. Configure the Amazon S3 bucket as the source backup for the Amazon Kinesis Data Firehose delivery stream failed data or all data, and configure any required tags for the delivery stream.

9. Review the settings and click on Create delivery stream.

In the example above, we are using the es_datastream_name parameter to pull in VPC Flow logs through the logs-aws.vpcflow-default datastream. Depending on your use case, this parameter can be configured with one of the following types of logs:

• logs-aws.cloudfront_logs-default (AWS CloudFront logs)
• logs-aws.ec2_logs-default (EC2 logs in AWS CloudWatch)
• logs-aws.elb_logs-default (Amazon Elastic Load Balancing logs)
• logs-aws.firewall_logs-default (AWS Network Firewall logs)
• logs-aws.route53_public_logs-default (Amazon Route 53 public DNS queries logs)
• logs-aws.route53_resolver_logs-default (Amazon Route 53 DNS queries & responses logs)
• logs-aws.s3access-default (Amazon S3 server access log)
• logs-aws.vpcflow-default (AWS VPC flow logs)
• logs-aws.waf-default (AWS WAF Logs)

Alternative approach using Terraform

Using the " aws_kinesis_firehose_delivery_stream" resource in Terraform is another way to create a Kinesis Firehose delivery stream, allowing you to specify the delivery stream name, data source, and destination - in this case, an Elasticsearch HTTP endpoint. To authenticate, you'll need to provide the endpoint URL and an API key. Leveraging this Terraform resource is a fantastic way to automate and streamline your deployment process, resulting in greater consistency and efficiency.

Here's an example code that shows you how to create a Kinesis Firehose delivery stream with Terraform that sends data to an Elasticsearch HTTP endpoint:

resource "aws_kinesis_firehose_delivery_stream" “Elasticcloud_stream" {
  name        = "terraform-kinesis-firehose-ElasticCloud-stream"
  destination = "http_endpoint”
  s3_configuration {
    role_arn           = aws_iam_role.firehose.arn
    bucket_arn         = aws_s3_bucket.bucket.arn
    buffer_size        = 5
    buffer_interval    = 300
    compression_format = "GZIP"
  }
  http_endpoint_configuration {
    url        = "https://cloud.elastic.co/"
    name       = “ElasticCloudEndpoint"
    access_key = “ElasticApi-key"
    buffering_hints {
      size_in_mb = 5
      interval_in_seconds = 300
    }

   role_arn       = "arn:Elastic_role"
   s3_backup_mode = "FailedDataOnly"
  }
}

Step 4: Configure VPC Flow Logs to send to Amazon Kinesis Data Firehose

To complete the setup, you'll need to configure VPC Flow logs in the VPC where your application is deployed and send them to the Amazon Kinesis Data Firehose delivery stream you set up in Step 3.

Enabling VPC flow logs in AWS is a straightforward process that involves several steps. Here's a step-by-step details to enable VPC flow logs in your AWS account:

1. Select the VPC for which you want to enable flow logs.

2. In the VPC dashboard, click on "Flow Logs" under the "Logs" section.

3. Click on the "Create Flow Log" button to create a new flow log.

4. In the "Create Flow Log" wizard, provide the following information:

Choose the target for your flow logs: In this case, Amazon Kinesis Data Firehose in the same AWS account.

• Provide a name for your flow log.
• Choose the VPC and the network interface(s) for which you want to enable flow logs.
• Choose the flow log format: either AWS default or Custom format.

5. Configure the IAM role for the flow logs. If you have an existing IAM role, select it. Otherwise, create a new IAM role that grants the necessary permissions for the flow logs.

6. Review the flow log configuration and click "Create."

Create the VPC Flow log.

Step 5: After a few minutes, check if flows are coming into Elastic

To confirm that the VPC Flow logs are ingesting into Elastic, you can check the logs in Kibana. You can do this by searching for the index in the Kibana Discover tab and filtering the results by the appropriate index and time range. If VPC Flow logs are flowing in, you should see a list of documents representing the VPC Flow logs.

Step 6: Navigate to Kibana to see your logs parsed and visualized in the [Logs AWS] VPC Flow Log Overview dashboard

Finally, there is an Elastic out-of-the-box (OOTB) VPC Flow logs dashboard that displays the top IP addresses that are hitting your VPC, their geographic location, time series of the flows, and a summary of VPC flow log rejects within the selected time frame. This dashboard can provide valuable insights into your network traffic and potential security threats.

Note: For additional VPC flow log analysis capabilities, please refer to this blog.

Step 7: Configure AWS Network Firewall Logs to send to Kinesis Firehose

To create a Kinesis Data Firehose delivery stream for AWS Network firewall logs, first log in to the AWS Management Console, navigate to the Kinesis service, select "Data Firehose", and follow the step-by-step instructions as shown in Step 3. Specify the Elasticsearch endpoint, API key, add a parameter (_ es_datastream_name=logs-aws.firewall_logs-default _), and create the delivery stream.

Second, to set up a Network Firewall rule group to send logs to the Kinesis Firehose, go to the Network Firewall section of the console, create a rule group, add a rule to allow traffic to the Kinesis endpoint, and attach the rule group to your Network Firewall configuration. Finally, test the configuration by sending traffic through the Network Firewall to the Kinesis Firehose endpoint and verify that logs are being delivered to your S3 bucket.

Kindly follow the instructions below to set up a firewall rule and logging.

1. Set up a Network Firewall rule group to send logs to Amazon Kinesis Data Firehose:

• Go to the AWS Management Console and select Network Firewall.
• Click on "Rule groups" in the left menu and then click "Create rule group."
• Choose "Stateless" or "Stateful" depending on your requirements, and give your rule group a name. Click "Create rule group."
• Add a rule to the rule group to allow traffic to the Kinesis Firehose endpoint. For example, if you are using the us-east-1 region, you would add a rule like this:json

{
  "RuleDefinition": {
    "Actions": [
      {
        "Type": "AWS::KinesisFirehose::DeliveryStream",
        "Options": {
          "DeliveryStreamArn": "arn:aws:firehose:us-east-1:12387389012:deliverystream/my-delivery-stream"
        }
      }
    ],
    "MatchAttributes": {
      "Destination": {
        "Addresses": ["api.firehose.us-east-1.amazonaws.com"]
      },
      "Protocol": {
        "Numeric": 6,
        "Type": "TCP"
      },
      "PortRanges": [
        {
          "From": 443,
          "To": 443
        }
      ]
    }
  },
  "RuleOptions": {
    "CustomTCPStarter": {
      "Enabled": true,
      "PortNumber": 443
    }
  }
}

• Save the rule group.

2. Attach the rule group to your Network Firewall configuration:

• Go to the AWS Management Console and select Network Firewall.
• Click on "Firewall configurations" in the left menu and select the configuration you want to attach the rule group to.
• Scroll down to "Associations" and click "Edit."
• Select the rule group you created in Step 2 and click "Save."

3. Test the configuration:

• Send traffic through the Network Firewall to the Kinesis Firehose endpoint and verify that logs are being delivered to your S3 bucket.

Step 8: Navigate to Kibana to see your logs parsed and visualized in the [Logs AWS] Firewall Log dashboard

Wrapping up

We’re excited to bring you this latest integration for AWS Cloud and Kinesis Data Firehose into production. The ability to consolidate logs and metrics to gain visibility across your cloud and on-premises environment is crucial for today’s distributed environments and applications.

From EC2, Cloudwatch, Lambda, ECS and SAR, Elastic Integrations allow you to quickly and easily get started with ingesting your telemetry data for monitoring, analytics, and observability. Elastic is constantly delivering frictionless customer experiences, allowing anytime, anywhere access to all of your telemetry data — this streamlined, native integration with AWS is the latest example of our commitment.

Start a free trial today

You can begin with a 7-day free trial of Elastic Cloud within the AWS Marketplace to start monitoring and improving your users' experience today!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic can make it easier for organizations to transform data into actionable insights and stop threats quickly with unified visibility across your environment — so mission-critical applications can keep running smoothly no matter what. From a free trial and fast deployment to sending logs to Elastic securely and frictionlessly, all you need to do is point and click to capture, store, and search data from your AWS services.

Monitoring EMR via Elastic Observability

In this article, we will delve into the following key aspects:

• Enabling EMR cluster metrics for Elastic integration: Learn the intricacies of configuring an EMR cluster to emit metrics that Elastic can effectively extract, paving the way for insightful analysis.
• Harnessing Kibana ^® dashboards for EMR workload analysis: Discover the potential of utilizing Kibana dashboards to dissect metrics related to an EMR workload. By gaining a deeper understanding, we open the doors to optimization opportunities.

Key benefits of AWS EMR integration

• Comprehensive monitoring: Monitor the health and performance of your EMR clusters in real time. Track metrics related to cluster status and utilization, node status, IO, and many others, allowing you to identify bottlenecks and optimize your data processing.
• Log analysis: Dive deep into EMR logs with ease. Our integration enables you to collect and analyze logs from your clusters, helping you troubleshoot issues and gain valuable insights.
• Cost optimization: Understand the cost implications of your EMR clusters. By monitoring resource utilization, you can identify opportunities to optimize your cluster configurations and reduce costs.
• Alerting and notifications: Set up custom alerts based on EMR metrics and logs. Receive notifications when performance thresholds are breached, ensuring that you can take action promptly.
• Seamless integration: Our integration is designed for ease of use. Getting started is simple, and you can start monitoring your EMR clusters quickly.

Accompanying these discussions is an illustrative solution architecture diagram, providing a visual representation of the intricacies and interactions within the proposed solution.

How to get started

Getting started with AWS EMR integration in Observability is easy. Here's a quick overview of the steps:

Prerequisites and configurations

If you intend to follow the steps outlined in this blog post, there are a few prerequisites and configurations that you should have in place beforehand.

1. You will need an account on Elastic Cloud and a deployed stack and agent. Instructions for deploying a stack on AWS can be found here. This is necessary for AWS EMR logging and analysis.

2. You will also need an AWS account with the necessary permissions to pull data from AWS. Details on the required permissions can be found in our documentation.

3. Finally, be sure to turn on EMR monitoring for the EMR cluster when you deploy the cluster.

Step 1: Create an account with Elastic

Create an account on Elastic Cloud by following the steps provided.

Step 2: Add integration

1. Log in to your Elastic Cloud on AWS deployment.

2. Click on Add Integration. You will be navigated to a catalog of supported integrations.

3. Search and select Amazon EMR.

Step 3: Configure integration

1. Click on the Add Amazon EMR button and provide the required details.

2. Provide the required access credentials to connect to your EMR instance.

3. You can choose to collect EMR metrics, EMR logs via S3, or EMR logs via Cloudwatch.

4. Click on the Save and continue button at the bottom of the page.

Step 4: Analyze and monitor

Explore the data using the out-of-the-box dashboards available for the integration. Select Discover from the Elastic Cloud top-level menu.

Or, create custom dashboards, set up alerts, and gain actionable insights into your EMR clusters' performance.

This integration streamlines the collection of vital metrics and logs, including Cluster Status, Node Status, IO, and Cluster Capacity. Some metrics gathered include:

• IsIdle: Indicates that a cluster is no longer performing work, but is still alive and accruing charges
• ContainerAllocated: The number of resource containers allocated by the ResourceManager
• ContainerReserved: The number of containers reserved
• CoreNodesRunning: The number of core nodes working
• CoreNodesPending: The number of core nodes waiting to be assigned
• MRActiveNodes: The number of nodes presently running MapReduce tasks or jobs
• MRLostNodes: The number of nodes allocated to MapReduce that have been marked in a LOST state
• HDFSUtilization: The percentage of HDFS storage currently used
• HDFSBytesRead/Written: The number of bytes read/written from HDFS (This metric aggregates MapReduce jobs only, and does not apply for other workloads on Amazon EMR.)
• TotalUnitsRequested/TotalNodesRequested/TotalVCPURequested: The target total number of units/nodes/vCPUs in a cluster as determined by managed scaling

Conclusion

Elastic is committed to fulfilling all your observability requirements, offering an effortless experience. Our integrations are designed to simplify the process of ingesting telemetry data, granting you convenient access to critical information for monitoring, analytics, and observability. The native AWS EMR integration underscores our dedication to delivering seamless solutions for your data needs. With this integration, you'll find the confidence to monitor, analyze, and optimize your EMR clusters, opening up exciting opportunities for your data-driven initiatives.

Start a free trial today

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Managing applications and the infrastructure they run on requires advanced observability into the diverse types of data involved like logs, traces, profiles, and metrics. General purpose generative AI large language models (LLMs) offer a new capability to provide human readable guidance to your observability questions. However, they have limitations. Specifically, when it comes to providing answers about your application’s distinct observability data like real-time metrics, the LLMs require additional context to provide answers that will help to actually resolve issues. This is a limitation that the Elastic AI Assistant for Observability can uniquely solve.

Elastic Observability, serving as a central datastore of all the observability data flowing from your application, combined with the Elastic AI Assistant gives you the ability to generate a context window that can inform an LLM’s responses and vastly improve the answers it provides. For example, when you ask the Elastic AI Assistant a question about a specific issue happening in your application, it gathers up all the relevant details — current errors captured from logs or a related runbook that your team has stored in the Elastic AI Assistant’s knowledge base. Then, it sends that information to the Amazon Bedrock LLM as a context window from which it can better answer your observability questions.

Read on to follow the steps for setting up the Elastic AI Assistant for yourself.

Set up the Elastic AI Assistant for Observability: Create an Amazon Bedrock connector in Elastic Cloud

Start by creating an Elastic Cloud 8.13 deployment via the AWS marketplace. If you’re a new user of Elastic Cloud, you can create a new deployment with a 7-day free trial.

Sign in to the Elastic Cloud deployment you’ve created. From the top level menu, select Stack Management.

Select Connectors.

Click the Create connector button.

Enable Amazon Bedrock model access

For populating the required connector settings, enable Amazon Bedrock model access in the AWS console using the following steps.

In a new browser tab, open Amazon Bedrock and click the Get started button.

Currently, access to the Amazon Bedrock foundation models is granted by requesting access using the Bedrock Model access section in the AWS console.

Select Model access from the navigation menu.

To request access, select the foundation models that you want to access and click the Save Changes button. For this blog post, we will choose the Anthropic Claude models.

Once access is granted, the Manage model access settings will indicate that access has been granted.

Create AWS IAM User

Create an IAM user and assign it a role with Amazon Bedrock full access and also generate an IAM access key and secret key in the console. If you already have an IAM user with a generated access key and secret key, you can use the existing credentials to access Amazon Bedrock.

Configure Elastic connector to use Amazon Bedrock

Back in the Elastic Cloud deployment create connector flyout, select the connector for Amazon Bedrock.

Enter a Name of your choice for the connector. Also, enter the Access Key and Key Secret that you copied in a previous step. Click the Save & test button to create the connector.

Within the Edit Connector flyout window, click the Run button to confirm that the connector configuration is valid and can successfully connect to your Amazon Bedrock instance.

You should see confirmation that the connector test was successful.

Add an example logs record

Now that the connector is configured, let's add a logs record to demonstrate how the Elastic AI Assistant can help you to better understand the diverse types of information contained within logs.

Use the Elastic Dev Tools to add a single logs record. Click the top-level menu and select Dev Tools.

Within the console area of Dev Tools, enter the following POST statement:

POST /logs-elastic_agent-default/_doc
{
    "message": "Status(StatusCode=\"BadGateway\", Detail=\"Error: The server encountered a temporary error and could not complete your request\").",
    "@timestamp": "2024-04-21T10:33:00.884Z",
    "log": {
   	 "level": "error"
    },
    "service": {
   	 "name": "proxyService"
    },
    "host": {
   	 "name": "appserver-2"
    }
}

Then run the POST command by clicking the green Run button.

You should see a 201 response confirming that the example logs record was successfully created.

Use the Elastic AI Assistant

Now that you have a log entry, let’s use the AI Assistant to see how it interacts with logs data. Click the top-level menu and select Observability.

Select Logs Explorer under Observability.

In the Logs Explorer search box, enter the text “badgateway” and press the Enter key to perform the search.

Click the View all matches button to include all search results.

You should see the one log record that you previously inserted via Dev Tools. Click the expand icon in the actions column to see the log record’s details.

You should see the expanded view of the logs record. Let’s use the AI Assistant to summarize it. Click on the What's this message? button.

We get a fairly generic answer back. Depending on the exception or error we're trying to analyze, this can still be really useful, but we can improve this response by adding additional documentation to the AI Assistant knowledge base.

Let’s add an entry in AI Assistant’s knowledge base to improve its understanding of this specific logs message.

Click the AI Assistant button at the top right of the window.

Click the Install Knowledge base button.

Click the top-level menu and select Stack Management.

Then select AI Assistants.

Click Elastic AI Assistant for Observability.

Select the Knowledge base tab.

Click the New entry button and select Single entry.

Give it the Name “proxyservice” and enter the following text as the Contents :

​​I have the following runbook located on Github. Store this information in your knowledge base and always include the link to the runbook in your response if the topic is related to a bad gateway error.

Runbook Link: https://github.com/elastic/observability-aiops/blob/main/ai_assistant/runbooks/slos/502-errors.md

Runbook Title: Handling 502 Bad Gateway Errors

Summary: This is likely an issue with Nginx proxy configuration

Body: This runbook provides instructions for diagnosing and resolving 502 Bad Gateway errors in your system.

Click Save to save the new knowledge base entry.

Now let’s go back to the Observability Logs Explorer. Click the top-level menu and select Observability.

Then select Explorer under Logs.

Expand the same logs entry as you did previously and click the What’s this message? button.

The response you get now should be much more relevant.

Try out the Elastic AI Assistant with a knowledge base filled with your own data

Now you’ve seen the complete process of connecting the Elastic AI Assistant to Amazon Bedrock. You’ve also seen how to use the AI Assistant’s knowledge base to store custom remediation documentation like runbooks that the AI Assistant can leverage to generate more helpful responses. Steps like this can help you remediate issues more quickly when they happen. Try out the Elastic AI Assistant with your own logs and custom knowledge base.

Start a 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.