Elastic and AWS Serverless Application Repository (SAR): Speed time to actionable insights with frictionless log ingestion from Amazon S3

blog-thumb-cloud-aws.png

As companies leverage the Amazon Web Services (AWS) cloud platform and services to drive operational efficiency and bring products to market, logs are often stored in Amazon Simple Storage Service (Amazon S3) then shipped to an external monitoring and analysis solution. Now AWS users can quickly ingest logs stored in Amazon S3 with the new Elastic serverless forwarder, an AWS Lambda application, and view them in the Elastic Stack alongside other logs and metrics for centralized analytics. 

Skip lengthy processes like provisioning a VM or installing data shippers — and reduce management overhead by ingesting data directly from AWS to Elastic. 

In this blog, we will teach you how to use the Elastic serverless forwarder —  that is published in the AWS Serverless Application Repository (SAR ) — to simplify your architecture and send logs to Elastic, so you can monitor and safeguard your multi-cloud and on-premises environments.

Monitor the health and performance of your AWS environment

In an increasingly complex hybrid and multi-cloud ecosystem, it is no surprise observability continues to be a critical business initiative and the number one challenge for DevOps teams, according to research from the Enterprise Management Associates (EMA) group. As many organizations choose various technologies from containers to serverless computing to bring products to market faster and reduce overhead, it is important to note the need for an observability solution that covers all architectures. Teams that deploy a comprehensive observability solution are able to develop 70% faster and maintain increased product velocity with four times the number of features, according to EMA research.

Elastic Observability unifies logs, metrics, and APM traces for a full contextual view across your hybrid AWS environments alongside their on-premises data sets —at scale— in a single stack. Track performance and monitor across a broad range of AWS services including AWS Lambda, Amazon Elastic Compute Cloud (EC2), Amazon Elastic Container Service (ECS), Amazon Elastic Kubernetes Service (EKS), Amazon S3, and more.

Elastic Observability platform detail and capabilities

Equip security teams to stop threats quickly and at cloud scale

A commissioned Forrester study showed that customers achieved up to 75% cost savings using Elastic Security and Observability solutions together, and was up ten times faster than incumbent solutions. With the Elastic Common Schema and single repository architecture, the same observability data from Amazon S3 and other data sets can also be used for extended detection and response (XDR) to drive mean time to detection towards zero. Elastic Security brings together SIEM and endpoint security, allowing organizations to ingest and retain large volumes of data from diverse sources, store and search data for longer, and augment threat hunting with detections and machine learning. Eliminate data silos, reduce alert fatigue, and ready the organization to quickly stop threats across their environment.

Store data cost-effectively for fast retrieval and future analysis

There is another way to leverage your Amazon S3 for cost efficiency. In addition to ingesting logs stored in S3 to Elastic, Elastic also enables organizations to retain large amounts of historical data in low-cost object storage like Amazon S3 — while keeping it fully active and searchable. Keep AWS and on-premises data — at any granularity, for any length of time — and then scale as the data grows. Data management and tiering is automated through index lifecycle management and autoscaling capabilities, based on the organization’s data performance, resilience, and retention requirements.

Simplify data ingestion

The Elastic serverless forwarder Lambda application supports ingesting logs contained in the Amazon S3 bucket and sends them to Elastic. The SQS queue event notification on Amazon S3 serves as a trigger for the Lambda function. When a new log file gets written to an Amazon S3 bucket and meets the criteria, a notification is generated that triggers the Lambda function.

Users can set up the SQS function trigger on their S3 bucket and provide Elastic connection information to let the logs flow and use the prebuilt dashboards and full analytics features of Kibana to bring logs data to life.

Architecture diagram:

architecture diagram

Let’s get started

In this section, we’ll go into a step-by-step tutorial on how to get started with the Elastic serverless forwarder to analyze Amazon Virtual Private Cloud (Amazon VPC) Flow Logs in the Elastic Stack.

For more detailed instructions review the Elastic serverless forwarder documentation.

Ingesting Amazon VPC Flow Logs into Elastic enables you to monitor and analyze network traffic within your Amazon VPC and make more informed decisions by:

  • Analyzing the flow log data in Kibana with the ability to quickly search, view, and filter logs
  • Assessing security groups rules and uncover security gaps
  • Setting alarms that alert you when certain traffic types are detected
  • Identifying latency issues and establish baselines to ensure consistent performance


Before you begin

  1. If you are not already using Elastic, create a deployment using our hosted Elasticsearch Service on Elastic Cloud. The deployment includes an Elasticsearch cluster for storing and searching your data, and Kibana for visualizing and managing your data. For more information, see Spin up the Elastic Stack.

  2. Enable AWS VPC flow logs to be sent to an S3 bucket. If you don’t have that setup you can easily create an S3 bucket and send VPC flow logs to that bucket. The steps will essentially be:
    • Create an S3 bucket (Example: vpc-flow-logs)
    • On the EC2 console select specific network interfaces and from Actions menu “create flow log”. Select the destination as the S3 bucket you created in the previous steps. For more details, review the AWS documentation.

  3. Now, let’s create an SQS simple queue (Example: flow-logs-queue) and set up an appropriate access policy so that S3 event notifications from S3 are sent to the queue. On the S3 bucket (vpc-flow-logs) configure event notifications for all objects “create events” to be sent to the SQS queue (flow-logs-queue). For more details, review the AWS documentation.

  4. Next, you’ll start with installing the Elastic AWS integration straight from the Kibana web UI, which contains prebuilt dashboards, ingest node configurations, and other assets that help you get the most value out of the logs you ingest.
Go to Integrations in Kibana and search for AWS. Click the AWS integration to see more details, select Settings and click Install AWS assets to install all the AWS integration assets.
install AWS assets

5. Deploy the elastic-serverless-forwarder from AWS SAR and provide appropriate configurations for the Lambda function to start ingesting VPC flow logs into Elastic.

From the Lambda console select Functions->Create a function, select Browse serverless app repository and search for elastic-serverless-forwarder, Select and Deploy the application.

configuration
6. Next let’s create a new S3 bucket and a configuration file that elastic-serverless-forwarder will use to know the input source and the Elastic connection for destination information.
S3 bucket

Here is a sample configuration file:

configuration file

Go to Elastic Cloud and copy Cloud ID from the Elastic cloud console to specify in the parameter “cloud_id”. Navigate to Kibana and create a Base64 encoded API key for authentication and specify in the parameter “api_key”. You should store any sensitive values in AWS Secrets Manager and refer to it from the configuration file.

7. After the Lambda deployment is complete, select the deployed Lambda function and go to the Configuration->Environment variables tab to add the environment variable S3_CONFIG_FILE. The value will be the S3 url in the format "s3://bucket-name/config-file-name" pointing to the configuration file (sarconfig.yaml) that you created in the last step.

configuration environment

8. Setup additional IAM policies to grant minimum permissions required for the Lambda to be able to use the configured continuing SQS queue, S3 buckets, Secrets Manager (optional) and replay SQS queue. The Execution role associated with your function can be seen in the Configuration->Permissions section and by default starts with the name “serverlessrepo-elastic-se-ElasticServerlessForward-”. On top of the basic permissions the following policies must be provided to the Execution role of the Lambda function. For more details review Lambda IAM permissions and policies section in the documentation.

  • For the SQS queue resources that are specified in SQS_CONTINUE_URL (continuing SQS queue) and SQS_REPLAY_URL (replay SQS queue) environment variable, make sure “sqs:SendMessage” permission is granted. The continuing SQS queue and replay SQS queue is set up by the Lambda automatically at deployment time and its URL is available in the Configuration->Environment variables section.
  • For the S3 bucket resource file that's set in the S3_CONFIG_FILE environment variable make sure “s3:GetObject” permission is granted.
  • For the S3 bucket resource that contains the VPC flow logs make sure “s3:GetObject” permission is granted for all objects.
  • For the SQS queue resource that you use as triggers of the Lambda function make sure “sqs:GetQueueUrl” permission is granted.
policies

9. In the Lambda Configuration->Triggers section add SQS queue (flow-logs-queue) as the Lambda function trigger.

trigger

The deployed Lambda will read the VPC flow log files as they get written to the S3 bucket and send it to Elastic.

10. Navigate to Kibana to see your logs parsed and visualized in the [Logs AWS] VPC Flow Log Overview dashboard.

VPC flow log

Wrapping up

Elastic is constantly delivering frictionless customer experiences, allowing anytime, anywhere access — and this streamlined integration with AWS is the latest example of that. For more information visit the elastic-serverless-forwarder documentation or download the Elastic Observability guide for AWS.

Start a free trial today

You can begin with a 7-day free trial of Elastic Cloud within the AWS Marketplace to start monitoring and improving your users' experience today!



The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

  • 채용 공고

    여러분과 같은 사람을 찾는 글로벌 분산형 팀에서 일하세요. Zoom 미팅만 하면 가능합니다. 유연하고 임팩트가 있는 작업을 원하세요? 처음부터 개발 기회를 갖고 싶으신가요?