Spike in AWS Error Messagesedit

A machine learning job detected a significant spike in the rate of a particular error in the CloudTrail messages. Spikes in error messages may accompany attempts at privilege escalation, lateral movement, or discovery.

Rule type: machine_learning

Machine learning job: high_distinct_count_error_message

Machine learning anomaly threshold: 50

Severity: low

Risk score: 21

Runs every: 15 minutes

Searches indices from: now-60m (Date Math format, see also Additional look-back time)

Maximum alerts per execution: 100

References:

Tags:

  • Elastic
  • Cloud
  • AWS
  • ML

Version: 7 (version history)

Added (Elastic Stack release): 7.9.0

Last modified (Elastic Stack release): 7.16.0

Rule authors: Elastic

Rule license: Elastic License v2

Potential false positivesedit

Spikes in error message activity can also be due to bugs in cloud automation scripts or workflows; changes to cloud automation scripts or workflows; adoption of new services; changes in the way services are used; or changes to IAM privileges.

Investigation guideedit

## Config

The AWS Fleet integration, Filebeat module, or similarly structured data is required to be compatible with this rule.

## Triage and analysis

### Investigating Spikes in CloudTrail Errors

CloudTrail logging provides visibility on actions taken within an AWS environment. By monitoring these events and understanding
what is considered normal behavior within an organization, suspicious or malicious activity can be spotted when deviations
are observed. This example rule triggers from a large spike in the number of CloudTrail log messages that contain a
particular error message. The error message in question was associated with the response to an AWS API command or method call,
this has the potential to uncover unknown threats or activity.

#### Possible investigation steps:
- Examine the history of the error. Has it manifested before? If the error, which is visible in the `aws.cloudtrail.error_message` field, only manifested recently, it might be related to recent changes in an automation module or script.
- Examine the request parameters. These may provide indications as to the nature of the task being performed when the error occurred. Is the error related to unsuccessful attempts to enumerate or access objects, data, or secrets? If so, this can sometimes be a byproduct of discovery, privilege escalation or lateral movement attempts.
- Consider the user as identified by the `user.name field`. Is this activity part of an expected workflow for the user context? Examine the user identity in the `aws.cloudtrail.user_identity.arn` field and the access key ID in the `aws.cloudtrail.user_identity.access_key_id` field, which can help identify the precise user context. The user agent details in the `user_agent.original` field may also indicate what kind of a client made the request.
- Consider the source IP address and geolocation for the calling user who issued the command. Do they look normal for the calling user? If the source is an EC2 IP address, is it associated with an EC2 instance in one of your accounts, or could it be sourcing from an EC2 instance that's not under your control? If it is an authorized EC2 instance, is the activity associated with normal behavior for the instance role or roles? Are there any other alerts or signs of suspicious activity involving this instance?

### False Positive Analysis
- This rule has the possibility to produce false positives based on unexpected activity occurring such as bugs or recent
changes to automation modules or scripting.
- Adoption of new services or implementing new functionality to scripts may generate false positives

### Related Rules
- Unusual AWS Command for a User
- Rare AWS Error Code

### Response and Remediation
- If activity is observed as suspicious or malicious, immediate response should be looked into rotating and deleting AWS IAM access keys
- Validate if any unauthorized new users were created, remove these accounts and request password resets for other IAM users
- Look into enabling multi-factor authentication for users
- Follow security best practices [outlined](https://aws.amazon.com/premiumsupport/knowledge-center/security-best-practices/) by AWS

Rule version historyedit

Version 7 (7.16.0 release)
  • Formatting only
Version 6 (7.15.0 release)
  • Formatting only
Version 5 (7.14.0 release)
  • Formatting only
Version 4 (7.13.0 release)
  • Formatting only
Version 3 (7.12.0 release)
  • Formatting only
Version 2 (7.10.0 release)
  • Formatting only