Auditing Security Events

You can enable auditing to keep track of security-related events such as authentication failures and refused connections. Logging these events enables you to monitor your cluster for suspicious activity and provides evidence in the event of an attack.

Important

Audit logs are disabled by default. To enable this functionality, you must set xpack.security.audit.enabled to true in elasticsearch.yml.

X-Pack security provides two ways to persist audit logs:

  • The logfile output, which persists events to a dedicated <clustername>_access.log file on the host’s file system.
  • The index output, which persists events to an Elasticsearch index. The audit index can reside on the same cluster, or a separate cluster.

By default, only the logfile output is used when enabling auditing. To facilitate browsing and analyzing the events, you can also enable indexing by setting xpack.security.audit.outputs in elasticsearch.yml:

xpack.security.audit.outputs: [ index, logfile ]

The index output type should be used in conjunction with the logfile output type Because it is possible for the index output type to lose messages if the target index is unavailable, the access.log should be used as the official record of events.

Note

Audit events are batched for indexing so there is a lag before events appear in the index. You can control how frequently batches of events are pushed to the index by setting xpack.security.audit.index.flush_interval in elasticsearch.yml.

Audit Event Types

Each request may generate multiple audit events. The following is a list of the events that can be generated:

anonymous_access_denied

Logged when a request is denied due to a missing authentication token.

authentication_success

Logged when a user successfully authenticates.

authentication_failed

Logged when the authentication token cannot be matched to a known user.

realm_authentication_failed

Logged for every realm that fails to present a valid authentication token. <realm> represents the realm type.

access_denied

Logged when an authenticated user attempts to execute an action they do not have the necessary privilege to perform.

access_granted

Logged when an authenticated user attempts to execute an action they have the necessary privilege to perform. When the system_access_granted event is included, all system (internal) actions are also logged. The default setting does not log system actions to avoid cluttering the logs.

run_as_granted

Logged when an authenticated user attempts to run as another user that they have the necessary privileges to do.

run_as_denied

Logged when an authenticated user attempts to run as another user action they do not have the necessary privilege to do so.

tampered_request

Logged when X-Pack security detects that the request has been tampered with. Typically relates to search/scroll requests when the scroll ID is believed to have been tampered with.

connection_granted

Logged when an incoming TCP connection passes the IP Filter for a specific profile.

connection_denied

Logged when an incoming TCP connection does not pass the IP Filter for a specific profile.

Audit Event Attributes

The following table shows the common attributes that can be associated with every event.

Table 12. Common Attributes

Attribute Description

timestamp

When the event occurred.

node_name

The name of the node.

node_host_name

The hostname of the node.

node_host_address

The IP address of the node.

layer

The layer from which this event originated: rest, transport or ip_filter

event_type

The type of event that occurred: anonymous_access_denied, authentication_failed, access_denied, access_granted, connection_granted, connection_denied, tampered_request, run_as_granted, run_as_denied.


The following tables show the attributes that can be associated with each type of event. The log level determines which attributes are included in a log entry.

Table 13. REST anonymous_access_denied Attributes

Attribute Description

origin_address

The IP address from which the request originated.

uri

The REST endpoint URI.

request_body

The body of the request, if enabled.


Table 14. REST authentication_success Attributes

Attribute Description

user

The authenticated user.

realm

The realm that authenticated the user.

uri

The REST endpoint URI.

params

The REST URI query parameters.

request_body

The body of the request, if enabled.


Table 15. REST authentication_failed Attributes

Attribute Description

origin_address

The IP address from which the request originated.

principal

The principal (username) that failed authentication.

uri

The REST endpoint URI.

request_body

The body of the request, if enabled.


Table 16. REST realm_authentication_failed Attributes

Attribute Description

origin_address

The IP address from which the request originated.

principal

The principal (username) that failed authentication.

uri

The REST endpoint URI.

request_body

The body of the request, if enabled.

realm

The realm that failed to authenticate the user. NOTE: A separate entry is logged for each consulted realm.


Table 17. Transport anonymous_access_denied Attributes

Attribute Description

origin_type

Where the request originated: rest (request originated from a REST API request), transport (request was received on the transport channel), local_node (the local node issued the request).

origin_address

The IP address from which the request originated.

action

The name of the action that was executed.

request

The type of request that was executed.

indices

A comma-separated list of indices this request pertains to (when applicable).


Table 18. Transport authentication_success Attributes

Attribute Description

origin_type

Where the request originated: rest (request originated from a REST API request), transport (request was received on the transport channel), local_node (the local node issued the request).

origin_address

The IP address from which the request originated.

user

The authenticated user.

realm

The realm that authenticated the user.

action

The name of the action that was executed.

request

The type of request that was executed.


Table 19. Transport authentication_failed Attributes

Attribute Description

origin_type

Where the request originated: rest (request originated from a REST API request), transport (request was received on the transport channel), local_node (the local node issued the request).

origin_address

The IP address from which the request originated.

principal

The principal (username) that failed authentication.

action

The name of the action that was executed.

request

The type of request that was executed.

indices

A comma-separated list of indices this request pertains to (when applicable).


Table 20. Transport realm_authentication_failed Attributes

Attribute Description

origin_type

Where the request originated: rest (request originated from a REST API request), transport (request was received on the transport channel), local_node (the local node issued the request).

origin_address

The IP address from which the request originated.

principal

The principal (username) that failed authentication.

action

The name of the action that was executed.

request

The type of request that was executed.

indices

A comma-separated list of indices this request pertains to (when applicable).

realm

The realm that failed to authenticate the user. NOTE: A separate entry is logged for each consulted realm.


Table 21. Transport access_granted Attributes

Attribute Description

origin_type

Where the request originated: rest (request originated from a REST API request), transport (request was received on the transport channel), local_node (the local node issued the request).

origin_address

The IP address from which the request originated.

principal

The principal (username) that passed authentication.

action

The name of the action that was executed.

request

The type of request that was executed.

indices

A comma-separated list of indices this request pertains to (when applicable).


Table 22. Transport access_denied Attributes

Attribute Description

origin_type

Where the request originated: rest (request originated from a REST API request), transport (request was received on the transport channel), local_node (the local node issued the request).

origin_address

The IP address from which the request originated.

principal

The principal (username) that failed authentication.

action

The name of the action that was executed.

request

The type of request that was executed.

indices

A comma-separated list of indices this request relates to (when applicable).


Table 23. Transport tampered_request Attributes

Attribute Description

origin_type

Where the request originated: rest (request originated from a REST API request), transport (request was received on the transport channel), local_node (the local node issued the request).

origin_address

The IP address from which the request originated.

principal

The principal (username) that failed to authenticate.

action

The name of the action that was executed.

request

The type of request that was executed.

indices

A comma-separated list of indices this request pertains to (when applicable).


Table 24. IP Filter connection_granted Attributes

Attribute Description

origin_address

The IP address from which the request originated.

transport_profile

The transport profile the request targeted.

rule

The IP filtering rule that granted the request.


Table 25. IP Filter connection_denied Attributes

Attribute Description

origin_address

The IP address from which the request originated.

transport_profile

The transport profile the request targeted.

rule

The IP filtering rule that denied the request.


Logfile Audit Output

The logfile audit output is the default output for auditing. It writes data to the <clustername>_access.log file in the logs directory.

Log Entry Format

The format of a log entry is:

[<timestamp>] [<local_node_info>] [<layer>] [<entry_type>] <attribute_list>
<timestamp>
When the event occurred. You can configure the timestamp format in log4j2.properties.
<local_node_info>
Information about the local node that generated the log entry. You can control what node information is included by configuring the local node info settings.
<layer>
The layer from which this event originated: rest, transport or ip_filter.
<entry_type>
The type of event that occurred: anonymous_access_denied, authentication_failed, access_denied, access_granted, connection_granted, connection_denied.
<attribute_list>
A comma-separated list of key-value pairs that contain data pertaining to the event. Formatted as attr1=[val1], attr2=[val2]. See Audit Entry Attributes for the attributes that can be included for each type of event.

Logfile Output Settings

The events and some other information about what gets logged can be controlled using settings in the elasticsearch.yml file.

Table 26. Audited Event Settings

Name Default Description

xpack.security.audit.logfile.events.include

access_denied, access_granted, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, run_as_granted

Includes the specified events in the output.

xpack.security.audit.logfile.events.exclude

Excludes the specified events from the output.

xpack.security.audit.logfile.events.emit_request_body

false

Include or exclude the request body from REST requests on certain event types such as authentication_failed.


Important

No filtering is performed when auditing, so sensitive data may be audited in plain text when including the request body in audit events.

Table 27. Local Node Info Settings

Name Default Description

xpack.security.audit.logfile.prefix.emit_node_name

true

Include or exclude the node’s name from the local node info.

xpack.security.audit.logfile.prefix.emit_node_host_address

false

Include or exclude the node’s IP address from the local node info.

xpack.security.audit.logfile.prefix.emit_node_host_name

false

Include or exclude the node’s host name from the local node info.


You configure also configure how the logfile is written in the log4j2.properties file located in CONFIG_DIR/x-pack. By default, audit information is appended to the <clustername>_access.log file located in the standard Elasticsearch logs directory (typically located at $ES_HOME/logs). The file rolls over on a daily basis.

Index Audit Output

In addition to logging to a file, you can store audit logs in Elasticsearch rolling indices. These indices can be either on the same cluster, or on a remote cluster. You configure the following settings in elasticsearch.yml to control how audit entries are indexed. To enable this output, you need to configure the setting xpack.security.audit.outputs in the elasticsearch.yml file:

xpack.security.audit.outputs: [ index, logfile ]

Table 28. Audit Log Indexing Configuration

Attribute Default Setting Description

xpack.security.audit.index.bulk_size

1000

Controls how many audit events are batched into a single write.

xpack.security.audit.index.flush_interval

1s

Controls how often buffered events are flushed to the index.

xpack.security.audit.index.rollover

daily

Controls how often to roll over to a new index: hourly, daily, weekly, or monthly.

xpack.security.audit.index.events.include

anonymous_access_denied, authentication_failed, realm_authentication_failed, access_granted, access_denied, tampered_request, connection_granted, connection_denied, run_as_granted, run_as_denied

The audit events to be indexed. See Audit Entry Types for the complete list.

xpack.security.audit.index.events.exclude

The audit events to exclude from indexing.

xpack.security.audit.index.events.emit_request_body

false

Include or exclude the request body from REST requests on certain event types such as authentication_failed.


Important

No filtering is performed when auditing, so sensitive data may be audited in plain text when including the request body in audit events.

Audit Index Settings

You can also configure settings for the indices that the events are stored in. These settings are configured in the xpack.security.audit.index.settings namespace in elasticsearch.yml. For example, the following configuration sets the number of shards and replicas to 1 for the audit indices:

xpack.security.audit.index.settings:
  index:
    number_of_shards: 1
    number_of_replicas: 1

Forwarding Audit Logs to a Remote Cluster

To index audit events to a remote Elasticsearch cluster, you configure the following xpack.security.audit.index.client settings.

Table 29. Remote Audit Log Indexing Configuration

Attribute Description

xpack.security.audit.index.client.hosts

Comma-separated list of host:port pairs. These hosts should be nodes in the remote cluster.

xpack.security.audit.index.client.cluster.name

The name of the remote cluster.

xpack.security.audit.index.client.xpack.security.user

The username:password pair to use to authenticate with the remote cluster.


You can pass additional settings to the remote client by specifying them in the xpack.security.audit.index.client namespace. For example, to allow the remote client to discover all of the nodes in the remote cluster you can specify the client.transport.sniff setting:

xpack.security.audit.index.client.transport.sniff: true