Docs
You are looking at preliminary documentation for a future release. Not what you want? See the current release documentation.
Elasticsearch Reference [master] » EQL for event-based search » Run an EQL search
« EQL requirements EQL syntax reference »

Run an EQL searchedit

This functionality is in development and may be changed or removed completely in a future release. These features are unsupported and not subject to the support SLA of official GA features.

To start using EQL in Elasticsearch, first ensure your event data meets EQL requirements. You can then use the EQL search API to search event data stored in one or more Elasticsearch indices.

Example

To get started, ingest or add the data to an Elasticsearch index.

The following bulk API request adds some example log data to the sec_logs index. This log data follows the Elastic Common Schema (ECS).

PUT /sec_logs/_bulk?refresh
{"index":{"_index" : "sec_logs", "_id" : "1"}}
{ "@timestamp": "2020-12-06T11:04:05.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "2"}}
{ "@timestamp": "2020-12-06T11:04:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file" }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "3"}}
{ "@timestamp": "2020-12-07T11:06:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "4"}}
{ "@timestamp": "2020-12-07T11:07:08.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file" }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "5"}}
{ "@timestamp": "2020-12-07T11:07:09.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "regsvr32.exe", "path": "C:\\Windows\\System32\\regsvr32.exe" } }

You also can set up Beats, such as Auditbeat or Winlogbeat, to automatically send and index your event data in Elasticsearch. See Getting started with Beats.

You can now use the EQL search API to search this index using an EQL query.

The following request searches the sec_logs index using the EQL query specified in the query parameter. The EQL query matches events with an event.category of process that have a process.name of cmd.exe.

GET /sec_logs/_eql/search
{
  "query": """
    process where process.name == "cmd.exe"
  """
}

Because the sec_log index follows the ECS, you don’t need to specify the required event category or timestamp fields. The request uses the event.category and @timestamp fields by default.

The API returns the following response containing the matching events. Events in the response are sorted by timestamp, converted to milliseconds since the Unix epoch, in ascending order.

{
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "events": [
      {
        "_index": "sec_logs",
        "_id": "1",
        "_score": null,
        "_source": {
          "@timestamp": "2020-12-06T11:04:05.000Z",
          "agent": {
            "id": "8a4f500d"
          },
          "event": {
            "category": "process"
          },
          "process": {
            "name": "cmd.exe",
            "path": "C:\\Windows\\System32\\cmd.exe"
          }
        },
        "sort": [
          1607252645000
        ]
      },
      {
        "_index": "sec_logs",
        "_id": "3",
        "_score": null,
        "_source": {
          "@timestamp": "2020-12-07T11:06:07.000Z",
          "agent": {
            "id": "8a4f500d"
          },
          "event": {
            "category": "process"
          },
          "process": {
            "name": "cmd.exe",
            "path": "C:\\Windows\\System32\\cmd.exe"
          }
        },
        "sort": [
          1607339167000
        ]
      }
    ]
  }
}

Search for a sequence of eventsedit

Many query languages allow you to match single events. However, EQL’s sequence syntax lets you match an ordered series of events.

Example

The following EQL search request matches a sequence that:

  1. Starts with an event with:

    • An event.category of file
    • A file.name of cmd.exe

  2. Followed by an event with:

    • An event.category of process
    • A process.name that contains the substring regsvr32 
GET /sec_logs/_eql/search
{
  "query": """
    sequence
      [ file where file.name == "cmd.exe" ]
      [ process where stringContains(process.name, "regsvr32") ]
  """
}

The API returns the following response. Matching events in the hits.sequences.events property are sorted by timestamp, converted to milliseconds since the Unix epoch, in ascending order.

{
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "sequences": [
      {
        "events": [
          {
            "_index": "sec_logs",
            "_id": "4",
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:08.000Z",
              "agent": {
                "id": "8a4f500d"
              },
              "event": {
                "category": "file"
              },
              "file": {
                "accessed": "2020-12-07T11:07:08.000Z",
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe",
                "type": "file",
                "size": 16384
              },
              "process": {
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe"
              }
            },
            "fields": {
              "@timestamp": [
                "1607339228000"
              ]
            },
            "sort": [
              1607339228000
            ]
          },
          {
            "_index": "sec_logs",
            "_id": "5",
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:09.000Z",
              "agent": {
                "id": "8a4f500d"
              },
              "event": {
                "category": "process"
              },
              "process": {
                "name": "regsvr32.exe",
                "path": "C:\\Windows\\System32\\regsvr32.exe"
              }
            },
            "fields": {
              "@timestamp": [
                "1607339229000"
              ]
            },
            "sort": [
              1607339229000
            ]
          }
        ]
      }
    ]
  }
}

You can further constrain matching event sequences using the by keyword.

The following EQL search request adds by agent.id to each event item. This ensures events matching the sequence share the same agent.id field value.

GET /sec_logs/_eql/search
{
  "query": """
    sequence
      [ file where file.name == "cmd.exe" ] by agent.id
      [ process where stringContains(process.name, "regsvr32") ] by agent.id
  """
}

Because the agent.id field is shared across all events in the sequence, it can be included using sequence by. The following query is equivalent to the prior one.

GET /sec_logs/_eql/search
{
  "query": """
    sequence by agent.id
      [ file where file.name == "cmd.exe" ]
      [ process where stringContains(process.name, "regsvr32") ]
  """
}

The API returns the following response. The hits.sequences.join_keys property contains the shared agent.id value for each matching event.

{
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "sequences": [
      {
        "join_keys": [
          "8a4f500d"
        ],
        "events": [
          {
            "_index": "sec_logs",
            "_id": "4",
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:08.000Z",
              "agent": {
                "id": "8a4f500d"
              },
              "event": {
                "category": "file"
              },
              "file": {
                "accessed": "2020-12-07T11:07:08.000Z",
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe",
                "type": "file",
                "size": 16384
              },
              "process": {
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe"
              }
            },
            "fields": {
              "@timestamp": [
                "1607339228000"
              ]
            },
            "sort": [
              1607339228000
            ]
          },
          {
            "_index": "sec_logs",
            "_id": "5",
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:09.000Z",
              "agent": {
                "id": "8a4f500d"
              },
              "event": {
                "category": "process"
              },
              "process": {
                "name": "regsvr32.exe",
                "path": "C:\\Windows\\System32\\regsvr32.exe"
              }
            },
            "fields": {
              "@timestamp": [
                "1607339229000"
              ]
            },
            "sort": [
              1607339229000
            ]
          }
        ]
      }
    ]
  }
}

Specify an event category fieldedit

The EQL search API uses event.category as the required event category field by default. You can use the event_category_field parameter to specify another event category field.

Example

The following request specifies file.type as the event category field.

GET /sec_logs/_eql/search
{
  "event_category_field": "file.type",
  "query": """
    file where agent.id == "8a4f500d"
  """
}

Specify a timestamp fieldedit

The EQL search API uses @timestamp as the required event timestamp field by default. You can use the timestamp_field parameter to specify another timestamp field.

Example

The following request specifies file.accessed as the event timestamp field.

GET /sec_logs/_eql/search
{
  "timestamp_field": "file.accessed",
  "query": """
    file where (file.size > 1 and file.type == "file")
  """
}

Filter using query DSLedit

You can use the EQL search API’s filter parameter to specify an additional query using query DSL. This query filters the documents on which the EQL query runs.

Example

The following request uses a range query to filter the sec_logs index down to only documents with a file.size value greater than 1 but less than 1000000 bytes. The EQL query in query parameter then runs on these filtered documents.

GET /sec_logs/_eql/search
{
  "filter": {
    "range" : {
      "file.size" : {
        "gte" : 1,
        "lte" : 1000000
      }
    }
  },
  "query": """
    file where (file.type == "file" and file.name == "cmd.exe")
  """
}

Run a case-sensitive EQL searchedit

By default, matching for EQL queries is case-insensitive. You can use the EQL search API’s case_sensitive parameter to toggle case sensitivity on or off.

Example

The following search request contains a query that matches process events with a process.path containing System32.

Because the case_sensitive parameter is true, this query only matches process.path values containing System32 with the exact same capitalization. A process.path value containing system32 or SYSTEM32 would not match this query.

GET /sec_logs/_eql/search
{
  "case_sensitive": true,
  "query": """
    process where stringContains(process.path, "System32")
  """
}
« EQL requirements EQL syntax reference »