EQL searchedit

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

Event Query Language (EQL) is a query language used for event-based, time-series data, such as logs.

Advantages of EQLedit

  • EQL lets you express relationships between events.
    Many query languages allow you to match only single events. EQL lets you match a sequence of events across different event categories and time spans.
  • EQL has a low learning curve.
    EQL syntax looks like other query languages. It lets you write and read queries intuitively, which makes for quick, iterative searching.
  • We designed EQL for security use cases.
    While you can use EQL for any event-based data, we created EQL for threat hunting. EQL not only supports indicator of compromise (IOC) searching but makes it easy to describe activity that goes beyond IOCs.

Required fieldsedit

EQL assumes each document in a data stream or index corresponds to an event. To search using EQL, each document in the searched data stream or index must include a timestamp field and an event category field.

Elasticsearch EQL uses the @timestamp and event.category fields from the Elastic Common Schema (ECS) as the default timestamp and event category fields. If your searched documents use a different timestamp or event category field, you must specify it in the search request. See Specify a timestamp or event category field.

Run an EQL searchedit

You can use the EQL search API to run an EQL search.

The following request searches my-index-000001 for events with an event.category of process and a process.name of cmd.exe. Each document in my-index-000001 includes a @timestamp and event.category field.

GET /my-index-000001/_eql/search
{
  "query": """
    process where process.name == "cmd.exe"
  """
}

The API returns the following response. Matching events are included in the hits.events property. These events are sorted by timestamp, converted to milliseconds since the Unix epoch, in ascending order.

{
  "is_partial": false,
  "is_running": false,
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "events": [
       {
        "_index": "my-index-000001",
        "_id": "OQmfCaduce8zoHT93o4H",
        "_score": null,
        "_source": {
          "@timestamp": "2020-12-06T11:04:05.000Z",
          "event": {
            "category": "process",
            "id": "edwCRnyD",
            "sequence": 1
          },
          "process": {
            "name": "cmd.exe",
            "executable": "C:\\Windows\\System32\\cmd.exe",
            "pid": 2012
          }
        }
      },
      {
        "_index": "my-index-000001",
        "_id": "xLkCaj4EujzdNSxfYLbO",
        "_score": null,
        "_source": {
          "@timestamp": "2020-12-07T11:06:07.000Z",
          "event": {
            "category": "process",
            "id": "cMyt5SZ2",
            "sequence": 3
          },
          "process": {
            "name": "cmd.exe",
            "executable": "C:\\Windows\\System32\\cmd.exe",
            "pid": 2012
          }
        }
      }
    ]
  }
}

Search for a sequence of eventsedit

You can use EQL’s sequence syntax to search for an ordered series of events.

The following EQL search request matches a sequence that:

  1. Starts with an event with:

    • An event.category of file
    • A file.name of cmd.exe
  2. Followed by an event with:

    • An event.category of process
    • A process.name that contains the substring regsvr32
GET /my-index-000001/_eql/search
{
  "query": """
    sequence
      [ file where file.name == "cmd.exe" ]
      [ process where stringContains(process.name, "regsvr32") ]
  """
}

The API returns the following response. Matching sequences are included in the hits.sequences property.

{
  "is_partial": false,
  "is_running": false,
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "sequences": [
      {
        "events": [
          {
            "_index": "my-index-000001",
            "_id": "AtOJ4UjUBAAx3XR5kcCM",
            "_version" : 1,
            "_seq_no" : 3,
            "_primary_term" : 1,
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:08.000Z",
              "event": {
                "category": "file",
                "id": "bYA7gPay",
                "sequence": 4
              },
              "file": {
                "accessed": "2020-12-07T11:07:08.000Z",
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe",
                "type": "file",
                "size": 16384
              },
              "process": {
                "name": "cmd.exe",
                "executable": "C:\\Windows\\System32\\cmd.exe",
                "pid": 2012
              }
            }
          },
          {
            "_index": "my-index-000001",
            "_id": "yDwnGIJouOYGBzP0ZE9n",
            "_version" : 1,
            "_seq_no" : 4,
            "_primary_term" : 1,
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:09.000Z",
              "event": {
                "category": "process",
                "id": "aR3NWVOs",
                "sequence": 5
              },
              "process": {
                "name": "regsvr32.exe",
                "executable": "C:\\Windows\\System32\\regsvr32.exe",
                "pid": 2012
              }
            }
          }
        ]
      }
    ]
  }
}

You can use the with maxspan keywords to constrain a sequence to a specified timespan.

The following EQL search request adds with maxspan=1h to the previous query. This ensures all events in a matching sequence occur within 1h (one hour) of the first event’s timestamp.

GET /my-index-000001/_eql/search
{
  "query": """
    sequence with maxspan=1h
      [ file where file.name == "cmd.exe" ]
      [ process where stringContains(process.name, "regsvr32") ]
  """
}

You can further constrain matching event sequences using the by keyword.

The following EQL search request adds by process.pid to each event item. This ensures events matching the sequence share the same process.pid field value.

GET /my-index-000001/_eql/search
{
  "query": """
    sequence with maxspan=1h
      [ file where file.name == "cmd.exe" ] by process.pid
      [ process where stringContains(process.name, "regsvr32") ] by process.pid
  """
}

Because the process.pid field is shared across all events in the sequence, it can be included using sequence by. The following query is equivalent to the previous one.

GET /my-index-000001/_eql/search
{
  "query": """
    sequence by process.pid with maxspan=1h
      [ file where file.name == "cmd.exe" ]
      [ process where stringContains(process.name, "regsvr32") ]
  """
}

The API returns the following response. The hits.sequences.join_keys property contains the shared process.pid value for each matching event.

{
  "is_partial": false,
  "is_running": false,
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "sequences": [
      {
        "join_keys": [
          "2012"
        ],
        "events": [
          {
            "_index": "my-index-000001",
            "_id": "AtOJ4UjUBAAx3XR5kcCM",
            "_version": 1,
            "_seq_no": 3,
            "_primary_term": 1,
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:08.000Z",
              "event": {
                "category": "file",
                "id": "bYA7gPay",
                "sequence": 4
              },
              "file": {
                "accessed": "2020-12-07T11:07:08.000Z",
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe",
                "type": "file",
                "size": 16384
              },
              "process": {
                "name": "cmd.exe",
                "executable": "C:\\Windows\\System32\\cmd.exe",
                "pid": 2012
              }
            }
          },
          {
            "_index": "my-index-000001",
            "_id": "yDwnGIJouOYGBzP0ZE9n",
            "_version": 1,
            "_seq_no": 4,
            "_primary_term": 1,
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:09.000Z",
              "event": {
                "category": "process",
                "id": "aR3NWVOs",
                "sequence": 5
              },
              "process": {
                "name": "regsvr32.exe",
                "executable": "C:\\Windows\\System32\\regsvr32.exe",
                "pid": 2012
              }
            }
          }
        ]
      }
    ]
  }
}

You can use the until keyword to specify an expiration event for sequences. Matching sequences must end before this event.

The following request adds until [ process where event.type == "termination" ] to the previous query. This ensures matching sequences end before a process event with an event.type of termination.

GET /my-index-000001/_eql/search
{
  "query": """
    sequence by process.pid with maxspan=1h
      [ file where file.name == "cmd.exe" ]
      [ process where stringContains(process.name, "regsvr32") ]
    until [ process where event.type == "termination" ]
  """
}

Specify a timestamp or event category fieldedit

By default, the EQL search API uses @timestamp and event.category as the required timestamp and event category fields. If your searched documents use a different timestamp or event category field, you must specify it in the search request using the timestamp_field or event_category_field parameters.

The event category field is typically mapped as a keyword or constant keyword field. The timestamp field is typically mapped as a date or date_nanos field.

You cannot use a nested field or the sub-fields of a nested field as the timestamp or event category field. See EQL search on nested fields.

The following request uses the timestamp_field parameter to specify file.accessed as the timestamp field. The request also uses the event_category_field parameter to specify file.type as the event category field.

GET /my-index-000001/_eql/search
{
  "timestamp_field": "file.accessed",
  "event_category_field": "file.type",
  "query": """
    file where (file.size > 1 and file.type == "file")
  """
}

Filter using query DSLedit

You can use the filter parameter to specify an additional query using query DSL. This query filters the documents on which the EQL query runs.

The following request uses a range query to filter my-index-000001 to only documents with a file.size value greater than 1 but less than 1000000 bytes. The EQL query in query parameter then runs on these filtered documents.

GET /my-index-000001/_eql/search
{
  "filter": {
    "range" : {
      "file.size" : {
        "gte" : 1,
        "lte" : 1000000
      }
    }
  },
  "query": """
    file where (file.type == "file" and file.name == "cmd.exe")
  """
}

Run a case-sensitive EQL searchedit

By default, matching for EQL queries is case-insensitive. You can use the case_sensitive parameter to toggle case sensitivity on or off.

The following search request contains a query that matches process events with a process.executable containing System32.

Because case_sensitive is true, this query only matches process.executable values containing System32 with the exact same capitalization. A process.executable value containing system32 or SYSTEM32 would not match this query.

GET /my-index-000001/_eql/search
{
  "keep_on_completion": true,
  "case_sensitive": true,
  "query": """
    process where stringContains(process.executable, "System32")
  """
}

Run an async EQL searchedit

EQL searches are designed to run on large volumes of data quickly, often returning results in milliseconds. For this reason, EQL searches are synchronous by default. The search request waits for complete results before returning a response.

However, complete results can take longer for searches across:

To avoid long waits, you can use the wait_for_completion_timeout parameter to run an asynchronous, or async, EQL search.

Set wait_for_completion_timeout to a duration you’d like to wait for complete search results. If the search request does not finish within this period, the search becomes async and returns a response that includes:

  • A search ID, which can be used to monitor the progress of the async search.
  • An is_partial value of true, meaning the response does not contain complete search results.
  • An is_running value of true, meaning the search is async and ongoing.

The async search continues to run in the background without blocking other requests.

The following request searches the frozen-my-index-000001 index, which has been frozen for storage and is rarely searched.

Because searches on frozen indices are expected to take longer to complete, the request contains a wait_for_completion_timeout parameter value of 2s (two seconds). If the request does not return complete results in two seconds, the search becomes async and returns a search ID.

GET /frozen-my-index-000001/_eql/search
{
  "wait_for_completion_timeout": "2s",
  "query": """
    process where process.name == "cmd.exe"
  """
}

After two seconds, the request returns the following response. Note is_partial and is_running properties are true, indicating an async search.

{
  "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  "is_partial": true,
  "is_running": true,
  "took": 2000,
  "timed_out": false,
  "hits": ...
}

You can use the the search ID and the get async EQL search API to check the progress of an async search.

The get async EQL search API also accepts a wait_for_completion_timeout parameter. If ongoing search does not complete during this period, the response returns an is_partial value of true and no search results.

The following get async EQL search API request checks the progress of the previous async EQL search. The request specifies a wait_for_completion_timeout query parameter value of 2s (two seconds).

GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?wait_for_completion_timeout=2s

The request returns the following response. Note is_partial and is_running are false, indicating the async search has finished and the search results in the hits property are complete.

{
  "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  "is_partial": false,
  "is_running": false,
  "took": 2000,
  "timed_out": false,
  "hits": ...
}

Change the search retention periodedit

By default, the EQL search API stores async searches for five days. After this period, any searches and their results are deleted. You can use the keep_alive parameter to change this retention period.

In the following EQL search request, the keep_alive parameter is 2d (two days). If the search becomes async, its results are stored on the cluster for two days. After two days, the async search and its results are deleted, even if it’s still ongoing.

GET /my-index-000001/_eql/search
{
  "keep_alive": "2d",
  "wait_for_completion_timeout": "2s",
  "query": """
    process where process.name == "cmd.exe"
  """
}

You can use the get async EQL search API's `keep_alive`parameter to later change the retention period. The new retention period starts after the get request executes.

The following request sets the keep_alive query parameter to 5d (five days). The async search and its results are deleted five days after the get request executes.

GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d

You can use the delete async EQL search API to manually delete an async EQL search before the keep_alive period ends. If the search is still ongoing, this cancels the search request.

The following request deletes an async EQL search and its results.

DELETE /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d

Store synchronous EQL searchesedit

By default, the EQL search API only stores async searches that cannot be completed within the period set by wait_for_completion_timeout.

To save the results of searches that complete during this period, set the keep_on_completion parameter to true.

In the following search request, keep_on_completion is true. This means the search results are stored on the cluster, even if the search completes within the 2s (two-second) period set by the wait_for_completion_timeout parameter.

GET /my-index-000001/_eql/search
{
  "keep_on_completion": true,
  "wait_for_completion_timeout": "2s",
  "query": """
    process where process.name == "cmd.exe"
  """
}

The API returns the following response. A search ID is provided in the id property. is_partial and is_running are false, indicating the EQL search was synchronous and returned complete results in hits.

{
  "id": "FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=",
  "is_partial": false,
  "is_running": false,
  "took": 52,
  "timed_out": false,
  "hits": ...
}

You can use the search ID and the get async EQL search API to retrieve the same results later.

GET /_eql/search/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=

Saved synchronous searches are still subject to the retention period set by the keep_alive parameter. After this period, the search and its results are deleted.

You can also manually delete saved synchronous searches using the delete async EQL search API.