Enable analyze event feature after an upgradeedit

After upgrading from Elastic Stack version 7.9.x from a previous minor release (7.8.x, etc.), you need to update .siem-signals* system index mappings to enable the analyze event feature, which shows graphical representations of process relationships.

If you are upgrading from a minor release to Elastic Stack version >= 7.11.0, there is now a Detection Alerts Migration API that you can use instead of the manual process described below.

To update the .siem-signals* index:

  1. Create a new index template for storing existing detections alerts (see Create index template).
  2. Deactivate all detection rules (see Disable all detection rules).
  3. Copy existing .siem-signals* detection alerts to a new index (see Add existing detection alerts to a new index).
  4. Delete the original .siem-signals* index, as well as the original index lifecycle management (ILM) policy and template (see Delete existing .siem-signals-* indices and templates).
  5. Enable detection rules, which automatically creates the new .siem-signals* system indices, ILM policy, and template (see Enable detection rules).
  6. Restore existing detection alerts to the new .siem-signals* index (see Restore previously existing detections alerts).
  7. Add an alias to the .siem-signals* index for the restored detection alerts index (see Set up index alias).

Create index templateedit

  1. In Kibana, go to ManagementDev Tools.
  2. Do one of the following:

    • Copy and paste this code into the console.

      OR

    • Follow these steps:

      1. Run the following command in the console:

        GET _template/.siem-signals-<space> 

        <space> is the name of the Kibana space in which the detection alerts reside (for example, GET _template/.siem-signals-default).

        The existing .siem-signals template is displayed in the console’s output pane.

      2. Copy the code in the console output pane and paste it into the console input pane.
      3. Edit the code in the input pane as follows:

        • Add this line immediately above the code:

          PUT _template/temp-signals
        • Delete this line:

          ".siem-signals-default" : {
        • Delete the lifecycle object:

          "lifecycle" : {
              "name" : ".siem-signals-default",
              "rollover_alias" : ".siem-signals-default"
          },
        • Change the value of the index_pattern element to temp-signals:

          "index_patterns" : [
              "temp-signals"
          ],
      4. Make sure the first few lines of the code are identical to this:

        PUT _template/temp-signals
        {
            "order" : 0,
            "version" : 1,
            "index_patterns" : [
              "temp-signals"
            ],
            "settings" : {
              "index" : {
                "mapping" : {
                  "total_fields" : {
                    "limit" : "10000"
                  }
                }
              }
            },
  3. Run the code (click on the run icon).

    The console output pane displays this text:

    {
      "acknowledged" : true
    }

Disable all detection rulesedit

To disable all detection rules:

  1. Go to DetectionsManage detection rules.
  2. Scroll to the bottom of the page.
  3. Click the Rows per page menu, and then select 300 rows.
  4. When the page reloads, select all the rules.
  5. Click Bulk actionsDeactivate selected.

Add existing detection alerts to a new indexedit

To enable restoring existing detection alerts after recreating the .siem-signals-* index mapping, add (reindex) existing alerts to the temp-signals index:

  1. Go to ManagementDev Tools.
  2. Run this code in the console:

    POST _reindex
    {
      "source": {
        "index": ".siem-signals-<space>-*" 
      },
      "dest": {
        "index": "temp-signals"
      }
    }

    <space> is the name of the Kibana space in which the detection alerts reside (for example, "index": ".siem-signals-default-*").

    The console output pane displays text similar to this:

    {
      "took" : 603,
      "timed_out" : false,
      "total" : 15,
      "updated" : 0,
      "created" : 15,
      "deleted" : 0,
      "batches" : 1,
      "version_conflicts" : 0,
      "noops" : 0,
      "retries" : {
        "bulk" : 0,
        "search" : 0
      },
      "throttled_millis" : 0,
      "requests_per_second" : -1.0,
      "throttled_until_millis" : 0,
      "failures" : [ ]
    }

Delete existing .siem-signals-* indices and templatesedit

The existing .siem-signals-* system indices, templates, and ILM policy must be deleted before the new indices are created:

  1. Go to ManagementDev Tools.
  2. Run these commands in the console:

    DELETE .siem-signals-<space>-* 
    DELETE _template/.siem-signals-<space>
    DELETE _ilm/policy/.siem-signals-<space>

    <space> is the name of the Kibana space (for example, DELETE .siem-signals-default-*).

    You must run each command individually.

    Foe each command, the console output pane displays this text:

    {
      "acknowledged" : true
    }

Enable detection rulesedit

Enabling detections rules automatically creates the new indices:

  1. Go to DetectionsManage detection rules.
  2. Select the rules you want to activate.
  3. Click Bulk actionsActivate selected.

    The selected rules are activated and the new index, index template, and ILM policy are created.

  4. To verify the new index has been created:

    1. Go to ManagementDev Tools.
    2. Run the following command:

      GET .siem-signals-<space>-*/_mapping/field/process.entity_id 

      <space> is the name of the Kibana space (for example, GET .siem-signals-default-*/_mapping/field/process.entity_id).

      The console output pane displays text similar to this:

      {
        ".siem-signals-default-000001" : {
          "mappings" : {
            "process.entity_id" : {
              "full_name" : "process.entity_id",
              "mapping" : {
                "entity_id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            }
          }
        }
      }

Restore previously existing detections alertsedit

To restore detection alerts:

  1. Go to ManagementDev Tools.
  2. Run the following command:

    POST _reindex
    {
      "source": {
        "index": "temp-signals" 
      },
      "dest": {
        "index": ".siem-signals-<space>-mappingfix" 
      }
    }

    The name of the index in which existing alerts were stored when you performed Add existing detection alerts to a new index.

    Name of the new index with the correct mappings to which previously existing alerts are restored. <space> is the name of the Kibana space (for example, "index": ".siem-signals-default-mappingfix").

    The console output pane displays text similar to this:

    #! Deprecation: index name [.siem-signals-default-mappingfix] starts with a dot
    '.', in the next major version, index names starting with a dot are reserved for
    hidden indices and system indices
    {
      "took" : 170,
      "timed_out" : false,
      "total" : 15,
      "updated" : 0,
      "created" : 15,
      "deleted" : 0,
      "batches" : 1,
      "version_conflicts" : 0,
      "noops" : 0,
      "retries" : {
        "bulk" : 0,
        "search" : 0
      },
      "throttled_millis" : 0,
      "requests_per_second" : -1.0,
      "throttled_until_millis" : 0,
      "failures" : [ ]
    }

Ignore the deprecation warning.

Set up index aliasedit

To display reindexed alerts in the Elastic Security app, create an index alias:

  1. Go to ManagementDev Tools.
  2. Run the following command:

    POST /_aliases
    {
        "actions" : [
            { "add" : { "index" : ".siem-signals-<space>-mappingfix",
            "alias" : ".siem-signals-<space>" } } 
        ]
    }

    The name of the index created when you performed Restore previously existing detections alerts. <space> is the Kibana space name (for example: .siem-signals-default-mappingfix).

    The console output pane displays this text:

    {
      "acknowledged" : true
    }
  3. To confirm the alias exists, run this command:

    GET .siem-signals-<space>/_alias

    The console output pane should display text similar to this:

    {
      ".siem-signals-default-000001" : {
        "aliases" : {
          ".siem-signals-default" : {
            "is_write_index" : true
          }
        }
      },
      ".siem-signals-default-mappingfix" : {
        "aliases" : {
          ".siem-signals-default" : { }
        }
      }
    }