Automating detection tuning requests with Kibana cases

Automating Detection Tuning Requests with Elastic Security

At Elastic, the Infosec team is "Customer Zero”. We use the newest version of Elastic products extensively to secure our organization, which gives us unique insights into how to solve real-world security challenges. One of the ways we've improved Security Operations Center (SOC) efficiency is by creating a seamless, automated workflow that allows our analysts to open a detection tuning request directly from Kibana Cases with a single click.

In any SOC, the feedback loop between security analysts and detection engineers is crucial for maintaining a healthy and effective security posture. Analysts on the front lines are the first to see how detection rules perform in the real world. They know which alerts are valuable, which are noisy, and which could be improved with a bit of tuning. Alert fatigue from noisy alerts increases the risk of missing a true positive alert. Quickly tuning false positives is critical to responding to true positives. Capturing this alert feedback efficiently can be a challenge – manual processes, like sending emails, opening tickets, or direct messages can be inconsistent, time consuming, and hard to track.

With Elastic Security, an analyst can attach alerts to a new or existing case in Kibana, conduct their investigation, and with some customization and automation they can initiate a tuning request with a single click directly from Kibana Cases. This article will walk you through how we built this automation, and how you can implement a similar system to close the feedback loop and optimize your detection and response program.

Custom Fields in Kibana Cases

Custom fields are a key component of this automation within the Kibana Cases. Using these custom fields, we can capture the necessary information directly from the tool that the analysts are already using. These custom fields will appear on all new and existing cases, providing a clear and consistent way for analysts to flag a detection for review.

Note: The ability to add custom fields to cases was introduced in version 8.15. For more details, refer to the official Cases documentation.

Every Kibana Case is a document stored in a dedicated Elasticsearch index: .kibana_alerting_cases. This means all your case data is available for querying, aggregation, and automation, just like any other data source in Elastic. Each case document contains a wealth of information, but a few fields are particularly useful for metrics and automation. The cases.status field tracks whether a case is open, in-progress, or closed, while cases.created_at and cases.updated_at provide timestamps crucial for calculating metrics like Mean Time to Resolution (MTTR). Fields like cases.severity and cases.owner allow you to slice and dice your metrics to see how the team is performing. Most importantly for this blog, the cases.custom_fields object contains an array of the custom fields you've configured. Runtime fields can be used to parse the array of custom fields, allowing you to build queries, dashboards, visualizations, and detection rules that trigger workflows.

Beyond tuning requests, custom fields are incredibly versatile for tracking metrics and enriching cases. For example, we have a "Complex Case" custom field to flag cases that take more than an hour to resolve, helping us identify rules that may need better investigation guides or automation to help reduce the investigation time. We also use custom fields like "Detection rule valid" and "True Positive Alert" to gather granular feedback on rule performance and fidelity, allowing us to build powerful dashboards in Kibana to visualize the operational effectiveness of our SOC.

If you have not already created a data view for the Cases information you will need to do that if you want to use runtime fields and data visualizations with your cases.

Navigate to Index Patterns: In Kibana, go to Stack Management > Data Views and click ‘create new data view’.

Configure the Data view to map the .kibana_alerting_cases system index. You will need to click the Allow hidden and system indices button to allow this. For the timestamp field I recommend using the cases.updated_at field so the cases are displayed by the most recent activity.

Creating Custom fields

There are two types of custom fields; Text fields for free-form input, or Toggle fields for simple yes/no feedback. For our Tuning Request automation, we use one of each. The text field is an optional field used to capture any additional feedback from the analyst, and the toggle field is used to trigger the automation.

In Kibana, go to Security > Cases, then click on Settings in the top right. In the settings page you will see a Custom Fields section where you can add the new fields you want. The fields are displayed in the cases UI in alphabetical order so we prefix our fields with numbers to keep them in the order we want.

You can create the new custom fields, the Labels added in the UI are only for the analysts and are not stored in the cases index. These can be any value you want.

Add Custom Fields: We need two fields for this workflow.

• Field 1: Tuning Required Toggle

  • This will be the button analysts click to initiate a tuning request.

    • Label: Open tuning request?
    • Type: Toggle
    • Default Value: Off

  • Field 2: Tuning Request Details

    • This field allows the analyst to provide specific details about what needs to be changed, such as adding an exception, lowering the severity, or adjusting the query logic.
    • Name: Tuning request detail
    • Type: Text

  • Default Value: Off

Using Runtime fields to map the custom fields

A challenge when working with custom fields in Kibana Cases is that the cases.custom_fields field is mapped as an array of objects, where each object represents a custom field with its name and value. This structure makes it difficult to query for specific custom fields directly in KQL. For example, you can't simply use a query like cases.custom_fields.open_tuning_request : "true". To overcome this, we can use runtime fields to parse and query the custom fields.

Runtime fields are fields that are evaluated at query time. They allow you to create new fields on the fly without having to reindex your data. We can define runtime fields on the .kibana_alerting_cases index to use a painless script to parse the cases.custom_fields array and extract the values we need into new, easily queryable fields.

For this workflow, we'll create two runtime fields that will map to the custom fields created above:
* TuningRequired: A boolean field that will be true if the "Open tuning request" toggle is on.
* TuningDetail: A text field that will contain the analyst's comments from the "Tuning request detail" field.

Before we can create the runtime fields, we first need to identify the unique ID (key) that Kibana assigns to each custom field. Currently, there isn't a straightforward way to view this ID in the UI. To find it, we used the following workaround:

1. Create the Fields. If you are using other custom fields you should create the custom fields one at a time to make it easier to identify the new field keys. If you only have the two fields mentioned above you can tell them apart using the type value which can be either text or toggle.
2. Create a new case. After adding the field, we created a test case in Kibana and added some data to the description field and toggled the tuning required field to true with all other custom fields set to false or blank.
3. Inspect the case document. We then navigated to Discover and queried the .kibana_alerting_cases index to find the document for the new case. By inspecting the cases.customFields array in the document's source, we could find the key associated with our new custom field. Save the values of the key fields to be used in the runtime scripts.

The cases.customFields data is formatted like this:

  [
    {
      "key": "4537b921-3ca4-4ff0-aa39-02dd6a3177bd",
      "type": "text",
      "value": "This alert is too noisy"
    },
    {
      "key": "cdf28896-c793-43d2-9384-99562e23a646",
      "type": "toggle",
      "value": true
    }
  ]

Creating the Runtime Fields

You can add runtime fields through the Kibana UI or by using the Elasticsearch API in the Dev Tools console. If you have not already created a data view for the Cases information you will need to do that first.

While viewing the new Kibana Cases Data view click the ‘Add Field’ button to open the flyout menu to create a new runtime field.

Enter the name of the field, in this example we are configuring TuningRequired as a new Boolean field type. Click the ‘Set Value’ toggle to configure this as a new Runtime field configured via a painless script. Update this painless script to replace TUNING_REQUIRED_FIELD_KEY_UUID with the key value from the Tuning Required custom field and paste it into the value field and save the new runtime field.

...
    if (params._source.containsKey('cases') &&
    params._source.cases != null &&
    params._source.cases.containsKey('customFields') &&
    params._source.cases.customFields != null) 
{
  for (def cf : params._source.cases.customFields) {
    if (cf != null &&
        cf.containsKey('key') &&
        cf.key != null &&
        cf.key.contains('TUNING_REQUIRED_FIELD_KEY_UUID') &&
        cf.containsKey('value') &&
        cf.value != null) {
      emit(cf.value);
      break;
    }
  }
}

Repeat this process for the TuningDetail field, remember to use the key value from the text field in this field’s painless script. If you have any additional custom fields in your cases that you want to use for dashboards or metrics you can map those as well with this same process.

If you control your cluster settings and data views ‘as code’ you can also add runtime fields to an index mapping using the Update mapping API from the Kibana Dev Tools console.

Automating the tuning request creation

We can trigger this automation in two ways: through a custom detection rule (that will create a new alert and send it to a connector when a case is updated with a tuning request) or via a scheduled external automation that queries the API.

This automation can be created using any automation platform such as Tines, Github Actions, or custom scripting. This is the logic we use for our automation:

Step 1: Find any cases recently tagged as TuningRequired

You can use this elasticsearch query to find any cases that have been updated within the last hour where the TuningRequired field has been set to true. This query uses the cases.updated_at field as the time range. The runtime field mappings must be included in the API request to query the custom fields.

This query will return all of the case documents from the .kibana_alerting_cases index that have been updated in the last hour and the TuningRequired field has been set to true

POST /.kibana_alerting_cases/_search  
{  
  "query": {  
    "bool": {  
      "must": [],  
      "filter": [  
        {  
          "bool": {  
            "should": [  
              {  
                "match": {  
                  "TuningRequired": true  
                }  
              }  
            ],  
            "minimum_should_match": 1  
          }  
        },  
        {  
          "range": {  
            "cases.updated_at": {  
              "format": "strict_date_optional_time",  
              "gte": "now-1h",  
              "lte": "now"  
            }  
          }  
        }  
      ],  
      "should": [],  
      "must_not": []  
    }  
  },  
 "runtime_mappings": {  
   "TuningDetail": {  
     "type": "keyword",  
     "script": {  
       "source": "if (\nparams._source.containsKey('cases') &&\nparams._source.cases != null &&\nparams._source.cases.containsKey('customFields') &&\nparams._source.cases.customFields != null\n) {\nfor (def cf : params._source.cases.customFields) {\nif (\ncf != null &&\ncf.containsKey('key') &&\ncf.key != null &&\ncf.key.contains('6cadc70a-7d68-4531-9861-7d5bc24c4c1c') &&\ncf.containsKey('value') &&\ncf.value != null\n) {\nemit(cf.value);\nbreak;\n}\n}\n}"  
     }  
   },  
   "TuningRequired": {  
     "type": "boolean",  
     "script": {  
       "source": "if (\nparams._source.containsKey('cases') &&\nparams._source.cases != null &&\nparams._source.cases.containsKey('customFields') &&\nparams._source.cases.customFields != null\n) {\nfor (def cf : params._source.cases.customFields) {\nif (\ncf != null &&\ncf.containsKey('key') &&\ncf.key != null &&\ncf.key.contains('496e71f2-2bce-47a2-93a8-00db0de2d1b4') &&\ncf.containsKey('value') &&\ncf.value != null\n) {\nemit(cf.value);\nbreak;\n}\n}\n}"  
     }  
   }  
 },  
  "fields": [  
    "TuningDetail",  
    "TuningRequired"  
  ]  
}

Any time a field is changed or a comment is made in a case it will update the updated_at field to the current time. Because any update or comment added to a case will update this timestamp, it is possible to have a single case returned multiple times by this automation if it is run regularly while the case is being updated. Any automation processes leveraged for this should have a deduplication process to prevent processing the same case multiple times in this scenario.

Step 2: Parsing each case

Loop through each of the cases returned by the previous query to process them one at a time. Each document returned will contain the fields array with the values from the custom fields, as well as other useful fields. Parse each of the following fields and store them for future use:

• The _id field will have a format like cases:{{case_ID}}. The case ID is used for future API requests in the automation to add comments to the case or retrieve all alerts attached to the case.
• cases.title is the title of the case
• cases.assignees is who the case is assigned to
• cases.updated_by is the last person to update the case, this is often the person submitting the tuning request and can be useful for knowing who to contact for more information.
• cases.tags can be useful if you are using tags to sort or identify your cases.

Step 3: Retrieving the alerts attached to the case

For each case you will want to know which alerts are attached to the case so you know which alerts need to be tuned. This can be done using the cases API with the _id field for the case.

/api/cases/{caseId}/alerts

This query will return an array of all alert id values that are attached to the case. Using this ID value you can query the .siem-signals* elasticsearch index to find the full information about each alert attached to the case that needs tuning.

POST /.siem-signals-*/_search  
{  
 "size": 1,  
 "query": {  
   "bool": {  
     "must": [],  
     "filter": [  
       {  
         "bool": {  
           "should": [  
             {  
               "match": {  
                 "_id": "{{alert_id}}"  
               }  
             }  
           ],  
           "minimum_should_match": 1  
         }  
       },  
       {  
         "range": {  
           "@timestamp": {  
             "format": "strict_date_optional_time",  
             "gte": "now-30d",  
             "lte": "now"  
           }  
         }  
       }  
     ],  
     "should": [],  
     "must_not": []  
   }  
 }  
}

From the results of this query you can extract information about the alert such as the name and creation date, along with any other information that could help for tuning such as the user.name or process.name fields. Because a case can have many alerts attached to it you will want to deduplicate the alerts by the signal.rule.name value.

Step 4: Opening a tuning request.

This step is dependent on the ticketing system you use in your environment. Our team uses github issues to track tuning requests and slack for notifications, but this could also be done with any ticketing or project management system that supports automation.

This is the logic flow we use for our automation using both Github and Slack to track tuning requests:

• Using the name of the alert we search for any existing open tuning requests.
  • If an existing tuning request exists we update that request with the details from the case and the new request
  • If no existing request exists we open a new tuning request issue and attach the information
• We then send a slack notification to the Detection engineering team’s slack channel containing a link to the tuning request, a link to the case, and details about the request and alert.
• We then use the Cases API to add a comment to the original case with a link to the tuning request issue
• Optional AI Agent: We are starting to experiment with the use of AI Agents to analyze the alert and case information and then provide even better context with the tuning request, potentially even recommending the changes to make to the detection rules.

The final result from this automation is that our SOC Analysts can create a detailed detection tuning request ticket with a single click from their case. We have seen a dramatic increase in the reduction of false positives and the overall efficiency of our detection rules because of this automation.

Conclusão

By using Kibana Cases with custom fields and integrating with automation platforms, you can optimize many of your manual processes. This automated workflow reduces the manual overhead associated with collecting analyst feedback, ensuring that valuable analyst insights are quickly translated into actionable improvements in detection rules. The result is a more efficient, accurate, and resilient SOC that can adapt rapidly to emerging threats and reduce alert fatigue.

Ready to optimize your SOC's efficiency and improve your detection posture? Explore Elastic Security and start building your own automated tuning request workflows today!