In a single click, users can install and start using the DGA model and associated assets, including ingest pipeline configurations, anomaly detection jobs, and detection rules. Read on for step-by-step instructions on installing and fully enabling the DGA package.

[Related article: Automating the Security Protections rapid response to malware]

What is a DGA?

A DGA is a technique employed by many malware authors to ensure that infection of a client machine evades defensive measures. The goal of this technique is to hide the communication between an infected client machine and the command and control (C & C or C2) server by using hundreds or thousands of randomly generated domain names, of which one will ultimately resolve to the IP address of a C & C server.

To more easily visualize what’s occurring in a DGA attack, imagine for a moment you’re a soldier on a battlefield. Like many soldiers, you have communication gear that uses radio frequencies for communication. Your enemy may try to disrupt your communications by jamming your radio frequencies. One way to devise a countermeasure for this is by frequency hopping — using a radio system that changes frequencies very quickly during the course of a transmission. To the enemy, the frequency changes appear to be random and unpredictable, so they are hard to jam.

DGAs are like a frequency-hopping communication channel for malware. They change domains so frequently that blocking the malware’s C2 communication channel becomes infeasible by means of DNS domain name blocking. There are simply too many randomly generated DNS names to successfully identify and block them.

This technique emerged in the world of malware with force in 2009, when the “Conficker” worm began using a very large number of randomly generated domain names for communication. The worm’s authors developed this countermeasure after a consortium of security researchers interrupted the worm’s C2 channel by shutting down the DNS domains it was using for communication. DNS mitigation was also performed in the case of the 2017 WannaCry ransomware global outbreak.

Getting started

We have released the model and the associated assets — including the pipelines, anomaly detection configurations, and detection rules — to the Integrations app in Kibana as of 8.0. We will be maintaining this format moving forward.

If you don’t have an Elastic Cloud cluster but would like to start experimenting with the released ProblemChild package, you can start a free 14-day trial of Elastic Cloud.

We will now look at the steps to get DGA up and running in your environment in a matter of minutes using the released DGA package.

Step 1: Installing the package assets

In Kibana, the Integrations app now includes the DGA detection package. To install the assets, click the Install DGA assets button under the Settings tab. This will install all of the artifacts necessary to use the DGA model to generate alerts when DGA activity is detected in your network data.

Once installation is complete, you can navigate to Stack Management > Ingest Pipelines and see that the <version-number>-ml\_dga\_ingest\_pipeline has been installed and can now be used to enrich incoming ingest data. The ingest pipeline leverages the <version-number>-ml\_dga\_inference\_pipeline to do this.

Similarly, the installed DGA model can now be seen in Machine Learning > Model Management > Trained Models.

Step 2: Enriching your data

Now you are ready to ingest your data using the ingest pipeline. The supervised model will analyze and enrich incoming data containing DNS events with a DGA score.

This pipeline is designed to work with data containing DNS events — such as packetbeat data — which contain these ECS fields: dns.question.name and dns.question.registered_domain. You can add the installed ingest pipeline to an Elastic beat by adding a simple configuration setting.

If you already have an ingest pipeline associated with your indices, you can use a pipeline processor to integrate the DGA ingest pipeline into your existing pipeline.

You will also want to add the following mappings to the beat you chose:

{
  "properties": {
    "ml_is_dga": {
      "properties": {
        "malicious_prediction": {
          "type": "long"
        },
        "malicious_probability": {
          "type": "float"
        }
      }
    }
  }
}

You can do this under Stack Management > Index Management > Component Templates. Templates that can be edited to add custom components will be marked with a @custom suffix. Edit the @custom component template for your Elastic beat by pasting the above JSON blob in the Load JSON flyout.

You should now see that the model enriches incoming DNS events with the following fields:

• Ml_is_dga.malicious_prediction: A value of “1” indicates the DNS domain is predicted to be the result of malicious DGA activity. A value of “0” indicates it is predicted to be benign.

• Ml_is_dga.malicious_probability: A probability score, between 0 and 1, that the DNS domain is the result of malicious DGA activity.

If you want an immediate way to test that the ingest pipeline is working as expected with your data, you can use a few sample documents with the simulate pipeline API and confirm you see the ml_is_dga fields.

Step 3: Running anomaly detection

The package includes a pre-configured anomaly detection job. This machine learning (ML) job examines the DGA scores produced by the supervised DGA model and looks for anomalous patterns of unusually high scores for a particular source IP address. These events are assigned an anomaly score.

To run this job on your enriched data, go to Machine Learning > Anomaly Detection. When you create a job using the job wizard, you should see an option to Use preconfigured jobswith a card for DGA. After selecting the card, you will see the pre-configured anomaly detection job that can be run. Note this job is only useful for indices that have been enriched by the ingest pipeline.

Step 4: Enabling the rules

To maximize the benefit of the DGA framework, activate the installed detection rules. They are triggered when certain conditions for the supervised model or anomaly detection job are satisfied. The complete list of the installed rules can be found in the Overview page of the package itself or in the latest experimental detections release.

To fully leverage the included preconfigured anomaly detection job, enable the complementary rule: Potential DGA Activity. This will create an anomaly-based alert in the detection page in the security app.

The preconfigured anomaly detection job and complementary rule are both available in the detection rules repo releases. To enable and use the installed rules, navigate to Security > Rules and select Load Elastic prebuild rules and timeline templates.

Get in touch

We’d love for you to try out ProblemChild and give us feedback as we work on adding new capabilities to it. If you run into any issues during the process, please reach out to us on our community Slack channel, discussion forums, or even our open detections repository.

You can always experience the latest version of Elasticsearch Service on Elastic Cloud and follow along with this blog to set up the ProblemChild framework in your environment for your Windows process event data. And take advantage of our Quick Start training to set yourself up for success. Start your free trial of Elastic Cloud today to get access to the platform. Happy experimenting!

As part of our security solution, we use a machine learning model to help us detect malware based on known characteristics. Since machine learning is never perfect, we deploy supplemental artifacts, including lists of known file hashes that range from malicious to benign. There are two scenarios when we might need to update these lists, which we’ll cover here. Using these, teams can experience a much faster response to malware.

[Related article: <u>Sandboxing anti-malware products for fun and profit</u>]

Scenario 1: The model isn’t perfect

This happens pretty rarely, since our model has True Negative rates of 99.8%. But no model is perfect, and our True Positive rates vary from 99% (which is great) down to 97.9%, due to noisy data — particularly large bursts of near-duplicates. In applying machine learning to cybersecurity, data drift is an ever-present challenge and part of what makes it such an interesting problem.

There are a couple of ways we can find out if our model is doing the wrong thing:

1. User feedback.

A user sends us an email telling us that a piece of custom software is being flagged as malware, but is actually safe.

2. Telemetry data coming back from our model running on customers’ machines.

We look at the volume and velocity of alerts, using Elastic’s Anomaly Detection feature to let us know if there is a sudden spike from a particular file. We also use patterns to help us confirm if an alert is coming from a single noisy machine, a test cluster, or a legitimate threat.

The advantage of using lists as a backstop to machine learning is that it’s a fast and easy way to make sure our customers are protected from new malware, as well as protecting them from unnecessary noise from false positives.

Scenario 2: New forms of malware appear

We typically find out about new kinds of malware from either security news/community resources, or our own Security Intelligence and Analytics team identifying new kinds of malware.

When this happens, we can easily deploy small changes to lists. We update the model separately, because that takes longer.

We’ve been steadily adding more automation to make this process of deploying updated lists to customers smoother and faster, with the goal of making it easy for anyone on the team to create and deploy an update. We’ve also been working on making the process of updating the model easier and faster.

Threats yet discovered…

The Elastic Security Intelligence and Analytics team continues to research and respond to groundbreaking threats in its mission to help Elastic customers and the broader security community. If you’re using <u>Elastic Security</u> already, you can expect to see our latest findings in the newsfeed within the platform. We’ll also post our latest findings on <u>elastic.co/blog</u>.

Ready to become part of the Elastic Security community and take advantage of the team’s leading threat research? Get started with a free <u>14-day trial of Elastic Cloud</u> or <u>download</u> the self-managed version of the Elastic Stack for free.

With the advent of Integration packages in the Elastic stack we can now deliver the full, customizable package that includes the LotL classification model, anomaly detection job configurations, detection rules, and inference pipelines to make it easier to install and get up and running the entire end-to-end data pipeline from collecting windows events to alerting on potential Lotl attacks. We will walk you through how we set it up so you can try it yourself.

ProblemChild: Recap

In an earlier blog post, we talked about how to usethe detection rules repository command line interface (CLI), to set up the ProblemChild framework and get it up and running in your environment. We have now added a Living off the land (LotL) detection package to the Integrations app in Kibana. In a single click, you can install and start using the ProblemChild model and associated assets including anomaly detection configurations and detection rules.

As outlined in the previous blog, ProblemChild is a framework built using the Elastic Stack to detect LotL activity. LotL attacks are generally tricky to detect, given that attackers leverage seemingly benign software already present in the target environment to fly under the radar. The lineage of processes spawned in your environment can provide a strong signal in the event of an ongoing attack.

The supervised machine learning (ML) component of ProblemChild leverages process lineage information present in your Windows process event metadata to classify events as malicious or benign using Inference at the time of ingest. Anomaly detection is then applied to detect rare processes among those detected as malicious by the supervised model. Finally, detection rules alert on rare parent-child process activity as an indication of LotL attacks.

The sheer volume and variety of events seen in organizations poses a challenge for detecting LotL attacks using rules and heuristics, making an ML-based framework such as ProblemChild a great solution.

Getting Started

We have released the model and the associated assets - including the pipelines, anomaly detection configurations, and detection rules - to the Integrations app in Kibana as of 8.0. We will be maintaining this format moving forward.

If you don’t have an Elastic Cloud cluster but would like to start experimenting with the released ProblemChild package, you can start a free 14-day trial of Elastic Cloud.

We will now look at the steps to get ProblemChild up and running in your environment in a matter of minutes using the released Living off the land (LotL) detection package.

Step 1: Installing the package assets

In Kibana, the Integrations app now includes the LotL Attack Detection package. To install the assets, click the Install LotL Attack Detection assets button under the Settings tab.

This will install all of the artifacts necessary to use the ProblemChild model to generate alerts when LotL activity is detected in your environment.

Once installation is complete, you can navigate to Stack Management > Ingest Pipelines and see that the <version-number>-problem\_child\_ingest\_pipeline has been installed and can now be used to enrich incoming ingest data. The ingest pipeline leverages the <version-number>-problem\_child\_inference\_pipeline in order to do this.

Similarly, the installed ProblemChild model can now be seen in Machine Learning > Model Management > Trained Models

Step 2: Enriching your data

Now you are ready to ingest your data using the ingest pipeline. This will enrich your incoming data with predictions from the machine learning model.

This pipeline is designed to work with Windows process event data such as Winlogbeat data. You can add the installed ingest pipeline to an Elastic beat by adding a simple configuration setting.

If you already have an ingest pipeline associated with your indices, you can use a pipeline processor to integrate the ProblemChild ingest pipeline into your existing pipeline.

You will also want to add the following mappings to the Elastic beat you chose:

{
  "properties": {
    "problemchild": {
      "properties": {
        "prediction": {
          "type": "long"
        },
        "prediction_probability": {
          "type": "float"
        }
      }
    },
    "blocklist_label": {
      "type": "long"
    }
  }
}

You can do this under Stack Management > Index Management > Component Templates. Templates that can be edited to add custom components will be marked with a @custom suffix. Edit the @custom component template for your Elastic beat by pasting the above JSON blob in the Load JSON flyout.

You should now see that the model enriches incoming Windows process events with the following fields:

problemchild.prediction

• A value of 1 indicates that the event is predicted to be malicious and a value of “0” indicates that the event is predicted to be benign.

prediction_probability

• A value between 0 and 1 indicating the confidence of the model in its prediction. The higher the value, the higher the confidence.

blocklist_label

• A value of 1 indicates that the event is malicious because one or more terms in the command line arguments matched a blocklist.

If you want an immediate way to test that the ingest pipeline is working as expected with your data, you can use a few sample documents with the simulate pipeline API and confirm you see the problemchild fields.

Step 3: Running anomaly detection

The package includes several preconfigured anomaly detection jobs. These jobs enable you to find the rarest events among those detected as malicious by the supervised model in order to decide which events require immediate attention from your analysts.

To run these jobs on your enriched data, go to Machine Learning > Anomaly Detection. When you create a job using the job wizard, you should see an option to Use preconfigured jobs with a card for LotL Attacks. After selecting the card, you will see several preconfigured anomaly detection jobs that can be run. Note these jobs are only useful for indices that have been enriched by the ingest pipeline.

Step 4: Enabling the rules

To maximize the benefit of the ProblemChild framework, activate the installed detection rules. They are triggered when certain conditions for the supervised model or anomaly detection jobs are satisfied. The complete list of the installed rules can be found in the Overview page of the package itself or in the latest experimental detections release.

In order to enable and use the installed rules, you can navigate to Security > Rules and select _Load Elastic prebuild rules and timeline templates_.

Note that there are search rules as well as ML job rules. The search rules are triggered by the supervised model, for example this rule:

The above rule matches on any Windows process event for which the supervised model or its blocklist has a prediction value of 1 (malicious).

The ML job rules are triggered by anomalies found by the anomaly detection jobs that you set up in Step 3 — for example, this rule:

The above rule is triggered each time the anomaly detection job problem_child_rare_process_by_host detects an anomaly with an anomaly score greater than or equal to 75.

Summary

As mentioned in the first blog post, the supervised ML component of ProblemChild is trained to predict a value of 1 (malicious) on processes or command line arguments that can be used for LotL attacks. This does not mean that everything that the supervised model predicts with a value 1 indicates LotL activity. The prediction value of 1 should be interpreted more as “this could be potentially malicious,” instead of “this is definitely LotL activity.”

The real beauty of ProblemChild is in the anomaly detection, wherein it surfaces rare parent-child process relationships from among the events the supervised model marked as suspicious. This not only helps in reducing the number of false positives, but also helps security analysts focus on a smaller, more targeted list for triage.

You could of course start with the search rules, which will alert directly on the results of the supervised model. If the number of alerts from these rules is manageable and you have the time and resources to drill into these alerts, you might not need to enable the anomaly detection jobs. However, if you then notice that these rules are producing too many alerts (which is usually the case in most large organizations), you may benefit from enabling the anomaly detection jobs and their corresponding rules.

Get in touch with us

We’d love for you to try out ProblemChild and give us feedback as we work on adding new capabilities to it. If you run into any issues during the process, please reach out to us on our community Slack channel, discussion forums or even our open detections repository.

You can always experience the latest version of Elasticsearch Service on Elastic Cloud and follow along with this blog to set up the ProblemChild framework in your environment for your Windows process event data. And take advantage of our Quick Start training to set yourself up for success. Happy experimenting!

The potential for beaconing detection is that it can serve as an early warning system and help discover novel persistence mechanisms in the initial hours or days after execution. This allows defenders to disrupt or evict the threat actor before they can achieve their objectives. So, while we are not quite "left of boom" by detecting C2 beaconing, we can make a big difference in the outcome of the attack by reducing its overall impact.

In this blog, we talk about a beaconing identification framework that we built using Painless and aggregations in the Elastic Stack. The framework can not only help threat hunters and analysts monitor network traffic for beaconing activity, but also provides useful indicators of compromise (IoCs) for them to start an investigation with. If you don’t have an Elastic Cloud cluster but would like to try out our beaconing identification framework, you can start a free 14-day trial of Elastic Cloud.

Beaconing — A primer

An enterprise's defense is only as good as its firewalls, antivirus, endpoint detection and intrusion detection capabilities, and SOC (Security Operations Center) — which consists of analysts, engineers, operators administrators, etc. who work round the clock to keep the organization secure. Malware however, enters enterprises in many different ways and uses a variety of techniques to go undetected. An increasingly common method being used by adversaries nowadays to evade detection is to use C2 beaconing as a part of their attack chain, given that it allows them to blend into networks like a normal user.

In networking, beaconing is a term used to describe a continuous cadence of communication between two systems. In the context of malware, beaconing is when malware periodically calls out to the attacker's C2 server to get further instructions on tasks to perform on the victim machine. The frequency at which the malware checks in and the methods used for the communications are configured by the attacker. Some of the common protocols used for C2 are HTTP/S, DNS, SSH, and SMTP, as well as common cloud services like Google, Twitter, Dropbox, etc. Using common protocols and services for C2 allows adversaries to masquerade as normal network traffic and hence evade firewalls.

While on the surface beaconing can appear similar to normal network traffic, it has some unique traits with respect to timing and packet size, which can be modeled using standard statistical and signal processing techniques.

Below is an example of a Koadic C2 beacon, which serves the malicious payload using the DLL host process. As you can see, the payload beacons consistently at an interval of 10 minutes, and the source, as well as destination packet sizes, are almost identical.

It might seem like a trivial task to catch C2 beaconing if all beacons were as neatly structured and predictable as the above. All one would have to look for is periodicity and consistency in packet sizes. However, malware these days is not as straightforward.

Most sophisticated malware nowadays adds a "jitter" or randomness to the beacon interval, making the signal more difficult to detect. Some malware authors also use longer beacon intervals. The beaconing identification framework we propose accounts for some of these elusive modifications to traditional beaconing behavior.

Our approach

We’ve discussed a bit about the why and what — in this section we dig deeper into how we identify beaconing traffic. Before we begin, it is important to note that beaconing is merely a communication characteristic. It is neither good nor evil by definition. While it is true that malware heavily relies on beaconing nowadays, a lot of legitimate software also exhibits beaconing behaviour.

While we have made efforts to reduce false positives, this framework should be looked at as a means for beaconing identification to help reduce the search space for a threat hunt, not as a means for detection. That said, indicators produced by this framework, when combined with other IoCs, can potentially be used to detect on malicious activity.

The beacons we are interested in comprise traffic from a single running process on a particular host machine to one or more external IPs. Given that the malware can have both short (order of seconds) and long (order of hours or days) check-in intervals, we will restrict our attention to a time window that works reasonably for both and attempt to answer the question: “What is beaconing in my environment right now or recently?” We have also parameterized the inputs to the framework to allow users to configure important settings like time window, etc. More on this in upcoming sections.

When dealing with large data sets, such as network data for an enterprise, you need to think carefully about what you can measure, which allows you to scale effectively. Scaling has several facets, but for our purposes, we have the following requirements:

1. Work can be parallelised over different shards of data stored on different machines
2. The amount of data that needs to move around to compute what is needed must be kept manageable.

Multiple approaches have been suggested for detecting beaconing characteristics, but not all of them satisfy these constraints. For example, a popular choice for detecting beacon timing characteristics is to measure the interval between events. This proves to be too inefficient to use on large datasets because the events can't be processed across multiple shards.

Driven by the need to scale, we chose to detect beaconing by bucketing the data in the time window to be analyzed. We gather the event count and average bytes sent and received in each bucket. These statistics can be computed in MapReduce fashion and values from different shards can be combined at the coordinating node of an Elasticsearch query.

Furthermore, by controlling the ratio between the bucket and window lengths, the data we pass per running process has predictable memory consumption, which is important for system stability. The whole process is illustrated diagrammatically below:

A key attribute of beaconing traffic is it often has similar netflow bytes for the majority of its communication. If we average the bytes over all the events that fall in a single bucket, the average for different buckets will in fact be even more similar. This is just the law of large numbers in action. A good way to measure similarity of several positive numbers (in our case these are average bucket netflow bytes) is using a statistic called the coefficient of variation (COV). This captures the average relative difference between the values and their mean. Because this is a relative value, a COV closer to 0 implies that values are tightly clustered around their mean.

We also found that occasional spikes in the netflow bytes in some beacons were inflating the COV statistic. In order to rectify this, we simply discarded low and high percentile values when computing the COV, which is a standard technique for creating a robust statistic. We threshold the value of this statistic to be significantly less than one to detect this characteristic of beacons.

For periodicity, we observed that signals displayed one of two characteristics when we viewed the bucket counts. If the period was less than the time bucket length (i.e. high frequency beacons), then the count showed little variation from bucket to bucket. If the period was longer than the time bucket length (i.e. low frequency beacons), then the signal had high autocorrelation. Let's discuss these in detail.

To test for high frequency beacons, we use a statistic called relative variance (RV). The rate of many naturally occurring phenomena are well described by a Poisson distribution. The reason for this is that if events arrive randomly at a constant average rate and the occurrence of one event doesn’t affect the chance of others occurring, then their count in a fixed time interval must be Poisson distributed.

Just to underline this point, it doesn’t matter the underlying mechanisms for that random delay between events (making a coffee, waiting for your software to build, etc.)— if those properties hold, their rate distribution is always the same. Therefore, we expect that the bucket counts to be Poisson distributed for much of the traffic in our network, but not for beacons, which are much more regular. A feature of the Poisson distribution is that its variance is equal to its average, i.e. its RV is 1. Loosely, this means that if the RV of our bucket counts is closer to 0, the signal is more regular than a Poisson process.

Autocorrelation is a useful statistic for understanding when a time series repeats itself. The basic idea behind autocorrelation is to compare the time series values to themselves after shifting them in time. Specifically, it is the covariance between the two sets of values (which is larger when they are more similar), normalized by dividing it by the square root of the variances of the two sets, which measures how much the values vary among themselves.

This process is illustrated schematically below. We apply this to the time series comprising the bucket counts: if the signal is periodic then the time bucketed counts must also repeat themselves. The nice thing about autocorrelation from our perspective is that it is capable of detecting any periodic pattern. For example, the events don’t need to be regularly spaced but might repeat like two events occurring close to one another in time, followed by a long gap and so on.

We don’t know the shift beforehand that will maximize the similarity between the two sets of values, so we search over all shifts for the maximum. This, in effect, is the period of the data — the closer its autocorrelation is to one, the closer the time series is to being truly periodic. We threshold the autocorrelation close to one to test for low frequency beacons.

Finally, we noted that most beaconing malware these days incorporates jitter. How does autocorrelation deal with this? Well first off, autocorrelation isn’t a binary measure — it is a sliding scale: the closer the value is to 1 the more similar the two sets of values are to one another. Even if they are not identical but similar it can still be close to one. In fact, we can do better than this by modelling how random jitter affects autocorrelation and undoing its effect. Provided the jitter isn’t too large, the process to do this turns out to be about as complex as just finding the maximum autocorrelation.

In our implementation, we’ve made the percentage configurable, although one would always use a small-ish percentage to avoid flagging too much traffic as periodic. If you'd like to dig into the gory details of our implementation, all the artifacts are available as a GitHub release in our detection rules repository.

How do we do this using Elasticsearch?

Elasticsearch has some very powerful tools for ad hoc data analysis. The scripted metric aggregation is one of them. The nice thing about this aggregation is that it allows you to write custom Painless scripts to derive different metrics about your data. We used the aggregation to script out the beaconing tests.

In a typical environment, the cardinality of the distinct processes running across endpoints is rather high. Trying to run an aggregation that partitions by every running process is therefore not feasible. This is where another feature of the Elastic Stack comes in handy. A transform is a complex aggregation which paginates through all your data and writes results to a destination index.

There are various basic operations available in transforms, one of them being partitioning data at scale. In our case, we partitioned our network event logs by host and process name and ran our scripted metric aggregation against each host-process name pair. The transform also writes out various beaconing related indicators and statistics. A sample document from the resulting destination index is as follows:

As you can see, the document contains valuable beaconing-related information about the process. First off, the beacon_stats.is_beaconing indicator says whether or not we found the process to be beaconing. If it is, as in the case above, the document will also contain important metadata, such as the frequency of the beacon. The indicator beacon_stats.periodic says whether or not the signal is a low-frequency beacon, while the indicator beacon_stats.low_count_variation indicates whether or not it is a high-frequency beacon.

Furthermore, the indicators beacon_stats.low_source_bytes_variation and low_destination_bytes_variation indicate whether or not the source and destination bytes sent during the beaconing communication were more or less uniform. Finally, you will also notice the beaconing_score indicator, which is a value from 1-3, representing the number of beaconing tests satisfied by the process for that time period.

Writing such metadata out to an index also means that you can search for different facets of beaconing software in your environment. For example, if you want to search for low frequency beaconing processes in your environment, you would query for documents where the beacon_stats.periodic indicator is true and beacon_stats.low_count_variation is false. You can also build second order analytics on top of the indexed data, such as using anomaly detection to find rare beaconing processes, or using a significant terms aggregation to detect lateral movement of beaconing malware in your environment.

Finally, we’ve included several dashboards for your threat hunters and analysts to use for monitoring beaconing activity in your environment. These can be found in the release package as well.

Tuning parameters and filtering

Advanced users can also tune important parameters to the scripted metric aggregation in the transforms, like jitter percentage, time window, etc. If you'd like to change the default parameters, all you would need to do is delete the transform, change the parameters, and restart it. The parameters you can tune are as follows:

• number_buckets_in_range: The number of time buckets we split the time window into. You need enough to ensure you get reasonable estimates for the various statistics, but too many means the transform will use more memory and compute.
• time_bucket_length: The length of each time bucket. This controls the time window, so the larger this value the longer the time window. You might set this longer if you want to check for very low frequency beacons.
• number_destination_ips: The number of destination IPs to gather in the results. Setting this higher increases the transform resource usage.
• max_beaconing_bytes_cov: The maximum coefficient of variation in the payload bytes for the low source and destination bytes variance test. Setting this higher will increase the chance of detecting traffic as beaconing, so would likely increase recall for malicious C2 beacons. However, it will also reduce the precision of the test.
• max_beaconing_count_rv: The maximum relative variance in the bucket counts for the high frequency beacon test. As with max_beaconing_bytes_cov, we suggest tuning this parameter based on the kind of tradeoff you want between precision and recall.
• truncate_at: The lower and upper fraction of bucket values discarded when computing max_beaconing_bytes_cov and max_beaconing_count_rv. This allows you to ignore occasional changes in traffic patterns. However, if you retain too small a fraction of the data, these tests will be unreliable.
• min_beaconing_count_autocovariance: The minimum autocorrelation of the signal for the low frequency beacon test. Lowering this value will likely result in an increase in recall for malicious C2 beacons, at the cost of reduced test precision. As with some of the other parameters mentioned above, we suggest tuning this parameter based on the kind of tradeoff you want between precision and recall.
• max_jitter: The maximum amount by which we assume that a periodic beacon is jittered, as a fraction of its period.

You can also make changes to the transform query. We currently look for beaconing activity over a 6h time range, but you can change this to a different time range. As mentioned previously, beaconing is not a characteristic specific to malware and a lot of legitimate, benign processes also exhibit beaconing-like activity.

In order to curb the false positive rate, we have included a starter list of filters in the transform query to exclude known benign beaconing processes that we observed during testing, and a list of IPs that fall into two categories:

1. The source IP is local and the destination is remote
2. For certain Microsoft processes, the destination IP is in a Microsoft block

You can add to this list based on what you see in your environment.

Evaluation

In order to measure the effectiveness of our framework as a reduced search space for beaconing activity, we wanted to test two aspects:

1. Does the framework flag actual malicious beaconing activity?
2. By how much does the framework reduce the search space for malicious beacons?

In order to test the performance on malware beacons, we ran the transform on some synthetic data as well as some real malware! We set up test ranges for Emotet and Koadic, and also tested it on NOBELIUM logs we had from several months ago. The results from the real malware tests are worth mentioning here.

For NOBELIUM, the beaconing transform catches the offending process, rundll32.exe, as well as the two destination IPs, 192.99.221.77 and 83.171.237.173, which were among the main IoCs for NOBELIUM.

For Koadic and Emotet as well, the transform was able to flag the process as well as the known destination IPs on which the test C2 listeners were running. The characteristics of each of the beacons were different. For example, Koadic was a straightforward, high-frequency beacon that satisfied all the beaconing criteria being checked in the transform i.e. periodicity, as well as low variation of source and destination bytes. Emotet was slightly trickier since it was a low frequency beacon with a high jitter percentage. But we were able to detect it due to the low variation in the source bytes of the beacon.

To test the amount of reduction in search space, we ran the transform over three weeks on an internal cluster that was receiving network event logs from ~ 2k hosts during the testing period. We measured the reduction in search space based on the number of network event log messages, processes, and hosts an analyst or threat hunter would have to sift through before and after running the transform, in order to identify malicious beacons. The numbers are as follows:

While the reduction in search space is obvious, another point to note is the scale of data that the transforms are able to churn through comfortably, which becomes an important aspect to consider, especially in production environments. Additionally, we have also released dashboards (available in the release package), which track metrics like prevalence of the beaconing processes, etc. that can help make informed decisions about further filtering of the search space.

While the released dashboards, and the statistics in the above table are based on cases where the beacon_stats.is_beaconing indicator is true i.e. beacons that satisfy either of the beaconing tests, threat hunters may want to further streamline their search by starting with the most obvious beaconing-like cases and then moving on to the less obvious ones. This can be done by filtering and searching by the beacon_stats.beaconing_score indicator instead of beacon_stats.is_beaconing, where a score of 3 indicates a typical beacon (satisfying tests for periodicity as well as low variation in packet bytes), and score of 1 indicates a less obvious beacon (satisfying only one of the three tests).

For reference, we observed the following on our internal cluster:

What's next

We’d love for you to try out our beaconing identification framework and give us feedback as we work on improving it. If you run into any issues during the process, please reach out to us on our community Slack channel, discussion forums, or even our open detections repository. Stay tuned for Part 2 of this blog, where we’ll cover going from identifying beaconing activity to actually detecting on malicious beacons!

Try out our beaconing identification framework with a free 14-day trial of Elastic Cloud.

In 8.3, our Elastic Stack Machine Learning team introduced a way to import third party Natural Language Processing (NLP) models into Elastic. As security researchers, we HAD to try it out on a security dataset. So we decided to build a model to identify malicious command lines by fine-tuning a pre-existing model available on the Hugging Face model hub.

Upon finding that the fine-tuned model was performing (surprisingly!) well, we wanted to see if it could replace or be combined with our previous tree-based model for detecting Living off the Land (LotL) attacks. But first, we had to make sure that the throughput and latency of this new model were reasonable enough for real-time inference. This resulted in a series of experiments, the results of which we will detail in this blog.

In this blog, we will briefly talk about how we fine-tuned a transformer model meant for a masked language modeling (MLM) task, to make it suitable for a classification task. We will also look at how to import custom models into Elastic. Finally, we’ll dive into all the experiments we did around using the fine-tuned model for real-time inference.

NLP for command line classification

Before you start building NLP models, it is important to understand whether an NLP model is even suitable for the task at hand. In our case, we wanted to classify command lines as being malicious or benign. Command lines are a set of commands provided by a user via the computer terminal. An example command line is as follows:

**move test.txt C:\**

The above command moves the file test.txt to the root of the **C:** directory.

Arguments in command lines are related in the way that the co-occurrence of certain values can be indicative of malicious activity. NLP models are worth exploring here since these models are designed to understand and interpret relationships in natural (human) language, and since command lines often use some natural language.

Fine-tuning a Hugging Face model

Hugging Face is a data science platform that provides tools for machine learning (ML) enthusiasts to build, train, and deploy ML models using open source code and technologies. Its model hub has a wealth of models, trained for a variety of NLP tasks. You can either use these pre-trained models as-is to make predictions on your data, or fine-tune the models on datasets specific to your NLP tasks.

The first step in fine-tuning is to instantiate a model with the model configuration and pre-trained weights of a specific model. Random weights are assigned to any task-specific layers that might not be present in the base model. Once initialized, the model can be trained to learn the weights of the task-specific layers, thus fine-tuning it for your task. Hugging Face has a method called from_pretrained that allows you to instantiate a model from a pre-trained model configuration.

For our command line classification model, we created a RoBERTa model instance with encoder weights copied from the roberta-base model, and a randomly initialized sequence classification head on top of the encoder:

model = RobertaForSequenceClassification.from_pretrained('roberta-base', num_labels=2)

Hugging Face comes equipped with a Tokenizers library consisting of some of today's most used tokenizers. For our model, we used the RobertaTokenizer which uses Byte Pair Encoding (BPE) to create tokens. This tokenization scheme is well-suited for data belonging to a different domain (command lines) from that of the tokenization corpus (English text). A code snippet of how we tokenized our dataset using RobertaTokenizer can be found here. We then used Hugging Face's Trainer API to train the model, a code snippet of which can be found here.

ML models do not understand raw text. Before using text data as inputs to a model, it needs to be converted into numbers. Tokenizers group large pieces of text into smaller semantically useful units, such as (but not limited to) words, characters, or subwords — called token —, which can, in turn, be converted into numbers using different encoding techniques.

• Check out this video (2:57 onwards) to review additional pre-processing steps that might be needed after tokenization based on your dataset.
• A complete tutorial on how to fine-tune pre-trained Hugging Face models can be found here.

Importing custom models into Elastic

Once you have a trained model that you are happy with, it's time to import it into Elastic. This is done using Eland, a Python client and toolkit for machine learning in Elasticsearch. A code snippet of how we imported our model into Elastic using Eland can be found here.
You can verify that the model has been imported successfully by navigating to Model Management \> Trained Models via the Machine Learning UI in Kibana:

Using the Transformer model for inference — a series of experiments

We ran a series of experiments to evaluate whether or not our Transformer model could be used for real-time inference. For the experiments, we used a dataset consisting of ~66k command lines.

Our first inference run with our fine-tuned RoBERTa model took ~4 hours on the test dataset. At the outset, this is much slower than the tree-based model that we were trying to beat at ~3 minutes for the entire dataset. It was clear that we needed to improve the throughput and latency of the PyTorch model to make it suitable for real-time inference, so we performed several experiments:

Using multiple nodes and threads

The latency numbers above were observed when the models were running on a single thread on a single node. If you have multiple Machine Learning (ML) nodes associated with your Elastic deployment, you can run inference on multiple nodes, and also on multiple threads on each node. This can significantly improve the throughput and latency of your models.

You can change these parameters while starting the trained model deployment via the API:

**POST \_ml/trained\_models/\\<model\_id\\>/deployment/\_start?number\_of\_allocations=2&threa ds\_per\_allocation=4**

number_of_allocations allows you to set the total number of allocations of a model across machine learning nodes and can be used to tune model throughput. threads_per_allocation allows you to set the number of threads used by each model allocation during inference and can be used to tune model latency. Refer to the API documentation for best practices around setting these parameters.

In our case, we set the number_of_allocations to 2 , as our cluster had two ML nodes and threads_per_allocation to 4 , as each node had four allocated processors.

Running inference using these settings resulted in a 2.7x speedup on the original inference time.

Dynamic quantization

Quantizing is one of the most effective ways of improving model compute cost, while also reducing model size. The idea here is to use a reduced precision integer representation for the weights and/or activations. While there are a number of ways to trade off model accuracy for increased throughput during model development, dynamic quantization helps achieve a similar trade-off after the fact, thus saving on time and resources spent on iterating over the model training.

Eland provides a way to dynamically quantize your model before importing it into Elastic. To do this, simply pass in quantize=True as an argument while creating the TransformerModel object (refer to the code snippet for importing models) as follows:

**# Load the custom model**
**tm = TransformerModel("model", "text\_classification", quantize=True)**

In the case of our command line classification model, we observed the model size drop from 499 MB to 242 MB upon dynamic quantization. Running inference on our test dataset using this model resulted in a 1.6x speedup on the original inference time, for a slight drop in model sensitivity (exact numbers in the following section) .

Knowledge Distillation

Knowledge Distillation is a way to achieve model compression by transferring knowledge from a large (teacher) model to a smaller (student) one while maintaining validity. At a high level, this is done by using the outputs from the teacher model at every layer, to backpropagate error through the student model. This way, the student model learns to replicate the behavior of the teacher model. Model compression is achieved by reducing the number of parameters, which is directly related to the latency of the model.

To study the effect of knowledge distillation on the performance of our model, we fine-tuned a distilroberta-base model (following the same procedure described in the fine-tuning section) for our command line classification task and imported it into Elastic. distilroberta-base has 82 million parameters, compared to its teacher model, roberta-base , which has 125 million parameters. The model size of the fine-tuned DistilRoBERTa model turned out to be 329 MB, down from 499 MB for the RoBERTa model.

Upon running inference with this model, we observed a 1.5x speedup on the original inference time and slightly better model sensitivity (exact numbers in the following section) than the fine-tuned roberta-base model.

Dynamic quantization and knowledge distillation

We observed that dynamic quantization and model distillation both resulted in significant speedups on the original inference time. So, our final experiment involved running inference with a quantized version of the fine-tuned DistilRoBERTa model.

We found that this resulted in a 2.6x speedup on the original inference time, and slightly better model sensitivity (exact numbers in the following section). We also observed the model size drop from 329 MB to 199 MB after quantization.

Bringing it all together

Based on our experiments, dynamic quantization and model distillation resulted in significant inference speedups. Combining these improvements with distributed and parallel computing, we were further able to reduce the total inference time on our test set from four hours to 35 minutes. However, even our fastest transformer model was still several magnitudes slower than the tree-based model, despite using significantly more CPU resources.

The Machine Learning team here at Elastic is introducing an inference caching mechanism in version 8.4 of the Elastic Stack, to save time spent on performing inference on repeat samples. These are a common occurrence in real-world environments, especially when it comes to Security. With this optimization in place, we are optimistic that we will be able to use transformer models alongside tree-based models in the future.

A comparison of the sensitivity (true positive rate) and specificity (true negative rate) of our tree-based and transformer models shows that an ensemble of the two could potentially result in a more performant model:

As seen above, the tree-based model is better suited for classifying benign data while the transformer model does better on malicious samples, so a weighted average or voting ensemble could work well to reduce the total error by averaging the predictions from both the models.

What's next

We plan to cover our findings from inference caching and model ensembling in a follow-up blog. Stay tuned!

In the meanwhile, we’d love to hear about models you're building for inference in Elastic. If you'd like to share what you're doing or run into any issues during the process, please reach out to us on our community Slack channeland discussion forums. Happy experimenting!