Is MITRE ATT&CK the new “next-gen”?

It’s been 18 months since Endgame became the first endpoint protection vendor to go through a publicly disclosed ATT&CK tactics-based simulation run by the MITRE Corporation. Our early adoption and commitment to the ATT&CK matrix is what makes us the only endpoint protection vendor capable of making nation-state level protection attainable by everyone, regardless of the size or skills of a security team.

Only a year ago, Crowdstrike became the second vendor to jump aboard the MITRE ATT&CK train, and I called on the rest of the industry to use this framework and evaluation to move away from scare-ware marketing and buzzword bingo. See also #nomorenextgen.

And then most recently, in November 2018, MITRE published the raw data associated with their first round of commercial evaluations against the ATT&CK elements used in an APT3-like simulation. Finally, we have some actual data that organizations can use to evaluate and compare the most relevant vendors in the Endpoint Protection space, right? Well, kinda.

Remember when everyone was "next-gen" and everyone was "machine learning"?

It turns out that most organizations don’t want to run their own analysis.

If you kept up with the various vendor responses, you will have noticed that with the exception of one or two vendors, the facts behind the evaluation and usefulness of the data are already starting to get lost among the desperate scrambling for the “ATT&CK certified” tick-box badge of honor. In the same way that “Next-Gen” became the non-sensical feature and capability everyone wanted in 2016, the huge value of ATT&CK is at risk of becoming diluted outside of the practitioner community.

But there are still signs of hope as Forrester prepares to release its independent analysis of the first round of MITRE ATT&CK evaluation data. Forrester’s Josh Zelonis already published the data-gathering python scripts behind the Measuring Vendor Accuracy blog post (which mapped to Endgame’s own analysis). With other analyst firms asking how they can use data in their market research, there are strong indications that ATT&CK is reaching peak adoption outside of the vendor industry.

Without considering your organization's own context, ATT&CK is just another data-point

Although ATT&CK is more useful than any other testing framework that’s come before it, it’s not perfect. You can argue that it rewards vendors who alert on everything, despite the noise and false-positive rate that would make it useless for almost anyone. It doesn’t evaluate how you use the detection data, nor how vendors streamline the response and remediation to a detection. And it doesn’t factor in the financial and human costs associated with actually using the solutions in the same way as the evaluation, such as data storage costs, MDR or managed alerting services, cloud-only analysis, or even the fact that every vendor provided their own dedicated Blue Team to sift through alerts and point out detections that weren’t immediately obvious.

Remember though, post-breach detection is but one part of a capable endpoint protection solution. It doesn’t evaluate the accuracy or effectiveness of the prevention capabilities to stop even the most common initial methods of compromise.

More on that next month when we will look forward to the vendor clickbait machine running overdrive as NSS Labs releases the 2019 AEP test the week of RSA.