Transparency In Third-Party Testing

Before making a major purchase, chances are you shop around, compare products with a critical eye, and rely heavily on the experiences and opinions of people you trust to inform your buying decision. For example, before purchasing a car, customers often turn to friends and third-party reviews, such as Consumer Reports, Kelley Blue Book, or the wisdom of crowds captured in sites such as Yelp and Angi. These sources can validate claims made by the automobile manufacturer or salesman, and are essential to help inform consumers and provide transparency.

But what happens if we’re talking about computer security instead of cars? A consumer seeking a third-party review might feel somewhat bewildered for a few key reasons. First, some vendors may not participate in public third-party testing. In addition, the methodologies employed by testing companies may not be fully transparent to customers or even agreed upon by those vendors whose products are being evaluated. More transparency is required to incentivize both vendors and testers to adopt uniform and mutually understood standards for third-party evaluations, as occurs in many other industries. The expanded participation in third-party assessments by vendors, and an improvement of third-party tests, can best be achieved by fully adopting testing standards, such as those drafted by the Anti-Malware Testing Standards Organization (AMTSO), of which Endgame is a participating member. These assessments are a key step toward providing better transparency across the industry, and arming consumers with the required knowledge to make more informed decisions.

Moving Beyond Self-Evaluations

At Endgame, we maniacally and rigorously test our platform internally in environments that we have set up to mimic conditions experienced by our customers, including nascent threats. We exercise our detection engines, probe our protection mechanisms, even proactively and adversarially probe our machine learning models for blind spots. As such, we believe our product provides superior protection to our customers. However, moving beyond self-evaluations is critical for at least three reasons.

First, we’re not naïve to the fact that---subtly, but importantly---our product has been built to protect against threats that fit our understanding of the contemporary threat landscape. We have designed our product to generalize to evolving threats using behavioral-based and signatureless detections. Layered protections minimize the impact of a miss at any single point in the attack chain. As such, we’re confident in our ability to protect customers. It’s probably safe to assume that most mature vendors believe that their product is superior to competitors based on self-critical internal evaluations and subsequent improvements. We certainly do. But dataset bias is real, and by nature invisible. So, it behooves the rigorous and objectively-minded to submit to third-party testing, even when the testing results are never made public. It explicitly benefits the vendor by identifying any hidden weaknesses and strengths.

Related is that fact that internal testing methodologies create a conflict of interest with inherent bias, either innocently or purposefully, because the vendor benefits from self-assigned high grades. Vendors optimize to their success metrics. Methodological massaging to boost metrics is a direct consequence of financial incentives to broadcast the metrics’ supremacy. Of course, anyone can score well on a test they write for themselves. As amateur statistician Homer Simpson rightly noted, “people can come up with statistics to prove anything.” This notion of juking the stats was popularized on The Wire, but reflects a serious challenge when defining testing methodology and evaluation criteria. Thus, integrity compels an honest vendor to participate in independent third-party tests.

Thirdly, publishing test results is critical to the consumer. While many customers have the resources to conduct their own extremely rigorous product evaluation, many must rely on third-party tests as an objective confirmation that products protect as advertised. For vendors who do not participate in public tests, the consumer is left to wonder whether private test results (if any) were unsatisfactory or would otherwise be embarrassing to the vendor if published.

To summarize, vendors should participate in public third-party tests with motivations of objectivity, integrity, credibility, in addition to the obvious value-add as a critical comparative tool for customers. Just like the attackers, these tests also must evolve to evaluate against the broader range of attacker techniques in addition to malware, such as fileless attacks. Customers can and should demand this. Endgame is proud to participate in public third-party testing, including AV Comparatives and SE Labs, and we look forward to sharing results of ongoing third-party public testing in the near future.

But Who is Testing the Test?

Just as imperative as the need for vendors to participate in independent third-party testing, the security community must also ensure credibility in how tests are performed. From a customer's perspective, there are several elements of testing in infosec, especially related to potential conflicts of interest, that should at least raise an eyebrow. To paraphrase Dennis Batchelder, current president of AMTSO, at the most recent AMTSO meeting, in what other industry do you find the following scenarios?

• A testing organization that is evaluating a product may get paid by the vendor (or by a competitor) that is authorizing the test.
• A testing organization may test an unwilling vendor’s product for a use case or in an environment not intended by the vendor.
• A testing organization may receive significant input (e.g., malware samples) from the vendor being tested.
• A vendor may monitor or influence the outcome of the test while it is being performed, even changing the behavior of the product while it is being evaluated.
• A vendor may pay the testing organization for additional privileges or upgrades to the basic test.

Let’s be clear, here. Nonprofit testing is important, but it is not the only option. Testing organizations are expending significant effort to develop realistic testing methodologies, which requires resources. At Endgame, we pay testers to test our product. Furthermore, the methodology by some testers may not be perfectly aligned with Endgame’s protection strategy, and we appreciate testers who seek our input into how to exercise our product’s functionality. Customers must be able to trust that money isn’t driving the outcomes of the test, and that the relationship of the vendor to tester is indeed independent rather than influential. In both cases---payment and methodology refinement---the best way to ensure that these relationships don’t undermine customers’ interests is for both parties to agree to the highest standards of transparency.

Specifically for this reason, in May AMTSO adopted a draft standards document that outlines appropriate protocol for testers and vendors when evaluating anti-malware solutions. Like many technical standards documents (e.g., ISO 27001), the standards do not mandate specific implementation details. Rather, the document establishes a foundational protocol that promotes tests that are impartial, transparent, and notably address the inherent conflicts of interest present in our industry. Without this basic foundation that outlines protocols for fairness and transparency, more detailed standards aimed to improve testing quality are a proverbial house upon the sand. When testing methodology is transparent and becomes the subject of public scrutiny, testing quality is bound to follow.

The draft status of the standards is intended to help AMTSO, testers, and vendors work through the implementation details in providing transparency and rules of engagement for vendors and testers. Through this process, AMTSO will revise the draft as needed. Once adopted, AMTSO (comprised of a consortium of vendors and testers and interested parties) then becomes an organization that, indeed, can implement testing standards, which have notably been absent in its decade-long history.

Endgame is proud to support AMTSO in the organization’s effort to set standards of transparency, fairness and impartiality in third-party validation of security products. I am hopeful that after any necessary minor revisions revealed during the exercise of the draft, AMTSO member organizations will come to a consensus and fully adopt the testing protocol standards document. Similarly, Endgame is committed to additional tests that extend beyond malware to cover a broad range of attacker techniques. Customers deserve it.

Moving Forward with Openness

The call to vendors and testers for openness and transparency is an invitation to somewhat unfamiliar territory in infosec. As Nate Fick, our CEO, sharply called out in a 2017 New America conference keynote: “Security is bedeviled by a dark arts culture that’s both self-serving and wrong. Security is no more a dark art than finance or real estate or tax policy or animal husbandry...and to wrap itself in a cape of black magic is nothing more than self-importance and a vain attempt at job security. It’s bad for customers.”

We are in a transformative stage of the infosec industry. As enterprises iterate on compliance with frameworks such as NIST, there is a parallel demand to ensure greater transparency of the products to ensure they perform as advertised. Just as Kelley Blue Book and Angi have provided independent means to assess various products, so too is infosec moving in this direction. Driven by the expectation of methodologically rigorous and fair independent evaluation, vendors are participating in these assessments, and consumers are demanding them. While testing methodologies may not be perfect, openly participating in them is crucial, and that’s why we have been and will continue to be actively engaged in a number of public tests with various vendors. Together as members of AMTSO, we’ll adopt what are now draft standards, and can finally move the community forward, removing the ‘dark arts’ mystique of the industry, and provide protection assurances to consumers, corporations and national security assets. Transparency is key. We should expect nothing less.