CrowdStrike and Malwarebytes slam NSS Labs AV report

Vendor duo score lower than the likes of Cylance, Sophos and Trend Micro for 'security effectiveness'

Security vendors CrowdStrike and Malwarebytes have lashed out at cybersecurity testing house NSS Labs after it published the results of its latest end-point security test.

NSS Labs released the report on Tuesday after CrowdStrike failed in a last-ditch attempt to obtain an injunction in court blocking the report from being made public, claiming that the testing house acquired CrowdStrike's software illegally, and accusing its methods of being "unethical" and "deeply flawed".

NSS Labs tested products from 13 security vendors, with CrowdStrike and Malwarebytes scoring lower than their competitors on both effectiveness and pricing.

CrowdStrike scored 73.2 per cent for security effectiveness and was rated "below average", with only Malwarebytes scoring lower at 57.9 per cent.

Both vendors were given a final rating of "caution", and were also accused of being considerably more expensive than the other 11 in the test.

CrowdStrike is noted as having a total cost of ownership (TCO) per protected agent of $1,404, while Malwarebytes comes in at $1,235.

All other vendors (full list of vendors and scores below) in the report cost between $200 and $600 per agent.

CrowdStrike

CrowdStrike had initially opposed the publication of the results and sought an injunction in court, claiming that NSS Labs had illegally obtained a copy of its software through a reseller.

The court ruled against CrowdStrike and did not grant an injunction, but the vendor is continuing with legal action.

In a statement published on its website, CrowdStrike said that, having reviewed the report, the tests carried out on its Falcon end-point security software were "incomplete" and therefore "invalid".

It also claimed that all the prevention capabilities of its software were disabled during the test.

The blog stated: "On 14 February 2017, NSS Labs released a report available for purchase for approximately $12,000 claiming to analyse and address various advanced end-point protection products in the security marketplace.

"NSS' report plainly states that testing of CrowdStrike Falcon was incomplete, and therefore, the results are invalid.

"Including Falcon in the report based on an incomplete analysis is contrary to basic industry standards for testing."

The comments came one day after a previous post in which CrowdStrike launched a scathing attack on NSS Labs' testing ethics, branding it "deeply flawed".

In a statement sent to CRN, NSS Labs CTO Vikram Phatak said that the testing house is currently gagged by further legal action from CrowdStrike, adding that this is stopping the issue being debated publicly.

"While CrowdStrike's request for a temporary restraining order and preliminary injunction were denied by the Federal court, they are still suing us at present, and so we are limited in what we can say," he said.

"Whether or not it is their intent, their suit has the effect of keeping us from debating the facts publicly."

"We obviously disagree and are disappointed with CrowdStrike's characterisation of NSS as portrayed in their recent blog post. We would direct you to the AEP Group Test findings we published this [week] and as far as CrowdStrike's suit against NSS, we believe the judge's ruling and memorandum speak for themselves."

Malwarebytes

Malwarebytes was the only other vendor in the test to receive the rating of "caution", scoring less than CrowdStrike.

In a statement sent to CRN Malwarebytes CEO Marcin Kleczynski claimed that the vendor had previously been contacted by NSS Labs and asked to pay "tens of thousands of dollars" to participate in the test, but declined because of the testing methods employed.

"Malwarebytes does not endorse the test results by NSS Labs as we believe the testing methodology is severely flawed," he said.

"We were contacted by NSS several months ago to participate in this test to the tune of tens of thousands of dollars.

"Our results were downgraded because the testbed of 'malware' contained Microsoft and Malwarebytes-built programs. Put simply, NSS Labs wanted us to detect our own intellectual property as malicious."

NSS says on its website that its tests are free for vendors to participate in.

The testing house's FAQ page responds to the question "Are NSS Labs Group tests pay to play?" with: "No. Group tests are free to the vendors tested. Our policy is that no NSS Labs publication will ever be the result of a paid engagement, so if you see it in print, the test or research conducted was not the result of a paid engagement."

When asked by CRN whether Malwarebytes was asked to pay to participate in the test, NSS Labs' Phatak reaffirmed that it is not a pay-to-play firm.

"NSS Labs does not charge for, and never will charge for, participation in public tests," he said.

"Malwarebytes was invited to participate in the public group tests when NSS Labs initiated test coverage of the Advanced Malware Protection market. Malwarebytes installed the product of their choosing on November 23, 2016 and testing commenced shortly thereafter.

"The use of a blend of malicious and non-malicious software is purposeful and intended to assess the ability of a product to operate in actual deployments."

Top of the pile

While CrowdStrike and Malwarebytes scored 73.2 per cent and 57.9 per cent respectively for security effectiveness, every other vendor in the test scored at least 89.5 per cent - with Carbon Black scoring 100 per cent, earning it 'security recommended' status.

However, Carbon Black failed to be rated 'recommended' overall because of its price of $538 TCO per protected agent.

ESET was rated 'neutral' with an effectiveness score of 89.5 per cent.

Cylance, Fortinet, Invincea (recently acquired by Sophos), Kaspersky, McAfee, SentinelOne, Sophos, Symantec and Trend Micro were all recommended by NSS Labs based on their test score and price.

Cylance

AV vendor Cylance has long been a vocal critic of the testing culture in the cybersecurity industry, and published a highly critical blog post in December accusing testing houses of employing "pay-to-play" tactics and scoring vendors more favourably if they pay to commission a report.

In one particular test commissioned by Symantec, next-gen vendors SentinelOne and Cylance scored poorly in relation to Symantec itself, which scored 100 per cent across the board, and other legacy vendors.

"It seems that legacy AV vendors are more than happy to collude with certain testing houses who offer them 100 per cent efficacy ratings in tests so that they can maintain market share, despite widespread industry acceptance that traditional AV struggles to perform at a fraction of that efficacy," the blog stated.

Cylance has responded more favourably to NSS' report, declaring it "the most comprehensive advanced end-point security public test to date".

Chad Skipper, vice president of product testing at Cylance, said: "We appreciate NSS Labs independently corroborating what our customers have been saying for years, that CylancePROTECT [anti-virus product] is a proven replacement for traditional antivirus with AI-powered prevention blocking today's most advanced cyber threats.

"This is a big step forward in evolving security testing methodologies to match real-world environments, and we look forward to more progress in the coming months and years."

Full list of vendors tested and results, taken from the NSS Labs report (click to enlarge)