NSS Labs hits back at FireEye 'untruths'

FireEye says NSS Labs' methodology 'severely flawed' following poor performance in test

A war of words has erupted between FireEye and an independent testing house that questioned the vendor's ability to catch malware.

NSS Labs handed FireEye a "caution rating" in its first ever comparative analysis of breach detection systems (BDS) after its box scored a "below average" security effectiveness rating of 94.5.

All but one of the other vendors tested scored more than 98, with Fortinet, Sourcefire and Trend Micro all scoring 99 or more.

NSS said its findings would help end users cut through the vendor marketing puff in what is a "rapidly evolving" market.

In a forceful blog, FireEye product boss Manish Gupta moved immediately to discredit NSS Labs' findings, claiming the methodology it used was "severely flawed". The FireEye product the NSS used wasn't even fully functional, used an old version of its software and did not have access to its threat intelligence, Gupta said.

But this was like a red rag to a bull for NSS Labs, whose founder Bob Walder last night responded with an equally robust blog designed to address "a number of untruths and misdirections" in Gupta's missive.

Walder responded to Gupta's claims one by one, saying many were untrue. The product it tested was indeed a fully functional product installed and configured by FireEye engineers, he claimed.

According to Gupta, FireEye insisted the only way to properly test was to run in a real environment, but that the NSS declined to change its testing methodology.

Walder, however, rejected this version of events. He claimed NSS Labs does use a real, live environment, "with real PCs going to real, live malicious URLs". Walder confirmed FireEye did ask for it to change its methodology, but claimed this happened only after it saw the results, something the NSS "clearly cannot do".

"In the grand scheme of things, FireEye's results were not that bad," Walder said. "The real issue here is that FireEye now has credible competition in the BDS marketplace and the data from this NSS test shows it."

Walder said it was rare for NSS to respond to criticism from vendors that have performed poorly in its tests, although a similar spat with WatchGuard erupted last year over its next-generation firewall report.