FireEye and AhnLab found wanting by NSS boffins

Independent testing lab puts six breach-detection systems through their paces

FireEye and AhnLab have been handed a rating of "caution" by NSS Labs following its first ever comparative analysis of breach-detection systems (BDS).

Four of the six vendors tested – Fidelis, Fortinet, Cisco-owned Sourcefire and Trend Micro – bagged a "recommended" rating from the testing house, which evaluated each for security effectiveness, performance and total cost of ownership (TCO).

But FireEye and AhnLab were given the lowest rating of "caution" after their systems were marked down on their ability to catch malware and their cost. FireEye has been quick to reject the findings of the report, claiming the NSS' methodology was "flawed".

The NSS said third-party comparative testing is critical to determine if vendors in such a "quickly evolving" space are delivering on their marketing spiel.

"Breach-detection systems are one of the most rapidly evolving security technologies out there today and with that comes a lot of marketing hype and vendor claims," said Vikram Phatak, chief executive of NSS Labs.

"We are excited to be the first to test and provide empirical insight into the performance and capabilities of the leading vendors in this emerging market."

FireEye and AhnLab were both given "below average" security effectiveness ratings, scoring 94.5 per cent and 94.7 per cent respectively. The other four vendors' security effectiveness scores ranged between 98.4 and 99.1 per cent.

AhnLab's MDS was found to have detected 100 per cent of HTTP malware, 94 per cent of email malware and 90 per cent of exploits. It misidentified seven per cent of legitimate traffic as malicious (false positives). FireEye gave no false positives and detected 95 per cent of HTTP malware, 96 per cent of email malware and 93 per cent of exploits.

But FireEye and AhnLab also finished bottom for TCO per protected-Mbps ($427 and $468 respectively). Meanwhile, the solution offering the lowest TCO ($231) – Sourcefire – boasted one of the highest security effectiveness ratings.

Trend Micro's Deep Discovery Inspector was awarded the highest security rating of the six – 99.1 per cent – with Fortinet and Sourcefire both scoring 99.0 per cent. Fidelis scored 98.4 per cent.

Phatak said the findings would allow executives to make "educated purchasing decisions".

In a blog, FireEye's senior vice president of products, Manish Gupta, hit out at the NSS report for its "poor" sample selection and test methodology, complaining that there were no zero-day exploits in the test sample.

"We declined to participate in this test because we believe the NSS methodology is severely flawed," he said. "In fact, the FireEye product they used was not even fully functional, leveraged an old version of our software and didn't have access to our threat intelligence (unlike our customers)."