Testing house rejects Palo Alto's "pay-for-play" accusations

NSS Labs says allegations over its objectivity and accuracy made by next-generation firewall vendor are "dead wrong"

Security testing lab NSS Labs has angrily dismissed accusations from vendor Palo Alto Networks that it operates a "pay-for-play" approach.

Palo Alto Networks hit out at NSS last week after it was the only one of 12 participating vendors to receive a "caution" rating in its latest group test on next generation firewalls (NGFWs).

NSS handed Palo Alto's PA-3020 box a security effectiveness score of just 60.1 per cent, which it said was due partly to its reduced ability to block recent exploits. The product's block rate came out at 92.5 per cent, compared to its score of 96.4 per cent in 2013, when NSS handed it a "recommended rating".

In a blog post, Palo Alto senior vice president Lee Klarich said his firm intentionally did not participate in the test and that this put it at the disadvantage of not being able to configure and tune its box for the bake off.

"The reason we did not participate in this test is that over time we have come to believe that the NSS model of allowing vendor test tuning prior to public test is a ‘pay to play' approach and produces questionable objectivity and accuracy in results," Klarich said.

It is not the first time NSS' test results - and methodology - have been questioned by vendors, with both WatchGuard and FireEye locking horns with the Texas-based outfit in recent years.

NSS founder Bob Walder returned fire yesterday in a blog post responding to Klarich's claims.

Klarich's blog evaded the main issue that Palo Alto's NGFW "misses several critical evasions that leave its customers at risk", Walder said, adding that it contained some "serious inaccuracies".

"Pay to play' is a very strong accusation and is dead wrong, and Palo Alto Networks knows it," Walder wrote, linking through to an apology Palo Alto founder Nik Zur made in 2010 in the wake of similar accusations.

"The fact is that NSS does not charge any vendor for participation in any of our public group tests," Walder added. "The entire test is done on our dime, and the only input we ask from vendors is support in terms of supplying the most appropriate device, along with engineering support before and during the test, should we need it."

All NGFW products were tested using the pre-defined vendor recommended settings and NSS does not allow vendors to tune their devices, Walder said. He also noted that Palo Alto was quick to trumpet the results of the equivalent test last year when it did well.

Klarich claimed that Palo Alto had invested even more into its NGFW's security capabilities since last year and that he was struggling to understand why NSS came to such a "drastically different" result this time around.

"Importantly, the issues they've raised have never been observed in other tests conducted internally or with our install base of over 19,000 global enterprises," he said.