NSS Labs hits back at WatchGuard impartiality claims

'Pay-for-play' allegations are 100 per cent false, says security testing house

NSS Labs has dismissed WatchGuard's claims that it is no longer impartial as sour grapes stemming from it doing badly in its next-generation firewall (NGF) test.

The testing lab recently pitted the security vendor's XTM 2050 appliance against nine other NGFs from Dell SonicWall, Fortinet, Juniper, Palo Alto, Sourcefire and Stonesoft.

Although in general, vendors' scores were up on last year, WatchGuard finished well behind the field after its box missed several evasions NSS' testers threw at it.

In the aftermath of the report's publication, WatchGuard questioned the independence of the Austin-based outfit, accusing it of moving from an impartial testing house into a "pay-to-play" operator. Specifically, WatchGuard's vice president of corporate strategy, Dave Taylor, alleged that an NSS Labs representative had given him the impression that his score would improve if he became a subscriber to its research.

"We wish there were a test house out there that was more objective and science-based," he said.

Must do better

In response, NSS wheeled out several of its top brass to clarify to us exactly why WatchGuard failed its test, defend itself against those allegations and to talk us through its business model in detail.

WatchGuard finished well behind any of its competitors on both the Y and the X axes, which measured enterprise management and security effectiveness, and TCO per protected Mbps respectively.

Its score on the Y axis was affected by it missing evasions in three of the nine evasion categories, including TCP stream segmentation - potentially dangerous given its function at a low level in the communications stack. Evasions were also missed in the RPC fragmentation and HTML obfuscation categories. So although WatchGuard achieved a block rate of 91.3 per cent against exploits, once the exploits were obfuscated using some basic evasion techniques, it failed to block them, hitting its score hard.

Chief executive Vikram Phatak said: "The reason why they failed is they missed a number of very serious evasions, which really impacted the effectiveness of the device.

"Our answer would be for WatchGuard to do better. They simply need to address the evasion issue and their score will go up dramatically. There are going to be winners and losers and some products will be better than others. If everyone were perfect, it would make us very happy as it would be great for the industry."

WatchGuard also finished last on the X axis, which NSS attributed to its product performing at 2.6GBs rather than the 10GB advertised.

Ryan Liles, director of testing services at NSS Labs, said: "There is an easy way for them to move to the right – either drop the price or improve the performance."

Rebuttal

Phatak branded Taylor's claim that scores could be influenced by how much subscription revenue each vendor pays as "100 per cent false". If an NSS representative insinuated that, they would be fired, he added.

"I get that he is trying to protect his product, but that is sinking to a new low," he said. "Our subscriptions start at $75,000 (£50,000) and go up to $250,000 but we make it available at a discount to avoid the perception of a conflict of interest. But we are not going to give them valuable information for free."

Founded in 1991, NSS Labs began adding analyst capabilities last year and it now produces SVMs [security value maps] and comparative analyses that are written by analysts.

But Phatak waved off claims this means NSS is losing its edge as a science-based testing house.

"From talking to clients [that are enterprise end users], one thing they told us was missing was what it means for them," he said. "So we added in that layer. We now have 20 testers and six analysts and their role is to help make sense of the test results. But it is not subjective. There are a number of analysts out there and they are good firms. But they do not touch products. We do not know how you can be an effective analyst in 2013 if you do not touch product."

Feedback

Phatak said many of the NGF vendors that carry its recommended rating had made dramatic improvements since the 2012 test because they had responded to the feedback NSS had given.

"We have a good relationship with most folk that come through our lab," said Phatak. "They may not like the results, but they respect them."

Meanwhile, NSS said it would imminently publish an update on its recent blog post criticising unnamed vendors for changing its graphics in their marketing material.

Phatak confirmed that Check Point was the vendor at which NSS' ire was directed but said his firm needed to be clear on what the vendor did and did not do, stressing it removed a couple of competitors from its graphic rather than altering its position.