Check Point has knuckles rapped for doctoring report

Security vendor whited out competitors in its presentation of recent NSS next-generation firewall report

Check Point has had its wrist slapped for doctoring the graphics of a recent NSS Labs report in its own marketing bumf.

According to various sources, Check Point whited out some of the competitors clustered around it in NSS Labs' 2013 next-generation firewall (NGF) report when presenting the results in a PDF on its website.

Last Friday, NSS president Bob Walder penned an angry blog reminding vendors of the strict guidelines they must follow when reusing its reports in their own marketing puff.

Although NSS did not cite Check Point by name, several channel sources said Walder's ire was directed at the Israel-based vendor, whose 12600 appliance was one of seven NGFs to be rated as above average by the independent testing lab. The subject has also stirred up heated discussion on LinkedIn. The vendor appears to have since modified the PDF containing the offending graphic.

Walder reminded vendors that they are forbidden from altering its words or changing its graphics or the way it presents results.

"Which is why it pains us greatly when vendors take liberties with our stuff," he said. "Like, say, reproducing the latest SVM [security value map] graphic from our NGFW report and... wait for it... removing the data points of its competitors. Surely no one would do that, would they?"

NSS Labs also ran the rule over NGFs from Dell SonicWall, Fortinet, Juniper, Palo Alto, Sourcefire, Stonesoft and WatchGuard, rating each for security effectiveness, performance, enterprise management capabilities and total cost of ownership.

Dave Taylor, vice president of corporate strategy at WatchGuard, branded Check Point's actions as "silly".

"I am sure the marketing people there thought it would make it easier for customers to see where Check Point was, but it came across that they were trying to take the competition out. NSS obviously took this seriously and called them on the carpet."

Most vendors' scores improved from 2012, which NSS said was a reflection of the time and effort they have invested in addressing many of the stability, leakage, performance and security effectiveness concerns it expressed last year. Eight out of the nine NGFs tested scored more than 90 per cent for security effectiveness.

WatchGuard was the only vendor to be handed a below-average rating on both axes – enterprise management and security effectiveness, and TCO per protected Mbps. Its XTM 2050 appliance was found to detect only about half of the evasion techniques NSS testers threw at it.

Taylor expressed his frustrations at the results and questioned whether or not NSS was truly independent from the vendors it tests.

"This industry really needs an impartial testing house that can do a real deep dive on how these boxes perform," he said. "NSS started off as that but the new owner has taken the reputation NSS has built up and parlayed that into revenue and taken it into the ‘pay to play' space."

He pointed out that WatchGuard and Juinper's positions had reversed between 2012 and 2013, despite claiming that each have "roughly the same products" as last year, when WatchGuard subscribed to NSS' research.

NSS was unavailable for comment when we contacted them last night, while we are awaiting comment from Check Point.