MSP launches AV testing website to stamp out 'misleading' reports
Cognition founder claims independent testing reports are rarely impartial because of vendor sponsorship
Security MSP Cognition has launched an anti-virus (AV) testing website to fight against the "misleading" testing culture in the cybersecurity industry.
Cognition - a Cylance and Palo Alto Networks partner - launched the testmyav.com website to give resellers and end users the resources to test AV products themselves instead of relying on tests sponsored or commissioned by vendors.
Cognition founder Carl Grottlieb told CRN that his motivation for setting up the site stems from a lack of integrity in the testing industry, claiming that vendor-sponsored testing reports can never truly be independent.
"The main issue is that the IT industry is filled with AV vendors and their paid-for tests claiming that they're all 100 per cent perfect, meanwhile customers of these products are getting infected with malware in dramatic numbers, so which is it?" he said. "Are the products perfect or are they fundamentally flawed?"
"There are some things that you won't see [in the reports], for example some testing organisations will give you editorial rights so once the testing is done they will offer you, for a fee, the ability to look at the results and retest.
"We've seen a test recently where one vendor retested nine times to get the result they wanted and you will never see that kind of thing because they don't show it, so there are some very strange practices going on."
Testmyav.com provides end users with access to tools, guides and malware to carry out their own tests on AV products, "rather than trusting vendors, testing companies and salespeople".
The website is funded by Cognition and receives no sponsorship or advertising revenue from vendors; and refers to no specific vendors.
Since launching around two months ago the website has been used by over 200 end user organisations around the globe, reaching as far as Australia and South America, including 11 AV vendors, Gottlieb said.
"We recognised that we need to get credibility back into the AV industry, and the most trustworthy and unbiased people out there are the end users themselves," he added.
"It's threatening to the vendors that don't have the quality to stand up against their marketing.
"The ones that haven't got a great product and are not confident in end users testing it are the ones that have gone very quiet."
Cylance slams test report
One particular Symantec-sponsored test has recently caused a stir in the cybersecurity sector, with Cylance publishing a damning blog on its website.
The report saw Symantec score highly in a number of tests, including an exploit protection test which saw it score 100 per cent on both protection and prevention.
"It must be said: some of the top players in the third-party AV testing industry have recently revealed themselves to be nothing more than pay-to-play capitulators who seek to line their pockets by perpetuating outmoded technologies while keeping more effective and innovative solutions out of the hands of the users who need them," the Cylance blog stated.
"Furthermore, it seems that legacy AV vendors are more than happy to collude with certain testing houses who offer them 100 per cent efficacy ratings in tests so that they can maintain market share, despite widespread industry acceptance that traditional AV struggles to perform at a fraction of that efficacy."
"In the most recent malware protection test performed by AV-Comparatives and the exploit test that was performed by MRG Effitas, which was 'commissioned' by Symantec, it comes as no surprise that the major benefactor for the report also turned out to be the major beneficiary of its findings."
While Symantec declined to comment when contacted by CRN, security research firm MRG Effitas CEO Chris Pickard defended its test strategy and said that Symantec had no control or influence on the testing process, other than choosing the products that were tested.
He also stated that all products were set up to use their default settings.
"MRG Effitas and AV-Comparatives take great care to ensure we test the latest available products in our comparatives tests," said Pickard.
"MRG Effitas and AV-Comparatives used Cylance Protect default configuration, as we did for all products that we tested.
"Symantec commissioned this MRG Effitas and AV-Comparatives review, which we make very clear in the report. Symantec had no prior access to the samples used, no visibility of our test process and no results have been falsified or adjusted in any way."