Sun admits error in benchmark results

The workstation giant has finally conceded that its Java benchmarkresults for JIT were unintentionally misleading

Sun has admitted that it 'committed an unintentional error' when it published Java benchmark results for its Just In Time (JIT) compiler in October.

In some benchmark results released last week, Sun suggested that its JIT compiler makes Java applications run 50 per cent faster on Sun's Solaris operating system than on Windows NT (PC Dealer, 12 November).

The controversy started when Pendragon Software, creator of the Caffeine-Mark benchmark, claimed last week that Sun's product was designed to detect parts of the benchmark, thus producing misleading results.

Sun's initial response to the accusation was a flat-out denial, stating its product had 'blown the doors off' the CaffeineMark benchmark, and hinting that Pendragon's criticism was inspired by pressure from Microsoft.

Sun claimed it had merely optimised its compiler.

However, independent experts have since verified Pendragon's findings, confirming that the JIT compiler contains a portion of the CaffeineMark benchmark code. This enables the compiler to detect and skip this code, thereby completing the program faster - a well known but frowned upon method of producing better results.

Sun has posted a brief message on its Website, admitting that in its press release, 'the effect of the benchmark-specific optimisation was described as if it were an indication of overall application performance'.

It also stated that it has retracted the press release.

Sun offered an explanation for the 'unintentional error', adding that it believed CaffeineMark 3 has a large section of dead code, which can be removed without changing the program's result. The Website message states that as an engineering experiment, the optimiser looked for and removed the code.

The optimisation was created primarily to demonstrate what the performance would be if the full general-purpose dead code elimination algorithm were implemented.

The message then goes on to state that: 'As originally created, it was not intended that the results of this prototype would be used, except as an engineering test of potential future development.'