Benchmarks can mean whatever you want them to mean, it has always seemed. Although useful as a rough guide to performance, and sometimes price/performance, technology companies are famous for interpreting complex benchmark results as victories over their competition and them employing visual aids such as planes, snails, and automobiles to demonstrate their point.

The latest benchmark brouhaha may be over the newly developed SPECjAppServer2004 benchmark from SPEC (the Warrenton, VA-based Standard Performance Evaluation Corp.) This benchmark is the latest in the "Java Client/Server" category, and replaces SPECjAppServer 2002, which has now been retired. It "is designed to measure the performance of J2EE 1.3 application servers, (and) includes an enhanced workload by adding a web tier, JMS, and other changes to SPECjAppServer2002," according to SPEC.

And it has lent itself to spectacular success by IBM, according to a recent release coming out of Somers, NY. In fact, IBM's WebSphere Application Server "outperformed the competition by 64 percent," according to the release. Competitor BEA was quick to dispute this claim, however, pointing to its own success with its WebLogic on this benchmark.

IBM executive Sandy Carter said "these latest results further demonstrate the industry leading capability of IBM WebSphere Application Server," said Sandy Carter, Vice President, IBM WebSphere Software. And the IBM release went onto say that the "results submission involved more than 22,000 concurrent clients and produced more than 2,921 complex business transactions per second. The IBM submission represented a complete IBM solution bringing together the latest version of WebSphere Application Server software, DB2 Universal Database and IBM System p5 550 servers running SUSE Linux. This powerful solution outperformed a combination of BEA and Oracle software running on Sun Solaris and at significantly lower total cost of ownership across the full configuration."

BEA executive Eric Stahl , the company's Senior Director of Investor Relations, begs to differ. He referred to IBM's results as "inefficient," and has developed charts that show WebLogic as the clear winner in this contest. In his blog, Stahl says "the new (SPECjAppServer2004) result from BEA sets a new scalability world record, beating the previous record which was already held by BEA," and he provides a chart in his blog to illustrate his point.

Neither Oracle nor Sun had a comment about the IBM release by the time this story was being posted. Oracle has also not published its results on this benchmark, although Sun has. The results table is available to anyone at the SPEC website.

Which leads to the price/performance question. Unlike the 2002 benchmark, the new one does not include a price/performance metric, so ladies and gentlement, please re-start your engines. IBM says it based its 64% margin of victory on the following comparison:

IBM/WAS/DB2
AppServer Hardware: $192,000
AppServer Software: $480,000
Database Hardware: $377,000
Database Software: $181,000
Total = $1,230,000
Total $/JOPS = $1,230,000/2921 JOPS = $421/JOP

SUN/BEA/Oracle
AppServer Hardware: $38,000
AppServer Software: $100,000
Database Hardware: $752,000
Database Software: $566,000
Total = $1,456,000
Total $/JOPS = $1,456,000/1781JOPS = $817/JOP

An IBM spokesperson stressed the importance of the $/JOP figure, which "gives customers an idea of the cost per transaction of the system setup and represents their TCO directly. Basically we achieved 64% better throughput at nearly 1/2 the cost of the Sun/BEA/Oracle benchmark solution."

Stahl, for his part, said IBM's computation makes it clear "that (BEA's WebLogic) is vastly cheaper than (IBM's) WebSphere. (BEA) only ate up $100k in app server software licenses vs. IBM needing $480K in application server software licenses. That's 4.8x the cost for only 64% more throughput. (Furthermore, BEA) only consumed $38,000 in hardware on the app server tier to their $192,000. That's more than 5x the cost for only 64% more throughput."

Stahl also said, "when you isolate the app server software and hardware cost, which is the only part that is relevant for us, you get the following:

BEA WebLogic: $100k + $38k = $138k @ 1781 JOPS = $77.48/JOPS
IBM WebSphere: $480k + $192k= $672k @ 2921 JOPS = $230/JOPS"

Stahl also said that the difference in IBM's calculations stems from the database cost, "and that's an IBM DB2 vs. Oracle issue..WebLogic Server runs on IBM hardware, SuSE Linux and DB2 as well as WebSphere does. We have thousands of customers using that configuration."

BEA has its own competitive battles with Oracle in this market as well, and the company is on record as saying that it thinks Oracle's database is overpriced.

IBM has convinced enough customers of its advantages to have steadily been gaining the upper hand in this market, gaining several points of market share from BEA (and others), even achieving what's been called a "slam dunk" by analyst Gartner Corp. Yet BEA, if the company's response to this latest news from IBM is any indication, shows no signs of backing off in developing and marketing its competing environment in a fast-growing market.

Both companies have been on the move recently, IBM announcing a new broad-based SOA strategy, and BEA acquiring transactional software company Plumtree, Eclipse-centric M7, and Java EE 5 persistence company Solarmetric.

Benchmark wars can appear to be risible at times, as the complexity of the benchmark process easily leads to companies emphasizing only those parts that cast their products in the best possible light. The result can seem to provide no illumination at all, when two companies can use the same figures to demonstrate opposing viewpoints.

Yet, certainly SPECmarks always have, and do, provide a valuable service in that they keep technology providers' noses to the grindstone to keep creating faster and more efficient hardware, software, and solutions for their customers.

Thus, it seems healthy for the industry to have heavyweights such as IBM and BEA trade verbal punches on a regular basis. Both companies (along with Sun, Oracle, Microsoft, and JBoss) will be represented during SYS-CON's AppServer Shootout Webcast from New York on November 18, and maybe their dialog over the relative strengths of their platforms can continue.