Welcome!

Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

WebSphere vs WebLogic: IBM and BEA Spar Over SPEC Results

SPECjAppServer2004 Results Published, But What Do They Mean?

Benchmarks can mean whatever you want them to mean, it has always seemed. Although useful as a rough guide to performance, and sometimes price/performance, technology companies are famous for interpreting complex benchmark results as victories over their competition and them employing visual aids such as planes, snails, and automobiles to demonstrate their point.

The latest benchmark brouhaha may be over the newly developed SPECjAppServer2004 benchmark from SPEC (the Warrenton, VA-based Standard Performance Evaluation Corp.) This benchmark is the latest in the "Java Client/Server" category, and replaces SPECjAppServer 2002, which has now been retired. It "is designed to measure the performance of J2EE 1.3 application servers, (and) includes an enhanced workload by adding a web tier, JMS, and other changes to SPECjAppServer2002," according to SPEC.

And it has lent itself to spectacular success by IBM, according to a recent release coming out of Somers, NY. In fact, IBM's WebSphere Application Server "outperformed the competition by 64 percent," according to the release. Competitor BEA was quick to dispute this claim, however, pointing to its own success with its WebLogic on this benchmark.

IBM executive Sandy Carter said "these latest results further demonstrate the industry leading capability of IBM WebSphere Application Server," said Sandy Carter, Vice President, IBM WebSphere Software. And the IBM release went onto say that the "results submission involved more than 22,000 concurrent clients and produced more than 2,921 complex business transactions per second. The IBM submission represented a complete IBM solution bringing together the latest version of WebSphere Application Server software, DB2 Universal Database and IBM System p5 550 servers running SUSE Linux. This powerful solution outperformed a combination of BEA and Oracle software running on Sun Solaris and at significantly lower total cost of ownership across the full configuration."

BEA executive Eric Stahl , the company's Senior Director of Investor Relations, begs to differ. He referred to IBM's results as "inefficient," and has developed charts that show WebLogic as the clear winner in this contest. In his blog, Stahl says "the new (SPECjAppServer2004) result from BEA sets a new scalability world record, beating the previous record which was already held by BEA," and he provides a chart in his blog to illustrate his point.

Neither Oracle nor Sun had a comment about the IBM release by the time this story was being posted. Oracle has also not published its results on this benchmark, although Sun has. The results table is available to anyone at the SPEC website.

Which leads to the price/performance question. Unlike the 2002 benchmark, the new one does not include a price/performance metric, so ladies and gentlement, please re-start your engines. IBM says it based its 64% margin of victory on the following comparison:

IBM/WAS/DB2
AppServer Hardware: $192,000
AppServer Software: $480,000
Database Hardware: $377,000
Database Software: $181,000
Total = $1,230,000
Total $/JOPS = $1,230,000/2921 JOPS = $421/JOP

SUN/BEA/Oracle
AppServer Hardware: $38,000
AppServer Software: $100,000
Database Hardware: $752,000
Database Software: $566,000
Total = $1,456,000
Total $/JOPS = $1,456,000/1781JOPS = $817/JOP

An IBM spokesperson stressed the importance of the $/JOP figure, which "gives customers an idea of the cost per transaction of the system setup and represents their TCO directly. Basically we achieved 64% better throughput at nearly 1/2 the cost of the Sun/BEA/Oracle benchmark solution."

Stahl, for his part, said IBM's computation makes it clear "that (BEA's WebLogic) is vastly cheaper than (IBM's) WebSphere. (BEA) only ate up $100k in app server software licenses vs. IBM needing $480K in application server software licenses. That's 4.8x the cost for only 64% more throughput. (Furthermore, BEA) only consumed $38,000 in hardware on the app server tier to their $192,000. That's more than 5x the cost for only 64% more throughput."

Stahl also said, "when you isolate the app server software and hardware cost, which is the only part that is relevant for us, you get the following:

BEA WebLogic: $100k + $38k = $138k @ 1781 JOPS = $77.48/JOPS
IBM WebSphere: $480k + $192k= $672k @ 2921 JOPS = $230/JOPS"

Stahl also said that the difference in IBM's calculations stems from the database cost, "and that's an IBM DB2 vs. Oracle issue..WebLogic Server runs on IBM hardware, SuSE Linux and DB2 as well as WebSphere does. We have thousands of customers using that configuration."

BEA has its own competitive battles with Oracle in this market as well, and the company is on record as saying that it thinks Oracle's database is overpriced.

IBM has convinced enough customers of its advantages to have steadily been gaining the upper hand in this market, gaining several points of market share from BEA (and others), even achieving what's been called a "slam dunk" by analyst Gartner Corp. Yet BEA, if the company's response to this latest news from IBM is any indication, shows no signs of backing off in developing and marketing its competing environment in a fast-growing market.

Both companies have been on the move recently, IBM announcing a new broad-based SOA strategy, and BEA acquiring transactional software company Plumtree, Eclipse-centric M7, and Java EE 5 persistence company Solarmetric.

Benchmark wars can appear to be risible at times, as the complexity of the benchmark process easily leads to companies emphasizing only those parts that cast their products in the best possible light. The result can seem to provide no illumination at all, when two companies can use the same figures to demonstrate opposing viewpoints.

Yet, certainly SPECmarks always have, and do, provide a valuable service in that they keep technology providers' noses to the grindstone to keep creating faster and more efficient hardware, software, and solutions for their customers.

Thus, it seems healthy for the industry to have heavyweights such as IBM and BEA trade verbal punches on a regular basis. Both companies (along with Sun, Oracle, Microsoft, and JBoss) will be represented during SYS-CON's AppServer Shootout Webcast from New York on November 18, and maybe their dialog over the relative strengths of their platforms can continue.

More Stories By Roger Strukhoff

Roger Strukhoff (@IoT2040) is Executive Director of the Tau Institute for Global ICT Research, with offices in Illinois and Manila. He is Conference Chair of @CloudExpo & @ThingsExpo, and Editor of SYS-CON Media's CloudComputing BigData & IoT Journals. He holds a BA from Knox College & conducted MBA studies at CSU-East Bay.

Comments (2)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


IoT & Smart Cities Stories
The Japan External Trade Organization (JETRO) is a non-profit organization that provides business support services to companies expanding to Japan. With the support of JETRO's dedicated staff, clients can incorporate their business; receive visa, immigration, and HR support; find dedicated office space; identify local government subsidies; get tailored market studies; and more.
As you know, enterprise IT conversation over the past year have often centered upon the open-source Kubernetes container orchestration system. In fact, Kubernetes has emerged as the key technology -- and even primary platform -- of cloud migrations for a wide variety of organizations. Kubernetes is critical to forward-looking enterprises that continue to push their IT infrastructures toward maximum functionality, scalability, and flexibility. As they do so, IT professionals are also embr...
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
Atmosera delivers modern cloud services that maximize the advantages of cloud-based infrastructures. Offering private, hybrid, and public cloud solutions, Atmosera works closely with customers to engineer, deploy, and operate cloud architectures with advanced services that deliver strategic business outcomes. Atmosera's expertise simplifies the process of cloud transformation and our 20+ years of experience managing complex IT environments provides our customers with the confidence and trust tha...
AI and machine learning disruption for Enterprises started happening in the areas such as IT operations management (ITOPs) and Cloud management and SaaS apps. In 2019 CIOs will see disruptive solutions for Cloud & Devops, AI/ML driven IT Ops and Cloud Ops. Customers want AI-driven multi-cloud operations for monitoring, detection, prevention of disruptions. Disruptions cause revenue loss, unhappy users, impacts brand reputation etc.
As you know, enterprise IT conversation over the past year have often centered upon the open-source Kubernetes container orchestration system. In fact, Kubernetes has emerged as the key technology -- and even primary platform -- of cloud migrations for a wide variety of organizations. Kubernetes is critical to forward-looking enterprises that continue to push their IT infrastructures toward maximum functionality, scalability, and flexibility.
Today's workforce is trading their cubicles and corporate desktops in favor of an any-location, any-device work style. And as digital natives make up more and more of the modern workforce, the appetite for user-friendly, cloud-based services grows. The center of work is shifting to the user and to the cloud. But managing a proliferation of SaaS, web, and mobile apps running on any number of clouds and devices is unwieldy and increases security risks. Steve Wilson, Citrix Vice President of Cloud,...
When Enterprises started adopting Hadoop-based Big Data environments over the last ten years, they were mainly on-premise deployments. Organizations would spin up and manage large Hadoop clusters, where they would funnel exabytes or petabytes of unstructured data.However, over the last few years the economics of maintaining this enormous infrastructure compared with the elastic scalability of viable cloud options has changed this equation. The growth of cloud storage, cloud-managed big data e...
Artificial intelligence, machine learning, neural networks. We're in the midst of a wave of excitement around AI such as hasn't been seen for a few decades. But those previous periods of inflated expectations led to troughs of disappointment. This time is (mostly) different. Applications of AI such as predictive analytics are already decreasing costs and improving reliability of industrial machinery. Pattern recognition can equal or exceed the ability of human experts in some domains. It's devel...