Welcome!

Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

WebSphere vs WebLogic: IBM and BEA Spar Over SPEC Results

SPECjAppServer2004 Results Published, But What Do They Mean?

Benchmarks can mean whatever you want them to mean, it has always seemed. Although useful as a rough guide to performance, and sometimes price/performance, technology companies are famous for interpreting complex benchmark results as victories over their competition and them employing visual aids such as planes, snails, and automobiles to demonstrate their point.

The latest benchmark brouhaha may be over the newly developed SPECjAppServer2004 benchmark from SPEC (the Warrenton, VA-based Standard Performance Evaluation Corp.) This benchmark is the latest in the "Java Client/Server" category, and replaces SPECjAppServer 2002, which has now been retired. It "is designed to measure the performance of J2EE 1.3 application servers, (and) includes an enhanced workload by adding a web tier, JMS, and other changes to SPECjAppServer2002," according to SPEC.

And it has lent itself to spectacular success by IBM, according to a recent release coming out of Somers, NY. In fact, IBM's WebSphere Application Server "outperformed the competition by 64 percent," according to the release. Competitor BEA was quick to dispute this claim, however, pointing to its own success with its WebLogic on this benchmark.

IBM executive Sandy Carter said "these latest results further demonstrate the industry leading capability of IBM WebSphere Application Server," said Sandy Carter, Vice President, IBM WebSphere Software. And the IBM release went onto say that the "results submission involved more than 22,000 concurrent clients and produced more than 2,921 complex business transactions per second. The IBM submission represented a complete IBM solution bringing together the latest version of WebSphere Application Server software, DB2 Universal Database and IBM System p5 550 servers running SUSE Linux. This powerful solution outperformed a combination of BEA and Oracle software running on Sun Solaris and at significantly lower total cost of ownership across the full configuration."

BEA executive Eric Stahl , the company's Senior Director of Investor Relations, begs to differ. He referred to IBM's results as "inefficient," and has developed charts that show WebLogic as the clear winner in this contest. In his blog, Stahl says "the new (SPECjAppServer2004) result from BEA sets a new scalability world record, beating the previous record which was already held by BEA," and he provides a chart in his blog to illustrate his point.

Neither Oracle nor Sun had a comment about the IBM release by the time this story was being posted. Oracle has also not published its results on this benchmark, although Sun has. The results table is available to anyone at the SPEC website.

Which leads to the price/performance question. Unlike the 2002 benchmark, the new one does not include a price/performance metric, so ladies and gentlement, please re-start your engines. IBM says it based its 64% margin of victory on the following comparison:

IBM/WAS/DB2
AppServer Hardware: $192,000
AppServer Software: $480,000
Database Hardware: $377,000
Database Software: $181,000
Total = $1,230,000
Total $/JOPS = $1,230,000/2921 JOPS = $421/JOP

SUN/BEA/Oracle
AppServer Hardware: $38,000
AppServer Software: $100,000
Database Hardware: $752,000
Database Software: $566,000
Total = $1,456,000
Total $/JOPS = $1,456,000/1781JOPS = $817/JOP

An IBM spokesperson stressed the importance of the $/JOP figure, which "gives customers an idea of the cost per transaction of the system setup and represents their TCO directly. Basically we achieved 64% better throughput at nearly 1/2 the cost of the Sun/BEA/Oracle benchmark solution."

Stahl, for his part, said IBM's computation makes it clear "that (BEA's WebLogic) is vastly cheaper than (IBM's) WebSphere. (BEA) only ate up $100k in app server software licenses vs. IBM needing $480K in application server software licenses. That's 4.8x the cost for only 64% more throughput. (Furthermore, BEA) only consumed $38,000 in hardware on the app server tier to their $192,000. That's more than 5x the cost for only 64% more throughput."

Stahl also said, "when you isolate the app server software and hardware cost, which is the only part that is relevant for us, you get the following:

BEA WebLogic: $100k + $38k = $138k @ 1781 JOPS = $77.48/JOPS
IBM WebSphere: $480k + $192k= $672k @ 2921 JOPS = $230/JOPS"

Stahl also said that the difference in IBM's calculations stems from the database cost, "and that's an IBM DB2 vs. Oracle issue..WebLogic Server runs on IBM hardware, SuSE Linux and DB2 as well as WebSphere does. We have thousands of customers using that configuration."

BEA has its own competitive battles with Oracle in this market as well, and the company is on record as saying that it thinks Oracle's database is overpriced.

IBM has convinced enough customers of its advantages to have steadily been gaining the upper hand in this market, gaining several points of market share from BEA (and others), even achieving what's been called a "slam dunk" by analyst Gartner Corp. Yet BEA, if the company's response to this latest news from IBM is any indication, shows no signs of backing off in developing and marketing its competing environment in a fast-growing market.

Both companies have been on the move recently, IBM announcing a new broad-based SOA strategy, and BEA acquiring transactional software company Plumtree, Eclipse-centric M7, and Java EE 5 persistence company Solarmetric.

Benchmark wars can appear to be risible at times, as the complexity of the benchmark process easily leads to companies emphasizing only those parts that cast their products in the best possible light. The result can seem to provide no illumination at all, when two companies can use the same figures to demonstrate opposing viewpoints.

Yet, certainly SPECmarks always have, and do, provide a valuable service in that they keep technology providers' noses to the grindstone to keep creating faster and more efficient hardware, software, and solutions for their customers.

Thus, it seems healthy for the industry to have heavyweights such as IBM and BEA trade verbal punches on a regular basis. Both companies (along with Sun, Oracle, Microsoft, and JBoss) will be represented during SYS-CON's AppServer Shootout Webcast from New York on November 18, and maybe their dialog over the relative strengths of their platforms can continue.

More Stories By Roger Strukhoff

Roger Strukhoff (@IoT2040) is Executive Director of the Tau Institute for Global ICT Research, with offices in Illinois and Manila. He is Conference Chair of @CloudExpo & @ThingsExpo, and Editor of SYS-CON Media's CloudComputing BigData & IoT Journals. He holds a BA from Knox College & conducted MBA studies at CSU-East Bay.

Comments (2)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


IoT & Smart Cities Stories
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
Digital Transformation is much more than a buzzword. The radical shift to digital mechanisms for almost every process is evident across all industries and verticals. This is often especially true in financial services, where the legacy environment is many times unable to keep up with the rapidly shifting demands of the consumer. The constant pressure to provide complete, omnichannel delivery of customer-facing solutions to meet both regulatory and customer demands is putting enormous pressure on...
IoT is rapidly becoming mainstream as more and more investments are made into the platforms and technology. As this movement continues to expand and gain momentum it creates a massive wall of noise that can be difficult to sift through. Unfortunately, this inevitably makes IoT less approachable for people to get started with and can hamper efforts to integrate this key technology into your own portfolio. There are so many connected products already in place today with many hundreds more on the h...
The standardization of container runtimes and images has sparked the creation of an almost overwhelming number of new open source projects that build on and otherwise work with these specifications. Of course, there's Kubernetes, which orchestrates and manages collections of containers. It was one of the first and best-known examples of projects that make containers truly useful for production use. However, more recently, the container ecosystem has truly exploded. A service mesh like Istio addr...
Digital Transformation: Preparing Cloud & IoT Security for the Age of Artificial Intelligence. As automation and artificial intelligence (AI) power solution development and delivery, many businesses need to build backend cloud capabilities. Well-poised organizations, marketing smart devices with AI and BlockChain capabilities prepare to refine compliance and regulatory capabilities in 2018. Volumes of health, financial, technical and privacy data, along with tightening compliance requirements by...
Charles Araujo is an industry analyst, internationally recognized authority on the Digital Enterprise and author of The Quantum Age of IT: Why Everything You Know About IT is About to Change. As Principal Analyst with Intellyx, he writes, speaks and advises organizations on how to navigate through this time of disruption. He is also the founder of The Institute for Digital Transformation and a sought after keynote speaker. He has been a regular contributor to both InformationWeek and CIO Insight...
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
To Really Work for Enterprises, MultiCloud Adoption Requires Far Better and Inclusive Cloud Monitoring and Cost Management … But How? Overwhelmingly, even as enterprises have adopted cloud computing and are expanding to multi-cloud computing, IT leaders remain concerned about how to monitor, manage and control costs across hybrid and multi-cloud deployments. It’s clear that traditional IT monitoring and management approaches, designed after all for on-premises data centers, are falling short in ...
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...