Author: Jason Gartner, Vice President, WebSphere Foundation and PureApplication Development at IBM
Benchmarks are boring, benchmarks are irrelevant and benchmarks are old school. I’ve heard them all and at times, I’ve said it myself. For years large benchmarks have been about leapfrogging one another, they’ve been about spinning the results to match a perspective the vendor wanted to provide. So why talk about benchmarks now? Simply said, this one is different. It’s different for all the same reasons why l normally do not like benchmarks. For the first time that I am aware we have a benchmark on terms that I will dub Apples to Apples. The timeline is the same, the processors are the same, the workload is the same, but the results are clearly NOT the same. When you slice the results, it does not matter if you slice by core, by processor or by system based on the recent results submitted on Intel Haswell EP processor family systems, IBM clearly outperforms Oracle. So why is this significant? Firstly, 31%(1) out performance per core is not just a victory, but I classify that as a significant victory. Secondly and most importantly, it begins to question the investment of Oracle into Java. In an arena where organizations are transforming themselves with new digital channels such as Mobile and Web and new insights from BigData, Social sources, and Internet of Things, the demand on the on-premise systems is growing at exponential rates. The need for performance on our core systems is more critical now than ever. Since 2010, IBM has increased its performance of core Java workloads more than 5.4x(2) on Intel on the compared benchmarks while Oracle has improved only 1.9x(3) and has not taken a leadership position in Java performance since 2012(4). If that is not enough to question IBM vs Oracle’s investments, how about the fact that the world record for all Java per core is on IBM’s very own POWER8(5) systems. POWER8 continues to be the flagship of performance for Java. Add on top of that, the recent IBM z13 launch where IBM announced its new line of mainframe “providing up-to 2X improvement in throughput-per-core for security-enabled applications and up-to 50% improvement for other generic applications(8).” You can see the full blog here:
Cloud, Mobile, Analytics and Social are driving our systems of record more than ever. Performance is a cornerstone of any application platform and its clear how IBM is investing in its core infrastructure in addition to its systems of engagement.
Here’s more detail regarding the SPECJEnterprise benchmark that I’m sure you’ll find enlightening.
IBM published a World Record single application server result of 19,282.14 SPECjEnterprise2010 EjOPS using Lenovo x3650 M5 system for the application tier and Lenovo x3850 X6 system for the database.
The WebSphere Application Server 126.96.36.199 running on Lenovo x3650 M5 server equipped with dual Intel Haswell EP Processors demonstrated 31% better performance on per core basis when compared to latest result from Oracle using WebLogic 12.1.3 Server running on Oracle Server X5-2 also equipped with dual Intel Haswell EP Processors
- WebSphere 188.8.131.52 produced 19282.14 SPECjEnterprise2010 EjOPS using 28-core x3650 dual processor system equipped with Intel Haswell Processors whereas WebLogic 12.1.3 produced 18800.76 SPECjEnterprise2010 EjOPS using much higher 36-core Oracle Server X5-2 dual processor system
- WebSphere 184.108.40.206 is able to beat Weblogic 12.1.3 even with 22% less number of cores both running on latest dual processor systems equipped with Intel Haswell EP Processors. Oracle Weblogic used 28% more cores
- With this result WebSphere clearly demonstrates its performance superiority beating WebLogic on any metric on the latest Intel Haswell processors whether it be per Processor, per Core or even per Intel Haswell EP System
- WebSphere running on Lenovo x3650 M5 beats WebLogic on latest Oracle SPARC T5-2 server by 29% and WebLogic on latest Oracle Server X5-2 by 31% respectively on per core basis(6)
- WebSphere running on POWER8 servers beat WebLogic running on latest Oracle SPARC T5-2 by 76% and WebLogic running on latest Oracle Server X5-2 by 79% respectively on a per core basis(7)
What Does this mean to the Client?
- Performance is a key driver of lower TCO via lower # of servers, administrative processes, electrical power requirements, real estate, etc.
- Delivering more EjOPS per core on similar Intel processors WebSphere clearly demonstrates lower TCO compared to WebLogic servers
- IBM is committed to improving the performance of its application server on realistic customer benchmarks plucked from production environments which in turn drives performance on SPECjEnterprise2010 higher
Benchmark Performance Results
|SPECjEnterprise2010 Performance Results Chart
as of 2/20/2015
|Submitter||EjOPS*||Application Server||Database Server|
|Oracle||18,800.76||1x Oracle Server X5-2
2x Intel Xeon E5-2699 v3 (Turbo frequency 3.6 GHz) Haswell EP
Oracle WebLogic 12.1.13
|1x Oracle ServerX5-22x Intel Xeon E5-2699 v3 (Turbo frequency 3.6 GHz) Haswell EP
Oracle 12c DB Enterprise Edition
|IBM||19,282.14||1x Lenovo System x3650 M5
2x Intel Xeon E5-2697 v3 (Turbo frequency 3.6 GHz) Haswell EP WebSphere Application Server v220.127.116.11
|1x Lenovo System x3850 X6
4x Intel Xeon E7-4890 v2 (turbo frequency 3.4 GHz) Westmere EX
IBM DB2 10.5 FP5
|IBM||22,543.34||1x IBM Powers824POWER8 Processors
WebSphere Application Server V18.104.22.168
|1x IBM Power s824
IBM DB2 10.5 FP3
|Oracle||17,033.54||1x OracleSPARCT5-2SPARCT5 ProcessorsOracle WebLogic 12.1.2||1x OracleSPARCT5-2SPARCT5 ProcessorsOracle 12c Database Enterprise Edition|
* SPECjEnterprise2010 EjOPS, bigger is better.
(1) Based on per core comparison of Oracle’s latest publish of 18,800.76 SPECjEnterprise2010 EjOPS (522.24 EjOPS/core) dated 2/19/2015 and IBM’s latest publish of 19,282.14 SPECjEnterprise2010 EjOPS (688.64 EjOPS/core) dated 2/18/2015
(2) Based on per core comparison of 1st WebSphere Application Server publish on Intel processor of 1013.40 SPECjEnterprise2010 EjOPS (126.67 EjOPS/core) dated 1/2/20110 and latest WebSphere Application Server publish on Intel processor of 19,282.14 SPECjEnterprise2010 EjOPS (688.64 EjOPS/core) dated 2/18/2015
(3) Based on per core comparisons of 1st WebLogic Application Server publish on Intel processor of 17,301.86 SPECjEnterprise2010 EjOPS (270.34 EjOPS/core) dated 3/9/2011 and latest WebLogic publish on Intel processor of 18,800.76 SPECjEnterprise2010 EjOPS (522.24 EjOPS/core) dated 2/19/2015
(4) Based on per core industry leadership publish of WebSphere Application Server on Intel processor of 6295.46 SPECjEnterprise2010 EjOPS (524.62 EjOPS/core) dated 4/26/2012
(5) Based on the highest per core result of the industry with WebSphere Application Server running on POWER8 processors producing 22,543.34 SPECjEnterprise2010 EjOPS (939.30 EjOPS/core) dated 4/22/2014
(6) Based on 36-core Oracle Sun Server X5-2 result of 18800.76 SPECjEnterprise2010 EjOPS (522.24 EjOPS/core) published on 02/19/2015, 32-core Oracle SPARC T5-2 result of 17033.54 SPECjEnterprise2010 EjOPS (532.29 EjOPS/core) published on 01/22/2014, IBM result of 28-core x3650 19282.14 SPECjEnterprise2010 EjOPS (688.64 EjOPS/core) of 2/18/2015
(7) Based on 24-core IBM Power S824 result of 22543.34 SPECjEnterprise2010 EjOPS (939.30 EjOPS/core) published on 04/22/2014, 36-core Oracle Sun Server X5-2 result of 18800.76 SPECjEnterprise2010 EjOPS (522.24 EjOPS/core) published on 02/19/2015, 32-core Oracle SPARC T5-2 result of 17033.54 SPECjEnterprise2010 EjOPS (532.29 EjOPS/core) published on 01/22/2014
(8) Based on a comparison of IBM SDK Java 8 on IBM z13 compared to IBM SDK Java 7 on zEC12
Detailed Configuration Info
Full Configuration information and Benchmark run results can be obtained at SPEC.org website.
The SPECjEnterprise2010 benchmark is a full system benchmark which allows performance measurement and characterization of Java EE 5.0 servers and supporting infrastructure such as JVM, Database, CPU, disk and servers.
The workload consists of an end to end web based order processing domain, an RMI and Web Services driven manufacturing domain and a supply chain model utilizing document based Web Services. The application is a collection of Java classes, Java Servlets, Java Server Pages , Enterprise Java Beans, Java Persistence Entities (pojo’s) and Message Driven Beans.
SPECjEnterprise2010 is the third generation of the SPEC organization’s J2EE end-to-end industry standard benchmark application. The new SPECjEnterprise2010 benchmark has been re-designed and developed to cover the Java EE 5.0 specification’s significantly expanded and simplified programming model, highlighting the major features used by developers in the industry today. This provides a real world workload driving the Application Server’s implementation of the Java EE specification to its maximum potential and allowing maximum stressing of the underlying hardware and software systems.
SPEC has paid particular attention to making this benchmark as easy as possible to install and run. This has been achieved by utlising simplification features of the Java EE 5.0 platform such as annotations and sensible defaulting and by the use of the opensource Faban facility for developing and running the benchmark driver.
SPECjEnterprise2010’s new design spans Java EE 5 including the new EJB 3.0 and WSEE component architecture, Message Driven beans, and features level transactions.
Key Features of WebSphere Publish on Lenovo x3650 X5
- Improved performance on WAS 22.214.171.124
- Performance improvements on IBM JDK 7.1 SR2
- WAS Exploitation of RDMA through JDK support of Java Sockets over RDMA
Key Features of POWER8 and WebSphere Exploitation that resulted in Best Performance
- Exploit Significant Parallelism Offered by POWER8
- Exploit Transactional Memory
- Improve Per Core Performance
- Reduce Virtualization Overhead with PowerVM
- Exploit Faster Networking and Storage Capabilities
- Improve Security Workload Performance
- Exploit Larger Cache including L4 Cache
*Note: All data and results are as of 2/20/2015
SPEC and the benchmark name SPECjEnterprise are registered trademarks of the Standard Performance Evaluation Corporation. Results from http://www.spec.org as of 02/20/2015 36-core Oracle Server X5-2 (2-Processors) result of 18800.76 SPECjEnterprise2010 EjOPS published on 02/19/2015, 32-core Oracle SPARC T5-2 (2-Processor chips) result of 17033.54 SPECjEnterprise 2010 EjOPS published on 01/22/2014, IBM result of 28-core Lenovo x3650 M5 (2-processors) 19282.14 SPECjEnterprise 2010 EjOPS of 2/18/2015 and 24-core IBM Power S824 (4-Processor Chips) result of 22543.34 SPECjEnterprise 2010 EjOPS published on 04/22/2014.