SPECjEnterprise2010 benchmark questions answered


specSmarterQuestions reader has asked several Smart questions in response to this WAS vs. WLS performance comparison article. Below are reader questions and my answers.

QUESTION:Is the operating system included in this comparison? I know Oracle includes Solaris with its hardware. Presumably, the cost of SUSE, VMware and associated support is included for the X3650 for this comparison? Are application and database licensing licensing costs and support included? If not, how would this affect the comparison?

My cost comparison used in that article includes all of the costs for the SUT (System Under Test) over 3 year period, including:

– Operating System (SuSe for IBM and Solaris for Oracle)
– Database software (DB2 for IBM, Oracle DBMS EE)
– App Servers software (WebSphere for IBM, WebLogic for Oracle)
– Storage hardware
– DBMS hardware server
– App Server hardware server
– Networking
– Support for all of the above components for 3 years

What was not included in the cost?
– VMware cost is not included as it was not used in the test
– Cost of real estate, power, cooling, labor, etc. was not included as this is not part of the Bill of Materials for the benchmark.

In other words, every single item listed in the BOM (Bill of Materials) was priced at list price level and included in the comparison (IBM BOM, Oracle BOM).

QUESTION:Do the costs compare if we were to run Oracle WebLogic and Oracle DB on the X3650 and / or WebSphere and DB2 on the SPARC T4-4?

In order for me to answer this, I would need to have Oracle do SPECjEnterprise2010 test on IBM x3650 and IBM do the test on SPARC. This aint gonna happen any time soon. There are several reasons for this:

1) It costs a lot of money to buy these hardware systems. While IBM does have a number of test SPARC servers in WebSphere development labs, IBM is not going to invest half a million dollars into this exact configuration on latest Oracle hardware just to do this one test.
2) Even if IBM did have this much money just laying around and did do the test and published the result – Oracle would come out and print large articles all over billboards that IBM is now “abandoning its own hardware” and using Oracle hardware to showcase WebSphere performance. This has happened in the past. I am not making this up.

Having said that, I understand your question and agree that cross-platform comparisons are always tricky, unless you consider all the factors, including cost, reliability, cost of power, cooling, space, labor, etc. None of these additional factors are measured in SPECjEnterprise2010.

QUESTION:What happens in the event of sub-capacity or ‘pay-as-you-go’ software licensing. Is it possible to use VMware on the X3650 as a license boundary for virtualized workloads?

Yes, you can install VMware, Xen or Red Hat KVM or other hypervisors on IBM x3650. You can run many different Operating Systems under those hypervisors and partition server in many different ways. On guest OS instances you can run WebSphere or WebLogic. The difference is that with IBM license policy you only pay for cores used by the OS guest, while with Oracle you pay for all cores on the server as they do not recognize VMware, KVM and other software based partitioning using hypervisors, except for their own Oracle VM.

For example, if you had a server with 8 cores and you installed VMware ESX and ran guest OS on single core, you would only pay for WAS for that one core. If you were to run WebLogic on that OS guest, you still have to pay for 8 cores. Even if your guest OS only used 1 core. This is true for all of the Oracle and IBM software, including databases, portals, app servers, ESBs, etc. It is just the way Oracle licenses their software.

QUESTION:The X3650 only scales to 6% of the performance of the SPARC T4-4 cluster. How would a cluster of X3650 compare when delivering a similar number of EJOPs to the Oracle configuration? Oracle used RAC for the 2 x SPARC T4-4 database configuration. How would the DB2 cluster costs compare for the X3650 at a similar number of EjOPs to the SPARC T4-4?

Scaling App Server is very easy both vertically (adding cores to the server) and horizontally (adding machines). Scaling database vertically is relatively easy, however scaling database horizontally is hard. In order for me to answer your question in terms of dollars, I would need to have performance data on that configuration. It wont be fare to just multiply the cost of IBM x3650 system by the scale factor to match Oracle number of total transactions. The other questions is – how many applications really do need to run 40,000 transactions per second? Which is important, scalability or cost per transaction? I think there are cases when one or another or both are important. I hope IBM will publish high scale Power7 result in 2012 so that we can better compare costs on the high end.

QUESTION:We also looked at the Power 780 SPECjEnterprise2010. SPECjEnterprise2010 doesn’t require a pricing disclosure, yet if we use the basis of your comparison as the terms of reference, we found the $ per EjOP for the Power 780 to be almost 10x higher than the SPARC T4-4 and the Power 780 delivers almost 60% fewer EjOPs. Is this why IBM is now focusing on x86 for WebSphere?

I have not priced IBM Power result, but I really doubt that it will cost more than Oracle per transaction. I did mention above that cross-platform comparisons are always tricky – you must consider reliability and other factors too. For example, if you had one very large application, such as this SPECj app and needed very large volume of transactions, then x86 platform might be a good choice since you are likely to dedicate number of servers to that task and consume most of the capacity on those servers. However in most enterprises there are dozens or hundreds of applications and they all need compute capacity. Hence folks tend to virtualize their systems and consolidate servers. This is where Power7 comes in. It is great at running hundreds of virtualized workloads on a single system, while x86 virtualization is nowhere near that – VMware on x86 simply can not run several dozens of guest OS instances efficiently, while Power7 platform can run hundreds. What would be the average annual utilization for the x86 server? Probably within 10-20%, while Power7 systems often run at 60+% of their capacity. This means they are at least three times (or more) efficient compared to x86 servers when used for consolidating workloads.

For example, consider moving cargo by truck or by train. Yes, trains cost a lot more, but they are much more cost efficient and have lower environmental impact, higher reliability if you have a lot of different cargo to move. Both trucks and trains have their pros and cons. You aren’t going to invest into a train if you do not have enough cargo volume. Same with Power7 and x86. There are two white papers on IBM website that compare Power and VMware virtualization technologies. Just scroll to the bottom of the page. Here is the direct link to one of those reports.

One other thing I forgot to mention is that some of the IBM benchmarks done on Power platform were done on the hardware that was available at hand and not optimized for cost analysis. What this means is that there could be a storage array with lots of hard drives and only very small percentage of that disc array was actually used for the benchmark. However Bill of Materials includes complete configuration used for the test, even if some components were not fully used and strictly speaking were not even needed in the size that was included in the Bill of materials. This is just one example, but you can see why pricing some of the benchmark submissions might not give you true cost per transaction.



Categories: Technology

Tags: , , , , ,

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: