Analyzing tricks competitors play – Unvalidated and unchallengeable performance claims

This is my second article that analyzes tricks I believe competitors play. In my first article titled “Analyzing tricks competitors play – making the claim that list price comparisons are misleading”1, I presented why comparing product list prices matters. In this article I want to tackle the issue of how competitors may use “performance benchmarks” to justify claims that more instances of Red Hat JBoss Middleware are required than a competitors software.

First, let me acknowledge that I do believe performance matters for most production systems. Unfortunately for customers, finding reliable performance benchmark results is a huge challenge for a variety of reasons, and if you are looking for a “one size fits all” benchmark for most use cases… ”fuhgeddaboudit”. Even worse, for technologies such as business process management, business rules, and integration, I am not aware of any widely accepted benchmarks.

Even when a public benchmark exists, it may not yield relevant results. SPECjEnterprise20102 in my opinion is one such benchmark. Still, it is a benchmark that I hear mentioned by Oracle and IBM who regularly try to on-up each others submitted results. For those who are not familiar with the benchmark, SPECjEnterprise2010 was published in 2005 with the intent to provide a performance measurement and characterization of Java EE 5.0 or later servers and supporting infrastructure. So an obvious question is, can this benchmark be considered a valid way to compare vendors? My answer to this question is no, and I have two major reasons why I believe this.

  1. Does the benchmark test criteria represent an application profile similar to what you will use?
    Probably not since new applications are not likely to be based on the Java EE 5.0 standard. In addition, relational databases popular in 2005 and required for use in the benchmark, are today slipping in popularity as primary data sources for modern applications.
  2. Is the hardware used to generate benchmark submission similar to what is common in production?
    Probably not since SPECjEnterprise2010 results are most often generated using the fastest hardware profiles possible. Such highly performant hardware profiles often are unjustifiably expensive to use in production environments.

So if SPECjEnterprise2010 is not a desirable performance benchmark, then what is an alternative? What about a benchmark published by Red Hat? Why doesn’t Red Hat publish performance benchmarks that compare competitive technologies that can be re-produced by third parties? That seems like a fair question to me. Let me share my opinion which should not be interpreted as an official Red Hat position.

The primary reason I don’t believe this is done is due to competitor legal terms and conditions that at best make it impractical for Red Hat to perform such benchmarks. For example, look at the following text from the IBM Process Server Advanced V8.5.5 license agreement:3

Benchmarking

Licensee may disclose the results of any benchmark test of the Program or its subcomponents to any third party provided that Licensee (A) publicly discloses the complete methodology used in the benchmark test (for example, hardware and software setup, installation procedure and configuration files), (B) performs Licensee’s benchmark testing running the Program in its Specified Operating Environment using the latest applicable updates, patches and fixes available for the Program from IBM or third parties that provide IBM products (“Third Parties”), and (C) follows any and all performance tuning and “best practices” guidance available in the Program’s documentation and on IBM’s support web sites for the Program. If Licensee publishes the results of any benchmark tests for the Program, then notwithstanding anything to the contrary in any agreement between Licensee and IBM or Third Parties, IBM and Third Parties will have the right to publish the results of benchmark tests with respect to Licensee’s products provided IBM or Third Parties complies with the requirements of (A), (B) and (C) above in its testing of Licensee’s products.

The above text makes clear that it is theoretically possible to perform a performance benchmark. However, IBM can throw up so many obstacles that it is unlikely Red Hat could ever get benchmark results published. For example using clause “B”, IBM could drag their feet auditing, verifying, and requesting changes be made to a benchmark to make sure “the latest applicable updates, patches and fixes available for the Program from IBM or third parties” are used. Even more effective though would be for IBM to question if “all performance tuning and ‘best practices’ guidance available in the Program’s documentation and on IBM’s support web sites” are being used.

What about Oracle? Are they as restrictive as IBM? Unfortunately, Oracle terms and conditions seem even more restrictive. Text in the Oracle Technology Network (OTN) developer license agreement4 associated with evaluation downloads states that you may not “disclose results of any Program benchmark tests without our prior consent.”

  • So how do you get Oracle’s prior consent?
    It is not specified.
  • Does Oracle have to provide consent?
    It is not specified.
  • Is the license agreement for purchased Oracle software any less restrictive?
    No. The Oracle License And Services Agreement5 contains text similar to the Oracle Technology Network agreement.

So, if benchmarking terms and conditions are so pervasively prohibitive, how can competitors run benchmarks against Red Hat? I consider the answer to be simple, Red Hat JBoss Middleware products are open source and do have not appear to have terms and conditions6,7 that prohibit benchmarks from being run and published. Plus, Red Hat competitors have relatively easy access to Red Hat JBoss Middleware. While this may seem unfair, it is consistent with the open nature of Red Hat and I have seen competitors take advantage of this.

For example, in January 2013, Prolifics–an IBM partner–published a competitive paper commissioned by IBM comparing Red Hat JBoss Enterprise Application Platform to WebSphere Application Server Network Deployment Edition 8.5 that included benchmark results.8 I reviewed that report and questioned several of the conclusions contained therein. In an attempt to validate multiple Prolifics conclusions, I contacted IBM and Prolifics repeatedly requesting access to the recorded steps (Camtasia screen captures) and associated time measurements, server and licensing assumptions, as well as formulas used in the study. Access to this material was offered in the Prolifics white paper9, but my requests for access via email and telephone message went unanswered by both companies. Without access to the content requested, I was unable to review the details of the performance results or perform a rigor of review similar to what IBM specifies in its WebSphere Application Server Network Deployment V8.5.5 license agreement. I don’t think asking to review what was presented as public information was an unreasonable request. Do you?

So what options are available to run and publish benchmarks when it is not practical, or legally allowed? One option I have observed is competitors engaging consulting companies to do commissioned customer studies. Specifically, commissioned studies that target customers who have in most cases used a competitors offering and the sponsoring companies product. Amazingly, every time I have seen Oracle or IBM do this, the competition always loses in the comparison. Imagine that!

Historically Oracle has used Crimson Consulting for studies against Red Hat, and IBM has used both Summa and Prolifics for its commissioned studies. Regardless of who performs the study, a recurring tactic leverages interviews where users are asked their opinion of metrics later used to compare products. Opinions. Not quantitative, and reproducible testing results that could potentially violate legals terms and conditions. Just subjective opinions. To make what I consider matters much worse, multiple studies exist where the customer opinion sample size is not statistically significant, but conclusions still get published. Conclusions that can be very specific with decimal point precision numbers and percentages provided. Conclusions that may read something like “we are 20% better than the competition for this”, “2.5 times better for that”, “10% better for something else”, and so on, and so forth. Then, opinion based measurements can get applied to metrics that translate to costs that may ultimately get used in ROI studies. All stemming from opinions that can lack any quantitative measure. Of course opinions are easy to collect, and as many sayings conclude, “everyone has one”.

I ask that when you review benchmarks and commissioned studies that you rigorously review the contents. Ask questions like does the benchmark reflect how you will use the middleware? Are the competitive benchmarks verifiable and reproducible? Did the person or group running the benchmark have much experience with the software being tested? It is not uncommon for the people who run competitive benchmarks to have lots of experience with a competitors middleware and little with Red Hat JBoss Middleware. For example, in the Prolifics competitive report commissioned by IBM10, there is no mention of any hands on Red Hat JBoss EAP experience by any of the authors.

So what is a customer to do who values performance data as part of a middleware software evaluation process? I believe the best performance numbers you can get are based on tests you define and run yourself based on expected use cases. Keep the scope of performance test to run concise with performance thresholds in-line with the service level agreements your production systems will need to meet. As an added bonus, as tests are set up and execute, you should be able to build first hand knowledge of the effectiveness of tools and utilities that will be regularly used.

What do you think? Your feedback is most welcome and encouraged!

1Naszcyniec, Richard. “JBoss Developer.” Analyzing tricks competitors play – making the claim that list price comparisons are misleading. planet.jboss.org, 22 Oct. 2014. Web. 27 Oct. 2014. <http://planet.jboss.org/post/analyzing_tricks_competitors_play_making_the_claim_that_list_price_comparisons_are_misleading>.

2“Standard Performance Evaluation Corporation.” SPECjEnterprise2010. spec.org, 5 Apr. 2012. Web. 27 Oct. 2014. <http://www.spec.org/jEnterprise2010/>.

3IBM license information form number L-JTHS-94HLS7 – “IBM License Information document.” IBM Software license agreements. N.p., n.d. Web. 22 Apr. 2014. <http://www-03.ibm.com/software/sla/sladb.nsf/lilookup/4D0F46D8A597A26485257B7E0063185A?opendocument&li_select=52C9A5247AA65C3885257B7E0063184D >.

4“OTN Developer License Terms.” OTN Developer License Terms. Oracle, n.d. Web. 27 Oct. 2014. <http://www.oracle.com/technetwork/licenses/standard-license-152015.html>.

6“Policies And Guidelines.” Red Hat Enterprise Agreement – US. Red Hat, n.d. Web. 27 Oct. 2014. <http://www.redhat.com/en/about/licenses-us>.

7“Terms and Conditions.” JBoss Developer. jboss.org, n.d. Web. 27 Oct. 2014. <http://www.jboss.org/terms-and-conditions/>.

8“IBM WebSphere® Application Server V8.5 vs. JBoss® Enterprise Application Platform V6 TCO Analysis.” WAS v8.5 vs. JBoss v6 TCO Analysis. Prolifics, 14 Dec. 2012. Web. 22 Apr. 2014. <http://prolifics.com/was-jboss-analysis.htm>.

9Ibid. 8, p. 7-8, 16.

10Ibid. 8

Leave a Reply

Your email address will not be published. Required fields are marked *