In today’s competitive environment, customers are turning to Open Source Software as “good enough” to meet their performance and financial goals. OSS initially became popular with Linux, Eclipse, HTTP servers, browsers, servlet engines, and later has moved into the domain of messaging and ESBs. Some companies get caught up in the myth of Open Source as a “free lunch”. To bring sanity into this discussion, we compared an IBM Integration Bus v10.0.0.1 (IIB) against WSO2 ESB v4.9.0 and Red Hat’s JBoss Fuse v6.2 in the following areas:
- User Experience – each ESB was examined in the areas of installation, development, management, documentation.
- Performance – maximum throughput for each ESB utilizing workflows with clients sending XML customer records over JMS.
Our study also included MuleSoft’s Mule ESB, but due to the license terms, we are unable to publish their results.
We ensured all ESBs were installed on equivalent hardware and configured and tuned in accordance with the best practices available to us as found in the documentation and forums of each respective ESB. All tests were run to maximize CPU utilization with results at or above 97% CPU utilization.
Please note, however, that differing situations may create differing results, and while we hope that your results match ours, we know that your mileage may vary. That being said, please feel free to contact us if you have any additional questions about our test configuration.
The traits that we thought stood out to us are highlighted in the table below, where you can see IIB’s thoughtful and intuitive, user-centric design over Open Source ESB counterparts.
|Installation||Excellent||Excellent||Excellent||All products are simple to install and get up and running.|
|Built in IBM MQ Connector||Excellent||Poor||Poor||WSO2 ESB’s connector must be created manually. Instructions are provided in documentation. Fuse must have a custom connector created to connect to MQ. No documentation is found laying out these instructions. Both also require custom builds to support MQ 8.0.|
|Intuitive Visual Builder||Excellent||Poor||Excellent||All have Eclipse-based visual builders, but WSO2 ESB’s is more complex and has a greater learning curve.|
|Extensive Pallet of Connectors||Excellent||Limited||Excellent||IIB and Fuse have ~100 connectors while WSO2 ESB only has ~55 connectors.|
|Runtime Deployment from Visual Builder||Excellent||Excellent||Poor||IIB and WSO2 ESB have drag and drop deployment from workspace to server. Fuse requires flows to be exported and imported into the runtime server.|
|Runtime Management from Visual Builder||Excellent||Excellent||Poor||IIB and WSO2 ESB allow integration server management directly from the developer studio. Fuse requires users to use a command-line based console and/or a web console.|
|Web Management Console||Excellent||Excellent||Limited||Fuse runtimes can be partially managed from its web-based console (high-level functions only, but not for low level functions).|
|Command Line Management||Excellent||Poor||Excellent||WSO2 runtimes cannot be managed from console.|
|Performance||Excellent||Poor||Excellent||WSO2’s performance is extremely inconsistent with large peaks and valleys and requires significant performance tuning.|
|Docs||Excellent||Poor||Poor||JBoss Fuse and WSO2 ESB have outdated or lacking documentation (for example when connecting to IBM MQ Fuse Documentation tells you to refer to IBM MQ documentation)|
|User Forums||Excellent||Limited||Limited||IIB forums have 90%, WSO2 ESB’s forums have 50%, and Fuse’s forums have 20% of questions answered.|
Each ESB was very straightforward to install, requiring approximately 15 minutes.
We began by developing simple primitive flows that pass data through endpoints, route the data based on the received message content, transform the messages, and then a combination of the above use cases to drive the ESB with a heavier workload. These flows will be explained in further detail below.
IBM Integration Bus
Right off the bat, IIB immediately stood out in crafting these flows. IIB came packed with a complete pallet of connectors and components that you can drag and drop onto a pallet for developing your integration solutions. The connectors we used in IIB plugged directly into IBM MQ with minimal and straightforward configuration with basic settings like queues, queue manager names, host names, ports, and channels. Within minutes, we had a basic pass through flow up and running. The development steps were straightforward: create a new application, build a flow by connecting components that required minimal configuration, then drag the flow onto the integration test server to build the Broker Archive file (BAR), deploy it and run it. IBM Integration Bus took the chore out of developing complex integration solutions: there were no libraries we had to hunt down, files to manually move, or ESB properties to set. The flow was up and running.
During the course of the performance study, we examined various transformation tools available to each ESB. IIB had a really nifty tool called the Mapping Node in the Transformation Section of its pallet. Setting up this connector was straightforward: double clicking the Mapping component starts a wizard that walks one through the mapping definition. The first screen named the mapping. The second screen asked you to select the XML Schema to use to define the input mapping. Then you can also select the XML Schema to define the output schema. On the final screen, you just take the default of XMLNSC for the Output Domain. We are then taken to a new canvas that shows the input Schema on the left hand side and the output schema on the right hand side. We dragged lines from the elements on the left to the elements on the right to map them. In our case, we just used the function on the line named “Move” to move the values from the input message to the generated output message. That was all there was to the mapping transformation.
WSO2 ESB was a maze compared to the other ESBs. Their complex structure and development environment coupled with their emphasis on web servers and proxy services made for a complex learning experience. When we first began, we had to wrap our head around the split of configuration between multiple locations. There were so many different things to set up, like the JNDI Directory and the axis2.xml file, which were separate from developing the flow itself. It was all very confusing and very different compared to the other ESBs. The axis2.xml file in itself is lengthy and difficult to understand.
Connecting to IBM MQ seemed pretty straightforward with the instructions provided by WSO2’s. When it came time to actually get a flow running, we discovered that there would be additional properties required to get it working, which was not stated or pointed out in documentation. Furthermore, we didn’t know what properties to use or even where to find them. We ended up finding the correct properties to connect IBM MQ as well as ActiveMQ from forums and blogs after weeks of sifting through several incorrect and nonworking answers. We found that the additional properties must be added either in the flow or the axis2.xml file. The properties provide the means of accessing various types of information or to control the behavior of the ESB on a given message. When we connected to ActiveMQ, we had to find the properties specific to those flows as well.
Another large issue with WSO2 ESB is that error messages are completely uninformative. There is nothing worse than having a seemingly unfixable problem with no error message. Our errors would produce a pop-up window saying “There is an error in your configuration.” It gave us no information or feedback on how to debug or fix the issue, so we were often stuck for lengthy periods of time.
Fuse was largely intuitive with a large pallet of connectors and components similar to IIB. A big difference however, is that Fuse does not come with an IBM MQ connector. It has instead been designed to fit more easily with ActiveMQ. Connecting Fuse to IBM MQ was extremely tedious and took a significant amount of time to figure out. What took us hours with IIB turned into weeks with Fuse to find a solution and to go through testing many would-be solutions. The solution we discovered involved custom beans, generic connectors, and a dependency nightmare. We created a custom bean with all the IBM MQ connection information needed as parameters. We then took a generic connector and renamed it so that it referenced the custom bean. Once the reference was created, parameters on the connector had to be adjusted properly to allow throughput. Finally, it was necessary to add all dependencies through maven and not through the class path.
Fuse has a similar mapping transformation tool as IIB, called Data Transformation; the tool is a simple drag and drop transformation interface that can go between different forms of data. However, at the time of this study, this tool is only available in beta and has several bugs. When testing we noticed several things that were not working as “expected.” When we checked against Fuse’s Jira database it was clear that several of the things we were doing had open, unresolved tickets. And we also helped define a few new tickets. This tool may be comparable to IIB’s Data Mapper, but it will have to be debugged and vetted before it can be used in enterprise environments. We did end up running tests with ActiveMQ later on. Connecting Fuse to ActiveMQ was intuitive with simple instructions much like IIB’s connection with IBM MQ.
After we finished developing each flow, we started deploying them onto our server. This presented its own set of challenges for each ESB. The ESBs also had various ways to manage runtimes in order to provide editing and administration abilities. Through these management portals, you could change things, such as deployed flows, dependency chains, and security settings. Like the ESBs themselves, not all management systems are created equal.
IBM Integration Bus
IIB offered a few ways to deploy flows. The IBM Integration Toolkit (Eclipse based Development Studio) was the most intuitive tool and allowed dragging the generated BAR file directly onto the desired integration server. In the toolkit, you could also right click the application and select deploy, which asked you to select the integration server for deployment. Alternatively, mqsi commands from the command line could be used for deployment.
IIB has a fully featured development toolkit complete with the capability to connect and manage processes that hosts message flows. We first had to create a new Integration Node (Broker), which was basically a set of execution processes that hosted one to many message flows. We did this by right clicking on the Integration Nodes item in the Integration Nodes View window of the toolkit. We could create either a local node or connect to a remote Node. Integration nodes enforce a degree of isolation between message flows in distinct integration servers by ensuring they run in separate address spaces, or as unique processes. This layer of isolation could provide load balancing or a division of responsibilities between lines of business flows. For example, we could have our financial flows run in a separate node from our day to day operating flows to mitigate any runtime error risk. For deployment of the flows into an Integration Node, we then had to define the Integration Server within that Node.
The Integration Server is the grouping of the message flows that are assigned to the Integration Node. As before, we right clicked on the integration node and then selected New to create an integration server on that node. We then had an integration server we could drag our Broker Archive (BAR) file onto for deploying our flows and all of their dependencies. We could manage these nodes and servers through the toolkit by being able to start and stop them, assign flows and the number of running instances of these flows to the servers. We could also open and manage the security policy sets of these servers. IIB also supplied a web interface into the Integration Nodes for reviewing all defined configuration parameters, starting and stopping the flows, as well as monitoring stats on the flows. Here, we could see the message flow rates through the various components comprising the flows. These stats are very useful for debugging flow performance issues.
WSO2 also had the ability to deploy flows by the Development Studio as well, much like IIB. You could also directly deploy on the Web Management Console. WSO2’s Developer Studio had a more complex structure of projects; it required several different types of projects to be manually created and configured together before a WSO2 ESB project could be deployed through a Carbon Application Archive File (CAR File). We noticed it took more time to complete deployment with WSO2 ESB, since much of it had to be done manually.
WSO2 ESB has full management capabilities through its Development Studio and Web Management Console. We found that the management system that was most documented in blogs and forums was the Web Management Console. The web-based console was accessed by deploying WSO2’s Java Virtual Machine (JVM) Server and could be used to edit and run flows. Unlike IIB, we were unable to manage multiple deployments of the JVM concurrently; each deployment instead had its own web portal with a unique port number. Additionally, we found that we couldn’t simultaneously use multiple web portals when we deployed multiple instances of WSO2’s JVM.
Fuse came up lacking when compared to the deployment options of the other ESBs. Fuse was not able to remotely deploy flows from its developer studio. It could however deploy to servers on the same PC. There was also a Hot Deploy System for Fuse, which allowed you to drop a deployment file into a folder. Being unable to deploy remotely, we defaulted to the hot deploy system. When we attempted to use the hot deploy system, we noticed that it was not working properly; the flow was not being added to the server’s runtime even though it was dropped into the hot deploy folder. After playing with this for a few days, we moved on to manual deployment for the rest of the project. Manual deployment in Fuse requires you to transfer the project to the server, build it through maven, and then install it into the Fabric environment using OSGI commands. This could be somewhat involved, especially when running multiple projects and flows on the same server. Compared to the other ESBs, deployment was undoubtedly more complex in Fuse. Fuse also had a conflicting jar with IBM MQ during deployment. In order to bypass this issue, the conflicting jar had to be refreshed through the management system. Through this refreshment the IBM MQ jar was seen as the primary jar and the conflicting jar was seen as secondary.
Fuse had high-level management functions made visible through a web portal, but the portal did not allow for editing or deploying of flows. Instead, Fuse had a command-line interface that was used to deploy and manage low-level functions of the ESB. While Fuse could run multiple instances concurrently, it was unable to manage them through the same portal or command-line. Instead each instance had to be managed separately.
Documentation and user forums
An important resource we had to use while testing the ESBs were their respective forums. For some, we were able to gain important insights about the product while others were unresponsive or uninformative. This is an issue, especially in open source, as the idea is that there is a community around each of these products that supports and develops them. In reality, the only way to get support is to pay a support fee.
IBM Integration Bus
Complementing the intuitive user interface was a vast knowledge base named the IBM Knowledge Center. Here, you could polish up on all the features of IIB, through such topics as Start Here, Product Overview, Samples, Installing, Configuring, Developing Integration Solutions, Performance, and Workload Management to cite a few. This thorough documentation clearly stepped you through the many features of IIB, including many step-by-step instructions for using most components.
In addition, the IIB support forums were abuzz with activity, fostering a strong community support presence with 90% of questions being answered.
WSO2 moved its forums to Stack Overflow around three years ago−where only 50-60% of the questions tagged with WSO2 or WSO2 ESB are even answered. The percentage decreases even more if you only look at the recent/active questions.
Fuse forums are relatively inactive with only around 20% of questions being answered. Not only are forums relatively inactive, but very few of our own questions have been answered except for by our own team mostly 1-2 weeks after posting. The questions that are answered are generally questions asking information that is already documented in forums in one place or another as opposed to more advanced questions.
In testing the ESBs, we set up identical systems for each product. These ESBs were tested using the Open Source IBM Hursley Lab Performance Harness tool.
The tests in this study were run on a 16-core Intel Sandy Bridge system; the system was broken up into two separate virtual machines—one VM, the Client, holding the performance harness software from IBM Hursley Team and the other VM, the Server, holding the ESB (IIB, Fuse, or WSO2 ESB) as well as the queuing software (IBM MQ v188.8.131.52 or Active MQ v5.12.1). The Client had four virtual CPUs with 8 GBs of RAM while the Server had four virtual CPUs with 32 GBs of RAM. Both the Client and the Server used version 6.7 of Red Hat Enterprise Linux OS. We had to use IBM MQ v7.5 for WSO2 ESB as it did not work with IBM MQ v8.
The flow of messages in each of these tests was the same across all test cases. First, the Client sends a message to a queue (IBM MQ or Active MQ). Then the ESB takes the message off of the input queue to process it (based on its use case) and pushes it onto the output queue. Lastly, the Client then retrieves the response off of the output queue. We focused on four major use cases: three primitives (Pass Through, Content Based Routing, and XSLT Transformation) and one as a combination of the three primitives:
- Pass Through: In this flow the ESB picks the message off of the input queue and then places the message on the output queue. No other internal processing is done in this flow. This test’s main purpose is to test the performance of the connector itself in isolation from other flow computations.
- Content-Based Routing: In this flow the ESB picks the message off of the input queue, routes the message based off of a value in the message body, and then (if the value is correct) places the message on the output queue.
- XSLT Transformation: In this flow the ESB picks the message off of the input queue, uses XSLT to transform the payload, and then places the message on the output queue.
- Combination: This flow is a combination of the Content-Based Routing and the XSLT Transformation flows. This flow first transforms the payload and then routes based on the newly transformed payload.
Below you can see that through testing IIB performs far better than the competition.
IIB performed well above the competition in JMS Pass Through, seeing as high as 17,500 messages per second while the next highest competitor was below 6,500 messages per second. That’s almost three times less than IIB.
In Content-Based Routing, IIB greatly outperformed the open source competitors. A huge takeaway that we noticed in content-based routing is that WSO2 ESB did not perform nearly as consistently as the other two ESBs. While IIB drops roughly 60% between 12k and 126k WSO2 ESB drops over 75%. At the closest comparison Fuse performed only 60% as well as IIB.
The ESBs were much closer comparatively in transformation. This flow was the most intensive primitive test, as a lot of the processing was done within XSLT. Still, IIB outperformed each competitor with at least a 13% increase in the worst case and upwards of a 100% increase in the best case.
The final test case was a combination of the above three. This was the most complex test case we used and provided the fullest utilization of the ESB. IIB won with at least a 45% increase over the closest competitor in the worst case (for IIB).
This was the most indicative of a customer scenario and should be used when determining the most efficient ESB.
Open Source ESBs Coupled with ActiveMQ
In addition to testing on IBM MQ, we also ran some baseline testing on Active MQ. We did this because as Open Source products, the thought was that they may be more naturally tuned to an open source message queue. As such, we didn’t do this test with IIB, since it comes paired with IBM MQ. As expected, Fuse had slight increases in performance as it is tightly integrated with ActiveMQ. However, WSO2 ESB performed extremely poor when paired with ActiveMQ, which was unexpected as ActiveMQ is more recommended and documented than IBM MQ.
IIB outperformed the open source competition in every case. In some tests IIB was up to 9.8 times faster than WSO2 ESB And up to 3.1 times faster than Fuse. IIB was also a distinct winner when it comes to user experience. IIB had a fully integrated environment from development to deployment to management, which provided one singular point of access. In addition, IIB was tightly coupled with IBM MQ, whereas WSO2 ESB and Fuse required extensive configuration to connect with MQ.
Finally, IIB had strong community support, while WSO2 ESB and Fuse forums go largely unanswered for extended periods of time.
Overall, IIB provided the most fully featured development studio, excellent documentation, ease of deployment, and integrated management. To top it off IIB had clear lead in performance when used with JMS driven flows and XML payloads.