Lightweight Java servers and developer view on the App Server (update)


It’s been two years since I wrote the original article comparing Liberty Profile, WAS classic, Tomcat, Jetty, WildFly, JBoss, GlassFish and WebLogic from a developer point of view. The point of this research was to measure the typical developer experience, not a “benchmarketing” with lots of tuning and special tricks to make certain application servers run faster simply for the sake of faster numbers – hence I tested servers with little or no changes to their default configuration. In this post I will show the results of my latest tests comparing the startup times, application deployment times, memory and disk footprint, and other developer focused characteristics of the latest versions of these products:

  1. IBM WebSphere Liberty Core 8.5.5.7
  2. IBM WAS classic for developers 8.5.5.6
  3. Apache Tomcat 8.0.26
  4. Apache TomEE 1.7.2
  5. WildFly 9.0.1
  6. Red Hat JBoss EAP v6.4.0
  7. Eclipse Jetty 9.3.2
  8. Oracle WebLogic 12.1.3 (developer zip package)
  9. Oracle GlassFish OSS 4.1

The test environment consisted of the following:

  • Lenovo ThinkPad W530, 2 CPUs @ 2.6Ghz (4 cores, 8 hyperthreaded cores), 16 GB RAM, Samsung SSD 840 PRO 256GB (same machine I had 2 years ago – perhaps time for an upgrade if my manager is reading this? 🙂 )
  • Windows 7 64-bit native install, 6.8 Windows Experience Index
  • 64 bit Cygwin – all tests were ran in Cygwin using my own set of bash scripts posted on GitHub here
  • Oracle JDK 8u60 64-Bit Server VM (used for Tomcat, TomEE, Jetty, JBoss EAP, WildFly, WebLogic, GlassFish)
  • IBM J9 JVM build R28_Java8_SR1_20150513_0846_B248471 64-Bit (used for Liberty Profile – by the way – for Liberty this showed slightly better results than Oracle JVM)
  • IBM JDK 1.7 64-Bit (used for WAS classic)

I tested all servers with default configuration, except for the hot-deployment scanning intervals decreased to a minimum for faster deployments (see directory “server_configurations“). The video below shows how to run the tests and explains the values in the table below in more details:

Table below shows the results of my testing. Unfortunately I am not able to show you WebLogic numbers as Oracle license agreement prohibits anyone from publishing test results without written permission from Oracle, and given the numbers I got, I know they wont allow me to publish their results.

weblogic jboss wildfly websphere liberty tomee tomcat glassfish comparisonNotes:

(1) – JDK size is not included in the measurement of the disk size (Oracle JDK 8u60 is 186 MB download size and 350 MB when installed)

(2) – Dynamic configuration updates means that configuration changes do not require server restart

(3) – WLS Developer version zip install

(4) – TomEE+ size includes web admin webapp war file

(5) – server start, stop and RAM are measured for a simple HelloWorldServlet application using Oracle JDK 8u60 64bit on Windows 7, Samsung SSD disk, 4 cores Intel. For Liberty profile the JDK used is IBM JDK 8. For WAS JDK is IBM JDK 1.7. No changes to the server startup script were made – all default JVM heap and other settings were used. Servlet was called 50 times before the memory size was measured. JVM memory was determined as shown by Windows Task Manager “Memory Private Working Set”.

(6) – Developer install time does not include JDK install (which takes less than a minute)

(7) – Free developer support is available to customers who have production license(s) of these IBM products

(8) – Red Hat provides support for 25 developers for each 16 cores of production subscription of JBoss

To automate the testing I wrote bash scripts that are designed to execute start, stop, deployment tasks, iterate multiple times, measure timings, check server log files, check application deployment and version changes, etc. These scripts can be customized to create your own tests. You can download these scripts as well as test application from this GitHub repository. Here is a brief description of files (you can run scripts on Windows Cygwin or on Linux):

  • setenv.sh – script used to initialize the environment before running any other command (you need to run it “. ./setenv.sh” before you use ./run.sh)
  • run.shmain script to be run with no arguments – will iterate all tests on all servers (see video above for details)
  • applications (folder) – container folder with test applications
  • one.sh – script to run one command at a time (called from run.sh – you can also run it standalone if you wish)
  • utils.sh – common script snippets and variable declarations (called from run.sh and one.sh)
  • liberty.sh, wildfly.sh, glassfish.sh, jboss.sh, weblogic.sh, jetty.sh, tomcat.sh, websphere.sh – scripts with environment settings and commands to control each server (automatically called from one.sh script)
  • test_results.txt – output with all measurements (produced by running run.sh script)

Each test was run multiple times using bash scripts described above with time markers in the script to capture the start and stop times of the test. The generated results file was processed in a spreadsheet to calculate averages and compare servers across the board.

Server restart

Server restart test was done with the HellowWorld Servlet based application deployed on the server. Here is a sequence of steps I used to test this scenario as can be seen in the “one.sh” file:

restartExistingServerWithApp()
{
    # --- prep work
    stop_server
    remove_server
    create_server
    start_server
    deploy_app
    wait_for_app_deploy
    long_sleep

    # --- actual timed test
    start_timer
    stop_server
    start_server
    wait_for_app_deploy
    stop_timer
}

In this test the winners were Jetty, followed by WAS Liberty Profile and Tomcat (see table above for details). Please note that “wait_for_app_deploy” function actually tests that the web application is deployed and returns correct value to an HTTP Get by using the “wget” (see file “utils.sh“):

wait_for_app_deploy()
{
    echo "wait_for_app_deploy()..."
    FULL_TEST_URL=$SERVER_HOST:$PORT/$APP_URL
    echo "Will be looking for the '$APP_TEST_STRING' string in response from the app: '$FULL_TEST_URL'"
    STATUS=0
    COUNT=0
    while [ $STATUS = 0 ]
    do
        # now check for the URL to be served by the server
        wget $FULL_TEST_URL --timeout $APP_DEPLOY_TIMEOUT -O - 2>/dev/null | grep "$APP_TEST_STRING" >/dev/null
        if [ $? = 0 ]; then
            STATUS=1
        else
            if [[ $COUNT -gt $APP_DEPLOY_TIMEOUT ]]; then
                echo "!!!!!!!! Error - we have waited long enough - the '$UNDEPLOY_WAR' app will never deploy"
                exit 1
            fi
            # since correct string is not found, need to wait a little more
            sleep 1
            (( COUNT++ ))
            echo "Waiting $COUNT seconds for application '$UNDEPLOY_WAR' to deploy..."
        fi
    done
    echo "Application '$APP_NAME' is now deployed OK"    
}

Unfortunately WAS Full Profile took 34 seconds to stop and start from command line, but half of that time was spent on starting the wsadmin script twice (once for stop and once for start) and the WAS re-start time is twice as fast if you use Eclipse (17 seconds is not too bad for stop + start of WAS classic, eh?).

Dynamic configuration

In development environment most of the changes are usually made to the application, not to the server configuration. Hot deployment and redeployment of applications is supported well by all servers I tested. However, in this test I wanted to see which servers provide the ability to keep JVM up and running while making changes to the server configuration itself, not the application code. Perhaps this is more important for production environments to avoid server downtime and loss of service, but may come handy for developers. I played with different server properties, such as changing HTTP port numbers, trace and logging settings, etc.

Liberty Profile was the most agile server and picked up all of the changes on the fly, including adding or removing server components, for example “JDBC feature”. I did not have to restart the server, except to change startup JVM options (heap size, etc.). WebSphere Full Profile and WebLogic were second best and handled many configuration changes without restart. Tomcat, Jetty, GlassFish and JBoss required server restart much more often (such as changing certain datasource settings, JMS configuration, port numbers, security settings, etc.)

App redeploy

For this test I built a very simple HelloWorld application with a single Servlet. I know, I know… – need to get something more interesting than that (your suggestions are welcome). I did test Jenkins app in all servers, but I do not think Jenkins is a typical web app that your average company builds and runs on an app server. It has way too much going on there in terms of its own configuration and initial setup during the first deployment. WebSphere Performance Lab has done tests with DayTrader and TradeLight and published results of their tests in this document.

Re-deployment of the HelloWorld test application was done by copying the compressed WAR file into the hot deploy directory for each server on top of an already existing file. Fastest re-deployment times were shown by Liberty Profile and JBoss EAP and WildFLy. WebSphere classic was the third fastest, but only when deploying the application from Eclipse (the number in the table is hot deploy using directory, which is slower because directory scanning is done every 5 seconds and can not be set to a lower number). Overall for this simple application application redeployment time was very fast for all tested servers.

RAM usage

Liberty Profile with IBM JDK was the winner with lowest RAM usage of all the servers tested. Jetty and Tomcat were second and third as tested on Oracle JDK (most typical setup), followed by WAS Full Profile. However this is default server with no tuning and is developer oriented – if you were to deploy real application in production environment and had to support multiple concurrent users you would see increased memory usage for all servers.

One thing to keep in mind is that Jetty and Tomcat started with JSP/Servlet container while WAS classic and some other servers are started with all services enabled for full Java EE conformance (albeit some servers do use lazy loading of components to minimize memory use).

Disk footprint

Jetty, Tomcat and Liberty Profile were winners in this category, but then again – Jetty and Tomcat are very limited in function and can’t be directly compared to other servers that provide loads of other features, including more complete programming model, admin UI, messages, extensive samples, etc.

One of the reasons why WAS classic footprint is so big is that it comes with code for legacy spec support for backward compatibility. You can run Java EE 5 and 6 applications and they are fully supported. WAS Full Profile also installs system message catalogs in 16 different languages and with National Language Support for things like DOJO, CEA and many others – all localized in some cases supporting up to 28 different languages. Obviously all of that takes space on disk. I did not bother to measure the size of all of those localized files, but I’d estimate that to be close to 1 GB, considering localized property files, help files, system messages, wizards, Admin GUI, etc. I wish there was a way to minimize the install footprint in WAS Full Profile by stripping out all localization and just leaving English related files. I bet that would easily get the size of the server down to about 400 MB. This is still a lot and if there was a way to strip out all of the backwards compatible stuff, you will probably be left with about 200MB server.

What I described above is pretty much what IBM has done with the Liberty Profile (except Liberty is under 100 MB in size). One of the useful features of Liberty is the ability to produce a minimal server configuration via “minify” command. This command scans given server configuration and produces the smallest package that applications deployed on that server. When I ran this command for my HelloWorld Servlet application I got 25 MB zip file that I can share with other people so they can unzip it and run my application (although it includes 1.5 MB worth of license files in 36 different languages):

server.bat  package  defaultserver  --include=minify  --archive=c:\liberty.zip

Disk space for 1 instance

Once you complete initial server install (as discussed above), all servers allow the user to create multiple instances (some call it domains, or profiles, or instances). The benefit is that the server binaries are shared between multiple instances and only specific configuration data is unique to each instance. The winners for the smallest size for additional instance are Jetty, Tomcat, WebSphere Liberty Profile, followed by JBoss EAP and WildFLy. To make things simpler, Liberty has a very simple command to create and remove additional instances and it takes less than a second to complete:

"$LIBERTY_HOME/bin/server create"

Tomcat and Jetty require manual creation of additional server instances. WebSphere classic and WebLogic have very easy to use GUI and command line based profile (domain) management tools, but it takes some time to run those commands.

Developer install

Production install typically involves a whole lot more than simply unzipping the server. My install measurements in this test were very basic and did not include any security or clustering or HTTP server related tasks. In other words I did not measure production type install and only tested developer experience.

Developer style “desktop install” took under 5 seconds for all, but WebSphere Full Profile and WebLogic. Those 5 seconds do not include environment settings (things like JAVA_HOME, PATH, etc.). I did not use silent install capabilities of WebSphere and WebLogic as these could dramatically cut down on install time.

# of config files

Clear winner was WAS Liberty Profile as it only has 1 configuration file, yet with ability to split it into multiple files if so desired. Jetty, Tomcat, TomEE, WildFly and JBoss and GlassFish have anywhere from handful to a couple of dozen config files. However measuring number of configuration files for WebLogic and WAS classic is tricky as most of those files do not need to be touched by developer and therefore the number of config files is a bit misleading.

One important consideration is that WebSphere classic and WebLogic provide very powerful scripting (wsadmin and wlst respectively) and Admin GUI facilities well above and beyond what other servers in my test could offer. This reduces the need to deal with configuration files to a minimum, provides many task oriented wizards, script capture facility and other advanced features. However most of those advanced features would be of interest to operation support folks and not as important on a developer desktop.

IDE

Eclipse plugins for all products have their roots in Eclipse Web Tools Project. WTP supports Tomcat out of the box. JBoss Developer Studio does provide additional productivity features over and above Eclipse WTP and many people find it adequate for their development needs. WebSphere Liberty and WAS classic both have free Eclipse plugins that add number of useful features on top of WTP. These plugins can be installed into latest Eclipse simply by searching in Eclipse marketplace (found under Eclipse Help menu item).

There are several 3rd party development tools available for all of the tested servers, including various editions of MyEclipse, IntellyJ IDEA, NetBeans, JDeveloper, etc.

Configuration Editor

From a developer perspective, I think the best configuration editor is provided as part of the Liberty Eclipse Plugin. This tool allows the developer to edit Liberty server.xml configuration directly in Eclipse without having to have the server running. The developer has a choice between a native XML editor with code assist (not a big deal as you can do that with Tomcat or JBoss config files), or most important – specialized “smart” offline configuration editor with validation and drop down lists and checkboxes and such. The configuration is validated and parts of configuration are automatically added (when needed) – depending on the existing components in the file. Similarly to other servers, Liberty does have runtime based browser Admin Center, but it is not as important for the developer as the Eclipse based configuration tool.

The best runtime browser based Admin GUIs are provided by WebSphere Application Server Full Profile and WebLogic Server. These two products offer robust features, nice looking UI, context sensitive help, powerful monitoring, configuration editors, wizards, script capture facility for replay and lots of other cool things.WAS goes a step further and offers ability to setup SLAs for certain URIs, health monitoring and much more. However all this heavy lifting machinery is not of interest to developers.

GlassFish has a nice looking browser Admin GUI, but no Eclipse based configuration editor similar to Liberty.

WildFly and JBoss do provide Admin GUI, but it is single server oriented and is restricted in function. Developers often need to edit certain property and XML configuration files by hand.

Jetty has no GUI at all, Tomcat GUI can’t really be called a GUI as it is simple web page with some basic server info. Almost all changes must be made by hand in XML and property files (true for Jetty and Tomcat).

Ant, Maven, Jenkins

All products support various devOps tools and will likely fit into whatever development procedures you have in your shop with the caveat of the startup time and the installation footprint possibly being the limiting factor as shown in the summary table above.

APIs provided

Programming model (API support) is hugely important to developers and this is the area of biggest difference between products. Jetty and Tomcat only provide very basic Servlet/JSP programming model and you need to look for 3rd party packages to add any other APIs that you need. This may not be a trivial task and can lead to server bloat, much longer startup times (1 minute or higher, instead of 2 seconds – depending on application size and complexity).

Read detailed analysis of the programming model in this post: “Programming model support in lightweight Java servers“.

Development license

All of the servers used in my tests come with the license that permits their use in development environment free of charge. Moreover, Jetty, Tomcat, TomEE, WildFly and GlassFish allow unlimited use not only in development, but also in production (no vendor support provided). This could make it attractive to uses these products in certain cases. However one must consider the implications of running self supported product vs. paying 3rd party vendor for support and getting more function out of the box.

In case you did not know, IBM and Oracle allow for the use of their products in development environment without you having to pay license cost. You do need to purchase the license when you move outside of the development sandbox, though. There are two exceptions to this licensed production rule: (1) Liberty Core for ISV is free in certain production cases as described in this post “WebSphere Liberty Core for ISVs – free as in beer” and (2) Liberty can be used for free for production with up to 2 GB of RAM.

Development support

Once you download Tomcat, Jetty, GlassFish or WildFly you can use google and user forums for self support. However if you wanted to be able to call someone and have critical issues resolved you can’t get that, unless you pay for Red Hat support or go to 3rd party vendors for such support.

In case of WebSphere Liberty and Full Profile as well as JBoss EAP you get full vendor support for your development environment as long as you have at least one supported production runtime license.

Overall developer experience

And the winner is?… Well, just look at the table above and map the capabilities and ratings for servers to your own requirements. From a DEVELOPER perspective here are some considerations (please do not forget to also evaluate OPERATIONAL capabilities, not discussed in this article):

WebSphere Application Server Liberty Profile (from developer point of view):

  • PROS
    • free development license for the server runtime and Eclipse plugin
    • free developer support from IBM (if you have production license)
    • disk footprint is very small (only slightly more than Tomcat)
    • memory footprint is the smallest of any server tested
    • ultra fast to start, stop, deploy – faster than any other server
    • supports full Java EE 7 as well as Java EE Web Profile 6.0 (much richer programming model compared to Jetty and Tomcat) – the only production server to support Java EE 7
    • developer install only takes a few seconds
    • the most advanced server configuration editor inside Eclipse
    • very clear and complete documentation (best in class)
    • support for maven, Jenkins, ant, chef, etc.
    • the most dynamic server – does not require restart ever (just kidding – you do need to restart it if you apply server fixes of change JVM options)
    • only one configuration file and even that is very simple – far easier to configure than any other server I tested
    • applications built on Liberty can be deployed to WAS classic with no changes. If your app can’t deploy directly from Liberty to WAS classic – open a bug report with IBM
    • strong backwards compatibility for applications – innovative zero migration architecture (new features do not override nor break old features and are clearly separated in server.xml)
    • production license of Liberty (being part of WAS Base) is cheaper than JBoss EAP and WebLogic
  • CONS
    • 3rd party support (applications, tutorials and articles) are not yet as widely available as for other products
    • source code is not open
    • production license costs money (however keep in mind that license cost is only 10% of the TCO)

WebSphere Application Server Full Profile (from developer point of view):

  • PROS
  • CONS
    • source code is not open
    • does not yet support full Java EE 7
    • production license costs money
    • disk footprint is rather large
    • server stop and start times are rather long
    • developer install takes a long time
    • too many configuration files that are rather complex to deal with (strong admin GUI and wsadmin scripting make it easier)

Apache Tomcat, TomEE and Eclipse Jetty (from developer point of view):

  • PROS
    • free server runtime license and Eclipse plugin (development and production, but no vendor support)
    • developer install takes few seconds
    • disk footprint is very small
    • fast to start, stop and deploy basic applications
    • support for maven, Jenkins, ant
    • active support forums with books, tutorials and articles being widely available
    • source code is open
    • small number of configuration files to deal with and they are relatively simple in nature
  • CONS
    • not very dynamic servers – most changes require server restart
    • no vendor support (actually there are supported versions of Tomcat, but they cost money)
    • no real Admin GUI or configuration editor – must be edited by hand
    • documentation is not always clear or current
    • no free developer support available from any vendor (however can purchase 3rd party support)
    • Tomcat and Jetty programming model is limited to JSP/Servlet only. For anything else one must hand-pick from a variety of OSS projects
    • Once you add loads of 3rd party libraries (CXF, Hibernate, Spring, etc.) – you kill startup and deployment time with repetitive jar file annotation scanning. You also become your own application server vendor
    • TomEE partially solves the problem above by providing Java EE 6 Web profile support, but is not full Java EE and is based on an older version of Tomcat

JBoss EAP and WildFly (from developer point of view):

  • PROS
    • free server runtime development license and Eclipse plugin
    • developer install takes few seconds
    • disk footprint is relatively small (bigger than Liberty, Jetty and Tomcat)
    • fast to start, stop, deploy
    • support for maven, Jenkins, ant
    • active support forums with books, tutorials and articles being widely available
    • source code is open
    • WildFly product can be used in production for free if so desired (self supported)
    • JBoss EAP can also be used in production for free (thanks to LGPL license), but one has to compile it from sources and self support (I am surprised there is still no “CentOS” version of JBoss EAP)
    • free developer support available provided that user has production licenses on paid contract
    • programming model in JBoss EAP is Java EE 6. WildFly supports Java EE 7
    • Admin GUI is quite adequate for development, however Eclipse configuration editor is not available
  • CONS
    • not a very dynamic server – many changes require server restart
    • JBoss EAP does not support Java EE 7
    • Poor backwards compatibility – most of the admin scripts, APIs and server configuration have changed considerably in every major upgrade breaking majority of the user created tools for installing, configuring and managing JBoss
    • cost of JBoss EAP in production is usually higher compared to WAS Liberty and WAS Full Profile

GlassFish (from developer point of view):

  • PROS
    • free development license and Eclipse plugin (as well as NetBeans, etc.)
    • developer install takes few seconds
    • disk footprint is relatively small (significantly bigger than Liberty, Jetty and Tomcat)
    • fast to start, stop, deploy
    • support for maven, Jenkins, ant
    • source code is open
    • can be used in production for free if so desired
    • feature rich browser based Admin GUI
    • programming model is full Java EE 7
  • CONS
    • not a very dynamic server – many changes require server restart
    • no offline server configuration editor in Eclipse
    • too many configuration files
    • RAM use is quite high
    • commercial support is no longer available per Oracle announcement

WebLogic Server (from developer point of view):

  • PROS
    • free development license and free Eclipse and JDeveloper tools
    • supports full Java EE 6
    • support for maven, Jenkins, ant
    • most of the time does not require restart when server configuration changes are made
    • advanced browser based Admin GUI (similar to WAS classic)
    • active support forums, number of books, tutorials and articles
  • CONS
    • no support for Java EE 7
    • source code is not open
    • production license is very expensive
    • disk footprint and RAM use is much larger compared to other servers (the only product that has larger disk needs is WAS Full Profile, but WAS uses 3 times less RAM)
    • very slow to start and stop the server, and twice as slow to deploy applications compared to other servers
    • developer install takes much longer than most other servers (except for WAS classic)

I would love to hear your feedback, comments and recommendations. Post your comments below or email me at whywebsphere@gmail.com.

PS. If Larry Ellison posts here and says he is ok with me publishing WebLogic numbers, I will do it :-).



Categories: Technology

Tags: , , , , , , , , , , , , , ,

4 replies

Trackbacks

  1. Developer point of view on the App Server debate | WhyWebSphere Blog
  2. Building RESTful services with WebSphere Liberty, Tomcat and WildFly – IBM Advantage Blog
  3. The state of WebSphere Application Server business – IBM Advantage Blog

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: