Developer point of view on the App Server debate


Update: There is a new version of this article available here that covers latest versions of these products as of 22 September 2015.


Old article from 13 November 2013 is shown below…


In the last few weeks I tested the latest versions of Tomcat, Jetty, JBoss, WebSphere, GlassFish and WebLogic and compared startup times, application deployment times, memory and disk footprint, and other developer focused features of these products. Back in May this year there was an interesting article from RebelLabs with follow-up discussion in the blogosphere discussing developer experience with different Java servers. The original article was quite useful and well written, however new versions of products became available since and I also wanted to add WebSphere and WebLogic into the mix and add some extra comparison metrics, such as memory usage, and few other things that are relevant for developers.

I went ahead and downloaded latest versions of all of the software as shown below:

App Servers:

  • IBM WebSphere Liberty Core 8.5.5
  • Apache-tomcat-7.0.42
  • JBoss.org AS 7.1.1
  • JBoss EAP v6.1.0
  • Eclipse Jetty Stable 9.0.6.v20130930
  • WAS for developers 8.5.5 with IBM Java 1.7.0, 64 bit
  • WebLogic 12.1.2.0 (developer zip package)
  • GlassFish OSS 4.0

Hardware and OS:

  • Lenovo ThinkPad W530, 2 CPUs @ 2.6Ghz (4 cores, 8 hyperthreaded cores), 16 GB RAM, Samsung SSD 840 PRO 256GB
  • Windows 7 64-bit, 6.8 Windows Experience Index (native install, no virtualization)
  • Oracle JDK 1.7.0 (build 1.7.0-b147) Java HotSpot 64-Bit Server VM (build 21.0-b17, mixed mode)
  • 64 bit Cygwin (latest as of this writing)

I have used all servers with default configuration, except for the hot-deployment scanning intervals decreased to a minimum (0.1 sec) where possible. Table below shows the results of my testing. Unfortunately I am not able to show you WebLogic numbers as Oracle commercial license agreement prohibits anyone from publishing test results without written permission from Oracle, and given the numbers I got, I know they wont allow me to publish their results (hint – WebLogic numbers were not terrible, but definitely not the best).

JavaEE_servers_compared

To do start, stop and deployment testing I wrote bash scripts that are designed to execute these tasks, iterate multiple times, measure timings, check server log files, check application deployment and version changes, etc. These scripts can be customized to create new tests or run your own tests. You can download these scripts as well as test application from this dropbox folder. Here is a brief description of files (you can run scripts on Windows Cygwin or on Linux):

  • setenv.sh – script used to initialize the environment before running any other command (you need to run it “. ./setenv.sh” before you use ./run.sh)
  • run.shmain script to be run with no arguments – will iterate all tests on all servers
  • applications (folder) – container folder with test applications
  • one.sh – script to run one command at a time (called from run.sh – you can also run it standalone if you wish)
  • utils.sh – common script snippets and variable declarations (called from run.sh and one.sh)
  • liberty.shwildfly.sh, glassfish.sh, jboss.sh, weblogic.sh, jetty.sh, tomcat.sh, websphere.sh – scripts with environment settings and commands to control each server (called from one.sh script)
  • test_results.txt – output with all measurements (will be produced by running run.sh script)

Server start

WAS Liberty Profile, Jetty and Tomcat were absolute winners and had very similar start times consistently in under 2 seconds. JBoss was just a bit slower, followed by WebLogic, GlassFish and WebSphere. I ran each test multiple times using bash scripts described above and used time markers in the script to capture the start and stop times, then generated results file which I processed in a spreadsheet to calculate averages and compare servers across the board. Server start test was done on empty server with no application deployed.

Server restart

Server restart test was done with the application deployed on the server. Here is a sequence of steps I used to test this scenario as can be seen in the “one.sh” file:

restartExistingServerWithApp()
{
    # --- prep work
    stop_server
    remove_server
    create_server
    start_server
    deploy_app
    wait_for_app_deploy
    long_sleep

    # --- actual timed test
    start_timer
    stop_server
    start_server
    wait_for_app_deploy
    stop_timer
}

Once again, the winners were Jetty and WAS Liberty Profile with very fast restart times. Please note that “wait_for_app_deploy” function actually tests that the web application is deployed and returns correct value by using the “wget” (see file “utils.sh“):

wait_for_app_deploy()
{
    echo "wait_for_app_deploy()..."
    FULL_TEST_URL=$SERVER_HOST:$PORT/$APP_URL
    echo "Will be looking for the '$APP_TEST_STRING' string in response from the app: '$FULL_TEST_URL'"
    STATUS=0
    COUNT=0
    while [ $STATUS = 0 ]
    do
        # now check for the URL to be served by the server
        wget $FULL_TEST_URL --timeout $APP_DEPLOY_TIMEOUT -O - 2>/dev/null | grep "$APP_TEST_STRING" >/dev/null
        if [ $? = 0 ]; then
            STATUS=1
        else
            if [[ $COUNT -gt $APP_DEPLOY_TIMEOUT ]]; then
                echo "!!!!!!!! Error - we have waited long enough - the '$UNDEPLOY_WAR' app will never deploy"
                exit 1
            fi
            # since correct string is not found, need to wait a little more
            sleep 1
            (( COUNT++ ))
            echo "Waiting $COUNT seconds for application '$UNDEPLOY_WAR' to deploy..."
        fi
    done
    echo "Application '$APP_NAME' is now deployed OK"    
}

Unfortunately WAS Full Profile took 28 seconds to stop and start from command line, but half of that time was spent on starting the wsadmin script twice (once for stop and once for start) and the real server re-start time is twice as fast if you use Eclipse (hence I used two numbers in the table).

Dynamic configuration

In development environment most of the changes are usually made to the application, not to the server configuration. Hot deployment and redeployment of applications is supported well by all servers I tested. However, in this test I wanted to see which servers provide the ability to keep JVM up and running while making changes to the server configuration itself, not the application code. Perhaps this is more important for production environments to avoid server downtime and loss of service, but may come handy for developers. I played with different server properties, such as changing HTTP port numbers, trace and logging settings, etc.

Liberty Profile was the most agile server and picked up all of the changes on the fly, including adding or removing server components, for example “JDBC feature”. I did not have to restart the server, except to change JVM options. WebSphere Full Profile and WebLogic were second and handled many configuration changes without restart. Tomcat, Jetty, GlassFish and JBoss required server restart more often (such as changing certain datasource settings, JMS configuration, etc.)

App deploy

For this round of tests I built a very simple HelloWorld application with only single Servlet. I know, I know… Need to get something more interesting than that. However a couple of apps that I tested had certain issues with some servers and will take me little extra time to resolve those issues. Once I test additional apps, I will post results and update this blog.

I did test deployment and start times for Jenkins for Tomcat and Liberty and got very similar results for both servers. Once I figure out the issues of Jenkins deployment on Jetty and other servers I will run detailed tests and publish results on this blog. My next step – find a decent web app that will run on all of the above servers and re-run test scripts to measure differences between products. I am open for suggestions. I tried running Jenkins on the latest versions of Jetty, WAS and WLS and had trouble making it work (quick google search did not help with the issue and I will need to spend some additional time on this one).

Deployment of the application was done by copying the file into the hot deploy directory for each server. Fastest deployment times were shown by JBoss and Liberty Profile. WebSphere Full Profile was the third fastest, but only when deploying the application from Eclipse. For WebSphere Full Profile I used two different methods of application deployment. When I used hot deploy via monitored directory, it had longer deployment time since the minimum scanning period in WAS Full Profile is 5 seconds (can be set to longer, but not shorter time). I also tested app deployment from Eclipse and it was very fast, hence I provided both numbers in the table above.

RAM usage

Jetty, Tomcat Liberty Profile and WAS Full Profile were winners in the RAM usage. However this is default server with no tuning and is developer oriented. If you were to deploy real application in production environment and had to support multiple concurrent users you would likely see completely different ranking in this category.

One thing to keep in mind is that Jetty and Tomcat started with JSP/Servlet container while WAS Full Profile, and all other servers are started with all services enabled for full Java EE conformance (albeit some servers do use lazy loading of components to minimize memory use). Liberty Profile and WAS Full Profile are examples of such lazy loading model. No tuning was done on servers, all servers tested with out of the box configuration.

Disk footprint

Jetty, Tomcat and Liberty Profile were winners in this category, but then again – Jetty and Tomcat are very limited in function and can’t be directly compared to other servers that provide loads of other features, including more complete programming model, admin UI, messages, extensive samples, etc.

JDK footprint is not included in any of the numbers. You should add about 250 MB for JDK 1.7. I had two JDKs installed in WAS Full Profile – JDK 1.6 and 1.7 (you don’t have to have both if you don’t want to).

One of the reasons why WAS Full Profile footprint is so big is that it comes with code for legacy spec support for backward compatibility. You can run J2EE 1.4 and Java EE 5 applications and they are fully supported. WAS Full Profile also installs system message catalogs in 16 different languages and with National Language Support for things like DOJO, CEA and many others – all localized in some cases supporting up to 28 different languages. Obviously all of that takes space on disk. I did not bother to measure the size of all of those localized files, but I’d estimate that to be close to 1 GB, considering localized property files, help files, system messages, wizards, Admin GUI, etc. I wish there was a way to minimize the install footprint in WAS Full Profile by stripping out all localization and just leaving English related files. I bet that would easily get the size of the server down to about 400 MB, not counting JVM. This is still a lot and if there was a way to also strip out all of the backwards compatible stuff, you will probably be left with about 200MB server.

What I described above is pretty much what IBM has done with the Liberty Profile. Liberty supports only latest specs, does not have localized runtime, except for 1.5 MB worth of license files in 36 different languages – I guess IBM legal department got the better side of this battle :-). It is beyond me why not just provide a 20 byte URL to the site that has all that license garbage and not waste 1.5 MB on all computers that have Liberty installed. That would be one small step towards a Greener and Smarter Planet, wouldn’t it? I know, I know… Apache and other OSS licenses require vendors to ship license files. But not in 36 different languages, right?

One of the cute features of Liberty is the ability to produce a minimal server configuration via “minify” command. This command scans the application deployed on the server and produce the smallest package that can run such application. When I ran this command for my HelloWorld Servlet application I got 21 MB zip file that I can share with other people so they can unzip it and run my application (although it still includes all of those license files in 36 different languages) – click on the hyperlink to download the result of this command below:

server.bat  package  defaultserver  --include=minify  --archive=c:\liberty.zip

Disk space for 1 instance

Once you complete initial server install (as discussed above), all servers allow user to create multiple instances (some call it domains, or profiles, or instances). The benefit is that most of the binaries are shared between multiple instances and only specific configuration data is unique to each instance. The winners for the smallest size for additional instance are Jetty, Tomcat, WebSphere Liberty Profile, followed by JBoss. To make things simpler, Liberty has a very simple command to create and remove additional instances and it takes less than a second to complete:

"$LIBERTY_HOME/bin/server create"

Tomcat, Jetty require manual creation of additional server instances. WebSphere Full Profile and WebLogic have very easy to use GUI and command line based profile (domain) management tools.

Developer install

Production install typically involves a whole lot more than simply unzipping the server. My install measurements in this test were very basic and did not include any security or clustering or HTTP server related tasks. In other words I did not measure production type install and only tested developer experience.

Developer style “desktop install” took under 20 seconds for all, but WebSphere Full Profile and WebLogic. Those 20 seconds do not include environment settings (things like JAVA_HOME, PATH, etc.). I did not use silent install capabilities of WebSphere and WebLogic as these could dramatically cut down on install time.

# of config files

Clear winner was WAS Liberty Profile as it only has 1 configuration file, yet with ability to split it into multiple files if so desired. Jetty, Tomcat and JBoss and GlassFish have anywhere from handful to a couple of dozen config files. However measuring number of configuration files for WebLogic and WAS Full Profile is tricky as most of those files do not need to be touched by developer and therefore the number of config files is a bit misleading.

One important consideration is that WebSphere Full Profile and WebLogic provide very powerful scripting (wsadmin and wlst respectively) and Admin GUI facilities well above and beyond what other servers in my test can offer. This reduces the need to deal with configuration files to a minimum, provides many task oriented wizards, script capture facility and other advanced features. However most of those advanced features would be of interest to operation support folks and not as important on a development desktop.

IDE

Eclipse plugins for all products have their roots in Eclipse Web Tools Project. WTP supports Tomcat out of the box. JBoss Developer Studio does provide additional productivity features over and above Eclipse WTP and many people find it very adequate for their development needs.

WebSphere Liberty and Full Profile both have free Eclipse plugins that add number of useful features on top of WTP. These plugins can be installed into latest Eclipse simply by searching in Eclipse marketplace (found under Eclipse Help menu item).

In addition to the free Eclipse plugins mentioned above, IBM offers very robust Rational Application Developer (RAD) with many advanced features, such as powerful code generation wizards, database tooling, Java code coverage and code analysis, profiling, batch, OSGi, Web 2.0, JCA, SCA, Portlet, cloud and many other enhancements well beyond the very basic features of WTP. The disadvantage of RAD is that all those advanced features bring significant increase in disk and memory footprint and slower reaction times for IDE. Having said this, there are two kinds of people in the world – those who absolutely love RAD and those who hate it :-).

There are several 3rd party development tools available for all of the servers listed above, including various editions of MyEclipse, IntellyJ IDEA, NetBeans, JDeveloper, etc. I must mention that RAD does support development for non-IBM runtimes, so you are not restricted to WebSphere only.

Admin GUI

The best Admin GUIs are provided by WebSphere Application Server Full Profile and WebLogic Server. These two products offer robust features, nice looking UI, context sensitive help, powerful monitoring, configuration editors, wizards, script capture facility for replay and lots of other cool things.WAS goes a step further and offers ability to setup SLAs for certain URIs, health monitoring and much  more. However all this heavy lifting machinery is not of interest to developers.

JBoss and GlassFish provide basic Admin GUI, but it is primarily single server oriented and quite restricted in function. Developers would still need to edit certain property and XML configuration files by hand from time to time.

Jetty has no GUI at all, Tomcat GUI can’t really be called a GUI as it is simple web page with some basic server info. Almost all changes must be made by hand in XML and property files (true for Jetty and Tomcat).

Liberty Profile offers very nice user interface for editing server configuration. This UI is part of Liberty Eclipse plugin and is great for development. However for production Liberty does not yet provide an Admin GUI (alpha version is available for download from http://wasdev.net). For now one could use JConsole provided by Java SE. This method can also be used for other servers listed above, but this method does not qualify for the “Admin GUI”, although it can be helpful for certain monitoring and administration tasks.

Ant, Maven, Jenkins

I have not done detailed analysis of support for ant, maven and Jenkins, but all products seem to provide this support and will likely fit into whatever development procedures you have in your shop.

APIs provided

Obviously programming model is hugely important to developers and this is the area of biggest difference between products. Jetty and Tomcat only provide very basic Servlet/JSP programming model and you need to look for 3rd party packages to add any other APIs that you need. This may not be a trivial task and can lead to server bloat, much longer startup times (1 minute or higher, instead of 2 seconds – depending on application size and complexity). At the same time Liberty comes with Java EE Web Profile 6.0 support (and some extras, including certain Java EE 7 APIs) and all other servers on the list support full Java EE 6, with GlassFish being first to support not only Java EE 6, but also Java EE 7.

Read detailed analysis of the programming model in this post: “Programming model support in lightweight Java servers“.

Development license

All of the servers used in my tests come with the license that permits their use in development environment free of charge. In case you did not know, IBM and Oracle allow for the use of their products in development environment without you having to pay license cost. You do need to purchase the license when you move outside of the development sandbox, though. There is one exception to this licensed production rule, and that is Liberty Core for ISV being free in certain production cases as described in this post “WebSphere Liberty Core for ISVs – free as in beer”.

In case of IBM, you can simply download full version of Liberty or developer version of the WebSphere Full Profile from the IBM website and install and use with no time limit. The product is the same as production version, the only difference for WebSphere Full Profile is that post installation task automatically configures it with smaller JVM heap size, hot deployment being enabled, security disabled, etc.

Development support

Once you download Tomcat, Jetty, GlassFish or JBoss you can use google and user forums for self support. However if you wanted to be able to call someone and have critical issues resolved you can’t get that, unless you pay for Red Hat support or go to 3rd party vendors for such support.

In case of WebSphere Liberty and Full Profile you get full commercial support for your development environment as long as you have at least one supported production runtime license. I think IBM wins hands down on this one.

Red Hat does provides free support for development, but limits number of support contacts to 1 for 16 cores subscription and up to 10 contacts for 160 cores. Support for WildFly (JBoss.org) version is not available, except perhaps via paid contract from 3rd party companies, but not via Red Hat (isn’t it funny that the biggest competitor to JBoss EAP is JBoss.org? I am still waiting for a day when Red Hat will stop losing money with JBoss EAP, but for now I am not buying their stock.)

Overall developer experience

And the winner is?… Well, depending on the application, you would likely need to make a careful choice – from a DEVELOPER perspective here are some considerations (please do not forget to also evaluate OPERATIONAL capabilities, not discussed in this article):

WebSphere Application Server Liberty Profile (from developer point of view):

  • PROS
    • free development license
    • free developer support from IBM if you have production license
    • disk and memory footprint is very small (only slightly more than Tomcat)
    • ultra fast to start, stop, deploy, etc. (on par with Jetty and Tomcat)
    • supports Java EE Web Profile 6.0 plus JAX-WS, JMS, etc. (much richer programming model compared to Jetty and Tomcat)
    • developer install only takes a few seconds
    • free Eclipse plugin
    • support for maven, Jenkins, ant
    • the most dynamic server – does not require restart ever (just kidding – you do need to restart it if you apply fixes of change JVM options)
    • applications built on Liberty can be deployed to WAS Full Profile with no changes. If your app can’t deploy directly from Liberty to WAS Full– open a bug report with IBM
    • free production license for ISV applications
  • CONS

WebSphere Application Server Full Profile (from developer point of view):

Tomcat and Jetty (from developer point of view):

  • PROS
    • free development license
    • free Eclipse plugin
    • developer install takes few seconds
    • disk footprint is very small
    • very fast to start, stop, deploy (except when you add loads of 3rd party libraries and kill it with repetitive jar file annotation scanning)
    • support for maven, Jenkins, ant
    • active support forums with books, tutorials and articles being widely available
    • source code is open
    • can be used in production for free if so desired
  • CONS
    • not a very dynamic server – most changes require server restart
    • no real Admin GUI
    • no free developer support available from any vendor (however can purchase 3rd party support)
    • programming model is limited to JSP/Servlet only. For anything else one must hand-pick from a variety of OSS projects

JBoss (from developer point of view):

  • PROS
    • free development license
    • free Eclipse plugin
    • developer install takes few seconds
    • disk footprint is relatively small (bigger than Liberty, Jetty and Tomcat)
    • very fast to start, stop, deploy
    • support for maven, Jenkins, ant
    • active support forums with books, tutorials and articles being widely available
    • source code is open
    • WildFly product can be used in production for free if so desired
    • JBoss EAP can also be used in production for free (thanks to LGPL license), but one has to compile it from sources
    • free developer support available provided that user has production licenses on paid contract
    • programming model is full Java EE 6
  • CONS

GlassFish (from developer point of view):

  • PROS
    • free development license
    • free Eclipse plugin (as well as NetBeans, etc.)
    • developer install takes few seconds
    • disk footprint is relatively small (bigger than Liberty, Jetty and Tomcat)
    • very fast to start, stop, deploy
    • support for maven, Jenkins, ant
    • source code is open
    • can be used in production for free if so desired
    • programming model is full Java EE 6 and is the only server today to support Java EE 7
  • CONS
    • not a very dynamic server – many changes require server restart
    • Admin GUI is fine for development, but wont be enough for serious production
    • last week Oracle announced that future versions of GlassFish will not have commercial support, but existing version of GlassFish is still supported (although not really…)

WebLogic Server (from developer point of view):

  • PROS
    • free development license
    • supports full Java EE 6
    • free Eclipse plugin (and JDeveloper)
    • support for maven, Jenkins, ant
    • relatively dynamic server – most of the time does not require restart
    • very complete and advanced Admin GUI only second to WAS Full Profile
    • active support forums, number of books, tutorials and articles
    • solid production features and performance, only second to WAS Full Profile (developers don’t care about these anyway, so I wont list it here)
  • CONS
    • source code is not open
    • production license costs money
    • disk footprint is much larger compared to other servers (the only product that has larger disk needs is WAS Full Profile)
    • not as fast to start, stop, deploy (but still reasonably fast)
    • developer install takes about 10 minutes

I would love to hear your feedback, comments and recommendations. Post your comments below or email me at whywebsphere@gmail.com.

PS. If Larry Ellison posts here and says he is ok with me publishing WebLogic numbers, I will do it :-).



Categories: Technology

Tags: , , , , , , , , , , , ,

9 replies

  1. Been using Glassfish for years in production, so here are my 2 cents on it.

    Why do you think the admin console is lacking, single-dev & non-production oriented? I can’t see anything that you can’t do easily from admin console (even if I’m a command-line / config file guy). I’ve administered cluster with it. It’s easy enough to do, you can even setup a new remote server instance from admin console by just providing ssh access.

    You also get a few extras that I’m not aware of in any other app server, e.g. rest administration & monitoring. I’ve been doing great things with those.

    Your number seem a bit high to me, but I didn’t check the scripts.

    Anyway, could you comment on this?

    Like

  2. My benchmark for “robust” IDE in this case is Rational Application Developer and I do not think JBoss Developer Studio is as capable as RAD. I guess I need to update this so that there is a difference between “Basic” features of WTP and JBoss Studio and RAD. I have not done detailed analysis of features of these plugins and tools and my score is based on my own an informal review and opinions of other folks that I respect. I would love to see someone do this kind of in-depth analysis and compare those tools at a deep level.

    Like

  3. JBoss servers actually have a very good IDE support via JBoss Tools plugin.

    Like

Trackbacks

  1. WebSphere vs. JBoss license cost calculator « Why WebSphere? Blog
  2. WAS vs JBoss license and support cost calculator (updated) | WhyWebSphere Blog
  3. Programming model support in Java servers (update) | WhyWebSphere Blog
  4. Lightweight Java servers and developer view on the App Server (update) | WhyWebSphere Blog
  5. MPower Softcomm – Dev Tools, Cycle, and an Insight into Their Craftsmanship – Part 2! | MPower Softcomm

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: