tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Armishaw <mike.armis...@btinternet.com>
Subject Problems upgrading from version 4.0.5 to 4.1.24
Date Wed, 03 Nov 2004 12:39:24 GMT
Hi,

this is the first time I have posted to this or any other discussion 
group. Apologies if I am not following correct protocol, hopefully 
someone will correct me if that is the case.

I have checked the archive but have been unable to find anything that 
answers my questions. Again, apologies if I've missed stuff.

First some background. We have a web app that runs on a government 
secure intranet. The app is essentially a forms application that allows 
users to submit a variety of expenses and excess hours claims to a 
legacy mainframe payroll system. The app uses only JSPs, servlets, beans 
and straight JDBC calls to a Mysql database backend. The hardware is 
made up of two quad 1.8Mhz windows 2000 servers each with 3Gb of memory. 
The app runs in parallel on each server (i.e. sticky sessions, not 
distributed) using a fairly crude but effective load balancing that 
splits users 50/50 between the servers. Each server also hosts a Mysql 
database, the live database on one and an archive database on the other. 
We realise that this is not an ideal configuration, however 
circumstances leave us no choice. This is a very heavily used 
application, it has over 120,000 registered users and regularly handles 
over 2,000 concurrent users.

The app currently runs very successfully on Tomcat 4.0.5 with the 
following configuration.

    minProcessors = 250
    maxProcessors = 750
    enableLookups = false
    acceptCount = 200
    connectionTimeout = -1
    application context reloadable = false

The Java heap size is set to a minimum of 512M and a max of 1024M.

We have been considering upgrading Tomcat to a later version, and for 
the latest release installed version 4.1.24 on our development and test 
machines. Unit and functional testing went well but we were unable to 
carry out any load testing due to the lack of suitable hardware to set 
up a test environment that mirrored live. We therefore decided to deploy 
to live using 4.0.5 again. However, on carrying out final acceptance 
testing in the live environment we experienced several unexplained 
crashes. Things that had worked fine in test were suddenly failing in 
live. My initial assumption (at 11pm at night with heavy pressure to get 
the app deployed ready for 4am next morning) was that the problem must 
be in Tomcat 4.0.5, as everything had worked fine in test. I therefore 
decided to upgrade live to 4.1.24 using the same configuration as we had 
for 4.0.5 shown above. Following the upgrade acceptance testing went 
fine with everything working well just as it had in test.

However, the major problems showed up next morning as soon as the load 
began to build up. As soon as concurrent usage went over 100 we received 
the error "threadpool full, no more threads available - please increase 
maxProcessors". The app ws very unstable with some users able to log on 
but most receiving a "page unavailable" message. All sorts of new / 
unexplained errors showed up in the logs, most of them due to attributes 
not being found in the httpsession where they should have been. Memory 
usage on the servers was much higher than it had been and performance 
degraded badly. The situation got so bad that we had to bring the 
application down.

We then decided to re-examine the problems we'd had the previous night. 
We discovered that the issues with 4.0.5 had actually been code errors 
that were found by the 4.0.5 jsp compiler/engine but were ignored by 
4.1.24. For example,
    1. A semi colon placed after a page import in a JSP, compilation 
failure in 4.0, no error in 4.1
    2. Forwarding to a URL for a JSP page that did not exist, 
servletException in 4.0, no error in 4.1, which appeared to treat the 
non-existent page as null, which as it happens was the correct behaviour 
for our app but might not have been.
    3. Doing an Integer.parseInt on a String value taken from the http 
session. When the String is null get a NumberFormatException in 4.0 but 
no error in 4.1, which appeared to treat null as zero, which again was 
the corect behavior for our app but again might not have been.

After we fixed these problems we rolled back the live servers to 4.0.5 
and within ten minutes the application was happily handling over 1200 
concurent users with response times back to well below half a second.

Sorry if this has turned into something of an essay, but I have tried to 
include all information that might be relevant.

I would be very grateful for answers to the following questions.
    1. Why was 4.0.5 able to detect these errors while 4.1.24 ignored 
them, are they changes to the JSP / Servlet specs or are they bugs?
    2. Why was performance and stability so much worse with 4.1.24, have 
we missed some additional configuration changes that are required or is 
4.1 just worse than 4.0.
    3. Are these problems general to all versions of 4.1 or is 4.1.24 
just a bad release?

The Apache Tomcat pages advise users to upgrade to version 5, following 
our experiences trying to go to 4.1 we don't feel confident at all about 
trying to move to version 5.

Thanks very much for your time,

Mike.


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org


Mime
View raw message