Mailing-List: contact tomcat-user-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: tomcat-user@jakarta.apache.org
Message-Id: <4.3.1.2.20010813141839.00d38a30@mail.baltimore.ie>
Date: Mon, 13 Aug 2001 14:18:59 +0100
To: tomcat-user@jakarta.apache.org
From: Eamonn Maher <emaher@baltimore.ie>
Subject: Re: Re: Tomcat crashes simultaneous more connection above 990+
In-Reply-To: <20010813223133.A16201@socialchange.net.au>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed

Hi folks,
coincidentally I am also having problems with large number of concurrent 
connections to Tomcat

getting errors when we try and connect many clients concurrently to Tomcat. 
Does anyone have experience of this sort of Tomcat usage or should we be 
looking at a different servlet container for heavy loads.
The setup is as follows
Tomcat 3.2.2. on WinNT SP 6 running as a standalone servlet container.
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-C)
Java HotSpot(TM) Client VM (build 1.3.0-C, mixed mode)

We have a test environment where a number (2-4) of other machines, with 
multi-threaded clients (30-1000 threads) bombard the servlet in an effort 
to test the servlet performance. This leads to a number of problems

1) First of all the mutli-threaded test clients (also written in Java) 
occasionally throw a NoClassDefFoundError when the following line of code 
is executed

         PrintWriter out = new PrintWriter(conn.getOutputStream());

The class that is not found is a Socket class (sorry I don't have the trace 
at the moment) is not found. This happens when the number of threads on the 
client is high (c. 1000). Obviously it is not a Tomcat problem so I 
shouldn't even mention it. It is easily worked around by catching for the 
error and retrying.

It looks slightly similar to a problem in JKD 1.2 where the Calender class 
would load a ResourceBundle containing Locale info (in particular Date 
format info) and would report a MissingResourceBundle exception 
occasionally with a high number of threads creating Calenders 
simultaneously. A chain of resources are created and loaded into a 
HashTable during a call to load the resource, but some of the resources may 
be garbage collected during recursive calls to ResourceBundle.getObject()
resulting in a MissingResourceException. Since garbage collection is not 
very frequent the chances of this occurring during a resource access are 
quite low. I only mention it since it has some surface similarities with my 
problem. For more info look here
http://developer.java.sun.com/developer/bugParade/bugs/4335714.html

2) The second problem may be an actual Tomcat problem.
It is caused by the same line of code and is an IOException saying that the 
connection has been refused.  Even when we increase various parameters in 
the server.xml file -
     <Parameter name="backlog" value="4096" />
     <!-- Make the threads work better for multiple connections. -->
     <Parameter name="max_threads" value="4096"/>
     <Parameter name="max_spare_threads" value="256"/>
     <Parameter name="min_spare_threads" value="128">
to ensure that Tomcat will accept more connections it does not seem to 
work. This problem has been demonstrated to be independent of out servlet 
and appears when accessing a resource such as a JSP file. Again this is not 
a critical problem since we can catch the exception and retry the 
connection but it is puzzling as to why the high backlog figure doesn't 
ensure that the socket connection attempts are not queued. Again sorry that 
I don't have a trace to hand at the moment.

3) We have more serious problems on the Tomcat side, after some while (a 
couple of hours) with a high number (1000 -1500) of clients continuously 
connecting we get errors in the tomcat logs
"2001-08-07 09:33:47 - Ctx( /certresolver ): IOException in: R( 
/certresolver+ /certs + null) socket write error (code=10055)"

A quick search for code=10055 reveals it to be an error code thrown by 
winsock, however it covers a couple of different situations including 
trying to increase the timeout on a socket. The one that appears most 
appropriate to this problem is a bug report at sun
http://developer.java.sun.com/developer/bugParade/bugs/4474929.html

"Embedded in the exception's message string is code 10055.  Under Win32, 
this error is WSAENOBUFS which means the TCP/IP stack has temporarily run 
out of buffer space and cannot complete the write.  Microsoft has verified 
this as a bug in the WinSock TCP/IP stack (see the MSDN article ID: 
Q201213).  This would be a recoverable error if not for the way the Java 
native code handles the error.  The recovery would simply be to wait for 
the stack buffers to empty and try sending the same data again at a later 
time."

It could possibly be related to this error but without more information in 
the exception we can't tell. Has anyone ever experienced something like this?

4) The final problem is that Tomcat reports is an OutOfMemoryException 
after continuous usage with a high number of clients. Increasing the memory 
available to the Tomcat process (using -Xmx500m) does not make the problem 
go away, the taskmanager reports that the Tomcat process is only using a 
fraction of its allocated memory. Could this be a related to the previous 
discussion on upper limits for blocking I/O connections per process

Many thanks if you can help in any way.
Eamonn


>Well I think you're completely nuts :) For 10,000 concurrent requests for
>*dynamic* data (or else you'd just be using Apache.. right?), you'd need a
>server farm full of fancy equipment, a load-balancing server, and stuff that
>even my wildly ignorant speculations can't conceive.
>
>Besides, I don't think any single server with simple thread-per-request,
>blocking IO model can handle that sort of load. A smart fellow called Matt
>Welsh did his PhD thesis on highly concurrent server apps, and the results of
>his thinking are here:
>
>http://www.cs.berkeley.edu/~mdw/proj/sandstorm/
>
>In particular, the paper "The Staged Event-Driven Architecture for Highly
>Concurrent Servers." is well worth reading.
>
>--Jeff
>(who thinks 50 concurrent users is pushing things)
>
>
>On Mon, Aug 13, 2001 at 01:54:28PM +0530, Santosh Pasi wrote:
> > Hi Jeff,
> >
> > Thanks for reply.
> > I am looking around 10,000 to 12,000 concurrent connections.
>[snip]


-----------------------------------------------------------------------------
Baltimore Technologies plc will not be liable for direct,  special,  indirect 
or consequential  damages  arising  from  alteration of  the contents of this
message by a third party or as a result of any virus being passed on.

In addition, certain Marketing collateral may be added from time to time to
promote Baltimore Technologies products, services, Global e-Security or
appearance at trade shows and conferences.

This footnote confirms that this email message has been swept by
Baltimore MIMEsweeper for Content Security threats, including
computer viruses.
   http://www.baltimore.com