tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christophe Fondacci" <christophefonda...@nextep-softwares.com>
Subject Tomcat bottleneck on InternalInputBuffer.parseRequestLine
Date Tue, 01 Jul 2008 15:23:59 GMT
Hello all,

We have a problem with tomcat on our production server.
This problem may be related to the one listed here :
http://grokbase.com/profile/id:hNxqA0ZEdnD-6GYFRNs-iIkKEvF907FNWdczKYQ719Q

Here it is :
- We got 2 tomcat servers on 2 distinct machines.
- 1 server is our application (let's call it A for Application server)
- The other server is hosting solr (let's call it S for Solr server)
- All servers are Tomcat 6.0.14 running on jdk 1.6.0_02-b05 on Linux (Fedora 
core 6)
- Server A performs http requests to server S by 2 ways :
     > A http get (using apache commons HttpClient) with URL like 
http://S:8080/solr/select/?q=cityuri%3AXEABDBFDDACCXmaidenheadXEABDBFDDACCX&facet=true&facet.field=price&fl=id&facet.sort=false&facet.mincount=1&facet.limit=-1
     > A http post with URL like :  http://S:8080/solr/select/ with a set of 
12 NameValuePair

When traffic is light on our server A, everything works great.
When traffic is high on our server A (simultation of 40 simultaneous users 
with Jmeter), some requests to our server S take more than 200 seconds. It 
happens randomly and we couldn't isolate an URL-pattern: an URL can return 
in less than 500ms and the exact same URL can take 300s before returning...

We performed deep jvm analysis (using jprofiler) to observe what was going
on on the Solr-server. When the problem occurs, we can see threads which are
stucked with the following call stack :

at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at 
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:700)
at 
org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:805)
at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)

Requests which returns in 200s+ seem to spend almost all their time reading 
this input stream...
The javadoc says parseRequestLine is used to parse the http header. As I 
stated above, our URL seem quite small so I can't understand why it happens. 
The response from server S is very small as well.

We are able to reproduce the problem with less than 40 threads, but it is 
more difficult to repoduce.
As I said at the beginning, I have found a user which had a similar problem 
but the mailing list thread does not give any solution...

Has anyone an idea of what is going on ? Is there settings we can use to 
avoid this problems ?
I am out of ideas on what to try to fix this...

Any help would be highly appreciated...thank you very much.
Christophe.


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message