tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Clovis Wichoski" <clovis.wicho...@gmail.com>
Subject Re: Tomcat bottleneck on InternalInputBuffer.parseRequestLine
Date Tue, 08 Jul 2008 14:32:11 GMT
Hi Christophe,

if in future you get more stucks, try to increase our open files to 4096,
using follow commands:

ulimit -Hn 10240
ulimit -Sn 4096

to persist on reboot edit your /etc/security/limits.conf and add follow
lines:

*       soft    nofile          4096
*       hard    nofile          10240

i have changed my /etc/sysctl.conf with follow lines too, maybe one of that
can help you if you have problems in the future:
# tuneTCP
net.ipv4.tcp_window_scaling=0
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_mem=786432  1048576 1572864
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_keepalive_time=1800
net.ipv4.tcp_max_syn_backlog=4096
net.core.wmem_max=8388608
net.core.rmem_max=8388608
net.ipv4.tcp_rmem=4096 87380 8388608
net.ipv4.tcp_wmem=4096 87380 8388608

i followed some tips from IBM RedPaper - Linux Performance and Tuning
Guidelines, http://www.redbooks.ibm.com/abstracts/REDP4285.html

best regards

Clóvis
On Mon, Jul 7, 2008 at 6:02 AM, Christophe Fondacci <
christophefondacci@nextep-softwares.com> wrote:

> Hi Clovis,
>
> Thanks for your answers.
>
> Open files on our production servers are 1024.
> Here is the complete output of ulimit -a :
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> max nice                        (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 36352
> max locked memory       (kbytes, -l) 32
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 1024
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> max rt priority                 (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 36352
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
>
> Our production servers are connected with gigabit ethernet. However the
> servers used to reproduce the problems are only 100MBPS ethernet. The
> problem occurs in both test and production environment.
>
> Our TCP keep alive settings are :
> tcp_keepalive_time is 7200.
> tcp_keepalive_intvl is 75
> tcp_keepalive_probes is 9
>
> I am monitoring threads call stack using jprofiler which displays the stack
> from my initial mail.
>
> I've tried the suggestion from Filip Hanik (maxKeepAliveRequests="1" in my
> tomcat connector). But I was still able to reproduce the problem.
>
> Then I switched the connector to the NIO connector (I was previously using
> the HTTP/1.1 default connector). I was not able to reproduce the problem
> after hours of test (it usually happens after 10-20 minutes of heavy load).
> Pushed the configuration change to one of our 4 productions servers to
> monitor the efficiency.
>
> So far we didn't have any problem on the server with NIO connector after 5
> production days...
>
> Christophe.
>
>
>
>
> ----- Original Message ----- From: "Clovis Wichoski" <
> clovis.wichoski@gmail.com>
> To: "Tomcat Users List" <users@tomcat.apache.org>
> Sent: Friday, July 04, 2008 4:17 AM
> Subject: Re: Tomcat bottleneck on InternalInputBuffer.parseRequestLine
>
>
> hi, Christophe,
>
> well, i still dont find the reason about what is my problem, but some
> things
> that help me to avoid the problem to occurs frequently,
>
> i checked the limit for open files on linux, you can check yous with ulimit
> -a, here i set to 4096,
>
> how the machines are connected? its with gigabit ethernet?
>
> please show us all your configuration on /proc/sys/net/ipv4/ the most
> important is:
>
> cat /proc/sys/net/ipv4/tcp_keepalive_time
>
> but the what more impact on the performance was the right configuration of
> JDBC driver on the pool, i use MaxDB and the driver have a problem, that
> when getting new physical connections, the drive have a singleton pattern,
> that we cant get connections in parallel (really parallel, with multi-core
> processors) and when this parallel attempts to get connections occurs, we
> get stuck threads, but note, maybe the problem isnt with driver, its just a
> suspect, since i dont have a solution for this.
> another suspect is that, for some strange reason the socket on java exists,
> but the socket on linux (inode) dont exists anymore, and until java know
> this, the system stucks, until timeout, but i cant figure or simulate this,
> since its really a rare case, and for me just occurs only one time, and i
> dont have a way to prove that, i'm try to check my problem using the follow
> script:
>
> #!/bin/bash
> today=`date +%Y%m%d%H%M%S`
> psId=`/opt/java/jdk1.6.0_06/bin/jps | grep Bootstrap |  cut -d' ' -f1`
> /opt/java/jdk1.6.0_06/bin/jstack -l $psId >
> /mnt/logs/stack/stack${today}.txt
> echo "--- pstack ---" >> /mnt/logs/stack/stack${today}.txt
> pstack $psId >> /mnt/logs/stack/stack${today}.txt
> echo "--- lsof ---" >> /mnt/logs/stack/stack${today}.txt
> lsof >> /mnt/logs/stack/stack${today}.txt
> echo "--- ls -l /proc/${psId}/fd/ ---" >> /mnt/logs/stack/stack${today}.txt
> ls -l /proc/${psId}/fd/ >> /mnt/logs/stack/stack${today}.txt
> echo "stack do processo $psId gravado em /mnt/logs/stack/stack${today}.txt"
>
> when users reports a stuck, i run this script manually, ten times, then
> compare outputs, IBM have a tool that you can use to read better the jstack
> output, i dont remember the name right now, but tomorrow i will post the
> link here.
>
> lets see if we can share knowledge to win this fight, ;)
>
> regards
>
> Clóvis
>
> On Tue, Jul 1, 2008 at 12:23 PM, Christophe Fondacci <
> christophefondacci@nextep-softwares.com> wrote:
>
>  Hello all,
>>
>> We have a problem with tomcat on our production server.
>> This problem may be related to the one listed here :
>> http://grokbase.com/profile/id:hNxqA0ZEdnD-6GYFRNs-iIkKEvF907FNWdczKYQ719Q
>>
>> Here it is :
>> - We got 2 tomcat servers on 2 distinct machines.
>> - 1 server is our application (let's call it A for Application server)
>> - The other server is hosting solr (let's call it S for Solr server)
>> - All servers are Tomcat 6.0.14 running on jdk 1.6.0_02-b05 on Linux
>> (Fedora core 6)
>> - Server A performs http requests to server S by 2 ways :
>>   > A http get (using apache commons HttpClient) with URL like
>>
>> http://S:8080/solr/select/?q=cityuri%3AXEABDBFDDACCXmaidenheadXEABDBFDDACCX&facet=true&facet.field=price&fl=id&facet.sort=false&facet.mincount=1&facet.limit=-1
>> <
>> http://s:8080/solr/select/?q=cityuri%3AXEABDBFDDACCXmaidenheadXEABDBFDDACCX&facet=true&facet.field=price&fl=id&facet.sort=false&facet.mincount=1&facet.limit=-1
>> >
>>   > A http post with URL like : http://S:8080/solr/select/<
>> http://s:8080/solr/select/>with a set of 12 NameValuePair
>>
>>
>> When traffic is light on our server A, everything works great.
>> When traffic is high on our server A (simultation of 40 simultaneous users
>> with Jmeter), some requests to our server S take more than 200 seconds. It
>> happens randomly and we couldn't isolate an URL-pattern: an URL can return
>> in less than 500ms and the exact same URL can take 300s before
>> returning...
>>
>> We performed deep jvm analysis (using jprofiler) to observe what was going
>> on on the Solr-server. When the problem occurs, we can see threads which
>> are
>> stucked with the following call stack :
>>
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>> at
>>
>> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:700)
>> at
>>
>> org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
>> at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:805)
>> at
>>
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
>> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> Requests which returns in 200s+ seem to spend almost all their time
>> reading
>> this input stream...
>> The javadoc says parseRequestLine is used to parse the http header. As I
>> stated above, our URL seem quite small so I can't understand why it
>> happens.
>> The response from server S is very small as well.
>>
>> We are able to reproduce the problem with less than 40 threads, but it is
>> more difficult to repoduce.
>> As I said at the beginning, I have found a user which had a similar
>> problem
>> but the mailing list thread does not give any solution...
>>
>> Has anyone an idea of what is going on ? Is there settings we can use to
>> avoid this problems ?
>> I am out of ideas on what to try to fix this...
>>
>> Any help would be highly appreciated...thank you very much.
>> Christophe.
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message