tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Emile Litvak" <emile.lit...@turn.com>
Subject NIO connector under heavy load drops incoming requests?
Date Tue, 26 Feb 2008 00:46:19 GMT
Hello everyone,

 

We have a high load environment where we are running tomcat 5.5.15
successfully. We are interesting in reducing the system CPU load and
switching to Tomcat 6 with NIO, but so far have run into a few issues.
After trying out the patch Filip recommended

(http://svn.apache.org/viewvc?view=rev&revision=618420) we did get
further (our CPUs were not pegged any more and exceptions went away),
but it appears that about 2/3rds of the requests are being dropped. I
was going to file a bug, but I thought I'd post it first to see if
anyone has any ideas.

 

We have many tomcat servers running 5.5.15. I took one of the machines,
and put 6.0.16 on it, patched it. Each server gets around 3K requests a
minute during the time window when I saw doing my experiments. Using
6.0.16 regular HTTP connections, I do get around 3K requests per minute
that we process. With NIO, we only see around 1.2K per minute. Here are
the two connector configurations.

 

Regular config

    <Connector port="8000"

               maxThreads="500" minSpareThreads="25"
maxSpareThreads="75"

               enableLookups="false" redirectPort="443"
acceptCount="100"

               compression="on" 

               compressionMinSize="2048" 

               noCompressionUserAgents="gozilla, traviata" 

               compressableMimeType="text/html,text/xml"

               connectionTimeout="0" disableUploadTimeout="true" 

               maxHttpHeaderSize="8192" />

 

NIO config

    <Connector port="8000"
protocol="org.apache.coyote.http11.Http11NioProtocol"
acceptorThreadCount="4"

               maxThreads="500" minSpareThreads="25"
maxSpareThreads="75"

               enableLookups="false" redirectPort="443"
acceptCount="100"

               compression="on" 

               compressionMinSize="2048" 

               noCompressionUserAgents="gozilla, traviata" 

               compressableMimeType="text/html,text/xml"

               connectionTimeout="0" disableUploadTimeout="true" 

               maxHttpHeaderSize="8192" />

 

I tried to tune various NIO parameters and had no luck. I can only
assume that acceptors turn away some requests for some reason without
passing them down to the worker threads. Or that I have mis-configured
something.

 

Here are some interesting threads

 

A worker thread

http-8000-exec-20
52   TIMED_WAITING  20000000    

.....omited....

javax.servlet.http.HttpServlet.service(690)


javax.servlet.http.HttpServlet.service(803)


org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(290)


org.apache.catalina.core.ApplicationFilterChain.doFilter(206)


org.apache.catalina.core.StandardWrapperValve.invoke(233)


org.apache.catalina.core.StandardContextValve.invoke(175)


org.apache.catalina.authenticator.AuthenticatorBase.invoke(433)


org.apache.catalina.core.StandardHostValve.invoke(128)


org.apache.catalina.valves.ErrorReportValve.invoke(102)


org.apache.catalina.valves.AccessLogValve.invoke(568)


org.apache.catalina.core.StandardEngineValve.invoke(109)


org.apache.catalina.connector.CoyoteAdapter.service(286)


org.apache.coyote.http11.Http11NioProcessor.process(879)


org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.proce
ss(719)


org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(2080)


java.util.concurrent.ThreadPoolExecutor$Worker.runTask(885)


java.util.concurrent.ThreadPoolExecutor$Worker.run(907)


java.lang.Thread.run(619)


 

Idle worker thread

http-8000-exec-1
21   TIMED_WAITING  50000000
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f
aa65fe  -1           41            -1           17           -1

sun.misc.Unsafe.park(-2)


java.util.concurrent.locks.LockSupport.parkNanos(198)


java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
aitNanos(1963)


java.util.concurrent.LinkedBlockingQueue.poll(395)


java.util.concurrent.ThreadPoolExecutor.getTask(944)


java.util.concurrent.ThreadPoolExecutor$Worker.run(906)


java.lang.Thread.run(619)


 

NIO endpoint


http-8000-ClientPoller
20   RUNNABLE       370000000
-1           285           -1           8            -1

sun.nio.ch.EPollArrayWrapper.epollWait(-2)


sun.nio.ch.EPollArrayWrapper.poll(215)


sun.nio.ch.EPollSelectorImpl.doSelect(65)


sun.nio.ch.SelectorImpl.lockAndDoSelect(69)


sun.nio.ch.SelectorImpl.select(80)


org.apache.tomcat.util.net.NioEndpoint$Poller.run(1473)


java.lang.Thread.run(619)


 

My 4 acceptors


http-8000-Acceptor-3
19   BLOCKED        50000000    java.lang.Object@5302067
16           71            -1           0            -1

sun.nio.ch.ServerSocketChannelImpl.accept(129)


org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(1198)


java.lang.Thread.run(619)


 


http-8000-Acceptor-2
18   BLOCKED        30000000    java.lang.Object@5302067
16           48            -1           0            -1

sun.nio.ch.ServerSocketChannelImpl.accept(129)


org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(1198)


java.lang.Thread.run(619)


 


http-8000-Acceptor-1
17   BLOCKED        40000000    java.lang.Object@5302067
16           51            -1           0            -1

sun.nio.ch.ServerSocketChannelImpl.accept(129)


org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(1198)


java.lang.Thread.run(619)


 


http-8000-Acceptor-0
16   RUNNABLE       80000000
-1           59            -1           0            -1

sun.nio.ch.ServerSocketChannelImpl.accept0(-2)


sun.nio.ch.ServerSocketChannelImpl.accept(145)


org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(1198)


java.lang.Thread.run(619)  

 

My workers go up all the way to http-8000-exec-272. It looked like a
handful of them were doing work, but must were idle. Our servlet
requests are very short lived - 30ms tops.

 

Our application works with normal HTTP polling socket connector, but we
would like to use NIO because it would reduce the load on each tomcat
node, and let us scale better. I am very interested in trying any
suggestions or patches you might have. We have a unique scalability
environment for Tomcat - many unique requests from the internet at a
pretty fast pace. If NIO can be made work in our environment, it would
be a major milestone of stability and performance for it. It is also
possibly a configuration issue. If anyone has any thoughts/comments,
please post.

 

Thanks,

-Emile

 

 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message