tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier ...@ice-sa.com>
Subject Re: Connection count explosion due to thread http-nio-80-ClientPoller-x death
Date Mon, 16 Jun 2014 15:56:21 GMT
Lars Engholm Johansen wrote:
> Our company are running several Tomcat 7.0.52 high volume Ubuntu 12.04
> production servers.
> We are using Tomcat WebSockets (JSR356 implementation) heavily with 100M
> text messages (100GiB) per day.
> 
> We monitor webserver health by measuring several key parameters every
> minute, including tomcat connection count using:
> 
>     mBeanServer.getAttribute(threadPool, "connectionCount");     //
> threadPool is MBean of "type=ThreadPool"
> 
> ______The problem______
> 
> Our sites can run for days without problems, but once in a while the tomcat
> connection count suddenly starts growing abnormally fast.
> See this graph:  http://imgur.com/s4fOUte
> netstat shows these TCP connections to be mostly in CLOSE_WAIT state.

And if at that moment, you force the JVM that runs Tomcat to do a Garbage Collection, do 
you still have these numerous connections in CLOSE_WAIT state after the GC completed ?

> 
> Our sites still functions normally with no cpu spikes during this build up
> until around 60,000 connections, but then the server refuses further
> connections and a manual Tomcat restart is required.
> 
> We have no output in tomcat or our logs at the time when this event occurs.
> The only sign is when comparing full java thread dump with a dump from a
> newly launched Tomcat:
> One of  http-nio-80-ClientPoller-0  or  http-nio-80-ClientPoller-1  is
> missing/has died.
> 
> We have observed this problem at least since Tomcat 7.0.48 and can not find
> indications in Tomcat 7.0.x change logs that it should have been fixed in
> newer releases.
> 
> Any help or advises are appreciated,
> Best regards,
> Lars Engholm Johansen
> 
> 
> Our connector configuration:
> <Connector
> acceptCount="1500"
> acceptorThreadCount="4"
> asyncTimeout="100000"
> connectionTimeout="60000"
> connectionUploadTimeout="120000"
> disableUploadTimeout="false"
> enableLookups="false"
> keepAliveTimeout="120000"
> maxConnections="100000"
> maxPostSize="3000000"
> maxThreads="300"
> port="80"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> socket.soKeepAlive="true"
> />
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message