hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sam Berlin" <sber...@gmail.com>
Subject Re: Keep-Alive timeout & persistent connections
Date Tue, 01 Jul 2008 23:23:24 GMT
Whew!  Thanks for the heads-up about the problem, and confirmation
that the fix works, Paul.

Sam

On Tue, Jul 1, 2008 at 7:16 PM, Paul Lindner <plindner@hi5.com> wrote:
> That did it.  I'm running svn head on one production server and seeing very
> good behavior.
>
> On Jun 30, 2008, at 6:53 PM, Sam Berlin wrote:
>
>> Hi Paul,
>>
>> I think I've fixed this one -- the problem was using deleteEntry
>> instead of dropEntry.  The former required the entry to be considered
>> 'free', but the entry had just been allocated and was definitely not
>> 'free'.  I'm pretty certain this will indirectly fix the
>> connection/memory leak -- a side effect of the problem was connection
>> requests would just pile up and never be serviced.
>>
>> Sam
>>
>> On Mon, Jun 30, 2008 at 10:33 AM, Paul Lindner <plindner@hi5.com> wrote:
>>>
>>> I tested this code using production traffic and the results are not so
>>> good.
>>> Ended up with a connection/memory leak.  The jmap histogram of a server
>>> in
>>> this bad state is attached.
>>>
>>> Also, I had some issues with one host, it used a keepalive with a
>>> timeout=1.
>>> It appears that the route pool filled up with connections.  It also
>>> appears
>>> that when it closes an expired free connection that it is not released
>>> back
>>> into the pool...  Here's the debug output showing the final connection
>>> allocation and the attempt to get a connection from a full pool.  It
>>> appears
>>> that Freeing the connection puts it back on the Idle pool, but it becomes
>>> inaccessible for some reason.
>>>
>>>
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager  -
>>> ThreadSafeClientConnManager.getConnection:
>>> HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com], timeout =
>>> 50002008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Getting free
>>> connection
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Closing expired free
>>> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.DefaultClientConnection  - Connection closed
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - No free connections
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Available capacity: 1
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Creating new
>>> connection
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]]
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.DefaultClientConnection  - Connection closed
>>> 2008-06-29 19:22:27,818 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Freeing connection
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:27,818 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.IdleConnectionHandler  - Adding connection at:
>>> 12147925478182008-06-29 19:22:27,818
>>> [http-80-55] DEBUG org.apache.http.impl.conn.tsccm.ConnPoolByRoute  -
>>> Notifying no-one, there are no waiting threads
>>>
>>> -----------------------------------------------
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager  -
>>> ThreadSafeClientConnManager.getConnection:
>>> HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com], timeout = 5000
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Getting free
>>> connection
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Closing expired free
>>> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.DefaultClientConnection  - Connection closed
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - No free connections
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Available capacity: 0
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Need to wait for
>>> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-79] DEBUG
>>> org.apache.http.impl.conn.DefaultClientConnection  - Connection shut down
>>> 2008-06-29 19:22:29,331 [http-80-79] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Freeing connection
>>> [HttpRoute[{}->http://ads.socialmedia.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-79] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Notifying thread
>>> waiting
>>> on any pool
>>> 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - No free connections
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Available capacity: 0
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Need to wait for
>>> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - No free connections
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Available capacity: 0
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute  - Need to wait for
>>> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>>
>>>
>>>
>>> Size    Count   Class description
>>> -------------------------------------------------------
>>> 684164112       558913  byte[]
>>> 492584016       2134710 char[]
>>> 209551728       4365661
>>> java.util.concurrent.locks.ReentrantLock$NonfairSync
>>> 209517696       4364952 java.util.concurrent.ConcurrentHashMap$Segment
>>> 144470200       4364956
>>> java.util.concurrent.ConcurrentHashMap$HashEntry[]
>>> 101379880       2534497 java.lang.String
>>> 65693112        628849  java.lang.Object[]
>>> 47067200        588340  com.friend.data.UserAppBean
>>> 41467120        272810  java.util.concurrent.ConcurrentHashMap$Segment[]
>>> 37081896        272661  org.apache.catalina.session.StandardSession
>>> 32830608        683971  java.util.HashMap$Entry
>>> 30965200        275343  java.util.Hashtable$Entry[]
>>> 24135760        603394  java.util.ArrayList
>>> 22612816        115641  java.util.HashMap$Entry[]
>>> 19871936        620998  java.util.Date
>>> 19642320        272810  java.util.concurrent.ConcurrentHashMap
>>> 18644208        224156  int[]
>>> 17608256        275129  java.util.Hashtable
>>> 14569296        303527  java.util.concurrent.ConcurrentHashMap$HashEntry
>>> 13089600        272700  java.beans.PropertyChangeSupport
>>> 13054072        97229   * ConstMethodKlass
>>> 12340480        154256  org.apache.tomcat.util.buf.MessageBytes
>>> 11679960        97229   * MethodKlass
>>> 10593856        165529  org.apache.tomcat.util.buf.ByteChunk
>>> 10529104        10652   * ConstantPoolKlass
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: dev-help@hc.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>> For additional commands, e-mail: dev-help@hc.apache.org
>>
>
> Paul Lindner
> plindner@hi5.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message