hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sam Crawford <samcrawf...@gmail.com>
Subject Re: Lots and lots of waiting threads
Date Fri, 04 Jun 2010 16:58:41 GMT
The fact that the connection counters are high suggests to me that the
problem lies within the OP's code.

Are you definitely consuming all of the response data from the target?
If you don't, the connection will never be freed, and they'll back up
(like you're finding). We had a similar problem many moons ago when we
didn't consume response data in certain cases.

Thanks,

Sam


On 4 June 2010 04:10, Ken Krugler <kkrugler_lists@transpac.com> wrote:
> Hi Ren,
>
> Here's my wild guess...
>
> You haven't adjusted the stack space for threads, and when you get thousands
> of them allocated at one time you get into disk swap hell.
>
> Eventually the JVM gets enough time to reclaim the connection threads, and
> you return to normal.
>
> So I'd check the status of your system with top when you find yourself in
> the situation below.
>
> If you are in swap hell, then you'd want to do the following:
>
> 1. Before running your Java code, from the same shell environment execute:
>
> ulimit -s 512
>
> This limits the native memory allocated per thread to 512K (assuming that's
> enough)
>
> 2. Use the -Xss512k Java parameter to limit Java memory allocated per
> thread.
>
> -- Ken
>
> On Jun 3, 2010, at 6:07pm, Renaud Waldura wrote:
>
>> I'm trying to track down a performance/resource consumption problem which
>> seems to involve HttpClient. After running for few days with low to medium
>> traffic (with low concurrency), my application stops responding. A thread
>> dump reveals many thousands of threads (between 3K and 4K) waiting for the
>> following condition (see below). The VM eventually recovers, the thread
>> count goes wayyy down (around 100 or so, which what I'd expect with the kind
>> of traffic we see.)
>>
>> "search" prio=10 tid=0x0000002ae20f6000 nid=0x38d4 waiting on condition
>> [0x0000002b35504000]
>> java.lang.Thread.State: WAITING (parking)
>>      at sun.misc.Unsafe.park(Native Method)
>>      - parking to wait for  <0x0000002aa9aa4dc0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>       at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>>       at
>> org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:159)
>>       at
>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:339)
>>       at
>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:238)
>>       at
>> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:175)
>>       at
>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:324)
>>       at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555)
>>       at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:487)
>>       at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:465)
>>       ... (truncated)
>>
>> The connection parameters are like so:
>>
>>           ConnManagerParams.setMaxTotalConnections(params, 400);
>>           ConnManagerParams.setMaxConnectionsPerRoute(params, new
>> ConnPerRouteBean(400));
>>           HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
>>           ConnManagerParams.setTimeout(params, 10);
>>           sharedConnectionManager = new
>> ThreadSafeClientConnManager(params, schemeRegistry);
>>
>> I've turned on logging, and I've seen the following counters go pretty
>> high:
>>
>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute     Total issued
>> connections: 300
>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute     Total allocated
>> connection: 300 out of 400
>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute     Total connections kept
>> alive: 46
>>
>> My application uses multiple instances of HttpClient, about a dozen or so,
>> each configured as above.
>>
>> Question: what is "total issued connections", and how does it compare to
>> "total allocated connections"?
>>
>> Environment:
>> HttpClient 4.0-beta1
>> HttpCore 4.0-beta2
>> Java HotSpot(TM) 64-Bit Server VM (16.3-b01 mixed mode):
>> Linux 2.6.9-78.ELsmp
>>
>> I know this isn't much to go on, and I apologize. But I'm hoping it may
>> ring a bell with someone who's got an idea why this is happening. Kind of a
>> reality check too. Does any of this smell funny?
>>
>> Thanks!
>>
>> --Ren
>>
>
> --------------------------------------------
> Ken Krugler
> +1 530-210-6378
> http://bixolabs.com
> e l a s t i c   w e b   m i n i n g
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message