hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Kunnumpurath <thomas.kunnumpur...@db.com>
Subject RE: ThreadSafeClientConnectionManager and KeepAlive connnections [I]
Date Thu, 08 Sep 2011 14:46:36 GMT
Classification: For internal use only 

Parameters we use:
httpClient.maxTotalConnections = 10
httpClient.maxConnectionsPerRoute = 10
httpClient.keepAliveTime = 60000

Only define it once on the route I am using.

Thanks for the pointer around the [null] state. Digging through the httpClient code, this
indicates the UserToken which is supposed to be set by the client. 
As such, I've added a UserToken to DefaultHttpClient as follows:
httpclient.setUserTokenHandler(new UserTokenHandler() {
                public Object getUserToken(HttpContext context) {
                    return "TESLAPOP";
                }
            });

The null is gone, but I'm still not reusing sockets.

Now interesting tidbid from the HttpClient docs:

7.2.2.1. Persistent stateful connections Please note that a persistent connection that carries
a state object can be reused only if the same state object is bound to the execution context
when requests are executed. So, it is really important to ensure the either same context is
reused for execution of subsequent HTTP requests by the same user or the user token is bound
to the context prior to request execution.



Which I am doing now, but its still not working... any other ideas?

Thanks,
____________________________________________________



Thomas Kunnumpurath



-----Original Message-----
From: Sam Crawford [mailto:samcrawford@gmail.com]
Sent: Thursday, September 08, 2011 10:02 AM
To: HttpClient User Discussion
Subject: Re: ThreadSafeClientConnectionManager and KeepAlive connnections

Thomas,

SO_REUSEADDRESS is not really relevent to HttpClient's connection pooling mechanism; it's
a hint to the OS's TCP stack to reuse an existing TCP socket to the same target in the TIME_WAIT
state.

I cannot see anything obviously wrong with what you are doing. Sanity check:
What parameters are you using for the TSCCM? i.e. What have you passed to TSCCM.setMaxTotal(),
setDefaultMaxPerRoute(), etc, or are you defining seperately on each HttpRoute?

One potential cause:
The third line of your log says "Getting free connection [HttpRoute[{s}->https://xxxx.intranet.com:410]][null]",
where the 'null' is 'state'. When you come to pool the connection on the final line, it looks
like the state has been set to some DN (perhaps a client/server SSL certificate?). I'm not
sure what this state variable indicates, but I suspect the fact that you're requesting the
connection with a null state and pooling it with a non-null state may be the cause of your
problem.

Thanks,

Sam



On 8 September 2011 14:21, Thomas Kunnumpurath
<thomas.kunnumpurath@db.com>wrote:

> Classification: *Public *
>
> Hi,
>
> I am using the ThreadSafeClientConnectionManager in the HttpClient 4.1 
> API to submit secure requests to a webserver (nginx). From the debug 
> logs, I do see that the connections are being pooled and reused, 
> however, if I use netstat or tcpview I see that new TCP connections 
> are constantly being created in the ESTABLISHED state and the old ones 
> are put in TIME_WAIT state with no further activity. What I would 
> expect to see is that the same TCP handle would be reused for a period 
> of time. Can someone explain to me why a KeepAlive connection kicked 
> off by HttpClient would not be reused? I can confirm that the web 
> server supports keep alive and that the server is reporting that my 
> program is closing the keep-alive connections****
>
> ** **
>
> Please see a sample of my DEBUG logs below:****
>
> ** **
>
> DEBUG 03:07:03.490 - 08/09/2011 [pool-1-thread-43] [] 
> o.a.h.i.c.t.ConnPoolByRoute - [HttpRoute[{s 
> https://xxxx.intranet.com:410]] total kept alive: 10, total issued: 0, 
> total allocated: 10 out of 10****
>
> DEBUG 03:07:03.490 - 08/09/2011 [pool-1-thread-43] [] 
> o.a.h.i.c.DefaultClientConnection - Connection closed****
>
> DEBUG 03:07:03.490 - 08/09/2011 [pool-1-thread-43] [] 
> o.a.h.i.c.t.ConnPoolByRoute - Getting free connection [HttpRoute[{s}->
> https://xxxx.intranet.com:410]][null]****
>
> DEBUG 03:07:03.490 - 08/09/2011 [pool-1-thread-43] [] 
> o.a.h.i.c.DefaultClientConnectionOperator - Connecting to :// 
> xxxx.intranet.com:410 DEBUG 03:07:03.549 - 08/09/2011 
> [pool-1-thread-43] [] o.a.h.c.p.RequestAddCookies - CookieSpec 
> selected: best-match****
>
> DEBUG 03:07:03.549 - 08/09/2011 [pool-1-thread-43] [] 
> o.a.h.c.p.RequestAuthCache - Auth cache not set in the context****
>
> DEBUG 03:07:03.549 - 08/09/2011 [pool-1-thread-43] [] 
> o.a.h.i.c.DefaultHttpClient - Attempt 1 to execute request****
>
> DEBUG 03:07:03.549 - 08/09/2011 [pool-1-thread-43] [] 
> o.a.h.i.c.DefaultClientConnection - Sending request: PUT 
> /bla/put?name=TG_AUD_3M  HTTP/1.1****
>
> DEBUG 03:07:03.552 - 08/09/2011 [pool-1-thread-43] [] 
> o.a.h.i.c.DefaultClientConnection - Receiving response: HTTP/1.1 201 
> Created
> ****
>
> DEBUG 03:07:03.552 - 08/09/2011 [pool-1-thread-43] [] 
> o.a.h.i.c.DefaultHttpClient - Connection can be kept alive for 60000
> MILLISECONDS****
>
> DEBUG 03:07:03.552 - 08/09/2011 [pool-1-thread-43] [] 
> o.a.h.i.c.t.ThreadSafeClientConnManager - Released connection is reusable.
> ****
>
> DEBUG 03:07:03.552 - 08/09/2011 [pool-1-thread-43] [] 
> o.a.h.i.c.t.ConnPoolByRoute - Releasing connection 
> [HttpRoute[{s}->https://xxxx.intranet.com:410
> ]][CN=xxxxx.uk.db.com:Tesla, O=xxxx, DC=xx, DC=com]****
>
> DEBUG 03:07:03.552 - 08/09/2011 [pool-1-thread-43] [] 
> o.a.h.i.c.t.ConnPoolByRoute - Pooling connection 
> [[HttpRoute[{s}->https://xxxx.intranet.com:410
> ]][CN=xxxxx.uk.db.com:Tesla, O=xxxx, DC=xx, DC=com];keep alive for 
> 60000 MILLISECONDS
>
> I have a PUT request twice every second and I see that on average 2 
> TCP connections are established on the server every second, with the 
> previous 2 being put in TIME_WAIT state.
>
> The Response received from the server is as follows:
>
> Response headers:****
>
> HTTP/1.1 201 Created****
>
> Server: nginx/1.0.4****
>
> Date: Thu, 08 Sep 2011 05:00:52 GMT****
>
> Content-Type: application/octet-stream****
>
> Content-Length: 8****
>
> Connection: keep-alive****
>
> ** **
>
> ** **
>
> HTTPSampleResult fields:****
>
> ContentType: application/octet-stream****
>
> DataEncoding: null****
>
> ** **
>
> Wire Logs below:****
>
> DEBUG 09:17:18.034 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> >> "PUT /tesla/put?name=TG_AUD_3M HTTP/1.1[\r][\n]"****
>
> DEBUG 09:17:18.034 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> >>
> "Content-Length: 3160[\r][\n]"****
>
> DEBUG 09:17:18.034 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> >>
> "Host: xxxxx:410[\r][\n]"****
>
> DEBUG 09:17:18.034 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> >>
> "Connection: Keep-Alive[\r][\n]"****
>
> DEBUG 09:17:18.034 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> >>
> "User-Agent: Apache-HttpClient/4.1.2 (java 1.5)[\r][\n]"****
>
> DEBUG 09:17:18.034 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> >>
> "[\r][\n]"****
>
> ..........****
>
> DEBUG 09:17:18.034 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> >>
> "[\n]"****
>
> DEBUG 09:17:18.112 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> <<
> "HTTP/1.1 201 Created[\r][\n]"****
>
> DEBUG 09:17:18.112 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> <<
> "Server: nginx/1.0.4[\r][\n]"****
>
> DEBUG 09:17:18.112 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> <<
> "Date: Thu, 08 Sep 2011 13:17:17 GMT[\r][\n]"****
>
> DEBUG 09:17:18.112 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> <<
> "Content-Type: application/octet-stream[\r][\n]"****
>
> DEBUG 09:17:18.112 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> <<
> "Content-Length: 8[\r][\n]"****
>
> DEBUG 09:17:18.112 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> <<
> "Connection: keep-alive[\r][\n]"****
>
> DEBUG 09:17:18.112 - 08/09/2011 [pool-1-thread-1] [] o.a.http.wire - 
> <<
> "[\r][\n]"****
>
> ** **
>
> Shouldn't the default behavior be to reuse the existing sockets (I've 
> even set the SO_REUSEADDRESS to true, but to no avail). Assistance in 
> troubleshooting this issue would be greatly appreciated.****
>
> ** **
>
> ** **
>
> Thanks,****
>
> ____________________________________________________
>
> [image: https://brandportal.intranet.db.com/img/modules/logo.gif]
>
> Thomas Kunnumpurath
> Assistant Vice President | Application Developer - AS Pricing
>
> Deutsche Bank AG, Filiale New York
> Global Rates IT
> 60 Wall Street, 10005-2836 New York, NY, USA Tel. +1(212)250-2269 
> Email thomas.kunnumpurath@db.com****
>
>
> [image: https://brandportal.intranet.db.com/img/modules/claim.gif]****
>
> ** **
>
> ---
>
> This communication may contain confidential and/or privileged information.
> If you are not the intended recipient (or have received this 
> communication in error) please notify the sender immediately and 
> destroy this communication. Any unauthorized copying, disclosure or 
> distribution of the material in this communication is strictly forbidden.
>
> Deutsche Bank does not render legal or tax advice, and the information 
> contained in this communication should not be regarded as such.
>
>
>

---
This communication may contain confidential and/or privileged information. If you are not
the intended recipient (or have received this communication in error) please notify the sender
immediately and destroy this communication. Any unauthorized copying, disclosure or distribution
of the material in this communication is strictly forbidden.
Deutsche Bank does not render legal or tax advice, and the information contained in this communication
should not be regarded as such.

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message