hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomek Maciejewski <tomek.maciejew...@gmail.com>
Subject Re: BindException problem in multithreaded http client
Date Tue, 02 Sep 2008 21:57:31 GMT
Thanks Oleg, now it works. Earlier I used System.setProperty to enable 
logging but I get only WARN level logging, I don't know if the 
properties are case-sensitive because I set the same properties. Only 
difference was that I wrote "debug" instead of "debug".
Generally, with the debug enabled I don't get the exception. Maybe it is 
because, the crawler runs a little slower because of logging operation 
and saving them to output?
In the logs I found that the connection is created 3715 times and in the 
client I have the limit of 100 connections. Generally I often see 
something like that:

[DEBUG] ThreadSafeClientConnManager - Released connection open but not 
marked reusable.
[DEBUG] DefaultClientConnection - Connection shut down
[DEBUG] ConnPoolByRoute - Freeing connection 
[HttpRoute[{}->http://www.mrw.interscience.wiley.com]][null]
[DEBUG] ConnPoolByRoute - Creating new connection 
[HttpRoute[{}->http://www.mrw.interscience.wiley.com]]
[DEBUG] ThreadSafeClientConnManager - 
ThreadSafeClientConnManager.getConnection: 
HttpRoute[{}->http://www.mrw.interscience.wiley.com], timeout = 0

So the connection is not marked as reusable? Could it be a reason of my 
BindException problems?
I execute connections very simultaneously, right now there may be about 
600 threads in executors and every of them invokes 'execute method' on 
httpclient object.

Tomek
> On Tue, 2008-09-02 at 16:12 -0400, Tomek Maciejewski wrote:
>   
>> Hi Oleg,
>>
>> I am using Windows XP 32bit. I turned on context logging but I didn't 
>> see anything interesting.
>>     
>
> Are you seeing any DEBUG logs? Anyways, pass the following parameters to
> the JRE upon the start up:
>
> -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.SimpleLog -Dorg.apache.commons.logging.simplelog.log.org.apache.http.impl.conn=DEBUG
-Dorg.apache.commons.logging.simplelog.log.org.apache.http.headers=DEBUG
>
> This should result in DEBUG logs getting printed to the stdout and
> should give you enough details to help understand what may be causing
> problems with the reuse of connections. 
>
> For instance, this is a good example of connection not being reused due
> to connections being closed on the server side
>
> [DEBUG] ThreadSafeClientConnManager -
> ThreadSafeClientConnManager.getConnection:
> HttpRoute[{}->http://www.yahoo.com], timeout = 0
> [DEBUG] ConnPoolByRoute - Total connections kept alive: 0
> [DEBUG] ConnPoolByRoute - Total issued connections: 0
> [DEBUG] ConnPoolByRoute - Total allocated connection: 0 out of 200
> [DEBUG] ConnPoolByRoute - No free connections
> [HttpRoute[{}->http://www.yahoo.com]][null]
> [DEBUG] ConnPoolByRoute - Available capacity: 20 out of 20
> [HttpRoute[{}->http://www.yahoo.com]][null]
> [DEBUG] ConnPoolByRoute - Creating new connection
> [HttpRoute[{}->http://www.yahoo.com]]
> [DEBUG] DefaultClientConnection - Connection closed
> [DEBUG] headers - >> GET / HTTP/1.1
> [DEBUG] headers - >> Host: www.yahoo.com
> [DEBUG] headers - >> Connection: Keep-Alive
> [DEBUG] headers - >> User-Agent: Apache-HttpClient/4.0
> [DEBUG] headers - << HTTP/1.1 200 OK
> [DEBUG] headers - << Date: Tue, 02 Sep 2008 20:54:27 GMT
> [DEBUG] headers - << P3P: policyref="http://p3p.yahoo.com/w3c/p3p.xml",
> CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR
> DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA
> POL HEA PRE GOV"
> [DEBUG] headers - << Cache-Control: private
> [DEBUG] headers - << Vary: User-Agent
> [DEBUG] headers - << X-XRDS-Location:
> http://open.login.yahooapis.com/openid20/www.yahoo.com/xrds
> [DEBUG] headers - << Last-Modified: Tue, 02 Sep 2008 20:50:37 GMT
> [DEBUG] headers - << Accept-Ranges: bytes
> [DEBUG] headers - << Content-Length: 9562
> [DEBUG] headers - << Connection: close
> [DEBUG] headers - << Content-Type: text/html; charset=utf-8
> [DEBUG] ThreadSafeClientConnManager - Released connection open but not
> marked reusable.
> [DEBUG] DefaultClientConnection - Connection shut down
> [DEBUG] ConnPoolByRoute - Freeing connection
> [HttpRoute[{}->http://www.yahoo.com]][null]
> [DEBUG] ConnPoolByRoute - Notifying no-one, there are no waiting threads
>
> Hope this helps
>
> Oleg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message