hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Becke <be...@u.washington.edu>
Subject Re: strange behavior of 2.0-rc1
Date Tue, 26 Aug 2003 02:18:02 GMT
As long as the connection is released, as it is via  
method.releaseConnection(), all is okay.

There are three ways to force a connection close:

1) Create an HttpMethod that either overrides  
HttpMethodBase.shouldCloseConnection() to always return true or that  
calls setConnectionCloseForced(true).
2) Implement or override an existing HttpConnectionManager that closes  
the connection in releaseConnection().
3) Call HttpMethod.setRequestHeader("Connection", "close") on each  
method before executing.  This one is not fool proof as the server must  
echo the header for it to be effective.  In all of the cases I've seen  
this is true.

Doing any of these things will ensure that all connections are closed  
when released.

Enjoy,

Mike

On Monday, August 25, 2003, at 09:03 PM, Leo Galambos wrote:

> OK, before I start with all the points, can you tell me whether the  
> following strategy may cause the problem?
>
> In Network.java you can see, that the response shorter than 64kB is  
> read completely, but when it is longer, just first 64kB are read. In  
> both cases, method.releaseConnection is called. Is this correct?
>
> If not, how can I close the connection? I had already asked here, but  
> I did not get a clear answer yet. You know, I cannot read the full  
> stream, because a bad boy could redirect /dev/random to a socket.
>
> Thank you
>
> Leo
>
>
> Michael Becke wrote:
>
>> Yea, "good news" :)
>>
>> So now we know that we're losing connections somewhere, and they are   
>> not being garbage collected.  There are a bunch of things we can try.  
>>   Here are a few ideas:
>>
>>  - enable debug logging for the MultiThreadedHttpConnectionManager.    
>> You will probably also want to print out the thread name with the  
>> log.   This will require Log4j or JRE 1.4 logging.
>>  - reduce the number of threads executing methods as well as the max   
>> number of   
>> connections(MultiThreadedHttpConnectionManager.setMaxTotalConnections( 
>> )) .  This should cause the error to occur more quickly.
>>  - if possible, try to come up with something standalone that   
>> reproduces the problem.  If I can reproduce the error I will be  
>> better  able to debug the problem.
>>  - run the application in debug mode so that when things get locked  
>> up  we can see where each of the threads are.
>>
>> Also, any more detail you can provide about your application and   
>> environment would be helpful.  Please let me know if you have any  
>> ideas  or questions.
>>
>> Mike
>>
>> On Monday, August 25, 2003, at 01:59 PM, Leo Galambos wrote:
>>
>>> Hi Mike,
>>>
>>> "good news":
>>>
>>> c-0.log:org.apache.commons.httpclient.HttpException: Timeout waiting  
>>>  for connection
>>> c-0.log-        at   
>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGe 
>>> tC onnection(MultiThreadedHttpConnectionManager.java:314)
>>> c-0.log-        at   
>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getC 
>>> on nection(MultiThreadedHttpConnectionManager.java:241)
>>> c-0.log-        at   
>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.jav 
>>> a: 609)
>>> c-0.log-        at   
>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.jav 
>>> a: 500)
>>> c-0.log-        at org.egothor.robot.Network.get(Network.java:109)
>>> c-0.log-        at  
>>> org.egothor.robot.Capek$Fetcher.run(Capek.java:186)
>>>
>>> What next?
>>>
>>> THX
>>>
>>> Leo
>>>
>>>
>>> Michael Becke wrote:
>>>
>>>> Hi Leo,
>>>>
>>>> These 20 CLOSE_WAIT sockets are from the 20 connections in the pool  
>>>>  held by MultiThreadedHttpConnectionManager.  When connections are   
>>>> released they are not necessarily closed.
>>>>
>>>> My guess is that connections are being lost somewhere.  If all 20   
>>>> connections are checked out and not released the 21st request for a  
>>>>  connection will block until one is available.  You can verify this  
>>>> by  setting httpConnectionFactoryTimeout on HttpClient.  This  
>>>> timeout is  how long HttpClient will wait when getting a connection  
>>>> from the  connection manager.  Setting this timeout to something  
>>>> big is  probably best, that way you won't run into it by accident.
>>>>
>>>> If connections are getting lost somewhere and   
>>>> httpConnectionFactoryTimeout is set you will start getting   
>>>> HttpExceptions with a message of "Timeout waiting for connection"  
>>>> on  every method execute.  Please give this a shot to ensure that  
>>>> the  problem is with lost connections.  Once verified we can start  
>>>> looking  for a cause.
>>>>
>>>> Also, in you example Network.java class I noticed the following  
>>>> lines:
>>>>
>>>>             while (r.status == -1 && attempt++ < RETRIES) {
>>>>                 try {
>>>>                     r.status = client.executeMethod(method);
>>>>                     r.msg = null;
>>>>                 } catch (Throwable x) {
>>>>                     method.recycle();
>>>>                     r.msg = url + ": " + x.getMessage();
>>>>                     if (logger.isLoggable(Level.FINE)) {
>>>>                         logger.log(Level.FINE, attempt + " of " +   
>>>> url, x);
>>>>                     }
>>>>                 }
>>>>             }
>>>>
>>>> HttpMethod.recycle() resets all values in the method, including the  
>>>>  URL and request headers.  Calling recycle in the exception case  
>>>> and  not resetting the various values will make the retry behave  
>>>> quite  differently.  My guess is that this is not what you were  
>>>> trying to >> do.
>>>>
>>>> Mike
>>>>
>>>> On Saturday, August 23, 2003, at 01:07 PM, Leo Galambos wrote:
>>>>
>>>>> Hi.
>>>>>
>>>>> I am using httpclient (HC) in a webcrawler. After 6 hours of run,  
>>>>> HC  stops working and I think, it is locked by some lock of a  
>>>>> critical  section in HC. The "problematic" code, I use, is here:   
>>>>> http://www.egothor.org/temp/Network.java
>>>>>
>>>>> When 20 threads are used, I have this in `netstat -an` after   
>>>>> stopping the robot from its console (note: the number of dead   
>>>>> connection is really *20*):
>>>>> tcp    23786      0 censored:60083    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24179      0 censored:60080    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24143      0 censored:60092    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24164      0 censored:60091    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24169      0 censored:60090    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24189      0 censored:60089    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24188      0 censored:60071    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    11697      0 censored:60066    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24152      0 censored:60078    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24180      0 censored:60076    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24208      0 censored:60074    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24154      0 censored:60073    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24150      0 censored:60044    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24150      0 censored:60041    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24194      0 censored:60101    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24206      0 censored:60099    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    24156      0 censored:60097    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    23643      0 censored:60096    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    23609      0 censored:60111    censored2:80            
>>>>> CLOSE_WAIT
>>>>> tcp    23961      0 censored:60110    censored2:80            
>>>>> CLOSE_WAIT
>>>>>
>>>>> Other connections are not allocated and the robot is waiting for   
>>>>> something (O/S does not report that the respective JAVA processes  
>>>>> >  run).
>>>>>
>>>>> Any thoughts?
>>>>>
>>>>> Thank you
>>>>>
>>>>> Leo
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------- 
>>>>> --
>>>>> To unsubscribe, e-mail:   
>>>>> commons-httpclient-dev-unsubscribe@jakarta.apache.org
>>>>> For additional commands, e-mail:   
>>>>> commons-httpclient-dev-help@jakarta.apache.org
>>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------- 
>>>> -
>>>> To unsubscribe, e-mail:   
>>>> commons-httpclient-dev-unsubscribe@jakarta.apache.org
>>>> For additional commands, e-mail:   
>>>> commons-httpclient-dev-help@jakarta.apache.org
>>>>
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:   
>>> commons-httpclient-dev-unsubscribe@jakarta.apache.org
>>> For additional commands, e-mail:   
>>> commons-httpclient-dev-help@jakarta.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:  
>> commons-httpclient-dev-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail:  
>> commons-httpclient-dev-help@jakarta.apache.org
>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:  
> commons-httpclient-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:  
> commons-httpclient-dev-help@jakarta.apache.org
>


Mime
View raw message