hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leo Galambos <l...@centrum.cz>
Subject Re: strange behavior of 2.0-rc1
Date Thu, 28 Aug 2003 17:45:06 GMT
Mike, the the logs are here: http://com-os2.ms.mff.cuni.cz/temp/ 
(example.*.*)

THX

-g-


>>
>> Michael Becke wrote:
>>
>>> Yea, "good news" :)
>>>
>>> So now we know that we're losing connections somewhere, and they 
>>> are   not being garbage collected.  There are a bunch of things we 
>>> can try.    Here are a few ideas:
>>>
>>>  - enable debug logging for the 
>>> MultiThreadedHttpConnectionManager.    You will probably also want 
>>> to print out the thread name with the  log.   This will require 
>>> Log4j or JRE 1.4 logging.
>>>  - reduce the number of threads executing methods as well as the 
>>> max   number of   
>>> connections(MultiThreadedHttpConnectionManager.setMaxTotalConnections( 
>>> )) .  This should cause the error to occur more quickly.
>>>  - if possible, try to come up with something standalone that   
>>> reproduces the problem.  If I can reproduce the error I will be  
>>> better  able to debug the problem.
>>>  - run the application in debug mode so that when things get locked  
>>> up  we can see where each of the threads are.
>>>
>>> Also, any more detail you can provide about your application and   
>>> environment would be helpful.  Please let me know if you have any  
>>> ideas  or questions.
>>>
>>> Mike
>>>
>>> On Monday, August 25, 2003, at 01:59 PM, Leo Galambos wrote:
>>>
>>>> Hi Mike,
>>>>
>>>> "good news":
>>>>
>>>> c-0.log:org.apache.commons.httpclient.HttpException: Timeout 
>>>> waiting   for connection
>>>> c-0.log-        at   
>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGe 
>>>> tC onnection(MultiThreadedHttpConnectionManager.java:314)
>>>> c-0.log-        at   
>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getC 
>>>> on nection(MultiThreadedHttpConnectionManager.java:241)
>>>> c-0.log-        at   
>>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.jav 
>>>> a: 609)
>>>> c-0.log-        at   
>>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.jav 
>>>> a: 500)
>>>> c-0.log-        at org.egothor.robot.Network.get(Network.java:109)
>>>> c-0.log-        at  
>>>> org.egothor.robot.Capek$Fetcher.run(Capek.java:186)
>>>>
>>>> What next?
>>>>
>>>> THX
>>>>
>>>> Leo
>>>>
>>>>
>>>> Michael Becke wrote:
>>>>
>>>>> Hi Leo,
>>>>>
>>>>> These 20 CLOSE_WAIT sockets are from the 20 connections in the 
>>>>> pool   held by MultiThreadedHttpConnectionManager.  When 
>>>>> connections are   released they are not necessarily closed.
>>>>>
>>>>> My guess is that connections are being lost somewhere.  If all 
>>>>> 20   connections are checked out and not released the 21st request 
>>>>> for a   connection will block until one is available.  You can 
>>>>> verify this  by  setting httpConnectionFactoryTimeout on 
>>>>> HttpClient.  This  timeout is  how long HttpClient will wait when 
>>>>> getting a connection  from the  connection manager.  Setting this 
>>>>> timeout to something  big is  probably best, that way you won't 
>>>>> run into it by accident.
>>>>>
>>>>> If connections are getting lost somewhere and   
>>>>> httpConnectionFactoryTimeout is set you will start getting   
>>>>> HttpExceptions with a message of "Timeout waiting for connection"  
>>>>> on  every method execute.  Please give this a shot to ensure that  
>>>>> the  problem is with lost connections.  Once verified we can 
>>>>> start  looking  for a cause.
>>>>>
>>>>> Also, in you example Network.java class I noticed the following  
>>>>> lines:
>>>>>
>>>>>             while (r.status == -1 && attempt++ < RETRIES)
{
>>>>>                 try {
>>>>>                     r.status = client.executeMethod(method);
>>>>>                     r.msg = null;
>>>>>                 } catch (Throwable x) {
>>>>>                     method.recycle();
>>>>>                     r.msg = url + ": " + x.getMessage();
>>>>>                     if (logger.isLoggable(Level.FINE)) {
>>>>>                         logger.log(Level.FINE, attempt + " of " 
>>>>> +   url, x);
>>>>>                     }
>>>>>                 }
>>>>>             }
>>>>>
>>>>> HttpMethod.recycle() resets all values in the method, including 
>>>>> the   URL and request headers.  Calling recycle in the exception 
>>>>> case  and  not resetting the various values will make the retry 
>>>>> behave  quite  differently.  My guess is that this is not what you 
>>>>> were  trying to >> do.
>>>>>
>>>>> Mike
>>>>>
>>>>> On Saturday, August 23, 2003, at 01:07 PM, Leo Galambos wrote:
>>>>>
>>>>>> Hi.
>>>>>>
>>>>>> I am using httpclient (HC) in a webcrawler. After 6 hours of 
>>>>>> run,  HC  stops working and I think, it is locked by some lock of

>>>>>> a  critical  section in HC. The "problematic" code, I use, is 
>>>>>> here:   http://www.egothor.org/temp/Network.java
>>>>>>
>>>>>> When 20 threads are used, I have this in `netstat -an` after   
>>>>>> stopping the robot from its console (note: the number of dead   
>>>>>> connection is really *20*):
>>>>>> tcp    23786      0 censored:60083    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24179      0 censored:60080    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24143      0 censored:60092    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24164      0 censored:60091    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24169      0 censored:60090    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24189      0 censored:60089    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24188      0 censored:60071    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    11697      0 censored:60066    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24152      0 censored:60078    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24180      0 censored:60076    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24208      0 censored:60074    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24154      0 censored:60073    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24150      0 censored:60044    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24150      0 censored:60041    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24194      0 censored:60101    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24206      0 censored:60099    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    24156      0 censored:60097    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    23643      0 censored:60096    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    23609      0 censored:60111    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>> tcp    23961      0 censored:60110    censored2:80            
>>>>>> CLOSE_WAIT
>>>>>>
>>>>>> Other connections are not allocated and the robot is waiting 
>>>>>> for   something (O/S does not report that the respective JAVA 
>>>>>> processes  >  run).
>>>>>>
>>>>>> Any thoughts?
>>>>>>
>>>>>> Thank you
>>>>>>
>>>>>> Leo
>>>>>>
>>>>>>
>>>>>> -------------------------------------------------------------------

>>>>>> -- 
>>>>>> To unsubscribe, e-mail:   
>>>>>> commons-httpclient-dev-unsubscribe@jakarta.apache.org
>>>>>> For additional commands, e-mail:   
>>>>>> commons-httpclient-dev-help@jakarta.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> --------------------------------------------------------------------

>>>>> -
>>>>> To unsubscribe, e-mail:   
>>>>> commons-httpclient-dev-unsubscribe@jakarta.apache.org
>>>>> For additional commands, e-mail:   
>>>>> commons-httpclient-dev-help@jakarta.apache.org
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail:   
>>>> commons-httpclient-dev-unsubscribe@jakarta.apache.org
>>>> For additional commands, e-mail:   
>>>> commons-httpclient-dev-help@jakarta.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:  
>>> commons-httpclient-dev-unsubscribe@jakarta.apache.org
>>> For additional commands, e-mail:  
>>> commons-httpclient-dev-help@jakarta.apache.org
>>>
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:  
>> commons-httpclient-dev-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail:  
>> commons-httpclient-dev-help@jakarta.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: 
> commons-httpclient-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: 
> commons-httpclient-dev-help@jakarta.apache.org
>
>



Mime
View raw message