hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From maomaode <maoma...@gmail.com>
Subject Re: Out of memory using ThrottlingHttpClientHandler
Date Tue, 03 Jun 2008 16:17:42 GMT
Hi Oleg,

That make sense to me. I'll create a patch later

https://issues.apache.org/jira/browse/HTTPCORE-162

James


> On Tue, 2008-06-03 at 23:17 +0800, maomaode wrote:
>   
>> Hi Oleg,
>>
>> Thanks for the valuable information.
>>
>> I would love to create a testcase for it.
>>
>> I'm already a commiter of Apache CXF, can I check in the code directly 
>> into the httpclient? or provide a patch?
>>
>>     
>
> ´╗┐James,
>
> We tend to be very open about granting project karma to the existing
> Apache committers, especially if they are interested in contributing to
> the project more than just casually. But it would still make sense that
> you posted the first couple of patches for a review first. Once you get
> to know the project and the people better you can easily get write
> access to the SVN repository if you are still interested.
>
> Cheers
>
> Oleg
>
>
>   
>> Thanks,
>> James
>>
>>
>>     
>>> On Tue, 2008-06-03 at 21:42 +0800, maomaode wrote:
>>>   
>>>       
>>>> Hi Oleg,
>>>>
>>>> Thanks for the quick response, I'm dev an intranet web 
>>>> crawling/indexing, previous I used the httpclient 3-x.
>>>> we use the nio feature in the new httpclient to increase the performance.
>>>>
>>>>     
>>>>         
>>> Please note NIO is actually _significantly_ slow compared to classic I/O
>>> _unless_ you are dealing with thousands of simultaneous high latency
>>> connections. If the number of simultaneous connections is less than 500
>>> NIO is almost certainly bound to perform worse than the classic blocking
>>> I/O.  
>>>
>>>
>>>   
>>>       
>>>> I don't know how much of simultaneous conn httpclient-nio can handle? 
>>>> for me, I hope as much as the system can handle.
>>>>
>>>>     
>>>>         
>>> Really a lot. BUT every ´╗┐connection usually takes from 100 to 150KB, so
>>> you have to ensure there is enough memory of the heap to support a great
>>> number of ´╗┐simultaneous connections.  
>>>
>>>
>>>   
>>>       
>>>> I don't know the max length of the content in our system, I guess 300KB max?
>>>>
>>>> Since the ThrottlingHttpClientHandler  can use the ThreadPool, I guess 
>>>> the workerCount in the DefaultConnectingIOReactor will take NO effect, 
>>>> is that right?
>>>>
>>>>     
>>>>         
>>> Pretty much. But I would still recommend to use the same number of I/O
>>> dispatch workers as CPU cores on the system to ensure better CPU
>>> utilization. 
>>>
>>> Could you please create a test app that reproduces the problem you have
>>> been seeing and post it to this list or to the dev list?
>>>
>>> Cheers
>>>
>>> Oleg
>>>
>>>
>>>   
>>>       
>>>> Thanks,
>>>> James
>>>>
>>>> http://people.apache.org/~mmao
>>>>
>>>>
>>>>     
>>>>         
>>>>> On Tue, 2008-06-03 at 20:48 +0800, maomaode wrote:
>>>>>   
>>>>>       
>>>>>           
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>> I'm hitting a Out Of Memory error when using ThrottlingHttpClientHandler

>>>>>> <http://hc.apache.org/httpcomponents-core/httpcore-nio/apidocs/org/apache/http/nio/protocol/ThrottlingHttpClientHandler.html>
>>>>>>
>>>>>> I gave a Executors.|*newCachedThreadPool 
>>>>>> <cid:part1.01010308.04040401@gmail.com>*(), should I give a
fixed thread 
>>>>>> pool?
>>>>>>
>>>>>>     
>>>>>>         
>>>>>>             
>>>>> James,
>>>>>
>>>>> What is it exactly you are trying to do? How many simultaneous
>>>>> connections are you trying to open? What is the size of the content
>>>>> buffer? it is 20480 (default)? 
>>>>>
>>>>>
>>>>>   
>>>>>       
>>>>>           
>>>>>> And BTW, what's the workerCount means in the |DefaultConnectingIOReactor?
>>>>>>
>>>>>>     
>>>>>>         
>>>>>>             
>>>>> The number of threads I/O reactor uses to dispatch I/O events to the
>>>>> protocol handlers. This number should be equal to that of CPU cores of
>>>>> the system.  
>>>>>
>>>>> Hope this helps
>>>>>
>>>>> Oleg
>>>>>
>>>>>
>>>>>   
>>>>>       
>>>>>           
>>>>>> Thanks in advance,
>>>>>> James
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>>>
>>>>>>     
>>>>>>         
>>>>>>             
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>>
>>>>>
>>>>>   
>>>>>       
>>>>>           
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>
>>>>     
>>>>         
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>>   
>>>       
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>     
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message