hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: problem with multithreading and proxy
Date Wed, 11 Feb 2009 16:46:34 GMT
Arash Joorabchi wrote:
> 
> 
> olegk wrote:
>> Arash Joorabchi wrote:
>>> Hi all,
>>>
>>> I am using MultiThreadedHttpConnectionManager in httpClient 3.1 to send
>>> 10
>>> get methods to the Library of Congress SRU server
>>> (http://z3950.loc.gov:7090/) concurrently each time. After running about
>>> 35
>>> queries I get “server failed to respond” exceptions. In this setting I do
>>> not set the httpClient  proxy parameters directly and instead use the
>>> following lines in the applications’ main method 
>>>
>>> System.getProperties().put( "proxySet",  config.get_proxySet() );
>>> System.getProperties().put( "proxyHost", config.get_proxyHost() );
>>> System.getProperties().put( "proxyPort", config.get_proxyPort() );
>>>
>>> If I set the httpClient proxy parameters directly instead of “server
>>> failed
>>> to respond” exception I get this exception: “HTTPclinet.fetch Method
>>> failed:
>>> HTTP/1.1 502 Proxy Error (The specified network name is no longer
>>> available. 
>>> )”.
>>>
>>> When this occurs even the Internet Explorer fails to fetch the specified
>>> URI
>>> (e.g.
>>> http://z3950.loc.gov:7090/voyager?version=1.1&operation=searchRetrieve&maximumRecords=1000&query=title=%22Nieuw+Archief+voor+Wiskunde%22
>>> ), 
>>>
>>> however it still works with other URIs like Google.
>>>
>>> To prevent this problem I have to send about 10 queries wait for 10
>>> seconds
>>> and then send another 10 queries.  Using this setting I ran about 20,000
>>> queries overnight successfully.
>>>
>>> The most puzzling is that if I do not change the URL and send the same
>>> query
>>> over and over again no exception happens.
>>>
>>> At this point I only can think of two things happening here:
>>>
>>> 1. My proxy server does not respond after a while because it thinks I am
>>> an
>>> internal DOS attacker.
>>>
>>> 2. The server thinks I am a DOS attacker and does not respond if I send
>>> the
>>> queries to fast.
>>>
>>> I would appreciate your suggestions.
>>>
>>> Notes:
>>>
>>> 1. I do release the connections after each use.
>>> 2. I have tried sending queries with the connection close header.
>>> 3. I also tried another setting where I run 10 queries close/release
>>> everything shutdown the connection manager and create a new one and send
>>> another 10 queries.   
>>>
>>> Thanks,
>>> Arash Joorabchi
>>>
>> Arash,
>>
>> Whatever the reason this does sound like a server side problem to me. 
>> Consider reducing the rate of requests per time unit.
>>
>> Oleg
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>>
> 
> Hi Oleg,
> 
> Thanks for your reply,
> 
> As I mentioned in the original email If I start 10 getMethod threads, wait
> for all of them to finish and wait for another 10 seconds before starting
> another 10 getMethod threads, then no exception is thrown, but if I reduce
> the waiting time to less than 10 seconds then I will get the following
> exception:
> “HTTPclinet.fetch Method failed: HTTP/1.1 502 Proxy Error (The specified
> network name is no longer available. )” or “Proxy timeout error”.
> 
> Do you think this is caused by my proxy server or the remote server that I
> am sending queries to?
> 

In my opinion the issue is very likely to be caused by the target server.

Oleg



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message