hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arash Joorabchi <arash.joorab...@ul.ie>
Subject Re: problem with multithreading and proxy
Date Tue, 10 Feb 2009 11:05:57 GMT



olegk wrote:
> 
> Arash Joorabchi wrote:
>> Hi all,
>> 
>> I am using MultiThreadedHttpConnectionManager in httpClient 3.1 to send
>> 10
>> get methods to the Library of Congress SRU server
>> (http://z3950.loc.gov:7090/) concurrently each time. After running about
>> 35
>> queries I get “server failed to respond” exceptions. In this setting I do
>> not set the httpClient  proxy parameters directly and instead use the
>> following lines in the applications’ main method 
>> 
>> System.getProperties().put( "proxySet",  config.get_proxySet() );
>> System.getProperties().put( "proxyHost", config.get_proxyHost() );
>> System.getProperties().put( "proxyPort", config.get_proxyPort() );
>> 
>> If I set the httpClient proxy parameters directly instead of “server
>> failed
>> to respond” exception I get this exception: “HTTPclinet.fetch Method
>> failed:
>> HTTP/1.1 502 Proxy Error (The specified network name is no longer
>> available. 
>> )”.
>> 
>> When this occurs even the Internet Explorer fails to fetch the specified
>> URI
>> (e.g.
>> http://z3950.loc.gov:7090/voyager?version=1.1&operation=searchRetrieve&maximumRecords=1000&query=title=%22Nieuw+Archief+voor+Wiskunde%22
>> ), 
>> 
>> however it still works with other URIs like Google.
>> 
>> To prevent this problem I have to send about 10 queries wait for 10
>> seconds
>> and then send another 10 queries.  Using this setting I ran about 20,000
>> queries overnight successfully.
>> 
>> The most puzzling is that if I do not change the URL and send the same
>> query
>> over and over again no exception happens.
>> 
>> At this point I only can think of two things happening here:
>> 
>> 1. My proxy server does not respond after a while because it thinks I am
>> an
>> internal DOS attacker.
>> 
>> 2. The server thinks I am a DOS attacker and does not respond if I send
>> the
>> queries to fast.
>> 
>> I would appreciate your suggestions.
>> 
>> Notes:
>> 
>> 1. I do release the connections after each use.
>> 2. I have tried sending queries with the connection close header.
>> 3. I also tried another setting where I run 10 queries close/release
>> everything shutdown the connection manager and create a new one and send
>> another 10 queries.   
>> 
>> Thanks,
>> Arash Joorabchi
>> 
> 
> Arash,
> 
> Whatever the reason this does sound like a server side problem to me. 
> Consider reducing the rate of requests per time unit.
> 
> Oleg
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 
> 
> 

Hi Oleg,

Thanks for your reply,

As I mentioned in the original email If I start 10 getMethod threads, wait
for all of them to finish and wait for another 10 seconds before starting
another 10 getMethod threads, then no exception is thrown, but if I reduce
the waiting time to less than 10 seconds then I will get the following
exception:
“HTTPclinet.fetch Method failed: HTTP/1.1 502 Proxy Error (The specified
network name is no longer available. )” or “Proxy timeout error”.

Do you think this is caused by my proxy server or the remote server that I
am sending queries to?

-- 
View this message in context: http://www.nabble.com/problem-with-multithreading-and-proxy-tp21912219p21931475.html
Sent from the HttpClient-User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message