hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: problem with multithreading and proxy
Date Mon, 09 Feb 2009 18:50:19 GMT
Arash Joorabchi wrote:
> Hi all,
> 
> I am using MultiThreadedHttpConnectionManager in httpClient 3.1 to send 10
> get methods to the Library of Congress SRU server
> (http://z3950.loc.gov:7090/) concurrently each time. After running about 35
> queries I get “server failed to respond” exceptions. In this setting I do
> not set the httpClient  proxy parameters directly and instead use the
> following lines in the applications’ main method 
> 
> System.getProperties().put( "proxySet",  config.get_proxySet() );
> System.getProperties().put( "proxyHost", config.get_proxyHost() );
> System.getProperties().put( "proxyPort", config.get_proxyPort() );
> 
> If I set the httpClient proxy parameters directly instead of “server failed
> to respond” exception I get this exception: “HTTPclinet.fetch Method failed:
> HTTP/1.1 502 Proxy Error (The specified network name is no longer available. 
> )”.
> 
> When this occurs even the Internet Explorer fails to fetch the specified URI
> (e.g.
> http://z3950.loc.gov:7090/voyager?version=1.1&operation=searchRetrieve&maximumRecords=1000&query=title=%22Nieuw+Archief+voor+Wiskunde%22
> ), 
> 
> however it still works with other URIs like Google.
> 
> To prevent this problem I have to send about 10 queries wait for 10 seconds
> and then send another 10 queries.  Using this setting I ran about 20,000
> queries overnight successfully.
> 
> The most puzzling is that if I do not change the URL and send the same query
> over and over again no exception happens.
> 
> At this point I only can think of two things happening here:
> 
> 1. My proxy server does not respond after a while because it thinks I am an
> internal DOS attacker.
> 
> 2. The server thinks I am a DOS attacker and does not respond if I send the
> queries to fast.
> 
> I would appreciate your suggestions.
> 
> Notes:
> 
> 1. I do release the connections after each use.
> 2. I have tried sending queries with the connection close header.
> 3. I also tried another setting where I run 10 queries close/release
> everything shutdown the connection manager and create a new one and send
> another 10 queries.   
> 
> Thanks,
> Arash Joorabchi
> 

Arash,

Whatever the reason this does sound like a server side problem to me. 
Consider reducing the rate of requests per time unit.

Oleg

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message