hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Krugler <kkrugler_li...@transpac.com>
Subject Setting timeouts in 4.0
Date Mon, 04 May 2009 23:30:15 GMT
Hi all,

In Http 3.1, the Nutch code base would configure timeouts using the 
following snippet of code:

     MultiThreadedHttpConnectionManager connectionManager =
           new MultiThreadedHttpConnectionManager();

     HttpClient client = new HttpClient(connectionManager);

     HttpConnectionManagerParams params = connectionManager.getParams();
     params.setConnectionTimeout(timeout);
     params.setSoTimeout(timeout);

     // executeMethod(HttpMethod) seems to ignore the connection 
timeout on the connection manager.
     // set it explicitly on the HttpClient.

     client.getParams().setConnectionManagerTimeout(timeout);

What's the functional equivalent in 4.0? I'm assuming that:

     HttpParams params = new BasicHttpParams();
     ConnManagerParams.setTimeout(params, timeout);

is equivalent to the 3.1 call to 
params.setConnectionTimeout(timeout). But what about the 
setSoTimeout() call?

One reason I'm asking is that I ran into a very long timeout while 
trying to fetch a page. The wire log looked like:

09/05/04 16:03:39 DEBUG client.DefaultRequestDirector:408 - Attempt 1 
to execute request
09/05/04 16:03:39 DEBUG http.wire:78 - >> "GET 
/noticias/elrostrodeanaliacanal9-argentina-elrostrodeanalia-canal9/ 
HTTP/1.1[EOL]"
09/05/04 16:03:39 DEBUG http.wire:78 - >> "Host: 
telenovelas.censuratv.net[EOL]"
09/05/04 16:03:39 DEBUG http.wire:78 - >> "Connection: Keep-Alive[EOL]"
09/05/04 16:03:39 DEBUG http.wire:78 - >> "User-Agent: bixo[EOL]"
09/05/04 16:03:39 DEBUG http.wire:78 - >> "[EOL]"
09/05/04 16:03:39 DEBUG http.headers:251 - >> GET 
/noticias/elrostrodeanaliacanal9-argentina-elrostrodeanalia-canal9/ 
HTTP/1.1
09/05/04 16:03:39 DEBUG http.headers:254 - >> Host: telenovelas.censuratv.net
09/05/04 16:03:39 DEBUG http.headers:254 - >> Connection: Keep-Alive
09/05/04 16:03:39 DEBUG http.headers:254 - >> User-Agent: bixo
09/05/04 16:13:32 DEBUG conn.DefaultClientConnection:160 - Connection closed
09/05/04 16:13:32 DEBUG client.DefaultRequestDirector:414 - Closing 
the connection.
09/05/04 16:13:32 DEBUG conn.DefaultClientConnection:160 - Connection closed
09/05/04 16:13:32 INFO client.DefaultRequestDirector:418 - I/O 
exception (org.apache.http.NoHttpResponseException) caught when 
processing request: The target server failed to respond
09/05/04 16:13:32 DEBUG client.DefaultRequestDirector:423 - The 
target server failed to respond
09/05/04 16:13:32 INFO client.DefaultRequestDirector:425 - Retrying request
09/05/04 16:13:32 DEBUG client.DefaultRequestDirector:433 - Reopening 
the direct connection.
09/05/04 16:14:24 DEBUG conn.DefaultClientConnection:147 - Connection shut down
09/05/04 16:14:24 DEBUG tsccm.ThreadSafeClientConnManager:223 - 
Released connection is not reusable.
09/05/04 16:14:24 DEBUG tsccm.ConnPoolByRoute:374 - Releasing 
connection [HttpRoute[{}->http://telenovelas.censuratv.net]][null]
09/05/04 16:14:24 DEBUG tsccm.ConnPoolByRoute:631 - Notifying no-one, 
there are no waiting threads
09/05/04 16:14:24 DEBUG http.HttpClientFetcher:267 - Exception while 
fetching url 
http://telenovelas.censuratv.net/noticias/elrostrodeanaliacanal9-argentina-elrostrodeanalia-canal9/
java.net.UnknownHostException: telenovelas.censuratv.net

So the first try failed after about 10 minutes with an I/O exception 
(org.apache.http.NoHttpResponseException), then the retry failed much 
faster (50 seconds) with an java.net.UnknownHostException.

I'm guessing that maybe the real cause of the first long timeout was 
my DNS system timing out while trying to resolve the invalid server 
address, and then this "bad hostname" result was cached so that the 
retry failed faster.

But independent of the above, I'm interested in the best way to 
prevent all cases of long timeouts, with 4.0.

Thanks much!

-- Ken

PS - I could do my own DNS resolver that maps hostnames to IP 
addresses, and wrap this with a timer to fail after 10 seconds or so.
-- 
Ken Krugler
+1 530-210-6378

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message