hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: Effectively disabling stale connection check
Date Wed, 14 Oct 2009 20:22:20 GMT
On Tue, 2009-10-13 at 23:17 +0100, Tony Poppleton wrote:
> Hi Oleg,
> 
> Thanks for the fast and excellent response, am implementing your 
> suggested changes now.
> 
> Incidentally, the problem I observed where it sometimes didn't appear to 
> retry, was in fact due to problem 2 retrying, and then immediately being 
> followed by problem 1.  The stack trace for problem 1 was on the console 
> stderr stream, so wasn't appearing the logs, which is why I thought it 
> was mysteriously exiting before retrying the full 5 times.  So case 
> closed on that one, it was my misinterpretation.
> 
> Regarding the stale connection check, I am sure you have already 
> considered this, but what is the problem with doing the following:
>  - removing the stale connection check (or at least disabling by default)
>  - making the default http client use a retry handler that silently 
> retries the IO exceptions

Tony,

The trouble is HttpClient cannot simply silently retry _any_ I/O
exception because the knowledge whether the failed method is safe to
retry is application specific.


>  - making the default http client have an IdleConnectionMonitorThread as 
> per your reference [2], that cleans up the stale connections
> 

This can be done, but is not enough.

> Wouldn't that cover all the bases and be just as effective at managing 
> connections as with the stale connection check enabled, and no one would 
> report any problems, and get a free performance benefit?  Or is there 
> something else that the stale connection check is doing that makes it a 
> better choice?
> 

The stale connection check is a lesser evil compared to the disastrous
effect an automatic retrial of non-idempotent methods can have on
transactional applications.

I think the best we can do is to add a section to the HttpClient
tutorial describing in details what needs to be done in order to make
the stale connection check unnecessary.

Cheers

Oleg

> Thanks again,
> Tony
> 
> 
> Oleg Kalnichevski wrote:
> >
> > Tony Poppleton wrote:
> >> Hi,
> >>
> >> I am eager to squeeze the most performance out of my usage of 
> >> HttpClient 4.0 and have been reading the archives where it was 
> >> suggested to disable the stale connection check.  I have done this, 
> >> and it does indeed significantly improve performance, however I am 
> >> now occasionally getting the following two exceptions:
> >>
> >> 1.  Exception in thread "main" java.net.SocketException: Connection 
> >> reset by peer: socket write error
> >>        at java.net.SocketOutputStream.socketWrite0(Native Method)
> >>        at 
> >> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> >>        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> >>        at 
> >> org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:106)

> >>
> >>        at 
> >> org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:113)

> >>
> >>        at 
> >> org.apache.http.impl.AbstractHttpClientConnection.doFlush(AbstractHttpClientConnection.java:260)

> >>
> >>        at 
> >> org.apache.http.impl.SocketHttpClientConnection.close(SocketHttpClientConnection.java:248)

> >>
> >>        at 
> >> org.apache.http.impl.conn.DefaultClientConnection.close(DefaultClientConnection.java:154)

> >>
> >>        at 
> >> org.apache.http.impl.conn.AbstractPooledConnAdapter.close(AbstractPooledConnAdapter.java:131)

> >>
> >>        at 
> >> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:130)

> >>
> >>        at 
> >> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447)

> >>
> >>        at 
> >> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)

> >>
> >>        at 
> >> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:730)

> >>
> >>        at 
> >> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:708)

> >>
> >>        at 
> >> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:699)

> >>
> >>        ...
> >>
> >> 2. DefaultRequestDirector:455 - I/O exception 
> >> (org.apache.http.NoHttpResponseException) caught when processing 
> >> request: The target server failed to respond
> >>    DefaultRequestDirector:462 - Retrying request
> >>
> >> The first one is problematic because the program terminates (I don't 
> >> handle the exception anywhere).  The second one usually retries as I 
> >> am using the DefaultHttpRequestRetryHandler, but on occasion the 
> >> program has just exited too (without doing the 5 retries that I have 
> >> requested - I am trying to capture this case in a debugger so I can 
> >> get a clue as to what is happening, will post an update if I find 
> >> anything).  The second error is far more frequent than the first, for 
> >> example today I have seen the second error about 50 times, but the 
> >> 1st error only once.
> >>
> >> Anyway, I presume that both these errors are only now apparent 
> >> because I have disabled the stale connection check.
> >>
> >
> > Tony,
> >
> > The best strategy to tackle the stale connection issue is very simple: 
> > do not let connections go stale in the first place by evicting 
> > connections from the connection pool that have been idle for too long. 
> > For details see [1] and [2].
> >
> >> What advice can you give me on how to recover from the first 
> >> problem?  It is important that I don't re-post my requests if they 
> >> did manage to get through before the error occurred.
> >>
> >
> > As long as the method being executed can be considered idempotent [3] 
> > it is safe simply to retry the request using a custom retry handler 
> > [4] and be done with it. However, if your application uses HTTP 
> > transport to execute transactional business logic (like placing an 
> > order for an item) you may have a problem. Actually you do have a 
> > problem anyways because HTTP transport is not transaction safe and 
> > HTTP connections can and do fail in a middle of a transaction (the 
> > request gets executed, an order gets placed but the response is never 
> > delivered back to the client). To sum things up this is an application 
> > design issue, and not an issue of the HTTP transport.
> >
> >> For the second one, is it 100% safe to resume from that error?  
> >
> > Yes, it is.
> >
> >
> > Is it
> >> fully equivalent in terms of network communications as having the 
> >> stale connection check enabled?
> >>
> >
> > Pretty much.
> >
> >> Out of interest, what is the history of the stale connection check?  
> >>  From reading the archives it appears it is a relic of an older 
> >> HttpClient, and its usage is not recommended anymore.  Is this correct?  
> >
> > Yes, it is
> >
> >> If so, why isn't it just removed from the latest version, and 
> >> substituted with a robust retry handler that can deal with all the 
> >> consequences?
> >>
> >
> > Firstly, there is no such thing as a robust retry handler for HTTP 
> > transport for the reason given above. HTTP request retrial logic is 
> > always application specific. Secondly, there are tons of people out 
> > there who do not really understand what persistent connections are all 
> > about and start complaining about HttpClient nor working if they 
> > occasionally see a request failing with an I/O exception. It is easier 
> > for the sake of everyone's sanity to have the damn stale check on per 
> > default.
> >
> > Cheers
> >
> > Oleg
> >
> > [1] 
> > http://hc.apache.org/httpcomponents-client/tutorial/html/connmgmt.html#d4e638 
> >
> > [2] 
> > http://hc.apache.org/httpcomponents-client/tutorial/html/connmgmt.html#d4e645 
> >
> > [3]
> > http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e255

> >
> > [4]
> > http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e280

> >
> >
> >> Many thanks,
> >> Tony
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> >> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> > ---------------------------------------------------------------------------------------

> >
> > Orange vous informe que cet  e-mail a ete controle par l'anti-virus 
> > mail. Aucun virus connu a ce jour par nos services n'a ete detecte.
> >
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message