Mailing-List: contact httpclient-users-help@hc.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpClient User Discussion" <httpclient-users@hc.apache.org>
Received-SPF: neutral (athena.apache.org: local policy)
Message-ID: <4AD4FC7F.4090701@wanadoo.fr>
Date: Tue, 13 Oct 2009 23:17:35 +0100
From: Tony Poppleton <tony.poppleton@wanadoo.fr>
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: HttpClient User Discussion <httpclient-users@hc.apache.org>
Subject: Re: Effectively disabling stale connection check
References: <4AD4CED5.9060907@wanadoo.fr> <4AD4DA09.1030207@apache.org>
In-Reply-To: <4AD4DA09.1030207@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi Oleg,

Thanks for the fast and excellent response, am implementing your 
suggested changes now.

Incidentally, the problem I observed where it sometimes didn't appear to 
retry, was in fact due to problem 2 retrying, and then immediately being 
followed by problem 1.  The stack trace for problem 1 was on the console 
stderr stream, so wasn't appearing the logs, which is why I thought it 
was mysteriously exiting before retrying the full 5 times.  So case 
closed on that one, it was my misinterpretation.

Regarding the stale connection check, I am sure you have already 
considered this, but what is the problem with doing the following:
 - removing the stale connection check (or at least disabling by default)
 - making the default http client use a retry handler that silently 
retries the IO exceptions
 - making the default http client have an IdleConnectionMonitorThread as 
per your reference [2], that cleans up the stale connections

Wouldn't that cover all the bases and be just as effective at managing 
connections as with the stale connection check enabled, and no one would 
report any problems, and get a free performance benefit?  Or is there 
something else that the stale connection check is doing that makes it a 
better choice?

Thanks again,
Tony


Oleg Kalnichevski wrote:
>
> Tony Poppleton wrote:
>> Hi,
>>
>> I am eager to squeeze the most performance out of my usage of 
>> HttpClient 4.0 and have been reading the archives where it was 
>> suggested to disable the stale connection check.  I have done this, 
>> and it does indeed significantly improve performance, however I am 
>> now occasionally getting the following two exceptions:
>>
>> 1.  Exception in thread "main" java.net.SocketException: Connection 
>> reset by peer: socket write error
>>        at java.net.SocketOutputStream.socketWrite0(Native Method)
>>        at 
>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>>        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>>        at 
>> org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:106) 
>>
>>        at 
>> org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:113) 
>>
>>        at 
>> org.apache.http.impl.AbstractHttpClientConnection.doFlush(AbstractHttpClientConnection.java:260) 
>>
>>        at 
>> org.apache.http.impl.SocketHttpClientConnection.close(SocketHttpClientConnection.java:248) 
>>
>>        at 
>> org.apache.http.impl.conn.DefaultClientConnection.close(DefaultClientConnection.java:154) 
>>
>>        at 
>> org.apache.http.impl.conn.AbstractPooledConnAdapter.close(AbstractPooledConnAdapter.java:131) 
>>
>>        at 
>> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:130) 
>>
>>        at 
>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447) 
>>
>>        at 
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641) 
>>
>>        at 
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:730) 
>>
>>        at 
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:708) 
>>
>>        at 
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:699) 
>>
>>        ...
>>
>> 2. DefaultRequestDirector:455 - I/O exception 
>> (org.apache.http.NoHttpResponseException) caught when processing 
>> request: The target server failed to respond
>>    DefaultRequestDirector:462 - Retrying request
>>
>> The first one is problematic because the program terminates (I don't 
>> handle the exception anywhere).  The second one usually retries as I 
>> am using the DefaultHttpRequestRetryHandler, but on occasion the 
>> program has just exited too (without doing the 5 retries that I have 
>> requested - I am trying to capture this case in a debugger so I can 
>> get a clue as to what is happening, will post an update if I find 
>> anything).  The second error is far more frequent than the first, for 
>> example today I have seen the second error about 50 times, but the 
>> 1st error only once.
>>
>> Anyway, I presume that both these errors are only now apparent 
>> because I have disabled the stale connection check.
>>
>
> Tony,
>
> The best strategy to tackle the stale connection issue is very simple: 
> do not let connections go stale in the first place by evicting 
> connections from the connection pool that have been idle for too long. 
> For details see [1] and [2].
>
>> What advice can you give me on how to recover from the first 
>> problem?  It is important that I don't re-post my requests if they 
>> did manage to get through before the error occurred.
>>
>
> As long as the method being executed can be considered idempotent [3] 
> it is safe simply to retry the request using a custom retry handler 
> [4] and be done with it. However, if your application uses HTTP 
> transport to execute transactional business logic (like placing an 
> order for an item) you may have a problem. Actually you do have a 
> problem anyways because HTTP transport is not transaction safe and 
> HTTP connections can and do fail in a middle of a transaction (the 
> request gets executed, an order gets placed but the response is never 
> delivered back to the client). To sum things up this is an application 
> design issue, and not an issue of the HTTP transport.
>
>> For the second one, is it 100% safe to resume from that error?  
>
> Yes, it is.
>
>
> Is it
>> fully equivalent in terms of network communications as having the 
>> stale connection check enabled?
>>
>
> Pretty much.
>
>> Out of interest, what is the history of the stale connection check?  
>>  From reading the archives it appears it is a relic of an older 
>> HttpClient, and its usage is not recommended anymore.  Is this correct?  
>
> Yes, it is
>
>> If so, why isn't it just removed from the latest version, and 
>> substituted with a robust retry handler that can deal with all the 
>> consequences?
>>
>
> Firstly, there is no such thing as a robust retry handler for HTTP 
> transport for the reason given above. HTTP request retrial logic is 
> always application specific. Secondly, there are tons of people out 
> there who do not really understand what persistent connections are all 
> about and start complaining about HttpClient nor working if they 
> occasionally see a request failing with an I/O exception. It is easier 
> for the sake of everyone's sanity to have the damn stale check on per 
> default.
>
> Cheers
>
> Oleg
>
> [1] 
> http://hc.apache.org/httpcomponents-client/tutorial/html/connmgmt.html#d4e638 
>
> [2] 
> http://hc.apache.org/httpcomponents-client/tutorial/html/connmgmt.html#d4e645 
>
> [3]
> http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e255 
>
> [4]
> http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e280 
>
>
>> Many thanks,
>> Tony
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
> --------------------------------------------------------------------------------------- 
>
> Orange vous informe que cet  e-mail a ete controle par l'anti-virus 
> mail. Aucun virus connu a ce jour par nos services n'a ete detecte.
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org