hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: Connection Reset errors
Date Fri, 25 Jan 2013 12:45:42 GMT
Mark

You do not need to do anything special other than closing the content
input stream of the response entity (if it is present) to ensure proper
resource deallocation. What is important that it has to be done no
matter if you need the content or not. I do not think the problem you
are dealing with has anything to do with resource management.

There are probably several things you might want to try out

(1) if you can confirm that this issue is HttpClient version specific
and tell me what version of HttpClient exhibited it first (say, after
upgrade from 4.0.3 to 4.1) I could analyze the differences and try to
figure out what could potentially be the cause of connection resets.  

(2) This is entirely possible this is somehow SSL related and connection
gets reset by the SSL stack. You might want to run your app with SSL
debug enabled and see if there is anything in the SSL logs that could
help explain connection being reset.

(3) Otherwise there is no way around getting very close and personal
with Wireshark. 

Oleg 



On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> Something else I thought of is this:  We have some transactions where the response is
not too important.  For instance, if we send a pulse to the server to let it know we are still
here.  We don't really care what the response is, if we didn't get an error then we did our
part.  If we did get and error, we will try again soon anyway, so no big deal.
> 
> Is there something that needs to be done on a connection that is closed before reading
all the data?  I have never experienced reading stale data, so this seems a bit unlikely to
me.  However, if there is implicit cleanup that is being done, could this be, in some way,
causing my problems?
> 
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com] 
> Sent: Thursday, January 24, 2013 10:29 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
> 
> I am just not convinced this is a network thing.  It happens too often for her, and never
with the java.net connection.  She has a newer machine, maybe threading issues are just more
apparent on her hardware.
> 
> So, maybe the problem is somewhere in my code.  Can you tell me what are the methods
that might release a connection back to the pool?  I am wondering if there is a way I could
be releasing the connection back to the pool, it gets picked up by something else, and then
the original thread tries to close it again or something.
> 
> The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> --
> 
> Another avenue I am investigating is trying to force the issue by making the connections
work harder.
> I am trying to force the issue doing a lot of requests all at once.  I reduced my MAX
down to 1 to force contention.  Everything seems to work pretty well.  However, I did get
the Connection Reset error twice on my machine!  The logs were fairly regular looking since
there is only one connection at a time.  However, when this happened, it looks like it is
sending 2 request at once.  This shouldn't be happening because there is only one connection.
(Granted, it is hard to tell if the logging is indicating a threading issue, or if the "Connection
Reset" error caused the logging abnormality.)
> 
> I don't need to block around the "httpClient.execute(request);" do I?
> 
> I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> --
> 
> I do this to establish the connection:
> 
> This is run once in the setup
> -----------------------------
> connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> connectionManager.setDefaultMaxPerRoute(MAX);
> connectionManager.setMaxTotal(MAX);
> 
> httpClient = new DefaultHttpClient(connectionManager, params);
> 
> 
> 
> This is called when I want to make a request
> --------------------------------------------
> response = httpClient.execute(request);
> 
> 
> 
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 5:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
> 
> Well, the user went all day without any errors while using the Java.net connector.  
> 
> The error using the HttpClient connector has been happening with HttpClient 4.1 and higher,
and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3).  I will
have to check on the upgrade path and see if she had been using a 4.0.x version before this,
or if they jumped from a version that used a 3.x version.
> 
> Mark
> 
> 
> 
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:47 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
> 
> Oh, I forgot to add that all my requests are POST requests.
> 
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:02 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
> 
> While she is testing the java.net stuff, I decided to look into the possibility of retrying
the request.  Looking at this more closely, I see that the error is on the reading of the
data, and the write seems to succeed.  This gives me something to check on, since I should
be able to see whether or not the servlet actually got her request or not.
> 
> However, is this a common manifestation of the Connection Reset error?  The write succeeds
(or fails silently) and then it is on the read the problem can be detected?  Or does this
mean that the socket was fine during the write operation, but then was somehow severed before
the read?
> 
> Mark
> 
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 12:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
> 
> Looks like the stale connection check is enabled by default anyway, so I must have been
using it all along.
> 
> > It may well be that HttpURLConnection silently retries failed requests
> Is this a common thing to do, like for browsers?  Granted, she is in our app more than
most others, but I don't think she experiences issues with other applications.
> 
> She is not getting a Connection Refused or some other unable to read exception, but a
Connection Reset.  The stale connection check passed, so HttpClient thinks this is a good
connection, right?  Or are there limits to how reliable a stale connection check can be?
> 
>     public boolean isStale() {
>         if (!isOpen()) {
>             return true;
>         }
>         if (isEof()) {
>             return true;
>         }
>         try {
>             this.inbuffer.isDataAvailable(1);
>             return isEof();
>         } catch (SocketTimeoutException ex) {
>             return false;
>         } catch (IOException ex) {
>             return true;
>         }
>     }
> 
> 
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 11:24 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
> 
> On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > Before I switched to using HttpClient years ago, I used HttpsURLConnection.  Not
sure how things were going to turn out with HttpClient, I kept that code in place and made
it so I could use either API.  I had her switch back to using the java.net stuff the other
day and the problem appeared to go away.  I have configured her system to use the java.net
connector again today.  Before I make any conclusions I need some more data from her on each
connector.  However, the previous data seems to indicate that it is only an issue when using
HttpClient (originally 4.1.3, now 4.2.3).
> > 
> 
> It may well be that HttpURLConnection silently retries failed requests without your knowing
about it. It certainly did that when I last touched it (which admittedly was a looooooong
time ago), and that was precisely the reason I had to stop using it and to look for a better
alternative.
> 
> 
> > Some other things of note: 
> > - She is using JRE 6 Update 13.  (There is a long story behind this.)
> > - The application is launched via Webstart
> > - The SocketFactory is created with: fact = new 
> > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),verif
> > ier);
> > 
> 
> I do not thing any of these should matter.
> 
> Oleg
> 
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 18, 2013 6:17 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> > 
> > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > Thanks for the explanation.
> > > > pro-actively evicting expired connections and connections that 
> > > > have been idle
> > > Seems like just doing the stale check would more reliable, and speed 
> > > does not seem to be a factory (especially with 4.2.3)
> > > 
> > > I got in configured and distributed to this user and she is still 
> > > having the same problem.  Any other ideas?  I double checked my 
> > > code, and it looks like everything is correct.  (She is not using a
> > > proxy.)
> > > 
> > 
> > This can also be genuine network stability issues. Essentially, connection resets
are inevitable and robust HTTP services have to be prepared to deal with them (most likely
by re-trying idempotent methods automatically and reporting the problem back to the caller
otherwise).
> > 
> > Oleg
> > 
> > 
> > > 
> > > 	Method that makes the request (from multiple threads)
> > > 	{
> > > 		response = httpClient.execute(request);
> > > 	}
> > > 	Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > 	{
> > > 			HttpParams params = new BasicHttpParams();
> > > 			params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45,
TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > 			params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT
* 1000);
> > > 			params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > 			params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > 			configureProxy(params,protocol);
> > > 
> > > 			connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > 			connectionManager.setDefaultMaxPerRoute(MAX);
> > > 			connectionManager.setMaxTotal(MAX);
> > > 
> > > 			httpClient = new DefaultHttpClient(connectionManager, params);
> > > 	}
> > > 	private void configureProxy(HttpParams params, String scheme) {
> > > 		Proxy proxy = getProxy();
> > > 		if (proxy.type() != Proxy.Type.DIRECT) {
> > > 			InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > 			//SSL or not, we set the proxy up as HTTP
> > > 			HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(),
SCHEME_HTTP);
> > > 			params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > 			LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > 			if (!scheme.equals(SCHEME_HTTP)) {
> > > 				//If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > 				schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > 			}
> > > 		}
> > > 
> > > 	}
> > > 
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Thursday, January 17, 2013 7:21 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > > 
> > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > > 
> > > Basically 'connection reset' errors are a direct result of a general limitation
of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket
to get notified of the opposite endpoint terminating the connection. While kept alive in the
pool blocking HTTP connections can get stale. However, the only way to find it out is to try
to attempt an I/O operation on such connection.
> > > 
> > > > I noticed that there is a stale connection check:
> > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK,
> > > > Boolean.TRUE);
> > > > 
> > > > Is this something that I should be using?  I am connecting to the same
source over and over again.
> > > > 
> > > > Mark
> > > > 
> > > 
> > > Yes, this is what you should be using. Another alternative would be pro-actively
evicting expired connections and connections that have been idle longer than a given period
of time.
> > > 
> > > Oleg
> > > 
> > > 
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: Connection Reset errors
> > > > 
> > > > I have a user getting a lot of Connection Reset errors.  I did not 
> > > > think this had do to with HttpClient, so much as other factors on 
> > > > her machine and the server.  However, I did a bit of searching and see
that people have commented on this error with HttpClient before.  I just upgraded to HttpClient
4.2.3 to see if that would help at all, and it didn't.
> > > > 
> > > > This user is fine for a while, and then will get the Connection 
> > > > Reset error.  She then just retries the request, and it works.  Any tips
on how to resolve this would be greatly appreciated.
> > > > 
> > > > Thanks,
> > > > Mark
> > > > 
> > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > >                 <Connector
> > > >                         port="5502"
> > > >                         protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > >                         server="Unknown"
> > > >                         connectionLinger="0"
> > > >                         socket.soTimeout="300000"
> > > >                         SSLEnabled="true"
> > > >                         maxThreads="150"
> > > >                         scheme="https"
> > > >                         secure="true"
> > > >                         clientAuth="false"
> > > >                         keystoreFile="-----"
> > > >                         keystoreType="PKCS12"
> > > >                         keystorePass="-----"
> > > >                         sslProtocol="TLS"
> > > >  
> > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS
> > > > _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_E
> > > > DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > > 
> > > > 
> > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > >    Message (Connection reset)
> > > >    at java.net.SocketInputStream.read(Unknown Source)
> > > >    at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > >    at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > >    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > >    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown
Source)
> > > >    at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > >    at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > >    at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > >    at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > >    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > >    at
> > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > ja
> > > > va:138)
> > > > 
> > > > 
> > > > 
> > > > 
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail: 
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail: 
> > > > httpclient-users-help@hc.apache.org
> > > > 
> > > > 
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail: 
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail: 
> > > > httpclient-users-help@hc.apache.org
> > > > 
> > > 
> > > 
> > > 
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > > 
> > > 
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > > 
> > 
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message