hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Hjalmarsson <hja...@sgh.se>
Subject Re: HttpAsyncClient - Connection forcibly closed by remote host
Date Sat, 25 Feb 2012 01:47:58 GMT
Thank you for all your answers. I've now solved some of the problems I had,
but I got another question now.. I don't know if this this is a bug or if
I'm thinking about this in the wrong way:

In this example Im sending ONE request to a server, and I'm blocked until
I've received a response from the callback (either fail or success). It
works most of the requests and when it do work, it looks something like
this:

2012/02/25 02:17:14:590 CET [DEBUG] DefaultHttpAsyncClient - Connection
request suceeded: [id:8][route:{}->http://example.com][state:null]
2012/02/25 02:17:14:591 CET [DEBUG] DefaultHttpAsyncClient - Attempt 1 to
execute request
2012/02/25 02:17:15:261 CET [DEBUG] DefaultHttpAsyncClient - Response:
HTTP/1.1 200 OK
2012/02/25 02:17:15:316 CET [DEBUG] DefaultHttpAsyncClient - Response fully
read
2012/02/25 02:17:15:316 CET [DEBUG] DefaultHttpAsyncClient - Connection can
be kept alive indefinitely
2012/02/25 02:17:15:316 CET [DEBUG] DefaultHttpAsyncClient - Response
processed

Then suddenly I can make a call that outputs only the first row:
2012/02/25 02:17:15:466 CET [DEBUG] DefaultHttpAsyncClient - Connection
request suceeded: [id:1][route:{}->http://example.com][state:null]

but it doesn't go any further than that and the request is neither failed
for any kind of timeout nor completed within many minutes. It seems like it
never gets the possibility to make the request. When I've let the request
run for some time (~10 minutes), I forcibly shutdown the connectionmanager
and receives

2012/02/25 02:30:50:897 CET [DEBUG] DefaultHttpAsyncClient - HTTP exchange
cancelled

Which indicates that there was a request that was cancelled at least. Since
the requests never terminates, my program never gets out of the lock (since
the callback function is never executed). It might be a terrible way of
handling requests, but I can't see the explanation why it behaves like
this.

And yeah, I'm using the 4.0-BETA1 :)


On Fri, Feb 24, 2012 at 5:58 PM, Oleg Kalnichevski <olegk@apache.org> wrote:

> On Fri, 2012-02-24 at 16:10 +0100, Alexander Hjalmarsson wrote:
> > But how can I catch this exception? Is there anywhere I can register an
> > exceptionhandler that catches this?
> >
>
> FutureCallback#failed should give you that exception. You might also
> consider using a custom HttpAsyncResponseConsumer if you want to exert
> as full control over the process of response processing. Standard
> response consumers shipped with HttpAsyncClient should always preserve
> the cause exception and return it with
> HttpAsyncResponseConsumer#getException.
>
> I assume you are using 4.0-BETA1
>
> Oleg
>
>
> > On Thu, Feb 23, 2012 at 8:20 PM, Oleg Kalnichevski <olegk@apache.org>
> wrote:
> >
> > > On Thu, 2012-02-23 at 16:22 +0100, Alexander Hjalmarsson wrote:
> > > > I've created a simple crawling framework with the help from
> > > HttpAsyncClient
> > > > and sometimes I get the exception that an existing connection was
> > > forcibly
> > > > closed by the remote host. I'm doing the requests rather polite (one
> > > > request every 15 seconds) so I'm not really overwhelming the server.
> What
> > > > could be the cause for this? I'd also like to know how/where I can
> catch
> > > > this exception myself, so I don't continue to send requests to a
> server
> > > > that closes the connections.
> > > >
> > > > 2012-feb-23 16:04:59
> > > > org.apache.http.impl.nio.client.LoggingClientProtocolHandler
> exception
> > > > SEVERE: http-outgoing-259 [ACTIVE] I/O error: An existing connection
> was
> > > > forcibly closed by the remote host
> > > > java.io.IOException: An existing connection was forcibly closed by
> the
> > > > remote host
> > >
> > > Things like that can happen. HTTP servers can and do drop connections
> > > for all sorts of reasons. Your crawler should be prepared to deal with
> > > such situations. I also think there is absolutely no reason to
> blacklist
> > > the server just because of one failed request. The server should
> respond
> > > with status 503 (Service Unavailable) if it is unable to process the
> > > request due to overload or a similar condition.
> > >
> > > Oleg
> > >
> > >
> > > >     at sun.nio.ch.SocketDispatcher.read0(Native Method)
> > > >     at sun.nio.ch.SocketDispatcher.read(Unknown Source)
> > > >     at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
> > > >     at sun.nio.ch.IOUtil.read(Unknown Source)
> > > >     at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
> > > >     at
> > > >
> > >
> org.apache.http.impl.nio.reactor.SessionInputBufferImpl.fill(SessionInputBufferImpl.java:95)
> > > >     at
> > > >
> > >
> org.apache.http.impl.nio.codecs.AbstractMessageParser.fillBuffer(AbstractMessageParser.java:115)
> > > >     at
> > > >
> > >
> org.apache.http.impl.nio.conn.DefaultAsyncClientConnection$LoggingNHttpMessageParser.fillBuffer(DefaultAsyncClientConnection.java:164)
> > > >     at
> > > >
> > >
> org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:153)
> > > >     at
> > > >
> > >
> org.apache.http.impl.nio.DefaultClientIODispatch.onInputReady(DefaultClientIODispatch.java:118)
> > > >     at
> > > >
> > >
> org.apache.http.impl.nio.DefaultClientIODispatch.onInputReady(DefaultClientIODispatch.java:49)
> > > >     at
> > > >
> > >
> org.apache.http.impl.nio.AbstractIODispatch.inputReady(AbstractIODispatch.java:113)
> > > >     at
> > > >
> > >
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> > > >     at
> > > >
> > >
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
> > > >     at
> > > >
> > >
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
> > > >     at
> > > >
> > >
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> > > >     at
> > > >
> > >
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> > > >     at
> > > >
> > >
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:599)
> > > >     at java.lang.Thread.run(Unknown Source)
> > > >
> > > > The requests I make are just basic GET-requests, executed like
> following:
> > > >
> > > > httpclient.execute(request, new FutureCallback<HttpResponse>() {
> > > >                         public void failed(Exception arg0) {
> > > >                             //Do nothing
> > > >                         }
> > > >                         public void completed(HttpResponse arg0) {
> > > >                             EntityUtils.consume(arg0.getEntity());
> > > >                         }
> > > >
> > > >                         public void cancelled() {
> > > >                             //Do nothing
> > > >                         }
> > > > });
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message