hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: I/O dispatch worker terminated abnormally, what to do?
Date Tue, 31 May 2016 08:37:26 GMT
On Mon, 2016-05-30 at 21:37 +0000, John Price wrote:
> We're using the HttpAsyncClient to make calls from within our own server
> application. Periodically (once a week or less), our service stops being
> able to make requests and we see errors of the form:
> 
> java.lang.IllegalStateException: Request cannot be executed; I/O reactor
> status: STOPPED
> at org.apache.http.util.Asserts.check(Asserts.java:46)
> at
> org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
> at
> org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
> ...
> 
> Digging further into the logs, it seems these begin after we see an
> exception that I've included at the bottom but basically seems to boil down
> to "I/O dispatch worker terminated abnormally" caused by "IOException:
> Connection reset by peer" in the depths of SSLIOSession.
> 
> I guess I've got two questions: first, is this a bug? I'm not sure how much
> additional info I can provide into the root cause of the IOException since
> it happens so infrequently, but I'll try to provide what I can if it's
> relevant.
> 
> Second, how should we be handling this issue? Some of the Apache
> documentation and older mailing list posts talk about "restarting the I/O
> Reactor" but I'm not clear what means in this context since I'm not
> interacting with it directly. Should I just be throwing away my instance of
> HttpAsyncClient and creating a new one?
> 
> We're using:
> httpclient 4.5.2
> httpasyncclient 4.1.1
> httpcore.nio4.4.4
> 
> John.
> 
> org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker
> terminated abnormally
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:358)
> 
> at
> org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
> 
> at
> org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
> 
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: rx.exceptions.OnErrorNotImplementedException: Connection reset
> by peer

The original 'java.io.IOException: Connection reset by peer' exception
was re-thrown as OnErrorNotImplementedException from inside the
FutureCallback#failed method, which killed the i/o dispatch thread and
caused the i/o reactor to shut down.  

This issue needs to be addressed either in your application code or in
the RX library.

Hope this helps

Oleg


> at rx.Observable$27.onError(Observable.java:7923)
> at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:159)
> at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:120)
> at
> rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71)
> at OurCode.SessionManager.lambda$null$10(SessionManager.java:174)
> at rx.Observable$10.onError(Observable.java:4407)
> at
> rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:65)
> at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:240)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.checkTerminate(OperatorMerge.java:776)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:537)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:526)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.onError(OperatorMerge.java:250)
> at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
> at
> rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71)
> at rx.apache.http.ObservableHttp$2$1.failed(ObservableHttp.java:195)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134)
> at
> org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:419)
> 
> at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:152)
> 
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:76)
> 
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:39)
> 
> at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:127)
> 
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164)
> 
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339)
> 
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317)
> 
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278)
> 
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> 
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)
> Caused by: java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_65]
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_65]
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_65]
> at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_65]
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> ~[na:1.8.0_65]
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.receiveEncryptedData(SSLIOSession.java:449)
> ~[antelope-server.jar:na]
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:503)
> ~[antelope-server.jar:na]
> at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:122)
> ~[antelope-server.jar:na]
> ... 7 common frames omitted



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message