hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremy Barlow (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HTTPASYNC-99) Response handling problems when too many open files as request executed
Date Mon, 04 Jan 2016 16:26:39 GMT
Jeremy Barlow created HTTPASYNC-99:
--------------------------------------

             Summary: Response handling problems when too many open files as request executed
                 Key: HTTPASYNC-99
                 URL: https://issues.apache.org/jira/browse/HTTPASYNC-99
             Project: HttpComponents HttpAsyncClient
          Issue Type: Bug
    Affects Versions: 4.1.1
            Reporter: Jeremy Barlow


We have encountered a few issues with handling an HttpAsyncClient request which is made when
the process has too many open file descriptors:

1) A .get() call made on the future returned from the httpclient's .execute() method hangs
indefinitely.

2) No call is made to any of the methods on the {{FutureCallback<HttpResponse>}} object
provided as an argument to the httpclient's .execute() method.  I was expecting a call to
.failed() to occur.

3) A call to the httpclient's .close() method does not error out but also does not result
in all of the IOReactor "I/O Dispatcher" threads being shut down.  In normal cases, these
threads would get shut down when the .close() call is made.

I've attached a couple of Java source files which demonstrate the problem - AbnormalIOReactor.java
and ResponseCallback.java.  AbnormalIOReactor.java is based on the example at https://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/examples/org/apache/http/examples/nio/client/AsyncClientHttpExchange.java.
 Just after the httpclient is started, this class allocates bare sockets repeatedly and holds
onto them.  This is done to force the application to get into the state where file descriptors
have been exhausted.  The httpclient.execute() call does not throw an error.

The class repeatedly calls .get() on the future response with a prolonged timeout, breaking
out of the loop when the client moves from the state of having .isRunning() be true to .isRunning()
being false.  Checking for a change in the .isRunning() state is a bit contrived here, ideally
something that an API client would not have to do.  I noticed from the implementation of the
{{CloseableHttpAsyncClientBase}} constructor at https://svn.apache.org/repos/asf/httpcomponents/httpasyncclient/tags/4.1.1/httpasyncclient/src/main/java/org/apache/http/impl/nio/client/CloseableHttpAsyncClientBase.java
that the reactor thread's run() method will catch an exception at request startup and set
the client status to {{STOPPED}} and this seemed like the only way that a client could detect
that a problem had occurred in this situation.

{code:java}
                @Override
                public void run() {
                    try {
                        final IOEventDispatch ioEventDispatch = new InternalIODispatch(handler);
                        connmgr.execute(ioEventDispatch);
                    } catch (final Exception ex) {
                        log.error("I/O reactor terminated abnormally", ex);
                    } finally {
                        status.set(Status.STOPPED);
                    }
                }
{code}

The implementation of ResponseCallback.java writes a message to stdout when any of the methods
is invoked.  In this case, no message is written to stdout even though the request has effectively
failed.

When this program is run from the command line like {{java abnormalioreactor.AbnormalIOReactor}},
it will hang after the word "Done" appears, which occurs after the .close() method is called
on the httpclient.  When running jstack from the command line, I see that various "I/O Dispatcher"
threads are still running for the client even though I had expected them to have been closed:

{noformat}
"I/O dispatcher 8" #19 prio=5 os_prio=31 tid=0x00007f978e800000 nid=0x6503 runnable [0x0000000120cf6000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x000000076ba79bf8> (a sun.nio.ch.Util$2)
        - locked <0x000000076ba79be8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x000000076ba79ac8> (a sun.nio.ch.KQueueSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:257)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)
        at java.lang.Thread.run(Thread.java:745)
{noformat}

I had been thinking that in this situation it would be better for the httpclient to not be
implicitly stopped - so it could be used for subsequent requests if/when the number of open
file descriptors has dropped.  Also, it would be good for either the httpclient.execute()
call to throw an exception right away or at least for calls to .get() on the future returned
from the .execute() call to throw an exception and for the .failed() method on {{FutureCallback<HttpResponse>}}
object to be called.

Note that I ran this application on my MacBook Pro, running OS-X Yosemite.  We've seen this
problem on other OSes as well, though, including CentOS 7.

I reproduced this problem with a few different versions of HttpAsyncClient - 4.0.2, 4.1.1,
custom build from trunk, custom build from the 4.1.x branch. 

This isn't a situation that we run into often with our applications - mostly in cases where
users haven't tuned their application settings properly to account for the number of file
descriptors that a Java process would need to use.   It would be nice if the HttpAsyncClient
handling could gracefully fail a request when open file descriptors have temporarily been
exhausted under load spikes but leave the client in a usable state for future requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message