hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremy Barlow (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HTTPASYNC-99) Response handling problems when too many open files as request executed
Date Wed, 06 Jan 2016 17:59:39 GMT

     [ https://issues.apache.org/jira/browse/HTTPASYNC-99?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jeremy Barlow updated HTTPASYNC-99:
-----------------------------------
    Attachment: AbnormalIOReactor_catchingbindexceptions.java

[~olegk], I've also uploaded a slightly modified example, AbnormalIOReactor_catchingbindexceptions.java.
 This version just tries to create new sockets going from the first port in the dynamic port
range on up until it fails.  Any java.net.BindExceptions encountered along the way are swallowed.
 Hopefully, this would allow you to see something like this in the output when you run it:

{noformat}
Creating sockets
Bind exception: Address already in use
Bind exception: Address already in use
Socket exception (class java.net.SocketException): Too many open files in system
...
{noformat}

> Response handling problems when too many open files as request executed
> -----------------------------------------------------------------------
>
>                 Key: HTTPASYNC-99
>                 URL: https://issues.apache.org/jira/browse/HTTPASYNC-99
>             Project: HttpComponents HttpAsyncClient
>          Issue Type: Bug
>    Affects Versions: 4.1.1
>            Reporter: Jeremy Barlow
>            Assignee: Oleg Kalnichevski
>         Attachments: AbnormalIOReactor.java, AbnormalIOReactor_catchingbindexceptions.java,
ResponseCallback.java
>
>
> We have encountered a few issues with handling an HttpAsyncClient request which is made
when the process has too many open file descriptors:
> 1) A .get() call made on the future returned from the httpclient's .execute() method
hangs indefinitely.
> 2) No call is made to any of the methods on the {{FutureCallback<HttpResponse>}}
object provided as an argument to the httpclient's .execute() method.  I was expecting a call
to .failed() to occur.
> 3) A call to the httpclient's .close() method does not error out but also does not result
in all of the IOReactor "I/O Dispatcher" threads being shut down.  In normal cases, these
threads would get shut down when the .close() call is made.
> I've attached a couple of Java source files which demonstrate the problem - AbnormalIOReactor.java
and ResponseCallback.java.  AbnormalIOReactor.java is based on the example at https://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/examples/org/apache/http/examples/nio/client/AsyncClientHttpExchange.java.
 Just after the httpclient is started, this class allocates bare sockets repeatedly and holds
onto them.  This is done to force the application to get into the state where file descriptors
have been exhausted.  The httpclient.execute() call does not throw an error.
> The class repeatedly calls .get() on the future response with a prolonged timeout, breaking
out of the loop when the client moves from the state of having .isRunning() be true to .isRunning()
being false.  Checking for a change in the .isRunning() state is a bit contrived here, ideally
something that an API client would not have to do.  I noticed from the implementation of the
{{CloseableHttpAsyncClientBase}} constructor at https://svn.apache.org/repos/asf/httpcomponents/httpasyncclient/tags/4.1.1/httpasyncclient/src/main/java/org/apache/http/impl/nio/client/CloseableHttpAsyncClientBase.java
that the reactor thread's run() method will catch an exception at request startup and set
the client status to {{STOPPED}} and this seemed like the only way that a client could detect
that a problem had occurred in this situation.
> {code:java}
>                 @Override
>                 public void run() {
>                     try {
>                         final IOEventDispatch ioEventDispatch = new InternalIODispatch(handler);
>                         connmgr.execute(ioEventDispatch);
>                     } catch (final Exception ex) {
>                         log.error("I/O reactor terminated abnormally", ex);
>                     } finally {
>                         status.set(Status.STOPPED);
>                     }
>                 }
> {code}
> The implementation of ResponseCallback.java writes a message to stdout when any of the
methods is invoked.  In this case, no message is written to stdout even though the request
has effectively failed.
> When this program is run from the command line like {{java abnormalioreactor.AbnormalIOReactor}},
it will hang after the word "Done" appears, which occurs after the .close() method is called
on the httpclient.  When running jstack from the command line, I see that various "I/O Dispatcher"
threads are still running for the client even though I had expected them to have been closed:
> {noformat}
> "I/O dispatcher 8" #19 prio=5 os_prio=31 tid=0x00007f978e800000 nid=0x6503 runnable [0x0000000120cf6000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>         at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
>         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x000000076ba79bf8> (a sun.nio.ch.Util$2)
>         - locked <0x000000076ba79be8> (a java.util.Collections$UnmodifiableSet)
>         - locked <0x000000076ba79ac8> (a sun.nio.ch.KQueueSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:257)
>         at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>         at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)
>         at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I had been thinking that in this situation it would be better for the httpclient to not
be implicitly stopped - so it could be used for subsequent requests if/when the number of
open file descriptors has dropped.  Also, it would be good for either the httpclient.execute()
call to throw an exception right away or at least for calls to .get() on the future returned
from the .execute() call to throw an exception and for the .failed() method on {{FutureCallback<HttpResponse>}}
object to be called.
> Note that I ran this application on my MacBook Pro, running OS-X Yosemite.  We've seen
this problem on other OSes as well, though, including CentOS 7.
> I reproduced this problem with a few different versions of HttpAsyncClient - 4.0.2, 4.1.1,
custom build from trunk, custom build from the 4.1.x branch. 
> This isn't a situation that we run into often with our applications - mostly in cases
where users haven't tuned their application settings properly to account for the number of
file descriptors that a Java process would need to use.   It would be nice if the HttpAsyncClient
handling could gracefully fail a request when open file descriptors have temporarily been
exhausted under load spikes but leave the client in a usable state for future requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message