Mailing-List: contact dev-help@hc.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpComponents Project" <dev@hc.apache.org>
Message-ID: <2051105862.1214575605064.JavaMail.jira@brutus>
Date: Fri, 27 Jun 2008 07:06:45 -0700 (PDT)
From: "Denis Rogov (JIRA)" <jira@apache.org>
To: dev@hc.apache.org
Subject: [jira] Commented: (HTTPCORE-164) DefaultConnectingIOReactor throws
 ConcurrentModificationException in processEvents() method
In-Reply-To: <930211512.1214558865119.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HTTPCORE-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608775#action_12608775 ] 

Denis Rogov commented on HTTPCORE-164:
--------------------------------------

Oleg, at this moment I still found no test case to reproduce this. The thing is that it is being reproduced while server is running for a few hours. 

But now I have few updates on this problem. First of all, this is not a root cause.

Here is the chronology of the incident:
1.  For some reasons not driven from application DefaultConnectingIOReactor#execute(IOEventDispatch eventDispatch); method exits without throwing IOException or any other exception. Code like this produced log output like this:
                try {
                    ioReactor.execute(ioEventDispatch);
                } catch (InterruptedIOException ex) {
                    getLog().info("IO reactor execution has been interrupted.");
                } catch (IOException e) {
                    getLog().error("IO exception occured.", e);
                }
                getLog().info("IO reactor execution has been finished.");

10:44:34,648 INFO  [nioEventListener] connectionOpen
10:44:34,651 INFO  [httpManagerAsyncEngine] IO reactor execution has been finished.
10:44:34,656 INFO  [nioEventListener] [closed] connectionClosed


After that we got few other exceptions from the httpcomponenets code, one to exception handler, next in the thread that started ioReactor.execute(ioEventDispatch); (named "IO Reactor Execution Thread")

10:44:34,669 ERROR [httpManagerAsyncEngine] RuntimeException occured while processing request
java.nio.channels.CancelledKeyException
        at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
        at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
        at org.apache.http.impl.nio.reactor.IOSessionImpl.setEvent(IOSessionImpl.java:111)
        at org.apache.http.impl.nio.DefaultNHttpClientConnection.submitRequest(DefaultNHttpClientConnection.java:214)
        at org.apache.http.nio.protocol.AsyncNHttpClientHandler.requestReady(AsyncNHttpClientHandler.java:183)
        at org.apache.http.nio.protocol.AsyncNHttpClientHandler.connected(AsyncNHttpClientHandler.java:111)
        at org.apache.http.nio.protocol.BufferingHttpClientHandler.connected(BufferingHttpClientHandler.java:94)
        at org.apache.http.impl.nio.DefaultClientIOEventDispatch.connected(DefaultClientIOEventDispatch.java:87)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.keyCreated(BaseIOReactor.java:182)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processNewChannels(AbstractIOReactor.java:246)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:153)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:70)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:321)
        at java.lang.Thread.run(Thread.java:619)
10:44:34,684 INFO  [nioEventListener] [closed] connectionClosed
10:44:34,686 INFO  [nioEventListener] [closed] connectionClosed
10:44:34,687 INFO  [nioEventListener] [closed] connectionClosed
10:44:34,689 INFO  [nioEventListener] [closed] connectionClosed
Exception in thread "IO Reactor Execution Thread" java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
        at java.util.HashMap$KeyIterator.next(HashMap.java:827)
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:93)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:162)
        at <>HttpManagerApacheAsyncEngineImpl$1.run(HttpManagerApacheAsyncEngineImpl.java:282)
        at java.lang.Thread.run(Thread.java:619)

The other important piece of code for this issue is reactor initializer block:
ioReactor = new DefaultConnectingIOReactor(3, httpParams);
((DefaultConnectingIOReactor) ioReactor).setExceptionHandler(new IOReactorExceptionHandler() {
            public boolean handle(IOException ex) {
                getLog().error("IOException occured while processing request", ex);
                // in any case we should preserve ioreactor alive
                return true;
            }

            public boolean handle(RuntimeException ex) {
                getLog().error("RuntimeException occured while processing request", ex);
                // in any case we should preserve ioreactor alive
                return true;
            }

As a result, the worst thing is that we got the reactor thread dead, which actually blocked all further application work. Do you have ideas why it could happen? And the most important question is how do you suggest to catch the cases when reactor or one of its workers crashed? How to resurrect it to go on with serving requests? Once it became in state shut down SHUT_DOWN I invoked execute method once again, but seems it is not a right way because it didn't got the control and returned from the method w/o exception immdiately.

Regards,
Denis
        });

> DefaultConnectingIOReactor throws ConcurrentModificationException in processEvents() method
> -------------------------------------------------------------------------------------------
>
>                 Key: HTTPCORE-164
>                 URL: https://issues.apache.org/jira/browse/HTTPCORE-164
>             Project: HttpComponents HttpCore
>          Issue Type: Bug
>          Components: HttpCore NIO
>    Affects Versions: 4.0-beta2
>         Environment: Win XP
> JDK 1.6.0-b105
> httpcore-nio-beta2-20080510.140437-10.jar
> httpcore-beta2-20080510.140437-10.jar
>            Reporter: Denis Rogov
>            Priority: Critical
>
> Using httpcomponents nio to serve as http client. 
> Occasionally getting exception that stops reactor and therefore all engine work:
> Exception in thread "IO Reactor Execution Thread" java.util.ConcurrentModificationException
> 	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> 	at java.util.HashMap$KeyIterator.next(HashMap.java:827)
> 	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:93)
> 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:162)
> 	at <...>HttpManagerApacheAsyncEngineImpl$1.run(HttpManagerApacheAsyncEngineImpl.java:294)
> 	at java.lang.Thread.run(Thread.java:619)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org