hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frédéric Passaniti (JIRA) <j...@apache.org>
Subject [jira] [Created] (HTTPCORE-343) ConcurrentModificationException into ProcessPendingRequest when queries are reinjected into the ResponseConsumer#failed event (after a Timeout failure of the original query)
Date Mon, 03 Jun 2013 08:14:19 GMT
Frédéric Passaniti created HTTPCORE-343:
-------------------------------------------

             Summary: ConcurrentModificationException into ProcessPendingRequest when queries
are reinjected into the ResponseConsumer#failed event (after a Timeout failure of the original
query)
                 Key: HTTPCORE-343
                 URL: https://issues.apache.org/jira/browse/HTTPCORE-343
             Project: HttpComponents HttpCore
          Issue Type: Bug
          Components: HttpCore NIO
    Affects Versions: 4.2.3
         Environment: a lot of client thread (32 in my test), a  lot of queries per second
per thread (64 QPS in my test), some servers (8 in my test) and very big queries (servers
respond between 10 to 25 seconds in my test)
            Reporter: Frédéric Passaniti


initial discution : 

https://issues.apache.org/jira/browse/HTTPCORE-164?focusedCommentId=13671852&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13671852




Here are the errors :

SEVERE: shard[shard_v5] target[ng51.exa.com:9810] take[1] resp failed : java.util.concurrent.TimeoutException
java.util.concurrent.TimeoutException
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exalead.elastic_ha.searng51.exa.com : 9810
ch_client_federate.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Unknown Source)
java.util.ConcurrentModificationException
May 31, 2013 5:55:25 PM com.exa.search_client.EHASearchClient getResults
SEVERE: problem with query: [q=expects&b=0&hf=150000] affinity: [null] : java.util.ConcurrentModificationException
java.util.ConcurrentModificationException
at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
at java.util.LinkedList$ListItr.next(Unknown Source)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Unknown Source)
187781 [Thread-14] WARN search.qps 11 - Search failed
May 31, 2013 5:55:25 PM com.exa.search_client.ShardQuery logError
SEVERE: shard[shard_v5] target[ng51.exa.com:9210] take[0] Could not reinject failover query
java.util.ConcurrentModificationException
at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
at java.util.LinkedList$ListItr.next(Unknown Source)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Unknown Source)
java.util.ConcurrentModificationException
May 31, 2013 5:55:25 PM com.exa.search_client.ShardQuery logError
SEVERE: shard[shard_v5] target[ng51.exa.com:9110] take[0] Could not reinject failover query
java.util.ConcurrentModificationException
at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
at java.util.LinkedList$ListItr.next(Unknown Source)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAs ..........




So : 
a concurrent exception is raised while the AbstractNIOConnectionPool is iterating over the
leasingRequest AND at the same time (I think) a thread is trying to remove an element of this
list.
This occurs after a timeout exception.


To reproduce it I used this setup :

32 clients thread, that try to execute a total of 64 QPS (query per second) to 8 servers (but
I think this part is not a problem)
and the MASTER KEY is to have very very long response size !!
If my clients are able to read the answer within less than 1 or 2 seconds the issue doesn't
occurs (or almost never).
If I ask for a very big result (response time between 10 and 25 seconds) it take 1 or 2 minuts
and then explode.

The error occurs when it's spending some times in this call : conn.consumeInput(this.handler);
(DefaultHttpClientIODispatch)


Here is my code and setup :

init :

params_ = new SyncBasicHttpParams();
params_
.setIntParameter(CoreConnectionPNames.SO_TIMEOUT, http_params.so_timeout_)
.setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, http_params.conn_timeout_)
.setIntParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE, http_params.so_buffersize_)
.setParameter(CoreProtocolPNames.USER_AGENT, "search-client");
// <SocketTimeout>120000</SocketTimeout> 
// <ConnectionTimeout>80000</ConnectionTimeout>
// <SocketBufferSize>4096</SocketBufferSize>

final HttpAsyncRequestExecutor protocolHandler = new HttpAsyncRequestExecutor();
final IOEventDispatch ioEventDispatch = new DefaultHttpClientIODispatch(protocolHandler, params_);
ioReactor_ = new DefaultConnectingIOReactor();

pool_ = new BasicNIOConnPool(ioReactor_, params_);
pool_.setDefaultMaxPerRoute(2);
pool_.setMaxTotal(4);

// Run the I/O reactor in a separate thread
reactor_daemon_ = new Thread(new Runnable(){
public void run(){
LOGGER.debug("IOReactor thread started");
try {
ioReactor_.execute(ioEventDispatch);
} catch (final Exception ex) {
LOGGER.error("error in the IOReactor thread: ", ex);
}
LOGGER.debug("IOReactor thread shutdown");
}
});
reactor_daemon_.setDaemon(true);
reactor_daemon_.start();

httpproc_ = new ImmutableHttpProcessor(new HttpRequestInterceptor[] {
new RequestContent(),
new RequestTargetHost(),
new RequestConnControl(),
new RequestUserAgent(),
new RequestExpectContinue()});
requester_ = new HttpAsyncRequester(httpproc_, new DefaultConnectionReuseStrategy(), params_);



query call :


final CountDownLatch latch = new CountDownLatch(queries.length);
for (final ShardQuery query : queries) {
requester_.execute(
new NioHttpRequestProducer(query),
new NioHttpResponseConsumer(query, latch, this),
pool_);
}
latch.await();



Just a particularity (AND IT SEEMS TO BE THE PROBLEM) :

into my NioHttpResponseConsumer, if my query failed (timeout), I invalidate the server, get
the next available one, and try to re-execute the query (by calling again the HttpAsyncRequester.execute


here is the code of the NioHttpResponseConsumer :


	@Override
	public void			failed (final Exception ex)
	{
		if (! close_1()) {
			return;
		}
		query_.logError("resp failed : " + ex, ex);
                ex_ = ex;
		query_.flagCurrentTargetFailure();
		try {
			query_.nextOKTarget(); // get the next server of my list
			client_.sendRequests(latch_, query_); // call the HTTP requester
		} catch (final Throwable inner_th) {
			query_.error_ = inner_th;
			latch_.countDown();
			query_.logError("Could not reinject failover query", inner_th);
		}
	}



 I'm using this version :

  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcomponents-core</artifactId>
    <version>4.2.3</version>
  </parent>
  <artifactId>httpcore-nio</artifactId>
  <name>HttpCore NIO</name>
  <inceptionYear>2005</inceptionYear>
 
thanks for your help

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message