hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frédéric Passaniti (JIRA) <j...@apache.org>
Subject [jira] [Updated] (HTTPCORE-343) ConcurrentModificationException into ProcessPendingRequest when queries are reinjected into the ResponseConsumer#failed event (after a Timeout failure of the original query)
Date Mon, 03 Jun 2013 08:16:22 GMT

     [ https://issues.apache.org/jira/browse/HTTPCORE-343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Frédéric Passaniti updated HTTPCORE-343:
----------------------------------------

    Description: 
initial discussion : 

https://issues.apache.org/jira/browse/HTTPCORE-164?focusedCommentId=13671852&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13671852




Here are the errors :

SEVERE: shard[shard_v5] target[ng51.exa.com:9810] take[1] resp failed : java.util.concurrent.TimeoutException
java.util.concurrent.TimeoutException
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exalead.elastic_ha.searng51.exa.com : 9810
ch_client_federate.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Unknown Source)
java.util.ConcurrentModificationException
May 31, 2013 5:55:25 PM com.exa.search_client.EHASearchClient getResults
SEVERE: problem with query: [q=expects&b=0&hf=150000] affinity: [null] : java.util.ConcurrentModificationException
java.util.ConcurrentModificationException
at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
at java.util.LinkedList$ListItr.next(Unknown Source)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Unknown Source)
187781 [Thread-14] WARN search.qps 11 - Search failed
May 31, 2013 5:55:25 PM com.exa.search_client.ShardQuery logError
SEVERE: shard[shard_v5] target[ng51.exa.com:9210] take[0] Could not reinject failover query
java.util.ConcurrentModificationException
at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
at java.util.LinkedList$ListItr.next(Unknown Source)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Unknown Source)
java.util.ConcurrentModificationException
May 31, 2013 5:55:25 PM com.exa.search_client.ShardQuery logError
SEVERE: shard[shard_v5] target[ng51.exa.com:9110] take[0] Could not reinject failover query
java.util.ConcurrentModificationException
at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
at java.util.LinkedList$ListItr.next(Unknown Source)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAs ..........




So : 
a concurrent exception is raised while the AbstractNIOConnectionPool is iterating over the leasingRequest AND at the same time (I think) a thread is trying to remove an element of this list.
This occurs after a timeout exception.


To reproduce it I used this setup :

32 clients thread, that try to execute a total of 64 QPS (query per second) to 8 servers (but I think this part is not a problem)
and the MASTER KEY is to have very very long response size !!
If my clients are able to read the answer within less than 1 or 2 seconds the issue doesn't occurs (or almost never).
If I ask for a very big result (response time between 10 and 25 seconds) it take 1 or 2 minuts and then explode.

The error occurs when it's spending some times in this call : conn.consumeInput(this.handler); (DefaultHttpClientIODispatch)


Here is my code and setup :

init :

params_ = new SyncBasicHttpParams();
params_
.setIntParameter(CoreConnectionPNames.SO_TIMEOUT, http_params.so_timeout_)
.setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, http_params.conn_timeout_)
.setIntParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE, http_params.so_buffersize_)
.setParameter(CoreProtocolPNames.USER_AGENT, "search-client");
// <SocketTimeout>120000</SocketTimeout> 
// <ConnectionTimeout>80000</ConnectionTimeout>
// <SocketBufferSize>4096</SocketBufferSize>

final HttpAsyncRequestExecutor protocolHandler = new HttpAsyncRequestExecutor();
final IOEventDispatch ioEventDispatch = new DefaultHttpClientIODispatch(protocolHandler, params_);
ioReactor_ = new DefaultConnectingIOReactor();

pool_ = new BasicNIOConnPool(ioReactor_, params_);
pool_.setDefaultMaxPerRoute(2);
pool_.setMaxTotal(4);

// Run the I/O reactor in a separate thread
reactor_daemon_ = new Thread(new Runnable(){
public void run(){
LOGGER.debug("IOReactor thread started");
try {
ioReactor_.execute(ioEventDispatch);
} catch (final Exception ex) {
LOGGER.error("error in the IOReactor thread: ", ex);
}
LOGGER.debug("IOReactor thread shutdown");
}
});
reactor_daemon_.setDaemon(true);
reactor_daemon_.start();

httpproc_ = new ImmutableHttpProcessor(new HttpRequestInterceptor[] {
new RequestContent(),
new RequestTargetHost(),
new RequestConnControl(),
new RequestUserAgent(),
new RequestExpectContinue()});
requester_ = new HttpAsyncRequester(httpproc_, new DefaultConnectionReuseStrategy(), params_);



query call :


final CountDownLatch latch = new CountDownLatch(queries.length);
for (final ShardQuery query : queries) {
requester_.execute(
new NioHttpRequestProducer(query),
new NioHttpResponseConsumer(query, latch, this),
pool_);
}
latch.await();



Just a particularity (AND IT SEEMS TO BE THE PROBLEM) :

into my NioHttpResponseConsumer, if my query failed (timeout), I invalidate the server, get the next available one, and try to re-execute the query (by calling again the HttpAsyncRequester.execute	

here is the code of the NioHttpResponseConsumer :


	@Override
	public void			failed (final Exception ex)
	{
		if (! close_1()) {
			return;
		}
		query_.logError("resp failed : " + ex, ex);
                ex_ = ex;
		query_.flagCurrentTargetFailure();
		try {
			query_.nextOKTarget(); // get the next server of my list
			client_.sendRequests(latch_, query_); // call the HTTP requester
		} catch (final Throwable inner_th) {
			query_.error_ = inner_th;
			latch_.countDown();
			query_.logError("Could not reinject failover query", inner_th);
		}
	}



 I'm using this version :

  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcomponents-core</artifactId>
    <version>4.2.3</version>
  </parent>
  <artifactId>httpcore-nio</artifactId>
  <name>HttpCore NIO</name>
  <inceptionYear>2005</inceptionYear>
 
thanks for your help

  was:
initial discution : 

https://issues.apache.org/jira/browse/HTTPCORE-164?focusedCommentId=13671852&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13671852




Here are the errors :

SEVERE: shard[shard_v5] target[ng51.exa.com:9810] take[1] resp failed : java.util.concurrent.TimeoutException
java.util.concurrent.TimeoutException
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exalead.elastic_ha.searng51.exa.com : 9810
ch_client_federate.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Unknown Source)
java.util.ConcurrentModificationException
May 31, 2013 5:55:25 PM com.exa.search_client.EHASearchClient getResults
SEVERE: problem with query: [q=expects&b=0&hf=150000] affinity: [null] : java.util.ConcurrentModificationException
java.util.ConcurrentModificationException
at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
at java.util.LinkedList$ListItr.next(Unknown Source)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Unknown Source)
187781 [Thread-14] WARN search.qps 11 - Search failed
May 31, 2013 5:55:25 PM com.exa.search_client.ShardQuery logError
SEVERE: shard[shard_v5] target[ng51.exa.com:9210] take[0] Could not reinject failover query
java.util.ConcurrentModificationException
at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
at java.util.LinkedList$ListItr.next(Unknown Source)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Unknown Source)
java.util.ConcurrentModificationException
May 31, 2013 5:55:25 PM com.exa.search_client.ShardQuery logError
SEVERE: shard[shard_v5] target[ng51.exa.com:9110] take[0] Could not reinject failover query
java.util.ConcurrentModificationException
at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
at java.util.LinkedList$ListItr.next(Unknown Source)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
at org.apache.http.nio.protocol.HttpAs ..........




So : 
a concurrent exception is raised while the AbstractNIOConnectionPool is iterating over the leasingRequest AND at the same time (I think) a thread is trying to remove an element of this list.
This occurs after a timeout exception.


To reproduce it I used this setup :

32 clients thread, that try to execute a total of 64 QPS (query per second) to 8 servers (but I think this part is not a problem)
and the MASTER KEY is to have very very long response size !!
If my clients are able to read the answer within less than 1 or 2 seconds the issue doesn't occurs (or almost never).
If I ask for a very big result (response time between 10 and 25 seconds) it take 1 or 2 minuts and then explode.

The error occurs when it's spending some times in this call : conn.consumeInput(this.handler); (DefaultHttpClientIODispatch)


Here is my code and setup :

init :

params_ = new SyncBasicHttpParams();
params_
.setIntParameter(CoreConnectionPNames.SO_TIMEOUT, http_params.so_timeout_)
.setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, http_params.conn_timeout_)
.setIntParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE, http_params.so_buffersize_)
.setParameter(CoreProtocolPNames.USER_AGENT, "search-client");
// <SocketTimeout>120000</SocketTimeout> 
// <ConnectionTimeout>80000</ConnectionTimeout>
// <SocketBufferSize>4096</SocketBufferSize>

final HttpAsyncRequestExecutor protocolHandler = new HttpAsyncRequestExecutor();
final IOEventDispatch ioEventDispatch = new DefaultHttpClientIODispatch(protocolHandler, params_);
ioReactor_ = new DefaultConnectingIOReactor();

pool_ = new BasicNIOConnPool(ioReactor_, params_);
pool_.setDefaultMaxPerRoute(2);
pool_.setMaxTotal(4);

// Run the I/O reactor in a separate thread
reactor_daemon_ = new Thread(new Runnable(){
public void run(){
LOGGER.debug("IOReactor thread started");
try {
ioReactor_.execute(ioEventDispatch);
} catch (final Exception ex) {
LOGGER.error("error in the IOReactor thread: ", ex);
}
LOGGER.debug("IOReactor thread shutdown");
}
});
reactor_daemon_.setDaemon(true);
reactor_daemon_.start();

httpproc_ = new ImmutableHttpProcessor(new HttpRequestInterceptor[] {
new RequestContent(),
new RequestTargetHost(),
new RequestConnControl(),
new RequestUserAgent(),
new RequestExpectContinue()});
requester_ = new HttpAsyncRequester(httpproc_, new DefaultConnectionReuseStrategy(), params_);



query call :


final CountDownLatch latch = new CountDownLatch(queries.length);
for (final ShardQuery query : queries) {
requester_.execute(
new NioHttpRequestProducer(query),
new NioHttpResponseConsumer(query, latch, this),
pool_);
}
latch.await();



Just a particularity (AND IT SEEMS TO BE THE PROBLEM) :

into my NioHttpResponseConsumer, if my query failed (timeout), I invalidate the server, get the next available one, and try to re-execute the query (by calling again the HttpAsyncRequester.execute	

here is the code of the NioHttpResponseConsumer :


	@Override
	public void			failed (final Exception ex)
	{
		if (! close_1()) {
			return;
		}
		query_.logError("resp failed : " + ex, ex);
                ex_ = ex;
		query_.flagCurrentTargetFailure();
		try {
			query_.nextOKTarget(); // get the next server of my list
			client_.sendRequests(latch_, query_); // call the HTTP requester
		} catch (final Throwable inner_th) {
			query_.error_ = inner_th;
			latch_.countDown();
			query_.logError("Could not reinject failover query", inner_th);
		}
	}



 I'm using this version :

  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcomponents-core</artifactId>
    <version>4.2.3</version>
  </parent>
  <artifactId>httpcore-nio</artifactId>
  <name>HttpCore NIO</name>
  <inceptionYear>2005</inceptionYear>
 
thanks for your help

    
> ConcurrentModificationException into ProcessPendingRequest when queries are reinjected into the ResponseConsumer#failed event (after a Timeout failure of the original query)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HTTPCORE-343
>                 URL: https://issues.apache.org/jira/browse/HTTPCORE-343
>             Project: HttpComponents HttpCore
>          Issue Type: Bug
>          Components: HttpCore NIO
>    Affects Versions: 4.2.3
>         Environment: a lot of client thread (32 in my test), a  lot of queries per second per thread (64 QPS in my test), some servers (8 in my test) and very big queries (servers respond between 10 to 25 seconds in my test)
>            Reporter: Frédéric Passaniti
>
> initial discussion : 
> https://issues.apache.org/jira/browse/HTTPCORE-164?focusedCommentId=13671852&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13671852
> Here are the errors :
> SEVERE: shard[shard_v5] target[ng51.exa.com:9810] take[1] resp failed : java.util.concurrent.TimeoutException
> java.util.concurrent.TimeoutException
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exalead.elastic_ha.searng51.exa.com : 9810
> ch_client_federate.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
> at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
> at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
> at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
> at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
> at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
> at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
> at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> at java.lang.Thread.run(Unknown Source)
> java.util.ConcurrentModificationException
> May 31, 2013 5:55:25 PM com.exa.search_client.EHASearchClient getResults
> SEVERE: problem with query: [q=expects&b=0&hf=150000] affinity: [null] : java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
> at java.util.LinkedList$ListItr.next(Unknown Source)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
> at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
> at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
> at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
> at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
> at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
> at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
> at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> at java.lang.Thread.run(Unknown Source)
> 187781 [Thread-14] WARN search.qps 11 - Search failed
> May 31, 2013 5:55:25 PM com.exa.search_client.ShardQuery logError
> SEVERE: shard[shard_v5] target[ng51.exa.com:9210] take[0] Could not reinject failover query
> java.util.ConcurrentModificationException
> at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
> at java.util.LinkedList$ListItr.next(Unknown Source)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
> at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
> at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
> at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
> at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
> at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
> at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
> at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> at java.lang.Thread.run(Unknown Source)
> java.util.ConcurrentModificationException
> May 31, 2013 5:55:25 PM com.exa.search_client.ShardQuery logError
> SEVERE: shard[shard_v5] target[ng51.exa.com:9110] take[0] Could not reinject failover query
> java.util.ConcurrentModificationException
> at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
> at java.util.LinkedList$ListItr.next(Unknown Source)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAs ..........
> So : 
> a concurrent exception is raised while the AbstractNIOConnectionPool is iterating over the leasingRequest AND at the same time (I think) a thread is trying to remove an element of this list.
> This occurs after a timeout exception.
> To reproduce it I used this setup :
> 32 clients thread, that try to execute a total of 64 QPS (query per second) to 8 servers (but I think this part is not a problem)
> and the MASTER KEY is to have very very long response size !!
> If my clients are able to read the answer within less than 1 or 2 seconds the issue doesn't occurs (or almost never).
> If I ask for a very big result (response time between 10 and 25 seconds) it take 1 or 2 minuts and then explode.
> The error occurs when it's spending some times in this call : conn.consumeInput(this.handler); (DefaultHttpClientIODispatch)
> Here is my code and setup :
> init :
> params_ = new SyncBasicHttpParams();
> params_
> .setIntParameter(CoreConnectionPNames.SO_TIMEOUT, http_params.so_timeout_)
> .setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, http_params.conn_timeout_)
> .setIntParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE, http_params.so_buffersize_)
> .setParameter(CoreProtocolPNames.USER_AGENT, "search-client");
> // <SocketTimeout>120000</SocketTimeout> 
> // <ConnectionTimeout>80000</ConnectionTimeout>
> // <SocketBufferSize>4096</SocketBufferSize>
> final HttpAsyncRequestExecutor protocolHandler = new HttpAsyncRequestExecutor();
> final IOEventDispatch ioEventDispatch = new DefaultHttpClientIODispatch(protocolHandler, params_);
> ioReactor_ = new DefaultConnectingIOReactor();
> pool_ = new BasicNIOConnPool(ioReactor_, params_);
> pool_.setDefaultMaxPerRoute(2);
> pool_.setMaxTotal(4);
> // Run the I/O reactor in a separate thread
> reactor_daemon_ = new Thread(new Runnable(){
> public void run(){
> LOGGER.debug("IOReactor thread started");
> try {
> ioReactor_.execute(ioEventDispatch);
> } catch (final Exception ex) {
> LOGGER.error("error in the IOReactor thread: ", ex);
> }
> LOGGER.debug("IOReactor thread shutdown");
> }
> });
> reactor_daemon_.setDaemon(true);
> reactor_daemon_.start();
> httpproc_ = new ImmutableHttpProcessor(new HttpRequestInterceptor[] {
> new RequestContent(),
> new RequestTargetHost(),
> new RequestConnControl(),
> new RequestUserAgent(),
> new RequestExpectContinue()});
> requester_ = new HttpAsyncRequester(httpproc_, new DefaultConnectionReuseStrategy(), params_);
> query call :
> final CountDownLatch latch = new CountDownLatch(queries.length);
> for (final ShardQuery query : queries) {
> requester_.execute(
> new NioHttpRequestProducer(query),
> new NioHttpResponseConsumer(query, latch, this),
> pool_);
> }
> latch.await();
> Just a particularity (AND IT SEEMS TO BE THE PROBLEM) :
> into my NioHttpResponseConsumer, if my query failed (timeout), I invalidate the server, get the next available one, and try to re-execute the query (by calling again the HttpAsyncRequester.execute	
> here is the code of the NioHttpResponseConsumer :
> 	@Override
> 	public void			failed (final Exception ex)
> 	{
> 		if (! close_1()) {
> 			return;
> 		}
> 		query_.logError("resp failed : " + ex, ex);
>                 ex_ = ex;
> 		query_.flagCurrentTargetFailure();
> 		try {
> 			query_.nextOKTarget(); // get the next server of my list
> 			client_.sendRequests(latch_, query_); // call the HTTP requester
> 		} catch (final Throwable inner_th) {
> 			query_.error_ = inner_th;
> 			latch_.countDown();
> 			query_.logError("Could not reinject failover query", inner_th);
> 		}
> 	}
>  I'm using this version :
>   <modelVersion>4.0.0</modelVersion>
>   <parent>
>     <groupId>org.apache.httpcomponents</groupId>
>     <artifactId>httpcomponents-core</artifactId>
>     <version>4.2.3</version>
>   </parent>
>   <artifactId>httpcore-nio</artifactId>
>   <name>HttpCore NIO</name>
>   <inceptionYear>2005</inceptionYear>
>  
> thanks for your help

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message