hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frédéric Passaniti (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HTTPCORE-164) DefaultConnectingIOReactor throws ConcurrentModificationException in processEvents() method
Date Fri, 31 May 2013 16:48:20 GMT

    [ https://issues.apache.org/jira/browse/HTTPCORE-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671635#comment-13671635
] 

Frédéric Passaniti commented on HTTPCORE-164:
---------------------------------------------

Hello everyone, I think 4 years and a half later.... I reproduced the bug....
or maybe I've something very very similar.


Here are the errors :

SEVERE: shard[shard_v5] target[ng51.exa.com:9810] take[1] resp failed : java.util.concurrent.TimeoutException
java.util.concurrent.TimeoutException
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exalead.elastic_ha.searng51.exa.com : 9810
ch_client_federate.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
	at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
	at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
	at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
	at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
	at java.lang.Thread.run(Unknown Source)
java.util.ConcurrentModificationException
May 31, 2013 5:55:25 PM com.exa.search_client.EHASearchClient getResults
SEVERE: problem with query: [q=expects&b=0&hf=150000] affinity: [null] : java.util.ConcurrentModificationException
java.util.ConcurrentModificationException
	at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
	at java.util.LinkedList$ListItr.next(Unknown Source)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
	at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
	at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
	at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
	at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
	at java.lang.Thread.run(Unknown Source)
187781 [Thread-14] WARN search.qps 11 - Search failed
May 31, 2013 5:55:25 PM com.exa.search_client.ShardQuery logError
SEVERE: shard[shard_v5] target[ng51.exa.com:9210] take[0] Could not reinject failover query
java.util.ConcurrentModificationException
	at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
	at java.util.LinkedList$ListItr.next(Unknown Source)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
	at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
	at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
	at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
	at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
	at java.lang.Thread.run(Unknown Source)
java.util.ConcurrentModificationException
May 31, 2013 5:55:25 PM com.exa.search_client.ShardQuery logError
SEVERE: shard[shard_v5] target[ng51.exa.com:9110] take[0] Could not reinject failover query
java.util.ConcurrentModificationException
	at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
	at java.util.LinkedList$ListItr.next(Unknown Source)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
	at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
	at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
	at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
	at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
	at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
	at org.apache.http.nio.protocol.HttpAs  ..........




So : 
a concurrent exception is raised while the AbstractNIOConnectionPool is iterating over the
leasingRequest AND at the same time (I think) a thread is trying to remove an element of this
list.
This occurs after a timeout exception.


To reproduce it I used this setup :

32 clients thread, that try to execute a total of 64 QPS (query per second) to 8 servers (but
I think this part is not a problem)
and the MASTER KEY is to have very very long response size !!
If my clients are able to read the answer within less than 1 or 2 seconds the issue doesn't
occurs (or almost never).
If I ask for a very big result (response time between 10 and 25 seconds) it take 1 or 2 minuts
and then explode.

The error occurs when it's spending some times in this call : conn.consumeInput(this.handler);
(DefaultHttpClientIODispatch)


Here is my code and setup :

init :

		params_ = new SyncBasicHttpParams();
		params_
		.setIntParameter(CoreConnectionPNames.SO_TIMEOUT, http_params.so_timeout_)
		.setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, http_params.conn_timeout_)
		.setIntParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE, http_params.so_buffersize_)
		.setParameter(CoreProtocolPNames.USER_AGENT, "search-client");
		// <SocketTimeout>120000</SocketTimeout> 
		// <ConnectionTimeout>80000</ConnectionTimeout>
		// <SocketBufferSize>4096</SocketBufferSize>

		final HttpAsyncRequestExecutor protocolHandler = new HttpAsyncRequestExecutor();
		final IOEventDispatch ioEventDispatch = new DefaultHttpClientIODispatch(protocolHandler,
params_);
		ioReactor_ = new DefaultConnectingIOReactor();
		
		pool_ = new BasicNIOConnPool(ioReactor_, params_);
		pool_.setDefaultMaxPerRoute(2);
		pool_.setMaxTotal(4);
		
		// Run the I/O reactor in a separate thread
		reactor_daemon_ = new Thread(new Runnable(){
			public void run(){
				LOGGER.debug("IOReactor thread started");
				try {
					ioReactor_.execute(ioEventDispatch);
				} catch (final Exception ex) {
					LOGGER.error("error in the IOReactor thread: ", ex);
				}
				LOGGER.debug("IOReactor thread shutdown");
			}
		});
		reactor_daemon_.setDaemon(true);
		reactor_daemon_.start();

		httpproc_ = new ImmutableHttpProcessor(new HttpRequestInterceptor[] {
				new RequestContent(),
				new RequestTargetHost(),
				new RequestConnControl(),
				new RequestUserAgent(),
				new RequestExpectContinue()});
		requester_ = new HttpAsyncRequester(httpproc_, new DefaultConnectionReuseStrategy(), params_);

		
		
query call :


	final CountDownLatch latch = new CountDownLatch(queries.length);
	for (final ShardQuery query : queries) {
		requester_.execute(
				new NioHttpRequestProducer(query),
				new NioHttpResponseConsumer(query, latch, this),
					pool_);
	}
	latch.await();
	


Just a particularity :

into my NioHttpResponseConsumer, if my query failed, I invalidate the server, get the next
available one, and try to re-execute the query (by calling again the HttpAsyncRequester.execute

	
		
		
		

What am I doing wrong doctor ? thanks by advance
                
> DefaultConnectingIOReactor throws ConcurrentModificationException in processEvents()
method
> -------------------------------------------------------------------------------------------
>
>                 Key: HTTPCORE-164
>                 URL: https://issues.apache.org/jira/browse/HTTPCORE-164
>             Project: HttpComponents HttpCore
>          Issue Type: Bug
>          Components: HttpCore NIO
>    Affects Versions: 4.0-beta2
>         Environment: Win XP
> JDK 1.6.0-b105
> httpcore-nio-beta2-20080510.140437-10.jar
> httpcore-beta2-20080510.140437-10.jar
>            Reporter: Denis Rogov
>            Priority: Critical
>
> Using httpcomponents nio to serve as http client. 
> Occasionally getting exception that stops reactor and therefore all engine work:
> Exception in thread "IO Reactor Execution Thread" java.util.ConcurrentModificationException
> 	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> 	at java.util.HashMap$KeyIterator.next(HashMap.java:827)
> 	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:93)
> 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:162)
> 	at <...>HttpManagerApacheAsyncEngineImpl$1.run(HttpManagerApacheAsyncEngineImpl.java:294)
> 	at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message