lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: Build failed in Jenkins: slow-io-beasting #5020
Date Mon, 29 Oct 2012 00:05:47 GMT
This is happening right at the start of the test - so all the nodes are 
not showing up as ready as part of the setup - before the test even starts.

I've added some test code to print the zk tree and all the current 
thread stack traces when this times out.

- Mark

On 10/28/2012 07:17 PM, Charlie Cron wrote:
> See <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/5020/>
>
> ------------------------------------------
> [...truncated 14006 lines...]
> [junit4:junit4]   2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket connection
for client /127.0.0.1:58646 which had sessionid 0x13aa99d68d00002
> [junit4:junit4]   2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket connection
for client /127.0.0.1:58708 which had sessionid 0x13aa99d68d00004
> [junit4:junit4]   2> 690757 T303 oaz.ClientCnxn$SendThread.run Unable to read additional
data from server sessionid 0x13aa99d68d00002, likely server has closed socket, closing socket
connection and attempting reconnect
> [junit4:junit4]   2> 690757 T331 oaz.ClientCnxn$SendThread.run Unable to read additional
data from server sessionid 0x13aa99d68d00004, likely server has closed socket, closing socket
connection and attempting reconnect
> [junit4:junit4]   2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket connection
for client /127.0.0.1:59242 which had sessionid 0x13aa99d68d00006
> [junit4:junit4]   2> 690757 T355 oaz.ClientCnxn$SendThread.run Unable to read additional
data from server sessionid 0x13aa99d68d00006, likely server has closed socket, closing socket
connection and attempting reconnect
> [junit4:junit4]   2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket connection
for client /127.0.0.1:58739 which had sessionid 0x13aa99d68d00005
> [junit4:junit4]   2> 690757 T343 oaz.ClientCnxn$SendThread.run Unable to read additional
data from server sessionid 0x13aa99d68d00005, likely server has closed socket, closing socket
connection and attempting reconnect
> [junit4:junit4]   2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket connection
for client /127.0.0.1:58674 which had sessionid 0x13aa99d68d00003
> [junit4:junit4]   2> 690757 T284 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory
exited run method
> [junit4:junit4]   2> 690757 T317 oaz.ClientCnxn$SendThread.run Unable to read additional
data from server sessionid 0x13aa99d68d00003, likely server has closed socket, closing socket
connection and attempting reconnect
> [junit4:junit4]   2> 690757 T282 oazs.FinalRequestProcessor.shutdown shutdown of request
processor complete
> [junit4:junit4]   2> 690757 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 58639
> [junit4:junit4]   2> 690757 T282 oasc.CoreContainer.shutdown Shutting down CoreContainer
instance=25232942
> [junit4:junit4]   2> 690757 T282 oasc.SolrCore.close [collection1]  CLOSING SolrCore
org.apache.solr.core.SolrCore@7e4f51
> [junit4:junit4]   2> 690757 T282 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
> [junit4:junit4]   2> 690757 T282 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
> [junit4:junit4]   2> 690757 T282 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState
ref count has reached 0 - closing IndexWriter
> [junit4:junit4]   2> 690757 T282 oasu.DefaultSolrCoreState.closeIndexWriter closing
IndexWriter with IndexWriterCloser
> [junit4:junit4]   2> 690757 T282 oasc.SolrCore.closeSearcher [collection1] Closing
main searcher on request.
> [junit4:junit4]   2> 690757 T305 oasc.Overseer$ClusterStateUpdater.amILeader According
to ZK I (id=88569636220567554-127.0.0.1:58639_solr-n_0000000000) am no longer a leader.
> [junit4:junit4]   2> 690866 T318 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4985d9
name:ZooKeeperConnection Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
> [junit4:junit4]   2> 690866 T344 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13c22b0
name:ZooKeeperConnection Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
> [junit4:junit4]   2> 690866 T356 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b66ab4
name:ZooKeeperConnection Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
> [junit4:junit4]   2> 690866 T318 oascc.ConnectionManager.process zkClient has disconnected
> [junit4:junit4]   2> 690866 T344 oascc.ConnectionManager.process zkClient has disconnected
> [junit4:junit4]   2> 690866 T356 oascc.ConnectionManager.process zkClient has disconnected
> [junit4:junit4]   2> 690866 T332 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b57ca1
name:ZooKeeperConnection Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
> [junit4:junit4]   2> 690866 T332 oascc.ConnectionManager.process zkClient has disconnected
> [junit4:junit4]   2> 690866 T304 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14a65f8
name:ZooKeeperConnection Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
> [junit4:junit4]   2> 690866 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00002 closed
> [junit4:junit4]   2> 690866 T304 oascc.ConnectionManager.process Client->ZooKeeper
status change trigger but we are already closed
> [junit4:junit4]   2> 690866 T304 oaz.ClientCnxn$EventThread.run EventThread shut down
> [junit4:junit4]   2> 690867 T282 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
> [junit4:junit4]   2> 690918 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 58669
> [junit4:junit4]   2> 690918 T282 oasc.CoreContainer.shutdown Shutting down CoreContainer
instance=2908606
> [junit4:junit4]   2> 690918 T282 oasc.SolrCore.close [collection1]  CLOSING SolrCore
org.apache.solr.core.SolrCore@d75f7a
> [junit4:junit4]   2> 690922 T282 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
> [junit4:junit4]   2> 690922 T282 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
> [junit4:junit4]   2> 690922 T282 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState
ref count has reached 0 - closing IndexWriter
> [junit4:junit4]   2> 690923 T282 oasu.DefaultSolrCoreState.closeIndexWriter closing
IndexWriter with IndexWriterCloser
> [junit4:junit4]   2> 690923 T282 oasc.SolrCore.closeSearcher [collection1] Closing
main searcher on request.
> [junit4:junit4]   2> 692065 T343 oaz.ClientCnxn$SendThread.startConnect Opening socket
connection to server 127.0.0.1/127.0.0.1:58626
> [junit4:junit4]   2> 692236 T317 oaz.ClientCnxn$SendThread.startConnect Opening socket
connection to server 127.0.0.1/127.0.0.1:58626
> [junit4:junit4]   2> 692580 T331 oaz.ClientCnxn$SendThread.startConnect Opening socket
connection to server 127.0.0.1/127.0.0.1:58626
> [junit4:junit4]   2> 692845 T355 oaz.ClientCnxn$SendThread.startConnect Opening socket
connection to server 127.0.0.1/127.0.0.1:58626
> [junit4:junit4]   2> 693063 T343 oaz.ClientCnxn$SendThread.run WARNING Session 0x13aa99d68d00005
for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException:
Connection refused: no further information
> [junit4:junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> [junit4:junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> [junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> [junit4:junit4]   2>
> [junit4:junit4]   2> 693314 T318 oaz.ClientCnxn$EventThread.run EventThread shut down
> [junit4:junit4]   2> 693314 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00003 closed
> [junit4:junit4]   2> 693325 T282 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
> [junit4:junit4]   2> 693376 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 58703
> [junit4:junit4]   2> 693376 T282 oasc.CoreContainer.shutdown Shutting down CoreContainer
instance=15813854
> [junit4:junit4]   2> 693376 T282 oasc.SolrCore.close [collection1]  CLOSING SolrCore
org.apache.solr.core.SolrCore@6b55c1
> [junit4:junit4]   2> 693380 T282 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
> [junit4:junit4]   2> 693380 T282 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
> [junit4:junit4]   2> 693381 T282 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState
ref count has reached 0 - closing IndexWriter
> [junit4:junit4]   2> 693381 T282 oasu.DefaultSolrCoreState.closeIndexWriter closing
IndexWriter with IndexWriterCloser
> [junit4:junit4]   2> 693383 T282 oasc.SolrCore.closeSearcher [collection1] Closing
main searcher on request.
> [junit4:junit4]   2> 693634 T332 oaz.ClientCnxn$EventThread.run EventThread shut down
> [junit4:junit4]   2> 693634 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00004 closed
> [junit4:junit4]   2> 693644 T282 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
> [junit4:junit4]   2> 693695 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 58734
> [junit4:junit4]   2> 693695 T282 oasc.CoreContainer.shutdown Shutting down CoreContainer
instance=18062375
> [junit4:junit4]   2> 693814 T355 oaz.ClientCnxn$SendThread.run WARNING Session 0x13aa99d68d00006
for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException:
Connection refused: no further information
> [junit4:junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> [junit4:junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> [junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> [junit4:junit4]   2>
> [junit4:junit4]   2> 695169 T355 oaz.ClientCnxn$SendThread.startConnect Opening socket
connection to server 127.0.0.1/127.0.0.1:58626
> [junit4:junit4]   2> 695169 T343 oaz.ClientCnxn$SendThread.startConnect Opening socket
connection to server 127.0.0.1/127.0.0.1:58626
> [junit4:junit4]   2> 696183 T355 oaz.ClientCnxn$SendThread.run WARNING Session 0x13aa99d68d00006
for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException:
Connection refused: no further information
> [junit4:junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> [junit4:junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> [junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> [junit4:junit4]   2>
> [junit4:junit4]   2> 696292 T344 oaz.ClientCnxn$EventThread.run EventThread shut down
> [junit4:junit4]   2> 696292 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00005 closed
> [junit4:junit4]   2> 696293 T282 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
> [junit4:junit4]   2> 696344 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 59235
> [junit4:junit4]   2> 696344 T282 oasc.CoreContainer.shutdown Shutting down CoreContainer
instance=33028132
> [junit4:junit4]   2> 696344 T282 oasc.SolrCore.close [collection1]  CLOSING SolrCore
org.apache.solr.core.SolrCore@14c2b2f
> [junit4:junit4]   2> 696353 T282 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
> [junit4:junit4]   2> 696354 T282 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
> [junit4:junit4]   2> 696354 T282 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState
ref count has reached 0 - closing IndexWriter
> [junit4:junit4]   2> 696354 T282 oasu.DefaultSolrCoreState.closeIndexWriter closing
IndexWriter with IndexWriterCloser
> [junit4:junit4]   2> 696356 T282 oasc.SolrCore.closeSearcher [collection1] Closing
main searcher on request.
> [junit4:junit4]   2> 698109 T355 oaz.ClientCnxn$SendThread.startConnect Opening socket
connection to server 127.0.0.1/127.0.0.1:58626
> [junit4:junit4]   2> 699232 T356 oaz.ClientCnxn$EventThread.run EventThread shut down
> [junit4:junit4]   2> 699232 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00006 closed
> [junit4:junit4]   2> 699248 T282 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
> [junit4:junit4]   2> 699501 T282 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
> [junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=FullSolrCloudDistribCmdsTest
-Dtests.method=testDistribSearch -Dtests.seed=F1B077980FD36247 -Dtests.slow=true -Dtests.locale=bg
-Dtests.timezone=Indian/Mahe -Dtests.file.encoding=ISO-8859-1
> [junit4:junit4] FAILURE  700s J4 | FullSolrCloudDistribCmdsTest.testDistribSearch <<<
> [junit4:junit4]    > Throwable #1: java.lang.AssertionError: There are still nodes
recoverying - waited for 600 seconds
> [junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([F1B077980FD36247:7056F980788C027B]:0)
> [junit4:junit4]    > 	at org.junit.Assert.fail(Assert.java:93)
> [junit4:junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:156)
> [junit4:junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:115)
> [junit4:junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:110)
> [junit4:junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:553)
> [junit4:junit4]    > 	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:65)
> [junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:695)
> [junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> [junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> [junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:597)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> [junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> [junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:662)
> [junit4:junit4]   2> 699517 T282 oas.SolrTestCaseJ4.deleteCore ###deleteCore
> [junit4:junit4]   2> 822211 T282 oas.SolrTestCaseJ4.endTrackingSearchers SEVERE ERROR:
SolrIndexSearcher opens=7 closes=6
> [junit4:junit4]   2> 822211 T281 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will
linger awaiting termination of 1 leaked thread(s).
> [junit4:junit4]   2> 842366 T281 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread
leaked from SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest:
> [junit4:junit4]   2> 	   1) Thread[id=345, name=searcherExecutor-163-thread-1, state=WAITING,
group=TGRP-FullSolrCloudDistribCmdsTest]
> [junit4:junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
> [junit4:junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> [junit4:junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> [junit4:junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> [junit4:junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> [junit4:junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> [junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:662)
> [junit4:junit4]   2> 842366 T281 ccr.ThreadLeakControl.tryToInterruptAll Starting
to interrupt leaked threads:
> [junit4:junit4]   2> 	   1) Thread[id=345, name=searcherExecutor-163-thread-1, state=WAITING,
group=TGRP-FullSolrCloudDistribCmdsTest]
> [junit4:junit4]   2> 845455 T281 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There
are still zombie threads that couldn't be terminated:
> [junit4:junit4]   2> 	   1) Thread[id=345, name=searcherExecutor-163-thread-1, state=WAITING,
group=TGRP-FullSolrCloudDistribCmdsTest]
> [junit4:junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
> [junit4:junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> [junit4:junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> [junit4:junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> [junit4:junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> [junit4:junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> [junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:662)
> [junit4:junit4]   2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=true,coord=no):
{}, locale=bg, timezone=Indian/Mahe
> [junit4:junit4]   2> NOTE: Windows Vista 6.0 x86/Sun Microsystems Inc. 1.6.0_23 (32-bit)/cpus=4,threads=2,free=6564744,total=32903168
> [junit4:junit4]   2> NOTE: All tests run in this JVM: [XsltUpdateRequestHandlerTest,
TestDocSet, TermVectorComponentTest, SolrCmdDistributorTest, TestPseudoReturnFields, URLClassifyProcessorTest,
NoCacheHeaderTest, TestSystemIdResolver, FileBasedSpellCheckerTest, TestIndexSearcher, TestStressReorder,
FastVectorHighlighterTest, TestLMJelinekMercerSimilarityFactory, StatsComponentTest, ScriptEngineTest,
LukeRequestHandlerTest, FullSolrCloudDistribCmdsTest]
> [junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=FullSolrCloudDistribCmdsTest
-Dtests.seed=F1B077980FD36247 -Dtests.slow=true -Dtests.locale=bg -Dtests.timezone=Indian/Mahe
-Dtests.file.encoding=ISO-8859-1
> [junit4:junit4] ERROR   0.00s J4 | FullSolrCloudDistribCmdsTest (suite) <<<
> [junit4:junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrIndexSearcher
opens=7 closes=6
> [junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([F1B077980FD36247]:0)
> [junit4:junit4]    > 	at org.junit.Assert.fail(Assert.java:93)
> [junit4:junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:242)
> [junit4:junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:97)
> [junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> [junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> [junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:597)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> [junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:662)
> [junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError:
1 thread leaked from SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest:
> [junit4:junit4]    >    1) Thread[id=345, name=searcherExecutor-163-thread-1, state=WAITING,
group=TGRP-FullSolrCloudDistribCmdsTest]
> [junit4:junit4]    >         at sun.misc.Unsafe.park(Native Method)
> [junit4:junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> [junit4:junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> [junit4:junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> [junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> [junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> [junit4:junit4]    >         at java.lang.Thread.run(Thread.java:662)
> [junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([F1B077980FD36247]:0)
> [junit4:junit4]    > Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError:
There are still zombie threads that couldn't be terminated:
> [junit4:junit4]    >    1) Thread[id=345, name=searcherExecutor-163-thread-1, state=WAITING,
group=TGRP-FullSolrCloudDistribCmdsTest]
> [junit4:junit4]    >         at sun.misc.Unsafe.park(Native Method)
> [junit4:junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> [junit4:junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> [junit4:junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> [junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> [junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> [junit4:junit4]    >         at java.lang.Thread.run(Thread.java:662)
> [junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([F1B077980FD36247]:0)
> [junit4:junit4] Completed on J4 in 845.50s, 1 test, 2 failures, 2 errors <<<
FAILURES!
> [junit4:junit4]
> [junit4:junit4]
> [junit4:junit4] Tests with failures:
> [junit4:junit4]   - org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch
> [junit4:junit4]   - org.apache.solr.cloud.FullSolrCloudDistribCmdsTest (suite)
> [junit4:junit4]
> [junit4:junit4]
> [junit4:junit4] JVM J0:     2.62 ..   273.28 =   270.66s
> [junit4:junit4] JVM J1:     2.62 ..   269.64 =   267.02s
> [junit4:junit4] JVM J2:     2.87 ..   341.30 =   338.43s
> [junit4:junit4] JVM J3:     2.62 ..   270.27 =   267.65s
> [junit4:junit4] JVM J4:     3.40 ..   974.88 =   971.47s
> [junit4:junit4] JVM J5:     2.62 ..   280.39 =   277.77s
> [junit4:junit4] JVM J6:     2.62 ..   275.62 =   273.00s
> [junit4:junit4] JVM J7:     2.62 ..   270.00 =   267.38s
> [junit4:junit4] Execution time total: 16 minutes 14 seconds
> [junit4:junit4] Tests summary: 233 suites, 965 tests, 3 suite-level errors, 1 failure,
13 ignored (8 assumptions)
>
> BUILD FAILED
> <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/build.xml>:39:
The following error occurred while executing this line:
> <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/solr/build.xml>:178:
The following error occurred while executing this line:
> <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/lucene/module-build.xml>:61:
The following error occurred while executing this line:
> <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/lucene/common-build.xml>:1159:
The following error occurred while executing this line:
> <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/lucene/common-build.xml>:827:
There were test failures: 233 suites, 965 tests, 3 suite-level errors, 1 failure, 13 ignored
(8 assumptions)
>
> Total time: 24 minutes 34 seconds
> Build step 'Invoke Ant' marked build as failure
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message