lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_45) - Build # 8438 - Still Failing!
Date Sun, 24 Nov 2013 00:20:26 GMT
I had hoped SOLR-5496 would help with this, but I don’t think it’s going to.

- Mark

On Nov 23, 2013, at 7:17 PM, Policeman Jenkins Server <jenkins@thetaphi.de> wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8438/
> Java: 32bit/jdk1.7.0_45 -client -XX:+UseParallelGC
> 
> 2 tests failed.
> FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
> 
> Error Message:
> 9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:     1) Thread[id=1039, name=qtp22943325-1039 Acceptor1 SelectChannelConnector@127.0.0.1:52401, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:225)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=1038, name=qtp22943325-1038 Acceptor0 SelectChannelConnector@127.0.0.1:52401, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=1042, name=qtp22943325-1042, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=1036, name=qtp22943325-1036 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)         at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=1037, name=qtp22943325-1037 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)         at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=1043, name=qtp22943325-1043, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=1035, name=HashSessionScavenger-38, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Object.wait(Native Method)         at java.util.TimerThread.mainLoop(Timer.java:552)         at java.util.TimerThread.run(Timer.java:505)    8) Thread[id=1041, name=qtp22943325-1041, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=1040, name=qtp22943325-1040, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)         at java.lang.Thread.run(Thread.java:744)
> 
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: 9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest: 
>   1) Thread[id=1039, name=qtp22943325-1039 Acceptor1 SelectChannelConnector@127.0.0.1:52401, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:225)
>        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>        at java.lang.Thread.run(Thread.java:744)
>   2) Thread[id=1038, name=qtp22943325-1038 Acceptor0 SelectChannelConnector@127.0.0.1:52401, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
>        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>        at java.lang.Thread.run(Thread.java:744)
>   3) Thread[id=1042, name=qtp22943325-1042, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at sun.misc.Unsafe.park(Native Method)
>        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>        at java.lang.Thread.run(Thread.java:744)
>   4) Thread[id=1036, name=qtp22943325-1036 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>        at java.lang.Thread.run(Thread.java:744)
>   5) Thread[id=1037, name=qtp22943325-1037 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>        at java.lang.Thread.run(Thread.java:744)
>   6) Thread[id=1043, name=qtp22943325-1043, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at sun.misc.Unsafe.park(Native Method)
>        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>        at java.lang.Thread.run(Thread.java:744)
>   7) Thread[id=1035, name=HashSessionScavenger-38, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at java.lang.Object.wait(Native Method)
>        at java.util.TimerThread.mainLoop(Timer.java:552)
>        at java.util.TimerThread.run(Timer.java:505)
>   8) Thread[id=1041, name=qtp22943325-1041, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at sun.misc.Unsafe.park(Native Method)
>        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>        at java.lang.Thread.run(Thread.java:744)
>   9) Thread[id=1040, name=qtp22943325-1040, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at sun.misc.Unsafe.park(Native Method)
>        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>        at java.lang.Thread.run(Thread.java:744)
> 	at __randomizedtesting.SeedInfo.seed([67AA78603D7545AD]:0)
> 
> 
> FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
> 
> Error Message:
> There are still zombie threads that couldn't be terminated:    1) Thread[id=1039, name=qtp22943325-1039 Acceptor1 SelectChannelConnector@127.0.0.1:52401, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at org.eclipse.jetty.server.nio.SelectChannelConnector.getConnection(SelectChannelConnector.java:160)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:934)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=1038, name=qtp22943325-1038 Acceptor0 SelectChannelConnector@127.0.0.1:52401, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=1036, name=qtp22943325-1036 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)         at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=1037, name=qtp22943325-1037 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)         at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=1035, name=HashSessionScavenger-38, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Object.wait(Native Method)         at java.util.TimerThread.mainLoop(Timer.java:552)         at java.util.TimerThread.run(Timer.java:505)
> 
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
>   1) Thread[id=1039, name=qtp22943325-1039 Acceptor1 SelectChannelConnector@127.0.0.1:52401, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at org.eclipse.jetty.server.nio.SelectChannelConnector.getConnection(SelectChannelConnector.java:160)
>        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:934)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>        at java.lang.Thread.run(Thread.java:744)
>   2) Thread[id=1038, name=qtp22943325-1038 Acceptor0 SelectChannelConnector@127.0.0.1:52401, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
>        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>        at java.lang.Thread.run(Thread.java:744)
>   3) Thread[id=1036, name=qtp22943325-1036 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>        at java.lang.Thread.run(Thread.java:744)
>   4) Thread[id=1037, name=qtp22943325-1037 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>        at java.lang.Thread.run(Thread.java:744)
>   5) Thread[id=1035, name=HashSessionScavenger-38, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>        at java.lang.Object.wait(Native Method)
>        at java.util.TimerThread.mainLoop(Timer.java:552)
>        at java.util.TimerThread.run(Timer.java:505)
> 	at __randomizedtesting.SeedInfo.seed([67AA78603D7545AD]:0)
> 
> 
> 
> 
> Build Log:
> [...truncated 10280 lines...]
>   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
>   [junit4]   2> 255837 T857 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_quc/x
>   [junit4]   2> 255841 T857 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
>   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyNothingIsSafeTest-1385251930572
>   [junit4]   2> 255841 T857 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>   [junit4]   2> 255842 T858 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>   [junit4]   2> 255942 T857 oasc.ZkTestServer.run start zk server on port:40409
>   [junit4]   2> 255943 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 256209 T864 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d9e086 name:ZooKeeperConnection Watcher:127.0.0.1:40409 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 256210 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 256210 T857 oascc.SolrZkClient.makePath makePath: /solr
>   [junit4]   2> 256214 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 256215 T866 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14edc3 name:ZooKeeperConnection Watcher:127.0.0.1:40409/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 256216 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 256216 T857 oascc.SolrZkClient.makePath makePath: /collections/collection1
>   [junit4]   2> 256218 T857 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
>   [junit4]   2> 256220 T857 oascc.SolrZkClient.makePath makePath: /collections/control_collection
>   [junit4]   2> 256221 T857 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
>   [junit4]   2> 256223 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
>   [junit4]   2> 256223 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
>   [junit4]   2> 256226 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
>   [junit4]   2> 256227 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
>   [junit4]   2> 256230 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 256231 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 256233 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
>   [junit4]   2> 256234 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
>   [junit4]   2> 256236 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
>   [junit4]   2> 256236 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
>   [junit4]   2> 256238 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
>   [junit4]   2> 256239 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
>   [junit4]   2> 256240 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
>   [junit4]   2> 256241 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
>   [junit4]   2> 256243 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
>   [junit4]   2> 256243 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
>   [junit4]   2> 256245 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 256246 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 256248 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
>   [junit4]   2> 256248 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
>   [junit4]   2> 256250 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
>   [junit4]   2> 256250 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
>   [junit4]   2> 256324 T857 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 256327 T857 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38893
>   [junit4]   2> 256328 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 256328 T857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 256329 T857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385251930984
>   [junit4]   2> 256329 T857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385251930984/'
>   [junit4]   2> 256344 T857 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385251930984/solr.xml
>   [junit4]   2> 256396 T857 oasc.CoreContainer.<init> New CoreContainer 13270187
>   [junit4]   2> 256397 T857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385251930984/]
>   [junit4]   2> 256398 T857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 256399 T857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 256399 T857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 256399 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 256399 T857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 256400 T857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 256400 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 256400 T857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 256401 T857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 256402 T857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 256402 T857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 256403 T857 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 256403 T857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40409/solr
>   [junit4]   2> 256403 T857 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 256404 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 256406 T877 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@192aa08 name:ZooKeeperConnection Watcher:127.0.0.1:40409 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 256406 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 256409 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 256410 T879 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@113c43b name:ZooKeeperConnection Watcher:127.0.0.1:40409/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 256410 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 256412 T857 oascc.SolrZkClient.makePath makePath: /overseer/queue
>   [junit4]   2> 256414 T857 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
>   [junit4]   2> 256416 T857 oascc.SolrZkClient.makePath makePath: /live_nodes
>   [junit4]   2> 256420 T857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38893__quc%2Fx
>   [junit4]   2> 256423 T857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38893__quc%2Fx
>   [junit4]   2> 256425 T857 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
>   [junit4]   2> 256429 T857 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
>   [junit4]   2> 256430 T857 oasc.Overseer.start Overseer (id=90783870522163203-127.0.0.1:38893__quc%2Fx-n_0000000000) starting
>   [junit4]   2> 256432 T857 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
>   [junit4]   2> 256442 T881 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
>   [junit4]   2> 256442 T857 oascc.SolrZkClient.makePath makePath: /clusterstate.json
>   [junit4]   2> 256445 T857 oascc.SolrZkClient.makePath makePath: /aliases.json
>   [junit4]   2> 256446 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 256448 T880 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
>   [junit4]   2> 256451 T882 oasc.ZkController.publish publishing core=collection1 state=down
>   [junit4]   2> 256451 T882 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 256452 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 256452 T882 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 256454 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 256454 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:38893/_quc/x",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:38893__quc%2Fx",
>   [junit4]   2> 	  "shard":null,
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"control_collection",
>   [junit4]   2> 	  "numShards":"1",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 256454 T880 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
>   [junit4]   2> 256455 T880 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
>   [junit4]   2> 256457 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 256457 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 257453 T882 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>   [junit4]   2> 257453 T882 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385251930984/collection1
>   [junit4]   2> 257454 T882 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
>   [junit4]   2> 257454 T882 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 257455 T882 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
>   [junit4]   2> 257456 T882 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385251930984/collection1/'
>   [junit4]   2> 257457 T882 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385251930984/collection1/lib/classes/' to classloader
>   [junit4]   2> 257457 T882 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385251930984/collection1/lib/README' to classloader
>   [junit4]   2> 257501 T882 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
>   [junit4]   2> 257541 T882 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 257543 T882 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 257551 T882 oass.IndexSchema.readSchema [collection1] Schema name=test
>   [junit4]   2> 257956 T882 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 257962 T882 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 257965 T882 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 257975 T882 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 257978 T882 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 257982 T882 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 257983 T882 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 257983 T882 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 257983 T882 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 257984 T882 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 257984 T882 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 257984 T882 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
>   [junit4]   2> 257992 T882 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385251930984/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/control/data/
>   [junit4]   2> 257992 T882 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@18f0d13
>   [junit4]   2> 257992 T882 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/control/data
>   [junit4]   2> 257993 T882 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/control/data/index/
>   [junit4]   2> 257993 T882 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/control/data/index' doesn't exist. Creating new index...
>   [junit4]   2> 257993 T882 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/control/data/index
>   [junit4]   2> 257994 T882 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=76.4794921875, floorSegmentMB=1.3642578125, forceMergeDeletesPctAllowed=19.690572121010746, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.347545062527452
>   [junit4]   2> 258003 T882 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/control/data/index,segFN=segments_1,generation=1}
>   [junit4]   2> 258004 T882 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 258006 T882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 258007 T882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 258007 T882 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 258007 T882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 258007 T882 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 258007 T882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 258008 T882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 258008 T882 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 258008 T882 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 258009 T882 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 258009 T882 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 258009 T882 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 258010 T882 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 258010 T882 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 258010 T882 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 258011 T882 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 258013 T882 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 258016 T882 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 258016 T882 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 258017 T882 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=13, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3773653177043339]
>   [junit4]   2> 258017 T882 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/control/data/index,segFN=segments_1,generation=1}
>   [junit4]   2> 258018 T882 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 258018 T882 oass.SolrIndexSearcher.<init> Opening Searcher@1110348 main
>   [junit4]   2> 258021 T883 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1110348 main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 258022 T882 oasc.CoreContainer.registerCore registering core: collection1
>   [junit4]   2> 258023 T882 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38893/_quc/x collection:control_collection shard:shard1
>   [junit4]   2> 258024 T882 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
>   [junit4]   2> 258029 T882 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
>   [junit4]   2> 258031 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 258032 T882 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
>   [junit4]   2> 258032 T882 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>   [junit4]   2> ASYNC  NEW_CORE C530 name=collection1 org.apache.solr.core.SolrCore@1e52c93 url=http://127.0.0.1:38893/_quc/x/collection1 node=127.0.0.1:38893__quc%2Fx C530_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:38893/_quc/x, core=collection1, node_name=127.0.0.1:38893__quc%2Fx}
>   [junit4]   2> 258032 T882 C530 P38893 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38893/_quc/x/collection1/
>   [junit4]   2> 258032 T882 C530 P38893 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
>   [junit4]   2> 258032 T882 C530 P38893 oasc.SyncStrategy.syncToMe http://127.0.0.1:38893/_quc/x/collection1/ has no replicas
>   [junit4]   2> 258032 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 258033 T882 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38893/_quc/x/collection1/ shard1
>   [junit4]   2> 258033 T882 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
>   [junit4]   2> 258035 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 258036 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 258040 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 258043 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 258146 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 258190 T882 oasc.ZkController.register We are http://127.0.0.1:38893/_quc/x/collection1/ and leader is http://127.0.0.1:38893/_quc/x/collection1/
>   [junit4]   2> 258190 T882 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38893/_quc/x
>   [junit4]   2> 258190 T882 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
>   [junit4]   2> 258191 T882 oasc.ZkController.publish publishing core=collection1 state=active
>   [junit4]   2> 258191 T882 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 258192 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 258192 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 258192 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 258193 T882 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 258194 T857 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
>   [junit4]   2> 258194 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 258194 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 258195 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"active",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:38893/_quc/x",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:38893__quc%2Fx",
>   [junit4]   2> 	  "shard":"shard1",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"control_collection",
>   [junit4]   2> 	  "numShards":"1",
>   [junit4]   2> 	  "core_node_name":"core_node1"}
>   [junit4]   2> 258197 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 258198 T886 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@87f292 name:ZooKeeperConnection Watcher:127.0.0.1:40409/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 258198 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 258198 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 258199 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 258201 T857 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
>   [junit4]   2> 258266 T857 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 258268 T857 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34995
>   [junit4]   2> 258269 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 258269 T857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 258270 T857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385251932932
>   [junit4]   2> 258270 T857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385251932932/'
>   [junit4]   2> 258282 T857 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385251932932/solr.xml
>   [junit4]   2> 258301 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 258301 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 258328 T857 oasc.CoreContainer.<init> New CoreContainer 28366621
>   [junit4]   2> 258329 T857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385251932932/]
>   [junit4]   2> 258330 T857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 258331 T857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 258331 T857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 258331 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 258331 T857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 258332 T857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 258332 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 258332 T857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 258333 T857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 258334 T857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 258334 T857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 258335 T857 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 258335 T857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40409/solr
>   [junit4]   2> 258335 T857 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 258336 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 258338 T897 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21d179 name:ZooKeeperConnection Watcher:127.0.0.1:40409 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 258338 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 258341 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 258342 T899 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ff824 name:ZooKeeperConnection Watcher:127.0.0.1:40409/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 258343 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 258346 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 259349 T857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34995__quc%2Fx
>   [junit4]   2> 259350 T857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34995__quc%2Fx
>   [junit4]   2> 259352 T879 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 259352 T886 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 259353 T899 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 259359 T900 oasc.ZkController.publish publishing core=collection1 state=down
>   [junit4]   2> 259359 T900 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 259450 T900 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 259450 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 259451 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 259451 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 259452 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 259453 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:34995/_quc/x",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:34995__quc%2Fx",
>   [junit4]   2> 	  "shard":null,
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"collection1",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 259453 T880 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
>   [junit4]   2> 259453 T880 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
>   [junit4]   2> 259458 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 259560 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 259560 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 259560 T899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 260451 T900 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>   [junit4]   2> 260451 T900 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385251932932/collection1
>   [junit4]   2> 260451 T900 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
>   [junit4]   2> 260453 T900 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 260453 T900 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
>   [junit4]   2> 260454 T900 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385251932932/collection1/'
>   [junit4]   2> 260455 T900 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385251932932/collection1/lib/classes/' to classloader
>   [junit4]   2> 260455 T900 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385251932932/collection1/lib/README' to classloader
>   [junit4]   2> 260497 T900 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
>   [junit4]   2> 260535 T900 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 260537 T900 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 260545 T900 oass.IndexSchema.readSchema [collection1] Schema name=test
>   [junit4]   2> 260925 T900 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 260931 T900 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 260933 T900 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 260952 T900 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 260957 T900 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 260960 T900 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 260962 T900 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 260962 T900 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 260963 T900 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 260965 T900 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 260965 T900 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 260965 T900 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
>   [junit4]   2> 260966 T900 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385251932932/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty1/
>   [junit4]   2> 260966 T900 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@18f0d13
>   [junit4]   2> 260967 T900 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty1
>   [junit4]   2> 260967 T900 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty1/index/
>   [junit4]   2> 260968 T900 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty1/index' doesn't exist. Creating new index...
>   [junit4]   2> 260968 T900 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty1/index
>   [junit4]   2> 260969 T900 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=76.4794921875, floorSegmentMB=1.3642578125, forceMergeDeletesPctAllowed=19.690572121010746, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.347545062527452
>   [junit4]   2> 261024 T900 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty1/index,segFN=segments_1,generation=1}
>   [junit4]   2> 261024 T900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 261030 T900 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 261030 T900 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 261031 T900 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 261031 T900 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 261031 T900 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 261031 T900 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 261032 T900 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 261032 T900 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 261032 T900 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 261033 T900 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 261033 T900 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 261034 T900 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 261034 T900 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 261034 T900 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 261035 T900 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 261035 T900 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 261038 T900 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 261040 T900 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 261040 T900 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 261041 T900 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=13, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3773653177043339]
>   [junit4]   2> 261042 T900 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty1/index,segFN=segments_1,generation=1}
>   [junit4]   2> 261042 T900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 261043 T900 oass.SolrIndexSearcher.<init> Opening Searcher@b13f67 main
>   [junit4]   2> 261046 T901 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b13f67 main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 261049 T900 oasc.CoreContainer.registerCore registering core: collection1
>   [junit4]   2> 261049 T900 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:34995/_quc/x collection:collection1 shard:shard1
>   [junit4]   2> 261050 T900 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
>   [junit4]   2> 261056 T900 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
>   [junit4]   2> 261057 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 261058 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 261058 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 261058 T900 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
>   [junit4]   2> 261058 T900 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>   [junit4]   2> ASYNC  NEW_CORE C531 name=collection1 org.apache.solr.core.SolrCore@1b014f6 url=http://127.0.0.1:34995/_quc/x/collection1 node=127.0.0.1:34995__quc%2Fx C531_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:34995/_quc/x, core=collection1, node_name=127.0.0.1:34995__quc%2Fx}
>   [junit4]   2> 261059 T900 C531 P34995 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:34995/_quc/x/collection1/
>   [junit4]   2> 261059 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 261059 T900 C531 P34995 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
>   [junit4]   2> 261060 T900 C531 P34995 oasc.SyncStrategy.syncToMe http://127.0.0.1:34995/_quc/x/collection1/ has no replicas
>   [junit4]   2> 261060 T900 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:34995/_quc/x/collection1/ shard1
>   [junit4]   2> 261060 T900 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
>   [junit4]   2> 261062 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 261063 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 261063 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 261063 T899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 261068 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 261070 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 261174 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 261174 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 261174 T899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 261217 T900 oasc.ZkController.register We are http://127.0.0.1:34995/_quc/x/collection1/ and leader is http://127.0.0.1:34995/_quc/x/collection1/
>   [junit4]   2> 261218 T900 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:34995/_quc/x
>   [junit4]   2> 261218 T900 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
>   [junit4]   2> 261218 T900 oasc.ZkController.publish publishing core=collection1 state=active
>   [junit4]   2> 261218 T900 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 261219 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 261220 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 261220 T900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 261220 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 261221 T857 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
>   [junit4]   2> 261221 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 261222 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 261222 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"active",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:34995/_quc/x",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:34995__quc%2Fx",
>   [junit4]   2> 	  "shard":"shard1",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"collection1",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "core_node_name":"core_node1"}
>   [junit4]   2> 261224 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 261291 T857 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 261293 T857 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52401
>   [junit4]   2> 261293 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 261294 T857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 261294 T857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385251935953
>   [junit4]   2> 261294 T857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385251935953/'
>   [junit4]   2> 261306 T857 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385251935953/solr.xml
>   [junit4]   2> 261349 T899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 261349 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 261349 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 261351 T857 oasc.CoreContainer.<init> New CoreContainer 11817994
>   [junit4]   2> 261351 T857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385251935953/]
>   [junit4]   2> 261353 T857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 261353 T857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 261354 T857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 261354 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 261354 T857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 261355 T857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 261355 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 261355 T857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 261355 T857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 261356 T857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 261357 T857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 261357 T857 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 261358 T857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40409/solr
>   [junit4]   2> 261359 T857 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 261359 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 261361 T913 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c09f29 name:ZooKeeperConnection Watcher:127.0.0.1:40409 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 261361 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 261363 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 261364 T915 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@916399 name:ZooKeeperConnection Watcher:127.0.0.1:40409/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 261365 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 261367 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 262370 T857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52401__quc%2Fx
>   [junit4]   2> 262372 T857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52401__quc%2Fx
>   [junit4]   2> 262376 T879 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 262376 T899 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 262376 T915 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 262376 T886 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 262384 T916 oasc.ZkController.publish publishing core=collection1 state=down
>   [junit4]   2> 262384 T916 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 262385 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 262385 T916 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 262386 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 262386 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 262387 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 262388 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:52401/_quc/x",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:52401__quc%2Fx",
>   [junit4]   2> 	  "shard":null,
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"collection1",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 262388 T880 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
>   [junit4]   2> 262388 T880 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
>   [junit4]   2> 262391 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 262493 T915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 262493 T899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 262493 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 262493 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 263386 T916 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>   [junit4]   2> 263387 T916 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385251935953/collection1
>   [junit4]   2> 263387 T916 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
>   [junit4]   2> 263387 T916 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 263388 T916 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
>   [junit4]   2> 263389 T916 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385251935953/collection1/'
>   [junit4]   2> 263389 T916 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385251935953/collection1/lib/classes/' to classloader
>   [junit4]   2> 263390 T916 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385251935953/collection1/lib/README' to classloader
>   [junit4]   2> 263425 T916 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
>   [junit4]   2> 263463 T916 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 263465 T916 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 263473 T916 oass.IndexSchema.readSchema [collection1] Schema name=test
>   [junit4]   2> 263853 T916 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 263859 T916 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 263862 T916 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 263871 T916 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 263874 T916 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 263877 T916 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 263878 T916 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 263878 T916 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 263878 T916 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 263879 T916 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 263879 T916 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 263879 T916 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
>   [junit4]   2> 263879 T916 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385251935953/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty2/
>   [junit4]   2> 263880 T916 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@18f0d13
>   [junit4]   2> 263880 T916 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty2
>   [junit4]   2> 263880 T916 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty2/index/
>   [junit4]   2> 263881 T916 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty2/index' doesn't exist. Creating new index...
>   [junit4]   2> 263881 T916 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty2/index
>   [junit4]   2> 263881 T916 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=76.4794921875, floorSegmentMB=1.3642578125, forceMergeDeletesPctAllowed=19.690572121010746, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.347545062527452
>   [junit4]   2> 263951 T916 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty2/index,segFN=segments_1,generation=1}
>   [junit4]   2> 263951 T916 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 263954 T916 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 263955 T916 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 263955 T916 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 263955 T916 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 263955 T916 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 263956 T916 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 263956 T916 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 263956 T916 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 263956 T916 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 263957 T916 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 263958 T916 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 263958 T916 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 263958 T916 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 263959 T916 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 263959 T916 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 263959 T916 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 263962 T916 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 263964 T916 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 263964 T916 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 263965 T916 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=13, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3773653177043339]
>   [junit4]   2> 263966 T916 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385251930572/jetty2/index,segFN=segments_1,generation=1}
>   [junit4]   2> 263966 T916 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 263967 T916 oass.SolrIndexSearcher.<init> Opening Searcher@137e869 main
>   [junit4]   2> 263971 T917 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@137e869 main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 263973 T916 oasc.CoreContainer.registerCore registering core: collection1
>   [junit4]   2> 263974 T916 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52401/_quc/x collection:collection1 shard:shard2
>   [junit4]   2> 263975 T916 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
>   [junit4]   2> 263979 T916 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
>   [junit4]   2> 263981 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 263981 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 263981 T916 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
>   [junit4]   2> 263981 T916 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>   [junit4]   2> 263981 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> ASYNC  NEW_CORE C532 name=collection1 org.apache.solr.core.SolrCore@1b2608e url=http://127.0.0.1:52401/_quc/x/collection1 node=127.0.0.1:52401__quc%2Fx C532_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:52401/_quc/x, core=collection1, node_name=127.0.0.1:52401__quc%2Fx}
>   [junit4]   2> 263982 T916 C532 P52401 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52401/_quc/x/collection1/
>   [junit4]   2> 263982 T916 C532 P52401 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
>   [junit4]   2> 263982 T916 C532 P52401 oasc.SyncStrategy.syncToMe http://127.0.0.1:52401/_quc/x/collection1/ has no replicas
>   [junit4]   2> 263982 T916 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52401/_quc/x/collection1/ shard2
>   [junit4]   2> 263983 T916 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
>   [junit4]   2> 263983 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 263985 T879 oasc.DistributedQueue$L
> 
> [...truncated too long message...]
> 
> readPool.runJob(QueuedThreadPool.java:608)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
>   [junit4]    >    5) Thread[id=1037, name=qtp22943325-1037 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>   [junit4]    >         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>   [junit4]    >         at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>   [junit4]    >         at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
>   [junit4]    >    6) Thread[id=1043, name=qtp22943325-1043, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
>   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>   [junit4]    >         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
>   [junit4]    >    7) Thread[id=1035, name=HashSessionScavenger-38, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>   [junit4]    >         at java.lang.Object.wait(Native Method)
>   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:552)
>   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
>   [junit4]    >    8) Thread[id=1041, name=qtp22943325-1041, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
>   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>   [junit4]    >         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
>   [junit4]    >    9) Thread[id=1040, name=qtp22943325-1040, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
>   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>   [junit4]    >         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
>   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([67AA78603D7545AD]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
>   [junit4]    >    1) Thread[id=1039, name=qtp22943325-1039 Acceptor1 SelectChannelConnector@127.0.0.1:52401, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>   [junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.getConnection(SelectChannelConnector.java:160)
>   [junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:934)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
>   [junit4]    >    2) Thread[id=1038, name=qtp22943325-1038 Acceptor0 SelectChannelConnector@127.0.0.1:52401, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>   [junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
>   [junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
>   [junit4]    >    3) Thread[id=1036, name=qtp22943325-1036 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>   [junit4]    >         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>   [junit4]    >         at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>   [junit4]    >         at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
>   [junit4]    >    4) Thread[id=1037, name=qtp22943325-1037 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>   [junit4]    >         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>   [junit4]    >         at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>   [junit4]    >         at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
>   [junit4]    >    5) Thread[id=1035, name=HashSessionScavenger-38, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
>   [junit4]    >         at java.lang.Object.wait(Native Method)
>   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:552)
>   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
>   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([67AA78603D7545AD]:0)
>   [junit4] Completed on J1 in 202.42s, 1 test, 2 errors <<< FAILURES!
> 
> [...truncated 695 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:420: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:400: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:492: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1272: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:908: There were test failures: 353 suites, 1558 tests, 2 suite-level errors, 954 ignored (5 assumptions)
> 
> Total time: 30 minutes 58 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 32bit/jdk1.7.0_45 -client -XX:+UseParallelGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message